Laird Close – Page 2 – Extreme Wavefront Control Lab

2017B Day 15: The End of the 2017B Run, and a Hike to See Petroglyphs

Hi Everyone,

As is tradition I’m writing the last blog. This run has been a bit challenging due to mostly poor weather. For example, there were a few nights so cloudy that only photos we actually saved were of “Hedwig” the owl landing on our all-sky camera…. Hedwig is great but we really came to look at exoplanets! Anyways by the end of the run we did get some good datasets.

The run had a bit of a rough start with the loss of our main glycol (antifreeze) cooling pump.

This is the state of the old pump. The motor and the bearing were bad shape and needed replacement. — This is the state of the old pump. The motor and the bearing were in bad shape and needed replacement.

The ASM puts out ~2KW of heat. All this heat needs to be removed by a liquid cooling system. So we had ASM cooling system in place that pumps about 11 liters/min of glycol through the ASM. But after more than 6 years of use the pump was starting to fail. Luckily LCO mechanical engineers Juan and Carlos realized this and I gave them the spare pump that I purchased 8 years ago (because eventually all pumps die). After a bit of engineering to make sure the pump pressure at the ASM was safe (we don’t want another leak inside the ASM), the happy final result was a new pump that works very well and keeps the ASM nice and cool.

Then on the first night on sky we had some issues with a bad capacitor (position) sensor (DSP 590) that was causing strange readings and leading to diagnostic “dumps” making the system hard to use. But, adding DSP 590 to the act_wo_pos array in the elec.txt file effectively added it to the list of “actuators that cannot read positions well”. Removing this sensor and replacing the pump allowed us to have a problem free run!

So it was great to see MagAO have such a strong run. The ASM, wave front sensor, VisAO camera, and the Clio camera all worked well for the whole run!

The star of MagAO: the Adaptive Secondary Mirror (ASM). Throughout a cold, windy, cloudy run it worked extremely well.

This was something of a relief after our tough run in Feb 2017 where we had 2 electronic cards in the ASM fail and our second fast tip-tilt mirror literally “burned up”. So it seems, for now at east, we are back normal science operations.

As usual the run was a success due to the hard work of Katie running Clio (and teaching the new astronomers how to run it), and Jared keeping all the complex software running excellently. This run we also had help from UA graduate student Jhen Lumbres who was a great help running the AO system (giving me a real break). Also Kate came with an undergraduate Clare Leonard. Kate (and Clare) ran the VisAO camera while she was here giving Jared a break. Also TJ and Alycia are expert Clio observers and helped out by running Clio during their whole runs. Ewan Douglas came from MIT and we also had some new observers too from Chile — everyone was great to work with!

And of course a big thanks to the entire LCO staff who keep all of MagAO running well!

Felix (who has helped us put on and off MagAO many of the 20 times we've done it) and the NAS coming off the telescope and going down the elevator. — Felix (who has helped us put on and off MagAO many of the 20 times we’ve done it) and the NAS coming off the telescope and going down the elevator.

I should mention that due to all the snow and rain this winter it is beautiful here –greener than I have ever seen LCO!

Three Guanocos and some donkeys enjoy the lush green spring we are having — Three guanacos and some donkeys enjoy the lush green spring we are having

Inspired by the lovely green springtime the MagAO team (Jared, Katie and I) headed out for a hike after the instrument was completely packed away. This was, in theory, a hike to see the petroglyphs in the valley carved long ago. But we didn’y really know where to go. And so it was unclear if we actually went down the right valley — but we did find a nice “bell rock” (these “special rocks” that ring like bells) outcropping that had trees (well just two trees but that is a lot here)!

Katie by the first real tree I've seen at LCO. This was near the rocks. — Katie by the first real tree I’ve seen at LCO. This was near the rocks.

And then we looked more closely at the rocks

A petroglyph of something sitting on top of a rock? Looking at the stars?

And we saw a interesting shape carved into the rock. Below is zoom into the carved region.

A zoom into the object sitting at the top of the carving -- a Vizzy? — A zoom into the object sitting at the top of the carving — a Vizzy?

Could this be a petroglyph of the ancient viscacha ?? Only the wise viscachas know for sure…

It is that time, time to go back, back to our “shack” of a lab and dream up new AO systems to build…

2017A Day 21: A Very Successful MagAO Science Run Ends

It has been a great recovery for MagAO from the terrible glycol leak of last year. We have now completed our first science run and it was a big success with a completion of the February targets that we failed to get last year now completed. As readers of this run’s blog posts know it was a tough run with losses of 2 DSP boards for the ASM and the loss of the PI TT mirror. But we fixed what was broken and carried on doing science. In fact, the last 5 days were straightforward with full AO uptime, and prove that MagAO is fully back in the science business.

So after 21 days and nights at LCO Jared, Katie, and I finally left LCO today. But first I got up very early because the most important piece of equipment in all of LCO is the 20 year old ISUZU flatbed truck. It spends its life running around the observatory and driving to La Serena and back. It is the only way to get the ASM from the Clay telescope to the clean room (some 500 years down the road from the telescope) where the ASM is stored when it is not used. The ISUZU had an hour free this morning to bring the ASM down the mountain! So I was excited to get the ASM down from the telescope.

The ASM is Happy to be going down — It is easier to get time on the Clay telescope than the Isuzu truck…

Yesterday we had a super full day of complex crane and mounting and cabling work (see yesterday’s blog post by Katie). Through it all Juan Gallardo has overseen almost every single mount and dismount of Clio, and NAS, and ASM that we have done over the last 6 years! The safety of MagAO is in great hands with Juan. Thanks Juan!

These guys are good — Juan and Nelson secure the 1040 lb ASM to the Isuzu truck at Clay

I should note that the head mechanics Nelson and Felix are the key people that help Juan with all these crazy difficult tasks. Since Nelson and Felix work different tournos (week long shifts) you never see them together (except at the magical Tuesday morning when the shifts change) so here is the photo of these two amazing individuals below. These are the only guys I trust with wrenches and 5/8th bolts over my optics.

These guys at great — A rare photo of Felix and Nelson head mechanics from each shift -together

Now the ASM is all packed away till the next run. It is nice to being back to doing science with the ASM after all the repairs of the last year.

rest easy big guy... — Here is the ASM packed away in the clean room with help from Charlotte.

Also the NAS was packed away by Jared in the AUX building

And Katie put Clio in its special office…

Clio was pretty great this run... — Clio in its personal office in the AUX building

Now Jared and Katie are finally getting some well deserved rest in the LAN VIP lounge (where I’m writing this post).

I remember that photo from the start of this run.... — Finally some rest !

And last but not least in honor of our visit ending and going down to the airport the cooks made a going away gift for all…

also it is a well known blog rule that the last post of the run doesn’t need to include a song if posted by the PI (hence no song)

2017A DAY 15: The Amazing Weather Predicting ASM

Tonight is my first calm night driving the AO system in 5 nights. It is great that everything in MagAO/VisAO/Clio is working well together again. Katie is getting great data and Kevin Wagner has joined us at Clay.

Earthquake! — Clio’s data as we went through an Earthquake tonight. Upper left image AO, all the rest are AO + earthquake — loop stayed locked!

I was having such an easy time running the AO that I was “volunteered” into writing tonight’s blog post.

Last night’s very rare high humidity night (60-80% humidity all night) was so well sensed by our capacitive sensors on the ASM that I was reminded how capacitors really are barometers. Then I got to thinking that the ASM has many unsung talents in weather prediction.

When I was a young lad I was always interested in science and part of that was a love of weather. My grandfather Jack Close who was a bit of contrarian realized this and purchased a “weather predicting donkey” for his home. I was very excited to meet this clever donkey. I was a bit less impressed when I finally got to meet it. We went outside and he pointed to a round piece of wood with donkey painted on it with a little tail made of wool stapled on to the wood for a tail. Below you can see a picture of a similar weather predicting donkey:

Now that I’m all grown up I’m pleased to present my very own weather predicting ASM. And, since long time readers of the blog know how Katie and Jared love rules: here are the rules of how the weather predicting ASM works (this could actually replace sections in our operating manual):

Take Care Of Me, I'm sensitive — The Amazing Weather Predicting ASM

So here are the rules of how the weather predicting ASM should be always used:
ASM state = weather: Action
______________________________________________________________________
ASM dry = Nice Night: keep observing
ASM can’t set = humid night: Close dome and warm up the ASM –it is cold and damp
ASM wet = Rain: stop observing!, and repeat what we did for the last 10 months to repair liquid in the gap
ASM frozen = Cold: Call LBT for advice
ASM’s tip-tilt loop goes crazy but stays locked = small earthquake: remain calm and ride it out (see Clio’s earthquake images above!)
ASM’s tip-tilt loop opens = big earthquake: HIT THE BIG RED EARTHQUAKE BUTTON (then, do as the day crew do, and run out of building)
ASM’s tip-tilt loop good on one side and bad on other = low airmass: blame ADCs for a while, then replace Tip-Tilt mirror
ASM’s Shell blows off: Tornado: OMG I hope MagAO-X gets built soon…

I think Papa Jack Close would approve…

A nice song that I like about fireflies

2017A Day 6: Better than OK

Hi all you MagAO fans, last night we had some excitement (which as long time readers of the blog know — that means something broke). After a fairly routine night we had just settled into a nice rhythm at around 3AM. Then the telescope rotator decided to stop rotating — which wasn’t a big deal, but then the AO database got a bit confused since it thought we were rotating (but we were not) and odd things started to happen. To clear all this we went to zenith (always a good idea to troubleshoot) and then the ASM started acting very odd. It was fairly clear (after a while) that there were 8 sensors that were well out of range and the shell would not SET or even have it program loaded without going to its panic state (yes that is a thing). This was new for us and totally unrelated to the AO problem we were actually trying to fix. So it was a bit disappointing to try and fix one problem and 3 others pop up…

As you can see above there are a group of bad actuators that are red in color — that means they are jumping around and fooling the ASM mirror into thinking it has a piece of dust trapped under those actuators. So the ASM will not run in that condition. This was new for us and we called in ASM expert Marco Xompero from Arcetri and we located that there was a problem likely with the cable or DSP (like a computer) in ASM crate 4 board 9. We had just spent the last 10 months repairing the ASM so Jared and I had a pretty good idea of where DSP board 9 in crate 4 was.

So all we needed to do was wait until the next day and tip the telescope over to 2 degrees above the horizon and position a scissor lift under the ASM and start taking the ASM apart. In the above image you can see Jared doing surgery on the ASM. In the end we found it was not the cable but indeed there was something wrong with the DSP board.

Above you can see our bad DSP board. Once we replaced it with a spare today all was good and the AO was working again… Ya-lin was the AO operator tonight and things went very smoothly.

All and all it was a good fix, and Jared could get back to taking visible images (see below)

can't you just leave me alone... — Viscacha Paparazzi

2016B Day 12: MagAO is back on-sky Closed Loop (400 modes 1kHz)

Great News MagAO’ers and lovers of the Blog

Today we went on-sky after a day of taking interaction matrices (IMATs).

We have made many improvements to MagAO since we had to also rebuild so much after the glycol leak. It has been a 10 month saga that has been quite a strain on all of us — but thanks to our great friends in Italy (Arcetri, Microgate and ADS) it has been a manageable storm to weather.

Here is a short list of how we got through our our repair saga:

On Feb 19, 2016 Marco and Runa (and Laird) flew, in a rush, to Chile to help Jared and Katie remove and clean the ASM (which was quite dirty from Glycol)

Then about 3 months later (May/June) we had another 3 week trip to Chile to further repair the ASM:
Brian Smith (U. Arizona) and I replaced 47 capacitors on the reference body with new silver ones (see the “Spray Silver” part of Katie’s MagAO-R post)
Katie and Jared cleaned and replaced ~100 actuators, and Mario replaced all the damaged electronics
We also replaced a failing computer (BCU) internet card in the ASM.
Mario and Enzo replaced and repaired all the ASM cooling that might fail again. All the fuses were replaced.

Then about 2 month later (September) we came again to Chile to replace the failed capacitor spring contact board and place new contacts on little 100×300 micron gold pads. This was very tricky and needed to be exactly the right size and shape (at the 10s of microns level) but was well executed by Federico of Microgate.
Then we set the ASM for full Electromech testing over the next 10 days with remote support from Marco and Runa (and Armando). Meanwhile the MagAO 2K PI (Jared) was hard at work trying to upgrade MagAO to run at 2kHz. He got our CCD39 running at 2kHz!

Then the ASM (in the clean room) was run remotely from Arcetri to continue testing.

Then our big November engineering run started (we are in the middle of that now). First we had to install an interferometer to calibrate the smallest (nm scale) motions of the repaired (and now miscalibrated) position sensors (the capacitors). This had never been done before in Chile –but Jared and I (with Manny from LBTI) designed a new mount for the 4D interferometer shipped from Arcetri. This all worked, and we had fringes from Runa and Marco’s interferometer. Meanwhile Fernando (GMT) came to help us construct an amazing new BASIS set with Simone and Armando to allow MagAO to work with a higher number of modes (even if we had many unilluminated actuators in the center of our ASM). Then we had to align the whole system again for the NAS and then flatten the shell again with the newly aligned wavefront sensor. Then today Alfio, Simone, and Armando took the IMATs need to test if MagAO still worked and if it is now possible to work with more than 300 modes on-sky (something we could not do before the repair upgrade). Also to help make that happen Jared’s 2K plans had a new camera lens fabericated so Enrico and Lauren could align the PWFS CCD to have 10% bigger pupils (to enable better sampled modes) see here for more on this successful (and very tricky) alignment. Also Mario and Jared got our new PI 331.04 Tip Tilt mirror running at 2 kHz as well with our new high voltage driver board. So you see we have been busy!

Today After one day of taking interaction matrices (IMATs). Alfio, Simone, and Armando made 400 mode IMATs with modulation radii of 3.0, 1.0 and even zero!

The Key IMAT team — Simone, Armando, and Alfio with great test results of their new calibrations.

So as you can see it has been 10 months of almost constant repair and upgrading work which has lead us to this moment.

So tonight it is so exciting to note that in bad 1.0 seeing and super thick clouds (very bad weather for LCO) we were able to close the loop beautifully at 400 modes at 1 kHz with ease with ~60 ph/subap/sample at 3 and 1 lambda/D modulation.

YES!! — Closed Loop 400 modes 1 kHz — new post repair mirror positions. Everything looks good.

The results are excellent, we are all very happy (and the PI is very relieved)

so pretty .... — A high Strehl z’ image of the first star we looked at post repair. More modes than ever before.

Here is one of my (and Emmeline’s) favorite songs: