MagAO-X 2024Aa Day 21: Time To Offload

Offloading is a major, but sometimes under-appreciated, part of an AO system. It makes sense when you really think about what we’re doing. The moving part of the Magellan Clay Telescope weighs more than 100 tons (I don’t know the exact number, but that is the OOM that has stuck in my head from somebody telling me that once). But what we actually do for a living is wiggle tiny little pieces of silicon with a few atoms of gold or silver on top, in response to massless particles streaming through the atmosphere, all in a quest to achieve nanometer or better control over the universe. Over time the little wiggles turn into big wiggles, and we have to send them to the next bigger thing. We call this offloading, and in MagAO-X there’s a whole chain of it from our coronagaph deformable mirror to our pyramid+tweeter DM system (where the real magic happens) to our woofer, and from there to the telescope itself. Inside those main control loops there are other control loops that are adding steps to the dance. Loops on loops. There’s a certain absurdity to our little concerns getting dumped onto the majestic Clay telescope and pushing it around.

A way to analogize offloading: imagine flying to the other side of the world to a remote mountain top, being given a bunch of urgent tasks (some of which are impossible but nobody knew that until you failed), then suddenly switching to a night schedule, sleeping for 4 or 5 hrs a day (not night) for two weeks straight, and then suddenly switching to a day schedule so you can undo everything you spent the last two weeks doing. You can imagine that you’ll build up a little … stress… that’s hard to hold and you need to … offload it. Well it’s time to hit the ol’ dump button. You might not hear from us for a while.

Maybe as long as six weeks.

The last sunset.

So that’s it. MagAO-X is off the telescope after a truly amazing run. We’ve really hit our stride as a productive scientific instrument. I lost track of how many new things about the universe we discovered on this run, and #nospoilers anyway. At the same time I am leaving, as always, a little frustrated by the ways in which MagAO-X isn’t yet perfect, not yet meeting my expectations. Every new discovery comes with a lesson for how to do it better next time.

An image of Crux and Alpha & Beta Centauri rising over the Babcock Lodge at Las Campanas Observatory

Without a doubt the best thing about this project is that it is fun. The fun started with MagAO, and a long running tradition on this blog is the “quotes of the day”. It comes and goes, even during this run. Back “in the day” it became a problem because we realized people were just saying things hoping to get famous. Later on, it got us kicked of the Steward front page (still banned AFAICT, ¯\_(ツ)_/¯). As a reader, you should know that some fraction of the quotes have always been just straight made up. The lessons are that the quote log can be used for good, for evil, and sometimes it gets things done. Well, so anyway, here’s a PI-curated selection of quotes from this run. #overheard

“The control room is likely to be completely overrun. Keep calm and make sure you have a chair”

“take that speckles”

“This is a public cheese. Everyone has to use this cheese.”

“Who is Jahlishus?”

“Blog ops is completely unrecognizable. All it is is fucking penguins and Dall-E viscachas.”

“Which is almost a triggering thing for Jared… I’m kind of disappointed he’s not here to hear that”

“Let’s see if I can install zoom before the sun sets”

“Oh. My. God. We’re over 2 arcseconds.”
“We’re going to need a bigger plot.”
“This is just all sadness…”
“I don’t want to go up there to the sadness party”

“When conditions are like this, it feels like nothing works… because nothing works.”

“I have way too many laptops”

“I spent my whole postdoc finishing my Ph.D.”

“If you’re okay I’m going to take a break … for breakfast … what time is it?”

“the units are ‘play around and find out’”
(Ed: shortly thereafter the SI unit FAFO was coined)

“We had everything wrong”

“g band is horrible.  I wouldn’t wish this on my worst enemy”

“We deserve strehls better than 20%”

“I’m not doing science, I’m doing AO”

“But I thought AO was easier in the visible?!”

“You see that?”
THAT?!
“I know.” 

“what’s that second speckle???”
“…. OH that’s Pi Pup B!”

“Who knew it would actually work?”

“You cannot hit the grad students”

“that is a really large blueberry”

“… ah this is the 4th time I’ve heard about ‘The little guy’, Can I see what you’re talking about?”

“I stared at him and thought ‘Why are you not making my salad’ “

“oh that tingled. oh wait I am electrocuting myself.”

“So the only thing misaligned in your system was the cardboard?”

“except that it imploded during shipping, it works really well!”

“You don’t do that with geese but I didn’t know that at the time”

“I’m stealing fruit for my next couple days of breakfasts. I’ve been stealing a little bit every day. I have a little pile of fruit in my room.”

“Zernike polynomials, you’re my bitch now”

“I found our nuts at Baade! Those Bastards!”

“I appreciate that you’re always working on bullshit when you’re here”

“I love it here”

“I’ll bet you all your empanadas for the rest of the run”
“… nothing is worth that”

“More people will read your blog than your PhD”

“Go ahead and take a dump.”


I love stories about a crew. LOTR, Firefly, The Expanse. I found this song through Yellowstone, which on its surface appears to be about one dude/family but I think is more properly understood as a crew story. The song itself sort of captures my mood about our ongoing attempt to achieve the fundamental limits of wavefront control. If it wasn’t hard, it wouldn’t be worth doing. We have a good crew, so we’ll get there.

or maybe, lol.

MagAO-X 2024Aa Day 13: Conjugate

We finally had an all-night good night. Started with what we call engineering, which means testing new ways of operating the instrument. The big news is that Sebastiaan got his “implicit Electric Field Conjugation” algorithm to work on-sky. This brings MagAO-X almost to its as-planned fully capability. Here’s what that looks like to one of our highly trained operators:

A fully armed and operational MagAO-X station.

Why would one want to conjugate an electric field? Let’s let Sebastiaan explain:

Speckles are why twinkling is bad for science

After the instrument-science experiments, we spent the rest of the night hunting planets. Tonight was almost one of those legendary Cerro Manqui nights that keeps us coming back over and over again. It was essentially 0.5″ seeing all night, but we were plagued by our version of the low-wind effect. However instead of Mickey Mouse, we get the Bouncey House: when the wind drops to 0.0 the telescope gets bored and starts to bounce. It makes our control loop more or less impossible to optimize.

We have been conjugating other things too.

Moonrise
Moonset
Telescopes day
Telescope night
Viscachas demonstrating the principle of reversibility.

The song:

I heard you missed me

It’s been a busy month in the eXtreme Wavefront Control Lab. The main event was the return of MagAO-X after a long wait for a ride on the mountain without us. Everyone’s favorite ExAO instrument finally showed up right at the beginning of May, which of course meant that we broke out the hard hats and steel toes.

Everyone is always happy to be here.
This is the box with the AO Operations Computer (AOC).

The day of the main event started at 0600 when we started working with the mirror lab crew to crane our shipping boxes open. But first we needed the stuff that makes astronomy go:

Jay brought coffee and bagels
The box must first be unbolted
We got the lid off and before the first rays of sunshine peaked over the loading dock wall.
Then we moved on to the electronics rack.
Not everybody was awake enough for power tool operation.

MagAO-X has been down below since October, so of course the lab had to be reorganized. You could barely tell that MagAO-X used to live there full time!

This is the HEPA blower we use to apply positive pressure to keep the dust out-ish.
One must first tear one’s lab apart before one can put it back together. Our clean room has to be partially disassembled to fit a crane into the lab to finish unloading MagAO-X.
A “clean” room doesn’t stay very clean when it just sits on the floor for 6 months.

Maybe the biggest excitement of the whole thing was evidence that we had water leak in to the box. It must have been rained on (we always blame Miami), and we saw water spots on the mylar blanket and some rust on the shipping frame. So we were super anxious until we finally got to unwrap the instrument in the clean clean room:

Regarding the circle, I wouldn’t worry about that little circle.

Once back in the lab, safe and sound, we began the long process of unpacking.

All the things are shrouded in plastic bags to keep them dust free. We call this step “de-baggie-ing”
It’s a lot of bags
Bag free. But does it work?

For me, the shipment isn’t over until we actually cable the 2040 actuator deformable mirror (I’ll let you work out how many wires it takes to move 2040 actuators . . .) and show that it has survived the trip.

Eden is tightening down a bolt to press the connectors together. It’s a good workout and low stress.

Thanks to the excellent touch of Eden and Sebastiaan we got all the connections made on the first try and sat down and closed the loop.

This is when I start sleeping again.

We were gone long enough that the university’s IT department didn’t recognize us:

Our computers are . . . complicated. When they draw up rules for campus computing systems they just don’t think about 2 kHz real-time control of turbulence and TB/day data rates.

Here is MagAO-X, Phase I, all set up and cabled back in its Tucson home.

But now the fun begins! We’re taking the next year off from the telescope to do some major upgrades, and long overdue repairs. Today we started tearing our MagAO-X apart. The first thing to go is the Pyramid Wavefront Sensor detector, which is on its way back to France for a timing board replacement and to get some (possibly) more sensitive operating modes installed.

Our venerable OCAM-2K EMCCD. This is a key part of what makes MagAO-X go and go fast. But it needs some TLC. Before shipping it, we of course have to drain the glycol from it.

When we head back to LCO for April (or so) 2024 we’ll have 1000 more actuators, new coronagraphs, new LOWFSs, and waaaaaay more software. Maybe fewer github issues too.

Going back into hardware mode is both exciting and a little terrifying. The XWCL will have to stay focused on the tasks at hand and be ready to deal with all the sh$t that won’t work the first time and go about figuring it out. But remember: “having more things just means more things can go right”.

It’s also a time of change in our group. Alex (v2), Joseph, and Avalon have all defended; and Warren is gearing up for his defense in a few weeks. If you’ve been following this blog you know they have all been crucial to the success of MagAO-x and been on many an adventure. The rest of us are now scrambling to scoop up all of their knowledge and make sure we can keep it going without them. We’re also going to miss them!

Here’s a bunch of the XWCL at the usual Friday afternoon spot, celebrating Avalon’s defense, and maybe showing some relief that MagAO-X made it home too.

The song of the day is “Guess Who’s Back” by BEGINNERS & Night Panda. It is, as the kids say, a banger.

Stay tuned for an action packed summer full of travel and conferences, new hardware, probably some more glycol spills, and all of our usual shenanigans.

MagAO-X 2023A Day 24: Homeward Bound

Well that was better. After our December run (trucking strike followed by extremely bad luck with the spatial power spectrum) we were all holding our breath for this run. This time, we got a good week in the cleanroom to tune MagAO-X up, and Cerro Manqui cooperated — delivering 0.27″ seeing at one point. As documented in previous posts, we got a huge amount of work done. We also detected a whole bunch of planets.

MagAO-X Phase 1 commissioning is now complete. XKID was the last thing on the official list, but we also got to the point where we can confidently act like a real instrument. We can point at a star, lock the high-order loop, align a coronagraph, lock the low-order loop, and start taking data, and do it all night long. We can target hop efficiently (for Alycia anyway). We’re achieving contrasts with “e-6” in them, and we are working within 2 lambda/D of stars (though not yet at the same time).

Don’t worry though, the excitement will continue. Everybody says the X in MagAO-X stands for eXtreme. But it really means eXperimental. Kinda like the the Billy o’ Tea, we’re never going to be done commissioning — because we don’t know what we don’t know and are still figuring out what works. The only way to find out is to keep coming back for more.

Here’s a viscacha doing an impression of the current state of the MagAO-X instrument and its keepers:

We’re all starting to blue shift.

After breakfast today we finished packing up. That meant putting the front door on the big box which holds the optical table, and then loading our electronics into its box and doing the always exciting dance with gravity to get it into the shipping position.

Smooth Criminal.
Apparently $1M worth of electronics being on a forklift isn’t as interesting as Sebastiaan’s footwear.

I was given permission to post a pic of myself “being PI”:

Look, if I bust out the label maker it was for a damn good reason and I f-ing meant it. T.B.C. the problem was bad threads on that lifting eye which make it extremely difficult to remove. To quote a wise man, “I’m not even mad…

After the exciting crane ops, the more tedious stuff has to get done. Applying shipping labels (which involves cleaning first) and bolting down various lifting fixtures we don’t want the shippers to use in transit (our entering assumption: if they can find a way to destroy it, they’ll do it).

Joseph cleans a spot for a new shipping label while I idiot proof some lifting fixtures that are for us only.
Eden attaching MagAO-X’s home address in case it gets lost and forgets.

After a quick tidy-up in the clean room, we’re finally bound for home.

LCO is an amazing place, with the best seeing in the world (… most of the time), gorgeous sunsets, comfy beds and great food, an awesome crew, and it somehow doubles as a zoo.

A pack of wild dogs ranged over the observatory this morning. I’ve seen a solitary dog run by up here before, but not a whole pack.
Right before I left an agave in our yard started to put up a stalk. I’m excited to get home to see how it’s going — even though it’s a bitter sweet display.
The internet knows this as “the spirit animal” pose. In my experience it’s actually kinda rare.

Laird hit the road after lunch:

Packing done, time to go.
Jay and Jialin are home. But studying needed to get studied even in the UA club in Houston.
Post-packing malaise.

Our last sunset for 2023A — 2023 in total — was as amazing as always:

We’ll be back — wait until you see what’s next.
One last breakfast.
Venus is still with us. Her friends have moved on after the spectacular “winter” display.

I brought up The Wellerman during our last-night-that-wouldn’t-end because I’ve been listening to Colm McGuinness a lot since my last “end of run” post. I think I gave Eden the wrong impression though: I wasn’t aware of whatever happened on Tik-Tok vis-a-vis sea shanties and t.b.h. am not at all disappointed that she wasn’t either.

Being even more honest, and probably ironic, I don’t actually like sea shanties as a genre. Mainly because it’s usually done in that comedic irish pub style where for some reason someone hiccups during each verse. But the songs are actually working songs, sung to pass the time or keep the beat during dangerous and hard work, far and long from home with only the hostile sea for company. (let’s leave aside that few of the songs we’re talking about here are actually shanties . . . )

I “discovered” Colm McGuinness last December because of his My Mother Told Me. Which is not a shanty, but I have to say Colm seems to get it the way I get it: these are songs that deserve a little bit of epic flare. Examples: Roll Boys Roll and Santiana. [Dude also does an amazing Jolene, background here].

Before going further, note that this all fits in my series of “music I rocked to with Ben” b/c I recently got into sea-oriented music when he sent this with the assertion that they are what we (including two other friends from high school) would look like if we decided to start a sea shanty band. YMMV. (longer story includes that Home Free recently played in Brookings South Dakota, the official hometown of MagAO).

Anyway, the song of the day is a ballad about the journey home from a long and arduous adventure at sea. I love this job, and we’re doing something amazing as a team. I think we’re on the road to achieving our very lofty goals. But every time we do this thing we do at LCO, I have to acknowledge that “it’s a damn tough life, full of toil and strife, we AOistas undergo…”

Colm McGuinness “Old Maui”

Thank god, we’re homeward bound.

MagAO-X 2023A Day 21: The Wind Down

Whelp, it’s getting to be that time. The sun is rising as I type, and it’s officially the 16th. Tonight is our last night on the Clay for 2023A, and tomorrow morning we tear it all apart, box it up, and GTFO.

Another night of awesome seeing let us all catch up on some coding and analysis.

Several coronagraphs where harmed in the making of this image (or at least made to deliver).
And note that none of us were the ones actually studying.
The cleanroom Viznitches were there to say hi as I walked up.
Uno Dos

The conclusion of my concert story, which started with the surprise KoRn and went though an awful Marilyn Manson performance, is the band we actually went to see: Danzig. It was every bit as good as one could hope. We were right in front for the whole show. At one point Glenn Danzig reached out and clasped Ben’s hand – legendary. Here is Danzig’s iconic “Mother”:

Danzig “Mother”. Sorry if you have to “Watch on YouTube”