MagAO-X Post-Cataclysm Day 0: It Was Almost Worse

7 weeks ago Chile was hit by a massive winter storm that dumped huge amounts of rain at lower elevations and snow higher up. This caused many disruptions to life here, with flooding, washed out roads, and many hardships.

The crew at LCO knew it was coming, so they went down to a skeleton crew who rode out the storm, but were stuck on the mountain for nearly 2 weeks until access could be restored. The mountain lost power, water, and internet. As part of managing this, MagAO-X was unplugged and left to hibernate while the rest of the observatory recovered.

Katie and I arrived today to begin plugging MagAO-X back in and bring it back to life. We’ll also take care of a few odds and ends while we’re here.

On the drive up it looked like we had arrived at the wrong place. Lush greenery and stunning wildflowers. The picture above doesn’t really do it justice. The picture below shows it a little better, but also captures the remains of one of the landslides that blocked the highway from La Serena to LCO.

By now the highway is mostly back to normal, though were several lane closures for crews still doing cleanup. The road from the highway to LCO is a different matter. Above you can see where it was totally washed out, and we had to take a freshly plowed dirt road through the wash downstream. No water there, but we saw many puddles, ponds, even a small lake, along the road.

Upon arriving, we discovered just how close we were to disaster. Water had clearly gotten into the cleanroom and run along the floor. However, it appears to have only touched one leg of the instrument and missed our electronics. Phew.

After arriving we did some glycol maintenance and got it flowing again, mopped the floor, and plugged everything back in and turned on the computers. So far so good. Tomorrow morning we’ll start moving things and close the loop.

We took a different route from the airport today, and saw this mural in La Serena
A nice surprise in the lodge! A selection of NA beer.
It’s still winter here, so this is the closest we got to a sunset pic.

This is a short trip so we’ll keep the rules simple: song of the day is required as usual.

In Residence

Part I: At The Grand Canyon

Drs Males and Morzinski (and a little squirrel) just had the amazing honor of being personally shown around the Grand Canyon by the Grand Canyon Conservancy’s Astronomer in Residence.

An absolutely packed theater at the Grand Canyon Visitor Center.

Said resident astronomer is none other than MagAO-X’s own Dr! Logan Pearce. After a pleasant road trip (via Nothern New Mexico) we arrived at the Grand Canyon visitor’s center just in time to catch one of her public talks:

It was really quite like the act everybody had been waiting for finally took the stage!

Logan covered a wide range of astronomy, and the audience’s rapt attention was broken only by their laughs at all the astronomy humor. (well, and a briefly fussy baby)

Of course PDS 70 made an appearance. +1.
We toured the rim trail and visited several of Logan’s favorite spots.

One of the fun facts we learned from the AIR is that the #1 treated injury at the Grand Canyon is squirrel bites.

The other fun fact is that there are no dinosaur bones in the Grand Canyon because the rocks are too old!

Katie, Jared, (squirrel for scale), Todd, Alycia, Logan, and Justin

On our 2nd day we had another fun MagAO-X experience as MagAO-X superstar Dr. Alycia Weinberger and fam joined us. I think Logan even managed to keep it a secret until they were in line at the gate (when I texted Alycia to hurry up!) We had a great time catching up and had a wonderful sunset dinner outside of the Yavapai Tavern.

Logan will be in residence through the end of July. Here’s the schedule of night-sky events which include her talks: https://www.nps.gov/grca/planyourvisit/upcoming-night-sky-events.htm

This is actually from a few days before, but the post-sunset planet show was really amazing for a few days there.

Part II: At LCO

It was implied but never stated: MagAO-X is off the telescope and set up in the clean room for remote ops. Thus endeth our first Nasmyth-West residency on the Clay telescope.

LCO and the Grand Canyon aren’t actually all that different in terms of scenery.

Not much to say other than “big success”. The jury is still out on whether it was actually better for our quality of life — no one cooks for you when you are just working late on campus, and other people expect you to be awake during the day.

Ready.

Song of the day:

The cover band at the Yavapai Tavern played this while we were there. It reminded me of the time I saw John Denver live — and he refused to play this song. Actually shouted “no” back at the crowd. I guess he was sick of it.

MagAO-X 2026A Day 32: UUUUU_

The poop-covered mirror selfie, with MagAO-X driving by.

As Katie reports, we had a smooth and efficient MagAO-X removal — even though Sebastiaan and Parker collected the very last photons of the run. Everything went great, that is, until we turned the computers back on in the cleanroom. At this point we found that the data storage system on the Instrument Control Computer (ICC) had essentially melted down, apparently it got car sick. We need that working to be able to test cabling results, so instead of getting all the way to a working lab setup in 1 day (our standard) we had to pause for a night of RAID rebuilding.

Lots of good friends hanging around.

So this morning we finished restoring the ICC data array, moved the important bits back onto it from our (just barely finished in time) fresh backup, and got to work. Parker and Katie cabled the DMs, and we had only one problem which was an actually failed “samtec” insert (the cheapest and easiest to replace piece of our DM systems). After replacing that we had the loop closed and got to work.

In addition to the IR camera packup, we also removed the experimental baffle tubes so we can make “professional” versions, and knocked out a few other odds-and-ends tasks to clean up.

The sunset from the gym. The green flash was amazing!
There was a herd of horses (and at least one mule) to negotiate with on the run. Parker almost didn’t make it.

After a dinner break, and some nice sunset jogging, we ended the night with some ADC experiments. It’s a long story, but something changed a few years ago. Katie has tried a bunch of things, which mostly just confused everyone. After a consultation with Laird, and digging through his small warehouse of optics down here, we installed a wedge to de-wedge another wedge, with encouraging results. As always, the permanent answer involves $, really $$$, but now we know what to do.

Those are some straight speckles. IYKYK.

With apologies, the color of the day is inspired by a real problem: the foxes are starting to become, let’s say, noticeable for the wrong reasons.

I’ll bet it’s been a while since this has been on the blog. My advice: crank it and dance like you’re 18 mos old:

she actually tries to do the horsey dance, not quite there though.

MagAO-X 2026A Day 17: And Nothing Else Matters

I heard you guys are finally leaving?

Well here we go, on our way home. Wow this run felt long, even though we only had 7 nights. The first group of us got here 10 days early, though as is typical we mostly sat around waiting for a shipment to arrive.

This captures how it always feels when we careen back down the mountain, hoping to find a sign of our old civilization somewhere ahead.

In truth we did an unreal amount of work. From overhauling our instrument’s glycol cooling to fixing all the vibrations to installing a new infrared camera, this has been action packed. And it all works!

A big accomplishment was all the people who got their driver’s license on this run.

Our last day/night was not without excitement. Over the last two nights our AO Operators Computer (AOC) has been randomly freezing up. My little buddy (read as either GPT-5 or Joesph, your choice) had lots of ideas. After a dawn reboot it was still spewing disk errors when I woke up this afernoon, so we decided to do some troubleshooting.

I feel like there is always a wrong choice made about “do it right, move it and have full access” vs. “touch as little as possible so you don’t break it more”. You never know until you’re done.

In the end we didn’t actually learn anything. We did stop the freezes, but not the disk errors, so we have fewer ideas than when we started.

At least the computer came back up and is operational except for a [_UUU] we have to fix before we leave.

The official group photo. We’re missing Miles (who left a few days before) and Laird (who I left a few minutes before).
MagAO-X will be here for you even after we’re gone.

Here’s the craziest part: this isn’t even the end of the run. We have 13 more nights coming up, but we’ll be driving MagAO-X from the friendly confines of Steward Observatory. Another adventure begins.

The color of day is a moonlit-sky-blue

The song of the day is Nothing Else Matters by Metallica

This makes a good lullaby (except for the yeah-yeah! part)

MagAO-X 2026A Day 3: Deadlock Funlock

Multi-threaded software is hard. You get all these different processes trying to access the same place in memory, and if they do it at the same time weird things happen. We have ways to stop these “data races” with tools like “atomics” and “mutexes” (mutices? plural of mutex for mutual exclusion) — but these add a new fun bug called “deadlock” when different threads try to lock the same mutex. So I’ve been spending lots of quality time with my new third best friend GPT-Codex. We make about 20 steps forward for every step back, but it’s in the time it used to take me to take about 1 step forward. Winning.

Miles has been hard at work getting his new polarization components installed.
Here we’re using a bright red laser to chase the beam path through the instrument.

Sebastiaan and Adam arrived from Leiden today. Sebastiaan is already complaining about having to do EFC.

The door to the lodge was being guarded this afternoon. In a very friendly fashion.

A nice sunset from my room tonight.

The color of the day is HeNe laser red.

The song of the day is The Emptiness Machine by Linkin Park: