MagAO-X Engineering Run 2024B Day 1: No Longer A Couple

One of the big goals for this visit to LCO was to troubleshoot a coupled pair of actuators on our precious 2040 actuator “tweeter” deformable mirror (DM). Being coupled meant that when we poked actuator 498, actuator 650 moved (not 498). Actuator 650 also moved when we poked 650. We first noticed this problem after shipping MagAO-X in March, when we ran a post-cabling function test. However, digging through our archives we found that it showed up back in November, 2023, but we just didn’t notice.

Now our tweeter is expensive. So we don’t like it when things go wrong, and it makes messing with it really stressful. But having those two actuators acting up means MagAO-X didn’t work perfectly, so we had to do what we can to fix it. We deferred it until after our March and April runs to give us time to plan and do it right.

We can all breathe a sigh of relief: with only minimal shenanigans we got it fixed. After testing various potential cabling problems, we found that a circuit board just needed to be replaced. We had a spare, popped it in, and the couple had split up and gone back to behaving normally.

And we got that done in time for Empanada Sunday:

Empanadas are just a little bit better fresh, compared to at midnight.

After lunch we got onto the next set of tasks, which included disassembling a rack shelf so we can get better airflow.

The shelf in question is actually Jay’s baby. He designed the hold-down system for the components on it. That also means he had to take it apart and put it back together.

We’re also upgrading the Instrument Control Computer (ICC) to go faster and do more stuff. We like to liquid cool our CPUs and GPUs. Normally we rig that up in Tucson, but we couldn’t get all of the components from our usual supplier in time, so we shipped what we had (and found some substitutes on Amazon!) and are doing some clean room plumbing.

An NVIDIA RTX 4090 in the middle of having a water cooling block attached to it. The shiny square in the middle is the business end.

Jay has also been leading the assembly and testing of new ICC. Other than the liquid cooling parts, it was fully working in Tucson before we took it apart to ship it down here. We’re getting it all back together.

Jay hard at work screwing ICC components to the custom rack mount.

Even after our delayed departure, we are making good time thanks to the relatively easy fix to the tweeter. I haven’t had enough running on this trip yet, so I took some time before dinner to go circle around the 100″.

Start of run, heading to the 100″ telescope which is down and left from Magellan.
The view back from the 100″. I’m surprised this came out, it was so windy I couldn’t hold my phone still.
From 1966.

MagAO-X Engineering Run 2024B Day 0: A Hero’s Journey to LCO

Let me tell you about our trip.

Jared and I were scheduled to leave on Thurs. 09/27, however our Tucson flight was delayed to the point where we would have missed our ATL-SCL connection, so we got the opportunity to rebook. Whether or not this was due to Hurricane Helene, we’re not sure. Because ATL-SCL on 09/27 left on time, more or less, despite being closer in time to peak hurricane’ing. Our original TUS-ATL flight arrived about 3 minutes before the plane to Santiago left the runway, so all in all, it was the right move to rebook.

Because we rebooked, the revised departure date was the next day, Friday 09/28. By about 9a on Friday, the flight was already delayed by 20 minutes (we thought: “here we go again”), but thankfully the delayed departure time didn’t change again…. all the way until the VP landed in TUS which (we painfully learned) shuts down the whole airport for half an hour. This new delay was coupled by the plane being overweight as well because of the unseasonably hot tarmac temperatures this late into September. So, the poor souls on standby who got onto the plane had to be escorted right back off to bring the curb weight of the plane down. After all that, we departed TUS *finally* at around 2:10p, a final delay of about 1hr 20mins. We were set to arrive at ATL by 8:15p EST at Terminal B. Our ATL–SCL flight was set to depart right at 8:55p, from Terminal… E….

This is NOT Air Force Two, but it was close to the airport, so, possibly related.
Cue the Mario star powerup music
Y’all ever see “Yes Man” with Jim Carrey? Maybe the group that practices running photography was onto something, this is pretty fun.

We ended up making it with mere seconds to Gate E14, but Delta misjudged our dedication (and ability to sprint) and automatically rebooked us to fly to Santiago the next day while we were still en route to ATL. After a few tense minutes, however, the gate agents were able to get us back on our original flight and thankfully they didn’t close the door on us as it was definitely after 8:40p by this point. After taking our seat, we left the ground a few minutes after 8:55p.

We landed at SCL without incident but about ten minutes past 7a local time. Normally not such a big deal, but because we had gotten rebooked initially, our new flight out SCL–LSC was at 9a on the dot. The line to get through customs was, of course, super long. We finally got through at 7:55a. Boarding began at 8a, so, more running…!

SCL has also moved the security checkpoint to near the LATAM check-in desks. Again, normally not such a big deal, but Jared was never given a boarding pass when we deplaned out flight from ATL, so he had to get one reprinted before we could get through security. It was 8:34a.

Boarding doors close at 8:45! and the B gates were, no joke, like a third of a mile from the security check point.

We. Just. Made it…!

After taking the transport up from La Serena (and sleeping for most of it), we were just in time for lunch and were greeted by some familiar views.

Truth be told, after all the delays and setbacks and just plain bad luck, I never thought we’d make it this far.

Bonus Content

For today’s Bonus Content we were greeted by a ravenous goaty parade at sunset…! This made for many, many small “landmines” when walking down later that evening in the dark. Make sure you wipe your shoes extra good before entering your room during goat season.

Blog Rule

Short trip, simple rule. The blog writer must pick the Song of the Day from the assigned decade below:

Day 0: 1950s
Day 1: 1960s
Day 2: 1970s
Day 3: 1980s
Day 4: 1990s
Day 5: 2000s
Day 6: 2010s
Day 7(?): 2020s

Song of the Day

The song of the day is Frank Sinatra’s “Come Fly with Me”, originally released in 1958.

MagAO-X Maintenance 2024-1 Day 2: In-N-Out

Caught the 0830 transport down this morning. Irony of ironies, this may have been the most stunningly clear blue Chile-wide sky I’ve ever seen. Caveat: I’m usually coming off a weeks-long night-schedule average 4 or 5 hrs of sleep, so not usually awake for any of this trip.

Not a cloud in the sky at sunrise, but a few on the ground.
Last view of the telescopes (until next time)
First view of the ocean. About here is where your skin breathes again.
The forms must be observed. Even after such a short stay, this is the stuff.
The La Serena / Coquimbo shoreline. Should have gone to the beach.
The Andes look very imposing when you can see them.
Challenge: find Vera C. Rubin!
The SCL succulent wall looked gorgeous with all the sunlight.
A closeup.
Song of the day.

MagAO-X Maintenance 2024-1 Day 1: That Was Quick

My short visit was a success. Got our cooling system back to nominal, and installed and tested cables for our new cam-yo-fiz.

Clouds aren’t so bad when you’re above them
A good lunch to warm up with. It’s cold.
Beef meatballs on pasta, and some very garlicy tomatoes and avocado.
My PhD project sits forlornly in the parking lot, not even fully covered.
Vizzy. What a tail!
Time to go to work up there.
The song of the day

MagAO-X Maintenance 2024-1 Day 0: 40 to 40 in 1 day

I popped down to LCO from Tucson for a quick visit to fix some cooling issues and test some new cables. I left 40 C in Tucson and arrived to 40 F at LCO.

On my flight down I was somewhat surprised to see that Laird was coming with me.

Laird and Shannon were seated in the row in front of me. Who knew.
The Starbucks SCL Tradition. I think that’s Yeri.
I don’t know what this is about, but I know we’re fans.
Santiago is surrounded by snow-capped mountains, but it had just rained when I walked between terminals.
The La Serena area is stunningly green.
A familiar, but greener, view driving up the coast
Our old friend is still here waiting for us. I probably won’t make it up there.
The post-sunset sky on my walk down to dinner.

There are blog rules. But since I’m the only one here I’m keeping them to myself. Don’t worry, I’ll follow them.

The song of the day