Blog

We’ve got MagAO-X mostly re-cabled in its temporary home in the LCO cleanroom, and Doctors Close and Knight are fresh off the plane and working on the optical alignment. But, earlier, we had a fun discovery: the instrument control computer (ICC) was getting almost no coolant flow.

Yesterday, we did some brain surgery on the real-time control computer. Today was more like heart surgery. We found that although our pump tried its hardest, almost no liquid coolant made it through the ICC, and temperatures remained stubbornly high. In other words, it was clogged. We really wanted the issue to be anywhere except the CPU liquid cooling block, so of course our troubleshooting pinpointed… the CPU liquid cooling block. Not any of the lines feeding it, but the very center of it.

See those three pink hoses in the center, under a bunch of crap? Those go to the blocks we removed. We took the computer out of the rack, the cooling blocks off the computer, then took them both out of the clean room entirely to try and blast the clog free.

But, to no avail.

After consulting reputable YouTubes, we were pretty sure these things came apart. The down-side is, according to the manufacturer, you lose your “leak-free guarantee.” (Well, it’s probably void after 5 years anyway.)

Readers, it was gross in there. We only do extreme adaptive optics, and this was extremely gross.

It turns out glycol does not enjoy being left in tiny channels without moving for a long time. And, while MagAO-X was on its two year shipping hiatus, it didn’t get the same twice-yearly flushing it evidently needed.

Fortunately, there was a solution: graduate student labor!

(Just kidding; it was a team effort.)

After we reassembled and pressure tested and reinstalled everything, we had great flow. We also had just spent a few hours on another unscheduled computer disassembly, and had to hustle to get the system ready for Laird and Justin. Fortunately, it was a two-viscacha day, which boded well for our efforts.

Another of today’s wins was figuring out what the “ultra-wide angle” camera on my phone is for: making an already long advisor look even longer.

Once the computers were back online again, we used 2.67 monitors per researcher in hopes of making everything go faster.

Fortunately, installing all of the cables between the electronics rack and the instrument went great. All our movable bits in the instrument moved when we asked them to, so we had time for a bit of sunset-watching before the clean room became part of Laird and Justin’s quarantine bubble.

I also had a wistful moment, taking a selfie in the tail plate. I definitely did not imagine that I would be the only non-faculty repeat visitor from the original team.

We rigged up a lab laptop (labtop) and left it signed in to Zoom™ so that when Laird and Justin got in we’d see them. We’re remotely supporting their alignment efforts by sitting upstairs with our laptops to move mechanisms as needed. (Or write blog posts, when not needed.)

As I write this, they’re still at it. I admire their tenacity.

Late breaking news: we have pupil images on the pyramid wavefront sensor!

Song of the Day

Your song of the day is brought to you by ~*~*flow*~*~.

We hit a pretty major roadblock today. Due to the sky high fuel prices around the globe, Las Campanas Observatory has had to drastically reduce use of most forms of energy, including electricity for cranes, propane for forklifts, and gas for trucks, except as needed for nighttime operations. So we essentially have no power when the sun is up.

Luckily, as we always do on the MagAO-X and LCO teams, we had a contingency plan. This is, of course, the burros who hang around the mountain top. The young lady above was our assigned crane and forklift motor for the day. It took a little longer than normal, but in the end we managed to get MagAO-X unpacked.

Yesterday Sebastiaan mentioned our misbehaving graphics cards (GPUs). When we turned the electronics on, 0/4 GPUs that we use for our super-fast real-time computations were alive. There is not much in common between these 4 devices in terms of where/how they are installed, so it seemed extremely unlikely that they all just quit on us. So after none of us getting much sleep, in part due to racing minds trying to figure out what it could be, we divided our duties today so Joseph could troubleshoot while Sebastiaan and I worked with the crew on unpacking.

In the end it was two different problems. One of them just needed a stern talking to (nerd code for we took it out and put it back in, and it worked). The other 3 were more complicated. We use a PCIe expansion system, which lets us attach more crap to our computers. The backplane of that failed, but luckily we had a backup burrito for this too, and once it was put into service we now have all 4 GPUs working.

When we went up to the top we saw a vulture soaring on thermals. It was coming right over us, and we could hear a whistling sound from its feathers.

The best news of the day was our friend Vizzy the cleanroom vizcacha making an appearance. This was when I knew the GPUs would be ok.

The song of the day accurately reflects my sentiments after what was a damn stressful 24 hours.

Two and a half years ago, I came to the University of Arizona to work with MagAO-X. The plan was that MagAO-X would go to the Magellan telescope twice a year. And then suddenly a global pandemic appeared. However, right now we are really at Las Campanas Observatory preparing for an observing run! We arrived here on Monday and we entered the Bubble Mode for 3 days. In the Bubble mode we had to stay away from the other staff and got food delivered by room service. Our initial expectation was that we had to stay in this bubble mode for 72 hours after arrival at LCO. But, the clock started counting from arrival in Santiago! We learned this after missing our special breakfast room service. The kitchen staff thought we were already good to go for normal breakfast. So for the final procedure to break out of the bubble, we went to the medical unit to get our COVID test 8 hours earlier than planned.

All three of us agreed that this was quite an unpleasant nose swab. I think they were trying to scrape out some brain material. So for all other MagAO-X observers; be warned! After getting our clearance, we were allowed to eat in the actual cafeteria during lunch.

With the covid-free clearance, we were allowed to come in contact with other humans. I could finally get a tour of the Magellan telescopes.

And because we were allowed to meet other people, we could actually start unpacking MagAO-X. We met up with the great technical staff of LCO to crane the electronics box that contains the driver electronics of pretty much every part of the instrument. This required a very delicate procedure and took a couple hours. But, we are very happy with the results; the electronics box is unpacked and plugged in.

But what is a commissioning run without any challenges? Apparently our graphics cards found our plan to observe most disagreeable. Luckily, more team members are flying out tomorrow and they have quickly scavenged the lab in Tucson for more spare graphics cards. We still have more than a week to go for our first night, and enough work to do. It is going to be an exciting run with many interesting projects. Stay tuned!

When we last left LCO we expected to be back in just 3.5 months. We’re nerds of the science and engineering type, so we tend to take lots of notes and pictures because we think we want to remember this or that. But there are always those things where you don’t even think about not remembering it because it’s so obvious, or around here it’s because we have done it that way every 6 months for the last 10 years.

Well, guess what. 2+ years of being totally distracted by global catastrophe fries your brain and you forget a lot of things. So I spent a lot of time today rummaging around all of our junk at LCO trying to piece my brain back together. Several times I came across a nicely labeled box, or a place where someone collected all the bolts that go with the thingy, and said out loud “way to go past us!”.

But also there are cases where there’s nothing on the hard drive and I just stand there like this:

The big accomplishment of the day was unpacking our AO Operators Computer (a.k.a. AOC, a.k.a. exao1) and getting it set up. Joseph and Sebastiaan assembled Megadesk (without breaking it) and got AOC plugged in, on the network, and monitors sorted out.

We also had our first Earthquake of the run. I was up in the cleanroom after dinner, and announced it on Slack. Sebastiaan was down in the lodge and saw my slack message before he felt it.

We also unveiled the new attire for cool-kids at LCO

There are some Burros around, as you’ve seen in previous posts. Today I’d like to highlight the Burritos. There is one hanging around with Ma up here at the lodge, and another down at the Bodega.

And of course, there are the sunsets. Oh how I have missed the LCO sunset ritual.

Us first 3 have just 1 more day of “bubble”. Assuming negative tests tomorrow evening we’ll get to join normal life at LCO. So far, we’ve just been able to say “hi!!!” to our friends from a distance. We’re really looking forward to actually getting to reconnect.

The song of day’s video sorta captures what it feels like to look back through a decade’s worth of blog posts to find that thing that you forgot.

(distanciamiento de un metro is distancing of 1 meter, a required anti-COVID-19 measure)

MagAO-X arrived on its truck from Santiago this afternoon, and a familiar team of forklift operators and engineers was here to unload it. The next shift will have some new faces, but this shift was people we knew from The Before Times. We think. They’d grown ponytails and everyone wore masks, so who’s to say?

When I got the message via Jared that MagAO-X had arrived, I was just finishing lunch. While in Bubble Mode™ we have to wait around and hand off our trays to the staff to be returned to the lodge cafeteria, so I said I’d stick around for a few minutes.

Turns out I couldn’t wait, so I wrote some (hopefully correct) Spanish on my health check-in form as a sign for the staff and took off.

And here it is!

You don’t see it? Computer: enhance 224 to 176.

There we go. Right there on the truck. See it? No?

Fortunately, I was distanced by more than the required amount, and I could get closer.

No, closer than that…

Look at those three boxes of big science! But it’s not doing us much good on the truck…

Fortunately, LCO has their own fork lift. And very good operators.

Next, let’s get our electronics…

They set everything down outside, ready for unpacking when we’re out of bubble mode. Fortunately, these boxes were made to be left outside for a bit.

(Not pictured: The removal of a small plastic box containing our operator computer by means of a totally overpowered forklift. Still better than trying to clamber down from the flatbed!)

We did notice some oddities… the couple of missing tip-n-tell indicators indicate someone tipped but would rather not tell. Aircraft takeoff/landing angle, or something more nefarious? Also, something happened to trip the 10-G acceleration sensor in the electronics rack, but fortunately not the 25 or 50 G!

Bonus Animals

Song of the Day