After that we ran up the hill and cabled our 2040 actuator MEMS DM. We got it on the first try! We are getting better at this procedure.
Sebastiaan and I cabling the tweeter. Photo by Joseph Long.Sebastiaan is testing the actuator response to see if we got any cable wrong. We didn’t.
After that, we turned the cleanroom over to our Bubbled comrades. Laird and Justin put in a loooong day of aligning the optics. We sat up in the library and watched. It’s sort of like a medical procedure drama.
Laird and Justin working on MagAO-X.At the end of the day we had a pretty nice PSF. Not perfect, but that’s why we have an AO system. Tomorrow we’ll clean this up with the DMs. That’s Laird at upper right.
We found this book in the library, which we might need if any REAL problems come up.
We don’t use FORTRANA panoramic view of LCO just before the sun dips below the horizon, with Dr. Justin Knight a bubble’s length away from the photographer.
After taking the above photo and circling around Justin, I started down the path to begin my walk up. Then I noticed a sliver of moon hanging over the valley. Then the following ensued:
Joseph got an up close and personal Vizcacha interaction.
It can take a second, but you’ll notice that Vizzy is looking into the library at Joseph.
The reference to bananas in the slack convo is to this blog post from the MagAO-Classic Age of Legends. We have a long history of shenanigans with the cleanroom vizcacha family.
The post-sunset show was extravagant tonight.
LCO, the Moon, and a plane.
Changing up the sound tonight for the song of the day. I think this is the best of the “new” Metallica, as good as the old classics.
We’ve got MagAO-X mostly re-cabled in its temporary home in the LCO cleanroom, and Doctors Close and Knight are fresh off the plane and working on the optical alignment. But, earlier, we had a fun discovery: the instrument control computer (ICC) was getting almost no coolant flow.
Yesterday, we did some brain surgery on the real-time control computer. Today was more like heart surgery. We found that although our pump tried its hardest, almost no liquid coolant made it through the ICC, and temperatures remained stubbornly high. In other words, it was clogged. We really wanted the issue to be anywhere except the CPU liquid cooling block, so of course our troubleshooting pinpointed… the CPU liquid cooling block. Not any of the lines feeding it, but the very center of it.
Disclaimer: this is actually a picture of the other computer, but it looks cooler. They’re basically the same though.
See those three pink hoses in the center, under a bunch of crap? Those go to the blocks we removed. We took the computer out of the rack, the cooling blocks off the computer, then took them both out of the clean room entirely to try and blast the clog free.
But, to no avail.
After consulting reputable YouTubes, we were pretty sure these things came apart. The down-side is, according to the manufacturer, you lose your “leak-free guarantee.” (Well, it’s probably void after 5 years anyway.)
Readers, it was gross in there. We only do extreme adaptive optics, and this was extremely gross.
It turns out glycol does not enjoy being left in tiny channels without moving for a long time. And, while MagAO-X was on its two year shipping hiatus, it didn’t get the same twice-yearly flushing it evidently needed.
Fortunately, there was a solution: graduate student labor!
Photo by Jared Males
(Just kidding; it was a team effort.)
After we reassembled and pressure tested and reinstalled everything, we had great flow. We also had just spent a few hours on another unscheduled computer disassembly, and had to hustle to get the system ready for Laird and Justin. Fortunately, it was a two-viscacha day, which boded well for our efforts.
Another of today’s wins was figuring out what the “ultra-wide angle” camera on my phone is for: making an already long advisor look even longer.
Once the computers were back online again, we used 2.67 monitors per researcher in hopes of making everything go faster.
Fortunately, installing all of the cables between the electronics rack and the instrument went great. All our movable bits in the instrument moved when we asked them to, so we had time for a bit of sunset-watching before the clean room became part of Laird and Justin’s quarantine bubble.
It only looks like a romantic twosome because Jared had to go take the picture.
I also had a wistful moment, taking a selfie in the tail plate. I definitely did not imagine that I would be the only non-faculty repeat visitor from the original team.
We rigged up a lab laptop (labtop) and left it signed in to Zoom™ so that when Laird and Justin got in we’d see them. We’re remotely supporting their alignment efforts by sitting upstairs with our laptops to move mechanisms as needed. (Or write blog posts, when not needed.)
As I write this, they’re still at it. I admire their tenacity.
Late breaking news: we have pupil images on the pyramid wavefront sensor!
Song of the Day
Your song of the day is brought to you by ~*~*flow*~*~.
We hit a pretty major roadblock today. Due to the sky high fuel prices around the globe, Las Campanas Observatory has had to drastically reduce use of most forms of energy, including electricity for cranes, propane for forklifts, and gas for trucks, except as needed for nighttime operations. So we essentially have no power when the sun is up.
Our assigned backup burrito for the day. She was game, and did all the heavy lifting we usually have a forklift and crane do. Thanks friend.
Luckily, as we always do on the MagAO-X and LCO teams, we had a contingency plan. This is, of course, the burros who hang around the mountain top. The young lady above was our assigned crane and forklift motor for the day. It took a little longer than normal, but in the end we managed to get MagAO-X unpacked.
Sebastiaan inspecting the inside of MagAO-X. Looks like nothing is broken.Our only small problem was a broken dessicant bag, which dribbled a bit. Here Joseph is holding a side panel up while Sebastiaan vacuums.
Yesterday Sebastiaan mentioned our misbehaving graphics cards (GPUs). When we turned the electronics on, 0/4 GPUs that we use for our super-fast real-time computations were alive. There is not much in common between these 4 devices in terms of where/how they are installed, so it seemed extremely unlikely that they all just quit on us. So after none of us getting much sleep, in part due to racing minds trying to figure out what it could be, we divided our duties today so Joseph could troubleshoot while Sebastiaan and I worked with the crew on unpacking.
In the end it was two different problems. One of them just needed a stern talking to (nerd code for we took it out and put it back in, and it worked). The other 3 were more complicated. We use a PCIe expansion system, which lets us attach more crap to our computers. The backplane of that failed, but luckily we had a backup burrito for this too, and once it was put into service we now have all 4 GPUs working.
Our control system electronics, with the comforting glow of a 2080 Ti FTW card visible.
When we went up to the top we saw a vulture soaring on thermals. It was coming right over us, and we could hear a whistling sound from its feathers.
We have some good close ups, but I like this background.
The best news of the day was our friend Vizzy the cleanroom vizcacha making an appearance. This was when I knew the GPUs would be ok.
Vizzy!
The song of the day accurately reflects my sentiments after what was a damn stressful 24 hours.
