After that we ran up the hill and cabled our 2040 actuator MEMS DM. We got it on the first try! We are getting better at this procedure.
Sebastiaan and I cabling the tweeter. Photo by Joseph Long.Sebastiaan is testing the actuator response to see if we got any cable wrong. We didn’t.
After that, we turned the cleanroom over to our Bubbled comrades. Laird and Justin put in a loooong day of aligning the optics. We sat up in the library and watched. It’s sort of like a medical procedure drama.
Laird and Justin working on MagAO-X.At the end of the day we had a pretty nice PSF. Not perfect, but that’s why we have an AO system. Tomorrow we’ll clean this up with the DMs. That’s Laird at upper right.
We found this book in the library, which we might need if any REAL problems come up.
We don’t use FORTRANA panoramic view of LCO just before the sun dips below the horizon, with Dr. Justin Knight a bubble’s length away from the photographer.
After taking the above photo and circling around Justin, I started down the path to begin my walk up. Then I noticed a sliver of moon hanging over the valley. Then the following ensued:
Joseph got an up close and personal Vizcacha interaction.
It can take a second, but you’ll notice that Vizzy is looking into the library at Joseph.
The reference to bananas in the slack convo is to this blog post from the MagAO-Classic Age of Legends. We have a long history of shenanigans with the cleanroom vizcacha family.
The post-sunset show was extravagant tonight.
LCO, the Moon, and a plane.
Changing up the sound tonight for the song of the day. I think this is the best of the “new” Metallica, as good as the old classics.
We’ve got MagAO-X mostly re-cabled in its temporary home in the LCO cleanroom, and Doctors Close and Knight are fresh off the plane and working on the optical alignment. But, earlier, we had a fun discovery: the instrument control computer (ICC) was getting almost no coolant flow.
Yesterday, we did some brain surgery on the real-time control computer. Today was more like heart surgery. We found that although our pump tried its hardest, almost no liquid coolant made it through the ICC, and temperatures remained stubbornly high. In other words, it was clogged. We really wanted the issue to be anywhere except the CPU liquid cooling block, so of course our troubleshooting pinpointed… the CPU liquid cooling block. Not any of the lines feeding it, but the very center of it.
Disclaimer: this is actually a picture of the other computer, but it looks cooler. They’re basically the same though.
See those three pink hoses in the center, under a bunch of crap? Those go to the blocks we removed. We took the computer out of the rack, the cooling blocks off the computer, then took them both out of the clean room entirely to try and blast the clog free.
But, to no avail.
After consulting reputable YouTubes, we were pretty sure these things came apart. The down-side is, according to the manufacturer, you lose your “leak-free guarantee.” (Well, it’s probably void after 5 years anyway.)
Readers, it was gross in there. We only do extreme adaptive optics, and this was extremely gross.
It turns out glycol does not enjoy being left in tiny channels without moving for a long time. And, while MagAO-X was on its two year shipping hiatus, it didn’t get the same twice-yearly flushing it evidently needed.
Fortunately, there was a solution: graduate student labor!
Photo by Jared Males
(Just kidding; it was a team effort.)
After we reassembled and pressure tested and reinstalled everything, we had great flow. We also had just spent a few hours on another unscheduled computer disassembly, and had to hustle to get the system ready for Laird and Justin. Fortunately, it was a two-viscacha day, which boded well for our efforts.
Another of today’s wins was figuring out what the “ultra-wide angle” camera on my phone is for: making an already long advisor look even longer.
Once the computers were back online again, we used 2.67 monitors per researcher in hopes of making everything go faster.
Fortunately, installing all of the cables between the electronics rack and the instrument went great. All our movable bits in the instrument moved when we asked them to, so we had time for a bit of sunset-watching before the clean room became part of Laird and Justin’s quarantine bubble.
It only looks like a romantic twosome because Jared had to go take the picture.
I also had a wistful moment, taking a selfie in the tail plate. I definitely did not imagine that I would be the only non-faculty repeat visitor from the original team.
We rigged up a lab laptop (labtop) and left it signed in to Zoom™ so that when Laird and Justin got in we’d see them. We’re remotely supporting their alignment efforts by sitting upstairs with our laptops to move mechanisms as needed. (Or write blog posts, when not needed.)
As I write this, they’re still at it. I admire their tenacity.
Late breaking news: we have pupil images on the pyramid wavefront sensor!
Song of the Day
Your song of the day is brought to you by ~*~*flow*~*~.
We hit a pretty major roadblock today. Due to the sky high fuel prices around the globe, Las Campanas Observatory has had to drastically reduce use of most forms of energy, including electricity for cranes, propane for forklifts, and gas for trucks, except as needed for nighttime operations. So we essentially have no power when the sun is up.
Our assigned backup burrito for the day. She was game, and did all the heavy lifting we usually have a forklift and crane do. Thanks friend.
Luckily, as we always do on the MagAO-X and LCO teams, we had a contingency plan. This is, of course, the burros who hang around the mountain top. The young lady above was our assigned crane and forklift motor for the day. It took a little longer than normal, but in the end we managed to get MagAO-X unpacked.
Sebastiaan inspecting the inside of MagAO-X. Looks like nothing is broken.Our only small problem was a broken dessicant bag, which dribbled a bit. Here Joseph is holding a side panel up while Sebastiaan vacuums.
Yesterday Sebastiaan mentioned our misbehaving graphics cards (GPUs). When we turned the electronics on, 0/4 GPUs that we use for our super-fast real-time computations were alive. There is not much in common between these 4 devices in terms of where/how they are installed, so it seemed extremely unlikely that they all just quit on us. So after none of us getting much sleep, in part due to racing minds trying to figure out what it could be, we divided our duties today so Joseph could troubleshoot while Sebastiaan and I worked with the crew on unpacking.
In the end it was two different problems. One of them just needed a stern talking to (nerd code for we took it out and put it back in, and it worked). The other 3 were more complicated. We use a PCIe expansion system, which lets us attach more crap to our computers. The backplane of that failed, but luckily we had a backup burrito for this too, and once it was put into service we now have all 4 GPUs working.
Our control system electronics, with the comforting glow of a 2080 Ti FTW card visible.
When we went up to the top we saw a vulture soaring on thermals. It was coming right over us, and we could hear a whistling sound from its feathers.
We have some good close ups, but I like this background.
The best news of the day was our friend Vizzy the cleanroom vizcacha making an appearance. This was when I knew the GPUs would be ok.
Vizzy!
The song of the day accurately reflects my sentiments after what was a damn stressful 24 hours.
Two and a half years ago, I came to the University of Arizona to work with MagAO-X. The plan was that MagAO-X would go to the Magellan telescope twice a year. And then suddenly a global pandemic appeared. However, right now we are really at Las Campanas Observatory preparing for an observing run! We arrived here on Monday and we entered the Bubble Mode for 3 days. In the Bubble mode we had to stay away from the other staff and got food delivered by room service. Our initial expectation was that we had to stay in this bubble mode for 72 hours after arrival at LCO. But, the clock started counting from arrival in Santiago! We learned this after missing our special breakfast room service. The kitchen staff thought we were already good to go for normal breakfast. So for the final procedure to break out of the bubble, we went to the medical unit to get our COVID test 8 hours earlier than planned.
Here we can see the PI checking in for his COVID test.
All three of us agreed that this was quite an unpleasant nose swab. I think they were trying to scrape out some brain material. So for all other MagAO-X observers; be warned! After getting our clearance, we were allowed to eat in the actual cafeteria during lunch.
The MagAO-X forerunners having their first post-bubble mode lunch together.
With the covid-free clearance, we were allowed to come in contact with other humans. I could finally get a tour of the Magellan telescopes.
Me checking out the Magellan Clay telescope.
And because we were allowed to meet other people, we could actually start unpacking MagAO-X. We met up with the great technical staff of LCO to crane the electronics box that contains the driver electronics of pretty much every part of the instrument. This required a very delicate procedure and took a couple hours. But, we are very happy with the results; the electronics box is unpacked and plugged in.
Taking the electronics box out of its shipping crate.
A happy PI with his electronics box.
But what is a commissioning run without any challenges? Apparently our graphics cards found our plan to observe most disagreeable. Luckily, more team members are flying out tomorrow and they have quickly scavenged the lab in Tucson for more spare graphics cards. We still have more than a week to go for our first night, and enough work to do. It is going to be an exciting run with many interesting projects. Stay tuned!
When we last left LCO we expected to be back in just 3.5 months. We’re nerds of the science and engineering type, so we tend to take lots of notes and pictures because we think we want to remember this or that. But there are always those things where you don’t even think about not remembering it because it’s so obvious, or around here it’s because we have done it that way every 6 months for the last 10 years.
Well, guess what. 2+ years of being totally distracted by global catastrophe fries your brain and you forget a lot of things. So I spent a lot of time today rummaging around all of our junk at LCO trying to piece my brain back together. Several times I came across a nicely labeled box, or a place where someone collected all the bolts that go with the thingy, and said out loud “way to go past us!”.
But also there are cases where there’s nothing on the hard drive and I just stand there like this:
You can briefly see Sebastiaan and Joseph sitting next to the fires of AOC there at the end wondering why I’m mumbling to myself
The big accomplishment of the day was unpacking our AO Operators Computer (a.k.a. AOC, a.k.a. exao1) and getting it set up. Joseph and Sebastiaan assembled Megadesk (without breaking it) and got AOC plugged in, on the network, and monitors sorted out.
Somehow, even after all of this time, you always see something that you haven’t before.
We also had our first Earthquake of the run. I was up in the cleanroom after dinner, and announced it on Slack. Sebastiaan was down in the lodge and saw my slack message before he felt it.
Just a long slow rumble that made the roof rattle, nothing moved though.
We also unveiled the new attire for cool-kids at LCO
MagAO-X rests in the background, waiting for crane day.These are the handiwork of MagAO-X’s own lithography expert Avalon McLeod. I’m 100% sure she’ll write a post about how she does it once she gets down here. (hint hint)
There are some Burros around, as you’ve seen in previous posts. Today I’d like to highlight the Burritos. There is one hanging around with Ma up here at the lodge, and another down at the Bodega.
And of course, there are the sunsets. Oh how I have missed the LCO sunset ritual.
So good to be back.In 10 days we won’t get to see this, cuz we’ll be working already.
Us first 3 have just 1 more day of “bubble”. Assuming negative tests tomorrow evening we’ll get to join normal life at LCO. So far, we’ve just been able to say “hi!!!” to our friends from a distance. We’re really looking forward to actually getting to reconnect.
The song of day’s video sorta captures what it feels like to look back through a decade’s worth of blog posts to find that thing that you forgot.
