MagAO-X 2022A Day 4: The Backup Burrito

We hit a pretty major roadblock today. Due to the sky high fuel prices around the globe, Las Campanas Observatory has had to drastically reduce use of most forms of energy, including electricity for cranes, propane for forklifts, and gas for trucks, except as needed for nighttime operations. So we essentially have no power when the sun is up.

Our assigned backup burrito for the day. She was game, and did all the heavy lifting we usually have a forklift and crane do. Thanks friend.

Luckily, as we always do on the MagAO-X and LCO teams, we had a contingency plan. This is, of course, the burros who hang around the mountain top. The young lady above was our assigned crane and forklift motor for the day. It took a little longer than normal, but in the end we managed to get MagAO-X unpacked.

Sebastiaan inspecting the inside of MagAO-X. Looks like nothing is broken.
Our only small problem was a broken dessicant bag, which dribbled a bit. Here Joseph is holding a side panel up while Sebastiaan vacuums.

Yesterday Sebastiaan mentioned our misbehaving graphics cards (GPUs). When we turned the electronics on, 0/4 GPUs that we use for our super-fast real-time computations were alive. There is not much in common between these 4 devices in terms of where/how they are installed, so it seemed extremely unlikely that they all just quit on us. So after none of us getting much sleep, in part due to racing minds trying to figure out what it could be, we divided our duties today so Joseph could troubleshoot while Sebastiaan and I worked with the crew on unpacking.

In the end it was two different problems. One of them just needed a stern talking to (nerd code for we took it out and put it back in, and it worked). The other 3 were more complicated. We use a PCIe expansion system, which lets us attach more crap to our computers. The backplane of that failed, but luckily we had a backup burrito for this too, and once it was put into service we now have all 4 GPUs working.

Our control system electronics, with the comforting glow of a 2080 Ti FTW card visible.

When we went up to the top we saw a vulture soaring on thermals. It was coming right over us, and we could hear a whistling sound from its feathers.

We have some good close ups, but I like this background.

The best news of the day was our friend Vizzy the cleanroom vizcacha making an appearance. This was when I knew the GPUs would be ok.


The song of the day accurately reflects my sentiments after what was a damn stressful 24 hours.