Blog

I was mistaken, but now I’m not: Sebastiaan, Avalon and I are the only team members on this run who haven’t been to Las Campanas Observatory (LCO) before. However, perhaps situation this extends to at least some of the avid blog readers, so I’d like to share what some of my experience has been like thus far.

Before that, I’d like to list a couple of additional tips to Joseph’s travel steps on the way to LCO, and then out of the bubble.

Bonus step:

Our last two comrades, Logan and Avalon, are due to arrive tomorrow, and when they are on the driving portion of the trip from La Serena to LCO they may be asked to stop and fill out the daily health check form in the first guard post building they encounter. That was something new to Laird and me as Joseph and company were only asked to do so once at LCO. It may behoove Logan and Avalon to preemptively fill out the form ahead of time while at the La Serena airport (if you can connect to WiFi, which I couldn’t) with something that looks like this:

• passport ID#
• place of work (mountain)
• any of the following symptoms (no symptoms)
• community exhibition (none of the above)

Now as far as the bubble procedure goes, you’re more or less relegated to your room and the ample outdoor spaces appropriately distanced from everyone – except maybe your bubble cohort(s). When it is your time, i.e. the morning of the 4th day you’ve been at LCO, you will be beckoned to the paramedic on the mountain who resides in the bodega (image featured below).

More precisely you walk to the portable building next to a warehouse and behind the ambulance (also featured below) where you’ll receive a rapid antigen test. Five minutes later you should theoretically have a negative test result and you’ll be free to move about the mountain facilities with a mask on.

As this was the case for Laird and me, we went back to our rooms for a short spell, and then reunited with the rest of the MagAO-X team at lunch time.

Now since it was and remains irrefutably true that the answer to Laird’s blog post from yesterday is yes, I had time to explore LCO today. (At least the pupil vignetting is no longer an issue, but what’s a few optical hardware re-positionings amongst friends? Nothing, right? Okay, so not nothing… we’re definitely going to realign the position of the science camera beam-splitter and possibly the LOWFS camera position tomorrow, but we’re practically done after that…) Here are my results:

I went for a run, and I found some lively road blocks.

If I’m not mistaken, I made my way to the site of the 40″ and 100″ telescopes where I found another surprise.

A Guanaco! Unfortunately I could only get within about 80 feet of it before it started to move away from me, so apologies for the poor image quality.

And finally after dinner while walking with Jared back to the cleanroom where MagAO-X currently resides, we snapped a few pictures of a culpeo. Jared definitely captured the best culpeo still, so kudos to him for lending it out for the blog.

I think that’s all of the animals I’m likely to see minus the airborne ones, so not bad for one day. And for anyone wondering about the viscachas, I was actually with Laird when he snagged that viscacha duo photo – it counts!

I’ve lived in Tucson for a while now, so I’m definitely used to some spaced out housing outside of city limits, but scenes like this are completely foreign to me:

In other words, we’re not in the suburbs anymore (Arcade Fire, The Suburbs).

So for the last 48 hours Justin Knight and I have been busy working hard at the optical alignment of MagAO-X. On Saturday, after 30 hours of travelling we arrived at LCO, and had a great dinner (delivered to our door here at LCO as we are in the bubble for the next 3 days). Then right after dinner we inserted the upper periscope mirror and aligned the upper optical table of MagAO-X to the lower table. This then allowed the rest of the team (led by Jared) to cable and optically test the Tweeter connections (with the PyWFS camera) on Sunday.

After some much needed sleep we checked all the optical beam paths on Sunday afternoon. We also set up the PEPS II floating table control system — which is cool since it allows the whole MagAO-X to float on a cushion of air — eliminating vibrations from the lab floor. That allows for really excellent stable images (no burring due to floor vibrations shaking optics).

The optics were well aligned with our bright laser (or at least we thought so). But… later that night we looked with our default white light laser and found that there was slight “clipping” of one beam in one place somewhere in the 7.5 meter optical train. But it wasn’t obvious where it was on Sunday night (and we were tired since it was midnight).

But by Monday morning (instead of working on my telescope proposal that was due at noon today) I realized where exactly the “clipping” was likely coming from. Indeed Justin and found this and fixed it (see photo below).

The beam was a few mm to far to the right and was just slightly clipping one of the mounts of the PyWFS optical train (there are 42 optical surfaces in MagAO-X’s science optical train –it can be exciting tracking just one slight optical clipping down!). But one way another we knew we’d find it eventually.

Anyways MagAO-X is realigned with lots of room for the beam to move around without any similar clipping (see photo above). To test this Jared rotated the K-mirror so that we simulated the Magellan telescope to move from vertical pointing to just 30 degrees above the horizon (K-mirror rotated from -55 to -25 deg) . Even though the beam sweeps through a large volume in this full simulation of observing conditions –no other “beam clipping” were observed. So the optical alignment looks like it is ready to go on sky. Now MagAO-X is dust proof with all its panels on. Sebastiaan was able to very quickly make a nearly perfectly flat wavefront. So like MagAO-X is making “outer space-quality” images–at least in the lab (see photo below).

But perhaps the most import data I took so far this run (so far) is seeing two vizzys just hanging out at the clean room

Still lots of work to do, but the optical alignment phase is mostly over until we align to the telescope on Saturday!

So here is my song of the day. One way or another we where going to align MagAO-X’s twin optical beams into parallel lines…

My day started with a classic LCO breakfast.

After that we ran up the hill and cabled our 2040 actuator MEMS DM. We got it on the first try! We are getting better at this procedure.

After that, we turned the cleanroom over to our Bubbled comrades. Laird and Justin put in a loooong day of aligning the optics. We sat up in the library and watched. It’s sort of like a medical procedure drama.

We found this book in the library, which we might need if any REAL problems come up.

After taking the above photo and circling around Justin, I started down the path to begin my walk up. Then I noticed a sliver of moon hanging over the valley. Then the following ensued:

Joseph got an up close and personal Vizcacha interaction.

The reference to bananas in the slack convo is to this blog post from the MagAO-Classic Age of Legends. We have a long history of shenanigans with the cleanroom vizcacha family.

The post-sunset show was extravagant tonight.

Changing up the sound tonight for the song of the day. I think this is the best of the “new” Metallica, as good as the old classics.

We’ve got MagAO-X mostly re-cabled in its temporary home in the LCO cleanroom, and Doctors Close and Knight are fresh off the plane and working on the optical alignment. But, earlier, we had a fun discovery: the instrument control computer (ICC) was getting almost no coolant flow.

Yesterday, we did some brain surgery on the real-time control computer. Today was more like heart surgery. We found that although our pump tried its hardest, almost no liquid coolant made it through the ICC, and temperatures remained stubbornly high. In other words, it was clogged. We really wanted the issue to be anywhere except the CPU liquid cooling block, so of course our troubleshooting pinpointed… the CPU liquid cooling block. Not any of the lines feeding it, but the very center of it.

See those three pink hoses in the center, under a bunch of crap? Those go to the blocks we removed. We took the computer out of the rack, the cooling blocks off the computer, then took them both out of the clean room entirely to try and blast the clog free.

But, to no avail.

After consulting reputable YouTubes, we were pretty sure these things came apart. The down-side is, according to the manufacturer, you lose your “leak-free guarantee.” (Well, it’s probably void after 5 years anyway.)

Readers, it was gross in there. We only do extreme adaptive optics, and this was extremely gross.

It turns out glycol does not enjoy being left in tiny channels without moving for a long time. And, while MagAO-X was on its two year shipping hiatus, it didn’t get the same twice-yearly flushing it evidently needed.

Fortunately, there was a solution: graduate student labor!

(Just kidding; it was a team effort.)

After we reassembled and pressure tested and reinstalled everything, we had great flow. We also had just spent a few hours on another unscheduled computer disassembly, and had to hustle to get the system ready for Laird and Justin. Fortunately, it was a two-viscacha day, which boded well for our efforts.

Another of today’s wins was figuring out what the “ultra-wide angle” camera on my phone is for: making an already long advisor look even longer.

Once the computers were back online again, we used 2.67 monitors per researcher in hopes of making everything go faster.

Fortunately, installing all of the cables between the electronics rack and the instrument went great. All our movable bits in the instrument moved when we asked them to, so we had time for a bit of sunset-watching before the clean room became part of Laird and Justin’s quarantine bubble.

I also had a wistful moment, taking a selfie in the tail plate. I definitely did not imagine that I would be the only non-faculty repeat visitor from the original team.

We rigged up a lab laptop (labtop) and left it signed in to Zoom™ so that when Laird and Justin got in we’d see them. We’re remotely supporting their alignment efforts by sitting upstairs with our laptops to move mechanisms as needed. (Or write blog posts, when not needed.)

As I write this, they’re still at it. I admire their tenacity.

Late breaking news: we have pupil images on the pyramid wavefront sensor!

Song of the Day

Your song of the day is brought to you by ~*~*flow*~*~.

We hit a pretty major roadblock today. Due to the sky high fuel prices around the globe, Las Campanas Observatory has had to drastically reduce use of most forms of energy, including electricity for cranes, propane for forklifts, and gas for trucks, except as needed for nighttime operations. So we essentially have no power when the sun is up.

Luckily, as we always do on the MagAO-X and LCO teams, we had a contingency plan. This is, of course, the burros who hang around the mountain top. The young lady above was our assigned crane and forklift motor for the day. It took a little longer than normal, but in the end we managed to get MagAO-X unpacked.

Yesterday Sebastiaan mentioned our misbehaving graphics cards (GPUs). When we turned the electronics on, 0/4 GPUs that we use for our super-fast real-time computations were alive. There is not much in common between these 4 devices in terms of where/how they are installed, so it seemed extremely unlikely that they all just quit on us. So after none of us getting much sleep, in part due to racing minds trying to figure out what it could be, we divided our duties today so Joseph could troubleshoot while Sebastiaan and I worked with the crew on unpacking.

In the end it was two different problems. One of them just needed a stern talking to (nerd code for we took it out and put it back in, and it worked). The other 3 were more complicated. We use a PCIe expansion system, which lets us attach more crap to our computers. The backplane of that failed, but luckily we had a backup burrito for this too, and once it was put into service we now have all 4 GPUs working.

When we went up to the top we saw a vulture soaring on thermals. It was coming right over us, and we could hear a whistling sound from its feathers.

The best news of the day was our friend Vizzy the cleanroom vizcacha making an appearance. This was when I knew the GPUs would be ok.

The song of the day accurately reflects my sentiments after what was a damn stressful 24 hours.