MagAO-X is alive! After being boxed up, shipped from Tucson to Phoenix to LA (we think — a little fuzzy) to Miami, with a long pause, then to Santiago, braving the dangers of revolution (and customs (and customs strikes!)), and a trip by truck (always touchy, this time with road blocks!), and finally being craned out of its box and carefully reconstituted, we can still close the loop.
Needless to say we are happy, a little bit relieved, and excited to get our new instrument on the Magellan Clay Telescope.
A clear example of this phenomenon is when we had to roll our electronics rack out of the clean room so Nelson and Emilio could cut off a stripped 3/8-16 bolt. The Aluminium L-bracket you see there is part of the earthquake restraint system, which, as Amali pointed out last night, is always a thing here.A pano of the cleanroom when we were just getting started this morning. That’s Kyle checking on the DM humidity before turning it on.
So here’s the big moment when we finally knew that we didn’t f it all up:
And here you can see our vAPP coronagraphs looking good in a deep exposure:
vAPPs looking eXtreme. Laird wants you to know that there are no optical ghosts in the dark hole. Maybe the eye doctor has a little work to do though.Moonrise over the Andes on my hike up.Laird and Alex watching tonight’s sunset. Photo by Kyle, I’d already given up — obvious non-green-flash night. That’s our electronics shipping crate, btw.
Tonight’s song is “Wanted Dead or Alive”, by Bon Jovi:
Tonight, gentle readers, a tale of drama and derring-do! A mystery was solved, a dastardly plot uncovered, an insidious impostor unmasked, several computers rebooted, a culprit nabbed, and, despite its most vicious efforts, the criminal’s final revenge borne without injury. I present: The Alpaca Saga!
THE CASE
Yesterday night the adaptive secondary — aka adsec, asm, shell, TS1, and as of last week Lollipop — began to display some troubling symptoms. For instance: every so often, even in the gentlest of conditions, the shell would suddenly RIP (don’t worry, the falling-to-pieces is only metaphorical). That in itself isn’t terribly uncommon, but the circumstances were a bit strange: the adsec would briefly report missing values for elevation and for windspeed. A shriek into the void! The wail of an injured creature in the darkness! But then the elevation and windspeed would both return, pretending that nothing was amiss.
Obviously, something was amiss.
Simultaneously, in the once-peaceful environs of the Clio workstation, bayside stage focus moves were being sent, as messenger pigeons into a clear blue sky…and then snapped up by an invisible space dragon, never to return. By which I mean, the bayside stage at the WFS never moved. And yet, it could be moved without problems from the WFS directly.
THE INVESTIGATION
What do absent elevation, unreadable wind speed, and unheard Clio commands have in common? They’re all communications sent through the computer that masterminds the MagAO processes: the Magellan adsec supervisor, or magadsecsup.
A CLUE!
In the dead of night, Jared bravely took his life in his hands and went down to the equipment room — is that its name? the one with the MagAO computers and racks and occasional unstable ladders — anyway, Jared found A CLUE!! Which was: the little blinky lights on the ethernet ports connecting magadsecsup and the network switch were neither happy-working-green or transmitting-information-amber, but instead an alarming shade of ORANGE. So Jared and Emilio beheaded a cable and put it in a different switch port.
But the crimes did not stop!
We hobbled through the night. The adsec wept. The adsec RIPped (still metaphorically). The notion of a criminal in our midst weighed heavily upon our minds. Dawn came and we fell into uneasy slumber…
A BREAKTHROUGH!
During the day, while the rest of us slept, Jared (life still in hands) disconnected magadsecsup. AND YET…he could still ping its IP.
Note, gentle readers: if you remove a computer from the network, you should most certainly NOT be able to ping it. Which meant that, instead of only magadsecsup responding to madadsecsup, we had … AN IMPOSTOR! GASP! SHOCK!
We put the sleuths of LCO onto the case. At dinnertime we received word that the culprit had been found…and it was none other than Alpaca, one of the TCS computers at Baade!
THE PART WHERE THE DETECTIVE EXPLAINS EVERYTHING
Somehow, alpaca.lco.cl had been given the same IP address — one ending in 35 — as is statically assigned to magadsecsup.lco.cl. Consequently, when the Clay TCS sent information — elevation, for instance — to magadsecup, they were sometimes intercepted by this nefarious agent and never made it to the real adsec. Chaos! Upheaval!
Here I provide a helpful diagram to assist in distinguishing a genuine adsec from a sneaky alpaca.
A guide to distinguishing a true adsec from an alpaca impostor.
Anyway, once the impostor was summarily booted from the scene of the IP, and Lollipop comforted and cajoled into returning to service, the night proceeded without further loss of elevation.
Emilio says the real solution is to update the hosts file on alpaca.
ALPACA’S LAST KICK
But! Just as we returned to science, we were subject to one last act of villainy! In a fit of rage and envy, deprived of its chance to be cool like magadsecsup, Alpaca threw a fearsome tantrum. Being somewhat larger, the actuators of an alpaca are capable of a much larger throw, and can cause much larger aberrations. And, because it’s networked, this particular alpaca managed to cause severe turbulence IN THE VERY EARTH, somewhere 33km northeast of Vallenar…
In other words, we had an earthquake! It was fascinating and bizarre! As though the world was swimming. I got to push the “earthquake button” on the adsec GUI, which Laird had pointed out to me only a few hours earlier!
Apparently the “Force enable TSS” button works by setting the shell windspeed reading to 500 m/s. Seeing that number was honestly more alarming than the earthquake…
Anyway, here are a few more helpful diagrams to cement the reader’s understanding of the situation. One is from the Chilean national seismological center. The other is…not.
Why yes, it was a perceptible earthquake. Even the Earthquake Officials agree.ALPACA SMASH
MORAL OF THE STORY
Don’t judge a villain by its actuators. And watch your IPs.
SONG OF THE DAY!
Here’s a song about another creature up to no good. May all of us here at MagAOs both -C and -X have more luck with our problems than the hapless Inspector Clouseau.
Well it was another long but successful day for MagAO-X. We fully integrated the electronics rack with the instrument and began the optical alignment! Impressively, the optical alignment looked almost exactly the way we left it in Tucson. After all that MagAO-X has been through on its way to Chile, we were worried that the inside of MagAO-X would be a mess. However this was not the case, and we were more than pleased by it. We are confident that we will complete the realignment tomorrow and close the loop!
We started the day with our walk up to the “halfway house” at 9:00 am. In the picture below, you can see Jared joined by a friendly goat. Then there’s Kyle catching up, and me not far behind taking the picture.
A typical walk up the hill at Las Campanas Observatory
After some final glycol testing, we started cabling the system. The cables run from the instrument to the electronics rack to power all of our electronics (deformable mirrors, cameras, stages, actuators, etc.).
Laird plugging in the cables
Laird and Jared plugging in cables
We completed all of the cabling except for our “Tweeter” deformable mirror (DM), so Laird and I were able to start on the optical alignment.
Laird working on the optical alignmentLooking good!
Here’s a picture of me with the instrument (front view):
Me and MagAO-X from the front
Here’s a picture of Laird with the instrument (back view). From this point of view it sort of looks like a “dollhouse” of optics. That makes it sound a lot simpler than it looks!
Laird and MagAO-X from the back
Finally, we began cabling the 2,048 actuator DM. This is the part that took us the longest, since cabling 2,000 wires is no easy task. This took multiple iterations in order to make sure all of the pins were aligned with the Samtec Connectors.
Kyle mounting the DM cables to the Samtec ConnectorsEl completo!The fully cabled MagAO-X system
And so that marks the end of a long day for MagAO-X. Tomorrow we may finally close the loop for the first time in Chile!
Just another beautiful sunset at Las Campanas Observatory
The song of the day is going to be Moonlight Serenade, by Glenn Miller:
I’m working on telescope domination this week. I started at Clay using MIKE (well, not really, I got clouded out). I moved to duPont using CAPSCam, I had a night off to visit with the MagAO team, and now I’m back at Clay using MagAO. Tomorrow I’m off to Baade with MagE. Have a telescope or instrument, I’m there!
It’s been a while, so I took last night to get reacquainted with my old friend Clio. It’s like riding a bike (or maybe like being back in the saddle again, though I wouldn’t really know), it turns out. I think I might even have muscle memory for the Camera Control GUI.
Last night was great, but tonight, well, we have some problems. The Clio pupil mask was out of place, though it was fine last night and no one should have touched the motor. Then, the MagAO systems stopped communicating with each other correctly. But by 1 AM, we were back in the saddle again.
So, let’s talk about the weather, animals, and classic songs instead. The sunset was beautiful.
Clouds make for lovely sunsets, and I’m fine with these on-the-horizon beauties as long as they don’t come overhead. If you look very carefully (at least, I could see them on my phone screen), you can see Venus, Jupiter and Saturn along the ecliptic. Yes, our planets seem to have formed in a disk.
We have a big group up here, possibly larger than I’ve seen since my first MagAO runs up here in 2013 or 2014. I love the number of women we’ve got working here. Last night there were none of those beautiful clouds at all, when we gathered on the catwalk at sunset:
Sunset with the whole gang: Logan, Me, Jared, Katie, Amali, Elisabeth, and Emily
This week, I’ve seen a fox, more than a dozen burros, a herd of goats, and at least three vizcachas, so I’m only missing a guanaco to get LCO animal bingo. I have never seen a vizcacha sunning him(her?)self like this before.
A vizcacha sunning itself on the wall next to the Astronomer Support Building this afternoon.
I am using the classic Gene Autry, “Back in the Saddle Again” as my song tonight.
I’m back in the saddle again Out where a friend is a friend [here’s looking at you Katie, Jared, MagAO and Clio] Where the longhornburro cattle feed On the lowly gypsum weed [or whatever those plants are outside] Back in the saddle again
Ridin’ the range sky once more Totin’ my old.44wavefront sensor Where you sleep out everynightday And the onlylaw is rightsky is fey Back in the saddle again
Unpacking Day 2 marks the 3rd day of MagAO-X unpacking activities, which only makes sense if you’re a computer scientist or you just try not to think about it too much.
Alex and Laird began the day by removing the teflon bars that had been installed in front of particularly expensive optics for protection during shipping.
Laird and Alex hard at work in the clean room
Meanwhile, Jared pressure-tested the electronics rack coolant system while I set up the AO Operator Computer.
The PI at work
Mega Desk II: Not Quite Mega Yet (But It Will Be Once We Get That Missing Monitor Bracket)
We filled the instrument coolant lines with glycol and hooked the table to air, giving MagAO-X its first breath and drink at LCO.
The instrument table’s first breath of mountain airFresh glycol adds a much-needed splash of color to the instrument.
After passing its pressure test, the electronics rack coolant system joined in on the activities. Things got a little out of hand when Jared fashioned a makeshift bong and started passing out mixed drinks (glycol and distilled water — not recommended for human consumption).
Jared and Alex fill the liquid-liquid exchanger with the titular bong
The glycol forest in the underbelly of the electronics rack
And, most importantly, the goats made another appearance.
Alex visited with goats during an afternoon stroll.
I present the MagAO-X song of the day: “Always Something There to Remind Me” by Naked Eyes (related to the most recent MagAO-C song by its year of release—a tenuous but legal connection) .
