Blog

MagAO-C 2019B Day 7: The Alpaca Saga

Tonight, gentle readers, a tale of drama and derring-do! A mystery was solved, a dastardly plot uncovered, an insidious impostor unmasked, several computers rebooted, a culprit nabbed, and, despite its most vicious efforts, the criminal’s final revenge borne without injury. I present: The Alpaca Saga!

THE CASE

Yesterday night the adaptive secondary — aka adsec, asm, shell, TS1, and as of last week Lollipop — began to display some troubling symptoms. For instance: every so often, even in the gentlest of conditions, the shell would suddenly RIP (don’t worry, the falling-to-pieces is only metaphorical). That in itself isn’t terribly uncommon, but the circumstances were a bit strange: the adsec would briefly report missing values for elevation and for windspeed. A shriek into the void! The wail of an injured creature in the darkness! But then the elevation and windspeed would both return, pretending that nothing was amiss.

Obviously, something was amiss.

Simultaneously, in the once-peaceful environs of the Clio workstation, bayside stage focus moves were being sent, as messenger pigeons into a clear blue sky…and then snapped up by an invisible space dragon, never to return. By which I mean, the bayside stage at the WFS never moved. And yet, it could be moved without problems from the WFS directly.

THE INVESTIGATION

What do absent elevation, unreadable wind speed, and unheard Clio commands have in common? They’re all communications sent through the computer that masterminds the MagAO processes: the Magellan adsec supervisor, or magadsecsup.

A CLUE!

In the dead of night, Jared bravely took his life in his hands and went down to the equipment room — is that its name? the one with the MagAO computers and racks and occasional unstable ladders — anyway, Jared found A CLUE!! Which was: the little blinky lights on the ethernet ports connecting magadsecsup and the network switch were neither happy-working-green or transmitting-information-amber, but instead an alarming shade of ORANGE. So Jared and Emilio beheaded a cable and put it in a different switch port.

But the crimes did not stop!

We hobbled through the night. The adsec wept. The adsec RIPped (still metaphorically). The notion of a criminal in our midst weighed heavily upon our minds. Dawn came and we fell into uneasy slumber…

A BREAKTHROUGH!

During the day, while the rest of us slept, Jared (life still in hands) disconnected magadsecsup. AND YET…he could still ping its IP.

Note, gentle readers: if you remove a computer from the network, you should most certainly NOT be able to ping it. Which meant that, instead of only magadsecsup responding to madadsecsup, we had … AN IMPOSTOR! GASP! SHOCK!

We put the sleuths of LCO onto the case. At dinnertime we received word that the culprit had been found…and it was none other than Alpaca, one of the TCS computers at Baade!

THE PART WHERE THE DETECTIVE EXPLAINS EVERYTHING

Somehow, alpaca.lco.cl had been given the same IP address — one ending in 35 — as is statically assigned to magadsecsup.lco.cl. Consequently, when the Clay TCS sent information — elevation, for instance — to magadsecup, they were sometimes intercepted by this nefarious agent and never made it to the real adsec. Chaos! Upheaval!

Here I provide a helpful diagram to assist in distinguishing a genuine adsec from a sneaky alpaca.

A diagram pointing out the important differences between an adaptive secondary mirror and an alpaca. The alpaca can be distinguished by its relative fluffiness, envious glances, hoard of stolen commands, and 581 fewer actuators.
A guide to distinguishing a true adsec from an alpaca impostor.

Anyway, once the impostor was summarily booted from the scene of the IP, and Lollipop comforted and cajoled into returning to service, the night proceeded without further loss of elevation.

Emilio says the real solution is to update the hosts file on alpaca.

ALPACA’S LAST KICK

But! Just as we returned to science, we were subject to one last act of villainy! In a fit of rage and envy, deprived of its chance to be cool like magadsecsup, Alpaca threw a fearsome tantrum. Being somewhat larger, the actuators of an alpaca are capable of a much larger throw, and can cause much larger aberrations. And, because it’s networked, this particular alpaca managed to cause severe turbulence IN THE VERY EARTH, somewhere 33km northeast of Vallenar…

In other words, we had an earthquake! It was fascinating and bizarre! As though the world was swimming. I got to push the “earthquake button” on the adsec GUI, which Laird had pointed out to me only a few hours earlier!

Apparently the “Force enable TSS” button works by setting the shell windspeed reading to 500 m/s. Seeing that number was honestly more alarming than the earthquake…

Anyway, here are a few more helpful diagrams to cement the reader’s understanding of the situation. One is from the Chilean national seismological center. The other is…not.

A chart from a Chilean earthquake tracking website. Magnitude 4.8, 33km NE of Vallenar.
Why yes, it was a perceptible earthquake. Even the Earthquake Officials agree.
An alpaca throwing a tantrum because it can't have IP address 35. A truly indignant alpaca may engage EARTHQUAKE MODE!!!
ALPACA SMASH

MORAL OF THE STORY

Don’t judge a villain by its actuators. And watch your IPs.

SONG OF THE DAY!

Here’s a song about another creature up to no good. May all of us here at MagAOs both -C and -X have more luck with our problems than the hapless Inspector Clouseau.

MagAO-X 2019B Unpacking Day 3: Cabling

Well it was another long but successful day for MagAO-X. We fully integrated the electronics rack with the instrument and began the optical alignment! Impressively, the optical alignment looked almost exactly the way we left it in Tucson. After all that MagAO-X has been through on its way to Chile, we were worried that the inside of MagAO-X would be a mess. However this was not the case, and we were more than pleased by it. We are confident that we will complete the realignment tomorrow and close the loop!

We started the day with our walk up to the “halfway house” at 9:00 am. In the picture below, you can see Jared joined by a friendly goat. Then there’s Kyle catching up, and me not far behind taking the picture.

A typical walk up the hill at Las Campanas Observatory

After some final glycol testing, we started cabling the system. The cables run from the instrument to the electronics rack to power all of our electronics (deformable mirrors, cameras, stages, actuators, etc.).

We completed all of the cabling except for our “Tweeter” deformable mirror (DM), so Laird and I were able to start on the optical alignment.

Laird working on the optical alignment
Looking good!

Here’s a picture of me with the instrument (front view):

Me and MagAO-X from the front

Here’s a picture of Laird with the instrument (back view). From this point of view it sort of looks like a “dollhouse” of optics. That makes it sound a lot simpler than it looks!

Laird and MagAO-X from the back

Finally, we began cabling the 2,048 actuator DM. This is the part that took us the longest, since cabling 2,000 wires is no easy task. This took multiple iterations in order to make sure all of the pins were aligned with the Samtec Connectors.

Kyle mounting the DM cables to the Samtec Connectors
El completo!
The fully cabled MagAO-X system

And so that marks the end of a long day for MagAO-X. Tomorrow we may finally close the loop for the first time in Chile!

Just another beautiful sunset at Las Campanas Observatory

The song of the day is going to be Moonlight Serenade, by Glenn Miller:

MagAO-C 2019B Day 6: Back in the Saddle Again

I’m working on telescope domination this week. I started at Clay using MIKE (well, not really, I got clouded out). I moved to duPont using CAPSCam, I had a night off to visit with the MagAO team, and now I’m back at Clay using MagAO. Tomorrow I’m off to Baade with MagE. Have a telescope or instrument, I’m there!

It’s been a while, so I took last night to get reacquainted with my old friend Clio. It’s like riding a bike (or maybe like being back in the saddle again, though I wouldn’t really know), it turns out. I think I might even have muscle memory for the Camera Control GUI.

Last night was great, but tonight, well, we have some problems. The Clio pupil mask was out of place, though it was fine last night and no one should have touched the motor. Then, the MagAO systems stopped communicating with each other correctly. But by 1 AM, we were back in the saddle again.

So, let’s talk about the weather, animals, and classic songs instead. The sunset was beautiful.

Clouds make for lovely sunsets, and I’m fine with these on-the-horizon beauties as long as they don’t come overhead. If you look very carefully (at least, I could see them on my phone screen), you can see Venus, Jupiter and Saturn along the ecliptic. Yes, our planets seem to have formed in a disk.

We have a big group up here, possibly larger than I’ve seen since my first MagAO runs up here in 2013 or 2014. I love the number of women we’ve got working here. Last night there were none of those beautiful clouds at all, when we gathered on the catwalk at sunset:

a whole lot of AO operators and astronomers
Sunset with the whole gang: Logan, Me, Jared, Katie, Amali, Elisabeth, and Emily

This week, I’ve seen a fox, more than a dozen burros, a herd of goats, and at least three vizcachas, so I’m only missing a guanaco to get LCO animal bingo. I have never seen a vizcacha sunning him(her?)self like this before.

A vizcacha sunning itself on the wall next to the Astronomer Support Building this afternoon.

I am using the classic Gene Autry, “Back in the Saddle Again” as my song tonight.

I’m back in the saddle again
Out where a friend is a friend [here’s looking at you Katie, Jared, MagAO and Clio]
Where the longhorn burro cattle feed
On the lowly gypsum weed [or whatever those plants are outside]
Back in the saddle again

Ridin’ the range sky once more
Totin’ my old .44 wavefront sensor
Where you sleep out every night day
And the only law is right sky is fey
Back in the saddle again

Gene Autry – Back in the Saddle Again

MagAO-X 2019B Unpacking Day 2: MagAO-X Does Bongs

Unpacking Day 2 marks the 3rd day of MagAO-X unpacking activities, which only makes sense if you’re a computer scientist or you just try not to think about it too much.

Alex and Laird began the day by removing the teflon bars that had been installed in front of particularly expensive optics for protection during shipping.

Laird and Alex hard at work in the clean room

Meanwhile, Jared pressure-tested the electronics rack coolant system while I set up the AO Operator Computer.

We filled the instrument coolant lines with glycol and hooked the table to air, giving MagAO-X its first breath and drink at LCO.

The instrument table’s first breath of mountain air
Fresh glycol adds a much-needed splash of color to the instrument.

After passing its pressure test, the electronics rack coolant system joined in on the activities. Things got a little out of hand when Jared fashioned a makeshift bong and started passing out mixed drinks (glycol and distilled water — not recommended for human consumption).

And, most importantly, the goats made another appearance.

Alex visited with goats during an afternoon stroll.

I present the MagAO-X song of the day: “Always Something There to Remind Me” by Naked Eyes (related to the most recent MagAO-C song by its year of release—a tenuous but legal connection) .

MagAO-C 2019B Day 5: Alycia and the Baysides

Tonight was Alycia’s first night. I told Emily it would be more relaxed than last night with all Jayne’s weird rotations because Alycia’s observations are more in MagAO’s wheelhouse — Alycia is searching for faint companions; keeping the rotator off for ADI mode; and is a Clio and general observational expert. But… hah.

So on the first Pre-set of the night, the Bayside-X stage decided to not work. There are 3 “Bayside” stages — they are the motors that move the AO system around so that the star is always on the tip of the pyramid WFS even when the star needs to be nodded around the detector on Clio for sky subtraction. We have replaced the X and Y stages with higher-power versions after we had observers who wanted to keep the rotator on and tracking, which caused the stages to overheat and fail. So I wonder if it’s a coincidence that we spent all last night with the rotator on, putting weight on the X-stage, and then tonight the X-stage refused to move…

Laird and Jared are now on pseudo-day schedules (see Alex’s great MagAO-X post about the unpacking today!) but this happened at the start of the night so Laird was still at the summit and Jared was down in the clean room with MagAO-X but he came up to help post-mortem. Laird went out to the platform with a walkie-talkie and we did the usual dance of moving the motors from the control room and Laird telling us what he saw or heard in the Nas. The current would spike but the motor wouldn’t move. Small movements and homing didn’t work; eventually we power-cycled them and in the end the X-stage did start responding again.

WFS hardware GUI, on the Bayside Stages tab. Each stage can be moved manually here, and the position is reported in mm and current in amps. The Bayside X and Y stages nod in Y and X on Clio, respectively, and the Bayside Z stage moves the W-unit towards or away from Clio, effectively producing a focus movement for Clio. Amali edited the code to enable “Control always active” on the X-stage after a nod rather than “Stage” (brakes). [Image description: A GUI on a Linux computer with status and function boxes and buttons.]

But Laird heard a noise that made him think there was some friction affecting the motion of the X-stage. Since the stage worked normally after the power-cycle, we started thinking about all the work we have done on the Baysides over the years. Originally we used the setting “Control always active” as that would position them precisely to the micron. But this was thought to contribute to the over-heating problem when we had high currents at strange rotator angles with all the weight of the instrument on the Y-stage. So we swapped the original out for the higher-power version, but we also started applying the brakes and turning off the active control after each move (“Stage” check box). But now we are thinking… maybe the brakes themselves are starting to fail, and the friction was the X-stage rubbing along its own brake. So tonight, Amali went into the code and reverted the procedure for the X-stage to now enable “Control always active” rather than enabling “Stage” (applying the brakes) after each move. And all stages worked perfectly for the rest of the night.

This is the “Board GUI” that shows the current state of the various subsystems on the W-unit, which includes the WFS (left) and VisAO (right). The rectangle at the bottom shows the current position of the pyramid with respect to the telescope center. The Bayside X, Y, and Z positions are listed. The entire W-unit is moved relative to Clio when we move the Bayside motors. [Image description: An optical diagram of the WFS and VisAO. Light from the telescope comes in from the top center, goes through the ADCs, and then is split by the selectable beamsplitter such that some portion goes to the right to VisAO and some portion goes to the left to the WFS. The WFS arm includes a rerotator that keeps the pupils aligned to the actuators, and a camera lens that keeps the pupils registered to the pixels on the CCD39. The VisAO arm includes science filters. Below the optical diagram is a rectangle with cross-hairs and a red dot showing the relative position of the pyramid.]

After Jared and Laird went to bed, at one point our TO Alberto went down to get some coffee, and I snapped this pic of the all-women control room. What a big difference from when I was the “Only Girl In the World” at LCO!

Astronomers hard at work. From left: Alycia, Logan, Emily, Amali. [Image description: Alycia is on her phone (don’t worry, Clio and VisAO are actively taking data). Logan is analyzing some data on her laptop. Emily is studying the telescope system to make sure she understands the big picture of MagAO. Amali is on her laptop debugging MagAO software, probably the FITS headers of Clio that aren’t being fully populated with the correct AO parameters.]
Vizzy by the library on my way up to the top. [Image description: A Vizcacha sits on a window ledge in the diffuse glow of pre-sunset.]
The Clay telescope at sunrise. [Image description: Photo taken from slightly downhill of Clay, the dome is to the side glinting in the sunrise while the sky has a beautiful stripey glow.]
Walking back to the village at dawn. [Image description: A cluster of terracotta-roofed buildings down a slope with a winding road. Beautiful mountains and hills. Quiet and still.]
The telescope ridges at dawn. [Image description: A stripey glow on the horizon. A mountain peak and a ridge are both covered in telescopes that are closed up to sleep for the day.]

Classic MagAO song from Day 40 of the super long cold winter run with so many technical problems and such long nights that I actually worked, really worked, 112 hours a week (trouble-shooting all afternoon, observing all night, and choosing each day either a shower or dinner): This Must Be the Place/ Naive Melody by the Talking Heads: