MagAO-C 2019B Day 8: Life At LCO

Tonight was finally the promised night of relaxed AO operation after multiple nights of troubleshooting. It is much more to an AO operators delight to only need to open and close the loop for new targets with the occasional MatLab pop up forcing a reset on the tip tilt. Alberto, our telescope operator, was able to tell us his operational comedies of the past and share pictures of his lovely family. Later today is the Mercury transit! Tonight is also my first blog post so an introduction to the outside world is due.

I’m Emily Mailhot, a new AO operator for not only MagAO-C but also LBTI and eventually MMT Maps, both in Arizona. My equally new counterpart, Jared Carlson, and I have been learning the ins and outs of adaptive secondary operations, concepts, and frustrations – and are excited to do so.

This trip to LCO has created numerous “first experiences” for me personally. First time leaving the United States, first time on a B787-8, first time seeing the southern sky, first time experiencing sub-arcsecond seeing (both AO operationally and visually), the list goes on. Considering this, a life at LCO post from brand new eyes seemed appropriate.

Food at LCO

When I was informed that our meals would be served on site I was both relieved and skeptical. At LBTI we are responsible for feeding ourselves through the night, vending machines on the 5th floor of Steward Observatory get a lot of traffic. But would this provided food be delightful or would I need to start preparing for numerous high school band-camp cafeteria flashbacks? I am happy to announce that the food is absolutely superb. From the empanadas to the night lunches to the cookie filled dome kitchens – you have to try very hard to go hungry. Meals are at set times, 730 breakfast, 1230 lunch, and 1830 dinner, but for most of the run breakfast and lunch is sleep for the observers. Instead we fill out night lunch forms which are located in the main kitchen for observers to fill out by 1600 each day. It’s a checklist of various sandwich combinations or your can write “plato de cena” if you’d prefer a plate of that evenings dinner. It is the telescope operators job to collect all the night lunches after dinner and bring them to the telescope.

For those at home wondering what we are eating here’s a selection:

My first empanada!
Empanadas are only served on Sunday for lunch so you better get your fill while you can. They come in two forms, carne or con queso, both equally delicious. This particular Sunday there was also veggie and seafood soup.

Clearly more than enough food to go around.

Wildlife at LCO

The wildlife here at the summit is diverse. This blog has many posts about the endearing vizzy’s, but the burros, horses, and guanacos are just as abundant. Today on a run around the grounds I encountered both guanacos and horses running along the ridge below the road. They seemed to be chasing each other in what resembled a wild wild west setting. It was so magnificent to capture that I stopped the video clip too soon and I instead watched in awe.

On another run earlier in the week, towards the du Pont Telescope, I encountered a large herd of burros controlling the road. One was particularly annoyed that I was trying to pass and gave me a stiff snort.

"I'm watching you, Wazowski. Always watching."
A herd of burros serving a strong stare-down.

Of course, this wouldn’t be complete without a vizzy. This sunrise viscacha (a Chilean cocktail?!) watched us sleepy observers wander back down the summit, only to be met by this eight legged, fanged creature. He was small but mighty, creeping up toward my hand each time I tried to open my door. Not usually being afraid of spiders, a small fear for them was gained after the long dance it took to be allowed into my home, room 5.

The fox on the other hand kept distance and the goats clumsily climbed to the summit with us.

The goats followed us all the way to the Clay Telescope dome where inside the control room us tired observers tried to troubleshoot through our engineering night after a long install day.

Work at LCO

Of course the animals only provide us entertainment between times of work – like our extremely long install day and engineering night (see Day 3!). Here’s a glimpse of the Clay control room, AO work station, and some pre-science pupils.

Smoking with science.
An image of the control room before sunset. The back left station is for science camera, Clio. To its right is the telescope operators station. Left front is the VisAO station and finally next to it is the adaptive operators station. During troubleshooting events this room can be full of people, standing room only.
A well behaved shell.
The AO station! GUI’s upon GUI’s upon GUI’s.

Beauty at LCO

LCO is nestled in the most beautiful landscape, best described through photographs.

Pretty in pink.
The first sunset of the trip from the porch of the “hotel” dining room.
Looking out towards to Andes.
Taken from behind the Summit near sunset. This was the first clear night since we had arrived.
The peak for the future Giant Magellan Telescope taken from the telescope operators housing.
Hair of JJ lens flare
Clay dome open for science.
Clay + ASM appreciation pic
Clay still at the zenith, the ASM backlit by the sky.
Baade and Clay awaiting a night of science.
Orange mountains
A picture of the open dome from the nasmyth platform, taken during the night of the misbehaving baysides.

And my favorite photograph of all, taken after dinner on our long day transitioning into our long night.

Stairway to Heaven
The stairway to the stars.

Keeping with classic song theme, here’s Led Zeppelin performing Stairway to Heaven.

MagAO-C 2019B Day 7: The Alpaca Saga

Tonight, gentle readers, a tale of drama and derring-do! A mystery was solved, a dastardly plot uncovered, an insidious impostor unmasked, several computers rebooted, a culprit nabbed, and, despite its most vicious efforts, the criminal’s final revenge borne without injury. I present: The Alpaca Saga!


Yesterday night the adaptive secondary — aka adsec, asm, shell, TS1, and as of last week Lollipop — began to display some troubling symptoms. For instance: every so often, even in the gentlest of conditions, the shell would suddenly RIP (don’t worry, the falling-to-pieces is only metaphorical). That in itself isn’t terribly uncommon, but the circumstances were a bit strange: the adsec would briefly report missing values for elevation and for windspeed. A shriek into the void! The wail of an injured creature in the darkness! But then the elevation and windspeed would both return, pretending that nothing was amiss.

Obviously, something was amiss.

Simultaneously, in the once-peaceful environs of the Clio workstation, bayside stage focus moves were being sent, as messenger pigeons into a clear blue sky…and then snapped up by an invisible space dragon, never to return. By which I mean, the bayside stage at the WFS never moved. And yet, it could be moved without problems from the WFS directly.


What do absent elevation, unreadable wind speed, and unheard Clio commands have in common? They’re all communications sent through the computer that masterminds the MagAO processes: the Magellan adsec supervisor, or magadsecsup.


In the dead of night, Jared bravely took his life in his hands and went down to the equipment room — is that its name? the one with the MagAO computers and racks and occasional unstable ladders — anyway, Jared found A CLUE!! Which was: the little blinky lights on the ethernet ports connecting magadsecsup and the network switch were neither happy-working-green or transmitting-information-amber, but instead an alarming shade of ORANGE. So Jared and Emilio beheaded a cable and put it in a different switch port.

But the crimes did not stop!

We hobbled through the night. The adsec wept. The adsec RIPped (still metaphorically). The notion of a criminal in our midst weighed heavily upon our minds. Dawn came and we fell into uneasy slumber…


During the day, while the rest of us slept, Jared (life still in hands) disconnected magadsecsup. AND YET…he could still ping its IP.

Note, gentle readers: if you remove a computer from the network, you should most certainly NOT be able to ping it. Which meant that, instead of only magadsecsup responding to madadsecsup, we had … AN IMPOSTOR! GASP! SHOCK!

We put the sleuths of LCO onto the case. At dinnertime we received word that the culprit had been found…and it was none other than Alpaca, one of the TCS computers at Baade!


Somehow, had been given the same IP address — one ending in 35 — as is statically assigned to Consequently, when the Clay TCS sent information — elevation, for instance — to magadsecup, they were sometimes intercepted by this nefarious agent and never made it to the real adsec. Chaos! Upheaval!

Here I provide a helpful diagram to assist in distinguishing a genuine adsec from a sneaky alpaca.

A diagram pointing out the important differences between an adaptive secondary mirror and an alpaca. The alpaca can be distinguished by its relative fluffiness, envious glances, hoard of stolen commands, and 581 fewer actuators.
A guide to distinguishing a true adsec from an alpaca impostor.

Anyway, once the impostor was summarily booted from the scene of the IP, and Lollipop comforted and cajoled into returning to service, the night proceeded without further loss of elevation.

Emilio says the real solution is to update the hosts file on alpaca.


But! Just as we returned to science, we were subject to one last act of villainy! In a fit of rage and envy, deprived of its chance to be cool like magadsecsup, Alpaca threw a fearsome tantrum. Being somewhat larger, the actuators of an alpaca are capable of a much larger throw, and can cause much larger aberrations. And, because it’s networked, this particular alpaca managed to cause severe turbulence IN THE VERY EARTH, somewhere 33km northeast of Vallenar…

In other words, we had an earthquake! It was fascinating and bizarre! As though the world was swimming. I got to push the “earthquake button” on the adsec GUI, which Laird had pointed out to me only a few hours earlier!

Apparently the “Force enable TSS” button works by setting the shell windspeed reading to 500 m/s. Seeing that number was honestly more alarming than the earthquake…

Anyway, here are a few more helpful diagrams to cement the reader’s understanding of the situation. One is from the Chilean national seismological center. The other is…not.

A chart from a Chilean earthquake tracking website. Magnitude 4.8, 33km NE of Vallenar.
Why yes, it was a perceptible earthquake. Even the Earthquake Officials agree.
An alpaca throwing a tantrum because it can't have IP address 35. A truly indignant alpaca may engage EARTHQUAKE MODE!!!


Don’t judge a villain by its actuators. And watch your IPs.


Here’s a song about another creature up to no good. May all of us here at MagAOs both -C and -X have more luck with our problems than the hapless Inspector Clouseau.

MagAO-C 2019B Day 6: Back in the Saddle Again

I’m working on telescope domination this week. I started at Clay using MIKE (well, not really, I got clouded out). I moved to duPont using CAPSCam, I had a night off to visit with the MagAO team, and now I’m back at Clay using MagAO. Tomorrow I’m off to Baade with MagE. Have a telescope or instrument, I’m there!

It’s been a while, so I took last night to get reacquainted with my old friend Clio. It’s like riding a bike (or maybe like being back in the saddle again, though I wouldn’t really know), it turns out. I think I might even have muscle memory for the Camera Control GUI.

Last night was great, but tonight, well, we have some problems. The Clio pupil mask was out of place, though it was fine last night and no one should have touched the motor. Then, the MagAO systems stopped communicating with each other correctly. But by 1 AM, we were back in the saddle again.

So, let’s talk about the weather, animals, and classic songs instead. The sunset was beautiful.

Clouds make for lovely sunsets, and I’m fine with these on-the-horizon beauties as long as they don’t come overhead. If you look very carefully (at least, I could see them on my phone screen), you can see Venus, Jupiter and Saturn along the ecliptic. Yes, our planets seem to have formed in a disk.

We have a big group up here, possibly larger than I’ve seen since my first MagAO runs up here in 2013 or 2014. I love the number of women we’ve got working here. Last night there were none of those beautiful clouds at all, when we gathered on the catwalk at sunset:

a whole lot of AO operators and astronomers
Sunset with the whole gang: Logan, Me, Jared, Katie, Amali, Elisabeth, and Emily

This week, I’ve seen a fox, more than a dozen burros, a herd of goats, and at least three vizcachas, so I’m only missing a guanaco to get LCO animal bingo. I have never seen a vizcacha sunning him(her?)self like this before.

A vizcacha sunning itself on the wall next to the Astronomer Support Building this afternoon.

I am using the classic Gene Autry, “Back in the Saddle Again” as my song tonight.

I’m back in the saddle again
Out where a friend is a friend [here’s looking at you Katie, Jared, MagAO and Clio]
Where the longhorn burro cattle feed
On the lowly gypsum weed [or whatever those plants are outside]
Back in the saddle again

Ridin’ the range sky once more
Totin’ my old .44 wavefront sensor
Where you sleep out every night day
And the only law is right sky is fey
Back in the saddle again

Gene Autry – Back in the Saddle Again

MagAO-C 2019B Day 5: Alycia and the Baysides

Tonight was Alycia’s first night. I told Emily it would be more relaxed than last night with all Jayne’s weird rotations because Alycia’s observations are more in MagAO’s wheelhouse — Alycia is searching for faint companions; keeping the rotator off for ADI mode; and is a Clio and general observational expert. But… hah.

So on the first Pre-set of the night, the Bayside-X stage decided to not work. There are 3 “Bayside” stages — they are the motors that move the AO system around so that the star is always on the tip of the pyramid WFS even when the star needs to be nodded around the detector on Clio for sky subtraction. We have replaced the X and Y stages with higher-power versions after we had observers who wanted to keep the rotator on and tracking, which caused the stages to overheat and fail. So I wonder if it’s a coincidence that we spent all last night with the rotator on, putting weight on the X-stage, and then tonight the X-stage refused to move…

Laird and Jared are now on pseudo-day schedules (see Alex’s great MagAO-X post about the unpacking today!) but this happened at the start of the night so Laird was still at the summit and Jared was down in the clean room with MagAO-X but he came up to help post-mortem. Laird went out to the platform with a walkie-talkie and we did the usual dance of moving the motors from the control room and Laird telling us what he saw or heard in the Nas. The current would spike but the motor wouldn’t move. Small movements and homing didn’t work; eventually we power-cycled them and in the end the X-stage did start responding again.

WFS hardware GUI, on the Bayside Stages tab. Each stage can be moved manually here, and the position is reported in mm and current in amps. The Bayside X and Y stages nod in Y and X on Clio, respectively, and the Bayside Z stage moves the W-unit towards or away from Clio, effectively producing a focus movement for Clio. Amali edited the code to enable “Control always active” on the X-stage after a nod rather than “Stage” (brakes). [Image description: A GUI on a Linux computer with status and function boxes and buttons.]

But Laird heard a noise that made him think there was some friction affecting the motion of the X-stage. Since the stage worked normally after the power-cycle, we started thinking about all the work we have done on the Baysides over the years. Originally we used the setting “Control always active” as that would position them precisely to the micron. But this was thought to contribute to the over-heating problem when we had high currents at strange rotator angles with all the weight of the instrument on the Y-stage. So we swapped the original out for the higher-power version, but we also started applying the brakes and turning off the active control after each move (“Stage” check box). But now we are thinking… maybe the brakes themselves are starting to fail, and the friction was the X-stage rubbing along its own brake. So tonight, Amali went into the code and reverted the procedure for the X-stage to now enable “Control always active” rather than enabling “Stage” (applying the brakes) after each move. And all stages worked perfectly for the rest of the night.

This is the “Board GUI” that shows the current state of the various subsystems on the W-unit, which includes the WFS (left) and VisAO (right). The rectangle at the bottom shows the current position of the pyramid with respect to the telescope center. The Bayside X, Y, and Z positions are listed. The entire W-unit is moved relative to Clio when we move the Bayside motors. [Image description: An optical diagram of the WFS and VisAO. Light from the telescope comes in from the top center, goes through the ADCs, and then is split by the selectable beamsplitter such that some portion goes to the right to VisAO and some portion goes to the left to the WFS. The WFS arm includes a rerotator that keeps the pupils aligned to the actuators, and a camera lens that keeps the pupils registered to the pixels on the CCD39. The VisAO arm includes science filters. Below the optical diagram is a rectangle with cross-hairs and a red dot showing the relative position of the pyramid.]

After Jared and Laird went to bed, at one point our TO Alberto went down to get some coffee, and I snapped this pic of the all-women control room. What a big difference from when I was the “Only Girl In the World” at LCO!

Astronomers hard at work. From left: Alycia, Logan, Emily, Amali. [Image description: Alycia is on her phone (don’t worry, Clio and VisAO are actively taking data). Logan is analyzing some data on her laptop. Emily is studying the telescope system to make sure she understands the big picture of MagAO. Amali is on her laptop debugging MagAO software, probably the FITS headers of Clio that aren’t being fully populated with the correct AO parameters.]
Vizzy by the library on my way up to the top. [Image description: A Vizcacha sits on a window ledge in the diffuse glow of pre-sunset.]
The Clay telescope at sunrise. [Image description: Photo taken from slightly downhill of Clay, the dome is to the side glinting in the sunrise while the sky has a beautiful stripey glow.]
Walking back to the village at dawn. [Image description: A cluster of terracotta-roofed buildings down a slope with a winding road. Beautiful mountains and hills. Quiet and still.]
The telescope ridges at dawn. [Image description: A stripey glow on the horizon. A mountain peak and a ridge are both covered in telescopes that are closed up to sleep for the day.]

Classic MagAO song from Day 40 of the super long cold winter run with so many technical problems and such long nights that I actually worked, really worked, 112 hours a week (trouble-shooting all afternoon, observing all night, and choosing each day either a shower or dinner): This Must Be the Place/ Naive Melody by the Talking Heads:

MagAO-C 2019B Day 4: ExoSagan

Today was our first science night, with visiting observers Jayne and Ben from Amsterdam in collaboration with Sebastian from Arizona. Jayne, Jared, and I were all Sagan fellows before becoming faculty, and Sebastian is a current Sagan(-Hubble) fellow at Arizona.

A picture of the 3 Sagan alums in the Clay control room. [Image description: Jared is operating his phone in selfie mode so he’s up close. I’m sitting at the VisAO workstation. Jayne is behind me sitting at the Clio workstation. There are around 20 computer monitors mounted to the desks and walls in the background.]

There were a few hiccups associated with trying to keep the rotator tracking like all other instruments at Clay, but Emily and Amali managed to keep the loop closed and we got great data all night long.

Sunset at Clay. The MagAO-C and MagAO-X teams participated in the tradition of hoping for a green flash. [Image description: Several people stand on a catwalk watching the orange sun set over the distant horizon. The horizon has a orange glow and the distant mountains are blurry.]

The song of the day is the Britney Spears classic “Oops I did it again”: