Blog

MagAO-C 2019B Day 8: Life At LCO

Tonight was finally the promised night of relaxed AO operation after multiple nights of troubleshooting. It is much more to an AO operators delight to only need to open and close the loop for new targets with the occasional MatLab pop up forcing a reset on the tip tilt. Alberto, our telescope operator, was able to tell us his operational comedies of the past and share pictures of his lovely family. Later today is the Mercury transit! Tonight is also my first blog post so an introduction to the outside world is due.

I’m Emily Mailhot, a new AO operator for not only MagAO-C but also LBTI and eventually MMT Maps, both in Arizona. My equally new counterpart, Jared Carlson, and I have been learning the ins and outs of adaptive secondary operations, concepts, and frustrations – and are excited to do so.

This trip to LCO has created numerous “first experiences” for me personally. First time leaving the United States, first time on a B787-8, first time seeing the southern sky, first time experiencing sub-arcsecond seeing (both AO operationally and visually), the list goes on. Considering this, a life at LCO post from brand new eyes seemed appropriate.

Food at LCO

When I was informed that our meals would be served on site I was both relieved and skeptical. At LBTI we are responsible for feeding ourselves through the night, vending machines on the 5th floor of Steward Observatory get a lot of traffic. But would this provided food be delightful or would I need to start preparing for numerous high school band-camp cafeteria flashbacks? I am happy to announce that the food is absolutely superb. From the empanadas to the night lunches to the cookie filled dome kitchens – you have to try very hard to go hungry. Meals are at set times, 730 breakfast, 1230 lunch, and 1830 dinner, but for most of the run breakfast and lunch is sleep for the observers. Instead we fill out night lunch forms which are located in the main kitchen for observers to fill out by 1600 each day. It’s a checklist of various sandwich combinations or your can write “plato de cena” if you’d prefer a plate of that evenings dinner. It is the telescope operators job to collect all the night lunches after dinner and bring them to the telescope.

For those at home wondering what we are eating here’s a selection:

My first empanada!
Empanadas are only served on Sunday for lunch so you better get your fill while you can. They come in two forms, carne or con queso, both equally delicious. This particular Sunday there was also veggie and seafood soup.

Clearly more than enough food to go around.

Wildlife at LCO

The wildlife here at the summit is diverse. This blog has many posts about the endearing vizzy’s, but the burros, horses, and guanacos are just as abundant. Today on a run around the grounds I encountered both guanacos and horses running along the ridge below the road. They seemed to be chasing each other in what resembled a wild wild west setting. It was so magnificent to capture that I stopped the video clip too soon and I instead watched in awe.

On another run earlier in the week, towards the du Pont Telescope, I encountered a large herd of burros controlling the road. One was particularly annoyed that I was trying to pass and gave me a stiff snort.

"I'm watching you, Wazowski. Always watching."
A herd of burros serving a strong stare-down.

Of course, this wouldn’t be complete without a vizzy. This sunrise viscacha (a Chilean cocktail?!) watched us sleepy observers wander back down the summit, only to be met by this eight legged, fanged creature. He was small but mighty, creeping up toward my hand each time I tried to open my door. Not usually being afraid of spiders, a small fear for them was gained after the long dance it took to be allowed into my home, room 5.

The fox on the other hand kept distance and the goats clumsily climbed to the summit with us.

The goats followed us all the way to the Clay Telescope dome where inside the control room us tired observers tried to troubleshoot through our engineering night after a long install day.

Work at LCO

Of course the animals only provide us entertainment between times of work – like our extremely long install day and engineering night (see Day 3!). Here’s a glimpse of the Clay control room, AO work station, and some pre-science pupils.

Smoking with science.
An image of the control room before sunset. The back left station is for science camera, Clio. To its right is the telescope operators station. Left front is the VisAO station and finally next to it is the adaptive operators station. During troubleshooting events this room can be full of people, standing room only.
A well behaved shell.
The AO station! GUI’s upon GUI’s upon GUI’s.

Beauty at LCO

LCO is nestled in the most beautiful landscape, best described through photographs.

Pretty in pink.
The first sunset of the trip from the porch of the “hotel” dining room.
Looking out towards to Andes.
Taken from behind the Summit near sunset. This was the first clear night since we had arrived.
GMT
The peak for the future Giant Magellan Telescope taken from the telescope operators housing.
Hair of JJ lens flare
Clay dome open for science.
Clay + ASM appreciation pic
Clay still at the zenith, the ASM backlit by the sky.
Twins
Baade and Clay awaiting a night of science.
Orange mountains
A picture of the open dome from the nasmyth platform, taken during the night of the misbehaving baysides.

And my favorite photograph of all, taken after dinner on our long day transitioning into our long night.

Stairway to Heaven
The stairway to the stars.

Keeping with classic song theme, here’s Led Zeppelin performing Stairway to Heaven.

MagAO-X 2019B Unpacking Day 4: Survival

MagAO-X is alive! After being boxed up, shipped from Tucson to Phoenix to LA (we think — a little fuzzy) to Miami, with a long pause, then to Santiago, braving the dangers of revolution (and customs (and customs strikes!)), and a trip by truck (always touchy, this time with road blocks!), and finally being craned out of its box and carefully reconstituted, we can still close the loop.

Needless to say we are happy, a little bit relieved, and excited to get our new instrument on the Magellan Clay Telescope.

Of course, it doesn’t always go exactly as planned. As the Captain said, “It never goes smooth. How come it never goes smooth?”

A clear example of this phenomenon is when we had to roll our electronics rack out of the clean room so Nelson and Emilio could cut off a stripped 3/8-16 bolt. The Aluminium L-bracket you see there is part of the earthquake restraint system, which, as Amali pointed out last night, is always a thing here.

A pano of the cleanroom when we were just getting started this morning. That’s Kyle checking on the DM humidity before turning it on.

So here’s the big moment when we finally knew that we didn’t f it all up:

And here you can see our vAPP coronagraphs looking good in a deep exposure:

vAPPs looking eXtreme. Laird wants you to know that there are no optical ghosts in the dark hole. Maybe the eye doctor has a little work to do though.

Moonrise over the Andes on my hike up.

Laird and Alex watching tonight’s sunset. Photo by Kyle, I’d already given up — obvious non-green-flash night. That’s our electronics shipping crate, btw.

Tonight’s song is “Wanted Dead or Alive”, by Bon Jovi:

MagAO-C 2019B Day 7: The Alpaca Saga

Tonight, gentle readers, a tale of drama and derring-do! A mystery was solved, a dastardly plot uncovered, an insidious impostor unmasked, several computers rebooted, a culprit nabbed, and, despite its most vicious efforts, the criminal’s final revenge borne without injury. I present: The Alpaca Saga!

THE CASE

Yesterday night the adaptive secondary — aka adsec, asm, shell, TS1, and as of last week Lollipop — began to display some troubling symptoms. For instance: every so often, even in the gentlest of conditions, the shell would suddenly RIP (don’t worry, the falling-to-pieces is only metaphorical). That in itself isn’t terribly uncommon, but the circumstances were a bit strange: the adsec would briefly report missing values for elevation and for windspeed. A shriek into the void! The wail of an injured creature in the darkness! But then the elevation and windspeed would both return, pretending that nothing was amiss.

Obviously, something was amiss.

Simultaneously, in the once-peaceful environs of the Clio workstation, bayside stage focus moves were being sent, as messenger pigeons into a clear blue sky…and then snapped up by an invisible space dragon, never to return. By which I mean, the bayside stage at the WFS never moved. And yet, it could be moved without problems from the WFS directly.

THE INVESTIGATION

What do absent elevation, unreadable wind speed, and unheard Clio commands have in common? They’re all communications sent through the computer that masterminds the MagAO processes: the Magellan adsec supervisor, or magadsecsup.

A CLUE!

In the dead of night, Jared bravely took his life in his hands and went down to the equipment room — is that its name? the one with the MagAO computers and racks and occasional unstable ladders — anyway, Jared found A CLUE!! Which was: the little blinky lights on the ethernet ports connecting magadsecsup and the network switch were neither happy-working-green or transmitting-information-amber, but instead an alarming shade of ORANGE. So Jared and Emilio beheaded a cable and put it in a different switch port.

But the crimes did not stop!

We hobbled through the night. The adsec wept. The adsec RIPped (still metaphorically). The notion of a criminal in our midst weighed heavily upon our minds. Dawn came and we fell into uneasy slumber…

A BREAKTHROUGH!

During the day, while the rest of us slept, Jared (life still in hands) disconnected magadsecsup. AND YET…he could still ping its IP.

Note, gentle readers: if you remove a computer from the network, you should most certainly NOT be able to ping it. Which meant that, instead of only magadsecsup responding to madadsecsup, we had … AN IMPOSTOR! GASP! SHOCK!

We put the sleuths of LCO onto the case. At dinnertime we received word that the culprit had been found…and it was none other than Alpaca, one of the TCS computers at Baade!

THE PART WHERE THE DETECTIVE EXPLAINS EVERYTHING

Somehow, alpaca.lco.cl had been given the same IP address — one ending in 35 — as is statically assigned to magadsecsup.lco.cl. Consequently, when the Clay TCS sent information — elevation, for instance — to magadsecup, they were sometimes intercepted by this nefarious agent and never made it to the real adsec. Chaos! Upheaval!

Here I provide a helpful diagram to assist in distinguishing a genuine adsec from a sneaky alpaca.

A diagram pointing out the important differences between an adaptive secondary mirror and an alpaca. The alpaca can be distinguished by its relative fluffiness, envious glances, hoard of stolen commands, and 581 fewer actuators.
A guide to distinguishing a true adsec from an alpaca impostor.

Anyway, once the impostor was summarily booted from the scene of the IP, and Lollipop comforted and cajoled into returning to service, the night proceeded without further loss of elevation.

Emilio says the real solution is to update the hosts file on alpaca.

ALPACA’S LAST KICK

But! Just as we returned to science, we were subject to one last act of villainy! In a fit of rage and envy, deprived of its chance to be cool like magadsecsup, Alpaca threw a fearsome tantrum. Being somewhat larger, the actuators of an alpaca are capable of a much larger throw, and can cause much larger aberrations. And, because it’s networked, this particular alpaca managed to cause severe turbulence IN THE VERY EARTH, somewhere 33km northeast of Vallenar…

In other words, we had an earthquake! It was fascinating and bizarre! As though the world was swimming. I got to push the “earthquake button” on the adsec GUI, which Laird had pointed out to me only a few hours earlier!

Apparently the “Force enable TSS” button works by setting the shell windspeed reading to 500 m/s. Seeing that number was honestly more alarming than the earthquake…

Anyway, here are a few more helpful diagrams to cement the reader’s understanding of the situation. One is from the Chilean national seismological center. The other is…not.

A chart from a Chilean earthquake tracking website. Magnitude 4.8, 33km NE of Vallenar.
Why yes, it was a perceptible earthquake. Even the Earthquake Officials agree.
An alpaca throwing a tantrum because it can't have IP address 35. A truly indignant alpaca may engage EARTHQUAKE MODE!!!
ALPACA SMASH

MORAL OF THE STORY

Don’t judge a villain by its actuators. And watch your IPs.

SONG OF THE DAY!

Here’s a song about another creature up to no good. May all of us here at MagAOs both -C and -X have more luck with our problems than the hapless Inspector Clouseau.

MagAO-X 2019B Unpacking Day 3: Cabling

Well it was another long but successful day for MagAO-X. We fully integrated the electronics rack with the instrument and began the optical alignment! Impressively, the optical alignment looked almost exactly the way we left it in Tucson. After all that MagAO-X has been through on its way to Chile, we were worried that the inside of MagAO-X would be a mess. However this was not the case, and we were more than pleased by it. We are confident that we will complete the realignment tomorrow and close the loop!

We started the day with our walk up to the “halfway house” at 9:00 am. In the picture below, you can see Jared joined by a friendly goat. Then there’s Kyle catching up, and me not far behind taking the picture.

A typical walk up the hill at Las Campanas Observatory

After some final glycol testing, we started cabling the system. The cables run from the instrument to the electronics rack to power all of our electronics (deformable mirrors, cameras, stages, actuators, etc.).

We completed all of the cabling except for our “Tweeter” deformable mirror (DM), so Laird and I were able to start on the optical alignment.

Laird working on the optical alignment
Looking good!

Here’s a picture of me with the instrument (front view):

Me and MagAO-X from the front

Here’s a picture of Laird with the instrument (back view). From this point of view it sort of looks like a “dollhouse” of optics. That makes it sound a lot simpler than it looks!

Laird and MagAO-X from the back

Finally, we began cabling the 2,048 actuator DM. This is the part that took us the longest, since cabling 2,000 wires is no easy task. This took multiple iterations in order to make sure all of the pins were aligned with the Samtec Connectors.

Kyle mounting the DM cables to the Samtec Connectors
El completo!
The fully cabled MagAO-X system

And so that marks the end of a long day for MagAO-X. Tomorrow we may finally close the loop for the first time in Chile!

Just another beautiful sunset at Las Campanas Observatory

The song of the day is going to be Moonlight Serenade, by Glenn Miller:

MagAO-C 2019B Day 6: Back in the Saddle Again

I’m working on telescope domination this week. I started at Clay using MIKE (well, not really, I got clouded out). I moved to duPont using CAPSCam, I had a night off to visit with the MagAO team, and now I’m back at Clay using MagAO. Tomorrow I’m off to Baade with MagE. Have a telescope or instrument, I’m there!

It’s been a while, so I took last night to get reacquainted with my old friend Clio. It’s like riding a bike (or maybe like being back in the saddle again, though I wouldn’t really know), it turns out. I think I might even have muscle memory for the Camera Control GUI.

Last night was great, but tonight, well, we have some problems. The Clio pupil mask was out of place, though it was fine last night and no one should have touched the motor. Then, the MagAO systems stopped communicating with each other correctly. But by 1 AM, we were back in the saddle again.

So, let’s talk about the weather, animals, and classic songs instead. The sunset was beautiful.

Clouds make for lovely sunsets, and I’m fine with these on-the-horizon beauties as long as they don’t come overhead. If you look very carefully (at least, I could see them on my phone screen), you can see Venus, Jupiter and Saturn along the ecliptic. Yes, our planets seem to have formed in a disk.

We have a big group up here, possibly larger than I’ve seen since my first MagAO runs up here in 2013 or 2014. I love the number of women we’ve got working here. Last night there were none of those beautiful clouds at all, when we gathered on the catwalk at sunset:

a whole lot of AO operators and astronomers
Sunset with the whole gang: Logan, Me, Jared, Katie, Amali, Elisabeth, and Emily

This week, I’ve seen a fox, more than a dozen burros, a herd of goats, and at least three vizcachas, so I’m only missing a guanaco to get LCO animal bingo. I have never seen a vizcacha sunning him(her?)self like this before.

A vizcacha sunning itself on the wall next to the Astronomer Support Building this afternoon.

I am using the classic Gene Autry, “Back in the Saddle Again” as my song tonight.

I’m back in the saddle again
Out where a friend is a friend [here’s looking at you Katie, Jared, MagAO and Clio]
Where the longhorn burro cattle feed
On the lowly gypsum weed [or whatever those plants are outside]
Back in the saddle again

Ridin’ the range sky once more
Totin’ my old .44 wavefront sensor
Where you sleep out every night day
And the only law is right sky is fey
Back in the saddle again

Gene Autry – Back in the Saddle Again