Category: MagAO-C

Tonight was finally the promised night of relaxed AO operation after multiple nights of troubleshooting. It is much more to an AO operators delight to only need to open and close the loop for new targets with the occasional MatLab pop up forcing a reset on the tip tilt. Alberto, our telescope operator, was able to tell us his operational comedies of the past and share pictures of his lovely family. Later today is the Mercury transit! Tonight is also my first blog post so an introduction to the outside world is due.

I’m Emily Mailhot, a new AO operator for not only MagAO-C but also LBTI and eventually MMT Maps, both in Arizona. My equally new counterpart, Jared Carlson, and I have been learning the ins and outs of adaptive secondary operations, concepts, and frustrations – and are excited to do so.

This trip to LCO has created numerous “first experiences” for me personally. First time leaving the United States, first time on a B787-8, first time seeing the southern sky, first time experiencing sub-arcsecond seeing (both AO operationally and visually), the list goes on. Considering this, a life at LCO post from brand new eyes seemed appropriate.

Food at LCO

When I was informed that our meals would be served on site I was both relieved and skeptical. At LBTI we are responsible for feeding ourselves through the night, vending machines on the 5th floor of Steward Observatory get a lot of traffic. But would this provided food be delightful or would I need to start preparing for numerous high school band-camp cafeteria flashbacks? I am happy to announce that the food is absolutely superb. From the empanadas to the night lunches to the cookie filled dome kitchens – you have to try very hard to go hungry. Meals are at set times, 730 breakfast, 1230 lunch, and 1830 dinner, but for most of the run breakfast and lunch is sleep for the observers. Instead we fill out night lunch forms which are located in the main kitchen for observers to fill out by 1600 each day. It’s a checklist of various sandwich combinations or your can write “plato de cena” if you’d prefer a plate of that evenings dinner. It is the telescope operators job to collect all the night lunches after dinner and bring them to the telescope.

For those at home wondering what we are eating here’s a selection:

• 

• 

• 

• 

• 

• 

• 

• 

Clearly more than enough food to go around.

Wildlife at LCO

The wildlife here at the summit is diverse. This blog has many posts about the endearing vizzy’s, but the burros, horses, and guanacos are just as abundant. Today on a run around the grounds I encountered both guanacos and horses running along the ridge below the road. They seemed to be chasing each other in what resembled a wild wild west setting. It was so magnificent to capture that I stopped the video clip too soon and I instead watched in awe.

On another run earlier in the week, towards the du Pont Telescope, I encountered a large herd of burros controlling the road. One was particularly annoyed that I was trying to pass and gave me a stiff snort.

Of course, this wouldn’t be complete without a vizzy. This sunrise viscacha (a Chilean cocktail?!) watched us sleepy observers wander back down the summit, only to be met by this eight legged, fanged creature. He was small but mighty, creeping up toward my hand each time I tried to open my door. Not usually being afraid of spiders, a small fear for them was gained after the long dance it took to be allowed into my home, room 5.

• 

• 

The fox on the other hand kept distance and the goats clumsily climbed to the summit with us.

• 

• 

The goats followed us all the way to the Clay Telescope dome where inside the control room us tired observers tried to troubleshoot through our engineering night after a long install day.

• 

• 

Work at LCO

Of course the animals only provide us entertainment between times of work – like our extremely long install day and engineering night (see Day 3!). Here’s a glimpse of the Clay control room, AO work station, and some pre-science pupils.

• 

• 

Beauty at LCO

LCO is nestled in the most beautiful landscape, best described through photographs.

And my favorite photograph of all, taken after dinner on our long day transitioning into our long night.

Keeping with classic song theme, here’s Led Zeppelin performing Stairway to Heaven.

Tonight, gentle readers, a tale of drama and derring-do! A mystery was solved, a dastardly plot uncovered, an insidious impostor unmasked, several computers rebooted, a culprit nabbed, and, despite its most vicious efforts, the criminal’s final revenge borne without injury. I present: The Alpaca Saga!

THE CASE

Yesterday night the adaptive secondary — aka adsec, asm, shell, TS1, and as of last week Lollipop — began to display some troubling symptoms. For instance: every so often, even in the gentlest of conditions, the shell would suddenly RIP (don’t worry, the falling-to-pieces is only metaphorical). That in itself isn’t terribly uncommon, but the circumstances were a bit strange: the adsec would briefly report missing values for elevation and for windspeed. A shriek into the void! The wail of an injured creature in the darkness! But then the elevation and windspeed would both return, pretending that nothing was amiss.

Obviously, something was amiss.

Simultaneously, in the once-peaceful environs of the Clio workstation, bayside stage focus moves were being sent, as messenger pigeons into a clear blue sky…and then snapped up by an invisible space dragon, never to return. By which I mean, the bayside stage at the WFS never moved. And yet, it could be moved without problems from the WFS directly.

THE INVESTIGATION

What do absent elevation, unreadable wind speed, and unheard Clio commands have in common? They’re all communications sent through the computer that masterminds the MagAO processes: the Magellan adsec supervisor, or magadsecsup.

A CLUE!

In the dead of night, Jared bravely took his life in his hands and went down to the equipment room — is that its name? the one with the MagAO computers and racks and occasional unstable ladders — anyway, Jared found A CLUE!! Which was: the little blinky lights on the ethernet ports connecting magadsecsup and the network switch were neither happy-working-green or transmitting-information-amber, but instead an alarming shade of ORANGE. So Jared and Emilio beheaded a cable and put it in a different switch port.

But the crimes did not stop!

We hobbled through the night. The adsec wept. The adsec RIPped (still metaphorically). The notion of a criminal in our midst weighed heavily upon our minds. Dawn came and we fell into uneasy slumber…

A BREAKTHROUGH!

During the day, while the rest of us slept, Jared (life still in hands) disconnected magadsecsup. AND YET…he could still ping its IP.

Note, gentle readers: if you remove a computer from the network, you should most certainly NOT be able to ping it. Which meant that, instead of only magadsecsup responding to madadsecsup, we had … AN IMPOSTOR! GASP! SHOCK!

We put the sleuths of LCO onto the case. At dinnertime we received word that the culprit had been found…and it was none other than Alpaca, one of the TCS computers at Baade!

THE PART WHERE THE DETECTIVE EXPLAINS EVERYTHING

Somehow, alpaca.lco.cl had been given the same IP address — one ending in 35 — as is statically assigned to magadsecsup.lco.cl. Consequently, when the Clay TCS sent information — elevation, for instance — to magadsecup, they were sometimes intercepted by this nefarious agent and never made it to the real adsec. Chaos! Upheaval!

Here I provide a helpful diagram to assist in distinguishing a genuine adsec from a sneaky alpaca.

Anyway, once the impostor was summarily booted from the scene of the IP, and Lollipop comforted and cajoled into returning to service, the night proceeded without further loss of elevation.

Emilio says the real solution is to update the hosts file on alpaca.

ALPACA’S LAST KICK

But! Just as we returned to science, we were subject to one last act of villainy! In a fit of rage and envy, deprived of its chance to be cool like magadsecsup, Alpaca threw a fearsome tantrum. Being somewhat larger, the actuators of an alpaca are capable of a much larger throw, and can cause much larger aberrations. And, because it’s networked, this particular alpaca managed to cause severe turbulence IN THE VERY EARTH, somewhere 33km northeast of Vallenar…

In other words, we had an earthquake! It was fascinating and bizarre! As though the world was swimming. I got to push the “earthquake button” on the adsec GUI, which Laird had pointed out to me only a few hours earlier!

Apparently the “Force enable TSS” button works by setting the shell windspeed reading to 500 m/s. Seeing that number was honestly more alarming than the earthquake…

Anyway, here are a few more helpful diagrams to cement the reader’s understanding of the situation. One is from the Chilean national seismological center. The other is…not.

MORAL OF THE STORY

Don’t judge a villain by its actuators. And watch your IPs.

SONG OF THE DAY!

Here’s a song about another creature up to no good. May all of us here at MagAOs both -C and -X have more luck with our problems than the hapless Inspector Clouseau.

I’m working on telescope domination this week. I started at Clay using MIKE (well, not really, I got clouded out). I moved to duPont using CAPSCam, I had a night off to visit with the MagAO team, and now I’m back at Clay using MagAO. Tomorrow I’m off to Baade with MagE. Have a telescope or instrument, I’m there!

It’s been a while, so I took last night to get reacquainted with my old friend Clio. It’s like riding a bike (or maybe like being back in the saddle again, though I wouldn’t really know), it turns out. I think I might even have muscle memory for the Camera Control GUI.

Last night was great, but tonight, well, we have some problems. The Clio pupil mask was out of place, though it was fine last night and no one should have touched the motor. Then, the MagAO systems stopped communicating with each other correctly. But by 1 AM, we were back in the saddle again.

So, let’s talk about the weather, animals, and classic songs instead. The sunset was beautiful.

We have a big group up here, possibly larger than I’ve seen since my first MagAO runs up here in 2013 or 2014. I love the number of women we’ve got working here. Last night there were none of those beautiful clouds at all, when we gathered on the catwalk at sunset:

This week, I’ve seen a fox, more than a dozen burros, a herd of goats, and at least three vizcachas, so I’m only missing a guanaco to get LCO animal bingo. I have never seen a vizcacha sunning him(her?)self like this before.

I am using the classic Gene Autry, “Back in the Saddle Again” as my song tonight.

I’m back in the saddle again
Out where a friend is a friend [here’s looking at you Katie, Jared, MagAO and Clio]
Where the longhorn burro cattle feed
On the lowly gypsum weed [or whatever those plants are outside]
Back in the saddle again

Ridin’ the range sky once more
Totin’ my old .44 wavefront sensor
Where you sleep out every night day
And the only law is right sky is fey
Back in the saddle again

Tonight was Alycia’s first night. I told Emily it would be more relaxed than last night with all Jayne’s weird rotations because Alycia’s observations are more in MagAO’s wheelhouse — Alycia is searching for faint companions; keeping the rotator off for ADI mode; and is a Clio and general observational expert. But… hah.

So on the first Pre-set of the night, the Bayside-X stage decided to not work. There are 3 “Bayside” stages — they are the motors that move the AO system around so that the star is always on the tip of the pyramid WFS even when the star needs to be nodded around the detector on Clio for sky subtraction. We have replaced the X and Y stages with higher-power versions after we had observers who wanted to keep the rotator on and tracking, which caused the stages to overheat and fail. So I wonder if it’s a coincidence that we spent all last night with the rotator on, putting weight on the X-stage, and then tonight the X-stage refused to move…

Laird and Jared are now on pseudo-day schedules (see Alex’s great MagAO-X post about the unpacking today!) but this happened at the start of the night so Laird was still at the summit and Jared was down in the clean room with MagAO-X but he came up to help post-mortem. Laird went out to the platform with a walkie-talkie and we did the usual dance of moving the motors from the control room and Laird telling us what he saw or heard in the Nas. The current would spike but the motor wouldn’t move. Small movements and homing didn’t work; eventually we power-cycled them and in the end the X-stage did start responding again.

But Laird heard a noise that made him think there was some friction affecting the motion of the X-stage. Since the stage worked normally after the power-cycle, we started thinking about all the work we have done on the Baysides over the years. Originally we used the setting “Control always active” as that would position them precisely to the micron. But this was thought to contribute to the over-heating problem when we had high currents at strange rotator angles with all the weight of the instrument on the Y-stage. So we swapped the original out for the higher-power version, but we also started applying the brakes and turning off the active control after each move (“Stage” check box). But now we are thinking… maybe the brakes themselves are starting to fail, and the friction was the X-stage rubbing along its own brake. So tonight, Amali went into the code and reverted the procedure for the X-stage to now enable “Control always active” rather than enabling “Stage” (applying the brakes) after each move. And all stages worked perfectly for the rest of the night.

After Jared and Laird went to bed, at one point our TO Alberto went down to get some coffee, and I snapped this pic of the all-women control room. What a big difference from when I was the “Only Girl In the World” at LCO!

• 

• 

Classic MagAO song from Day 40 of the super long cold winter run with so many technical problems and such long nights that I actually worked, really worked, 112 hours a week (trouble-shooting all afternoon, observing all night, and choosing each day either a shower or dinner): This Must Be the Place/ Naive Melody by the Talking Heads:

Today was our first science night, with visiting observers Jayne and Ben from Amsterdam in collaboration with Sebastian from Arizona. Jayne, Jared, and I were all Sagan fellows before becoming faculty, and Sebastian is a current Sagan(-Hubble) fellow at Arizona.

There were a few hiccups associated with trying to keep the rotator tracking like all other instruments at Clay, but Emily and Amali managed to keep the loop closed and we got great data all night long.

The song of the day is the Britney Spears classic “Oops I did it again”: