MagAO-X Post-Cataclysm Day 0: It Was Almost Worse

7 weeks ago Chile was hit by a massive winter storm that dumped huge amounts of rain at lower elevations and snow higher up. This caused many disruptions to life here, with flooding, washed out roads, and many hardships.

The crew at LCO knew it was coming, so they went down to a skeleton crew who rode out the storm, but were stuck on the mountain for nearly 2 weeks until access could be restored. The mountain lost power, water, and internet. As part of managing this, MagAO-X was unplugged and left to hibernate while the rest of the observatory recovered.

Katie and I arrived today to begin plugging MagAO-X back in and bring it back to life. We’ll also take care of a few odds and ends while we’re here.

On the drive up it looked like we had arrived at the wrong place. Lush greenery and stunning wildflowers. The picture above doesn’t really do it justice. The picture below shows it a little better, but also captures the remains of one of the landslides that blocked the highway from La Serena to LCO.

By now the highway is mostly back to normal, though were several lane closures for crews still doing cleanup. The road from the highway to LCO is a different matter. Above you can see where it was totally washed out, and we had to take a freshly plowed dirt road through the wash downstream. No water there, but we saw many puddles, ponds, even a small lake, along the road.

Upon arriving, we discovered just how close we were to disaster. Water had clearly gotten into the cleanroom and run along the floor. However, it appears to have only touched one leg of the instrument and missed our electronics. Phew.

After arriving we did some glycol maintenance and got it flowing again, mopped the floor, and plugged everything back in and turned on the computers. So far so good. Tomorrow morning we’ll start moving things and close the loop.

We took a different route from the airport today, and saw this mural in La Serena
A nice surprise in the lodge! A selection of NA beer.
It’s still winter here, so this is the closest we got to a sunset pic.

This is a short trip so we’ll keep the rules simple: song of the day is required as usual.

MAPS 2nd and First Night of 2026B

First Night we did not open and did check out of hardware and called it an early night.

I wanted to provide a quick update from last night and fill everyone in on what happened.

We were able to get on sky early in the night. After locating the target star, which was initially a little high, we guided it to the center of the dichroic. We then moved on to alignment and were able to proceed to closing the loop and getting the system operational.

A few highlights from the night:

We successfully got Suresh’s gain feature on the IR wavefront sensor working, and it was used throughout the night.
The gain feature helped us work with dimmer guide stars and improved our ability to acquire and correct on fainter targets.
We successfully closed the loop with three modes and were able to bootstrap up to 20 modes. However, the correction was still very unstable, indicating that additional work is needed before we can push to higher-order correction.

Issues encountered:

We appear to have some portions of the shell missing. We are investigating this today to better understand the cause and determine the appropriate corrective actions.
The flat does not appear to be where we would expect it to be, and we will be looking into that as well.
We also believe the optical alignment of the wavefront sensor is not as good as it could be. We plan to spend time today improving the alignment and evaluating its impact on performance.

Overall, the night was productive. We achieved on-sky acquisition, completed alignment, closed the loop, validated the IR wavefront sensor gain feature, and demonstrated bootstrapping from 3 to 20 modes. Today’s focus will be on understanding the shell issue, investigating the flat, improving the wavefront sensor optical alignment, and stabilizing the higher-order correction

MAPS 3rd night of 2026B

Instrument: ARIES + MMTAO  ·  Sensor: IR wavefront sensor


Overview

The third night of this AO engineering run focused on closing the loop at higher mode counts than previous nights, refining the alignment, and acquiring on-sky data for an ASU science target and a photometric reference for Suresh’s gain comparison.

At the start of the night there were lingering alignment issues from Night 2, so we budgeted time to revisit the bootstrap sequence before pushing to high mode counts.


Planned sequence

  1. Improve pupil alignment
  2. Confirm alignment on the IR wavefront sensor
  3. Check and select flat files
  4. Close loop — bootstrap from 3 modes up to maximum achievable
  5. Observe ASU science target

Night log

Coma and alignment

Significant coma was present at the start of the night. The coma adjustment pushed the hexapod near its travel limit, which caps how much we can correct going forward. A full realignment of both the AO system and ARIES is warranted before the next run — there is something mechanically misaligned upstream.

Flat selection

We tried several flat configurations. None showed a clear advantage over the others, so we settled on the 35 μm gap flat and used it for the remainder of the night. This will be our reference for the run.

WFS alignment

After the alignment iteration, the four pupils on the WFS were better centered and the slope maps looked more symmetric. The improvement was visible by eye.

Bootstrap

Redid the 3-mode and 20-mode loops, going back and forth between mode counts to bring the mirror to a position where the WFS could robustly pick up the surface. The iterative bootstrap was slower than ideal but ultimately got the mirror into the right regime.


Science target — MASCARA-1 b

ParameterValue
V magnitude8.3
K magnitude7.7
IR WFS Detector signal~200 counts (~1% of detector)

Summary and follow-up

The coma and hexapod limit situation is the most pressing issue.

The 32-minute loop run at 50 modes is encouraging — the system is stable when it’s up. The wild coil events are shorter and less frequent than early in the run, suggesting the system is warming up to a more thermally stable state.

MAPS 4th night of 2026B

Coma investigation, telescope nudging, and variable clouds

Instrument: ARIES + MMTAO  ·  Conditions: Variable clouds, intermittent AO


Overview

Night 4 was a challenging one. Variable clouds prevented stable adaptive optics for most of the night, forcing us to work on engineering tasks in the gaps. We used the time productively: we revisited the coma issue first flagged on Night 3, inspected the hardware for a physical cause, and made meaningful progress on coordinated telescope nudging — a capability needed for precise slit acquisition and AGPM coronagraphy with MIRAC.


Coma investigation

Initial hypothesis: ASM–ARIES misalignment

Coming into the night, our working theory was that the coma seen in the wavefront sensor signal originated from a misalignment of the wavefront sensor. If one system was significantly more misaligned than the other, that would point to a clear culprit and a straightforward fix.

Result: both systems contributing equally

When we measured the coma contribution from each system independently, both the ASM and ARIES showed a similar magnitude of error. Neither was the obvious offender — the coma was distributed across both systems.

Hardware inspection

We physically inspected the ASM and the top box to look for anything obviously wrong — a shifted element, a loose mount, anything that could account for a coma error of this magnitude. Nothing was found. There was no visible mechanical cause that explained the level of aberration we are seeing.


Coordinated telescope nudging with AO closed

The problem

When the AO loop is closed, you cannot simply nudge the telescope to move a star — the AO system sees the resulting tip/tilt as a wavefront error and immediately corrects it, moving the mirror to put the star back where it was on the WFS. The star stays pinned to the WFS reference, not to the slit or coronagraph mask. This is fine for AO correction but a problem when you need to precisely position a target on a spectrograph slit or on the AGPM coronagraph with MIRAC.

The solution: coordinated WFS motor and telescope moves

We worked out the combination of moves needed to shift the star on sky while keeping the AO loop closed and the WFS in a valid sensing state. The sequence works as follows:

  1. A coordinated nudge is applied to the WFS motors, shifting the WFS reference position.
  2. The AO system now sees an apparent tip/tilt error — the star is no longer centered on the new WFS reference.
  3. The AO system offloads this tip/tilt to the telescope, commanding the telescope to move.
  4. The telescope slews to re-center the star on the WFS, effectively moving the star on sky to the new position.

The result is a controlled, AO-stable nudge: the star moves on sky, the WFS remains happy, and the loop stays closed throughout. This is the correct approach for keeping a target precisely on a spectrograph slit during long integrations and for AGPM coronagraphy with MIRAC, where sub-diffraction-limit centering on the mask is required.

Scaling factors for nodding

We also performed a large move with the AO system off to establish the scaling factor between the commanded move and the resulting on-sky displacement. This gives us the starting point for nodding variables — the numbers needed to define a nod amplitude that corresponds to a known angular offset on sky.


Weather and on-sky time

Clouds were thick enough for several hours that no light was reaching the WFS, and we were closed for that period. When the clouds thinned toward the end of the night, we recovered a few hours of on-sky time. We were able to test the nudging scheme on a real star — with the AO system both on and off — and confirm that the coordinated move sequence behaved as expected.


Summary and follow-up

Despite the weather, Night 4 produced two concrete results: a clearer picture of the coma problem and a working recipe for closed-loop telescope nudging. The nudging capability is now understood well enough to be used operationally for slit work and AGPM acquisition.

Follow-up items

  • Schedule daytime realignment of ASM and ARIES together to address the distributed coma
  • Document the scaling factor obtained from the large open-loop move for use as nodding starting values
  • Formalize the WFS motor + telescope nudge sequence into a repeatable procedure for slit acquisition and AGPM work
  • Monitor whether the coma level changes after the realignment to confirm the fix

Observation Run October 2026

Sorry the goal for me was to try to get one of these out every night, but here is a download of my notes fro this run.

Goals

  • IRWave front Sensor
  • Skipped frames
  • Larger Gains
  • Flat and configuration file.
  • CACAO/Do Crime
  • Push beyond 50 modes.
  • Telescope Alignment
  • Coordinated moves for mount and AO system

Nights

Night 1

  • Alignment
  • There was another offset on the secondary. We could not offload all of it to the hexapod and where caring about -40 arc secconds on the mount. And the hexapod was at its limits at zenith
  • Did not have to use the paper where able to slew the telescope in the elevation in and was able to find it.
  • Calibration
  • Started off with a fresh calibration 3 modes 2026_10240 91018. NOt from the past calibration. The unit did seem to be a little more stable.
  • Fresh calibrations from last night:
  • 3 modes: MAPS336_fkl2c0_3-modes_20261024_091018
  • bootstrapped 3 modes: MAPS336_fkl2c0_3-modes_20261024_092854
  • bootstrapped 20 modes: MAPS336_fkl2c0_20-modes_20261024_094650
  • bootstrapped another 20 modes: MAPS336_fkl2c0_20-modes_20261024_101253
  • Calibrations built on top of 3 modes calibration from December 2025 (MAPS336_fkl2c0_3-modes_20251205_021722):
  • 3 modes: MAPS336_fkl2c0_3-modes_20261024_080343
  • bootstrapped 3 modes: MAPS336_fkl2c0_3-modes_20261024_081245
  • bootstrapped 20 modes: MAPS336_fkl2c0_20-modes_20261024_104817
  • Able to close the loop on 20 modes. The images seemed to be improved, but the images did seem to not be stable and would go in and out. Boot strapped from 3 twice and then up to 20 modes and bootstrapped that one also.
    • Problems
  • z stage motor did not home when it was comanded to be moved.

Night 2

  • Weather
  • There was high humidity from the start of the night. And able to get on target for the first of the night.
  • A lot of clouds where blowing over and this made it impracticle for any AO work
  • Did a cordinated move through REDIS and confirmed that a command was sent to the AO System and the Telescope mount.

Night 3

  • Started off by looking at the flats and closed the loop and then created a flat from that position
  • Then pushed the mirror with robin s maximizing flux algorithium
  • For better pupil ilumination
  • This took a lot of current on the mirror to make themoves
  • Then santized the flat and used this to start a new reconstructor
  • This was abanondend because the mirror was not stable and a lot of current was bing put to the mirrror.
  • Created a new calibration with the new on sky flat.
  • Bootstapped 2X 4 modes 2X 20 modes 500 iterations each. The 20 modes we doubled the amplitude and for the 50 we 1.5 the first 3 to 20.
  • Boostrapped to 50.
  • Bootstraped to 100.
  • This did not have very good results
  • Turned focus offloading on.
  • Closed early the seeing had really gotton behone 3 arcseconds and the ao loop could not improve.

Night 4

  • Started the night with very nice seeing condtions. The min. was around 0.53 arcseconds. Did a 100 mode calibration
  • Avalanche implemented gain on a 9th magnitude and it the closed the loop.
  • 0.6 v K9 star able to close the loop 100 m
  • 12.4 v k6 star able to close the loop 100 modes
  • Ross 614

Night 5

  • mflt loop limit was used to keep it more consistent kept large commands from being sent to the mirror.
  • hr8799
  • Tested new flats to see which one would work better
  • The 100 modes from last night had a problem that above 50 modes it did not work. We just turned on the 50 modes and trying to bootstrap from there.
  • The system does seem to struggle after 50 modes. Its starts to elogate the image.
  • Z stage in the periscope where able to get it working and map out what kind of motion is needed for a focus adjustment on ARIES
  • Exoplanet WASP33