Incident

The power event that took the platform with it

On 2026-08-04 a power event dropped three of five Proxmox nodes. The node carrying routing, one domain controller, the repository, and the control host was among them, and four HA-managed services later remained in an error state. I was hundreds of miles away with no out-of-band path to the hardware. This account keeps the first remote evidence, the recovery completed on 2026-08-09, and the physical power-domain question that remains open.

Timeline

All times are UTC on 2026-08-04 unless another date is given. They come from three sources that survived because none of them lives on the failed node: the WireGuard handshake log on my laptop, the agent fleet's append-only run ledger in the repository, and Cloudflare's own error pages. The ledger is a committed file, so the timestamps below are copied out of it rather than remembered.

18:35:15 The agent fleet's scheduled cycle finishes normally and dispatches four worker sessions. That detail matters later.
19:16:46 The last WireGuard keepalive that my laptop and the lab's firewall both agree on.
19:17:15 The first WireGuard timeout. This is the start of the outage, and the gap between these two lines is the whole of what the evidence can say about the moment of failure.
From about 19:20 Every hostname behind the Cloudflare Tunnel returns error 1033, which is Cloudflare saying the tunnel connector on the far end is gone. Internal addresses time out. The firewall's own console is unreachable.
20:02 to 20:04 A manually started fleet cycle cannot resolve the git server and stops without dispatching anything.
00:02 on 08-05 The next scheduled cycle runs. It fails DNS three times, checks public DNS and finds it healthy (so the internet is fine and the lab is not), refuses to dispatch work against a repository state it cannot confirm, writes one line in its ledger, and exits. Every scheduled cycle for the rest of the outage repeats that refusal.
2026-08-05 onward The lab stays down. I am away from the hardware with no out-of-band path to it. Work continues against an offline copy of the repository, which turned out to be a story of its own.
2026-08-08 The first version of this page is written while the lab is still down. Its root-cause and recovery sections stay open because the hardware has not been reached.
2026-08-09 On-site recovery shows that the power event dropped three nodes. Cold boots reapply stale declared network settings, and four HA-managed services, including both domain controllers, remain in an error state until they are reset. Routing, DNS, the repository, the control path, and the secrets service return after those conditions are corrected.

Impact

The first remote symptom looked like one failed node because that node carried the firewall, one domain controller, the repository, and the control host. Recovery established a wider event: three nodes had dropped. The concentrated node still made the blast radius immediate by removing routing, source control, and the control path together.

High availability did not return the estate on its own. Four managed services remained in an error state after the nodes returned, and both domain controllers were in that set. The cold boots also reapplied stale declared network settings on several virtual machines. Restored power was therefore the start of recovery, not the end.

Nothing a reader of this site would notice was lost. The public site is served by Cloudflare Pages from a mirror repository and has no runtime dependency on the lab, so it stayed up for the entire outage, checked from outside the failed network. That was not luck. It is the reason the site is built that way, and this is the first time the design has been under real load.

This page was first written from remote evidence while the hardware was still unreachable. The timeline keeps that initial view, while the resolution uses what the on-site recovery established. The difference is part of the incident record.

What worked

Two designs were on trial, and both were written down before they were tested.

The site does not depend on the lab. Cloudflare Pages serves the site from a mirror repository that the platform pushes to, so no machine in the lab is in the path of a visitor. The portfolio survived its own platform's worst day, which is the entire point of splitting them.

The agent fleet refused to act blind. Every four hours the scheduled cycle woke up, tried to confirm the state of the world, could not reach the repository, and declined to dispatch anything. Each time it wrote a fully explained refusal into the ledger, and it made zero blind dispatches for the whole outage. An estate that stops cleanly when it cannot see is the property the rest of this site is trying to earn, and the dispatch half of it passed its first real test.

The honest asterisk The refusal discipline held. Some of the machinery around it did not. The same week's reviews found commit hooks that failed open rather than closed, and a ledger that could not push its own lines. Both became fixes. "The fleet declined correctly" and "every guard around the fleet was sound" are different claims, and only the first one is proven here.

The outage also handed the fleet something strange: several days of a frozen, perfectly stable copy of the repository, with nothing merging under anyone's feet. It became the most productive review and build window the platform has had. The outage cost the lab days of uptime and paid back weeks of engineering, which is a trade I did not choose and would not repeat.

What failed

Three designs failed. Each is written below in the shape the lab uses for a failure mode: what it looked like, what it actually was, how recovery worked, and which control now carries the gap.

Placement: 1 node held the front door

Symptoms. The power event dropped three nodes. The node holding routing, source control, one domain controller, and the control host made the wider failure look like one concentrated platform loss.
Diagnosis. Routing, one domain controller, the repository and the control host had all been placed on the same node. The concentration was known and had never been examined. HA cannot move a guest across a network the dead node was providing.
Recovery. On-site work restored the nodes and routing, then corrected stale network state and explicitly reset the HA-managed services that remained in error.
Prevention. A placement rule, written as an acceptance criterion so it can be tested rather than admired: no single node's loss may remove routing, repository, DNS and control at the same time.

Reach: there was no out-of-band path

Symptoms. Nothing to try. No console, no way to power-cycle a node, no way to confirm what was broken without being physically present.
Diagnosis. Every path I had into the lab ran through the lab. The VPN terminated on the failed node, the management interfaces sat behind it, and the tunnel connector needed the routing that had just gone away.
Recovery. Travel to the hardware. That is the honest answer, and it is why this outage is measured in days rather than minutes.
Prevention. Internal dead-man checks now make missing scheduled work visible. Console access and a watcher outside the lab remain open because they must survive the lab network and power domain being unavailable.

Sight: the monitoring was inside the blast radius, and so was my browser

Symptoms. No alert fired. From outside, a dead estate and a quiet estate look identical. On top of that, my own laptop reported the public site as down while it was up for everybody else.
Diagnosis. Two versions of the same mistake. The monitoring runs inside the estate it monitors, so when the estate died the monitoring died before it could say anything. And my laptop resolved the public domain through the lab's internal DNS, so with the lab gone the name did not resolve at all and the browser drew the obvious wrong conclusion.
Recovery. Check from a network that owes the lab nothing before believing what your own machine tells you. A phone on mobile data settled the question in seconds, once I thought to ask it.
Prevention. Client resolver fallback and an outside watcher are separate controls. The internal dead-man path can catch a stopped job, but it cannot report the loss of the estate that runs it.

Resolution

Confirmed cause. A power event dropped three of five cluster nodes. The outage expanded through shared-fate placement, stale declared network settings applied during cold boot, and four HA-managed services that stayed in an error state. The UPS engaged, but the physical reason three nodes still lost power remains open pending outlet and load verification.

Recovery. On 2026-08-09 I restored the nodes and routing, corrected stale network state, reset the HA error services, restored internal DNS, and verified the repository, control path, tunnel, and secrets service before normal automation resumed.

Corrective actions. The follow-on work now covers declared network-state drift, HA error alerting, sealed or unreachable secrets-service alerting, placement, dead-man monitoring, outside console access, and the UPS power domain. Several checks exist in code; the external hardware path and physical power review remain open.

Root cause

The initiating event was power loss, not an isolated operating-system or network failure. Three nodes dropped while the UPS engaged. The remaining physical question has two credible answers: the connected load exceeded what the battery could carry, or the affected nodes were not on protected outlets. The current evidence does not distinguish them, so the outlet and load review stays open.

The service outage lasted longer than the power event because the recovered estate did not match its intended state. Cold boots applied stale network declarations, and four HA-managed services stayed in an error state until they were reset. Both domain controllers were in that group, which kept internal name resolution down and made the remaining services harder to diagnose.

Recovery

The order drafted during the outage was directionally right, but it missed the two conditions that consumed most of the recovery: stale declared network settings and HA services parked in an error state.

Inspect the hardware on site and establish that the event affected three cluster nodes.
Restore routing first so the surviving and recovered systems can be reached.
Compare running and declared network state. Correct stale settings before another reboot can reapply them.
Inspect HA state explicitly. Reset the four services in error, including both domain controllers, then verify internal DNS.
Restore and verify the repository, control path, tunnel, and secrets service after the platform underneath them is stable.
Audit the remaining virtual machines and the golden template. That pass found more stale resolver declarations that had not yet caused an outage.

The recovery changed the runbook in two places. Declared configuration now needs an assurance check before a reboot exposes drift, and HA error state must be queried directly because it does not clear itself or appear as healthy silence.

Action items

Each item below is tracked separately. The status here distinguishes code that exists from a control that has been applied and tested against the running environment.

Declared-state assurance. An offline checker compares virtual-machine network declarations with inventory. Live collection and the address-management comparison remain to be completed.
HA and secrets-service alerting. The checks distinguish HA error, sealed, and unreachable states. Applying them and proving delivery from a real fault remain operator work.
Placement and live-fire review. The acceptance criterion remains that one node loss cannot remove routing, repository, DNS, and control together. A deliberate drill still has to prove the resulting placement.
Outside observation and console access. Internal dead-man monitoring is live for scheduled work. Hardware-independent alerting and console access remain open.
UPS power-domain review. Physically verify protected outlets, connected load, runtime, and the UPS event record before assigning the initiating failure to battery capacity or wiring.

The one-sentence version

A power event dropped three cluster nodes, and shared placement, stale declared network state, HA error services, and monitoring inside the same blast radius turned that event into a multi-day platform outage.

Other postmortems → Watching the watchdog → The security posture →