Postmortems

The power event that took the platform with it

A power event dropped three of five cluster nodes. The first remote symptom looked like one failed node because it held routing, one domain controller, the repository, and the control host. On-site recovery exposed stale declared network state and four HA-managed services stuck in an error state, including both domain controllers. The physical UPS and outlet question is still open. Read the full postmortem →

Backup datastore corruption

A backup datastore was thin provisioned, meaning it was told it had more space than physically existed, on the bet that not everything would fill up at once. The bet lost. As real usage grew, the underlying pool reached its true capacity. A low-space protection then did exactly what it is designed to do and blocked further writes to stop the situation getting worse. But the pool was already at the edge, and the filesystem took damage in the window between filling up and being frozen.

Two lessons came out of it. The first is that thin provisioning is a loan, not a gift, and a loan you have to watch: the number to alert on is the real pool usage, well before the cap, not the apparent free space the thin volume reports. The second is subtler. The protection that blocked I/O was not the bug; it was the seatbelt. The bug was letting the pool get close enough to the cap that the seatbelt had to engage at all. The fix was to watch the real usage and alert early, so the cap is a line you see coming rather than one you hit.

The commit that switched off CI

A pull request went green with no failures. It went green because nothing ran. A single control token in the commit message told the automation to skip scheduling its workflows, and instead of skipping 1 noisy job it suppressed every check for that pull request. The pull request looked clean because none of the checks that would have failed it had executed at all.

This is the most dangerous shape a CI failure can take, because it does not look like a failure. A red build tells you something is wrong. A build that silently did not run looks identical to a build that ran and passed, and it quietly invites you to merge. The fix is a guard that now refuses to accept that skip token in a commit message in the first place, both as a local pre-commit check and as a check in the pipeline, so the silent switch-off cannot be reintroduced by accident. The rule underneath it is the one this whole lab keeps relearning: a check that did not run is not a check that passed, and the system has to tell the two apart.

A toolchain vulnerability, caught in the pipeline

A vulnerability scanner runs on pull requests and reads the dependencies that go into the lab's container images. It flagged a known vulnerability in one of the build toolchain's dependencies. The fix was to fold the patched version into the build so the vulnerable code is no longer part of the image, and the scan on the pull request then confirmed the finding was gone from the rebuilt image.

Where this stands right now, told straight. The image has been rebuilt with the fix and the scan confirms the vulnerability is gone from that rebuilt image. The running workload has not yet been repointed to the rebuilt image, so the fix is proven in the build but is not live in production until that repoint happens. Calling it done today would be the exact kind of overclaim these writeups exist to avoid, so it is recorded as built and scanned clean, repoint pending.

What these four share

Each incident began with visible state that hid the operating state underneath it. The remote outage looked like one failed node when three had dropped. The datastore reported space the pool could not supply. The pull request reported success without running its checks, and the image could have carried a known flaw behind a clean build. The corrective controls expose the underlying state and keep missing or unreadable evidence from becoming a pass.