Rebuilding the secrets server into a cluster

What a secrets manager does here

Nothing in the lab has a password typed into it. Every pipeline, every monitoring check, every scheduled job asks the secrets server for what it needs at the moment it runs, proves who it is, and gets back a short-lived credential. That is the whole point: a password that lives in one place can be rotated in one place, and a job that has to ask for its credential can have that permission taken away later.

The cost of that design is concentration. If the secrets server is down, the backup job cannot log in to the backup system, the certificate renewal cannot talk to the certificate authority, and the build pipelines cannot pull anything. It is the most important machine in the lab by a wide margin, and for a long time there was exactly one of it.

The old design, and what was actually wrong with it

One virtual machine. Its data lived in a directory on that machine's own disk. When Vault starts it comes up sealed, meaning it holds the encrypted data but cannot read it until someone provides the key that opens it, so a small service on the same machine read the key from a file on that same disk and unsealed it automatically at boot.

That worked, and it is worth being honest about why it was replaced rather than dressing it up. Two problems, and the second is the interesting one.

The obvious problem is that one machine is one machine. Reboot it and everything that needed a credential during those two minutes failed.

The subtler problem is that the unseal key sat in a plain file on the same disk the backups imaged every night. That meant every backup of that machine contained the key to its own contents, so the security of the backup store had quietly become the security of the secrets store. Nobody decided that. It was the natural consequence of automating the unseal the easy way, and it is the kind of thing you only notice when you sit down and ask what a stolen backup would actually give someone.

Three nodes instead of one

The cluster now stores its data using Raft, which is a way for several machines to agree on the same list of changes even when they cannot all talk to each other. Every write goes to a leader, the leader tells the others, and a write only counts once a majority have written it down. Because a majority has it, any survivor can become the new leader without losing anything.

Three is the smallest number that gives you that. With two machines there is no majority: split them and neither side can safely claim to be in charge. With three, one can fail and the remaining two still form a majority, elect a leader among themselves, and carry on. Lose two and the cluster deliberately stops accepting writes rather than risk two halves disagreeing about what is true.

The three nodes are placed on different physical hosts on purpose, with a placement rule that keeps them apart. Three copies on one hypervisor is one power supply away from zero copies.

The locked safe problem

Vault starts sealed and needs a key to open. If you automate that by leaving the key next to the door, you have a lock that does not lock. If you do not automate it, every reboot needs a human with the key, at whatever hour the reboot happens.

The way out is to have something else hold the key. A second, much smaller Vault runs on its own machine, running only Vault's Transit secrets engine: it holds an encryption key and will encrypt or decrypt small pieces of data for callers it trusts, never storing them itself. This is Vault's own transit auto-unseal pattern. The cluster stores its own unseal material in a form only that second server's Transit engine can decrypt. On boot, each node asks it to decrypt, gets what it needs, and unseals itself. Nobody types anything, and no readable key sits on any of the three nodes.

This is a real improvement and it is not free, so here is the honest version of the trade. The cluster now depends on that small server being up. If it is down when a node boots, that node comes up sealed and stays sealed, and getting it open is a manual recovery using the operator key that is kept offline. That path is written down step by step. It has not been rehearsed yet, and until it has, it is a documented procedure rather than a proven one. The distinction matters enough to print: a procedure nobody has walked is a plan, and this page will say something different once it has been walked.

The operator key itself is held offline, away from the machines it opens. That is all this page will say about it, which is deliberate.

One address that follows the healthy node

Three nodes is not much use if everything in the lab has to know which of the three to talk to today. So the cluster answers on a single address that is not permanently tied to any node. The nodes run keepalived, using VRRP (virtual router id 51) to decide among themselves who currently owns that address. Each node carries a different priority, one at 150 and the other two at 140 and 130, so ties always resolve the same way. If the owner goes quiet the others hold an election and the highest remaining priority takes it over. To the rest of the lab, nothing changed: the same address still answers.

The interesting failure is not a node that dies. It is a node that stays perfectly alive and stops being useful. A machine whose Vault process has sealed itself is still pingable, still answering on the network, still winning the election, and serving nothing. Every host-level health check stays green while the service behind it is dead.

So the election is not decided by whether the machine is up. Each node runs a check against its own Vault asking whether it is actually unsealed and answering, and a node that fails that check drops its own priority and loses the address on purpose. The check runs every five seconds and gives up after two consecutive failures, so a node that seals itself hands the address over in about ten seconds without anyone watching. The check also records why it stepped aside, sealed versus not running at all, because those two look identical from outside and want different fixes.

Certificates from the lab's own authority

Every endpoint in the cluster is served over TLS with a certificate issued by AD CS, Active Directory Certificate Services, the lab's own internal certificate authority, not a self-signed certificate generated on the box.

The difference is who decides to trust it. A self-signed certificate has to be trusted one machine at a time, which in practice means someone eventually passes the flag that turns verification off, and from then on nothing is being verified anywhere. With AD CS as a shared authority, machines are told once to trust it, and every certificate it issues is trusted automatically. Rotating a certificate stops being a fleet-wide errand.

These certificates are issued by hand and expire after a year, which is a quiet trap: nothing fails until the day it does, and then everything that verifies fails at once. So every endpoint, the shared address, all three nodes, and the small Transit server, is watched by the blackbox exporter, which reads the certificate each server is actually presenting rather than trusting a config file. Prometheus turns that into two alarms: a warning at 30 days from expiry and a louder one at 7. A fifth alarm watches that all five endpoints are still reporting at all, so the watcher itself going quiet does not read as a clean bill of health.

Backups and failure domains

PBS, Proxmox Backup Server, takes a nightly backup of every VM in the lab, the secrets cluster and the small Transit server included. That is the ordinary path, and it means no single machine's own disk is where the fleet's protection actually lives.

The backup store itself is not the last line either. It gets mirrored every night onto a second, separate Synology, so a bad day for the primary backup store does not also take out its own backups. That mirror covers the whole fleet, not one server carved out for special treatment; the failure it closes is a shared backup store and the thing it backs up going down together.

Separately, the weekly restore test that already existed for the old single server still runs. A backup nobody has restored is a hope, not a backup.

What was proven on the night

The migration ran in one sitting and each claim below was tested rather than assumed.

Nothing was lost in the move. Before starting, every secret path in the old server was listed and a set of known values recorded. After the cluster was up, the same listing was taken again and compared. The comparison came back empty, which is the only evidence that actually answers "did we lose anything" without reading every secret out loud.

Failover works, and it was tested the awkward way. A node was sealed deliberately while staying up and reachable, which is the case a simple up-or-down check cannot see. The address moved to a healthy node and callers kept getting answers through it. That test was chosen because a setup that only survives a node being switched off looks identical, right up until the day a node fails in a more interesting way.

Self-unsealing works. A node was restarted and came back unsealed on its own, with nobody entering anything.

Both of the machines that keep the cluster honest were also checked: the certificate on every endpoint, and the placement rule that keeps the three nodes on different hosts.

One thing is deliberately not claimed here. The recovery path for the case where the unseal-key server itself is gone has been written but not rehearsed, so it is not on this list.