How I removed the secrets server as a single point of failure
The lab now asks one stable Vault address for credentials even when a cluster node fails. The cluster copies accepted writes across 3 nodes, unseals itself after a reboot, and moves the service address away from a node that is alive but unable to serve secrets.
This rebuild started because the old server concentrated two risks. Every pipeline stopped during its reboot, and the file that unlocked its encrypted storage lived on the same disk captured by the backup. I moved the data into a Raft cluster, moved unseal authority to a separate Transit service, and tested the failure case that a normal ping check misses.
Built from the Vault HA design, apply record, sizing decision, recovery design, monitoring rules, and the current site source in the repository.
Vault is behind nearly every automated change
Provisioning, backups, certificate deployment, monitoring, and identity jobs fetch credentials when they run. A pipeline authenticates with a narrow machine role, reads the fields allowed for that job, and keeps the values in memory. The same password or token does not have to be copied into every Jenkins credential, configuration file, or script.
That is useful centralization, and it also makes availability matter. When the old single Vault server rebooted, a backup job could not authenticate to the backup API, a certificate job could not read deployment credentials, and provisioning could not reach its infrastructure providers. The pipelines were healthy. Their shared dependency was not.
The old server automated its own unseal by reading an unlock key from its local disk. Vault data was encrypted, but a full image backup contained both the encrypted data and the material needed to open it. That backup-store dependency was never an explicit design decision. It appeared because local auto-unseal was convenient.
Five mechanisms solve five different problems
- Vault
- Authenticates people and workloads, enforces path policies, encrypts stored data, and serves the secrets each caller is allowed to read.
- Integrated Raft
- Replicates Vault's ordered changes. The active node proposes a write and a majority records it before the write is accepted.
- Transit auto-unseal
- A separate, limited Vault service performs the cryptographic operation that opens a cluster node. The unseal key is not stored on the cluster node.
- Keepalived and VRRP
- The cluster nodes elect the owner of one shared service address. Health checks lower the priority of a node whose Vault process is sealed or unavailable.
- Prometheus and blackbox exporter
- Measure seal state, cluster health, served TLS certificates, and missing endpoints. A stopped observer is reported separately from an unhealthy Vault service.
Raft lets the surviving majority continue
Raft is a consensus protocol built around an ordered log. Vault sends writes to the active node. That node assigns the next log position and asks its peers to record the same entry. Once a majority has it, Vault can acknowledge the change. A surviving majority therefore has the complete accepted history needed to elect another active node.
The current 3-member cluster is the smallest useful shape for this design. An even pair cannot retain a majority after either member fails. The current shape can lose a member and continue. If it loses its voting majority, the remaining member refuses new writes because it cannot know that another partition is not also acting alone.
The nodes are spread across physical compute hosts. Raft protects against a node or guest failure; separate placement prevents one hypervisor failure from removing the whole voting majority.
What happens to a read and a write
- A workload authenticates to the stable service address. Keepalived has placed that address on a node whose local Vault health check passes.
- The receiving node identifies the active cluster member. Requests reach the active node, directly or through Vault's request forwarding.
- A read is checked against the caller's policy. Vault returns only the paths and fields granted to that identity.
- A write enters the Raft log. A majority records the entry before Vault reports success.
- Monitoring checks the service from outside. It asks whether the endpoint answers, whether it is sealed, and which certificate it serves.
The key that opens Vault lives somewhere else
A Vault process starts sealed. It can see encrypted storage but cannot derive the keys needed to read it. An operator can enter recovery material manually, but that would turn every restart into attended work.
The cluster uses Vault's Transit secrets engine for auto-unseal. Transit stores an encryption key and performs approved encrypt or decrypt operations for authenticated callers. Each cluster node stores its unseal material in wrapped form. At startup, the node authenticates to Transit and asks it to decrypt that wrapped material. Transit performs the operation without returning its own underlying key.
The Transit service is smaller than the main cluster and has a narrower job. That separation removes a readable unseal key from the cluster disks and their backups. It also creates a dependency that has to be named: a cluster node that boots while Transit is unavailable stays sealed.
The service address follows Vault health
Keepalived uses VRRP, a small election protocol, to place one address on the preferred healthy node. Callers do not need to know which cluster member is active or which machine currently owns the address.
A host-level check would be too weak. A sealed Vault node can keep answering ping and keep its network interface up while every secret request fails. Each node therefore checks its own Vault process. A sealed or unavailable process lowers the node's election priority and causes the address to move.
The check runs every five seconds and requires consecutive failures before moving the address. The expected handoff is about ten seconds. The check also records whether the process was sealed or unreachable, because those states produce the same external symptom and need different repairs.
| Condition | Cluster response | Caller experience |
|---|---|---|
| Healthy preferred node | Owns the address and participates in Raft. | The normal Vault URL answers. |
| Vault seals while host stays up | Local check lowers priority; a healthy peer takes the address. | Requests resume through the same URL after election. |
| A member is powered off | The remaining members retain a voting majority. | Reads and writes continue. |
| Voting majority is unavailable | No majority accepts writes. | The cluster stops rather than risk divergent state. |
| Transit unavailable during reboot | Booted node stays sealed. | Existing unsealed peers continue; recovery is needed if quorum cannot be restored. |
Clients verify the cluster they reached
The shared address, the cluster members, and the Transit service use certificates issued by the internal Active Directory Certificate Services authority. Managed callers trust that root, which lets them verify both the certificate chain and the name they intended to reach.
Blackbox exporter performs a TLS handshake against the endpoints that clients use. Prometheus warns when a certificate has 30 days remaining and escalates at 7. A coverage alert counts the endpoints that should report, so a failed scrape cannot disappear from the expiry view.
The certificate lifecycle project now automates the wider renewal and propagation paths. Vault still needs the same last-mile check as every other consumer: the certificate presented on the network must match the material intended for that endpoint.
Backup and restore cover different failure domains
The backup platform captures the Vault cluster guests and the separate Transit guest. A second storage system receives a nightly mirror, so loss of the primary backup server does not remove the only recovery copy.
The weekly restore-verification pipeline rotates through protected systems and restores into an isolated scratch network. That path checks whether the backup can produce a booted, usable guest. Vault also received a separate live-fire recovery on 2026-07-12 that reached initialized and unsealed state in 97 seconds. That measurement covered the tested single-server recovery path, not a full loss of both the cluster and Transit service.
How it runs now
| When | What runs | What a quiet result means |
|---|---|---|
| On every client request | Vault authenticates the caller and enforces its policy. | The requested field is returned only to the registered role. |
| Every five seconds per node | Keepalived asks whether local Vault is unsealed and answering. | The healthy owner keeps the shared address. |
| Continuously | Prometheus records service, seal, cluster, and TLS signals. | Missing evidence is visible as a coverage failure. |
| Nightly | Backup storage captures the guests and mirrors the protected data. | Both backup layers received current material. |
| Weekly rotation | The recovery pipeline restores the next registered workload. | Boot, filesystem, service, and teardown checks passed. |
What broke and what remains limited
The old auto-unseal weakened the backup boundary
The single server read its unseal key from the same disk that held Vault data. Nightly image backups captured both. The correction moved cryptographic unseal authority to the separate Transit service and kept operator recovery material offline.
A live host could still be a dead Vault endpoint
The first address design centered on node availability. A sealed process left the machine reachable, so a machine-level election could keep the address on a useless node. The current check asks Vault directly and makes a failed service surrender the address.
The unseal-service loss procedure is still unmeasured
The written operator path has not been rehearsed. The next useful recovery exercise is to remove Transit during a controlled cluster restart, follow the offline procedure, and record the result without exposing recovery material.
If I were recreating this from scratch
- Inventory every caller before changing storage. Record each authentication method, policy, and consumer so migration validation can compare complete path lists without printing secret values.
- Place the voting nodes before building them. A majority must survive the physical failure the cluster is meant to cover.
- Build and restrict Transit first. Give the cluster nodes only the encrypt and decrypt operations required for auto-unseal.
- Create the Raft cluster and stable address. Test service health, not host reachability, in the address election.
- Migrate with before-and-after inventory comparison. Do not retire the old server until the path sets match and known consumers authenticate through the new address.
- Test a sealed-but-running node. Power-off tests miss the failure that a ping check cannot see.
- Exercise recovery outside the cluster. Back up, restore, and rehearse loss of the unseal dependency as separate tests.