Disaster recovery walkthrough

How I verify that a backup becomes a working server

The scheduled job rotates through a recovery manifest, restores a protected guest from backup into a fenced scratch network, boots it, runs operating-system and service assertions, removes the temporary copy, and confirms that the copy is gone. The report and dashboard below show the outcome without exposing the systems being recovered.

The path took twelve reruns to reach the intended assertion set. The failures were not variations of one bug. The job bypassed its scoped credential, the runbook assumed missing preconditions, effective permissions collapsed to an empty intersection, TLS verification targeted the wrong identity, and a bare boolean destroyed the reason a restore assertion failed.

last run 2026-09-20 · PASS · 181 seconds

Built from the weekly restore design, Jenkins pipeline, restore verifier, dated live-fire result, failure saga, backup digest template, and public recovery dashboard.

View the code →

Backup verification and restore verification answer different questions

Proxmox Backup Server can verify that stored chunks and indexes are readable. That is necessary, but it does not show that Proxmox can reconstruct a guest, that the guest boots, or that the application inside it can perform its job.

The scheduled restore pipeline closes that gap. It chooses the next protected guest, chooses a backup source, creates a temporary guest under a reserved scratch identity, attaches it to a network with no production uplink, and boots it. Common assertions inspect the operating system. A service profile adds the checks that matter for the recovered workload.

The temporary machine is destroyed after success or failure. Cleanup then asks the platform whether the guest is absent. The delete command returning zero is supporting evidence; the absence check is the teardown result.

Separate live-fire measurement. On 2026-07-12, a controlled Vault recovery reached initialized and unsealed state in 97 seconds. That was a destructive single-service exercise. The weekly pipeline uses fenced copies so it can keep testing different protected systems without placing duplicates on the production network.

PBS, Proxmox, Jenkins, and the guest each own one part

Recovery manifest: Names the protected system class, eligible backup sources, required assertions, and scratch constraints. Rotation prevents the easiest guest from being tested forever.
Proxmox Backup Server: Stores deduplicated backups, verifies stored data, and supplies the selected snapshot for restore.
Proxmox: Creates the scratch guest, attaches the fenced network, starts and stops it, reports guest state, and removes it.
Jenkins: Coordinates preflight, selection, restore, boot, verification, evidence, notification, and unconditional cleanup.
Guest agent and verifier: Expose boot, filesystem, operating-system, identity, and service facts from inside the restored machine.
Prometheus and Grafana: Record run result, duration, age, coverage, and failure state independently of the email report.

The scratch guest can boot without joining production

Weekly restore path from manifest and backup selection through fenced boot, assertions, evidence, and verified teardown

The fenced network is a bridge with no uplink to production. A restored directory server, secrets server, or application can therefore use its original configuration without answering on the live address or contacting production peers.

The eight phases of one run

Preflight the control path. Jenkins verifies the manifest entry, backup source, reserved scratch identity, fenced network, scoped credential, and absence of an old scratch guest.
Select the next protected target. The rotation record chooses a system and backup datastore according to the manifest rather than operator convenience.
Resolve the exact backup. The pipeline records the selected snapshot and asks PBS to verify the stored material used for this run.
Restore into the reserved scratch identity. Proxmox reconstructs a new guest. The live guest is never overwritten.
Attach the fenced network and boot. The restored machine can initialize and answer its guest agent without reaching the production network.
Run common and service assertions. The verifier checks boot, filesystem, operating-system identity, guest-agent response, and the profile-specific service facts.
Write evidence and send the digest. Result, duration, target class, datastore class, assertion results, and cleanup state are recorded with public aliases.
Destroy and verify absence. Cleanup runs on success and failure, then queries the platform to confirm the scratch guest no longer exists.

Blast-radius rule. The run must never restore over a live guest, attach the scratch copy to a production uplink, reuse an unreserved identity, or continue when the expected target cannot be determined.

“Booted” is the first assertion, not the last

A hypervisor can mark a guest running while the operating system is stuck, the filesystem is damaged, the guest agent is absent, or the application service cannot read its data. The verifier divides checks into a common set and a registered service profile.

Assertion group	What it checks	Failure it catches
Hypervisor state	The restored guest exists, starts, and reaches the expected runtime state.	Restore or boot failure before the operating system responds.
Guest agent	The agent answers through the platform control channel.	A running guest that cannot report internal state.
Operating system	Expected family, disks, filesystems, and protected paths are present.	The wrong backup, incomplete disk, or damaged filesystem.
Identity	The recovered guest reports the expected system class without joining live peers.	A restored image that does not match the selected manifest entry.
Service profile	Registered processes, ports, files, or data checks succeed inside the fence.	A machine that boots but cannot perform the workload’s job.
Teardown	The reserved scratch guest is absent after deletion.	A leaked duplicate left behind by a successful command.

The verifier also records whether each planned assertion executed. That audit catches a conditional branch that silently skipped its check while leaving the surrounding stage green.

The daily report is designed to be read before the dashboard

The email names the selected recovery class, backup source class, result, duration, assertion summary, and teardown result. It carries enough context to decide whether the run was quiet, whether one assertion failed, or whether the pipeline stopped before it created anything.

Sanitized backup and restore digest rendered from the production email template — Backup email template rendered with sanitized sample recovery data. The first version embedded an HTML email document that rendered as a blank frame on the public page. The published image keeps the report visible without depending on nested document behavior.

A separate dead-man monitor watches the age of the latest successful digest. If the report disappears, silence does not read as a healthy recovery week.

Twelve reruns exposed different layers of wrong

The recovery path did not turn green by fixing one restore command. Each correction exposed the next assumption.

Attempt 1: the scoped credential was not on the execution path

The repository defined a narrow recovery token, but the command that performed the restore still ran through a broader administrative path. The job could pass while bypassing the permission boundary it claimed to test. The execution code was changed to use the scoped identity.

Attempt 2: the runbook assumed its preconditions

The first live walk found identity creation ordered after the rule that referenced it, a missing custom role, and an endpoint check that assumed failover had already been established. The runbook was rewritten in the order the platform actually requires.

Attempt 3: effective permission collapsed to empty

The platform calculates a token’s rights from the intersection of user, group, token, path, and role permissions. The configuration looked granted in individual places while the effective result was no permission. The verifier now queries effective access instead of reading one ACL in isolation.

Attempt 4: TLS checked an identity the cluster did not present

The client reached a cluster endpoint whose certificate names did not satisfy the original assertion. The correction aligned the test with the endpoint identity and trusted chain the recovery client actually uses.

Attempt 5: a bare False erased the cause

The restore assertion returned only a boolean. Jenkins could report failure but not the provider response that explained it. The verifier now carries a typed result with the failed step, response detail, and safe context.

Later reruns: assertions and cleanup had their own drift

A missing guest-agent permission stopped internal checks. One operating-system assertion asked for a fact that could never appear and waited through the full retry window. Cleanup trusted command success without verifying absence. Each became a named check with bounded retry and a separate teardown query.

One final development pass recorded 197 of 200 assertions passing. The remaining three stayed in the report as tracked gaps. That number belongs to that dated run and is not presented as a current fleet count.

Live backup and recovery evidence

The public dashboard shows freshness, result, duration, restore coverage, and active failure state under stable service classes. Guest names, infrastructure identifiers, addresses, backup paths, job parameters, and management links are removed before the public data source.

Live public Grafana view

Open full screen

Checking the live backup dashboard.

How to read it

Did the latest run complete?

Start with result and age

A pass needs a recent timestamp. An old pass is stale evidence, not current recovery coverage.

Which protection layer is behind?

Compare backup sources

Freshness by datastore shows whether the primary and mirrored paths both contain current recovery material.

Did recovery slow down?

Use the duration history

A longer restore can reveal storage, network, or guest-initialization change before it becomes a timeout.

Did the job clean up?

Use teardown state

The run is incomplete while the scratch guest still exists, even when every application assertion passed.

How it runs now

When	What runs	What happens next
On the weekly schedule	Jenkins selects the next manifest entry and eligible backup source.	Preflight must pass before a scratch guest is created.
On restore	PBS supplies the snapshot and Proxmox creates the isolated guest.	Boot and assertion phases begin under the scoped identity.
On an assertion failure	The verifier records the failed check and safe reason.	The run is red and still enters cleanup.
On cleanup	Proxmox deletes the scratch guest; the verifier queries for absence.	Notification distinguishes verified teardown from leaked state.
On rerun	Preflight checks that no previous scratch guest remains.	The same manifest entry can be exercised without manual state repair.
Between runs	Prometheus evaluates freshness, outcome, duration, coverage, and missing heartbeat.	An old or absent result cannot remain green.

If I were recreating this from scratch

Write a recovery manifest. Name the protected classes, backup sources, scratch constraints, and useful service assertions.
Reserve and fence the scratch environment. Prove that it cannot join the production network before booting a restored copy.
Create the scoped identity before the pipeline. Test effective permissions through the exact code path that will restore and delete.
Make every assertion return a reason. A boolean is not enough for a scheduled recovery failure.
Audit assertion execution. Record which planned checks ran so a skipped branch cannot produce an empty green stage.
Put cleanup in the unconditional path. Query for absence after deletion and report cleanup independently from application verification.
Add freshness monitoring. A passing run expires as evidence when the next expected run does not arrive.