Template factory

The Windows template factory

I use Jenkins, Kubernetes, and Packer to rebuild Windows Server templates in Proxmox every week. Each replacement is installed, updated, generalized, and verified before provisioning starts using it.

The current template stays available throughout the build. Provisioning resolves a stable OS tag, so I can replace the image underneath it without changing the provisioning pipeline.

View the code →

Jenkins runs the factory. Packer builds the image.

A weekly Jenkins pipeline checks out the Packer definitions and build scripts from Git, then starts disposable build agents in Kubernetes. Each Windows release has its own agent and a pair of template slots in Proxmox.

Packer installs Windows into the inactive slot, applies Windows Updates, adds runtime initialization tooling, and runs Sysprep to generalize the guest. Vault supplies the build credentials. A Python lifecycle script checks the resulting template against the Proxmox API before it can be promoted.

Jenkins starts a disposable Kubernetes Packer agent to build and verify a Windows template in Proxmox. — Jenkins schedules the work; the Kubernetes agent runs Packer; the lifecycle script controls promotion.

A slow build does not hold up provisioning

Last successful build: September 15, 2026 · about 77 minutes. That is time spent preparing the next image. A request for a new server can still clone the current verified template while that work runs.

The separation comes from template tags. Provisioning asks for a role such as os-2025 instead of a fixed VM ID. That tag stays on the current template while its replacement is built in the inactive slot.

There is always a verified template available during a rebuild. The candidate receives the live tag only after verification. The previous image is then marked as the fallback. If the build fails, the live tag stays where it was and provisioning continues from the same image.

Current and candidate templates coexist. The live tag moves to the candidate after verification, and the previous image remains a fallback. — Build in the inactive slot. Verify the replacement. Move the tag. Keep the previous image for recovery.

I still care about build time for capacity and backup scheduling. It just does not put image maintenance in the path of every new server request.

The templates are updated, not left to age

Weekly rebuilds create a fresh Windows image and apply updates before promotion. New servers inherit the updated baseline as soon as their provisioning run resolves the new live tag.

For an on-demand patch, a separate Jenkins job clones the current template into a working VM. Ansible applies Windows Updates over WinRM, runs Sysprep, and shuts the guest down before it is converted back into a template and promoted.

Both paths prepare the replacement separately from the image serving provisioning. Updating a template changes the baseline for future clones; existing servers still need their own patching.

Jenkins + Kubernetes: Schedule the rebuild and provide disposable, isolated Packer agents.
Packer + Sysprep: Build a clean, updated Windows image that is ready to clone.
Ansible + WinRM: Apply updates to a working copy when I need an on-demand template patch.
Proxmox + tags: Keep the current image available and let provisioning follow the verified replacement.

Why I stopped patching the same image in place

The first version repeatedly booted, patched, generalized, and shut down the same template. Pending Windows work and Sysprep failures accumulated until image maintenance could block provisioning.

I changed the design so that an image is a replaceable release artifact. The build can fail without taking the current template with it. The stable interface is the OS tag, backed by a verified image.

Build safeguards and recovery

Build definitions. The repository contains 3 Windows variant definitions; the Jenkins matrix selects the weekly releases.
Check the backup window. The factory reads active and scheduled Proxmox backups before starting a long build.
Keep releases isolated. Each OS uses a separate template pair and Kubernetes agent. Matrix builds can run alongside one another; overlapping pipeline runs are disabled.
Verify authoritative state. The lifecycle script reads the node configuration, because the cluster cache can briefly report a converted template as a VM.
Bound failed work. Per-image timeouts stop a stalled build. Failure cleanup targets staging and leaves the current live template in place.
Recover a promoted weekly image. Move the live role tag back to the retained previous template so subsequent provisioning uses it.