Bill Gernert
Platform & Orchestration

Proxmox VE

The 5-node high-availability cluster that provides the compute foundation for the lab's VM estate.

What it is and whyHow I use it inArchitectureLessons learned and gotchasImpactFurther reading

What it is & why I use it

Proxmox VE is an open-source virtualization platform that combines KVM virtual machines and LXC containers with clustering, high availability, and software-defined storage under one API. I chose it as the base layer for AutomationLab because it gives me a real, API-driven hypervisor cluster - the same primitives a production platform team works with - without a per-socket licensing model getting in the way of experimentation.

The point of the lab is to treat infrastructure as something you declare and automate, not something you click together. Proxmox exposes everything through a REST API, which is exactly what lets Terraform stand VMs up from code instead of from a wizard.

How I use it in the lab

  • 5 nodes, clustered. The cluster runs as a single management plane with a shared quorum, so I administer all nodes as one system rather than 5 separate hosts.
  • High availability. Selected VMs are enrolled in HA groups; if a node fails, the cluster restarts those guests elsewhere automatically. Enrollment is the final, deliberate step of provisioning - not the default - so I control exactly what is allowed to migrate.
  • Terraform-driven. Each VM gets its own Terraform workspace using the bpg/proxmox provider. A pipeline plans and applies against the cluster API; there is no hand-built VM anywhere in the lab.
  • Dynamic load balancing. The cluster rebalances running guests across nodes based on load, so no single node becomes a hotspot as the lab grows.
  • Clean decommission. The provisioning flow reverses cleanly - HA de-registration, then destroy - so tearing a VM down is as automated as building it.

Architecture

Proxmox VE architecture: Five-node HA cluster Terraform calls the cluster API, the five-node HA cluster places the guest, and shared quorum and storage support migration and recovery. ARCHITECTURE FLOW FIVE-NODE HA CLUSTER Terraform provider: Sends the approved VM change TERRAFORM PROVIDER Sends the approved VM change Cluster API: One control surface for the nodes CLUSTER API One control surface for the nodes Five HA nodes: Guests can move between hosts FIVE HA NODES Guests can move between hosts Shared quorum: The cluster agrees on membership SHARED QUORUM The cluster agrees on membership Shared storage: Guest data stays available SHARED STORAGE Guest data stays available HA-managed guest: Placement and recovery stay coordinated HA-MANAGED GUEST Placement and recovery stay coordinated Cyan. Automated step or path. Green. Recorded or healthy outcome. Dashed box. Stored state or record. Rounded box. Actor or process. Arrow. Direction of work or data.
The cluster view shows the HA nodes, shared quorum and storage, and the Terraform provider at the API boundary. Addresses and node hostnames are omitted.

Lessons learned & gotchas

Gotcha Quorum is not optional. In a small cluster it is tempting to run an even node count, but you want a reliable majority for the quorum to survive a single-node loss without the whole cluster going read-only. Plan the vote layout before you need it, not during an outage.
  • HA is a promise you have to mean. Enrolling a guest in HA means the cluster will happily restart it on another node - including after a transient blip. Anything with node-local state has to be designed for that before you enroll it, or you will be surprised at the worst time.
  • The API is the contract. Once every VM comes from Terraform against the API, drift shows up as a plan diff instead of a mystery. Clicking a change in the UI just means the next apply wants to undo it - so I stopped doing that.

Impact

5
HA nodes, one plane
100%
VMs from code
0
hand-built guests

Proxmox turned "the lab" from a pile of machines into a single programmable substrate. Everything above it - GitOps, monitoring, identity - assumes it can ask for a VM and get one, reliably, from code. That assumption is what makes the rest of the platform possible.

Further reading

Proxmox VE