How the lab fits together from the public edge to recovery
A browser request enters through an outbound tunnel, crosses identity policy, and reaches an application in Kubernetes. Infrastructure jobs start in Jenkins or Gitea Actions, read credentials from Vault, change virtual machines through Terraform and Proxmox, and configure operating systems with Ansible. Monitoring observes each layer, and the recovery path can reconstruct protected guests from two backup stores.
The map below is organized by responsibility. The 5-node Proxmox cluster supplies compute. K3s supplies the application platform across 3 nodes, including 1 control node. Directory, certificate, secret, mail, monitoring, backup, and AI services sit beside those paths because applications depend on them without owning them.
Built from the current architecture page, generated topology facts, provisioning and certificate paths, Vault HA design, segmented network inventory, monitoring stack, and recovery workflow.
Also in Platform:
- Engineering capabilities. A short overview of the skills used across the projects. Read more →
- Roadmap. What is already live, what is still being built, and where the next trade-offs sit. Read more →
- Vault. The secrets system, its measured recovery, and the limits that are still named plainly. Read more →
- Windows identity. Directory, certificate authority, MFA, and the identity failure that is still shown as an active investigation. Read more →
- Mail. Self-hosted mail with the delivery path, DNS records, and operational limits explained. Read more →
- Recovery test. The restore check that destroys a VM on purpose and times the recovery path. Read more →
The map separates traffic, execution, trust, and evidence
Reading downward follows a normal application request. Reading across the middle shows the shared services an application or pipeline calls. Reading upward from the bottom follows monitoring and recovery evidence back to the operator.
The diagram uses roles rather than internal names. The live operator map carries hostnames, addresses, guest placement, and management links. The public page keeps the relationship and leaves out the identifiers needed to operate the environment.
A browser request never starts with an inbound port-forward
- Public DNS resolves the application name. The record points to the edge service, not to a private cluster address.
- Cloudflare applies the route policy. Public portfolio and status routes are allowed. Protected applications redirect to Entra sign-in and evaluate group membership.
- An existing outbound tunnel carries approved traffic inward. A connector inside the lab established the connection first, so the firewall does not need a general inbound rule for the application.
- Traefik receives the hostname in Kubernetes. This ingress controller selects the service and presents the public certificate.
- The application handles the authenticated request. It may read secrets from Vault, identity from directory claims, data from its own store, and metrics through the platform libraries.
- Blackbox and application monitoring observe the result. They check the page, certificate, protocol, and workload state from paths outside the application process.
The public wildcard certificate follows the same boundary. Let’s Encrypt issues it through cert-manager, one Kubernetes Secret owns the source pair, and Jenkins distributes that pair to registered services outside the source namespace. A handshake against each public name verifies what a browser receives.
An infrastructure request moves through owners, not one all-powerful tool
Jenkins is the main front door for infrastructure work. Backstage can create the same reviewed request and let Gitea Actions apply it. Both paths use the same underlying scripts and definitions.
| System | What it owns | What it hands to the next step |
|---|---|---|
| Jenkins or Gitea Actions | Request inputs, approval, stage order, logs, notification, and failure handling. | A validated request and execution context. |
| NetBox | Address allocation and the intended infrastructure inventory record. | A reserved address and canonical server facts. |
| Terraform | Requested infrastructure definition, plan, apply, and state. | An approved provider change. |
| Proxmox provider | Translation from Terraform resource operations to virtualization API calls. | A create, update, move, or delete request to Proxmox. |
| Proxmox | Virtual machine lifecycle, compute placement, disks, and virtual networks. | A running guest reachable for configuration. |
| Ansible | Operating-system configuration, packages, domain integration, monitoring, and security controls. | A configured server and verification results. |
| Vault | Credentials read at run time by each narrow workload identity. | Only the secret fields permitted for that stage. |
The committed host record and monitoring enrollment make the result discoverable after the pipeline ends. Backup policy and the generated topology then pick up the same recorded server.
Public and internal trust have separate authorities
Let’s Encrypt signs names that public browsers must trust. Cloudflare provides DNS validation for the wildcard request. cert-manager owns renewal inside Kubernetes, and the fan-out pipeline moves the source pair to the registered consumers.
Active Directory Certificate Services signs private service identities. Managed Windows systems use templates and auto-enrollment. Linux systems create keys locally and submit approved certificate requests. The internal root is distributed to the managed callers that need it.
Vault handles application credentials and API tokens. A client secret is not a certificate, but the lifecycle shape is similar: create a replacement, place it in custody, activate the consumer, test the live path, keep overlap, and retire the recorded predecessor.
Recovery starts below the application and verifies above it
Proxmox Backup Server stores protected guest backups. A second storage system receives a mirrored copy so the primary backup server is not its own final recovery point.
The weekly restore pipeline selects a protected system class, restores it into a reserved scratch guest, attaches a network with no production uplink, boots it, checks the operating system and application, removes the scratch copy, and verifies absence. That path begins at storage and ends at application behavior.
Configuration and source code follow a different recovery path through Git, Argo CD, Terraform state, and Ansible. The architecture uses both approaches because restoring a guest image and reconciling a declared workload solve different recovery problems.
What each region is for
- Edge
- Public DNS, identity policy, and the outbound tunnel decide how a browser reaches the platform.
- Network
- Firewall policy, routing, private mesh connectivity, and segmented server, management, storage, and user networks limit which systems can talk.
- Compute
- The Proxmox cluster runs virtual machines and can move or restart guests across physical nodes.
- Kubernetes
- K3s schedules platform applications, controllers, monitoring exporters, and internal services across its control and worker nodes.
- Delivery
- Gitea stores code and review state; Jenkins and Gitea Actions coordinate pipelines; Argo CD reconciles registered Kubernetes applications from Git.
- Secrets
- Vault authenticates workloads, applies path policy, rotates registered credentials, and serves sensitive values at run time.
- Identity
- Active Directory, Entra, Duo, and application roles identify people and machines and apply sign-in policy.
- Certificates
- Let’s Encrypt covers public names; AD CS covers private identities; live probes verify the certificate presented by each service.
- Monitoring
- Prometheus, Grafana, Zabbix, logs, and dead-man checks report state, trend, missing evidence, and drift.
- Backup
- PBS, mirrored storage, configuration repositories, and scheduled restore tests provide recovery paths with different failure boundaries.
- AI
- Subscription-backed repository agents and gateway-routed operational inference run under the delivery, identity, budget, and evidence controls above.
The architecture is also a list of failure boundaries
| Failure | Boundary that contains it | Evidence after containment |
|---|---|---|
| One physical compute node | Cluster placement and shared storage let eligible guests run elsewhere. | Node, guest, and service state in monitoring. |
| One Vault cluster node | Raft majority and the service-aware shared address keep the credential endpoint available. | Seal state, quorum, address owner, and client request. |
| Public certificate renewal | One source certificate feeds registered copies with consumer-specific reload. | Live handshake fingerprint and expiry per application name. |
| One Kubernetes pod | The controller recreates the workload from its declared state. | Rollout state, service endpoint, and application probe. |
| Monitoring producer | Dead-man items alert on missing heartbeat or stale result. | Age, no-data trigger, and second-plane host check. |
| Primary backup storage | The mirrored store preserves a separate recovery copy. | Freshness by source and a restore from the selected copy. |
| Configuration drift | Argo CD, CI checks, and scheduled reconciliation compare declared and live state. | Named resource or dashboard field that differs. |
How it runs now
| When | What coordinates the work | What changes |
|---|---|---|
| On a browser request | DNS, edge policy, tunnel, ingress, and application identity. | No infrastructure change; the request reaches one registered service. |
| On a reviewed infrastructure request | Jenkins or Gitea Actions with NetBox, Terraform, Proxmox, Ansible, and Vault. | Compute, configuration, inventory, monitoring, and repository state. |
| Continuously in Kubernetes | K3s controllers and Argo CD compare desired and running resources. | Missing or drifted workloads are reconciled within declared authority. |
| Continuously in monitoring | Exporters, Prometheus, Zabbix, and Grafana update state and alerts. | Evidence changes; the monitored services are not mutated by the dashboard. |
| On a lifecycle schedule | Certificate and credential workers evaluate renewal, overlap, and retirement. | Registered trust material is replaced and consumers reload. |
| On recovery schedule | PBS, Proxmox, Jenkins, guest verifiers, and cleanup checks. | A fenced scratch guest is created, tested, and removed. |
If I were recreating this map
- Start with three journeys. Trace a browser request, an infrastructure change, and a recovery run from beginning to end.
- Name the owner at each handoff. The coordinator, source of truth, executor, credential source, and verifier should be distinct.
- Draw network and trust boundaries before product boxes. Show which paths are public, private, identity-protected, or fenced.
- Add observation beside execution. Every important change needs evidence from outside the process that made it.
- Mark shared dependencies. Secrets, identity, DNS, certificate authority, storage, and Git should be visible where several paths depend on them.
- Generate counts and labels that drift. The site build should read current topology facts rather than preserving a hand-drawn fleet size.