Bill Gernert
Platform & Orchestration

Kubernetes / K3s

A 3-node K3s cluster, one control plane and two workers, running the platform's own services - a real Kubernetes, sized for the lab, with tradeoffs made on purpose rather than by accident.

What it is and whyHow I use it inArchitectureLive Kubernetes evidenceLessons learned and gotchasImpact

What it is & why I use it

K3s is a lightweight, fully conformant Kubernetes distribution. It is the service plane of AutomationLab: the CI controller, the GitOps engine, ingress, and the lab's own applications all run as workloads on it. Terraform makes VMs; Kubernetes is where the things that run on those VMs live.

I chose K3s over full upstream Kubernetes because it gives me the same API and the same mental model with a fraction of the operational weight - the right size for a lab that needs to be real Kubernetes, not a toy, without a control plane that dwarfs the workloads it runs.

How I use it in the lab

  • The platform's services run here. The automation controller, GitOps reconciler, ingress, and lab applications are all Kubernetes workloads, deployed and reconciled from git.
  • Control plane plus spread workers. A control-plane node runs the API and scheduler; worker nodes carry the workloads and are spread across separate physical hosts so a single machine failure does not take everything with it.
  • Declarative, GitOps-managed. What runs on the cluster is declared in git and reconciled continuously - the cluster is cattle, described in code, not a hand-tended pet.
  • Secrets and ingress wired in. Secrets come from the secrets manager and traffic arrives through the ingress behind the zero-trust edge - the cluster is a citizen of the wider platform, not an island.

Architecture

Kubernetes / K3s architecture: Git-reconciled K3s platform Git defines the desired state, Argo CD reconciles it through the K3s control plane, workers run the workloads, and ingress presents them behind the edge. ARCHITECTURE FLOW GIT-RECONCILED K3S PLATFORM Git: Desired workload state GIT Desired workload state Argo CD: Diff and reconciliation ARGO CD Diff and reconciliation K3s control plane: Schedules and manages workloads K3S CONTROL PLANE Schedules and manages workloads Worker nodes: Run the requested workloads WORKER NODES Run the requested workloads Ingress: Routes requests to services INGRESS Routes requests to services Edge: Published access path EDGE Published access path Cyan. Automated step or path. Green. Recorded or healthy outcome. Dashed box. Stored state or record. Rounded box. Actor or process. Arrow. Direction of work or data.
The control plane and workers are spread across hosts, with workloads reconciled from Git and ingress behind the edge. Node addresses and hostnames are omitted.

Live Kubernetes evidence

The public dashboard shows component availability, running workload counts, and CPU and memory use by purpose. Pod hashes, namespaces, node names, images, addresses, and management links are removed before Grafana can query the public series.

Checking the public Kubernetes dashboard.

Open Kubernetes full screen →

Lessons learned & gotchas

Gotcha - a stale kubeconfig makes every command fail identically One of the most time-wasting failures is not in the cluster at all - it is in the client config. When a kubeconfig points at a control-plane endpoint that no longer exists (a rebuilt or re-addressed control node), every kubectl and every tool that shells out to it fails with the same timeout, and it looks like the cluster is down when the cluster is fine. The tell is that everything fails the same way at once. The fix is to point the client at the live API endpoint - I keep an explicit API server override handy so tooling can reach the real control plane even when a config on disk is stale.
  • The single control plane is a deliberate SPOF. Spreading workers buys resilience for workloads, but 1 control-plane node means losing it stops scheduling and API access. That is an accepted lab tradeoff - the point is that it was chosen and documented, so nobody mistakes it for high availability it does not have.
  • Distinguish "cluster down" from "I cannot reach it." Client-side, DNS, and edge problems all masquerade as an outage. Before declaring the cluster broken, prove you can actually reach the API from a known-good client.
  • Lightweight is a feature, not a compromise. Because the distribution is the same API, everything learned here transfers to full Kubernetes - the smaller footprint just removes weight the lab does not need.

Impact

1
service plane for the lab
Spread
workers across hosts
GitOps
managed workloads

K3s is the substrate everything user-facing runs on. The lasting lesson is less about Kubernetes and more about honesty: name your single points of failure out loud, and learn to tell a real outage from a client that is simply pointed at the wrong place - because those two look identical right up until you check.

Further reading

K3s docs