Bill Gernert
CI/CD & GitOps

Kaniko

Daemonless builds for the lab's 5 custom images, each run as an ordinary pod inside the cluster - no privileged Docker socket, and every build trusted only after its digest changes.

What it is and whyHow I use it inArchitectureLessons learned and gotchasImpact

What it is & why I use it

Kaniko builds container images from a Dockerfile inside a Kubernetes pod, without a Docker daemon and without needing the privileged access that mounting the host's Docker socket would require. In AutomationLab it is how the custom images the pipelines depend on - like the provisioning agent image - get built and pushed to the registry.

I use it because the alternative, bind-mounting a Docker socket into CI, hands the job a root-equivalent foothold on the node. Kaniko keeps image builds inside the same least-privilege, scheduler-managed world as everything else the cluster runs.

How I use it in the lab

  • Builds as pods. A build is a one-shot pod: it checks out the context, runs the Dockerfile, and pushes the result to the registry, then completes.
  • Agent images for the pipelines. The tools a provisioning run needs - the IaC binaries, configuration-management collections, cloud SDKs - are baked into an agent image built this way, so pipeline runs start from a known, versioned toolset.
  • Registry push with scoped credentials. The push credential is provided to the build pod as a mounted secret, not embedded in the image or the manifest.
  • Digest-verified promotion. A build is only "done" once I have confirmed the pushed digest changed from the previous build - see the gotcha below, which cost me real time to learn.

Architecture

Kaniko architecture: Rootless image build A Git context enters a Kaniko build pod, the pod builds and pushes the image, a digest check records the immutable result, and pipelines consume it. ARCHITECTURE FLOW ROOTLESS IMAGE BUILD Git context: Dockerfile and build inputs GIT CONTEXT Dockerfile and build inputs Build pod: Kaniko runs without a Docker daemon BUILD POD Kaniko runs without a Docker daemon Image build: Assemble and snapshot layers IMAGE BUILD Assemble and snapshot layers Registry push: Publish the completed image REGISTRY PUSH Publish the completed image Digest check: Record the immutable identity DIGEST CHECK Record the immutable identity Pipeline use: Consumers pin the verified image PIPELINE USE Consumers pin the verified image Cyan. Automated step or path. Amber. Operator or policy gate. Green. Recorded or healthy outcome. Dashed box. Stored state or record. Rounded box. Actor or process. Arrow. Direction of work or data.
The build flow runs from Git context through a Kaniko build pod and registry push to digest verification and pipeline use. Registry addresses and hostnames are omitted.

Lessons learned & gotchas

Gotcha - a "successful" build log can be a lie A one-shot Kaniko build pod can silently reuse a stale, already-completed pod. If the pod already exists, kubectl apply reports configured, not created, and does not re-run anything - so you happily tail logs from an old build and conclude the new one worked. Nothing errors. You just deployed the previous image and believed it was the new one.

The rule I now follow every time, without exception:

  • Delete the build pod before applying. Never rely on apply to replace a completed pod; remove it first so the new build actually runs.
  • Confirm apply reports created, not configured. That one word is the difference between a fresh build and a no-op.
  • Trust the digest, not the log. Verify the pushed image digest changed from the prior build. A green log proves a pod ran; only a changed digest proves this build produced this image.
  • Verify by digest everywhere downstream. The same principle carries into how the pipelines pin images: a tag can move, a digest cannot, so pinning by digest is what makes "which image ran?" answerable after the fact.

Impact

0
privileged Docker sockets
Digest
the real proof of a build
1 image
versioned pipeline toolset

Kaniko lets the lab build its own tooling images the same way it runs everything else - as scheduled, unprivileged workloads. The lasting lesson was less about Kaniko and more about verification discipline: a log that says "success" is not evidence the thing you intended actually happened. Checking the digest turned a class of silent, confusing failures into something I can no longer fool myself about.