How a reviewed manifest becomes the running Kubernetes state

I added Argo CD after deployment and desired state had started to separate

Argo CD keeps part of my Kubernetes cluster matching what git says it should be. The manifests for eight general application directories live in the lab repo. Argo CD watches those files, compares them against what the cluster is actually running, and reports where the two disagree. Those paths wait for a reviewed sync. A separate credential-lifecycle application self-heals inside a bounded monitoring scope because its rotators are already live and change frequently as they are proved.

The boundary is worth stating plainly, because most GitOps writeups skip it. Not everything on this cluster is Argo CD's. Jenkins deploys itself and the metrics stack through its own Helm pipelines, and those are deliberately kept outside the GitOps surface so the two systems never fight over ownership of the same objects. Durable changes for the registered Argo CD paths begin in git. Other cluster surfaces still use Jenkins or an operator runbook, so GitOps is a defined slice of the cluster rather than a claim about the whole platform.

The first thing to unlearn: Argo CD never builds anything. Jenkins builds the images, Kaniko does the container work inside the cluster, and the Gitea registry holds the result. Argo CD's whole job starts after all of that, when a manifest in git changes.

What reconciliation actually means

Reconciliation is a loop, not an event. Argo CD holds two pictures at all times. The desired state is what the manifests in git declare: this deployment, this image digest, three replicas. The actual state is what the Kubernetes API reports is really running. Every few minutes, and immediately on a push, Argo CD diffs the two.

When they match, the app shows Synced and nothing happens. When they differ, that difference has a name: drift. Drift comes from two directions. Either git moved (someone merged a new image digest) and the cluster is behind, or the cluster moved (someone hand-edited a deployment, a node ate a pod's config) and git is being disobeyed. Argo CD treats both the same way. Git is right.

Self-heal is the setting that decides how hard git wins. It is off for the eight general application directories. Several had not been applied through Argo CD when the comparison path was introduced, and they carry values meant to be resolved at apply time. Enabling self-heal there could have pushed them live as a side effect of adoption. Those applications therefore ship detection first, with action behind a reviewed sync.

The credential-lifecycle application has a narrower boundary. It alone uses automated self-heal, within the monitoring namespace and the lab-raw-manifests project, while pruning remains disabled. The root application also has no automated sync. This gives a frequently changing live control path automated correction without widening that authority to the rest of the registered applications.

Where it sits in the platform

This is one code change traveling the whole road. Every box is a real system in this lab, and the diagram marks which steps run on their own and which ones wait for me.

Two handoffs matter more than the rest. The first is the merge: nothing below it happens without the gates passing, so a leaked secret or a manifest that does not validate dies at the pull request. The second is the sync, and that is where the diagram stops being automatic. Argo CD will sit on a correct, reviewed, already-merged change indefinitely and deploy nothing until someone syncs it. That is the trade this lab took deliberately, and it is the one thing a diagram of somebody else's GitOps setup will not prepare you for.

One detail worth naming because most writeups skip it: images are pinned by digest, not by tag. A tag like latest can move under you silently. A digest cannot. The build writes the digest it actually pushed to a file, a promote script reads that digest and rewrites a committed pin, and a dependency bot turns any later change to that pin into a reviewable pull request. Two honest limits on that. The pin currently feeds the Jenkins build agent image, which is one of the Helm surfaces Argo CD does not manage, and the promote step is a command a person runs rather than a pipeline stage. Wiring the same digest pin into the manifests Argo CD does track is the obvious next move, and it is not built.

How it relates to the rest of the lab

Kubernetes. Argo CD does not place pods, pull images, or route traffic. It edits Kubernetes objects and lets the cluster's own controllers do the work. When Argo CD syncs a Deployment, the Deployment controller makes a ReplicaSet, the scheduler picks nodes, the kubelet on each node pulls the image and starts containers, and the readiness probe decides when traffic arrives. Argo CD is a client of the Kubernetes API like everything else. That is why it can be down without anything breaking: the cluster keeps running the last synced state.

Gitea. Gitea is three things to this flow at once: the source of truth (the repo holding the manifests), the gatekeeper (Actions runs the merge checks), and the registry (the built images live there too). A push fires a webhook that tells Argo CD to re-read git, so the comparison is prompt instead of waiting on the poll interval. Argo CD reads that repository, and nothing in the GitOps path writes a commit back.

Jenkins. Jenkins and Argo CD split the world, but not cleanly, and not yet. Jenkins builds the custom images in this delivery flow, and it still applies two Helm releases into the cluster itself, so it does still hold cluster credentials. What it does not do is deploy the eight general directories or the bounded credential-lifecycle path Argo CD tracks; no Argo CD application points at a Jenkins-owned path, the Argo CD project is fenced to this repository and a named set of destinations, and no application is allowed to prune. Those three checks were run live. Narrowing Jenkins to build-only is the direction of travel, not the current state, so the honest version of the security argument is that the blast radius of a stolen pipeline credential got smaller, not that it went to zero.

Vault. Secrets never enter git, so they never enter this flow. Manifests reference Kubernetes Secrets by name; the values come from Vault through committed create-secret scripts run at deploy time. Wiring External Secrets Operator to make that pull automatic is committed and not yet applied, and this page will say so until it is running.

The boundary matters: Argo CD governs a slice of the Kubernetes half of this lab and nothing else. The VMs, the domain controllers, the mail platform, and the hypervisor cluster are Ansible and Jenkins territory. One platform, two reconciliation stories, and only one of them is Argo CD.

How it runs now

What broke while I was building it

Argo CD is down. Symptoms: apps stop syncing, the interface is unreachable. What does not happen: nothing running breaks. The cluster holds the last synced state because Kubernetes, not Argo CD, runs the workloads. Diagnosis: pod status in its namespace. Recovery: bring the pods back; Argo CD re-reads git and catches up, because git never forgot. Prevention: the same dead-man monitoring every other component here gets.

Git is unreachable. Symptoms: apps show Unknown, comparisons stall. Again nothing running breaks, changes just freeze. This one matters because it is the failure I have actually lived: when the lab lost power in August 2026, the fleet's own rule was to refuse to act on state it could not verify. Argo CD behaves the same way by design. No git, no changes.

A bad manifest merges. Symptoms: the sync fails or the app goes Degraded, the readiness probe never passes, old pods keep serving. Diagnosis: the app's sync status names the exact resource and error. Recovery: git revert, then sync. That is the whole rollback story, one commit and one click, and Argo CD walks the cluster back. Prevention: a manifest schema check and the other merge gates catch most of this class before it merges at all.

The registry is unreachable. Symptoms: new pods stick pulling the image while old pods keep serving. Recovery: fix the registry; the rollout resumes on its own. This is the quiet argument for readiness probes: Kubernetes never routed traffic to the broken half.

Drift war. This is the failure self-heal can create. In this lab the risk is limited to the credential-lifecycle application. A hand-edited value in that scope can snap back seconds after you change it, and the fix is to change the manifest rather than the cluster. If a controller and Argo CD genuinely disagree about a field, that field gets an ignore rule, which is a one-line declaration in the app's spec. The neighbouring lesson I did live is worth more: a sync only reconciles the fields the committed manifest actually owns. The first version of my drift demo changed a label git had never set, the sync correctly left it alone, and that would have read as a broken demo. Moving the demo to the replica count, a field git does own, made it honest.

Why Argo CD, and when it is the wrong tool

The alternative I actually had was Jenkins running apply and Helm, and it worked. It still runs, for the surfaces Argo CD does not manage. What it could not give me: a deploy path that survives Jenkins being compromised, drift detection between deploys, and a rollback that is just a git revert. A push-based deployment path keeps mutation credentials available to the build system. Pull-based reconciliation lets the cluster read git instead. For the paths moved behind Argo CD, a stolen build credential alone cannot mutate the registered resources; a git change still has to pass the merge gates. This lab is partway along that road, not at the end of it: eight general directories are review-gated and one credential-lifecycle path self-heals.

When it is the wrong tool: anything that is not Kubernetes. Most of this lab is VMs, and forcing Argo CD into that story would be resume-driven engineering. Ansible reconciles the VM estate on a schedule for the same reason Argo CD reconciles the cluster continuously. Same idea, different clock. A single-node app on one box needs neither; a service unit and a git pull get you surprisingly far, and pretending otherwise is how platforms grow parts nobody can explain.

What I would do differently

The concept that unlocked it for me was realizing reconciliation is not deployment. Deployment is an action someone performs. Reconciliation is a standing disagreement-resolver that happens to deploy things as a side effect. Once I saw that, the rest of the design fell out: of course Argo CD does not build, of course git is the door, of course rollback is revert.

The mistake I nearly made: adopting Argo CD one directory at a time, I reached one that commits an empty Secret to git whose real content is created on the live cluster. Argo CD would have seen drift it could never win, forever. The fix was a ruling rather than a workaround: never commit an object whose real content is born outside git. Drop the Secret, adopt the ServiceAccount and the access rules around it. That directory is still not adopted, which is the honest state of it, and the ruling is the reason it is not adopted badly.

The advice I would give: do not start with Argo CD. Start with your manifests in git and apply them from CI, feel the drift problem for real, then adopt reconciliation and let it sting you once. Self-heal reverting your own hand-edit at two in the morning teaches GitOps better than any diagram. I enabled it first on the credential-lifecycle application because that path is live, narrow, and changes frequently. Any expansion should remain an application-by-application decision after the scope and object-level diff are understood.

Live Kubernetes evidence

This public dashboard shows the running component inventory plus CPU and memory use by purpose. Pod hashes, namespaces, nodes, images, addresses, and management links are removed before the dashboard can query the data.

Checking the live Kubernetes dashboard.