A change is checked for exposed credentials and vulnerable dependencies before it can merge. The resulting container is identified by an immutable digest, signed, and verified from a separate machine. That proves the image being inspected is the image that was built. Runtime rejection of unsigned images is designed but not enabled, so the page keeps that boundary visible.
This page follows the controls from a proposed code change to the container that actually runs. It also keeps the two checks that gave false confidence in the story, because they explain why a green job is not evidence until it measures the right artifact.
Built from the repository workflows, image-signing scripts, verification records, deployment manifests, and the accepted-risk register.
You did not write most of the code you run. A container image is your handful of files sitting on top of a base image, a language runtime, and dozens of libraries, all pulled from the internet. Supply chain security is the work of knowing what is in that stack, checking it for known problems, and proving that the thing you tested is the exact thing that runs. If you skip that, a bug in a library you have never heard of is now your bug.
The digest is the container's content fingerprint. Unlike a label such as latest, it
cannot quietly move to different bytes. The pipeline and deployment therefore use the same digest as
the common identity for scanning, signing, verification, and startup.
Two scans stand between a change and a merge, and they run on different clocks, which is worth being precise about. The first looks for secrets, an API key or password accidentally committed, and blocks the merge if it finds one. That one really does run on every push and every pull request, as a pre-commit check and again in the pipeline, so a developer who skips the local hook still gets caught. The second reads the code, the infrastructure definitions, and the built container image, and reports known vulnerabilities in any library or base image they pull in. It fires on pull requests that touch a container definition, and on a weekly sweep besides. So a change that can pull in a new library is checked while it is small and fresh in someone's head, and everything else is caught on the sweep.
The image that ships gets signed. The signature is a small cryptographic stamp that says "this specific image, byte for byte, is the one that passed." Today that signing is a deliberate step an operator runs against a specific build, not something that fires automatically on every green one. The signing key never sits in the CI system: it lives in the secrets server, and the signing step reads it from there when it runs.
The part that makes any of this mean something is checking it from somewhere else. On a cold machine, trusting nothing but the published public key, both the signature and the attached build record verify, and the build record points at the same image by digest. That is the whole point of signing. Someone who was not there, and who trusts none of your infrastructure, can still confirm that the image in front of them is the one you built. That check has been run and it passes.
There is one more link in that chain, and it is the one that closed most recently. The build system's worker containers are now started from that exact image by digest rather than by a moving label, and the machine that starts them pulled precisely those bytes. So the image that was signed and cold-verified is provably the image that runs, which is the claim signing exists to support. Worth being precise about what that is and is not. Pinning by digest closes the gap where a label quietly moves to different bytes. It does not check the signature at start time, and nothing here does. That check is the next section, and it is still off.
The intended end state is that the cluster refuses to start an image that does not carry a valid signature from the lab's own key, and that the check fails closed: if the verification service cannot make a decision, the answer is no, not "let it through and hope." The policy is written and committed, set to audit mode first so it records what it would have blocked before it blocks anything, because the alternative is locking yourself out of your own cluster on the first bad match.
None of that is running. This is the part of the page most worth stating flatly, because "we sign our images" is exactly the kind of claim that implies more than it delivers. The admission controller that would do the checking is not installed, so nothing is being blocked and nothing is being audited either. And the policy could not pass yet even if it were applied: only one image in the lab has a signing path so far, so an audit run today would correctly report the others as unsigned. The unsigned images are the real prerequisite, not the soak. What is proven is the chain for that one image, end to end and cold-verified. Enforcement is the next piece of work.
| When | What runs | What the result means |
|---|---|---|
| Before a commit leaves a workstation | The local secret scanner checks the proposed files. | A detected credential stops the commit before it reaches the server. |
| On a pull request | CI repeats the secret scan and checks changed container definitions and dependencies. | The review cannot rely on a developer having run the local hook. |
| On the scheduled sweep | The workflow rebuilds against current upstream packages and scans the fresh result. | Newly disclosed upstream risk can appear even when this repository has not changed. |
| When an image is promoted | The selected digest is signed and receives a signed build record. | The signature and provenance describe one immutable image, not a moving label. |
| Before release evidence is accepted | A separate machine verifies the signature, build record, and digest. | The proof does not depend on trusting the build worker that produced it. |
| At container startup | The deployment selects the verified digest. | The correct bytes run, but the cluster does not yet reject an unsigned digest on its own. |
Provenance is the paper trail: which pipeline built this image, from which commit, at what time. That record is attached to the image itself as a signed attestation rather than left in a build log somewhere, so it travels with the image, and it is the half of the chain that already cold-verifies. It is what lets you trace a running container back to the line of code and the pull request that produced it. That is the difference between "we think this is fine" and "here is exactly what it is and here is how it got here."
The registry, where built images are stored, was scanned once a week. That felt thorough and it was quietly wrong, in a way that took a while to see. The scan read the image already sitting on the shelf. But the risk that moves without anyone touching the repo lives in the base image underneath: a vulnerability gets published against a package the base pulls in, nobody in the lab changes a line, so nothing triggers a scan, and the stored image scans clean next week because it is byte for byte the same image it was. A build run that same day would pull a newer base and see something different.
The fix was to build fresh on a schedule and scan that, instead of only re-reading what was already stored, plus a scan on any pull request that changes a container definition. Worth being honest about the shape of that fix: it is still a weekly clock. What changed is which artifact gets scanned, not how often. The lesson was not "scan more often." It was that a scan can be perfectly on schedule and still be measuring the wrong object.
Some vulnerabilities have no fix available yet, and some do not apply to how you actually use the library. The tempting move is to add them to an ignore list and never look again. The problem is that an ignore list nobody reads becomes a place where real risk goes to hide. So an accepted finding is treated as a documented decision with an owner and a reason, reviewed rather than buried. The goal is not zero findings. The goal is that every finding was either fixed or consciously accepted by a named person who can say why.
Scan so problems are caught small. Sign what ships so the tested thing and the running thing are provably the same, and verify that signature from a machine that trusts none of your infrastructure, because a signature nobody checks from the outside is just decoration. Keep the signing key in the secrets server, not in CI. Refusing unsigned images at the door is written, committed, and not yet turned on, and it is worth saying so in those words rather than letting the rest of the page imply otherwise. And treat both the scan schedule and the accept list as places where a check can quietly stop checking, because those are exactly where the last two real gaps were.
I would have defined the evidence object before I wrote the first scan. The important question is not whether a scanner ran. It is whether every later control can point to the same immutable digest. I also would have separated “the policy exists” from “the policy is enforced” in the first status model. That distinction now stays visible until runtime verification is actually enabled and tested.