Certificate automation with Kubernetes, Jenkins, and Ansible

At a glance

Outcome

The shared public certificate renews ahead of expiry and reaches every registered consumer. Proven credential paths rotate on a 60-day cycle with a seven-day recovery overlap.

Operating scope

The certificate source fans out to Kubernetes, servers, and appliances. Credential workers update Vault and the applications registered for unattended activation; explicit exceptions remain on the same board.

Evidence

The live dashboard measures source and served expiry, rotation state, and coverage. Fresh TLS handshakes and real authentication requests test what consumers use.

Primary lesson

Issuance or provider success is supporting evidence. The job verifies the live consumer before it retires the working predecessor.

One wildcard certificate now feeds every copy

The wildcard covers the first level of application names. Let’s Encrypt signs it, cert-manager renews it in Kubernetes, Cloudflare carries the DNS validation record, and Jenkins sends the resulting certificate and key to the services that use them.

This division matters because cert-manager finishes its work at the source Secret. It has no knowledge of a certificate file on a Linux VM, an appliance certificate object, or a copy in another Kubernetes namespace. The Jenkins job knows those destinations and how each service reloads.

The six-step certificate renewal run

These are the six things one run has to accomplish. Steps one through three create a replacement. Steps four and five put it into use. Step six proves the live result.

1. cert-manager opens a renewal order with Let’s Encrypt. It uses ACME, the automated certificate request protocol, and starts early enough to leave recovery time. No application changes yet.
2. Cloudflare publishes the DNS validation record. cert-manager creates a temporary TXT record with a scoped API credential. That record shows Let’s Encrypt that the requester can change DNS for the requested name, so it signs the certificate.
3. cert-manager writes the pair into a Kubernetes Secret. The Secret holds sensitive workload data: the certificate in tls.crt and private key in tls.key. Jenkins verifies the pair and remaining lifetime before deployment.
4. Jenkins sends the pair to the registered destinations. Kubernetes isolates resources in namespaces, so the monthly job updates an existing TLS Secret in each registered namespace and stages the same pair for services outside the cluster.
5. Each service loads the pair. Traefik, the Kubernetes ingress controller, watches Secret changes. Linux web servers swap complete files and reload or restart. Appliances use a restricted API or helper.
6. Jenkins checks what a new client receives. A fresh TLS handshake returns the served certificate. Jenkins compares its fingerprint, a unique hash, with the source and fails if the service still presents the previous certificate.

Why one source matters

Two independent wildcard Certificates once existed behind the same Traefik controller. Both were valid, but Traefik’s global TLS selection could serve the older lineage for a hostname whose namespace contained the newer copy. The fix was architectural: keep one cert-manager owner, copy its result, and verify every destination against that source.

How each consumer reloads

Atomic replacement swaps complete files, so a service cannot read a half-written pair. A forced-command SSH key can run the certificate helper but cannot open a general shell.

OPNsense stores certificates as Trust objects. Its API updates one object and restarts only the web interface.

Public and internal certificates solve different trust problems

Public browsers already trust Let’s Encrypt. Private services use Active Directory Certificate Services, Microsoft’s domain certificate authority, whose root is distributed to managed computers.

Windows auto-enrollment uses Group Policy and an AD CS template, which defines purpose, names, lifetime, and enrollment permission. Windows requests and renews without moving the private key between machines.

Linux creates its private key locally and submits a CSR, a request containing the public key and names. AD CS returns the signed certificate and chain; Linux installs them and reloads the service.

Blackbox exporter validates the name, trust chain, and live expiry on both paths, catching an expired leaf, untrusted chain, or missed reload. Some internal leaves are still watched rather than automatically renewed.

Secrets needed the same handoff and a different last mile

A client secret is a password an application uses when it authenticates as itself. API tokens and service keys serve a similar purpose for other products. Creating a replacement at the provider does not make an application use it. The replacement still has to reach the application’s configuration, the process has to reload, and a real authentication request has to succeed.

I call the system that creates and revokes a credential its provider. Microsoft Graph manages the Entra client secrets. Grafana, Zabbix, Jenkins, NetBox, Cloudflare, OPNsense, and LiteLLM expose product APIs for their own credentials. Where a product has no secret-safe update API, a small helper on the host accepts the value through standard input and can change one allowlisted setting.

The seven-step secret rotation run

Certificates share one deployment source. Credentials do not because each provider creates them differently and each application stores them differently. A reviewed registry describes those differences so the worker does not infer where a value belongs.

1. Jenkins reads one registry entry. It names the provider, Vault field, consuming application, activation method, and required checks. An unmatched credential stays in inventory but cannot enter an automatic write path.
2. The worker decides whether rotation is due. The policy starts at age 60 days and creates a 90-day replacement, leaving time for the seven-day overlap and a failed attempt.
3. The provider creates one replacement. The worker retrieves its scoped provider authority through a Vault AppRole. The returned value is not printed or sent to metrics; its safe provider identifier is recorded for retirement.
4. The worker updates Vault with compare-and-set. It changes the registered field only if the version it read is still current, preserves neighboring fields, and stops when another process edited the record first.
5. The registered adapter updates the application. It changes one Kubernetes Secret, supplies the next script invocation, sends stdin to a forced host command, or updates one provider resource.
6. The worker tests the new path. It checks provider authentication, the Vault-to-consumer binding, workload health, and the real login or API route.
7. The previous credential remains for seven days. The worker repeats every check after overlap and deletes the recorded predecessor. A failed check leaves both credentials and marks the rotation pending.

The activation adapters are the practical part

I verify the result outside the process that made the change

Jenkins could prove that it wrote a file or Secret, but not what a browser received. The observation path measures the service independently.

Each exporter converts one source into metrics. Blackbox exporter opens the TLS connection; the Entra and Vault exporters collect metadata without receiving existing secret values.

Prometheus stores the metrics and evaluates alerts. Grafana groups the resulting series by application and renewal path.

How it runs now

Most of the time, no rotation is happening. The controllers and exporters watch. Jenkins wakes only when a schedule or an operator starts a run.

What broke