Jenkins: the automation behind AutomationLab

At a glance

Outcome

Infrastructure work has a recorded path from request to result, with reviewed definitions and explicit approval boundaries.

Operating scope

Provisioning, identity workflows, image builds, repository tests, backups, monitoring maintenance, and website publication.

Evidence

Published test reports, retained build records, delivery dashboards, and a controller recovery exercise measured at 6 minutes 55 seconds.

Primary lesson

Recreating the controller is one part of recovery. Triggers, credentials, history, and every worker type need their own checks.

Why I put the work into Jenkins

A script can create a server or change an account. Operating that script repeatedly raises more questions: which inputs were used, who approved the change, which version ran, and what happened before it failed?

I use Jenkins to make those questions answerable. Pipeline definitions live beside the infrastructure code in Gitea. Jenkins coordinates the stages and records the run, while Terraform, Ansible, PowerShell, and Python do the work. The same job becomes both an operating tool and a record I can return to during an investigation.

What I ask it to run

A server request starts one provisioning workflow across inventory, infrastructure state, the virtual machine, and operating-system configuration. Identity jobs create or disable access. Other pipelines build worker images, validate repository changes, verify restores, scan websites, maintain monitoring, and publish this site.

Some runs begin with a form, others with a source change or a schedule. The common thread is a finite piece of work with inputs, stages, and a result. A failed stage leaves a build record that shows where the run stopped.

The Jenkins features I use, and why

Views make the inventory navigable

I group jobs into Jenkins views such as VM, Platform, Monitoring, and CI. The tabs in the public console use the same view names for the jobs whose membership can be shown. Open the VM view to follow provisioning, or Platform to find the job that recreates the job definitions.

Descriptions explain the operation before it runs

A job description explains its purpose, stages, and operating limits beside its build history. I manage the descriptions in Git with the job definitions. The public console uses reviewed versions that leave out internal addresses, credential references, and operational links.

Jobs and controller settings in Git

Jenkins Configuration as Code defines controller settings. Job DSL creates jobs, descriptions, parameters, and schedules. Jenkinsfiles define execution. Together they let me review an operational change before it runs and recreate job definitions after replacing a controller.

Shared libraries for recurring behavior

I use shared steps for runtime access to Vault, build notifications, and common pipeline setup. Routine jobs share a standard pipeline wrapper. Provisioning and identity flows keep their own stages where their approval and recovery requirements differ. A common fix can reach several jobs without copying a new block into every Jenkinsfile.

Development and production controllers

The controllers have separate storage and share a configuration base with environment overrides. Development follows the development branch; production follows protected main. I can exercise configuration changes before promoting them through a reviewed pull request. Development uses restricted credentials and suppresses production schedules so it does not duplicate backups or maintenance.

Workers that match the task

Kubernetes supplies short-lived workers with a separate workspace for each supported build. Retained Linux and Windows agents handle jobs that need their operating systems and installed tools. The controller coordinates execution; the workers provide the environment the job needs.

Parameters, approvals, and bounded concurrency

Parameters turn an operation into a repeatable request. Pipeline validation checks those values before use, and sensitive workflows stop at an approval gate. Independent pull-request checks can run concurrently, while the dispatcher limits active work and prevents duplicate submissions for the same pending commit. Timeouts keep a stuck run from occupying capacity indefinitely.

Test reports and the artifact that passed

Jenkins publishes JUnit results so a failed check identifies individual tests. Reports are archived with fingerprints, and the result is attached to the exact pull-request commit. For the provisioner worker image, promotion records and verifies an immutable digest, then uses that same digest in the agent configuration. Rollback selects a previously verified image without rebuilding it during an incident.

Follow one request: disable an account

The offboarding workflow shows why the pipeline needs more than a command that disables a user. I need a known requester, a different authorized approver, a reversible first action, and a record of the decision.

The same principle appears in provisioning: planning and validation precede the infrastructure change, and verification follows it. The details belong to each workflow, so I keep their deeper walkthroughs in the related projects.

How I know it is working

A read-only exporter supplies Prometheus with build reliability, queue depth, waiting time, duration, and executor capacity. Grafana makes those measurements visible, including whether the exporter can still reach Jenkins. Delivery metrics use an explicit list of production deployment jobs, so maintenance and test runs do not inflate the deployment count.

Pull-request testing also has a trust boundary. Pipeline definitions come from protected main. The worker that executes proposed code does not receive the status-writing token or Vault role; separate workers publish the result. The check tests the proposed merge and refuses a changed head or conflicting merge.

Recovery exposed what Git could not recreate

On September 2, 2026, I deleted the production controller and its storage during an approved maintenance window. The recorded rebuild took 415 seconds, or 6 minutes 55 seconds. It recovered the job catalog, checked selected schedules, and completed a pipeline on an ephemeral worker.

That time measures the controller exercise. It excludes later repairs, historical build import, and recovery of the full estate. A later import restored 9,814 build records from a retained legacy backup while preserving the freshly seeded job configuration. On September 6, I reconnected the retained Linux and Windows workers.

The failures were useful. External copies of tokens had survived, but Jenkins had lost the server-side records needed to accept them. Source notifications and permanent worker definitions also needed recovery. The procedure now checks those dependencies explicitly. Retained history has a separate backup path because source code cannot recreate old console output or fingerprints.

The expanded recovery procedure has not yet completed another full destructive drill. The controller measurement and later repairs remain separate evidence.

What I would do differently

I would define a working delivery system before designing its recovery test: source changes trigger jobs, each worker type can execute, approvals still hold, and historical evidence remains available. A controller that answers HTTP and can run one Linux job does not prove the Windows path or the next automatic build.

Jenkins publishes this account of Jenkins

This page follows the workflow it describes. I change source in the private repository, review and merge the change, and let Jenkins prepare the public site. A publication check scans the staged content before it reaches the public repository. The pipeline retains the outcome, so the page visitors read is also an example of the delivery system at work.