Bill Gernert
How it works

The platform

I built AutomationLab around Gitea, Jenkins, and Ansible. Gitea holds the code and review history, Jenkins coordinates the jobs, and Ansible configures the hosts. CI checks connect a proposed change to the version I’m willing to run.

Source and review

Gitea

Pipeline definitions, playbooks, and infrastructure code live together. A pull request holds the change, its discussion, and the check results before it reaches protected main.

Branches · Pull requests · Required checks

Gitea in the lab →
Jobs and approval

Jenkins

Jenkins tests proposed changes and runs infrastructure workflows. It coordinates the tools, stops for approval where needed, and retains logs and reports with the run.

Jenkinsfiles · Shared libraries · Job DSL

Inside Jenkins →
Host configuration

Ansible

Playbooks and roles configure Linux and Windows hosts from reviewed files. The same configuration can be applied again during a rebuild or a scoped repair.

Inventory · Playbooks · Reusable roles

Ansible in the lab →
Before a change reaches main

CI checks are part of the change path.

Gitea Actions and Jenkins report check results on the pull request. Checks cover the proposed code and the files generated from it, so review includes more than reading a diff.

Lint and validation YAML, Ansible, shell, and Kubernetes manifests
Security and policy Secret scanning and infrastructure policy checks
Tests Behavior checks for the affected code
Generated files Freshness, references, and public-site safety

Required checks gate the merge. Results belong to a specific commit. A changed pull-request head needs results for that new revision, followed by review before merge.

From code to a running server

A Linux or Windows build

Jenkins runs the provisioning workflow from reviewed definitions. NetBox allocates the address, Terraform plans the infrastructure change, and I approve the plan before apply.

Proxmox creates the VM. Ansible then configures its operating system, with monitoring registered before the agent starts. A basic build takes about five to ten minutes.

Follow the provisioning job →
  1. 01 / Gitea holds the definition The pipeline and roles have a reviewed revision.
  2. 02 / Jenkins runs the request Validate inputs, prepare the plan, and wait for approval.
  3. 03 / Ansible configures the host Apply the OS-specific roles after the VM exists.
  4. 04 / Keep the result Retain the run log and host record. After apply, a failed run keeps the VM so the next run can continue.
The rest of the platform

The services those workflows use

Vault supplies credentials, the monitoring stack checks the estate, and Argo CD reconciles Kubernetes workloads.

Secrets and recovery

Pipelines fetch credentials from Vault at run time. A measured recovery brought the secrets VM back initialized and unsealed in 97 seconds, with the canary secret intact. That was a single-node VM restore, not an HA failover.

Vault and the recovery test →
Runtime credentials, renewal jobs, and tested backups.
Monitoring and repair

Zabbix and Prometheus collect signals, and Grafana brings them together. Host repairs normally wait for approval; one scoped service restart has a two-attempt cap. The repair engine does not yet verify health after dispatch.

View the live platform →

Live infrastructure evidence

This public dashboard shows node availability, guest state, per-node CPU and memory, storage use, and recovery-test state under stable descriptive aliases. Hypervisor names, numeric guest IDs, addresses, exporter labels, and management URLs are removed before the dashboard can query the data.

Checking the live infrastructure dashboard.

Signals, dashboards as code, and scoped repairs.
Identity and access

Joiner and leaver pipelines manage Active Directory and Entra ID. Offboarding saves an account snapshot before removing access. A blank requester now stops the approval gate after an audit found it could bypass the identity check.

Windows identity →
Account lifecycle jobs with approval and recovery records.
Kubernetes and applications

A paved road for application teams

Most of the platform began as automation for operating the lab. The next layer treats an application team as the customer. A developer should be able to start from a supported service shape, see a proposed change in an isolated environment, and understand the reliability target after the service is running.

The golden path works as a scaffolder

The new-service command renders the same Python service skeleton used by the software catalog. The result includes an importable application, a digest-pinned container base, CI scoped to that service, documentation, dependency updates, catalog metadata, and a script that refers to a secrets path without writing the secret into source control. It can create a branch and open a review request, or render locally for inspection. Repository creation and an automatic production deployment are still outside this command, so this is the service-entry part of the paved road rather than a one-click production claim.

Preview environments are designed and held at the apply gate

The committed Argo CD design watches review requests carrying an opt-in label and gives each one its own namespace, route, quota, and limit range. An AppProject confines the controller to preview namespaces, and a time-to-live cleanup job provides a second teardown path if normal pruning misses one. The control plane remains on manual sync and has not completed the live create, isolate, and teardown acceptance run. It is source-controlled work, not yet a live developer service.

Platform SLOs have definitions but no live error budgets yet

Pyrra definitions cover CI job success, application restore success, source-control availability, and the monitoring API. Pyrra would turn those definitions into Prometheus recording rules and multi-window burn-rate alerts, then show remaining error budget in its own interface. The definitions and metric adapters are committed, but the Pyrra layer has not been applied. Until a controlled failure produces a burn alert and the budget view is checked live, these are staged objectives rather than operating SLOs.

Continuous reconciliation and the developer platform.
Explore the architecture →
Repository counts and related pages

Counted from the repository every time this page is built: 95 pipeline definitions, 67 of them wired into the seed script as code today, and 351 helper scripts the pipelines call. These three numbers are generated, never typed, so the page cannot quietly drift away from the tree it describes.

Also in Platform:

  1. Architecture diagram. The map of the lab's shape and the pages that explain each region. Read more →
  2. Engineering capabilities. A short overview of the skills used across the projects. Read more →
  3. Roadmap. What is already live, what is still being built, and where the next trade-offs sit. Read more →
  4. Vault. The secrets system, its measured recovery, and the limits that are still named plainly. Read more →
  5. Windows identity. Directory, certificate authority, MFA, and the identity failure that is still shown as an active investigation. Read more →
  6. Mail. Self-hosted mail with the delivery path, DNS records, and operational limits explained. Read more →
  7. Recovery test. The restore check that destroys a VM on purpose and times the recovery path. Read more →