Pipelines
69 Jenkins jobs. Work as pipelines, not checklists - with the private-to-public publish pipeline as the centerpiece: a one-way trust boundary whose scrub gate checks 6 categories of internal detail before anything crosses from the private source of truth to the public web.
What it is & why I use it
In AutomationLab everything meaningful runs as a pipeline - a defined, repeatable sequence with gates, secrets pulled at run time, and an audit trail - rather than a sequence of manual steps. This page is centered on the one I find most instructive: the publish pipeline that ships this very site from a private repository to the public web. (The provisioning flow that builds VMs is the other big one, sketched below.)
The publish pipeline is interesting because it crosses a trust boundary. Private source and public output are two different worlds, and the pipeline's whole job is to make sure only content that has passed a hard scrub ever moves from the first to the second. That makes it a small, concrete example of a supply-chain control.
How I use it in the lab
- Publish pipeline (the centerpiece). The private git server is the source of truth; a build job takes the site's public subtree, runs a fail-closed scrub gate over it, and only then pushes it to a public repository that a static host serves.
- Fail-closed scrub gate. The same publishing checklist that governs the content runs as a pipeline step: if it finds a forbidden pattern (an internal address, hostname, or secret shape), the build fails and nothing is published. The safe default is "do not ship."
- Secrets fetched, never stored. The push to the public target uses a fine-grained token fetched from the secrets manager at run time and held only in memory - it is never written to disk or baked into a job.
- Provisioning pipeline (the other one). The lab's build-a-VM flow is also a pipeline: allocate an address, apply infrastructure-as-code, register for monitoring, configure, gate on approval, and enroll - rolling forward on partial failure and reversing cleanly on decommission.
Architecture
The publish pipeline as a one-way trust boundary:
- Private source of truth. The self-hosted git server holds the full repo, including things that must never be public.
- Scrub gate as the crossing. The build extracts only the publishable subtree and runs the scrub gate. A single forbidden match aborts the publish - fail closed.
- Public target, then served. On a clean scrub, the build pushes to a public repository, which a static host builds and serves. Content only ever flows private -> public, never back.
- Deliberately plugin-light trigger chain. A change notification reaches the build controller through the git plugin's commit-notify endpoint plus a polling backstop, rather than by installing a source-forge-specific controller plugin - fewer plugins is a smaller attack surface for something that holds a publish credential.
Lessons learned & gotchas
200, so I assumed commits were kicking off builds. They were not.
The commit-notify endpoint was answering, but its response body said
No git jobs using repository - because the job did not have Poll SCM
enabled, so nothing had registered as a polling consumer of that repo. A webhook can only
nudge jobs that are already watching; with none watching, the nudge is delivered and
discarded, and the delivery log is green the whole time. The fix was to enable polling so
the job registers as a consumer - and the lasting lesson is that the commit-notify
response body is the real diagnostic, not the HTTP status of the delivery.
- Delivery success is not effect success. "The webhook returned 200" tells you the endpoint was reached, not that a build ran. Verify the thing you actually wanted - a build started - not the hop before it.
- Fail closed on a publishing gate. A scrub gate that failed open - publishing when the check errored or was skipped - would be worse than none, because it would feel safe. The gate must treat "could not confirm clean" as "do not publish."
- Prefer fewer plugins on anything that holds a credential. Reaching the trigger through a generic endpoint plus polling, instead of a bespoke plugin, keeps the publish controller's attack surface small - a deliberate security tradeoff, not an accident of setup.
- Keep the publish token in memory only. A token that pushes to the public world is exactly the thing you never want on disk or in a job definition - fetch it at run time, use it, discard it.
Impact
The publish pipeline is what lets a private, internal-detail-heavy repository safely feed a public website: every deploy passes a hard scrub, the credential that crosses the boundary never lands on disk, and the trigger path is kept deliberately simple. The webhook gotcha is the reminder that ties it together - a pipeline is only trustworthy when you verify the outcome you wanted, not the green light one step before it.