Bill Gernert
Platform & Orchestration

Ansible

Post-provision configuration and hardening for managed hosts, split across 28 roles and applied in a deliberate order that preserves management access.

What it is and whyHow I use it inArchitectureLessons learned and gotchasImpactFurther reading

What it is & why I use it

Ansible is the configuration-management layer of AutomationLab. Terraform brings a VM into existence; Ansible turns that bare VM into a configured, hardened, monitored host. It runs as the configure stage of the provisioning pipeline, after the machine exists and before it is handed over as ready.

I use it because configuration should be declarative and idempotent: the same play run twice leaves the host in the same state, so "converge the host to what the role says" replaces a pile of one-off setup commands nobody can reproduce.

How I use it in the lab

  • Roles, not scripts. Each concern is a role with documented defaults - a firewall role, a time-sync role, a monitoring-agent role - composed into a single host-configuration play.
  • Runs from the pipeline, as an unprivileged user. The configure stage runs Ansible from the CI agent under a dedicated automation user, not root, with the collection path pinned so runs are reproducible regardless of which agent picks up the job.
  • Secrets pulled at runtime. Anything sensitive a role needs comes from the secrets manager during the run - nothing sensitive is stored in inventory or vars in the repo.
  • Deliberate role order. The play applies firewall, then time-sync, then the monitoring agent - and that order is load-bearing, not cosmetic (see the gotcha).

Architecture

Ansible architecture: Ordered host convergence The pipeline connects through a scoped automation identity and applies the baseline roles in order until the host reaches the declared configuration. ARCHITECTURE FLOW ORDERED HOST CONVERGENCE Pipeline stage: Starts configuration after build PIPELINE STAGE Starts configuration after build Automation identity: Scoped access to the new host AUTOMATION IDENTITY Scoped access to the new host Firewall role: Apply the baseline policy FIREWALL ROLE Apply the baseline policy Time sync role: Set the trusted time source TIME SYNC ROLE Set the trusted time source Monitoring role: Install and register the agent MONITORING ROLE Install and register the agent Converged host: A second run changes nothing CONVERGED HOST A second run changes nothing Cyan. Automated step or path. Green. Recorded or healthy outcome. Rounded box. Actor or process. Arrow. Direction of work or data.
The configuration stage uses a scoped automation identity and ordered roles for firewall, time synchronization, and monitoring. Addresses and hostnames are omitted.

Lessons learned & gotchas

Gotcha - firewall ordering can lock you out The firewall role has to ensure the SSH allow rule is in place before the default zone flips to DROP. Get that backwards and the play sets a default-deny policy while it still needs SSH to finish - and it severs its own connection mid-run, leaving a half-configured, unreachable host. The fix is baked into the role order: firewall first, and within it, permit management access before tightening the default. Never let "deny by default" land before "allow the thing you are connected over."
  • Idempotence is a feature you have to test. A role is only trustworthy once a second run reports zero changes. A task that keeps reporting "changed" is usually lying about convergence and will fight you later.
  • Pin the collection path. If roles depend on collections, the path they load from must be fixed and owned by the run user - not left to whatever happens to be installed on a given agent - or the same play behaves differently on different runners.
  • Run as a real service account, not root. Least privilege applies to configuration management too; a role that needs elevation should ask for it explicitly, so "what can this play change?" stays answerable.

Impact

1 play
bare VM to ready host
0
changed tasks on re-run
Order
enforced, not hoped

Ansible is what makes "provisioned" mean "configured and hardened" through a repeatable path. The firewall-ordering lesson is the one I keep closest: automation that can lock itself out of the very host it is configuring is a trap, and the defense is encoding the safe order into the roles to reduce the chance of an ordering mistake.

Further reading

Ansible playbooks