Bill Gernert
Security & Identity

HashiCorp Vault

The secrets backbone of the lab, reached through 19 committed runtime credential-fetch scripts. Registered pipelines and services authenticate with scoped AppRoles and read the paths assigned to their work.

What it is and whyHow I use it inArchitectureLessons learned and gotchasImpactFurther reading

What it is & why I use it

Vault is the custody system for automation secrets in AutomationLab: API tokens, per-host monitoring keys, service credentials, and the personal access tokens that automation uses to open pull requests. Static secret values are kept out of repositories, pipeline definitions, and container images; they are fetched from Vault when needed.

The reason it matters is not "encryption at rest" - it is identity and scope. Each registered consumer proves who it is, gets a short-lived token, and is authorized to read a deliberately narrow slice of the secret tree. A leaked credential is then a bounded problem, not a master key.

How I use it in the lab

  • AppRole for machines. Pipelines and services authenticate with the AppRole method - a role ID plus a secret ID - rather than a person logging in. The credentials are injected as environment or mounted secrets, never baked into an image.
  • KV v2 for secret storage. Secrets live in a versioned key-value store under a consistent <service>/... path convention, so each managed secret has an obvious, predictable home and a version history to roll back to.
  • Least-privilege policies. Each role is bound to a policy that grants read on exactly the paths that role needs. A monitoring agent cannot read PR tokens; the PR helper cannot read monitoring keys.
  • Vault-backed credential helpers. Small scripts fetch a secret at runtime and hand it straight to the tool that needs it - the secret is never written to disk and never printed. The PR-opening helper is the clearest example (below).

Architecture

A concrete, scrubbed example: the portable PR-token AppRole that lets the automation open pull requests from any box without a token file on disk.

  • A dedicated read-only policy grants read on a single KV path - the one holding the PR token - and nothing else.
  • A dedicated AppRole is bound to that one policy. Its role ID and secret ID are themselves stored as a cluster secret, so a pipeline can bootstrap an AppRole login over Vault's HTTP API using only its in-cluster identity.
  • The helper logs in, reads the one path, hands the token to the PR tool in memory, and exits. No secret is written to disk at any step.
HashiCorp Vault architecture: One scoped secret read A workload proves its identity through AppRole, receives a short-lived token limited by policy, reads one secret, uses it in memory, and exits. ARCHITECTURE FLOW ONE SCOPED SECRET READ Workload identity: The running job supplies identity WORKLOAD IDENTITY The running job supplies identity AppRole login: Vault checks the role binding APPROLE LOGIN Vault checks the role binding Scoped policy: Permission covers one required read SCOPED POLICY Permission covers one required read KV read: Fetch the required value KV READ Fetch the required value In-memory use: The job calls the target system IN-MEMORY USE The job calls the target system Token expires: No credential file remains TOKEN EXPIRES No credential file remains Cyan. Automated step or path. Amber. Operator or policy gate. Green. Recorded or healthy outcome. Dashed box. Stored state or record. Rounded box. Actor or process. Arrow. Direction of work or data.
The AppRole login path turns workload identity into one policy-scoped KV read and an in-memory credential. Addresses and secret paths are omitted.

Lessons learned & gotchas

Gotcha - KV v2 hides a "data" segment On a KV v2 mount, the CLI path and the API path are not the same. You read kv get <mount>/foo on the command line, but the HTTP API path is <mount>/data/foo - and the JSON response wraps the secret one level deep under .data.data. Every hand-rolled API client I wrote hit this: a 403 or an empty result that looks like a policy problem is often just the missing data/ segment. Worse, a policy written for <mount>/foo will not authorize the API call to <mount>/data/foo.
  • Scope the policy to one path, then prove it. It is tempting to grant a role a broad wildcard "so it just works." The discipline that pays off is granting exactly one path and then testing that the role can read it and cannot read a neighbour. Least privilege you have not tested is just a hope.
  • Policy paths must match the auth method's real request path. Because KV v2 inserts data/, the metadata and data planes are separate capabilities. Granting read on the data path does not let you list versions, and vice versa - decide which the consumer actually needs.
  • Keep secrets out of argv. A token passed on a command line leaks into process listings and shell history. Helpers fetch and pipe in memory; the token never becomes a visible argument or a file.
  • Separate role per consumer. One shared "automation" role that can read everything defeats the point. A role per job means revocation and rotation are surgical, and an incident stays contained to one blast radius.

Impact

0
secrets in git or images
1 path
the PR role can read
0
tokens written to disk

Vault is what lets the rest of the platform be config-as-code without being secret-as-code. Pipelines run from public definitions, yet still reach real credentials at runtime - each one scoped so tightly that "what could this leak reach?" has a short, boring answer. That is the whole goal of a secrets manager, and wiring it up per-consumer is where the real work lives.

Further reading

Vault AppRole KV v2