Security engineering walkthrough

How security decisions are enforced in the lab

A person signs in once, applications receive only the identity claims they need, workloads fetch credentials through narrow Vault roles, and infrastructure jobs stop when approval, inventory, budget, or verification is missing. Public traffic reaches applications through an outbound tunnel, so there is no ordinary inbound application port to discover.

The controls are split across layers because they answer different questions. Edge access decides who may reach an application. The pipeline decides whether the requested change is complete and approved. Vault limits which credential the workload can read. The target system applies the final permission. I learned why that separation matters when an approval check I wrote could fail open on execution paths with no requester identity.

Built from the security page, identity lifecycle policy, publication gates, workload policies, certificate automation, edge access configuration, and the dated approval-gate failure in the repository.

The security model starts with boundaries, not products

The lab is public enough to be useful and private enough to operate. The portfolio and scrubbed dashboards are intentionally readable. Management interfaces, infrastructure APIs, secrets, identities, and operational dashboards sit behind controls that can identify the caller and deny the request.

I use several products because one product cannot answer every question. Cloudflare Access can decide whether a signed-in person belongs to an allowed group, but it cannot decide whether a Terraform destroy plan is safe. Vault can decide whether a Jenkins workload may read a provider token, but it cannot know whether the operator approved the target. CI can scan a public page for internal strings, but it cannot validate the certificate a browser receives after deployment.

The design therefore follows the request. Each control has one place in the path, one failure behavior, and evidence that can be checked after the request finishes.

A request crosses four independent decisions

Security control path from identity at the edge through a pipeline gate and Vault role to the target system, with audit evidence beside the path

Edge identity: Cloudflare Access redirects protected browser requests to Entra ID. Sign-in and group membership are checked before traffic reaches the application.
Pipeline policy: Jenkins or Gitea Actions validates the request, approval, collision result, budget, plan, and expected change before it calls an infrastructure API.
Workload identity: A Kubernetes service account, Vault AppRole, or forced-command SSH identity identifies one automation path. It cannot borrow a broad operator login.
Target authorization: The provider, operating system, Kubernetes API, or application accepts only the actions granted to that machine identity.
Independent observation: Monitoring, live login probes, audit logs, and repository checks record what the request changed and whether the consumer worked afterward.

A request is allowed only when each applicable layer can answer. An unavailable identity provider, unreadable budget, failed collision source, missing approver, or unknown target state stops the path that depends on it.

The public edge has no general inbound application rule

A connector inside the network opens an outbound connection to Cloudflare. Approved traffic returns over that tunnel to the Kubernetes ingress controller, which routes by hostname to the intended service. The firewall does not expose a matching inbound port-forward for those applications.

Protected hostnames require Entra sign-in at the edge. The application may also maintain its own authorization, but unauthenticated internet traffic never reaches its login handler. Remote administrative access uses a private mesh VPN.

There are two public surfaces by design. This portfolio has no login because it exists to be read. The status host serves fixed dashboard documents, their public data route, health checks, and required static assets. Other routes on that host meet a block policy with no login flow.

Public-dashboard boundary. Internal labels are removed in Prometheus recording rules before the public Grafana instance can query them. Hiding a hostname in a panel would be too late because the browser-facing data source could still return it.

People and workloads have separate identities

Human access is group-driven. A role maps to directory groups and application assignments, which gives each access decision a written source. Multi-factor authentication protects remote administration and selected application paths. The user lifecycle pipeline creates, changes, disables, and later removes directory accounts through reviewed requests.

Offboarding disables access first. Before removing memberships or application assignments, the pipeline records a machine-readable snapshot. That snapshot keeps the information needed to explain what was removed and to reverse an incorrect request during the grace period. Deletion is a later action with its own checks.

Machines receive narrow identities for the same reason. A certificate-sync job can update registered certificate destinations but cannot open a general shell. A credential rotator can read and update its own Vault records but cannot enumerate unrelated secret values. Kubernetes service accounts bind jobs to namespace permissions. Product API tokens are limited to the action and resource the job manages.

Certificate identity is part of authorization

The internal certificate authority issues private service identities from templates. A template defines permitted subject names, purposes, enrollment identities, and lifetimes. That configuration is the privilege boundary. A template that lets a low-privileged requester choose an arbitrary high-value subject can turn certificate enrollment into account impersonation.

Public certificates come from Let’s Encrypt. Internal certificates come from Active Directory Certificate Services. Both paths are monitored from the network so the certificate a service actually presents must match the expected name, chain, and lifetime.

Secret values have one custody path

Vault stores provider credentials, application secrets, and the lifecycle metadata used by automatic rotation. Pipelines authenticate at run time and receive only the fields permitted by their role. The values are kept out of the repository, command arguments, metrics, and public logs.

Two scanners protect the repository path. A local check catches an accidental secret before commit when the developer environment supports it. CI scans the committed tree and is the enforcement gate because it does not depend on the editor or workstation that created the change.

Rotation follows overlap. A worker creates a replacement, updates Vault with a version check, activates the consumer, performs a real login or API request, and keeps the predecessor during a seven-day recovery window. The predecessor is retired only after the same path passes again. A new value is never printed as evidence; the provider identifier, timestamps, state, and validation results are enough.

Fail-closed behavior is written into the control path

The most useful question for a gate is what happens when one of its dependencies is broken. These are the current answers.

Gate	Question	Unknown or failed result
Public-site scrub	Does the complete publish tree contain a private string, identifier, or unsafe route?	The build stops before the public mirror is updated.
Provisioning collision	Do DNS, inventory, virtualization, directory, and infrastructure state agree that the requested identity is unused?	An unreachable source stops the request. Unknown is not treated as available.
Approval identity	Are the requester and approver known, allowed, and different where two-person review is required?	A blank or matching identity fails the gate.
Decommission plan	Does the plan contain the deletion the request claims it will perform?	An empty destroy plan is a failure, not a successful run.
AI capacity	Is provider usage current and below the configured reserve?	Missing or stale capacity closes dispatch.
Scanner triage	Did the model return a valid, bounded disposition for a scanner finding?	The original finding remains active.
Credential retirement	Did provider auth, Vault binding, consumer health, and the real application path pass after overlap?	Both credentials remain and the lifecycle state stays pending.

Some checks run at the workstation and help prevent mistakes. The corresponding pipeline gates inspect the committed or deployed result. I keep that distinction because a convenient check can be skipped; a branch requirement or deployment validation decides whether the change moves.

The two-person approval check had a path around it

User deprovisioning requires a requester and a different authorized approver. The original implementation compared those identities only when the requester field contained a value. Browser-started work normally supplied one, so the expected path behaved correctly.

API, timer, and replay paths could start without a requester attached. In that case the comparison short-circuited. The gate did not say that one person was allowed to request and approve. It simply failed to evaluate the rule because one operand was absent.

What failed

The control treated a missing identity as a reason to skip the separation check.

Why normal testing missed it

The interactive route always carried a requester. Tests centered on same-person and different-person comparisons, not on the execution paths that had no person attached.

What changed

The pipeline now rejects a blank requester before it compares requester and approver. Trigger paths that cannot supply the required identity cannot enter deprovisioning.

What the test suite checks

Interactive, API, timer, and replay inputs all reach the same identity precondition. A missing requester, missing approver, unauthorized approver, or matching pair fails before account mutation.

Scanning finds candidates; context decides priority

Repository secrets, container images, and infrastructure code are scanned in CI. Required checks stop delivery for findings that cross policy. A local model can classify scanner output into likely exploitable paths and likely noise, but it cannot create a clean result when the scanner failed or the model returned malformed output.

Secret values are removed before a model sees scanner or log context. The model receives bounded text as data, not as instructions. Its disposition, source finding, and downstream decision are recorded separately.

A central security-event correlation layer is still under development. Wazuh and the existing monitoring paths provide current host and service signals, but this page does not describe the planned correlation work as deployed.

How it runs now

When	What runs	What the result means
On a protected browser request	Cloudflare Access and Entra evaluate sign-in and group policy.	Allowed traffic enters the tunnel; rejected traffic never reaches the application.
On a repository change	CI runs tests, security scans, privacy checks, and policy checks.	The committed result satisfies the gates required for its branch.
On an infrastructure request	The coordinator validates identity, approval, inventory, plan, and target permissions.	Mutation begins only after the full request is known.
When a workload needs a credential	It authenticates to Vault with its machine role.	Vault returns the registered fields allowed for that path.
After a change	Monitoring and application probes inspect the live consumer.	A write is not considered complete until the outside path works.
On scheduled audit	Drift, credential age, certificate expiry, missing heartbeats, and repository state are reevaluated.	A previously good system must continue producing current evidence.

If I were recreating this from scratch

Draw the request path before choosing tools. Mark where identity enters, where approval is required, where a credential is read, where mutation happens, and where independent validation occurs.
Give each actor one identity. Separate people, pipelines, Kubernetes workloads, host helpers, and provider integrations.
Write unknown-state behavior beside every gate. An unreachable inventory, missing requester, unreadable budget, or failed scanner needs an explicit stop decision.
Put secret custody in place before automation. Define Vault paths and policies, then build the worker that consumes them.
Separate guardrails from enforcement. Use local checks for feedback and pipeline gates for decisions.
Test the alternate trigger paths. API, schedule, replay, and retry inputs are where identity and approval assumptions tend to disappear.
Observe the consumer from outside. Test the login, certificate, API request, or service response that a person or dependent system receives.