How I run the lab with AI and automation

The work has to keep moving when I am busy elsewhere

My part

Choose priorities, define the scope, investigate unresolved failures, and decide which operational actions are allowed.

Automation's part

Collect evidence, launch bounded jobs, run tests, enforce identities, and record where a workflow stopped.

AI's part

Read the supplied context, implement a scoped change, review another provider's work, or explain a finding.

Current limit

The fleet is experimental. A green build and a second model's review still leave room for a shared misunderstanding, an incomplete test, or a defect that appears only when the change runs.

The server provisioning story shows what Ansible and the delivery pipelines own once a request is approved. This page follows how I use AI to develop and support those workflows.

A public map of the experimental fleet

The private fleet map groups the controller, builders, and assigned reviewers so I can follow a piece of work through the system. This version keeps that hierarchy and shows both review directions. Each box represents a role. The number of running agents changes with available work and operating limits.

On a narrow screen, scroll the map sideways or open the fleet map at full size. The map shows the workflow; it does not display live status.

The controller is Python code

I set the priorities and operating limits. The controller checks the queue and recorded state, assigns eligible work, and starts the appropriate command-line agent. A builder runs in an isolated Git worktree, which gives it its own branch and files. The launch and publication code records the result and opens the pull request.

That separation is useful when an agent stops halfway through. The controller can distinguish a running job from a completed process with no deliverable, arrange a bounded retry or repair, and park work that has exhausted its attempts. I can then inspect the recorded failure instead of starting from an agent's claim that it finished.

Delivery has its own checks

A passing review is one input to delivery. The separate merge executor also requires successful CI on the current commit and a mergeable branch, subject to the fleet's operating controls. If a repair changes the commit, the old verdict cannot authorize that revision. Repository delivery and an infrastructure apply remain separate operations.

I am still testing whether this arrangement reduces the time I spend getting useful work delivered. More agents also mean more opportunities for conflicting work, stalled jobs, and reviews that repeat the same mistake. I keep the scope and retry budgets bounded, and I need a way to stop the fleet and inspect its state.

Cross-vendor review gives the change a second reading

I ask the other provider to review because the author has already settled on an explanation of the problem. A fresh reviewer can question that explanation, inspect the failure path, and compare the implementation with the original scope. Changing vendors is useful for that second reading, although it cannot guarantee that the models will make different mistakes.

The task gives the reviewer something to test

A task names the objective, allowed files, expected evidence, and stop conditions. For an Ansible change, that might mean checking the intended operating-system behavior and the rerun after a partial failure. The reviewer needs to know what success means before it can judge whether the branch delivers it.

1. Build on an isolated branch. The agent reads the task and repository instructions, makes the scoped change, and runs the relevant checks. Publication preserves the diff and evidence in a pull request.
2. Review with the other provider. The reviewer reads the task and the current diff with fresh context. Its verdict names the commit under review and any defects it found.
3. Repair a named defect. A failed review returns findings to the builder through a bounded repair path. The corrected commit needs new CI and a fresh review.
4. Check delivery state again. The merge process evaluates the current commit, its checks, its review, and whether the branch can merge. Exhausted repairs or unresolved failures need my attention.

That is the kind of review I want: a specific input, the behavior that breaks, and evidence that the correction covers it. Agreement between two models is not enough by itself.

CLI sessions and API requests have different controls

I use subscription-backed Claude and Codex command-line sessions for repository work and selected unattended pipeline analysis. Services that need an API use LiteLLM, the shared model gateway. These paths have different credentials, usage measurements, and failure behavior.

The subscription routing project follows the pipeline selector. The LiteLLM page explains the API budgets and their limits.

Morning triage gives me somewhere to start

I do not want to start the morning by reading every overnight log separately. The AIOps path starts with collectors. Monitoring, CI, backup, and delivery systems produce structured records with source timestamps and status. Code normalizes those records, applies known-noise dispositions, checks freshness, and groups exact relationships that do not require a model.

The model receives the remaining ambiguous bundle. It can summarize likely relationships, identify missing context, and draft an actionable finding. The digest keeps separate categories for actionable work, needs information, and known noise. The source records remain available for each claim.

1. Collectors read each source. They write timestamped records and completion state. A failed collector does not produce an empty healthy input.
2. Code normalizes and correlates known cases. Stable identifiers, time windows, duplicate suppression, and approved dispositions run before inference.
3. The analysis runner receives a bounded bundle. The subscription-backed analysis path receives the collected evidence. Source text is treated as data, and the result still has to pass validation.
4. The model returns a constrained structure. Findings name their evidence, disposition, severity, and missing information.
5. The pipeline validates and stores the result. Malformed output fails. Valid findings enter the digest and tracking path.
6. The delivery monitor expects the report. The dead-man switch alerts when the daily digest or its checker goes silent.

The morning triage project follows collection through the digest in more detail. The incident analyst uses the same shape at a smaller scale. It summarizes the evidence attached to one failure so the first response starts with context. It does not change the alert or perform the repair.

Scanner triage cannot turn a broken scan green

Secret, container, and infrastructure scanners find candidate problems. A local model can classify a result against the deployed context and reduce noise where the policy allows. The model is downstream of the scanner.

If the scanner did not run, the input is incomplete, or the model response is missing or malformed, the original finding keeps its severity. The model cannot invent a clean scan. Secret values are removed before the request, and the model receives the finding text inside a delimiter that marks it as untrusted data.

Routing considers the task, its data, and available capacity

I use a local open-weight model for frequent classification that can stay inside the lab. Approved external models handle repository-scale reasoning and harder analysis when the consumer's route allows it. The data supplied to a request still has to fit that route.

LiteLLM gives API consumers separate virtual keys, per-consumer budgets, and a shared spend ceiling. It records spend and failures, and its virtual keys follow the credential-rotation system. Those dollar budgets are separate from the subscription windows used by the command-line agents.

The fleet and pipeline selector handle missing usage differently

The fleet reads remaining subscription capacity and reset times to rest a depleted provider, balance work, and adjust concurrency within configured bounds. Fresh evidence of low capacity can stop new work for that provider. If the feed is missing or stale, the fleet records quota as unknown and continues using its configured fallback behavior. This quota check is an operating optimization, not a safety boundary.

The shared subscription selector used by unattended pipelines has a stricter reserve and freshness decision. If neither provider is eligible, that pipeline path stops. I keep these behaviors separate because saying that all AI work stops on stale usage would misdescribe the fleet.

Live subscription-capacity dashboard

The public board shows remaining capacity and the time-based ideal line for both subscription providers. Account identity, runtime tokens, worker labels, exporter internals, and private alert annotations are removed before the public data source.

How to read it

A recommendation does not authorize its own action

Prompts are not treated as security boundaries. The boundary is the data supplied, the identity used, the network route, the output schema, and the action the next process is permitted to take.

What broke

How it runs now

If I were recreating this from scratch

1. Choose one bounded decision. Start with classification, summarization, or code review where the input and output can be checked.
2. Build deterministic collection first. The model should not be responsible for discovering whether its evidence exists.
3. Define the identity and data boundary. Decide which Vault role, provider key, network path, and fields the request requires.
4. Validate the output before using it. Use a schema and keep the source evidence beside the model result.
5. Put action in a separate process. A recommendation, pull request, or disposition should cross a policy or operator gate before mutation.
6. Measure provider capacity and freshness. Define the fallback explicitly. The pipeline selector stops on missing eligibility; the fleet currently continues on unknown quota.
7. Add a dead-man signal. A silent collector or digest should become a named monitoring failure.