Models can classify alerts, explain a failure, and prepare a repair plan. They do not receive unrestricted credentials or a direct execution path. Paid requests pass three spending limits, infrastructure changes wait for an operator, protected identity systems stay outside the tool surface, and monitoring checks the result after an approved action.
I built these boundaries while adding AI to the morning triage and self-healing workflows. The useful output is still an explanation or a plan. Deterministic code owns collection and validation, LiteLLM owns the provider route and budget controls, Jenkins owns the recorded workflow, and a narrow helper owns the exact action.
Built from the AI budget decisions, LiteLLM gateway configuration, AIOps collectors, self-healing pipeline, read-only tool server, and approval-gate tests in the repository.
The controls arrived with the first useful model call
The first useful case was morning triage. Code collected overnight alerts, build failures, and backup results, then a model summarized the cases that were hard to explain with fixed rules. That raised two separate risks immediately. A scheduled job could spend money while nobody watched it, and a plausible model answer could be mistaken for permission to change a machine.
I kept those risks separate. Budget controls decide whether a hosted request may leave the lab. Tool controls decide which information a model can read. The workflow decides whether a proposal reaches an operator, and the execution helper allows one named action. A model score or confident sentence never grants access.
The model sits inside a workflow, not in front of the infrastructure
Jenkins starts the scheduled or attended job and keeps the log. Collectors read monitoring, build, and backup state through scoped credentials. Python handles known classifications first. Routine language work stays on the local model; difficult cases can reach a hosted model through LiteLLM. An approved remediation then passes to an allowlisted helper, and monitoring checks what happened from outside that helper.
The model is one stage in the middle. It does not collect with unrestricted access, approve its own plan, choose a new target, or decide that its action worked.
Collectors
Read a defined set of monitoring, CI, backup, and inventory facts. Missing or stale input becomes a finding instead of an empty success.
Python
Groups known symptoms, checks schemas, removes duplicate findings, and decides when a case genuinely needs language-model judgment.
LiteLLM
Provides one internal API for local and hosted models, meters virtual keys, and enforces the gateway-wide ceiling.
Jenkins
Records the run, holds the approval gate, passes only the approved target and action, and reports a failed or timed-out gate.
Restricted helper
Accepts one allowlisted operation. It is not a general shell, and identity-core systems remain excluded from automatic repair.
Monitoring
Observes service health after the action. A successful helper exit is supporting evidence, not the final result.
The budget gate that fails closed
In place today. Every paid model call goes through a budget gate, and the important part is how it
behaves when something is wrong. It denies not only when the budget is exceeded but also when the
budget is missing or unreadable. A broken secret or an unreachable counter closes the tap rather than
opening it. The default answer is no. That is the whole idea of fail-closed: the failure of the
safety check must not be the thing that lets spending through.
One gateway, one hard cap, one pinned image
In place today. All the AI consumers reach hosted models through a single gateway rather than each
holding its own key and calling out on its own. That gateway carries a hard spending cap for the
whole lab, so no single runaway job can spend past a ceiling the others do not know about. The
gateway also runs from a pinned image digest, not a moving tag, so the exact build in production is
the one that was reviewed, and it does not silently change under you when an upstream tag is
repointed.
Cheapest model first, escalate only the hard cases
In place today. Routing is two-tier. A free local model reads everything first and handles what it
can. A paid hosted model is only ever reached for the cases the local one cannot handle with
confidence, and even then behind the budget gate and a per-run cap. The paid model never sees the
easy work. This keeps the common path free and reserves spending for the genuinely hard decisions,
which is also where a second, stronger opinion is actually worth paying for.
Machines automation is never allowed to touch
In place today. Some machines are too important to hand to unattended repair. The domain controllers
and the certificate authority are permanently excluded from auto-remediation. Not "excluded unless
overridden," permanently, in the floor of the system. If one of them is unhealthy, the automation
classifies it and escalates to an operator; it does not act. The reasoning is blunt: the blast radius of
an automated mistake on the identity core is the whole estate, so that core is off the table for
automation by design.
An operator approves before anything changes
In place today. The self-healing pipeline detects and plans without touching anything, then emails
the full plan and waits for an operator to approve it within a time limit. An unapproved plan touches
zero machines. So the model and the automation are free to be wrong in the plan, because the plan is
a proposal, not an action, until a person says yes. The approval is the line between a system that
suggests and a system that acts.
Exclusions written in code for the tool surface
Incoming with the integration work. As models get a broader tool surface into the lab, the list of
what a model may and may not reach moves into code rather than living in a prompt. A prompt asking a
model not to do something is a request; an exclusion enforced in the tool layer is a wall. This is
being built as part of the integration work so the boundary is a property of the system, not of the
model's willingness to obey an instruction.
Spend you can actually see
Partly in place, being extended. The metrics endpoint that would report spend directly is behind a
paid license tier, so observability is built on polling the data that is available rather than on the
endpoint that is not. It works, and it is one of the pieces the integration work is set to improve as
the gateway matures. The honest note here is that the visibility is good enough to enforce the cap
today and is on the list to get better, not that it is finished.
Why current versus incoming is on the page at all
The AI stack is being migrated: a broader integration effort is changing how consumers reach models
and how the tool surface is governed. It would be easy to describe the finished picture as if it were
all live today. That would be a nicer page and a less true one. The controls marked in place today
are running now. The ones marked incoming are designed and being built. Keeping the two apart is the
same honesty the rest of the lab runs on: say what is proven, mark what is planned, and never let the
plan borrow the credibility of the proof.
How it runs now
When
What runs
What happens next
Every collection cycle
Deterministic collectors and classifiers read the current evidence.
Known conditions stay in code; only unresolved cases continue to a model.
Routine language work
The local model classifies or summarizes without a hosted-provider charge.
A usable result returns to the workflow with its source evidence.
Hard case
The consumer, virtual-key, and global budget checks all run before LiteLLM sends a hosted request.
Any denial or gateway failure stops the escalation and reports why.
Proposed infrastructure change
Jenkins presents the target, evidence, and exact action to the operator.
Approval dispatches the bounded helper; timeout or rejection changes nothing.
After an approved action
The workflow reads health from the monitoring side.
A healthy check closes the run. A failed check keeps the incident visible.
What broke while I was building it
The first gateway dashboard watched metrics this build did not expose
The dashboard and alerts used documented LiteLLM Prometheus series. The open-source gateway image returned no series for those names, so the panels showed no data and the alerts could never fire. The endpoint being reachable had been mistaken for evidence that it contained the measurements.
I replaced that path with an exporter that polls the gateway's spend API and emits the measurements the dashboard actually uses. The old scrape target was removed. The check now verifies that the response contains the expected series.
What I would do differently
I would define the tool exclusions and result-verification path before connecting the first model. I built them alongside the first use, but writing the boundary first would have made every later integration a registration exercise.
I would also instrument budget decisions before drawing the dashboard. The first attempt began with expected metric names. Starting from one captured live response would have exposed the open-source metrics gap immediately.
