Putting a hard ceiling on AI spending

The failure mode is an invoice

An interactive model session ends when the person using it stops. A scheduled job can keep sending requests through a noisy night without anyone watching the logs. The triage workload costs almost nothing when deterministic rules explain the findings, and more when many cases need hosted-model judgment. The unusual night is therefore the expensive one.

Worse, a runaway does not look like an outage. Every job stays green, every result is delivered, and the only symptom is a number on a card statement three weeks later. So the control has to be preventive rather than an alert, and it has to sit somewhere the individual jobs cannot talk their way around.

Before the gateway, each job held its own API key and its own budget check. That works right up until you want to know what the lab spent in total, and then you find that nothing can answer, because every job could only ever see its own ledger.

One endpoint in front of every model

The gateway is LiteLLM, an open-source proxy that speaks one API and forwards to many providers. It runs in the cluster as an internal-only service on a pinned image (v1.74.3-stable, referenced by digest rather than tag so the running version cannot drift), backed by its own PostgreSQL database, which is what makes per-key budgets and spend metering possible at all.

It presents four model names. One, qwen2.5-7b, routes to Ollama running on a machine in the lab, so it costs nothing per token and takes the cheap high-volume work. The other three, claude-opus, claude-sonnet and claude-haiku, route to the Anthropic API and take the judgment-heavy work.

The useful consequence is that which model is now a config value rather than an architectural decision. Every consumer asks for a name; the mapping from that name to a local model or a hosted one lives in one file. Swapping what a job runs against, or moving work from a paid model to the free local one, is a one-line change for every consumer at once instead of a code change in each.

It fails loud. If the gateway is down, the jobs behind it fail and say so. There is deliberately no fallback to calling the provider directly, because a silent fallback would route around every limit described below at the exact moment something is wrong.

Three limits, most restrictive wins

There are three separate places a request can be refused, and a request has to pass all three. They are not summed, they are independent ceilings, and any one of them can say no.

The job's own credit gate. Each consumer keeps the fail-closed check it had before the gateway existed: read the monthly allowance, read month-to-date spend from its own ledger, refuse at or over the limit. Deleting these once the gateway arrived was the obvious tidy-up and it was rejected on purpose. This is the only layer still standing if the gateway is bypassed, misconfigured, or has its budget set wrong, and it fails closed on a budget it cannot read rather than assuming zero.

The per-job allowance at the gateway. Each consumer holds its own virtual key with its own monthly cap, metered centrally. This is the view the individual jobs structurally cannot have: per-tenant caps and per-tenant spend, measured in one place, by something other than the job doing the spending.

The global ceiling. One monthly limit across every consumer together. This is the number that no per-job gate could ever enforce, because none of them can see the others. It is the answer to "what is the worst this lab can cost me this month" and it is the reason the layering exists.

The cost of doing it this way is real and worth stating: spend is now computed in more than one place, and the figure a job's own ledger reports is not the same figure the gateway reports for it. They measure different scopes and must never be added together. A request can also be refused by a layer other than the one it logs to, which is confusing the first time it happens.

What the caps are set to

The global ceiling is 50 dollars a month. The live stack does not spend the whole ceiling on per-job allowances today. Headroom under the ceiling is deliberate. If the live allowances added up to exactly the ceiling, the ceiling would never be the thing that fired and would be decoration.

The job-search scorer's allowance started at 10 dollars and was raised to 20 after it hit the gate in normal use. That is the system working rather than a problem: the gate stopped the spend, the raise was a deliberate decision with a reason, and it happened once rather than drifting.

One standing rule keeps the layers honest. A job's own credit gate and its gateway allowance are always set to the same number and changed in the same sitting. Letting them diverge recreates the confusion above, where a job is denied by a limit it cannot see and reports a budget that is not the one that stopped it.

The 50 dollar figure is a reviewed operating value, not a law of nature. It was set as a starting point to be revisited, and the config says so where it is defined.

The metrics finding

The plan was ordinary: have the gateway expose Prometheus metrics, scrape them, build a dashboard showing all three layers side by side so the precedence is visible. A dashboard and alerting rules were written against those metric names.

Then the gateway was checked live after real traffic had gone through it, and its metrics endpoint returned nothing at all. Not stale numbers, not zeroes: zero lines. LiteLLM's per-model and per-key Prometheus series are gated behind its enterprise build, and this is the open-source one, so it emits none of them and never will.

The dashboard and the alert rules were therefore built entirely against series that do not exist. They did not error. They rendered "No data", and the alerts sat there never firing, which is the worst possible shape for a monitoring failure because it is indistinguishable from everything being fine. They are marked inert rather than left looking functional, and the alert rules were not applied against names never seen live.

The fix is to stop asking for metrics that are not there and read the source that is. The gateway does expose a spend endpoint, and the replacement exporter polls that instead of scraping. The scrape target that was pointed at the empty endpoint has been retired, because a monitoring target that always returns nothing is worse than no target: it occupies the slot where a real one would go.

The reusable lesson is about what "verified" means. The metric names were correct in the documentation and the endpoint was reachable. What had not been checked was whether anything came out of it, and nothing in a green build would ever have said otherwise.

What it is allowed to talk to

The gateway holds the upstream API key for the whole lab, which makes it worth being strict about where it can send traffic. Network policy alone cannot express "only this vendor", because it works in addresses and a large API is a moving set of them behind a name.

The approach taken is a small proxy in front of it, running Squid, with an allowlist of exactly one hostname. Network policy then blocks the gateway from reaching the public internet directly at all and permits it to reach only that proxy, so the allowlist becomes the single place the destination is decided, and it is committed configuration rather than something baked into an image.

This one is built and committed, and applying it to the running gateway is an operator step that has not been taken. Until it is, the gateway's egress is the broader rule it started with. That is a known accepted position rather than an oversight, and stating which of the two is live seemed more useful than describing the design as though it were.

How it runs now

What is running and what remains limited

Verified against the running gateway: the global ceiling is in effect and both budget-limiter callbacks are active, all four model names resolve, an end-to-end request through the proxy to the local model succeeded, and the per-job keys exist with their allowances attached. Every scheduled consumer that calls a paid model now goes through the gateway rather than holding its own provider key.

Not proven, and not claimed: no ceiling has actually been hit in production, so the enforcement has been confirmed as configured and active rather than by watching it refuse a real request. The replacement spend monitoring is in progress, so there is currently no dashboard view of the three layers together. The egress restriction above is committed and not applied.

The spending controls are active. The remaining work is the full three-layer operator view and the narrower outbound path. Those limits stay on the page until the running configuration changes.

What I would do differently

I would capture a live response from every planned monitoring endpoint before building panels or alerts. The first dashboard used documented metric names and a reachable endpoint, but this image emitted none of the expected series.

I would also finish the restricted egress path before moving the last consumer. Centralizing a provider key makes the gateway easier to govern, but it also makes the gateway's outbound reach more consequential.