Technical project walkthrough

How I turned the first hour of triage into one morning report

Last run 2026-09-21 · 45 findings · 3 actionable · 41 needs info · 1 known noise

Before I start work, one read-only job checks alerting, monitoring, failed automation, and backups, then sends a report that separates work to do, questions to investigate, and noise already understood. The report names any source it could not read, keeps the evidence that produced each decision, and cannot make a change to the environment. The result is a short attention list instead of a morning spent opening consoles and reconstructing the same timeline by hand.

AIOps is the use of software and machine-assisted analysis to reduce operational signals into decisions a person can review. In this project it means collecting evidence, applying repeatable rules, asking a language model to explain ambiguous cases, and stopping at a report.

Built from the current collectors, database migrations, deterministic correlation and disposition code, local-first classification path, Jenkins orchestration, production email template, public Grafana view, and the incident that proved a missing run needs its own alarm.

View the code →

At a glance

Outcome: One daily report separates actionable work, incomplete evidence, and known noise before the workday starts.
Operating scope: The live job reads alerting, monitoring, CI, and backup evidence, stores its own history, uses bounded model analysis, and stops at a report.
Evidence: The site build fills in the latest run totals from live facts. The public dashboard shows remaining subscription usage; private per-source heartbeats show whether collection actually ran.
Primary lesson: A quiet source and a collector that stopped are different states. Each collector records a heartbeat before an empty result can be treated as quiet.

The report is the product

I designed the email first because it is the part a person has to trust. The top line says whether collection and triage ran cleanly. The next line gives the measured finding totals for that run. The body then leads with the item that deserves action, keeps incomplete evidence in a separate needs-info section, and leaves suppressed noise visible enough to audit without letting it dominate the morning.

Actionable

The evidence supports a concrete next step. The report explains why it matters and gives a diagnostic starting point. It still does not execute the step.

Needs info

The signal may matter, but a source is stale, context is missing, or the classification is not trustworthy enough to recommend action.

Known noise

History or an operator ruling identifies a recurring non-actionable condition. Critical new evidence can override that suppression and return the item to view.

The three outcomes are not model moods. They are stored classifications with a rationale, confidence, engine, prompt version, source finding, and run identifier. That history makes a later report explain whether the same signature has appeared before and whether a person has already ruled on it.

Provider usage replaces API cost

I now switch between ChatGPT and Claude subscriptions based on which eligible provider has the most usage remaining. Ollama still handles the first classification locally; hard cases can receive a second opinion through the subscription router. There is no metered API fallback.

The router compares weekly usage remaining after setting aside each provider's reserve for interactive work. The five-hour allowance breaks a tie. Failed, stale, incomplete, or reserve-breaching usage observations make a provider ineligible.

How to read it: the gauges show weekly and five-hour usage remaining for each provider. The pace charts compare weekly allowance with the rate needed to last until reset. These are subscription percentages, not API costs, and they do not prove that a triage run completed.

Checking the live subscription-usage dashboard.

The public view contains aggregate provider usage. Finding text, prompts, host identities, and management links remain private.

Run health is tracked separately. Each collector writes a private run heartbeat before it asks its source for findings. A current heartbeat with zero findings means the source was reached and the period was quiet. A missing or stale heartbeat means the evidence was never gathered. That distinction cannot be inferred from an empty chart.

The morning run follows one evidence path

Jenkins, the automation server, starts the job on a daily schedule after the backup window and before the workday. It launches an ephemeral Kubernetes agent, a short-lived container with the Python libraries and runtime identity the chain needs. The agent checks out the reviewed repository, reads narrowly scoped credentials from Vault, reaches the source systems, and writes the normalized result to PostgreSQL.

Operational evidence becomes one read-only morning decision list

Phase 1CollectRead each operational source and open a run heartbeat before the source call begins.

Phase 2NormalizeConvert source-specific events into one finding shape with identity, severity, status, time, and host when known.

Phase 3CorrelateDeduplicate repeated signatures, group same-host findings, and load earlier classifications and operator rulings.

Phase 4ClassifyApply a ruling when one matches, otherwise use the local model and reserve second opinions for hard cases.

Phase 5ExplainBuild actionable, needs-info, and known-noise sections with rationale, source health, and the first diagnostic check.

Phase 6Deliver and watchSend the email, record a digest heartbeat, and expose job, collector, and triage health to monitoring.

Every arrow moves data, not authority. The job can read source systems, write its own evidence store, and send a report. It has no path back from a classification to an operational change.

Each source keeps its own meaning

Alertmanager

The Prometheus alert router supplies alerts that are firing now. Its fingerprint becomes the stable identity. It does not claim to reconstruct alerts that fired and resolved before collection.

Zabbix

The infrastructure monitoring system supplies problem events and whether each one is still active. The collector maps source severity into the common vocabulary and keeps the original tags as evidence.

Jenkins

The automation server exposes failed, unstable, or aborted builds in the reporting window. The collector walks jobs and nested folders. A build is recorded as work awaiting triage, not as a live infrastructure alert.

Backup

The backup platform supplies stale and failed guest results through the same code used by the backup report. Healthy guests do not become findings.

A normalized finding keeps the source-native identifier and a trimmed raw payload. Normalization makes the records comparable without pretending they are identical. A failed build and an active host alert can share a host and a time window, but the original source evidence remains available when the report needs to explain the difference.

Code narrows the work before a model explains it

The pipeline does not send an unfiltered event dump to a model. Deterministic code creates a signature from the source identity, updates an existing finding when the same event is collected again, and groups findings that share a host. It also loads classification history and active operator dispositions before any inference runs.

Use a human ruling first. A matching disposition can mark a recurring signature, title and host, or entire host as actionable, needs-info, or known noise. The most specific active ruling wins. The decision is versioned and can expire or be revoked.
Protect critical evidence. A general known-noise ruling does not hide a new critical finding unless the operator deliberately created that stronger exception. The report calls out the conflict between the old ruling and the new severity.
Use the local model for the common path. Ollama, a language-model runner on my own hardware, receives one finding, its group context, and its history. The response must match a strict JSON shape with a valid classification, rationale, and confidence.
Escalate only the hard cases. Invalid local output, low confidence, or a high-stakes actionable result can receive one provider-neutral second opinion when recent subscription measurements show capacity above the protected reserve.
Fail closed when the explanation path is weak. If a second opinion is unavailable, a valid local result remains. If the local output itself is invalid, the finding becomes needs-info rather than disappearing.

AllowedSources → evidence store → report

Read alerts, problems, builds, and backup health. Write findings, run records, classifications, dispositions, and digest heartbeats to the AIOps database. Send HTML mail.

Not presentReport → operational change

No shell, deploy, restart, acknowledge, delete, ticket-closing, or pull-request action is reachable from a model classification. Suggestions remain text for a person to review.

The model is inside the analysis path, not the control path. Vault policies and the job design give collectors read access to their sources. The only writes are to the dedicated evidence database and the mail system.

What runs where

Component	Where it runs	Responsibility	Failure posture
Jenkins	Kubernetes controller and an ephemeral job agent	Schedules the run, orders collection, triage, and digest stages, keeps logs, and marks partial source failures unstable.	Collector or triage failures do not block the digest. A digest failure fails the job because a missing report is the alarm.
Collectors	Python processes in the job agent	Use read-only APIs to gather alerts, monitoring events, CI results, and backup health, then normalize them.	Each records its own running, ok, or error row. An unreachable source cannot look like a quiet source.
PostgreSQL	A dedicated AIOps database	Stores findings, collector runs, classifications, triage runs, digest runs, operator dispositions, and retained history.	A database failure makes the evidence chain untrustworthy and prevents a normal digest.
Vault	Central secrets manager	Issues the job’s runtime identity and supplies read-only source credentials plus the mail credential.	A missing credential fails the source that needs it. Values are not placed in the repository or the report.
Ollama	A local inference host	Classifies the common path without a metered API credential.	If it is unavailable, triage records its own failure finding and the digest reports that triage did not run.
Headless second-opinion router	The same job agent, using installed subscription-backed clients	Reads fresh usage observations, protects interactive reserves, and selects an available provider only for a hard case.	Stale or exhausted measurements fail closed. There is no automatic metered API fallback.
Mail service	The existing authenticated SMTP path	Sends the rendered report after collection and triage state have been assessed.	A send failure fails the digest stage. The absent morning email is intentionally treated as a signal.
Prometheus and Grafana	The monitoring stack	Track job status, activity, source freshness, shared service health, and public aggregate evidence.	Freshness alerts catch a chain that goes silent even when no Jenkins failure notification was produced.

How it runs now

When	What runs	What a person sees
On the daily schedule	The source collectors run in sequence, triage classifies new evidence, and the digest renders and sends.	One email arrives before the workday with collector health and the prioritized findings.
When one source fails	That collector records an error and the Jenkins build becomes unstable, but the remaining collectors and digest continue.	The report is marked degraded and names the missing source. Existing evidence is not presented as a fully healthy run.
When the local model fails	The triage run records an error and creates a stable finding about the triage engine. The job still proceeds to the digest.	The report says triage did not run. It does not reinterpret an empty classification table as no work.
When report delivery fails	The digest stage fails instead of being softened to unstable.	No report arrives, and the normal Jenkins failure notification becomes the signal to investigate the reporting path.
When a source heartbeat becomes stale	A separate freshness check compares the latest run record for every expected collector with the daily-run threshold.	An attention alert names the stale or absent source. This catches silence even if the job history itself looks inconclusive.
On a manual preview	The job renders the report from existing database state and archives the HTML without collecting, classifying, or sending.	A reviewer can inspect the exact template and data grouping without changing source or evidence state.

What broke while I was building it

2026-07-20The entire collection chain went quiet

Every configured collector stopped writing run records at the same timestamp. Because they failed together, the evidence pointed to one chain-level problem rather than independent source failures. The daily job produced no new collection evidence for nineteen days, and nothing alerted on the silence.

2026-08-09A different audit found the missing reports

The gap was discovered while reading the live database for another task. That exposed the larger design failure: monitoring the Jenkins build was not enough. The system needed to ask whether each collector had written a recent run record, because a healthy quiet run still writes one.

2026-08-09The fix moved the alarm to the evidence boundary

A source-by-source freshness check now reads the latest collector-run timestamp. It distinguishes current, stale, absent, and unreadable states and sends an attention report when the evidence is old. One fresh source cannot hide the others. The original incident record remains honest that the historical stop’s exact Jenkins-side cause was not proven from repository evidence alone.

If I were recreating this

Design the human report before the model prompt. Decide what a person needs to know, what evidence belongs beside it, and how missing evidence will look.
Give every collector a heartbeat. Store run status and finding count before treating an empty result as a quiet system.
Normalize without flattening provenance. Use one finding shape for comparison, but keep source identifiers and trimmed raw evidence for explanation.
Make correlation deterministic first. Deduplicate signatures and group obvious shared-host events in code. Use models for explanation and ambiguous classification, not for operations that can be reproduced exactly.
Let human rulings outrank model history. Store scoped, expiring dispositions and keep critical-severity exceptions visible.
Separate analysis from authority. The scheduled job should have no credential or code path that can turn text into a change.
Monitor the product and its ingredients. Watch report delivery, collector freshness, triage runs, and the services they depend on. A missing email and a stale source are different failures.