Observability

Zabbix

The lab's monitoring brain - hosts register themselves through the API with a unique encrypted key, and a classify-and-heal engine that recognizes 5 distinct failure classes turns common failures into auditable, automatic fixes.

What it is and whyHow I use it inArchitectureLessons learned and gotchasImpactFurther reading

What it is & why I use it

Zabbix is the monitoring backbone of AutomationLab. Every provisioned host is registered into it automatically as the last step of the pipeline, so "monitored" is not a thing someone remembers to do later - it is part of what "provisioned" means. Beyond collecting metrics, it is the trigger source for the lab's self-healing.

I use it because it has a first-class API. That lets registration, templating, and remediation all be driven from code rather than clicked in a UI, which is the only way monitoring keeps up with infrastructure that is itself built from code.

How I use it in the lab

API-driven registration. The pipeline creates the host via the API, places it in the right host group, and attaches the standard agent template - no manual host creation.
Unique encrypted key per host. Each host gets its own pre-shared key, generated at registration and stored in the secrets manager; agent traffic is PSK-encrypted and required in both directions, so no host shares another's key.
Classify-and-heal engine. Triggers feed an engine that classifies a failure and applies an OS-aware remediation - for example, restarting a stuck service - and records every attempt and outcome so heals are auditable, not magic.
Tiered response. A fast Tier-1 path restarts common stuck services within minutes; anything it cannot safely fix escalates to an operator with the context already gathered.

Architecture

Monitoring registers the host, detects a failure, classifies it, applies the permitted repair, and records the result. Addresses, hostnames, and key material are omitted.

Lessons learned & gotchas

Gotcha - the API is a machine door, not the front door The monitoring UI sits behind an identity-aware SSO proxy, which is correct for interactive users. But a script calling the API cannot complete an interactive SSO challenge - so automation has to reach the API endpoint directly, on the internal service path, rather than through the public, SSO-fronted hostname. Point a registration script at the pretty public URL and it gets an auth redirect page instead of JSON, and fails in a way that looks like a broken token. Machine-to-machine and browser sign-in are two different doors on purpose.

A unique key per host is worth the bookkeeping. Sharing one PSK across hosts is easier on day one and a disaster the day one host is compromised. Generating and storing a key per host at registration keeps a key leak scoped to a single machine.
A heal you cannot audit is a liability. Auto-remediation that does not record what it did, and whether it worked, erodes trust fast. Every heal writes an outcome, so the engine is accountable and I can see when it is masking a recurring problem instead of fixing it.
Register at the end, but never optionally. Making monitoring the mandatory final provisioning step is what prevents the classic "that box was never in monitoring" outage.

Impact

Auto

host registration

1 key

per host, encrypted

Minutes

Tier-1 auto-recovery

Zabbix is where the lab stops just watching itself and starts fixing itself. The combination that matters is automatic registration (nothing escapes monitoring), per-host keys (a leak stays contained), and audited self-healing (the automation is accountable). Together they turn monitoring from a dashboard you check into a control loop that keeps the platform healthy on its own.