Bill Gernert
Observability

Prometheus

Pull-based metrics and alerting for the cluster and its hosts, across 16 committed alerting-rule files - and a hard-won respect for the difference between a metric that is zero and a metric that is not there at all.

What it is and whyHow I use it inArchitectureLessons learned and gotchasImpactFurther reading

What it is & why I use it

Prometheus is the time-series metrics engine in AutomationLab. It scrapes instrumented targets and exporters on an interval, stores the samples, and evaluates alerting rules over them. It sits alongside the broader monitoring stack, feeding dashboards and firing alerts on the numeric health of the cluster and the workloads on it.

I use it because the pull model and the query language are a clean fit for infrastructure: targets expose metrics, Prometheus discovers and scrapes them, and rules turn raw series into "this is wrong, tell someone." Exporters let me bring things that do not speak Prometheus natively - like hypervisor stats - into the same model.

How I use it in the lab

  • Scrape targets and exporters. Cluster and host metrics come from instrumented endpoints and purpose-built exporters that translate a system's stats into the Prometheus format.
  • Alerting rules over the series. Rules encode what "unhealthy" means - saturation, error rates, a target that has stopped responding - and fire when the condition holds.
  • Feeds the dashboards. The series back the visual layer, so the same numbers drive both the at-a-glance view and the alerts.
  • Exporters for the non-native. Where a system does not expose Prometheus metrics itself, an exporter bridges it in - keeping everything in one query model instead of a special case.

Architecture

Prometheus architecture: Scrape, store, evaluate, alert Prometheus scrapes exporters, stores time series, evaluates rules, feeds dashboards, and sends alerts when the rule state requires attention. ARCHITECTURE FLOW SCRAPE, STORE, EVALUATE, ALERT Targets and exporters: Expose the measured signals TARGETS AND EXPORTERS Expose the measured signals Scrape: Collect on the configured interval SCRAPE Collect on the configured interval Time-series store: Keep samples by label and time TIME-SERIES STORE Keep samples by label and time Rule evaluation: Compare series with declared logic RULE EVALUATION Compare series with declared logic Dashboards: Present current and historical state DASHBOARDS Present current and historical state Alerts: Send the evaluated condition ALERTS Send the evaluated condition Cyan. Automated step or path. Green. Recorded or healthy outcome. Dashed box. Stored state or record. Rounded box. Actor or process. Arrow. Direction of work or data.
Targets and exporters feed the time-series store, rule evaluation, dashboards, and alerting. Target addresses and hostnames are omitted.

Lessons learned & gotchas

Gotcha - a down target goes absent, not to zero The alert that quietly does not work: you write a rule like "fire when the metric drops to 0," reasoning that a dead service is a zero. But when a scrape target is down, its series does not become zero - it stops existing. The rule has no series to evaluate, so it never fires, and the exact outage you built it for slips through silently. The fix is to alert on absence and scrape health - on the target being unreachable (its up signal going to 0) or on the series being missing - not on the value hitting a number that a vanished series will never report. "No data" and "zero" are different states, and conflating them is how a monitor misses the thing it exists to catch.
  • Alert on scrape health. A target that has stopped responding is often the most important alert of all. Scrape health exposes that condition after the business metric disappears.
  • Watch cardinality. A label with unbounded values (an id, a timestamp) quietly explodes the series count and the memory with it. Keep labels to bounded, meaningful dimensions - high cardinality is a slow-motion outage.
  • Rule for the state you fear, then test it by causing it. The only way I trust a "target down" alert is to actually stop a target in a safe window and watch it fire. A rule you have never seen trigger is a guess.

Impact

Pull
based scrape model
up == 0
catches a dead target
Exporters
bridge the non-native

Prometheus gives the lab a rigorous, queryable view of its own numbers - and the absent-versus-zero lesson is the one I would hand to anyone new to it. The most dangerous monitoring failure is not a wrong number; it is an alert that silently never fires because the thing it watches disappeared instead of changing value.

Further reading

Prometheus docs