Monitoring walkthrough

How the public NOC moves from one state to the evidence behind it

Each operational domain is reduced to healthy, degraded, critical, or no signal. A tile opens the domain dashboard that produced that state. The panels are live Grafana views from the running lab, with saved queries restricted to reviewed public recording rules and database views. An edge filter removes private response metadata before it reaches the browser.

The hub is generated from registered domains and then compared with the Grafana definition that is actually serving. A missing series cannot render green, and a stopped reconciliation job has its own no-data alarm. The page therefore distinguishes a service failure, a missing observer, and dashboard drift.

Built from the public NOC generator, recording rules, scrubbed Grafana dashboards, public-route checks, nightly reconciler, no-data policy tests, and the live dashboard endpoints below.

The hub is a starting point, not a wall of tiny graphs

The operator dashboards are detailed enough to troubleshoot. They include targets, labels, alert text, management links, and the queries behind each panel. Putting all of that into one overview would create a dense screen and would expose details the public site should not carry.

The NOC hub answers a smaller first question: where should I look? Each operational view has its own tile. The tile is reduced to one of four states before it leaves the monitoring system. Click any tile to drill into that domain's live dashboard. The domain view gives expiry, capacity, workload, delivery, or recovery evidence the space it needs.

Nothing here is a screenshot. A reader can hover a series, change the time range allowed by the dashboard, zoom a graph, and follow the drill-down links. When the public monitoring route is unavailable, the portfolio page remains online and says that the live board could not be reached.

Live status board

The board updates on its own. The badge reports whether the public monitoring route answered on the last check. If the route fails, the written page remains available and the dashboard does not present a cached state as current.

Checking live state.

Checking the live dashboard.

← Back to NOC overview

Live public Grafana hub

Open full screen

How to read the first screen

Is anything urgent?

Start with red and amber

The tile state is the worst active condition in that operational view. Open it for the panel and time series behind the severity.

Why is a tile grey?

No signal is not healthy

The registered public series is missing or stale. The domain cannot claim a healthy state until measurement returns.

Is the whole service down?

Open the domain dashboard

A domain can be degraded because one dependency, certificate, backup, or scrape is wrong while the user-facing path still answers.

Where are the hostnames?

The public results omit them

Recording rules and reviewed database views reduce operational identity before the saved public queries select it.

Four states carry different operating meaning

State	Required evidence	What a reader should do
Healthy	Current public measurements exist and no warning or critical condition is active.	No action. Open the tile only for detail or trend.
Degraded	Current data shows a warning threshold, reduced redundancy, approaching expiry, or partial failure.	Open the domain and read the named warning.
Critical	Current data shows an outage, failed recovery, urgent expiry, blocked delivery, or another high-severity condition.	Open the domain immediately and follow the operational evidence.
No signal	The public series is missing, stale, or its exporter cannot be confirmed.	Treat the observer as failed. Do not infer service health.

The no-data rule exists because Prometheus queries over an empty result can look like zero, and zero often maps to green. The generated dashboards and tests require an explicit no-signal state before color thresholds are applied.

The supporting facts answer three cross-domain questions

Beneath the domain tiles, the hub carries a small facts row for active alerts, public web-console reachability, and backup freshness. These are cross-domain signals. A single unreachable public console may belong to Identity or Core Services; one stale backup may affect a system whose application remains healthy.

Certificate horizon and secrets-service state live together in the certificate and secret lifecycle view. That domain can show a healthy portal beside an approaching credential expiry, or a current source wildcard beside a destination still serving an older copy.

The hub keeps these facts small because the detailed panels already exist. Their purpose is to expose an exception that may not raise the whole domain to critical yet.

Saved queries and the edge enforce the public boundary

From operational data to a public dashboard

Exporters collect the full operational measurements. Private monitoring retains the labels needed for troubleshooting.
Recording rules and database views construct the approved results. They select, aggregate, and replace identifying labels with the public vocabulary.
Grafana's shared dashboards use fixed saved queries. They select approved public series or database views on the existing monitoring services.
The Cloudflare boundary restricts requests and responses. The Worker checks routes and methods, removes private bootstrap metadata, and validates complete dashboard responses before releasing them.
The browser opens the curated views. Readers follow drill-down links without receiving query editing or administration controls.

What you are not seeing

Before Grafana can query anything on the public host, the shared dashboard must have its fixed queries saved and its public share enabled. Those queries are checked against the approved recording rules and database views. Private response fields are then checked at the edge. Remaining application paths match the catch-all Access application, where the only policy is block.

Prometheus and Grafana also serve private monitoring. The boundary depends on the public query contract and edge controls, not on a separate public Prometheus instance. Read the container and request path explanation and the security testing scope for the enforced checks and remaining limitations.

A tile opens the domain where the evidence has room

The Infrastructure view shows node state, guest capacity, and resource trends under stable aliases. The AI usage view compares remaining subscription capacity with the time-based ideal line. Other tiles use the same pattern for delivery, Kubernetes, certificates, backups, identity, mail, and core services.

Infrastructure drill-down

Checking the live Infrastructure dashboard.

Live public Infrastructure view

Open full screen

AI subscription-capacity drill-down

Checking the live subscription usage dashboard.

Live public provider-neutral usage view

Open full screen

The site build generates the hub, and a scheduled job checks the live copy

Each operational domain declares its public dashboard role and drill-down destination in the repository. The generator builds the hub from that registry. CI fails when a domain lacks the required tag, has no single primary destination, contains an unsafe link, or produces a panel that violates the public query rules.

Generation protects the repository version. A nightly Jenkins job then reads the dashboard Grafana is actually serving and compares it with the generated definition. The comparison covers panel identity, query, threshold, color steps, links, variables, annotations, and dashboard settings.

The reconciler pushes drift state to monitoring and sends its own heartbeat. A live dashboard that differs from Git produces a drift alert. A reconciler that stops running produces a no-data alert.

Change	Check	Failure result
Domain added	Registry must identify one public overview and safe route.	Site generation fails until the domain is complete.
Dashboard source edited	Tests inspect query labels, thresholds, no-data behavior, links, and panel structure.	The pull request cannot pass public-dashboard CI.
Grafana edited by hand	Nightly reconciliation compares the live JSON with generated intent.	Drift alert names the panel and field.
Exporter or series disappears	Coverage and no-signal rules evaluate current data.	The tile becomes grey or raises a coverage failure.
Reconciler stops	Zabbix no-data trigger watches its heartbeat.	The watcher failure is reported separately from dashboard agreement.

What broke

A dead exporter made many services look down

A merge left duplicate YAML keys in the blackbox exporter configuration. The exporter rejected the file and stopped. Prometheus lost the shared probe target, so many application probes failed together even though the applications were still serving.

The configuration was corrected, the exporter rolled out, and a direct probe plus the Prometheus target returned healthy. CI now uses strict duplicate-key parsing because the previous YAML library silently accepted the last duplicate.

The dashboard needed a separate observer-failure state

The incident showed that “all probes failed” and “the probe service failed” need different presentation. Coverage signals and no-data handling now prevent one missing observer from reading as a simultaneous application outage or as a clean green state.

A hand-edited board can drift while staying online

The nightly reconciler exists because HTTP success says only that Grafana served a dashboard. It does not say that the panel queries and links still match the reviewed definition.

How it runs now

When	What runs	What a reader sees
Continuously	Exporters, Prometheus, recording rules, and Grafana refresh domain state and detail panels.	Interactive live views with public service classes.
When data is missing	Coverage and no-data rules override ordinary threshold coloring.	No signal or a named observer failure, never an empty green.
On repository change	Generator and public-dashboard tests rebuild and validate the hub.	New domains and panels appear only after their public contract passes.
Nightly	The reconciler compares generated intent with live Grafana.	Unreviewed live drift becomes an alert.
When the public route fails	The portfolio’s embed check reports the unavailable live source.	The written page remains available without presenting stale state as current.

If I were recreating this from scratch

Define operational domains before building panels. Each domain needs one primary state and one drill-down destination.
Create the public metric contract. List the measurements and labels saved public queries may return. Check the recording rules, database views, and edge response filter together.
Model no signal explicitly. Test an empty query, a failed exporter, and a stale series before adding colors.
Generate the hub from the domain registry. Do not hand-copy tiles and links.
Keep detail in domain dashboards. The hub should answer where to look, then open the evidence.
Validate the served dashboard. Compare live Grafana with the generated definition on a schedule and monitor the reconciler.
Test the observer outage. Stop a shared exporter and confirm that the hub identifies coverage loss without declaring every service dead.