Bill Gernert
Observability

Grafana

The single pane over the lab, 23 dashboards strong - pulling from metrics, APIs, and JSON endpoints, including a hand-tuned Infinity data source.

What it is and whyHow I use it inArchitectureIn actionLessons learned and gotchasImpact

What it is & why I use it

Grafana is the visualization and dashboarding layer for AutomationLab. It sits on top of the monitoring stack and turns raw signals into something an operator can actually read at a glance. I use it as the front door to the platform's health: cluster state, provisioning outcomes, self-healing activity, and service-level views all land here.

What makes it interesting is that not everything I want to chart comes from a time-series database. Some of it lives behind plain REST/JSON APIs, so I lean on the Infinity data source to pull JSON directly into panels - which is exactly where the sharp edges live.

How I use it in the lab

  • Metrics dashboards. Cluster and host health from the metrics backend, with alerting thresholds visualized alongside the raw series.
  • API-sourced panels. The Infinity data source pulls JSON from internal service APIs so operational data that never went into a TSDB is still charted.
  • Provisioning & healing views. Panels that summarize pipeline runs and auto-heal outcomes, so the lab's automation is observable, not a black box.
  • Access-controlled. Grafana is published behind the same zero-trust edge and SSO as every other app in the lab - no bare port on the internet.

Architecture

Grafana architecture: Two data paths into one dashboard Prometheus metrics and Infinity JSON data meet in Grafana, where one dashboard presents both sources through the published SSO path. ARCHITECTURE FLOW TWO DATA PATHS INTO ONE DASHBOARD Prometheus: Time-series metrics PROMETHEUS Time-series metrics Infinity JSON: Operational facts from APIs INFINITY JSON Operational facts from APIs Grafana: Queries both data sources GRAFANA Queries both data sources Dashboard: Panels share one operational view DASHBOARD Panels share one operational view SSO boundary: Published access is authenticated SSO BOUNDARY Published access is authenticated Reader: Receives the scrubbed public view READER Receives the scrubbed public view Cyan. Automated step or path. Amber. Operator or policy gate. Green. Recorded or healthy outcome. Dashed box. Stored state or record. Rounded box. Actor or process. Arrow. Direction of work or data.
The dashboard combines the metrics backend and an Infinity JSON data path before publication behind SSO. Addresses and hostnames are omitted.

In action

Static configuration evidence coming Use a panel definition or data-source configuration view here. Do not publish a screenshot of a live graph.

Lessons learned & gotchas

Gotcha - Infinity backend parser The Infinity plugin's backend parser is a Go implementation of JSONata, and it is not the full language. It handles a path-plus-filter root selector and plain top-level columns - but it does not support object-construction expressions, computed columns, or nested column definitions. Queries that work in a JSONata playground can silently return nothing here.
  • Prove it with live probes, not docs. The fastest way I found the real boundary was to fire small, deliberate probe queries at a known endpoint and watch what parsed and what came back empty. The behavior, not the documentation, is the spec.
  • Shape the JSON at the source when you can. If the parser cannot compute a column, the cheaper fix is to have the upstream API return the field already shaped, rather than fighting the query language to derive it.
  • Backend vs. browser parsing differ. Choosing the parser mode changes which expressions are legal. Pin the mode you are testing against so you are not debugging two engines at once.

Impact

1
pane over the lab
2
source types unified
SSO
on every dashboard

Grafana is where the lab stops being a set of independent services and starts being a system I can watch. The Infinity work in particular meant I could bring non-time-series operational data - things that only exist as API responses - into the same view as metrics, without standing up a whole extra pipeline to store it first.

Further reading

Infinity data source