Prometheus Metrics Simulator

Six metrics of different types on two running copies of a service, a guided PromQL tool that explains each query from what it did to the samples, and a small Grafana panel.

The four metric types

Counter:
A total that only goes up, until its process restarts.
- The program adds to it and never subtracts.
- Here they are http_requests_total and process_cpu_seconds_total.
Gauge:
A value at one moment, which can go up or down.
- Prometheus sees only the value at each scrape.
- Here they are http_connections_open and process_resident_memory_bytes.
Histogram:
A set of counters, one per range of values.
- Each _bucket line counts the observations up to its le limit, so the buckets are cumulative.
- _sum adds up the observed values.
- _count counts the observations.
- Here it is http_request_duration_seconds.
Summary:
Quantiles computed by the program itself, with a sum and a count.
- Each quantile line behaves as a gauge.
- Here it is http_request_latency_seconds.

Reading each type with PromQL

Type	What to ask	Query
Counter	How fast it grows	`rate(http_requests_total[1m])`
Counter	Several series aggregated	`sum by (route) (rate(http_requests_total[1m]))`
Counter	A share of the total	`sum(rate(x{status="500"}[1m])) / sum(rate(x[1m]))`
Gauge	Its value	`http_connections_open`
Gauge	Smoothed	`avg_over_time(http_connections_open[2m])`
Histogram	The average	`rate(x_sum[1m]) / rate(x_count[1m])`
Histogram	A percentile	`histogram_quantile(0.95, sum by (le) (rate(x_bucket[1m])))`
Summary	A percentile of one copy	`x{quantile="0.95"}`

The PromQL basics define instant and range selectors.
The function list covers rate() and the others.

Mistakes that are easy to make with metrics

Graphing a raw counter: the line only rises, and its height depends on how long the process has run.
- Use rate().
rate() on a gauge: every fall is read as a restart, so the result means nothing.
- Read the gauge directly.
sum before rate: PromQL refuses rate(sum(x)[1m]), and a restart of one copy would be hidden in the sum.
- Use sum(rate(x[1m])).
A window under two samples: rate(x[15s]) on a 15 s scrape returns nothing.
- Use at least four scrape intervals, or $__rate_interval.
Dropping le: histogram_quantile() needs it to tell the buckets apart.
- Write sum by (le).
Averaging summaries: the average of two copies' 95th percentiles is not the 95th percentile of their requests.
- Use a histogram when copies must be aggregated.
Labels with many values: each value makes a new series.
- Keep user IDs and full URLs out of labels.

How a Grafana panel gets its data

A range query evaluates the query once per point, from the start of the time range to its end, one step apart.
- The time series and heatmap panels use it.
- The step comes from the time range and the panel's width.
An instant query evaluates it once, at the end of the time range.
- A table is commonly set to it.
The Stat and Gauge panels show the last value of a range query by default.
The Heatmap panel needs the query's format set to Heatmap, which un-cumulates the buckets.
$__rate_interval is the larger of the step plus one scrape interval, and four scrape intervals.

How this simulator was made

The service is a simulation, the same on every load until we press one of its buttons.
Its samples were loaded into a real Prometheus v3.15.1, and 33 queries were compared at 27 moments each: every result matched the simulator's.
The panel is drawn by the simulator in the manner of Grafana's, and is not Grafana itself.
The query box handles selectors, rate, irate, increase, delta, the *_over_time functions, histogram_quantile, sum, avg, min, max, count and arithmetic.