PromQL

PromQL#

PromQL, the Prometheus Query Language, is the query DSL for Prometheus and the broader Prometheus-compatible ecosystem (VictoriaMetrics, Mimir, Thanos, Cortex). Designed for time-series metrics, with operations for aggregation, rate-of-change, joins between series, and alerting predicates.

The standard for cloud-native metrics in 2026; even non-Prometheus backends (M3, Grafana Cloud Metrics, AWS Managed Prometheus) speak PromQL.

The Data Model#

  • Metric name, http_requests_total.

  • Labels, {method="GET", status="200", path="/api"}.

  • Sample, numeric value at a timestamp.

A time series is uniquely identified by metric_name + labels.

PromQL operates on either instant vectors (one sample per series at a given instant) or range vectors (a sliding window of samples).

Selectors#

http_requests_total                                 // all series
http_requests_total{status="200"}                   // exact match
http_requests_total{status!="200"}                  // negated
http_requests_total{path=~"/api/.*"}                // regex match
http_requests_total{path!~".*health.*"}             // regex non-match
http_requests_total{job="web", env="prod"}          // multiple labels

Range Vectors and rate()#

A range vector is [duration] after a selector:

http_requests_total[5m]      // raw samples in last 5 minutes per series

The hottest function in PromQL is rate(), per-second rate over a window.

rate(http_requests_total[5m])               // requests/s, per series
sum(rate(http_requests_total[5m]))          // total RPS

For counters, rate handles counter resets automatically. For short-term spikes, irate uses just the last two samples. For unrelated cumulative-but-not-counter metrics, increase() gives the increase over the window.

Aggregation#

The verbs that collapse many series down to one or a few. sum, avg, max, min, topk, count – each can be paired with by (...) to keep selected labels or without (...) to drop them. Aggregation is where PromQL queries earn their compactness.

sum(rate(http_requests_total[5m])) by (status)
avg(http_request_duration_seconds) by (job)
max(node_memory_MemAvailable_bytes) by (instance)
topk(5, rate(http_requests_total[5m]))
count(up == 0)                              // number of down targets

Aggregators: sum, min, max, avg, stddev, stdvar, count, count_values, bottomk, topk, quantile, group.

Histograms#

Histograms are aggregated server-side via histogram_quantile:

histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, path))

The le (less-than-or-equal) bucket label is required by the function. Modern Prometheus also supports native histograms with their own functions (histogram_avg, histogram_quantile).

Math and Joins#

PromQL has arithmetic, comparison, and logical operators that work on matching series.

# Cache hit rate
sum(rate(cache_hits_total[5m]))
/
sum(rate(cache_lookups_total[5m]))

# Filter
up == 0
node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1

# Join with on() / ignoring()
sum(rate(http_requests_total[5m])) by (job)
  / on (job) group_left
sum(rate(http_requests_capacity[5m])) by (job)

Alerts#

PromQL is the language of Prometheus alerting rules.

- alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
    /
    sum(rate(http_requests_total[5m])) by (service)
    > 0.05
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "{{ $labels.service }} 5xx > 5%"
    description: "Above SLO for {{ $value | humanizePercentage }}"

The for clause requires the predicate to hold for that duration before firing, which mitigates flapping.

Recording Rules#

Pre-aggregated time series make dashboards and alerts cheaper.

- record: job:http_request_rate:5m
  expr: sum(rate(http_requests_total[5m])) by (job)

Then dashboards and alerts query the recorded series instead of the raw data.

Common Patterns#

The handful of PromQL expressions that show up over and over in dashboards, alerts, and runbooks. Counter rates, error ratios, histogram quantiles, up/down detection, and absence-tests cover most of an SRE’s working vocabulary.

Pattern

Expression

Rate of a counter

rate(metric[5m])

Sum across labels

sum(rate(metric[5m]))

Per-job rate

sum(rate(metric[5m])) by (job)

Average

avg(metric)

99th percentile latency

histogram_quantile(0.99, sum by (le)(rate(metric_bucket[5m])))

Error ratio

rate(errors[5m]) / rate(total[5m])

Up / down detection

up == 0

Time since last scrape

time() - timestamp(metric)

Detect missing series

absent(metric{job="x"})

Tooling#

The tools an operator reaches for when authoring or debugging PromQL. Prometheus and Grafana are the everyday surfaces; promtool lints rules and runs unit tests; promlens is the visual builder. Backend choice (Cortex, Mimir, Thanos, VictoriaMetrics) doesn’t change the language.

  • Prometheus, the original; ships with a query UI.

  • Grafana, the dominant visualization layer; PromQL completion in the editor.

  • promtool, promtool query instant ..., promtool check rules for unit-testing alerts.

  • Cortex / Mimir / Thanos / VictoriaMetrics, horizontally- scalable backends speaking PromQL.

  • promlens (now part of Grafana), visual PromQL builder.

Strengths#

The reasons PromQL won. It’s purpose-built for time series, labels are first-class throughout, alerting and recording rules are baked in, and a single query language now travels across many backends. Knowing PromQL transfers cleanly to LogQL and TraceQL in the same Grafana stack.

  • Built for time series, every operation respects label sets.

  • Powerful, aggregations, joins, predicates in one expression.

  • Industry-standard, the same PromQL works across many backends.

  • Excellent for alerting, the for clause and recording rules are first-class.

Weaknesses#

The flip side. High-cardinality labels destroy Prometheus performance silently; the rate-vs-increase-vs-irate question trips up newcomers; range-vector and instant-vector confusion produces baffling errors; and string operations beyond label matching simply aren’t there.

  • Cardinality explosion, adding high-cardinality labels (user IDs, request IDs) breaks Prometheus performance. Cardinality awareness is a core skill.

  • Learning curve, rate(counter[5m]) vs. increase() vs. irate() confuses everyone at first.

  • Range-vector / instant-vector confusion, many functions take one or the other, not both.

  • No string operations beyond label matching; not for log analytics.

PromQL vs. LogQL vs. TraceQL#

Grafana Labs uses similar forms for related signals.

  • PromQL, metrics.

  • LogQL (Loki), logs; PromQL-influenced syntax with stream selectors.

  • TraceQL (Tempo), distributed traces; selectors over span attributes.

Knowing one accelerates the others.

Pitfalls#

The traps that catch teams in production. High-cardinality labels are the silent killer; window-size choices on rate trade smoothing against latency; counter resets break naive subtraction; aggregating quantiles directly is mathematically wrong; always sum the buckets first.

  • High-cardinality labels, user_id, request_id, free-form strings. Avoid; cardinality is the silent killer.

  • ``rate(counter[5m])`` with too small a window misses recent changes; too large smears them.

  • Counter resets, rate handles them; raw subtraction does not.

  • Aggregation on histograms, always sum the buckets first (sum by (le) (...)) before histogram_quantile; aggregating quantiles directly is wrong.

See Also#

Contents