Monitoring

Monitoring is the production-time half of observability, the running operation of watching systems, surfacing change, and paging the operator when a fault crosses a threshold. The instrumentation layer (Instrumentation) produces the telemetry; this layer collects, stores, displays, alerts on, and routes it.

For the operator monitoring is two things at once. On defended estates it is the early-warning network for everything from disk-full to lateral movement. On operator-built capability it is the only honest view of whether the kit on target is doing what it was sent to do, with how much budget and at what cost.

Pillars

The three signals plus the responses that close the loop.

Signal	Use
Metrics	Numerical time series for rates, errors, latency, saturation. Cheap to store, fast to query, the substrate every alert rule sits on.
Logs	Discrete event records. Where the operator goes when a metric tells them something is wrong and the next question is “what happened”.
Traces	Causal records across services. Where the operator goes when “what happened” spans more than one process.
Events / change feed	Deployments, config changes, feature flags, certificate rotations. The chronology behind every “what changed at 02:14”.
Alerts	Rules that turn signal into a page. Page on symptoms (user-facing), not causes.
Dashboards	Pre-built views for the operator on shift. Should be readable at 03:00 without a Slack thread.

The Stack

A typical operator-facing monitoring stack assembles four layers. The components named below are the most common; substitutes exist for every slot.

        flowchart LR
  subgraph src[Sources]
    direction TB
    APP[Workload]
    HOST[Host / Node]
    NET[Network / Cloud]
  end
  subgraph collect[Collection]
    OTEL[OpenTelemetry Collector]
    FB[Fluent Bit / Vector]
  end
  subgraph storage[Storage]
    PROM[(Prometheus / Mimir / VictoriaMetrics / Thanos)]
    LOGS[(Loki / Elasticsearch / OpenSearch / ClickHouse)]
    TRACE[(Tempo / Jaeger / Honeycomb)]
  end
  subgraph view[View + Act]
    GRAF[Grafana]
    ALERT[Alertmanager]
    PAGE[PagerDuty / Opsgenie / Slack]
  end
  APP --> OTEL
  APP --> FB
  HOST --> OTEL
  HOST --> FB
  NET --> OTEL
  OTEL --> PROM
  OTEL --> TRACE
  OTEL --> LOGS
  FB --> LOGS
  PROM --> GRAF
  LOGS --> GRAF
  TRACE --> GRAF
  PROM --> ALERT
  ALERT --> PAGE
    

Layer	Role
Sources	The application, host, network, and cloud-provider APIs producing telemetry.
Collection	Daemons and sidecars that scrape, tail, batch, and forward telemetry. OpenTelemetry Collector is the cross-signal default; Fluent Bit, Vector, and Logstash for logs specifically.
Storage	Time-series stores for metrics, indexed stores for logs, span stores for traces. Operator picks open-source self-hosted or vendor-managed per signal.
View	Dashboards, ad-hoc query, alert routing. Grafana is the cross-vendor default UI; provider-managed dashboards (CloudWatch, Cloud Monitoring, Azure Monitor, VMware Aria Operations) ship out of the box on each cloud.

The Tools

A short catalog of the components an operator picks among.

Metrics

Tool	Role
Prometheus	The standard. Pull-based scraping, PromQL, alerting. Single-binary; horizontally scaled with Thanos, Cortex, Mimir, or VictoriaMetrics.
VictoriaMetrics	Drop-in Prometheus-compatible store, faster on large fleets.
Thanos / Cortex / Mimir	Long-term horizontal storage on top of Prometheus.
Datadog / New Relic / Dynatrace	SaaS metrics + APM. Pay per host or per metric series.
CloudWatch / Cloud Monitoring / Azure Monitor	Provider-native metrics. Bundled, integrates with provider IAM, billed per metric.

Logs

Tool	Role
Loki	Grafana Labs’ log store. Indexes labels, not contents; cheap at scale.
Elasticsearch / OpenSearch	Full-text indexed search. Heavier; the SIEM substrate for many estates.
ClickHouse	Columnar OLAP that doubles as a log store. SignalFX, Cloudflare, Uber log here.
CloudWatch Logs / Cloud Logging / Azure Monitor Logs	Provider-managed. The path of least resistance inside a single cloud.
Splunk	The enterprise SIEM. Strong correlation language (SPL), strong price tag.

Traces

Tool	Role
Tempo	Grafana Labs’ span store. Object-storage backed; pairs with Grafana.
Jaeger	The CNCF tracing project. Self-host with Cassandra or Elasticsearch.
Honeycomb / Lightstep	SaaS, query-first tracing. Drill into events rather than canned dashboards.
Datadog APM / New Relic / Dynatrace	SaaS APM with traces alongside metrics and logs.

Dashboards and alerting

Tool	Role
Grafana	The default open-source dashboard. Plugs into every store above.
Alertmanager	Prometheus’s alert router. Groups, dedupes, and forwards.
PagerDuty / Opsgenie / Splunk On-Call	Paging and on-call rotation.
Slack / Microsoft Teams	Where most non-paging alerts land.

Alerts

Page on symptoms (user-facing), not causes. The four signals worth alerting on directly are SRE’s golden signals.

• Latency, p99 of every external request.

• Traffic, request rate per service.

• Errors, 5xx rate, error budget burn rate.

• Saturation, CPU / memory / disk / queue depth headroom.

Cause-based alerts (disk-full, certificate-expiring-soon) belong on the dashboard, not the pager. They become alerts only when no symptom-based alert would catch the failure in time.

Alertmanager rule shape:

- alert: HighErrorRate
  expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
        / sum(rate(http_requests_total[5m])) by (service)
        > 0.05
  for: 10m
  labels: { severity: page }
  annotations:
    summary: "{{ $labels.service }} 5xx rate above 5%"
    runbook: "https://runbooks.example.com/high-error-rate"

Operator Practice

• SLOs over alerts. Define service level objectives; let burn- rate alerts wake the operator. A static threshold becomes wrong the moment traffic shifts.

• One source of truth per signal. A metric in two stores will drift between them; pick one, mirror as needed.

• Cardinality discipline. Every high-cardinality label (user ID, request ID, full URL) multiplies storage cost. Move high-cardinality data to logs or traces.

• Runbook links in every page. A page without a runbook is a request for the operator to invent one at 03:00.

• Post-incident review every page. What signal caught it, what signal should have caught it earlier, what alert change closes the gap.

References

• Instrumentation for the data-collection side.

• Orchestration for the substrate most monitoring stacks watch.

• SRE for the discipline that surrounds monitoring.

• Prometheus

• Grafana

• OpenTelemetry

• Google SRE Workbook: Alerting on SLOs