Observability

Observability#

Observability is the eyes and ears of the operator on a running system, the ability to ask new questions about deployed infrastructure without redeploying it. It is the difference between an outage caught in minutes and one guessed at for hours, and the feedback loop the operator mines to improve the tools and sharpen the training that come next.

Three pillars are universally cited, logs, metrics, and traces, plus their derivative, alerts. The blue-team operator owns the pipeline; the developer owns the emissions; the on-call operator watches the dashboards.

        flowchart LR
  subgraph Apps["Applications"]
    App1[Service A]
    App2[Service B]
    App3[Service C]
  end

  App1 -->|"logs"| LC[Log Collector]
  App2 -->|"logs"| LC
  App3 -->|"logs"| LC
  LC --> LogStore[(Log Store)]

  App1 -->|"metrics"| Prom[Prometheus / OTel]
  App2 -->|"metrics"| Prom
  App3 -->|"metrics"| Prom
  Prom --> TSDB[(Time-Series DB)]

  App1 -->|"traces"| OTel[OTel Collector]
  App2 -->|"traces"| OTel
  App3 -->|"traces"| OTel
  OTel --> Tempo[(Trace Store)]

  LogStore --> Dash[Dashboards]
  TSDB --> Dash
  Tempo --> Dash
  TSDB --> Alerts[Alerting]
  Alerts --> OnCall[On-call]

Logs#

The first observability pillar. Logs record discrete events in textual or, better, structured form. They are also the operator’s primary collection on defense, the record an attacker works not to appear in and the first place an intrusion shows.

  • Structured over textual, JSON or logfmt with consistent field names.

  • Levels, DEBUG, INFO, WARN, ERROR, and don’t overlog at INFO.

  • Correlate, include request, user, and trace IDs.

  • No secrets, never log credentials, tokens, or PII.

  • Sample noisy events, but never drop the rare ones.

Stack

Components

ELK

Elasticsearch, Logstash, Kibana

Loki

With Grafana

Splunk

Enterprise

Datadog Logs

SaaS

CloudWatch Logs

AWS

Cloud Logging

GCP

Metrics#

The second pillar. Numeric time series are cheap to store and query, within cardinality limits. Four metric types cover most needs. A metric spike is often the first sign of an intrusion, and a flat line where there should be noise is its own tell.

Type

Meaning

Counter

Monotonically increasing (http_requests_total)

Gauge

Value that rises and falls (queue_depth)

Histogram

Bucketed observations (request_duration_seconds_bucket)

Summary

Precomputed quantiles

Stacks include Prometheus with Grafana, OpenTelemetry, Datadog, New Relic, Honeycomb, AWS CloudWatch Metrics, and Google Cloud Monitoring. Three frameworks give a starting set of signals.

Framework

Signals

USE (resources)

Utilization, Saturation, Errors

RED (services)

Rate, Errors, Duration

Four Golden Signals

Latency, Traffic, Errors, Saturation

Traces#

The third pillar. Traces follow a single request through a distributed system, recording the parent and child spans and their timing. A trace reconstructs exactly what a request did, an intruder’s footprints included, which makes it the sharpest forensics the operator has.

  • Sample, 100% in dev, around 1% or tail-based in production.

  • Propagate context, pass traceparent headers across service boundaries.

  • Annotate spans with attributes like tenant, user, feature flag, and version.

Stack

Note

OpenTelemetry

Vendor-neutral SDK and protocol

Jaeger

Open-source tracing

Tempo

Grafana’s trace store

Zipkin

Open-source tracing

Honeycomb

High-cardinality analysis

Alerts#

The action layer on top of metrics and logs. Each rule earns its place against these checks before it reaches production. The alert set is the operator’s trip-wire grid, and what does not page is the gap an intruder works inside.

Rule

Why

Alert on symptoms

User-facing latency above SLO beats CPU above 80%

Pageable means actionable

If the on-call can do nothing, it should not page

Tune ruthlessly

Every false page erodes trust in the system

Runbook-linked

Every pageable alert links to a doc on what to do

SLIs / SLOs / Error Budgets#

The targets that define healthy and the budget for being unhealthy. A service already burning its error budget freezes feature work until it recovers.

Term

Meaning

SLI

A numeric indicator, the percent of fast or successful requests

SLO

A target, e.g. 99.9% of requests succeed over 30 days

Error budget

One minus the SLO, the unreliability you can spend on shipping faster

When you are under budget, ship features. When you are over, slow down and invest in reliability.

OpenTelemetry#

OpenTelemetry is the cross-language, cross-vendor standard for emitting traces, metrics, and logs. Instrument once with the OTel SDK; ship to any backend via the Collector.

Contents