Instrumentation

Instrumentation is the side of observability that lives inside the workload. The agents, exporters, SDKs, and probes that extract telemetry from a running process, host, or network and hand it off to the monitoring stack (Monitoring).

For the operator the discipline is twofold. On operator-built capability the choice of what to instrument determines what the operator can prove later when something on target misbehaves. On a defended estate the instrumentation layer is the first thing an attacker tries to blind; knowing how each agent reports heartbeat, where its config lives, and what its absence looks like is non-negotiable.

Three Surfaces

        flowchart LR
  subgraph wkl[Workload signals]
    direction TB
    APP[App metrics, logs, traces]
    LIB[Auto-instrumentation libs]
    SDK[OpenTelemetry SDK]
  end
  subgraph host[Host signals]
    direction TB
    NE[node_exporter / Telegraf]
    FB[Fluent Bit / Vector]
    SYS[systemd / journald / auditd]
    EBPF[eBPF probes]
  end
  subgraph net[Network signals]
    direction TB
    PCAP[packet capture]
    FLOW[NetFlow / IPFIX / sFlow]
    NF[node-level flow agents]
  end
  COLL[OpenTelemetry Collector / Vector]
  APP --> COLL
  LIB --> COLL
  SDK --> COLL
  NE --> COLL
  FB --> COLL
  SYS --> COLL
  EBPF --> COLL
  PCAP --> COLL
  FLOW --> COLL
  COLL --> STORE[(metrics / logs / traces stores)]
    

Surface	What it reports
Workload	The application’s view: request rates, latencies, errors, business events, traces. The signal the operator’s own code produces.
Host	The kernel’s view: CPU, memory, disk, network counters, syscalls, audit events. Reported by per-node agents.
Network	The fabric’s view: flow records, captured packets, DNS queries, TLS handshakes. Reported by switches, taps, eBPF, or hypervisor inspection.

OpenTelemetry

The CNCF-graduated cross-vendor standard for instrumenting and emitting telemetry. Three pieces matter to the operator.

Piece	Role
OTel API	Language-agnostic interface the application calls to record metrics, logs, spans.
OTel SDK	The language implementation behind the API. Exports telemetry over OTLP.
OTel Collector	Vendor-neutral pipeline daemon. Receives OTLP, processes (sampling, batching, attribute injection, redaction), exports to anywhere.

OTLP (the wire format) is the lingua franca; every modern backend speaks it. The operator standardises on OTLP at the collector boundary and pushes vendor-specific exporters behind the collector, swappable without touching code.

Minimal Python instrumentation:

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("my-service")

with tracer.start_as_current_span("handle_request") as span:
    span.set_attribute("user.id", user_id)
    do_work()

Auto-instrumentation libraries cover most of the stdlib and popular packages for Python, JavaScript / Node, Java, Go, .NET, Ruby. Start there before writing manual spans.

Host Agents

Agent	Role
node_exporter	Prometheus’s standard host metrics agent. CPU, memory, disk, network, filesystem, NTP.
Telegraf	InfluxData’s agent. Pull or push, several hundred input plugins; pairs with InfluxDB or any Prometheus-compatible store.
Datadog Agent	Vendor-proprietary, covers metrics, logs, traces, and processes in one binary.
Elastic Agent / Beats	Elastic’s per-signal agents (Metricbeat, Filebeat, Packetbeat, Auditbeat).
Fluent Bit	Log forwarder. Tails files, journald, kubelet, parses, filters, ships to anywhere.
Vector	Datadog’s Rust-based log + metric router. Drop-in for Fluent Bit + Logstash on high-throughput pipelines.
Promtail	Loki’s log shipper. Tail + parse + ship.

Each is a per-node DaemonSet (on Kubernetes) or a systemd unit (on classic hosts).

Application Libraries

The patterns the operator’s code uses to emit signal.

Metrics

Library	Role
prom-client (Node), prometheus-client (Python), micrometer (JVM)	Native Prometheus exposition. Application exposes /metrics; Prometheus scrapes.
OpenTelemetry Metrics SDK	Same API across languages. Pushes via OTLP; the collector translates to Prometheus, OTLP, or vendor formats.

Logs

Library	Role
Structured logger	pino (Node), structlog (Python), slog (Go), logback (JVM with JSON encoder). Always JSON.
Correlation	Inject trace ID, span ID, request ID into every log line so logs and traces join at query time.
PII redaction	Redact at the library layer, not downstream. Email addresses, tokens, request bodies.

Traces

Library	Role
OpenTelemetry SDK + auto-instrumentation	The default. One library covers HTTP clients, servers, DB drivers, message queues.
Vendor SDKs	Datadog APM, New Relic, Honeycomb beelines. Pre-OTLP era; still common.
Custom spans	Wrap operator-built work that is not covered by auto-instrumentation: external API calls, internal state machines, long-running jobs.

eBPF Instrumentation

eBPF programs attach to kernel hooks (kprobes, uprobes, tracepoints, network filters) and emit telemetry without a sidecar or recompilation. The current source of low-overhead, deep host and network instrumentation.

Tool	Role
bpftrace	One-liner DSL for ad-hoc kernel tracing. Read it like awk for syscalls.
Pixie	Auto-instrumentation for Kubernetes, runs as a DaemonSet. Captures HTTP, MySQL, Postgres, gRPC, DNS, TLS by sniffing on the kernel side.
Cilium Tetragon	Security observability: process exec, file access, network connections. Pairs with Cilium’s CNI.
Parca / Pyroscope	Continuous CPU profiling via eBPF stacktrace sampling. Always-on, low-overhead.

eBPF is privileged. On hostile networks the operator audits the loaded programs (bpftool prog show) and treats them as part of the trust boundary.

Network Telemetry

Source	Detail
NetFlow / IPFIX / sFlow	Flow records from switches, routers, hypervisors. Aggregate; cheap; lossy on detail.
Packet capture	Full content. Expensive at scale. tcpdump for forensics; Suricata or Zeek for analysis.
eBPF socket / TLS	Per-connection metadata without taps. Cilium / Pixie / Tetragon.

Sampling and Cost

Telemetry is unbounded; storage is not. The operator’s three levers:

• Sample at the source. Tracing libraries drop 90 to 99% of spans before they leave the process. Keep parent-based or rate-based; never random per span (breaks causality).

• Filter in the collector. Drop high-cardinality labels, low- value paths, internal probes. Cheaper to drop early.

• Tier the store. Hot store for the last 7 days, archive to object storage beyond that. Most queries hit the hot tier.

Sampling skews metrics. Compute rate and error from unsampled counters; use traces for the slow-path detail.

Operator Practice

• Instrument before you need to. Adding signal during an incident is too late.

• Three signals on every service. Request rate, error rate, latency p50 / p99. Anything more is detail.

• Correlate. Trace ID in every log, metric label on every span. One query crosses signals.

• Test the absence. A killed agent should page; “no data is good data” hides outages.

• Redact at the source. PII never reaches storage.

• Version the schema. Metric names are an API; rename with a deprecation cycle, never silently.

References

• Monitoring for the storage and alerting side.

• Orchestration for the cluster surface most agents instrument.

• OpenTelemetry

• OpenMetrics

• Fluent Bit

• Vector

• eBPF

• Pixie