SRE

SRE#

Site Reliability Engineering, a discipline that came out of Google and was popularized by the 2016 Site Reliability Engineering book. SRE applies a software-engineering mindset to operations: instead of manual interventions, automate; instead of guessing, measure; instead of all-or-nothing reliability, set targets.

In 2026 SRE practices are widespread, even where the title isn’t.

Core Concepts#

The vocabulary that lets a team talk about reliability with precision. SLIs measure; SLOs target; SLAs contract; error budgets convert reliability into a tradable resource; toil names the work to automate away; postmortems convert incidents into learning. Each is the ground floor of SRE.

  • Service Level Indicator (SLI), a numeric measure of service health (e.g. % of requests succeeding within 200 ms).

  • Service Level Objective (SLO), a target on an SLI (e.g. “99.9% of requests succeed within 200 ms over a rolling 30 days”).

  • Service Level Agreement (SLA), the contract version, with consequences for missing it; usually less strict than the SLO.

  • Error budget, 1 - SLO. The amount of unreliability you get to spend on shipping fast.

  • Toil, manual, repetitive, automatable operational work; SRE reduces it.

  • Postmortem, blameless analysis of an incident; the deliverable is a list of action items.

  • Eliminating sources of toil is a core practice; SRE teams reserve significant time for this work.

The Error Budget Idea#

A 99.9% SLO over 30 days allows 43.2 minutes of “down”. That’s the error budget.

  • If you’re under budget, you can ship faster, take more risk, experiment.

  • If you’re over budget, you slow down: feature freezes, focus on reliability work, no new services.

This converts the perennial “ops vs. dev” tension into a number both sides agree on. When the budget is gone, both sides agree that reliability work takes priority. When the budget is plentiful, both sides agree that it’s safe to ship.

Picking SLIs#

The dimensions an SLI measures. Availability and latency are the obvious starts; throughput, quality, freshness, coverage, and correctness fill in the rest. The right SLI reflects what users actually care about, not internal resource metrics like CPU.

Common SLI families:

Family

Examples

Availability

% of requests not failing

Latency

% of requests faster than target

Throughput

requests / s sustained

Quality

% of responses with full data (vs. degraded)

Freshness

% of records younger than target

Coverage

% of jobs that ran on schedule

Correctness

% of records that pass validation

The Google SRE book frames everything as: what do users care about? A user cares that their request succeeded and was fast. They don’t care about CPU utilization. SLIs reflect user-visible quality.

The Four Golden Signals#

Google’s starter framework for instrumenting any service. Latency, traffic, errors, and saturation cover the surface area where most incidents start. Pair with USE for resources and RED for services to triangulate; each is a different lens on the same data.

  • Latency, how long requests take.

  • Traffic, how many requests / second.

  • Errors, how many requests fail.

  • Saturation, how full the resource is (CPU, memory, queue).

Pair with USE (Utilization / Saturation / Errors, for resources) and RED (Rate / Errors / Duration, for services). Each is a different lens on the same data.

Toil Reduction#

The SRE book defines toil precisely: manual, repetitive, automatable, tactical, devoid of enduring value, scaling linearly with service growth.

If 100% of an SRE team’s time is toil, you have a manual ops team in new clothes. Google targets <50% toil; the leftover time goes into engineering work that reduces future toil.

A useful test: track time-spent for a sprint or two; categorize as toil or project work. Honest measurement is half the fight.

Postmortems#

The artifact that turns an incident into improvement. The properties below distinguish a real postmortem from a blame exercise: blameless framing, timely writing, specific timeline, actionable findings with owners, and wide distribution so the lessons spread.

Every significant incident gets a postmortem. Properties:

  • Blameless, on systems and process, not individuals.

  • Timely, written within a week of the incident.

  • Specific, timeline with timestamps; root causes (often multiple); contributing factors.

  • Actionable, each finding produces an action item with an owner and a due date.

  • Read, distributed widely; the lessons spread.

The action items are the deliverable. A postmortem with no follow- through is theater.

On-Call#

The practices that separate a healthy rotation from a burnout factory. Time-bounded shifts, compensation, actionable pages with linked runbooks, page tracking, handoff rituals, and rehearsed escalation paths; each is a known fix to a known failure mode.

Practices that distinguish a sustainable rotation from a burnout factory:

  • Time-bounded, one week per N engineers, not 24/7 forever.

  • Compensated, pay or comp time off.

  • Page-able means actionable, if there’s nothing on-call can do, it isn’t a page.

  • Runbooks, every page links to one.

  • Tracking, pages per shift; if it’s miserable, fix the noisy alerts before adding more.

  • Handoff rituals, the outgoing on-call briefs the incoming.

  • Escalation paths, clear, documented, rehearsed.

Chaos Engineering#

Deliberately inject failures to verify the system handles them:

  • Kill instances at random.

  • Inject latency between services.

  • Block egress to a downstream.

  • Saturate CPU / memory.

  • Pause a database.

  • Drop network packets.

Tools: Chaos Mesh, AWS Fault Injection Simulator, Gremlin, Toxiproxy, Litmus.

The first run always finds something. Repeat regularly. Pair with game days where the on-call team practices the response.

Capacity Planning#

The practice of staying ahead of traffic. Headroom buffers absorb spikes; forecasts model expected growth; load tests find the latency knee before customers do; saturation alerts on queue depth and connection count cover what CPU misses; autoscaling automation is verified, not assumed.

  • Headroom, run at 50-70% utilization so spikes are absorbed.

  • Forecasting, model expected growth; provision ahead.

  • Load testing, find the knee of the latency curve before a customer does.

  • Saturation alerts, not just CPU; queue depth, connection count, file descriptors.

  • Scaling automation, HPA, cluster autoscaler, Karpenter; verify they actually fire under load.

Releases#

Reliability and release engineering converge:

  • Progressive delivery, canary, blue/green, feature flags.

  • Automated rollback, on SLI / golden-signal regression.

  • Pre-production environments, shadow traffic, integration tests against real dependencies.

  • Tracking, DORA metrics: deployment frequency, lead time, MTTR, change failure rate.

The DORA metrics correlate with team performance and well-being. Track them; nudge them in the right direction.

SRE vs. DevOps vs. Platform Engineering#

Three overlapping disciplines, often in the same team:

  • DevOps, the cultural movement: shared responsibility, automate the toil, ship fast and safely.

  • SRE, a specific implementation: SLOs, error budgets, toil reduction, postmortems. Originated at Google.

  • Platform Engineering, builds the IDP that makes the above achievable for product teams at scale.

In a small org, one team does all three. In a large one, an SRE team focuses on reliability; a platform team builds the IDP; product teams practice DevOps culture on top.

Common Pitfalls#

The traps that catch teams adopting SRE in name only. Cargo-cult SLOs no one believes; toil-only teams that just got renamed; postmortems with no follow-through; vanity five-nines targets that fall apart on math; treating SRE as a tier instead of a practice; each defeats the purpose.

  • Cargo-cult SLOs, numbers no one believes; targets ignored during real incidents.

  • All-toil teams branded as SRE, without engineering time, the function is just ops.

  • Postmortems without follow-through, generates documents, not improvements.

  • Vanity reliability targets, “five nines” with no engineering rigor; the math doesn’t survive contact with the budget.

  • Treating SRE as a tier instead of a practice, “send it to the SRE team to fix” defeats the purpose.

Reading#

  • Site Reliability Engineering (Google, 2016).

  • The Site Reliability Workbook (Google, 2018).

  • Building Secure & Reliable Systems (Google, 2020).

  • Seeking SRE (David Blank-Edelman, 2018).

  • Implementing Service Level Objectives (Alex Hidalgo, 2020).

The first two are free online and worth reading even if you never adopt the title.

See Also#

Contents