Reliability

Reliability is what the operator’s tools deliver when the network is hostile, the upstream is degraded, and nobody is awake to nurse the job along. Things break (a campaign’s collection pipeline, a defended estate’s auth service, a redirector at 03:00) and the system has to keep working anyway, ideally in ways the user does not notice.

A handful of patterns do most of the work: timeouts, retries, circuit breakers, bulkheads, idempotency, graceful degradation. Each is cheap to apply at design time and brutally expensive to retrofit during the incident.

Timeouts

Every network call needs a timeout. Always. Defaults in most clients are too long or infinite, which means that under real-world failure modes the call hangs the caller until something else times out instead. Three rules make timeouts useful.

• Set timeouts per call, not just globally.

• Pick them based on the caller’s budget, not the callee’s typical latency: “if I haven’t heard back in N ms, I have to give up to honor my own SLA”.

• Add a small jitter to avoid synchronizing retries.

Latency distributions are heavy-tailed; the operator sizes timeouts against the tail, not the mean. The ratio of p99 to p50 is the number that matters.

        xychart-beta
    title "Latency percentiles (illustrative ms)"
    x-axis [p50, p90, p95, p99, p99.9]
    y-axis "latency (ms)" 0 --> 600
    bar [40, 95, 140, 320, 560]
    

Retries

Retry idempotent calls only. Retry on transient failures (timeouts, 5xx, 429), not on permanent ones (4xx non-rate-limit). The discipline below avoids the most common retry pathology, coordinated retry storms after a partial outage.

• Exponential backoff with jitter: min(cap, base * 2^n) * (0.5..1.5).

• Cap the number of retries; let the caller above decide whether to re-try the bigger operation.

• Budget retries, don’t retry inside every layer or you’ll get N^L total attempts.

Circuit Breakers

When a downstream is failing, fast-fail instead of piling on requests. The breaker treats a service like a fuse: trip when errors exceed a threshold, refuse calls for a cooldown, then half-open to probe whether things have recovered before resuming normal traffic.

• States: closed (normal), open (fail fast), half-open (probe).

• Trip on error rate or latency, not raw count.

• Pair with retries, not in place of them.

Bulkheads

Compartmentalize so one failure doesn’t sink the whole ship. The naval-architecture analogy is exact: bulkheads contain damage to the failing compartment so the rest of the system stays operational. In software, that means dedicated resources per dependency rather than shared pools that propagate trouble.

• Thread/connection pools per dependency, a slow downstream consumes its own pool, not all of them.

• Queues with max length, shed load instead of unbounded buffering.

• Tenant isolation, one noisy customer can’t degrade another.

Idempotency

A retry should be safe. See the Communication chapter for idempotency keys, dedupe on event id, state-transition operations rather than counters.

Graceful Degradation

When something fails, degrade rather than crash. Users tolerate “the recommendations are missing” or “the search is using yesterday’s index” much better than a 500. The patterns below are the standard graceful-degradation moves; building them in is cheaper than retrofitting them after an outage.

• Return cached data with a “stale” indicator instead of erroring.

• Drop optional features (recommendations, analytics) before core ones.

• Serve a read-only mode rather than 500-ing.

Backpressure

When a producer outpaces a consumer, do something other than buffer forever. Unbounded queues are the easy default and the single most common cause of cascading failure; they hide the imbalance until a queue full of stale work suddenly becomes the incident. Three approaches bound the buffer instead.

• Bounded queues, with explicit overflow policy (drop, reject, oldest-out).

• Reactive streams style (request-N) for explicit demand.

• Rate limits at the edge, token bucket, leaky bucket, sliding window.

Health Checks

The two-question split that orchestrators (Kubernetes, ECS, Nomad) use to decide what to do with a process. Conflating them is the most common mistake; a readiness check that calls downstream services will cascade an upstream outage into restart loops on every consumer.

• Liveness, “am I alive?” Restart on failure.

• Readiness, “am I ready to take traffic?” Remove from load balancer on failure.

• Distinguish them. Failing readiness while you load caches is fine; failing liveness restarts the pod.

• Do not make readiness checks call downstream services; you’ll cascade failures.

Failure Domains

Cluster failures have characteristic blast radii. Knowing which domain a given failure can take down is the basis for every replication, multi-zone, and multi-region decision; you spread along the dimensions you actually need to survive, not all of them.

• Process, a single instance.

• Pod / VM, one host.

• Availability zone, one datacenter inside a region.

• Region, entire geographic area.

• Provider, the whole cloud.

• Account / project, billing and IAM blast radius.

Spread replicas across zones; spread environments across accounts; consider multi-region only for the small fraction of services that genuinely need it.

Chaos / GameDays

The best way to know your reliability story works is to break things on purpose, in a controlled way. A planned exercise with the team in the room and a stop button finds problems an incident would have, without the production cost. Tools and patterns.

• Inject latency, drop traffic, kill instances, pause databases.

• Run regularly. The first run always finds something.

• Tools: Chaos Mesh, AWS Fault Injection Simulator, Gremlin, Toxiproxy.

Disaster Recovery

Two numbers define the conversation. Set them per service before the disaster, document them somewhere durable, and design backups and replication to actually meet them. The biggest pitfall. backups that have never been restored are hopes, not backups.

• RTO (Recovery Time Objective), how long until we’re back up?

• RPO (Recovery Point Objective), how much data are we willing to lose?

Document targets per service and design backups, replication, and runbooks to meet them. Test restores; backups that have never been restored are hopes, not backups.