Resilience

Resilience#

Resilience is how much failure the infrastructure absorbs before the mission feels it. Availability zones cover a datacenter loss, regions cover a geographic one, and the architecture decides which failures are survived automatically and which need a human and a runbook.

Two numbers anchor the design. The recovery point objective (RPO) is how much data the operator can afford to lose, measured backward from the failure; the recovery time objective (RTO) is how long recovery may take. Tighter targets cost more, active-active across regions buys near-zero RPO and RTO at a steep price, while backup-and-restore is cheap but slow. The operator sizes the targets to the mission, not to a brochure.

        flowchart TB
  subgraph R1["Region 1 (primary)"]
    A1[AZ a]
    A2[AZ b]
    A3[AZ c]
  end
  subgraph R2["Region 2 (failover)"]
    B1[AZ a]
    B2[AZ b]
  end
  R1 -. async replication .-> R2

Domains#

A failure domain is the blast radius of one fault. The architecture stacks them so a fault is contained at the smallest level that can absorb it, and only the rare large fault escalates to the expensive recovery path.

Domain

Fault contained

Covered by

Host

A single machine or rack

Redundant instances behind a balancer

Availability zone

One datacenter, its power and network

Spreading instances across zones

Region

A whole geography

A second region with replicated data

Provider

The cloud account or provider itself

A second provider or an offline copy

Targets#

RPO and RTO are set per workload, not once for the whole infrastructure. A billing ledger tolerates almost no data loss; a cache can be rebuilt from scratch. The operator records the targets, then proves the architecture meets them, because an untested target is a guess.

  • RPO, the data the operator can lose, set by how often state is replicated or backed up.

  • RTO, the downtime the mission can absorb, set by how fast traffic shifts to healthy capacity.

  • Per workload, tier the targets so spend follows the data that actually matters.

Strategies#

Disaster recovery runs on a spectrum from cheap and slow to expensive and instant. The operator picks the cheapest strategy that still meets the workload’s RPO and RTO.

Strategy

RPO

RTO

Cost

Backup and restore

Hours

Hours

Lowest

Pilot light

Minutes

Tens of minutes

Low

Warm standby

Seconds to minutes

Minutes

Medium

Active-active

Near zero

Near zero

Highest

Failover#

Failover is the mechanism that moves traffic off the failed domain. It only counts as resilience when it triggers without a heroic intervention, and when failing back afterward is itself a planned operation.

  • Detect, health checks decide a domain is gone before users do.

  • Shift, DNS or a global load balancer moves traffic to healthy capacity.

  • Quorum, stateful systems need an odd member count to avoid split-brain.

  • Fail back, returning to the primary is rehearsed, not improvised.

Testing#

A recovery path that is never exercised is a recovery path that does not work. The operator proves resilience on a schedule rather than discovering its gaps during the real incident.

  • Game days, rehearse the runbook under realistic, time-boxed pressure.

  • Chaos, inject host, zone, and dependency failures to surface bad assumptions.

  • Restore drills, recover a backup end to end and verify the data.

Pitfalls#

  • Untested backups, a backup never restored is a hope, not a plan.

  • Single-region control plane, a multi-region data plane steered by one region.

  • No quorum, an even member count that splits the brain on partition.

  • RTO fantasy, a target written on paper and never measured.

  • Silent replication lag, replicas that drift because nobody watches the lag.

References#

Contents