Distributed Systems#
The moment the operator’s infrastructure spans more than one machine (a redirector chain in front of a C2, multi-region collection hitting a central analyst plane, a defended estate of sensors reporting to a SIEM), a set of properties that held trivially in a single process get hard. The constraints are well-studied; the implications are easy to forget on contact.
The Eight Fallacies#
Operators building distributed systems repeatedly assume things that aren’t true. Sun’s L. Peter Deutsch wrote them down in 1994 and the list still describes the assumption traps that catch every newcomer to distributed systems. All eight are false in production.
The network is reliable.
Latency is zero.
Bandwidth is infinite.
The network is secure.
Topology doesn’t change.
There is one administrator.
Transport cost is zero.
The network is homogeneous.
All eight are false. Designs that ignore them break in production.
CAP and PACELC#
Brewer’s CAP theorem: under a network Partition, a system must choose between Consistency and Availability. CAP is the most-cited (and most-misquoted) result in distributed systems; PACELC extends it to cover the no-partition case where you still trade latency for consistency.
CP, refuse writes during a partition (leader-based DBs without async followers).
AP, accept writes; reconcile later (Dynamo-style stores, multi-master).
CA, only meaningful in a non-distributed system.
PACELC extends it: even when there’s no partition (E**lse**), there’s a trade-off between Latency and Consistency. Synchronous quorums across regions are slow; async replication is fast but stale.
Consistency Models#
A spectrum of guarantees, from strongest to weakest. Stronger models are easier to reason about and harder to scale; weaker models go further on coordination cost and tolerate more failure. Pick the weakest model that satisfies the business requirement.
Linearizable, as if there were a single copy, with operations happening one at a time in real time. Hard to scale.
Sequential, as if a single global order exists, but not necessarily matching real time.
Causal, causally related operations are seen in order; concurrent ops may be seen in different orders by different observers.
Read-your-writes / monotonic reads, per-session guarantees.
Eventual, replicas converge if writes stop; in the meantime, anything goes.
Pick the weakest that satisfies the business requirement.
Time#
Wall clocks drift. Two machines on the same network rarely agree on “now” to better than tens of milliseconds, and across regions the gap is worse. Never trust wall clocks for ordering; use the logical-time mechanisms below when ordering matters.
Logical clocks (Lamport), give a partial order over events.
Vector clocks, give partial order plus the ability to detect concurrency.
Hybrid Logical Clocks (HLC), combine wall and logical time; used in CockroachDB and others.
TrueTime (Spanner), bounded uncertainty; “wait out” the bound for global consistency.
Practical rules.
Don’t do “latest wins” on wall clocks across machines.
Don’t compute durations across processes by subtracting timestamps.
Use NTP and accept that it has tens-of-milliseconds error.
Consensus#
How do multiple nodes agree on a value when any of them might crash, lag, or partition off? Consensus algorithms are the formal answer. You almost never implement them yourself; instead, you adopt one of the systems that does.
Paxos, the original; correct, hard to understand.
Raft, equivalent power, easier to explain. Used in etcd, Consul, CockroachDB, TiKV.
Zab, ZooKeeper’s variant.
You rarely implement consensus yourself. You use a system that does.
Quorums#
The arithmetic that decides whether a read sees the latest write in a replicated store. N replicas, W writes acknowledged on each insert, R reads from each query. To guarantee a read sees the latest write.
W + R > N
Pick to balance availability and latency.
W=N, R=1, writes wait for everyone; reads cheap.W=1, R=N, writes cheap; reads expensive.W = R = ceil((N+1)/2), balanced majority quorums.
Failure Detection#
You can’t tell a slow node from a failed one with certainty. The best you can do is make a probabilistic call: how long has it been since I heard from you, and how unusual is that for this node? Three approaches dominate, in increasing sophistication.
Heartbeats with timeouts, approximate but practical.
Phi-accrual, adaptive failure detection based on heartbeat history (Cassandra, Akka).
Suspicion vs. certainty, gossip-style memberships (SWIM, Serf) treat node states as probabilistic.
Idempotency, Again#
In a distributed system, “exactly once” is mostly a marketing term; you cannot deliver every message exactly once across an unreliable network. Real systems offer “at least once” delivery plus idempotent handlers, which is observably equivalent for the caller and actually achievable.
Carry an idempotency key on commands.
Dedupe events on a stable id.
Make state updates state transitions, not deltas.
The Two Generals’ Problem#
You can never be certain a message was delivered to a remote party. Any “are you sure?” reply itself can be lost. Practical systems live with this. acknowledge, retry, idempotent receivers, and accept that the world is eventually consistent.
Practical Heuristics#
The rules of thumb that separate a useful distributed system from an over-engineered one. Most teams overshoot by an order of magnitude on complexity; the heuristics below are the “actually start here” defaults the experienced practitioner reaches for first.
Single-master with synchronous replicas inside a zone covers an enormous range of needs.
Multi-region active/active is much harder than it looks; usually active/passive is enough.
The hardest problems are usually about state: data ownership, consistency boundaries, schema evolution; not about request routing.
When debugging, look for clock skew, partition events, and retry storms before assuming bugs in your code.