The Distributed Systems Failure Catalog

Every distributed system fails in the same fourteen ways. The names differ by company, the post-mortems get written from scratch each time, and the fix is usually already known to someone three teams over.

This is that list. Each entry gives you the trigger, the signature you will actually see in telemetry, the mitigation, and the trap — the fix that looks right and makes things worse.

Read it once now so you recognise the shape at 3 a.m., not so you can recite it.

How to use this catalog

The fastest path from an outage to a diagnosis is matching a signature, not a cause. Start from what you can see:

What you observe	Likely entries
Load spike at a round-number interval	Thundering herd, Cache stampede
Traffic to a dependency far exceeds traffic to you	Retry storm
System stays down after the trigger is removed	Metastable failure
One shard/partition hot, the rest idle	Hot partition
p50 fine, p99 catastrophic	Head-of-line blocking, Coordinated omission
Latency climbs steadily until timeout	Unbounded queue
Two nodes both believe they are primary	Split brain
Health checks green, users complaining	Gray failure
One bad record halts an entire consumer	Poison pill
Failure spreads to services that share nothing	Cascading failure, Connection pool exhaustion

---

1. Thundering herd

What it is. A large number of clients act at the same instant because something synchronised them.

Trigger. A cache TTL that was set at deploy time, so every entry expires together. A cron at :00. A dependency recovering, releasing every blocked caller at once. A mobile app that refreshes on the hour.

Signature. Load that is flat, then vertical, at a suspiciously round timestamp. The spike height is roughly the number of clients, not the number of users.

Mitigation. Jitter everything that has a schedule. A TTL of 600s becomes 600s ± 10%. A retry at 2s becomes random(0, 2s). On recovery, ramp admission rather than opening the gate — a token bucket refilled over 30 seconds beats an instant release.

Trap. Jittering only the retry and not the TTL. The cache is usually the bigger herd, because it synchronises on deploy time and deploys hit every instance simultaneously.

2. Retry storm

What it is. Retries multiply through layers until a struggling dependency receives many times its normal load — precisely when it can least handle it.

Trigger. Every layer retries three times. Three layers means 27x amplification. The dependency slows, more calls time out, more retries fire.

Signature. Inbound request rate at the dependency is a multiple of the caller's request rate, and the ratio climbs as latency climbs. This ratio is the single most useful metric you are probably not recording.

Mitigation. Three things, all of them required:

• Retry budgets. Cap retries as a fraction of successful traffic — 10% is a common ceiling. When the budget is exhausted, fail fast instead of retrying.
• Retry at one layer only. Usually the outermost one that knows the request is idempotent. Middle layers propagate the failure.
• Circuit breakers on the calling side, so a dependency that is failing stops receiving traffic entirely for a cooling period.

Trap. Exponential backoff alone. Backoff spaces retries out; it does not reduce their number, and with enough clients the steady-state amplification is unchanged. Backoff with jitter and a budget is the actual fix.

3. Cache stampede

What it is. A hot key expires and every concurrent request for it misses simultaneously, so all of them recompute the same value.

Trigger. High-traffic key + TTL expiry + expensive recomputation. A key served 5,000 times per second with a 200 ms recompute produces 1,000 concurrent identical computations at the moment it expires.

Signature. Periodic database CPU spikes at exactly the cache TTL interval, with the origin query being the same query thousands of times.

Mitigation.

• Request coalescing (single-flight): the first miss computes, the rest wait on that computation. This is the highest-leverage fix and most languages have a library for it.
• Probabilistic early expiry: refresh a key before it expires with a probability that rises as expiry approaches, so one unlucky request refreshes for everyone.
• Stale-while-revalidate: serve the expired value and refresh in the background. Requires that you can tolerate seconds of staleness — usually you can.

Trap. Raising the TTL. It makes the stampede rarer and larger, and it delays the moment you notice.

4. Metastable failure

What it is. The system enters a state where the load it generates from being broken is enough to keep it broken, even after the original trigger disappears.

Trigger. Anything that pushes the system past a threshold — a brief traffic spike, a slow deploy, a dependency blip. Then retries, queue growth, and cache misses form a sustaining loop.

Signature. The defining one: you remove the cause and the system does not recover. Traffic is back to normal. The bad deploy is rolled back. It is still down.

Mitigation. The only reliable escape is to shed load below the threshold — usually well below, because the recovery threshold is lower than the failure threshold. Drop traffic, drain queues, restart consumers, then ramp. Design for it in advance with admission control and bounded queues, because in the moment the instinct to "just add capacity" makes it worse by giving the loop more to feed on.

Trap. Restarting the service without shedding load. It comes up, immediately receives the full backlog, and re-enters the same state within seconds. You will do this three times before someone suggests turning traffic off.

5. Unbounded queueing

What it is. A queue with no limit converts an overload problem into a latency problem, and then into a total-failure problem.

Trigger. Arrival rate exceeds service rate for any sustained period. With no bound, the queue absorbs the difference — growing memory, growing latency, and doing work whose results nobody is waiting for any more.

Signature. Latency that climbs linearly and without limit while throughput stays flat. Eventually, work completing successfully for clients that timed out minutes ago.

Mitigation. Bound every queue — in-memory, thread pool, connection pool, message consumer. Then choose the shedding policy deliberately: reject on full (fail fast, honest), or LIFO under overload, which is counter-intuitive but correct — the newest request is the one most likely to still have a caller waiting. Add deadline propagation so work whose deadline has passed is dropped rather than executed.

Trap. Making the queue bigger. A bigger queue is a longer wait before the same failure, with more wasted work.

6. Hot partition

What it is. One shard receives a disproportionate share of traffic, so cluster capacity is irrelevant — you are limited by one node.

Trigger. A partition key with skewed distribution. tenant_id where one tenant is 40% of volume. date where all writes go to today. A celebrity user. A sequential ID that always lands on the newest shard.

Signature. One node at 100% CPU or IOPS, the rest under 20%. Throughput does not improve when you add nodes.

Mitigation. Choose keys by access distribution, not by what is convenient to look up. Salt hot keys (tenant_id:bucket where bucket is hash(request) % N) and scatter-gather on read. Split known-hot tenants onto dedicated infrastructure. For read-hot keys, a local in-process cache in front of the shard removes most of the load for a few seconds of staleness.

Trap. Adding shards. Rebalancing a skewed key space across more shards leaves the hot key exactly as hot; you have simply added idle machines.

7. Head-of-line blocking

What it is. One slow item blocks everything behind it in a strictly ordered channel.

Trigger. A single connection multiplexing many requests. An ordered partition where message 4,001 is slow. A thread pool where one endpoint's slow calls occupy every thread.

Signature. p50 healthy, p99 terrible, and the slow requests have no common feature except when they arrived. Tail latency correlates with the presence of one slow operation, not with the request itself.

Mitigation. Separate the channels. Bulkheads: a dedicated thread pool or connection pool per dependency, so one slow dependency cannot consume the shared budget. Separate queues by expected cost — a slow lane and a fast lane. Concurrency limits per endpoint rather than one global limit.

Trap. Increasing the pool size. It raises the number of slow operations required to block you, which delays the problem to a busier moment.

8. Split brain

What it is. A network partition leaves two halves of a cluster each believing the other is dead, and both accept writes.

Trigger. Any partition, plus a failover mechanism that promotes on timeout without a quorum.

Signature. After healing: conflicting records, lost updates, duplicate primary keys, two nodes with a PRIMARY role in their own view. Frequently discovered days later by a reconciliation job or a customer.

Mitigation. Quorum. A cluster of 2f+1 nodes tolerates f failures and refuses to make progress without a majority, which mathematically prevents two majorities existing at once. Fencing tokens — monotonically increasing epoch numbers that storage rejects if stale — stop a deposed primary whose writes are still in flight. Even-numbered clusters are a bug: four nodes tolerate the same single failure as three, and add a tie.

Trap. Resolving conflicts with last-write-wins. LWW is not conflict resolution; it is silent data loss with a timestamp, and clock skew decides which write survives.

9. Cascading failure

What it is. One component's failure raises load or latency on others, which fail in turn, until the failure reaches services that had nothing to do with the original fault.

Trigger. A shared resource — a database, a thread pool, a service discovery layer, a config store — or a retry pattern that converts one failure into load on the next tier.

Signature. A widening blast radius over minutes, and services failing in an order that has no relationship to the deploy that caused it.

Mitigation. Bulkheads to contain, circuit breakers to stop propagation, load shedding at the edge to protect the core, and graceful degradation so a non-critical dependency failing means a missing widget rather than a 500. Test it: fault injection on your top ten dependencies, one at a time, in production, on a Tuesday morning.

Trap. Treating the last thing to fail as the cause. In a cascade, the loudest failure is usually the furthest downstream.

10. Gray failure

What it is. A component is degraded but not down. It passes health checks and fails real work.

Trigger. A partially failed disk, a NIC dropping 2% of packets, a node with a corrupted route table, an instance whose GC pauses run to seconds, a dependency returning 200 OK with an empty body.

Signature. The gap that defines it: your dashboards are green and your users are not. Error rates elevated but below alert thresholds. One instance with subtly worse latency for weeks.

Mitigation. Health checks that exercise the real path — a query against the actual database, not return 200. Client-side health: let callers report per-instance success rates and let the load balancer act on outlier detection, because the caller sees the failure and the instance does not. Alert on user-visible SLIs, not on component liveness.

Trap. Deep health checks wired to automatic removal. A dependency blip then fails every instance's health check at once and the load balancer removes the entire fleet. Deep checks should inform routing weight and page a human; they should not be permitted to empty the pool.

11. Connection pool exhaustion

What it is. Every connection in a bounded pool is held by a slow or stuck operation, so healthy requests fail while the resource behind the pool is idle.

Trigger. A dependency slows from 20 ms to 2 s. Concurrency demand rises 100x. A pool of 50 is gone. Or: one leaked connection per error path, and enough error paths.

Signature. Timeouts acquiring a connection while the database reports low CPU and few active queries. The pool metric is the tell — pool.pending climbing while db.active_queries is flat.

Mitigation. Separate pools per dependency and per criticality, so batch work cannot starve the request path. Acquisition timeouts short enough to fail fast. Statement-level timeouts on the server so a stuck query cannot hold a connection indefinitely. Always monitor pending and wait_time, not just active.

Trap. Enlarging the pool. Usually the database's connection limit is the real ceiling, and a larger pool moves the failure from your process to a shared resource where it takes out everyone.

12. Poison pill

What it is. One malformed message that crashes the consumer, which restarts, re-reads the same message, and crashes again — indefinitely.

Trigger. A schema change, a null in a field the producer swore was non-null, an oversized payload, an encoding bug.

Signature. Consumer lag climbing linearly with a restart loop in the logs and an offset that never advances. Zero throughput on one partition, normal on the others.

Mitigation. A dead-letter queue with a retry counter, and a hard rule that deserialisation failures are never retried in place. Validate at the boundary. Make the DLQ alertable and drainable — an unmonitored DLQ is a data-loss mechanism with extra steps.

Trap. Skipping the offset manually and moving on. It clears the alert and loses the message, and the same class of message arrives again next week.

13. Clock skew

What it is. Two machines disagree about the time, and something correctness-critical depends on them agreeing.

Trigger. NTP drift, a VM pause, a leap second, or simply the fact that "simultaneous" is not a property distributed systems have.

Signature. Events ordered impossibly — a response logged before its request, a token rejected as expired the second it was issued, a cache entry that never expires because it was stamped in the future.

Mitigation. Never use wall-clock time for ordering or for mutual exclusion. Use logical clocks (Lamport, vector) for causality, monotonic clocks for durations, and fencing tokens for exclusion. If you genuinely need bounded-error physical time, you need explicit uncertainty intervals in the style of TrueTime — and then you must wait out the interval, which is the part people skip.

Trap. A distributed lock with a TTL and no fencing token. The holder pauses for GC past the TTL, the lock is granted elsewhere, and now two processes hold it. The TTL bounds the lock's lifetime, not the holder's belief in it.

14. Coordinated omission

What it is. A measurement failure, included here because it hides most of the others. Your load generator waits for a response before sending the next request, so when the system stalls, the requests that would have arrived during the stall are never sent — and never measured.

Trigger. Any closed-loop benchmark. Most load-testing setups, by default.

Signature. Benchmarks that look excellent and production tail latency that does not match. A p99 that improves when the system gets slower — the giveaway.

Mitigation. Open-loop load generation: send at a fixed rate regardless of responses. Record latency from intended send time, not from actual send time. Measure at the client, outside your infrastructure, and prefer p99.9 over p99 for anything that fans out — a request touching 100 services hits its slowest dependency's p99 more often than not.

Trap. Averages. An average latency figure is compatible with every failure in this catalog and diagnostic of none.

---

The pattern behind the patterns

Read the mitigations together and three ideas cover most of them:

1. Bound everything. Queues, pools, retries, concurrency, deadlines. Unbounded resources convert overload into collapse.
2. Break synchronisation. Jitter, coalesce, stagger. Correlated behaviour is what turns load into a spike.
3. Isolate. Bulkheads, per-dependency pools, cells, quorums. Isolation determines blast radius, and blast radius is what an incident is actually measured by.

The fourth is not a mechanism: assume the failure will happen and decide the behaviour in advance. Every entry above has a graceful version and an undignified version, and the difference is whether someone chose before the incident or during it.