Thundering Herd
Also known as: herd effect, retry storm
A thundering herd is a surge of simultaneous requests to one resource, triggered by an event that releases many waiting clients at once — a cache expiry, a service restart, a synchronised retry, a cron. The spike is caused by correlation in timing, not by a rise in real demand.
Last reviewed · Part of the Architecture Glossary
In practice
Correlation is the whole mechanism, and it is usually something you built. A deploy restarts 200 pods within the same second; all 200 open their connection pools, load the same config, and warm the same cache entry simultaneously. Or a downstream service returns 503 for three seconds and 5,000 clients all retry after exactly one second — together.
The fixes all break the correlation:
- Jitter. Full jitter (
sleep = random(0, base × 2^attempt)) rather than fixed exponential backoff. Plain exponential backoff keeps a synchronised herd synchronised; it just spreads it over longer intervals. - Request coalescing. One in-flight fetch per key; the rest wait on it (Go's
singleflight, a per-key mutex). - Staggered startup and staggered TTLs. A small random offset on boot delay and on every cache TTL.
- Retry budgets. Cap retries at a fraction of the request rate — say 10% — so a downstream blip cannot become a self-inflicted DDoS.
When it matters
Deploys, autoscaling events, regional failovers, cron-driven batch jobs, and any client fleet you do not control but whose retry policy you specify in an SDK.
Common mistake
Adding retries without a budget. Three retries per request turns a partial outage into 4x load exactly when the dependency has the least headroom — the retry becomes the outage.
See also
- Cache StampedeA cache stampede occurs when a popular cache entry expires and every concurrent request for it misses simultaneously, sending the full uncached load to the origin at once.
- BackpressureBackpressure is the mechanism by which an overloaded component signals upstream to slow down, rather than accepting work it cannot complete.
- Circuit BreakerA circuit breaker wraps calls to a dependency and stops making them once the failure rate crosses a threshold, failing fast for a cool-down period before letting a trial request through.