Performance & Latency

Coordinated Omission

Also known as: load generator bias

Definition

Coordinated omission is a measurement error in which a load generator waits for a slow response before sending the next request, so the requests that would have been issued during the stall are never recorded. The resulting latency distribution omits exactly the worst samples and can understate p99 by an order of magnitude.

Last reviewed · Part of the Architecture Glossary

In practice

A closed-loop generator with 100 threads at a target of 1,000 req/s. The server stalls for 1 second. During that second every thread is blocked, so instead of 1,000 requests experiencing a delay, 100 are recorded as slow and 900 are simply never sent. The average of the recorded set looks acceptable; a real user population, which does not politely wait, would have seen 1,000 slow requests.

Symptoms that you are looking at an omitted distribution:

  • Reported p99 is close to p50.
  • Achieved throughput dips during the stall instead of the queue growing.
  • The client's p99 is far better than the server's own histogram.

Fixes: use an open-loop generator that maintains the intended arrival rate regardless of responses (wrk2, Gatling's open model, k6 with arrival-rate executors), or correct after the fact — HdrHistogram's recordValueWithExpectedInterval synthesises the missing samples.

When it matters

Every load test whose numbers will be quoted in a capacity or SLO decision.

Common mistake

Signing off on a latency SLO using JMeter or wrk defaults. Both are closed-loop. The test passes, production misses the same target, and the gap is a measurement artefact rather than an environment difference.

See also

Go deeper