Latency Numbers Every Engineer Should Know — 2026
The orders of magnitude that decide your architecture, re-measured for NVMe, DDR5, 25GbE and LLM inference.
Last reviewed ·revision 1·first published
This is a living document. It is revised when the underlying numbers or practice change, not on a publishing schedule.
Show revision historyHide revision history
First publication. Storage rows re-derived for datacenter NVMe; HDD rows retained for context only. Added LLM inference and serverless cold-start sections.
Latency spans ten orders of magnitude, from a 1 ns L1 cache hit to a 200 ms trip to Sydney. The numbers matter less than the ratios: DRAM is ~100x L1, NVMe is ~1,000x DRAM, a same-datacenter round trip is ~3x NVMe, and a cross-continent round trip is ~200x that. Design against the ratios; they move far slower than the absolute figures.
The canonical version of this table is over a decade old, and the storage rows in it are wrong by three orders of magnitude. Spinning disks are gone from the request path. NVMe, DDR5, 25GbE fabrics and — new to this table — LLM inference have redrawn the landscape.
What has not changed is the reason the table exists. You do not memorise these numbers to predict performance. You memorise them so that when someone proposes putting a network call inside a loop that runs a million times, the answer arrives before the meeting ends.
How to read this table
Every figure is an order of magnitude for typical datacenter-class hardware, not a benchmark. Assume:
- Modern x86 server, DDR5, datacenter NVMe, 25 Gbps NIC.
- Warm caches, uncontended, p50 unless stated.
- All times are one-way except round trips, which are labelled RTT.
If your number is within 2x of the table, the table is working. If it is 100x off, something in your architecture is wrong — not the table.
The numbers
Inside the CPU
| Operation | Latency | Relative to L1 |
|---|---|---|
| L1 cache reference | 1 ns | 1x |
| Branch mispredict | 3 ns | 3x |
| L2 cache reference | 4 ns | 4x |
| Uncontended mutex lock/unlock | 17 ns | 17x |
| L3 cache reference (shared) | 30 ns | 30x |
| Main memory reference (DDR5) | 80 ns | 80x |
The DRAM row is the one people underestimate. Memory has barely improved in latency for fifteen years — bandwidth has grown enormously, latency has not. A pointer-chasing data structure that misses cache on every hop runs at roughly 12 million hops per second, single-threaded, no matter how fast your CPU is.
Memory and compression
| Operation | Latency |
|---|---|
| Compress 1 KB with LZ4/Snappy | 1–2 µs |
| Read 1 MB sequentially from memory | 30 µs |
| Decompress 1 KB with LZ4/Snappy | 0.5 µs |
Compression is nearly free relative to any I/O. If you are sending 1 KB over a network, compressing it first costs about 1% of the transmission cost.
Storage
| Operation | Latency | vs. DRAM |
|---|---|---|
| NVMe random read, 4 KB (datacenter) | 80 µs | ~1,000x |
| NVMe sequential read, 1 MB | 200 µs | ~2,500x |
| HDD seek (context only) | 8 ms | ~100,000x |
| HDD sequential read, 1 MB | 5 ms | ~60,000x |
This is the block that changed most. The classic table's "disk seek: 10 ms" made caching mandatory for almost everything. At 80 µs, NVMe is fast enough that a cache in front of it is a decision, not a given — and one you should justify, because a cache is a correctness liability.
Network
| Operation | Latency |
|---|---|
| Send 1 KB over 25 Gbps link | 0.4 µs |
| Same-rack RTT | 100 µs |
| Same-availability-zone RTT | 250 µs |
| Cross-AZ RTT (same region) | 1 ms |
| TLS 1.3 handshake, same region (1-RTT) | 1 ms |
| Cross-region RTT, us-east ↔ us-west | 65 ms |
| Cross-continent RTT, New York ↔ London | 75 ms |
| Cross-continent RTT, New York ↔ Sydney | 200 ms |
The cross-region figures are set by the speed of light in fibre, roughly 200,000 km/s, plus routing overhead. No amount of engineering will fix them. If your architecture requires a synchronous cross-region round trip on the request path, you have chosen a floor of 65 ms and the only remaining question is how much you add on top.
LLM inference
New section, and the one most likely to be wrong by next year.
| Operation | Latency |
|---|---|
| Time-to-first-token, frontier model, streaming | 200–600 ms |
| Per output token, frontier model | 10–30 ms |
| Full response, 500 output tokens | 5–15 s |
| Embedding call, single short document | 20–50 ms |
| Vector search, 1M vectors, in-memory HNSW | 1–5 ms |
Sit these next to the network table for a moment. A single LLM call costs more than a round trip to Australia. Everything about how you treat a cross-region dependency — make it asynchronous, stream it, cache aggressively, never put it in a loop, never put it between the user and a page render — applies to an LLM call with more force, not less.
Cold starts
| Operation | Latency |
|---|---|
| Serverless cold start, Node/Python, small bundle | 150–300 ms |
| Serverless cold start, JVM/.NET container | 2–10 s |
| Container pull + start, warm image cache | 1–3 s |
| Kubernetes pod scheduled → ready | 5–30 s |
The ratios that matter
Memorise five, not thirty:
- DRAM is ~100x L1. Cache locality is a real optimisation, not a micro-optimisation.
- NVMe is ~1,000x DRAM. An in-memory index still beats disk decisively.
- A same-AZ round trip is ~3x an NVMe read. Reading local disk is usually cheaper than asking another service — which is the entire argument against chatty microservices.
- Cross-region is ~250x same-AZ. Region boundaries are architectural boundaries. Treat them as such.
- An LLM call is ~10x cross-region. It belongs off the request path by default.
What this means in practice
Count your sequential network hops. A request that fans out to 20 internal services sequentially spends 5 ms on network alone at 250 µs a hop — before any service does work. Parallelise, or collapse the call graph.
Stop caching reflexively. With NVMe at 80 µs, a cache saves you roughly 78 µs and costs you an invalidation bug. Cache to reduce load, or to absorb a slow upstream — not out of habit.
Put p99 in the budget, not p50. Every number here is p50. Tail latency on shared infrastructure runs 10–100x higher, and a request touching 50 services will hit someone's p99 almost every time. Budget with tail numbers or your budget is fiction.
Draw the region boundary before you draw the service boundary. Splitting a service is reversible in a sprint. Splitting it across regions puts 65 ms in the middle of it permanently.
Sources
- Latency Numbers Every Programmer Should Know (original talk data) — Jeff Dean, Google (verified )
- Systems Performance, 2nd Edition — latency and time-scale tables — Brendan Gregg (verified )
Frequently asked
Are Jeff Dean's original latency numbers still accurate?
The ratios largely hold; several absolute values do not. Main memory is faster than the classic 100 ns figure only marginally, but storage changed by three orders of magnitude — a 4 KB random read moved from a ~10 ms HDD seek to roughly 80 microseconds on datacenter NVMe. Network round trips inside a datacenter also dropped from ~500 to ~250 microseconds. Anything derived from the disk row of the original table should be re-derived.
Why use order-of-magnitude latency numbers instead of benchmarking?
Because most architecture decisions are decided by a factor of 100, not a factor of two. Order-of-magnitude numbers tell you whether to cache, whether to co-locate, and whether a call belongs on the request path at all. Benchmark once the shape is fixed and you are tuning — not while you are choosing the shape.
What is a realistic same-datacenter network round trip in 2026?
Roughly 100 to 500 microseconds for a TCP round trip between two hosts in the same availability zone, depending on hop count and whether the path crosses a spine. Treat 250 microseconds as the planning figure, and remember that a single user request fanning out to 20 sequential internal calls has already spent 5 milliseconds on network alone.
How does LLM inference latency compare to traditional service latency?
It is three to four orders of magnitude worse. A frontier-model time-to-first-token of 200 to 600 milliseconds is comparable to a cross-continent round trip, and full generation of a few hundred tokens runs into seconds. An LLM call is not a service call; budget for it like a batch job or stream it.