Sharding
Also known as: horizontal partitioning, data partitioning
Sharding splits a dataset across independent database instances by a partition key, so each shard holds a disjoint subset. It is the only way past a single machine's write and storage ceiling, and it costs you cross-shard joins, cross-shard transactions and global uniqueness.
Last reviewed · Part of the Architecture Glossary
In practice
Choosing the key is the entire decision, and it is very expensive to change later. The test: can the majority of your queries be answered from one shard?
| Strategy | Good | Bad |
|---|---|---|
| Hash of entity ID | Even distribution | Range scans hit every shard |
| Tenant ID | Natural isolation, single-shard queries | Whale tenants create a hot partition |
| Geography | Data residency, local latency | Uneven population, cross-region users |
| Time range | Trivial archival | All writes hit the newest shard |
What you lose, and what replaces it:
- Joins across shards → denormalise, or fan out and merge in the application.
- Transactions across shards → a saga or two-phase commit.
- Auto-increment IDs → UUIDv7 or Snowflake IDs.
SELECT count(*)→ a maintained counter, or an approximation.
When it matters
Past roughly a single primary's write ceiling, or when one tenant's data must be physically isolated for compliance.
Common mistake
Sharding before exhausting the alternatives. Read replicas, connection pooling, partitioning within one instance, archiving cold rows and buying a bigger machine are all reversible; sharding is not. Modern hardware runs a single Postgres primary far further than most teams assume — see scaling to millions for where the real ceilings sit.
See also
- Hot PartitionA hot partition is a single shard receiving a disproportionate share of traffic, so it saturates while the rest of the cluster idles.
- Consistent HashingConsistent hashing maps keys and nodes onto the same circular hash space, assigning each key to the next node clockwise.
- Replication LagReplication lag is the delay between a write committing on the primary and becoming visible on a replica.
- QuorumA quorum is the minimum number of replicas that must respond for an operation to count.