Distributed Systems

Split Brain

Also known as: partition with two leaders

Definition

Split brain is the state where a network partition leaves two halves of a cluster each believing it is the authoritative one, so both accept writes. When the partition heals the two histories conflict, and some accepted writes must be discarded or manually reconciled.

Last reviewed · Part of the Architecture Glossary

In practice

The classic trigger is a two-node HA pair with a failover heartbeat. The link between them drops. Neither node is down; each simply cannot see the other. Each concludes its peer has failed and promotes itself. Both now serve writes to the same virtual IP range, and the shared floating address ends up split across two live primaries.

The defences, in the order they should be reached for:

  1. Majority quorum. A node only serves if it can see ⌊N/2⌋ + 1 members. Two minorities cannot both be majorities, so this makes split brain structurally impossible — at the price of a minority partition being unavailable.
  2. Fencing tokens. Every leadership term gets a monotonically increasing number; storage rejects writes carrying a stale token. This catches the paused-then-resumed old leader that a heartbeat check misses.
  3. STONITH. Power off the other node before taking over. Crude, effective, and requires out-of-band access.

When it matters

Any leader-elected system: Postgres/MySQL failover, Redis Sentinel, Elasticsearch masters, Kubernetes control planes, distributed locks.

Common mistake

Running an even-numbered cluster, or a two-node pair with no witness. There is no majority to be had, so the design has to pick a loser in advance — which is what a tiebreaker or arbiter node exists to do.

See also

Go deeper