Distributed Systems

Transactional Outbox

Also known as: outbox pattern

Definition

The transactional outbox pattern writes an outgoing message into an `outbox` table inside the same database transaction as the state change it describes. A separate relay then publishes those rows to the broker. It removes the dual-write problem: the state change and the message commit or fail together.

Last reviewed · Part of the Architecture Glossary

In practice

The problem is the dual write:

db.commit(order)         # succeeds
broker.publish(event)    # process dies here

The order exists and nothing downstream will ever hear about it. Swapping the order just gives you the opposite bug — an event for an order that does not exist.

The outbox collapses both writes into one:

BEGIN;
  INSERT INTO orders ...;
  INSERT INTO outbox (id, topic, payload, created_at) VALUES (...);
COMMIT;

A relay then reads unpublished rows and publishes them — either by polling with FOR UPDATE SKIP LOCKED, or by tailing the write-ahead log with change data capture (Debezium). CDC is the better option at volume: no polling interval to tune, no extra load on the primary.

Delivery is at-least-once, because the relay can publish and then crash before marking the row sent. Consumers must be idempotent.

When it matters

Any service that must both persist state and tell someone about it — which is most services in an event-driven system.

Common mistake

Never pruning the outbox table. It is a write-heavy append-only table in your primary OLTP database; without partitioning or a delete job it becomes the largest table you own and drags vacuum, backups and restores with it.

See also

Go deeper