The Outbox Pattern: Your Safeguard Against Lost Messages in Distributed Systems

The dual-write problem

A common-looking pattern: inside a request handler, update the database, then publish an event to Kafka or RabbitMQ. It reads as one operation. It is actually two independent systems, each with its own failure modes, and there is no transaction that spans both. If the database commit succeeds and the publish call then fails (network blip, broker down, process crash), your database says the order was placed - and no downstream consumer ever hears about it. If the publish happens first and the database write then fails, you get the opposite problem: consumers react to an event for something that never actually happened.

The fix: write the event, don't publish it, in the same transaction

The outbox pattern moves the "publish" step out of the request path entirely. Instead of publishing to Kafka directly, the request handler writes a row describing the event into an outbox table - in the exact same database transaction as the business write. Since it's the same transaction, the database itself guarantees both happen or neither does. A separate, independent relay process is then responsible for reading unpublished outbox rows and actually publishing them to the queue.

CREATE TABLE outbox (
  id           BIGSERIAL PRIMARY KEY,
  aggregate_id VARCHAR(80)  NOT NULL,
  event_type   VARCHAR(80)  NOT NULL,
  payload      JSONB        NOT NULL,
  published    BOOLEAN      NOT NULL DEFAULT false,
  created_at   TIMESTAMPTZ  NOT NULL DEFAULT now()
);

-- Inside the request handler, ONE transaction:
BEGIN;
  UPDATE orders SET status = 'confirmed' WHERE id = $1;
  INSERT INTO outbox (aggregate_id, event_type, payload)
  VALUES ($1, 'OrderConfirmed', $2);
COMMIT;
-- The business fact and the intent-to-publish either both land, or neither does.

Two ways to build the relay

async function relayOutbox() {
  const unpublished = await db.query(
    'SELECT * FROM outbox WHERE published = false ORDER BY id ASC LIMIT 100'
  );
  for (const row of unpublished.rows) {
    await kafka.publish(`events.${row.event_type}`, row.payload);
    await db.query('UPDATE outbox SET published = true WHERE id = $1', [row.id]);
  }
}
// Run on an interval (e.g. every 500ms) or trigger via CDC instead of polling.

What "guarantee" actually means here

The outbox pattern guarantees the event will eventually be published if the business write committed - it does not guarantee the consumer only sees it once. The relay can crash after publishing but before marking the row published, causing a redelivery on restart. This is why outbox-based publishing is inherently an at-least-once guarantee, and consumers on the other end still need the idempotency-key pattern to handle the occasional duplicate safely.

The combination that actually works in production: outbox pattern for guaranteed publishing + idempotency keys on the consumer side for safe handling of the occasional duplicate. Neither one alone is sufficient - together, they give you effectively exactly-once processing without needing true distributed exactly-once delivery, which doesn't exist end-to-end across independent systems.

Key takeaways

• Writing to a database and publishing to a queue as two separate calls in one request risks silent inconsistency between them - the dual-write problem.
• The outbox table turns "publish this event" into a database write, so it's covered by the same transaction as the business change.
• A relay process (polling or CDC-based) is responsible for actually publishing outbox rows - Debezium is the most common CDC choice for this.
• The outbox pattern gives you guaranteed, at-least-once publishing - not exactly-once. Pair it with idempotency keys downstream.
• This pattern underpins reliable notification delivery and SAGA event choreography alike - it's foundational, not niche.