Why naive retries make things worse

The simplest retry - "if it fails, try again immediately, up to 3 times" - is often worse than no retry at all. When a provider is struggling under its own load, a wave of immediate retries from every failed caller adds load precisely when the provider needs less. This is the thundering-herd problem, and it's the reason production retry logic needs structure, not just a loop.

Time-bucketed retries with backoff

Instead of retrying immediately, queue the failed message into a delayed queue with an increasing TTL: retry at 30 seconds, then 1 minute, then 5 minutes, then 30 minutes - each bucket implemented as its own queue (or a delayed-message feature in Kafka/RabbitMQ), with a small jitter added to each delay so retries from many callers don't all land in the same instant. 

const RETRY_SCHEDULE_MS = [30_000, 60_000, 300_000, 1_800_000]; // 30s, 1m, 5m, 30m

async function scheduleRetry(message, attempt) {
  if (attempt >= RETRY_SCHEDULE_MS.length) {
    await moveToDeadLetterQueue(message, attempt);
    return;
  }
  const baseDelay = RETRY_SCHEDULE_MS[attempt];
  const jitter = Math.random() * baseDelay * 0.2; // +/-20% jitter
  await delayedQueue.publish(message, { delayMs: baseDelay + jitter, attempt: attempt + 1 });
}

Dead letter queues: don't lose it, quarantine it

After the retry schedule is exhausted, the message shouldn't vanish - it moves to a Dead Letter Queue (DLQ), a holding area for messages that consistently failed. A DLQ turns "silent data loss" into "visible backlog you can inspect and replay," which is the entire point.

Visibility

A growing DLQ is a metric worth alerting on - it usually means an upstream integration is broken, not that individual messages are unlucky.

Manual or automated replay

Once the underlying issue is fixed (a rotated API key, a restored provider), DLQ messages can be replayed back into the main queue - nothing about the DLQ deletes the original event.

Idempotency still applies

Replayed DLQ messages must still resolve through the same idempotency-key check as any other retry - a DLQ is not an excuse to bypass dedupe.

Circuit breakers: stop calling a provider that's down

Retries and DLQs handle individual message failures. A circuit breaker handles the case where the provider itself is degraded - if 50% of calls to Twilio are failing in the last minute, continuing to send new calls just burns time and connections on requests that are very likely to fail too. A circuit breaker tracks the failure rate and "opens," short-circuiting new calls immediately (often to a fallback provider or a queue) until a cooldown period passes and a test call confirms the provider has recovered. 

class CircuitBreaker {
  constructor(threshold = 0.5, windowMs = 60_000, cooldownMs = 30_000) {
    this.threshold = threshold;
    this.windowMs = windowMs;
    this.cooldownMs = cooldownMs;
    this.state = 'closed'; // closed | open | half-open
    this.failures = [];
  }

  async call(fn) {
    if (this.state === 'open') {
      if (Date.now() - this.openedAt < this.cooldownMs) throw new Error('circuit open');
      this.state = 'half-open'; // allow one test call through
    }
    try {
      const result = await fn();
      if (this.state === 'half-open') this.close();
      return result;
    } catch (err) {
      this.recordFailure();
      throw err;
    }
  }

  recordFailure() {
    const now = Date.now();
    this.failures.push(now);
    this.failures = this.failures.filter(t => now - t < this.windowMs);
    const failureRate = this.failures.length / (this.windowMs / 1000);
    if (failureRate > this.threshold) { this.state = 'open'; this.openedAt = now; }
  }

  close() { this.state = 'closed'; this.failures = []; }
}

How the three layers fit together

1

Circuit breaker decides whether it's even worth attempting a call to the provider right now.

2

Retry with backoff handles individual call failures that pass the circuit breaker, spacing attempts to avoid pile-on.

3

Dead letter queue catches whatever survives every retry, keeping it visible and replayable instead of lost.

Key takeaways

  • Immediate, unbounded retries can make an outage worse - always back off, and add jitter so retries don't synchronize.
  • A DLQ is a holding area, not a trash can - growing DLQ depth is a metric to alert on, and messages should stay replayable.
  • Circuit breakers protect the provider (and your own connection pool) from being hammered during an outage - they're a different layer from retries, not a replacement for them.
  • Idempotency keys must survive every layer - a retried, DLQ'd, or replayed message still needs to dedupe correctly.
  • Monitor all three: retry counts, DLQ depth, and circuit-breaker state changes are the leading indicators of an integration going bad, often hours before customers notice.
If your current integration with a payment or messaging provider has no circuit breaker and no DLQ, the next provider outage will surface as a production incident instead of a metric spike you saw coming.