Why exactly-once is impossible across a third-party boundary

Kafka can offer exactly-once semantics within its own cluster - between producers, the broker, and consumers, using idempotent producers and transactional writes. That guarantee stops at your system's edge. The moment a consumer calls Twilio's API to actually send an SMS, you're making a network call to a system you don't control, over a connection that can fail after the SMS was sent but before you receive the success response. There is no way to distinguish "the call failed" from "the call succeeded but the response was lost" without asking the provider - and even that ability isn't universal.

This is a known result in distributed systems, not a gap in any particular vendor's SDK: exactly-once delivery to an external system, over an unreliable network, cannot be guaranteed end-to-end. The practical goal is not to achieve it - it's to make at-least-once delivery behave like exactly-once from the user's perspective.

Design for at-least-once, enforce idempotency at the edges

The workable pattern: assume every send might happen more than once, and make "happening more than once" harmless. That means persisting every notification with an idempotency key before sending (covered in detail in our companion piece), and having every worker check that key before acting - not after. 

async function processNotificationEvent(event) {
  const idempotencyKey = deriveKey(event); // event_type + user_id + event_id

  // Check the dedupe store FIRST - before any side effect.
  const alreadyProcessed = await redis.get(`processed:${idempotencyKey}`);
  if (alreadyProcessed) {
    return { status: 'skipped_duplicate', notificationId: alreadyProcessed };
  }

  const result = await sendViaProvider(event);

  // Mark processed only after a successful send, with a TTL long enough
  // to cover realistic redelivery windows (e.g. 24h for most queues).
  await redis.set(`processed:${idempotencyKey}`, result.id, 'EX', 86_400);
  return { status: 'sent', notificationId: result.id };
}

The subtle bug this avoids: if you mark an event "processed" before confirming the send succeeded, a crash between those two steps causes the event to be silently dropped on retry (it looks already-processed, but was never actually sent). Order matters: check before, mark after success.

At-least-once at every layer, not just one

Queue layer

Kafka/RabbitMQ redeliver on consumer crash or rebalance by design - this is expected, not a bug to "fix" by disabling redelivery.

Retry layer

Backoff-based retries (covered in our resilience piece) intentionally resend failed calls - the dedupe store is what makes that safe.

Provider layer

Some providers (Twilio, Stripe) accept their own idempotency key on the request itself - pass yours through when supported, as a second line of defense.

Passing idempotency keys through to the provider

Several providers - Stripe most notably - accept a client-supplied idempotency key on the request and will return the original result for a repeated key without re-executing the action. Where available, this is a strong second layer, catching duplicates even if your own dedupe store was somehow bypassed. 

await stripe.paymentIntents.create(
  { amount: 4999, currency: 'usd', customer: customerId },
  { idempotencyKey: idempotencyKey } // same key you stored on the notification/payment row
);

Key takeaways

  • True exactly-once delivery across a third-party API boundary is not achievable - design for at-least-once instead.
  • Idempotency keys, checked before any side effect and marked only after success, are what make at-least-once behave like exactly-once to the end user.
  • Queue redelivery is expected behavior, not a bug - your dedupe layer is what should absorb it, not "fixing" the queue.
  • Pass your idempotency key through to providers that support it (Stripe, some SMS gateways) as a second layer of protection.
  • This is the same underlying discipline as the outbox pattern and the notification audit-trail piece - they all rely on the same idempotency-key foundation.
If your integration currently assumes "the queue only delivers once" or "the API call either fully succeeds or fully fails," that assumption is the gap - and it's usually invisible until a redelivery actually happens in production.