The $40K Mistake: What Happens When You Skip Idempotency in Payment Webhooks

Why this keeps happening

Every major payment provider - Stripe, PayPal, Razorpay - documents that webhooks can be delivered more than once for the same event. It's not a defect; it's how they guarantee at-least-once delivery on their side, the same pattern covered in our exactly-once vs at-least-once piece. A webhook handler that treats every delivery as a brand-new event, with no memory of what it already processed, will eventually process the same payment confirmation twice.

What the naive handler looks like

// The version that causes the problem
app.post('/webhooks/payment', async (req, res) => {
  const event = req.body;
  if (event.type === 'charge.succeeded') {
    await fulfillOrder(event.data.orderId);   // grants access, ships the order, etc.
    await creditWallet(event.data.userId, event.data.amount); // <- runs again on redelivery
  }
  res.sendStatus(200);
});

Nothing here looks wrong on a single read. It only becomes wrong under redelivery - which is exactly why this gap survives code review so often. It doesn't fail in testing; it fails under the provider's real retry behavior, which most local dev environments never exercise.

Where the cost actually comes from

The fix: a five-line check that changes everything

app.post('/webhooks/payment', async (req, res) => {
  const event = req.body;
  const idempotencyKey = `webhook:${event.provider}:${event.id}`; // provider's own event ID

  const inserted = await db.query(
    `INSERT INTO processed_webhooks (idempotency_key, event_type)
     VALUES ($1, $2) ON CONFLICT (idempotency_key) DO NOTHING RETURNING id`,
    [idempotencyKey, event.type]
  );

  if (inserted.rowCount === 0) {
    return res.sendStatus(200); // already processed - acknowledge, don't re-execute
  }

  if (event.type === 'charge.succeeded') {
    await fulfillOrder(event.data.orderId);
    await creditWallet(event.data.userId, event.data.amount);
  }
  res.sendStatus(200);
});

The detail that matters most: use the provider's own event ID as (or as part of) your idempotency key, not something you generate yourself - the provider's ID is what's identical across every redelivery of the same event, which is exactly the property you need.

Cheap to prevent, expensive to clean up

The fix above is a database table and a handful of lines, addable to an existing handler in under a day once you know to look for the gap. Cleaning up after it's caused duplicate charges or credits at scale - reconciling records, refunding or clawing back the difference, and rebuilding customer trust in the ones who noticed - is a materially larger undertaking, and one that's much harder to scope in advance.

Key takeaways

• Webhook redelivery is documented, expected provider behavior - not an edge case worth deprioritizing.
• A handler with no idempotency check will eventually double-process a redelivered event; it's a matter of when, not if.
• Use the provider's own event ID as your idempotency key - it's the one value guaranteed identical across redeliveries.
• The fix is small and cheap when added proactively; the cleanup after an incident is neither.
• This is the same foundational pattern as our notification idempotency piece - webhook handlers are just another entry point that needs the same discipline.