Designing Reliable Payment Webhooks for Production Systems

Payment webhooks are the backbone of any modern billing system. When a payment is processed, refunded, or fails, your system receives a webhook notification from the payment provider. Getting this right is critical — a missed or double-processed webhook can mean lost revenue or duplicate charges.

The Core Problem

Webhooks are HTTP POST requests sent by payment providers to your endpoint. The fundamental challenge is that the same event can be delivered multiple times. Network issues, timeouts, and provider retry logic mean you cannot assume a webhook arrives exactly once.

Idempotency is Everything

The most important property your webhook handler must have is idempotency — processing the same event twice should produce the same result as processing it once.

async function handleStripeWebhook(event: Stripe.Event) {
  const existing = await db.processedEvents.findUnique({
    where: { stripeEventId: event.id }
  });

if (existing) {
return { status: 'already_processed' };
}

await db.$transaction(async (tx) => {
await processPaymentEvent(tx, event);
await tx.processedEvents.create({
data: { stripeEventId: event.id, processedAt: new Date() }
});
});
}


Signature Verification

Always verify the webhook signature before processing. Every major provider gives you a signing secret.

const sig = request.headers['stripe-signature'];
const event = stripe.webhooks.constructEvent(
  request.rawBody,
  sig,
  process.env.STRIPE_WEBHOOK_SECRET
);

Queue-Based Processing

For high-volume systems, don't process webhooks synchronously. Acknowledge quickly (200 OK), then queue for async processing.

Key Lessons from Production

1. Store raw event payloads — you'll need them for debugging

2. Implement retry-with-backoff for downstream service calls

3. Alert on processing failures — a dead webhook means lost business logic

4. Test with provider simulators — Stripe CLI is excellent for this

5. Monitor webhook latency — providers retry if you're slow to respond

Production webhook reliability comes down to: verify early, acknowledge fast, process idempotently, and monitor everything.