Payments Webhooks Don't Get to Be Slow: Building an Idempotent Serverless Pipeline

Building the payments system for an early-stage property platform on Razorpay, the naive version of a webhook handler is a tempting shortcut: add a route to the main API, verify the signature, update the database, send a confirmation, done. It works in a demo. It's the wrong design for anything real, for reasons that don't show up until traffic does.

The problem with handling webhooks inline

Razorpay, like most payment gateways, expects your endpoint to acknowledge a webhook quickly — a few seconds — or it assumes delivery failed and retries. If your handler does real work inline before acking — writing to the database, sending an SMS or email, reconciling internal subscription state — you're racing that timeout on every single event. Miss it under load, and Razorpay retries an event you actually already received, which is exactly the kind of duplicate delivery that causes real damage if your system isn't built to expect it.

There's a second, quieter cost: payment-processing work has nothing to do with normal user-facing traffic, but handling it inline on the main API means it competes for the same request-handling capacity as everything else. A burst of webhook traffic — a wave of monthly renewals, a retry storm from an upstream hiccup — shouldn't be able to degrade the search experience for someone browsing listings at the same moment.

The design: decouple receipt from processing

The webhook listener is its own service — a Lambda function deployed via the Serverless Framework, entirely separate from the core Fastify API. Its job at the moment a webhook arrives is deliberately narrow: verify the Razorpay signature, run a fast idempotency check, persist the raw event, and ack. Nothing slow happens on that path. The actual business logic — updating subscription state, sending confirmations, reconciling records — runs afterward, asynchronously, decoupled from whether Razorpay is still waiting on a response.

Lambda specifically fits this because payment webhook traffic is bursty by nature, not constant. Provisioning a dedicated worker in the main API sized for peak load — the 1st of the month, or a gateway retry storm — means paying for that capacity the other 29 days too. Pay-per-invocation serverless matches a workload that's naturally spiky far better than a long-running process sized for its worst day.

Idempotency is the actual hard part, and "check for duplicates" isn't enough

Payment gateways generally guarantee at-least-once delivery, not exactly-once. The same event — the same Razorpay event ID — can legitimately arrive twice: a network blip, a retry because your ack was slow that one time, an operator manually resending an event from the Razorpay dashboard. If you don't design for this explicitly, a duplicate delivery double-processes a payment: a subscription gets extended twice, a confirmation email goes out twice, or worse.

The naive fix — "check if we've seen this event ID, and if not, process it" — sounds sufficient and isn't, because of a race condition that only shows up under concurrent delivery:

// looks safe, isn't:
const seen = await events.findOne({ eventId });
if (seen) return ack();       // <- two near-simultaneous
await processPayment(event);  //    deliveries can both pass
await events.insertOne({ eventId }); //  this check before either
                               //    marks itself done

Two copies of the same event arriving close enough together both read "not seen yet" before either one writes its own idempotency record. Both proceed to process the payment. The check-then-write pattern isn't atomic, and payment processing is exactly the domain where a rare race condition is still a real customer being charged twice, not just a theoretical edge case.

The fix is making the check-and-claim a single atomic operation instead of two separate ones, using the database's own concurrency guarantee rather than application-level logic:

// atomic: the unique index does the work, not application logic
try {
  await events.insertOne({ eventId, status: "processing" }); // unique index on eventId
} catch (e) {
  if (isDuplicateKeyError(e)) return ack(); // already claimed, no-op
  throw e;
}
await processPayment(event);
await events.updateOne({ eventId }, { $set: { status: "processed" } });

A unique index on eventIdturns "has this been claimed" into something the database enforces atomically, rather than something two racing requests both get to answer independently. Whichever delivery gets there first wins the insert; the other gets a duplicate-key error and simply acks without reprocessing. No distributed lock required — the database's own uniqueness guarantee is the lock.

What happens when processing fails after ack

Acking fast doesn't mean processing can't fail — a downstream email service can be down, a database write can time out. Because the raw event is persisted before anything else happens, failure downstream doesn't mean losing the event: it means a record sitting in "processing" that a retry pass can pick back up, without needing to ask Razorpay to redeliver anything. That persisted-event log is also what makes a manual replay possible when something needed a fix after the fact — you're replaying your own stored event, not hoping the gateway will resend it on request.

Why this deploys separately from the core API too

This is the same shape of decision as how the rest of the platform is architected: the three product-facing apps share one backend because they genuinely share logic and a solo engineer can't afford three deploy pipelines for no benefit. The payments webhook listener is the opposite call, deliberately — its own Lambda, its own deploy, its own scaling — because the stakes and the traffic pattern are different enough to justify the physical boundary immediately rather than deferring it. A bug or a traffic spike in payment processing can't exhaust capacity the main API needs for ordinary requests, because there's no shared capacity to exhaust.

Why this gets the strict treatment

Money-adjacent code is the one place I hold a stricter bar on how much I let move fast, including how much I let AI-assisted scaffolding handle unsupervised — an idempotency bug in a pipeline like this doesn't surface in code review as an obvious syntax problem. It surfaces months later as a support ticket that says "I was charged twice," after the bug has already run in production against real transactions. Designing for at-least-once delivery and atomic idempotency from the first version is dramatically cheaper than discovering the gap after an incident does the discovering for you.