Webhooks are just HTTP, and that is why they keep breaking

When a payment provider tells you your integration is "webhook based", it sounds like a technology. It is not. A webhook is a POST request, from their server to yours, that you did not request and did not ask for. It is HTTP, with all of HTTP's reliability, which is to say it can be delayed, duplicated, reordered, lost, or delivered to the wrong place. The providers know this, which is why their webhook documentation is a list of warnings: verify the signature, handle retries, deduplicate, process asynchronously, be ready for the same event twice.

None of this is mysterious. It is the same set of problems you solve for any distributed interaction, wearing a slightly different costume. And yet the webhook integration is where a lot of payment systems leak money, because teams treat the webhook handler as a regular endpoint and discover the hard way that the sender is not your friend, the network is not reliable, and the event is not going to arrive exactly once, in order, on time.

The sender is a stranger with a key

The first thing to internalize is that the webhook endpoint is public. Anyone who can reach your server can send a POST to it. The payment provider signs their requests, and your only defense is verifying that signature before you do anything with the payload. The standard shape is an HMAC: the provider hashes the raw body with a secret, puts the hash in a header, and your handler recomputes the hash and compares.

Two details matter here, and they are the ones that trip people up. First, the signature is over the raw body, exactly as received. If your framework parses the JSON before you get the raw bytes, or if it re-serializes and changes the whitespace, the signature breaks. You have to capture the raw body and verify against it before parsing. Second, the comparison should be constant time, because a naive string comparison is a timing side channel, and the signature is the only thing standing between your endpoint and an attacker who wants to fake a "payment succeeded" event.

Verify before trusting, and verify before even logging the body as structured data. The endpoint is public, and the first thing it should do is check the credential, like any other authenticated route. The webhook is not special because it is a webhook. It is special because the sender is a stranger holding a key, and the key check is the whole security model.

At least once is the contract

Here is the sentence to tattoo somewhere near the code: webhook delivery is at least once, not exactly once. The provider will retry, and the retry will be a full duplicate of the event. A "payment succeeded" event that the network delays can be delivered twice, once from the original attempt and once from a retry, and your handler cannot tell the difference by looking at the event, because it is the same event.

The deduplication is your job. The standard move is an idempotency record: store the event ID in a table with a unique constraint, and skip the event if it has already been processed. The event ID is the provider's identifier, and it is stable across retries. The handler that processes "payment succeeded" must be safe to run twice, which means the processing itself needs to be idempotent as well, because the dedup check and the processing are not atomic, and a crash between them will replay the event.

There is a classic failure that this design prevents and a naive design invites. The naive handler processes the event, then marks it processed, and a crash between the two means the retry processes it again. The correct design makes the processing and the marking one transaction, or makes the processing itself idempotent so the duplicate is harmless. The word that matters is "or", and the team that relies on the first without the second is one crash away from a duplicate charge.

The order is not a promise

Events from a single provider are not guaranteed to arrive in the order they happened. A payment can be created, then refunded, and the "refund" webhook can arrive before the "payment" webhook, because they took different paths through the network and one got delayed. If your handler processes "refund" without having seen the "payment", it will try to refund a payment it does not know about.

The fix is not to hope the order works out. It is to make the handler tolerate any order. That usually means the handler is a state transition, not a step in a sequence: "payment succeeded" sets a flag, "refund issued" sets another flag, and the system is consistent regardless of which arrives first. If the ordering genuinely matters, the event processor needs to hold events and reorder them, or the handler needs to be able to reject an event it is not ready for and wait for the retry.

This is where the "process asynchronously" advice comes from. The handler should not block on the network, call the provider, or do slow work, because the provider has a timeout, and a slow handler is a failed delivery, which is a retry, which is more load. The pattern is: verify, enqueue, respond fast, and process the event in a queue where ordering, retries, and backpressure are handled by the queue instead of by the webhook endpoint.

The retry budget is a contract, too

Providers retry on a schedule, with a limited budget, and then they stop and tell you the delivery failed. The "then they stop" part is important. A webhook that fails after all the retries is not silently lost, it is an event you need to reconcile. The mature integration has a reconciliation pass: a periodic job that asks the provider for the list of recent events, and compares it with what you have processed, and finds the gaps.

The reconciliation pass is the safety net under the whole webhook system. No signature, dedup, or ordering discipline catches the event that was never delivered. The provider's API is the source of truth, and the periodic diff is how you find out that the truth includes events your webhook never saw. Teams that skip reconciliation are betting that the delivery network never drops anything. The teams that have been in production for a while know the bet is a bad one.

The debugging story

Webhook debugging is a special kind of painful, because the failure is remote: the provider saw something, your server did something, and the evidence is scattered across two log systems. The integrations that are debuggable have two properties. First, the raw payload is logged before parsing, so the "what did we actually receive" question is answerable. Second, the processing has a trace ID that survives from the incoming request through the queue job to the database write, so the "what did we actually do with it" question is answerable. Without those two, every webhook incident is an argument about what happened.

The other thing that makes webhooks debuggable is a test mode. The providers all have one, and the integrations that use it ruthlessly are the ones that do not burn out on staging. Test mode lets you force the failures: duplicate delivery, delayed delivery, signature errors. A team that has seen the failure modes in test mode does not panic when they appear in production.

Treat the webhook like a stranger at the door

The right mental model is a stranger at the door who claims to be from the payment company. You check their ID before letting them in, which is the signature. You do not trust them to have said everything exactly once, so you keep your own records, which is the dedup. You do not trust the order of their visits, so you process the information in a way that works regardless of sequence. And every so often, you call the company and ask what they actually sent you, which is the reconciliation.

The webhook is just HTTP. The reliability work is yours, and it is the same work you would do for any system where the message can be delayed, duplicated, or lost. Do that work, and the webhook is boring. Skip it, and the webhook is how the money leaks out.