A webhook endpoint that parses a payload and does the work is fifteen lines and wrong in six ways. Here is each one and what it costs.
One: verify the signature over the raw body
Follow the provider's documented verification scheme. For schemes that sign the raw request body, parsing and reserialising JSON can change the bytes and invalidate the signature. Do not disable verification to make a parsed body work.
Configure the route to retain the raw body where required. Prefer the provider's maintained verification library. If implementing a supported shared secret scheme yourself, use the required algorithm and a constant time comparison rather than an ordinary string equality check.
An unverified endpoint is an open write path into your system. Anyone who learns the URL can create orders.
Two: check the timestamp
Signature verification alone allows replay. A captured request stays valid forever if the only check is the signature.
Where the provider includes a signed delivery timestamp, enforce its documented tolerance and keep the receiver's clock accurate. Not every provider has this mechanism. Apply the replay controls its protocol actually supports, and deduplicate events independently. A delivery timestamp is not necessarily the time the business event occurred.
Three: acknowledge fast, process after
Providers enforce a delivery timeout, commonly a few seconds. If your handler does the business work inline and takes longer, the provider records a failure and retries, and now you are processing the same event twice while already running slowly.
Store the event and return success immediately. A worker reads unprocessed events and does the work. This decouples your processing time from the provider timeout entirely, and it means a slow downstream system does not generate duplicates.
Four: deduplicate on the event identifier
Providers retry, and some deliver at least once by design. Your endpoint will receive the same event more than once.
Insert the provider event identifier into a table with a unique constraint. If the insert conflicts, you have seen it: return success and do nothing.
Then mark the event processed inside the same transaction as the business effect. If those are separate writes, a crash between them leaves work done and the event unmarked, and the retry duplicates it.
Five: handle out of order delivery
Events are not guaranteed to arrive in the order they occurred. A subscription updated event can reach you before the created event. A shipped status can arrive before production started.
If your state machine applies whatever it last received, a late event moves the record backwards, and that is a bug that appears once a month and is almost impossible to reproduce.
Use a documented monotonic object version where the provider offers one. Do not treat an event creation timestamp as a universal version number. For providers without suitable versions, retrieve current object state and serialize application updates for that object so concurrent handlers cannot overwrite newer information with stale reads.
Six: reconcile
Webhooks are missed. A deployment window, a certificate problem, an outage at either end, or a provider incident. None of these produce an error on your side, because on your side nothing arrived.
A scheduled job that lists recent objects from the provider API and compares them against your records catches it. Run it on a schedule regardless of whether anything looks wrong.
The endpoint shape
Put together, the handler does very little.
Read the raw body. Verify the provider’s signature and replay controls. Store the event identifier and the work payload in one durable transaction before acknowledging receipt. A duplicate can be acknowledged only if the original work was durably recorded. A crash between recording the identifier and saving the payload must not discard the event permanently.
The worker does the rest: read unprocessed events oldest first, check the ordering against applied state, perform the business effect and mark the event processed in one transaction, and on repeated failure move the event to a dead letter queue with the payload and the error.
Operational details worth having
Log the event identifier with everything, so a support question traces through in one search.
Retain the work payload only as long as recovery requires, with access controls and appropriate protection. Redact secrets from diagnostic logs. A replay facility does not justify retaining personal information indefinitely.
Expose a replay action for operators, made safe by the idempotency already in place.
Alert on unprocessed event age rather than on individual failures. A single failure is noise. Events sitting unprocessed for an hour is a signal.
Providing webhooks to others
If you are the sender, the same list inverts. Sign with a per endpoint secret, include a timestamp, include a stable event identifier, retry with backoff over a documented window, give consumers a delivery log they can inspect, and provide a replay endpoint.
A valid signature is not the whole authorisation decision
After verification, check that the event belongs to the expected account and environment. A legitimate test event must not update a live order. In a platform serving several tenants, resolve the provider account to the tenant using trusted configuration rather than accepting a tenant identifier from an arbitrary payload field.
Limit body size before buffering the request, and validate the event structure before creating work. Preserve unknown event types for a bounded diagnostic path or deliberately ignore them according to the integration contract. Do not route an unrecognised type through the nearest familiar handler.
Exercise secret rotation with the provider's supported overlap procedure. Verify both a newly signed delivery and a delayed legitimate retry during the transition, then retire the old secret. This is separate from replaying an old captured request: a provider can sign a fresh delivery attempt for an older business event. Tests should distinguish the delivery signature's freshness from the event's business age.