One important cause of duplicates is an operation that succeeded without its caller receiving the response. The caller cannot distinguish a failed operation from a lost acknowledgement, so something retries.
The customer clicked submit, the order was created, the connection dropped before the confirmation arrived, and they clicked again. The payment processor sent a webhook, your handler processed it, the acknowledgement timed out, and the processor sent it again. The workflow called an API, the call succeeded, the network hiccuped, and the retry policy fired.
In each case the system behaved correctly and produced a duplicate. Adding more retries makes it worse. The fix is idempotency.
What idempotency means here
An operation is idempotent when performing it more than once has the same effect as performing it once. Reading is naturally idempotent. Setting a value to a specific number is idempotent. Incrementing a counter is not. Creating an order is not, unless you make it so.
You make it so with a key. The caller generates a value that identifies this specific business operation, sends it with the request, and the server stores it alongside the result. A second request with the same key returns the stored result instead of doing the work again.
Choosing the key
This is where implementations go wrong. The key must identify the business operation, not the attempt.
A random value generated on each retry is useless, because every attempt looks new. A value generated once when the user opens the form and reused across retries is correct.
Derive it from the operation where you can. Submitting a specific cart for a specific customer. Sending the quote follow up for opportunity forty seven at step two. Creating the fulfilment job for order one two three. Those descriptions are stable across retries, which is exactly the property you need.
Storing it correctly
The critical detail: the key must be recorded in the same transaction as the side effect.
If you write the record and then separately mark the key as used, a crash between the two produces a state where the work was done and the key is not recorded, so a retry duplicates it. One transaction, both writes, no window.
Store the key, a fingerprint of the request, the result and a timestamp. The fingerprint catches the case where the same key arrives with different content, which indicates a client bug and should be an error rather than silently returning the old result.
Keys need a retention period long enough for the operation's retry and reconciliation windows. Do not copy a provider's retention period without considering later manual replays. A permanent unique business reference may still be needed after a temporary request cache expires.
The inbound side is a separate layer
Idempotency keys protect operations you initiate. Deduplication protects operations others initiate against you.
Providers retry webhooks. Some deliver the same event more than once by design. Your handler must record the provider event identifier and process each one only once.
One useful pattern is to verify the provider's signature over the raw body, then durably store both the event identifier and the work payload in one transaction. A unique constraint prevents duplicate intake. Acknowledge only after that transaction commits. A worker then claims the stored work and applies local changes transactionally. A previously recorded event can be acknowledged again because its work remains recoverable.
Acknowledging fast and processing after also solves a second problem. If the handler does the work inline and takes too long, the provider times out and retries, which means a slow system generates duplicates purely because it is slow.
Ordering is a different problem
Idempotency stops the same event being applied twice. It does not stop two different events arriving out of order.
A subscription updated event can reach you before the created event. A shipped status can arrive before the production status. If your state machine applies whatever it last received, a late event moves the record backwards.
If the provider supplies a documented monotonic object version, compare it with the version already applied. An event timestamp alone is not necessarily an object version: events may describe different changes or share a timestamp. Otherwise retrieve current object state and serialize updates for that object. Test concurrent handlers as well as sequential delivery.
Where to apply it
Everywhere a repeat would be visible or expensive.
Order creation. Payment capture. Refunds. Fulfilment submission. Outbound email and messages. Task creation in a workflow. Any write to a third party system. Record creation from an inbound form.
The cheap test: if this ran twice, would anybody notice? If yes, it needs a key.
The cost of not doing it
It is not theoretical. The failure mode is a customer charged twice, a garment printed twice, a follow up sequence that sends four copies of the same email, and a support conversation that starts with an apology.
Retries need a design that accounts for the effect, not only the request.
A remote side effect is outside your transaction
Writing a local key and calling an email provider cannot usually be committed as one database transaction. If the provider accepts the email and your process crashes before storing the response, the local record does not establish whether it was sent. Marking it complete before the call simply creates the opposite problem: a crash can lose the send.
Use the provider's idempotency facility where the operation supports one, with a stable key and a documented retry window. Preserve the provider result identifier. If the provider has no such facility, look for a reliable operation lookup or design an operator review state for an uncertain outcome. A database flag alone cannot guarantee exactly one external effect.
Test the boundary deliberately: stop the worker after the remote acceptance but before the local result update. Restart it and inspect the external system, not just your job table. This exercise reveals whether the implementation prevents duplicates or merely records them neatly after they happen.