Integrations fail. That is not the problem. The problem is the failure mode where nothing happens and nobody knows.
An order that errors visibly can be investigated. An order that was never submitted and produced no signal may remain unnoticed until somebody compares the systems or a customer asks about it. The difference is observability, not a guaranteed recovery time.
Three layers, in order
Retry. Transient failures are common and most resolve on their own. Exponential backoff with jitter, so a provider recovering from an outage is not hit by every client retrying in lockstep. A circuit breaker so a provider that is clearly down is not hammered while it recovers.
Dead letter queue. Anything that exhausts its retry budget is written to a durable store with the information needed for safe recovery, a redacted error, the attempt count and the timestamp. Protect retained payloads with access and retention controls. It is not logged and discarded: it is work an authorised operator can inspect.
Reconciliation. A scheduled comparison between systems that reports anything present in one and missing in the other. This is the layer that catches what the event pipeline never knew about, and it is the layer most often skipped.
The third layer exists because the first two only handle failures the system noticed. A webhook that was never delivered, a job that was accepted and then stalled at the provider, an event dropped during a deployment: none of them produce an error on your side, because on your side nothing happened.
Making the queue usable
A dead letter queue that nobody opens is a log file with extra steps. It needs to be an operational surface.
Show the payload, the error message, the attempt history and the related business record, so an operator can see which customer and which order this is without running a query.
Provide a replay action that re enters the item into normal processing. Recheck permissions, current business state and external idempotency retention before replaying. The original request may no longer be appropriate even when repeating it would not create a duplicate.
Provide a discard action with a required reason, because some failures are genuinely not recoverable and leaving them in the queue trains people to ignore it.
Separate the recoverable from the not recoverable. A timeout is worth retrying. A rejected print file or an invalid address needs a person and should not sit in the same list as transient errors.
Alert on depth, not on events
Alerting on every failure produces noise, and noise trains people to dismiss alerts.
Alert on the shape of the problem instead. Queue depth above a threshold. Queue depth growing over a period. The oldest item in the queue exceeding an age. Reconciliation reporting a mismatch. A circuit breaker staying open beyond a duration.
Those signals mean something is wrong in a way that needs attention, which is what an alert is for.
What reconciliation looks like
Pick the things that must agree and compare them on a schedule.
Paid orders against fulfilment jobs. Records in the source system against records in the destination. Processor payouts against recorded transactions and fees. Stock movements against a physical count for a sample of lines.
Report the differences rather than trying to resolve them automatically. An automatic resolution for a discrepancy nobody has understood is a good way to make a small problem symmetrical.
Run it whether or not anyone suspects a problem. The entire value of reconciliation is that it runs on days when everything seems fine.
Design for the deployment window
A specific and common cause of silent loss: a deployment restarts a worker mid job, and the item was taken from the queue but not completed.
Acknowledge work only after it is done, or use a visibility timeout so an unacknowledged item returns to the queue. Implement graceful shutdown so a process finishes what it holds before exiting. Both are small and both prevent a class of loss that is very hard to trace afterwards.
The test
Ask one question about any integration you run: if this stopped working right now, how would you find out, and how long would it take?
If the honest answer involves a customer, build the three layers.
Recovery work needs its own state machine
Separate waiting, claimed, resolved and discarded items. A claim should identify the operator and expire or be releasable if they leave the task. Without it, two people can both investigate the same order and independently trigger corrective work.
Preserve the failed operation as evidence. If a correction needs a different destination or quantity, record a new corrective operation linked to the original rather than editing history until the failure looks successful. Show the difference before dispatch. This matters when a fulfilment address has changed after an order was placed: replaying the old payload and silently replacing it are both decisions with consequences.
Do not retain confidential payloads forever merely because a job failed. Where possible, keep a reference to the business record and the minimal snapshot needed to explain the error. Restrict access to any snapshot that must remain, and define what happens when its retention period ends. A recovery queue can otherwise become an overlooked second database of personal information.
Measure the oldest unresolved business impact as well as item count. One blocked order can deserve attention even when the queue is small. Conversely, many duplicate delivery notices may represent one upstream incident. Group related failures so an operator can repair the cause before replaying the affected work.