Payment webhooks that survive duplicates and disorder

Payment providers do not promise that a webhook will arrive once, in order, while your database is healthy. Production systems have to assume the opposite.

Checkout.com says its webhooks are delivered at least once and may arrive out of order. It retries failed deliveries up to eight times. Stripe retries undelivered events for up to three days. Adyen warns that duplicate events can carry a later eventDate even when the eventCode and pspReference are unchanged.

A handler that immediately runs business logic will eventually send the same order twice, move a refunded payment back to "paid", or acknowledge an event it never stored. The safer design is an inbox: authenticate the request, persist the event once, return success, then process it asynchronously under explicit state rules.

Cover and infographic: original SultanByte editorial artwork.

Treat delivery and payment state as separate problems

A webhook delivery answers one question: did the provider attempt to tell us something? It does not prove that your local payment row is current.

Keep at least three identifiers:

• the provider's unique event or delivery ID, when it supplies one;
• the provider's payment or transaction reference;
• your own order or payment ID.

They have different jobs. The event ID deduplicates one notification. The provider reference groups events about the same remote payment. Your internal ID connects that payment to fulfilment, accounting and support.

Do not use an event type such as payment.succeeded as the idempotency key. Thousands of payments can share the same type. Do not use a timestamp either. Retries, clock precision and provider behaviour make it a poor identity.

When a provider has no stable event ID, derive a fallback from documented immutable fields and the raw payload hash. Keep that adapter provider-specific. A universal concatenation rule tends to hide collisions.

Verify the exact bytes before parsing JSON

Webhook signatures authenticate bytes, not your application's reconstructed object.

Stripe requires the raw UTF-8 request body, the Stripe-Signature header and the endpoint secret. Whitespace changes, key reordering or parsing the body before verification can break the check. Checkout.com sends an HMAC in the hex-encoded Cko-Signature header. Adyen's scheme depends on the webhook type: Standard webhooks carry a signature in additionalData, while some other webhook types carry it in headers.

That difference belongs in a provider adapter, not scattered across route handlers:

interface VerifiedPaymentEvent {
  provider: "stripe" | "checkout" | "adyen";
  eventId: string;
  paymentRef: string;
  eventType: string;
  occurredAt: string;
  payload: unknown;
}

interface WebhookAdapter {
  verify(rawBody: Buffer, headers: Headers): Promise<boolean>;
  parse(rawBody: Buffer): VerifiedPaymentEvent;
}

Capture the raw bytes first. Verify them with the provider's supported library where possible. Parse only after verification succeeds. Reject an invalid signature without putting the payload on the business queue.

Use a timing-safe comparison when an SDK does not handle it for you, and rotate secrets without creating a hard cut-over. Adyen notes that a newly generated HMAC key can take time to propagate, so receivers should temporarily accept the previous key too. Record which key version verified the request, but never log the secret or signature material.

HTTPS is still required. HMAC proves message integrity and knowledge of the shared secret; TLS protects the request and headers in transit. The OWASP REST Security Cheat Sheet recommends HTTPS-only REST endpoints.

Persist before you acknowledge

The ingress transaction should do very little:

1. verify the signature;
2. extract the provider event identity;
3. insert the immutable envelope into an inbox table;
4. enqueue or mark it for processing;
5. return a successful status.

Stripe tells endpoints to return a 2xx before complex logic. Adyen recommends storing the message in a database or queue, acknowledging it with 200 or 202, and applying business logic afterward. Adyen treats a response that does not arrive within 10 seconds as a failed delivery.

A minimal PostgreSQL table could look like this:

create table payment_webhook_inbox (
  id bigserial primary key,
  provider text not null,
  provider_event_id text not null,
  payment_ref text not null,
  event_type text not null,
  occurred_at timestamptz not null,
  raw_payload jsonb not null,
  received_at timestamptz not null default now(),
  status text not null default 'pending',
  attempts integer not null default 0,
  last_error text,
  unique (provider, provider_event_id)
);

Make the insert and queue hand-off atomic. A database-backed work queue can use the new row itself. If a separate broker is required, use an outbox record in the same transaction and relay it afterward. "Insert, publish, then return" has a gap if the process dies after the insert but before the broker publish.

On a duplicate unique-key conflict, return success. The provider has done what you asked; repeating the side effect will not improve delivery.

Production webhook inbox pattern. Provider-specific contracts are based on Stripe, Checkout.com and Adyen documentation linked in this article. Visual: SultanByte editorial artwork.

Idempotency needs two locks

The unique inbox constraint prevents the same event from being inserted twice. It does not stop two different events for one payment from racing.

Use a second guard when applying business state. Lock the local payment row or use an optimistic version check, then evaluate the transition against the current state:

await db.transaction(async (tx) => {
  const payment = await tx.payment.lockForUpdate(event.paymentRef);

  if (!canApply(payment.status, event.eventType, event.occurredAt)) {
    await tx.inbox.markIgnored(event.eventId, "stale-or-invalid-transition");
    return;
  }

  await tx.payment.apply(event);
  await tx.inbox.markProcessed(event.eventId);
});

Define transitions instead of assigning whatever status arrived last. A capture can move an authorised payment forward. A later refund can move it to refunded. A delayed authorisation event should not move that payment back to authorised.

Timestamps help, but they are not the whole rule. Adyen tells receivers to check event timestamps and, for some webhook types, sequence numbers. It also says duplicate events with the same eventCode and pspReference can have different dates, and advises using details from the latest event. Checkout.com explicitly says webhook order may vary. Your adapter should expose the provider's ordering evidence, while the domain layer decides whether the transition is valid.

If the sequence is ambiguous or an event skips an expected state, fetch the current payment from the provider API before changing fulfilment. That extra read is slower, which is another reason to keep it out of the acknowledgement path.

Keep money and fulfilment behind separate gates

A durable inbox reduces duplicate work, but side effects need their own idempotency keys.

For each external action, store a business operation key such as:

ship-order:{order_id}
issue-invoice:{payment_id}:{capture_id}
send-receipt:{payment_id}:{successful_version}

Enforce each key with a unique constraint or an idempotent downstream API. The webhook event ID is usually too narrow: two legitimate provider events may describe one business action, and a replay tool may use a new delivery identity.

Do not make "webhook processed" synonymous with "order fulfilled". Mark the inbox event processed after its transaction commits. Track fulfilment, invoicing and notifications separately so one failed email does not cause the payment mutation to run again.

Reconciliation is part of the design

A webhook system without reconciliation trusts a delivery channel to be perfect forever.

Run a scheduled job that compares local records with provider data for a bounded window. Look for:

• remote payments with no matching inbox event;
• local payments stuck in a transitional state;
• amounts or currencies that disagree;
• events that exhausted processing retries;
• fulfilment actions missing after a confirmed payment.

Stripe's undelivered-events guide shows why this matters. Manual recovery can run while Stripe is still retrying, so Stripe recommends marking events as processing or processed and returning success when an already processed event is delivered again. Checkout.com allows past webhooks to be resent through its Dashboard and API. Recovery traffic must pass through the same inbox and state guards as live traffic.

Reconciliation should repair state through normal commands, not direct SQL updates. That keeps audit records, notifications and business invariants consistent.

What to measure in production

Monitor the boundary, not only the queue depth:

• signature failures by provider and endpoint;
• time from receipt to durable insert;
• acknowledgement latency and non-2xx rate;
• duplicate insert rate;
• age of the oldest pending event;
• stale or rejected state transitions;
• reconciliation mismatches and repair outcomes.

Do not put full webhook payloads, card data, secrets or signature headers into ordinary application logs. Log event identity, payment reference, adapter version, verification result, processing status and a redacted error code. Keep the immutable payload in a restricted store with a retention period tied to operational and compliance needs.

Test the failure paths deliberately. Send the same fixture twice. Reverse two events. Kill the worker after it starts a transaction. Make the provider API time out during ambiguity resolution. Rotate a signing secret. Run reconciliation while delivery retries are still arriving.

The Standard Webhooks specification recommends authenticating the payload together with a timestamp and unique identifier. Payment providers do not expose one universal format, but the receiver architecture can still be consistent: provider-specific verification at the edge, one durable inbox, guarded domain transitions and a separate reconciliation loop.

That design is less clever than a route handler that updates an order in one pass. It is also far easier to explain when a customer says they paid, the provider agrees, and your database does not.