Out-of-order webhooks: what the handler must do

Webhook handlers fail in production in a specific way: payment.captured is applied, then an older payment.pending arrives and overwrites it. The customer was charged. Your ledger says they were not.

Arrival order is not causal order. Providers retry, queues reorder, and a slow handler on one event does not hold the next. Stripe documents that endpoints can receive the same event more than once, and that you should make handlers idempotent. The stronger rule, which follows from retries even when a provider never states it, is that you also cannot trust the sequence the requests showed up in.

Verification still comes first. An authentic event in the wrong order is a different bug from a forged one, and both need handling. Signature checks are covered in validating webhook signatures.

What you need

• The provider’s event id, and a timestamp or object version they generated. Your receipt time is not a substitute.
• A unique index on that event id.
• An explicit state machine for the payment. “Last write wins” is not one.

Apply transitions, not payloads

Store the payment as a state, and define which moves are legal.

pending → authorised → captured → settled
                    ↘ failed
         captured → refunded

An event is accepted only when it is a legal move from the state you already have, or when it is a no-op because you have already applied it. An event that would move backwards is stored and ignored.

public enum PaymentState { Unknown, Pending, Authorised, Captured, Failed, Refunded }

public static bool TryApply(PaymentState current, PaymentState incoming, out PaymentState next)
{
    next = current;
    if (incoming == current) return true; // duplicate delivery

    var legal = (current, incoming) switch
    {
        (PaymentState.Unknown, PaymentState.Pending) => true,
        (PaymentState.Unknown or PaymentState.Pending, PaymentState.Authorised) => true,
        (PaymentState.Unknown or PaymentState.Pending or PaymentState.Authorised, PaymentState.Captured) => true,
        (PaymentState.Pending or PaymentState.Authorised, PaymentState.Failed) => true,
        (PaymentState.Captured, PaymentState.Refunded) => true,
        _ => false,
    };

    if (!legal) return false;
    next = incoming;
    return true;
}

Unknown → Captured is legal on purpose. If the captured event arrives before the pending one, you are captured. The late pending event then fails TryApply and must not rewind you.

Do this in the same transaction as the “have I seen this event id” insert. Check the id after you have moved the money, and a crash between the two steps replays the capture.

await using var tx = await db.BeginTransactionAsync(ct);

if (!await db.TryInsertEventAsync(providerEventId, ct))
{
    await tx.CommitAsync(ct);
    return Results.Ok(); // already applied; ack so they stop retrying
}

var payment = await db.LockPaymentAsync(providerPaymentId, ct);
if (!TryApply(payment.State, incoming, out var next))
{
    await db.RecordIgnoredAsync(providerEventId, "backward_transition", ct);
    await tx.CommitAsync(ct);
    return Results.Ok();
}

payment.State = next;
await db.SaveAsync(payment, ct);
await tx.CommitAsync(ct);
return Results.Ok();

Return 2xx for an event you deliberately ignore. A 500 tells the provider to retry, and they will, forever, with the same stale payload.

When a timestamp is the right tool

A state machine fits payments because the object has a lifecycle. It is the wrong tool for a snapshot you are mirroring, such as a customer record, where any newer version should replace any older one.

For those, compare the provider’s created or updated timestamp to the one you stored, and drop the event when theirs is older. Use their clock. Ordering by the time your server received the POST is how a delayed retry beats a newer event that arrived on time.

If the provider has neither a version nor a timestamp, you cannot order the events. Say so in the handler and alert. Do not invent an order from the HTTP log.

What a retry schedule does not tell you

Moyasar’s webhook reference, as checked for the signature guide, retries a non-2xx six times over about two hours. That schedule is about delivery, not about sequence. Six attempts of an old event can still land after a newer event you have already applied. Deduping on event id stops the duplicate. Only the state machine stops the rewind.

The same split applies to Checkout.com’s Cko-Signature flow in the Checkout.com guide: a valid signature proves the body was theirs. It does not prove this is the latest thing that happened to the payment.

Failure modes

Symptom	Cause	Fix
Captured payment flips back to pending	Handler assigned the payload status blindly	Accept only legal transitions
Gateway retries one event all day	Ignored event returned 5xx	Ack it and record why it was ignored
Duplicate fulfilment	Event id checked after the side effect	Insert the event id in the same transaction
Snapshot updated by a stale event	Compared your receipt time, not theirs	Compare the provider timestamp
Two events, no way to order them	Provider has no id and no timestamp	Alert; do not pretend the HTTP log is an order

Production checklist

• Unique event id, inserted in the same transaction as the state change
• Explicit legal transitions, including the skip from unknown to a later state
• 2xx on ignore and on duplicate
• Provider timestamp for snapshots; never your receipt time
• A log of ignored events you can read when a merchant disputes the status
• Signature verification before any of this runs

Sources

• Stripe webhooks — duplicate delivery, and the instruction to make handlers idempotent. Checked 26 September 2026.
• Moyasar webhook reference — retry schedule. Checked 27 July 2026 for the signature guide; the ordering point does not depend on that schedule having changed.

Nothing here is a claim that a particular Gulf gateway reorders events in a specific way. The handler has to be safe if they do.