Test Hop 1 idempotency with your payout payloads
Clone the Formance Ledger repository on GitHub and run the duplicate-post test locally to see how Hop 1 idempotency behaves against your own payout payloads.
The transactional outbox pattern keeps your financial events in sync with the money you actually moved. The outbox works by writing the event and the ledger posting in a single database transaction, so one cannot exist without the other. Every payments team that sends settlement or payout events to a broker or a webhook endpoint eventually runs into the problem it solves, and most explanations of the pattern stop at the first sentence.
A $485,000 payout posts to the ledger at 02:14, but the broker is down, and the event never leaves the service. The treasury team never learns the money is waiting. Or the reverse: the event goes out, the ledger write fails, and a payout fires against money the ledger never moved.
Either outcome ends the same way: a reconciliation break the team cannot explain, a merchant who did not get paid, and an auditor asking which system is wrong.
Writing the event and the posting in one transaction closes the dual-write gap, but a correct outbox has more moving parts than one transaction. Four design decisions turn that single transaction into a complete outbox, including how to commit the row, how to relay it, how to deduplicate, and how to preserve the record for audit.
Publishing after a commit is a dual-write problem because the ledger write and the broker publish are two independent durable acts with no transaction spanning them. This leaves a crash window where money can move incorrectly or not at all.
Take one marketplace payout of $485,000. A partial failure can happen in two ways, depending on which write succeeded before the crash:
Both failure orders share a root cause: a service other than the record described by the event produced it.
The core ledger is the system of record for balances and has to be the only producer of settlement and payout events. A payout.settled event emitted by the payment service before the ledger commits is a prediction about what the ledger will accept.
The ledger can still reject the write on a balance check.
The outbox pattern makes three promises.
When the ledger commits a transaction, it emits a COMMITTED_TRANSACTION event. The event carries each posting's source account, destination account, amount, and asset.
The ledger publishes these events via HTTP webhooks, so the treasury, notification, and reconciliation consumers subscribe directly to the ledger. The ledger may send the same event more than once, so consumers have to handle duplicates.
A "hop" is one leg of the journey an event takes from one system to the next.
The event does not travel from the payment service straight to the treasury consumer in a single step. It travels in two legs, with the ledger in the middle.
Each leg is a hop:
Both hops are required. If the payment service publishes an outbox event directly to consumers, it predicts what the ledger will do rather than reporting what the ledger did. Any ledger rejection then breaks the system.
Hop 1 has an acceptance record it can trust, which is the ledger's idempotency response. Hop 2 doesn't, so the consumer has to build its own.
Formance is an open-source, programmable core ledger that unifies fiat and digital assets with regulatory-grade traceability. In the two-hop architecture, it acts as the source of truth and the origin of Hop 2.
Four design decisions turn the single-transaction commit into a complete payments outbox. You need to decide how to commit the outbox row alongside the payout intent, whether to use polling or change data capture (CDC) as the relay transport, how to key the consumer inbox and enforce per-account ordering, and how to keep the published event record safe for audit.
Commit the outbox row with the payout intent in one database transaction (Hop 1), then have a background relay process post the row to the ledger with an Idempotency-Key header.
The row carries five columns: event identifier (ID), aggregate account (the merchant), per-account sequence, payload, and created timestamp.
The event ID doubles as the idempotency key for the ledger write and as the payment instruction ID that the consumer inbox keys on in Decision 3.
The payload is a double-entry posting with a source and a destination account.
The source, @merchants:4471:available, holds merchant 4471's available balance. The destination, @merchants:4471:payouts:pending, holds the same merchant's funds pending payout.
For a marketplace settlement payout of $485,000 to merchant 4471, the background relay process posts the following transaction with reference payout:settlement-2026-10-07:4471.
The block below is written in Numscript, Formance's purpose-built language for describing financial transactions:
// MERCHANT_PAYOUT_REQUEST
// Event: move merchant 4471's available funds into pending payout funds
send [USD/2 48500000] (
source = @merchants:4471:available
destination = @merchants:4471:payouts:pending
)
set_tx_meta("event_type", "payout.requested")
set_tx_meta("payout_id", "payout:settlement-2026-10-07:4471")
If the background relay process crashes after the ledger accepts the write but before the relay advances its cursor, the relay re-sends the row with the same Idempotency-Key header.
The ledger recognizes the replayed key and returns the original transaction response with an Idempotency-Hit: true header rather than posting a second $485,000 movement.
Use polling as the default, and switch to CDC only when polling is too slow for a specific consumer. Both options recover after a crash, because the relay records its progress in a separate step from sending the event.
Polling works when the relay reads outbox rows that have not been sent yet, resends them, and then updates a cursor in a separate progress table to mark where it left off. It never changes the event row itself (which Decision 4 forbids).
CDC skips the scan entirely. It tails the database's change log, forwards each committed row as soon as it appears, and records its progress in a separate position file. A crash mid-send resumes from the last recorded log position, and is not resumed from a new scan.
Both options can send the same event twice after a crash, so the consumer (Decision 3) must handle duplicates.
Polling adds a small delay (the polling interval) to every consumer and runs one scan per interval. CDC removes that delay and reads events in the order they were committed, but requires more database configuration.
A scheduled payout batch doesn't care about the extra delay. But a card-authorization service that reacts to ledger events in real time does.
The consumer's inbox is a deduplication table. Key it on the payment instruction ID (the outbox event ID from Decision 1) and do not key the amount or destination.
A second real payout of $485,000 to the same merchant is a different instruction and gets processed. A redelivered copy of the first instruction is the same instruction and gets rejected.
In June 2024, the Co-operative Bank experienced a computer glitch that caused duplicate transactions, pushing some small business owners into overdrafts after previous payments made several weeks earlier were duplicated.
The Co-op incident originated on the sending side, but whichever system is the authoritative applier of an instruction (the sending bank's core, the receiving bank's rail, or a downstream consumer in your own stack), it needs an inbox keyed on the instruction ID.
If any of those layers had rejected the second batch as a replay of the first, the duplicates would have stopped at that boundary instead of posting to customer accounts.
Keying the inbox is half the action, because you must write the processed marker and the impact in a single transaction.
Ordering comes from a per-account sequence number on the outbox row. The consumer stores the last sequence it applied for each account, rejects anything with a lower number, and holds anything that skips a number until the gap is filled or an alarm fires.
The ledger also emits an event when a prior transaction is reversed. Give the reversal a higher sequence number than the posting it reverses. If the reversal arrives first, the consumer holds it in the gap queue instead of refunding money the consumer never received.
The event record is a receipt that money moved. Never change or delete it. Instead, track the relay's progress in a separate cursor table and keep the event record immutable.
Flipping a processed flag or dropping a day's worth of rows destroys the only record that an event was ever emitted. On an outbox carrying money, that is the wrong record to lose.
The usual argument against append-only tables is that they grow forever. The financial argument weighs more: an auditor asking why a payout fired twice needs the row itself, not a flag you overwrote on the second delivery.
The outbox table lives in the payment service's database and holds the emission record. The ledger holds a separate, append-only record of the committed transaction, with its reference, metadata, and tamper-evident history.
An append-only core ledger already has the right shape. Set the outbox event retention equal to the ledger's retention, so "was this event emitted" has an answer for as long as "was this posting made" does.
Reconciliation captures events that slip past the inbox and land on the wrong side of a balance. The reconciliation step runs two checks to catch missed events: per-account sequence continuity owned by the consumer, and an aggregate balance comparison owned by the reconciliation function. Ownership of the broader reconciliation process splits across three teams.
The consumer runs the first check by walking the per-account sequence numbers it has applied and looks for missing or repeated values. A missing number means a dropped event, while a repeated number means the inbox should have absorbed a redelivery.
Because the inbox keys on the instruction ID, a repeated instruction that reached the impact points to a bug in the inbox transaction itself.
The second check is owned by the reconciliation function. It compares aggregate balances per asset between the core ledger and the provider.
In the merchant 4471 payout example, a $485,000 difference on the merchant's pending payout account one morning means either the bank executed a payout the ledger never recorded, or the bank never acted on a posting the ledger shows as sent.
Ownership splits across three teams. The ledger team owns the outbox table and the relay, while the consuming team owns the inbox and the sequence check. The reconciliation function owns the balance check and the exception queue.
Each exception includes the account and sequence number for the discrepancy, so the on-call engineer opens a specific row rather than searching for the problem.
A missed or doubled event shows up as a balance difference on the affected asset before anyone has to trace a single posting.
Formance’s Reconciliation automates comparisons between internal Formance Ledger records and external bank or payment service provider (PSP) statements and supports balance-level checks.
Five outbox checks ensure that the four design decisions implemented run effectively. Run your outbox through the five checks, in the order the money moves, to discover any weak link before the next batch leaves the service.
If any of the five checks above fails, hold the payout batch until the underlying control is in place. Do not ship the release with a retry loop in place of the missing control.
Retries re-send the same event, but they cannot invent an idempotency key the ledger never received or a sequence number the inbox never stored, so the same crash window reopens on the next failure.
If any of these controls are missing in your stack, start with the ledger. A ledger that enforces idempotency, append-only writes, and per-account sequencing turns four of the five checks into storage-layer properties.