How Not to Build a Ledger
Part 2: Race Conditions, Throughput Cliffs, and the Hidden Risks of Financial Software
Race conditions and throughput cliffs are the hidden dragons of fintech systems. Learn how to design ledgers that stay both correct and fast.
In finance, a few milliseconds can decide whether a transaction clears, duplicates, or conjures money from nothing. The systems moving those funds are made of code, but it’s helpful to think of them as economic machinery running at machine speed. And like any complex machine, ledgers can fail in unanticipated ways. So let’s anticipate those failures, to strengthen your ledger.
Two of the most dangerous and least understood failure modes are race conditions and throughput cliffs. One threatens correctness; the other, capacity. Together, they can turn well-intentioned payment platforms into financial black holes. At Formance, we call them dragons. And it’s necessary to uncover and slay them to build great financial software.
The Race Condition: When Time Becomes Your Enemy
A race condition happens when two operations touch the same balance at nearly the same time, and the final result depends on which one “wins.” In a ledger, that usually means both actions read the same state before updating it, breaking the rule that every dollar (or coin, or token) can move only once.
For example, imagine two tellers checking your account and seeing that you have a balance of $10. On this basis, both tellers approve a $9 withdrawal simultaneously. Each believes there’s enough in your account. Both transactions complete successfully. You now owe $8.
This scenario is going to be very common in a high-throughput ledger, and has led to real-world problems. In 2015, a researcher exploited a flaw in Starbucks gift cards by simultaneously sending two balance-transfer requests in rapid succession. Each saw the same starting amount and duplicated credit before reconciliation could catch up.
As financial systems grow faster and more connected, these timing errors can create or destroy real money in milliseconds.
How Race Conditions Emerge in Modern Fintech
Race conditions don’t usually appear as obvious coding mistakes. They hide in the seams between services, where small timing gaps or retries overlap. In modern financial systems built on APIs and microservices, that overlap is constant.
Common patterns include:
- Split operations: Deducting a wallet balance and updating a transaction record in separate queries creates a gap where one succeeds and the other fails.
- Retries without idempotency: A timeout triggers a duplicate charge because the same request is processed twice.
- Weak transaction IDs: Timestamp-based identifiers collide under heavy load, overwriting previous records.
- Cache misuse: Storing balances in Redis or memory for speed creates inconsistencies when caches are not invalidated.
Each seems minor in isolation, yet together they erode the guarantees financial systems depend on: conservation, uniqueness, and order.
Designing Against the Dragon
Preventing race conditions starts with intent. Every system that moves money, such as those used at an NBFI, should have an explicit concurrency model that describes what can run in parallel and what must be serialized. When that intent is missing, unpredictable behavior fills the gap.
A few proven design principles anchor reliable systems:
- Atomic transactions: Treat every transfer as a single database transaction. Either all operations succeed together, or none do.
- Conditional updates: Let the database enforce limits, such as “update only if balance ≥ amount.”
- Idempotency keys: Ensure that retrying a payment or API call produces the same result instead of duplicating it.
- Scoped locks: Serialize actions per account or service, not across the entire system. This preserves safety without freezing throughput.
- Unique identifiers: Use UUID or ULID schemes to prevent accidental collision or overwriting.
- Database constraints: Block negative balances and out-of-order operations at the schema level.
The goal is simple: Make correctness automatic and irreversible. If every path through the system respects those constraints, timing no longer decides the outcome.
The Throughput Cliff: When Scale Turns on You
If race conditions threaten correctness, throughput cliffs threaten capacity. A throughput cliff appears when performance suddenly collapses instead of degrading smoothly. At light load, everything runs fine…then one sudden increase in traffic causes lock contention, retry storms, and latency that spirals out of control.
The danger is that the very tools used to prevent race conditions, such as locks, coordination, and synchronization, can also choke performance. If every transaction waits for every other, the system becomes safe but potentially unusable.
In high-volume environments, this collapse can cause a cascade, leading to payment queues backing up, card authorizations timing out, or cutoff windows being missed. Each of these failures triggers a retry, adding more traffic to an already strained system until response times explode.
Avoiding a throughput cliff demands thoughtful architecture that limits coordination to only what is necessary, allowing thousands of transactions to proceed safely in parallel without sacrificing accuracy or speed.
The Balancing Act: Correctness vs. Performance
Designing financial systems means living with a permanent tension between safety and speed. Each layer of protection adds friction. Yet removing them risks breaking the most fundamental invariant of all: Money cannot simply appear or disappear.
The art is in applying control only where it matters. Service-specific locking allows a user to pay for a ride and top up a wallet simultaneously, while ensuring that two wallet withdrawals do not overlap. A two-phase commit pattern, where funds are first reserved and then confirmed, delivers atomicity without blocking every operation in line.
At the database level, row-level locking and conditional updates let concurrent transactions proceed safely. Monitoring completes the picture: measuring lock contention, transaction latency, and negative balance alerts in real time prevents minor issues from snowballing.
“Here Be Dragons”: Why Formance Builds for the Unknown
On old maps, unexplored regions were marked with the warning Here be dragons. It was a signal to navigators: What lies beyond is unpredictable, and possibly dangerous.
Building your own ledger from scratch is much the same. The territory looks familiar at first, with rows, balances, and transactions, but the hazards are ultimately hidden in the timing.
Sometimes, people believe a ledger is “just a database.” In reality, it’s a distributed, real-time accounting engine where every write must reconcile across systems that move faster than humans can review. That’s where dragons live: In the race conditions that slip through concurrency models, and in the throughput cliffs that appear without warning at scale.
Formance exists to make those dangers predictable. Its ledger architecture was built by engineers who have already fought and contained these problems. The goal is not blind trust, but transparency, helping teams see the tradeoffs, constraints, and patterns that keep value flows safe, no matter how complex the system becomes.
