Order book development: where the exchange core breaks
An order book is not a table in the UI, but a book of orders that the core rebuilds on every event: placement, cancellation, partial execution. While the flow is steady, the discrepancy is invisible, and the load headroom seems excessive.
It all breaks in the first minutes of a peak: the queue of incoming orders grows faster than the core can process it, and then a cumulative effect kicks in.
Core response latency increases with every second of overload, some orders time out on the client side, and market makers stop seeing current depth and pull their quotes.
The exchange formally still has throughput, but it is spent recalculating stale states rather than on new trades.
Worst of all is the discrepancy between the state of the order book and what the UI shows: the user sees a price that no longer exists, the trade does not go through, and reconciliation has to be done manually against the event log.
Venues fix this by halting trading in the instrument — no longer an engineering loss, but a reputational one.
Core resilience rests on three things: what happens in the hot path of order processing, how the queue and backpressure are designed under overload, and how quickly the book state is restored from a snapshot and log.
Below we break down how this is addressed in order book development — from the data model to testing against a peak load profile.
What a reworked core delivers: measurable effects
A reworked exchange core is measured not in general terms but in values captured from a load-test bench before and after.
We compare throughput in orders per unit of time, queue behavior under peak, and recovery time after node failure — the customer sees the effects in a report, not in words.
Handling peak order bursts
Throughput grew many times over: where the previous core hit its limit and dropped orders, the new scheme sustains a peak several times higher. Ingestion and validation run in parallel, while matching remains strictly sequential — the book does not "drift" under load.
The foundation of the approach, known from the LMAX architecture, has been adapted to the venue's requirements.
The queue is no longer a bottleneck: the hot matching path is separated from logging and distribution of updates to subscribers.
A peak burst no longer raises latency for other participants, and the failure class "order accepted but did not make it into the book" has been eliminated entirely.
Predictable response and transparent book state
Response time under peak is no longer a lottery — the gap between typical and worst-case values has narrowed to single-digit multiples.
The book state at any moment can be restored from the event log, so reconciling a snapshot with history takes minutes, not hours of manual investigation.
Exchange core resilience delivers an effect that cannot be reduced to a single number: venue downtime after node failure is reduced to tens of seconds, and participants usually do not have time to notice the gap. The recovery procedure is rehearsed and tested in drills, rather than written during an incident.
Options for building an order exchange core: comparison
The choice of exchange core defines the scaling boundaries and the set of available trading types. If the engine initially supports only limit and market orders, then an opening auction, stop orders, or iceberg orders cannot be "bolted on" later without a rewrite.
That is why the option is chosen not by speed of the first release, but by the trading modes the venue plans to support.
| Option | When it fits | Limitations |
|---|---|---|
| Venue's own core | Non-standard trading rules, proprietary order types, strict latency requirements | Higher scope of work, requires a team for ongoing support |
| Configuration of an off-the-shelf solution | Standard set of rules, fast launch, limited list of instruments | New modes are limited by vendor capabilities, part of the logic lives in external services |
| Module inside an existing platform | The venue is already operating, a new class of instruments is added | Inherits the platform core's limitations and its load distribution scheme |
Venue's own core
Deterministic execution order, an operations log, and periodic snapshots enable recovery after failure by recalculation rather than reconciliation against exports.
Scaling is handled by sharding by instrument: active pairs move to separate nodes, and you design the state synchronization between them.
Configuring an off-the-shelf engine for your trading rules
An off-the-shelf core covers the standard set: limit, market, stop, and iceberg orders, several settlement modes.
Non-standard features — auctions, priority within a price level, custom cancellation rules — are implemented via a wrapper around the engine, and here it is important to check in advance whether it conflicts with the internal execution order.
Work stages: from load audit to go-live
We work in iterations: first measure, then change, then test on a copy of the production environment. This order enables go-live without halting trading — the exchange continues accepting orders while we prepare the switchover and keep a rollback plan to the previous scheme.
- Load audit. We capture the profile: how many orders per second arrive at peak, what the cancellation share is, how instruments are distributed by activity, and which operations increase response latency. Without these numbers, any change to the execution core becomes guesswork.
- Order flow scheme. We break down the path from ingestion to confirmation: gateway, rights and balance checks, queuing, matching, publication of order book changes, snapshot capture. We record where the flow blocks and what happens when a connection drops.
- Order execution rules. We verify order types, price-time priority, partial execution, behavior under insufficient liquidity, opening and closing auctions. We test every rule against adversarial scenarios, not only the happy path.
- Queue and priority configuration. We separate what must not be lost (trades, cancellations) from what can be batched (order book updates). We configure buffering so that a peak does not turn into a failure.
- Load tests. We reproduce peak scenarios on a copy: we measure the failure share, response time at each segment, and behavior when a consumer falls behind. We run a series with load headroom.
- Switchover and monitoring. We roll it out in stages: part of the flow goes through the new scheme, metrics are compared with the old ones. Then we remove restrictions and run on-call monitoring with rollback ready.
What we verify at each stage
- Consistency of numbers: the load profile matches the actual metrics from the production environment.
- Completeness of the flow scheme: no order is lost when a failure occurs at any segment.
- Correctness of execution rules: prices, volumes, and priorities match expectations.
- Queue resilience: a peak does not cause failure, and consumers catch up without data loss.
- Rollback readiness: returning to the previous scheme takes minutes and does not require halting trading.
Artifacts and documentation: what the customer retains
After handover, the customer retains not only a working service but also a set of documents by which the operations team can run it without us.
The order book lives in constant motion: order flow, matching, depth publication — and any intervention without understanding these relationships ends in desynchronization or a stuck queue. Therefore we hand over:
- Order flow scheme — the path from ingestion to execution and publication: where buffering happens, where the state commit point is, and how the system behaves when an individual node fails.
- Core configuration and execution parameters — all values are externalized into files, with a description of how each parameter affects throughput and response time.
- Access policies and roles — who changes matching parameters, who sees metrics, who releases updates; permissions are separated so that routine work does not require elevated privileges.
- Core documentation — queue design, execution and cancellation rules, behavior during partial volume closure.
- Load test logs — load profiles and run results, so that the test can be repeated after any change and compared with the initial state.
- Maintenance regulations — planned work, data schema changes, updates without halting trading.
Regulations and instructions for the on-call shift
A separate set is for those who come on shift, including the night shift. It contains step-by-step actions for typical situations: queue growth, order book divergence between nodes, delivery channel failure, planned restart.
For each case, it specifies the sign by which the on-call engineer distinguishes normal behavior from degradation, the sequence of actions, and the point at which the task must be handed over to engineers.
This way the shift does not act at random, and the decision is made according to a document, not from a specific person's memory.
Case study: venue throughput before and after
What preceded the rework
This exchange venue case study began with a monolithic core: the order book and participant balances were calculated in one transaction with writes to a relational database, and order matching ran in a single thread.
It worked under planned loads, but during news bursts the command queue grew faster than the core could process it.
Measurements in pre-production confirmed the hypothesis: above 1,200 orders per second, peak response stretched to one and a half seconds, and the share of rejected orders reached 9%. After a restart, the trade feed was reconciled manually — recovery took tens of minutes.
The hot path and accounting were separated. The order book lives in memory, order matching runs sequentially in Matching Core, and trades and balance changes are recorded in a write-only log and asynchronously projected into the database.
A replica picks up the log from the last confirmed record, so after a failure the venue returns to service without manual reconciliation.
Throughput before and after
| Metric | Before | After |
|---|---|---|
| Sustained throughput | 1,200 orders per second | 9,000+ orders per second |
| Peak load response | up to 1.5 seconds | fractions of a second |
| Share of rejected orders at peak | about 9% | less than 0.1% |
| Recovery after failure | tens of minutes, manually | up to a minute, automatically |
How to verify order book resilience under load
Order book resilience is verified by an experiment with a reproducible flow, not by a one-off attempt to "flood it with more orders." We stand up a test environment next to production, record the real event stream, and replay it at a controlled rate — the event log in Kafka provides the same sequence of orders, so the measurement is compared with a benchmark, not with the previous mood.
Scenarios differ in the point at which the system breaks, so each is run separately, and then in mixed mode.
Only repeatability helps distinguish true core degradation from a one-off dip.
We run the same profile several times and see whether tail latencies and queue length hold between runs: if, after the burst is removed, metrics do not return to the baseline, it is drift, not a random outlier.
We verify order book integrity by reconciliation — snapshot plus change stream must yield the same state as the benchmark, on every failure and reconnection.
The output is a set of scenarios and threshold values by which a release is accepted or rejected without arguing about whether we loaded the bench enough.
Technical concerns: resilience, failures, enhancements
Engineering concerns at this stage are usually the same: how much of the core will have to be rewritten, how to migrate live orders, what happens on node failure, and what the customer's team will retain after handover. Below are substantive answers.
Is a full core rewrite needed, or is enhancement enough?
We look at invariants: if the core does not lose orders on restart and correctly handles concurrent changes, we keep it, changing the matching and data publication layer.
A full rewrite is justified when logic is mixed with balances and settlements — then changing one part breaks adjacent parts.
How are open orders migrated?
The book and event log are captured at a consistent point so that no order is processed twice. Before switchover, acceptance of new orders is paused, then the snapshot is loaded into the new core and reconciled by number and volume of positions.
What is the behavior on node failure?
Matching is led by the primary node; the standby receives the change stream and, if the primary is lost, continues from the same log point. The client sees a connection break rather than vanished orders: after reconnection, it receives a current snapshot and delta.
How does the core operate after handover?
We hand over the configuration, data schema, and description of invariants along with a run of load scenarios — the customer's team can reproduce any core state and knows which checks must pass before deployment.
What does post-handover support include?
We analyze incidents using logs and metrics, and handle fixes and expansion: a new order type, an additional market data channel. Handover does not close the work — access to those who designed the core remains.
Let's discuss the task: where to start and what to prepare
Before the conversation, it is useful to gather facts about the current order book — from them we calculate the load profile and find the bottleneck: the matching core, update distribution, or the order book storage layer. This way the discussion proceeds at the level of numbers and metrics, not general wishes.
What is worth preparing before the start:
- Current load indicators: peak and average values for orders per second, order book depth, number of active connections, and market data volume.
- Environment diagram: where matching nodes live, how the order book is stored, how changes are replicated, and how read and write flows are separated.
- Trading rules: order types, auction and halt modes, price-time priority, cancellation and partial execution logic.
- Incidents over recent months: discrepancies between the order book and trade feed, distribution delays, outages, and how they were resolved.
- Expected peak profile: by what factor growth in orders and connections is planned, and which events cause bursts.
- Hosting and access constraints: where deployment is allowed, and what requirements apply to node isolation and operation logging.
After the conversation, we return with an assessment of the load profile, a work plan for the core, and acceptance criteria: which latencies and what processing order we consider normal for the target profile.