Matching engine development: where it breaks under load
A self-built or third-party trading core holds up only until the first real peak. In a calm market, everything adds up: the order book in memory, the event log, the stream of confirmations to clients.
But peak load arrives on the exchange engine—a volatile pair, a flurry of cancellations, an avalanche of stop orders—and order processing time grows by orders of magnitude, while the event queue gets stuck: new orders wait while the core finishes calculating the previous batch.
The first characteristic failure is order book desynchronization. The core updates the book and sends it to clients through two different paths, and if a failure occurs in the middle of a batch, the state snapshot drifts away from the event stream.
Recovery proceeds by replaying the log, and all this time the liquidity provider and arbitrageur see different market depth—trades are executed at a price that no longer exists.
The second is duplicate orders. Delivery of confirmations with retries during a connection loss leads to the same order passing through the core twice: filled volume doubles, the client's position drifts, and investigation is performed manually using the log.
Both failures come from the same architectural mistake: the core lacks a deterministic event order and a single owner of the order book.
As soon as processing is parallelized “by eye” and locks are taken pointwise, behavior under peak trading becomes unpredictable—this is a question of design, not capacity headroom. We look at how matching engine development is structured to withstand peaks by design.
What a well-designed core delivers: throughput and latency
We measure not “it became faster” but a fixed set of values on the same load profile: order flow, queue depth, share of rejected orders, time to return to operating mode after a failure.
The set is captured before and after core rework—otherwise the performance discussion remains a matter of taste.
- Trading core (matching engine) throughput — tens of thousands of orders per second per instrument; adding a node increases throughput rather than redistributing the same queue.
- Order response time — p50 and p99 stay within the agreed range when the average flow is exceeded tenfold: the queue grows, not the tail of the distribution.
- Eliminated failure classes — races when matching opposite orders, order loss when a client reconnects, duplicates after resending, order book hangs during partial fills. Each class is covered by a separate check, not by a promise to “write carefully”.
- Resilience under peak trading — at the open and during sharp moves, the system enters predictable degradation: queue limits, priorities by order type, an explicit error instead of silent order loss.
- Recovery and observability — order book and position state reconcile from the operations log in minutes, and stage latency (intake, validation, matching, publication) is visible on the monitoring dashboard.
Acceptance testing is one scenario on the old and new core, comparing distributions rather than averages: averages hide the tails that drive traders to a competitor. We run load tests in k6 before release, not after the first incident.
Matching engine: core design options
There are few options for core design: an off-the-shelf solution, modification of an existing engine, in-house development, and a hybrid where matching is in-house while the surrounding modules are off-the-shelf.
The difference shows up at the third asset class, when the execution model and event log requirements change.
| Option | When it fits | Limitations |
|---|---|---|
| Off-the-shelf core, including open source | Standard orders, one asset class, fast launch | Matching model is hardcoded: priority and partial fills change only by editing source code |
| Modification of an existing engine | The system works, and you need to add instruments and reporting | Technical debt, poor documentation, regressions on every change, vendor lock-in |
| In-house trading core development | Non-standard execution order, order audit, many asset classes | Longer time to market, a team needed for development and maintenance |
| Hybrid: in-house core and off-the-shelf modules | You need control over matching, but not over gateways and clearing | Component seams: version desynchronization, double validation, boundary debugging |
The choice is determined not by the stack but by the venue's rules: order priority, order types, partial fills, trade audit, and reporting scope.
If the product's value lies in the execution rules themselves, an off-the-shelf core quickly hits the limits of its model, and deviating from it means rewriting someone else's code.
Modifying an existing engine costs less upfront, but the limitations come from within: incomplete test coverage, weak documentation, dependence on a single team.
We start with the order protocol and load profile, run each option against them, and show where it will start to break.
Implementation stages: from audit to handover
Before we start, we map the current environment: where orders come from, where position and balance are calculated, which external systems read the trade result.
Without this map, core rework turns into blind editing—a discrepancy in a single executed order costs more than the entire design stage.
Then we proceed sequentially, without skipping load testing: the audit produces a flow diagram, the diagram sets the boundaries of changes, and the changes are tested at extreme modes.
Each stage is closed with an artifact that can be reviewed and challenged before it enters the production environment.
- Trading system audit: we build a map of order and settlement flows, measure current latencies and failure points, and record discrepancies between the core and the accounting environment.
- Core design: we define the order book model, matching rules, priorities, and partial-fill behavior—as a specification, not verbal agreements.
- Interface agreement: we describe message formats, error codes, and the recovery scenario after a connection loss with the external environment.
- Implementation and configuration: we build the core for the target configuration, set up queues, logging, and limits, and stand up a test environment next to production.
- Core load testing: we run peak profiles—order bursts, mass cancellations, liquidity taking—and look at the worst case and latency tail, not the average.
- Project handover to the customer: we provide documentation, diagrams, deployment and rollback procedures, train the team, and stay in touch during the first releases.
After handover, the customer's team runs the system on its own: configuration changes, adding instruments, incident investigation. We get involved selectively—when the external environment changes or load grows.
Deliverables: configurations, diagrams, procedures
After completion, the customer receives not only a working environment but also a set of artifacts that can be used to develop it, accept it into operation, and hand it over to another team without rebuilding context from scratch.
Below is the scope of delivery for each environment: from core configurations to exchange operations procedures.
- Matching engine core configurations: order types, matching rules, limits, and trading modes—as versioned files, not edits “on the server”.
- Order and trade flow diagrams: the path from intake to confirmation, failure points, and service responsibility zones.
- Access policies and roles: a “role → action → resource” matrix, service accounts, and the procedure for granting and revoking permissions.
- Trading engine documentation: interface descriptions, message formats, error codes, logs, and metrics.
- Exchange operations procedures: scheduled maintenance, degradation, failover to standby, and on-call actions during an incident.
- Migration and rollback procedure: how a new configuration is rolled out and how the previous one is restored without losing trade history.
Handover is closed by a verifiable criterion: an engineer who did not participate in the project uses this documentation to bring up the environment and run a test trading session.
Such an acceptance test filters out “checkbox documentation” and makes further maintenance predictable.
Case study: trading core under peak load on settlement days
What was the situation
On settlement days, order flow increased many times over, and the trading core could no longer keep up: the matching queue grew, some orders were rejected by timeout, and the order book diverged from the trade log.
The cause was in the design, not the load—a single shared order book with locks, synchronous database writes on every trade, and a complete absence of an incoming flow limit.
Recovery required halting trading and manually reconciling positions: while it was underway, participants left for competitors. Below is what we changed as part of trading core development and what metrics we achieved as a result.
What we changed
The order book was partitioned by instrument and pinned to a single owner thread per instrument: locks on shared structures disappeared, and order sequence within an instrument became strictly deterministic.
Trade recording moved to an asynchronous write-ahead log, and order book snapshots allow the core to be brought up and state restored without manual reconciliation.
The core was built in Rust—the ownership memory model removes the class of failures on which the trade flow previously stalled.
An explicit flow limit with priorities appeared at intake: cancellations and market orders are processed before others, and excess is cut off predictably rather than by collapse.
Before rollout, the daily profile of settlement days was reproduced in the test environment—by replaying an anonymized order flow.
Metrics before and after
Peak throughput increased sixfold, order processing time dropped from seconds to fractions of a second, and the share of timeout rejections fell from a few percent to zero.
No order book divergences from the log occurred again, position reconciliation runs automatically, and there were no trading halts on settlement days at all throughout the observation period.
How is an exchange engine different from an order queue in a database?
An order queue in a database is a table plus transactions and locks. While the flow is weak, it works: insert, read, debit. Under activity, conflicts arise over the same rows, and execution order begins to depend on the database scheduler rather than trading rules.
Two opposite orders may execute in a different order than the one in which the book registered them—and this is no longer latency but a violated priority.
An exchange engine separates recording and execution. The core keeps the order book in memory and processes events strictly sequentially in a single thread, so order execution order is deterministic: the same input produces the same result on any run.
Trades, book changes, and settlements are recorded in a log—not as a state snapshot, but as a list of events.
Priorities and calculation logic live in the core, not in database queries. Price-time priority, limit and market orders, stop conditions, partial fills, funds reservation for a trade—all are a set of rules applied to the book step by step.
Calculations run in a separate environment: the core only matches orders, while positions, fees, and mutual settlements are calculated afterward and do not slow down matching. Reference implementations of this separation are described by LMAX.
The core survives a failure by replaying the log: events are replayed from the last checkpoint, and the book returns to the same state it had before the failure. Recovery takes seconds because it is reading a sequential file, not scanning indexes and rolling back transactions.
Divergence after recovery is excluded—otherwise the exchange cannot prove the correctness of trades to participants.
If you already have your own database-queue solution, you do not need to rewrite the entire environment.
Usually it is enough to isolate the order matching core and the event log, keeping the familiar interfaces and accounting—then execution order and recovery are covered, while the rest of the system continues to work as before.
Technical concerns: fault tolerance and settlement
The engine is the venue's core: its error is visible not as an interface failure but as a balance discrepancy for all participants at once.
That is why, before starting, the most common questions are about behavior during node failure, integration into a running platform, and the guarantee that every order is accounted for exactly once.
What happens if the engine node fails in the middle of a session?
The primary node continuously streams orders to a hot standby with the same state, and on failure the standby takes over within seconds. Recovery proceeds from the snapshot and log: an order either goes through completely or is not accepted.
Will the engine integrate into an already running platform?
The engine is installed behind an adapter to your order gateway and accounting system and synchronizes balances with yours. We switch over in stages—first one instrument under real load.
How is the correctness of settlements between participants confirmed?
Settlements are deterministic: the same order log produces the same result for all participants. Any trading day can be replayed from the log and reconciled with the accounting system, and a discrepancy on an account blocks clearing until it is investigated.
How are duplicate orders excluded during a connection loss?
Each order carries the participant's idempotency key: a retry with the same key does not create a second record but returns the result of the first. Deduplication is in place both at the gateway and in the core, so a connection loss does not turn into a double trade.
Let's discuss the task: where the project begins
The first conversation is about your trading system, not our tooling.
To come back with an estimate and work plan, we need inputs: which instruments are traded and in which modes, the set of order types and their execution rules, requirements for placement and cancellation time, how funds accounting is arranged, and what should happen to open positions during node failure.
We also discuss the projected load profile separately: it determines which parts of the core need redundancy and which scale horizontally.
After the conversation, you receive a working document you can take both to us and to another team: stage boundaries, the commissioning procedure, and the points where decisions are needed on your side.
- an estimate of the scope of work and a stage-by-stage plan—from order matching to reporting and exports;
- a trading core diagram: where the boundary lies between order matching, funds accounting, and the trade feed;
- the scope of the first stage—starting the core project in a test environment with an order log and peak load replay;
- a list of risks: performance bottlenecks, balance divergence scenarios, and the procedure for rolling back to the previous version;
- requirements for access, environments, and test data needed from you to start without delays;
- the acceptance format: which checks confirm correct order execution and accounting integrity.
Describe the task: what is traded, how funds are calculated, and what requirements exist for reliability and timelines. We will come back with a scope estimate and work plan—and immediately outline where we will start with the core.