Why crypto exchange architecture breaks under peak loads
The exchange looks stable until the first spike. In calm mode, the trading core matches orders faster than participants send them, and the gap between intake and confirmation is imperceptible.
Peak load on the exchange — a listing, a sharp price move, a liquidation cascade — increases the flow by an order of magnitude: the input queue grows, order execution latency goes into seconds, and some orders arrive at a stale price.
Crypto exchange architecture does not break at a single point: a bottleneck in the trading core is almost always connected to adjacent contours.
The order book is held in one thread — only this guarantees a deterministic execution order — and that same thread becomes a serialization point. Anything that falls into the same cycle: funds checks, reserves, trade recording — starts slowing it down.
The second line of failures is the settlement contour and market data dissemination. While the balance check is synchronous with execution, every order waits for write confirmation, and throughput hits storage.
Subscriptions to order book updates compete for the same resources as trading: overflow of the dissemination queue adds latency for all participants.
Core failure at peak rarely remains local. If service permissions are not separated, analytics and the admin panel read the same tables as the trading core, and at peak they compete with it for locks — the cascade reaches order cancellations.
Below we analyze how this is closed: by separating the hot path and auxiliary contours, asynchronous writes, resource isolation, and controlled degradation instead of complete failure.
Effect of rebuilding the core: speed, resilience, failure modes
In the redesigned trading core, the order book is not recalculated entirely on every event — only deltas are published, and reads are separated from writes. Because of this, trading core performance grows with the number of worker threads instead of hitting one lock.
On historical snapshots, peak load is sustained without response latency growth: the order queue stopped accumulating by the end of the active session.
Exchange platform resilience now relies on a deterministic event log: state is raised by replay from the last snapshot plus replica catch-up.
Recovery time after failure dropped from tens of minutes to units — the standby node starts with an already warm order book and without manual balance reconciliation.
At the same time, a whole class of failures related to desynchronization between trading settlement and funds accounting disappeared.
Idempotent operations and quorum writes to the log do not allow a trade to be executed twice after failover, and degradation under load became controlled: the queue is bounded and slows intake instead of collapsing.
Balances and reference data are stored in PostgreSQL, hot snapshots live in memory, so exports and reporting do not slow trading. Platform downtime during active hours is measured in seconds and planned in advance, rather than happening unexpectedly.
Crypto exchange architecture options: single core, cluster, hybrid
A single core — one active node that sequentially applies orders: the queue is predictable, audit is simple, execution logic lives in one place.
The price for this is that throughput hits one thread, and process restart halts the entire market and opens a window for order reconciliation after failure.
A cluster scheme spreads instruments across independent groups and removes the load ceiling, but adds inter-group operations: funds transfers between accounts, cross-margin positions, consolidated order book.
They have to be coordinated separately — and this is exactly where balance discrepancies most often surface, which reconciliation later catches.
A hybrid core separates two contours: trading, where response time is valuable, and settlement, where balance state is valuable.
Trades match in the fast contour, while debits and credits go asynchronously through the transaction log — discrepancy is closed by reconciliation, not by halting trading. The cost of the solution is consistency of the two contours and a higher support entry threshold.
| Option | When it fits | Limitations |
|---|---|---|
| Single core | Spot trading, one or two dozen instruments, few linked accounts | Only vertical scaling; node failure halts trading |
| Cluster with partitioning by instrument | Many instruments and uneven load, need headroom for peaks | Inter-group operations and consolidated order book are more expensive, reconciliation is harder, more failure points |
| Hybrid: trading contour plus settlement | High order frequency, linked accounts, strict reconciliation requirements | Need a transaction log with repeatable processing, coordinating two contours is more expensive |
It makes sense to compare schemes by the number of instruments, share of linked accounts, and reconciliation requirements — not by peak load in isolation from everything else.
Implementation stages: from load audit to handover to operations
We build implementation stages so that every architectural decision relies on a measured load profile, not assumptions. First a load audit, then a fixed order flow diagram, and only after that — core configuration and load testing. Handover to operations is a separate stage, with documentation and on-call runbooks.
- Load audit. We capture the actual profile: peak order flow, read/write ratio, share of cancellations and amendments, order book depth by instrument. Here we also fix target metrics and growth points so as not to design blindly.
- Order flow diagram. We lay out the path from entry point to confirmation: validation, limit checks, matching, balance recalculation, market data publication. The diagram shows where a queue is appropriate and where a synchronous response is needed.
- Core design and configuration. We separate matching by instrument, configure queues, priority rules, and system behavior under overload. Separately determine which nodes can be scaled horizontally and which are stateful.
- Load testing. We replay the real traffic profile and peaks above the calculated ones, look at 99th percentile latency, stability under prolonged load, and behavior when a separate node fails. Discrepancies return us to the previous step.
- Security and fault-tolerance check. We delineate access, configure key storage and rotation, and rehearse recovery from replicas. We fix the incident response runbook.
- Handover to operations. We provide documentation, dashboards, alerts, and instructions for the on-call shift, conduct training, and remain in touch during the stabilization period.
Deliverable package: diagrams, configurations, runbooks, documentation
Crypto exchange architecture work ends not with a demo, but with a package of artifacts that the operations team can work with without us. Below is exactly what remains in your hands and at what level of detail.
- Integration diagram. How the order gateway, matching core, balance service, risk control, and withdrawal are connected: data flows, failure points, service responsibility boundaries.
- Service configurations. Configuration files with a description of each parameter — queue limits, timeouts, connection pools, retries — and separate values for development, pre-production, and production environments.
- Access policies. Roles and permission scopes, separation of duties for administrator, operator, and auditor, key rotation procedure, and cold storage handling rules.
- Deployment runbook. Step-by-step release, migration sequence, rollback procedure, and checklist after restart.
- Monitoring and alerts. Metrics: order processing latency, order book depth, balance discrepancy, failure rate. For each — a threshold at which the on-call engineer receives a notification.
- Recovery plan. Target time and recovery point, failover to the standby contour, replica reconciliation, and processing of the accumulated queue.
- Operational documentation. Description of software interfaces, limits and error codes, instructions for the on-call shift, architecture decision log with rationale for trade-offs.
Each document is tied to a specific service and versioned alongside code — otherwise the package quickly turns into an archive of outdated diagrams.
Case study: how the trading core stopped losing orders at peaks
The client is a spot venue with its own order book. In calm mode, the trading core sustained load, but during sharp market moves the input event queue fell apart: some orders did not reach the order book, and trades executed at a price worse than stated.
There were three causes: synchronous log writes in the matching thread, a single thread for all instruments, and no deduplication on repeated event delivery.
What was changed in the core
The hot path was freed from everything except matching: events go through a ring buffer, the log is written by a separate thread in batches, and instruments are separated into independent threads.
Log replay was added — the order book state is raised to the last confirmed record, so node failure no longer requires manual reconciliation.
Switchover without halting trading
The new core was launched in shadow mode: it received the same events in parallel with the old one, and discrepancy for each instrument was calculated automatically. We switched one market at a time, with instant rollback to the previous core on deviation. Numbers before and after were captured from the same dashboards in Grafana:
| Metric | Before | After |
|---|---|---|
| Orders that did not reach the order book at peaks | 0.3% | zero |
| Order processing time at peak | seconds | fractions of a second |
| Support requests about discrepancies | dozens per day | isolated |
| Rollback to previous core | planned window | seconds |
Recovery was tested separately: the node comes up from the log and catches up with the event stream without halting trading.
How does the trading core differ from the settlement module and what should be changed first?
The trading core holds the order book and matches orders in memory, without knowing about user funds. The settlement module credits and debits funds, holds locks for open orders, and maintains transaction history.
The boundary between them is the first thing checked during a crypto exchange architecture audit: if the core itself adjusts balances, any matching delay turns into an accounting discrepancy.
Starting changes with the core is almost always a mistake. It is the only place where failure is immediately visible to all market participants, and rollback requires halting trading.
First fix the invariants: deterministic processing of the same order sequence, end-to-end event sequencing, and cancellation idempotency.
The settlement module is changed first — it is isolated, can be duplicated, replayed on historical data, and reconciled line by line.
The key element here is balance reconciliation: user account funds must reconcile with balances in hot and cold wallets and the volume of open orders. A discrepancy is not a reason to panic, but a signal that an event was lost or applied twice.
The order of changes is built backward: first observability — event log, queue metrics and response latency, ability to replay a trading day. Then the settlement module, then the matching core isolated from external calls, and only after that — market data access.
The readiness criterion is simple: you enable the shadow core, compare its output with production, and explain every discrepancy. Until that exists, it is too early to change trading logic. We analyze how such an order maps to your contour.
Technical concerns: access, audit, zero-downtime migration
Before the start, the client wants to understand three things: who has access to keys and balances, what remains in the logs after each action, and whether it is possible to migrate to a new contour without halting trading. Below — how these questions are closed architecturally.
The answers are not "trust us," but reproducible procedures: roles are described by configuration, the runbook — by document, migration — by rehearsal on a copy of data. Everything is tested on staging before production rollout.
How are service permissions separated?
Each service has its own account and only the permissions needed for its task. Only the signing service has access to key storage; others contact it on request. Granting extended permissions goes through second engineer approval — no one can withdraw funds or change limits alone.
Where are keys stored and how is the action log maintained?
Keys do not get into code, configs, or logs: they are stored in secure storage, and the hot contour is limited by a withdrawal limit.
Every action — login, permission change, transaction signing — is written to an immutable log with timestamp and initiator; anomalies raise an alert to the on-call engineer. Security audit is performed before launch and after changes in the signing contour.
Is zero-downtime trading migration realistic?
Yes. The new contour is raised alongside, data is continuously replicated, part of traffic goes to it in shadow mode and is reconciled by results. Switchover takes seconds, and a rollback path remains: if metrics diverge, traffic returns to the old contour without losing orders.
Let's discuss the task: what we need from you and where we start
So that the timeline estimate and work plan rely on facts, not guesses, we need a snapshot of the current state and target requirements.
Below is the input data we request at the first discussion of the task: some is already described in your documentation, some we reconstruct together.
- How the trading contour is structured now: monolith or separate services, where orders, balances, and trade history live.
- Order model and order types to support: limit, market, stop, conditional, margin — the matching core and risk calculation depend on this.
- Target load profile: number of accounts and trading pairs, peak spikes during volatility, latency requirements for order placement and cancellation.
- Fault-tolerance requirements and allowable downtime: what is more critical — trading continuity or balance integrity during an incident.
- Regulatory contour: what checks will need to be passed, what reporting and transaction history to store.
- External integrations: deposits and withdrawals, quote providers, external liquidity venues, custodial key storage.
- Responsibility boundaries and access: what we cover, what remains with your engineers, which repositories and staging environments need access, whether there are time-to-market constraints.
After reviewing the input data, we return with a phased work plan, timeline estimate, and start sequence — what we do first, what can run in parallel. Describe the task in its current form, even if some data is not yet collected: we will close gaps with clarifying questions.