CEX Exchange Development: Matching Engine, Security, Infrastructure

Crypto exchange development: where order execution breaks

Crypto exchange development hinges not on the interface but on the behavior of the order matching engine at peak load. While the trade flow is steady, everything looks healthy: orders fill in an instant, and order book depth matches trade history.

As soon as a wave arrives — a listing, a sharp price move, a mass influx of trading bots — the command queue grows faster than the engine can process it, and response time stretches into seconds.

Then a cascade begins. Some orders hang in an intermediate status: the user cannot tell whether it has been executed and submits a duplicate.

Funds have already been debited but not yet confirmed in the accounting system, while the matching credit has gone through — this creates a balance discrepancy that a retry cannot fix.

The cause is usually not the speed of the language but the architecture: a single node handles both matching and balance calculation, debits are written without idempotency, and the engine state lives only in process memory.

After a restart, recovery comes from the event log — on Kafka this is a replayable stream, while on a custom queue it more often results in a set of gaps in the history.

The worst outcome is manual reconciliations: engineers halt trading, reconcile the transaction ledger against balances, and manually close stuck orders. That means hours of downtime, market makers leaving, and a stream of support requests demanding to know where the money is.

Below we break down how this is addressed: separation of matching and settlement, idempotent debits, a deterministic log, and automated reconciliation instead of manual.

What a reworked engine delivers: latency, failures, downtime

Reworking the trading loop begins with separating the hot path — order intake and matching — from the settlement loop that maintains balances, limits, and history.

The hot path keeps the order flow in memory and records only the trade result, while settlement follows and does not block intake.

This is what provides headroom in engine throughput: peak flow is several times higher than average trading flow, and a burst of activity does not turn into an intake queue.

We confirm the headroom not in words but on a test bench with a real trading profile: sustained load at the limit, short bursts above average, and series of order cancellations and replacements.

The engine sustains this profile without increased latency on trade confirmation, and if one node fails, the load is redistributed among the others.

Service recovery time is measured in seconds, not minutes. A hot-standby replica takes over the flow, failover is automatic, state is reconciled from the event log, and incomplete operations are replayed.

Repeating a command does not create a second trade — operations are idempotent, and recording the result and updating the balance are committed consistently.

Entire classes of failures that used to be caught only in production are gone: balance discrepancies after a partial failure, double execution of an order on retry, and stuck orders when connectivity to the exchange gateway drops. Each such case is now handled at the loop level rather than through manual incident triage.

Exchange resilience also means predictable degradation instead of failure: under load, secondary functions are disabled, but order intake and settlement remain available.

Scheduled updates take place without halting trading — the new version is brought up on the standby, traffic is shifted over, and returned after verification. Observability at every stage shows where latency accumulates and which path degrades first.

Approaches to crypto exchange development: custom engine, off-the-shelf, hybrid

The choice between a custom engine, an off-the-shelf solution, and a hybrid is determined not by taste but by how much non-standard logic is in the trading loop and how quickly a launch is needed.

A custom matching engine gives full control over the order queue, order types, and risk rules, but it pushes out timelines: fault tolerance and failure recovery are debugged from scratch, without ready-made groundwork.

An off-the-shelf solution removes that layer, but it locks trading logic within the vendor's capabilities.

A hybrid scheme is a compromise: a standard trading loop is taken off the shelf, and custom services live on top of it — limits, auctions, specific order types. The difference between the options is in the table.

We separately check license restrictions: off-the-shelf platforms are often tied to a jurisdiction or number of instruments, or prohibit modification of the source code, and some vendors provide only a compiled loop with no right to modify it.

This affects the choice more than the list of ready-made features, so the terms are reviewed before the start, not after.

Implementation stages: scheme analysis, configuration, verification, handover

We carry out the work in fixed stages: each is closed with a verifiable artifact, not a status report. The sequence is arranged so that design errors are uncovered before real orders and client funds enter the loop, not after trading launches.

1. Analysis of the current loop. We review what is already working — the order matching engine, balance accounting, gateways to network nodes — and which scenarios must be supported: spot, futures, deposits and withdrawals.
2. Scheme design and definition of load boundaries. We fix throughput for the order flow, target trade confirmation latency, and peak headroom as a multiple of the estimated load.
3. Assembly of loops and service configuration. We deploy separate environments, configure trade log replication, wallet isolation, and access role separation.
4. Load and failure test runs. We test behavior during an order surge, loss of connectivity to a network node, and failure of an individual service — the system must remain in a consistent state.
5. Acceptance checks. We reconcile balance and fee calculations against reference values, test change rollback, and record the results in a protocol.
6. Handover to operations. We hand over configurations, schemes, incident response procedures, and the update process without halting trading.

What you keep: schemes, policies, procedures

The work is considered delivered not at the moment of demonstration but when your team releases changes and handles incidents without our engineers.

Therefore we hand over not only the code but also the descriptions by which it is operated: how orders flow, who has access to what, and in what order a release goes out.

• Order flow diagram — the path from intake at the client gateway to a trade in the matching engine: where the signature and limits are validated, where load is balanced, at what point an order is rejected, and how that is reflected in the status shown to the client.
• Access policies and key management — roles of services and people, least-privilege principle, issuance and rotation of signing keys, access to cold storage, and administrator action log.
• Release procedure — build order, data schema migrations and backward compatibility, update windows, rollback procedure to the previous version, and release stop criteria.
• Exchange documentation — description of services and their contracts, data model, queues and topics, matching and limit settings, and a glossary of metrics and dashboards.
• On-call instructions — troubleshooting typical failures (balance discrepancy, stuck order, matching degradation), recovery points, and escalation procedure.
• Infrastructure description and test profiles — environment configuration, how to bring up a test bench from scratch, and a set of checks for settlement correctness and fault tolerance.

Documents are written during the work, not at the end, so at handover we go through them together with your team and close the gaps — including questions that engineers will have during their first independent release.

Case study: how an exchange stopped losing orders during peak hours

The Arvex platform — a spot exchange — faced the same picture during peak hours: order matching ran in a single write thread, and balance recalculation ran in the same cycle.

During activity spikes in major pairs, orders queued up and some were rejected on timeout — the client saw an error instead of a trade.

Point of failure

At peak, up to 5% of orders were lost, and trade confirmation stretched from a fraction of a second to several seconds. The bottleneck was not hardware but a shared resource that every write operation waited on; a failure in the settlement module halted trading entirely. The platform went down for 30–40 minutes and was brought back up manually.

What was changed and what the result was

We separated the two loops: the order log is written sequentially and serves as the source of truth, while balances are recalculated from it asynchronously — a settlement failure no longer halts trading.

Matching was split by trading pair, inbound flow limiting and priority for position-closing orders were added; settlement nodes were moved to a hot replica with automatic failover.

The share of rejected orders at peak fell to zero, and downtime was reduced from tens of minutes to tens of seconds during failover. Throughput increased sevenfold on the same loop, and a peak in one pair no longer affected neighboring pairs.

A peak-activity profile run became mandatory before release — degradation is caught before deployment, not at the moment trading opens.

How does a custom matching engine differ from an off-the-shelf one?

An off-the-shelf engine is an engine with a fixed market model: order book, price/time priority, a standard set of order types, and ready-made liquidity gateways.

It covers the standard spot scenario, but flexibility ends where non-standard order execution logic is needed — opening auctions, custom partial execution rules, and separate trading halt modes by instrument.

A custom engine is chosen when trading mechanics are part of the product, not an implementation detail. Then order execution logic is designed for the rules of a specific venue: priority queue, stop-order cascade, and risk limits directly at the matching level. The price for this is your own load test runs and full responsibility for maintaining the engine.

In practice, the choice is rarely binary: an off-the-shelf engine is taken as the base, and custom modules are placed around it — risk control, clearing, gateways, reporting.

This scheme lasts until the first modification inside the engine: changing someone else's logic turns into a custom branch that has to be maintained with every update.

Therefore the decision is made not by flexibility but by engine maintenance: who responds to an incident under peak load, how changes are verified before deployment, and how rollback is organized.

If the answer is "the vendor on request," custom order execution logic will not appear quickly, and the venue will remain on standard mechanics.

Technical concerns: resilience, audit, data migration

Exchange security begins not with channel encryption but with a threat model: what exactly we are protecting — client funds, keys of the custody loop, integrity of the order log.

It then determines operator permission separation, the split between hot and cold storage, and which actions require a second signature and which are impossible without manual confirmation.

Code verifiability is not a promise but a mode of operation: review for every change, automated tests on historical data, reproducible artifact builds, and an external audit before the system sees real funds.

Migration of a live exchange goes through a shadow loop: it receives the same order flow and reconciles balances, while the switchover fits into a scheduled maintenance window with a pre-rehearsed return to the previous loop.

What can be checked before work starts?

The data schema, migration plan, failure scenarios, and threat model report. If an external code audit is needed, it is scheduled before going live with real funds, not after.

What should be done if a defect is discovered after launch?

Releases are reversible: changes are rolled out to a share of traffic, and metrics for balance discrepancies and latency are monitored continuously. If a threshold is exceeded, the version is rolled back without losing log data.

Let's discuss the task and come back with a plan and estimate

To come back with a timeline estimate and work plan rather than general statements, we need a minimum of input data. The more specific the description, the more precise the stage boundaries and the fewer clarification rounds before the project starts. Below is what to send and what to discuss in the first review.

• First-phase scenarios: spot, derivatives, margin positions — what we launch immediately and what as a separate stage.
• Liquidity and settlement model: where the order book is filled from, who acts as market maker, and how clearing and asset withdrawals are handled.
• Load profile: expected number of concurrent sessions, peak order flow, and acceptable response time under peak.
• Security loop: access roles, mandatory two-factor authentication, key storage procedure, and requirements for auditing and activity logging.
• Infrastructure constraints: own servers or cloud, data residency jurisdiction, and regulatory reporting requirements.
• What already exists: prototypes, current code, ready-made integrations with payment and custody services, and open interfaces.
• Priorities and horizon: timeline for the first version and features that can be postponed without compromising launch.

The review takes one or two conversations with the engineers who will lead the project. After them, we send a work plan with stages, team composition for each area, and a timeline estimate — with assumptions and risks, not a single number.