Why an HFT algorithm does not deliver results on real data
An HFT algorithm that looks good on historical data most often breaks not in the mathematics but at the junction with the real exchange infrastructure.
In high-frequency trading, the result is determined not by the model itself but by how accurately it sees the market and how quickly its decisions reach the order book. Let us examine three places where the logic stops working.
The first is desynchronization with the quote stream. The strategy makes a decision based on an order book snapshot that is already outdated: by the time the data reaches the handler, the best prices have moved.
Response latency accumulates across the entire chain "receive — parse — calculate — send," and the output is a decision based on a price that no longer exists on the market.
The second is order execution. In a test run, an order usually fills instantly and entirely at the quote price.
In reality, it joins a queue where priority is determined by submission time, part of the volume is not filled, and a limit order remains outstanding after the opportunity is gone. This creates a discrepancy between the calculated and actual entry price.
The third is strategy drawdown when the market regime changes. Parameters selected during a calm period fall apart when volatility rises, the spread widens, and the structure of the event flow changes. The model continues trading by inertia, and the statistical edge disappears.
That is why work on an HFT algorithm begins not with return optimization but with measurements: where exactly time is lost, how the execution queue forms, and under which regimes the strategy degrades. Below we examine how these places are addressed in implementation.
What managed high-frequency trading delivers
The first changes are visible even before real trading: the loop is run on archived quotes and under synthetic load, so bottlenecks are found in advance rather than at the moment of a market spike. After that, the effect accumulates in two directions, and both are measured.
Predictable response time
The key effect is that the spread between average and worst execution is reduced severalfold. The HFT algorithm receives an event, checks limits, and places an order along the same route, without "floating" segments and manual confirmations.
Therefore, system behavior in a calm market and at the moment of a spike differs within a specified range, not by an order of magnitude.
Resilience under peak load and controlled failure class
Load tests set the ceiling: it is clear at what number of simultaneous events the loop begins to lag, and this reserve is built in with a multiple of an ordinary day.
Continuous trading monitoring tracks not only the result but also the discrepancy between actual and expected orders — an anomaly is visible in the first minutes, not from the end-of-day results.
Risk control describes what happens during a failure: a channel break, node failure, or data desynchronization leads not to a series of erroneous orders but to strategy shutdown and correct position closure.
This is the controlled failure class — predictable in consequences and debuggable from event logs, instead of an unnoticed position deviation. A few weeks after launch, it is clear that the resilience of the trading system rests on repeatable procedures, not on luck.
HFT algorithm: execution options and their limitations
The operating mode of the trading logic determines everything else: requirements for market data stream processing, risk control, and how the algorithm places orders.
The same HFT algorithm cannot be switched from market making to price arbitrage without rework — the position holding horizon and the nature of the load on the exchange gateway change.
Choosing the operating mode of the logic
| Option | When it fits | Limitations |
|---|---|---|
| Market making | Liquid instrument, steady order flow | Inventory control required: on sharp price movement quotes are pulled, otherwise position grows |
| Price arbitrage | Two related venues or pairs | The discrepancy persists longer than the order takes; time synchronization and commission accounting are critical |
| Market data stream logic | There is a stable trade flow and order book | Sensitive to subscription gaps and channel latency; a fail-safe auto-stop is needed |
Order type is a separate decision within the mode. Limit orders hold the price but leave execution risk and require cancellation management; market orders close the position immediately but cause slippage on a thin order book.
Compatibility with an existing system loop
An algorithm is rarely deployed into a vacuum: a gateway to the exchange, position accounting, and risk limits are already running alongside.
We separate the logic by data exchange — the algorithm receives the stream and sends orders without touching the internal structures of the existing system.
Before launch we verify instrument formats, risk-control rules, and order cancellation procedures so the old loop does not block the new one.
Implementation stages: from data audit to handover
We structure the work so that no decision enters the live loop until it has been verified on data. The stage order is the same for a new HFT algorithm and for reworking an existing one:
- Trading logic audit — we analyze which market event creates the edge, what data is needed for it, and what we consider a signal error versus normal behavior.
- Data quality check — completeness of instrument history, gaps, duplicate orders, correctness of order book reconstruction. Without this, any historical test yields a false result.
- Infrastructure diagram — we determine where the logic runs, how market data and order-sending flows are separated, and where latency from event to decision is measured.
- Connection setup — access and permissions, test and production loops, position and request-rate limits, behavior on failures and retries.
- Historical and test-loop validation — on a holdout period and on a rolling window.
- Solution handover — code, configurations, documentation, shutdown criteria.
Historical validation: what we consider proof
We run the logic on a period that was not used for parameter tuning and on a rolling window — this shows whether the edge holds when the market character changes.
We separately calculate execution costs and latency: a strategy that survives only under ideal execution does not go into the loop.
Acceptance and solution handover
Acceptance follows pre-agreed criteria: share of profitable periods, drawdown, behavior on connection loss, and on trading session change.
We hand over the source code, loop diagram, start/stop procedures, and failure response procedure — the client team then runs the system independently.
Delivery package: configurations, diagrams, procedures
After handover, the client team is left not with a "working instance on our server" but with a package that lets them deploy, verify, and develop the system on their own.
Everything configured during the project is recorded in the repository and handed over together with access credentials.
Configurations, diagrams, and access
- System configuration: environment files, connection parameters to exchanges and brokers, instrument and volume limits, risk-control settings.
- Data flow diagram — from market feed to order placement, with points where latency and connection loss are possible.
- Access policies: roles, rights to modify the strategy, environment separation, procedure for issuing and revoking keys.
- Set of checks: how to verify that after changes the HFT algorithm behaves the same as on the reference.
- Deployment instructions for a clean environment — from dependency installation to first launch.
Procedures and documentation
- Start and stop procedure: pre-launch checks, sequence of actions, behavior on loss of connection to the exchange.
- Update procedure: how to roll out a new strategy version without stopping trading, and how to roll back to the previous one.
- Algorithm logic documentation: which signals are used, which parameter controls what, and where the acceptable risk boundary lies.
- Incident and known-limitations log, so the next team does not have to rediscover the same causes of failures.
The package is assembled for your loop: if some systems are already running, we integrate with them rather than duplicate them.
Implementation case study: metrics before and after
The client was a trading team running a strategy on several venues.
Before implementation, orders were generated in the same process as the signal logic: at peaks, the share of rejected orders reached 4.2% — entries were lost, and the venue responded with rejections to the excessive request flow. A failure in the market data feed stopped trading for 20–30 minutes, and recovery was manual.
Where the system lost time
The bottleneck turned out to be not in the model but in the surrounding infrastructure. Market data and orders went through one stream with synchronous risk checks: logging latency blocked the entire cycle, and the command queue grew faster than it was processed.
Part of the latency was removed at the network level — stream reception was moved to Solarflare adapters with hardware packet processing.
Signals, risk control, and order sending were separated into independent loops: limits are checked against a precomputed account state and do not block data reception.
The hot path was rewritten without dynamic structures, and order key verification was added — duplicate orders stopped being sent.
| Metric | Before | After |
|---|---|---|
| Share of rejected orders | 4.2% | 0.3% |
| Recovery after failure | 20–30 minutes, manually | about 40 seconds, automatically |
| Manual interventions per week | 17 | 1 |
Three months in operation — the metrics hold, rather than diverging after one lucky week.
The loops are fault-tolerant individually: a market data feed failure no longer brings down order sending, and the account state is restored from the reconciliation log without manual commands.
How does low latency differ from high order frequency?
These are two independent characteristics, and confusing them is expensive. Low latency is how much time passes from a market event to order submission; order frequency is how many orders are sent per unit of time.
A loop can be made very fast but narrow, or wide but with noticeable reaction to an event; the choice is always about priority.
Low latency matters where the result depends on position in the event queue: reaction to a price shift, order cancellation and replacement, arbitrage between venues.
The gain here comes from a short signal path — fewer intermediate layers, pre-established connections, lock-free processing. Every extra step in this path directly eats into the edge, so it is designed first.
Throughput matters in strategies where volume counts: dense order ladders, maintaining many instruments, holding a required share of presence.
The bottleneck becomes not the speed of a single request but the order queue — confirmations arrive in a stream, and an algorithm waiting for each one loses pace. Batch sending, asynchronous response processing, and correct handling of venue frequency limits help here.
That is why the loop is often split: a thin layer for urgent decisions stays close to the venue, while position accounting, risk control, and bulk replacements go into a separate stream and communicate with the first through a queue.
Latency is measured by percentiles, not by the average — rare spikes in the tail cost more than a smooth baseline.
Limitations, risks, and frequently asked questions
The venue sets stricter boundaries than it seems at the prototype stage: orders-per-second limit, minimum price increment, order cancellation rules, and automatic shutdown when going beyond the risk profile.
Some of these rules change without a separate announcement, so the HFT algorithm keeps limits in the venue adapter configuration, not in the logic core. Otherwise the strategy catches order rejections and loses position for no reason.
The second source of risk is an abnormal market: liquidity gap, price jump, trading halt.
Logic designed for a smooth flow behaves unpredictably at such moments, so safeguards are built in advance: pause on price deviation, cancellation of stuck orders, daily loss limit, and position recovery after connection loss.
What happens if the venue changes rules or limits?
Access rules and limits are moved into the adapter configuration — an update does not require rewriting the strategy core. Order rejections and latency growth are visible in monitoring, so parameter substitution is tracked within the first minutes, not at the end of the day.
Who is responsible for a failure of an external data provider?
The provider's liability is almost always limited by a failure disclaimer, so redundancy is on our side: two independent quote channels and stream reconciliation. If the discrepancy exceeds the threshold, the algorithm goes into stop mode instead of trading on corrupted data.
How do you validate the strategy before launching at real volume?
First, a run on historical data with a reconstructed order book, including days with anomalies. Then shadow mode on the live market without placing orders — we compare the algorithm's decisions with actual outcomes. Only after that is the volume limit raised in steps.
Which failure is considered the worst, and what happens then?
Connection loss with an open position: without control, it remains open. That is why the algorithm has a communication heartbeat watchdog timer and an exit plan — cancel orders, close the position at market, stop until manual confirmation.
Let's discuss the task: hypothesis review and work plan
To discuss the task substantively, no presentation is needed — how the strategy works now is enough. The more accurate the input data, the faster the project estimate turns into a plan rather than a list of assumptions.
- Quote and trade logs for several weeks of observations — with timestamps and anomaly labels: liquidity gaps, auctions, trading halts.
- Formalized signal description: which order flow events trigger it, under what conditions it stays silent, and how often it fires.
- Risk-control rules: position and instrument limits, maximum order frequency, behavior on channel break and data discrepancy.
- Venue and broker requirements: available order types, cancellation restrictions, reporting composition, and access rules.
- Target benchmarks: share of successful entries, acceptable drawdown, average position holding time, acceptable response lag.
Then we work in three steps. First, we rebuild the logic on historical data and check whether the result is reproducible across repeated runs. Then we prepare a pilot implementation in observation mode: the algorithm calculates decisions but does not send orders. We move the flow into the real loop only after the calculated and actual results match.
As a result, you receive:
- hypothesis map: where the logic is sensitive to data quality and response latency;
- stage-by-stage work plan with control points and acceptance metrics;
- rollback scenarios and emergency shutdown procedure for the algorithm.
Describe the task — we will return with an estimate and work plan.