Why backtesting AI strategies breaks on real data
Backtesting an AI strategy on history almost always shows better returns than subsequent trading. The gap does not arise from the model: a backtest and a live market execute the same order differently.
In history, an order is filled at the bar price; in the market, at the price remaining in the order book after everyone who arrived earlier.
Unaccounted execution costs quietly kill the result. Commission, spread, slippage on volume, and the delay between signal and order submission accumulate on every trade, and a strategy that draws a smooth curve in a backtest goes negative in real trading.
A separate cost line in the report matters more than total return: it shows where the system loses money, not where it promises it.
The third reason is the lack of experiment repeatability. If the data snapshot, instrument universe, feature version, and random states are not fixed, two runs of one strategy produce different curves, and comparing hypotheses loses meaning.
Repeatability here is not about convenience, but about whether the conclusion can be trusted at all.
Usually all this runs into the bottleneck of the computation pipeline: parameter search takes hours, robustness testing to a window shift does not fit into the research cycle, and the divergence between backtest and real trading is discovered only after launch. Let's break down how this is closed at the platform architecture level.
What the platform delivers: measurable effect at every stage
The platform's effect is measured, not described in words: every run leaves metrics that show what changed. Below are four areas where the difference is noticeable immediately after the first month of work.
Time of one run
Caching intermediate calculations and parallel instrument evaluation compress a full portfolio run from hours to tens of minutes. This changes the mode of work: you can run a strategy across several market states on the same day, rather than "by next week."
Reproducibility of results
A run fixes the code version, data snapshot, parameters, generator seed, and environment — a repeated launch produces the same metrics. We keep the experiment log in MLflow, so it is clear which hypothesis produced what and to which code state to roll it back.
Speed of hypothesis testing
A unified data format, ready strategy templates, and auto-reporting shorten the path from idea to number to a few hours. In a day, 10–15 configurations are tested instead of one or two: weak hypotheses are discarded before development is invested in them.
Eliminated failures and overfitting
Moving away from manual exports removes data and metric desynchronization, loss of artifacts, and crashes on long history: a run either completes entirely or fails with diagnostics.
Testing on a holdout sample and by time steps filters out strategies that look good on history and fall apart on live data — robustness metrics are displayed next to return.
Comparison of backtesting modes for AI strategies: selection table
The choice of run mode determines not only speed but also which error classes you will see at all. Batch calculation computes the entire history in one pass over the return matrix — it is suitable for rough hypothesis screening but does not reproduce order execution sequence.
Event-driven calculation runs the tape bar by bar and repeats the real cycle: order, trade, commission, slippage.
Streaming mode keeps the strategy in the same form in which it will go into trading, so only it honestly shows the model's response lag to new data. The backtesting mode is chosen for the task, not by default — see the table below.
| Run mode | When it fits | Mode limitations |
|---|---|---|
| Batch calculation | Initial screening of hundreds of hypotheses and finding patterns in features on long history | Does not account for order queue and partial fills — the result is more optimistic than reality |
| Event-driven calculation | Final check before deployment: commissions, slippage, limits, ban on short positions | Full parameter search on large history takes hours, and run resource cost grows with the number of events |
| Streaming mode | Testing real-time reaction: model response lag, feature drift, data feed interruption | Harder to reproduce: requires a deterministic log of the incoming stream and replay of the same window |
| Hybrid run | You need both broad search and honest evaluation: batch screening, event-driven final | Requires consistent feature definitions in two pipelines, otherwise results do not converge |
How we build the platform: stages of engineering work
We start work not with code but with a data audit: where the price history comes from, how gaps, splits, and dividends are processed, and whether the time intervals of sources match.
Until this is verified, a strategy looks good on history and diverges from real trades — therefore we put the backtesting calculation scheme and metrics in writing in advance.
- Data audit. We reconcile sources, check history completeness, find gaps and duplicates, and bring trading calendars and time zones to a single form. The output is a data suitability report and a list of limitations.
- Calculation scheme. We fix metrics, entry and exit rules, accounting for commissions, execution delays, and slippage — so that calculation does not diverge from real trading.
- Prototype on a narrow slice. We assemble the computation pipeline on one instrument and a short period: the hypothesis is tested quickly, and logic errors are visible before scaling.
- Environment setup. We deploy isolated environments, configure run reproducibility and storage of results together with launch parameters — any calculation can be repeated and compared.
- Testing on control sets. We separate validation data in advance, look at robustness across different periods and market regimes, and discard options that hold up only due to overfitting to history.
- Production pipeline. We schedule runs and add reports and alerts about divergences between expected and actual dynamics — the platform works without manual launches.
- Handover and team training. We provide documentation, access, architecture walkthroughs, and conduct joint runs on your tasks so that your specialists can change strategies and data independently.
What we hand over: configurations, schemes, regulations
We consider a backtesting AI strategies project delivered not when the model has worked on a demo set, but when your team reproduces any run without us.
Therefore we hand over not a presentation with charts, but a working pipeline: configurations, calculation schemes, access policies, and regulations by which the platform continues to live.
- Platform configurations — a versioned set of parameters: price sources, rebalancing windows, commissions and slippage, position size limits.
- Calculation schemes — the pipeline from data loading to metrics: where return, drawdown, and turnover are calculated, and how look-ahead bias is excluded.
- Feature specification — which series and derived indicators are fed into the model, and how weight versions are fixed during retraining.
- Access policies — participant roles, data rights, permission to launch runs and edit configurations, action log.
- Documentation — architecture, local launch, breakdown of typical failures with symptoms and procedure.
- Maintenance regulations — retraining frequency, monitoring divergence between test and production results, rollback to the previous strategy version.
- Acceptance runs — reference scenarios with fixed results: after your edits, it is visible that calculation has not diverged.
We go through the pipeline together with your team: the engineer walks through each artifact and answers questions about calculations and access. Describe the task — we will come back with an estimate and a plan.
Case: from scattered scripts to a unified computation pipeline
Before: scattered scripts
Before the migration, calculation lived in dozens of separate Jupyter notebooks and local scripts: each strategy had its own set of historical data and its own intermediate exports.
A run of one parameter set took about six hours and did not fit into the working window, so testing was launched at night, and edits were made afterward.
The result depended on the environment: two copies of one script on different machines produced different returns, and up to a third of the numbers had to be rechecked manually.
After: a unified computation pipeline
We assembled a unified backtesting computation pipeline: one historical data source with versioned snapshots, immutable environments for each launch, and a parameter log that unambiguously restores any run.
Heavy calculations were split into independent tasks and parallelized, and results are stored in a common repository together with the data snapshot and code version.
Run time decreased from six hours to forty minutes — a hypothesis is tested several times a day, not once a day.
The share of reproducible results grew from 70 to 99%: the same parameter set on the same snapshot produces the same outcome, and divergences are explained by new market data, not by the environment.
The number of disputed results that had to be investigated manually fell by almost five times — strategies are compared by common metrics, not by correspondence.
What to do if part of the calculations is already automated
An automated computation pipeline is not an obstacle but a foothold: usually it already has verified data preparation rules, historical price arrays, and order execution logic.
We start with an inventory — we record data sources, storage formats, run schedules, and which metrics the team considers reference. After that, for each block we decide separately: migrate as is, wrap in a thin compatibility layer, or leave in place and only connect to it.
Existing work is preserved: indicators, filtering rules, processing of gaps and corporate events move into the new pipeline without rewriting the logic.
We bring them to a unified interface so that the AI backtesting engine calls them the same way as its own modules — otherwise comparison of old and new results loses meaning.
Seamless integration is verified by a parallel run: the old and new pipelines calculate the same historical segment, and we compare the return curve, maximum drawdown, number of trades, and distribution of results by period.
A divergence above the agreed threshold stops the switch — first we find the cause. In practice, this is most often different gap filling, the moment of commission accounting, or a mismatch in overnight position rollover rules.
The migration goes strategy by strategy, not all at once: until the last step, there remains the ability to return to the previous calculation and compare both pictures side by side.
We maintain reproducibility through versioning of historical data and a log of all run parameters — a month later, any past result can be retrieved and explained without archaeology through scripts.
Technical doubts engineers have before starting
Before the first launch, an engineer asks the same questions: is there enough history, what guarantees that the calculation is correct, and will there be a beautiful curve on past data that falls apart on the live market.
We resolve these doubts before development — here an error in calculation costs not a bug, but a wrong decision to launch a strategy.
Below are short answers to technical objections that most often come up during the evaluation stage.
Will history have to be cleaned manually?
The loader brings the series to a single time zone, recalculates corporate actions, and marks gaps rather than silently filling them. We separately pull delisted instruments; otherwise the calculation is biased toward survivors.
How can we tell that a strategy is not fitted to the past?
The training and validation windows are separated, walk-forward testing runs over consecutive periods, and the result is calculated only on unseen data. Commissions and slippage are included in the calculation, not in the report.
How are exchange and broker sources connected?
We work with both streaming connections and file exports: the field schema, time zone, and timestamp precision are described for each source. Duplicates are filtered by instrument-time key, gaps are reloaded, and divergences between sources go to the log.
How do you verify that the calculation is correct?
There are reference scenarios with a known answer — sequence of trades, commissions, dividends. The same run is reproduced by an independent implementation, and we compare the trade log; on a short window with the live market, we compare calculated trades with actual ones.
Let's discuss the task and come back with an estimate and plan
To start discussing a task, it is enough to describe it in terms of strategies, data, and result — then the estimate relies on engineering reality, not general considerations.
There is no need to prepare a document package specially: exports, analyst notes, or a brief description of the idea will do.
- Strategies. Entry and exit logic, instrument set, holding horizon, risk management rules, signal frequency.
- Source data for evaluation. Which historical series already exist, over what period and with what detail, in what format and where they are stored; if there is no data — from which sources to take it and whether there is access to them.
- Reporting requirements. Key metrics — return, maximum drawdown, share of profitable trades; whether breakdowns by period and instrument are needed and in what form the report is convenient for you.
- Quality criteria. How we distinguish a working strategy from fitting to history: holdout-period testing, rolling window, comparison with a baseline scenario.
- Constraints. Deadlines, access to data and infrastructure, deployment — cloud or closed environment, number of simultaneous runs.
- Users and scenarios. Who works with the platform — analyst, trader, risk manager: the interface and access separation depend on this.
The order of further steps is the same: we review the inputs, clarify what is unclear, and come back with a work plan — stages, scope, and what will be needed from you at each.
If part of the data is still closed, we will estimate based on what is available and separately note what will affect the timeline.