Why crypto strategy backtesting lies on historical data
A smooth equity curve on the chart and a loss in the very first days of live trading is the most common story clients come to us with.
The issue is usually not the strategy itself, but how strategy validation on historical data is built: the test measures not what will happen to the money.
Most often, the calculation has look-ahead bias. The signal is calculated using the candle close price, and the trade is executed at that same price, although in reality the order is sent after the close.
If indicators are recalculated across the entire history at once, then at each point in time information that has not yet occurred is implicitly used—and the return is inflated.
Next is survivorship bias: the list of coins is taken as of today, so pairs that went to zero and were delisted do not enter the test.
Plus data quality: gaps in candles, different exchange time zones, and different period alignment methods when stitching create signals where no trade physically existed.
Costs in the test are usually understated or absent: commissions, spread, slippage, funding rate on perpetual contracts, and order book depth on illiquid pairs. In reality, these are exactly what eat away profit on frequent trades.
Finally, overfitting: if you search thousands of parameter sets on the same history, the best result is almost certainly fitted to noise. This is where the discrepancy between the test and live trading comes from, and it cannot be fixed by tweaking thresholds. We break down how this is addressed at the platform level itself.
What honest strategy validation on historical data provides
Strategy run speed determines how many hypotheses you can test before the market regime changes.
The vectorized core on VectorBT computes the entire parameter matrix over candle history in parallel: a parameter sweep that took hours in an event loop fits into minutes, and the grid stops being a bottleneck.
Reproducibility of testing results rests on three fixed elements: a market data snapshot with a checksum, the strategy code version, and a fixed random seed.
Any run from the log can be brought up again and yields the same equity curve—a discrepancy of a fraction of a percent is already a signal of an implicit source of randomness that must be removed before putting the strategy into production.
We check the strategy's robustness to overfitting with a rolling window scheme: parameters are selected on one segment and validated on the next one that the calculation has not seen.
If the optimum looks like a lone peak rather than a plateau, the strategy rests on noise and will fall apart on live data. Additionally, we run the set through a trend, sideways market, and sharp drawdown periods separately.
The run log stores for each run the dataset with its version, the full parameter set, accounted commissions and slippage, the trade curve, and risk metrics.
From it, you can see which validation screened out a hypothesis and at which step—this closes entire classes of errors: forward-looking bias, testing only on surviving instruments, and underestimation of costs.
Run modes: vectorized, event-driven, and streaming
The three modes differ not in the format of the final report, but in what exactly they model.
A vectorized run computes trades over arrays of historical candles and works faster than all—it is needed to triage thousands of parameter combinations and discard obviously weak hypotheses.
An event-driven order book run reproduces the order flow accounting for depth, partial fills, and response time, while streaming real-time calculation runs on live data and checks that the model matches actual execution.
Choosing a single mode is pointless: comparison of testing modes is built as a pyramid—from fast screening to precise validation.
| Mode | When it fits | Limitations |
|---|---|---|
| Vectorized run | Initial hypothesis screening, sweeping a large number of parameter sets | Does not see price movement within a candle, order queue, or slippage—the result is more optimistic than reality |
| Event-driven order book run | Execution validation: slippage, partial fills, behavior on illiquid instruments | Requires a market depth archive, calculation takes noticeably longer, data volume grows quickly |
| Streaming real-time calculation | Final validation before putting the strategy into production volume | History cannot be replayed, observation window is limited, resilience to connection breaks is required |
To make the results comparable, we run all three modes on the same dataset, with the same commission calculation rules and the same position management logic.
The discrepancy between the vectorized and event-driven runs is a measurable estimate of how much the execution model differs from the market; if the gap is large, the strategy goes back for refinement before being brought to the live environment.
Platform development stages: from data audit to handover
Before writing code, we analyze where the data comes from and exactly how the trade result is calculated. A platform breaks not on mathematics, but on discrepancies: different exchange time zones, missing candles, pair delistings, unaccounted funding rate.
That is why the first stages are a historical data audit and an agreed calculation scheme: without them, any return figure on the chart means nothing. Below is the sequence we follow to run the project.
- Historical data audit. We cross-check series with an independent source, look for gaps, duplicates, breaks at listings and delistings, and bring timestamps to a single time zone. We record which pairs and periods are covered, and where the strategy cannot be validated.
- Calculation scheme. We agree on the execution model: commissions, spread, slippage, funding rate, and the order trigger moment within the candle. The scheme becomes the reference against which every figure is subsequently checked.
- Implementation of the calculation core. Event-driven run with a deterministic result: the same dataset and parameters yield the same report, which can be reproduced.
- Calculation correctness validation. Reference scenarios—a no-trade strategy, buy and hold, a pre-calculated manual case. Plus cross-checking with an independent implementation and regression tests on every change.
- Out-of-sample run. We split history into training and holdout parts and check the result's robustness to execution delay and slippage.
- Handover. Documentation, calculation scheme, test bench, deployment on your infrastructure, and training the team to work with the platform.
Delivery package: data schemas, run logs, procedures
We assemble the delivery package so that the platform can be maintained and evolved without us. Each artifact answers its own question: where the data comes from, how to repeat a specific run, under which procedure to launch it, and what to check after changes to the strategy.
- Data schema — entities and relationships: instruments, historical quotes, signals, trades, metrics, normalization and storage rules. With it, a new source is connected without interfering with the calculation core.
- Run procedure — launch parameters, history window, handling of gaps and outliers, and the order of fixing the strategy version. A run is reproduced from the same input even months later.
- Run logs — for each one: strategy version, data slice, risk and return metrics, full trade log. Variants are compared using saved results, without recalculation.
- Platform documentation — description of modules, the calculation core interface, and the procedure for adding indicators and new strategies.
- Deployment instructions — bring up the bench from scratch, configure backup and recovery after failure.
- Acceptance checks — a set of tests for the correctness of metric calculation that you run yourself after every change.
- Access matrix — roles and rights: who changes strategies, who launches runs, who exports results.
We hand over the package with acceptance on your data: you run your strategies and compare the metrics with expected ones before closing the work.
Case study: how the discrepancy between the test and live trading was removed
Where accuracy was lost
The client manages an algorithmic strategy on spot and futures at Binance. The historical run produced a smooth curve, while actual trading delivered a different result: the discrepancy by the end of the day reached 15-20%. We looked for the cause not in the strategy logic, but in the execution engine—and found three.
First: the decision was made at the bar close price, although in reality the order is sent within the bar. Second: slippage, commissions, and order book liquidity were not accounted for in the model at all.
Third: tick series came with different time zones and gaps, and the ClickHouse storage did not check their continuity.
What was changed
We rewrote the event engine: the bar is parsed step by step, and the signal is calculated only on data available at that moment.
We added a slippage and commission model based on actual execution conditions, and order book snapshots—so that trade size does not move the price in the test more than in real life.
We built a trade reconciliation module: the signal, the sent order, and the actual execution are linked by identifier, and any discrepancy is highlighted automatically.
The result: the discrepancy between the test and live trading fit within 1-2% by the end of the day, and manual trade reconciliation went from two days to 15 minutes per report.
Strategies that looked profitable on the old engine due to execution errors are now screened out before launch—this class of “beautiful” failures is gone.
How does backtesting differ from a forward test on a demo account?
Backtesting runs a strategy over historical data: trades are calculated on already known prices, so the result is obtained in minutes, not weeks.
A forward test works with the same code, but on a live quote stream—orders go through real checks, and execution depends on current liquidity. This is the difference of a forward test: it checks not the signal logic, but its interface with the market.
Testing on a demo account is vulnerable to fitting: if you search parameters over the same historical segment, you find a set that worked perfectly in the past and breaks on new data. A forward test cannot be fitted that way—future quotes are unknown.
But it has its own cost: to accumulate a significant number of trades, weeks are needed, and for slow strategies—months.
Therefore, the question “backtesting or forward test” is usually not posed as a choice of one. We build both modes on a shared execution core—the same order logic, the same calculation of commissions, slippage, and funding rates.
Otherwise, the very transition from history to the live stream changes the result, and comparing runs is meaningless.
The choice of validation method depends on the strategy frequency: for intraday ones, a long history first, then a short forward; for positional ones—the opposite.
The practical readiness criterion is the discrepancy of metrics. If return and drawdown in the forward stay within what the backtest shows including costs, the strategy can be released into trading.
A systematic deviation is a signal that something was missing in the history: incomplete data, absent slippage, or a market regime that no longer repeats.
Common concerns: data, delays, liquidity, overfitting
Engineers' concerns come down to one thing: will the test result be reproduced on a real account. We break down four topics that come up before work starts.
Where does historical data come from and how is its quality checked?
We consolidate trades and candles for the required pairs and exchanges, and mark gaps, duplicates, and anomalous spikes. Separately, we remove survivorship bias: pairs that left the exchange remain in the sample—otherwise the test overstates return.
Will the strategy not overfit to history?
We train on one time segment and validate on the next one that the model has not seen. We limit the number of parameters and run across different market regimes—trend, sideways, sharp drop.
A result that holds on a single set of settings is overfitting, and we will tell you about it before launch.
Are liquidity and execution delay taken into account?
We build spread, commissions, slippage, and the delay between signal and execution into the test, and compare order size with available liquidity. A strategy that is profitable only with instant entry at the close price is screened out here.
How do you know the test matches actual execution?
We compare calculated trades with actual ones by time, price, volume, and commission. We investigate any discrepancy down to its cause: data, execution model, or strategy logic. The reconciliation log remains with the client.
Let's discuss the task and come back with an estimate and work plan
To make the conversation about estimating the platform project substantive, send a description of the task in free form: markets and instruments, signal frequency, position holding horizon, and required validation modes. This is enough to understand the load class and the solution outline.
The minimum set to start:
- Formulations of two or three strategies—entry and exit rules: what value the signal is calculated from, where the stop is placed, and how position size is allocated.
- Sources of historical quotes: exchange exports, your own archives, or aggregators, history depth, and candle frequency.
- Requirements for calculation realism: commissions, slippage, liquidity constraints, execution delay.
- Solution metrics: risk-adjusted return, drawdown, share of profitable trades, behavior on an out-of-sample segment.
- Validation scenarios: out-of-sample run, stress on volatile periods, comparison with a passive approach.
- Roles and access: who launches runs, who sees results, whether team isolation from each other is needed.
- Infrastructure and timeline constraints: where we host, what data volumes we keep, and by what date a working environment is needed.
After discussing the task, we come back with a work plan: composition of data loading modules, the run engine, and reporting, the sequence, delivery points, and an estimate as a range.
The earlier the materials arrive, the fewer clarifications at the start—usually one call and the list above are enough to fix the stages and prepare for launch.