Why backtesting in Python hits a speed limit
On a test interval of one week of trading, a run fits into minutes; on several years of history, it takes hours.
Backtesting in Python does not hit the limit because of the volume of data as such: every regroup, rolling window, and slice creates frame copies, and an element-by-element loop over bars adds overhead to every call.
The calculation bottleneck is almost always not in the strategy mathematics but in moving data between structures.
The second limit is memory. The entire history is loaded as one frame, signals and trade results are joined to it, and metrics are written for every bar — consumption grows to the machine limit, and the run crashes mid-test.
What helps is not adding memory but streaming window processing and explicit numeric types instead of objects.
Optimizing blindly is pointless: performance measurement shows which section consumes most of the time, and only after it do changes yield a multiple effect, rather than rewriting for rewriting's sake.
Reproducibility of results is a less obvious problem. Floating-point operation order, unstable sorting, unseeded random number generators, and different traversal order cause a repeated run on the same data to produce different numbers.
Comparing two versions of a strategy loses meaning: it is unclear whether this is an improvement or rounding noise.
A slow run and diverging numbers are not properties of the language but consequences of the calculation pipeline design: where data lives, who owns the copies, what is calculated and in what order.
Below we break down how each of these areas is addressed — from measurements to pinning down the run environment.
What reworking a backtesting engine in Python delivers
Run speed on historical data
Engine rework starts with profiling: we find where the calculation is limited by the interpreter and where by unnecessary data copying. We vectorize indicator computation, eliminate repeated passes over quotes, and move heavy sections to compilable Python code.
Run speed on multi-year history increases severalfold: what used to take tens of minutes now fits into minutes, and the entire parameter grid is checked in one run instead of a night of waiting.
Calculation stability
The second effect is predictability of the result. Peak memory load stops depending on history length: data is read in chunks, and intermediate series are released immediately.
A whole class of failures disappears — out-of-memory crashes and “silent” divergences between identical runs. With identical input data, the same curve is produced, so a divergence in reports becomes a signal about strategy logic rather than about the engine.
The freed-up time is spent by the team on hypotheses, not on waiting and troubleshooting crashes.
The report for every run is reproducible, rollback to a previous configuration takes one run — idea validation proceeds continuously, without manual workarounds and environment rebuilds.
Options for organizing calculations: comparing modes
The calculation mode determines not only speed but also which validations are reproducible at all. Event-driven run walks history in the same order as trading logic works, vectorized computes the array entirely in one pass, distributed spreads independent scenarios across nodes.
The choice of mode is dictated by the number of scenarios and requirements for execution model accuracy — below are the alternatives and the trade-offs for each.
| Option | When it fits | Limitations |
|---|---|---|
| Event-driven run | Logic depends on order flow, partial fills, and stop triggering inside a bar | One full pass through history per scenario, expensive when sweeping parameters |
| Vectorized calculation | Mass validation of signals and indicators on broad history | Does not reproduce order queue, slippage, and order priority |
| Distributed computing across multiple nodes | Hundreds of independent parameter sets and runs across different instruments | Requires a shared market data cache, state serialization, and verification that the cluster result matches a single run |
| Incremental recalculation | The last data segment or one rules block changed | Requires a dependency map: part of the results will have to be fully recalculated |
Combining modes in one engine
Modes are rarely chosen once and for all.
Usually a vectorized pass filters out obviously weak combinations across the entire history, an event-driven run validates the selected ones with a precise execution model, and incremental recalculation keeps the result up to date as new data arrives.
Distribution is connected a layer above — at the task scheduling level, without modifying strategy logic: the same set of rules produces an identical report both on a local machine and on a cluster.
Rework stages: from calculation audit to handover
Rework of the calculation core proceeds in iterations: first we measure, then we change, then we verify on control runs. After each stage, an artifact remains that shows whether it is possible to move on.
- Engine audit and bottleneck measurement. We profile the calculation on your history: where time is spent waiting, where data is calculated repeatedly, where the result depends on traversal order.
- Calculation schema. We pin down the data flow: quote sources, cleaning rules, decision point, commission and slippage accounting. The schema becomes the acceptance contract.
- Core rework. We eliminate repeated calculations and unnecessary copies, separate data preparation from the run, and move heavy sections to vectorization. Python remains the language for describing strategies — the engine internals change.
- Historical validation. We run the new engine on the same data as the old one and compare metrics trade by trade. We investigate divergences down to the cause, not down to “roughly similar.”
- Results acceptance. You run your scenarios; we show the match and break down divergences according to the calculation schema, not by feel.
- Handover and training. We deliver the code, configurations, and run procedures, and walk your team through how to add strategies and data without us.
Parallel work with your infrastructure
While the rework is in progress, we close some steps on your side: we connect to your data storage and run calculations in your environment. This way the result is visible on real data before final handover, not after it.
Deliverables: calculation core, schemas, procedures
The package is handed over not as an archive “as it turned out,” but as a set of artifacts that your team can use to run the engine, reproduce the calculation, and extend it without us. Below is what exactly goes to the customer.
- Engine source code — position and metrics calculation modules, tests, example runs on historical data.
- Calculation schemas — call graph, order of processing candles and ticks, points for accounting commissions, slippage, and execution delays.
- Data flow schemas — the path from quote source to results mart: file formats, batching, where metrics are written.
- Environment configurations — run parameters, datasets, profiles for single and distributed runs.
- Access policies and roles — who runs the calculation, who changes strategy parameters, who reads the results; secrets are stored outside the repository.
- Run and recovery procedures — starting a run, resuming from a checkpoint, rolling back to a previous core version.
- Documentation and team instructions — interface descriptions, parameter reference, procedure for adding a new strategy.
How acceptance works
Acceptance is based on control runs, not on a file list. On your side, we run the agreed set of strategies, compare metrics against the reference result, and verify recovery after a forced stop.
Separately, we check that a new engineer can bring up the environment and perform a calculation from the documentation without hints.
Case study: how historical data run time was reduced
The client came with a backtesting engine in Python that worked as long as the history fit into one trading year.
A full run over a five-year window took about fourteen hours and often did not make it to the end: the calculation crashed on long series, and two runs of the same strategy produced different results at the third decimal place. Hypotheses had to be compared manually, and no one trusted decisions based on such numbers.
Measurements by code section showed that most of the time was consumed by line-by-line loops over bars and recalculating indicators at every step, while instability came from floating addition orders and state leaking between runs.
What exactly was reworked
The calculation logic was moved to vectorized operations over arrays: indicators are calculated once across the entire history, not anew on every bar.
Heavy sections were moved into separate processes, intermediate results are saved, so a repeated run does not recalculate what is already done.
We pinned down the aggregation order and isolated state: every run starts from a clean environment, and the result is reproduced bit for bit.
After the rework, a full run over the same window fits into forty minutes, the spread between runs is zero, and on long history over a quarter of observations there was not a single crash out of several hundred runs.
We state the limitations upfront: on tick history spanning several years, the calculation still takes hours, so such tasks are scheduled in a night window, and the number of parallel runs is limited by memory — parameter sweeping proceeds in queues, not across the entire grid at once.
How does vectorized calculation differ from event-driven?
Vectorized calculation applies a rule to the entire price series at once: the indicator and signal are computed as arrays, without a loop over time.
Such a run passes several years of history in seconds, so it is used to sweep tens of thousands of parameter combinations and filter out obviously weak ones. There is one limitation, but it is strict: the logic must be unconditional, without relying on position state.
An event-driven run proceeds candle by candle and restores state at each one: open positions, remaining funds, submitted orders. Only this way are stops and takes inside a candle, partial fills, overnight position carry, commission, and volume-dependent slippage accounted for.
This cannot be expressed vectorized: the outcome depends on what happened earlier, and the order of events changes the result.
The difference in calculation accuracy shows up where a decision depends on the current account. The vectorized approach will show entry and exit at the closing price, while an event-driven run will show that some orders were not filled and the stop triggered at a worse price.
The first answer is suitable for selecting ideas; the second is for deciding whether the strategy can be trusted.
Hence the combination of approaches in one engine: vectorized calculation as a filter over a broad parameter grid, event-driven run for finalists and validation of robustness to costs.
Both modes must read the same data and apply the same logic; otherwise the results will diverge and it will be unclear which one to trust.
A practical check is to run one strategy both ways: a divergence within costs confirms the correctness of the execution model; a noticeable divergence indicates an error in it.
Common technical questions about the calculation core and data
Before the first run, the same doubts come up: whether to rewrite everything from scratch, whether the calculations will match the market, whether the results will repeat. Below are five questions from a technical meeting and how we address them.
Will the existing code have to be rewritten?
No. We separate the calculation core from data loading and strategy description with a stable interface: we migrate current rules and replace only the bottlenecks — indicator recalculation and history traversal. The strategies themselves remain in their familiar form.
How closely will the calculation match real conditions?
We build commission, order book slippage, execution delay, and partial order fills into the cost model. Without this, backtesting in Python shows returns that did not exist in the market.
How do you ensure run reproducibility?
The run is pinned down in full: dataset version, strategy parameters, initial random number generator state, cost settings. Repeating it a month later yields the same result — this is the basis for comparing hypotheses.
What should be done with incomplete history?
We catch gaps and timeframe desynchronization before calculation: cross-checking with a second source, bar restoration, explicit marking of periods with low coverage. We flag disputed sections rather than silently discarding them.
Will the core handle growth in the number of instruments?
The calculation is split into independent tasks and runs in parallel across cores and machines, with shared indicators cached. Adding a hundred instruments increases run time linearly rather than causing it to collapse.
What is needed to start: discuss the task and come back with a plan
An accurate estimate for a backtesting engine rests not on guesses but on inputs.
As long as it is not clear which scenarios are run and on what volume of history, any timeline will be approximate — so the start is always the same: a short discussion of the task, after which we come back with an implementation plan and a breakdown by stages.
To estimate the effort, send us:
- a description of the run scenarios — which strategies are checked, how often parameters change, whether combination sweeping is needed;
- the volume and format of history — number of instruments, series depth, data source and its quality: gaps, breaks, corporate events;
- the current calculation schema — how you calculate now: custom scripts, an off-the-shelf library, or spreadsheets, and what exactly stopped working;
- requirements for full run time — what pass duration is considered workable and on what configuration you expect it;
- infrastructure constraints — where the calculation will run: local machines, your own servers, or cloud, and whether there are data isolation requirements;
- result format — reports, metrics, exports, whether visualization and an interface for adjacent systems are needed.
Based on these inputs, we assemble a work estimate, propose a calculation architecture, and give timelines. The implementation plan is broken into stages so that the result of each can be verified on your data, not on a demo example.