Why backtesting diverges from live trading results
Backtesting almost always looks better than live trading, and this is no coincidence. The report is built on history where there are no failures, delays, or unexecuted orders.
The gap between the curve in the report and actual equity is not “the market changed,” but specific assumptions baked into the model.
The first source of divergence is incomplete history and survivorship bias. The database includes instruments that survived to today, while delisted and wiped-out ones drop out: the strategy is tested only on winners.
Gaps in candles, different time zones, unfiltered outliers are typical defects in historical data quality, which make historical returns non-reproducible.
The second is accounting for commissions and slippage. A static per-trade rate does not describe actual execution: on a thin order book, an order passes through several levels, and the spread and delay eat part of the move.
The higher the turnover, the greater the gap — on frequent trades, historical profit turns into actual loss.
The third is different rules for calculating metrics. Sharpe, drawdown, and P&L are calculated from daily returns or from trades, from closed positions or from intraday equity, with or without reinvestment, with funding and dividends or in a vacuum.
Two teams on the same dataset will get different numbers, and both reports will be formally correct.
Therefore, the divergence between report and fact is a reproducibility problem: a single methodology, a complete dataset, identical rules for calculating metrics. We break down how this is closed in practice.
What changes when metrics are calculated by a single set of rules
As long as return, maximum drawdown, and Sharpe ratio are calculated by different scripts and at different times, the report ceases to be proof.
Two versions of the same period diverge by fractions of a percent, and the argument is not about the strategy but about whose calculation is correct. A unified metric methodology removes this fork — here is what we check at the output.
- Single source of trades: commissions, slippage, and funding rate are pulled from one repository rather than manually entered into a spreadsheet.
- One calculation pipeline: return, Sharpe ratio, and maximum drawdown are calculated by a shared library; changing the period or instrument set does not change the formula.
- Reproducibility of calculations: the report is saved together with a data snapshot, instrument list, and run parameters — the result repeats on another machine without edits.
- Reconciliation with fact: the historical report is compared with the executed trades journal, and the divergence is explained by execution costs, not a code error.
- Data control: gaps, duplicates, and quote breaks are filtered out before calculation rather than surfacing later in the return curve.
- Risk next to return: drawdown, recovery time, and share of profitable trades appear in the same report as capital growth.
We run the calculation part on a proven tool — Backtrader — and it also serves as the reference against which we validate our own modules.
The check is simple: a report rebuilt a month later matches the previous one, and the difference with actual trades fits within execution costs.
Backtesting report formats: comparison of approaches
The choice of report format is primarily a choice of calculation methodology, not presentation.
Sharpe, maximum drawdown, and the return curve produce different numbers depending on how commissions and slippage are accounted for, at what frequency the portfolio is recalculated, and how gaps in the data are handled.
A report that does not fix these assumptions cannot be compared with another report, and a strategy decision is made on incomparable numbers.
Options and their limitations
| Option | When it fits | Limitations |
|---|---|---|
| In-house calculation pipeline | Non-standard strategy logic, custom metrics, sensitive data | Your own validation, your own look-ahead checks, methodology versioning — all on the client side |
| Ready-made backtesting library reports | Typical strategy, result needed quickly | Methodology is closed: the set of metrics and rebalancing rules are fixed, drawdown cannot be redefined |
| External metric calculation service | Independent recalculation and a unified report format for several strategies are needed | Data leaves the client’s perimeter, the methodology cannot be changed, the report appears with a delay |
In practice, a combination is more often assembled: calculation lives in an in-house pipeline, the result is exported in a stable format with a fixed set of metrics, and an external service is used as a methodology control.
Then ready-made reports remain reproducible, and the limitations of any methodology are visible explicitly rather than hidden in the calculation code.
Order of work: from metric definition to regular backtesting
We fix the order of work before the first line of code: if a metric is calculated differently in the report and in the trading pipeline, the argument about the result is unresolvable.
Therefore we go from data audit to launch policy, and each stage is closed with an artifact — a report, a calculation scheme, or a configuration.
-
Audit of trade history. We export trades and cash movements, reconcile with exchange reports, and mark gaps and duplicates. We fix the time zone, commissions, and adjustments — without this, return is distorted for no reason.
-
Metric calculation scheme. We agree on formulas before implementation: Sharpe ratio — by daily returns, maximum drawdown — by the profit and loss curve, not by individual trades. We describe open positions and idle periods separately.
-
Setup of the calculation pipeline. A history repository, an instrument reference, and a service that assembles the report on request and on schedule using the same rules.
-
Consistency check. We run a reference period and compare with an independent recalculation. Every divergence is either explained or eliminated — otherwise the metric cannot be trusted.
-
Handover to the client. A report with metrics, a calculation scheme, and a reading guide — the number can be defended before an investor or risk committee.
-
Policy for regular launches. Recalculation schedule, configuration versioning, and failure notification: the result does not depend on who launched the report and when.
After that, backtesting becomes routine — new trades enter the report on schedule, and the effect of strategy changes is visible by comparing periods.
What we hand over: schemes, calculation rules, and policies
A backtest report is useless if only the calculation author can reproduce it. Therefore, together with the results, we hand over a kit that your team can use to recalculate the metrics itself: without correspondence and clarifying questions.
- Description of metrics and assumptions — formulas for Sharpe, drawdown, and P&L, return window, base rate, what we consider the risk-free return and why exactly so.
- Calculation schemes — the chain from raw quotes to the final table: where we clean data, where we aggregate, at which step each metric appears.
- Rules for handling anomalies — gaps in the series, outliers, breaks on non-trading days, corporate events: when data is excluded, when it is interpolated, and what goes into the calculation log.
- Launch instructions — environment parameters, order of steps, entry point, and expected recalculation time on a historical period.
- Report verification policy — reference run, checksums for metrics, and a list of divergences that are considered errors rather than the norm.
- Reporting templates — export format, field composition, frequency, and procedure for passing results to risk control and management.
- Methodology change log — configuration version for each report, so that comparison between periods remains correct.
Acceptance follows the same documents: your team runs the calculation in a clean environment and compares the result with the reference.
We resolve divergences before handover, not a month later in correspondence — thanks to this, reporting remains reproducible even after a change of contractor.
Case study: how the backtesting report stopped diverging from fact
Before work started, metrics were calculated in three places: a research notebook, risk-management scripts, and back-office reconciliation.
Different assumptions on commissions and slippage produced different P&L on the same trades, while Sharpe and maximum drawdown differed depending on whose calculation was taken as the basis.
Every strategy release ran into several days of manual reconciliation, and until it finished, the result could not be shown to either the portfolio manager or the investor.
We assembled a unified calculation module: one trade model, one fixed set of assumptions, and the model version printed in the report header next to the metrics themselves.
We brought the methodology to GIPS requirements — the calculation became reproducible: the same input and the same model version produce the same Sharpe, the same drawdown, and the same P&L for any engineer on the team.
What we changed
| Indicator | Before | After |
|---|---|---|
| Reconciliation with actual trades | manually, 2–3 days per release | automatically, every night |
| Divergence between report and fact | 4–7% on a daily sample | less than 1% |
| Sharpe and drawdown | two independent calculations | one module, one formula |
| Scheduled reports | on request, some releases were skipped | issued on schedule, without manual launches |
Next, we added divergence analysis to the report: it shows which trades diverged and on which assumption — the engineer goes to the log, not someone else’s notebook. The time to investigate a divergence dropped from days to tens of minutes.
Why drawdown in the report is smaller than in live trading
Short answer: the report calculates drawdown from interval closing prices and on ideal execution.
Within the interval, price goes deeper than is visible in the calculation, and the order is executed at a worse price — therefore the reported drawdown is a lower bound, not an expected value. The gap is set by calculation assumptions, and each one shifts the metric in its own direction.
The same assumptions also overstate the Sharpe ratio and the share of profitable trades: the softer the cost model, the smoother the profit and loss curve looks.
We check differently — we recalculate drawdown using intra-interval lows, add costs at the worst observed liquidity, and run the strategy on segments with different market regimes.
In the report we show two numbers: base and conservative, so the client sees the boundary beyond which the metric should not go.
Divergence is not a calculation error but a measure of hidden risk. If the gap between base and conservative drawdown is large, we do not allow the strategy to trade: first we reduce position size and add execution control, and only then look at return.
Technical doubts about metric calculation and reporting
Metric calculation usually breaks not on the Sharpe ratio formula but on input data quality.
The client arrives with incomplete trade history — exports from the terminal, broker report, and internal journal diverge in time and volume, some executions are lost, and some are duplicated.
Until the source is reduced to a single event journal, any drawdown and return curve is calculated on a truncated sample.
Below are the engineering questions most often raised before work starts.
What should be done if the trade history is incomplete or broken?
We consolidate data into a single journal keyed by “instrument + execution time + volume,” collapse duplicates, and mark gaps explicitly.
We load missing intervals from an alternative source or exclude them from the window — with a note in the report, so that metrics do not look more reliable than the input data.
How often is the report recalculated, and how expensive is it?
The base mode is incremental: only affected trades and the tail of the rolling metric window are recalculated. A full pass over the entire history is launched on schedule or on request when the calculation methodology changes. This keeps report update time in minutes regardless of history depth.
Which metrics are considered disputed, and how is ambiguity removed?
Drawdown, Sharpe ratio, and commission accounting have several definitions. We fix them in the calculation policy: what we consider a peak, how we account for intraday movements, whether we include costs and slippage. The policy is versioned, and the report always references the version it was built on.
Is the calculation connected to an already operating trading pipeline?
Yes, as a separate analytics pipeline: the trading pipeline remains the source of truth, the calculation reads it through a secure read-only interface and cannot affect execution. A failure in analytics does not stop trading, and data lag is visible by timestamp.
Let’s discuss the task and come back with an estimate and calculation plan
To come back with an estimate and calculation plan rather than general considerations, we need your current analytics: otherwise we will have to guess metric definitions again.
Below is a list of what is sufficient to send from your side — we use it to assemble the reporting scope and implementation stages.
- Metric specification: which values you calculate (Sharpe, Sortino, maximum drawdown, P&L by trades, share of profitable trades), at what frequency, and over what window.
- Historical data export: trades and quotes in the form in which they exist, with a description of time zones, gaps, and corporate events.
- Accounting rules: commissions, slippage, leverage, reinvestment, timing of position entry and exit.
- Current calculation: the script, notebook, or spreadsheet you use now — so we do not break the familiar logic but transfer it into a reproducible form.
- Reporting requirements: export formats, breakdowns by strategy, periods, and instruments, whether decomposition into market and active return components is needed.
- Verification policy: backtest horizons, whether rolling recalculation by windows is needed, requirements for result repeatability and for who reads the report.
Describe the task in free form — that is enough to start. We will come back with an estimate, calculation plan, reporting scope, and implementation stages: from a report prototype on your data to regular scheduled recalculation with divergence control.