Backtesting Arbitrage Strategies: Where Calculations Diverge from Reality
The historical equity curve looks smooth: the arbitrage spread between venues is consistently positive, drawdowns are short and recover quickly.
The same algorithm on a live setup over a month does not reproduce a single large trade — the signal arrives, but there is no execution at the price on which the calculation was built.
The cause is almost always in three places. Historical data is collected from different sources with different timestamps: where the trade tape is stitched to order book updates, prices appear that never existed, and such errors in historical data draw profit that was not there.
Next — quote delays: between receiving the signal and placing the order, the spread manages to close, and this is not market noise but a systematic minus not reflected in the model.
Plus incomplete history across instruments: delisting, changes in trading conditions, and venue pauses drop out of the sample, and the strategy is tested only on those that survived.
Reproducibility appears when backtesting arbitrage strategies is built on a single data snapshot with identical execution rules, and divergence between the model and actual trades is measured per trade, not by the final curve.
Below we break down how this is closed and how testing trading hypotheses becomes a measurable procedure rather than an argument about a nice chart.
What Changes After Implementing Correct Strategy Testing
When historical data is collected manually — via exchange exports, with different time zones and formats — testing one hypothesis stretches over weeks.
Worse, the result is not reproducible: a repeated run on the "same" sample diverges from the previous one by 15–20%, and it is unclear whether the strategy changed or the data did.
Test automation starts with the ingestion pipeline: order book and trade tape snapshots are brought to a single schema, duplicates are merged, and gaps are flagged.
At the input there are quality checks — time monotonicity, anomalous spreads, liquidity gaps — and a window with bad data does not enter the calculation. Running a hypothesis takes hours instead of weeks, and manual exports and spreadsheet recalculation leave the process.
Reproducibility rests on the run specification: version of the normalized market data set Tardis, window, fees, execution delay, order book depth, anomaly filtering rules.
The same set with the same specification yields the same result months later — divergence between repeated calculations drops from percentages to statistical noise.
A unified trade registry stores every simulated trade with run ID, data version, and entry and exit reason.
The class of errors related to manual data assembly — shifted timestamps, lost hours, merged duplicates — stops reproducing in reports, and strategy comparison happens on a single market slice.
Hypothesis Testing Modes: Comparing Approaches and Limitations
An arbitrage strategy breaks on three things: latency between venues, fees, and order book depth. Therefore testing a hypothesis on one history proves nothing — strategy testing modes answer different questions, and each has its own limitations.
| Option | When it fits | What it will not show |
|---|---|---|
| Historical data testing | First screening of hypotheses, pattern discovery, sensitivity assessment to fees and latency | Order queue and impact of your own order on the order book: some opportunities look available in history although in reality they were already gone |
| Real-time testing | Behavior of logic and timings on a live quote stream | No execution: orders are not sent to market, so slippage and rejections remain out of frame |
| Paper-trading environment | Testing connectivity with venues, access keys, handling of connection breaks and partial fills | Execution rules are set by the environment, not by the venue itself — divergence from live trading remains |
| Partial deployment | Final check on a small share of capital under strict limits | Scale: on small volume the strategy does not hit liquidity and venue limits |
No single mode gives the full picture — divergence between history and real orders accumulates precisely on delays, partial fills, and rejections.
Therefore we build them as a sequence with a verifiable transition: history, then live stream, then paper-trading environment, and only after that limited capital.
Implementation Stages: From Data Review to Testbed Handover
We structure the implementation stages of the testbed so that each ends with a verifiable artifact, not a verbal report. The client sees at which step work is happening and can correct calculation logic before it becomes fixed in code.
- Data audit. We review quote and order book sources: at what frequency records arrive, how series behave during connection breaks and trading session transitions. Output — a source map with a list of gaps that will need to be restored during calculation.
- Calculation scheme. We fix the execution model: reaction delay, partial order fill, fees, and available volume on each side. The agreed scheme becomes the reference against which any divergence is later explained.
- Environment setup. We assemble the testbed: historical data storage, calculation service, isolated jobs for long runs, and separate logging of each step.
- Running reference scenarios. We test the calculation on pre-labeled periods with known results — this way logic errors surface before real strategies are placed on the testbed.
- Reconciliation with real trades. We compare the calculation with actual executions for past periods and analyze each divergence: some will be model error, some an honest market feature.
- Acceptance and handover. We run the full cycle on client data, fix control metrics, hand over documentation on the calculation scheme, and train the team to work with the testbed independently.
Delivery Package: Configurations, Schemes, and Regulations
The delivery package is not a code archive but documentation by which the system can be deployed, run, and put into operation without the author on call. Below is what is actually in hand by the time of handover.
- Environment configurations — versioned files with connection parameters, request limits, timeouts, and a switch between test and production environments.
- Integration schemes — the path of market data from source to storage, order book and trade capture points, normalization rules, and weak points where tape breaks are possible.
- Access policies — analyst, operator, and deployment roles, moving keys outside the repository, rotation, and prohibition of edits in the production environment.
- Operational regulations — procedure for preparing historical data, launching a run, reproducing the result, and accounting for fees with slippage.
- Report description — metric composition (return, drawdown, turnover, share of executed signals), export format, and interpretation of indicators.
- Registry of known limitations — where the model simplifies reality: history depth, delays, thin liquidity, partial fills.
- Deployment and rollback instructions — bringing up the environment from scratch and returning to the previous configuration.
We verify the package in practice at acceptance: deploy the environment on a clean machine strictly according to instructions and repeat the reference run. Divergence in metrics means the documentation is incomplete — and this is fixed before signing, not after.
Case Study: How We Reduced Divergence Between Model and Real Trades
Where accuracy was lost
The strategy was built on manual quote collection: data from the exchange and pools was consolidated in spreadsheets, fees and slippage were accounted for as estimates.
Divergence between calculated return and actual trades reached 18–20%, and preparing a report for one hypothesis took two to three days. Because of this, one or two ideas were tested per month, and some of them fell apart already on a real account.
What was rebuilt
We assembled a unified pipeline: trade history and order books are loaded in batches, then run in the same order as in real time, accounting for fees, execution delay, and pool depth.
The run became reproducible — using a checkpoint from the journal, it can be repeated and produce the same numbers. Arbitrage between pools on Uniswap and a centralized exchange was calculated as a separate scenario with its own assumptions.
What was achieved
Divergence between calculated and actual results decreased from 18–20% to 4–6%. Testing a hypothesis takes 20–30 minutes instead of two to three days, and manual operations in data preparation dropped to about five instead of forty.
The report is generated automatically from the run journal. The system is fault-tolerant: if the connection to the exchange breaks, an unfinished step resumes from the last checkpoint rather than starting over.
How Does Historical Testing Differ from Real-Time Testing?
Historical testing is running a strategy over a recorded stream: order book snapshots, trade tapes, fees, and exchange rules at that time. It answers the basic question — is there any pattern in a pair of signals at all — and does so quickly and reproducibly. Any arbitrage idea starts with history, but it cannot end there.
History almost always overestimates. The main reason is model overfitting: when there are many entry and exit parameters and only one segment, fitting captures the noise of a specific week instead of the spread, and on the next segment the result resets to zero.
The second source of optimism is ideal run conditions: instant execution at the best price, full volume, and no other participants closing the same opportunity.
In a live environment, data delays are added. In a run, the signal and order live in the same order book snapshot, while in the market fractions of a second pass between quote update, calculation, and order placement — during which the spread usually disappears.
The thinner the opportunity, the stronger the divergence, so the model must be tested on delays taken from a real channel, not on zero delays.
The transitional step is a paper-trading environment: the same live stream and the same decision code, but orders are not placed — they are written to the journal with a calculated fill price.
Comparing live and historical results shows what share of the opportunity is consumed by delays, fees, and slippage.
If divergence is multiple times, the issue is not the market but the model: it is simplified, the number of parameters is reduced, and it is tested on several independent segments.
Technical Doubts: Data, Delays, Reproducibility
Doubts about backtesting arbitrage strategies come down to one thing: will the calculation match real trading. Historical calculation does not promise profit — it shows under which conditions the strategy works and where it breaks. Below are the questions asked most often.
How is data from different sources consolidated?
We collect history for each market separately, bring it to a single time grid, and reconcile overlapping periods. Gaps and divergences between exchanges are not smoothed — they go into the data quality report.
How can you tell that the model is not overfitted?
We run the strategy on a segment of history it has not seen and look at the result in a rolling window, not on a single successful segment. Drawdown on the validation period is shown separately.
How are quote delays accounted for?
Calculation uses the time when data reached the system, not the trade time on the exchange. We add channel latency and see whether arbitrage remains profitable under worst-case execution.
Will the calculation be reproducible?
Yes: we fix the data snapshot, instrument composition, and calculation parameters, so a repeated run produces the same report — strategy versions can be compared with each other.
Can the strategy be tested before trading and what remains after handover?
Yes — running on live quotes without deploying funds shows divergence between calculation and market. After handover, configurations, reports, and the procedure for re-running remain; support covers changes in trading conditions.
Let's Discuss the Task: Where to Start Testing Trading Hypotheses
For a backtest to answer the question "will the strategy work" rather than fit a curve to history, specific inputs are needed at the start.
During task discussion we clarify: which exchanges and markets are in scope, which types of arbitrage are of interest — cross-exchange, funding, triangular — how to account for fees, execution delays, and order book depth, and what trade and order book history is available for the required period.
The logic of the simulator and the set of metrics depend on these answers.
Then we return with an estimate and work plan: break the task into stages and discuss dependencies — for example, if order book history is incomplete, some scenarios will have to be assembled from your own archive.
- Task discussion: hypothesis, markets, horizon, and signal frequency — you provide inputs, we assess whether there is enough data for an honest run.
- Estimate and work plan: stages, dependencies, what you provide — access and history, what we do.
- Pilot environment: one or two markets and a narrow set of pairs, checking convergence of data, fees, and signal logic.
- Scope of work: strategy run module, accounting for fees and delays, metric reporting, reproducibility of each run.
- Project start: configurations, access separation, acceptance regulation, fixing reference runs.
- Handover: code, data schemes, and documentation — the team launches and changes strategies without us.
Describe the task in general terms — we will return with an estimate and work plan, timelines, and boundaries of the pilot environment.