Why a DeFi backtesting system diverges from real trading
The divergence between a run and the market is not a rounding error, but a consequence of how data is collected and how a trade is executed.
In a DeFi backtesting system, the picture repeats: the test curve is smoothed, while on-chain the strategy loses part of its return or goes into drawdown. The causes are measurable and reproducible on the same history.
Incomplete history: a pool exists for a couple of years, and the strategy is checked on exactly that window. Periods of abnormal returns make it into the data, while liquidity contractions do not.
Testing DeFi strategies on such a segment overestimates robustness: rare events that determine the result are absent from the sample.
Substitution of closing prices: the calculation takes a snapshot of the pool state once per interval, whereas a trade is executed against the curve between those points.
Price movement within the interval is not accounted for, and the more volatile the window, the greater the divergence.
Liquidity depth and slippage in the calculation: with a fixed price shift, the model ignores the drawdown that the order itself creates. In a thin pool, a large trade moves the price, while fees and higher execution cost amid competition for block space are reduced to a constant.
Survivorship bias: selection is based on pools and tokens that survived to today — closed and zeroed-out ones do not enter the sample. The set looks more robust than it was in real time, and parameter fitting to the same history raises the metric and reduces transferability.
Next — how this is closed: what data is needed, how execution is calculated, and what is checked before the strategy goes on-chain.
Measurable effect: run speed, convergence, and drawdown breakdown
Recomputing a parameter set across the entire history takes minutes instead of a day: a run is calculated on pre-collected and normalized data, rather than querying a node on every iteration.
The parameter grid is computed in parallel and cached by slices — change one coefficient, and only the affected part is recalculated. An engineer can run dozens of configurations per shift and see where the strategy's response breaks.
Convergence of run results with real execution is the main criterion for trusting a test.
Trades are reconstructed from on-chain events; fees, slippage, and gas cost are calculated as in the actual trade, so the divergence stays within the model's error margin, rather than differing by multiples and flipping the conclusion about profitability. It is on this basis that a configuration is chosen, not on a pretty curve.
We structurally exclude look-ahead bias: the strategy sees only the data that was available at the moment of decision. The holdout period does not participate in parameter selection at all — it remains untouched until the final run.
We assess strategy robustness across several market regimes: growth, sideways movement, and a sharp drop in liquidity in Uniswap pools.
Drawdowns are explained by data, not guesses. You can see what share of the loss came from the asset price, what share from slippage in a thin pool, and what share from execution delay.
Instead of the conclusion "the strategy didn't work," there is a breakdown with each cause's contribution — and the fix goes into a specific node of the logic, not the entire model.
Run modes: vectorized, event-driven, and portfolio
The run mode determines which effects make it into the result at all.
Vectorized calculation processes price history entirely in arrays, event-driven reconstruction replays the block stream and contract responses, and portfolio simulation adds the overall balance, limits, and order queue.
Fast and complete at the same time does not happen — the mode is chosen for the question that needs an answer.
| Mode | When it fits | Limitations |
|---|---|---|
| Vectorized calculation | Mass hypothesis screening, rough estimation on long history | Does not see intra-block state, slippage, and liquidations — execution is idealized |
| Event-driven reconstruction | Checking protocol mechanics: fees, oracles, liquidations, order sequence in a block | Slower by orders of magnitude, requires a state archive, thousands of variants cannot be run |
| Portfolio simulation | Multiple strategies on a shared balance: competition for capital, risk limits | An error in the model of a trade's price impact distorts the result across all strategies at once |
What cannot be obtained the cheap way
Intra-block state, the impact of a large trade on price, and order sequence under network congestion are not reproduced by a cheap mode: they are either modeled approximately or obtained through an event-driven run with archived states.
The working scheme is hypothesis screening with vectorized calculation, final candidates through an event-driven run, and the portfolio as a whole on an agreed strategy composition.
Divergence between modes on a simple strategy points to an error in the fee, oracle, or order sequence model — until it is resolved, the results cannot be trusted.
How we build the test bench: from data audit to runbook handover
The test bench is built for a specific strategy, not "in general": first we pin down what decisions it makes and on what data, then we build a pipeline that reproduces those decisions.
The order of stages is not accidental — an error in preparing the history devalues any validation after it.
- Strategy breakdown. Entry and exit logic, the set of signals, position management, and what we consider execution — interval close or contract state change.
- Source audit. History depth, gaps, discrepancies between independent price providers, reorg handling: without this check, you get beautiful but false results.
- Data schema. We bring exports to a single form with block-time alignment, add pool fees, slippage, and the liquidity available at the moment of the trade.
- Building the computation pipeline. Reproducible run: the same input gives the same result, state is recorded step by step, and the trade and metric log is saved.
- Holdout validation. Part of the history does not participate in tuning — it shows where the strategy is fitted to the past.
- Stress scenarios. Volatility spike, liquidity failure, gas cost increase, execution delay, source failure: we look at where the pipeline breaks.
- Handover and run procedure. We deliver the test bench with documentation, describe the launch order, stopping and data update criteria, and train the team.
Then you run the backtests yourself — and see how the strategy's behavior changes when assumptions change.
Handover artifacts: data schemas, parameter sets, reports
The handover ends not with a repository, but with a set of artifacts your team can use to repeat any run without us and explain the result to an external audit. Below is what remains with the client and why each element is needed.
- Data schema — tables of trades, pool states, and quotes: field types, keys, relationships, history depth, and rules for handling gaps in the source.
- Source reference — where data comes from, update frequency, delay relative to the moment of the trade, and behavior when two sources diverge.
- Parameter sets — ready run profiles: pool fees, slippage, holding horizon, recalculation step, and position exit rules.
- Report field dictionary — what each metric means, how return is calculated including fees, and what exactly we call drawdown.
- Report templates — strategy summary, distribution of results by period, and a list of worst trades with trigger reasons.
- Access separation — the analyst changes parameters and runs backtests, the risk manager sees the history, and a recorded result cannot be rewritten after the fact.
- Run documentation — launch log with the parameter set version, data snapshot, and list of changes relative to the previous run.
- Launch procedure — the sequence of actions: bring up the environment, run the strategy, compare the result with the recorded one, and understand where the divergence comes from.
Such a kit removes dependence on the contractor: months later, you reproduce a figure from the report and see which parameters changed it.
Case study: model-to-market convergence tripled
The client came with an arbitrage strategy across several Uniswap liquidity pools: on historical data the chart grew steadily, but in real trading the result was not reproduced.
The divergence between model and market came from two places — swap fees and the timing of trade execution relative to price movement within the pool. We rebuilt the backtesting case from scratch, splitting the history into training and holdout segments.
What we changed in the model
We added the lag between signal and the trade entering a block, slippage at every step, and actual fee rates instead of an averaged constant.
We separately described the stress period: when liquidity sharply shifts between pools, the price in one diverges from the price in the other.
The model now treats this not as an anomaly but as a working scenario — and checks whether the arbitrage has time to close before convergence.
Metrics before and after
Convergence of calculated return with actual return on the holdout period tripled, and the share of deviations beyond the acceptable threshold fell from 21% to 7%.
The error in estimating the result of an individual trade decreased from 0.9% to 0.3% — the calculation stopped showing profit where there is none.
Validation on historical data runs across several independent segments, including episodes of sharp liquidity shifts.
If the strategy fails even one, it is rejected or sent for revision with an explicit applicability boundary — the client sees where the model breaks before withdrawing funds.
How does historical backtesting differ from real-time validation?
Historical runs and real-time observation answer different questions. The first shows how a strategy goes through tens of thousands of trades over a long horizon; the second shows whether a signal survives to real execution. In a DeFi backtesting system, they go sequentially because they do not replace each other.
A historical run provides statistics across entire market cycles: trend changes, prolonged drawdowns, behavior under thin liquidity. It is reproducible — the same configuration is replayed with different risk thresholds and the results are compared.
The weak spot is ideal conditions: a trade is considered executed at the block price, without an order queue, slippage, or competition for liquidity. The gap is noticeable where return depends on rates and pool depth — for example, in Aave.
Shadow mode covers exactly this gap. The strategy connects to a live stream and generates signals, but does not send transactions: you can see the actual delay between signal and block, chain reorganization, and moments when liquidity leaves the pool.
The price of accuracy is time: a meaningful sample takes weeks to accumulate, and rare scenarios enter it in single digits.
Small-size validation is the next step, where real funds come into play. It confirms actual execution prices, network fee spend, and strategy behavior under failures and partial fills. Risk is limited by position size, so such a run is safe for debugging operational details.
Bottom line: history answers whether the strategy has an edge; shadow mode — whether the signal reaches the block; small size — whether the calculation matches reality.
We arrange these stages into a single validation cycle so that the strategy goes to full size with clear statistics already in hand.
Doubts about data, liquidity, and reproducibility of runs
Before the first run, the client wants to make sure the output numbers mean anything at all. So the questions are usually the same: where the history comes from, how execution is accounted for, and whether the result will repeat tomorrow. Below are answers without generalities.
Where does trade history come from and how is its quality checked?
We collect data from several independent sources and cross-check them: discrepancies in volume and price per block are flagged, not silently averaged.
We separately handle chain reorganizations and liquidity "holes" — otherwise, over a missed interval, the strategy would show a trade that did not exist in reality.
A quality flag is stored for each asset, and a run can be stopped if the share of questionable candles is above a given threshold.
Are execution costs and pool depth taken into account?
Yes, and this is the main difference between a pretty and a truthful result. Pool fees, gas price at the moment of the trade, and slippage are calculated from the actual depth of the specific pool, not from an average market percentage.
If the trade size exceeds the liquidity available for that block, it is either split into parts or marked as unexecutable — it does not make it into the report.
Can the system be run on a strategy that is already live?
Yes. We restore the state of positions for the required date and calculate further only from new data, without recalculating history after the fact.
This shows where actual trades diverge from calculated ones, and that divergence is the most useful signal when fine-tuning the rules.
Will the numbers repeat if the run is launched again?
A run is tied to a data snapshot and a version of the execution rules — if they match, the result matches down to the last trade. If the numbers diverge, then one of three layers changed: data, rules, or the source of randomness.
We fix all three, so divergence is always explainable and is not blamed on "the market was different."
Let's discuss the task and come back with an estimate and run plan
So that the project estimate is based on your pipeline rather than averaged assumptions, we need a minimum of inputs.
Below is what we request at the start: using these points, we calculate the effort and scope of work, after which we come back with a run plan and test bench handover.
- Data access. Trade exports, pool state snapshots, quote history — in raw form and with field descriptions: settlement currency, time zone, units of measurement. Any readable format will do: columnar files, database export, stream from a node.
- History volume. What horizon is needed and which periods are mandatory — calm market, crashes, low-liquidity moments. This determines how many scenarios we cover and where gaps in the data will appear.
- Strategy logic. Entry and exit rules, position sizing, rebalancing moments, stopping conditions. The more formal the description, the fewer assumptions we build into the model.
- Cost accounting. Pool fees, slippage, transaction cost, execution delay. Without this, a run shows a result that did not exist on the real market.
- Infrastructure and access. Where we deploy the test bench, what permissions are needed, and whether there are restrictions on exporting data externally.
- Acceptance criteria. Which metrics we consider reliable and what we compare the calculation against — your manual runs or an independent source.
- Test bench handover. What should remain with you: run configurations, data schema, reproduction instructions, and the launch procedure without our involvement.
Describe the task in free form and attach whatever you already have for the data. We will come back with a project estimate and work plan — by stages, timelines, and test bench composition.