Development of an A/B testing system for trading models

A/B testing of trading models: where comparison breaks

Two trading models are rarely compared under equal conditions. Usually they are launched on one account and one historical interval, and the conclusion is based on final return.

Such a test measures not the quality of the model, but how much it coincided with the market regime of that interval.

The key trap is intersection of trades. If models open positions on the same instrument almost simultaneously, their results correlate, and the difference between them goes into execution variance. The observed advantage remains within margin of error, although it looks convincing.

Without assessing statistical significance of the result, it is impossible to separate a stable advantage from random deviation.

They look not at the final figure, but at the distribution of returns, the number of independent observations and behavior when the start date shifts: a model winning in one window usually does not win in any other.

Separately, strategy drawdown is analyzed. Two models with similar returns give different depth and duration of drawdown, and sometimes a different nature of it — market or technical, related to execution failures and data delays. From one equity curve this is not visible.

The picture is completed by survivorship bias: out of a dozen tested variants, the one that accidentally showed the best historical result goes into work.

When comparing pairs, this is doubly dangerous — brute-forcing pairs almost guarantees a "winner" that has no reason to work further.

Therefore the testbed separates models into independent data streams, fixes the hypothesis before launch and accounts for multiple comparisons — we discuss how this is arranged.

What a controlled model comparison loop provides

The comparison loop is built around a control group: the signal stream is split deterministically — by a key from the instrument identifier and entry time — so the candidate model and reference model receive the same market conditions.

Stream splitting fixes the share of cut-off signals, and the difference in result is explained by model logic, not a different trade sample. Without this, A/B comparison of trading models degenerates into an argument about whose period was luckier.

The experiment journal stores everything that determines a run: data snapshot, risk parameters, model version, input series and execution settings.

The run is repeated a month later on the same snapshot and gives the same result — reproducibility stops depending on the engineer's memory and local file copies.

For the journal we use MLflow: metrics, artifacts and parameters are tied to a specific run, and any conclusion can be rechecked.

The hypothesis testing cycle shrinks: the candidate undergoes comparison on historical and holdout segments without going into real trading.

Previously one check took weeks of waiting for statistics, now — hours of loop operation, and the number of tested hypotheses grows with the same team composition. A hypothesis that fails the control group is cut off before it occupies capital.

Separately goes the class of errors "model chosen randomly". A decision is made only if the advantage exceeds variance, is stable across several segments and does not disappear when the start shifts.

The share of models rejected at this step shows how many times previous selection let statistical noise into work.

Comparison formats: live run, replay, shadow mode

Comparison methods differ not by accuracy, but by the price paid for it: risk on real trades or bias on historical data. Live stream splitting gives the most honest answer, but a configuration error is immediately seen by the market.

Replay and shadow mode allow mistakes without consequences — in return they require a prepared archive and a separate monitoring loop.

Applicability conditions dictate the volume and quality of historical data, instrument liquidity and how reversible the decision is.

Where which format is appropriate

Formats don't compete, but line up into a route: hypothesis is screened on historical data, confirmed in shadow mode and only then goes to split stream with limited participation share. Such order reduces the cost of error and preserves metric comparability between stages.

Stages of implementing model comparison loop

Trading model comparison loop is a reproducible experiment: one market history, same order execution rules and different decision-making logic. The point is that the difference in result is explained by the model itself, not by sample specifics, launch moment or account settings. Work proceeds sequentially — from data review to handing the loop over to the customer's team.

1. Data review and hypothesis statement. We check what series, trades and signals are available, how they are labeled, where gaps and outliers are. We formulate what exact effect we test and what we consider result improvement.
2. Experiment design and group labeling. We define control and test groups, rules for their formation, observation window and early stopping criteria, so comparison isn't tuned to result.
3. Setting up metric collection. We fix a unified set of indicators per trade and per period: return, drawdown, number of triggers, share of rejected signals. Logs are written identically for all groups — otherwise comparison loses meaning.
4. Checking on historical snapshots. We run loop on historical data across several independent segments, including calm and volatile periods. We look whether gap between groups is stable or rests on one lucky period.
5. Distortion control. We remove influence of randomness: repeat runs with different initial conditions, compare distribution of group results. Model is allowed to work only with reproducible advantage.
6. Handing over to team. We give group configurations, launch scripts, metric descriptions and regulations for adding new models. Customer engineers run next pass themselves, we stay in touch on disputed cases.

Such order gives the main thing: comparison result can be rechecked at any moment and repeated without us.

What remains with the team: configurations, schemes, regulations

Work is considered complete not when loop is launched, but when customer's team conducts experiment itself and stops it if necessary.

Therefore together with system we hand over a set of artifacts — each is versioned in your repository and updated via usual change review procedure.

• Loop configuration — environments, queues, storages and model launch parameters. With it loop is raised on new site without our engineers.
• Experiment map — which hypotheses are tested, which groups are compared and how long measurement runs. It shows which tests are already closed, and which overlap and interfere with each other.
• Labeling rules — what is considered a correct trade, which records are discarded and why. Unified for all models, so results are comparable between reports.
• Stop regulation — conditions for emergency test termination: drawdown from slippage, divergence of expected and actual return, data source failure. It specifies who makes decision and what happens to open positions.
• Metric documentation — calculation formulas, averaging windows, outlier and commission handling. One metric isn't calculated two ways in different reports.
• Roles and access scheme — who changes parameters, who stops test, who sees intermediate results. Plus procedure for rolling configuration back to previous version.

We hand over artifacts together with walkthrough: customer engineers launch test, stop it and edit labeling themselves — we answer questions, but don't do anything hands-on. After that loop develops inside team, and we join only for new tasks.

Case: cut model selection cycle by a factor of three

Customer — quantitative strategies team that selected trading models almost manually.

Hypothesis queue grew, and cycle of checking one took 3–4 weeks: half the time went to rebuilding environment and sorting out discrepancies when two analysts on same dataset got different numbers.

4 out of 10 models went into real trading, and every second of them was rolled back — decision was made by average result, without variance assessment.

What changed

We assembled a unified A/B testing loop: launch manifest fixes dataset version, code hash, strategy parameters and group size, so any run is reproducible.

Historical check and live market test go one route through Airflow, groups are compared by one statistics module — with confidence intervals, not averages. All runs are written to journal: you can see who tested what and on which data.

What became

Hypothesis testing cycle shortened from 3–4 weeks to 5–7 days, and bottleneck shifted from environment preparation to hypothesis formulation itself.

Share of rollbacks after going live fell from 40% to 12%: models with wide variance are screened out before launch, not after losses. Repeat run on same manifest gives identical result in 100% of cases, whereas before discrepancies occurred in almost every third run. Model selection stopped depending on who exactly calculated.

What to do if several models already run on one account?

Several strategies on one trading account give a shared trade flow in which result cannot be honestly separated: one model chooses risk limit, second because of this doesn't enter position, and equity curve reflects their mutual influence, not quality of each. While trades go into one journal, comparison remains guesswork.

Separation of models is done on three levels, and choice depends on how strictly results need to be isolated.

Step-by-step transition takes one or two trading sessions and doesn't require stopping trading. First trade labeling and shadow journal are enabled — at this step nothing changes, only breakdown per each model accumulates.

Then one model is moved to separate subaccount with reconciliation of trade match before and after, and only after that share of capital is transferred with execution divergence control.

Preservation of working loop rests on fixing baseline metrics before changes: share of profitable trades, maximum drawdown, average position holding time.

If after separation breakdown diverges from shadow journal, weights are returned by configuration, without rolling out new code. In Ethereum loop execution is fixed on separate addresses, so trade history remains verifiable and isn't rewritten retroactively.

Technical questions: significance, data, rollback

Launching trading model test rests on three things: whether history is enough, whether difference between variants is significant and how to return back if hypothesis isn't confirmed. Below are answers to engineering doubts that come up before start.

How do you check significance of difference?

Returns are unevenly distributed — rare large trades outweigh long series of small ones. Therefore we compare distributions with permutation test, give confidence interval and correction for multiple comparisons when iterating many parameter sets.

What does experiment rollback mean and how long does it take?

Each version of rules and dataset is fixed by separate artifact, so return is switching pointer to previous build, not manual editing. New variant first goes to shadow mode and doesn't mix with live portfolio.

How does data quality affect result?

Quote gaps, history breaks and trading calendar discrepancies give false return growth that disappears on live market. We reconcile data with independent source and mark segments with incomplete history.

Which distortions do you look for separately?

Look-ahead bias — feature calculated on close of same bar, selection by surviving instruments, different liquidity between variants. Each is checked with separate scenario: we run model on data with signal delay.

Let's discuss the task: what to send for assessment

To come back with assessment and plan, not general questions, we need your current trading model testing loop: how it's arranged, on what data it works and where it hits limitations. Below are source data by which we calculate scope of work and start order.

Send what you already have in any form: scripts, tables, journal exports — we'll sort it out ourselves.

• How model versions are compared now: manual runs, set of scripts or own wrapper — and how long one full comparison cycle takes.
• Description of trading logic: which parameters change between versions, and which are considered fixed and shouldn't go into experiment.
• Data: depth of quote history, intervals, sources, presence of train and validation period split and rule by which boundary was chosen.
• Rules for calculating result: set of metrics (return, maximum drawdown, profit factor, share of profitable trades), accounting for commissions and slippage.
• Journals of one full cycle: trades, signals, entry refusals, error log — they show where run comparability is lost.
• Risk limitations: position limits, daily stops, rules for stopping test when going beyond set boundaries.
• Execution loop: test account, broker sandbox or working account with small volume, and how results return to report.
• Who on your side makes decision to move model to working mode and by what criteria.

Describe task in free form — this list is enough to understand scope of work. We'll come back with assessment, stage plan and start order.