Trading Models A/B Testing System Development
A/B testing of trading models is controlled comparison of two strategies on live market. Unlike backtesting, A/B test accounts for real conditions: slippage, latency, market impact, regime changes. But it's more complex: market changes, so can't just run models sequentially.
Principles:
- Simultaneity: both models must trade simultaneously, otherwise comparison unfair
- Capital allocation: divide trading capital between two models (50/50 or 70/30)
- Symbol separation: each symbol uses one model (A or B), distributed evenly by characteristics
- Statistical significance: can't decide after 10 trades. Need sufficient data for statistically significant conclusions
Router: deterministic hash-based symbol assignment to model version ensures stability.
Statistical tests:
- Frequentist: Welch's t-test, Mann-Whitney U test, effect size (Cohen's d)
- Bayesian: P(B > A), expected lift, credible intervals
- Sequential testing: SPRT (Wald) for early stopping
Guardrail metrics: protect from harm. Max drawdown, max daily loss, minimum trades, minimum win rate. Immediate stop if violated.
Decision framework:
- P-value < 0.05 AND N trades > min_trades → can decide
- Bayesian P(B > A) > 95% → confident B wins
- Cohen's d < 0.1 → no practical difference, choose by other criteria
Dashboard: realtime equity curves, P-value and confidence intervals, Bayesian probability, metrics table (Sharpe, Win Rate, Max DD).
Develop A/B testing system with capital allocation routing, statistical significance testing (frequentist + Bayesian), sequential stopping rules, guardrail monitoring and decision dashboard.