Why an RL agent for trading breaks in the live market
An agent based on A2C/A3C learns to maximize average reward over trajectories from the training sample. The market, however, is non-stationary: the volatility regime, liquidity, and participant composition change, and history contains only one realization of the process.
A policy learned for a specific regime behaves randomly in a new one — hence the failure of an RL agent in trading when launched in a real loop.
The costliest issue is the gap between backtest and live trading in execution costs.
The environment fills a trade at the candle close, while the real loop adds spread, commission, and slippage, plus latency to execution — this can be reproduced only according to the logic of a specific broker, whether exchange access or a terminal like MetaTrader.
If average profit per trade is comparable to friction, the strategy goes negative without a single model error.
Another source is overfitting to historical data: feature selection over the entire series, normalization computed using the future, parameter tuning on the same segment as the control. A beautiful curve more often means data leakage than a discovered pattern.
It is treated by an honest validation scheme: a rolling window without overlap, a separate holdout period, and testing on regime change.
On a live account this yields unstable returns: a streak of winning trades is replaced by a drawdown the agent does not recognize. We discuss how this is addressed.
What reinforcement learning provides: effect in metrics
The effect of reinforcement learning is measured not by returns in a vacuum but by the stability of the agent's behavior.
Asynchronous actor-critic methods, going back to the work of DeepMind, allow dozens of copies of the environment with different market conditions to be run in parallel — the agent learns a class of regimes, not one lucky trajectory.
A change from trend to flat or a rise in volatility no longer requires rewriting rules from scratch.
We check strategy robustness on holdout periods: the share of scenarios where behavior matched expectations, Sharpe ratio dynamics, and the number of violations of the position size limit.
Randomization of initial conditions and commissions prevents the policy from memorizing a specific history — the gap between training and validation falls to a few percent.
Drawdown control is embedded in the reward: a risk penalty, a limit on the share of capital in a position, and mandatory stop conditions.
Maximum drawdown becomes a configurable parameter and is checked by stress runs on crisis periods, rather than remaining a byproduct of luck in history.
Fewer manual edits are a direct consequence: a market regime change is handled by incremental policy training, not code edits, the cycle takes minutes, and metrics are compared before and after.
Regime adaptation relies on an ensemble of environments and deterministic policy application — the same input data yields the same decision, so the result is reproducible in the test loop.
In the report we record the number of manual interventions per month, maximum drawdown, share of time out of market, and retraining time.
Comparison of approaches: A2C, A3C, and learning on history
The choice between A2C, A3C, and learning on history is a choice between gradient stability, speed of experience collection, and the cost of hypothesis testing.
Synchronous actor-critic updates the policy on a synchronized step of all environments, asynchronous collects experience with independent copies of the agent, and an offline dataset provides an initial policy without any access to the environment.
In practice, these modes do not compete but stand in one pipeline: quotes warm up the policy, the simulator refines behavior, and the choice between synchronous and asynchronous training is determined by how many parallel environments you can actually maintain and how critical run reproducibility is.
| Mode | When it fits | Where it hits a wall |
|---|---|---|
| A2C, synchronous training | Need even gradient and reproducible run; several parallel environments available | Each step waits for the slowest copy; scales worse than asynchronous |
| A3C, asynchronous training | Many nodes, need fast experience collection from different market states | Stale gradients from lagging copies reduce stability, debugging is harder |
| Learning on history | No access to environment; need policy warm-up and hypothesis testing on a long quote series | Limited by sample size: rare market regimes are poorly represented, no feedback |
| Simulator on historical data | Need validation with commissions and slippage before real trades | Realism of the execution model determines whether the result transfers to the live loop |
Therefore, the volume of historical data is not a goal but a constraint: the rarer the desired market regime in the sample, the more expensive it is to confirm that the agent learned exactly what we mean.
Implementation stages: from data audit to model handover
We fix the work order before start: a reinforcement learning agent is sensitive to the quality of historical data and to how the execution environment is described.
An error in trade labeling or in the reward function manifests not in training but in real trades — so we close each stage with a verifiable artifact, not a verbal “done”.
- Data audit. We consolidate quote and trade history by timeframes, looking for gaps, duplicates, time shifts, and signs of future value leakage. The output is a cleaned dataset, a description of gaps, and a verdict on the period's suitability for training.
- Training scheme. We agree on the action space (entry, exit, position size), the reward function accounting for commissions and slippage, episode length, and the feature set. Here we also define what we consider success: risk-adjusted return, not absolute profit.
- Environment assembly. We build a reproducible environment: pinned dependencies, a broker emulator with execution latency, and run logging. Any run is repeated from one configuration in the repository.
- Training and tuning. We run A2C/A3C variants, compare them by curve stability and the number of episodes to convergence, and select a configuration without overfitting to individual market regimes.
- Validation on a holdout period. We test with a rolling window: training on one segment, validation on the next one the model has not seen. We look at drawdown, behavior in a trend and in a sideways market, not one final number.
- Model handover and support. We hand over code, weights, configurations, a validation report, and retraining instructions. We describe degradation monitoring and the rollback procedure to a previous version.
The stages proceed sequentially, but with returns: if the data audit reveals leakage, the training scheme is rebuilt before the next run.
Implementation artifacts: configurations, models, and regulations
After implementation, the client is left not with a “trained model” but with a reproducible loop: it can be used to retrain, roll back, and hand the agent to another team without the original authors.
We fix the set in the acceptance act, and the list is closed — if an item cannot be reproduced on your bench, the work is not accepted.
- Configuration files — training hyperparameters, instrument composition, feature window, risk limits, and auto-stop thresholds; versioned together with code.
- Training scheme description — number of parallel environments, weight synchronization frequency, stopping criteria, and rules for selecting model checkpoints.
- Run logs — metrics by epoch, saved weights, and a report on returns and drawdown on the holdout period.
- Access separation — separate roles for training, signal execution, and changing risk limits; keys to the trading gateway are stored in a secret vault, not in the repository.
- Operating instructions — launch order, “signals only” mode, and rollback to the previous checkpoint with one command.
- Retraining regulation — triggers (feature distribution drift, metric degradation), frequency, and the procedure for putting a new version into operation.
- Data documentation — feature dictionary, sources, and rules for handling gaps and corporate actions.
The point of the list is not reporting: using these files, a new team reproduces training and continues work without us.
Case: instrument portfolio, metrics before and after implementation
Before: policy on a single asset
In this case — a portfolio of twelve instruments: stocks, index futures, and currency pairs. The agent was trained on a single asset, and on transfer the policy skewed: up to 70% of weight was taken by the most liquid instrument. On the holdout segment of history, Sharpe stayed around 0.6, and maximum drawdown reached 28%.
After: what changed and how it was measured
We built a common policy for the entire portfolio: instrument features were added to the state — class, liquidity, correlation with the index — and the reward was made risk-adjusted with a concentration penalty.
Training used the actor-critic family (A2C/A3C): several parallel environments ran different history segments and volatility regimes, and quotes and order books were taken from Interactive Brokers.
Metrics were calculated only on the holdout segment and compared with an equal-weight portfolio. Sharpe 0.6 → 1.3, maximum drawdown 28% → 11%, share of time in market dropped from 85% to 60% — the agent waited more often. Instrument contribution leveled out: none provides more than a fifth of the result.
Separately, we ran stress scenarios: a liquidity gap, an opening gap, a prolonged sideways market, and a trend reversal. In each, the agent lost less than the baseline strategy, and recovery took less time.
How an RL agent for trading differs from a rule-based strategy
A rule-based strategy is an explicitly written condition: if the price broke a level, volatility is below the threshold, and the position is empty, we buy. The logic can be read in code and explained to the risk committee.
Reinforcement learning removes the condition and puts a policy in its place: the agent sees the market and account state, chooses an action, and receives a reward.
Thus, in algorithmic trading, a model appears that determines for itself which features are significant and which are noise.
Formally, the task is described as a Markov process. The state is a window of market features, the current position, unrealized result, and free funds. Actions are entry, holding, exit, and position size. The reward is the step's return adjusted for drawdown and costs.
An agent based on A2C/A3C learns through an actor-critic: the actor proposes a distribution of actions, the critic evaluates the promise of the state, and the gradient corrects the policy.
There is no ready-made “if — then” in the decision: the rule is replaced by parameters tuned on data.
The boundary is most noticeable on new data. A set of conditions selected on history breaks along with a regime change: the same thresholds in a trend and in a sideways market produce different quality, and they can only be rewritten by hand.
The agent transfers behavior if the features describe the regime, not a specific segment. But flexibility comes with its own failures: the policy degenerates into constant holding when the reward does not account for costs, or it is fitted to historical anomalies.
Therefore, training goes hand in hand with validation discipline. The reward includes commission and slippage, quality is measured on consecutive time segments rather than on a shuffled sample, and policy degradation under regime change is calculated separately.
The working scheme is a hybrid, not a replacement: the agent makes decisions, while hard rules remain as limits on position, drawdown, and trade frequency.
Technical doubts: overfitting, slippage, risk
Before launch, the client's engineering teams ask the same set of questions: how to make sure the model is not fitted to the past, how real order execution is accounted for, and who is responsible for risk in case of failure.
Below are the answers we verify on a prototype before industrial launch, rather than declare in words.
How to distinguish a working model from an overfitted one?
We split history by time, not randomly: training runs on one segment, validation on the next one the agent has not seen. We look not at final return but at the gap between these segments and at behavior under market regime change.
If the result holds under inflated execution costs and on instruments not involved in training, the model is robust. If it falls apart, we return to the feature set and regularization, not “tweak” it with fitting.
How is slippage and real execution accounted for?
The simulator executes an order not at the closing price but accounting for spread, commission, and the latency between decision and the order entering the order book. We add random latency and a limit on available volume so the agent does not count on a full fill. A policy that is profitable only on paper is filtered out at the validation stage.
What to do if trade history is short?
Short history is a statistical limitation, not a reason to wait. We expand the sample using other instruments and timeframes, and validate in a rolling window and on synthetic scenarios.
We honestly record that with a small sample some conclusions have wider dispersion, and at the start we rely on risk control and minimum position size.
Who and how controls risk?
Risk lives outside the agent. A separate layer of rules limits position size, daily drawdown, number of trades, and holding through news and weekends.
The agent proposes an action, but the order passes through a control module: if a limit triggers, trading stops rather than continuing in the hope of recovery.
How does connection to the trading system work?
Through the broker's API with a separate key and reduced permissions — without withdrawal of funds. First, there is a long run in observation mode: the agent generates signals, and we compare them with actual prices and real latencies.
Only after deviation is within tolerance do we enable minimum volume. Switching back to manual mode is one action.
Let's discuss the task and come back with an estimate and implementation plan
To come back with a project estimate and implementation plan, rather than general words, we need your picture of the market and infrastructure. Below is the minimum set that is usually sufficient; we will clarify the rest on a call and, if needed, request it specifically.
- Data: instrument, timeframe, history depth, tick data or bars, export source, and whether there are gaps in the series.
- Trading constraints: allowable risk per trade, maximum drawdown, position limits, and periods when trading is not conducted.
- Execution loop: own servers or cloud, availability of a test mode, how the gateway to the broker is arranged, and who manages access.
- What already works manually: entry and exit rules that produce results now — the training scheme relies on them more reliably than on blind search.
- Acceptance criteria: by what metric you consider the model successful — risk-adjusted return, share of profitable trades, behavior in a falling market.
- Responsible persons on your side: who is responsible for data, who accepts the model, and who approves the risk framework.
We will come back with a training scheme (environment, feature set, reward function, validation on a segment not involved in training), a stage-by-stage implementation plan with control points, and a model support regulation — drift monitoring, retraining, and manual trading stop.
Describe the task — we will come back with an estimate and plan.
