PPO trading: why an agent loses robustness in the live market
A trained agent almost always looks convincing on historical data: the return curve is smooth, drawdowns stay within set boundaries, and metrics on the holdout segment match the training ones. The problem is that the market is not stationary.
Volatility, liquidity, and relationships between instruments change, while the policy found by reinforcement learning remains tuned to the past price distribution.
PPO updates the policy in small steps, limiting deviation from the previous version — this stabilizes convergence but does not protect against overfitting of the agent.
The reward function is tailored to a specific historical segment, and the model begins to reproduce its peculiarities: rare spikes, seasonal effects, noise during narrow hours.
When the market regime changes, the strategy's drawdown exceeds the limits shown by any historical backtest run.
The causes are usually engineering-related, not that "the market has changed." Features are calculated with a lag relative to the decision moment; execution costs and slippage are accounted for coarsely; the validation segment is too similar to the training one in volatility.
Libraries such as Stable-Baselines3 provide a working training loop, but the choice of environment, the historical split, and regime checks remain the team's responsibility.
Therefore, the decisive argument is live testing: the agent runs on real quotes in real time with a limited trade volume, and we compare the expected policy behavior with the actual one. The discrepancy is visible only here.
Next, we break down how this is addressed: which robustness checks we introduce and how we transfer the agent from history to real trades.
Agent robustness, risk control, and reproducibility of the result
An agent trained on a single market regime loses robustness when volatility changes: the policy memorizes the statistics of the period rather than the logic of behavior. That is why we train with a rolling window and at each step validate the model on data it has not seen.
We measure robustness not by average return but by drawdown on these slices — if it grows faster than on the training window, we stop training and investigate the cause.
Risk control is built into two points. Into the reward function — a penalty for variance and for exceeding the position limit, so that the agent does not earn through rare large bets.
Into execution — hard constraints on top of the policy: share of capital per trade, daily drawdown stop, and strategy shutdown when the threshold is breached, regardless of the current forecast.
We maintain reproducibility at the infrastructure level: we fix the random seed, the composition and order of market data, environment parameters, and hyperparameters — all in the repository next to the code.
A repeated run on the same dataset produces metrics within a specified tolerance, and a discrepancy is read as a configuration error rather than as "the market has changed." We use Stable-Baselines3 as the reference PPO implementation to separate the effect of our modifications from the effect of the library itself.
The output is a set of quality metrics with history: maximum drawdown and recovery time after it, share of shutdowns due to risk limits, and dispersion of results between runs.
From these, it is clear where the agent is robust and where an additional filter is needed — this is managed risk, not a promise of returns.
Options for building the agent: training from scratch, fine-tuning, hybrid
The choice of mode determines how much data will be required at the input, how quickly the first working policy will appear, and how painful a change in the instrument set will be.
Training from scratch with PPO gives full control over the agent's behavior but requires a long price history and tolerance for unstable returns in the early iterations.
Fine-tuning the model starts from an already assembled policy: it is noticeably shorter in time to result, but the agent inherits biases from previous training and easily "forgets" regimes that were rare in the new data.
A hybrid scheme removes some of these risks by distributing roles between rules and the learnable part.
| Option | When it fits | Limitations |
|---|---|---|
| Training from scratch | Your own instrument set and holding horizon that are not available in ready-made datasets | Long start, sensitivity to risk penalty tuning, overfitting to the past with a short history |
| Fine-tuning the model | There is already a working policy; you need to add an instrument or a new market regime | Transfer of old biases, risk of degradation on previous regimes, a control run on a holdout segment is required |
| Hybrid scheme | Rules set the risk framework; the agent allocates capital within it | More code and checks; analysis of each layer's contribution to the final result is needed |
The choice of mode begins with an audit of the data and the target metric: where the boundaries of acceptable drawdown are, how often the instrument set changes, and whether there is something with which to validate the policy outside the training segment.
From these answers, it is clear whether fine-tuning is enough or a hybrid scheme is needed — and the scope of work is already planned around it.
How the work is structured: data audit, scheme, tuning, validation
The process proceeds in stages, and each has a control point: you see exactly what has been checked and what conclusions follow. Training starts only after the data and task statement are fixed — otherwise the metrics cannot be interpreted.
- Data audit. We check the continuity of quotes, align intervals and time zones, and look for gaps, outliers, and breaks around corporate events. Separately, we exclude survivorship bias: the sample retains only instruments that were trading at that time.
- Task statement. We fix the objective — risk-adjusted return or drawdown control — as well as the horizon, the set of observed features, and limits on volume and leverage.
- Execution environment and reward. We build a simulator with commissions, slippage, and order latency. The reward penalizes excess turnover and abrupt rebalancing so that the agent does not overfit to ideal execution.
- PPO training tuning. We select the advantage horizon, clipping threshold, batch size, and learning rate. Several runs with different initial seeds show the dispersion, not the best result.
- Robustness validation. We run the agent on a holdout period and in regimes that were not in training: trend, sideways, volatility spike. We measure degradation under increased slippage and latency.
- Solution handover. We hand over the environment and training code, run configurations, saved weights, fine-tuning rules on fresh data, and a report with metrics and applicability boundaries.
Control points are after the data audit, before training starts, and before handover: at each one, we fix what result we consider acceptable.
Handover composition: agent code, configurations, documentation, regulations
The handover is not an archive with code but a working loop that your engineers run and maintain without us. Each artifact covers its own class of tasks and is checked at acceptance: the PPO agent completes training and enters a test market segment on your infrastructure.
- Agent code — training and policy application: environment with market data, policy and value model, risk filters, order plumbing. It is brought up according to instructions in a clean environment.
- Training configurations — horizon, policy step limit coefficient, sample size, number of parallel environments, features, randomness fixation. A repeated run reproduces the same policy.
- Run logs and metrics — reward, return, and drawdown curves, trade frequency, distribution of agent actions. From them, it is clear whether the model degraded or the market regime changed.
- Schemes and decisions — data flow from quotes to order, risk filter logic, branching points, and boundaries between modules.
- Documentation — description of the environment, observations, and reward function, risk constraints, and the procedure for fine-tuning on new data.
- Maintenance regulations — criteria for stopping trading, retraining procedure, rollback to the saved state of the previous model.
- Access rights — roles and keys to the data store, model registry, and broker loop are issued to you; our access is revoked.
Acceptance — your engineers themselves deploy the loop and repeat the training run. After that, maintenance follows the regulations, without contacting us.
PPO trading under load: metrics before and after implementation
What was before implementation
The baseline version of the PPO agent was trained on a single market regime — a trending period with low volatility.
When the regime changed, the policy continued to increase positions in the same direction: the drawdown reached 34% of the calculated capital, and recovery took twice as long as the decline itself.
Risk control was outside the model — hard limits cut positions after they were opened, so the agent learned on one reward distribution but traded in another.
What was changed
The risk layer was moved inside the training environment: constraints on exposure and position change step became part of the reward function rather than an external filter.
We added regime detection based on volatility and return — when the regime changes, the policy switches to a protected profile instead of continuing the trend.
We trained and validated on historical data from the public Binance archive, separately running sharp reversal scenarios.
What became after
Maximum drawdown decreased from 34% to 11% over the same testing horizon, and recovery after the decline accelerated approximately threefold. The return-to-risk ratio increased from 0.8 to 1.5 — the agent's robustness no longer depended on the current market regime.
The number of position changes fell by 40%, execution costs no longer eat into the result, and risk control became part of the policy: the agent does not exceed risk limits in any session.
How does an agent differ from a strategy defined by manual rules?
Manually defined rules are a set of conditions: moving average crossover, volatility threshold, stop level. Their strength is predictability; each decision is explained by a single phrase.
Their weakness appears when the market regime changes: parameters selected on a trend break in a flat market, and rebuilding turns into manual enumeration of options that can easily be overfit to the past.
An agent for PPO trading is structured differently. It does not store conditions but trains a policy — a probability distribution over actions, maximizing expected return with penalties for risk.
A continuous action space allows setting position size, not just entry and exit, and accounting for dozens of features in combinations that a human would not write out as rules.
Learning wins where dependencies are multidimensional and dynamic: position size, simultaneous consideration of correlations, order book microstructure, and multiple holding horizons.
Fine-tuning on a fresh data window allows faster reaction to a regime change, but only if a similar shift has already occurred in history — otherwise the agent extrapolates noise just as confidently.
The agent loses in interpretability. Network weights are not read like a rule, so explaining to a risk committee why a position is open is harder — a separate analysis of feature contribution is needed.
Requirements for data and the reward function are also higher: a hard penalty for drawdown makes the agent passive, while a soft one provokes leverage that will not survive real costs.
In practice, this is not a choice between one of the two. Rules remain the risk framework — position limits, stops, a ban on trading in certain windows — while the agent works within them, where there are too many rules for manual description.
Both schemes are validated in the same way: on a holdout period, accounting for commissions and slippage, with separate evaluation on a trend and in a flat market.
Technical doubts: robustness, data, risk control
Five technical doubts are most often resolved before work starts: whether the agent will withstand a market regime change, whether there is enough data, where the risk boundaries lie, what happens with maintenance, and how to validate the result. Below is a brief note on each.
Robustness: what if the market changes character?
We validate not on a single historical segment but on a series of windows, including sharp moves and low liquidity. We consider the agent suitable if its behavior persists when the regime changes, not only on a favorable segment.
Data quality: is there enough history?
We look at completeness, frequency, and gaps in quotes, and the correctness of corporate events. We separate training and validation data in time — otherwise we get a beautiful but useless result.
Risk constraints: who enforces the framework?
We build limits on position, drawdown, and number of trades into the training environment — the agent learns within the framework. Additionally, we set up external control that triggers independently of the model's decisions.
Maintenance: what happens after launch?
The model loses accuracy as the market changes, and this is normal, not a failure. Therefore, we determine in advance which metrics we monitor, at what deviations we retrain, and how we roll back to the previous version of the agent.
Validation methods: how to be sure before launch?
Three levels: a test on a holdout period, a run under stress scenarios, and operation at small volume in the live loop. Agreement of results across all three is the condition for moving to full scale.
Let's discuss the task: what is needed for assessment and a work plan
To assess a PPO trading task, you do not need the full archive of the trading system — a description of the environment and the objective is enough.
The more precise the input data, the fewer assumptions in the work plan: it is immediately clear where the simulator will match real execution and where separate validation will be required.
What to send so that we return with an assessment:
- instrument and horizon: what we trade, on what timeframe, how often the agent makes a decision;
- historical data: depth of the series, composition of fields (prices, volumes, order book), how gaps are marked;
- target metric: return adjusted for drawdown, Sharpe ratio, or share of profitable trades;
- cost model: commissions, slippage, execution latency — without them, the agent will train on nonexistent returns;
- risk constraints: maximum drawdown, position limit, trading stop rules;
- infrastructure: where data comes from, where the agent sends signals, whether there is a test loop;
- what you have already tried and why the result was unsatisfactory.
Based on this data, we assemble a work plan: environment preparation and history recalculation, reward function tuning, policy training and validation, holdout-period validation, and stress tests on volatile segments.
We fix the stages and timelines together with acceptance criteria — at what result we consider the agent ready for a limited launch.
Describe the task in free form — we will return with an assessment of the task and a stage-by-stage work plan.
