SAC trading: why the agent loses robustness on new data
An agent trained with SAC often shows a smooth curve on historical data — and loses it in the first weeks of live trading.
It's not about the architecture itself: the entropy-regularized algorithm is designed to explore the environment, but when training a trading agent the environment remains fixed — one period, one set of volatility regimes.
When a market regime shift occurs, the return distribution on which the policy was built no longer matches the current one.
The agent keeps choosing actions optimal for the past: the share of trades where entry is justified only by the logic of the old period grows, and the unstable strategy turns into a series of agent drawdowns.
Worst of all, the policy is trained on a narrow range of states and outside its boundaries behaves almost randomly — and that is exactly where live trading happens.
The second reason is the evaluation methodology. If quality is measured by a single run on a holdout segment, the model that fit it better comes out higher, not the one that is robust.
Several independent training episodes with different initial conditions and testing across different regimes reveal this before the discrepancy becomes visible in a live account.
SAC trading is robust exactly to the extent that the training environment is diverse. Data covering a trend, range, and volatility spikes, a penalty for abrupt position changes, and an explicit risk constraint yield a policy that does not fall apart when the regime changes. We break down how this is addressed in the problem statement and in training.
Continuous position management: what it changes for a strategy
An agent with continuous position management outputs not an "entry/exit" signal but a share of capital in the instrument.
It changes smoothly, following the model's confidence, so there is no gap between "in the market" and "out of the market" — and that is exactly where risk management is usually lost. This is fundamentally different dynamics compared to discrete signals.
Drawdown reduction is a consequence of the position size being cut earlier than a protective stop would trigger.
We measure drawdown amplitude on a rolling window and look at behavior during sharp moves: the depth of the decline shrinks, and recovery proceeds without manual intervention.
The strategy's robustness to regime change rests on entropy regularization — the foundation of Soft Actor-Critic. The agent continues to explore and does not get stuck in one behavior when volatility and instrument correlations change.
We test this on regimes that were not present in training: what matters is not average quality but behavior at the boundaries.
Reproducibility of training is a separate workstream.
We fix the order of episodes, the composition and version of historical data, the configuration, and the random seed: a repeated run yields the same set of model parameters, not "roughly the same." Then changes are tested one at a time and the result can be explained, rather than retraining blindly.
We separately capture a class of failures rarely visible in final reports: opening price gaps and liquidity breakdowns. The position is reduced before the event materializes, so drawdown remains within the limits set during training.
Comparison of approaches to training a trading strategy
The approach is chosen based on which signal actually distinguishes profitable trades from losing ones, not on the algorithm's "power." If the response lives in position size, a continuous action formulation is needed; if the decision boils down to entry, exit, or pause, the discrete one trains faster and more stably. Below is a working comparison we use to choose the setup for a specific instrument and horizon.
| Option | When it fits | Limitations |
|---|---|---|
| Continuous actions (SAC) | Position size is part of the strategy, there is a liquidity and slippage model | Sensitive to noise and execution delays, trains longer, critical to the reward shape |
| Discrete signals | The decision is binary: entry, exit, or pause | No risk management via volume, many false entries in a range |
| Training on trade labels | There is a history of trades with quality assessment | Does not transfer to market regime change, inherits labeling bias |
| Rules and heuristics | Transparency, reproducibility, and auditability of decisions are needed | Do not adapt to new volatility, require manual revision |
| Hybrid | Rules set the risk framework, the agent manages the position within it | Harder to validate: both the rules and the agent's behavior are tested |
We check compatibility before training: how the approach fits current execution, fees, and delays, and what happens if the model fails.
A hybrid scheme usually provides a controllable start — the rules prevent the agent from exceeding limits while the strategy accumulates statistics. Next we break down how a working pipeline is assembled from this.
Work stages: from problem statement to agent handover
Each stage of agent implementation ends with a verifiable artifact: a metric description, a prepared dataset, an environment configuration, a validation protocol.
We do not change the order — if you skip data preparation or holdout validation, agent quality will rest on a lucky run rather than a reproducible result.
- Problem and data review. We clarify instruments, position holding horizon, costs, and liquidity constraints; we fix the metric by which we measure the result.
- Market data preparation. We align timeframes, handle gaps and outliers, and split history into training, validation, and holdout periods — without leaking the future into the past.
- Training environment setup. We describe the market state and the agent's permissible actions, set the reward function adjusted for risk and costs — on this basis SAC learns to manage the position continuously rather than select signals.
- Training and parameter tuning. Several runs with different settings, control for overfitting to a single market regime, selection of robust configurations.
- Holdout validation. A run on data the agent has not seen, accounting for fees and slippage; we compare with the baseline and examine behavior in calm and volatile markets.
- Launch preparation and solution handover. Position and drawdown limits, automatic braking, integration with the brokerage pipeline, then code, configurations, documentation, and a retraining procedure with a rollback order.
Solution handover: training code, configurations, procedures
At the output, the project is closed not with a demonstration but with a package by which your team can retrain the agent, verify any of its decisions, and stop it at any moment.
Everything we did during the development of the SAC agent is handed over as source code, configurations, and written procedures — without verbal agreements that leave along with the contractor.
- Training code and launch scripts: a reproducible run on a clean machine — from data preparation to saving weights.
- Environment and reward configurations: state and action space, fees, slippage, risk penalties. Change a parameter and see how the agent's behavior changes.
- Experiment log: runs with hyperparameters, return, drawdown, random seed, and a link to the artifact — the best configuration is restored exactly.
- Risk control rules: position and daily loss limits, automatic stop conditions, manual veto point.
- Retraining procedure: triggers (data drift, metric degradation), history window, validation and new version release order.
- Agent documentation: features, model application scheme, logging points, analysis of typical failures.
- Maintenance and rollback instructions: return to the previous artifact and launch the agent in shadow mode.
- Acceptance checks: a holdout run and comparison with a reference strategy — we fix acceptance criteria before handover.
Case: strategy behavior before and after agent rework
A SAC agent was already trading a basket of liquid instruments on Moscow Exchange, but the result held up in only one market regime.
The reward function accounted only for step return, so the agent willingly took on risk, looked convincing in trending sections, but in a prolonged range gave back the accumulated gains: strategy drawdown reached a third of capital, losing streaks — up to 13-14 trades in a row.
What was reworked
We kept the architecture and changed what shapes behavior. We added a drawdown penalty and a term linking return to volatility to the reward — the agent began to treat risk as part of the result rather than a side effect.
We added scenarios from different regimes with priority for rare ones to training, limited position size based on current volatility, and prohibited entries in a range without confirmation.
After the rework, drawdown decreased from 34% to 11%, the share of losing trade streaks — from 47% to 22%, and the return-to-risk ratio roughly doubled.
The number of trades fell by a third: lower turnover means lower costs and fewer reasons for error, while the result over the same period did not decline.
Robustness was tested separately — on data not involved in training, with a regime change from trend to range and back.
The agent began to reverse less often on noise and hold a position in its direction longer, the dispersion of results between regimes decreased — that is what the rework was for.
How continuous management differs from discrete
Continuous management — the agent sets the position share at each step: 0.4, 0, minus 0.7. Discrete signals — a finite set of events: enter, exit, hold.
Hence the first difference — sensitivity to execution delay: the continuous pipeline recalculates the target constantly, and a stale quote turns into a mismatch between desired and actual position.
The discrete scheme forgives delay — the signal lives until the next event, not until the next observation. The price for this is coarse control: the agent cannot partially reduce risk; during an adverse move, either the full position remains or there is a full exit.
The rebalancing frequency is also different: a continuous policy changes the portfolio at almost every step, and the fee becomes part of the reward function rather than an external factor.
| Option | When it fits | Management limitations |
|---|---|---|
| Continuous actions | Liquid instrument, smooth quote flow | Sensitivity to delay and slippage, noise in the reward signal |
| Discrete signals | Rare entries, wide thresholds, expensive trade | No partial risk, profit is lost on reversals |
| Hybrid approach | A discrete trigger sets the regime, the policy sets position size | Conflict between rules and policy during regime change |
The hybrid approach separates roles: the rules layer decides whether we trade at all, the continuous policy decides how deep. Data requirements are higher here — a smooth stream of observations without gaps is needed, otherwise the policy behaves unpredictably at breaks.
In SAC trading, a continuous pipeline is usually built: the algorithm works with a distribution of actions and penalizes excessive confidence, which produces a smoother return curve, while the Stable-Baselines3 library covers the basic training scaffolding.
Limitations of the approach and technical concerns
Doubts about reinforcement learning arise not at the prototype stage but when the agent goes live on real data and real orders. Below is what engineers check first.
Is there enough historical data for training?
SAC learns from state-action-reward tuples, and data quality matters more than volume here. We fix gaps and outliers in quotes, align sources to a single time grid, mark periods of low liquidity, and ensure features do not peek into the future.
With a short history, the agent is first trained in a simulated environment and only fine-tuned on market data.
How do you know the agent has overfit?
Overfitting is visible from the divergence between training and holdout results, so validation proceeds by time, not by a random slice. Limiting network complexity, regularization, and early stopping on an out-of-sample metric help. An agent that wins only on the historical segment does not go into production.
What about execution delay?
There is always a gap between signal and trade, and the training environment must live with the same delay as the production pipeline. We build slippage, fees, and broker response time into the simulator, and limit the frequency of position changes. This way the model learns on execution close to real.
Can the agent be integrated into an existing system?
The agent outputs not ready-made orders but target weights that pass through your risk module and brokerage gateway: position limit, daily stop, automatic shutdown when metrics leave the corridor.
Before capital is deployed, the model runs in shadow mode on a live stream, and rollback means returning to the previous version of the agent.
Let's discuss the task and come back with a work plan
To make the project estimate rest on facts, not guesses, we need data and an understanding of the environment's constraints. Data audit is the first step: we look at history depth, gaps, market regime changes, and how features behave at different horizons. Only after that do we state the timeline, stage composition, and risks.
- Historical quotes and trades: instruments, minimum price increment, depth, and periods of sharp volatility without fail.
- Composition of the observable state: which features the agent sees at each step and how often they are updated.
- Target metric and risk boundaries: what we consider success, what drawdown is acceptable, what position limits apply.
- Execution constraints: trading windows, available instruments, prohibitions on short positions and leverage.
- Method of outputting decisions: where the agent sends the signal and how trade execution is confirmed.
- Emergency stop rules: under what conditions trading stops and what happens to open positions.
- A responsible person on the client side who approves disputed decisions on the reward function.
The first discussion of the task takes about an hour and follows your domain, not a template.
At the output you receive a project estimate and a work plan by stages with verifiable results at each: data collection and cleaning, training environment, baseline comparison strategies, SAC policy training, holdout validation.
Describe the task — we will come back with a plan and the boundaries of the solution's applicability.
