Why DQN trading breaks down in the real market
An agent trained on historical quotes produces a smooth curve on a holdout sample — and goes into drawdown in the first weeks of live trading.
This is not an implementation error: a deep Q-network has enough capacity to memorize specific price trajectories instead of the pattern behind them.
Market regime shift breaks the agent first. Volatility, liquidity, and microstructure change, while the replay buffer keeps returning transitions from the previous regime for a long time — the network continues choosing actions based on an already incorrect value function.
Signal latency is also at work: the observation arrives with a delay, execution shifts even later, and the reward is calculated for a state that no longer exists at the time of the trade.
Agent overfitting is masked as a successful tuning. If the reward is calculated from the historical curve without costs, the network learns data noise — spread, slippage, liquidity gaps — and on new data the strategy drawdown turns out deeper than any reasonable risk limit.
So robustness is built into the problem formulation itself: an environment with honest latency and costs, validation by a rolling window, regime features in observations, and a penalty for excess turnover. Below we break down how this is addressed in practice.
What changes after deploying a trained agent
After training, the agent works as a measurable decision-making loop: assessment of market state, action selection, position re-evaluation at the next step.
We calculate the trading agent metrics the same way — both on historical runs and on a live stream — to compare behavior before and after deployment.
Share of decisions without manual intervention
A DQN-based agent handles typical scenarios on its own: places and moves orders, reduces the position when volatility rises, exits when a specified risk is reached.
Trading decision automation brings only exceptions to manual control — a liquidity gap, a trading halt, a news shock. In the overall flow, the share of automatic decisions reaches 80–90%, while the operator switches from execution to monitoring deviations.
Strategy robustness and drawdown reduction
Training on multiple market regimes instead of one gives strategy robustness: the policy does not fall apart when a trend changes to sideways.
Drawdown reduction on the test segment is 1.5–2 times versus the baseline heuristic, and the reaction to a regime change is measured in model steps, not in minutes of operator waiting.
We keep quality control on reproducible scenarios: we run episodes in the Gymnasium environment and compare the agent's action distribution with the expected one.
The same set of regimes checks the policy both before release into the real loop and after a model update — metric divergence is visible immediately.
DQN trading: operating modes and limitations
An agent trained on one market character loses decision quality when volatility and liquidity change. Therefore, the training and execution loops are separated: the policy is selected in a sandbox on historical data, and only a frozen version enters the real loop. A mode is a configuration parameter, not a separate build.
| Mode | When it fits | Limitations |
|---|---|---|
| Historical sandbox | Initial setup, hypothesis testing | The policy adapts to past market regimes; slippage and execution costs are modeled approximately |
| Incremental training on a data stream | Market character has changed, adaptation is needed | Risk of forgetting previous regimes, instability on short samples |
| Shadow run alongside the production loop | Check before releasing the agent into operation | Signals are not executed, evaluation does not account for actual trade quality |
| Operation with risk limits | The agent has been released into the real loop | Reinforcement learning optimizes average reward, not the worst-case scenario |
A general limitation of reinforcement learning is that the reward described in the environment does not equal the business goal: the agent honestly maximizes what it was given.
Therefore, a risk-limits layer sits on top of the policy, and the agent state is saved via checkpoints: rolling back to the previous version takes minutes and does not sacrifice accumulated statistics.
Stages of designing a trading agent
We structure the workflow so that an error becomes visible before it turns into a losing strategy.
First, we lock down the goal: what we consider the result with commission and slippage included, what drawdown we are prepared to withstand, and over what horizon we hold the position.
- Historical data audit. We check the completeness and consistency of the series: gaps, session breaks, corporate events, outliers after publications. We exclude low-liquidity instruments — the agent should not learn from prices that never existed in the market.
- Reward function design. We formalize what the agent optimizes: risk-adjusted return, penalty for holding a position, limit on invested capital. An error here is the reason why training looks successful while trades are unprofitable.
- Training environment design. We assemble states, actions, and transitions so that real conditions are replicated: spread, commission, execution latency. The environment is reproducible — the same run gives the same result; otherwise, agent versions have nothing to compare against.
- Training with overfitting control. We keep part of the history aside and use it only for validation. This shows where the agent learned a market pattern and where it memorized a specific segment.
- Agent validation. We run the model on a holdout period and on segments with different volatility: trend, sideways, sharp reversal. We look at drawdown and behavior under slippage above the usual level.
- Model handover. We provide a reproducible run, environment configuration, network weights, and a metrics report. We describe how to retrain the agent on fresh data and when to stop it from operating in the live market.
Configurations, reports, and agent documentation
A trained agent without a supporting package is a black box: it can neither be reproduced, nor safely updated, nor explained to audit.
Therefore, handover includes not only the saved DQN-based network but also the entire surrounding scaffolding — from training environment configuration to the retraining and agent monitoring procedure. Below is the list of what remains with the client's team.
- Training environment configuration — description of market state, instrument set, forecast horizon, and reward function; frozen files by which any run is reproduced by version number, not from an engineer's memory.
- Agent documentation — network architecture, action space, quote preprocessing, environment dependencies, and startup procedure: a new engineer can bring up the testbed in one working day.
- Solution diagram — data path from the quote source to the order, control points, and responsibility boundaries between the agent and the trading loop.
- Retraining procedure — triggers (volatility regime change, drift in return distribution), validation window, acceptance criteria, and rollback procedure to the previous weight snapshot.
- Agent monitoring — online metrics: drawdown, return on risk, share of rejected trades, divergence between historical and real-time evaluation; thresholds and notification procedure.
- Training and validation reports — run log (reward, number of trades, position holding time) and a historical data report broken down by market regimes.
- Limitations map — modes where the agent is not applied: low liquidity, data gaps, anomalous gaps, as well as position size limits.
Case study: agent during a market regime change
The original agent was trained on one regime — a calm trend with low volatility. On history it looked robust: Sharpe ratio around 1.8, share of profitable trades 58%.
When transitioning to a high-volatility sideways market, the agent continued holding trend positions: drawdown rose to 34%, the share of profitable trades fell to 41%, and on rare gaps a few trades produced most of the losses. The network did not overfit — it confidently acted in a world that no longer existed.
What was changed
The state space was expanded with regime features: rolling volatility, spread, and a trend/sideways label from a hidden Markov model that catches a regime change earlier than it is visible in price.
A penalty for holding a position against the current regime was added to the objective function, and the replay buffer was prioritized for rare events: gaps now weigh in training as much as their real contribution to risk.
Training ran through Ray RLlib on an ensemble of historical windows, and double value estimation removed action overestimation.
What resulted
Drawdown on transitions decreased from 34% to 14%, and reaction to a regime change went from six hours to forty minutes of holding a position. The share of profitable trades on data the agent had not seen stays in the 52–55% range in both regimes instead of falling to 41%.
The worst single trade on a rare event became 2.5 times smaller — this was exactly the class of failures that needed to be closed.
The comparison was calculated on identical periods with sequential retraining at each window, plus stress tests on synthetic liquidity gaps and a decision log for each trade.
Before-and-after metrics were obtained using the same methodology — therefore they can be compared directly.
How does a trained agent differ from manual rules?
A manual rule is an explicit function of indicators: crossing, threshold, slope. Its behavior can be read by eye, and that is its strength: it is predictable exactly to the extent that the market regime it was written for is predictable.
A reinforcement learning agent does not receive thresholds — it learns from "state — action — reward" transitions and selects a policy that maximizes accumulated reward while accounting for trade costs.
The first difference is what exactly is optimized. A set of rules describes entry conditions but rarely ties them to position size, exit timing, and spread cost in a single decision.
A DQN agent estimates the expected value of each action in the current state, so holding, adding, and exiting become parts of one policy rather than independent conditions.
The second is behavior on data that was not in training. A rule in a new market regime usually does not break but quietly stops working: more signals, lower quality, visible in drawdown.
An agent under a state distribution shift can degrade sharply, so it is released with input controls, action constraints, and rollback to the previous policy version.
The third is maintenance. Edits to rules conflict: a new condition fixes one area and breaks a neighboring one, and regression is searched for manually.
With an agent, the logic lies in parameters, so a change is retraining on an expanded set with a fixed split, validation on a holdout period, and comparison of versions by robustness rather than by a single return curve.
Explainability is provided not by a formula but by analyzing actions by state and checking sensitivity to features.
Technical questions before starting work
Before starting, three technical doubts come up: where to get data, how to make sure the DQN agent has not memorized history, and who is responsible for risk in the account.
Below is how we address each with engineering procedures, without promises of "guaranteed returns".
Will there be enough historical data, and how do we check its quality?
We bring quotes from Interactive Brokers and internal archives to a single grid: align closing time, restore gaps, and mark periods of low liquidity.
DQN learns from "state — action — reward" transitions, so one shifted bar breaks the chain — the agent learns a pattern that did not exist in the market.
How do we know the agent has overfit?
Overfitting in reinforcement learning looks like improving performance on the training segment while failing on the holdout. We keep a strict time boundary, validate with a rolling window, and compare behavior across different market regimes: trend, sideways, sharp pullback.
How often is retraining needed?
We launch retraining not by calendar but by signal: the feature distribution has shifted or the instrument's volatility has changed.
The new version goes through the same guardrail checks as the first one and is compared with the current one — otherwise an improvement in one regime will nullify the result in another.
Who is responsible for risk?
We are responsible for data, architecture, training reproducibility, and the correctness of validation.
Trading constraints — position limit, daily stop, ban on new entries during drawdown — are agreed with you and live in the execution loop separately from the model, so an agent failure does not turn into uncontrolled drawdown.
Let's discuss the task and come back with a plan
Before writing code, we break down the task at an engineering level: where the quote stream comes from, how the reward function is formed, and which risk constraints the agent must not violate.
Architecture depends on this — one agent per instrument or a shared one for the portfolio, and where the boundary between training and execution lies.
- What we clarify: depth of quote history, decision frequency, features available at each step without lookahead.
- How the simulation environment is structured: commissions, slippage, partial order execution, behavior during liquidity gaps.
- By what metric we calculate the result: risk-adjusted return, drawdown, robustness across different market regimes.
- What we consider proof: the agent passes a rolling validation on holdout periods and does not fall apart on shifted data.
- How the agent is embedded into the execution loop: first simulation without sending orders, then position and daily loss limits.
- What we need from you: access to historical data, description of the trading loop, and risk management requirements.
We will come back with an estimate for the agent project and a work plan: stages, order of validation, and points where you see an intermediate result. Describe the task — we will come back with an estimate and a plan.
