Custom gym environment: where strategy training breaks down
An environment with a Gymnasium interface is a market model on which a strategy gains experience: it returns an observation, accepts an action, and returns a reward. If the model is simplified, reinforcement learning converges to a solution that the real market does not confirm.
The error manifests late—already on a live account, when overfitting cannot be rolled back with a couple of parameters.
The first point of failure is synthetic quotes. Candles in the market simulator are assembled from the last trade, while an order is executed at its own price: the spread, broker response delay, and partial execution diverge. A strategy trained on "ideal" execution loses on every entry on real data.
The second is liquidity. In the environment, the agent buys any volume at the closing price, so a large position looks free. In reality, volume moves the price, and a limit order may not be filled at all—slippage and the order queue are simply not described in the model.
The third is the reward function and future leakage. If returns are calculated without commissions, financing, and position holding, the agent chooses schemes that are entirely eaten up by costs.
Normalization across the entire dataset and random shuffling of series mix the future into the features: test metrics look excellent, while the divergence between tests and reality becomes systemic.
We examine how this is addressed: exactly what needs to be reproduced in a custom gym environment so that a trading strategy transfers to the production loop without surprises.
Effect: convergence, robustness to failures, speed of checks
Before moving a strategy into the live loop, we define which shifts we consider evidence: training convergence, robustness to market anomalies, hypothesis testing time, and risk metrics.
The checklist below is what is measured from the custom gym environment before and after switching to it: an item that cannot be expressed as a number does not make it into the report.
Comparison makes sense only with the same set of random numbers and the same slice of market data. Separately, we check reproducibility of a repeat run: if the results diverge, part of the logic remained outside the environment and is not visible in the report.
| Metric | What we record | Shift after switching |
|---|---|---|
| Training convergence | number of episodes until reaching a plateau in returns | 2–4 times shorter |
| Robustness to market failures | drawdown during liquidity gaps and anomalous candles | anomalies go into training, not only into validation |
| Hypothesis testing speed | time from logic change to a finished report | hours instead of days, runs in parallel and without manual exports |
| Strategy risk metrics | maximum drawdown, share of losing trades, position holding time | calculated on every run and included in the report |
A class of failures that is the most expensive to fix goes away: divergence between behavior on historical data and in the live loop, manual stitching of exports, and lost intermediate runs.
When the environment is the same for both training and validation, a hypothesis is accepted or rejected in the time of a run, not in a week.
Environment formats: full simulator, trade history, hybrid
A custom gym environment is built around one of three modes, and it determines which actions the agent can explore at all. A full simulator provides a response to any action, but requires its own model of how a trade affects the market.
Trade history reproduces real data accurately, but does not let the agent go beyond what has already happened. The hybrid mode removes some limitations of both, but is harder to configure.
They choose by task, not by default.
If you need to assess a strategy's impact on liquidity and execution costs, take the full simulator; if the goal is to refine existing behavior, take trade history; to search for new scenarios with validation on validated data, take the hybrid.
| Mode | When it fits | Level of realism | Limitations |
|---|---|---|---|
| Full simulator | Testing position size, limits, and market reaction to agent actions | Controllable, set by the market model | The model diverges from reality; calibration and validation are required |
| Trade history | Reproducing and refining behavior from actual data without risk | Maximum within the recorded window | New actions cannot be tested, only scenarios that occurred |
| Hybrid mode | Pretraining on history and fine-tuning in the simulator, testing new scenarios | Medium, with control of divergence between loops | Harder to reconcile both loops: a unified format of observations and actions is mandatory |
Realism is set not by the mode itself but by a set of costs: spread, execution delay, partial execution, order rejection. If they do not match actual trades, the agent will train on an ideal market and fall apart on the first real trades.
That is why we run the environment on a historical window and compare the result with records—the divergence shows what to calibrate.
Work stages: market model, agent logic, hypothesis testing
A custom Gym environment is a formalized description of your market: what the agent observes, what it controls, and by what rule it receives feedback.
The work proceeds from your data and regulations, not from a ready-made template, so the route starts with an inventory and ends with a reproducible testbed that your team launches and develops on its own.
-
Customer data review. We check history depth, gaps, time zone shifts, and different source frequencies. Separately, we look for future leaks—because of them, the agent later "predicts" an already known result.
-
Market model. We define the composition of the observation vector, time step, commissions, slippage, and position limits. The rules become part of the environment, not a comment in the report.
-
Agent logic. We define the action space: continuous position size or discrete levels, risk constraints per episode, and the episode termination condition.
-
Reward setup. The reward function is assembled from risk-adjusted returns and penalties for excessive activity—otherwise training converges to a single lucky trade.
-
Hypothesis testing and baseline. First, we run simple policies: holding, uniform entry, random action. Without this benchmark, it is impossible to tell whether the agent adds anything beyond noise.
-
Stress checks. We run the environment on synthetic regimes: volatility spikes, liquidity gaps, execution delays. We look at the dispersion of results across episodes, not just the average.
-
Handover to the team. We deliver code with tests for step correctness, a pinned environment, a description of environment extension, and the procedure for repeating experiments on your data.
Code, configurations, and the hypothesis testing procedure
The environment does not stay on our side: upon completion of the work, the customer receives the same set of artifacts that the engineers worked with.
This is not a demo script but a reproducible loop—it can be handed over to an internal team, deployed in your own environment, and used to continue training without us.
- Environment code: observation space, simulation step, reward function, terminal state handling—with comments for each module and extension points.
- Training configurations: agent parameters, episode lengths, cutoffs, random seed sets—edited in files, without rebuilding code.
- Market schema: instruments, timeframes, bar formation rules, accounting for commissions and slippage by which the reward is calculated.
- Data access policy: quote sources, storage method, roles and permissions, composition of log entries.
- Testing procedure: order of runs—validation, holdout period, stress scenarios—and formal criteria by which a hypothesis is rejected.
- Environment documentation: starting from scratch, interface descriptions, common errors, and environment limitations.
- Reference agent and saved runs—a baseline for comparing new ideas.
Along with this, we hand over a decision log: which hypotheses were tested and why they were closed. The procedure defines what counts as an improvement and what counts as fitting to history, so a new version of the strategy passes the same gates as the first.
Case study: how we removed the divergence between tests and live trades
Before: validation did not match execution
The strategy was validated on history with ideal assumptions: an order was executed instantly and in full at the desired price, and commissions were accounted for on average.
In live trading, the same models produced a divergence in daily returns of 30–40%, and only six or seven prototypes out of ten made it to a real account. The custom gym environment smoothed execution, and the agent learned on conditions that do not exist in the market.
What we changed and what it delivered
We rewrote the environment on Gymnasium: in the agent step we added order confirmation delay, partial fills by order book levels, commissions and slippage as a function of volume, position limits, and connection loss with recovery.
The reward penalizes abrupt portfolio reallocations, and a fixed random seed makes the run reproducible—rejection happens for a reason, not by random chance.
- Divergence between the testbed and live execution—from 30–40% to 4–6%.
- Rejection at the testbed—from three or four to eight prototypes out of ten.
- Solution preparation—from several weeks to 5–7 days per iteration.
- Stress testing became regular: liquidity gaps, exchange outages, and price gaps are played out before launch.
The testbed stopped being a formality: strategies are compared with each other under identical conditions, and the decision to move into trading relies on the numbers obtained in the same place where the models learn.
What to do if training is already running in a ready-made environment?
A ready-made environment is convenient at the start, but its dynamics are foreign: transitions, observation scale, and reward function are set by the authors for their task.
A policy trained in it inherits these assumptions, so transferring to your environment is not copying weights but reconfiguring. Let us break down what is preserved and what has to be retrained.
More is reused than it seems: the network architecture and feature scheme, selected search parameters, episode evaluation code, and the harness for runs and logging.
All of this does not depend on the transition model and transfers almost without edits—this saves the bulk of the time that would otherwise go into repeated selection.
The policy itself and normalization will have to be retrained. The scale of observations and rewards in the new environment is different, so statistics are collected anew; otherwise training diverges.
The previous checkpoint is suitable as initial initialization: it speeds up convergence, but it cannot be considered the final result.
Strategy migration is built as gradual complication. First, the agent is fine-tuned in a simplified version of the environment, then friction, costs, and execution delays are enabled.
This way, training transfer is noticeably faster than starting from scratch, and behavior is checked at every step, not after the entire run.
The workflow is as follows: audit existing work, measure the divergence between the dynamics of the old and new environments, and only then decide what to transfer as weights and what to retrain.
We do this on Gym-compatible interfaces, so the existing training loop connects to the new environment without rewriting.
Frequently asked questions about agent training and hypothesis testing
Agent training is a series of experiments, and the main risk here is not in implementation but in measurement: random profit is easy to mistake for a pattern.
That is why we design the environment so that validation cannot be bypassed—separate periods for tuning and confirmation, a fixed set of initial states, and costs and slippage built into the execution model from day one.
Below are the questions most often asked before the start. If your case is not in the list, we will review it separately and come back with an estimate and plan.
How can we be sure the agent is not overfitted to history?
The main criterion is behavior on a confirmation segment that was not involved in either tuning or feature selection. Additionally, we run the trained policy on other instruments and market regimes where the pattern should hold, and we look at drawdown relative to turnover. A result that holds only on one segment is not yet a hypothesis.
How long does custom environment development take?
First, the specification of state, actions, and execution model, then calibration of costs and validation on degenerate strategies: a simple policy should not show profit.
From specification to an environment with automated tests—usually several weeks; the timeframe depends on the required detail of execution.
What are the data requirements?
Trade or bar history reduced to a single time step, with correct corporate actions and without gaps, is sufficient. A quote stream improves the accuracy of the execution model but is not mandatory.
Who owns the code, and will we be able to develop the environment ourselves?
We transfer the repository, rights, and trained configurations to the customer. The environment comes with documentation, automated tests, and a list of parameters: adding an instrument, changing costs, or changing the position holding horizon can be done without our involvement.
Let's discuss your strategy: data, risk profile, work plan
A business-like conversation starts before code: we need to understand what decision the agent makes and within what boundaries it is allowed to make mistakes.
The more precisely the problem statement is formulated and the more complete the customer's data, the fewer iterations will go into the environment and the sooner a verifiable result will appear.
To come back with an estimate and plan, we expect from you:
- Problem statement—what actions the agent chooses, what it sees at the input, when a training step is considered complete, and what constitutes a bad outcome.
- Customer data—historical series and trades in their original form, with frequency, gaps, and known breaks; there is no need to manually bring them into a convenient form.
- Strategy risk profile—permissible drawdown, position limits, prohibited behavior scenarios, and what is worse for you: shortfall in profit or sharp loss.
- Reward logic in general terms—what the agent receives a plus for, what it receives a penalty for, which constraints are hard and which can be violated.
- Training and validation loop—where we calculate, on what history we validate, and how we separate the validation period from the training period.
- Acceptance criterion—by what numbers you will understand that the custom gym environment is ready for work, and not merely starts without errors.
Next, we clarify details on a call, fix the stages, and come back with an estimate, work plan, and team composition. Describe the task in a convenient form—we will respond to the point and tell you which data will be needed first.
