Why LSTM time series fail on real data
The classic symptom: on historical data, an LSTM model shows consistently low error, but a few weeks after launch the forecast diverges from reality.
The cause is rarely the architecture or the amount of training — the divergence is generated by data preparation and the way quality is measured.
Future value leakage is the most expensive of errors. It occurs when normalization is calculated over the entire sample at once, features are aggregated over a window that captures the future, or the target shift is made in the wrong direction.
On validation, such a model looks accurate because the answer is already mixed into it: the error on live data turns out to be several times higher than the validation error.
Non-stationarity of the series finishes off the rest: the mean level and volatility change, market regimes switch, and the distribution of inputs on live data stops matching the training one.
Random splitting into training and validation masks this — temporally adjacent observations almost duplicate each other, so validation confirms what the model has already seen.
The third cause is rare but significant events: gaps, volatility spikes, regime changes. There are only a few of them in history, so the model learns to average the tail of the distribution and systematically underestimates risk exactly where it is critical.
Next, we break down how this is addressed: sequential validation over time, a gap between the training and validation segments, regime features, and continuous monitoring on the live stream.
Measurable effect: what changes in forecast quality
We measure forecast quality on LSTM not by a single number on a holdout sample, but by a set of checks repeated on different parts of the history.
The baseline is the naive forecast “tomorrow will be like today”: if the model does not beat it on the required horizon, it has no value.
Quality over the horizon
- Error is calculated separately for 1–5 steps ahead — it is clear up to what point the forecast is suitable for a decision and where noise begins.
- The spread of error between validation windows is limited: the metric does not diverge when the history segment changes.
- The share of missed peaks falls, false positives become fewer — the model stops smoothing a spike down to the average.
Regime change and interpretation
- Rolling-window validation: training on the past, validation on the next block — this reveals overfitting that is invisible with random splitting.
- The contribution of features and lags shows what the model relies on at a specific point — the forecast can be explained rather than taken on faith.
- During a regime change, the error grows smoothly and predictably: there is time to retrain the model before deviations become systemic.
We record each item in the validation report together with an error chart by window — the client sees where the model is stable and where its readings should be double-checked.
Forecast model options and their technical limitations
The model class determines which patterns of the series it can see in principle — and mistakes here are more common than during training.
Below are the real alternatives chosen for financial series: we build recurrent branches in PyTorch, and statistical ones on standard libraries.
| Option | When it fits | Limitations |
|---|---|---|
| Recurrent networks (LSTM) | Long memory across a sequence: intraday quotes, volumes, order flows | Requires a long homogeneous series; training is slow, feature contribution is poorly explained |
| Gradient boosting on lag features | Short horizons, tabular features, limited history | Does not see order within the window; the lag window is set manually |
| Statistical forecast models | Seasonality and trend, small sample, interpretability needed | Linear assumptions; nonlinear relationships are lost |
| Hybrid forecast models | The series is decomposed into trend, seasonality, and residual | Double tuning; block errors accumulate |
| Attention models for series | Long horizons, several interrelated series | Requires a lot of data, sensitivity to scale, more expensive to maintain |
In practice, the choice is rarely binary: boosting provides a baseline level, and a recurrent or hybrid extension is further trained on the residual.
We finalize the decision not by the metric on a historical segment, but by the validation scheme — a rolling window plus a period that the model has never seen.
Workflow: from data audit to model handover
The order of stages is fixed: each subsequent one makes sense only after the previous one has produced a verifiable result. We do not start with architecture — on real financial series it runs into the quality of history and the correctness of splitting.
We break the work into seven steps; after each one, the client sees an artifact, not a progress report.
- Data audit. We check gaps, duplicates, breaks in quotes, weekends and holidays, trading regimes, splits, and corporate events. We record what we clean and what we restore — and why.
- Feature scheme and rolling window. We determine the history depth and window step, the set of lags, derived and regime features. We calculate normalization only on the training segment; otherwise the future leaks into the past.
- LSTM assembly and training. A baseline configuration with regularization and early stopping — as a reference point against which all further options are compared.
- Hyperparameter tuning. Number of layers and blocks, window size, dropout rate, learning rate. The search runs on the validation segment; the test segment is not opened until the end.
- Honest validation scheme. A rolling window with training on the past and validation on the next period, without shuffling. We calculate metrics over several periods, not one lucky one.
- Robustness check. Period shift, volatility change, stress test on sharp movements. The model must beat the naive forecast; otherwise it is useless.
- Model handover. Weights, preprocessing scheme, launch and retraining script, metrics by period, reproducibility documentation — so the model lives with you, not with us.
This sequence addresses the main risk of forecasting tasks: beautiful metrics obtained on data that the model should not have seen.
Artifacts and policies that remain with the client
A project is considered delivered not when the metric on a holdout window is satisfactory, but when the model can be retrained and explained without us.
Therefore, acceptance includes not only code but also everything that makes the behavior of the series verifiable: fixed data, environment configurations, and rules for working with time slices.
- Reproducible training — fixed random seeds, library versions, and data processing order: a repeated run on the same slice produces the same metric.
- Feature scheme — windows, lags, normalization, and missing-value handling rules with justification: it can be extended without rewriting the architecture.
- Environment configurations — descriptions of training and forecast calculation environments, so the model can be raised on your infrastructure without our involvement.
- Data access policies — roles, masking of sensitive fields, and an access log for training samples.
- Model documentation — architecture, hyperparameters, metrics on holdout time segments, applicability boundaries, and known failure modes.
- Retraining policy — what counts as a trigger (distribution drift, metric drop), at what frequency, and how the new version is checked before rollout.
- Validation log — time-based splitting without shuffling, so the future does not leak into training.
Each artifact is accepted by fact, not by the presence of a file: we run training on your side and compare metrics with the reference run. If the discrepancy exceeds the agreed tolerance, the item is revised until delivery.
Case study: demand forecasting before and after model rework
The client is a distributor with several hundred SKUs and pronounced seasonality. Procurement planning rested on manual rules: sales history, a coefficient for the previous period, manual adjustment for promotions.
Over a two-week horizon this worked; beyond that, the forecast diverged from reality, and peaks — the start of the season, a sale, a large shipment — were not seen by the rule at all.
What it was like before the rework
A typical model on lag features reduced the average error but smoothed away the main thing: peaks were smeared across adjacent weeks. Validation with random splitting by rows gave a decent metric on the control sample, but on the real horizon the model fell apart. Safety stock was kept “by eye” — excessive for calm SKUs and insufficient for peak ones.
What was changed and what the result was
We switched to a recurrent scheme in PyTorch with separate series by product class, rolling time-based validation, and a loss function where missing a peak is penalized more heavily than overestimating.
We added a promo calendar, lag features, and rolling statistics, and we measure quality not by overall error but separately by horizon and by peak weeks.
- error over an eight-week horizon: 23% → 11%;
- share of missed peaks: 38% → 9%;
- false spikes: down by a third, excess stock for calm SKUs was reduced;
- forecast preparation: two days of manual work → twenty minutes of computation.
Model maintenance is part of the work, not a separate service: monitoring feature drift, an alert when accuracy drops on peak weeks, a regression set from past spikes, and scheduled retraining. During the season, the model remains predictable rather than degrading by the next cycle.
What to do if there is little data to train LSTM
A short history is primarily an imbalance between the number of parameters and the volume of observations: the network memorizes the noise of the training segment and on the holdout segment loses to simple rolling statistics.
Therefore, we start not with architecture but with a baseline — autoregression or a linear model on returns. If the network does not beat it over several consecutive windows, there is nothing to complicate further: this is a normal engineering conclusion, not a failure.
Regularization with a small sample gives more than depth.
We leave one layer, limit the number of hidden units, enable dropout between layers and separately on recurrent transitions, limit the weight norm, and stop training by the validation segment rather than by the number of epochs.
Changing the representation also helps: instead of prices, we feed increments and a volatility estimate scaled to a comparable magnitude.
Transfer learning closes the history deficit: we pretrain on long series of similar instruments, then fine-tune on a short target segment, freezing some layers.
The technique works as long as the regimes are comparable — otherwise the network drags along someone else's dynamics, and this is visible in the very first validation windows.
Validation is built only over time, without shuffling: an expanding window, separate segments for tuning and for final validation, normalization calculated strictly on the training part.
We analyze rare events separately and evaluate them by the share of correct signals, not by average error. With a short history, a hybrid more often wins: the baseline model sets the level, and the network learns on its residuals.
Technical concerns: overfitting, drift, and online forecasting
A metric on a holdout sample does not say how an LSTM model will behave on a live stream of quotes. The engineering team asks substantive questions: is there overfitting, how is drift caught, when is retraining triggered, and what happens in online forecasting mode.
How do you control model overfitting?
Validation is over time, not by random splitting: an expanding window and a gap between the training and validation segments, so future values do not leak into features. We limit capacity with dropout and early stopping by the metric on the validation window.
How is data drift tracked?
On the input side, we compare the feature distribution in the current window with the training one — stability index and Kolmogorov–Smirnov test. On the output side, we monitor the error on accumulated actuals.
Exceeding the threshold triggers retraining, while the pipeline continues to calculate with the current version.
How often do you retrain the model?
Frequency is set by triggers, not by the calendar: a scheduled window plus unscheduled events — drift above threshold, metric drawdown, market regime change. The new version goes through the same time-based selection and is launched in parallel with the current one.
What happens in online forecasting mode?
Features are calculated on a rolling window, the forecast is returned with a fixed latency budget, and upon degradation the pipeline switches to a backup version.
In production, we monitor error, trigger frequency, and input stability — the problem is visible before the business notices it.
Let's discuss the task: we will return with an estimate and work plan
To return with an estimate and a stage plan, rather than general considerations, we need a minimum of inputs from you — enough to break the work into blocks and determine the team composition. Below is what we usually clarify in the first conversation about LSTM time series forecasting.
- Target variable and horizon: what exactly we forecast — return, price, volume, volatility — and how many steps ahead the result is needed.
- Data: which series are already collected, at what frequency and history depth, and how gaps, breaks, and corporate events are handled in them.
- External factors: whether macroeconomic indicators, an event calendar, or a text stream are available — the feature set and model complexity depend on them.
- Operating mode: batch recalculation on a schedule or online forecasting with a response-time constraint.
- Validation scheme: rolling window, number of validation windows, metrics for each horizon, and the baseline model against which we compare quality.
- Deployment pipeline and access: where the data is stored, whether there are restrictions on moving it outside the perimeter, and who is responsible for compute resources.
- Acceptance: which metric values we consider sufficient and how the forecast enters reporting or the trading pipeline.
Describe the task in free form — goal, data, usage mode, constraints. We will return with an estimate, a stage plan, and team composition: who is responsible for which block and at what stage you join the review of interim results.
