Why time series forecasting breaks on real data
A forecast on historical data converges almost perfectly, but on a live stream it diverges within the first weeks.
The reason is usually not the model architecture, but the nature of the data itself: non-stationarity of the series means that the distribution of returns and the relationships between features drift.
A model trained on a calm trend carries its assumptions into a period of high volatility and systematically errs precisely at turning points.
The second most common cause is leakage of future data into the training sample.
Normalization over the entire history, lags calculated with look-ahead, feature selection on the full dataset, joining timestamps from different time zones — all of this produces beautiful metrics on the holdout set and complete uselessness in operation.
Such leakage is caught only by strict time-based splitting and by checking that at the moment of each forecast the feature was actually available.
Feature quality matters more than model choice. Macro indicators are released with a lag and revised retroactively, corporate events and the news background arrive irregularly, and sources have their own frequency and their own gaps.
If the series are not aligned in time and it is not labeled what was known at the moment of the forecast, even a careful architecture like Temporal Fusion Transformer begins to fit noise instead of signal.
A change in market regime, leakage, and lagging features produce one result: a plausible backtest on history and a loss-making forecast in reality. We break down how this is closed — from time-window labeling and feature selection to drift control in production.
What a managed forecasting model delivers: accuracy and intervals
Forecast accuracy over a long horizon sags not because of a lack of data, but because the model does not separate short-term noise from slow market regimes.
Temporal Fusion Transformer solves this explicitly: the attention mechanism selects which parts of history to rely on for each step of the horizon, while gates cut off features that stop working at that step.
Forecast error at distant points of the horizon decreases more noticeably than at near ones — precisely where classical approaches fall apart fastest.
The second part of the effect is interval estimation instead of a single number. A model trained on quantile losses returns not a point value but a range: a conservative scenario, the median, and an optimistic one.
Planning begins to rely on the width of the interval rather than a “magic” number, and the client sees where the forecast is confident and where the model honestly states uncertainty.
This directly changes the planning horizon: decisions that were previously made two steps ahead become justified six to eight steps ahead.
The model’s resilience to anomalous days is a separate metric. Holidays, technical failures, and outliers in history usually inflate the error, and the forecast has to be discarded.
Here the quantile loss function and the separation of features into static and dynamic ones produce a smaller error spread on “dirty” days without manual cleaning of each series.
Manual recalculations go away with this. Previously an analyst compared the forecast with actuals and corrected the number by hand; now recalculation runs on schedule, and intervention is needed only when the actual value falls outside the interval bounds.
Forecast error, interval width, and the share of actuals outside its bounds go into the report as measurable quantities — they show whether the model is degrading or holding accuracy.
Financial market time series: comparing forecasting approaches
The choice of approach is determined by three properties of the series: forecast horizon, amount of available history, and how often the market regime changes.
Stable seasonality over a short horizon is handled by classical statistics, while heterogeneous signals — prices, volumes, macro indicators, event calendar — require a model that weighs their contribution itself.
The key difference between families is how they work with memory. Statistical models describe the series with a formula: they are reliable on small samples and extrapolate a trend well, but they do not carry across a regime change.
Attention models choose for themselves which parts of history to look at, so they capture long dependencies and multi-horizon forecasting — but they require an order of magnitude more data and careful regularization.
| Approach | When it fits | Where it hits limitations |
|---|---|---|
| Statistical models (ARIMA, ETS, GARCH) | Short horizon, stable seasonality, little history | Do not handle nonlinearity and regime shifts, external features connect poorly |
| Gradient boosting on lags | Many features, intraday signals, tabular data | Does not see long dependencies and order in time |
| Recurrent networks | Medium-length sequences, one or two variables | Fading memory, sensitivity to structural shifts |
| Attention models | Heterogeneous series, multiple horizons, cross-assets | Requires a large amount of history, risk of overfitting to noise |
| Hybrid: statistics + attention | A series with trend and rare spikes | Harder to maintain and reproduce the baseline |
In practice we do not choose one family — we calculate a baseline with statistics and see whether the attention model gives an improvement on the holdout period for each horizon separately. If there is no improvement, the complex architecture does not go to production.
How implementation is built: audit, training, acceptance
A forecasting model is not a single training run on history, but a sequence where each step relies on the previous one. If a feature is unavailable in real time, test accuracy will be a fiction. Therefore we carry out the work in seven stages.
- Audit of historical data: completeness, gaps, changes in market regimes, corporate events. We check that each feature was known at the moment of the forecast rather than added retroactively.
- Feature scheme: which series, lags, and calendar factors enter the model, for which horizons, and at what granularity we calculate the forecast.
- Sample splitting: the training part is the past, the validation part is the holdout period the model has not seen. We do not allow shuffling across time.
- Model training: Temporal Fusion Transformer with quantile losses — the output is not only a point forecast but also an uncertainty interval.
- Validation on the holdout period: metrics for each horizon, comparison with baseline models, and review of cases with maximum error.
- Acceptance of the solution: we fix the achieved metrics, degradation thresholds, retraining procedure, and the order of actions in case of failure.
- Handover to operation: forecast calculation on new data, comparison of forecast with actuals, documentation, and training for your team.
The order of stages is not accidental: a missed data audit comes back as an error already in operation, when the model confidently errs at every step.
What we hand over: configurations, schemes, procedures
We consider a time series forecasting project delivered not when the model shows quality on the holdout period, but when your team can launch it, update it, and explain its behavior without us.
Therefore handover is not a report but a package of artifacts, each of which closes a specific operational risk.
- Model configuration — architecture, hyperparameters, training window, fixed random seeds: training reproduces on the same history and gives the same result.
- Feature passport — for each input, the source, window, appearance lag, handling of gaps and outliers, and permissible distribution shift.
- Data flow scheme — the path from sources to data marts, quality control points, responsibility zones, and behavior when a source is unavailable.
- Access rules — roles, service accounts, environment separation; forecast calculation does not have rights to change training data.
- Monitoring procedure — quality and stability metrics, trigger thresholds, notification order, and deviation review.
- Retraining procedure — schedule and triggers, version selection order, validation on historical periods, rollback to the previous version.
- Failure response procedure — quality drop, input data shift, missing source: calculation stop points and recovery order.
We hand the package over to your repository and go through it with the team — after that the model is maintained internally, without external dependency.
Case study: how forecast accuracy grew on exchange data
The client’s team was building a forecast on exchange data from several sources: quotes, trading volumes, macroeconomic releases, event calendar.
There was a separate model for each horizon, and gaps and outliers were closed with analysts’ manual assumptions — when they were revised, the forecast history changed retroactively, and there was no way to compare two versions.
We consolidated this into a single loop: one model instead of a set of point forecasts, explicit separation of features into groups, and input data quality checks before calculation.
The core was Temporal Fusion Transformer: the architecture weighs feature contributions itself and returns not only a point but also interval bounds — the forecast reads as a range with a confidence level, not as a number that cannot be checked.
Metrics before and after
| What was measured | Before | After |
|---|---|---|
| Error at daily horizon | 12–18% above baseline | 9% below baseline |
| Interval coverage | not calculated | 90% of intervals cover actuals |
| Forecast preparation | 2 days of manual assembly | 40 minutes according to procedure |
| Recovery after data failure | up to a week manually | overnight automatically |
An equally important part is the update procedure. Retraining runs on schedule on a rolling window, feature drift is checked before release, and the model version is linked to the data version, so any export can be reproduced.
If a new version loses to the previous one on the control period, release is blocked and the working build remains — the forecast does not degrade silently.
How does Temporal Fusion Transformer differ from classical models?
Classical models — ARIMA, exponential smoothing, boosting on lags — give a point forecast and assume a homogeneous series. External factors are introduced with manually engineered features, and on heterogeneous data quality rests on lag selection that is hard to explain.
A model with an attention mechanism is structured differently: the input is divided into three groups — static characteristics of the object, features known in advance for the entire horizon, and features observed only in the past.
Attention itself decides which time points and which features to rely on at each point, so heterogeneous data enter one model without being reduced to a common form.
Importance weights are exposed as well: you can see which input shifted the forecast and how its influence changed over time. This removes the main pain of classical schemes — the inability to understand why the model erred precisely here.
The second difference is interval forecasting. The output is not one number but a set of quantiles, so the decision is made based on the distribution: how wide the spread is and how often actuals fall outside the bounds.
Where a margin of safety is needed rather than an exact hit, this changes the problem statement entirely.
The price for this is model complexity: training takes longer, the result is sensitive to the amount of history and the quality of event labeling, and parameter selection requires separate validation across time slices.
On short series or with two or three features, the gain does not pay off — there, boosting on lags remains.
Therefore it makes sense to compare approaches on a specific data structure and forecast requirements, not by the name of the architecture.
Limitations and common doubts about a forecasting model
Time series forecasting does not end with model training: beyond the historical segment, the result depends on data preparation and control. Below are the doubts raised before the start, and how we close them.
How much history does the model need?
You need a series covering a full market cycle — growth, correction, and sideways movement. On a homogeneous segment the model memorizes the regime rather than the dependency, so with a short history we narrow the forecast horizon.
What should be done with gaps?
We do not fill gaps with the mean — that distorts volatility. For each series we choose a rule: calendar alignment, interpolation with a limit on gap length, or a “no data” feature as a separate signal.
How is data protected?
Training runs on your infrastructure: exports do not leave the environment, and access is differentiated by roles. We connect external sources in anonymized form — the model sees aggregates and derived features, not raw records.
What happens during a market reversal?
In a regime change, any model degrades — this is a limitation of the approach, not a defect of implementation. We monitor error and distribution drift: when thresholds are exceeded, the forecast switches to a cautious mode and retraining begins.
What is included in support?
We hand over a reproducible training loop, retraining procedure, and quality control dashboard. After that — scheduled model updates, signal review, and feature adjustment when data changes.
Let’s discuss the task: we will select a configuration for your data stream
A substantive conversation does not require a finished technical specification. It is enough to have an export of a series with actual values, a calendar of events affecting behavior, and a description of frequency, forecast horizon, and permissible response lag.
If a log of past models with their errors has been preserved — send it, it saves us a round of hypotheses.
Data review comes before any promises about quality. We look at distribution, gaps, regime changes, and whether the signal separates from noise at your horizon. Selecting a solution after that is a matter of justification, not taste.
Based on the review, we come back with specifics:
- data review: frequency, gaps, anomalies, changes in market regimes;
- solution selection: from baseline models to an ensemble with Temporal Fusion Transformer — with justification for why this class;
- validation scheme: rolling window, holdout period, metric for your task;
- estimate and work plan: stages, team composition, quality control points;
- collaboration order: roles, data access, reporting and acceptance format;
- applicability boundaries: at what horizon the model is reliable, and where manual control is needed.
Describe the task and send what you have. We will come back with an estimate and work plan that shows what we are responsible for and at what point you can stop.
