Why an AI investment system fails on live data
On historical data, an AI investment system produces an accurate forecast, but on a live stream the forecast error grows severalfold. The reason is usually not the model itself, but that training was done on a cleaned and synchronized series that does not exist in real trading.
Market data quality is the first source of discrepancies. Quotes contain gaps and stale last trades: for an illiquid security, the price has not updated for hours, while the model reads it as a confirmed signal.
Different sources diverge on closing prices and on the treatment of corporate events, and this discrepancy goes straight into the inputs.
The second factor is model drift when the market regime changes. Features that worked in a calm trend lose significance during high volatility: the distribution of inputs shifts, while relationships learned in the previous regime continue to be applied without revision.
The third is data arrival lag and manual adjustments in asset accounting.
While quotes and corporate events reach the system, positions have time to change: splits, dividends, and transfers between accounts are entered manually, and the model evaluates a portfolio that no longer exists.
Together this produces a systematic forecast error that is invisible on the test sample. We break down how it is closed — from data quality control to drift detectors and regular reconciliation of asset accounting.
What changes after implementation: the effect in measurable terms
Recommendations stop being assembled manually from disparate spreadsheets. The system pulls quotes and corporate events itself, calculates metrics for each instrument, and gives the analyst a ready draft with rationale.
The time to prepare one recommendation drops from hours to minutes, and manual operations are reduced to review and confirmation.
Portfolio risk control is built into the workflow before trade execution.
Limits on a single instrument's share, sector concentration, correlation with current positions, and allowable drawdown are checked automatically — some orders are rejected before they reach a person, and the reason for rejection is stored for each. Risk stops depending on who is on duty today.
Reproducibility of decisions rests on versioning: input data, the rule set, and the model version are recorded at the time of calculation. A month later, any recommendation can be pulled up and shown on which figures it was built. This is both protection against claims and material for tuning the strategy.
The data source is normalized quotes and corporate events from external providers such as Bloomberg; discrepancies with the accounting system are handled by a separate report, not a person.
The result is visible in three figures: the speed of preparing recommendations, the share of automatically rejected risky trades, and the share of decisions reproducible without manual review.
Build formats: ready-made core, fine-tuning, own model
The choice of format is set not by ambition, but by how much your logic differs from the standard. If the rules fit standard scenarios — risk profile, rebalancing corridors, instrument limits — a ready-made core wins: it is faster to put into operation and easier to maintain.
Fine-tuning on your own data is needed when the advantage comes from accumulated trade history and analyst decisions: without them, the model will not see your patterns.
An own model is justified where a ready-made workflow does not describe the process — non-standard risk logic, multiple signal sources, a requirement to explain every decision.
| Format | When it fits | Limitations of the approach |
|---|---|---|
| Ready-made core with configuration | The process fits standard scenarios: risk profile, rebalancing corridors, limits | Logic is limited by core capabilities; non-standard rules are handled by external add-ons |
| Fine-tuning on your own data | There is accumulated trade and decision history that needs to be turned into signals | Labeled data is required; when the strategy changes, the model must be retrained |
| Own model | The logic is unique, explainability of decisions and full control over calculation are important | Longer to put into operation; ongoing support and forecast quality control are required |
We lock the format before development starts: the team composition, timelines, and how the system will explain its decisions depend on it. If you are unsure between fine-tuning and an own model, we will run both options on your data and show where the boundary lies.
Implementation stages: from data audit to handover to operation
Implementation of an AI system for a portfolio begins not with the model, but with an inventory of the data it will learn from.
The order of stages is fixed: if you skip the audit of sources or historical validation, the output will be plausible but non-reproducible recommendations and drawdown on real trades.
-
Data source audit. We collect everything already available: quotes, corporate actions, issuer reports, macroeconomic series, exports from brokerage systems.
We check completeness, update frequency, gaps, retroactive restatements, and usage rights — without this, the model learns from a leaky series.
-
Feature schema. From raw series we build features: returns over different horizons, volatility, correlations, market regimes, risk metrics. For each feature we fix the calculation window and feed delay so that future data does not leak into training.
-
Objective functions and risk rules. We agree on what we optimize — return at a given risk, drawdown, or asset class shares. We immediately set boundaries: limits on a single instrument, sector, and issuer share.
-
Configuration and training. We label the history, split periods into training and validation without shuffling, and tune model parameters.
-
Validation on historical periods. We run the system across several market regimes, including crisis periods, and compare with baseline strategies. We look at drawdown, stability of the result, and number of trades.
-
Acceptance and handover to operation. We hand over the retraining procedure, data and metric drift control, automatic shutdown threshold, and documentation to the customer's team.
Delivery scope: configuration, data schemas, and procedures
The model itself is only part of the system: configuration, data, rights, and procedures remain around it.
If the customer receives only a trained model and a set of scripts, the system lives until the first failure — it is unclear which features are fed as input, who has access to historical data, and by what procedure the model is updated.
That is why we fix delivery as a set of artifacts: each can be accepted and handed over for maintenance to another engineer without verbal explanations.
- Solution configuration — calculation horizons, risk constraints, and portfolio formation rules are moved into separate files rather than hardcoded. Changing a limit does not require a rebuild and a new release.
- Data flow schemas — the path from the quote and corporate event source to the feature store and recommendation output: where normalization happens, where quality control happens, where the failure point is and how it is handled.
- Access policies — roles of analyst, risk manager, and support: who sees historical data, who changes configuration, who starts retraining, with an action log for each change.
- Documentation — model description, list of features and their sources, applicability limitations (which market regimes are not covered), and operating instructions for the on-duty shift.
- Retraining procedure — triggers (drift in feature distribution, quality drop on the holdout sample), frequency, validation order, and conditions for rollback to the previous version.
- Set of checks and recovery plan — automated tests for gaps, outliers, and divergence from the reference calculation, plus a description of where previous versions of the model and configuration are stored and within what time the system returns to operation.
Case study: how the share of manual portfolio checks was reduced
Before: calculations and reconciliation lived in different spreadsheets
The management team at Aurum Capital assembled model portfolios manually: an analyst transferred target weights from the calculation model, a risk manager checked them against limits in a separate file, and a trader entered orders in yet another system.
A full review took about four hours, and the discrepancy between the approved and actual portfolio structure surfaced only after trades — when it was more expensive to fix.
What was changed in the decision-making workflow
We consolidated the workflow into one service: calculation of target weights, checking limits by instrument, sector, and currency, an approval log with the author of each edit, order placement, and rollback to the previous model version.
Risk rules are described declaratively, so a new limit is added without rewriting logic, and a deviation from the model cannot be made silently — it is stored with the reason and the approver's signature.
- Share of manual portfolio checks: 100% → 18% — automation reconciles weights and limits, and the analyst handles only exceptions.
- Full portfolio review: 4 hours → 25 minutes including approval.
- Limit violations found after the trade: from 7 per quarter to zero — the check occurs before order placement.
- Rollback to the previous model version: instead of restoring spreadsheets from memory — one operation with saved decision context.
How a recommendation system differs from an automated trading robot
The difference is not in the model, but in the point where calculation turns into action. A recommendation scheme issues a signal with rationale — instrument, share, reason for triggering — and a person enters the order.
An automated trading robot closes the loop itself: the signal immediately becomes an order, and there is no person in the cycle.
This is where reaction speed and risk class diverge. A recommendation scheme is fine with recalculation once a day or on an event — the pace is set by a person.
Automatic execution needs constant recalculation and a stable channel to the broker; otherwise slippage eats the strategy's effect. A model error in the first case costs a missed trade, in the second — an executed order.
| Criterion | Recommendation scheme | Automated trading robot |
|---|---|---|
| Level of control | trade is confirmed by a person | decision is made by an algorithm |
| Reaction speed | hours, pace set by a person | seconds, pace set by the model |
| Human oversight | review of signals and deviations | limits, risk circuits, stop by trigger |
| Risk class | missed trade | executed order |
In practice, the choice is not "either/or" but where the boundary lies. Usually a recommendation scheme is responsible for changing the portfolio structure, while automatic execution handles rebalancing and protective actions under strict limits.
The more expensive the error and the less time for reaction, the higher the oversight requirements: a log of every decision, signal attribution, emergency stop, and a responsible operator.
Technical concerns: model drift, data, responsibility
Three questions come back at every project review: what will happen when the market changes regime, what to do about gaps in data, and who is responsible for the final decision.
All three are closed not with promises but with system design — model drift control, data quality rules, and a clear division of responsibility.
Model drift is the divergence between the data it was trained on and current market behavior.
We track the distribution of input features, stability of factor importance, and divergence between forecast and actual on a rolling window; when metrics cross a threshold, retraining is triggered.
Each version is saved together with the dataset and settings, so returning to a previous version does not require a rebuild.
Incomplete data does not arrive entirely missing, but with different frequency, delay, and gaps. Therefore each source has its own set of checks: completeness, freshness, consistency with a backup channel.
If data is insufficient, the system does not fill it in but marks the calculation as incomplete and switches to observation mode.
How do you know the model is outdated?
By the rolling divergence between forecast and actual, stability of factor importance, and the share of signals rejected by risk rules.
Retraining runs on schedule and by trigger; the new version first works in shadow mode — its calculations are counted but do not affect decisions.
What to do if there is no data for an instrument?
No recommendation is issued, the calculation is marked as incomplete, and the event goes into the monitoring log. The discrepancy between the primary and backup source is visible in the report, so the gap is detected before it reaches the portfolio.
Who is responsible for the investment decision?
The system prepares the calculation, rationale, and list of assumptions; the decision is made by a person — an analyst or portfolio manager.
The division is fixed by procedure: we are responsible for data quality, model operation, and the decision log; your side is responsible for interpretation and the trade.
What to discuss at the first meeting about your task
The first meeting is not for a presentation but for gathering inputs on which the estimate of work volume directly depends.
The more specific the answers about data, constraints, and roles, the fewer assumptions remain in the plan, and the timelines to the first working workflow become predictable.
- Data sources: what exports from brokerage and banking portals exist now, in what form they arrive, how often they are updated, and who grants access.
- Depth of transaction history: for what period trades exist and how corporate actions are reflected in them — deposits, payouts, splits.
- Current decision-making process: who decides on buying or selling, which reports they look at, and how much time this takes today.
- Project constraints: which tools are already in use, what cannot be changed, and what requirements exist for data storage and access separation.
- Roles in the team: who on the customer side is responsible for data, who confirms the investment logic, and who accepts the result.
- Timelines and milestones: by what date the first working workflow is needed and which stages can be run in parallel.
After the meeting, we fix the scope, break it into stages, and return with an estimate and implementation plan — with an explicit indication of what we do and what remains on the customer's side.
