Feature Store: why features diverge between training and production
Features for a model are computed in two different places. During training they are collected by querying historical data — in batch, across all available depth. In model operation, the same features come from an online store, one entity at a time, with minimal latency. As long as both implementations match, metrics on the holdout set and in production agree.
Feature divergence begins where these two branches differ in details. In training, an aggregate is taken over the entire history with exact event time; online, it is taken over the latest window, which late-arriving events no longer enter.
Different missing-value handling rules produce different default values, and the model receives an input set of values different from the one it was trained on.
A separate cause is feature recomputation. The aggregate rule is changed, history is recomputed retroactively, but the model remains trained on the previous definition.
Training and production live on different versions of the same logic, and offline validation metrics do not reveal this.
The cost is silent degradation of model quality in production. Accuracy on tests stays the same, but the share of erroneous decisions grows.
Finding the cause takes weeks: there is nothing to compare online and offline values with, and no one stores the correspondence between them.
Feature Store removes this by describing the feature definition once and serving both loops. Feast populates offline and online stores from a single description, accounts for event time, and stores definition versions. Next we will look at how this works in practice.
What a unified feature layer provides during model development
A feature almost always lives in several places: a calculation in an analyst's script, a copy in a data scientist's notebook, and a third version in service code.
The divergence is invisible while the model trains on one sample and receives another at input — and surfaces in production when quality drops without changes to the model itself.
A unified feature layer (Feature Store) removes this fork: the definition is described once and materialized in offline and online stores — open catalogs like Feast are built around such a registry.
- Data preparation time. A new feature is described once: the cycle from idea to value in the model shrinks from weeks to days — there is no need to reassemble and reconcile calculations.
- Feature reuse. The catalog shows what has already been computed, who owns it, and how fresh the values are — duplicates do not appear, and a new calculation builds on top of an existing one.
- One source of values for training and model operation. Training and online serving read consistent definitions, so the feature distribution in production matches the validation one.
- Eliminated class of failures due to desynchronization. Different versions of calculation logic and manual copies stop breaking quality: the scenario is closed at the platform level, not by process.
- Reproducibility. Each training sample records a feature snapshot at a point in time — a previous result can be repeated and explained.
- Access control. One catalog with roles, owners, and change history instead of approvals in chat.
This can be checked before implementation: if at least two features are computed in two places, desynchronization is a matter of time.
Feature Store: comparing implementation options
Feature Store implementation options differ along two axes: whether features are needed at request time and how often their composition changes. If the model is recomputed on a schedule and a delay of hours is acceptable, a complex loop is not needed.
If features go to an online service and one calculation logic serves both training and serving, a registry and layer synchronization are indispensable — the divergence between them is visible in production. The difference between options is not in the amount of work, but in what remains with your team.
Options and their limitations
| Option | When it fits | Limitations of the approach |
|---|---|---|
| In-house build | Unique calculations, strict latency requirements, there is a platform team | Layer consistency, versioning, and drift control are on you |
| Platform solution | Need online serving and a unified registry without in-house development | Dependence on an external loop and its source limitations |
| Layer on top of a data lake | Data is already in the lake, analytics needs feature history | No serving at request time, a separate synchronization loop is needed |
| Batch mart only | Scheduled recomputation, delay of hours is acceptable | Features become stale between recomputations, some scenarios fall away |
In practice, a hybrid more often wins: the data lake remains the source of truth, the registry stores definitions, and batch recomputation populates the feature mart. We fix the limitations of the chosen option before start, not after the first divergence.
How we implement the feature layer: work stages
Implementing a feature layer is not installing a library but bringing order to where models get input data. First we examine sources: which tables, events, and exports actually feed training and inference, who owns them, how often they update, and where values diverge. This determines the flow schema and which features are even worth computing as reusable.
- Data source audit. Inventory of tables and events: owner, update frequency, appearance latency, change history. Result — a source map marking which are suitable for training and which only for analytics.
- Flow schema design. We fix keys, granularity, windows, and versions: the same logic must be computed identically for historical data and for an online request.
- Calculation and storage setup. We stand up a feature store (Feature Store): an offline layer for training, an online layer for serving at request time, and a definition registry. We describe features declaratively so the formula is not rewritten from scratch in every team.
- Feature quality validation. We compare offline and online: the same event must produce the same vector in both layers. We catch missing values, distribution drift, and stale values on a test loop, not in production.
- Deployment. We configure update schedules, latency control, and divergence alerts so data degradation is visible before the model notices it.
- Documentation handover and training. Feature registry, flow schemas, process for adding new features and incident review. We conduct a review with engineers and analysts so the layer expands within the team.
What we hand over: configurations, schemas, processes
Someone will have to maintain this six months after launch — and not the person who set it up.
Therefore we close each loop with a set of artifacts by which an engineer without the author can stand up the environment from scratch, add a feature, and roll back a failed change without manually parsing logs.
- Environment configurations — test, pre-production, and production loops with separate sources and schedules; deployment is reproduced from the repository, not “from memory.”
- Feature map — a registry with owner, source, type, update frequency, and list of consumers; it is clear in advance which models an edit or deletion will affect.
- Data flow schemas — the path from source to training sample and online serving: where transformation happens, where the snapshot is stored, where the failure point is.
- Team access policies — roles for reading, writing, and changing a feature definition; rights are granted by role, not by handing out shared keys.
- Update processes — the procedure for adding a feature, decommissioning it, and checking compatibility with already trained models, with approvers specified.
- Data quality control — checks for freshness, missing-value share, and divergence between the training snapshot and online value; degradation is visible before prediction quality drops, not after.
- Support documentation — recovery points, procedure for rolling back a feature version, source owners, and typical failures with ready actions.
Case: how feature divergence between environments was removed
Before
Data for training and for online calculation was collected by two independent branches: analysts — by querying the warehouse, the model service — by its own preprocessing scripts.
A mart with dozens of features was reassembled for each new model, and missing-value and categorical field handling diverged between these branches.
Preparation took 4–5 days, and about a third of models had to be rolled back after deployment — feature values in training and in the working loop did not match.
What was changed
Features were consolidated into a single management layer: the description lives in the registry, and from it both historical snapshots for training and fast access for online calculation are built.
The registry was based on Feast; on top we added distribution comparison between loops, checks for missing values and drift, and the rule “a feature without an owner and tests does not go to production.” Rebuilding is launched by process, not manually for a release.
Mart preparation shrank from 4–5 days to several hours, and the share of incidents due to value divergence fell from a third to isolated cases per quarter. Online calculation response time decreased threefold and stopped depending on the size of the historical window.
Model rollback became a standard operation: features are versioned together with the model, and the result is reproduced on any previous version.
How does a feature mart differ from a data warehouse?
A data warehouse solves the problem of preservation and access: how to record events without loss during a failure and how to provide analytics with a consistent snapshot.
A feature mart solves a different problem — reproducibility of values: what number the model received at input at the moment of a specific event and where it came from.
Hence the first difference — calculation is tied to time. A warehouse returns the current state of a record; a feature layer returns its state at the moment of the event, without data from the future.
If this boundary is absent, values that did not exist then leak into the training sample, and model quality on validation turns out higher than in production.
The second difference — one definition serves two loops. Training reads historical snapshots, production receives current values, but both are computed by the same formula.
The logic is not rewritten from scratch in a separate service, and divergence between training and serving stops being a standard situation.
The third — a contract instead of a request. A feature is described by name, type, source, aggregation window, recomputation schedule, and owner.
Until this exists, each team computes the metric in its own way, and during model quality review it turns out there is nothing to compare.
The practical criterion is simple: if the same calculation logic is repeated in queries for the third time, its place is in the feature mart.
A definition catalog at the level of Feast or Tecton stores them as code, synchronizes both loops, and keeps change history — a data warehouse does not solve this problem.
Common technical questions about the feature layer
Before start, engineers ask the same questions: how the feature layer (Feature Store) will fit on the current data lake, what to do with already accumulated history, and how to separate teams by rights. Below are short answers for each point.
Does the feature layer integrate with our data lake?
We do not move data: feature definitions describe sources that already lie in the lake, and the update schedule is set alongside. Materialization goes to a separate fast store — training and inference serving read the same values.
What to do with historical data for past periods?
We pull history in batch from accumulated snapshots and build point-in-time values for the required dates, so features for training are collected retroactively. Divergence between the training sample and production serving is removed: both sides read one calculation.
How are feature definitions versioned?
Each change is a new definition version; the previous one remains available. A model is pinned to a specific version, so feature recomputation does not break working models, and rollback is switching a reference to the previous version.
How to separate access between teams?
Each team has its own namespace and read/write rights; other teams' features are available only through published sets. The feature owner is specified in the description — it is clear with whom to coordinate a change.
Let's discuss the task: what is needed to start work
To come back with an estimate and implementation plan, not general words, we need to see your feature workflow. To start work, it is enough to show what is listed below — we will clarify the rest with questions during task review.
- Data sources: which systems generate events, at what frequency and in what form they reach analytics.
- Current feature calculation: scripts, notebooks, or periodic exports from which training samples are assembled, and who maintains them.
- Freshness requirements: which features are needed online at request time, and which are acceptable to compute in batch once a day.
- Infrastructure: where training and model deployment to production run, which stores are already used and what cannot be changed.
- Volumes and growth: number of entities, number of features, required history depth for past periods.
- Access separation: requirements for data isolation and for who gets access to features.
- Divergences already caught: how feature values differed during training and in online serving and what it led to.
- Responsible parties: who confirms the scope of work and accepts the result by stages.
This is enough to estimate effort and risks. After review, we will come back with the scope of work, the feature store (Feature Store) schema, and a stage-by-stage implementation plan — with result checkpoints and a rollback procedure if something goes wrong at some step.
