ML Model Versioning: Turnkey Model Registry Setup

Model Registry: why model versions diverge across environments

A model shows a good metric on a run, but a week later in the service it behaves differently. The data is the same, the code is the same, but reproducing the result doesn't work: weights from one run, feature preparation from another, and the data schema has already changed. People sort this out manually, and prediction quality is what pays for it.

The first source of divergence is manual artifacts. Weights are dumped into a shared folder under a name like final_v2, next to it sits the training script, and no one remembers anymore which dataset and which parameters produced that particular file. Across dozens of runs this is tolerable; across hundreds, the link is lost for good.

The second is environment desync. In training and in the prediction service, features are prepared differently: a different column order, different handling of missing values, a different library version. Formally it's one model; in practice it's two different behaviors on the same input data.

The third is model rollback. If the previous version wasn't saved together with the data schema and dependencies, when degradation occurs all that's left is guessing what to roll back to. A rollback should take minutes, not turn into a recovery from logs and the engineer's memory.

A model registry closes this gap as a single point of truth: versioning of ML models, binding to data and run parameters, reproducibility of experiments, and predictable rollback. Below we break down exactly how this is structured.

The Effect of Versioning: Reproducibility and Model Rollback

A model version ceases to be just a set of weights in a separate folder: it is tied to a snapshot of training data, the run configuration, and metrics on fixed slices of the validation set.

Retraining on the same snapshot yields the same metric — the discrepancy is visible during validation, not a week later in production.

Rollback ceases to be an operation with an indefinite timeline. Previously, finding the last working state took hours — artifacts, datasets, and configs were verified manually.

With a registry, it is enough to move the label to the previous version: rollback takes minutes, and the same data snapshot is restored along with the model.

The failure rate after release drops because data and model desynchronization is caught before rollout. Model versions are compared on the same slices, so it is clear where there is an improvement in target metrics and where there is noise on a convenient subsample.

A registry such as MLflow Model Registry maintains this link by default: versions, stages, labels, and artifacts are stored in one repository, and transitions between stages are governed by rules rather than agreements in chat. Permissions to promote a version are assigned by roles, and change history is preserved.

Answering the question "what exactly is running now and on what data" takes a single query to the registry, while incident investigation takes minutes instead of an overnight investigation.

Model storage and serving modes: comparing the options

The choice of how to store versions is almost never dictated by the model itself — it is dictated by how many people and environments work with the artifacts at the same time. While a team has one engineer and one environment, a directory with a date in the file name is enough.

As soon as test and production environments appear, training reproducibility stops being a matter of convenience and becomes a matter of incident investigation.

Below are three ways of organizing versions that teams realistically choose between at the start.

We deploy any of them, but a registry in your own environment covers what a directory fundamentally cannot: an immutable artifact version with metadata, permissions, and a history of releases to production.

The difference shows up at rollback: with a directory, the path to the previous version is reconstructed from run logs; with a registry, it is retrieved by version number in a single request, together with the training environment and metrics. Below we break down what turnkey setup includes.

Implementation stages: from model inventory to handover

The work sequence is structured so that the registry works on real releases, not in a demo. We start with an audit: until it is known which models live in production and who is responsible for each, setup turns into guesswork.

1. Model audit. We collect everything that is trained and launched: services, regular retrainings, one-off experiments. We record where the version is stored today — in weights on disk, in a file name, in someone's memory.
2. Version inventory. We match artifacts to releases: which code and data produced a specific version, which are still in use, which are due for archiving. The output is a mapping table that is used to configure the registry.
3. Schema design. We define the hierarchy “project → model → version”, naming rules, and required metadata: dataset, quality metric, owner, stage. The schema is fixed before setup, otherwise the registry quickly turns into a dump.
4. Registry setup. We deploy artifact storage, role-based access, integration with the version control system and the build pipeline. Separately — a retention policy: how many versions we keep at the ready, what goes to the archive.
5. Release validation. We run a full scenario on one model: training, registration, promotion through stages, rollback to the previous version. The readiness criterion is rollback in minutes and without manual edits.
6. Team training and handover to support. We show engineers and analysts how to register a model and find the needed version; we hand over the policy, responsible roles, and incident investigation procedure.

Configurations, Access Policies, and Version Release Procedure

After project handover, you retain not only a working registry but also all artifacts by which it can be maintained, evolved, and handed over to another team.

Below is the set we record at acceptance: it is enough to connect a new data source or change release rules without contacting us.

• Registry configurations — parameter files for artifact storage and the metadata database, version retention and cleanup rules, binding to object storage, limits on artifact size and the number of concurrent publications.
• Integration schema — how the build pipeline passes the trained model, at which step quality metrics are checked, who registers a version and under which account, which external systems receive notification about a new version.
• Access policies — roles and service accounts with separation of rights for reading, publishing, and promoting a model to production, token lifetimes, procedure for granting and revoking access.
• Release procedure — version numbering rules, mandatory validation stages, approval points, conditions for promotion to production, procedure for rollback to the previous version and its maximum period.
• Registry documentation — description of the structure, mandatory fields of the model card (data source, metrics, owner, purpose), instructions for connecting a new project.
• Instructions for the on-call shift — what to check during a publication failure, how to restore access, where to look first when versions diverge between the registry and production.
• Responsibility matrix — who owns the registry, who approves the release, to whom escalation goes during an incident.

All of the above is maintained in your repository and handed over with change history. Describe the task — we will come back with an estimate and a plan for the scope in which this set can be assembled for your environment.

Case study: reduced failure rate after model releases

Case study of implementation: the customer had 14 models in production, and each version was released via chat.

Weights were stored in shared folders, metrics in a separate spreadsheet, training code in someone else's branch, and the only way to understand how the working version differed from the one that showed numbers at the defense was manually.

What was changed

Deployed MLflow Model Registry as the single source of truth for versions: each entry has an ID, links to the dataset, commit, and quality metrics, and the stages "training → validation → production" are set automatically.

The manual step "copy weights to the server" was removed entirely: release became promotion of a version through stages, and rollback became switching the pointer to the previous entry.

Metrics before and after

The reduction in failures was not due to the registry itself, but to the disappearance of manual artifacts: a class of errors disappeared — "the wrong version went to production" and "metrics were calculated on a different dataset." Residual failures are related to data quality and load — they are now visible at the validation stage, before release, rather than on live users.

How does Model Registry differ from code versioning?

Code and a model are versioned differently because a code version is text and configuration, while a model version is a combination of weights, training parameters, dataset version, and quality metrics.

A commit reproduces the build but does not reproduce the model: the same code on a different data slice yields different accuracy. Therefore, the difference between the approaches lies in what exactly is considered the unit of version.

Practical takeaway: a registry versus code versioning is not a replacement but a complement. Training code lives in a version control system, while the registry stores what does not fit into it and maintains links between model versions, data, and metrics.

Another difference is lifecycle stages. A registry entry goes through clear states: draft, under review, in production, archived. This answers the question of which model is currently running and on what data it was trained — without digging through chats and experiment logs.

Compatibility, artifact storage, and permissions: frequently asked questions

The model registry (Model Registry) is layered on top of the current stack rather than replacing it: training runs in the familiar environment, while the version, metrics, and artifact are added to the registry as an automatic step of the build pipeline or via the API.

Already trained models are migrated by import — weights, metadata, and version owner are attached to the record, and experiment history is not lost.

Artifacts are described by reference, not by copy: weights, configuration, feature schema, and metrics reside in object storage, while the registry keeps the version, lineage, and status.

We assign permissions by roles — training, release to production, rollback — and every action leaves an entry in the log.

Is the registry compatible with our training tools? Connection is made via the API and standard libraries, and the training environment remains the same. If there are multiple environments, we consolidate versions into a single list with metrics and artifacts.

What about artifact volume? A lifecycle policy keeps the latest versions per branch and cleans up intermediate builds — storage stops growing, while rollback to the previous release remains possible. Large weights go to the cold tier.

Let's discuss the task: where to start with model versioning

To make the implementation assessment based on facts rather than guesses, we need input data on the environment and models. Collect what you already have at hand — we'll clarify the rest in conversation.

• Which models run in production: task types, artifact formats, number of active versions for each.
• How models get from training to launch: manual deployment, scripts, build pipeline — and where this path breaks most often.
• Where artifacts and metadata are stored: file or object storage, image registry, experiment tables.
• Launch infrastructure: Kubernetes or virtual machines, cloud or own servers, isolated environments.
• What is required for reproducibility and audit: whether to store the dataset, artifact hash, training metrics, and release author.
• Who has the right to publish and revoke a version: roles, external contractors, access restrictions.
• Timelines for bringing a model into production and requirements for rollback to the previous version.

Based on this data, we'll return with a work plan: registry schema for your environment, naming and version lifecycle policies, procedure for migrating current models, integration points with training and launch, acceptance criteria. The implementation assessment will be tied to stages, not an abstract scope.

Describe the task and send the input data — at least as draft notes: we'll clarify details and return with a plan and timelines.