Why without a Model Registry model versions diverge from production
The model is trained, holdout metrics are acceptable — but the artifact goes into the production environment manually: by copying a directory, a script from an engineer's notes, sometimes on top of the previous version.
Two weeks later, no one can confidently say which set of weights and which feature preparation are running in production.
This leads to environment mismatch: one model version runs on the test bench, another in production, and the difference surfaces only during an incident. Manual deployment leaves no trace — there is no record of who, when, and exactly what was rolled out.
Model rollback has to be done blind, because no one saved the previous artifact together with its dependencies in advance.
MLflow and similar solutions have shown that a version registry is a separate service, not a folder on disk. A model registry stores for each version the artifact, metrics, input data schema, and stage: testing, production, archive.
Promotion to production becomes an explicit event with a log and checks, and the previous version remains available — rollback takes minutes instead of hours.
We break down how such a registry is built in your environment: from naming conventions to automatic stage transitions.
What a managed model registry provides: measurable effect
A model registry is not a file catalog, but a contract between those who train models and those who bring them into production.
While versions live in personal directories and chat threads, the team spends hours figuring out which artifact is running in the working environment and how it differs from what came out of the experiment.
Experiment reproducibility is measured by the share of runs that can be repeated weeks later.
In the registry, each version is tied to its dataset, training parameters, and environment, so a repeated run yields comparable metrics, not a result "of approximately the same order".
This is the condition under which a successful model can be defended before the business, rather than rechecked from scratch.
Model release time is reduced thanks to a ready-made route: the version is registered, passes checks, and reaches production without manually assembling the environment at every step.
Rollback time drops from hours to minutes — the previous version remains in the registry together with its dependencies, and you can return to it with a single operation.
Version transparency provides attribution: you can see who registered the model, what data it was trained on, what checks it passed, and where it is used.
This eliminates a whole class of failures — rolling out the wrong artifact, rolling back to a build that no longer exists, arguing over whose result is closer to the truth. Incident analysis comes down to comparing two records, not reconstructing history from logs and messages.
The result: the registry turns working with models from a set of personal practices into a repeatable process with measurable indicators.
Model Registry options: cloud, self-hosted environment, hybrid
The choice of model registry scheme is determined not by interface convenience, but by where the data resides and who is responsible for the infrastructure. The registry itself stores metadata: versions, stages, training metrics, links to artifacts, and their checksums. The weights remain in object storage — the scheme depends on its location.
The options are compatible with each other: the registry can be started in the cloud and later moved to your own environment without rebuilding models, if artifacts are addressed by hash rather than by file system path.
| Option | When it fits | Limitations |
|---|---|---|
| Cloud registry | No own infrastructure, short experiment cycle, data does not require isolation | Artifacts and metadata outside the perimeter; uploading large weights is limited by the channel; dependence on provider availability |
| Local environment | Sensitive data, regulator requirements, predictable load | Storage, backup, and updates are on the team's side; manual scaling; slower rollout of a new test environment |
| Hybrid scheme | Metadata and training runs in the cloud, weights and data remain inside the perimeter | Synchronizing environments, separate authorization, risk of version mismatch if the channel fails |
How to choose
In practice, the decision comes down to two questions: is it acceptable to store weights outside the perimeter, and are you ready to maintain object storage yourself.
A mismatch in the answers produces a hybrid — then it is worth defining in advance what is considered the source of truth: the registry or the artifact storage.
Implementing a model registry: from audit to handover
A model registry is not deployed "on a blank slate": first you need to understand what is already running in production and what exists within the team at all. Below is the engineering path from inventory to handover that implementation follows.
- Model audit. We compile a list: which models serve the product, who owns them, from which artifact deployment occurs. We separately mark duplicates and "orphans" — we decide their fate before setting up the registry.
- Registry scheme. We define the hierarchy "project → model → version → stage", naming rules, and mandatory metadata: training data, metrics, author. Reproducibility and rollback capability depend on this.
- Permissions setup. We differentiate roles by actions: version publication, promotion to production, read-only. We link the registry to the deployment environment so production does not pull a model from an experimental branch.
- Test environment. We run an end-to-end scenario on a copy: publication, transition between stages, rollback to the previous version. We check that permissions really cut off what they should and that metrics are reproducible.
- Knowledge transfer. We transfer working versions with history and record what remains outside the registry. We give the team the publication regulations, instructions for typical operations, and the incident analysis procedure.
What remains in your environment: artifacts and regulations
The work ends not with a demonstration, but with handover of the environment: the model registry, artifact storage, and the rules for working with them remain in your infrastructure and under your access control. Below is exactly what the team receives to continue releasing versions without our involvement.
- Metadata schema — mandatory fields of the model card: dataset, quality metric, owner, stage, training date.
- Registry and storage configurations — separation of development, acceptance, and operation environments, artifact backup settings.
- Access policies and roles — who publishes a version, who promotes it to operation, who has read-only access. Roles are linked to your user directory.
- Release regulations — the procedure for moving a version between stages, mandatory checks before deployment, and conditions for rollback to the previous release.
- Retention and cleanup rules — how many versions we keep per tag, what goes to the archive, and how rollback capability is preserved.
- Automation templates — templates for the typical path from training to delivery into operation, so a new model can be connected along a ready-made route.
- Documentation and acceptance checklist — environment diagram, instructions for engineers and analysts, and the procedure for incidents.
- Team handover — walkthrough of the environment with your specialists and an agreed support procedure after launch.
We fix the handover scope in advance: acceptance is based on a list, not on feelings, and each item has an owner on your side.
Case study: bringing order to model versions in production
Before
In production, 38 models were running, 14 of them lived outside the registry: weights were stored in directories on training servers, and the version was determined by file name and modification date.
Inventory took a week and still produced discrepancies — disputed versions had to be confirmed manually with the authors.
Rollback after a failed release proceeded without regulations: an engineer searched for the previous artifact, replaced the file, and restarted the service, hoping to replace exactly what had been working. On average 3–4 hours at night, twice a quarter — with a complete shutdown of the scoring service.
What was changed
First, an inventory was conducted: weights, quality metrics, and release dates were reconciled, dead versions were marked and taken out of circulation. Then a unified model registry based on MLflow was set up — each model got an owner, training set, metrics, and stage.
Release goes only through the registry: the build itself registers the artifact and transitions the stage, manual file upload to production is closed.
After
A quarter later, under registry management — 100% of models, version release was reduced from 4 hours to 40 minutes, rollback — from 3–4 hours to 8 minutes, during the day and without service shutdown.
Recurring incidents of the kind "an outdated model returned to production" stopped. Inventory now takes minutes: the list of versions, stages, and owners is exported from the registry with a single query.
How does a model registry differ from artifact storage?
Artifact storage answers the question "where is the file", while a model registry answers "what model is this and can it be rolled out".
The former operates with paths, sizes, and checksums: object storage will reliably preserve weights, but it does not know what data they were obtained from or which artifact was the previous one.
The registry does not duplicate binaries but holds a link to the artifact plus metadata: version number, parent version, hyperparameters, holdout metrics, training set hash, run author, and current stage.
It is the stages that turn a set of files into the model lifecycle: training, validation, production, archive.
The difference in approaches manifests at the moment of an incident. In the registry, rollback is moving the previous version to the production stage: a single operation with a clear trace in the log.
In file storage, rollback is searching for a directory by naming convention, and it works exactly until the first overwritten artifact.
Tool choice depends on scale, not fashion. While there are only a few models and one team, storage with a strict naming convention is enough.
As soon as parallel experiments appear, several teams, and audit requirements, reproducibility is lost without a registry — no one will answer which version is currently in production.
In practice, the scheme is hybrid: weights are stored in object storage, and the registry — for example, MLflow or its analogues — stores the catalog of versions, stages, and links between them.
The registry itself does not fix reproducibility: if training code and data are not versioned, the metadata will merely honestly record a non-reproducible run.
Technical doubts: addressing common questions
A model registry is almost never deployed from scratch: it is integrated into the existing environment — artifact storage, training system, deployment pipeline. Below are four questions most often asked before work starts.
We have our own artifact storage — will the registry have to be kept separately?
No. The registry stores version metadata, while the weights themselves remain in your storage — S3-compatible or file-based. The deployment pipeline pulls the model by tag via API; no duplicate storage appears.
How to differentiate access between teams and environments?
Through permissions at the level of projects and environments: read, publish, promote to production. Publishing to production is a separate permission, usually assigned to the pipeline's service account rather than a developer.
Will inherited models without metadata have to be rebuilt?
No rebuild is needed. Existing artifacts are imported with version assignment and a card: description, metrics, dataset. Models without source code continue to work but are marked as archived — the status in the registry is visible to everyone.
How much space will all model versions take?
The main volume is weights. A lifecycle policy keeps the latest versions per project and cleans the rest, rollback to the previous one remains possible; similar checkpoints are deduplicated by layers.
Let's discuss the task and return with an estimate and work plan
So that the model registry estimate is based on your environment rather than average assumptions, we structure the first conversation as a review of inputs. We need facts about how models currently live — then we calculate effort by stages, not as a single averaged figure.
- Problem statement: what the registry should provide — version traceability, reproducible rollback, access control to artifacts, or all together.
- Current state: where weights and metadata are stored, how experiments are recorded, who and how publishes a model to the production environment.
- Inputs from you: list of models and their purpose, requirements for separating test and production environments, version retention policies.
- Constraints: cloud, own servers or hybrid, permission differentiation, personal data regulations.
- Team composition: what roles are needed on your side — model owner, infrastructure administrator, acceptance owner.
- Stage timelines: critical dates, alignment with product launch, whether parallel migration of already running models is needed.
- Acceptance points: what we consider the result of each stage and by what checks we accept it.
We return with a work plan: sequence of stages, team composition with responsibility areas, timelines, and acceptance criteria for each step. If we see a risk during the review, we name it immediately, not in the middle of the project.
