Setting up DVC for data and ML model versioning

DVC setup: where model reproducibility breaks

A telling picture: a model shows good quality in validation, it is rolled out, and a month later the same result no longer reproduces.

The code in the repository has not changed, but the dataset on which it was trained has been overwritten — some labels were corrected, some exports were replaced, some files were deleted.

There is nowhere to return: no one tracked data changes, and the old weights have nothing to compare against.

The cause is not the model, but that code and data live by different rules. Code goes through review, has history and rollback points, while data sits in a network folder or object storage, where a new export overwrites the previous one.

Model training is launched "on what is there now," and the link between the data version and the resulting metric is kept only in the engineer's memory or in a folder name.

The cost of such a scheme is diverging results and lost reproducibility of experiments. An engineer spends days trying to repeat a previous run, compares metrics that cannot be compared, and the argument about "what changed" turns into a message thread. When an unsuccessful update needs to be rolled back, it turns out there is nothing to roll back to.

DVC setup closes exactly this gap: versioning datasets and configurations, binding model weights to a specific data snapshot, and predictable run repetition. Below is how it works and where it most often breaks.

What data and model versioning gives you in practice

Versioning data and weights closes a narrow but expensive class of problems: the divergence between what the model was trained on and what is currently in storage.

As long as the dataset and the training artifact live separately, every repeated run is a lottery: metrics diverge, and the cause is searched for days.

DVC links the dataset, configuration, and weights into one commit, so the state of an experiment is restored by identifier, not by the engineer's memory.

The first measurable effect is the share of reproducible runs. Data is addressed by content hash rather than by a path in the file system, so one commit produces matching metrics both on the developer's machine and on the training server.

Incident analysis starts with a version identifier rather than a survey of the team: divergence between runs drops from "every other one" to isolated cases.

The second effect is time. Unchanged stages are not recomputed: data preparation, training, and validation are cached by input hashes, and a repeated run brings up ready artifacts instead of a full cycle.

Environment setup also stops being manual — dependencies and parameters are fixed in configuration, and a new engineer gets a working environment with one command.

The third is model version history and rollback to a previous version. Weights are tied to a specific data state, so returning to a previous release is a pointer switch, not retraining.

The class of errors "the model in production was trained on different data" goes away: a mismatch between data and weights becomes visible before rollout.

Version storage options: comparing approaches

The storage location for versions determines how quickly the team rolls back to a previous dataset and what happens when two engineers edit labels at the same time. We examine four options that actually occur in projects and their limitations.

What we consider when choosing

In practice, these options are combined: object storage becomes the main location for data, the model registry becomes the point for releasing versions into production, and the network drive remains a cache to speed up reads.

The key limitation here is not write speed but predictability: a version must be restored by identifier, not by a file name with a date in it — otherwise rollback turns into a manual search for the right copy.

DVC setup step by step: from audit to handover

The order of stages here is not a formality: if you start with storage but do not agree on a scheme, the repository becomes overgrown with copies of datasets, and no one can say which data version corresponds to which commit. So we go from audit to policy.

1. Audit of the current process. We examine what is already in Git, which datasets and weights live outside the repository, how often they change, and how much space they take. We record bottlenecks: manual artifact transfer, lost intermediate samples, non-reproducible training runs.
2. Versioning scheme. We determine what to bring under control — raw data, intermediate slices, training configurations, model weights — and by what rules a data version is linked to a commit and tag.
3. Storage connection. We configure remote storage, delimit access by role, enable a local cache so repeated exports do not move the same files again, and set a retention policy for old versions.
4. Description of reproducible steps. Data preparation and training steps are described declaratively: any past experiment is brought up with one command rather than restored from memory.
5. Automation of checks. Checksums signal when data changes, and in continuous integration we configure build and metric comparison: a degraded result does not pass into the main branch.
6. Handover to the team. We provide the policy, instructions for real scenarios, and the person responsible for storage. After that, the team versions data itself and rolls back to the required version without external support.

What the client receives: configurations and policies

We hand over not only working files but also everything your team needs to work independently with data and model versions. We agree on the package contents at the start, and at acceptance we check against it, not from memory.

• Pipeline configuration files: description of data preparation, training, and evaluation stages — repeating an experiment is launched with one command rather than assembled manually.
• Storage schemes: a map of where datasets, metrics, and model weights are stored, what is versioned, and what is restored from cache.
• Storage settings: connection of remote storage and cache, path separation by dataset and version, rules for evicting outdated objects.
• Access policies: roles for engineers, analysts, and contractors, separation of read and write permissions, procedure for issuing and revoking keys.
• Version handling policy: when an experiment branch is created, how a model is moved into production, and how rollback is performed.
• Documentation: repository structure, instructions for onboarding a new participant and for restoring the working environment from scratch.
• Acceptance checklist: scenarios we run during handover — reproducing training, rolling back a version, granting access to a new engineer.

The package remains in your environment: external dependencies are listed separately so that nothing ends up outside it.

Case study: reduced time to reproduce results

Before

The project was model training on data updated weekly. Dozens of experiments lived "however it turned out": samples on disk, weights next to the code, launch parameters in the engineer's notes.

Re-running training took 6–8 hours: bring up the environment, find the required data version, reassemble the configuration. In 6 out of 10 cases, metrics diverged from the previous run, and rolling back a model meant half a day of manual rebuilding.

What changed

Sample and weight versions were moved from the working folder to separate storage; only pointers to them remained in the repository. One configuration file describes the entire run: sample splitting, preprocessing, stage order.

DVC was configured so that a repeated run pulls the same data and restores the environment, and returning to a previous version is a pointer switch, not manual rebuilding.

Project result: an experiment stopped being disposable — it can be repeated and defended. Version rollbacks stopped being a risky operation: the previous state is restored precisely, together with data and environment.

How does data versioning differ from code versioning?

Code and data change differently, and this difference determines everything else. A code edit is a few lines of text, so a version control system stores only the difference between adjacent states.

A dataset is a binary file of tens of gigabytes, and the difference between two exports is almost always comparable to the full volume.

That is why a dataset is not put in the same repository as code: line-by-line comparison does not exist for it, the delta is not calculated, and every commit doubles the storage size.

Models behave the same way — comparing them line by line is meaningless, but linking them to the data and parameters they were trained on is meaningful.

For data, pointer files with a checksum go into the repository, while the datasets and weights themselves go into object storage; this is how DVC setup works.

Version matching is checked by checksum: the same dataset produces the same identifier, and rollback to the required combination of data and model is performed with one synchronization command.

The criterion for matching results is also different. For code, it is enough for tests to pass on the same revision; for a model, it matters that the metric reproduces on a fixed dataset with the same training parameters.

If at least one of the three components — data, code, parameters — is not pinned, the training result stops being repeatable.

Technical concerns: volumes, storage, access rights

Engineering questions before setting up versioning are almost always the same: large data volume, several teams with different permissions, an already working training process.

DVC is designed so that only pointer files remain in the repository, while the artifacts themselves live in separate storage.

Below are answers to what is most often clarified before work starts. If your case is not in the list, we will address it separately.

How do we delimit access to data and models?

Permissions are granted at the storage level: roles for read, write, and delete, separate keys for service builds and for engineers. Keys are not stored in the repository — only references to environment variables. Access to artifacts remains in logs, so it is visible who uploaded a version and when.

We already have versioning deployed and our own training process. Will we have to break it?

No — versioning is layered on top of the current folders and scripts: data is indexed as is, and training is launched in the usual way. The data version is fixed in the same commit as the training code, so reproducibility appears without rewriting the process. The transition is step by step, with rollback to the original scheme.

Let's discuss the task: where to start versioning

To come back with a scope estimate and work plan, we need to discuss the task concretely — based on your current process of working with data and models. Some answers can be approximate; we will clarify the rest ourselves when reviewing the repository and storage.

• Where data and models currently reside: local disks, network directories, object storage — and what volume with monthly growth rate.
• What counts as an artifact: model weights, training metrics, configurations, prepared samples — and which of them are needed in production.
• How many engineers work with one dataset and whether permission delimitation and change review rules are needed.
• How the current training cycle is structured: scripts, notebooks, an orchestrator, manual steps — where versioning will fit and what will have to be rewritten.
• Requirements for artifact storage: external or internal environment, retention and deletion policy, access from an external network.
• What infrastructure already exists: a repository with change history, a build server where training runs.

These answers form the starting order: audit of current storage, branching and tagging scheme, remote storage setup, migration of existing datasets, and a work policy for the team. These are the project inputs — without them, the scope estimate will be approximate. Describe the task in a convenient form, and we will come back with an estimate and work plan.