Setting up MLflow for experiment tracking in production

MLflow: why experiments are not reproducible in production

Model training almost always starts in a notebook: a quick hypothesis check, a metric chart, a successful run.

Then branches with different feature sets appear, and results settle in personal folders — each engineer has their own numbering, their own exports, and their own “final” weights file.

A few weeks later, no one can say which data produced the best metric and what exactly was changed in the code at that time.

Scattered notebooks produce two failures at once. The first is reproducibility of runs: a repeated run shows a different value because the data version, split order, and initialization parameters are not fixed.

The second is handover to production: there is no object to which code, environment, metrics, and trained weights are tied, so the model is rebuilt from scratch, and comparison with the previous release is done by eye.

MLflow closes this by making a run a standalone record in a shared experiment store: parameters, metrics, artifacts, and code version sit side by side.

A model is registered as a version with its own stage, and what goes to production is not a file from a chat but a reference to a specific build with a full history — you can see how it differs from the previous one.

We set up experiment tracking so that it is used every day, not once a quarter: a server and separate artifact storage, a common naming convention for runs and tags, automatic logging of parameters from code, a model registry with stages and promotion-to-production rules.

Below we break down what parts such a configuration consists of and what it changes over the long run.

What centralized experiment tracking provides

Debug runs, notebooks on local machines, and metrics scattered across tables are the main reason a successful result cannot be reproduced a month later.

Centralized tracking in MLflow closes this gap: for every run, parameters, metrics, data version, environment, and model artifacts are stored in one registry.

A single registry of runs and artifacts

• Re-running a successful experiment takes hours instead of days: parameters, weights, and environment come from the record rather than being reconstructed from memory.
• The share of reproducible experiments is measured, not debated: the same set of parameters yields the same metric within an acceptable variance.
• Model artifacts are tied to the run, so the “model — data — code” link is not lost during handover to production.
• The time to hand a model over from development to the production environment is reduced: the supporting team does not need to figure out what was trained and on what.

Comparing models by the same metrics

• Quality, response speed, and robustness on the holdout set are calculated the same way for all runs and compared in one table.
• Quality discrepancies are visible before rollout — replacing a model is decided by numbers, not by the latest successful demo.
• The class of failures “trained with the wrong script or on the wrong data” goes away: mismatches between runs are highlighted automatically.

Let's review how this can be addressed in your environment.

MLflow deployment options: comparing operating modes

Deploying MLflow is not about installing a package but choosing where run metadata and artifacts live and who has network access to them. The mode is determined by two requirements: data isolation and who maintains the tracking server — your team or an external platform.

Data isolation and network access

The tracking server exposes a REST interface, so it is placed inside a private network and exposed externally through a reverse proxy with authentication and a role-based access model.

If artifacts contain sensitive samples, access proxying is enabled: the client works through the server and does not receive direct links to the storage.

Artifact and metadata storage

Metadata is kept in a relational DB, not in files — parallel runs and metric search on file storage quickly hit locking limits. Artifacts go to object storage with lifecycle rules: old runs are cleaned up, while rollback to the required model version remains possible.

How we structure implementation: engineering stages

Implementing tracking is not installing a service but moving model training into a managed form: every run gets a code version, parameters, metrics, and artifacts that can be revisited months later. We go through six stages, and each ends with an artifact that can be verified.

1. Audit of the training process. We examine how experiments are currently launched, where logs and metrics are stored, and how someone else's result is reproduced. The output is a process map and a list of gaps that tracking closes.

2. Design of the tracking scheme. We define the structure of experiments and runs, naming rules, the mandatory set of parameters and metrics, and access by team. We agree on the scheme before deployment — otherwise accumulated data will have to be relabeled.

3. Setting up storage and server. We bring up the MLflow server, connect object storage for artifacts and a database for metadata, and configure backups and cleanup of old runs. We link the model registry to the same storage so that a version and its files do not diverge.

4. Onboarding teams. We embed tracking into current training scenarios: parameters, metrics, and configurations are logged automatically. Engineers work in their usual environment, but every run goes into the shared registry.

5. Reproducibility check. We repeat a control run using the saved parameters, code version, and data, and compare metrics. We investigate discrepancies and fine-tune how the environment and data sources are recorded.

6. Handover and training. We provide access, naming policies, and instructions for typical scenarios, run walkthroughs with teams, and stay in touch until the work becomes routine.

The result is that tracking works as part of the normal development cycle, not as a separate tool.

What the customer receives: configurations and policies

We hand over not a running service but a reproducible set of artifacts: any engineer at the customer's side can deploy and maintain experiment tracking on MLflow.

Everything is stored in a version-controlled repository — configurations, schemes, and policies change through a normal change review.

• Configuration files for the server, artifact storage, and metadata database with descriptions of parameters and default values.
• An integration scheme with the training pipeline: logging points for metrics, artifacts, and model versions at each stage.
• Access policies and a role matrix: who launches experiments, who sees results, who changes settings.
• Documentation for deployment, updates, and onboarding new teams.
• Experiment management policy: run naming, tags, comparison and archiving rules.
• Recovery instructions: copies of metadata and artifacts, and the procedure for returning to a working state.

Access policies and roles

Access is based on least privilege: an engineer sees their own experiments, a department lead sees a project summary, and an administrator sees storage settings.

Roles are described in configuration, not by verbal agreement, so a new employee gets the required level in one operation, and someone leaving does not leave open access behind.

Experiment management policy

The policy establishes what counts as a completed experiment: a set of parameters, metrics, model artifacts, and a link to the data.

Runs without a description and tags are archived on a schedule — the registry remains readable, and comparing results takes minutes instead of searching through dozens of similar records.

Case study: how the share of non-reproducible runs was reduced

The customer is a team that maintains a churn prediction model and updates it every two weeks.

Before implementing tracking, experiment results lived in notebooks and chats: parameters were selected from scratch, some runs could not be repeated, and handing the model over to support stretched to three weeks — the engineer reconstructed context from the author's stories.

The share of reproducible runs hovered around four out of ten: the rest diverged in metric because the data slice changed, while hyperparameters and random seeds were not recorded anywhere. Re-running a disputed result took up to three days of manual reconstruction.

What we did and where we stumbled

We deployed MLflow in the customer's closed environment: a tracking server, artifact storage, and a model registry.

Each run began logging parameters, metrics, code version, and dataset, while the trained model went into the registry with a transition status — from draft to maintained version.

We stumbled on data versioning: at first we only logged the path to the dataset, but the slice at the same path was updated, and two runs with identical parameters produced different metrics.

We added a data snapshot and its fingerprint to the run metadata — the class of failures “same code, different result” stopped being a mystery.

The second snag was access: without permissions to write artifacts, runs failed midway until we separated the roles of author and maintainer.

After implementation, the share of reproducible runs is 96%, a repeated run is assembled in 20–30 minutes instead of three days, and model handover dropped from three weeks to two days.

The class of failures where a model behaves differently in the production environment than in the experiment went away.

How MLflow differs from custom experiment tracking

Custom tracking starts with CSV or JSON: a script appends run parameters, the final metric, and the path to saved weights. While there are dozens of experiments, this is enough — the file is opened and compared by eye.

The scheme breaks on parallel runs: several processes write to the same file, columns diverge, and some runs are lost without an error in the logs.

Moving to an SQL table removes concurrent writes but adds manual work. You need to maintain schema migrations, monitor field types, and link the run to the code version, dataset, and model artifact yourself.

The link relies on the convention “the date is in the file name” — the first unplanned rerun breaks it.

MLflow separates these roles: tracking server, metadata database, and object artifact storage. Each run gets an identifier, and parameters, step-by-step metrics, tags, environment version, and model files are tied to it.

Comparison and search across runs are built in, not assembled by a separate script on top of a table.

On top sits the model registry: a model has versions and stages, and transition between stages is logged as an event.

This answers the question that custom journals usually do not hold — which version is currently in production, on what data it was trained, and which run to roll it back to.

The switch is justified when several engineers work on experiments, reproducibility of runs is required, and managed model rollback is needed. For individual research, a CSV next to the weights remains simpler — moving it to MLflow just for a single location makes no sense.

Technical concerns before implementing tracking

Concerns before implementing experiment tracking on MLflow are usually the same: will it fit into the current stack, will the load increase, where will data reside, and who will stay in touch after handover. We address them point by point — without generic promises.

Will the infrastructure load increase?

The metadata server and artifact storage live in a separate environment and do not compete with training for accelerators. The retention policy keeps the latest versions per experiment and cleans up the rest, so occupied space stops growing.

Where does the data reside and who has access to it?

Metadata and artifacts remain in your environment, with no outbound traffic to external services. Access is delimited by roles: a researcher sees their own runs, a manager sees a summary, and exports and changes go into the audit log.

Who is responsible for the server and updates after handover?

We hand it over together with documentation and backup and recovery instructions. Our engineers remain on support at an agreed level — from incident analysis to updates, with a response time.

Let's discuss the task: what to prepare before work starts

So that the first conversation yields a concrete work plan rather than general considerations, it helps to collect several inputs in advance.

They determine how many stages will be included in the first iteration and where separate work with your infrastructure team will be required.

• Description of the training process: what data is used as input, how often you retrain the model, who launches the run and with which command.
• Current approach to running experiments: tables, notebooks, file logs — to understand what we migrate to tracking and what we leave as is.
• Integration points: where training is launched (on a workstation, in a cluster, in an orchestrator) and where model metrics and artifacts should go.
• Data isolation requirements: what cannot leave the environment, which datasets are anonymized, who has access to saved model versions.
• List of tasks for the start: what must work in the first iteration — parameter logging, run comparison, model registry with versions.
• Environment constraints: a closed environment without external access, an internal image registry, and an adopted access control policy.
• Team composition and roles: who owns the infrastructure, who is responsible for data, who accepts the result at the end of a stage.
• Acceptance criteria: by which indicators you will understand that experiment tracking in production operation works as intended.

With this set, we will come back with an estimate and plan: stages, tentative timelines, and areas of responsibility at each step.

If some inputs are missing, it still makes sense to start the conversation: MLflow deployment allows phased rollout, and the first iteration can be limited to run logging, while the model registry and automation can be moved to the second. Describe the task as it stands now — then we work according to the agreed plan.