Setting Up MLflow for Experiment Tracking in Production

Setting Up MLflow for Experiment Tracking in Production

Imagine a team of five data scientists running 50 experiments every week. Parameters saved in Jupyter notebook comments, metrics in Google Sheets, artifacts in random Google Drive folders. When after a month you need to reproduce the best model with F1=0.924, nobody remembers the exact hyperparameters: learning_rate 0.001, batch_size 32 or 64? Sound familiar? MLflow is an open-source platform that centralizes this chaos: logs parameters, metrics, artifacts, and models for each run. Our engineers (certified ML specialists with five years of experience) deploy MLflow turnkey with PostgreSQL and S3, turning disparate records into a structured system. We've implemented MLflow in 30+ projects, guaranteeing 99.9% SLA for production infrastructure.

Why MLflow became the MLOps standard

MLflow is backed by a community of 10,000+ commits, supports any ML framework (PyTorch, TensorFlow, Hugging Face, scikit-learn), works with Kubernetes, and has APIs for CI/CD. Alternatives like homegrown solutions require constant maintenance and often break. MLflow is a reliable tool that saves teams up to five hours per week on logging. Unlike Bash or Python scripts, MLflow provides a unified UI, flexible API, and integration with 50+ tools.

How to set up MLflow for production

Backend selection

For production we choose PostgreSQL — it handles dozens of concurrent sessions, supports SQL queries, and scales easily.

Artifact storage

Deployment with Docker

Here's a typical Docker Compose configuration we use in production:

---
services:
  mlflow:
    image: ghcr.io/mlflow/mlflow:v2.14.0
    ports:
      - "5000:5000"
    environment:
      - MLFLOW_S3_ENDPOINT_URL=https://storage.yandexcloud.net
      - AWS_ACCESS_KEY_ID=${YC_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${YC_SECRET_KEY}
    command: >
      mlflow server
      --backend-store-uri postgresql://mlflow:${DB_PASS}@postgres:5432/mlflow
      --default-artifact-root s3://mlops-bucket/mlflow
      --host 0.0.0.0
---

How to automate logging with autologging?

Simply call mlflow.autolog() at the beginning of your script. Autologging automatically records parameters, metrics, artifacts, and the model for PyTorch, TensorFlow, scikit-learn, Hugging Face. For fine-tuning, use mlflow.sklearn.autolog(log_models=True). This eliminates manual coding for each experiment.

mlflow.autolog()
mlflow.sklearn.autolog(log_models=True, log_input_examples=True)
mlflow.pytorch.autolog(log_every_n_epoch=1)
mlflow.transformers.autolog()

Implementation process

1. Analysis — we discuss your infrastructure, number of users, model types.
2. Design — we choose backend, storage, security level.
3. Implementation — deploy MLflow, configure autologging, integrate with Git and CI/CD.
4. Testing — conduct load testing, verify 99.9% SLA.
5. Deployment — hand over documentation, access credentials, and train your team.

Case study

A computer vision company with 5 data scientists previously logged experiments in Google Sheets. After implementing MLflow, experiment comparison time dropped from two hours to 10 minutes, and the share of reproducible runs rose to 90%. We confirm results with SLA — the standard that ensures transparency.

Timeframe and scope

Roughly 2 weeks to 2 months depending on complexity. Includes: MLflow installation with PostgreSQL and S3, autologging setup, documentation, team training (remote or on-site), and support during the rollout phase (1 month).

Local run and example experiment

For a quick start:

pip install mlflow mlflow server --host 0.0.0.0 --port 5000 

Example experiment logging:

import mlflow
mlflow.set_tracking_uri("http://mlflow-server:5000")
with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_params({"batch_size": 32, "epochs": 10, "optimizer": "adam"})
    for epoch in range(10):
        train_loss = train_one_epoch(model, train_loader)
        val_loss, val_acc = evaluate(model, val_loader)
        mlflow.log_metrics({"train_loss": train_loss, "val_loss": val_loss, "val_acc": val_acc}, step=epoch)
    mlflow.log_metric("test_f1", 0.924)
    mlflow.log_artifact("confusion_matrix.png")
    mlflow.log_dict({"feature_importance": feature_imp}, "artifacts/feature_importance.json")
    mlflow.sklearn.log_model(model, "model", registered_model_name="my-classifier")

MLflow Model Registry: managing model versions in production

MLflow includes a built-in model registry — a tool for managing the model lifecycle from experiment to production. Each registered model goes through stages: Staging → Production → Archived.

Transition a model to Production via API:

client = mlflow.tracking.MlflowClient()
client.transition_model_version_stage(
    name="my-classifier",
    version=3,
    stage="Production"
)

Automatic CI/CD rules for promotion: F1 on hold-out set > 0.92 and p99 latency < 100 ms. If a new version fails, it remains in Staging, production is not affected.

For teams of 5+, the model registry reduces the time to roll out a new version from 2–3 hours (manual artifact transfer) to 10–15 minutes via an automated pipeline. Each model version is stored with full metadata: hyperparameters, test-set metrics, dataset hash, and a link to the experiment run.

Conclusion

MLflow is a proven tool for experiment tracking that pays off within the first few weeks. Contact us for an assessment of your infrastructure — we'll prepare an architecture in 2 days. Get a consultation right now. We guarantee results: 99.9% SLA and transparent documentation.