Setting up MLflow in Kubernetes: a complete DevOps guide

Why MLflow in Kubernetes loses data under load

While MLflow lives on a single node and there are few runs, everything looks functional: metadata in local storage, artifacts nearby, experiments are readable.

As soon as several parallel trainings appear in Kubernetes, the metadata database becomes a point of failure — when it overflows, records of runs are lost, and some experiments simply disappear from the interface.

The second class of failures is conflicts of concurrent runs: several pods write to the same table, transactions are blocked, some attempts fail due to timeout. Worse, metrics manage to be written, but parameters are not, and the experiment becomes incomparable with the rest.

A unified queue of runs and proper isolation are needed, otherwise the race for resources turns into a race for writing.

The third is a restart of a training pod. If artifacts and state are written to the container layer, everything that did not manage to get to external storage is lost, and rollback to the previous checkpoint is impossible because it does not exist.

Separately, the divergence of training and serving environments: the model is trained on one build of libraries, and serving is launched in another image — and the result differs with the same input data. Below we analyze how this is closed by configuration, not by heroism.

What changes in operations after switching to a managed environment

While the model lives next to the application code, its rollout is a manual operation: an engineer copies the artifact to the required server, edits the configuration, and hopes that the version of weights matched the version of the script.

A line 'updated' remains in the log, and which revision is currently in operation is a question for the one who did it. This gives rise to a whole class of failures: divergence of environments, in which the model behaves differently in test and production environments.

The managed environment removes the manual step. The model is registered in MLflow as a version tied to a code revision and training parameters, and the environment is assembled from a description — the same artifact goes to both environments without manual file substitution.

The model rollout time is reduced from hours to minutes: release becomes a routine operation, not an event with on-call duty.

Recovery after a failure ceases to depend on the memory of the on-call engineer. Rollback is switching to the previous registered version according to the regulation, not reassembly and guesses about previous settings.

Recovery time is measured in minutes, and the exact revision remains in the analysis: code, data, parameters.

The frequency of model releases changes as a consequence. When release is reversible and does not require manual edits, updates happen several times a week instead of once a quarter, and each feature change passes validation before entering the production environment.

Deployment modes of MLflow in Kubernetes: comparison and limitations

The choice of MLflow deployment option in Kubernetes comes down to two questions: how many teams share one model registry and where metadata and artifacts physically reside.

The answer determines whether a failure of one environment stops the work of others and how quickly the training history is restored.

How to choose

In practice, they more often come to a hybrid: metadata and artifacts are moved to external services with a separate lifecycle, and separation is done by namespaces.

We start with an inventory — how many teams, what requirements for isolation and recovery time — and from it grows a scheme that will not have to be redone as the number of projects grows.

Implementation order: from environment audit to handover to operations

We start with an inventory, not with installation: otherwise deployment simply transfers the accumulated mess into the cluster — duplicate experiments, artifacts in different places. The order of work is the same for a new environment and for migration from a local tracking server.

1. Environment audit. We calculate the load profile: how many runs go in parallel, how many metrics are written per run, average artifact size and retention period. We record external consumers — training systems, builds, analytical dashboards.
2. Component placement scheme. We determine what lives in the cluster (tracking server, metadata database, entry point) and what we give to managed services and object storage for artifacts. The scheme is agreed upon before the first line of manifests — changing placement later is the most expensive.
3. Environment assembly. We prepare deployment templates, separate development and operations environments, set resource limits and readiness probes. The cluster must see when a pod is actually ready to accept traffic.
4. Setting up access and storage. We differentiate rights by roles, move secrets to the cluster storage, connect object storage for artifacts and the metadata database. We immediately set the lifecycle: old artifacts are archived, not disappear.
5. Fault tolerance testing. We delete tracking pods under load, disable a node, break the connection to artifact storage. We check whether metric writing is restored, whether past runs are lost, and how quickly access returns.
6. Handover to operations. We provide the on-call regulations, analysis of typical failures and rollback procedure, train the team on real scenarios. The environment is accepted — with clear boundaries of responsibility.

Handover artifacts: configuration, schemes, policies and regulations

Handover does not end with the stand being launched and model training ongoing. Then the customer's team lives with this system on their own: brings up the environment after a failure, issues access to new engineers, changes configuration for load growth.

Therefore, the package includes not only working services, but also documents according to which all these actions are performed without us.

• Component scheme — which services are deployed, how they are interconnected, where experiment metadata and artifacts themselves reside, which nodes are considered points of failure.
• Deployment configurations — chart values files, service manifests, resource limits and requests, readiness probes, container restart policy.
• Access policies — roles and service accounts in the cluster, rights to namespaces, procedure for granting access to secrets and external storages.
• Recovery documentation — step-by-step bringing up the stand from scratch, procedure for restoring the metadata database and artifacts, target recovery time.
• Backup regulations — what is copied, at what frequency, where it is stored and how regularly recovery from a copy is checked.
• On-call regulations — who responds to monitoring signals, response time, escalation procedure, rules for changes outside working hours.
• Instructions for typical incidents — overflow of artifact storage, unavailability of metadata database, rollback to previous image.

Acceptance is based on these same points: the customer's engineers restore the stand on a test environment according to our documents, and if something does not match, we refine the wording before closing the work.

Case study: reduction of failures and recovery time after migration

A recommendation platform launched model training from MLflow on dedicated servers. The environment was assembled manually each time: dependencies, paths to storages, transfer of artifacts between machines.

Bringing up from scratch took up to two working days, and divergence of library versions caused about a third of failed training runs.

We moved management to the cluster: configuration was described with declarative manifests, metadata and artifacts were moved to external object storage with lifecycle rules. Teams have separate namespaces with quotas, training jobs have resource limits and readiness probes. A failed job is restarted automatically, and experiment state survives node failure.

What changed in operations

The share of failed runs dropped from a third to isolated cases, and the cause of failure is now read in the job logs, not reconstructed from memory.

The time to bring up the environment from scratch instead of two days takes less than an hour — the same set of manifests as in the production environment is deployed.

Manual interventions per week decreased from fifteen-twenty to two-three planned ones: updating images and rotating secrets.

The main thing here is reproducibility: the stand according to the template behaves identically for any engineer on the team, and rollback to a previous configuration version takes minutes.

Operational metrics — node load, job queues, restart frequency — are collected automatically, so degradation is visible before a user reports it.

How does launching experiments differ from deploying a model to service?

Training and deploying a model to service are different in the nature of the load, even when they live in the same cluster and look at a shared model registry. Batch training is a finite task: started, computed, wrote artifacts, finished.

Model serving is a continuously running process that must respond to requests at any moment, including during its own update.

The first consequence is the scaling mode. Batch training grows with queue length: tasks appeared — workers were raised, queue empty — resources returned to others.

Serving scales by incoming traffic, and here scaling lags: a new replica first loads the model and only then passes the readiness check.

The second is limits and scheduling. Training has peak consumption over a short interval, serving has even and predictable. On shared nodes, a batch job displaces serving and drops response, so different node pools and different resource requests are allocated for them.

The third is the update strategy. For training, recreation of the pod and restart from the last checkpoint is sufficient.

For serving, gradual replacement of replicas, liveness and readiness probes, canary release of a new version and quick rollback to the previous one are needed.

MLflow covers both halves — tracking, model registry with versions, artifact storage; the engineering work is to separate them into different objects and policies.

What is clarified before starting work: technical doubts

Before starting, we close questions that determine whether MLflow in Kubernetes will integrate into your existing environment or require rebuilding the environment. Below are the things that engineering teams most often clarify at this stage.

How are environments separated?

Each environment gets its own namespace, its own metadata database and its own artifact storage. The production model registry is isolated: debugging and experiments physically cannot write an artifact there bypassing the agreed publication path.

What about metadata backup?

Scheduled dumps of the metadata database plus versioning of artifacts in object storage. We verify recovery in full: the model registry and run history are brought up in a test environment, not just a copy is created.

Who is responsible for the system after handover?

You receive manifests in the repository, description of environment policies and instructions for update and rollback. We remain on support in the agreed scope: updates, incident analysis, configuration refinement.

Let's discuss the task and come back with an estimate and work plan

To ensure the estimate is based on your real environment, not an average template, for a start a description of the task and what already works is sufficient. The more accurately you show the current scheme, the fewer clarifications will remain before starting work.

What to send for calculation:

• description of the task: only experiment tracking in MLflow is needed, model registry, or full cycle with model deployment to service;
• current environment scheme: where training takes place, where data is read from, where artifacts are stored and how the result reaches the production environment;
• how experiments and models are currently versioned and what exactly hinders this process;
• environment limitations: own nodes or cloud, network policies, access isolation requirements, own image registry;
• load profile: how many runs per day, average and peak artifact size, how many of them need to be stored;
• failover requirements: whether the ability to return to a previous model version is needed and in what time it should happen;
• acceptance criteria on the customer side and who accepts the work.

In response, you will receive a solution scheme for your environment, a composition of stages with the order of work and acceptance criteria — what is considered a result at each step. Describe the task in free form: we will come back with an estimate and work plan.