Setting Up ClearML for Experiment Tracking and MLOps

ClearML setup: where experiment tracking breaks

An experiment shows a metric improvement, but two weeks later it cannot be reproduced: runs were launched from local machines, hyperparameters remained in a notebook, and weights in a folder on disk.

Experiment tracking is enabled only after training, when reconstructing the history is too late. This is exactly where ClearML setup begins — not with installing the server, but with answering the question of where exactly results are lost.

Scattered training logs are the second loss point. Metrics are written simultaneously to the console, a text file, and a table maintained manually by an analyst; comparing twenty runs turns into a manual summary where some rows disappear during transfer.

Add different environments among engineers — and the same set of parameters yields different numbers.

The third point is data and model versions. The dataset is edited "in place," without a snapshot or checksum, the model is saved under a name with a date, and the run configuration lives separately from the code.

The link "data — parameters — metric — artifact" breaks, so rolling back to a successful run and proving comparability of results become impossible.

The cost of such organization is measured in time, not convenience: reproducing a successful run takes days, some conclusions have to be verified anew, and no one counts lost run results — there is simply nothing to compare them against. Below, we break down how this is addressed.

What managed tracking provides: measurable effects

Without managed tracking, the result of a run depends on the engineer's memory: which parameters were set, which dataset was supplied. A week later, reproducing a successful run is no longer possible — it is unclear which configuration produced the desired metric.

ClearML records each run as an entry with code, parameters, environment, and metrics, and it can then be handled like a version in a repository.

Experiment reproducibility

Before implementation, about half of the runs were reproduced; the rest diverged due to unrecorded parameters and data versions.

Now the configuration, commit, and dataset checksum are saved automatically, so the same run produces the same result, and any divergence is explainable rather than random.

Data versioning and a unified artifact storage

Weights, checkpoints, and learning curves are stored in one storage rather than on engineers' local disks. Data versioning ties the model to a specific sample slice, so it is clear what it was trained on and what changed between versions. Finding the required model version is reduced from hours to a few minutes.

Comparing model runs

Metrics and hyperparameters from dozens of runs are consolidated into one table where the contribution of each change is visible: what produced an improvement and what turned out to be noise.

A whole class of errors disappears — "the wrong model version went into production" and "it's unclear what was changed last time." Decisions are made based on data, not on the memory of the most senior person in the team.

Comparison of deployment modes: ClearML setup for the task

The tracking server stores metrics, artifacts, and the model registry — where it lives determines which data leaves your perimeter and how quickly compute nodes connect to it.

The mode is chosen not by preference, but by data isolation requirements and by where tasks are physically computed.

The difference shows not at the start, but during maintenance. The local option provides full control over isolation but requires your own infrastructure.

The external one removes that burden but changes the risk profile: you decide what exactly is permissible to transfer outside the perimeter. Hybrid meets both requirements but pays for it with a more complex access and synchronization scheme.

We start with an inventory: which data cannot be taken outside, where compute nodes live, who needs access to the results.

Based on this set, the mode is determined unambiguously — and switching between local and mixed schemes remains possible without rebuilding tracking processes.

Implementation stages: from audit to handover to the team

First, we understand how the team currently conducts experiments: where run logs accumulate, how hyperparameters are recorded, and from which run the model can be reproduced. Without this, the tracker quickly turns into a dump of heterogeneous records. Below is the sequence of work by which we conduct ClearML implementation.

1. Audit of the experiment process. We collect current points: local notebooks and scripts, manual tables with metrics, folders with artifacts. We note what is set directly in code and not recorded anywhere.
2. Infrastructure scheme. We determine the placement of tracking services, artifact storage, and the task queue, their integration with compute resources and the network. We agree on the scheme before setup begins.
3. Access separation. Roles by project: who launches training, who sees metrics and artifacts, who changes configuration. We connect corporate authentication so accounts are not created manually.
4. Configuration and templates. We configure automatic collection of hyperparameters and metrics, artifact retention rules, cleanup by number of runs, and comparison of experiments with each other.
5. Scenario validation. We run the team's real tasks: training from scratch, fine-tuning, returning to a previous run, recovery after interruption. A scenario is closed only with a reproducible result.
6. Handover to the team. We provide documentation and regulations for working with the tracker, train engineers, and designate those responsible for configuration. We stay in touch until the processes stabilize.

The outcome of each stage is an artifact: a process map, a perimeter scheme, an access matrix, a configured stand, a validation report, and regulations.

What remains in the repository after implementation

ClearML setup ends not with a working server, but with a set of artifacts that the team uses to continue without us. Everything lies in the infrastructure repository under version control — with edit history and review, not in chat threads and someone else's memory.

• Server configurations — declarative deployment files for the server, task queue, metadata and artifact storage; the stand is brought up with one command on a clean environment.
• Perimeter schemes — how training, the prediction service, and data storage are separated, where network boundaries run, and which services are accessible from outside.
• Access policies — roles and scopes: who sees others' experiments, who can delete artifacts, how tokens are issued to agents and external systems.
• Experiment regulations — rules for naming projects and runs, mandatory tags, linking results to a commit and data version, and what counts as a reproducible run.
• Documentation — architecture, failure points, update and rollback procedures, requirements for backing up metadata and artifacts.
• Instructions for the team — short scenarios: bring up an agent on a workstation, compare two runs, move a model to the prediction service.
• Templates and scripts — run configuration templates, automatic checks before queuing a task, export of metrics to a shared dashboard.

We deliver each item based on actual results: we deploy the configuration on a clean environment, walk through the instruction together with your engineer, and record the result.

After that, the team adds experiments and changes policies itself — without contacting us for explanations.

Case study: how a team stopped losing experiment results

Before: a successful run was lost after a week

The model showed the required quality, but a week later no one could say what data and parameters it was trained on.

Run configurations lived in notebooks and chat threads, metrics in scattered files, and artifacts were put into a shared directory with names like "final_v2." A successful run could be reproduced in about one third of cases, and finding the needed result took from 40 minutes to an hour — the run history was reassembled from colleagues' messages.

What was changed and what was achieved

We deployed ClearML in the customer's perimeter: server, task queue, and artifact storage, with role-based access separation and integration into the existing training pipeline. Setup took several days.

Each run now automatically records parameters, data versions, per-epoch metrics, and the final model — without manual actions by the engineer. We added run comparison and a retention rule: we keep the last 10 checkpoints per task, and the rest goes to archive.

The share of reproducible runs increased from one third to more than 90%: configuration and data are restored from history in one click. Result search time decreased from 40–60 minutes to 2–3 minutes.

Storage volume stopped growing linearly — the artifact registry was reduced by a factor of three, while rollback to previous model versions remained available.

How does managed tracking differ from results spreadsheets?

A results spreadsheet is manual accounting: an engineer fills in a row, and the link between the row, configuration, and trained model file is kept in their head or in a cell comment.

While there are dozens of runs and one person handles experiments, this is enough — the spreadsheet is flexible, visible to the whole team, and requires nothing to be set up.

It breaks down during parallel work. Two engineers edit hyperparameters simultaneously, the summary diverges from what was actually run, and the best checkpoint is overwritten by the next iteration.

Reproducing someone else's result a month later is no longer possible: the spreadsheet contains neither the code version, nor the environment, nor the order of runs.

The growth threshold is visible by three signs: more than two or three people involved in experiments, dozens of runs per week, and an external requirement to explain where a metric in a report came from.

From that moment, the spreadsheet becomes not the source of truth but a retelling of it, and divergence from reality costs more than a deployed server.

Managed tracking does not eliminate the spreadsheet — it remains a convenient summary for management.

The data source changes: ClearML itself records the parameters, metrics, and artifacts of each run, so returning to a previous checkpoint or analyzing a disputed result can be done in minutes, not a day of searching through chat threads.

Let's discuss your task: assessment and implementation plan

An accurate estimate of the scope of work comes from facts, not general statements, so a short set of inputs is needed before the start. We gather them in one conversation so we don't have to come back with clarifications later or rebuild the implementation plan midway.

• Which models are being trained now and where configurations, datasets, and weights are stored.
• On which machines training runs: your own servers, rented accelerators, or a cluster, and who has access.
• Who is involved in the work: researchers, engineers, and the person who will remain to support the system after launch.
• Which metrics and artifacts need to be recorded for each run and how long to retain history.
• What is already used for experiment tracking: whether we migrate accumulated data or build from scratch.
• Requirements for access, data placement, and logs: what must remain inside your perimeter.
• Time constraints: when the first working result is needed and where rollback to the previous scheme is mandatory.

In response, we send an implementation plan: stages, scope of work, areas of responsibility, and points where your participation is required.

We provide a preliminary estimate within two business days after the conversation, and a detailed plan after access to repositories and infrastructure, so the numbers are based on the actual state of the project rather than assumptions.

The call format is simple: 40–60 minutes with the engineer who will lead the project and one person on your side who knows the infrastructure and constraints. Describe the task — we will return with an estimate and implementation plan.