Why AI development environment setup breaks on the second project
On the first project, everything looks harmless: one machine, one shared interpreter, libraries are installed on top of system ones and work.
The problem is that such an environment implicitly encodes the state of a specific person — what they installed and in what order, what they installed manually afterward, and what they forgot about.
The second project brings a second model and a different branch of libraries. The shared interpreter turns into a point of failure: one dependency requires the old line, another requires the new one, and installing the second breaks the first.
This is the environment dependency conflict — it manifests not at startup, but hours after training begins.
Then the machine factor comes into play. AI development environment setup is repeated manually on every machine: locally, on a server with an accelerator, for each engineer on the team.
Each time, a different combination is produced, and incompatible library versions come back as a discrepancy in model quality on the same code.
The result is lost reproducibility. A run cannot be repeated a month later, rollback to a previous state is unavailable, and any update turns into a rebuild from scratch.
Therefore, per-project environment isolation, pinned dependency descriptions, and build verification before training starts are not hygiene for hygiene's sake.
Below we look at how this is solved: PyTorch and similar training libraries live in a separate environment for each project, and the environment state is restored with a single command rather than manual steps from memory.
What a reproducible environment gives an engineering team
While dependencies live in the system environment, every machine on the team is a separate configuration: one has an updated driver and a fresh library build, another has leftovers from a previous project.
The error reproduces for the author and disappears for colleagues, and finding the discrepancy takes days. Dependency isolation removes this entire class of failures: the environment is assembled identically on a workstation, server, and cluster.
The isolation method determines how long it takes to deploy the environment from scratch and how predictable the result is.
| Option | When it fits | Limitations |
|---|---|---|
| System packages | One-off experiment | Breaks neighboring projects, rollback impossible |
| Virtual environment | One project, one engineer | Manual dependency pinning, version drift |
| Manifest with pinned dependencies | Team development, multiple projects | Requires discipline for updating the pin |
| Container image | Running on a server and in a cluster | Longer first build, accelerator passthrough required |
In practice, such a manifest reduces bringing up an environment from scratch to tens of minutes instead of days of manual installation, and onboarding a new engineer to one day instead of weeks.
The share of launch failures due to version discrepancies between machines drops to almost zero: the library version is pinned in a file, not in the developer's memory. An experiment conducted a month ago is repeated with the same result.
Our engineers build such environments on PyTorch and keep a separate manifest for each project — updating one library does not drag along a rebuild of the others.
Isolated environments vs. shared installation: comparison of approaches
How the environment is organized determines whether the result will reproduce on another machine and how many person-hours will go into sorting out dependency discrepancies. Below are four options that are realistically chosen between at the start, and the limitations of each.
| Option | When it fits | Limitations |
|---|---|---|
| Isolated environments on each machine | Multiple projects with different library versions, one or two engineers | Reproducibility rests on discipline: version discrepancies surface already during model training |
| Shared environment for everyone | Short one-off tasks, quick prototype | A library update by one engineer breaks work for the rest, version conflict stops the whole team |
| Containerization of the workstation | Reproducibility is needed between machines and when handing over a project | Accelerator access is configured separately, image is rebuilt when base layers are updated |
| Remote compute node | Local accelerators are insufficient, there are long trainings | Delay during debugging, access separation and a task queue are required |
In practice, the most stable combination is: a container as the unit of reproducibility, inside it an isolated environment for fast iterations and version swapping, and heavy runs go to a dedicated node.
We leave a shared environment for everyone only where tasks are short and not reused. This arrangement removes the main pain point — 'it works on my machine, not yours' stops being a reason for downtime.
How implementation proceeds: stages of engineering work
A local environment breaks not on the model, but on environment discrepancies: on one machine training runs, on another it fails on an incompatible library version, and a month later the previous result no longer reproduces.
That is why we conduct work from inventory to acceptance — each stage ends with a check, not a report.
- Environment audit. We inventory machines, accelerator drivers, and current environments, and record where training runs and where it fails.
- Dependency scheme. We build a map of packages and versions, bring training and model inference to a common set, isolate projects, and pin versions in manifests.
- Accelerator and access setup. We bring up drivers, check device visibility by the environment, distribute them between tasks, and separate rights to data and model storage by role.
- Reproducibility check. We deploy the environment from scratch on a clean machine and run the reference scenario: a discrepancy in metrics with the control run is grounds to stop acceptance.
- Update policy. We change versions only through a test environment, with metrics recorded and the ability to roll back to the previous set.
- Environment acceptance. We hand over deployment scripts, manifests, instructions, and a verification checklist — your engineers assemble the environment themselves.
What remains with you after handover
The environment stops depending on a specific person: it is assembled from manifests, checked against a checklist, and updated according to policy. A new engineer gets into work with one installation, not after a week of manually debugging 'why won't it start for me'.
Configurations, dependency schemes, and handover policies
The output artifacts are not a report, but a working kit: from it the environment is brought up, updated, and handed over further without the author's involvement.
Everything lies in the repository next to the code, so that a discrepancy between the description and the real environment is caught at review, not in a production run.
- Environment description files — a manifest with pinned versions and separate branches for the base layer and the project; deployment on a clean machine with a single command.
- Scheme of connections between projects — where the common base layer is, where the isolated environment with its own set of libraries is, and why the boundary is drawn that way.
- Access policies for compute nodes — roles, accelerator quotas, who is allowed to launch long-running tasks and change the common base layer.
- Environment documentation — directory map, variables, entry points, placement of datasets and saved models.
- Dependency update policy — frequency, test node for compatibility checks, rollback criteria, and procedure for a version conflict.
- Instructions for handing over the environment to the team — a new engineer's path from cloning the repository to the first training run, with typical errors.
- Acceptance checklist — a set of tests that are run on a clean machine before the environment is moved into the shared contour.
The kit is ready when the customer's engineer deploys the environment according to the instructions without prompts and reproduces the result on a test set.
We eliminate discrepancies between the description and reality before handover — otherwise the documentation becomes outdated in the very first week.
Case study: sped up environment deployment and removed discrepancies
The project is computer vision model training: six engineers, some work on local machines, some on a server with accelerators.
The environment was assembled manually according to instructions in the repository: environment manager Conda, training framework, binding to accelerator drivers. A new machine or OS reinstall turned into a separate day-long task.
Before the changes, building the environment took from 6 hours to two days if library versions conflicted.
About a third of runs failed not on code, but on dependency incompatibility, and discrepancies between the local machine and the server were searched for manually — reproducing someone else's experiment did not succeed on the first try.
What was changed
We converted environments to a declarative description: one file with pinned versions, separate sets for machines with and without accelerators, a local package cache, and compatibility checking at build time.
Deployment became a single command, the same for an engineer's laptop and the server, and a version update goes through review like a regular code change.
Building the environment from scratch takes 12-15 minutes, the share of failures due to versions dropped from a third of runs to zero and remains at that level. A new engineer gets to their first task the same day, and experiment results reproduce on any machine on the team.
What to do if dependencies have already diverged between machines
Dependency divergence manifests in a recognizably identical way: on one machine training runs, on another it fails already at import, and on the server the result differs from local.
The cause is almost always the same — the environment was assembled by hand and is not pinned anywhere, so updating a library for a new task quietly breaks a neighboring project.
The first step is not to rebuild everything from scratch, but to take an exact snapshot of the working environment: a full list of packages with versions, binding to bitness and system libraries, and separately — accelerator driver versions.
Such a snapshot deploys on a new machine as is, without re-resolving dependencies — this is the transfer of an existing environment without rebuilding and without surprises.
Next, we enable partial dependency isolation.
Keeping every project in a separate environment entirely is expensive and breaks the usual process, so the common base layer — interpreter, accelerator, typical libraries — remains single, and only conflicting groups are separated: those that pull different versions of the same packages.
On top — version freezing per project: each has its own pinning file, including transitive dependencies.
An update in one project stops affecting neighboring ones, and migration between machines turns into a repeatable operation: deploy the snapshot, run the control set of tasks, get the same result.
The effect is measurable: deploying a new workstation takes minutes instead of hours of manual installation, and the share of tasks that reproduce on a second machine approaches full.
AI development environment setup stops being a one-off act of heroism by one engineer and becomes a reproducible process.
Technical concerns: compatibility, access, updates
Environment isolation removes most conflicts, but not all: discrepancies remain at the level of system libraries and accelerator drivers. Therefore, before starting, we fix a compatibility matrix — which version combinations are verified in a pair, and which conflict.
Below are the things most often asked at the evaluation stage: how environments coexist, who gets access to nodes, what happens during an update.
How to handle version conflicts between projects on one machine?
Each project gets a separate environment with a pinned set of dependencies and its own library tree.
The environment is assembled from a manifest, so the same result is produced on the test bench and on the node with an accelerator — the 'it works on my machine' discrepancy stops being a problem.
Who gets access to compute nodes, and how?
We grant access by role: analyst, engineer, on-call — each has their own set of rights, and we do not use shared accounts. Heavy jobs run from service accounts with a short-lived key, and actions are logged: you can see who occupied a node and when they released it.
Can the environment be updated without downtime?
Environments are built as immutable images. The new version is rolled out next to the old one and run through the control set of tasks, after which traffic is switched to it.
If quality has dropped, returning to the previous image takes minutes instead of hours — no downtime is needed.
Part of the work happens on our own machines — is that compatible?
We describe the environment with a manifest, so the local developer environment repeats the server one: the same library versions and the same path scheme.
The only difference remains in the amount of accelerator memory, and code run locally does not have to be rewritten when moved to the shared node.
Let's discuss the task and come back with an assessment and plan
To keep the discussion substantive, it is enough to send what you already have at hand: a full set is not needed, even a partial set gives an understanding of the scale of the environment setup task.
- a list of models and scenarios that must work: training from scratch, fine-tuning on your own data, or inference only;
- equipment composition: accelerator models, their count, one machine or several nodes;
- data constraints: where they are stored, whether export outside the perimeter is allowed, whether environment isolation between teams is needed;
- reproducibility requirements: is it enough to bring up the environment with a single command from pinned versions, or is a portable image needed;
- operating mode: one-off setup or support with planned updates of libraries and drivers;
- who accepts the result and where the run will happen — cloud resources or your own servers.
In response, we prepare an environment scheme divided into base, working, and experimental contours, a staged implementation plan for environments with checkpoints, and the composition of handed-over artifacts: configurations, instructions for restoring the environment from scratch, update policy, and rollback procedure.
The assessment of engineering work is built on this scheme, so it shows which steps we do and which remain with your infrastructure team. Describe the task — we will come back with an assessment and plan.
