Why CI/CD Python Falls Apart on Dependencies
A classic scenario: a developer runs tests locally, everything is green, the branch goes into the shared pipeline — and the dependency build fails during package installation.
The reason is not in the code, but in the developer's environment: it has accumulated over months and is not described in any project file.
The first source is the absence of a pinned version set. If dependencies are declared as ranges, and the lock file is not updated with every change, CI/CD Python installs what is in the index at the time of the run.
Two runs on different machines produce different dependency trees, and a package version conflict surfaces in someone else's branch.
The second is mixing tools. pip, Poetry, and Conda resolve dependencies differently: Conda pulls binary builds from its channels, pip from indexes, Poetry pins its own tree. One package installed bypassing the main manager makes the build non-reproducible.
A separate pain point is the interpreter version: a mismatch between the local machine and the image in the pipeline changes dependency resolution rules and breaks binary packages built for another version.
The cost of such instability is not just one failed build. A build failure at the installation stage eats up the whole team's time, teaches people to ignore red runs, and moves the search for the real error to release day. Let's break down how this is addressed.
What Changes in the Team After a Well-Tuned Build and Release
The effect of a well-tuned pipeline is visible in daily numbers: how long the build takes, how long rollback to a previous release takes, what share of release steps are done manually.
Below are four shifts that the client notices within the first weeks, without a separate metrics implementation project.
Builds in Minutes Instead of Tens of Minutes
Dependency and layer caching, a fixed base image, separation of package installation and test runs — the build stops re-downloading and recompiling the same thing.
The developer gets feedback on their commit, not at the end of the day, and edits stop piling up while waiting for a run.
Reproducibility Across All Environments
Poetry with a pinned lock file, package hashes, and one image for the local machine, test environment, and production environment remove "it works on my machine" discrepancies.
The dependency set is the same everywhere, so a failure reproduces locally, not only on the server five minutes before release.
Rollback to a Previous Release in Minutes
The artifact is built once and promoted through environments without rebuilding: exactly what passed checks goes into production. If a release goes wrong, rollback is switching to the previous tag, not rebuilding and searching for the right branch in a hurry.
The Class of Failures That Disappears
Environment drift, a forgotten dependency, an incompatible library version, a step skipped manually — these failures are caught during checks and do not reach users.
The share of manual operations in release drops, and with it variability goes away: the scenario runs the same way both by a person and on a schedule.
Ways to Build Environments for Python Projects
Python has no built-in environment pinning: there is one interpreter, and the set of libraries in the system branch is shared by all projects. If you bring up a second service alongside it with an incompatible major version of a shared dependency, the already working one fails.
A virtual environment provides isolation but not reproducibility: without version pinning, a build on another machine will pull in fresh transitive dependencies and code behavior will change.
The choice of dependency manager determines what exactly goes into isolation and how repeatably this is reproduced in the package build.
| Option | When It Fits | Limitations |
|---|---|---|
venv + pip + requirements.txt |
Small services, dependencies listed manually | No pinning of transitive versions — reinstalling produces a different set |
pip-tools |
requirements.txt already exists, version pinning is needed |
Direct and transitive lists are maintained separately, conflicts are resolved by rebuilding |
| Poetry / PDM | Libraries and services with dependency groups for tests and documentation | Its own version resolution algorithm, discrepancies with pip when building artifacts |
| Conda | Scientific stack and binary dependencies outside PyPI | Heavy channels, the environment compresses poorly into a minimal image |
| Container image as environment | Services where the build environment and runtime must match | Without layered cache, build time grows multiple times on every code change |
In practice, the choice goes along two axes: who resolves version conflicts — pip or the manager's own mechanism, and where the environment boundary lies — in a virtual directory or in an image.
For a service deployed as a container, it is reasonable to pin versions with a lock file and install dependencies as a separate layer: then a code change does not drag libraries along, and a version mismatch surfaces during the build, not at runtime.
Implementation Stages: From Repository Review to Handover
The order of work is the same for any stack: first understand what already exists in the repositories, and only then build the pipeline.
A skipped audit leads to the build being set up in one project and falling apart in the others, while environment drift surfaces during deployment. Below is the sequence we follow during implementation.
- Repository audit. We look at project structure and how dependencies are declared: requirements.txt with exact versions, pyproject.toml with Poetry, or a separate Conda environment. We note where the environment is assembled manually and how much time this takes from the team.
- Pipeline design. We agree on stages and their order: environment preparation, code checks, tests, artifact build, publication, delivery to environments. For each stage, we immediately define what counts as a blocking error.
- Environment setup. We bring the interpreter version and dependency composition to a unified form, enable package caching between runs — the next build no longer pulls everything from scratch.
- Test run. We run the pipeline on one repository or branch, measure stage times, and look for failures on empty ground: floating dependencies, lack of test isolation.
- Acceptance check. We go through scenarios critical for the client: failed tests block delivery, rollback to the previous artifact, parallel builds do not interfere with each other.
- Handover and training. We hand over the pipeline design, instructions for editing stages, and a breakdown of typical failures, and go through them with the team — after that, support is handled in-house.
Configurations, Diagrams, and Regulations: What Remains in the Project
A configured pipeline without documentation lives only until the first departure of the engineer who built it: a month later, no one remembers why dependencies are pinned this way and what will happen during rollback.
Therefore, together with a working build, we hand over a set of artifacts to the project repository and the team's knowledge base — they allow the process to be reproduced without the author.
- Pipeline configuration files with comments: where the build and test stages are, what launch conditions exist, what is cached between runs.
- Dependency management files for the chosen tool (pip, Poetry, or Conda): pinned versions, separation of sets for development and release.
- Environment diagram: what environments exist, what is deployed in each, how parameters differ, how the environment is brought up from scratch.
- Secret access policies: where keys are stored, who issues them, how rotation works, and what is guaranteed not to get into logs.
- Release procedure: the path from branch to production environment, who makes the decision, at what points rollback is possible, and what blocks release.
- Instructions for the on-call engineer: typical build failures, procedure for a failed run and rollback, escalation points.
The set removes dependence on verbal agreements: a new engineer enters the project using the documents, and an auditor sees who changed access and releases and on what grounds.
The release procedure and environment diagram remain working documents — they are edited together with the code, not restored after a failure.
Case Study: Release Pipeline for a Distributed Team
In a distributed team, building one service took from 18 to 25 minutes, and this time almost did not depend on the volume of changes.
The reason was not tests: half the step was eaten up by dependency installation, which each environment solved in its own way — services pulled fresh versions from the shared index, some projects were built with Poetry, and the analytics environment lived in a separate Conda environment.
The main result was this: the build agent and the developer machine produced different package sets.
What We Changed
Dependencies were brought to a single source of truth: versions are pinned by a lock file, installation comes from it, not from ranges.
Layer caching is configured so that environment rebuild starts only when this file changes, and the build itself in GitLab CI runs in an immutable image tied to a specific code revision. Checks were split into a fast set for every change and a full run before release.
Before: about 20 minutes per build, every fourth release failed — environment drift, failure at the installation step, manual fixes on the server.
After: the build fits into 6–7 minutes, one out of twenty releases fails, and rollback returns the previous image in minutes instead of manual debugging.
How Does Building a Library Differ from Building Services?
A library and a service differ not in repository contents, but in what is considered the build result. For a library, it is an immutable package that is published to a private registry and then lives in other people's dependencies.
For a service, it is an image that is deployed in a specific environment and remains running there for months. This is where requirements for versioning, artifact publication, and rollback diverge.
Library builds are checked with a matrix: several interpreter versions, both supported dependency managers, and separately — installing the package from the built artifact, not from source.
The versioning scheme here is semantic and strict: a change that breaks backward compatibility must raise the major part. Other teams depend on the package, and they update on their own schedule, not simultaneously with you.
Artifact publication for a library is irreversible in meaning: one version is released once, overwriting is forbidden, otherwise two different builds end up with the same name.
If an error is found in a released version, a new patch version is released; rollback is performed on the consumer side, which pins the previous number in its dependency file.
Service builds solve a different problem: obtain a reproducible image that behaves the same in the test environment and in production. Here they check application startup, data schema migrations, configuration and secret reading, and behavior under load.
Versioning is tied to the repository state and release tag, not to a package number — tracing "which code revision is currently running" matters.
Service rollback and library rollback are also different procedures. A package cannot be taken back in the registry: you can only mark the problematic version as unusable, and dependent projects will update their pins.
A service is rolled back by switching to the previous image — quickly, but only if schema migrations are compatible with the previous code. Therefore, data changes are prepared so that one release step allows both versions to operate.
Technical Doubts Before Starting Work
The questions that most often come up before starting work concern not tools, but risks: what will happen to the already working pipeline, how dependency conflicts are resolved, and what happens after handover. Below are short answers for each.
We already have a build pipeline set up — will we have to rewrite everything from scratch?
No. First we take an inventory: what environments exist, where dependencies are pulled from, where steps are duplicated. Usually most of it is reused — only dependency installation and caching stages change. We make the transition in stages so that the main branch remains working.
What should we do if dependencies of two services conflict by version?
We separate them into isolated environments and pin incompatible pairs in separate build targets. A conflict that cannot be removed by an update is localized in one service and does not block the others from building. We analyze conflicts using the dependency tree, not by guessing.
Can we do without our own package repository?
Yes, but with caveats: on every build we pull from external indexes and keep a local cache. Your own repository is worth bringing up when you need private packages, strict version control, or builds without access to the external network. We make the decision based on the number of services and isolation requirements.
How is build reproducibility ensured?
We pin exact versions, not ranges, and store the lock file next to the sources. The build runs in a clean environment, with pinned base images. This gives the same result on the developer machine and on the build agent, and also allows the release to be repeated months later.
Who maintains the pipeline after handover?
We hand over configurations, stage diagrams, and a description of typical failures with procedures. The client's team can maintain the pipeline itself — the documentation is designed for that.
If support is needed, we stay on for maintenance: fix build failures, update dependencies, and refine stages for new services.
Let's Discuss the Task and Come Back with a Diagram and Plan
To come back with a pipeline diagram rather than general words, one discussion of the task is enough for us. We break down how code is currently built and published, where checks most often fail, and which steps before a version release are done manually.
We need a minimum of inputs from you:
- repository or its structure: branches, tags, version release order;
- dependency files — what you use: pip, Poetry, or Conda, whether private package mirrors exist;
- list of environments and how they differ in access and environment variables;
- weak points over recent months: failed builds, manual steps, rollbacks;
- constraints: closed networks, access requirements, and security procedures;
- desired build and release time — the numbers you consider acceptable.
In response, we send the full pipeline diagram: stages, launch conditions, dependency cache, separation of checks and artifact publication.
Along with it — the scope of work with "what is included and what is not," a stage-by-stage work plan, and an estimate of timelines in working days, separately for setup and for handover to operation. If we see a risk at some step, we talk about it before the start, not in the middle.
Describe the task in free form — we'll come back with an estimate and plan.