Builds on Drone CI: what breaks as projects grow
While there are five projects, the build pipeline rests on one person: they remember the order of steps, know which secrets to substitute, and edit configuration by hand.
At thirty projects and two dozen engineers, the same pipeline starts to slow down not because of code, but because of how work is organized.
The first bottleneck to emerge is build capacity: a shared pool of runners is divided among all branches, and the queue of build jobs grows faster than the team.
An application build that took minutes yesterday waits for a runner to free up, and the developer stares at a status instead of working.
The second source of failures is environment drift and hidden dependencies. Libraries, cached artifacts, and variables that someone set in the UI and did not move into configuration: locally everything is green, but on a clean runner it fails.
Such dependencies are not visible in the repository until the environment changes or access to someone else's account is revoked.
Finally, manual deployment: rollout depends on a person who “knows how to do it right,” and on vacation they become a single point of failure.
Drone CI removes this dependency — the build pipeline is described as code: steps, environments, rollout rules, and approvals live in the repository next to the application. We explain how this is addressed.
Build and deployment automation: a measurable effect
The first thing that changes is the time from commit to verified artifact.
Stages are split by dependencies, layer and dependency caches survive between runs, so a typical change builds in minutes rather than tens of minutes, and the developer gets feedback before switching context.
The share of parallel steps is the second measurable parameter. Linters, unit and integration tests, and image checks run as independent branches, and the critical path is reduced to the longest of them: total machine time even grows, while calendar time falls.
Release frequency rises as a consequence. When deployment is one button and rollback is also a single operation, the team ships in small batches several times a day instead of one risky release per week.
A small batch is checked faster and rolled back more cheaply, and incidents become fewer simply because a release stops being an event.
Separately, about the class of failures that is eliminated. Environment drift and manual edits on servers go away: Drone CI runs the same set of steps for the test and production environments, and cluster state is described declaratively.
The “it worked in test, not in prod” class is closed at the build stage, not during a post-incident review, and recovery time is measured in minutes — rollback is a re-run of a pinned version, not manual archaeology.
Build execution modes: cloud, self-hosted agents, hybrid
The execution mode determines where build steps physically run, which networks and secrets a job can access, and who is responsible for the node.
In Drone CI, each job runs in a separate container, so build isolation is provided at the step level — but the agent that starts those containers is chosen by the team.
There are usually two questions when choosing: is access needed to private resources — an internal image registry, a secrets store, services in a closed environment — and how strictly do jobs from different teams need to be separated?
| Option | When it fits | Limitations |
|---|---|---|
| Provider cloud agents | Need a fast start, no own infrastructure, builds of public code | No direct access to a closed environment, resources are shared with other tenants |
| Own persistent agents | There are internal dependencies and own access control requirements | Nodes persist between runs: leftover files, cache, and working directory state need cleaning |
| Ephemeral agents in a cluster | Need a clean environment for each build and load management | Require a cluster, permissions and limits setup; cold start is longer |
| Hybrid scheme | Branch builds go to the cloud, release builds go to the internal environment | Two pools need to be maintained and synchronized |
In practice, the hybrid approach is chosen most often: ordinary builds do not occupy internal nodes, while release and secret handling remain in a controlled environment.
There is one condition — the same images, plugin versions, and cache rules in both pools, otherwise the result stops being reproducible. The mode is set in the agent configuration, so the scheme can be changed without rewriting build files.
Drone CI implementation: stages of engineering work
The switch happens without stopping releases: while the new pipeline is being tested, the old one continues to build branches. Below is the sequence of stages we go through when implementing Drone CI in Kubernetes.
-
Build process audit. We look at what is still built manually, at which build steps failures happen most often, how long the path from commit to environment takes, and where access keys are scattered. The output is a map of the current state and a list of builds to migrate first.
-
Pipeline design. We determine where the control server and agents are located, how they access the cluster, which branches correspond to which environments, and who can see secrets. The design is fixed before setup — it defines all other decisions.
-
Agent and policy setup. We separate agents by job types, enable build layer caching and resource limits, and define run rules: what builds on every commit, what builds on a tag, and what waits for manual approval before the production environment.
-
Build migration. We migrate repositories gradually, one service at a time, with the old pipeline running in parallel. The team sees the result on the first build, not after a full switchover.
-
Verification on a real release. We run a normal release, not a demo one: from commit to environment, then rollback to the previous version and a re-run. We check whether the build is reproducible with the same code and how permissions and secrets behave.
-
Documentation handover. We provide the pipeline design, agent configurations, run policies, and instructions for on-call engineers: how to triage a failed build, how to restart a step, and how to grant access to a new engineer.
What you keep: configurations, designs, policies
We do not leave the customer with a working but opaque pipeline: the pipeline on Drone CI is fully described as code in your repository, with designs and documentation next to it.
An engineer on your team can change a build step, add an environment, or roll back a release without contacting us.
- Build configurations — descriptions of steps and run conditions: by tag, by branch, manually. With comments explaining why the order is the way it is and where the rollback point is.
- Access policies — who runs builds, who changes configuration, and who deploys to production; developer and on-call roles are separated.
- Infrastructure designs — the entire pipeline: repository, build agents, image registry, cluster and environments, secret storage locations.
- Recovery procedure — the order of actions when an agent fails, registry access is lost, or a release fails, with time allocated for each step.
- Team instructions — how to reproduce a build locally, add a new service to the pipeline, and what to do with a failed build at night.
- Secrets and variables map — where a value is stored, who owns it, how and when to rotate it, so you do not have to search for it during an incident.
- Acceptance checklist — checks by which you can verify yourself that builds and rollback work without us.
Case study: shortened builds and removed manual deployments
A product pipeline came to us: several services, a shared build server, and a deployment procedure in a wiki.
Releases were started manually from an on-call engineer's local machine, so build and rollout stretched to more than an hour, and the result depended on whose machine in the team “worked.”
Pipeline before changes
The median build took about 38 minutes: steps ran sequentially and pulled dependencies from scratch every time. Automated deployments were less than a fifth; the rest were manual, with manual rollback as well.
A failed release meant a night shift: returning to the previous version took hours because no one remembered exactly which deployment steps had been skipped.
What we changed and what we got
Build and deployment were moved to Drone CI: the pipeline is described declaratively and lives next to the code. We added dependency caching between runs, parallelized independent steps, and made images immutable and pushed to the registry for a specific commit.
Promotion between environments goes through the same pipeline, and deployment starts by tag, not by button.
The median build dropped from 38 to 9 minutes, and the share of automated deployments rose to 95%. Rollback is a re-run of the previous successful pipeline: a couple of minutes instead of hours.
If the test check or the pod readiness check after deployment fails, the rollout does not reach production and the environment stays on the previous version — the failure class “configurations diverged between environments” is gone entirely.
Technical concerns: secrets, cache, self-hosted agents
Before starting, the same set of practical questions usually comes up: where secrets are stored, whether cache breaks reproducibility, whether self-hosted agents are needed, and what to do with an already working pipeline. Below are short answers on how we handle this in Drone CI projects.
Where are secrets stored and who can see them?
Secrets live in Drone CI itself — at the organization or individual project level, split by environment. They are injected into a step only for the duration of execution and masked in logs, so values never end up in the repository or in built artifacts. Access is granted by member permissions, not to everyone who has access to the code.
Cache speeds up builds — does it break reproducibility?
The cache is keyed by a hash of the dependency file and the target branch: if the manifest changes, the cache is rebuilt from scratch. We cache only third-party dependencies and do not cache the results of our own build, otherwise stale code could leak into the artifact.
For release builds, cache is forcibly disabled: slower, but the result is verifiably repeatable on the same revision.
Are self-hosted agents needed, or are cloud ones enough?
Cloud agents are enough for typical tasks: build, tests, image publishing. Self-hosted ones are needed when a step requires access to the internal network, a specific environment, or when the volume of dependencies makes it more cost-effective to keep the cache close to the code. We start with one or two agents and add more as parallel jobs grow.
Another pipeline is already running — should everything be migrated at once?
No. We do not turn off the working pipeline: we migrate one repository at a time, both exist in parallel, and build results are compared on the same code changes. Rollback is moving the trigger back to the previous system; history and artifacts stay in place. This way the transition does not block releases.
Let's discuss your task: what to send for a quick estimate
For the estimate to be substantive rather than a guess, a short set of inputs is enough for us. We look at the current build and delivery pipeline, environments, and how changes are rolled out today — this shapes the scope of work, stages, and timelines.
What to send:
- Links to repositories and read access to pipeline settings — we will see how builds, tests, and image publishing are organized.
- Build and deployment configurations: image build files, Kubernetes manifests, environment and variable designs.
- Environment description — how many clusters, which environments (development, pre-production, production), who gets access and how.
- The workflow for secrets and the image registry: where keys are stored, how permissions are granted, and whether there is separation by environment.
- The current manual release process: how long build, checks, and rollout take, and at which step failures happen most often.
- Security requirements: vulnerability checks, image signing, permission separation, and action logging.
- Expected release mode: rollout frequency, update windows, rollback requirements, and acceptable downtime.
Separately, we need an engineer who will walk us through the infrastructure and answer questions during the review — this saves weeks of back-and-forth.
Based on this data, we will come back with a task estimate and an implementation plan: migrating builds and rollouts to Drone CI in Kubernetes, broken into stages, with owners and timelines for each.
How do self-hosted build agents differ from managed ones?
Managed agents are someone else's infrastructure: you get the image you were given and a queue you do not control. Self-hosted agents are nodes under your control, where you define the image, access, and utilization policy.
The difference shows up not in convenience, but in what exactly you can fix when a build is stuck, and who is responsible for downtime.
Isolation determines what survives a single build.
A disposable agent in a separate namespace takes everything with it: package cache, temporary files, intermediate image layers — the next job starts from a clean state, and substituting the result with someone else's build is impossible.
A reusable pool is faster due to a warm cache, but it requires discipline: cleaning volumes and working directories becomes your task.
Control over the environment determines whether a build can reach internal services. On managed agents, a private registry and database are accessible only through a public endpoint, and secrets live in someone else's store.
Self-hosted nodes in the cluster see internal addresses directly, keys are stored in your vault, and the system library version is pinned by your image, not by someone else's update policy.
The flip side is the burden on the team. A self-hosted pipeline requires runtime updates, key rotation, scaling for peak hours, and disk cleanup after failures; managed removes that but takes away flexibility.
The cost of maintenance is measured in engineer hours, not number of nodes. In practice, with Drone CI, teams keep a hybrid: managed agents for checks and builds, self-hosted ones for rollout and tasks that need internal network access.