Own CI/CD runners: setup, security, and savings

Why self-hosted CI/CD runners derail releases at peak times

Shared build machines serve all projects at once — and this works exactly until the first peak.

If several events coincide: a mass merge of branches before a release, a nightly regression run, and a scheduled dependency update — the build queue grows at peaks, and jobs wait a long time for a free machine.

A run that took only a few minutes during quiet hours stretches out many times over by evening, and its duration can no longer be predicted.

As long as GitLab has one pool with a common default tag, all jobs compete for the same resources. Machines share cache and working directories, so checks start failing for no apparent reason — timeouts, lack of memory, someone else's garbage in the build directory.

These are unstable check runs: an engineer restarts the job, the queue gets even longer, and trust in the red status disappears — people get used to "forcing it through" with a rerun.

The cost of such a scheme is a missed release window. A change fully ready for rollout waits for a build longer than the window itself lasts and slips into the next one; an urgent fix goes out outside the process, manually and without a full set of checks.

We'll look at how this is solved: dedicated runners matched to the load profile, separating jobs by tags, and isolating work environments.

What an in-house build environment provides: speed, isolation, control

An in-house build environment is a pool of machines under our control, where jobs run next to the code, dependency cache, and a local package mirror.

GitLab runners in such a scheme start in seconds: the environment image is preloaded, and dependency installation comes down to verifying checksums.

Queue wait time stops depending on someone else's load — it is determined by the number of machines in the pool and the priorities we set ourselves.

Job isolation is the second effect. Each job gets a disposable environment: a container or virtual machine that is destroyed after the run. Parallel branches do not share a common directory and variables, so one team's build cannot affect another's.

CPU, memory, and disk limits are set at the executor level, and a heavy job does not crowd out the others.

This eliminates a whole class of failures — environment drift and "dirty" state between runs: unwritten files, someone else's variables, stuck locks, different versions of system libraries.

At peaks, when a series of changes from several teams goes through in an hour, the pool scales, and the queue does not grow avalanche-like.

In numbers, it looks like this: the share of runs requiring a manual restart drops several times over, and median build time shrinks thanks to cache and parallel steps.

A single machine failure does not stop the environment — the job moves to a free executor, and this is visible in recovery time.

Runner placement options: cloud, own machines, hybrid

A runner is where pipeline speed, access to internal test environments, and the way source code is handled meet. The three schemes below are not a theoretical set: we have deployed each one, and each has its own maintenance cost.

A closed development environment deserves separate mention: if regulations prohibit code and artifacts from leaving, cloud machines are ruled out entirely, and hybrid comes down to your own capacity with local dependency mirrors. Hybrid pays off only when routing is described explicitly — by labels, paths, or data sensitivity.

Otherwise, the team gets two inconsistent systems instead of one: different environment variables, different caching rules, and as a result — build reproducibility that cannot be guaranteed.

Implementation stages: from build audit to handover to operations

Implementation starts not with installing agents, but with analyzing what is already happening in the build environment: where time is lost, which steps are duplicated, which jobs run on shared executors with someone else's access.

We follow a fixed sequence — each stage ends with a verifiable artifact, so the project does not stretch out indefinitely.

1. Build process audit. We collect step durations, queues at peak hours, and the share of failures caused by infrastructure rather than code. The result is a map of bottlenecks and a list of jobs that should be moved to dedicated executors.
2. Build environment design. We design the topology: how many runner groups are needed, where they are located, how networks and access are separated between teams, and how the dependency cache and image storage are organized.
3. Runner setup. We deploy executors with job isolation, labels for build types, resource limits, and queue-based autoscaling. Secrets are issued for the duration of a job and are not stored on the host.
4. Build migration. We migrate projects in groups, starting with the least risky, with the old environment running in parallel and a quick rollback to it if results diverge.
5. Load testing. We run several release cycles at the peak number of concurrent builds: we look at cache degradation, autoscaling behavior, and time to job start.
6. Handover to operations. We hand over configurations, the environment design, the update procedure, and the incident response procedure, train on-call staff, and record the metrics by which the environment is accepted.

What the team keeps: configurations, diagrams, procedures

We hand over the build environment as a working system, not as a set of verbal agreements: everything needed for support and development is in your repository and internal documentation.

An engineer on your team will be able to change pipeline steps, add new projects, and troubleshoot failures without contacting us.

• Build configurations — pipeline descriptions, templates for typical applications, and sets of environment variables. Each step includes a comment: why it is needed, what is cached, and which checks block the release.
• Access and key scheme — a map of where credentials live, how an agent gains access to the image registry and artifact storage, what rights each role has, and how they are issued to a new engineer.
• Maintenance procedure — frequency of agent updates, cache and old image cleanup, secret rotation, and actions to take when disk space runs out or an executor hangs.
• Environment documentation — agent placement diagram, network zones and access rules, failure points, and what to do when an external service is unavailable.
• Recovery instructions — how to bring up an agent from scratch step by step, where to get the configuration, and how to verify that the pipeline is building and publishing the artifact again.
• Knowledge transfer to the team — a walkthrough of the environment with your engineers, including typical failures and how to diagnose them, plus a meeting recording.

We separately record what remains outside automation and why: it is a list of development tasks, not hidden debt. A month after handover, we answer questions about the transferred materials — after that, the environment lives on your side.

Case study: build time cut by a factor of three, peak queue eliminated

Builds ran through a shared set of runners: a commit after a merge got into the same queue as the nightly regression and image builds. The average run took 14 minutes, and at peaks waiting reached 22 minutes — resources were held by jobs no one was waiting for anymore.

The share of failed runs stayed at 18%: timeouts and races for the shared dependency cache forced checks to be restarted, and restarts made the queue even longer — one peak overlapped another.

What we changed

We separated runners by load groups: short checks, image builds, and heavy nightly regression got separate sets of machines and limits on concurrent jobs.

Autoscaling based on queue length on Kubernetes brings up disposable agents for peaks and removes them during idle time — each build's environment is clean, and the dependency cache is moved to a persistent volume and is not overwritten by neighboring jobs.

Results

Average build — 14 → 4 minutes, peak wait — 22 minutes → 40 seconds, share of failed runs — 18 → 5%. Nightly regression finishes before the start of the workday, and an incident patch goes through a full run before release, not a stripped-down set of checks.

The queue is no longer a bottleneck: its length in reports is stable, and new repositories are connected without manually expanding the machine fleet — agents are added based on actual load.

What to do if self-hosted CI/CD runners are already deployed

If the environment is already deployed, it is worth starting not with rewriting, but with a review of the current scheme: machine inventory, jobs performed, accounts, access to secrets.

It usually turns out that some runners were started manually, labels were assigned "by eye," and jobs access production storage with a shared token. This is the first layer of problems, and it is visible only on the environment map, not in individual configurations.

The second step is separating permissions on machines. Runners are distributed across pools: build, tests, artifact publication, and deployment get different accounts and different access scopes.

Secrets are issued to a job for the duration of execution rather than stored in machine variables, and executors are made disposable — after the build, the working environment is destroyed along with temporary data.

Scaling for load relies on queues and labels: machines are added based on queue length, not "by eye," heavy builds go to a separate pool and do not block fast checks.

Dependencies and shared volumes are moved outside, so a new executor starts working immediately, without reloading the entire environment.

Improvement of an existing environment proceeds one pool at a time while maintaining compatibility: old jobs continue to work until the new pool confirms stability on real workload, and rollback remains possible at any step.

At the same time, the environment stops depending on one engineer: configuration is stored as code in a shared repository, is checked before being applied, and the scheme of pools, network, and permissions is described in documentation. Then changes stop being the personal responsibility of one person.

Technical questions asked before the start

Before the start, the same set of concerns usually comes up: whether keys will leak, what will happen to releases during migration, and who is responsible for machines after launch. Below are direct answers to what is most often asked before signing the work plan.

Is the build environment isolated from the rest of the infrastructure?

Each run takes place in a clean environment that is deleted after the job — one project's code does not remain on the machine and does not get into someone else's build.

The runner sees only the image and artifact registries allowed by policy, and it has no access to internal services and databases. Network rules are described separately and tested on a test pipeline before being put into operation.

What will happen to releases during migration?

We bring up the new environment in parallel — the old one continues to build and release. For several days, jobs run on both, artifacts are compared, then projects are switched over one by one.

The downtime window is zero: if a build on the new runner behaves differently, the project returns to its previous location until the cause is investigated.

Who maintains the machines after launch?

We fix the procedure in writing: runner and base image updates, monitoring of the queue and free resources, cleanup of old artifacts, response to job failure.

The same document defines the boundary: what remains on our side and what requires your team's involvement — access to internal systems, approval of network rules, release issuance. This way, responsibility is not blurred a month after handover.

Let's discuss the task: builds without queues and leaks

To come back with an estimate and work plan rather than general thoughts, we just need introductory information about builds — it is enough to calculate the runner configuration, estimate effort, and propose a starting order.

• Number of repositories and active teams, typical commit frequency — from these we calculate parallelism and load headroom.
• Where exactly the queue accumulates: current build and test times, and during which hours waits are longest at peak.
• How builds run now: cloud runners, developers' local machines, or a dedicated server.
• Isolation requirements: which jobs get access to secrets and artifacts, who sees what in logs.
• Where you plan to place runners — an in-house environment, dedicated machines, or hybrid — and which networks are available to them.
• Who on your side is responsible for access, and who maintains the configuration after launch.

In response, we send an estimate and work plan, a runner placement diagram, and a starting order. As a first step, we propose a pilot on one repository: we migrate its build, measure time and behavior under load, and only after that roll it out to the rest. Describe the task — we will come back with an estimate and plan.