Setting Up Scalable CI/CD on Buildkite with Self-Hosted Agents

Setting Up a Scalable CI/CD Pipeline via Buildkite

Buildkite follows a model that most CI platforms never properly implemented: orchestration in the cloud, execution on your infrastructure. Buildkite knows which job to run and in what order, but it never sees the code — it only goes to your agent running inside your network, on your hardware, or in your cluster.

We deploy such pipelines turnkey with guaranteed stability and performance. We have 5+ years of DevOps experience and 50+ successful CI/CD implementations across various platforms. Contact us for a project assessment.

Why Buildkite is the Best Choice for CI/CD with Self-Hosted Agents

Shared runners from GitHub or GitLab are a convenient start. But once builds start taking 15–20 minutes and the queue holds 10 jobs, it becomes obvious: there's no room to scale on someone else's machines with shared CPU and memory. Buildkite tackles this radically — your agents are yours, so if you want 50 parallel builds, spin up 50 agents. Want ARM for mobile builds and x86 for backend — tag agents accordingly and route jobs to appropriate hardware via agents: { queue: "arm64" }.

A second use case is private dependencies and registries. When a build accesses an internal npm registry, private Docker Hub, or PyPI mirror inside a VPN, shared runners simply cannot reach it. A Buildkite agent inside the network solves this without VPN tunnels or IP whitelisting. Infrastructure cost savings reach 30–50% compared to paid CI solutions by using your own agents.

Comparison: Shared Runners vs. Buildkite Agents

How a Buildkite Pipeline Works

Pipeline configuration is a YAML file .buildkite/pipeline.yml in the repository. The basic structure looks simple, but the platform supports dynamic pipelines: one step runs a script that generates YAML for subsequent steps and uploads it via buildkite-agent pipeline upload. This allows matrix builds where the list of components to build is determined on the fly based on changed files, not hardcoded.

# .buildkite/pipeline.yml
steps:
  - label: ":docker: Build"
    command: "scripts/build.sh"
    agents:
      queue: "docker-builders"
    plugins:
      - docker#v5.10.0:
          image: "node:20-alpine"
          volumes:
            - "/tmp/cache:/root/.npm"
  - wait
  - label: ":test_tube: Test"
    command: "scripts/test.sh"
    parallelism: 4
    agents:
      queue: "test-runners"
  - block: ":rocket: Deploy to Production"
    branches: "main"
  - label: "Deploy"
    command: "scripts/deploy.sh"
    branches: "main"
    agents:
      queue: "deploy-production"

parallelism: 4 — Buildkite automatically splits the test job into 4 parallel shards. Each shard receives variables BUILDKITE_PARALLEL_JOB and BUILDKITE_PARALLEL_JOB_COUNT, allowing the test script to determine which part of the tests to run. Jest, RSpec, pytest — all support such splitting.

Plugins are a separate ecosystem. docker, ecr, kubernetes, cache, artifacts — each plugin wraps typical logic into reusable configuration. No need to write a bash script for pushing to ECR; just attach the ecr#v2.7.0 plugin and specify parameters.

How to Deploy Buildkite Agents

The Buildkite agent is a Go binary that installs in a minute. For a production setup, we recommend running via systemd on dedicated VMs or as a DaemonSet in Kubernetes.

For Kubernetes, the Buildkite Agent Stack for Kubernetes (official operator) allows each job to run in a separate pod. The agent holds no state between jobs, providing full isolation. buildkite-agent-stack-k8s is deployed via Helm:

helm repo add buildkite https://buildkite.github.io/helm-charts
helm install buildkite-agent buildkite/agent \
  --set agentToken="$BUILDKITE_AGENT_TOKEN" \
  --set config.tags="queue=k8s-builders,arch=amd64"

For caching dependencies between jobs, use a separate PVC or S3 bucket mounted via the cache plugin. Unlike GitHub Actions, where cache is stored on GitHub servers with a limited TTL, here the cache lives as long as you need.

What's Included

• Audit of current pipelines and infrastructure
• Agent deployment (VM or K8s)
• Pipeline setup, caching, artifacts
• Test parallelization with sharding
• Integration with Slack, GitHub, PagerDuty
• Operations documentation and runbook
• One month of post-launch support

Process Steps

1. Audit current pipelines — what exists, where time bottlenecks are, which jobs require access to internal resources.
2. Choose agent infrastructure — VM vs Kubernetes, on-premise vs cloud, isolation requirements.
3. Deploy agents — installation, systemd/DaemonSet, queue tag configuration.
4. Migrate pipelines — convert .gitlab-ci.yml or GitHub Actions workflows to .buildkite/pipeline.yml.
5. Configure cache and artifacts — S3/GCS bucket for cache plugin, retention policy.
6. Parallelize tests — set up sharding, integrate with Test Analytics for slow test analysis.
7. Notifications and integrations — Slack, PagerDuty, GitHub commit statuses.

Timeline Estimates

Cost is calculated individually after infrastructure audit. Request a consultation for your project assessment.

Buildkite gives you full control over CI/CD infrastructure without compromising speed or security. Self-hosted agents solve scalability, private resource access, and caching issues that cloud rivals cannot handle. Dynamic pipelines and plugins accelerate development and maintenance.

Contact us to receive a preliminary migration plan.