Harness for CI/CD: Deployment and Automatic Rollback Based on Metrics
We have seen this scenario: a pipeline successfully deploys a new version, and five minutes later the error rate spikes in Prometheus, users are complaining, the admin is panicking. Traditional CI/CD cannot roll back on its own—only scripts and manual triggers. Our experience implementing Harness shows that the platform solves this: after deployment, it analyzes metrics and automatically triggers a rollback on degradation. This is not just CI/CD—it is intelligent deployment with continuous verification. MTTR (Mean Time to Recovery) drops from 2 hours to 10 minutes, and rollback costs are reduced by 80%.
How Continuous Verification Works in Harness
After each deployment, Harness runs a Verify stage that, for a set period (e.g., 15 minutes), compares new metrics against the baseline of the previous version. ML models account for time of day, day of week, and historical patterns. If the error rate rises or latency degrades beyond the threshold, rollbackSteps trigger without human intervention. This cuts MTTR from hours to minutes. Compared to Jenkins, where rollback requires custom scripts and separate monitoring setup, Harness accelerates response by 3–5 times.
Example of configuring a Verify stage with Prometheus:
- stage:
name: Verify Staging
identifier: verify_staging
type: Deployment
spec:
execution:
steps:
- step:
type: Verify
name: CV Check
identifier: cv_check
spec:
type: Rolling
monitoredServiceRef: myapp-staging-monitored
healthSources:
- identifier: prometheus-staging
spec:
sensitivity: MEDIUM
duration: 15m
deploymentTag: <+artifacts.primary.tag>Why the Delegate Matters for Security
Harness does not connect directly to your Kubernetes cluster from the cloud. Instead, a Delegate is installed—an agent in the cluster that receives tasks from the Harness Platform and executes them locally. This solves the problem of private clusters without public access. The Delegate runs in an isolated network, requiring no inbound ports. It scales horizontally and updates automatically. Each Delegate instance requires 2 vCPU and 4 GB RAM, supporting up to 10 parallel tasks; for production, at least 2 replicas are recommended.
Installation via Helm:
helm install harness-delegate harness/harness-delegate \
--namespace harness-delegate-ng \
--create-namespace \
--set delegateName=my-delegate \
--set accountId=$HARNESS_ACCOUNT_ID \
--set delegateToken=$DELEGATE_TOKEN \
--set managerEndpoint=https://app.harness.ioHarness Structure: Pipelines, Stages, Steps
A Harness Pipeline consists of Stages. The stage type defines the context: CI, CD, Feature Flags, Approval, Custom.
Configuration via YAML (Harness YAML):
pipeline:
name: myapp-ci-cd
identifier: myapp_ci_cd
projectIdentifier: myproject
orgIdentifier: myorg
stages:
- stage:
name: Build and Test
identifier: build_and_test
type: CI
spec:
cloneCodebase: true
platform:
os: Linux
arch: Amd64
runtime:
type: Cloud
spec: {}
execution:
steps:
- step:
type: Run
name: Install and Test
identifier: install_and_test
spec:
connectorRef: docker-hub-connector
image: node:20-alpine
command: |
npm ci
npm run lint
npm run test -- --coverage
- step:
type: BuildAndPushDockerRegistry
name: Build and Push Image
identifier: build_push
spec:
connectorRef: docker-registry-connector
repo: registry.example.com/myapp
tags:
- <+pipeline.executionId>
- latest
dockerfile: Dockerfile
- stage:
name: Deploy Staging
identifier: deploy_staging
type: Deployment
spec:
deploymentType: Kubernetes
service:
serviceRef: myapp-service
environment:
environmentRef: staging
infrastructureDefinitions:
- identifier: staging-k8s
execution:
steps:
- step:
type: K8sRollingDeploy
name: Rolling Deploy
identifier: rolling_deploy
spec:
skipDryRun: false
pruningEnabled: false
rollbackSteps:
- step:
type: K8sRollingRollback
name: Rollback
identifier: rollback
spec: {}
- stage:
name: Verify Staging
identifier: verify_staging
type: Deployment
spec:
execution:
steps:
- step:
type: Verify
name: CV Check
identifier: cv_check
spec:
type: Rolling
monitoredServiceRef: myapp-staging-monitored
healthSources:
- identifier: prometheus-staging
spec:
sensitivity: MEDIUM
duration: 15m
deploymentTag: <+artifacts.primary.tag>
What Metrics Does Harness Analyze for Verification?
Harness integrates with Prometheus, DataDog, New Relic, Splunk, and other monitoring systems. By default, it uses error rate, latency, traffic, and saturation—the so-called Golden Signals. The ML model compares them with a historical baseline and detects anomalies. For example, if the error rate increases by 5% with a typical variation of 2%, this triggers a rollback. Sensitivity is adjustable via the sensitivity parameter (LOW, MEDIUM, HIGH).
Metrics Comparison for Verification
| Metric | Source | Typical Threshold |
|---|---|---|
| Error rate | Prometheus, DataDog | +5% from baseline |
| Latency (p95) | Prometheus, DataDog | +20% from baseline |
| Traffic drop | Grafana, CloudWatch | -10% over 15 min |
Connectors and Secrets
Harness uses Connectors to connect to external systems: Docker Registry, Kubernetes, Git, Helm Chart repos. Secrets are stored in Harness Secrets Manager or delegated to an external Vault:
connector:
name: vault-connector
identifier: vault_connector
type: Vault
spec:
vaultUrl: https://vault.example.com
authType: AppRole
appRoleId: harness-app-role
secretId: harness-vault-secret
secretEngine: secret
secretEngineVersion: 2After that, any secret from Vault is available in the pipeline via <+secrets.getValue("vault://myapp/prod/db_password")>.
What Our Work Includes
We provide the full cycle of Harness implementation. Our experience includes more than 40 successful projects in large clusters (10 to 500 nodes). We configure GitOps pipelines, automated rollback, and deployment monitoring. The service includes:
- Audit of current infrastructure and pipelines
- Design of CI/CD architecture with security and scalability in mind
- Deployment of Delegate and configuration of Connectors (Docker Registry, K8s, Git, Vault)
- Building Pipelines with CI, CD, and Verify stages
- Integration with monitoring (Prometheus, DataDog, New Relic) for Continuous Verification
- Pipeline documentation and developer guide
- Team training on Harness
- Support during the first month after launch
Process of Work
- Analysis—infrastructure audit, requirement gathering, defining success metrics.
- Design—pipeline architecture, secure access scheme, deployment strategy (rolling, blue-green, canary).
- Implementation—Delegate deployment, Connector setup, Pipeline creation with CI, CD, Verify.
- Testing—rollback verification, anomaly detection, load testing.
- Deployment—going live, training, documentation handover.
Estimated Timelines
| Scale | Timeline |
|---|---|
| Basic setup + one service | 2-3 days |
| Multiple services + CV + Vault | 4-6 days |
| Enterprise setup with multiple clusters | 6-10 days |
Cost is determined individually after an infrastructure audit. Contact us for a project assessment—get an engineer consultation. We guarantee pipeline stability and minimal rollback time.
Typical Mistakes in Harness Setup
- Incorrect healthSources configuration—metrics don't arrive, verification doesn't trigger.
- Too low sensitivity—anomalies are missed.
- Missing rollbackSteps in CD stage—deployment doesn't roll back on error.
- Delegate without sufficient permissions—cannot perform cluster operations.
- Ignoring baseline—if baseline is built on non-representative data, ML triggers falsely.
Order Harness implementation from our team. Get an engineer consultation and timeline estimate for your project.