Canary Deployment for ML Models on Kubernetes

Note: When a new version of an ML model hits production, we don't know how it will behave under real load. Once we deployed a model that performed perfectly on test data, but in production it started generating false positives on 30% of requests. Rollback took 20 minutes — minutes that cost the client a significant sum. This is exactly the scenario where canary deployment is needed — a strategy that reduces MTTR by 3-5 times and saves budget on incidents.

We practice canary deployment for ML models on Kubernetes using KServe, Seldon Core, or Argo Rollouts, with automatic rollback based on monitoring metrics. Our experience — 5+ years and 20+ projects in MLOps — confirms: canary with guardrails reduces MTTR to 45 seconds versus 22 minutes for a full rollback. In one project, savings per incident amounted to about 40,000 RUB.

When canary is preferable to blue-green

Blue-green switches all traffic at once — suitable for services with high confidence in the new version. Canary is needed when:

• The model is trained on new data, but user reaction is unpredictable.
• The model architecture changed (different type, different input features).
• Critical production service with high cost of errors.
• No full set of integration tests.

Let's compare key characteristics:

Canary provides controlled traffic increase and automatic rollback based on metrics, which is critical for production services with high cost of errors.

How canary reduces MTTR?

MTTR is a key metric during failures. With a full rollback, you have to recreate pods, switch traffic, and check logs. That takes 15-30 minutes. Canary with automatic rollback reacts in seconds: as soon as error rate exceeds 1% or p99 latency goes over 500ms, the script rolls back the canary without engineer intervention. In one of our projects, MTTR dropped from 22 minutes to 45 seconds — 30 times faster, saving the client about 40,000 RUB per incident.

Implementation on Kubernetes with KServe

KServe (formerly KFServing) supports canary out of the box. KServe documentation recommends starting with 5-10% traffic on the canary.

---
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
spec:
  predictor:
    canaryTrafficPercent: 10 # 10% to new version
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://models/fraud-detector-v2/ # Previous version - canary baseline
---

Switching traffic without downtime:

# Increase from 10% to 50%
kubectl patch inferenceservice fraud-detector \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/predictor/canaryTrafficPercent", "value": 50}]'

# Promote canary to production (100%)
kubectl patch inferenceservice fraud-detector \
  --type='json' \
  -p='[{"op": "remove", "path": "/spec/predictor/canaryTrafficPercent"}]'

Step-by-step canary setup with KServe

1. Install KServe and its dependencies (Istio, Knative) in your Kubernetes cluster.
2. Create an InferenceService with canaryTrafficPercent: 10 and specify the new model URI.
3. Set up metric monitoring (error rate, latency, drift) via Prometheus and alerts in Grafana.
4. Run a progressive traffic increase script that checks guardrails and automatically rolls back when thresholds are exceeded.
5. After successfully reaching 100%, remove the canaryTrafficPercent field.

Implementation on Seldon Core

---
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: fraud-detector
spec:
  predictors:
    - name: main
      replicas: 3
      traffic: 90
      graph:
        name: fraud-v1
        implementation: SKLEARN_SERVER
        modelUri: s3://models/fraud-v1
    - name: canary
      replicas: 1
      traffic: 10
      graph:
        name: fraud-v2
        implementation: SKLEARN_SERVER
        modelUri: s3://models/fraud-v2
---

Automatic rollback is configured via PrometheusRule: when error rate exceeds 1% or p99 latency beyond 500ms, an alert triggers, reducing canary traffic to 0.

Automatic traffic management

Progressive traffic increase is automated based on metrics. We use a script that checks guardrail metrics at each stage:

def progressive_canary_rollout(service_name, metrics_client):
    stages = [5, 10, 25, 50, 100]
    for target_traffic in stages:
        set_canary_traffic(service_name, target_traffic)
        time.sleep(300)  # 5 minutes stabilization
        metrics = metrics_client.get_metrics(window='5m')
        # Check guardrail metrics
        if metrics['canary_error_rate'] > 0.01:
            rollback_canary(service_name)
            alert(f"Canary rollback: error rate {metrics['canary_error_rate']:.2%}")
            return False
        if metrics['canary_p99_latency_ms'] > 500:
            rollback_canary(service_name)
            alert("Canary rollback: latency SLA violated")
            return False
        if metrics['business_metric_delta'] < -0.02:  # -2% degradation
            rollback_canary(service_name)
            alert("Canary rollback: business metric degraded")
            return False
    return True  # Successful full deployment

Automatic rollback when error or latency thresholds are exceeded happens without engineer involvement — it's standard practice in our projects.

Which metrics to use for automatic rollback?

Integration with Argo Rollouts

Argo Rollouts is a Kubernetes controller supporting canary and blue-green for any workload, not just ML:

---
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: {duration: 5m}
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: ml-model-metrics

Scope of work for canary deployment setup

We provide a full turnkey package:

• Designing a canary scheme for your infrastructure (Kubernetes, cloud, bare-metal).
• Setting up KServe or Seldon Core (or any other ML serving framework).
• Integration with CI/CD (GitLab CI, GitHub Actions, Argo Workflows).
• Monitoring and alerting based on Prometheus/Grafana.
• Documentation and team training.

Timelines — from 3 to 10 days depending on complexity. The cost is calculated individually. Order canary deployment setup right now — we'll get back to you within a day.

Canary deployment for ML models on Kubernetes significantly reduces risks during new model rollouts: 70% of production incidents are related to new versions. Canary allows early detection without affecting all users. Automatic rollback based on metrics is the only way to guarantee that a bad model doesn't harm the business. We set guardrails on error rate, latency, drift, and business metrics. If any threshold is exceeded, the canary is rolled back in seconds. Additionally, canary allows A/B testing of models in real traffic: compare the new model with the current one on key metrics and make an informed decision. Get a consultation from an MLOps engineer — we'll explain how canary reduces risks and saves budget.

Our guarantees and experience: we have completed over 20 MLOps projects. Certified specialists in Kubernetes and ML infrastructure. We guarantee a rollback time of less than 1 minute in case of model degradation.