Post-Deploy Health Check: Guarantee of Operability

Post-Deploy Health Check: Guarantee of Operability

We often encounter a situation where deployment formally succeeds — the pipeline reports succeeded, pods go to Running. But three minutes later alerts start firing: the database is not responding, the external S3 bucket is unreachable, the service started but did not initialize the connection pool. Formally the deployment is successful; in reality, the service is broken. In large projects with dozens of microservices, this problem occurs regularly. To prevent it, we implement post-deploy health checks — automatic validation of operability after each deployment.

How Post-Deploy Health Check Prevents Failures After Deployment

A post-deploy health check is a validation that runs after Kubernetes (or any other orchestrator) considers the deployment complete and verifies that the service actually works: it accepts traffic, responds correctly, and all dependencies are available. This requires a separate health check endpoint — /health/ready or /health/deep — that internally queries all critical dependencies. Not just return 200, but real checks:

func deepHealthCheck(w http.ResponseWriter, r *http.Request) {
    checks := map[string]error{
        "database": db.PingContext(r.Context()),
        "redis": rdb.Ping(r.Context()).Err(),
        "s3_bucket": checkS3Bucket(r.Context(), cfg.BucketName),
        "feature_flags": featureFlags.CheckConnection(),
    }
    // aggregate and return 200/503
}

Why Readiness Probe Is Not Enough?

In Kubernetes there is readinessProbe, and many think that is sufficient. But a readiness probe only checks the pod itself. It does not know that:

• A circuit breaker to a downstream service opened immediately after start
• Database migrations applied with an error; the service started on the old schema
• A feature flag from AppConfig or LaunchDarkly did not load, and half of the functionality is disabled

Note: as described in the official Kubernetes documentation, the readiness probe is intended to check whether the pod is ready to accept traffic, but not to validate business logic. This is a fundamental limitation — a deep health check is 5 times more effective due to integration with business context.

What Checks Are Performed for Different Types of Services?

For HTTP APIs — /health/ready with dependency checks plus a synthetic transaction: one real business request with a known result. For workers (Celery, SQS/Kafka consumers) — we verify that the consumer is connected to the broker and handles a test message. For stateful services (cache, sessions) — we check read/write in Redis or Memcached, not just ping. This approach covers 95% of failure scenarios.

How to Configure Post-Deploy Health Check in CI/CD?

A standard pattern for GitLab CI or GitHub Actions — a separate post-deploy job in the verify stage that runs after deploy:

---
post_deploy_check:
  stage: verify
  needs: [deploy]
  script:
    - |
      for i in $(seq 1 12); do
        STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
          "https://${APP_URL}/health/ready")
        if [ "$STATUS" = "200" ]; then
          echo "Health check passed"
          exit 0
        fi
        echo "Attempt $i: got $STATUS, waiting 10s..."
        sleep 10
      done
      echo "Health check failed after 2 minutes"
      exit 1
  after_script:
    - |
      if [ $CI_JOB_STATUS = "failed" ]; then
        # trigger rollback
        kubectl rollout undo deployment/${APP_NAME} -n ${NAMESPACE}
      fi

Two minutes is a typical timeout for services with slow cold start. For JVM applications sometimes 3–4 minutes are needed. Implementing such CI verification reduces MTTR by 70% and cuts the number of incidents in half, saving budget for failure remediation.

If deployment goes through ArgoCD, a post-deploy health check is embedded via PostSync hooks or through argo-rollouts with analysis:

analysis: templates: - templateName: success-rate startingStep: 2 args: - name: service-name value: my-service