Health check endpoints: where a failure breaks the entire service
The failure of a single node rarely stays local. The load balancer keeps sending traffic to an instance that can no longer serve requests — connections pile up, the thread pool is exhausted, and a few minutes later the entire service goes down.
This is a classic cascading failure: health check endpoints are either missing or respond "ok" regardless of the application's actual state.
The opposite extreme is no better. If the liveness probe checks an external dependency — a database, a queue, a third-party API — then during a brief database outage the platform will start restarting healthy instances.
The result is false restarts: the service drops connections, warms up the cache again, and the root cause of the failure is still there.
Separating the probes removes this confusion: liveness is responsible only for "the process is alive and not hung," readiness — for "ready to accept traffic right now."
The readiness probe is the point where traffic is cut off before the user sees an error.
While an instance is warming up its cache, waiting for a migration, or losing its connection to the database, it honestly reports "not ready" — the load balancer takes it out of rotation, and requests go to working nodes. Without this mechanism, the degradation of a single pod turns into a service outage for everyone.
The cost of a mistake is measurable: with correct probes, recovery time after a node failure is seconds; without them, it is minutes, until the on-call engineer manually takes the instance out. Below we break down how this is addressed at the configuration level.
What a properly configured health check delivers
Recovery time drops from tens of minutes to seconds
The liveness probe stops being a formality: it checks not whether "the process is alive," but whether the service can respond to a real request — the availability of the database connection pool, the state of internal queues, cache readiness.
When a container hangs in a degraded state, the orchestrator gets an honest signal and restarts it immediately, rather than half an hour later via an alert from the on-call engineer.
In practice, this is the difference between recovery in 20-40 seconds and a manual investigation taking 15-30 minutes.
False restarts disappear from the on-call schedule
A readiness probe with correct thresholds (startup delay, number of allowed failures, polling period) separates "the service is still coming up" from "the service is broken." Without it, a typical picture emerges: the application takes longer to start than the timeout, the orchestrator kills it mid-initialization, the cycle repeats — and the service never comes online at all.
Proper configuration eliminates this class of incidents entirely, and along with it, nighttime restarts and false monitoring alerts.
Traffic routing stops sending requests to dead pods
Until readiness is confirmed, the load balancer does not route traffic to the instance — and conversely, when the probe fails, it removes it from rotation until restart.
This closes off failures that would otherwise look like random errors for some users: requests go to a container that is no longer serving connections. Service availability stops depending on whether someone managed to manually remove the node in time.
Separating the probes enables controlled rollback
When liveness, readiness, and the startup probe are configured separately, a restart stops being the only reaction to any failure.
A temporary loss of the database fails readiness but does not kill the container — the service waits out the failure and returns to rotation on its own. This reduces the load on the infrastructure and preserves state that is lost during a hard restart.
Approaches to health checking: comparing modes
Checks fall into two levels: the orchestrator's internal probes, which decide whether to restart a container and whether to route traffic to it, and external monitoring, which looks at the service through the user's eyes.
The two must not be mixed: a probe that hits the database or an external API will take down a healthy pod when a dependency fails, while monitoring that only pings the site root will not see that the application is returning errors in half of its scenarios.
| Option | When it fits | Limitations |
|---|---|---|
| Liveness probe | The process may hang: deadlock, infinite loop, thread pool exhaustion | Only answers "is the process alive." It does not check dependencies — otherwise a database failure triggers a cascading restart of the entire service |
| Readiness probe | You need to remove a pod from load balancing during warm-up, migrations, or overload | Does not restart the container. A logic error can easily leave a pod stuck outside traffic forever |
| Startup probe | Slow start: cache warm-up, model loading, applying migrations | Runs once per container lifetime. If startup time is unstable, the threshold will have to be raised with a margin |
| External monitoring | End-to-end scenario checking: login, payment, data retrieval from storage | Does not control the orchestrator. False positives during network spikes; needs headroom in polling frequency |
In practice, the modes are combined: startup covers slow launches, readiness regulates traffic, liveness catches hangs, and external monitoring confirms that the service works for the client.
Separating endpoints is a mandatory condition: /healthz for the process and /ready for readiness to accept requests must not hit the same dependencies.
The main limitation of any mode is the cost of a false positive. A liveness probe that is too strict turns a brief dip into a restart storm; one that is too lenient leaves a non-working pod in rotation.
That is why thresholds, delays, and retry counts are tuned to the real load profile rather than left at their defaults.
Stages of implementing health check endpoints
Implementing health check endpoints begins with an audit of what already exists: which checks respond to requests, what exactly they check, and how the load balancer reacts to them.
It often turns out that the endpoint always returns "ok" — even when the database is unavailable or the queue is overflowing, it still returns a successful response, and traffic keeps going to the broken instance.
At this stage, we record the service's dependencies and agree on which of their failures should take the instance out of rotation and which should only signal.
- Audit of current checks. We compile a list of endpoints, look at what they check, how the intervals and thresholds are configured on the orchestrator side, and which failures have already led to downtime.
- Designing the check scheme. We separate liveness and readiness: the first answers the question "is the process alive," the second — "is it ready to accept traffic." We design the startup check separately for services with long initialization so that the probe doesn't kill the instance prematurely.
- Configuring the probes. We set the intervals, timeouts, number of retries before transitioning to a failed state, and the thresholds at which the instance is taken out of rotation. We keep the checks cheap — without heavy database queries, so that the probe itself doesn't create load.
- Testing failure scenarios. We deliberately bring down a dependency, flood the queue, revoke access to storage — and watch how the system behaves: does traffic leave the problematic instance, does it recover after the dependency returns, are there false positives during short-term failures.
- Documentation. We hand over the check scheme, parameter values, and the logic of reacting to failure — so that the on-call engineer understands why the instance was taken out of rotation and can distinguish a real incident from a false positive.
After implementation, the failure of a single instance stops being noticeable to the user: traffic moves to healthy nodes, and recovery happens without manual intervention. Describe your task — we'll get back to you with an estimate and a work plan.
Artifacts and configurations you receive
As a result of the work, you receive not a "configured monitoring setup," but a reproducible set of artifacts: configurations, diagrams, policies, and procedures by which the system operates and is handed over to your team without verbal explanations. Below is what exactly remains in the repository and in the documentation.
- Probe configuration — manifests with liveness, readiness, and startup checks: endpoint path, intervals, failure thresholds, initial delay. Separately for each service, with comments explaining why these values were chosen.
- Dependency diagram — a map of which service depends on which and how this is reflected in readiness: checking the database, queue, external APIs. It shows exactly what blocks traffic acceptance during a failure.
- Access policies — rules defining who can hit the endpoints and from where: internal network, orchestrator, monitoring systems. Public access is closed, and the checks do not expose the service state to the outside.
- Response procedures — what each probe status means, in which scenario a restart is triggered, in which one the service is removed from load balancing, who receives the notification, and within what timeframe.
- Operations instructions — how to add a new probe, change a threshold, roll back a configuration. With examples of edits and the expected result.
- Set of test scenarios — how to reproduce a failure: take down a dependency, slow down the response, crash the process. Verification that the probes react as intended.
- Implementation report — which services are covered, which thresholds are set, what was left out of scope and why.
Configurations are stored in the version control system alongside the service code — changes go through review and are captured in history. Diagrams and procedures are in the project documentation, in a format that can be read without access to the infrastructure.
Describe your task — we will come back with an estimate and a plan for which artifacts you specifically need.
Case study: how health check endpoints reduced downtime
The project is a high-load service with a background task queue: several instances behind a load balancer, deployments several times a week.
Before the checks were introduced, containers were restarted based on the external sign "the process is alive," even though the application was no longer responding: the database connection pool was exhausted, background handlers had stalled, but the process had not crashed.
The load balancer kept sending traffic to the unhealthy instance, clients got errors, and the on-call engineer learned about the failure from incoming reports. Each such incident stretched into dozens of minutes of manual diagnostics.
What was changed
Two types of checks were separated by purpose. Liveness answers only the question "is the process completely hung" — if so, the platform restarts the container.
Readiness describes readiness to accept traffic: database availability, connection pool saturation, and the state of background handlers.
While readiness does not pass, the instance is removed from load balancing but is not killed — this is fundamental, otherwise a restart during warm-up causes a cascade.
A startup check with an extended window was configured separately: the application warms up caches and establishes connections longer than the usual interval lasts, and without this the platform killed the container before it was ready.
The thresholds were tuned so that a single dependency failure would not take the instance out of service, while sustained degradation would.
What happened as a result
Kubernetes stopped restarting healthy containers and began removing only genuinely unready ones from load balancing. The share of requests that hit an unhealthy instance dropped from 4–5% to fractions of a percent.
The average recovery time after a dependency failure shrank from dozens of minutes to 1–2: the instance returns to service on its own as soon as readiness passes again.
The number of container restarts per week decreased by roughly five times — false triggers during warm-up and peak load were eliminated.
Stability no longer depended on whether the on-call engineer noticed the problem. The checks detect degradation before clients feel it, and endpoint metrics provide a ready-made picture for analysis.
How does liveness differ from readiness?
Liveness answers the question "is the process alive," readiness — "is it ready to accept traffic." These are different questions, and confusing them is costly: a probe that checks the wrong thing either throws a healthy container under suspicion into a restart loop, or keeps an instance in the load balancer that hasn't yet established connections to the database.
A liveness probe looks only at the well-being of the process itself: whether it's blocked, whether it's deadlocked, whether the control thread is responding.
If the check fails, the platform kills the container and brings up a new one — this is an emergency lever, not a fine-tuning tool.
The classic mistake is to put an external dependency check into liveness: the database blinks for ten seconds, and the entire fleet of containers goes into simultaneous restart, even though the application was fully operational.
A readiness probe, on the other hand, controls routing. Until it passes, the instance is excluded from load balancing but continues to live: it loads the cache, waits for the connection pool to warm up, applies migrations.
A failed readiness does not restart the container — it merely removes it from rotation, and when the dependency returns, the instance just as smoothly returns to service.
Hence the practical rule: in liveness — minimal logic, only "the process responds"; in readiness — everything without which a request won't be served correctly.
Timeouts also diverge: readiness can be checked frequently and with a short threshold, liveness — less often and with a margin, so as not to kill the container under peak load, when responses have naturally slowed down.
Common technical concerns
Probes are integrated into an already running service without rewriting business logic: a separate endpoint is added that checks dependencies and returns a status.
If the application is already deployed in a cluster, we first capture the current behavior — how it responds when the database is unavailable, when the connection pool is exhausted, when an external service is slow. Based on this picture, we configure thresholds rather than using defaults.
Probes do have limitations, and it is important to discuss them before implementation. Liveness must not call the database or an external service: if a dependency hiccups, restarting the container will not fix it, and the service will lose request processing.
Readiness, on the contrary, must account for the readiness of dependencies — otherwise traffic will go to an instance that cannot yet respond. Separating these roles is the main work.
How to avoid false positives during short-term failures?
The trigger threshold is set not by a single request but by a series: for example, three consecutive failed checks with an interval and a timeout period. Plus a separate timeout for the endpoint itself — it must respond faster than the probe expires. This way, a short-term database delay does not cause a restart, while sustained degradation does.
What will happen to an already configured solution if we come in?
We analyze the current probes and see what they actually check. Often liveness and readiness point to the same endpoint — that is the first thing we separate.
We move the configuration into manifests so that thresholds and timeouts are versioned rather than set by hand in the panel. Rollback is reverting the previous manifest, without rebuilding the image.
Do we need to change the application code for probes?
In most cases — no. The endpoint is added as a separate handler, and business logic is not affected. If the service cannot report dependency status, we add a lightweight check: connection availability and response time.
This is enough for the probe to reflect actual readiness rather than the mere fact that the process started.
Let's discuss your health check setup task
To come back with an estimate and a plan rather than generalities, we need a minimum of inputs about your environment. Describe the task: which services are involved, how deployment is currently structured, whether there is an orchestrator, and how probes are configured today. That is enough to understand the scope and propose a concrete approach.
What is useful to include with the description:
- a list of services and their roles — which ones receive traffic, which ones run in the background;
- current manifests or deployment configurations, if any;
- how availability is currently checked: external monitoring, manual checks, or nothing;
- recovery time requirements — how much downtime is acceptable when a pod fails;
- access and environment constraints: how many environments there are, who owns the cluster;
- known issues: false restarts, stuck pods, long warm-up after startup;
- a contact person on the infrastructure side for clarifications during the work.
Based on this set, we estimate the effort, name the stages and the rollout order. Then we agree on the work plan: mapping probes to services, configuring thresholds and timeouts, testing on staging, and a phased rollout to production. The project starts after we lock down the boundaries and rollback points.
Describe the task in whatever form is convenient — we will come back with an estimate and a work plan.
