Service Discovery in Kubernetes: where connectivity breaks
Pods are running, readiness probes are green, but the service won't come together: one participant gets a name resolution error, another knocks on an address that no longer exists.
Service Discovery in Kubernetes doesn't break entirely, but partially — some calls succeed, some hang until timeout.
The first class of failures is a failure of the name layer. CoreDNS doesn't respond when it hits resource limits, loses connection to the API server, or remains a single instance after pod eviction.
In application logs, this looks like a spike in name resolution errors, even though the service itself is alive and responds at a direct address.
The second class — stale pod addresses. The client keeps the response in cache for the TTL, the pod is recreated during that time and gets a new address: traffic goes nowhere, and the connection hangs until timeout.
With headless services, the same thing is more noticeable — the client does balancing itself using an endpoint list that has become stale.
The third class — configuration mismatch: the selector didn't match pod labels, the port in the service and in the container diverged, the readiness check lets through an application that hasn't come up yet.
In this case, Endpoints are empty not because of the network, but because the service selected no one.
The cost of all three is the same: partial failures that aren't caught by infrastructure alerts but are visible in the user error share. Below we break down how this is closed.
What configuration gives: release speed and eliminated failures
While service addresses live in application configs and environment variables, any pod recreation turns into manual reconciliation: addresses changed, clients hold stale values and keep going nowhere.
Managed discovery removes the very reason for this work — the application contacts the service by name, not by a concrete address.
What happens at the infrastructure level: CoreDNS serves stable names for regular services, and headless mode gives a fixed entry point with direct enumeration of addresses for sets where each instance is addressed separately.
When pods are replaced, only the contents of records change — names and connection points stay the same, so application and balancer configs are no longer rewritten for every release.
The key effect is visible in two metrics. Release time is reduced from hours to tens of minutes, because the step “update addresses and check that nothing was forgotten” leaves the process — it was performed manually and was the most frequent source of delays.
The share of manual config edits falls almost to zero: only meaningful changes remain, not a reaction to address changes.
A whole class of failures also goes away. Previously, after planned pod recreation, the application returned connection errors or responded partially — some instances were already up, while clients still held old addresses.
Now the stale record expires by time to live, and traffic switches to live instances. This directly affects availability: pod recreation becomes a routine operation, not an event with a risk of degradation.
Discovery modes: DNS records, headless mode, external registries
Discovery inside the cluster solves one task — let a service find a neighbor without hardcoding its address into the config.
There are several methods, and they are not interchangeable: a regular DNS record, headless mode with a direct address list, and an external registry for services outside the cluster. The choice depends on who holds balancing — the platform or the client.
| Option | When suitable | Limitations |
|---|---|---|
| Regular service DNS record | Many short requests, the client doesn't care about a specific instance | The platform balances; client DNS cache smooths rebuilds with delay |
| Headless mode | Stateful workloads, DBs, brokers, long connections | The client traverses the list itself; its own client-side balancing is needed |
| External registry | Services outside the cluster, multiple environments, unified catalog | Separate system with its own failures; synchronization must be maintained |
| Static addresses | Emergency environment, one-off dependencies | Failed node is not excluded automatically; replacement is a manual edit |
In headless mode, CoreDNS returns the full list of addresses in response to a DNS query, and balancing is handled by the client.
This is a deliberate trade-off: the platform stops being an intermediary and a point of failure, but the client must be able to repeat the query when membership changes.
That's why we move to headless only those services where direct access to a specific instance is needed, and leave static addresses for emergency scenarios.
Work stages: audit, scheme, configuration, verification, handover
Work proceeds from analyzing the current topology to handing runbooks to the customer's team, without stopping the production environment. The order is arranged so that each stage gives a verifiable result and serves as input for the next.
- Cluster audit — collect a map of services and namespaces, look at how names are currently resolved, where manual records remain, and which network policies cut DNS traffic. Separately record points of failure: one CoreDNS replica, no cache, hard timeouts in clients.
- Discovery scheme — decide where a regular service is enough and where headless is needed for direct access to pods. Determine zones, forwarding rules, and what goes to external DNS.
- CoreDNS configuration — edit the configuration for this scheme: zones, forwarding, cache, limits, readiness probes. Replicas are spread across nodes and availability zones so that failure of one doesn't bring down name resolution.
- Fault tolerance verification — shut down a replica and the node with it, see whether resolution persists and how clients survive response delay. Separately run headless services and endpoint list updates.
- Observability — expose request, failure, and response time metrics, set thresholds for on-call so that degradation is visible before complaints.
- Handover to operations — scheme, runbooks, and the procedure for typical operations: how to connect a new service, how to roll back a change, what to check first during a failure.
Deliverables: configurations, diagrams, runbooks
The project result is not only a working cluster, but also a set of artifacts your team uses to run it further on its own. Everything is placed in your repository and handed over with explanations: what is responsible for what, where it changes, and how it rolls back. Below is what exactly remains with the customer after handover.
- Working configurations: manifests and ConfigMap for CoreDNS, headless services, and related objects — in your version control system, with change history.
- Discovery topology diagram: how services find each other, which records are created for StatefulSet, how names are resolved inside and outside the cluster.
- Access policies: RBAC roles, NetworkPolicy, and restrictions on zone edits — who and from which namespaces can read and modify records.
- Fault tolerance diagram: number of replicas, distribution across nodes, behavior when a node or zone is lost.
- Operations runbook: typical operations — connecting a service, changing a zone, certificate rotation — with checkpoints and rollback procedure.
- Set of checks: metrics and scripts for monitoring name resolution, response delays, and error shares, with thresholds for on-call.
- Documentation: description of the scheme, decisions made, and known limitations — so a new engineer can get up to speed with the setup without verbal explanations.
Case: how manual address edits after release were removed
What was A platform of several dozen services on Kubernetes kept connectivity on hand-written addresses.
Before each release, an engineer collected the current list of pods and transferred addresses into the configurations of dependent services — 25-40 minutes for synchronization and a constant risk that part of the list would become stale between the edit and the rollout.
Unplanned pod recreation cost more: on new nodes addresses changed, but clients continued to access old records. The service returned errors for 5-8 minutes until the on-call engineer found the stale address.
Over the quarter, eleven such investigations accumulated, each with partial unavailability for users.
What changed The reason was not the number of services, but static addresses as a method of communication.
We introduced CoreDNS as the cluster's standard name server and headless Services for services with multiple instances: the service returns the list of addresses of all live pods, and the client chooses an available one.
Calls between services were switched to names; for long-lived connections, periodic list re-reading was configured — otherwise the client would hold an old address until its own restart.
What became Address synchronization disappeared as a separate stage: time for edits during release — zero, changes are picked up together with the cluster state.
Connectivity after emergency pod recreation is restored in 10-20 seconds instead of 5-8 minutes of downtime, and there were no incidents due to stale addresses over a comparable period. Measurements were taken from failure logs and on-call response time.
What to do if CoreDNS is already deployed in Kubernetes?
A deployed CoreDNS is not a reason to rebuild the cluster: it is a working component that is changed in place, without stopping services. The main rule — do not touch the kube-dns service name and its address inside the cluster.
Application manifests access exactly it, so the discovery scheme can be changed under the hood while the external resolution contract remains the same.
The order of changes is built from read to write. First we take the current Corefile configuration and record which zones, plugins, and upstream resolvers are used in it — this is the rollback point.
Then we edit a copy in ConfigMap without applying it to the production environment, and check from a test namespace: resolution of internal and external names, behavior of search domains and the ndots parameter.
A separate step — headless services. Converting a regular service to headless changes only the record in the manifest: instead of one address, the client gets a set of pod addresses, while the name remains the same.
Applications that access by service name continue to work without edits if the previous search domains and resolution order are preserved.
We perform the update via sequential pod replacement: at least two replicas, a readiness check before removing the next one, and configuration reload without restart. At any step, some pods respond by the old scheme, some by the new one, and requests are not lost.
After the switch, we verify the share of responses from cache, the number of SERVFAIL failures, and the completeness of address lists for headless services. Rollback — return to the previous configuration and reload; in time it is comparable to a direct change.
Technical doubts: cache, timeouts, mode limitations
Record caching is the first question that comes up when a service continues to access the old address after pod migration. A record has a time to live, and the client holds it until the timeout expires — this is not a cluster failure but expected behavior.
The second objection is timeouts when a node is unavailable: readiness checks, spreading replicas across zones, and disruption budget help. A node is removed from operation only after it has stopped being polled.
Cache holds old address after pod migration — what is done?
The cache doesn't need treatment: we reduce the record's time to live for unstable services so that the address list update keeps up with the migration. We add a client-side readiness check — it doesn't send traffic to an unready pod.
What happens if a node with DNS replicas drops out?
Nothing catastrophic: CoreDNS replicas are spread across nodes and zones, and the client re-queries the next address from the list. Some requests are lost only during the switchover — we cover the interval with alerts and checks.
Headless returns address list without balancing — what's the limitation?
There is no virtual address and traffic distribution here: the client gets pod addresses directly. Database clusters need exactly this, but a regular application will have to do balancing on the client side.
These scenarios are closed at the design stage: record time to live, replica distribution, and service mode. Below are the remaining questions that come up during configuration handover.Let's discuss your task: where to start
To make the assessment concrete rather than “by eye,” let's start with a description of your topology: how many clusters and environments, how namespaces are divided, who accesses services and by which names, what you've already tried, and which symptoms remain. The format doesn't matter — text, a diagram, or diagnostic command output.
In response, we'll return not general considerations, but an analysis: where exactly name resolution breaks, what changes are needed, and in what order to roll them out. Below is what we'll clarify first and what will be included in the work plan.
- Task boundaries: name resolution only inside the cluster, or also access from external networks and neighboring clusters.
- Topology questions: number of environments, namespace split, your own name servers and zones under your control.
- Current state: services without a balancer, local node access mode, records that are edited manually.
- What has already been checked: name query response time, failure share, behavior during node restarts and settings changes.
- Work estimate: list of configuration edits, scope of changes, effort, and rollout order across environments.
- Implementation plan: stages, rollback points, and checks that old names continue to work after the transition.
- Post-launch operations: naming rules, monitoring and alerts, documentation, and handover to your team.
Describe the task in free form — we'll return with an estimate and implementation plan. If it's more convenient by voice, we'll set up a call with an engineer and go through your scheme together.