Why External DNS Kubernetes lags behind cluster changes
You recreated the service — pods came up, checks pass, but traffic is not flowing. The record in the zone still points to the previous load balancer address, which no longer exists: the client gets a timeout, even though everything looks green from the cluster side.
And until the record's time to live expires, caching servers keep returning the old address — the failure lasts longer than the release itself.
The reason is the gap between the lifecycle of cluster objects and the contents of the zone. A service lives for minutes, while a record lasts until the first manual edit, so the divergence between records and services accumulates with every rollout.
Stale records are not deleted by themselves after recreation: garbage remains that resolves and sends some requests nowhere and some to the previous environment.
This makes diagnostics harder: externally the domain responds, inside the cluster everything works, but between them there is a mismatch.
Then manual zone edits begin. The on-call engineer reconciles the list of services with the contents of the zone, cleans up what is unnecessary, adds what is new — at night and under pressure, following a runbook no one has checked for completeness.
There is no reverse reconciliation: no one confirms that no pointers to nonexistent addresses remain in the zone. Automatic DNS management removes exactly this work — External DNS links cluster objects directly to the zone; we'll break down how it works and what it provides.
What changes in operations: measurable effects of automation
The External DNS controller connects to the cluster and works as a reconciliation loop: it reads services and entry points marked with an external name and brings the records in the zone to that state.
There remains a single source of truth — the manifest, not the registrar panel, so changes do not need to be duplicated by hand. The effect then breaks down into four measurable areas.
The speed at which records appear no longer depends on a person: a name is published within tens of seconds after the manifest is applied, rather than in the next manual edit window, and a reduced record TTL makes it possible to quickly remove a stale response from resolver caches.
The share of manual operations in the zone drops to isolated cases — usually domain delegation and records for resources outside the cluster, which are deliberately excluded from automation.
The main gain is an eliminated class of failures caused by human error: a typo in a name, a forgotten record after deleting an environment, a conflict between two owners over the same name, a mismatch between what actually runs in the cluster and what is published in the zone.
Such mismatches are almost invisible in application monitoring — they surface as “the service is alive but unavailable from outside,” and the investigation goes off in a direction away from the real cause.
Service recovery time is reduced: when a service is recreated or migrated, the workload gets a new entry point, and the record is updated without an on-call engineer with access to the registrar panel.
The reverse operation is also automated — deleting an environment frees up names, and dead pointers do not accumulate in the zone that would later have to be cleaned up manually.
DNS synchronization options: from manual edits to full automation
A zone can be synchronized with a cluster in four ways, and the choice among them is a choice between speed of reaction to changes and predictability of the zone's contents.
External DNS reads descriptions of cluster objects and creates or updates records, but its operating modes and policies determine what it is allowed to change and delete.
| Option | When it fits | Limitations |
|---|---|---|
| Manual zone management | Stable set of services, infrequent releases | Edits lag behind rollout, record errors are caught only in production, everything depends on one person |
| Scheduled jobs | Many services, address changes once a day | A record appears with a delay, parallel runs conflict, there is no reconciliation with the actual state |
| Automation based on service descriptions | Dynamic environments, frequent scaling | Write access to the zone is required, an explicit deletion policy and labels on objects are needed |
| Hybrid management scheme | Critical records must remain with the engineer | More complex separation rules, a list of zones and records untouched by automation is needed |
In practice, teams more often arrive at a hybrid scheme: development services are managed automatically, while root and mail records remain with the engineer, because an error in them brings everything down at once.
The boundaries are fixed by an explicit list of zones that automation does not touch — then rolling out a new service does not require human involvement, and the zone remains in a predictable state.
Implementation stages in a Kubernetes cluster: from audit to handover
We structure the order of work so that DNS automation gets exactly the permissions needed to publish records and cannot affect zones that live outside the cluster. Each stage ends with an artifact that is checked before moving to the next.
- Audit of zones and permissions. We collect the zones served by the cluster, look at which records were created there manually and who is responsible for them. For the cloud provider, we create a separate account with a limited role: writes are allowed only to designated zones, everything else is read-only.
- Service-to-record mapping scheme. We fix which name each service gets — by annotation, by label, or by the team owner's template. Here we also resolve conflicts with existing records and agree on who owns deletion.
- Configuration. We determine name sources, the domain filter, the synchronization mode, and the reconciliation interval. We add a service marker record: by it the controller understands that the record was created by it and does not overwrite someone else's.
- Testing on a staging environment. On a copy of the zone, we run scenarios: publishing a new service, renaming, deletion, name conflict, provider unavailability. We separately look at what happens to records when a release is rolled back.
- Handover to operations. We provide the runbook, access, metrics, and the procedure for mismatch — what is fixed automatically and where a human is needed.
Configurations, schemes, and runbooks in the handover package
The handover package is not an archive with manifests, but a set of artifacts your team can use to add zones, revoke access, and troubleshoot failures without contacting us.
We put everything in your repository with change history and run scenarios on staging together with the on-call shift.
- Working configurations — External DNS controller manifests with actual values: record sources, zone and domain filters, synchronization interval, record editing mode.
- Zone map — what is delegated, where the authoritative DNS is hosted, which records are managed automatically, which were created manually and must not be touched, and where responsibility boundaries lie.
- Record ownership policies — the controller edits only its own records: owner label, separate registry, separation of zones between teams.
- Access policies — a service account for the controller, permissions only for the required zones, read and write separated, keys in secrets with a rotation runbook.
- Incident response runbook — what to check during desynchronization, missing updates, and provider access failure; how to return to the previous state and whom to escalate to.
- Instructions for on-call engineers — how to view the change plan without applying it, confirm that a record is managed automatically, and what to look at in logs during a conflict.
- Connecting a new zone — step order: delegation, access, filters, test run, enabling edits in the production environment.
- Observability — controller metrics, alerts for authorization errors and synchronization delay, dashboard for the on-call shift.
We write the documentation for your environment, not for a generic staging setup: the zone names, account names, and alert thresholds in it are already yours, so handover does not turn into redesign.
Case study: from manual record edits to automatic synchronization
A product team kept 40+ services in a cluster and managed zones manually in the registrar panel: A, CNAME, and TXT records were created via tickets and edited by an on-call engineer.
Decommissioning a service, changing the load balancer, or migrating a node turned into a chain of approvals — an edit reached the production zone in 30–40 minutes.
Over a quarter, 11 record-related incidents accumulated: a forgotten record when decommissioning a service, a pointer to a removed node, duplicates after manual zone merging. Total downtime due to incorrect routing was about 9 hours.
We implemented External DNS: service and Ingress object annotations became the source of truth, and access to the registrar API was granted to a separate service account with permissions only for the required zones.
In the test environment, we first enabled preview mode — the service showed a change plan and did not apply it; reconciliation with actual records revealed 14 mismatches left over from the manual period, and they were cleaned up before enabling it on production zones.
Development, pre-production, and production environment zones were separated by distinct names, and we keep TTL short.
Now the route changes together with the rollout: an engineer edits the manifest, the record updates in the same cycle, and the window is minutes. Rollback is returning the previous manifest, not manually searching for the old address.
Over the next quarter, the number of record-related incidents dropped from 11 to 1, and that one came from the registrar side, not from our zones.
Downtime due to incorrect routing decreased from 9 hours to half an hour, and pointers to decommissioned nodes disappear along with the objects — the zone registry stopped growing.
What to do if the cluster is already deployed and records are managed manually?
The cluster is already running, and records in zones were created manually — switching to automation does not require a maintenance window.
We start with an inventory: export records, map them to cluster objects, mark what was created by hand, what by the load balancer, what the TTLs are, and who owns the domain.
Without this map, the very first synchronization run will wipe out what was configured manually over years.
Then External DNS is enabled in observation mode: we grant read-only access to the zone, and we compare decisions with the actual state without changes.
The key is the ownership label: a managed record gets a service TXT record with the owner identifier, and the service considers only those where the label matches as its own.
Other records remain untouched: the upsert-only policy (update and add without deletion) prohibits erasure, and the domain filter limits responsibility to already-verified subdomains.
The transition happens in stages — first the non-production environment, then production one zone at a time with a pause for reconciliation. Before switching, the TTL of key records is reduced in advance, so rollback returns traffic within minutes.
Rollback is cheap in general: synchronization is disabled, the records in the zone remain working because we did not delete any of them.
As a result, services do not go down, existing records are preserved, and manual zone editing becomes the exception — only for what is not managed by the cluster.
Technical concerns: limitations, fault tolerance, permissions
ExternalDNS is often perceived as a way to give teams access to DNS zone management.
In reality, the controller sees only the cluster objects it is allowed to see and edits the zone within the boundaries of the assigned domain filter — everything else is outside its responsibility.
Below are the concerns most often raised during acceptance: behavior during provider failures, protection against deleting other people's records, separation of permissions between teams, and the boundaries of the approach itself.
What happens if the DNS provider is unavailable or returns errors?
The controller retries requests with increasing backoff and does not touch records it could not confirm: the zone retains the last successfully applied state, and services continue to resolve against it.
We set alerts for the duration of an error series and for divergence between the cluster and the zone — the failure is visible before the next release, not after user complaints.
How does the controller know which records are its own, and will it delete anything extra?
Each created record is accompanied by a service TXT record with the owner identifier. Only what is marked with this identifier is changed and deleted, so records created manually remain untouched.
The upsert-only mode prohibits deletion entirely, and the domain filter limits operation to specific zones.
How do you separate permissions between teams?
We separate sources: one team publishes services only in its own namespace, another in its own subdomain. Permissions for cluster objects are granted by roles, and permissions for the zone are granted by a separate provider account with access to one zone. A mistake in a manifest does not take a team outside its perimeter.
What are the limitations of the approach?
Automation covers records derived from cluster objects. Static records, mail routes, and geographic balancing remain with the DNS administrator: if there are many such records, we automate only the service part of the zone.
What to prepare for the discussion and how we'll start work
Estimating automatic DNS record management can fit into one conversation if the input includes a zone inventory and clear access boundaries.
Below are the inputs that remove most of the uncertainty: some are resolved directly in correspondence, the rest through access to a test environment.
- A list of domains and subdomains that should be managed from the cluster, and where manual management ends.
- The zone owner and a DNS access token with permissions only for the required zone, not for the entire account.
- A list of services and entry points that need records: only external or also internal.
- Synchronization mode: a record for every object or only for labeled ones, and what to do with existing records.
- The number of environments and how zones are separated between them.
- The change window and rollback rule: what happens to a record when a service is deleted.
Then the order is simple: a short conversation on the inputs, an inventory of zones and access, a pilot on a test domain, rollout across environments, and handover of the operations runbook.
The pilot takes several days — in it we verify creation, update, and deletion of records and separately the behavior during rollback. Full rollout across environments usually takes one to two weeks, taking into account the change window.
As a result, you are left with a reproducible scheme and a clear procedure if you need to return to manual management.