DNS Failover: why one node brings down the entire service
A typical failure looks predictable: an A record points to a single address, and everything—the application, database, queue—lives on one site. The node or its link goes down, but traffic keeps going to the dead address.
Failover is done manually by the on-call engineer: they hear by phone, open the panel, change the record—and all this time the service is down for everyone at once.
The reason is not the speed of editing the record, but the cache. Recursive resolvers, provider caches, and the operating system's local cache hold the answer for exactly as long as specified in the TTL.
With typical values, that is hours, so instantly editing the zone does not help: some clients keep hitting the decommissioned node for a long time, and recovery time ends up fluctuating.
A separate point of failure is the zone itself: one authoritative server, one registrar account from which records are edited. If the panel is unavailable or the domain has expired, there is nothing to switch traffic with, even when the backup site is fully ready.
Anycast removes part of this problem—one address is announced from multiple points, and routing diverts traffic away from the failed site automatically, but only at the network level.
We break down how this is addressed: short TTLs for failover, automated health checks, multiple authoritative servers at different providers, and a pre-defined action sequence without a phone call.
What managed traffic failover provides
Managed failover removes the manual step from recovery. Node health checks run continuously, and as soon as a site stops responding to them, traffic automatically moves to the backup—without calling the on-call engineer or editing records manually.
Recovery time is now determined not by human reaction speed, but by the DNS record TTL and the check trigger threshold.
In practice, this turns recovery from tens of minutes into single-digit minutes, and with a short record TTL—into seconds.
The failure class “site went down—service went down” stops reproducing: the service keeps responding from the backup, and the user sees at most a single error at the moment of failover. Service availability stops depending on who exactly is on shift.
Load distribution is set by weights between sites. The backup is not idle: it continuously carries part of the traffic, so at the moment of failover it does not get overwhelmed by a sudden volume—its capacity has already been proven by real operation. At the same time, the risk of a “backup that has not been started for six months” is eliminated.
We configure node health checks based on an application-level signal, not just whether the network responds: a node may be up but returning errors—that scenario is caught as well. The trigger threshold is set with a margin so that a short spike does not cause a false failover.
In reporting, we calculate service availability by the share of successful responses, not by declared uptime.
Returning traffic to the primary site is especially important—it is done gradually, in weight steps, while monitoring latency and errors.
DNS Failover options: comparing operating modes
Choosing a mode is a trade-off between response speed and predictability. An active-passive scheme keeps one address in operation, while the second waits in hot standby and is enabled based on site health checks.
An active-active scheme serves both addresses at once, so a failure of one site does not require waiting for the mechanism to trigger. Manual failover leaves the decision to the engineer and is used where automation could do harm.
| Option | When it fits | Limitations |
|---|---|---|
| Active-passive scheme | Primary and backup site, different capacity, rare failures | Failover takes from one to several minutes: while the record TTL is alive, some clients go to the failed address |
| Active-active scheme | Two equal sites, zero-downtime requirement, traffic is growing | Both sites must hold identical data and configuration, otherwise some requests go to an unprepared node |
| Manual failover | Scheduled maintenance, migrations, test environments | Speed depends on the on-call engineer; at night and on weekends the response window grows |
The key parameter in any mode is record TTL. A short TTL speeds up failover but increases the load on name servers and query traffic; too long a TTL stretches the outage across the entire cache time of intermediate resolvers.
In practice, it is selected based on the acceptable downtime window and lowered in advance, before scheduled work, not at the moment of an incident.
Failover is further constrained: we check multiple points and require several failed responses in a row so that a single network failure does not divert traffic away from a live site.
Implementation sequence: from audit to handover to support
Traffic failover when a node or an entire region fails is not a single setting but a sequence of work where each step relies on the result of the previous one.
Until the zone is described, there is nothing to configure health checks against, and without a drill failure you cannot claim that failover will work in a real incident rather than on paper.
- Zone audit. We inventory records: which point to production nodes, which are used by mail and services, where the zone is delegated, and who has the right to change it. The output is a dependency map and a list of records that automation must not touch.
- Failover scheme. We determine how traffic is distributed between sites, by what signal the primary node is selected, and how quickly records stop pointing to the failed one. We separately record session behavior and what happens to clients at the moment the address changes.
- Health check configuration. We define exactly what is checked—service response or application-level response—how many confirmations are needed before triggering and before returning, so that we do not fail over on a short failure and do not “flap” between sites.
- Drill failure. We disable the primary node and look at the actual time to recovery, the share of lost requests, and whether the backup performs correctly under load. A discrepancy with the calculation is a reason to revisit the thresholds.
- Documentation. Diagram, responsibility matrix, manual failover and rollback procedure, and the points where automation stops and a human is needed.
- Handover to support. The on-call shift receives alerts, instructions for typical failures, and a regular verification procedure: every quarter we repeat the drill failure, otherwise the setup quietly drifts from reality.
Configurations, diagrams, and procedures delivered
The point of the work is that after handover, all failover logic remains with your team, not in a chat thread with a contractor. We deliver configurations in your repository, relationship diagrams, and procedures by which the on-call engineer makes decisions without us. Below is what exactly is recorded in the handover package.
- Zone configurations: zone files with a full set of NS, SOA, and service records, TTL values by record type, split-horizon views for external and internal resolvers.
- Health checks: probe addresses and ports, polling intervals, trigger threshold, record priority and weight order, rules for automatic return to the primary node after recovery.
- Topology diagram: where primary and backup nodes are located, how zones are synchronized between them, which network routes and addresses are involved, and what happens if a site fails entirely.
- Access policies: roles and rights for changing records, separation of environments, zone transfer signing keys, management token limits and lifetimes, and a log of all changes.
- On-call procedure: which events generate an alert, who responds and within what time, escalation order, and decision points for manual failover.
- On-call instructions: step-by-step scenarios for resolution degradation, manual traffic failover, result verification, and rollback to the previous state.
- Monitoring: a set of metrics for response time and successful resolution rate, synthetic probes from external points, and a status dashboard segmented by service.
The package is structured so that a new engineer can reproduce the scheme from documents rather than from colleagues’ memory. When the topology changes, we update both the diagrams and the scenarios—otherwise the documentation drifts from reality in a single release.
How this works in practice: before and after metrics
Before: manual failover and cold standby
The infrastructure lived on one active node: records in the zone had a long TTL, and the backup site was brought up manually when the on-call engineer was alerted.
When the primary site lost network connectivity, traffic kept going to the unavailable address until the engineer changed the records—failover took 3–7 minutes.
The backup was “cold”: it served only static content, without current data and without a warmed cache, so it would not have handled the full load.
After: automation and hot standby
Failover was moved to automation: external monitoring from several points takes health checks from both nodes, and records in Cloudflare are switched based on their results, with record TTL reduced to one minute.
The backup node was made hot—data replication, cache warming, and a load test for the full traffic volume, so that it could handle 100% of requests with headroom, not “as it goes.”
Probe thresholds were configured so that one lost response would not trigger failover: confirmations from several points are required within the observation window.
False triggers almost disappeared, and the load on the backup node at the moment of an incident fits within its design capacity—without response time degradation.
Over a quarter, availability increased from 99.5% to 99.97%, average failover time decreased from 3–7 minutes to 45–70 seconds, and the share of failures that reached the user fell from 0.4% to 0.03%.
There are no more manual night wake-ups: the on-call engineer only confirms that automation worked as expected, and false triggers are analyzed using the failover log.
What to do if the backup environment is already deployed
The backup environment is deployed, services in it respond, but emergency failover does not trigger—the infrastructure is ready, but the procedure is not.
Three things should be checked in order: what signal the failover decision is based on, whether the data state between environments matches, and whether the picture from internal checks matches what users see.
Usually the third is what breaks: the check hits the service from inside the network, while client traffic comes through external resolvers.
| Failover scenario | When it fits | Limitations |
|---|---|---|
| Automatically based on health checks | Backup is always warm, data is synchronous | False triggers during network failures and degradation of the external route |
| Manual failover mode by on-call command | Critical services where the cost of an error is higher than downtime | Requires a 24/7 responsible person and an unambiguous procedure |
| Incident detection by automation, confirmation by a human | Need failover without downtime, but without “flapping” the zone | More complex procedure, explicit confirmation criteria required |
A readiness check answers only the question “is the backup alive?” Failover without downtime requires more: drills with real traffic switching, control of record TTL in resolver caches, and confidence that data replicates before the incident rather than catching up after.
Requirements for zone delegation and working with a registrar are set out in ICANN rules—it is useful to verify them in advance, not during an outage.
Manual failover does not cancel automation; it sets the boundary beyond which a human makes the decision.
A workable option is automatic incident detection with mandatory confirmation by the on-call engineer: the zone does not “flap” from a brief network failure, and failover fits within the recovery window.
Common technical concerns before starting
At the start, the same questions almost always come up: will failover get stuck because of cache, will there be false triggers, and will mail be affected? We answer briefly—assuming you will verify the solution yourself.
Will failover get stuck because of provider cache?
Record TTL is the key parameter: on failover records, we keep it low so that provider servers re-query the address every few minutes. Too short a TTL increases the load on the authoritative server, so we select the value based on the traffic profile.
How frequent are false check triggers?
A single check itself becomes a point of failure. Therefore, the state is confirmed by a series of failed responses from several independent nodes in different networks—a single timeout or brief packet loss does not trigger failover.
Can only part of the services be failed over, and what about mail?
The policy is set per record: only service subdomains are failed over, while the main site remains static.
We do not include mail records in failover—when a node is unavailable, messages accumulate in the sender’s queue, and a sharp route change hurts domain reputation and authentication checks.
How does return to the primary environment work?
Return is the same procedure in reverse, but with a preliminary check of the backup: the recovered environment must pass tests before it receives traffic. Otherwise failovers will start going back and forth, and clients will see fluctuating latency.
Let’s discuss your fault-tolerance scheme
To come back with an assessment and work plan rather than clarifying questions, we need enough initial information about your fault-tolerance scheme.
Below is the minimum set that covers 90% of unknowns: our engineers will pick up the rest themselves, including a test environment for validating failover.
- List of zones and subdomains that must survive a failure: production domains, service ones (API, personal account, static), mail records.
- Where the zones currently live: registrar, managed DNS provider, or your own name servers, and who holds delegation and NS records.
- What is behind the records: address pools, load balancers, external services, CDN—this determines at what level traffic should be switched.
- Target failover time and acceptable downtime window: from this we calculate check frequency, trigger thresholds, and record TTL.
- Control points: from which regions and networks availability is checked, by what signals a node is considered down, and whether a manual confirmation mode is needed.
- Work sequence and procedure: who approves changes in production, whether a failover log and notifications to the on-call shift are needed.
- Access and restrictions: registrar roles, rights to edit zones, availability of a test domain, and prohibitions on automatic changes.
With this set, we assemble a fault-tolerance scheme, estimate the scope, and provide implementation timelines. Describe the task in whatever form is convenient—we will come back with an assessment and work plan.