Automatic Failover: Setup, Testing, and Split-Brain Protection

Last week, a client's production database went down at 3:15 AM. The on-call engineer woke up, but Wi-Fi was glitchy. After logging in 20 minutes later, the recovery script failed due to a configuration error. Downtime was 45 minutes — each hour of downtime cost the company an average of $10,000, so they lost $7,500 on this incident alone. Automatic failover could have reduced this to 30 seconds — 30-60 times faster than manual switching. But only if it's configured correctly. We've encountered cases where automation made things worse: split-brain, data divergence, manual recovery taking 4 hours. Therefore, failover configuration must be approached with an understanding of the pitfalls.

We develop and implement automatic failover — a system that detects the failure of the primary site and switches traffic to the backup without human intervention. This is a nontrivial task: protection from split-brain, false positives, and correct order of service stop and start are required. A mistake here is more dangerous than manual switching.

How Split-Brain Protection Works

Split-brain occurs when both sites consider themselves active and accept writes simultaneously. The result is data divergence that is hard to reconcile. Classic case: the network between regions was down for 30 seconds. The DR site promoted the replica, while the primary remained alive and continued accepting transactions. After recovery, we had two masters with different data.

Protection — quorum via external arbiter. For PostgreSQL we use Patroni with etcd, Consul, or ZooKeeper. For clusters — witness node or STONITH (Shoot The Other Node In The Head) — guaranteed shutdown of the node before promote.

---
# patroni.yml — automatic failover configuration
scope: pg-production
bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    # Maximum replica lag at which it can become master
    maximum_lag_on_failover: 1048576  # 1 MB
    # Synchronous mode: at least one replica must acknowledge write
    synchronous_mode: true
    synchronous_mode_strict: false  # false = do not block write if no replicas
postgresql:
  use_pg_rewind: true  # allows returning old master as replica
  # Script called on role change
  post_failover:
    - name: notify_ops
      command: /usr/local/bin/notify-failover.sh
---

Why DNS Failover Doesn't Always Help

DNS failover via Route53 Health Check or Cloudflare Load Balancing is a popular solution, but with a serious drawback: TTL caching. Even with TTL=60, clients may keep the old IP for several minutes. For critical applications, we use Anycast (Cloudflare Argo, AWS Global Accelerator) — switching at the network level, bypassing DNS cache. Anycast gives switchover times under 10 seconds compared to 60-300 seconds for DNS.

# terraform/route53-failover.tf
resource "aws_route53_health_check" "primary" {
  fqdn = "app-primary.company.internal"
  port = 443
  type = "HTTPS"
  resource_path = "/health"
  failure_threshold = 3
  request_interval = 10
  tags = {
    Name = "primary-health-check"
  }
}

resource "aws_route53_record" "app_primary" {
  zone_id = aws_route53_zone.main.zone_id
  name = "app.company.com"
  type = "A"
  failover_routing_policy {
    type = "PRIMARY"
  }
  health_check_id = aws_route53_health_check.primary.id
  set_identifier = "primary"
  ttl = 60
  records = [var.primary_ip]
}

resource "aws_route53_record" "app_dr" {
  zone_id = aws_route53_zone.main.zone_id
  name = "app.company.com"
  type = "A"
  failover_routing_policy {
    type = "SECONDARY"
  }
  set_identifier = "dr"
  ttl = 60
  records = [var.dr_ip]
}

How to Automate Failover with AWS Lambda

For complex scenarios, we orchestrate failover via Lambda, triggered by EventBridge from a CloudWatch alert. The script promotes the Read Replica, updates config in SSM, restarts ECS services, and switches Route53. Execution time is about 2 minutes. More details in the AWS Lambda documentation.

# lambda/failover-orchestrator.py
import boto3
import json

def lambda_handler(event, context):
    """ Called by EventBridge on CloudWatch alert: DatabaseDown severity=critical duration=5m """
    rds = boto3.client('rds', region_name='eu-central-1')
    r53 = boto3.client('route53')

    # Step 1: Promote RDS Read Replica in DR region
    response = rds.promote_read_replica(
        DBInstanceIdentifier='prod-dr-replica',
        BackupRetentionPeriod=7,
        PreferredBackupWindow='03:00-04:00'
    )

    # Wait for replica to become standalone
    waiter = rds.get_waiter('db_instance_available')
    waiter.wait(DBInstanceIdentifier='prod-dr-replica')

    # Step 2: Update application config via SSM Parameter Store
    ssm = boto3.client('ssm', region_name='eu-west-1')
    dr_endpoint = rds.describe_db_instances(
        DBInstanceIdentifier='prod-dr-replica'
    )['DBInstances'][0]['Endpoint']['Address']
    ssm.put_parameter(
        Name='/app/database/host',
        Value=dr_endpoint,
        Type='String',
        Overwrite=True
    )

    # Step 3: Restart ECS services in DR region with new config
    ecs = boto3.client('ecs', region_name='eu-west-1')
    ecs.update_service(
        cluster='prod-dr',
        service='app',
        forceNewDeployment=True
    )

    # Step 4: Switch Route53
    # (call update_health_check or change_resource_record_sets)
    return {'status': 'failover_initiated', 'dr_db': dr_endpoint}

How to Test Failover Without Risk

We periodically verify that automation works. The testing process consists of steps:

1. Disable the health check endpoint for the primary resource.
2. Ensure traffic has switched to the DR site (check logs and metrics).
3. Re-enable the health check.
4. Bring primary back and perform switchback using patronictl switchover.

For Patroni, use patronictl failover --master current-master --candidate replica-dr --force. After each test failover, always verify return to primary mode. We recommend running tests at least quarterly, and monthly for critical systems. Savings from failover implementation can exceed $100,000 per year by preventing downtime.

Comparison of Failover Methods

What Our Work Includes

• Audit of current infrastructure and identification of single points of failure
• Architecture design for failover (DNS/Anycast, database, application)
• Configuration of Patroni, Route53, Lambda, Ansible
• Development of runbooks and rollback procedures
• Testing scenarios: primary failure, replica failure, network failure
• Documentation and knowledge transfer to your team

Estimated Timelines

Project cost is discussed after an audit. Get a consultation on failover setup — contact us to evaluate your project. Our engineers have over 10 years of production infrastructure experience. Order an infrastructure audit today.