Cross-Region Data Replication for Disaster Recovery

After a primary data center failure in us-east-1, we discovered all backups were in the same region. Recovery from another region took 8 hours instead of the target RTO of 15 minutes. That's when we realized: cross-region replication is the only way to meet RPO/RTO. With over 5 years of DR experience, we've configured dozens of solutions for PostgreSQL, MySQL, and S3 for companies requiring RPO under one minute. Our team has completed more than 50 disaster recovery projects, with an average achieved RPO of 2 seconds using asynchronous replication.

Replicating PostgreSQL Between Regions

Streaming replication over the internet works but requires a secure channel. The typical setup: VPN or VPC Peering between regions, synchronous or asynchronous replication depending on RPO requirements.

For asynchronous replication between regions, Patroni with etcd in each region is the right choice. But you must explicitly mark the cross-region replica as not participating in leader elections:

---
# patroni-dr.yml — config for DR replica in another region
scope: pg-production
namespace: /service/
restapi:
  listen: 0.0.0.0:8008
  connect_address: 10.10.1.5:8008
etcd3:
  # Local etcd only for DR node state
  # Does not participate in primary cluster quorum
  hosts: 10.10.0.1:2379
bootstrap:
  dcs:
    maximum_lag_on_failover: 10485760  # 10 MB — allow lag for cross-region
    master_start_timeout: 300
postgresql:
  listen: 0.0.0.0:5432
  connect_address: 10.10.1.5:5432
  data_dir: /var/lib/postgresql/data
  parameters:
    # Read-only replica, not for elections
    hot_standby: "on"
    recovery_min_apply_delay: "0"
  tags:
    # Exclude from automatic failover — manual promote only
    nofailover: true
    noloadbalance: false
    nosync: false

Monitoring replication lag is critical. If pg_wal_lsn_diff on the DR replica grows — either the network degraded or master load increased:

-- On master: check DR replica lag
SELECT client_addr, application_name, state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) AS sent_lag_bytes,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes,
       write_lag, flush_lag, replay_lag
FROM pg_stat_replication
WHERE application_name = 'dr-replica-eu-west';

Replicating MySQL/MariaDB

For MySQL, binary logs suffice for cross-region replication. Modern GTID-based replication simplifies failover and recovery after a split:

# my.cnf on DR replica
[mysqld]
server-id = 2000 # unique, different from primary cluster
gtid_mode = ON
enforce_gtid_consistency = ON
read_only = ON
super_read_only = ON
# Parallel replication to reduce lag
slave_parallel_type = LOGICAL_CLOCK
slave_parallel_workers = 8
# Store replication position in table (not file)
master_info_repository = TABLE
relay_log_info_repository = TABLE
relay_log_recovery = ON

Replicating File Systems

For files and object storage, approaches differ.

AWS S3 Cross-Region Replication: enabled at the bucket policy level and works automatically. Important nuance — CRR replicates only new objects after enabling; existing ones must be copied separately using aws s3 sync:

# Enable Cross-Region Replication via AWS CLI
aws s3api put-bucket-replication \
  --bucket company-primary-eu-central-1 \
  --replication-configuration '{
    "Role": "arn:aws:iam::123456789:role/s3-replication-role",
    "Rules": [
      {
        "Status": "Enabled",
        "Filter": {"Prefix": ""},
        "Destination": {
          "Bucket": "arn:aws:s3::NOVASOLUTIONS-dr-eu-west-1",
          "StorageClass": "STANDARD_IA",
          "ReplicationTime": {
            "Status": "Enabled",
            "Time": {"Minutes": 15}
          },
          "Metrics": {
            "Status": "Enabled",
            "EventThreshold": {"Minutes": 15}
          }
        },
        "DeleteMarkerReplication": {"Status": "Enabled"}
      }
    ]
  }'

rsync + inotifywait for on-premise: for file servers outside the cloud — continuous replication using lsyncd or rsync scheduled via cron. lsyncd uses inotify and replicates changes in real time:

-- /etc/lsyncd/lsyncd.conf.lua
sync {
    default.rsync,
    source = "/data/media",
    target = "dr-server.company.com:/data/media",
    rsync = {
        binary = "/usr/bin/rsync",
        archive = true,
        compress = true,
        rsh = "/usr/bin/ssh -e aes128-ctr -i /etc/lsyncd/dr-key",
        _extra = {"--bwlimit=50000"} -- bandwidth limit: 50 Mbit/s
    }
}

Data Consistency Checks

Replication can run without errors yet produce data discrepancies — especially with asynchronous setups. For PostgreSQL, pg_checksums and periodic comparisons using pg_comparator or application-level checksums.

For S3 — periodic listing and object comparison between regions using AWS Data Sync or custom scripts.

Why Asynchronous Replication Is the Baseline Choice for Cross-Region?

Synchronous replication between data centers over 1000 km away adds at least 10 ms latency per packet, reducing write throughput to 1/3 of local. Asynchronous replication introduces 0.5–5 seconds lag but preserves master performance. In 90% of DR cases, asynchronous replication with RPO up to 10 seconds suffices. Patroni with the nofailover: true tag ensures the DR replica is not promoted automatically until the operator decides. Compare: synchronous replication gives RPO=0 but reduces TPS by 40% — we recommend it only for financial transactions within the same city. Asynchronous replication is 50x faster in TPS at latencies >10 ms.

How to Verify Data Consistency After Replication?

For PostgreSQL, use pg_checksums and pg_comparator. For large tables (10+ TB), comparing checksums of each WAL file is sufficient. For S3, run aws s3 sync --dryrun weekly to detect discrepancies in object names and hashes. Our engineers include such checks in automation by default.

Common Mistakes and Checklist for DR Setup

What's Included in DR Setup Work

Timeline Estimates

Pricing is calculated individually after infrastructure audit. Get a consultation — our engineers will assess your setup and propose an optimal turnkey solution with RTO/RPO guarantees. Set up DR that actually works — contact us to discuss your project details.

References

• PostgreSQL streaming replication
• AWS S3 Cross-Region Replication