Geo-distributed cluster: where fault tolerance breaks
A geo-distributed cluster is built for fault tolerance: sites are separated, and the failure of one should not stop the service. In practice, it is precisely on this separation that what worked for years in a single data center breaks. Let's look at three typical failures that surface after migrating to a geo-distributed cluster.
The first is data desynchronization. While a write goes to one site, replication catches up with a delay, and when reading from the second site, the client sees a stale cart, an outdated inventory balance, or an order status that has already changed.
The error does not reproduce on a single-node test environment and only manifests under load, when the replication queue grows.
It is fixed not by "beefing up the server," but by choosing a consistency model for specific entities: in some places, reads strictly from the leader node; in others, a lag of fractions of a second is acceptable.
The second is session breakage. A session written on one site is not visible on another, and the load balancer that redirects traffic during a failure throws all authenticated users back to the login form.
For an online store, this means lost carts; for a portal, aborted requests. What matters here is a shared session store, correct cookie domains and paths, and the order in which the load balancer switches traffic relative to node readiness.
The third is degradation during node failure. The cluster is formally alive, but the second site cannot handle the full load: there is not enough capacity, caches are not warmed up, background tasks and agents are not running.
The failover is "successful," but the service responds several times slower or partially. That is why fault tolerance is tested not by the fact of failover, but by how long the service remains healthy after it.
All three failures share one thing: they manifest at the junction between sites, not within each of them. Let's break down how this is addressed at the level of architecture and operational procedures.
What a turnkey geo-distributed cluster delivers
A turnkey geo-distributed cluster is not "a site on two servers" but a pre-engineered scheme for placing the application and data across multiple sites with automatic traffic failover.
We fix the target recovery time before work begins and select the topology to match it: an active node in one location with a hot standby in another, replication of the database and shared files, a single entry point with load balancing and health checks for each node.
The first measurable effect is reduced recovery time. A site failure stops being an event requiring manual troubleshooting: the load balancer removes the node from rotation based on the health check, traffic moves to the standby, and the database comes up from the replica.
Recovery is measured in minutes instead of hours of downtime, rather than "however it turns out."
The second is an eliminated class of failures. A single server fails entirely: disk, power, network, overheating — all of these are a single point of failure. In a geo-distributed cluster, these failures are localized to one site and do not stop the service.
We separately close the class of failures related to updates: a release is rolled out to one node, verified, and only then goes to the rest, so an error in the code does not bring down the entire project.
The third is stability under peak load. A distributed scheme allows adding nodes for seasonal spikes and parallelizing the delivery of static content and cache across sites, so traffic growth does not turn into response time growth. We model the peak in advance and verify that the cluster can handle it.
We break down how this is covered: from auditing the current infrastructure and choosing a topology to configuring replication, failover switching, and operating procedures.
Distribution schemes: active-active, active-passive, hybrid
The choice of scheme is determined not by the "fashionability" of fault tolerance, but by how the application works with data. A 1C-Bitrix cluster keeps state in two planes: file storage (uploads, cache, sessions) and the database.
As long as writes go to a single node, any schemes with two active sites run up against how you synchronize writes between them. Hence three working modes that are actually chosen between.
Active-active
Both sites accept traffic, and the balancer distributes requests by geography or weights.
It works predictably when writes are reduced to a single point: sessions are moved to shared storage, the cache is not tied to the node's local disk, and content changes go through a single master site.
The limitation is that database replication between sites is asynchronous, so there may be a delay when reading from a replica. For the catalog and static content this is unnoticeable, but for the cart and the personal account it is already a risk.
Active-passive
Traffic goes to the primary site, while the second holds a replica in hot standby and is brought online in the event of a failure. Simpler to maintain: a single write point, no replication conflicts, predictable behavior when updating content.
The price is downtime during the switchover and the fact that the standby sits idle. For projects with intensive editorial work and integrations with an accounting system, this is the most honest mode.
Hybrid scheme
The primary site accepts writes, while the second serves reads and delivers static content closer to the user. A compromise between cost and response time: part of the load is taken off the master, but the write logic remains in one place.
It requires careful configuration of routing — which requests can be sent to the replica and which only to the master.
| Option | When it fits | Limitations |
|---|---|---|
| Active-active | High read traffic, geographically distributed audience | Asynchronous replication, write conflicts, harder to maintain |
| Active-passive | Integrations with an accounting system, active editing, predictability as a priority | Downtime during switchover, the standby carries no load |
| Hybrid | Fast response time is needed without fully duplicating writes | Requires separating read and write requests, fine-tuned routing configuration |
The operating mode is chosen to match the load profile, not the other way around. Next we break down exactly what needs to be synchronized in each of the options.
Implementation stages: from audit to handover
Work on a geo-distributed cluster proceeds in stages with fixed deliverables at each one: the client sees exactly what was done and how it was verified, rather than "work is underway." Between stages there are checkpoints where priorities can be adjusted without redoing what has already been built.
- Infrastructure audit. We map the current landscape: where the web servers, database, file storage, and cache live, and how traffic is distributed between sites. We identify bottlenecks — a shared disk, a single point of failure, inter-region latency.
- Cluster design. We define the topology: which sites are active, where the database master is, how replication propagates, and what survives a complete failure. The design is agreed upon before work begins — then we build and verify against it.
- Environment preparation. We deploy servers at the sites, configure the network between them, and bring configurations into a uniform state. Separately, we verify that the environments are reproducible: the cluster is built from a description, not "by hand from memory."
- Replication and synchronization setup. We link the database, files, and cache so that data converges between sites predictably. We configure replica lag, behavior during a channel outage, and the recovery procedure after a disconnect.
- Failover testing. We bring down nodes and entire sites according to a scenario, measuring how long it takes for traffic to move to the live part and back again. Separately, we verify that failover does not lose data or break user sessions.
- Load testing. We run a profile close to real-world conditions: peak requests, heavy queries, simultaneous operation from multiple regions. We look at where things hit the limit first — the database, disk, or network.
- Handover and documentation. We deliver the design, configurations, failover and recovery procedures, and instructions for the on-call shift. We train the client's team so the cluster can be maintained without us.
Each stage closes with a verifiable deliverable: a design, a configuration, a test report. Describe your task — we will come back with an estimate and a stage-by-stage plan.
What the customer receives
The cluster is not handed over "as is," but as a package that your team can use to operate and evolve it without us.
Each artifact addresses its own class of tasks: configurations — reproducibility, diagrams — understanding of the topology, procedures — predictability of actions in normal mode and during a failure.
- Configurations of all nodes — files for the web servers, load balancer, application servers, and databases in a version control system. Deploying a new node comes down to applying a configuration rather than manually assembling it from memory.
- Placement diagrams — the topology of the geo-distributed cluster with availability zones, traffic routes, failure points, and failover logic. It separately notes which nodes hold state and which can be brought up and shut down freely.
- Operational procedures — the order of scheduled maintenance, adding a node, updating the kernel and modules, and rotating backups. Step by step, with checks indicated after each step.
- Recovery procedures — scenarios for the failure of a zone, an application node, and a database. For each: trigger indicators, sequence of actions, expected time to return to service.
- Monitoring and alerts — configured checks for availability, replication lag, and queues, with thresholds and notification routes. The on-call engineer sees degradation before the user notices it.
- Access documentation — a map of accounts, roles, and permissions by component, with the procedure for granting and revoking them. An employee's departure does not leave any doors open.
- Implementation log — what changed and in what sequence, what decisions were made and why. Six months later, this saves days when figuring out "why it was done this way."
The package is agreed upon with your engineers before handover: we go through the diagrams and procedures together with those who will remain to operate the cluster. Describe your task — we will come back with an estimate and a plan.
Case study: a cluster for an online store across two sites
The client is an online store with peak load during sales, one site, one application server, and a shared database.
At peak hours, the cart and checkout responded with delays, and a node failure meant a complete halt in sales: recovery took hours because everything was brought up manually and without a hot backup.
1C-Bitrix in such a setup is a single point of failure: the web layer, the database, and the sessions all live on one circuit.
We split the system across two geographically remote sites, moved the database into "master — replica with automatic promotion" replication, and moved static content and sessions to a shared layer accessible to both sites.
What changed in numbers
| Metric | Before | After |
|---|---|---|
| Recovery time on node failure | hours of manual work | minutes, automatic switchover |
| Availability at peak hour | dips and timeouts | steady response under load |
| Failure of one site | complete halt | the store keeps selling |
| Backups | scheduled, with gaps | continuous replication |
Why this works
The key is in the separation of roles: traffic is balanced between the sites, the database replicates synchronously for critical operations and asynchronously for analytics, and sessions are stored outside the application servers.
That is why the failure of any one site does not bring down the store — the remaining one takes on the load, and the database replica is promoted to master.
Stability grew not through "more hardware," but through eliminating the single point of failure.
We ran failure scenarios on a staging environment before production: we disabled a node, the database, and the link between the sites — and confirmed that switchover happens without losing orders. After launch, the store weathered several peaks without manual intervention from engineers.
How is a geo-distributed cluster different from ordinary replication?
Ordinary replication solves one problem: keeping a hot copy of the database so that if the primary node fails, you can switch to the copy and not lose data.
Geography here is secondary — the replica can sit in a neighboring data center, and from the application's point of view it is the same perimeter.
A geo-distributed cluster begins where nodes are physically spread across different sites and each of them must serve traffic independently, without waiting for a response from another region.
The key difference is in the write model. Replication is asynchronous by default: the master accepts the write, and the replica catches up with a delay.
As long as the delay is measured in seconds, everything looks great, but if the channel between sites breaks, you get data divergence, and on failover you lose the last transactions.
A geo-distributed cluster is built around a quorum: a write is confirmed only when a majority of nodes has accepted it, so switching between sites does not roll back data already shown to the user.
The second difference is read routing. In ordinary replication, the load balancer often sends read requests to any available node, and the user may see a stale page right after saving.
In a distributed perimeter, reads are tied to the user's site, and the session is pinned to the node that confirmed the last write.
This noticeably changes the behavior of the cart, the account area, and the admin panel: actions stop "disappearing" when switching between regions.
A typical misconception is to think that it is enough to spin up a replica in another region and declare it a distributed cluster.
Without a quorum, without coordinated failover, and without control over replication lag, such a scheme gives only faster reads, but not site-level fault tolerance.
The opposite mistake is building strict synchronicity between all nodes: over a long distance the write confirmation delay grows, and admin operations start to slow down noticeably.
A workable compromise is a synchronous quorum within a site and asynchronous transfer between regions with lag monitoring.
The third difference is operations. Replication lives by the "master and followers" principle: one node is primary, the rest read.
In a geo-distributed cluster there are more roles — there are nodes that accept writes, read-only nodes, arbiters for the quorum, and a disaster recovery site.
This leads to separate monitoring requirements: it is important to see not only "is the node alive," but also the lag between sites, the state of the quorum, and which site currently holds leadership.
Technical doubts before starting
Distributing across sites almost always raises three questions: what about the compatibility of the current code, how the load will behave, and what will happen to the data during the switchover. Let's break down each one — to the point.
Compatibility
The cluster does not require rewriting the site: the application stays the same, only the storage and routing layer changes. Problems surface in two places — sessions and uploaded files.
We move sessions to shared storage, otherwise the user will be logged out when switching between sites. Files — to object storage or a synchronized volume, so that an image uploaded on one site is available from another.
If the code has direct writes to local paths or a file cache without a shared backend — these are the points we check during the audit, before migration.
Distribution and load
The sites hold a synchronous database replica and asynchronous copies for reads — writes go to a single node, reads are distributed. This takes the main load off the database and provides headroom for concurrent visitors.
The load balancer checks the health of the nodes and removes from rotation the one that has stopped responding — traffic goes to the live site without manual intervention.
What happens to the data if one site fails?
The synchronous replica guarantees that a confirmed write exists on at least two nodes. If a site fails, the load balancer switches the traffic, and the database continues to operate on the remaining node. The asynchronous copies catch up after recovery — the divergence is closed automatically.
Can the sites be distributed in stages, without downtime?
Yes. First we bring up the second site and configure replication, then we switch reads to it, and only after that — writes. Each step is reversible: as long as the traffic has not been switched, the main site operates in its previous mode.
Let's discuss your task
Deploying a geo-distributed cluster makes sense when a site no longer fits within a single node: the database and web servers start competing for resources, and a site failure means a complete shutdown of the service. We build such a setup turnkey — from topology design to commissioning and ongoing support.
To come back with an estimate and a work plan, we need a minimum of inputs: the current architecture, peak loads, fault-tolerance requirements, and user geography. That's enough to propose a specific topology rather than general considerations.
- A scheme for distributing roles across nodes: where the web servers, database, cache, load balancer, and file storage live.
- A data replication and fault-tolerance configuration plan, indicating failure points and behavior when a site goes down.
- Traffic balancing and user routing configuration between regions.
- A migration procedure from the current environment without downtime or with a minimal unavailability window.
- Backup, monitoring, and alerting regulations for each node.
- A resource estimate: how many nodes, what class of servers, what communication channels between sites.
- A phased work plan with responsible parties and acceptance criteria at each step.
Describe your task — the current load, audience geography, and availability requirements. We'll come back with an estimate, topology options, and a work plan that shows what we do and in what sequence.