Cassandra Multi-Datacenter Setup

When moving to a geo-distributed architecture, many face the challenge that a Cassandra cluster running in a single datacenter can no longer handle the load or fault tolerance requirements. We help set up a multi-datacenter configuration turnkey — it can withstand the loss of an entire datacenter without data loss or downtime. Our experience: 5+ years, over 30 projects with multi-datacenter Cassandra. Over this time, we've accumulated methodologies that avoid common mistakes: incorrect snitch selection, wrong seed nodes, and suboptimal consistency levels. We'll assess your project and propose an architecture aligned with your RPO and RTO.

Multi-datacenter Cassandra is not just "placing nodes in two locations." It's a separate discipline: snitch selection, replication strategy, query consistency, handling network partitions between datacenters. Without understanding the CAP theorem in the context of Cassandra, a multi-datacenter cluster becomes a machine for generating hint queues and stale data. Below, we break down the key aspects of multi-datacenter configuration.

How to Ensure Geo-Redundancy?

For a single datacenter, SimpleStrategy is used. For multi-datacenter configuration, only NetworkTopologyStrategy. It allows you to set different replication factors for each datacenter:

CREATE KEYSPACE app_data WITH REPLICATION = { 'class': 'NetworkTopologyStrategy', 'DC1': 3, 'DC2': 3 }; 

RF=3 in each datacenter means: the cluster will survive the complete loss of one datacenter without data loss. With DC1 in Amsterdam and DC2 in Frankfurt — this is real geo-redundancy. Compare: using SimpleStrategy, losing one datacenter would result in losing all data in it.

Which Snitch to Choose for Production?

Cassandra must know in which datacenter and rack each node resides. That's the snitch's job. GossipingPropertyFileSnitch is the standard choice for production. Each node reads /etc/cassandra/cassandra-rackdc.properties:

dc=DC1 rack=RACK1 

In cassandra.yaml on all nodes, specify the same cluster name and seeds from both datacenters:

cluster_name: 'global-cluster'
endpoint_snitch: GossipingPropertyFileSnitch
seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "10.0.1.10,10.0.1.11,10.1.1.10,10.1.1.11"

Seeds must be in both datacenters — this is important for bootstrapping new nodes and recovery after split-brain.

Why Choose LOCAL_QUORUM?

Here start the real architectural decisions. QUORUM considers quorum from all replicas across all datacenters. With RF=3 in two datacenters (6 replicas total) — you need acknowledgment from 4 replicas. If one datacenter is unavailable, the write fails. For most scenarios, this is too strict.

LOCAL_QUORUM considers quorum only in the local datacenter. A write succeeds if a majority of local replicas acknowledged. This is the standard choice for multi-datacenter clusters — the application writes with LOCAL_QUORUM, reads with LOCAL_QUORUM, and continues working if a datacenter is lost. In fact, LOCAL_QUORUM gives a 2x performance improvement compared to global QUORUM when one datacenter fails: for a write, only 2 out of 3 replicas in the local DC are needed, not 4 out of 6.

EACH_QUORUM requires quorum in each datacenter — used when even temporary data divergence between datacenters is unacceptable. In practice: write with LOCAL_QUORUM, read with LOCAL_QUORUM. If you need to guarantee reading fresh data from another region — use SERIAL or LOCAL_SERIAL for lightweight transactions.

Comparison of consistency levels:

How We Do It: A Case Study

Recently we deployed a multi-datacenter setup for a client with nodes in AWS (us-east-1) and Azure (west-europe). Requirement: RPO < 1 minute, automatic failover when one DC fails. We used GossipingPropertyFileSnitch, RF=3 in each datacenter, LOCAL_QUORUM for writes and reads. To minimize latency, we configured cross-datacenter repair with throttling at 50 MB/s. As a result, when the AWS region failed, the application continued working without data loss and without manual intervention. Time saved on recovery — up to several hours manually.

Typical Problems with Multi-Datacenter Clusters

• Different Cassandra versions in datacenters. During a rolling upgrade, you need to update nodes one by one, avoiding incompatible protocol versions between datacenters. Cassandra supports rolling upgrade only between adjacent minor versions.
• Incorrect seeds. If seeds point only to one datacenter, nodes in the second datacenter may not see the first one on restart. Seeds must be in each DC.
• Firewall between datacenters. Cassandra needs port 7000 (inter-node communication) and 9042 (CQL) open between all nodes in all datacenters. Partial blocking leads to split-brain.
• Timezone with TTL. Cassandra calculates TTL in UTC. If servers in different datacenters have different timezones and time is not synchronized via NTP — data with TTL is deleted asynchronously.

Commands for repair:

# Repair only local datacenter
nodetool repair -dc DC1 -pr keyspace_name
# Full cluster repair (will load the inter-DC link!)
nodetool repair keyspace_name

Inter-datacenter repair is a heavy operation. On clusters with terabytes of data, run during minimal load with throttling:

nodetool setstreamthroughput 50 nodetool setcompactionthroughput 50 

Deployment Process

We start with one datacenter — deploy the cluster, test replication, load test. Only then add the second datacenter.

When adding the second DC: start nodes in the second datacenter with the same seeds, wait for gossip to propagate (nodetool status shows nodes from both DCs), then run nodetool rebuild -- DC1 on each node in DC2 to copy data from the first datacenter. After rebuild, alter the keyspace to include replication in DC2 and run repair.

What's Included in the Work

• Audit of current architecture and topology recommendations.
• Cluster deployment in one or multiple datacenters.
• Configuration of NetworkTopologyStrategy, snitch, consistency levels.
• Monitoring setup (Prometheus + Grafana) and alerts.
• Operations documentation and failover runbook.
• Client team training (2-3 sessions).
• Post-release support for 2 weeks.

Pricing is determined individually after analysis of RPO/RTO requirements and data volume. Contact us for a project assessment — get a consultation on your architecture.

Apache Cassandra is a distributed database management system. Wikipedia