Redis Cluster: where scaling breaks under load
A single instance hits three walls almost simultaneously. Memory is limited by the size of one machine: when the dataset approaches the maxmemory limit, the eviction policy kicks in — and keys that no one planned to lose are evicted.
Nearby, a second effect appears: background saving with fork on a large volume freezes command processing for hundreds of milliseconds, and this shows up in the latency graph, not in the logs.
Hot keys finish it off differently. A counter, a session, or a reference dataset that all requests access lives in one thread — no matter how many cores there are, the load falls on one.
Spreading such a key across replicas does not help: writes still go to the master, while reading from replicas returns desynchronization and stale values.
Memory overflow is felt as a cascade, not as a failure. Eviction switches from keys with an expiration time to arbitrary ones, some requests start going directly to the database, its latency grows, and at peak everything falls apart.
Plus, memory shortage goes into swap — response time is measured in seconds, but formally the service is alive.
This gets caught at the moment when a load peak coincides with writes: one bottleneck pulls the others with it, and there is nowhere to get a rollback to the previous state. That is why scaling Redis is planned in advance, not after an incident.
Below we examine how this is addressed by moving to Redis Cluster: sharding by slots, splitting hot keys, and predictable behavior when a node fails.
What the cluster provides: growth by nodes and failure without downtime
In Redis Cluster, the key space is divided into slots, and each master is responsible for its share.
As long as keys are evenly distributed, adding nodes raises the operations ceiling almost proportionally — the bottleneck moves from one process to the network and consistency between nodes.
We check the key schema before scaling: hot keys and operations across multiple slots consume gains faster than nodes are added. This is the case when horizontal database scaling runs into not hardware but the data model.
Cluster fault tolerance rests on replicas: each master has a copy, and if it fails, the other masters vote to promote the replica.
A whole class of failures goes out of operations — the failure of one node no longer stops writes or requires manual recovery from a backup with loss of recent operations.
Recovery time is reduced from tens of minutes of manual work to automatic failover in seconds. The window during which writes are unavailable shrinks, and clients reread the topology themselves and continue working without an on-call intervention.
Throughput also grows due to replicas: reads are distributed across copies if the application is ready for this.
Before changes, we take measurements — operations per second, 95th percentile latency, share of failures handled automatically — and repeat them after, so growth is measurable rather than claimed.
Redis Cluster or replication: comparison of operating modes
Replication and a cluster solve different tasks, so they are chosen by data profile, not by "power." Replication duplicates the same dataset across several nodes: fault tolerance and read throughput grow, but not volume. A cluster adds sharding — data is distributed across nodes, and writes also scale.
| Option | When it fits | Limitations |
|---|---|---|
| Replication with manual key splitting | Volume fits on one node, the application knows where to write | Node memory ceiling, role switching by an external arbiter, splitting logic lives in application code |
| Cluster with automatic slot distribution | Data does not fit on a node, writes to all masters are needed | Multi-key operations require a common tag, some commands are unavailable, rebalancing causes latency spikes |
| Hybrid: cluster, replicas, and a separate read contour | Heavy reports and exports must not hit production nodes | Harder maintenance, need to monitor replication lag and consistency |
| Application-side sharding | Full control over key placement is needed | Routing and rebalancing are on the application, topology changes require a migration plan |
The practical criterion is simple: if data no longer fits in node memory or writes hit a single master, replication does not solve the task. The opposite mistake is no less common: a cluster on small volumes brings command limitations and maintenance complexity with no gain.
We examine how to choose a mode for the load and data profile and how to plan a transition between them without downtime.
Implementation stages: from audit to handover to operations
Cluster implementation begins not with installation but with an audit of the load profile: we collect statistics on commands, read-to-write ratio, value size, key lifetime, and daily peaks.
This data determines how many nodes are actually needed, where the boundary between hot and cold data will run, and which keys cannot be spread across different nodes without losing operation atomicity.
- Load audit. We collect the traffic profile over a full cycle — working hours, night windows, reporting peaks. We record the share of heavy operations, the working set size, and memory headroom on each node.
- Data splitting schema. We design key placement by segments: choose a naming pattern, exclude hot keys, check that related structures land on one node. At this step we determine the number of segments and headroom for growth.
- Topology and fault tolerance. We distribute primary and replica nodes across different availability zones so that a rack failure does not take down a pair entirely. We configure quorum, timeouts, and client behavior during failover.
- Deployment and migration. We bring up the configuration and move data without stopping the service: gradual switching of reads, then writes, with discrepancy control at each step.
- Fault tolerance testing. We shut down nodes one by one and in groups, measure failover time and write loss, and check application behavior under load. We separately run the scenario of returning a node to service.
- Handover to operations. We hand over monitoring with thresholds, degradation alerts, a maintenance procedure, and documentation on the splitting schema — the customer's team then runs the cluster on its own.
The order of steps is not accidental: each next one relies on data from the previous one, so problems are caught before production rollout, not at the peak.
What remains with you: configurations, schemas, procedures
After cluster handover, you are not left with someone else's repository and oral explanations.
We hand over the complete set into your version control system; any further edits go through review: using it, your team can bring up a stand from scratch, add a new node, and restore data after a master failure — without contacting us.
- Configurations of all nodes: working files with cluster mode enabled, timeouts, memory limits, and key eviction policy, with comments — why this value was chosen and what will break if it is changed.
- Topology map: slot distribution across masters, replica binding, network zones, and failure points on a single diagram that can be shown to the on-call shift.
- Access policies: ACL-based separation — what each service reads and writes, separate accounts for monitoring and administrative operations, password rotation procedure.
- Backup scripts: scheduled snapshots from replicas, integrity checks, export outside the cluster, and regular test restores.
- Maintenance procedure: planned master failover, adding a node, slot rebalancing, and zero-downtime upgrades — with windows and readiness criteria.
- Incident response order: what to look at first, how to distinguish a failed node from network degradation, when to intervene manually, and when to wait for automatic failover.
- Cluster documentation: architectural decisions and why they are so, change log, acceptance checklist by which you accept the work.
Case: how a cluster removed failures under peak load
A single Redis instance held the cache, task queues, and user sessions. At peak — a sale, a mass mailing, morning app logins — the operation flow tripled, the node hit its ceiling, and scenarios unrelated to the cache went down with it. For the client, it looked not like "slow" but like "not working."
The analysis showed three bottlenecks: memory for data with fragmentation, command processing on a single core, and a complete absence of replicas. There was no spare node — when the process exited, the service did not come back up by itself; an on-call engineer did it manually.
What was changed
We deployed a cluster of several primary nodes with replicas, distributed keys across slots, and moved queues to a separate group of nodes so heavy operations would not evict user sessions.
Client libraries were switched to cluster topology mode: when the primary node changes, the connection is rebuilt without changes in the application.
Metrics before and after
- Failure rate under peak load: 12% of requests to zero — no unavailable operations remained at peak.
- Recovery time after a node exits: from manual 30-40 minutes to automatic replica promotion and failover in seconds.
- Operations ceiling: a single node stopped being the limiter — adding a node increases throughput manyfold.
- Service behavior when a node is lost: instead of complete failure — short degradation and rereading part of the keys from the database.
What to do if the application already works with a single server?
A single server hits not failures but a ceiling: the working set takes up memory, a replica saves you from a crash but does not provide room for growth.
While there is headroom, the transition is postponed, and when it runs out, it is migrated in a hurry at the worst possible moment. That is why migration is planned in advance and carried out without stopping the service.
First, inventory: which keys live long, where operations touch several keys at once, which structures cannot be spread across nodes. Multi-key commands and scripts require related keys to reside on one node — otherwise they are rewritten to common hash tags. This is the only part of the transition that really requires changes in application code.
We move data in the background: take a dump from the single server, load it into the cluster, distributing keys by slot ranges. Then we enable dual writes — the application writes to both the old instance and the cluster, while reading from the old one for now.
Migration runs on a live system, and the discrepancy between copies does not exceed the latency of one operation.
We verify integrity by comparing keys and values on a sample, then run a shadow read stream to the cluster and compare responses with the single server.
While discrepancies exist, we do not switch: client backward compatibility is ensured — they can already follow redirects between nodes, but work via the old address.
The rollback plan is a flag in the configuration: it returns reads to the single server, which remains warm and receives all writes.
If after switching the error rate or response time grows, the flag is removed in seconds, and the service continues working on the previous contour.
Technical doubts: node failure, migration, and compatibility
Before cluster handover, we go through what most often comes up at acceptance: behavior when a node is lost, the scope of application changes, network requirements, and what remains on our side after handover.
What happens if a node is lost?
Each master has a replica configured, and slots are reassigned automatically within seconds: the client receives a redirect and repeats the request. Data is preserved if the replica had time to synchronize, so we distribute nodes across different failure zones.
Will code need to be changed?
Current drivers work with a cluster, so code changes come down to configuration: a list of nodes instead of one address, cluster mode, timeouts, and retries.
We separately check operations across several keys — outside one slot they do not pass, so such places are converted to hash tags.
What are the network requirements?
Nodes need direct connectivity over the service port and even latency: state exchange is constant, and packet loss causes false master elections. We fix static addresses, exclude translation between nodes, and set health check thresholds with headroom.
What is included in maintenance after handover?
We hand over the configuration, slot map, and procedure for typical operations: adding a node, planned migration, restore from a copy, and upgrade one replica at a time.
The customer's engineers run scenarios on the stand together with us, so further expansion proceeds on their own.
Let's discuss the task: what is needed to start and estimate
Estimating a Redis Cluster deployment project takes less time the more accurate the input data. Below is what we need from you so that we come back with a work plan and timelines, not with clarifying questions a week later.
- Load profile: operations per second at peak and average, read/write share, typical value size — the number of nodes and memory headroom depend on this.
- Current schema: single instance, leader with replicas, or already a cluster; which applications and client libraries are connected to it.
- Expected growth: how load and data volume will change over six months — we lay in headroom so the configuration does not need to be rebuilt right after launch.
- Maintenance window: when a brief write interruption is acceptable, and when the transition must be invisible to users.
- Recovery requirements: acceptable downtime when a node fails and acceptable loss of recent writes.
- Infrastructure and access: how many machines are available, whether there is zoning, who on the customer side accepts changes, and what permissions our engineers need.
In response you receive a staged work plan, a node placement diagram with fault tolerance rules, and a transition scenario tied to your maintenance window.
The earlier the window and recovery requirements are agreed, the lower the risk that data migration will have to be planned twice.