Tarantool administration: where the cluster loses resilience
Tarantool keeps the working set in RAM, while the write-ahead log and periodic snapshots go to disk.
This provides predictably low read response time, but by itself does not create fault tolerance: a cluster consisting of a single node with the leader role fails entirely — together with writes, reads, and all services that depend on it.
Failure of the leader node is the first typical point of failure.
If the replica is raised for read only, and automatic leader election is not configured, writes stop until manual intervention: an engineer switches the role, the application reconnects, and some requests are lost in that window.
More dangerous is the case when the replica has lagged — switching to it rolls the data back, and the loss of the last confirmed operations is discovered already in production.
The second point is data volume growth beyond the allocated memory. An in-memory cluster does not tolerate uncontrolled growth: when it fills up, eviction begins, writes slow down, and under peak load some operations drop off by timeout.
Degradation under load manifests as a gradual increase in response time, and it can only be noticed in advance by fill metrics and retention rules.
The third is manual switchovers and lack of state control: roles in the cluster change by agreement rather than by actual availability, and no one tracks replica lag or snapshot integrity.
Next, we break down how this is addressed — from topology and automatic leader election to memory control and failure recovery.
What Tarantool administration delivers: the effect in numbers
What changes under regular cluster operation is visible in three metrics: how long the service is down when a node fails, what share of switchovers happen without human involvement, and how long an update takes without stopping operations.
Support of an in-memory cluster is built on replication and quorum — then the loss of one node becomes a routine event rather than an incident with root-cause analysis.
The first effect is reduced downtime. When fault tolerance is configured at the level of the sharding scheme and replicas, switchover to a standby node takes seconds, and client applications survive it through request retries.
Without such configuration, the same failure stretches into minutes of manual operations: find the failed node, bring it up, wait for data to catch up, and return traffic.
The second is the share of automatic switchovers. We describe leader election rules and quorum control so that the decision is made without an engineer: the cluster itself excludes the failed node and itself appoints a new one.
Only scenarios where automation is dangerous remain manual — for example, when there is a risk of data split-brain.
The third is updates without stopping the service. Rolling restart node by node allows rolling out a new configuration or current branch on a live cluster: while one node restarts and catches up with the others, traffic goes to the remaining ones. The downtime window is replaced by a slight reduction in load headroom.
These metrics are kept at the target level specifically by node state monitoring and timely support of the in-memory cluster.
Replication lag, in-memory data volume, quorum state, and response time for each node are collected into a single picture: you can see which node degrades before it fails, and memory does not hit the limit. It is on this data that the expansion plan is built — based on actual growth, not on gut feeling.
Fault-tolerance modes: comparison of options
Choosing a mode is a trade-off between recovery time after failure and operational complexity. Replication provides a second copy of data but does not answer the question of who is leader: quorum decides that.
Below are four schemes that are realistically chosen between in Tarantool administration, with their limitations.
Options and their limitations
| Option | When it fits | What limits it |
|---|---|---|
| Single node | Load is predictable, downtime during recovery from a copy is acceptable | No replication mode: node failure means data loss since the last copy |
| Leader with standby and manual switchover | A hot copy is needed and control over the switchover moment | No automation: an engineer switches, and the standby node does not accept writes |
| Cluster with automatic quorum-based leader election | The service must survive node failure without human involvement | Requires an odd number of nodes and a stable network: loss of quorum stops writes |
| Sharded scheme | Data and load do not fit on one node | Data placement scheme complicates queries: some operations go to multiple shards, and rebalancing requires a procedure |
The scheme is chosen based on acceptable downtime: the stricter the requirement, the more nodes and the higher the network requirements.
The chosen mode is fixed in the data placement scheme and switchover procedure — otherwise the very first update or node failure will force a revision of the decision.
Work order: from audit to handover to operation
Work proceeds in stages, and each ends with a verifiable result, not with a note "configured". This order is needed so that by handover to operation the cluster has already survived controlled failures on the test bench, not during working hours under load.
-
Cluster audit. We capture the topology: number of nodes and replicas, quorum, memory and disk log settings, places where latency grows. The output is a risk map: single points of failure, nodes without redundancy, configurations that have diverged between environments.
-
Cluster scheme design. We determine the number of replicas per shard, write quorum, placement of nodes across zones, and read mode. We fix the scheme before configuration, otherwise the topology is assembled "as it turned out".
-
Node and role configuration. We deploy configurations, assign roles, enable replication, monitoring, and backup with recovery verification.
-
Switchover testing. We take nodes out of service one by one and measure how long it takes to elect a new leader node and restore writes. Each scenario has a recorded result.
-
Load testing. We run a profile close to production: growth in the number of connections, write bursts, behavior when memory fills up.
-
Handover to operation. We train the on-call shift, hand over instructions for typical operations, the monitoring scheme, and the procedure for failure response.
We agree on each stage before the next one starts: the customer sees the scope and status of work as it progresses, not only in the final report.
Schemes, access policies, and switchover procedures
A cluster that relies on the memory of specific engineers breaks when they go on vacation. At acceptance, we hand over not only a working topology but also a set of artifacts that your staff and on-call shift can use to maintain the system without us.
- Node and link map — instance roles, placement of replicasets across failure zones, addresses and ports, who replicates with whom. With it, the on-call engineer understands within a minute what exactly failed and which services are affected.
- Access policies and roles — separate accounts for the application, monitoring, read-only diagnostics, and administrative operations. Routine checks do not require rights to modify data, and administrator actions are logged.
- Cluster documentation — space and index schemas, the procedure for applying schema migrations, configuration of each instance with an explanation of why the parameters are set exactly this way.
- Switchover procedure — conditions for moving the master to another node, who makes the decision, how replica lag is checked before switchover, and how rollback is performed if the new leader does not hold.
- Backup and recovery procedure — what is backed up and how often, where copies are stored, how regularly their suitability is checked, and how long it takes to bring the system up from scratch.
- Instructions for the on-call shift — step-by-step scenarios for typical failures: node not responding, replica lagging, out of memory for data. What to look at, what to do, and when to escalate.
Case study: an online store's cluster under sale load
Before the seasonal sale, the store ran on a single Tarantool node. The storage kept carts and order stock in memory but lived without redundancy: a restart or crash meant manual recovery, data reconciliation, and hours of downtime.
The very first traffic spike at the start of the promotion brought the node down — recovery stretched over several hours, and during all that time orders were not placed.
What was changed
We deployed a cluster: several instances with replication, even distribution of data across nodes, and automatic switchover on failure.
Now the loss of a node is not an event for the business: traffic moves to live instances within seconds, and data is restored from copies without manual actions.
Availability checks and recovery rules are configured so that under peak load the cluster does not go into cascading degradation.
Implementation results
Downtime when a node fails was reduced from hours to seconds — switchover happens automatically, without the on-call engineer's involvement. Failures under peak sale load stopped: the cluster withstands the seasonal spike and does not lose cart data.
Manual recovery has left regular practice, and the time to return to a working state no longer depends on which engineer is on shift.
How sharding differs from replication in practice
Replication copies the same data set to several nodes — each copy holds the entire volume. It covers node failure: when the master goes down, one of the replicas takes the leader role.
But it does not cure volume growth or writes hitting a single core — that requires another mechanism.
Sharding divides data by key between segments: each stores only its own part, and the cluster router routes the request to the required segment. Volume and throughput grow with the number of segments, but a single segment by itself remains a point of failure.
In practice, you do not choose one of the two schemes — you combine them. In Tarantool, the unit of scaling is not a node but a set of replicas: a segment is a master plus replicas connected by replication within the group.
Sharding distributes volume and writes across groups, replication within the group maintains availability when the master fails, so the cluster scales in both load and reliability at the same time.
Quorum is needed where node failure must not cost the last writes: the master confirms a transaction after waiting for a response from part of the replicas — switchover to a follower then loses nothing.
The price is write latency; if quorum is not reached, writes stop entirely, and that is a deliberate choice in favor of consistency.
Cluster scaling goes through adding a segment with migration of part of the key range. How evenly the load will fall on the new nodes depends on the sharding key chosen before the first write — changing it on a live cluster is almost impossible.
Technical doubts before starting cluster work
Before work starts, the questions are almost always the same: how a Tarantool cluster behaves when the leader node fails, what will happen during a scheduled update, and what will remain for the operations team after handover.
What happens if the leader node fails?
The remaining nodes will re-elect the leader automatically, and writes will continue after a short coordination pause — the application will see an increase in response time, not an error. We run failure and network partition scenarios on the test bench before going into operation.
Can a node be added and the cluster updated without stopping writes?
The update proceeds one node at a time: a node is taken out of service, returns and catches up with the state, then the next one is taken — quorum is preserved, and writes are not interrupted. A new node is included in traffic processing only after synchronization.
What to do if there is no cluster documentation?
We start by restoring the map: topology, node roles, placement of snapshots and logs, current access rules. The output is a scheme and a procedure for typical operations that can be handed over to a new engineer.
How is memory consumption controlled?
We set limits for the whole instance and for individual spaces, and enable eviction by record lifetime. This removes the main class of failures in in-memory systems — process termination when memory is exhausted.
Who is responsible for support after implementation?
We hand over the cluster with an on-call procedure, alert thresholds, and escalation order. After that — either support by our team, or your operations with our support according to an agreed response time.
Let's discuss the task: what is needed to assess your cluster
For the assessment of cluster work to rely on facts rather than assumptions, a few inputs are enough. They do not require exports and schemes at the start — usually a free-form description and a couple of configs are sufficient. Describe the task, and we will come back with an estimate and a work plan.
- Current scheme: number of nodes, router role layout, storage and queues, replication and quorum settings, whether sharding is used.
- Load profile: peak write and read operations, share of requests served from memory, whether there are heavy queries that slow down others.
- Data composition: how much memory is occupied, what is moved to disk, whether spaces with limited record retention are needed.
- Downtime requirements: acceptable unavailability window for scheduled work and expected behavior on node loss — preserve writes or read-only.
- Maintenance constraints: maintenance windows, prohibition of restarts during peak hours, server access procedure, and network restrictions.
- Current incidents: what has already happened — degradations under load, replica divergence, manual operations and recovery time after them.
- Target state: expected load growth, new services on the same cluster, backup and recovery verification requirements.
If part of the data is unavailable, this does not block the start: we record what is missing as assumptions in the work plan and resolve it during the survey stage.