Why the declared RTO diverges from actual downtime
The declared RTO is usually taken from a design document, where recovery time was calculated for a single service in isolation.
In a real failure, downtime is composed not of one operation but of a chain: diagnostics, decision to switch over, the switchover itself, warm-up, verification.
Measurements during drills almost always produce a figure several times greater than the design one, and the gap is explainable.
The first source of losses is unaccounted service dependencies.
A critical component is brought up in minutes, but it does not serve traffic: the message queue is unavailable, the cache does not respond, an internal name does not resolve, a secret from the vault has not been issued. Each dependency adds its own timeout to downtime, and during sequential recovery they add up.
The second is manual steps when switching to a standby circuit. While the operation runs as a script, it takes minutes; while an engineer performs it according to instructions, coordinating every action in the on-call chat, it takes tens of minutes.
Some access to the standby is requested via a ticket and unavailable at night. This is where most of the time is lost, not in the failure itself.
The third is untested standby. A circuit that was brought up only during commissioning diverges from the primary one: configurations, data schema, network policies, certificates.
Recovery does not follow a proven scenario but proceeds by trial and error, with manual verification of each step. Without regular drills, the standby remains a formality in documentation.
The gap between declared and actual RTO is closed not by a new regulation but by measurement: the failure scenario is played out in full, together with dependencies and switchover, and the real time is recorded. Then the most expensive steps are removed — first manual ones, then those amenable to automation.
What managed RTO provides: predictable service recovery
Managed RTO is when the target recovery time is set as a number and assigned to a specific class of failures, rather than formulated as a wish for “fast.” For each service, we fix the allowable downtime window, tie it to a failure scenario, and verify it during drills: until recovery time is measured under load, the figure in the regulation guarantees nothing.
The first effect is that entire classes of failures leave the “bring it up manually” mode.
The failure of an individual node is resolved by switching to a replica, loss of a zone by moving traffic to a live zone, data corruption by restoring a copy to a separate circuit with reconciliation.
An orchestrator like Kubernetes recreates containers on its own, but the real RTO rests on state: services with data run on replicas with automatic failover, and that is where it is decided whether we fit within the target window or not.
The second effect is predictability of recovery. Bring-up follows the regulation: steps are numbered, roles are assigned, the switchover decision point is known in advance.
We compare the actual time of each drill with the target, and the discrepancy becomes an input for refining the scheme, not just a line in a report.
The reduction of the downtime window is visible over time: the share of failures resolved without an on-call engineer grows, and recovery time stops depending on who happened to be on shift. Fault tolerance becomes a property of the architecture, not the luck of a particular night.
Protection modes: cold standby, hot standby, continuous replication
The protection mode is chosen based on two values: how long the system may be unavailable and how much data may be lost.
The difference between cold standby and continuous replication is a difference of multiples in recovery time and a qualitatively different amount of maintenance work.
Below are three basic options that are actually chosen between. They are not mutually exclusive: the critical core is usually kept on replication, while internal circuits are kept on cold standby.
| Mode | When it is suitable | Limitations |
|---|---|---|
| Cold standby | Internal systems, reporting, test circuits: downtime of several hours is not critical | Data is restored from copies, the standby is not deployed. Downtime is measured in hours, a proven regulation and regular recovery checks are needed |
| Hot standby | Client services where a downtime window of minutes is acceptable | Resources are deployed and synchronized, but idle and require updates. Standby and primary site configurations diverge without a single regulation |
| Continuous replication | Payments, orders, billing — a stop is critical, data loss is unacceptable | Replication lag control and protection against double writes are needed. Load switchover requires drills, otherwise the decision is made in panic |
The choice is dictated not only by acceptable downtime but also by how much engineering effort the team is willing to spend continuously.
A mode that has never been tested in drills does not work during a failure — that is why requirements for RTO and RPO immediately specify the frequency of switchover checks.
How to bring a system to the target RTO: stages of work
Target RTO is not a figure in a document but a verified property of the system. It appears when there is an accurate picture of the current state, a redundancy scheme reproducible under load, and people who have practiced switchover hands-on.
We run the project sequentially, and at each step the customer receives an artifact that shows progress.
- Infrastructure survey. We collect a map of services, dependencies, and failure points, examine what is already replicated and what runs as a single instance, where backups exist but have never been tested.
- Agreeing on RTO and RPO for each service. We divide systems into groups: for some, hours of downtime are acceptable; for others, minutes. We record the goals in writing, otherwise they cannot be verified.
- Designing the redundancy scheme. We determine the topology of the standby circuit, the replication mode, the order of automatic and manual switchover, and system behavior under partial degradation.
- Developing the recovery plan. Step-by-step procedures with checkpoints: what starts first, how data integrity is checked, at which step the rollback decision is made.
- Test switchover. We conduct drills on the production circuit: measure the actual recovery time, see where minutes are lost, and adjust the plan based on the results.
- Regulation and handover to operations. We describe roles, on-call duties, and escalation criteria, train the customer's team, and hand over documentation and access.
After handover, the scheme continues to live: switchovers are repeated on schedule, and the discrepancy between declared RTO and actual time is visible from the drill results.
A set of documentation and regulations for recovery
We assemble the documentation set so that the on-call shift can bring the service up without us: every regulation is tested in drills, not just on paper.
Everything described is stored in the customer's repository next to the configuration, so the document and reality do not diverge after the very first infrastructure change.
- Dependency map — what fails because of what: external services, queues, storages, shared network segments, and the order in which they are brought up. Without it, recovery turns into trial and error and consumes half of the RTO.
- Redundancy schemes — active and standby sites, replication modes, failure points, and which of them survive the loss of a single node and which do not.
- Switchover regulation — a step-by-step procedure with initiation criteria: who makes the decision, based on which metrics, what to do on a false trigger, and how to switch back.
- Access matrix — roles during an incident: who has switchover rights, who can only read logs, who approves routing changes. Sorting out authority in advance removes the “waiting for approval” pause at the worst possible moment.
- RTO and RPO profile by service — target recovery time and allowable data loss for each circuit: critical, internal, auxiliary.
- Configurations as code — descriptions of environments, routing rules, and backups in a version control system, with change history.
- Check log — drill dates, scenario, actual recovery time, and discrepancies from the plan.
A switchover regulation without drills is a hypothesis: by comparing the document with a real switchover, we correct the steps where the order turned out differently. The customer receives a set that can be handed over to a new team or an auditor.
Case study: reducing recovery time for a critical service
The order placement service relied on a replica of a managed database in the neighboring availability zone: fault tolerance existed, but switchover remained manual — an engineer pulled up the instructions, edited the connection string, and restarted the connection pool.
During the first real incident, recovery took 3 h 40 min with a declared RTO of 4 hours: formally within target, in reality orders were not being placed all that time, and some of them were later reconstructed manually from logs.
Measurements before and after
| Metric | Before | After |
|---|---|---|
| Service RTO | 3 h 40 min | 11 min |
| RPO (data loss) | up to 15 min | a few seconds |
| Manual steps | 12 according to instructions | launch of a proven scenario |
| Standby checks | once every six months | monthly on a copy of the production circuit |
| Replication lag monitoring | after an incident occurs | daily measurement |
What we changed: the switchover scenario was described as code, a check for replica readiness before traffic switching was added, and automatic return of the application to the new node was added.
Managed failover of PostgreSQL ceased to be a “from memory” procedure — it is launched with a single command and logs every stage with a timestamp.
Regular standby checks became mandatory: once a month, the scenario is run on a copy of the production circuit, actual RTO and RPO are compared with target values, and discrepancies are analyzed.
It is these measurements that turned the recovery case from a one-off success into a repeatable result — downtime was reduced by more than twenty times, and this is confirmed not by calculation but by measurements.
How RTO differs from RPO and what to standardize first
RTO and RPO measure different losses, and confusing them is costly at the design stage. RTO is how long a system may remain unavailable while the consequences for the process are still tolerable.
RPO is how much operational data may be lost, that is, how old a data copy is considered an acceptable recovery point.
It is logical to standardize RTO first: it defines the architecture. It determines whether a hot standby is needed, which components are duplicated, where the failure boundary lies, and how many switchover steps are automated.
RPO mainly affects replication frequency and log depth — that is, parameters within an already chosen scheme.
Both metrics are derived from business requirements, not assigned out of industry habit. First we describe the process that will stop and its sensitivity to downtime by the hour, then we assign the system to a criticality class: settlement circuit, warehouse, reporting.
For each class, RTO is calculated according to the worst-case scenario — together with failure detection, escalation, and the on-call decision to switch over, which often take more time than the procedure itself.
Next we refine RPO: with asynchronous replication, the latest operations remain only in the primary system, so some transactions will either have to be lost or manually reconstructed from logs.
This is a separate customer decision, and it is worth making it at the requirements stage, not during an incident.
Standardization ends not with figures in a document but with a verification procedure: a drill scenario, a success criterion, and the procedure for recording actual downtime and loss volume. Until the metrics are confirmed by drills, they remain a wish.
Technical concerns: replication, tests, and switchover
The questions when choosing a redundancy scheme are usually the same: where exactly data is lost during an incident, who decides on switchover, and how to avoid getting two independent masters after a communication failure.
These decisions are derived from RTO and RPO requirements, not chosen by taste, and are tested on a test bench before launch.
Below are the concerns that most often come up among the customer's engineers, and how we address them.
Will synchronous replication slow down writes?
Confirmation from the standby enters the write path, so response latency increases. The choice is not binary: a quorum of confirmations on several nodes, a semi-synchronous mode with one catch-up standby, or an asynchronous scheme with an allowable data loss window.
Zero RPO requires write confirmation from the standby; allowing the loss of a few seconds lets you keep the standby at a distance and avoid paying with latency.
Who switches the load, and how?
Automatic switchover is justified where recovery is needed within minutes: the decision is made by quorum, and the former master is fenced off from writes — otherwise we get two independent nodes and data divergence.
Where a false trigger is more expensive than downtime, switchover is performed by the on-call engineer according to the regulation: slower, but predictable.
How do we make sure the standby works?
Recovery tests are a regular procedure, not a one-off. We bring up a test bench from a copy, move traffic to the standby, measure time to readiness, and verify the count and checksums of records, the state of indexes and queues.
Separately, we monitor standby lag under peak load: if it grows, recovery will restore an outdated state.
What about data consistency after switchover?
After a forced switchover, the standby lags, and some transactions may not arrive.
In operations, this is addressed by reconciling key tables with the source, reapplying the change queue, and using application failback points to the standby — without these steps, divergence accumulates silently.
Let's discuss the task: we'll assemble a recovery scheme for your service
We suggest starting with a discussion of the task: to propose a recovery scheme rather than general recommendations, we need baseline data about the service. Send what you have — we will clarify some answers together with your engineer.
- list of services and their criticality: what stops first, what can wait, what must not be lost at all;
- target RTO and RPO by service group — the allowable downtime window and depth of data loss, agreed with the product owner, not set by eye;
- current scheme: where services are hosted, whether replicas exist, where backups are written, how often recovery is tested;
- dependencies: databases, queues, external integrations, and authentication systems — without them, the part brought up does not provide a working service;
- constraints: data retention regulations, maintenance windows, periods of minimum load;
- roles on the customer side: who confirms priorities, issues access, and accepts the drill results.
Then we proceed in the order of work: compare goals with the architecture, design backup and switchover, assemble the scheme in a test circuit, conduct drills with measurement of actual recovery time, and hand over documentation.
The output is an assessment and a plan with stages, responsible parties, and acceptance criteria for each metric.
Describe the task — we will come back with an assessment and a plan.