Failover AI agents: where production actually breaks
An agent in production breaks not when the model responds poorly, but when an external step does not respond at all. A request goes to OpenAI, the connection is open, there is no response — and the stuck agent holds the workflow until the timeout. While it hangs, the processing queue grows, and the user gets neither a result nor a clear error.
The second common place is a failure in the call chain. The agent calls a tool, the tool goes to the database, the database responds slowly; an error at one step is not visible from the outside, and the chain returns an empty or partial result.
This is more dangerous than an explicit error: the client considers the data correct and makes decisions based on it.
Third is an exhausted limit and queue degradation. The provider returns a quota rejection, retries go out simultaneously from all workers, and the load grows manyfold instead of decreasing. One such episode turns service degradation into complete failure.
Failover of AI agents begins with acknowledging: any external step may fail to respond.
Then come timeouts on every call, breaking the chain into isolated steps, automatic replacement with a backup executor, and rules by which the system returns an honest error instead of a hung response. We explain how this is addressed.
What changes after implementation: time, losses, failures
Before resilience was implemented, the user noticed the failure of an agent instance: the request did not go through, the response came back incomplete.
The on-call engineer saw the incident in monitoring a few minutes later, then came manual investigation, traffic switching, and restart. Recovery time in such a scheme is measured in tens of minutes, and every one of them adds to service downtime.
The failover scheme changes this in measurable terms.
A node that fails health checks is removed from load balancing, traffic goes to the backup, and unfinished tasks are replayed from the queue — recovery time is reduced to tens of seconds and no longer requires manual switching.
An orchestration platform such as Kubernetes recreates failed instances according to the desired state, so recovery proceeds the same way during working hours and at night.
Separately, the class of failures known as "silent degradation" goes away, when the process is alive but has stopped responding meaningfully. Previously such cases were found by indirect signs and investigated manually; now they are covered by the same set of checks. The share of manual operations during a failure drops to isolated, planned ones.
Overall, the availability of the system stops depending on which engineer is on shift and how quickly they react. This changes not only downtime numbers but also the team's operating mode: on-call shifts shift from firefighting to root-cause analysis and prevention.
Redundancy modes: active, hot, and cold
The choice of redundancy mode is determined not by habit, but by the cost of downtime and the amount of state the agent holds.
The difference between schemes is in what is already running and warmed up by the time of failure, and how long failover takes: switching traffic to the backup system.
| Mode | When it fits | Limitations |
|---|---|---|
| Active redundancy | Both systems accept requests; failure of one does not interrupt dialogues. For scenarios with strict continuity requirements | Requires idempotent operations and state coordination; more expensive to maintain, discrepancies between systems are difficult to investigate |
| Hot standby | The second system is running and warmed up but does not accept traffic. Switching takes seconds and covers most agent scenarios | Requires a degradation detector: standard readiness checks in the orchestrator (Kubernetes) see a crash but not a drop in response quality |
| Cold standby | Non-critical and batch tasks, scheduled reports, internal assistants | Readiness takes minutes; actual startup time is verified only through drills |
Combining modes is normal: keep the critical path on active redundancy, background tasks on cold. Hot replacement is justified where there is somewhere to shift traffic and something to measure quality with; otherwise the reserve remains unproven.
Separately, discuss behavior when both systems fail: an honest request queue with a clear response to the user is better than silently losing requests.
Failover schemes: implementation order for your system
Implementing failover is not installing a library, but a sequence of steps where each relies on the artifacts of the previous one. Skip the failure map — redundancy will cover the wrong failures; skip failure testing — switching will first trigger in production.
-
System audit and failure map. We inventory the nodes: model, external tools, queues, state storage. For each, we record the failure mode — timeout, error, quality degradation, unreliable response — and its cost to the scenario.
-
Choosing the failover scheme. Active redundancy, hot replica, route cascade, or degradation to a simpler model. The scheme is determined by the criticality of the scenario, not by ease of implementation.
-
Designing the switchover. Detection rules — timeout, response code, error rate over a window — and retry limits. Idempotency of calls so that a retry does not create a duplicate action in an external system.
-
Implementation and state externalization. Session state moves to shared storage: the backup route picks up the context instead of starting the dialogue over. At the same time, we limit the blast radius of a single node.
-
Failure testing. Drills on staging: we take down a node, cut the network, introduce delays, and return incorrect responses. We record whether the switchover worked, in what time, and whether actions completed without duplicates.
-
Observability. Switchover metrics, share of degraded responses, recovery time. Alerts — on trigger frequency, otherwise the noise of one-off failures will hide real degradation.
-
Commissioning and handover to on-call. Manual intervention procedure, on-call instructions, enhanced monitoring window, and criteria for rolling back to the previous scheme.
What we hand over: schemes, policies, procedures
We deliver not only a working system but also everything needed for its independent operation.
Without this, the very first agent failure turns into downtime until it is determined who should do what: automation triggered, but the on-call engineer does not know where to look and what to switch manually.
Therefore, together with the solution, a set of artifacts is handed over that the customer's team uses without us.
- System diagram — request flows, failure points, points where automatic switchover is enabled, and the logic for automatic agent replacement during degradation.
- Access policies — roles, service account boundaries, procedure for granting and revoking rights, separate access to logs and configuration.
- On-call procedure — who responds to a trigger, within what timeframes, what automation covers, and what remains for a human.
- Team instructions — step-by-step scenarios: manual traffic shifting, rollback to the previous configuration, returning an agent to the pool after a failure.
- Configuration documentation — thresholds, timeouts, number of retries, and an explanation of why these values were chosen.
- Pre-release checklists — what to check in routing and policies so that a change does not break switchover.
- Known failures log — which modes have already been observed, how they were mitigated, and which thresholds changed afterward.
The artifacts remain in your repository and are updated together with the system: when we change logic, in the same change we update the diagram and the procedure.
Case study: automatic replacement eliminated nighttime manual switchovers
The customer is a 24/7 service where assistants respond to customer inquiries and any pause in their work is immediately visible.
Before implementation, a nighttime agent failure meant manual on-call work: alert, diagnostics, switch to a backup agent, check responses, and only then sleep. The cycle took an average of 18–22 minutes, up to 40 on bad nights, and there were 3–4 such cases per month.
After implementing automatic replacement, switchover was no longer a human decision.
Agent health checks run every 15 seconds, three unsuccessful responses in a row take it out of rotation, a replacement is brought up from the warmed pool and immediately receives traffic, and the original returns to service after a series of successful checks.
The conditions are described declaratively, so behavior is the same for all agents, and metrics and switchover history are collected in Prometheus — in the morning you can see exactly what happened, without manual log analysis.
Metrics before and after
- Time to traffic switchover: 18–22 minutes → 40–60 seconds.
- Manual nighttime switchovers: 3–4 per month → zero over eight months of observation.
- Downtime per month: 60–90 minutes → single-digit minutes.
- Share of failures resolved automatically: about 95%, the rest are cases requiring a data-driven decision rather than switchover.
The reduction in downtime did not just produce a metric: the on-call engineer stopped being part of the nighttime process. An engineer gets involved when automatic replacement has exhausted its options and root-cause analysis is needed, rather than emergency service rescue.
What to do if the system is already deployed as a single instance?
If the system is already running as a single instance, you cannot stop it just for redundancy — users, not just monitoring, will notice the downtime.
Therefore, the first step is inventory: where the agent's state lives, what is stored in process memory, and what has already been externalized.
Everything that remains inside a single instance — dialogue context, task queue, limit counters — will have to be moved to shared storage. Until this is done, a second instance can be brought up, but it will not be able to take over the failed one's work.
Next, we align processing so that a repeated call does not break the result.
An agent that restarts a step after a break must either continue from the same point or safely complete the step again — otherwise, after switching to the backup, duplicate requests, emails, or records in the accounting system will appear.
This is also where liveness checks with different levels are configured: separately process unavailability, separately degradation of the external model, separately queue overflow. This avoids sending traffic to the backup because of a short network failure.
Backward compatibility keeps implementation running without downtime. External systems continue to call the same addresses and response formats — the only thing that changes is that there are now multiple instances behind the entry point.
We add new fields and do not remove old ones until the transition is complete, and we explicitly document differences in behavior versions so that the caller does not depend on which instance responded.
The migration plan itself is built in steps: first the backup is brought up in shadow mode and processes a copy of traffic, then a small share of real requests goes to it, and only after that does the switchover become complete.
A rollback point remains available at every step — until the backup has proven its behavior under real load, the primary instance is not taken out of service. This way we migrate without downtime and with predictable behavior during failure.
Technical concerns: frequently asked questions about fault tolerance
Most concerns about agent fault tolerance come down to one thing: how the system distinguishes a real failure from a brief slowdown and whether it switches earlier than needed. Below are answers to the questions engineers most often ask before launch.
How quickly does switchover happen, and what does the user see at that moment?
Health monitoring uses several independent signals: response availability, response time, and error rate over a sliding request window.
The switchover threshold is set not by a single failure but by a series of failed checks in a row, so the backup takes over within seconds and does not twitch on a random timeout. Unfinished requests are queued and retried on the backup; duplicate operations are excluded.
How do you deal with false positives?
The cause of false positives is usually one thing: the threshold is set based on a single signal.
We separate temporary network delays from failures of the agent itself, introduce different thresholds for different types of checks, and require failure confirmation from two observation points.
If a node responds slower than usual but correctly, there will be no switchover — only a note in the log.
What if the backup is also unavailable?
Then degradation triggers rather than failure: the agent responds according to a simplified scenario or returns a pre-prepared response, and the event is recorded with full context.
Handover to support happens automatically — with check history, node status, and recent requests, so the on-call engineer starts analysis rather than data collection.
Let's discuss the task: we'll review your system and come back with an estimate
To come back with a work plan and timeline estimate rather than general promises, we just need to understand your system: which agents are running, what has already failed, and what is critical not to lose when switching to the backup.
The more specific the initial data, the more accurate the estimate — by stages, not "for the project as a whole."
What you get from the task discussion:
- a review of the current system: where agents lose state when the model, node, or external integration fails;
- a fault-tolerant configuration design — backup executors, automatic replacement rules, degradation thresholds;
- a stage-by-stage work plan: implementation order, checkpoints, dependencies between steps;
- a timeline estimate and team composition for your load and availability requirements;
- a list of initial data and access needed to start the project;
- an operations procedure: how failures are recorded, who triggers switchover and by what signal;
- acceptance criteria — what we consider a successful recovery and what is grounds for rollback.
Describe the task in free form: where the agents live, which failure interferes the most, and what failure margin is needed.
We will review your system, identify the risks, and come back with a work plan, timeline estimate, and proposal for starting the project — then you decide whether it fits in terms of scope and priorities.
