Why Yandex Compute Cloud runs unstably
Failures under load rarely look like a spectacular crash — more often it is degradation: response time grows, a queue of requests accumulates, and some users leave with an error. The cause is usually the same: virtual servers sit in a single availability zone and have no reserve.
While peak load is moderate, the scheme holds; but a seasonal spike brings it down entirely — together with the database and the balancer on the same node.
The second source is hand-built configuration. Yandex Compute Cloud provides ready-made placement groups, autoscaling and snapshots, but they have to be enabled and linked to each other.
If machines are edited point by point through the console, drift accumulates between environments: on test the behavior is one thing, in production another, and a successful experiment cannot be reproduced.
The third cause is the absence of limits and observability. Without resource quotas, a single process eats the node, and neighboring services degrade along with it.
Without metrics and load alerts, the team learns about a problem from users rather than from the system, and puts out the consequence instead of the cause.
We break down how this gets closed: distribution across zones, declarative infrastructure description, resource quotas and alerts with working thresholds.
What a properly designed infrastructure delivers
An infrastructure built from individual virtual machines in Yandex Compute Cloud delivers effects measured in minutes and fractions, not in impressions.
A new environment for a task or a stand for testing a hypothesis is provisioned within minutes: a ready-made image with the runtime is already assembled, and all that remains is to deploy a copy and supply the network parameters.
A developer gets a working environment the same day instead of waiting for someone to manually bring up a server and configure access.
Fault tolerance stops being a slogan when nodes are spread across availability zones and the balancer distributes traffic.
Health checks pull traffic off an unhealthy instance within seconds, so the failure of one node or an entire zone does not turn into service downtime — the user simply does not notice the switchover.
Recovery time also becomes measurable: state lives in replicas and snapshots, and configuration is described declaratively rather than kept in an administrator's head.
Returning after a failure means reapplying the same description, not manually rebuilding a server under pressure. The fewer manual steps, the lower the chance of making a mistake at the moment of an incident.
A separate result is the share of failures that no longer bring the service down entirely. A full disk, a crashed process or the degradation of one node remain an event in monitoring with a clear response, not an incident for the client.
Pre-defined thresholds, automatic restarts and backups turn some of the accidents into routine system operation.
Placement schemes in Compute Cloud: what to choose
A placement scheme defines not "the level of reliability in general" but the specific class of failure the system will survive without human intervention.
One availability zone, two or three zones, an autoscaling group or a standby server are different trade-offs between resilience, maintenance complexity and switchover predictability. We break down what each option covers and where it starts to get in the way.
In Yandex Compute Cloud these options are combined: the front end can live in an autoscaling group across several zones, while stateful services run on standby servers with a rehearsed switchover scenario. The key question is where the state lives and how quickly it recovers after a failure.
| Placement scheme | When it fits | Limitations |
|---|---|---|
| One availability zone | Pilots, internal services, test environments | A zone failure means downtime until manual migration; the reserve cannot be tested without a separate stand |
| Multiple zones with distribution | Production with a continuity requirement | More expensive in resources, more complex networking and balancing, data replication between zones is required |
| Autoscaling groups | Uneven load, web front end, task queues | Inertia when starting new machines, an external balancer and correct health checks are required |
| Standby servers | Databases, stateful services | The reserve does not take part in the load, switchover requires rehearsal and accounting for replica lag |
A working template usually looks like this: core services are distributed across two zones, autoscaling by load is layered on top of them, and data is held by a reserve with regular switchover testing.
Such a set survives a zone failure and peak growth without manual actions, and the only remaining manual operation — a planned switchover — takes minutes instead of hours.
Implementation stages: from audit to handover
We work in iterations: each stage ends with a verifiable artifact, not with "it got better". This order is needed so that changes in Yandex Compute Cloud can be rolled back at any step, not only after a full production switchover.
-
Infrastructure audit. We build an inventory: what is already deployed, what load each machine carries, where data and secrets are stored, what dependencies exist between services.
We separately record the failure points — usually a single availability zone, manual edits on servers, and backups nobody has ever tested.
-
Network design. We design segmentation: subnets per zone, security group rules, routing, moving the balancer and the service network into separate segments. The scheme is agreed before resources are created — redoing the topology on a live environment is more expensive.
-
Server configuration. We describe resources as code (Terraform) and the initial configuration as Ansible roles. Images are pinned, configs are versioned, changes go through review and are reproduced from scratch rather than "patched by hand".
-
Fault tolerance testing. We shut down one node at a time in each group and watch how traffic switches. We measure service recovery time and compare it against the stated requirements.
-
Backup. We set up a snapshot policy and verify the reverse path: we deploy a server from a copy on a separate environment and confirm the data is intact.
-
Monitoring. We connect metric collection and alerts on key thresholds — node unavailability, disk exhaustion, growing latency. Without this, operations start blind.
-
Handover and training. We hand over access, the scheme, documentation and incident reviews, and walk the customer's team through typical operations: updating, scaling, recovery.
Schemes, access and policies that stay with you
An infrastructure in Yandex Compute Cloud lives not in the web console but in a description: if it is not recorded anywhere, the next change turns into archaeology, and recovery turns into negotiations with whoever set it up.
That is why we hand over not only working servers but a set of artifacts that let the environment be reproduced from scratch.
What stays with your team:
- Infrastructure scheme: availability zones, subnets, routes, the balancer and external entry points — on a single page, not in an engineer's head.
- Resource description as configurations: virtual machines, disks, snapshots and templates live in version control, and changes are applied after review.
- Access policies: roles, account federation, service accounts, least privilege and the key rotation procedure.
- Backup policy: what goes into the copies, frequency, retention depth and how integrity is verified.
- Recovery policy: a step-by-step scenario with target recovery time and recovery point, including the failure of a separate zone.
- Monitoring and alerts: the list of metrics, trigger thresholds, owners and the escalation procedure.
- Instructions for the team: bring up a new node, roll out a release, roll back to a previous version, grant and revoke access.
We hand over every artifact in a format your team can edit without us: a change in the configuration, not a letter to a contractor. This is what distinguishes a manageable environment from dependence on a single engineer.
Case study: withstood load growth without rebuilding servers
The project is a service with day-to-day load variance: at peak hours there are three times as many requests as on average. The machines ran on fixed virtual instances in Yandex Compute Cloud, and every new unit of capacity appeared only after manual actions.
An engineer would bring up a replacement, add it to the balancing and only then pull traffic off the old machine. At peak this turned into a queue of tasks, and a planned update turned into a downtime window for the entire duration of the work.
What it was like before the changes
The share of successful requests held at 97%: errors were caught precisely during peak hours, when capacity was insufficient. Recovery after a failure took 40 minutes on average — until the problem was noticed, a replacement brought up and returned to service.
More than ten manual operations when a single machine failed, each with a risk of an error in the rush.
What was changed
The scheme was described as code: a group of virtual instances, autoscaling rules by load, health checks, binding to availability zones. Backup and state snapshots became regular and verifiable, and rollouts go through rotating machine replacement without service downtime.
What was achieved
The share of successful requests rose to 99.9% and no longer dips at peak. Recovery takes 2-3 minutes and happens automatically, without an on-call engineer. Manual operations when a node fails: zero, capacity is added automatically as traffic grows, not on request.
How do managed databases differ from databases on your own servers?
The difference between a managed database and a database on your own server is not in the engine itself but in where the boundary of responsibility runs. The data sits in the same storage, but the set of tasks the team covers differs fundamentally.
That set determines how many engineers you need on call and how quickly the database returns to service after a failure.
On your own servers you are responsible for the entire path: security updates, major version upgrades, replication, regular backups and restore testing, monitoring and response to node failure. Each item is a procedure that has to be maintained, otherwise it quietly stops being followed.
A managed service takes this part upon itself: updates are installed in a maintenance window, copies are taken on schedule, and point-in-time recovery is available without a separate stand.
What remains with the client is what requires domain knowledge: schema, indexes, access rights, optimization of heavy queries and control over data growth. The platform does the mechanical work; the decisions about how the data is structured are yours.
Own servers are justified when non-standard extensions, special requirements for data placement or file-system-level access are needed.
In that case we deploy databases on virtual servers of Yandex Compute Cloud and automate exactly what a managed service usually takes on: copy schedules, restore testing, replica switching. Full control, but the maintenance is entirely on us.
Scaling works the same way: in the managed variant it is changing the node class and adding read replicas; in your own, it is partitioning and manually redistributing data. We choose the approach by the required recovery mode and the volume of manual operations, not by habit.
What clients most often clarify before work starts
Before the start, customers almost always ask the same set of questions: what will happen to the service during migration, who gets the access, will the system withstand growth and will the application have to be rewritten.
We answer briefly — the full breakdown is done for a specific configuration in Yandex Compute Cloud.
Will the migration happen without downtime?
Yes, with a careful order of work. We deploy the target group of nodes in parallel with the current one, set up data replication and verify integrity.
We switch traffic on the balancer after control checks, and keep the old nodes in reserve during the observation period — rollback takes minutes.
Will the application need to be modified?
In most cases, no. Everything that depends on the environment — database addresses, connection parameters, paths to files — we move into variables and secrets.
Changes are needed only if the application stores state on a local disk or is designed for a single node: we look into that separately and propose a scheme.
How is access organized?
We grant access by roles, not "one key for everyone": a developer, an on-call engineer and an external contractor have different sets of rights.
Keys are stored in a secure vault, actions are logged, and on termination or a change of contractor, access is revoked with a single operation.
What happens as the load grows?
We combine nodes into a group with automatic scaling by metrics — request count, CPU utilization, queue length. We set thresholds with headroom so that a new node has time to come up before degradation. The database and storage scale separately, without stopping the service.
How is data restored?
Disk snapshots are taken on schedule and stored separately from the working infrastructure, so a failure in one place does not take the copies with it.
Once a quarter we test recovery on a test environment — what matters is not a log entry but a service actually brought up from a copy.
Still unsure about your scheme — describe the task, and we will come back with an estimate and a work plan.Let's discuss the task and come back with an estimate and a plan
To make the conversation concrete, not much is needed: access to the current configuration, an understanding of the critical loads, and a person who knows the history of changes. The rest is our job, including an inventory of what is deployed in Yandex Compute Cloud today.
The first step is a review of the placement scheme: machine roles, availability zones, network, backup, image update procedure.
After that we discuss the task in terms of the result and come back with an estimate of the work and an implementation plan where every stage has a deadline and a checkpoint.
What you get at the end:
- a placement scheme for virtual machines with roles and failure boundaries;
- resource sizing for peak load and recommendations on disk types;
- a backup policy and a tested recovery scenario;
- a description of the network, access rules and environment separation;
- a procedure for updating images and rolling back to the previous release;
- a staged plan for putting the system into operation, with work windows and owners;
- a documentation set and handover to support with training for your team.
Describe the task: what runs now, what gets in the way and what result you need. We will come back with an estimate and a plan, and the engineer who will run the project will join the conversation — not a manager with a presentation.