Setting up Yandex Managed Kubernetes: where production breaks
Setting up Yandex Managed Kubernetes rarely breaks at the "cluster is up" stage. Problems begin later, when production traffic and the first incidents arrive.
The managed control plane removes API server and cluster state storage maintenance from the team, but it does not eliminate responsibility for worker nodes, permissions, and application configuration — that is exactly where the failure points live that are invisible in a test environment.
The first and most common is manual configuration edits.
Editing through the web interface during an outage closes the symptom but leaves no trace in the repository: the next recreated node or namespace returns the system to its previous state, and the gap between desired and actual accumulates. As a result, environments diverge, and the test environment stops predicting production behavior.
The second point is insufficient cluster resources. Without explicit requests and limits for containers and configured autoscaling, pods either crash due to memory or are evicted from nodes, and the waiting queue grows faster than machines are added.
Diagnostics are complicated by the fact that the root cause is usually not in the application code, but in how resources are distributed among workloads.
The third layer is access rights. Broad roles and shared cluster access for the whole team mean that anyone can change production, but no one can identify the author of a change.
Below we break down how this is closed: infrastructure as code, environment separation, and least-privilege access.
Measurable effects: rollout speed, downtime, failure class
Before automation, releasing a new version is a manual sequence: build the image, update configurations, apply manifests, verify behavior.
The rollout stretches into hours, is scheduled in a night window, and rollback is kept "just in case" as a separate scenario that no one has rehearsed.
After moving environments to declarative descriptions and a managed cluster, rollout is triggered by a commit and fits into 10–15 minutes, including checks.
Service recovery time is determined not by the speed of hands, but by how quickly the system notices a problem. Readiness and liveness probes remove an unhealthy instance from traffic, the cluster recreates it in a new state, and nodes are controlled at the platform level.
Unavailability is reduced from tens of minutes to single digits, and some incidents are resolved without an on-call engineer at all: the system reacts before anyone opens the monitoring dashboard.
The most expensive effect is an eliminated failure class, not acceleration. This refers to configuration drift between environments, a forgotten parameter, a manual edit "on the server" that makes the test environment behave differently from production.
When environments are built from a single description, these failures stop reproducing, rather than simply being fixed faster.
Update predictability is a consequence of the same. Updating the cluster and workloads stops being an event with an unknown outcome: changes follow one scheme, state is checked by metrics, and a failed release rolls back to the previous version.
Updates begin to be planned by calendar, rather than postponed until "there is no time for risk."
Deployment options: managed cluster, own nodes, hybrid
Choosing a mode is not about "more powerful or cheaper," but about which parts of the cluster you are ready to administer yourself.
A managed cluster takes on the control plane: its updates, fault tolerance across multiple availability zones, and recovery after a node failure.
Own nodes leave you the choice of image, kernel, kubelet parameters, and network settings — in exchange, you handle updates, scaling, and hardware replacement.
A hybrid scheme combines both approaches when part of the workload must run in your environment due to regulatory requirements or latency to internal systems.
The key criterion is where the boundary of your responsibility lies and how predictable the load is. Below are the real alternatives chosen between at project start.
| Option | When it fits | Limitations |
|---|---|---|
| Managed cluster | Standard services, variable load, need a quick start and autoscaling of node groups | Control plane is hidden — no access to its parameters; non-standard network scenarios are solved with workarounds |
| Own nodes on top of managed | Specific images, GPU profiles, local disks, or a custom runtime are required | You handle updates, configuration drift, and node replacement; higher operational overhead |
| Hybrid scheme | Part of the data must remain in your environment, part in the cloud | Need a unified network and policy layer, a separate access scheme, and monitoring on both sides |
| Fully own cluster | Strict regulations, isolated environment, own hardware fleet | Control plane, etcd, backup, updates — all on your team |
Practice: most services fit into a managed cluster with nodes divided into pools for different load profiles.
Own and hybrid schemes are justified where there is an external requirement, not an engineering preference — otherwise the team pays with time for what the platform already maintains itself.
Work stages: audit, design, setup, verification, handover
Work proceeds in iterations: each stage ends with a verifiable artifact, not a verbal "done." This way the customer sees which step the project is at and can stop or adjust course before changes reach the working environment.
- Infrastructure audit. We capture the current state: what was set up manually, where configurations have diverged from reality, which services depend on each other, which quotas are already at their limit. The output is a dependency map and a prioritized list of risks.
- Cluster design. We design node pools for different load classes, distribute them across availability zones, and define update and autoscaling policies. We agree on the design before creating resources, not after.
- Infrastructure as code. We move the cluster, networks, access rights, and secrets into declarative configurations. The environment starts deploying from the repository, and discrepancies between environments are caught during verification, not in production.
- Setup and launch. We bring up Yandex Managed Kubernetes, separate service account roles, connect metric and log collection, and set alerts for business-relevant thresholds. We verify every alert — it should fire for a reason, not call at night without cause.
- Load verification. We run a profile close to production and see where the system hits limits: node limits, the database, the queue. We separately test fault tolerance — take down a node and record how long the load moves over without losing requests.
- Handover to operations. We provide on-call regulations, rollback and recovery instructions, the monitoring scheme, and an analysis of typical failures. We train the team so that further changes can be made on their own.
After handover, one question remains — what exactly needs to be improved first. Describe the task, and we will come back with a work plan and estimate.
What remains with you: configurations, designs, policies, regulations
Handover does not end with issuing access to the cluster.
Everything we configured remains with you as artifacts that can be read, changed, and reproduced without our involvement: infrastructure is described as code, policies and access rules are recorded, and updates and recovery follow regulations, not the memory of a specific engineer.
- Infrastructure as code — repositories with the configuration of the cluster, node pools, networks, and rights: any change goes through review, and the environment is brought up from scratch with a single command.
- Access policies — a model of roles, service accounts, and namespace separation: it is clear who can do what, including contractor access and action audit.
- Backup scheme — what is copied (cluster state, object descriptions, data disks), at what frequency, how long it is stored, and how recovery is tested.
- Update regulations — order and windows for updating the control plane and worker nodes, safe workload eviction, and conditions for rollback to the previous version.
- Monitoring and alerts — dashboards, metric sets and thresholds, notification routes by on-call; configuration is also stored as code and versioned.
- Cluster documentation — environment diagram, map of services and their dependencies, list of failure points, and instructions for routine operations.
- Escalation procedure and responsibility matrix — who receives the signal, who has rights to change production, and how incidents are recorded.
We verify the handover in practice: we deploy the environment from scratch using your repositories and regulations, without connecting to previous settings. If there are no discrepancies, the team continues working independently, and we remain available for specific tasks.
Case study: how cluster recovery time was reduced
What it was
The client was a service with load that tripled during peak hours. The cluster lived on manual changes: configuration was edited directly in the control panel, infrastructure state was not recorded anywhere, and access was granted with a margin.
Recovery after a node failure or failed release took 4–6 hours — an engineer manually restored order step by step, checking their own notes.
In a quarter, the team handled 3–4 incidents, each taking the on-call engineer almost a full day.
Releases went out in night windows, one or two per week: no one dared touch the running infrastructure during the day, and everything rested on one person who knew how the cluster was built.
What was changed and what was achieved
The cluster on Yandex Managed Kubernetes was moved to an infrastructure-as-code description: nodes, groups, access roles, and service accounts are reproduced from the repository with a single command.
Rights were narrowed to the minimum necessary, changes go through review and are applied only through automation — no one goes into the control panel manually anymore.
Observability was added: node and application metrics, degradation alerts, and readiness checks before switching traffic to a new version.
Cluster recovery was reduced to 30–40 minutes, and the system does most of the work itself. Failure frequency dropped: instead of several incidents per quarter — isolated ones, and almost all are caught by alerts before users notice them.
Releases left the night window: several times a week, during the day, with rollback to the previous image in minutes.
The team's workload shifted from firefighting to product development. On-call stopped depending on one specialist: the configuration is described and verified, and a new engineer brings up the environment using documentation, not stories.
How is infrastructure as code better than manual configuration?
Manual configuration in the console leaves no traces: node group composition, role bindings, network rules, and autoscaling parameters live only in the current cluster state.
A month later, no one will answer why one node group has a different disk type and where an exception in an access rule came from. This is configuration drift — the discrepancy between what was intended and what actually runs.
Infrastructure as code moves the description into a repository: the same configuration brings up identical clusters in test and production environments.
Environment reproducibility stops depending on an engineer's memory — the environment is created by applying the description, not by recreating it from a checklist.
A change goes through configuration review: the diff shows exactly what is being added — a node group, network policy, service account role, monitoring rule. Rollback becomes a commit revert with re-application, not manual reverse operations, each of which is easy to forget.
Regular reconciliation works against console edits: the plan shows the discrepancy between the description and the cluster, and it is either moved into code or rolled back. Terraform stores state and on every run checks whether it matches the actual configuration.
The answer to the question is short: code gives reproducibility, a verifiable change history, and controlled rollback; manual configuration gives none of the three.
Addressing doubts: updates, availability zones, access rights
Technical doubts among engineering teams arise not at the design stage, but at the operations stage: who updates the cluster and how, what happens if a zone fails, and how strictly rights are separated. Below are answers to questions that come up at almost every Managed Kubernetes project start.
Will the cluster have to be recreated during an update?
No. We update the control plane, then node groups one by one: a node is removed from balancing, pods move to neighboring nodes, and the node returns already on the new version.
With correct readiness checks and replica distribution across zones, the user does not notice the transition. We define the update order and window in advance.
What happens if an availability zone fails entirely?
Nodes are distributed across at least three zones, and workloads have placement rules and replica spread defined. If one zone fails, some pods restart in the remaining ones, and the service remains available.
We separately calculate whether quotas and capacity reserves are sufficient to take on the load without degradation.
How are access rights separated?
Infrastructure rights and in-cluster rights are separated: a developer sees only their own namespace, an on-call engineer has read access to all but no access to secrets, and infrastructure changes go through cloud roles. Action audit is enabled, so it is visible who changed what.
What will be included in monitoring and alerts?
Node, pod, and control plane metrics, application logs, and alerts to the on-call engineer: service unavailability, error growth, resource exhaustion, autoscaling triggers. For each alert, we add a short instruction — what exactly to check first.
How are backup and recovery arranged?
We copy disks and cluster object descriptions on a schedule, and once a quarter we deploy a copy in a separate environment. Having a copy and confirmed recovery are different things, so we test the entire process, not just the fact that an archive was created.
If some answers diverge from your current configuration — let's discuss the task and come back with an estimate and migration plan.Let's discuss the task and come back with an estimate and work plan
To come back with a timeline estimate and implementation plan, we need to see the current state: how the cluster is deployed now, where Terraform state is stored, how access is arranged, and what is already collected in monitoring.
Describe this in free form — then we will clarify details on the network, node groups, and access policies.
At the output, you will receive concrete artifacts, not general promises:
- implementation plan by stages — from preparing the network and access to bringing the workload into the production environment;
- timeline estimate for each stage and scope of work with responsibility boundaries;
- architecture diagram: cluster, node groups, network zones, failure points;
- Terraform configurations and IAM policies in a form suitable for review by your team;
- a set of metrics, alerts, and monitoring dashboards with trigger thresholds;
- regulations for updating and rolling back cluster and application versions;
- knowledge transfer: configuration walkthrough with your engineers and operations documentation.
Describe the task — what already works, where the bottleneck is, and what result is needed. We will discuss it concretely and come back with an estimate and plan.