Managed Kubernetes in Google GKE: where the cluster breaks
Production cluster problems start not at launch, but at the third or fourth node update.
Engineers pull new versions selectively: a pool is left old somewhere, a driver does not match the kernel somewhere else, and somewhere the control plane has moved ahead and no longer supports lagging worker nodes. The update turns into a night operation with the risk that some workloads will not come up.
The second bottleneck is configuration drift. Changes go directly past the repository: an engineer edits an object spec on the live cluster, someone expands a service account's permissions or adds exceptions to network policies.
The environment stops being reproducible — deploying from scratch does not produce the same result as the working environment, and diagnostics hit the question of what is actually configured here.
The third is node failure without workload rescheduling. If replicas are pinned to a specific node and disruption budgets are not set, task eviction during node maintenance happens all at once: the service fails entirely, even though free resources still exist in the cluster. When changing a node pool, we get the same one-time dip instead of a gradual transfer.
All three scenarios are about the absence of declarative management, not about lack of capacity.
When configuring GKE, managed Kubernetes addresses them by describing node state in files, rolling out updates in waves within a specified maintenance window, and basing workload migration on disruption budgets and placement rules. Let us break down how this works and what is configured first.
What a managed cluster provides: time, resilience, downtime
A managed cluster removes from the team the work that does not create product value: node maintenance, updating control plane components, investigating incidents caused by diverged configurations.
The control plane is maintained by the provider — Google Kubernetes Engine — and it updates it itself, on its own schedule. Cluster fault tolerance no longer depends on whether the on-call engineer remembers the sequence of manual steps.
What changes in measurable terms
Automatic node scaling adjusts the number of nodes to actual load: the task queue grows — nodes are added; at the nightly traffic minimum — extra ones are removed.
The speed of delivering changes increases because deployment is done with a single artifact instead of manually assembling the environment on multiple servers.
Service recovery time is determined by the restart policy and replica distribution across zones, not by the on-call engineer's reaction speed.
A whole class of failures disappears along with manual administration: a forgotten configuration step, configuration drift between environments, "it worked in test, not in prod." Nodes are replaced automatically on failure, pod state is recreated from the manifest rather than restored from memory.
What remains with the team
A managed cluster does not eliminate engineering work — it shifts it. We define resource boundaries and quotas, configure readiness and liveness probes, distribute load across zones, and establish access and update policies. This part is responsible for ensuring scaling works on time, not after an incident.
Deployment options: managed, self-managed, hybrid
The difference between options is not in the level of "advancement" but in where the boundary of the provider's responsibility lies.
In a managed cluster, for example in GKE, it maintains the control plane, version updates, and zone availability; you are left with workloads, quotas, network policies, and access rights.
In a self-managed cluster, you administer both the state store and certificates, but you have full control over the kernel, network, and pod placement.
The choice is dictated by the workload: regulatory requirements for data placement, sensitivity to latency between services, and the availability of your own on-premises hardware.
Hybrid infrastructure appears where some systems must remain in the customer's environment, while the rest is more reliable and faster to keep in the cloud.
| Option | When it fits | Limitations |
|---|---|---|
| Managed cluster | No dedicated operations team, launch speed and predictable updates matter | Lock-in to one cloud, some kernel and network settings unavailable, versions set by the provider |
| Self-managed cluster on your own capacity | Strict data placement requirements, non-standard network or on-premises hardware | Updates, state backup, certificates, and on-call duty are entirely on you |
| Hybrid | Some services must remain in the environment, the rest is moved to the cloud | Harder to connect networks and distribute rights, need a single build and observability point |
For each option, we document in writing who is responsible for updates, availability, and recovery — this determines the real cost of ownership.
Implementation stages: from audit to handover to operations
Implementation begins not with deploying the cluster but with analyzing what will go into it.
We look at the service composition, load profile, availability requirements, and maintenance windows — the number and types of node pools, scaling policies, and resource headroom depend on this.
- Infrastructure audit — inventory of services and dependencies, measurement of traffic peaks, analysis of current bottlenecks. The output is a map: what we move into the cluster, what we leave outside, and why.
- Cluster design — availability mode, node pools for different profiles, resource requests and limits, pod and node autoscaling, eviction rules. Everything is described in the repository and versioned.
- Network and access setup — private environment, subnets and routes between services, traffic restrictions between pods, granting permissions to service accounts through cloud roles, separate environments for development and operations.
- Baseline environment — entry point, storage, metric and log collection, backup with recovery verification. Every component has health probes and predictable failure behavior.
- Load testing — running the real profile: peaks, node failure, zone loss. We see how probes and autoscaling perform, and adjust parameters based on facts, not estimates.
- Handover to operations — on-call regulations, alerts for meaningful metrics, rollback and recovery instructions, customer team training. We run alongside for the first releases, then the cluster is operated by the customer's team.
What you receive: configurations, diagrams, regulations
After handover, your team has not a set of screenshots but a description of the Managed Kubernetes cluster from which the environment can be recreated.
Configurations are stored in your repository under version control: changes go through review rather than being made manually through the control panel.
- Infrastructure as code: descriptions of the cluster, node pools, networks, and load balancers — the environment is reproduced from the same files in a new project.
- Infrastructure diagrams: connections between the cluster, zones, services, and storage with the failure boundary and points where zone loss does not bring down the service.
- Access policies: roles, separation of rights between teams and service accounts, integration with the corporate directory, network rules between services.
- Operations regulations: procedure for updating the cluster branch and nodes, version rollout, certificate rotation, actions during zone failure.
- Monitoring and alerts: dashboards for key metrics, trigger thresholds, notification routes, and diagnostics procedures for typical failures.
- Backup and recovery: what is saved, how often, where it is stored, and how it is deployed in a clean project.
- Documentation and handover: decisions made with rationale, limitations, known trade-offs, and walkthroughs of configurations with your engineers.
We agree on the package contents before the start — it is clear in advance what goes into your repository and what becomes part of on-call regulations.
Case study: migrating a service to a managed cluster
A service of 14 microservices lived on a fleet of virtual machines assembled manually and updated according to regulations.
Rolling out a new version took 40–60 minutes and only happened in the nightly window, with an on-call engineer at the console — half of the changes were postponed until the end of the week.
Nodes failed 3–4 times a month, most often due to memory leaks in neighboring services, and recovery stretched to 1.5–2 hours.
We migrated the workload to Google Kubernetes Engine: separated node pools by load profile, described services declaratively, enabled autoscaling based on actual resource consumption, added readiness probes and disruption budgets — planned node maintenance stopped bringing down the application.
We separately configured rolling updates with a delay between batches so an error had time to manifest before rollout to the entire cluster.
| Metric | Before | After |
|---|---|---|
| Version rollout | 40–60 min, nightly window | 6–8 min at any time |
| Rollback to previous version | 30–40 min by instructions | one operation, up to 2 min |
| Share of failed releases | ~18% | ~3% |
| Incidents caused by nodes | 3–4 per month | 1 per quarter |
| Service recovery | 1.5–2 hours | 4–5 min |
The previous version now lives in the same cluster and is available for rollback immediately, so nightly release duty stopped as a practice: no call-outs in a quarter.
The same workload fit on 40% fewer nodes — due to dense packing and correct limits, not due to "just in case" headroom. Let us break down how to reproduce this transition in your environment.
How does managed Kubernetes differ from self-managed?
Managed Kubernetes — for example, Google Kubernetes Engine — shifts the control layer to the provider: the API server, cluster state store, scheduler, and controllers run in its infrastructure, are replicated across availability zones, and are updated without your involvement.
The team is left with worker nodes, resource specs, network policies, and everything that lives inside the cluster.
The main difference is in the split of responsibility for the control layer. The provider maintains its availability and applies fixes, but is not responsible for the compatibility of your add-ons, permission separation, or correctness of specs.
In a self-managed cluster, this layer is entirely yours: state backup, certificate rotation, failure recovery, and upgrades require an on-call engineer and regulations.
Cluster updates are the most illustrative area. In the managed option, the provider offers stable versions and updates the control layer invisibly, but you choose the timing for nodes, compatibility checks, and rollout order yourself.
In a self-managed cluster, the upgrade is planned end to end — from change analysis to rollback if something goes wrong.
Practical conclusion: a managed cluster removes routine work but does not eliminate engineering discipline — versions, permissions, network, and observability remain the team's responsibility.
The self-managed option is justified when you need full control over the control layer or have strict placement requirements.
Common technical concerns before starting
Concerns before starting relate not to choosing a platform but to operations: who updates nodes, how not to grant excessive permissions, what happens under peak traffic, and whether it is possible to migrate without service downtime.
Below are short answers on each point, including questions most often asked before migrating to Google Kubernetes Engine; we go into details based on your design.
Who updates nodes and what about rollback?
The control plane is updated by the platform, nodes — within a specified maintenance window. We set the window during hours of minimum load, and the eviction policy gives the service time to finish connections and move to another node.
Configuration is stored as code, so rollback to a previous state is reproduced without manual reassembly.
How do you separate permissions between teams?
We grant permissions by role: a developer can see logs and restart services in their namespace but cannot change network policies or read another team's secrets. Access auditing is enabled — you can see who changed the configuration and when. This way one careless access does not open the entire environment.
Can a running cluster be migrated without downtime?
Migration goes through a parallel environment: we bring up a new cluster, service replicas run in both, and traffic is switched on the load balancer gradually.
Data is synchronized before the switch, and the old environment is kept as a hot standby until the new one runs steadily. Users do not notice the migration.
What happens when traffic grows?
The cluster scales based on metrics: when CPU load rises or the queue grows, nodes are added; when it falls, they are removed. Resource requests and limits for each service are set in advance so one service does not consume its neighbors' memory. Behavior under load is tested before switching production traffic.
Let's discuss the task: where we start and what we need from you
Assessing a managed Kubernetes project does not start with a full audit — it starts with your input data.
The more specific the description of the current design and constraints, the fewer assumptions in the architecture and the more accurate the stage-by-stage implementation plan. Below is the minimum that is usually enough to come back with a timeline estimate and scope of work.
- Current design: where services run today, how the network and balancing are structured, which external systems access the application.
- Load: peak number of concurrent users or requests, traffic profile — even or seasonal, sensitivity to response latency.
- Availability requirements: acceptable downtime, whether a standby node in another zone is needed, what is considered a disaster scenario.
- Constraints: security and regulatory requirements, data and secret storage rules, prohibitions on external services and the chosen deployment region.
- Change release process: how releases go live now, who is on call, which mandatory checks must pass before deployment.
- Integrations: image registry, monitoring and logging systems, queues, databases, task queues, and background processes.
- Team and access: who will maintain the cluster after launch, which roles and access levels need to be separated, which accounts you provide.
Describe the task — we will come back with a project estimate, implementation plan, and distribution of responsibility areas. If some data is not collected yet, that is no reason to wait: we close gaps during the design stage, documenting the decisions made.