AWS EKS production: where the cluster breaks
A managed cluster in production breaks not at deployment time, but in the second or third month of operation: capacity runs out, traffic grows, someone tweaked the config manually once.
Such a failure is harder to diagnose than a pod crash — the failure manifests not as an error but as degradation: the API responds more slowly, some requests start failing, and a node silently drops out of the pool.
The first common spot is garbage collection on nodes. Container images and logs accumulate on disk; when the threshold is reached, the node starts evicting pods, and by the time of peak there is no free capacity left.
Persistent image cleanup rules and log rotation close this in advance, not at the moment of the incident.
Second is balancing. On every Ingress change, the controller rebuilds rules and health checks, and while the rebuild is in progress, some traffic bypasses.
New pods that have not yet warmed up receive requests on par with ready ones — hence the spike of errors exactly at rollout time.
Third and most expensive is configuration drift. Edits via kubectl, manually changed security groups, temporary limits: a month later, neither the manifest nor the cluster matches reality, and rollback to the previous state is impossible both in the cluster and in the repository.
A separate story is manually created secrets: when a namespace or nodes are recreated, pods start without environment variables, and this is discovered already in production. Next, we'll look at how these cluster bottlenecks are closed in AWS EKS production.
What setting up a production environment provides
The production environment in AWS EKS is assembled from managed node groups, an ingress traffic controller on ALB, a volume driver on EBS, and an external secrets store.
This combination removes manual edits: versions, resource limits, and network policies are described in manifests and applied identically in any environment, and the cluster state converges to the declared state without human involvement.
Recovery time and fault tolerance
A faulty replica is removed from balancing before it stops — readiness checks do not let traffic reach a warming-up pod, so the user does not see errors.
The managed group recreates a node from the template on its own, and distributing replicas across availability zones prevents the application from going down entirely if one of them is lost.
Recovery time is determined by the restart policy and application warm-up, not by the speed of the on-call engineer: a typical failure is resolved in minutes.
Updates without downtime
Node rollout happens through pod eviction. Disruption budgets guarantee to the scheduler that some replicas remain available, and new nodes are put into operation only after passing checks — traffic moves without dropping connections.
If the new version fails checks, the rollout stops on its own, and rollback returns the previous set of images.
A separate effect is the elimination of a class of failures caused by manual edits. Secrets are pulled from the store and do not sit in manifests, permissions are granted to service account roles, and changes go through a repository with history.
Incidents of "fixed it on the server and forgot" no longer occur, and metrics and logs are collected in one place, so the cause of a failure is found from data, not from the memory of participants.
Operating modes and cluster assembly options
The launch mode determines who is responsible for updating the OS on nodes, replacing them on failure, and how workloads are placed on machines.
A single cluster is usually assembled from several options at once: services with constant traffic are placed on managed groups, short tasks on serverless launch.
Choosing an operating mode is not about "faster or slower", but about the boundaries of the team's responsibility. The more the platform takes on, the less day-to-day work engineers have and the less freedom there is in configuring the OS, storage, and network add-ons.
| Option | When it fits | What it imposes on the team |
|---|---|---|
| Managed node groups | Constant traffic and predictable consumption | Node replacement and updates follow platform rules; custom settings are done through group configuration and a limited set of add-ons |
| Self-managed nodes | Non-standard OS image, special drivers, custom workload placement across machines | OS maintenance, updates, and security are entirely on the team; a custom process for replacing nodes without downtime is required |
| Serverless launch | Short tasks and rare bursts for which it is unprofitable to keep permanent machines | Restrictions on workload type and access to the OS; persistent volumes are not mounted directly; task startup takes time |
Mixing modes in one cluster is a workable practice: services with traffic live on managed groups, one-off tasks go to serverless launch, and self-managed nodes are left for what the platform cannot do.
We choose the mode before assembly: update schemes, permission distribution, and the monitoring model depend on it.
Implementation stages: from audit to handover
We carry out the work in sequential stages, and each ends with a verifiable artifact: a design, configuration, or test protocol. The customer sees at which step the project is and what they will get as a result.
- Infrastructure audit. We analyze the current state: how the cluster is structured, where secrets and volumes are stored, how metrics and logs are collected, and what failure points already exist. The output is a map of bottlenecks and priorities for fixes.
- Environment design. We design the target architecture: distribution across availability zones, namespaces for workloads, network design, load balancer, and secret rotation. We agree on the design before configuration begins.
- Environment setup. We describe the infrastructure as code: cluster, load balancer, persistent volumes, secrets store, node and pod autoscaling, resource limits. Secrets come from an external store rather than sitting in configuration.
- Monitoring and logs. We configure cluster, node, and application metrics, log collection, dashboards, and alerts for critical thresholds — a problem should be detected before a user notices it.
- Fault tolerance testing. We run load tests, simulate the failure of an availability zone and an individual node, pod restarts, and recovery from backups. We measure recovery time and record the result.
- Handover to operations. We provide the environment design, instructions, and change procedure with rollback order, and train the team. After that, support is by agreement.
Configurations, designs, and procedures as outputs
Work on the cluster ends not with running services, but with a set of artifacts by which the platform can be developed, handed over to a shift, and restored after a failure.
Everything configured is recorded in the repository and the customer's documentation: an engineer who has not previously worked on the project can bring up the environment according to the procedure, not from a colleague's memory.
- Configuration description: node pools and their purpose, initial sizes, autoscaling rules, what is defined declaratively and what is configured manually and why.
- Network design: virtual network boundaries, public and private subnet division, routing, traffic entry point, and request path to the container.
- Access policies: roles and service accounts, namespace bindings, permissions in the cluster and cloud services, and the procedure for granting and revoking access.
- Secret management: source of truth, method for delivering values to pods and updating them without restart, and an audit log of reads and rotations.
- Update procedure: order of node and cluster version updates, maintenance windows, readiness checks, and conditions for rolling back to the previous state.
- Incident response procedure: which alerts go to whom, which monitoring dashboards are used for diagnostics, and first-line steps during degradation.
- Documentation: instructions for common operations, acceptance checklist, owners for each environment, and escalation points.
Case study: how recovery time was reduced
The customer came with a working cluster on Amazon EKS that was maintained manually: nodes were added via the console, and secrets sat in manifests in plain text.
Monitoring was limited to CPU utilization, alerts went to a general chat without an owner — the team learned about a pod crash from users.
The first joint incident review showed the cost of that mode: recovery time stayed at 45–50 minutes, and service downtime during peak hours stretched to several hours.
Half the steps were done manually — from finding the current manifest to reattaching a volume, and the result depended on who was on call.
The cluster was moved to infrastructure as code: node configuration and autoscaling groups, an ingress controller for the load balancer with liveness checks, and a persistent disk driver.
Secrets were moved to an external store and mounted into pods as volumes — manifests kept references, not values.
Monitoring was rebuilt around symptoms: availability and latency metrics, an alert for target level violations, on-call with a checklist and ready rollback commands.
Backup snapshots and practice recoveries were made regular — the procedure stopped being tested for the first time in a real incident.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Recovery time | 45–50 min | 7–9 min |
| Manual operations in an incident | about 70% of steps | less than 15% |
| Service downtime per quarter | several hours | tens of minutes |
| Release rollback | manually per instructions | one command, about 3 min |
The environment retains the ability to roll back to any of the latest releases, and new services are brought up from a template, not by copying someone else's manifest.
How does a managed cluster differ from manual assembly?
In a managed cluster, the control plane is not your concern: the vendor maintains its availability, updates it, spreads replicas, and restores it after a failure.
Everything that lives above remains with the team: workloads, node pool composition and size, autoscaling rules, access rights, network policies, secrets, and observability. In this sense, Amazon EKS is renting the control layer, not the entire cluster.
With manual assembly, this is added to what the vendor usually covers itself: fault tolerance of control nodes, backups of cluster state, planned updates without downtime, and compatibility checks for storage add-ons and drivers.
Such options are compared not by "convenience", but by the share of engineering time for maintenance and by recovery time after a control layer failure.
Manual assembly is justified when the requirement is strict: an isolated environment with no external access, a non-standard network stack, a custom load balancing stack that the managed layer does not support, or a special kernel update policy.
Then the responsibility zone expands to the entire infrastructure, and maintenance becomes a separate role in the team, not a side task for the on-call engineer.
In practice, the boundary is documented: what the managed layer covers, what our team covers, and where the interface lies — node updates, secret rotation, response to degradation of the external load balancer and volumes, and incident analysis at the application level.
Without this design, both options produce the same result: it is unclear who fixes production when it goes down at night.
Therefore, the choice comes down to one question: are you ready to keep people on staff for whom cluster support is their main job. If not, a managed layer plus a transparent maintenance responsibility zone is cheaper to operate than the time saved on the control plane.
FAQ on cluster configuration
Before starting, the same doubts usually remain: how to pass secrets, how to update nodes without outages, how the cluster will coexist with an already operating environment, and how long it will take. Below are short answers; if your case is different, we will go over it separately.
How to pass secrets without keeping them in the repository?
Values are injected by an external store; only references to them and permissions tied to service accounts remain in the cluster. Access is separated by namespace, keys are encrypted in the cluster state, and rotation happens without restarting services. The repository contains descriptions, not passwords and tokens.
How to update nodes without downtime?
Updates proceed in waves: first new nodes are brought up, workloads move to them, and only then are old ones drained. Availability rules prevent evicting more than necessary, so the service continues to respond to requests.
Volumes are attached in advance — otherwise a pod waiting for a disk will not be able to move to another node.
Does the cluster integrate into an existing environment?
Yes. Networks are joined with the current ones, traffic comes through the load balancer according to the already accepted scheme, and certificates and domains are reused.
Logs and metrics go to the same observability environment as the other systems — the on-call engineer does not need a second window and a separate instruction.
What is the scope of work and where do we start?
First we discuss the task and the current environment — then we come back with a plan: stages, scope of work, and rollback points.
The base environment (cluster, network, secrets, observability) is usually assembled in one or two iterations; after that comes service migration and tuning under load.
Let's discuss your task and come back with a plan
An estimate for such a task is not done "by eye" — to calculate the scope of work, we need to see the actual state of the environment and the agreements around it. Below is what we request from the customer before starting.
A team that operates a working cluster has all of this, even if some information has to be gathered along the way.
- A role in the cloud account with limited read permissions — we do not request or use root keys.
- A design of the current environment: cluster, ingress load balancer, volumes, secrets store — or an honest "there is no design, everything was assembled manually".
- Load profile: peak values, daily rhythm, expected growth, and service criticality by group.
- Requirements for secrets and access: who receives sensitive values, whether there is an external secrets store, and how rotation is organized.
- Where to send monitoring alerts and who is on call: channel, trigger thresholds, and what is considered a reason to call at night.
- Change restrictions: maintenance windows, approval procedure, and services that must not be touched without warning.
- An engineer on your side who will confirm the details and accept the result.
We come back not with an abstract proposal, but with a work plan: stages with a result at each, target architecture design, list of changes in rollout order, acceptance criteria in measurable terms, and rollback procedure.
Dependencies and risks are a separate item — not a surprise in the middle of the project. Describe the task — we will come back with an estimate and a plan.