Setting up EKS on AWS: where the cluster loses resilience
An EKS cluster can be brought up in an evening: a node group, a load balancer, basic roles. Resilience does not appear from that — it only shows under load, when the cluster stops handling the peak, nodes fail, and services begin responding with delay.
Setting up EKS on AWS is not an installation but a set of decisions about how the system behaves in the worst-case scenario.
The first bottleneck is resource allocation. If containers do not have requests and limits set, the scheduler considers nodes empty and packs more pods onto them than they can handle.
At the first spike, memory eviction kicks in, pods restart in a wave, and the service enters a "started — crashed" cycle.
The second is networking. Pods take addresses from subnets, and with dense packing free addresses run out: new replicas hang in pending, and horizontal scaling no longer helps.
On top of this come delays between services — extra traffic hops between availability zones add to every call and accumulate in a chain of a dozen dependencies.
The third is node failure. Without eviction policies and spreading replicas across zones, the loss of one node group takes down the entire service, not just one of its replicas.
We explain how this is addressed: spread the load across zones, set resource boundaries, keep a reserve of addresses, and describe in advance how the cluster behaves when a zone is lost.
Measurable effect: resilience and recovery speed
A properly assembled cluster is evident not on handover day but during the first incident or planned update.
Therefore, we measure the effect in four values: recovery time after node failure, service downtime during updates, share of successful rollouts, and behavior at peak load.
Resilience and recovery
The fault tolerance of an Amazon EKS cluster relies on nodes being spread across availability zones and workloads being evicted from a node before it is stopped.
Rotation happens without losing requests: the application comes up on the remaining nodes with the required number of replicas.
Recovery time is reduced from hours of manual actions to a few minutes — recovery is performed by automation, not by an on-call engineer following a runbook.
Scaling and updates
Automatic scaling reacts to load and queue length, not only CPU utilization, so the peak is handled without a permanent reserve of nodes. Version updates proceed in waves with a limit on the number of unavailable replicas, and rollout without rollbacks becomes the norm. Service downtime during planned maintenance stops being an expected event.
What we check at acceptance:
- nodes are distributed across zones — losing one does not bring down the service;
- health and readiness checks are configured, unhealthy instances do not receive traffic;
- updates proceed in controlled waves with the ability to roll back the version;
- scaling triggers on load, not on a schedule;
- eviction policies prevent critical services from being evicted;
- incidents are investigated using metrics and logs, not guesses.
Deployment approaches: AWS-managed or manual assembly
The deployment mode is chosen along two axes: who is responsible for the cluster control plane and where the worker nodes physically live.
This determines how many availability zones can actually survive a failure and which operations the provider handles versus which remain with your team.
| Option | When it fits | Limitations |
|---|---|---|
| Managed cluster | No dedicated operations team, need a quick start and updates handled by the provider | Less control over versions and control plane settings, some parameters unavailable |
| Self-managed assembly | Strict isolation requirements, custom node images, non-standard network topologies, closed environment | Updates, control plane fault tolerance, and state backups are entirely on you |
| Hybrid scheme | Managed control plane plus your own worker nodes on your own machines | You update the nodes yourself and ensure network interconnection between environments |
In practice, the managed or hybrid option more often wins: worker nodes are spread across three availability zones, and losing one zone does not stop services — restarting the workload takes minutes, not hours.
Self-managed assembly is justified where isolation or regulatory requirements matter more than operational speed. In that case, we separately plan the effort for maintaining the control plane, updates, and state recovery.
Implementation stages: from audit to handover to support
Setting up an EKS cluster on AWS is a sequence of steps where each next step relies on the artifacts of the previous one. The order is not a formality: if access policies are designed before the network scheme, they will have to be redone along with the subnet breakdown.
- Infrastructure audit. What already exists: accounts, networks, image registry, running workloads, who gets access and how. The output is a list of constraints that the design must account for.
- Cluster design. Number of node pools, distribution across availability zones, autoscaling requirements. We agree on this before assembly: rework on a live cluster is more expensive.
- Network and access policies. Subnets, routing, roles, namespace separation, integration with corporate authentication. We grant rights on a least-privilege basis, not "to make absolutely sure it works".
- Cluster assembly. Control plane, network plugin, ingress gateway, persistent storage, admission policies. We describe everything as configuration, not console actions.
- Fault tolerance testing. We shut down nodes and entire zones one by one, observe workload rescheduling and recovery time. Separately — behavior when external dependencies fail.
- Observability and policies. Metrics, logs, alerts, monitoring dashboards for the on-call shift, runbooks for typical incidents.
- Handover to support. Access, documentation, walkthrough of a training incident, escalation procedure. After that, the cluster runs without our involvement.
Delivered artifacts: configurations, diagrams, policies
After handover, the customer's team retains not only a working environment but also a complete set of artifacts in your repositories: code, diagrams, and policies.
This is enough to bring up a copy of the environment, pass a security review, and change settings without the contractor.
- Infrastructure as code — cluster and related resources described in Terraform: modules by layer (network, cluster, nodes, access), state in remote storage with locking, changes through review and plan before apply.
- Cluster configuration — manifests and Helm charts of system components: node and pod autoscaling, ingress controller, metrics and log collection, quotas and limits for namespaces.
- Network diagrams — subnet addressing, separation of public and private zones, routing, internet egress, access rules to external services: a diagram plus a mapping table.
- Update policy — upgrade frequency for the control plane and nodes, maintenance windows, component compatibility checks, procedure for rolling back to the previous version, approval points.
- Access and secrets policies — roles and bindings, separation of team and service account permissions, key storage and rotation.
- Monitoring and alerts — dashboards, key metrics, trigger thresholds, notification routes, and indication of what the on-call engineer responds to.
- Documentation — runbooks for typical operations (node recovery, pool expansion, release rollout), architecture diagram, decision log, and acceptance checklist.
Case study: how the cluster stopped failing under peak loads
Initial picture
The Amazon EKS cluster was assembled through piecemeal changes via the management console: node groups of the same size, autoscaling only by average CPU utilization, readiness check thresholds set "by eye".
At peaks this produced predictable degradation — response time grew from 400 ms to 3 seconds, the share of requests with errors reached 4–5%, and new service replicas appeared 8–12 minutes after traffic began to grow.
The on-call engineer brought up nodes manually, so incidents lasted from 20 to 40 minutes. Updates took down too many replicas at once: a release during peak hours reliably led to short-term unavailability.
What was changed
The configuration was moved to a declarative description so that cluster state could be reproduced from the repository rather than restored from memory. Nodes were divided into profiles for different load patterns: services with a steady request flow and queue handlers.
Autoscaling was switched to queue length and response time signals, and disruption budgets were added so that planned updates and node consolidation would not take down working replicas all at once. Readiness checks received realistic timeouts instead of default values.
How results were measured
Before each major release, load tests were run on a copy of the environment with a peak traffic profile, and the same metrics were captured: error rate, response time at the 95th percentile, scaling delay, and recovery time after forced node failure. Comparison was made against a baseline run on the previous configuration.
Over the quarter, the peak failure rate dropped from 4–5% to 0.2%, response time from 3 seconds to 700 ms, replica addition accelerated from 12 minutes to 90 seconds, and service recovery took 4 minutes instead of 35. The number of incidents requiring manual intervention fell from nine to one.
What to do if the cluster is already deployed and running
An existing cluster almost never needs to be torn down — we start with an audit of the current state. We look at resource distribution across nodes, application limits and requests, update policies, access separation, and how the system behaves when one node fails.
The audit gives a direct answer: what can be fixed locally and what will have to be changed fundamentally.
If the framework is working, the task is closed by targeted improvements.
Usually this is enough: automatic scaling for the real load profile, aligning limits with actual consumption, separating service permissions, network policies, state backup, and verification that recovery from backup actually works.
Such changes do not require stopping services and produce a measurable result — the share of failures at peaks drops, and releases accelerate.
When it cannot be closed locally, we extend: allocate separate node pools for sensitive workloads, move state out, separate services by namespaces so that a heavy neighbor does not take resources from a critical one.
These are incremental steps — each is tested in a live environment and rolled back separately, without overall downtime.
Rebuilding without downtime is traffic switching, not shutdown. A new environment is brought up alongside, a real scenario is run on it, traffic is shifted in parts, and the old one is shut down only after metrics and checks confirm health.
As long as there is doubt about a specific service, traffic is returned to the previous environment — that is the safety net.
We choose the order of actions by the cost of a mistake and the reversibility of the step, not by the volume of work. This then turns into a plan: what we fix immediately, what we extend, and where a parallel environment is needed.
Technical questions: access, updates, node failure
Operational concerns about a cluster come down to a few topics: who can change what in it, what happens when an availability zone fails, how to update without stopping services, and how to keep resource consumption under control. Below are short answers on how we address each of these areas when setting up AWS EKS.
How to separate access and roles?
We describe permissions declaratively and bind them to groups in an external directory, not to individuals: an engineer gets access to their environment, a contractor — for the duration of the work.
We do not issue long-lived personal keys, and actions are recorded in the audit log with attribution to the user.
What happens when an availability zone fails?
Worker nodes are distributed across multiple zones, and placement rules prevent all service replicas from being collected on a single node. When a zone is lost, instances are rescheduled to the remaining ones — this is verified by drills, not just by configuration.
How to update without downtime?
First we update the control plane, then the worker nodes — one by one, with graceful eviction of instances and a disruption budget. Services are rolled out gradually, so part of the capacity always remains under load, and the previous version remains available for rollback.
How to control resource consumption?
Each service is assigned requests and limits, autoscaling works by metrics, not by schedule. Separate rules address disk growth and forgotten environments — storage volume stops growing uncontrollably.
How to migrate existing services?
We migrate in stages: remove dependencies and move state out, then shift part of the traffic to the cluster and compare metrics. While the indicators match, the traffic share grows — rollback to the previous scheme remains available at every step.
Let's discuss the task and come back with an estimate and work plan
The estimate for a cluster depends not on the number of nodes but on what already exists: how many services, how they get into deployment, where secrets are stored, and how network routes are set up.
To discuss the task substantively and come back with an estimate and work plan, rather than a range "from and to", we need inputs — these are what we use to assemble the scope of work.
- List of services and their dependencies: what moves into the cluster, what stays outside, who communicates with whom directly.
- Current manifests and deployment charts — they show the maturity of the configuration and the scope of migration.
- Network diagram: subnets, availability zones, environment isolation requirements, traffic entry points.
- Access model: how cloud accounts and service accounts are linked, who can do what in the cluster.
- Load profile: peak values, traffic patterns, scaling and recovery requirements.
- Constraints: security policies, audit requirements, mandatory checks before rollout.
- Target indicators: release deployment time, share of automated checks, acceptable downtime.
Next — review of the inputs, clarifying questions on controversial points, and a written plan: scope of work, stages, order of further steps, and what you get as a result. Tasks already handled within the team are marked separately and not included in the scope. Describe your task — we will come back with an estimate and work plan.