Why Calico CNI fails as the cluster grows
Calico CNI is usually deployed on a cluster of a dozen nodes — and for a while it works unnoticed. Problems begin when the number of nodes increases and the topology stops being static: address pools are added, zones change, autoscaling is enabled. That is when connectivity between nodes turns into a separate operations task.
Calico builds subnet routes on top of BGP: each node advertises its own address block and learns neighboring blocks. If a session with a neighbor is re-established but the routing table is not recalculated, some traffic between nodes simply does not arrive.
From the outside this looks strange: the service responds from one node and is silent from another, while application logs are clean.
When the network scales, a new node gets its own address block, but the other nodes do not always learn about it in time. The same picture appears when the topology changes: migration between zones, pool replacement, node decommissioning.
Accumulated rules and stale routes cause mutual service unreachability, which is easy to blame on an application error.
The cost of cluster network failures is higher than it seems: degradation looks like a floating error rather than an outage, so diagnosis stretches into hours.
The engineer looks for the cause in the code, while the problem is in inconsistent routes and an incomplete neighbor table.
Below we break down how this is addressed: predictable network scaling, BGP session health control, subnet route recalculation, and monitoring of connectivity between nodes — before the failure manifests in production.
Managed cluster network: what changes in practice
Network policies as a single point of control
Previously, access between services was regulated by edits in manifests and host settings, and no one could answer the question “which connections are allowed now.” Calico network policies describe these connections declaratively and are applied to the entire cluster: the set of permissions lives in the repository, goes through review, and is rolled back together with the release.
A pre-deployment check catches an attempt to open an extra port, and the discrepancy between the declared and actual network state is visible in the diff rather than in incident analysis.
A single source of truth removes an entire class of failures — access that was once opened “temporarily” and forgotten to be closed.
Environment isolation and reducing the blast radius
We separate environments with policies and non-overlapping address spaces: test workloads cannot reach the production database, and a developer’s stand cannot reach someone else’s environment.
A node or service failure remains within its segment and does not drag along neighbors on the shared network: routes are rebuilt automatically, and recovery time is measured in minutes, not in the time it takes to find the culprit.
A managed network provides predictability of changes: a new connection is added by an explicit rule rather than by weakening control, and it does not surface as a side effect in another environment.
Calico CNI modes: encapsulation versus direct routing
The network mode in Calico is chosen based on topology: where the nodes are located and who manages routing between them. This determines the per-packet overhead and how much effort will go into incident diagnosis.
When traffic encapsulation is chosen
Encapsulation wraps a pod packet in an additional header, and to the network it looks like ordinary traffic between nodes.
It is used when nodes live in different subnets or zones and routing cannot be changed: connectivity appears immediately, without coordination with network engineers.
The cost is that the header consumes the useful packet size, so MTU on nodes and in Calico is coordinated manually: otherwise large responses are fragmented, and the application “sometimes works, sometimes hangs” on large files.
Direct routing and subnet advertisements between nodes
Here nodes themselves advertise their subnets to routers, and traffic to the pod goes without wrapping — full MTU and lower latency on the same hardware.
There are two constraints: a managed segment with BGP support is required, and there must be headroom for the number of routes on switches. An error in advertisements breaks connectivity immediately, so the configuration is checked before rollout.
| Network operation option | When it fits | Limitations |
|---|---|---|
| Encapsulation (IP-in-IP, VXLAN) | nodes in different subnets and zones, routing cannot be changed | additional header, manual MTU coordination, lower network performance |
| Direct routing with BGP advertisements | managed segment, BGP to routers is available | requires access to network configuration, route limit, higher operational complexity |
| Mixed mode by zone | some nodes are in the managed network, some are not | two models at once, more difficult diagnosis and documentation |
Implementation stages: from topology analysis to handover of regulations
Work proceeds from inventory to documentation handover: we record what exists, design the target scheme, change the live environment, and after checks hand over the regulations.
Each stage ends with an artifact that can be verified, not with the phrase “everything has been configured.”
Addressing scheme and a migration plan without downtime
- Network audit. Inventory of node subnets, pod and service address pools, routes, and external network exits. The result is a map of intersections and a list of address conflicts that would otherwise surface during scaling.
- Addressing scheme. We assign blocks to zones and node roles, leaving headroom for growth. We coordinate with the network team before changes so we do not have to redo them under live load.
- Configuration. We bring up Calico, switch network policies to default-deny mode, and describe permissions by service. We establish BGP sessions to upstream routers.
- Migration without downtime. We migrate nodes in waves: test environment first, then one node per group, checking service availability after each wave. Rollback is possible at any step.
Policy verification and degradation scenarios
- Policy verification. We run traffic between all pairs of services: what should be closed is closed, what should pass passes. We separately review rule updates and node unavailability.
- Degradation scenarios. We simulate node failure, route loss, and session breakage — policies must not block recovery. Based on the results, we refine the on-call procedure.
- Handover of regulations. We hand over the addressing scheme, configurations, policy descriptions, and instructions for typical changes. After that, the customer’s team edits the rules themselves.
What the customer’s team has after completion
After the work is handed over, the customer’s team has not a set of files but a working network management system.
Policies, topology, and access are described so that a new engineer enters the environment without verbal explanations and does not break connections between services on their first change.
Configurations and topology descriptions
- Calico CNI network policy manifests in version control, arranged by environment and namespace, with a single naming rule.
- Topology diagram: address pools, routing sessions, nodes, and external traffic exit points — indicating which segment is responsible for what.
- Service relationship map: which applications exchange traffic and in which direction, where default-deny is enabled.
- Diagnosis instructions: how to distinguish a policy block from route loss and what to check in the state of network agents.
Change regulations and access matrix
- Access matrix: which teams and services access what, who is allowed to change rules, and in what order.
- Change regulations: edits through a branch and testing on the stand, a mandatory rollback plan, and a change window.
- Decision log: why exactly this isolation model was chosen and what trade-offs were accepted.
- Checklist for connecting a new service: the minimum set of rules without which it will not enter the network.
- Procedure for rotating access when an engineer leaves or the contractor changes.
Every artifact is verified in the live environment: before handover, we run typical changes through the documentation together with your engineer. Describe the task — we will return with an estimate and a plan.
Case study: how the cluster network stopped being a bottleneck
The cluster grew to several dozen nodes and three environments, while the network still ran on manual routes: an engineer added static entries between subnets and edited access lists in different places.
Connecting a new team or deploying an environment turned into night work with the risk of affecting a neighboring environment, and the rules themselves gradually diverged between stands.
Before: manual routes and diverging access rules
There was no single description of which services were accessible to whom, so pre-change verification took hours, and rollback was done “from memory.” A monitoring and event log extract for the quarter before implementation: recovery after a network failure — about 40 minutes, packet loss share during a new application version rollout — up to 1.8%, network incidents requiring manual intervention — 3–4 times per month.
Metrics after: recovery time and packet loss share
After moving the network to Calico, access and routing are described declaratively: policies live next to the service code, and routes between nodes are distributed automatically.
According to the same extract for the following quarter: recovery — 4–6 minutes, packet loss share during rollout — below 0.05%, network failures requiring manual intervention — zero.
Divergence of rules between environments became impossible: stands are assembled from a single set of descriptions.
What to do if another network plugin is already deployed in the cluster
You cannot switch the network plugin in a running cluster with a single command: each CNI owns IPAM, routes on nodes, and rule chains in the kernel.
Switching “live” breaks pod connectivity within seconds, so there are only two workable paths — planned node recreation or managed coexistence of solutions.
Coexistence is possible, but only with separation of responsibility zones: one plugin holds addressing and routes, while the second — for example, Calico CNI — connects in network-policy-only mode and does not touch IPAM.
Then the new pool gets a separate segment, and workloads migrate to it in groups, without a shared downtime window.
A phased migration of access rules begins with inventory: we collect policy manifests and compare semantics — default-deny, permissions, egress, selection by labels and subnets.
Then a pilot on one namespace or node group: rules are duplicated, and actual traffic is verified against rejection logs. We migrate services in groups, preserving rollback until statistics on both sides match.
The selection criterion is simple. If you need BGP route exchange with equipment outside the cluster, a single source of policies, and predictable behavior as you grow — we change the plugin.
If the task comes down to segmentation inside the cluster and standard network policies are sufficient, it is cheaper to keep the current CNI than to maintain two controllers.
We separately check compatibility: the required Kubernetes branch, cloud load balancer, and interaction with kube-proxy replacement mode.
Technical doubts: compatibility, rollback, diagnostics
Network policies and routing are the part of the cluster where an error is visible not through a build but through silence: some connections simply stop passing, and it is impossible to find the culprit in application logs. That is why all our changes go through a pre-prepared rollback, not “we’ll fix it live.”
Below are questions engineers ask before work starts, and our substantive answers.
How do you roll back changes if policies have blocked traffic?
Calico configuration is declarative: it is stored in version control, and the state before the edit is captured by a dump of rules and routes.
Rollback is applying the previous revision; the state is restored to the original within seconds, without recreating pods or losing connections.
How do you diagnose traffic loss?
We follow the packet path: routes on nodes, dropped packet counters per rule, rejected connection log, latency and successful response rate metrics. Each failure gets a specific culprit rule, not a version of “the network is noisy.”
Is this compatible with the current cluster?
We look at the address allocation method, the route propagation mode, and the presence of other network layers. We perform the migration step by step — by nodes and namespaces, with connectivity checks after each step, so production services do not stand idle.
How do you keep changes predictable?
Edits go through the same path: plan, apply on the test environment, then on production with observation of dropped packet and latency graphs. If metrics go beyond the baseline — rollback and root cause analysis.
Let’s discuss the task: what we need from you for an accurate estimate
The fewer inputs there are, the more assumptions go into the estimate — so we start with a short review of your infrastructure.
Send what you have at hand, even in draft form: based on this data, we will return with a work estimate and a stage-by-stage implementation plan, without a “rough guess range.”
- Topology: how many clusters there are and where they live, what node and pod subnets are, whether addressing overlaps with the corporate network.
- Routing scheme: whether you work through an overlay network or directly, whether BGP is needed, whether there are upstream routers and agreed autonomous system numbers.
- Isolation requirements: which namespaces and services we close, which connections between them must remain, whether access from external networks is needed.
- Current state: how Calico is configured now, what was changed manually on top of the configuration, where manifests and policies are stored.
- External perimeter: whether there is a firewall at the boundary and whether its rules need to be synchronized with the cluster’s network policies.
- Constraints: change windows, whether pod restarts are allowed, requirements for rollback and for preserving traffic during work.
- Observability: what you use to collect metrics and logs, where you output them, and by what signs you will understand that routing and policies are operating normally.
- Priorities: what is more important at the first stage — segmentation, routing resilience, or throughput.
Describe the task to the extent it currently exists — we will ask clarifying questions ourselves. As a result, you will receive the work sequence, the scope of changes, and the points where coordination on your side will be required.