Flannel CNI in Kubernetes: why the overlay network breaks
Flannel as a Kubernetes CNI plugin solves one task: give each worker node its own subnet and connect these subnets over the physical network so that pods on different nodes communicate directly.
As long as routing and neighbor tables on all nodes are consistent, the cluster overlay network behaves predictably. Failures begin where state diverges: one node has already updated its subnet map, another still holds the previous one.
The first typical failure is loss of connectivity between nodes: each node sees only its own pods, and inter-pod traffic goes nowhere.
There are two causes: a stale subnet lease after node re-creation, and pod routing failure when rules in CNI chains are overwritten by another tool or reset when the runtime restarts.
The second class of failures manifests as instability: pod addresses change after restart. A freed address is returned to the pool with a delay, a new pod gets a different one, and neighboring nodes still hold the old record — some connections go along a nonexistent route. From the outside it looks like "sometimes works."
The third and most expensive case is connections hanging without obvious application errors. When packet size exceeds what the tunnel allows and fragmentation is prohibited, large requests are silently lost: service metrics are clean, the user waits for a response until timeout.
Such failures are not visible in application logs. Below we examine how we find the source and resolve it.
What changes after proper overlay network configuration
An overlay network is not "turned on" once — it is a set of parameters that determine cluster behavior under load: encapsulation mode, per-node subnet size, MTU consistency on all interfaces.
Flannel handles this predictably if the parameters are set deliberately rather than left at defaults.
New node join time
A fresh node receives a subnet from the shared pool and must synchronize routing tables before the scheduler places workloads on it.
With a properly sized pool and connectivity verification before commissioning, this takes seconds instead of minutes, and the node does not sit idle in a non-operational state due to a race during network layer initialization.
Packet loss and recurring failures
The main source of loss in an overlay is mismatched MTU: a packet leaves, gets fragmented or silently dropped, and the application sees a break on long requests.
After aligning MTU and configuring routing rules, the loss rate drops to fractions of a percent, and inter-service calls stop "floating" in response time.
Separately, the most expensive class of failures goes away: a node is formally alive, but some routes to other nodes diverge, and traffic goes nowhere.
Network diagnostics become measurable — interface state, dropped packet counters, reachability checks for each node pair instead of manual enumeration.
Options for organizing an overlay network: comparison of modes
The overlay mode determines how a packet leaves one node and arrives at another, and it is exactly what determines overhead, topology requirements, and the amount of manual work when adding a server. Below are the options actually chosen between when deploying Flannel in a cluster.
| Option | When it fits | Limitations |
|---|---|---|
| VXLAN | Nodes in different subnets and behind routers, cloud networks without shared L2 | Encapsulation consumes part of the payload: MTU is reduced, otherwise large packets are silently lost |
| host-gw | All nodes in one L2 network, throughput prioritized | Routes are built on each node; L2 segmentation or intermediate routing breaks the scheme |
| IPIP | Encapsulation with lower overhead than VXLAN is needed | IPv4 only, some NAT devices do not pass this protocol |
| WireGuard | Traffic between nodes in an untrusted environment must be encrypted | Keys and kernel module on each node, rotation, additional CPU load |
How to choose for your topology
If L2 is shared — direct routing gives maximum with minimum overhead; if nodes are distributed — overlay, and encryption is layered on as a separate mode.
The hybrid option automatically switches to direct routing for neighbors in a shared subnet and falls back to encapsulation for the rest, but it is harder to debug: some paths go one way, some another.
Order of network setup in Kubernetes: six steps
The order of work here is not accidental: the network is the only layer that cannot be restarted "live." An addressing scheme error manifests not during installation but weeks later, when the number of nodes and traffic volume between services grows.
Therefore we follow a fixed sequence for Flannel CNI setup — from inventory to handover to operations, with checks at each transition.
- Configuration audit. Collect node state, occupied address ranges, and routing rules so the overlay network does not overlap with office and cloud subnets.
- Addressing scheme. Choose the pod network range and per-node subnet size — it determines the maximum number of containers and headroom for cluster growth; plan service addresses as a separate range.
- Flannel deployment. Install CNI as a managed component, set the network mode and addressing parameters declaratively — the configuration can be reproduced on a clean cluster.
- Connectivity check. Run node-to-node and container-to-container scenarios across zones, separately verify that network policies do not allow extra traffic.
- Load testing. Apply growing traffic between services, monitor latency, packet loss, and behavior when one node fails, record the threshold after which the network degrades.
- Handover to operations. Provide the addressing scheme, configurations, new node join procedure, and a set of checks for the on-call shift.
This order gives the main thing: the cluster scales by adding a node, and a failure is localized to a specific layer rather than searched across the entire stack.
Configurations, network diagrams, and operational procedures
After handover, the network should not depend on one engineer's memory. We provide a set of artifacts by which the cluster is brought up from scratch, tested under load, and handed over to another team without losing context.
- Working configurations. Manifests and network plugin parameters (Flannel) for the control plane and agents on each node: encapsulation mode, packet size, state refresh intervals. Applied idempotently — re-running does not break a working cluster.
- Addressing scheme. Pod and service address ranges, their split across nodes and zones, reserve for future environments. Ranges occupied outside the cluster are marked separately so new ones do not overlap with them.
- Connectivity diagram. How nodes find each other, where traffic goes directly and where through intermediate routes, where egress to the external network is — with marking of the segments that will hit bandwidth limits first.
- Access rules. Network policies: who talks to whom and on which ports, what is isolated by default. A new service is connected according to the described scheme, not "like the neighbor."
- Diagnostics procedure. Step-by-step order for loss of connectivity: what to check on the node, in the control plane, which metrics and logs to examine first and by which signs the cause is localized.
- Change procedure. In what order parameters are edited and nodes updated so as not to drop traffic, and how to roll back to the previous set of configurations.
- Recovery instructions. Order of bringing up the network on a clean cluster and after node replacement, with checks that confirm connectivity, not just pod startup.
Case: unstable cluster network — what it was and what it became
Initial picture
A cluster of several dozen nodes served dozens of services, but connectivity between pods on different nodes periodically dropped: applications went into retries, some requests did not arrive, alerts came in waves.
The share of failed inter-service calls remained at several percent, and after pod rescheduling the network recovered in up to half an hour and only manually. To the customer it looked like "floating" short outages without a clear cause in the logs.
Causes stacked up: overlapping address ranges of node subnets, manual route editing when adding a new machine, lack of packet size control during traffic encapsulation, and zero observability into tunnels.
Flannel was present in the cluster but configured partially — hence the "flickering" network instead of a predictable channel.
What we did and what we got
Addressing was rebuilt: each node gets its own non-overlapping range from the shared pool, routes are formed by the CNI plugin itself when a node appears and leaves.
Encapsulation was configured with packet size checking so large service responses are not fragmented and lost along the way, and on top we added loss and retransmission metrics on tunnels with alerts for their growth.
The failed call rate dropped from several percent to fractions of a percent, recovery after rescheduling is automatic and within a minute.
Service downtime during scheduled node maintenance decreased from hours to minutes, and the class of failures "connectivity between pods on different nodes lost" stopped recurring.
How does direct routing mode differ from encapsulation?
Direct routing and encapsulation solve one task — connect pods on different nodes — but pay for it at different costs. With direct routing, a node writes routes to neighbors' subnets and forwards the packet as is: headers are not rewritten, no extra parsing layer appears.
With encapsulation, the pod frame is packed into an outer packet, delivered to the destination node, and unpacked there.
Hence the performance difference: direct mode saves CPU cycles on encapsulation and does not give up part of the packet to a service header — throughput and latency are better.
The cost is topology requirements: nodes must be in one Layer 2 segment or have routing between them, otherwise routes must be maintained manually on both nodes and network equipment.
Encapsulation, which Flannel implements via VXLAN, works over ordinary IP connectivity: nodes can be in different subnets, with different providers, in different clouds. This is its advantage and source of overhead — extra header, smaller MTU, more work with the connection table.
Topology dictates the choice, not preferences. One L2 segment and latency-sensitive services — direct mode; distributed sites and hybrid cloud — encapsulation, because maintaining routing between them is longer and more fragile.
Changing mode without downtime is possible, but the mode is set in cluster network configuration and must match on all nodes — mixing breaks connectivity.
Nodes are migrated one at a time: removed from load balancing, configuration changed, returned, and pod connectivity across different nodes checked before moving on.
Common technical doubts before network setup
Before changing the network layer, engineers ask the same questions: what to do with an already deployed overlay, whether services must be stopped, whether the network will withstand cluster growth and whether we will hit access restrictions. Below are short answers so you can compare them with your configuration before starting work.
Let's state upfront: the operating mode and address pools are chosen based on your topology, not the plugin defaults. Therefore we first capture a map of the current network, then propose a switching scheme.
We already have our own network overlay deployed — will it have to be removed?
Not necessarily. It is enough to separate pod address ranges: some nodes remain on the previous overlay, some move to Flannel. We change in waves — first a test pool and nodes without persistent state.
Will services have to be stopped during the switch?
Not across the entire cluster. A node is removed from load balancing, the network layer is updated on it, pods migrate, and the node is returned. Downtime affects only workloads without replicas.
What about cluster expansion and access policies?
We plan the address pool with headroom for several growth waves so new nodes join without reconfiguration. Network policies and access rules are migrated together with workloads and tested on a test bench.
Let's discuss the task: how we estimate the scope of work
Estimating cluster network work starts with discussing the task: we need a brief technical snapshot, usually one conversation and access to a description of the current state. Then we calculate the scope — the customer does not need to prepare a technical specification.
For calculation, the following is sufficient:
- topology: number of nodes, zones, managed cluster or self-managed;
- how the network is provisioned now and what exactly is unsatisfactory — connectivity failures, manual config edits, lack of addresses;
- requirements for isolation between namespaces and for external access;
- whether service downtime is acceptable or zero-downtime migration is needed;
- constraints: cloud provider, security policies, procedure for granting access.
In response we come back with an implementation plan, not an approximate range: order of work by stages, what is done at each, where maintenance windows are needed, and which checks confirm the result — pod connectivity between nodes, route preservation when a node restarts, absence of loss on the overlay network.
We fix conditions before start: scope boundaries, responsibilities of the parties, accesses, rollback procedure to the previous configuration, and acceptance point.
Then the scope does not grow during the project, and you see in advance at what point the Flannel cluster can be handed over to operations.