Why Weave Net in Kubernetes loses packets during scaling
On ten to fifteen nodes, the cluster behaves predictably, but after crossing a few dozen, services start to "flicker".
Some TCP sessions break off midway, individual SYNs do not reach the pod, inter-service calls fail by timeout — while node metrics are clean and the pods themselves are alive.
Packet loss in the cluster looks like random noise: it does not reproduce on all node pairs and not every time.
The cause usually consists of three layers. The first is encapsulation: the overlay wraps traffic in its own header, the effective interface MTU drops, and large packets with the don't-fragment flag are silently lost on the tunnel.
The second is topology exchange: as the number of nodes grows, agents take longer to converge to a consistent routing picture, and during the convergence window some paths lead nowhere.
The third is NetworkPolicy with an incorrect selector: traffic is dropped without a single entry in the application logs.
Overlay connectivity failures are unpleasant because cluster network diagnostics run up against the lack of a diagram.
An engineer sees truncated packets on one node but does not understand exactly where the path is lost: at the pod egress, on the tunnel between nodes, or already at the receiver.
There is no map of subnets, pod and service addressing, no list of overlapping ranges — and the investigation turns into trial and error directly on a live cluster.
That is why we start not with editing the configuration, but with restoring the network picture — we break down how this is resolved.
What the cluster overlay network delivers: measurable effect
The cluster overlay network defines two measurable parameters: node recovery time and the share of lost requests during route rebuilding.
While the network is assembled from disparate settings, each new node or pod rescheduling produces a spike in errors — losses are counted in percentages, not tenths of a percent.
We deploy Weave Net as a single network layer: addressing, routing between nodes, and access rules live in one configuration rather than in three different places. We take measurements on a test environment before and after implementation — and build our conclusions on them.
Encryption of traffic between nodes without speed loss
Traffic encryption between nodes is enabled at the network level: the application does not know about it and requires no modifications.
Before enabling it, we run a load scenario and compare throughput — with correct settings, the degradation remains within measurement error, and traffic between nodes no longer travels in plaintext.
Predictable recovery after node failure
Node failure stops being an event with an uncertain outcome: routes are rebuilt in seconds, pods move to live nodes, and the share of lost requests remains in tenths of a percent.
At the same time, a whole class of failures goes away — when after a restart some traffic goes nowhere and the service is formally up but responds with errors.
We verify this not in words: we shut down a node under load, look at recovery time and losses, and separately make sure that access rules do not let through extra traffic. If the numbers differ from expected, we fix the settings, not the report.
Overlay network operating modes: comparison of options
The choice of mode determines how packets travel between nodes and how many resources are spent on each new node. Overlay on top of the node network encapsulates traffic, direct mode gives routing to the kernel, and hybrid combines both. Below is a comparison along two axes: when the option is suitable and what it limits.
| Option | When it is suitable | Limitations |
|---|---|---|
| Overlay on top of the node network | A cluster in the cloud or in several subnets where a direct route between nodes is unavailable | Encapsulation takes part of the bandwidth, the maximum packet size is lower — large packets are fragmented |
| Direct routable mode | A single connectivity domain or a managed network where nodes see each other directly | Depends on network policy: changing the subnet or filters breaks routes between nodes |
| Hybrid | Mixed infrastructure: some nodes in the cloud, some on own hardware | Configuration is more complex, failures must be labeled separately for each subnet |
When the overlay is justified
The overlay is enabled when nodes live in different subnets and the network does not allow advertising routes between them.
In this mode, Weave Net provides connectivity and traffic encryption, and network policies are described the same way for any option — changing the mode does not require rewriting the rules from scratch.
Limitations under real load
On streaming workloads, encapsulation is noticeable in response time, so the database and event bus are more often moved to direct mode. But an error in routes there leads to some nodes no longer seeing each other.
Therefore, we run both profiles on a test environment with production-like traffic before switching.
Deployment procedure: from network audit to handover
Deployment begins not with installation but with an audit of the cluster network: which address ranges are in use, how routing between nodes is arranged, and where trust boundaries lie.
An error here surfaces later — some pods cannot see services, and diagnostics stretch over days. Therefore, the six stages follow strictly in order, and each ends with a verifiable result.
What we check at the input
We assemble a cluster map: used and free ranges, intersections with office and cloud site networks, and current access rules between services. We record which namespaces remain isolated and who has the rights to change the network configuration.
- Cluster network audit — inventory of nodes, ranges, and routes, search for intersections with existing infrastructure.
- Addressing scheme — address pool for pods and services, plan for namespaces.
- Installation and encryption setup — deployment of Weave Net in the cluster, enabling traffic encryption between nodes, key rotation procedure.
- Access policies — allowed connections between services and default deny for sensitive namespaces.
- Connectivity testing under load — tests between nodes and sites, behavior during node failure, latency measurement.
- Team training and handover — review of configurations, diagrams, and diagnostic scenarios.
Readiness criteria for handover
Handover is possible when traffic between nodes is encrypted, policies are applied to all required services, and node recovery has been verified in practice. The team is left with an addressing scheme, instructions for typical failures, and a change procedure.
Handover package: configurations, diagrams, procedures
A working cluster and a cluster that can be handed over for maintenance are not the same thing.
Therefore, handover is not a demonstration of running containers, but a set of documentation by which your team can deploy the environment from scratch, expand the network, and troubleshoot a failure without us.
What is included in the package
- Network configurations: installation manifests, overlay parameters, and values moved into separate files — the environment deployment is reproduced one-to-one.
- Addressing map: pod and service ranges, node addresses, used subnets — when expanding the cluster, we do not conflict with the corporate network.
- Access policies: which namespaces and services are accessible to whom, on which ports, and which traffic is encrypted between nodes.
- Traffic path diagram: the packet path from pod to pod, encryption points, and nodes where diagnostics are taken.
- Maintenance procedure: planned node updates, the procedure for taking a node out for maintenance, and encryption key rotation.
- Failure response procedure: scenarios for loss of communication between nodes, overlay degradation, and node loss — with verification commands at each step.
- Acceptance checklist: a set of checks before each release so that a configuration change does not break connectivity.
Procedures for typical incidents
We separately record who does what during a failure: where to look at metrics, how to isolate the problematic node, and at what point to roll back the configuration to the previous version.
The maintenance procedure covers planned work, and the failure response procedure covers unplanned work, so the on-duty engineer does not look for a solution from scratch.
Case study: cluster network under load before and after
Initial state and limitations
A cluster of several dozen nodes was assembled in stages, and the network could not keep up: some traffic went directly over L2, and some through a custom overlay layer.
Encryption between nodes was disabled, access separation relied on subnets and manual firewall rules, and MTU desynchronization on different interfaces caused packet loss.
During peak hours, connectivity degraded: services reconnected, some requests ended in timeouts, and the on-duty engineer could not quickly separate a network problem from an application problem.
Result in metrics before and after
We deployed Weave Net with traffic encryption between nodes, aligned MTU on all interfaces, and moved segmentation into declarative access policies. Additionally, we exposed connectivity metrics so that degradation is visible before users notice it.
| Metric | Before | After |
|---|---|---|
| Packet loss between nodes during peak hours | 8-12% | 0.1% |
| Connectivity incident resolution | about 5 hours | about 35 minutes |
| Incidents without a clear cause | 40% | 5% |
| Adding a new node to the network | 40 minutes manually | 3 minutes automatically |
Connectivity stopped being a source of random failures, and the troubleshooting procedure came down to three steps: overlay metrics, the state of channels between nodes, and policy triggering.
On-duty engineers stopped escalating network issues at random, and the load on the support team leveled out with the same traffic volume.
What to do if another network plugin is already running in Kubernetes
You can migrate, but not by taking a "brute force" approach — removing one plugin and installing another will not work. During the network plugin change, pods lose addresses, and together with the old plugin's objects, network policies also disappear.
Therefore, the migration is carried out in stages: the new plugin is brought up alongside, the rules are transferred, and only then is the old one removed.
The first step is an inventory of what actually runs in the cluster.
They look at which network policies are applied (some objects usually turn out to be drafts), whether there are pods with direct binding to the node network, and whether the address ranges of the old and new plugins overlap.
Overlapping subnets and mismatched routing modes are the most common reason why changing the network plugin requires manual editing rather than a simple replacement.
Compatibility of network plugins is checked before migration, not during it. The new plugin is installed in a mode where it sees nodes and reads cluster state but does not assign addresses — routing remains unchanged. An error at this stage costs one node, not the entire cluster.
Network policies are transferred separately: selectors and rules differ between plugins, and a policy that previously blocked traffic may be interpreted differently in the new plugin.
Each rule is tested in a test environment, and only when services respond are the old objects deleted.
A migration without downtime is done node by node: the node is taken out of service, switched over, connectivity is checked, and it is returned to the cluster. Capacity headroom is needed — the cluster must survive the loss of one node.
The old plugin's configuration is kept fixed as a rollback plan and removed only after all checks have passed.
Common technical doubts about the overlay network
Doubts about the Weave Net overlay network in a cluster usually surface not during installation but during operation: how it behaves under load, what happens when a node is lost, and what enabling encryption will entail.
Below is what our engineers verify on the test environment before putting the cluster into production operation.
How much does encapsulation add to latency between services?
Encapsulation adds a header to each packet, and part of the path goes outside the normal kernel stack, so latency increases. On typical web workloads, this is fractions of a millisecond, but with frequent exchange of small requests, the difference becomes noticeable.
We measure latency before and after enabling the overlay and leave direct routes where nodes are in the same subnet.
How is encryption enabled and what does it change?
Encryption is enabled at the overlay level and covers all traffic between nodes, including service traffic. The cost is additional CPU load and reduced throughput on large flows.
That is why we enable it selectively: for environments where traffic passes through untrusted segments, and always with measurements taken before and after.
What happens during node failure?
The remaining nodes rebuild routes to the lost subnets, and controllers recreate pods on live machines. Recovery time consists of probe timeouts and container startup time — we measure it and record it as a target indicator so that degradation is predictable rather than random.
Can subnet addressing be changed on a live cluster?
The address range is changed in stages: first a new pool is brought up, then nodes are moved to it one by one, checking connectivity after each step.
A complete addressing change with cluster recreation is simpler and more predictable — the choice depends on how many services are tied to the old addresses.
How to diagnose network problems?
We go from simple to complex: connectivity between pods on different nodes, the state of the network agent on each machine, dropped packet counters, then access rules between namespaces. This way, the overlay problem is separated from errors in service configuration and policies.
Let's discuss the task: we will select a mode and return with a plan
To make the assessment substantive rather than "from a photo", the first conversation only needs inputs about the cluster: how many nodes and in which subnets, which network layer is running now, whether there are downtime restrictions, and who owns the routes between segments. Access "just in case" is not needed — a description and current manifests are sufficient.
Based on the conversation, we return with an assessment and work plan:
- placement scheme: how Weave Net Kubernetes will fit the existing addressing and not overlap with service ranges;
- selection of network mode — overlay on top of routing or direct mode, and where an encrypted channel between nodes is needed;
- step-by-step migration plan with rollback points to the previous network layer;
- list of traffic isolation policies between namespaces: what is already covered and what remains to be described;
- assessment of timelines and scope of work broken down by stages;
- list of what is required from your side: accesses, approval window, on-duty contact.
Cluster maintenance is a separate track: updating the network layer, monitoring node connectivity, responding to degradation, and handing over documentation to your team so that the configuration does not remain a "black box".
Describe the task and send the inputs — we will return with an assessment, work plan, and stage composition.