Cilium CNI with eBPF: Kubernetes Performance and Security

Cilium CNI: where the network inside the cluster breaks

Kubernetes networking most often breaks not on physical links, but higher up — at address translation and service proxy.

Rule chains grow linearly with the number of services, the connection tracking table overflows, and packet loss between services begins long before bandwidth runs out.

Diagnostics run into the fact that the cluster network bottleneck is not visible either in node metrics or on switch graphs: the counters are clean, while the client gets errors.

The second source is encapsulation between nodes.

The overlay wraps traffic in an additional header, the useful frame size drops, and the application does not know about it: some packets are fragmented or silently lost, TCP goes into retransmissions, and response time grows in steps under load.

The third is route reconvergence when a node fails. Rules are updated sequentially, and in this window some connections go nowhere: pods are alive, checks pass, but requests to them do not arrive. For services with short timeouts, this looks like random errors with fully healthy infrastructure.

Cilium CNI changes the point at which decisions are made: instead of long address translation chains — eBPF programs directly in the kernel, attached to network interfaces, with connection tables and socket-level load balancing. Next, we examine exactly where traffic is lost in such schemes and what changes when a node fails.

What an eBPF-based network model provides: latency, failures, isolation

The difference is visible at the packet path level. In the traditional model, every packet traverses rule chains, and their length grows with the number of services and policies.

An eBPF-based network model makes routing and permission decisions at the point where traffic is attached, bypassing long chains — the processing cost stops depending on the number of declared rules.

Latency between services decreases: some connections go directly between instances, without an intermediate node that accepts and re-establishes the connection.

As the number of services grows from dozens to hundreds, the latency curve remains almost flat instead of growing linearly, so there are fewer cascading timeouts at peaks — the service does not wait for the network.

Network fault tolerance rests on a distributed service map: each node knows which instances are alive.

When a node goes down, neighbors redirect traffic to surviving replicas without central coordination or going around the entire network — recovery time is measured in seconds, and some requests manage to be resent instead of failing.

Traffic isolation is described by service labels, not addresses and ports. An access rule is defined once and applies to new replicas, including those created during scaling; by default, traffic between namespaces is denied until access is explicitly allowed.

Acceptance relies on before-and-after measurements: latency between services under load, network recovery time after a node failure, share of successful requests at the moment of failure, and coverage of services by isolation policies.

Without these numbers, the migration remains a matter of faith — we capture them on a test bench that replicates the production environment.

Cilium CNI and alternative network modes: comparison

In a cluster, the network mode determines how a packet travels from pod to pod: via encapsulation, direct routing, or kernel-side processing.

The choice almost always depends not on throughput, but on what already exists in the infrastructure: L2 segment, BGP peering, cloud provider policies. Below is a comparison of network modes by when each is appropriate and where it starts to get in the way.

Settings are configured at the Cilium level and can be switched reversibly, but not without traces: changing the mode affects routes on nodes and on external equipment.

Before switching, we check whether the selected tunnel type passes through the network, whether there is packet size headroom, and whether the routers are ready to accept pod address announcements.

Implementation order: from network audit to handover of regulations

The work follows an order in which the cluster network changes predictably and does not take down production. First, we record the current state, then design the target scheme, test it on a cluster copy, and only after that migrate production workloads. At every step, there is a path back.

1. Network audit. We capture a map of flows between services, inventory policies, and look for "allow all" rules and services without restrictions. The output is a list of risks and bottlenecks, not generic recommendations.
2. Scheme design. We define address spaces, traffic exit points, and the default policy: deny everything, with permissions granted selectively. We agree on the scheme with your team before configuration begins.
3. Configuration on a cluster copy. We deploy Cilium CNI on a test environment, migrate policies, and enable observability: flow metrics and a log of dropped connections. We also practice the rollback procedure here.
4. Validation on the test bench. We run load close to production, check behavior during node failure and external link loss, and review latency and packet loss rate.
5. Switchover without downtime. We migrate workloads by node pools: new nodes join the target scheme, old ones are drained after traffic to them stops. Services remain available throughout.
6. Handover of regulations. We deliver the scheme, configurations, and policies in a version control system, describe diagnostics and rollback procedures, and train the on-call shift.

Access policies and network scheme: what remains in the project

A working cluster on Cilium CNI is not yet a result for the team that will remain to maintain it.

Therefore, along with the configured network, we hand over a set of artifacts that make it possible to reproduce the configuration in another environment, verify it during acceptance, and analyze an incident without the author: everything is in the repository next to the manifests, not in a chat thread.

• Network scheme — segments, pod and service address ranges, routes between nodes, exit points to external networks and to internal systems.
• Access matrix — a summary table of "source — destination — port" showing every permission and the explicit default deny rule.
• Network policies in declarative form — default deny for ingress and egress, label-based permissions, separate rules for DNS.
• Network layer configuration — kernel-level packet processing parameters, load balancing, encapsulation between nodes, and binding to the cluster operating mode.
• Observability scheme — failure metrics broken down by policy, monitoring dashboards, and alerts tied to specific rules.
• Maintenance regulations — procedure for updating the network layer, mandatory checks before rollout, and the procedure for reverting to the previous configuration.
• On-call shift instructions — how to view current rules, determine which policy rejected traffic, and what to do when connectivity degrades.

Technical documentation is written not "for a checkmark," but for specific operations: policy changes go through review against the access matrix, and failure analysis follows the scheme and logs, not the memory of a particular engineer.

Case study: how network failures were eliminated and recovery time was reduced

The customer's cluster is several dozen nodes, dozens of services, and high pod density. The network ran on a classic plugin: routing rules were assembled into kernel chains and rebuilt entirely on every change.

The more services there were, the longer any edit took, and at peaks a new pod waited seconds for the network.

We started not with replacement, but with measurements. We collected incident history over several months and broke it down by cause — network, application, infrastructure.

It turned out that about two-thirds of availability failures were attributable to the network layer, and recovery after a node failure took more than ten minutes: routes converged gradually, and some traffic went to unreachable addresses.

What was changed

We moved the network layer to Cilium: routing and load balancing moved into the kernel, and the intermediate chains that were rebuilt on every edit were eliminated.

Access policies were described declaratively at the service level rather than by addresses — the rule "service A sees only service B" is enforced at pod ingress, without an intermediate proxy.

We enabled observability: flow tracing between pods so that an incident is analyzed from logs, not guesswork.

Result over a comparable period: the share of network-related failures dropped from two-thirds to one-tenth of total incidents.

Recovery time after a node failure decreased from minutes to tens of seconds — routes are reconverged in a distributed manner rather than by rebuilding shared rules.

The cluster handles peaks without network degradation, and the on-call engineer analyzes an incident using ready-made tracing.

What to do if another network module is already deployed in the cluster?

Yes, replacing the current network module with Cilium without service interruption is feasible — but not by installing over the existing one; it is done by migrating nodes in stages.

The key check here is not the performance of the new module, but compatibility with the current cluster: whether pod and service address ranges overlap, who handles traffic forwarding, whether the built-in load balancer remains, and whether MTU values and encapsulation modes match.

We check compatibility before the first change: compare address pools, the list of network policies, load balancing rules, integration with the ingress controller, and the address allocation method.

Discrepancies in MTU and routing mode cause packet loss and unnoticed latency degradation, so we align such parameters in advance, on a test environment with the same set of workloads.

Zero-downtime migration is based on node-by-node separation: a node is removed from load balancing, the new network module is brought up on it, traffic is checked against control scenarios, and only then is the node returned to service.

Then the next node follows, until the entire pool moves to the new scheme; some rules initially operate in compatibility mode so that nothing is lost at the boundary.

We plan the reverse rollback from the start: the previous module remains in the cluster in an inactive state, node configurations are stored as versions, and switching is described by a single command.

If latency grows or connections drop after migrating a group of nodes, the node returns to the old scheme within minutes, without rebuilding the cluster.

As a result, we take before-and-after measurements: network policy processing, packet loss rate, connection establishment time, and load on compute cores.

Technical questions on compatibility and load

Before changing the network layer, four things are checked: compatibility with the current Kubernetes version, node load, update order, and behavior of running services. Below are answers to the doubts that arise most often in this analysis.

Checks follow a fixed order: inventory of policies and network modes, a test bench with a production load profile, then rolling updates.

How will the migration affect node load?

Some rules are resolved in the kernel rather than expanded into routing chains: on nodes with hundreds of services, the number of rules and response time decrease. The trade-off is a constant overhead for the Cilium agent on each node. We capture metrics before and after the migration.

In what order should the cluster be updated?

In waves: first the control plane, then worker nodes in groups with readiness checks between steps. We migrate policies in advance so that traffic is not left without rules; if there is a failure, we roll back only that group.

Will running applications break?

Applications are not rebuilt — only the network layer beneath them changes. The risks are elsewhere: policy behavior, preservation of the client's source address, and name resolution. We compare allowlists before and after.

Let's discuss the task: we will return with an estimate and work plan

To make the estimate substantive rather than "based on feelings," we need input: how many clusters and nodes, what Kubernetes version, how the network is designed today, what requirements exist for isolation between namespaces, and what is critical not to lose during the migration to Cilium CNI.

Maintenance windows and acceptable downtime are especially important — this determines whether we proceed in stages, environment by environment, or change the network layer on a dedicated test bench and then migrate production.

Based on the analysis, we return with a package that includes:

• an implementation plan by stages: node preparation, migration of some workloads, enabling network policies, decommissioning the old network layer;
• a work estimate — scope, timelines, and team composition for your environment;
• a migration scheme with rollback points at every step and a clear criterion for "time to roll back";
• a list of requirements for access, the test environment, and the schedule of production changes;
• a pre-start checklist — what should be done on your side before the first edit;
• a set of acceptance metrics: latency, throughput, share of traffic under policies, and recovery time after a node failure.

We deliver the plan as a document with stages, dependencies, and areas of responsibility, and also walk through it on a call with your engineers — so that contentious points are resolved before the start, not at the moment of switchover.

Describe the task and the current state of the clusters — we will return with an estimate and a work plan.