Cilium service mesh: why the sidecar architecture hits a ceiling
The classic service mesh is built on sidecar proxies: a separate proxy is spun up next to each application container, and all inbound and outbound traffic flows through it.
On a cluster with hundreds of pods, that means hundreds of extra processes, each with its own connection buffers and routing tables.
The first thing that starts to hurt is network latency: every request takes an extra network hop inside the pod, and across chains of a dozen services the delays add up to a noticeable amount.
The second bottleneck is memory consumption. A sidecar reserves resources regardless of how much traffic actually flows through it: a service with one request per minute incurs the same overhead as a heavily loaded one.
On large clusters this adds up to tens of gigabytes that go not to the application but to maintaining the mesh itself. On top of that, every proxy has to be updated, restarted, and synchronized whenever the configuration changes.
The third problem is debugging and fault tolerance. When a request doesn't get through, the cause may be the application, its sidecar, the recipient's sidecar, or the control plane.
Logs are scattered across pods, tracing breaks at proxy boundaries, and a failure of a control component leaves the cluster in an undefined state.
Cilium service mesh removes the very source of these costs: routing and policies are enforced at the kernel level via eBPF, with no intermediate proxy in the data path. Let's break down how this changes latency, memory consumption, and fault tolerance.
What the sidecar-less eBPF architecture delivers
A classic service mesh is built on sidecar proxies: next to each application container, a separate intermediary container is spun up, through which all inbound and outbound traffic passes.
On a cluster of a hundred pods, that's a hundred extra processes, each with its own memory and CPU consumption, plus two extra network hops per request. Latency grows, resources go toward maintaining the mesh itself rather than the application.
Cilium Cilium moves the same logic into the kernel: network policies, routing, load balancing, and encryption are performed at the level of eBPF programs attached to network interfaces.
No proxy intermediary is needed in the pod — traffic is processed where it already flows, without additional context switches between user space and kernel space. This yields three measurable effects.
Latency. A request between services no longer passes through two user-space processes. On typical microservice chains, this removes a noticeable share of overhead per call, and the longer the chain, the greater the cumulative gain in response time.
Resources. The sidecar container leaves the pods: memory it held is freed, along with the CPU time it spent on proxying. On a dense cluster, this frees up resources for the workloads themselves rather than for the mesh infrastructure.
Transparency. The kernel-level event stream sees every request — who talked to whom, over which protocol, with which response code — without instrumenting application code.
This provides observability by default: the dependency map and the causes of failures are visible immediately, rather than being pieced together from logs across different sources.
The practical bottom line: the same set of capabilities — mutual authentication, access control, telemetry — with fewer moving parts and predictable performance under load.
Deployment modes: sidecar, sidecarless, and hybrid
Cilium can operate in three modes, and the choice between them is primarily a question of compatibility with what is already running in the cluster.
The classic sidecar places a proxy container next to each pod: the application does not change, but resource consumption grows and the pod lifecycle becomes more complex.
The sidecarless mode moves traffic processing into the kernel via eBPF — there is no proxy in the pod, latency is lower, but some L7 scenarios require explicit enablement. The hybrid mode keeps the sidecar only where it is actually needed, and hands the rest to the kernel.
| Variant | When it fits | Limitations |
|---|---|---|
| Sidecar | Non-standard L7 filters are needed, applications without support for eBPF interception, phased migration | A proxy per pod: higher memory and CPU usage, slower pod startup |
| Sidecarless | Homogeneous cluster, priority on latency and pod density, L3/L4 traffic with selective L7 | Some advanced L7 rules require explicit configuration, not all third-party extensions are compatible |
| Hybrid | Mixed workload: critical services with sidecar, the rest without | More difficult debugging and unified policy, discipline in pod labeling is required |
A practical limitation of sidecarless is that not all applications are ready to hand traffic over to interception at the kernel level, and some L7 scenarios (complex header-based routing, custom filters) are still more reliably handled by a proxy.
The hybrid mode removes this contradiction, but requires explicitly specifying which pods go through the sidecar, otherwise behavior becomes unpredictable.
Compatibility with existing network policies and mTLS is verified before switching: first on a single namespace, then expansion. Rollback is possible at any stage — the mode is set by an annotation on the pod or by a cluster-level setting, without rebuilding applications.
Stages of implementing Cilium service mesh in a cluster
Implementing Cilium service mesh begins not with installation, but with an inventory of what is already running in the cluster.
At this stage, it is important to understand the network topology, current CNI plugins, the set of network policies, and how applications communicate with each other.
Mistakes at this step are the most costly: unaccounted-for traffic or a conflict with an existing CNI leads to loss of connectivity after rollout.
After that, the work proceeds iteratively, with verification at each step and the ability to roll back. Below is a typical plan that we adapt to a specific cluster.
- Cluster and network audit. We build a map of nodes, versions, the current CNI, network policies, and traffic entry points. We identify which services are critical and which can be migrated first.
- Network design. We determine the operating mode (overlay or native routing), address ranges, and the routing policy between pods and nodes. The design is agreed upon before any changes begin.
- Environment preparation. We deploy Cilium in a test environment, check compatibility with current components, and configure traffic encryption between pods and baseline policies.
- Service mesh configuration. We enable L7 routing, mTLS between services, access rules, and observability via Hubble. Policies are described declaratively and versioned alongside code.
- Validation on pilot services. We migrate 2–3 non-critical services, examine latency, the success rate of connections, and behavior during node failure. We compare against baseline metrics.
- Gradual rollout. We expand the scope to the remaining services in waves, with monitoring and a ready rollback plan at each step.
- Documentation handover. We deliver the network design, configurations, policy descriptions, update procedures, and incident diagnostics instructions.
After handover, the customer's team independently manages policies and handles incidents using the prepared scenarios. If you would like to discuss an implementation for your cluster, describe the task, and we will come back with an estimate and a work plan.
Access policies and IAM roles in Cilium
Access policies in Cilium are described declaratively — as CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy objects, rather than as a set of rules in a cloud interface.
This means the customer receives not a "setting" but a versioned artifact: it can be committed to git, run through review, and rolled back together with a release.
IAM roles in this case remain on the side of the cloud or cluster and are linked to network policies through an explicit mapping of service accounts.
Below is what exactly is handed over after implementation.
- Network policy configurations: L3/L4 rules by pod and namespace labels, L7 rules for HTTP/gRPC with method and path restrictions, a separate set for DNS and egress.
- Traffic flow diagrams: a map of "who talks to whom and on which port," built from actual observation data rather than assumptions.
- IAM role and service account mapping matrix: which role is bound to which service and which policies restrict it.
- Policy change procedure: the review order, mandatory checks before applying, and the procedure for rolling back to a previous version.
- Default policies: a deny-all inbound traffic mode with an explicit list of allowed directions — so that a new pod does not gain access "just by being started."
- Diagnostics instructions: how to read the log of dropped connections and distinguish a policy block from a network failure.
- Acceptance checklist: a list of scenarios that are verified after every change — service availability, namespace isolation, and the operation of external integrations.
Each artifact is accompanied by a short description: why it is needed and what will break if it is not maintained. Such a kit allows the customer's team to change rules independently, without coming back to us for every edit.
Case study: migrating to Cilium service mesh with zero downtime
The client is a product team with dozens of services in a cluster and a strict availability requirement: releases happen daily, and no maintenance window is provisioned. Initially, traffic between services went through a sidecar proxy in each pod.
On paper the scheme worked, but in practice it produced three systemic problems: a noticeable increase in latency at every hop, a tangible memory overhead for the proxies themselves, and a complete lack of a clear picture of who talks to whom and how inside the cluster.
The migration was done to Cilium — a networking layer on eBPF that moves routing, load balancing, and encryption to the kernel level, without a proxy container next to the application.
The key point for the customer: this is not "yet another component on top," but a replacement of the existing networking layer, so the migration can be carried out gradually, without switching the entire cluster at once.
What it was like before implementation
- Latency between services at every hop — due to the double pass through the proxy.
- Memory and CPU time went to sidecar containers rather than to the payload.
- Incident diagnostics — via logs from disparate proxies, without a single map of connections.
- Traffic encryption was configured manually for each service.
What was changed
Cilium was deployed in a mode where network policies and routing work at the kernel level, and mTLS between services is enabled by configuration rather than by manually editing each manifest.
Hubble was connected as a source of observability: it shows flows between services and allows an incident to be investigated via the connection graph rather than by grepping logs.
The migration was carried out namespace by namespace — first non-critical services, then the rest, with metrics checked at every step.
What it became after
Latency between services dropped by roughly a third on chains with several hops. The memory that sidecar containers occupied was freed up — pod density on nodes increased. Traffic encryption became default behavior rather than a separate task for each service.
Incident investigation shrank from hours to minutes: the flow graph in Hubble immediately shows where the connection breaks.
There was no downtime during the migration — it proceeded in parallel with the production workload, and a rollback to the previous layer was possible at any stage.
How does Cilium service mesh differ from Istio?
Cilium service mesh is built on eBPF — programs execute in the Linux kernel at network packet hook points, so routing, load balancing, and policies are applied without a proxy in user space.
Istio historically works through sidecar containers: an Envoy is spun up next to each pod, intercepting traffic and applying rules.
Hence the main architectural divergence — Cilium embeds into the kernel and uses the existing network path, whereas Istio adds a separate proxy layer for each pod.
The practical difference shows up in overhead and in what the engineer sees. A sidecar per pod is an extra container that must be updated, have resources reserved for it, and be factored into latency: every request goes through an extra hop.
Cilium removes this layer for L3/L4 policies and load balancing, and connects L7 parsing selectively, only where HTTP-based rules are needed.
At the same time, Istio provides more mature and predictable L7 functionality out of the box — header-based routing, fine-grained retry and timeout rules — and this is its strong suit.
The approaches to mTLS also diverge. Istio issues certificates through its own authority and keeps them in the sidecar, encrypting traffic between pods at the proxy level.
Cilium can do mutual TLS by means of the kernel and through integration with external certificate issuance systems, which reduces the number of components but requires careful rotation configuration.
Observability: with Cilium it's Hubble with a map of service dependencies and flows straight from the kernel; with Istio it's Envoy telemetry with rich labels, but at the cost of that same proxy layer.
The selection scenarios are as follows. Cilium service mesh wins when the cluster is large and density matters: fewer containers, lower latency, a unified network and service policy in a single layer.
Istio is chosen when the most complete L7 control is needed, complex routing between versions, and established operational practices. In practice, these approaches are not mutually exclusive — Cilium often covers networking and security, while L7 tasks are handled selectively.
We analyze a specific cluster and say which option will deliver a predictable result without extra layers.
Common technical concerns during adoption
Cilium sits on top of the standard Kubernetes CNI interface, so there is no need to abandon the familiar network layer — only the component that enforces the rules changes.
Pods, services, and NetworkPolicy continue to work as before, while routing and load balancing move to the kernel level. In practice, this means existing manifests migrate without rewriting, and new capabilities are enabled as needed.
Compatibility with current clusters and clouds?
Cilium deploys on managed clusters of major cloud providers and on bare metal. Limitations arise where the cloud itself manages the network layer — in such cases some features are disabled while the rest work normally.
Before adoption, we check the kernel version on the nodes and the routing mode: this determines which capabilities are available immediately and which require configuration.
How does rollback work if something goes wrong?
Deployment proceeds in stages: first a control group of nodes, then the rest. Routing rules and policies are stored declaratively, so returning to the previous state is a matter of applying the prior configuration rather than manual editing.
If the new layer causes problems, nodes revert to the original network mode one by one, without stopping services.
What about security and traffic encryption?
Access policies are described at the service level rather than by addresses, and are enforced on each node before a packet reaches the application. Mutual authentication between services is enabled without rewriting code — certificates are issued and rotated automatically.
This closes the class of attacks where an intruder is already inside the perimeter and moves between services.
How to debug if traffic does not pass?
Every packet decision is visible in the observation log: which service contacted which, whether the connection was allowed or denied, and by which policy. Instead of cycling through hypotheses, the engineer looks at the specific event and finds the cause within minutes.
The same observability provides a map of dependencies between services, which usually has to be assembled manually.
If you still have questions about your environment — describe the task, and we will come back with an assessment and an adoption plan.
Let's discuss your task and come back with a plan
To make the conversation substantive rather than "tell us about Cilium," start with three inputs: what the cluster is and what workload it runs, where it hurts right now — networking, security, or observability — and what result you want to see a quarter from now.
That is enough to understand whether you need a service mesh on top of eBPF or whether the standard Kubernetes networking layer covers the task.
From there we work step by step, and at each stage you see an artifact, not just a status update in chat:
- We analyze the current state: how networking between services is structured, where mTLS is enabled, which policies are already in effect, and what Hubble shows.
- We define the goal in measurable terms: the share of traffic under mutual authentication, incident response time, and the depth of observability history.
- We propose an implementation scheme: the mode of replacing the networking layer, the order in which policies are enabled, and rollback points at each stage.
- We agree on a phased plan with clear readiness criteria — what must work before moving to the next step.
- We deploy on a pilot environment: one environment, a limited set of services, full control of metrics before and after.
- We migrate to production while preserving the ability to return to the previous configuration without downtime.
- We hand over documentation and set up monitoring so the team can run the system on its own going forward.
We send the assessment and implementation plan after a short review — with stages, timelines, and what will be required from your side: access to the cluster, a change window, and an engineer contact for coordination.
Describe the task in whatever format is convenient — we will come back with specifics and next steps.