Why CRI-O installation breaks at the network integration stage
Installing the runtime itself goes cleanly — it is the integration that breaks.
The node does not become ready: kubelet contacts the container runtime socket, the runtime brings up the pod sandbox, but the cluster network component does not find its config in the expected directory or reads it from a different version — and the sandbox remains without an address.
Pods hang waiting, runtime configuration errors flood the events, and after them internal DNS and everything that depends on it goes down.
The second cause is desynchronization of paths and settings. The runtime and the CNI plugin are updated separately, and after another edit the configuration directory shifts: the runtime looks for files in one place, the plugin in another.
On the test bench everything matches, but on a production node the discrepancy surfaces a day later, so the cause is searched for long and blindly.
And the most costly habit is manual edits directly on live nodes. Runtime and network component configuration is edited on an operating node to bring it back into service faster; the next node is deployed from a template and repeats the same error.
Nodes diverge from one another, the cluster stops being reproducible, and rollback has to be done manually under load.
It is not the installation that breaks, but the integration between the runtime and the network.
We explain how this is closed: a single source of configuration instead of edits on nodes, verification of the integration before production, and an application order in which a node either comes up completely or does not come up at all.
What changes in cluster operation after runtime configuration
Switching to CRI-O changes not only the launch command but also the very interface between kubelet and the container environment.
CRI-O implements the container interface directly, without an intermediate daemon layer: one process per node, minimal steps between a kubelet request and pod sandbox creation.
In practice, this reduces node onboarding time — the node reaches the ready state seconds after the service starts, rather than after warming up several daemons and synchronizing their states.
A separate effect is that an entire class of failures at the runtime-network boundary disappears.
The CNI plugin connects to the runtime as the single source of configuration, and the initialization order of the network and sandbox is fixed in configuration files, so after a node restart there are no pods that formally came up but remained without an address or route.
Previously such discrepancies were caught manually from logs — now there is nowhere to reproduce them.
Cluster recovery time after a failure is calculated differently. Sandbox state is stored on disk, and at startup the runtime brings them up in the original order without recreating containers entirely: workloads return in a strict sequence — first system ones, then applications.
A routine node restart fits into tens of seconds; an emergency one follows a predictable scenario without manual intervention.
The load profile on nodes becomes more even: a single runtime process holds a fixed amount of memory, spikes from image reindexing disappear, and workload placement density grows without replacing hardware.
Stability shows in daily container restart statistics — their number drops to isolated cases, and each already has a clear cause.
Runtime deployment modes: comparison of options
Installing CRI-O is only part of the solution: the deployment mode determines who is responsible for runtime updates and what can be changed on the node at all. Below are four real alternatives: two ways to obtain the runtime and two ways to place it.
| Option | When it fits | Limitations |
|---|---|---|
| Packages from the distribution repository | Standard environment, updates together with the OS | Only the version from the distribution branch is available, patches arrive with a delay |
| Build from source | A custom set of modules is needed, extra capabilities are disabled | Kernel compatibility and updates are shouldered by the team |
| Runtime on each node | Own environment, full control over parameters | Nodes must be updated and synchronized in a coordinated way: diverged versions break workload scheduling |
| Provider-managed cluster | The result is needed, not maintenance of the control layer | The runtime version and part of the configuration are set by the provider; custom kernel modules cannot be connected |
A separate limitation is rootless isolation: the runtime starts as an unprivileged user, does not rewrite the node's network rules, and cannot see some devices.
Privileged workloads and some network plugins do not come up in this mode — if the environment requires access to block devices, the mode will have to be reconsidered.
The mode is fixed before deployment: changing the installation method on a running environment means reinstalling the runtime and restarting all workloads on the node.
Therefore the option is chosen based on the answer to the question of who maintains the environment — your own team or a provider.
CRI-O installation: work stages from node audit to handover
CRI-O installation on a live cluster is not a single command but a sequence in which an error in the middle stops production workloads.
The runtime is linked to kubelet, the network plugin, and process isolation settings, so first we record the current state, then change one node at a time while preserving rollback capability. Below is the work order that we repeat on every cluster.
- Cluster node audit. We collect kernel and package versions, storage drivers, kubelet settings, active network plugins, and a list of images with checksums. The result is a report with bottlenecks: where CRI-O will install without rebuilding the node, and where a preliminary upgrade is needed.
- Runtime configuration design. We design the image pull policy, process and file descriptor limits, storage placement on a separate volume, and logging and log rotation parameters.
- Installation on a pilot node. We install from the official repository, verify the runtime socket address against kubelet settings, and run container start and stop checks.
- Cluster test environment. We deploy a copy of the production profile and run through scenarios: rollout, pod restart, image update, network isolation, and node failure behavior.
- Migrating production nodes one by one. The node is taken out of balancing, the runtime is changed, and the node is returned to rotation — with a pause for metrics so that the previous state remains reproducible.
- Handover of the operations procedure. We provide the scheme, update instructions, diagnostic checklist, and rollback map, and walk the on-call shift through them.
Handover package: configurations, network diagrams, and procedures
We assemble the handover package so that an on-call engineer can bring up the runtime from scratch on a new node and restore it after a failure without messaging us.
Each file is versioned together with the cluster configuration and accompanied by a comment: what the parameter is, why this value was chosen, and what will change if it is edited.
- Runtime configuration files: base config, limits on the number of pods and open file descriptors, logging parameters, and restart policy — with an explanation for every non-obvious value.
- Pod and service addressing scheme: address pool, subnet size per node, service network ranges, and a calculation of how many nodes the scheme will support before expansion.
- Node access policies: roles and bindings, login restrictions, separation of on-call and administrator rights, and procedures for working with keys and certificates.
- Runtime update procedure: taking the node out of service, image compatibility checks, rollback window, and criteria for stopping an update.
- Network and connected network plugin configuration: manifests, selected routing mode, and connectivity checks for pods between nodes and with external services.
- Diagnostics instructions: typical failures, a set of commands for capturing state, and thresholds after which an incident is escalated.
- Component map and acceptance checklist: what runs where, what it depends on, and which changes require approval.
Case study: cluster stopped failing after network configuration
What was the situation
The cluster operated in mixed mode: some nodes ran on the old runtime, some on the new one, and the network layer was configured separately and manually.
Containers came up but did not get an address — a restart loop would begin, and up to 7% of starts during peak hours ended in failure. The on-call engineer edited configurations on specific nodes almost every shift, and recovery after such a failure took up to 40 minutes.
What we did
We migrated the cluster to CRI-O as the only runtime and fixed the initialization order: first the network module comes up, then the runtime picks up the already prepared network configuration.
Node configurations stopped being “live” — they are stored in a version control system, rolled out with a single command, and validated before application.
Isolation policies and network parameters are described declaratively, so manual edits on nodes are no longer needed: any deviation from the reference is visible immediately.
What the result was
The share of successful starts is 99.9%; there were no failures at the runtime-network boundary during the observed period. Recovery time decreased from 40 to 4-6 minutes, and manual interventions went from two dozen per month to one or two planned ones.
The cluster no longer requires an on-call engineer to be present at night, and migrating subsequent nodes to the same runtime took one procedure instead of a separate project.
What to do if another runtime is already deployed on the nodes?
The runtime for a node is an implementation detail: the orchestrator communicates with it through a standard interface, so manifests, services, and network policies are not rewritten when migrating to CRI-O.
Three things need to be checked before the first switch: matching the cgroup driver with kubelet settings, compatibility of the selected network plugin, and private registry access — otherwise pods will hang waiting for an image or remain without network.
The switch is done one node at a time without stopping workloads. First the node is marked unschedulable, then pods are evicted from it — neighboring nodes take them over.
This requires spare resources in the pool, and disruption policies are configured so that eviction does not cut a service's quorum. While the node is empty, the old runtime is stopped, CRI-O is installed and configured, and kubelet switches to the new socket.
The node is brought back only after checks: a pod starts, exec works, logs are read, volumes mount.
The first one is a single node in the test pool — it runs scenarios with private images and storage attachment, and the node returns to the general pool when behavior matches expectations.
We keep rollback ready: the old runtime remains installed, kubelet returns to the previous socket, and the node is taken out of service again. The node removal order does not change: one at a time, with a pause for verification, without switching the entire pool at once.
Technical concerns: frequently asked questions about the container runtime
Before cluster acceptance, the customer checks not “did the runtime come up” but the behavior of the entire integration: orchestrator, runtime, network plugin, image storage.
Most incidents after launch are not failures of the runtime itself but mismatched versions and parameters between components, which does not manifest on a three-node test bench.
Below are the questions most often asked during handover. Some answers depend on your topology, so we check the configuration on your cluster, not on a standard scheme.
Is the runtime compatible with our orchestrator version?
CRI-O implements the standard interface and is updated together with the orchestrator according to the official compatibility matrix. We fix the allowed version pairs in the configuration and in the update plan — then upgrading one component does not break the other.
Do you configure the network plugin, or does it remain on our side?
The network lives separately from the runtime: the runtime only calls the plugin when creating a container.
We bring the network configuration to a unified form on all nodes, check the load order, the absence of conflicts with system components, and correct address resolution when a pod restarts.
What happens when the runtime is updated on a live cluster?
We update node by node with workload eviction so that applications move to neighboring nodes without downtime. We plan separately for containers with local volumes and node affinity. Rollback is to the previous stable build until the node returns to service.
What remains after handover — a working cluster or documentation?
A package: runtime and network manifests under version control, an update and rollback plan, a set of node health checks, and a procedure for typical operations. Your engineers reproduce the environment from scratch using these materials.
How can we be sure the cluster configuration is correct?
We run the acceptance checklist: compare runtime and network parameters on each node, check node restart, recovery after a workload failure, and behavior when the image registry is unavailable. The output is a report with discrepancies before and after edits.
Let's discuss the task: work plan and handover scope
To make the conversation concrete, describe the environment: how many nodes are in the cluster and what role each plays, which container engine runs now, where images are stored, and whether network policies between namespaces are needed.
If you have a network diagram, working configurations, and a log of recent failures, attach them — that removes half the questions before the call.
- OS and kernel version on the nodes, available modules and storage drivers;
- rollout scenario: a separate environment first or straight to the production environment;
- who owns the network segment and in what order rules are approved;
- change window and allowable service downtime;
- requirements for image updates and rollback to the previous release;
- who on your team accepts the work and by what criteria.
With this, we return with a package that can be used to make a decision:
- implementation stage plan with checkpoints after each step;
- layout diagram of components, network zones, and image storage;
- estimate of cluster work: node composition, scope, and sequence;
- handover scope: configurations, security policies, maintenance procedure, and instructions for the on-call shift;
- change approval procedure and responsibility matrix.
Describe the task in free form — we will return with a plan and estimate.