Containerd setup for Kubernetes: configuration and troubleshooting

containerd Failures in a Cluster: What Breaks and Why

The most common picture: a node was updated, the service was restarted — and the container won't start.

Pods hang in the pending creation state, although the image is present and the registry is available, and the logs show that the request to the container runtime ended with an error.

For the service, this means lost capacity: some replicas don't come up, and the rest receive increased load.

The cause is almost always a mismatch in settings, not a "raw" update. In containerd, the cgroup driver setting must match what kubelet uses: if one works via systemd and the other via cgroupfs, pods silently fail to start.

Add to that the chosen snapshotter, limits on parallel operations, and garbage from old layers — and the cluster loses nodes one by one.

The second group of container runtime failures is when the service stops responding rather than crashing.

The socket is occupied by a hung process, garbage collection takes longer than usual, the sandbox image isn't downloaded because a mirror is unavailable — and some pods enter a restart loop.

At the same time, runtime errors don't always reach monitoring: the node is listed as healthy, readiness probes fail, so the service quietly loses a share of traffic.

The cost of such incidents is not cluster downtime, but a drawn-out investigation. Auto-restart masks the root cause, node metrics look flat, and degradation accumulates for hours. Below, we break down how this is closed off by configuration and procedure.

What a Managed Container Runtime Provides

A configured runtime shows up primarily in container startup time. The containerd profile determines how image layers are laid out, how many unpack operations run in parallel, and how long the cache lives on the node.

With well-calibrated values, most registry requests during replica rollout disappear — pod startup drops from seconds to tenths of a second, and a mass application restart no longer stretches into minutes.

Cluster predictability rests on configuration being described as code and rolled out identically to all nodes. A runtime update first goes through on one node, and only after a full application startup cycle does the profile move to the remaining groups.

Behavior during updates no longer depends on who edited the node manually and when, and divergence between nodes is visible before it becomes an incident.

A separate class of failures that look spontaneous is removed: a node goes into an unavailable status, pods move, and the cause can't be read in the journal. This is usually behind disk I/O limits, uncontrolled growth of image storage, and filesystem snapshot leaks.

From the outside it looks like a node crash, but in fact it's a few lines in the configuration, after fixing which node failures stop.

The effect of configuration is measurable: the share of startup failures falls, reinstalling a node from a profile takes minutes instead of manual assembly, and incidents become reproducible.

Storage stops growing uncontrollably — old images are cleaned on a schedule, while rollback to the previous build remains available.

As a result, runtime stability ceases to be the achievement of a specific engineer. The profile is versioned together with the infrastructure code, changes are validated in a test environment, and every new node comes into service in the same state as the previous ones.

Comparison of containerd Operating Modes and Their Limitations

containerd as a runtime defines node behavior through configuration: which plugin accepts requests from kubelet, how image layers are stored, and what limits container resources.

One profile across all nodes is a condition for predictable scheduling; different profiles on neighboring nodes produce divergences that surface under load.

Where Configuration Hits Limitations

The storage profile determines node density: layered storage saves space and speeds up startup, but requires a compatible filesystem; disk-based storage is more predictable for writes, but images take up noticeably more space.

Changing the storage scheme or resource limit driver is not applied on a live node: a service restart is needed, which means container eviction and rescheduling — such work is planned as maintenance, not as an on-the-fly edit.

A separate limitation is consistency: a profile that is missing on some nodes breaks scheduling and monitoring. We lock down the profile matrix, validate it in a test environment, and only then roll it out across node groups.

Implementation Order: From Audit to Handover to Operations

Implementation follows a single route: first we capture the actual state of the runtime, then design the target configuration, and only then touch production. This order eliminates live edits: at each step, an artifact remains that can be used to verify the result or roll back.

1. Cluster configuration audit. We capture the containerd version, plugin composition, resource limits, garbage collector and logging parameters, and snapshot driver settings. Divergences between nodes are usually visible here.
2. Target configuration design. We agree on a single profile for all nodes and mark every deviation from the current state: why it is needed and what it will affect.
3. Runtime configuration in staging. We move the profile to a test environment, check compatibility with the current Kubernetes branch, mounted volumes, and workload images.
4. Load testing. We run a dense stream of container creation and deletion, looking at startup delays, memory consumption, and disk I/O spikes. We catch degradation before production, not after.
5. Production rollout. We update nodes one at a time with pod eviction and pauses for metric verification; if there is a deviation, the node returns to the previous profile.
6. Handover to operations. We provide the on-call team with version-controlled configurations, a rollback scheme, and an analysis of typical failures, and conduct a drill on a real incident.

Deliverable Configurations, Diagrams, and Procedures

All the settings we make during the work remain with the customer as files, not in chat threads and engineers' heads.

This is a key condition: if in six months the task is led by another team or your in-house specialist, they get a working picture of the environment, not a set of guesses. Below is exactly what is handed over along with the cluster.

• Cluster configuration files — declarations for each environment separately, explicitly stating what is inherited from the base layer and what is overridden locally; edits are made only through the version control system.
• Service relationship diagrams — what communicates with what, through which entry points traffic flows, where trust boundaries lie, and which nodes become a single point of failure.
• Access restriction rules — roles and access scopes for services and people, a default ban on excess privileges, the procedure for granting temporary elevated access and its automatic revocation.
• Environment documentation — node composition, current runtime branch (including containerd), addresses of image storages and logging systems, and the procedure for connecting new nodes.
• Update procedures — rollout sequence, compatibility check before applying, stop conditions, and step-by-step rollback to the previous state.
• Diagnostic scenarios for typical failures — what the on-call engineer checks when a service is unavailable, when a node is hung, when there is not enough space for images — with commands and expected output.
• List of acceptance checks — which tests must pass green in each environment before handover, and who records the result.

The document set is updated together with changes, not drawn up retroactively. If something in the environment changes without being reflected in files and diagrams, it is considered unfinished work.

Case Study: Cluster Stopped Losing Nodes Under Peak Load

A cluster under peak load was losing nodes: the agent on a node stopped responding to health checks, the node went to NotReady, and the scheduler restarted its containers on the remaining machines. Those received double traffic and dropped next — the cascade went in circles.

During peak hours, 12–18% of nodes were in NotReady, bringing a node back to operating mode took up to 40 minutes, and service availability dropped to 99.3%.

The investigation showed that the issue was not hardware, but the runtime configuration.

The standard containerd profile with basic limits could not handle the container density on a node: the logging driver wrote synchronously and blocked processes, images were pulled from an external registry on every start, and readiness probes fired before the application came up.

We raised the descriptor and process limits to match real density, switched logging to asynchronous mode with a file size limit, added a local image cache with preloading onto nodes, and brought probe timeouts in line with actual initialization time.

Metrics Before and After

Changes were applied in groups of 3–5 nodes, the configuration is versioned together with the manifests — if metrics deviate, the group rolls back to the previous profile.

Over a month of peak loads, there was not a single cascading failure: nodes went out only for scheduled maintenance.

How containerd Differs from Alternative Runtimes

<cite>containerd</cite> is a separate daemon that kubelet calls via the Container Runtime Interface (CRI): it pulls the image, prepares the root filesystem, and starts the process with the low-level runc runtime.

Docker Engine is structured differently — the same containerd inside, but with image building, a CLI, volumes, and user-defined networking added on top.

The cluster doesn't need this add-on: it adds an extra daemon between kubelet and the container, and divergences in its state between nodes produce different behavior with the same manifest.

The closest alternative is CRI-O, and it is deliberately narrower: only the minimum required by the interface, without build plugins and multiple snapshotters.

The selection criterion is simple: CRI-O when an extremely thin runtime is needed, containerd when different snapshotters, namespaces for tenants, and isolated sandboxes are required.

Compatibility is determined not by the runtime's name, but by support for the required interface version and the configuration of networking and volumes. Switching to another runtime in a running cluster is not just replacing the executable.

Containers created by the previous runtime will not be seen by the new one: the node must be removed from load balancing, the workload stopped, reconfigured, and pods restarted, and they are created in the new storage.

That is why images are downloaded again, and logs and metrics move to a different format — collection must be reconfigured in advance.

The switch is justified when the add-on itself gets in the way: version divergences between nodes, extra memory consumption, or the need for isolation that the previous runtime does not support.

After it, the startup chain is shorter, configuration is uniform across all nodes, and there is one fewer point of failure.

Common Doubts About Configuration and Compatibility

Changes in the container runtime almost always affect a running cluster, so teams have the same doubts: will services have to be stopped, what happens if nodes diverge, and how to make sure the edits didn't break startup. Below are short answers, then we move on to the work plan.

What should be done if node versions have diverged?

First we align them: the same set of runtime parameters on all nodes, otherwise the same image behaves differently depending on where the scheduler placed it.

Divergence usually surfaces on old nodes that were not rebuilt during an update — we update them first and only then move the load.

How can we make sure the changes are correct?

We validate every edit on a test node with the same set of workloads as in production: container startup, volume mounting, network access, correct process termination. Then a control run on one cluster node, and only after that a rollout to the rest. This way a configuration error is visible before it affects the entire cluster.

How to roll back the configuration?

We keep configuration files under version control, and before every change we take a copy of the current node state. Rollback means returning the previous set and rolling it out again in the same order, one node at a time. It takes the same amount of time as the update itself and does not require rebuilding the cluster.

Let's Discuss the Task: Configuration Audit and Work Plan

Let's Discuss the Task: Configuration Audit and Work Plan

For the assessment of cluster work to be based on facts rather than guesses, engineers need only a minimal set of inputs.

There is no need to send full exports — the items listed below are enough: from this data we will reconstruct the environment picture and understand where the configuration diverges from the actual load.

• The current containerd configuration file and output of current parameters. It shows what was manually overridden and what remains at default values.
• The method of node creation: image templates, installation scripts, or provisioning configuration. This makes it clear where edits will stick and where they will diverge during the next scaling.
• A description of the symptom with timing and full node logs for the failure window — they provide more than a retelling.
• Environment composition: the number of nodes and their groups, whether the profiles of control plane and worker machines differ, and how many environments are serviced.
• Image registry details: whether a local cache is used, how authentication is organized, and whether there are restrictions on pulling.
• Limitations: maintenance window, rollback requirements, security policies, and privilege separation.
• What has already been tried and with what result — this eliminates the repeat diagnostic stage.

After reviewing the inputs, we will return with an estimate and a configuration plan: what we change at the first stage, where maintenance windows are needed, and in what order we proceed.

The configuration audit comes first — it defines rollback points, so the start of work does not depend on how neglected the current state is. Describe the task — we will discuss the scope of work and timelines.