Where an LXC container breaks and how to notice it before a failure
Most often, a node with containers goes down not from a traffic spike but from contention for a shared resource.
If memory limits are specified in the configuration but not enforced at the kernel level, one process inside a container starts evicting memory pages from its neighbors — and it is not just that process that stalls, but the entire service on the node.
Worse, the event is visible only in the host's system log, not in the metrics of the affected container.
The second bottleneck is disk I/O and the network connection table. On a shared kernel, containers share the file system cache and bridges, so a “quiet” neighbor with active log writes degrades the response time for the others.
Such degradation under load grows gradually: response time increases, but alert thresholds have not yet triggered, and the failure comes as a surprise to the on-call shift.
The third class of failures is isolation leaks. Unclosed file descriptors, forgotten mount points for host directories, temporary file systems inside the container: LXD allows such scenarios, and restarting the container masks the problem — a day later it returns.
On a node with dozens of containers, this turns into infrastructure instability that is blamed on “hardware.”
Separately, there is the growth in the number of containers on a single server: aggregate limits begin to exceed real resources, the scheduler can no longer cope, and any launch of a new instance becomes a risk for the running ones.
We'll examine how each of these scenarios is closed by configuration rather than by on-call reaction.
What managed container virtualization on servers provides
System-level container virtualization changes service placement density: where several virtual machines used to run, several dozen isolated environments can be hosted.
Each gets its own processes, file system, and network, but shares one kernel with the host — hence the savings in memory and launch overhead.
We work with LXC: an environment comes up from a prepared image in seconds. This directly determines recovery time — returning to a working version after a failed update takes seconds instead of half an hour to rebuild and reconfigure the service.
Taking a state snapshot before every change makes rollback a routine operation: the engineer restores both the configuration and the data in full.
The same mechanism works when moving an environment to another server — the image moves, rather than the configuration being restored from memory.
Process isolation closes a separate class of failures: a memory leak or a hung worker process inside one service no longer takes down neighbors on the same host.
Quotas on memory, CPU time, and process count limit each environment's appetite, while permission separation prevents an application from looking into others' data.
In total, the effect is measurable: placement density is several times higher, the recovery window is seconds, and an incident is localized to a single service.
Before moving the production environment, we verify precisely these indicators: how many environments the host reliably holds, how long a rollback takes, what happens when a quota is exhausted, and how a neighboring service behaves.
Comparison of isolation modes: LXC, virtual machines, sandboxes
The isolation mode determines three things at once: how many workloads fit on one physical host, how quickly an instance comes up, and what exactly breaks during a failure. Below is a comparison without marketing, including cases where containers should not be used.
| Mode | When it fits | Limitations |
|---|---|---|
| LXC — system containers | Similar services where density and second-scale startup matter | Shared kernel with the host: you cannot load your own kernel or module, isolation is weaker than with VMs |
| Virtual machines on a hypervisor | Different kernels, strict isolation requirements, inherited workloads | Hypervisor overhead: part of the resources goes to virtualization, startup is slower, density is lower |
| Application-level sandboxes | Untrusted code, system call isolation within one host | Reduced set of calls: not every application will work without modification |
In colocation, it is not the mode itself that decides but the limits: quotas on CPU, memory, disk, and network bandwidth for each container, so that a noisy neighbor does not take resources from the others. A container without limits next to someone else's workload is not a saving but a shared failure.
Therefore, we make the choice based on the workload profile, not habit: if a service needs its own kernel or a hard boundary for security requirements, it stays in a virtual machine; everything else moves into containers.
A mixed fleet is not a problem if the boundaries are drawn deliberately. Describe your task — we will come back with an assessment of which modes should get what.
Implementation procedure: from infrastructure audit to handover
LXC implementation begins not with installing packages but with inventory: what production runs on, how disks are allocated, and which services cannot be stopped even for a minute.
An infrastructure audit provides a dependency map and an understanding of how many workloads containerization can sustain without degradation. Then comes a sequence in which each stage ends with a verifiable result rather than a report of work performed.
- Survey and infrastructure audit. We collect an inventory of services, actual CPU, memory, and disk consumption, and dependencies between nodes. We separately mark what cannot be stopped — such workloads are migrated last and with hot standby.
- Target design. We describe what moves into containers and what remains on separate hardware, how roles are distributed across nodes, where data resides, and how the network is organized. We agree on the design before configuration to avoid redoing it on a live environment.
- Cluster configuration. We connect nodes, define storage pools, network bridges, and profiles with limits on CPU, memory, and disk I/O. Container templates lock in uniform parameters — environments stop diverging between test and production environments.
- Workload migration. We start with non-critical services, check behavior under real traffic, and remove bottlenecks. Only after that do we migrate the main part.
- Failure testing. We power down a node, disconnect a disk, break a network link — and see how long it takes the cluster to redistribute containers and restore availability. For each scenario, we record the recovery time in a table.
- Acceptance and verification. A run through the checklist: state snapshots, rollback to the previous configuration, limits, access rights, backups. Anything that fails verification is refined before handover.
- Team training. We cover typical operations and incident scenarios, hand over configuration documentation and an update procedure without service downtime.
Describe your task — we will come back with an assessment and work plan following this procedure.
What we hand over: configurations, designs, procedures
After handover, your team has not a set of scattered commands but a working kit: configurations under version control, designs, procedures, and a clear sequence of actions for normal and emergency scenarios.
Using it, an on-call engineer can bring up an LXC container, take a node out for maintenance, or roll back a failed change without contacting us.
- Configurations as code. LXD profiles, declarative container descriptions, network and storage templates are stored in a repository: changes go through review rather than being edited by hand on the host.
- Network designs. Bridge topology, addressing, filtering rules, egress points — with notes on where our responsibility ends and yours begins.
- Update procedure. The order for rolling out images and updating the host, maintenance windows, rollback criteria, who approves and who executes.
- Infrastructure documentation. Inventory of hosts and containers, resource headroom, dependencies between services, known limitations.
- Monitoring and alerts. Node and container metrics, trigger thresholds, notification routes — the on-call engineer learns about a problem before the user does.
- Access and roles. Permission separation by hosts and container groups, the procedure for issuing and revoking keys.
- Runbooks for typical operations. Step-by-step scenarios: add a container, expand storage, move to another node, restore from a snapshot.
- Handover to operations. A joint run on your environment, review of emergency scenarios, and answers to the team's questions — before we disable our access.
The kit lives together with the infrastructure: when the topology changes, we update the designs and procedures rather than leaving them in the state they were at launch.
Case study: how we eliminated server sprawl and reduced downtime
The client came with a fleet of twelve machines: three environments, plus separate hosts for build and monitoring.
Settings were edited by hand, backups were taken selectively, and within a couple of months no one could answer how the production configuration differed from pre-production.
A typical failure: a task passes verification in the test environment and fails in production due to diverged library versions.
The cost of such a scheme is time. Recovery after a failure took hours because the sequence of actions was kept in one engineer's head. Any change on a live machine was irreversible: it could only be rolled back manually.
What we changed
Services were migrated to LXC containers, each configuration was described declaratively and placed under version control. Environment preparation ceased to be a manual operation: it is assembled from the description, and divergence between environments is visible before release.
Fault tolerance was ensured by distributing services across nodes — when one fails, containers come up on a free one without manual intervention.
What we achieved
| Metric | Before | After |
|---|---|---|
| Machines in the fleet | 12 | 4 |
| Recovery after failure | 4–6 hours | 15–20 minutes |
| Environment preparation | 1–2 days | 20 minutes |
| Services with autostart | some | all |
Manual edits on live machines stopped, and rollback to the previous configuration takes minutes. The engineer is no longer a single point of failure.
What to do if some services already run in virtual machines?
A hybrid infrastructure is not a transitional stage to get through as quickly as possible but an operating mode for years.
Some services are tied to a virtual machine: a license counts hardware binding, an application needs its own kernel or a specific version of system libraries. Breaking that for the sake of uniformity means weeks of debugging and a risk of downtime.
Therefore, we frame the question differently: what do we migrate immediately, what do we leave, and how do we separate them across nodes.
The selection criterion is simple. A container wins where a service does not store state on the local disk and does not require its own kernel: it starts in seconds and fits more densely on a node.
Anything that keeps data next to itself, works with direct device access, or is tied to a license stays on the machine — at least until a separate data migration project.
The coexistence of containers and machines requires separate placement of services. For them, we create separate node groups and labels so that the scheduler does not place a container on a hypervisor and does not take resources from machines.
We keep manageability shared: one service description layer, unified access policies, and one log collection — otherwise, instead of a hybrid, you get two disconnected environments.
The migration plan proceeds in waves of one or two services: a shadow run alongside the production one, traffic switching, and a rollback window to the previous machine.
The readiness criterion for a wave is startup time, no errors in logs for a day, and predictable behavior under load.
Technical concerns before moving to containers
Engineers rarely argue about containerization itself — questions begin in operations: how to move a workload without a downtime window, what happens when a node is updated, who gets access inside, and whether the system will survive a machine failure. Below are short answers to the four most common concerns.
How do you migrate containers without downtime?
Live migration moves a running container together with process state — downtime is reduced to traffic switching, usually seconds. For a stateless workload, it is enough to bring up a copy on another node and point the load balancer to it.
For containers with local data, a shared volume or storage replica is needed; otherwise, migration goes through a snapshot and a short unavailability window.
What happens to the workload when a node is updated?
Containers share the kernel with the host, so rebooting a node stops everything running on it.
The procedure is as follows: the node is removed from load balancing, containers move to the remaining machines, the kernel and containerization service are updated, and the node returns to operation. During this time, services live on other nodes, and guests are not restarted.
Who gets access inside, and how?
Containers run unprivileged: users inside are mapped to a separate range of identifiers on the host, so root in a container is not equal to root on the node. On top of that, there is a security profile and permissions at the containerization service level.
Access is granted to roles, not to everyone indiscriminately, and is recorded in configuration rather than in agreements.
Will the system survive a single node failure?
It depends on storage. With volume replication and liveness checks, containers automatically come up on surviving nodes — downtime is measured by switchover time. Without a replica, in-memory state is lost, and recovery proceeds from a snapshot.
At the start, we show which services survive a failure without loss and which do so with a stop, and we plan placement accordingly.
Let's discuss your task and come back with an assessment and implementation plan
To come back with an assessment and implementation plan, we need a minimum of initial information: how many nodes you currently maintain, what workload they carry, which services already run in containers, and which remain on virtual machines.
It is also important to understand the isolation requirements between environments, the backup procedure, and how updates are currently rolled out.
If part of the infrastructure is described only in the administrators' heads, that is also a valid starting point — we will work through it using accesses and configurations.
Based on the discussion of your task, you receive:
- a diagram of the current workload placement with failure points and bottlenecks;
- a target architecture: where LXC container virtualization is appropriate and where a full virtual machine is needed;
- a phased implementation plan — the order of service migration, maintenance windows, and readiness criteria for each step;
- a project estimate: labor inputs, timelines, and dependencies on your side;
- a procedure for backup, rollback to the previous state, and image updates;
- an option for infrastructure support after launch or handover to your team along with documentation and training.
Describe your task in free form — send the current design, a list of services, or simply tell us what is unsatisfactory in the current operation. We will come back with an assessment and implementation plan, and we will examine contentious points in detail.