Setting up libvirt/virsh for managing KVM virtual machines

Configuring libvirt: failure points in machine management

A virtualization node goes down at the most inconvenient moment — after a scheduled kernel update, storage pool overflow, or network storage failure. Some machines on it remain in a state that libvirt considers running, even though the guest systems no longer respond.

Worse, console access is lost: there is a single entry point and it lives on the same node, so along with the host, the ability to manage virtual machines disappears. Diagnostics turns into walking through hosts one by one via virsh right in the terminal.

The second, less noticeable problem is inconsistent configurations. Domain descriptions are edited pointwise on one node to “bring the service up faster”: changing disk type, limits, network binding.

These manual edits bypassing the regulations are not recorded anywhere, and a month later no one remembers how the production node differs from the standby one.

During live migration, stuck machines surface: the domain on the new node is listed as active, but resources are not allocated for it, and it silently does not work until manual investigation.

We examine how this is closed: a unified fleet description, managed console access, and state reconciliation that catches discrepancies before they become an incident.

What a careful hypervisor configuration changes

A careful hypervisor configuration is not “install and run,” but a set of decisions, each of which pays off over its own horizon.

We check the places where careless libvirt setup turns into manual work: storage pools, templates, limits, startup order, backups, and reproducibility.

• Storage is placed in a separate pool on a fast volume: cloning from a template takes seconds, and deploying a new machine is one operation instead of manual installation.
• One updatable template with a guest agent: machines receive the same base configuration, and no post-first-boot tuning is needed.
• Resource limits and priorities are distributed across machine groups: a node holds more guests without “silent” overloads, and density grows predictably, not until the first peak load.
• Autostart and startup order are explicitly described: after a node reboot, the infrastructure comes up on its own, and no operator at the console is needed.
• Observability over machine state, storage, and resources: degradation is visible before node failure — the “learned from users” class of incidents goes away.
• Configuration copies and scheduled snapshots: recovery after node failure is measured in minutes, not in searching for surviving disks.
• Node and machine descriptions are stored declaratively next to the code: a new node comes up repeatably, without “make it like the first one.”

We check these points before handover to operation. The result is visible in three metrics: deployment time, number of machines per node, and recovery time after failure.

libvirt configuration options: comparing approaches

The libvirt management method is chosen not by fleet size, but by the number of nodes and by who changes the configuration and how often.

Local virsh accesses the daemon on a single node directly — convenient for debugging, one-off tasks, and test benches where one engineer does everything.

As soon as there are more than two or three nodes and machines migrate between them, direct access becomes a source of discrepancies: storage pools, bridges, and virtual network models are configured manually and gradually drift apart.

Automation on top of such inconsistency breaks at the first step — that is why the limitations of the options are more important than their convenience.

Three working options

The boundaries between the options are fluid: local access remains a debugging tool within a centralized scheme, and automation is built on top of a shared machine description.

We fix the choice by the number of nodes, the required reproducibility, and the speed of bringing up a new machine.

Implementation procedure: from audit to handover

The work order is structured so that libvirt configuration relies on the placement scheme, not the other way around.

Otherwise, hypervisors are configured blindly: guests end up on the wrong nodes, pools are overfilled, and migration runs into configuration discrepancies between hosts.

1. Infrastructure audit. Inventory of nodes and guests: which networks and storage pools are defined, where parameters were edited manually and diverged between hosts, which machines do not come up after a reboot. The output is a list of discrepancies and bottlenecks with priorities.

2. Machine placement scheme. We distribute guests across nodes taking resources into account, mark up networks and storage, and determine what lives on a shared pool and what lives on local storage. The scheme is fixed before configuration — it sets the profile parameters.

3. Node and policy configuration. A unified libvirt profile on all hosts: networks, pools, access rights, guest autostart, behavior on power loss. Manual edits are closed out, and the configuration is brought to a single baseline.

4. Failure testing. We simulate node, disk, and network failure: we look at how the guest behaves, whether migration proceeds, and how quickly the service returns to operation. We close the weak points found before handover.

5. Handover to operation. We provide regulations, a placement map, and instructions for standard operations — creating a machine, migration, taking a backup, and responding to failure.

What the customer retains: schemes, policies, regulations

After the stand is delivered, the customer retains not only a working hypervisor but also a set of artifacts that its engineers can use to bring in new nodes, grant access, and recover after a failure without contacting us.

Everything configured in libvirt is recorded in text form and versioned — not “in the administrator’s words.”

• Node configurations — domain descriptions, templates for typical roles, CPU and memory pinning parameters: a new node is brought in by copying a template and editing a few fields.
• Network schemes — topology of bridges and virtual networks, VLAN binding, addressing, outbound connectivity; it is recorded which traffic goes where.
• Storage schemes — pools, placement of images and data by tiers: where the fast disks are, where the archive is, how snapshots are attached.
• Access policies — roles and groups, permission separation rules: who may start, shut down, and migrate domains, and who only views the console and logs.
• Engineer instructions — scenarios: create a domain, extend a disk, migrate to another node, take and roll back a snapshot.
• Recovery regulations — procedure for node failure: where configurations come from, how disks are deployed, and what recovery time and recovery point are considered normal.
• Change log — configurations in a versioning system, revisions, who changed them and when.

We separately walk through the regulations with the customer’s engineers: a document that no one has practiced remains just a file.

Case study: how machine density per node grew

The customer is an internal platform of a service company: about ten virtualization nodes for products and test environments, with machines deployed manually.

The inventory lived in a spreadsheet, domain parameters were edited with commands directly on running nodes, and after migration configurations were checked by eye.

For 40–60 manual operations per month, there were 3–4 on-call escalations, and almost each one involved service downtime of 40+ minutes.

What was changed

Management was moved to libvirt with declarative descriptions: domains, networks, and storage live in a version control system, and the same profile is deployed to each node.

Typical roles received templates, so a new machine comes up from a description rather than being assembled manually from a checklist.

Storage was re-partitioned into pools with thin provisioning and release policies, and CPU and memory limits were set according to the load profile, not “with headroom for every machine.” Any change is tested on a test node and rolled back to the previous description in a single operation.

Metrics before and after

Density grew because profiles were aligned and double reserve was removed: some nodes were freed up and became hot standby instead of buying new hardware. Each incident is now investigated using the description from the change history — you can see what changed and when.

How centralized management differs from manual commands

The difference is not in convenience, but in where the configuration state lives.

When working through libvirt, a virtual machine is described declaratively — disk layout, network interfaces, resources — and this description is applied to the node as a whole, not reproduced step by step.

Manual commands on nodes define the same thing, but only as a sequence of actions: after several edits in other people’s sessions, no one can say how the current node differs from the original intent.

Manual commands win where a single operation is needed on the spot: shut down a stuck machine, view a log, bring up a stand on a single node. They lose at scale — with two dozen nodes, the same steps are repeated with discrepancies, and configuration drift accumulates unnoticed.

A separate cost is recovery: a node cannot be reproduced from the command log, and it has to be reassembled from scratch.

Centralized management moves changes to a single entry point. Descriptions sit next to the inventory and edit history, so repeatability of operations no longer depends on who exactly logged into the node today.

One operation yields the same result on the first node and the last: rollback is returning the previous description, not searching for the guilty command in history.

Centralization loses in exactly two cases: a single node in a lab and emergency diagnostics when the management layer is unavailable.

That is why direct access to nodes is retained — but every manual change is then transferred into the description; otherwise, the next apply will overwrite it, and the discrepancy will become unpredictable.

Technical concerns: examining bottlenecks

libvirt is the management layer over KVM: the daemon maintains a registry of domains, stores their descriptions in configuration files, and provides a unified interface for virsh, programmatic calls, and control panels.

Implementation begins not with reinstalling the hypervisor, but with inventorying what already runs on the nodes.

Below are the concerns that most often come up at the start: compatibility with the current fleet, migration risks, behavior during node failure, and site requirements.

What happens if a node fails?

libvirt itself does not “revive” machines — that is handled by the cluster layer: node liveness checks, anti-affinity, and restart rules. We describe in advance which machines migrate automatically and which wait for a manual decision — failure stops being a surprise.

What are the requirements for the site and nodes?

Nodes with hardware virtualization and shared storage are needed if machines are planned to move between them, plus access for management services. We separately check that disks and network devices are correctly visible from guest systems.

Important limitation: the management layer does not replace backup and monitoring — we design those separately; otherwise, machine migration is left without a rollback point.

Let’s discuss the task and come back with an estimate and plan

So that the estimate is based on facts rather than assumptions, we collect an inventory and constraints at the input. Send what you already have at hand — that is enough to understand the scale and come back with a work plan and implementation scheme.

• List of hypervisors: how many hosts are in the environment, which virtual machines run on them, and which of them are critical for services.
• Current management: whether you work via virsh manually, with scripts, or through a panel — and what exactly is getting in the way now.
• Storage: where images and volumes are stored, whether libvirt pools are used, and whether snapshots and thin provisioning are needed.
• Network: bridges, VLANs, addressing, whether port forwarding and isolation between machine groups are needed.
• Templates: how new machines are created, whether there are reference images and XML domain descriptions.
• Access: how you are prepared to open access to hypervisors — a separate gateway, temporary keys, read-only at the first stage.
• Constraints: maintenance windows, rollback requirements, and backup retention rules.

Next, we clarify what is not visible from the correspondence: the placement scheme, rights, the order of rolling out changes, and checkpoints after each step.

As an output, you receive a stage-by-stage work plan, an implementation scheme, and the scope of infrastructure work — with explicit readiness criteria.

Describe the task in free form — we will come back with an estimate and a plan.