Configuring VMware vSphere: where the cluster breaks
A cluster more often breaks not during installation, but during high availability configuration: the host is there, storage is connected, virtual machines are running — until the first hypervisor failure.
At that moment it turns out that restart priorities are not set, and the standby host either does not exist or lacks resources. Infrastructure downtime is then measured not in minutes, but in the time until the on-call engineer notices the problem.
A single point of failure almost always hides in one of four places. A separate host that carries the load but has no pair in the cluster. Shared storage connected via a single path: failure of a switch or port makes volumes unavailable to all nodes at once.
A management node or directory service in a single instance. And a network where management, migration, and virtual machine traffic goes over the same physical channels.
Manual restart of virtual machines looks safe as long as there have been no failures. In practice, recovery depends on a person: night, weekend, vacation — and the service is down until morning.
Add startup order: the database must come up before the application, otherwise the application will go into a restart loop and add another half hour to the downtime.
And the absence of failure tests. Placement rules, restart priorities, path redundancy, and network isolation are configured once and not tested — and a real hypervisor failure reveals a discrepancy between expected and actual behavior.
We explain how this is closed by configuring VMware vSphere: from placement scheme to drills with forced node shutdown.
What changes after implementation: downtime, recovery, failures
A cluster on VMware vSphere is evaluated not by the number of configured nodes, but by how services behave at the moment of failure. Below are four metrics that we take before and after implementation and use to accept the work.
Service recovery time: minutes instead of hours
When a physical node fails, virtual machines automatically come up on the remaining ones — the service returns to operation in a few minutes, not in the time it takes for the on-call engineer to reach the site and manually reconfigure startup. For critical systems, a restart priority is set: what others depend on comes up first.
Eliminated class of hardware failures
A whole class of accidents related to a single node goes away: failure of a power supply, disk, network port, or storage channel no longer stops the service.
The arbitration node preserves quorum when the primary site is lost — the cluster does not split into two halves and does not start competing for resources.
Automatic failover without manual operations
The share of manual actions by the on-call engineer during a failure drops to almost zero: restart, node selection, and resource return are performed by cluster rules.
A person remains in the process only where a decision is needed — for example, to roll back a release after an unsuccessful update.
Service availability and on-call workload
Availability is measured by fact: the proportion of time the service responds grows because short failures stop turning into downtime. Night alarm wake-ups are reduced — most incidents are closed by automation before the on-call engineer opens the phone.
vSphere high availability modes: what to choose
The cluster mode is determined by two constraints: where virtual machine data resides and how far apart the nodes are. Hence the forks — local disk or shared storage, one site or two, one management node or a quorum.
| Option | When it fits | Limitations |
|---|---|---|
| Separate nodes on local disks | pilot, test environment, isolated service | manual migration, downtime until recovery from backup |
| High availability cluster on shared storage | primary environment from three nodes | shared volume and redundant access paths required, sensitivity to latency to the array |
| High availability cluster and placement planning | heterogeneous load, different service classes | compatible instruction sets on all nodes, placement rules set manually |
| Stretched cluster across two sites | continuity during site failure | third witness site, low latency between sites, synchronous replication |
The number of nodes is subject to quorum: we take at least three for a high availability cluster so that the loss of a host does not leave the management environment in an uncertain state.
We separate the network into management traffic and storage traffic — with a shared volume, this removes bandwidth competition and false monitoring triggers.
Compatibility is limited by four checks before deployment: array support for multipathing, node boot mode, drivers for the selected platform branch, and correct quorum between sites.
If at least one fails, the mode is downgraded — it is better to find this out during design than during a failure.
Implementation procedure: from audit to handover to operation
A cluster is not assembled from a standard template: the configuration depends on the number of hosts, storage class, and load profile. Therefore, we fix the work order before the start, and each stage ends with a verifiable artifact, not a report on the fact of work performed.
- Infrastructure audit. We inventory hosts, network segments, arrays, and current virtual machines, and take a dependency map between services. The output is a list of bottlenecks: overloaded storage, single paths to the storage system, lack of redundancy on switches.
- Node placement scheme. We distribute hosts across racks, power zones, and network pairs so that failure of any one element does not bring down the cluster. We calculate resource reserve for the N+1 rule and balancing rules.
- Platform preparation. We mark up networks: service traffic, migrations, access to storage via iSCSI or NFS, separate VLANs, coordinated frame size. We configure storage profiles and high availability policies.
- Cluster configuration. We enable high availability with reserve control, set heartbeat storages, priorities and restart order for virtual machines, and node isolation upon loss of communication.
- Test bench verification. We run test loads, planned migrations, and storage degradation, and measure response time under peak. We make edits before moving to production.
- Failure drills. We take turns shutting down a host, rack power, breaking an uplink, and simulating array unavailability. We record the actual time of service restart and data loss.
- Handover to operation. We provide maintenance regulations, recovery maps, configured monitoring and alert thresholds, conduct training for the on-call shift, and agree on maintenance windows.
Documentation, diagrams, and access policies
The cluster is handed over not as a "black box": along with the working nodes, you receive a set of materials according to which it can be maintained, expanded, and restored without us. Below is the minimum that we document at the end of the work.
- Placement scheme. Fault tolerance groups, where cluster data resides, what capacity reserve remains upon failure of one node, and how machines are distributed across hosts.
- Network diagrams. Separation of management, migration, storage access, and guest segment traffic, backup paths, and link aggregation.
- Configurations. Exports of node settings and the common management layer with notes on what was changed relative to factory values and why.
- Administrator roles and rights. Separation by areas of responsibility: network, machine startup, log reading. Service tasks run under separate accounts.
- Access policies. Entry points to management interfaces, mandatory multi-factor authentication, action logging, disabling of idle accounts.
- Maintenance regulations. Frequency of updates, checking disks and sensors, revision of snapshots and free space, procedure for agreeing on a maintenance window.
- Recovery regulations. Order of bringing up, storage locations for configuration backups, verification of recovery on the test environment.
- Health monitoring. Critical events and thresholds, who receives alerts, typical actions of the on-call shift upon node or path failure.
Case study: cluster withstood node failure on a workday
The customer is a manufacturing company where a virtualization cluster on VMware vSphere of six nodes served about 120 virtual machines: accounting systems, file exchange, analytics for workshops.
Failure of one host before the work meant manual troubleshooting — the on-call engineer brought up virtual machines on surviving nodes, downtime during working hours reached 40 minutes, and the startup order depended on who noticed the problem first.
The cluster was reassembled for automatic failover: high availability with restart priorities was enabled, a share of resources was reserved for restart so that surviving nodes would reliably take the load, and availability signals were moved to a separate network segment.
Virtual machines were distributed into priority groups — first databases and controllers, then secondary services.
Failure in the middle of a workday
A few weeks after handover, a node lost power in the middle of a shift. The cluster switched machines automatically: the critical environment was available in 2 minutes, the remaining services — in 4, and the startup order was maintained without human involvement.
Metrics before and after: downtime of critical services — 40 minutes versus 2, full recovery — an hour and a half versus 4 minutes, the share of failures handled by the cluster without an engineer — 100%.
Planned maintenance now runs without a maintenance window: a node is removed from the cluster, updated, and returned, while the load moves to the others.
How does a high availability cluster differ from backup?
These are two different mechanisms with different tasks, and one does not replace the other.
A high availability cluster is responsible for continuity of operation: when a physical server fails, its virtual machines automatically come up on live nodes, and the service continues to respond.
Backup is responsible for data protection — the ability to return the system to a state at a previous point in time.
The cluster works with the current state: a shared resource pool in VMware vSphere, load distribution, automatic restart upon failure. Downtime is reduced to the time it takes to bring up a virtual machine — minutes instead of hours while engineers repair hardware.
But the cluster does not protect against logical errors: if a database is deleted by mistake, a ransomware infection arrives, or an update corrupts data, the replica will honestly repeat this on all nodes — there will be nothing to roll back to.
Hardware failure and data loss are different classes of failures, and protection from the first does not cover the second.
A backup is stored separately and captures the state at the time of taking: it can be deployed on the same or another cluster, a separate copy can be brought up for testing, one file or the entire system can be restored.
Therefore, we plan both layers at once — the cluster, snapshots for short operations, and copies to independent storage on a schedule.
We write down recovery scenarios and test them in drills: node failure, site failure, accidental deletion, data corruption. Each has its own recovery source and its own time within which the service returns to operation.
Technical questions asked before work begins
Before the assessment, we go through four questions that most often change the cluster scheme. The answers are short, but they determine whether additional equipment will be needed and how many maintenance windows to include in the preparation plan for implementation.
Will the equipment already in the rack be suitable?
During the preparation stage, we take an inventory: virtualization support in the firmware, memory capacity for the required machine density, separate network ports for storage traffic and live migration.
Based on the result, we say directly which nodes will go into the cluster and which will remain for the test environment.
Is it possible to build high availability without an external storage system?
Yes. Local disks of each node are combined into a fault-tolerant pool, and data is replicated between servers. If one node fails, machines come up on the others, but free space is reserved differently — we take this into account when planning capacity.
How does planned maintenance proceed without downtime?
One node at a time: we move virtual machines to free ones, remove the node from the cluster, update the hypervisor and firmware, and return it back. We fix the node order and maintenance windows in advance to avoid peak loads.
How to make sure failover actually works?
We check not by indicators, but by simulating a failure: we de-energize a node or break the network and see how long it takes for critical services to come up and whether data was lost. The result is a report with measured recovery time and a list of issues.
Let's discuss the task: what is needed for an assessment
To return not with general words, but with a solution scheme and work plan, we need a snapshot of the current infrastructure. Below is the minimum set; if something is missing, tell us right away, and we will advise how to collect it.
- Cluster composition and load: number of physical hosts, their generation, placement across racks and zones, number and profile of virtual machines, peak values for CPU, memory, and disk.
- Storage: local disks, shared storage system or software-defined, what data protection levels are configured and how backups are organized.
- Network: switching scheme, separation of management, machine, migration, and storage traffic, availability of redundant channels and backup paths.
- Downtime requirements: allowable service unavailability time, recovery points, what is critical and what can be restarted manually.
- Access for survey: read-only accounts for hosts and the management system (VMware vSphere), access to storage and switch interfaces, a window for taking metrics.
- Target task and constraints: new deployment, migration, environment separation, or platform update; work windows and security rules that must not be violated.
From this data, the cluster composition, placement scheme, and high availability policies take shape. Describe the task and send what you already have — we will return with a solution scheme, work plan, and assessment.