Software RAID (mdadm) Setup for Servers

Why Software RAID (mdadm) setup on servers does not go to plan

Setting up mdadm often ends where the array has already assembled and is responding to requests. Formally, a software RAID on the server exists: the utility shows clean status, partitions are readable, data is in place.

But the array has no hot spare, the configuration is not fixed in a file that is read at boot, and no one has verified the device composition after a reboot.

The first software RAID failure does not manifest immediately.

Array degradation is silent: one disk drops out of the mirror, the kernel logs the event, but mdadm is not switched to notification mode — there is simply no one to deliver the message to, and the disk loss is discovered months later.

At the same time, it turns out that scheduled media checks have never been run: the log is empty, and unreadable sectors have long been growing on the second disk.

The cost of such a scenario is not one failed disk, but the entire array and dozens of hours for recovery.

Rebuild competes with production requests for I/O operations, service response time increases severalfold, and the window of unplanned downtime goes far beyond the regulations.

Worse, when the check is run for the first time exactly during an incident: the array rebuilds from a damaged member and replicates the error across all copies.

We break down how this is closed: from assembling an array with a pre-provisioned spare and predictable device names to scheduled checks with load limits and notifications that arrive before a failure becomes final.

What an assembled array gives: downtime and failure classes

A software array on mdadm is assembled so that when one drive fails, the service continues to run: requests go through the remaining disks, and the server switches to a reduced redundancy mode.

For the customer, this is the difference between "a nighttime call and recovery from a backup" and "a morning disk replacement according to the procedure". Service downtime at the moment of failure is measured in seconds of switchover, not hours.

The second effect is array recovery time. With a configured hot spare, the rebuild starts automatically, without waiting for an engineer and manual replacement selection; the window is reduced from several hours to tens of minutes, and all this time the service serves requests.

Entire classes of failures disappear altogether: physical death of a drive, read errors on damaged sectors (covered by a copy from the mirror) and gradual degradation, which is caught by regular integrity checks.

It runs on a schedule, so inconsistencies surface before they become a failure, not at the moment of peak load.

What the array does not cover is failure of the controller, cable, power supply, and erroneous administrator actions. Backups remain mandatory: the array is responsible for continuity, not for data recovery. We state this boundary upfront so as not to create a false sense of security.

The third measurable result is planned maintenance without downtime: a disk is replaced on a running server, integrity checks do not interfere with serving requests, and the performance drop in reduced redundancy mode is visible in monitoring in advance. The server survives a drive failure as a routine event, not as an incident.

Options for organizing a disk array: comparison of approaches

Three approaches solve one task — to survive a disk failure without stopping the service. The difference is who manages mirroring and what happens when replacing a drive: a system-level array, a hardware controller, or an external block volume with replication.

The scheme is chosen by the level of redundancy, and it is determined by the number of disks and acceptable recovery time. Mirroring provides predictable rebuild, parity provides more usable capacity with the same disk fleet.

Neither option cancels the other: the array protects against drive failure, but not against failure of the machine itself.

The practical choice comes down to three questions: whether portability of the set between servers is important, whether rebuild time during disk replacement is critical, and whether the array must survive failure of the machine itself.

How implementation proceeds: audit, scheme, assembly, verification

The work order is structured so that the array does not have to be rebuilt from scratch after the very first failure.

First we record the actual state of the server, then design a redundancy scheme for the load profile — and only after that do we assemble the array and verify that it actually recovers, not just "comes up".

1. Audit of server configuration and drive inventory. We look at disk models and wear, partitioning, file systems, what is already assembled into arrays, which partitions participate in booting, what the I/O load is, and whether there are free bays for replacement without downtime.
2. Design of the array scheme. We choose the redundancy level and stripe size for the real read-write profile, the number of working and spare disks, and partition service and boot partitions so that the system comes up even with a failed drive.
3. mdadm assembly. We create the array, partition and mount by identifiers rather than device names, and add dependencies to the bootloader and initial boot image so that the order of detected disks does not break anything.
4. Recovery verification. We intentionally remove a drive, see how the rebuild proceeds and how long it takes, whether the service behaves predictably during this time, and whether the spare returns to service automatically.
5. Monitoring and maintenance. We configure alerts for array degradation, keep an event log, and schedule planned parity checks — this way block mismatch is caught before failure, not during recovery.
6. Handover of procedures. We provide documentation on the scheme, a step-by-step drive replacement procedure, a "what to do on degradation" matrix, and the verification order after any changes.

Configurations, array schemes, and maintenance procedures

After the array is assembled, the infrastructure retains not only a working configuration but also a set of artifacts by which it can be reproduced and maintained without the author.

Without them, disk replacement turns into a blind teardown: it is unclear which drives are part of the array, how it will assemble after a reboot, and who is authorized to change its settings.

• Array configuration file: disk composition, redundancy level, block size, and binding to drive identifiers rather than device names. After a reboot, the array assembles itself regardless of disk detection order.
• Drive connection map: controller, slot, serial number, and role of each disk; hot-swap slots are marked separately. An engineer removes the right drive without touching the neighboring array.
• Disk replacement procedure: what to check before removal, how to introduce a new drive into the array, which states to watch during rebuild, and when the operation is considered closed.
• Access policies: who is allowed to perform array operations, how permissions are restricted, and where the command log is written. Accidental disk exclusion stops being irreversible.
• Order of scheduled checks: polling of drive and array status on a schedule, checksum verification, thresholds at which an engineer intervenes before failure.
• Recovery instructions: degradation and complete array loss scenarios, the assembly order on a new server, and points where data is lost irretrievably.
• Server documentation: the array's relationship to partitions, mount points, and boot, and what changes during expansion.

The set lives together with the server in a configuration repository: the next disk replacement follows the document, not memory.

Practice: array assembly and recovery after degradation

An illustrative project is a service with a database and file storage, an array on four disks with parity, assembled on mdadm.

There was no hot spare, and no one polled the state: the dropped disk was discovered in the morning from write errors, when the array had already been running degraded for several hours.

The rebuild ran under live load, the disk queue was busy, service response time increased severalfold, and some jobs had to be restarted manually.

What was changed

As a first step, we brought the array out of degradation and recorded at what level it is stable. Then we added a hot spare and enabled state polling: the utility monitors events and sends a notification to the on-call engineer on any change in composition.

We separately configured the change log — a returning disk is brought back into the array by rebuilding only the missing blocks, not by a full rebuild. Parity checks were put on a schedule: they detect "silent" read errors before they surface during recovery.

Result in numbers

The share of incidents that the on-call engineer closes without traveling on-site grew to the majority: the array picks up the spare itself, and the engineer gets a ready picture of disk composition and state.

mdadm or hardware controller: what to choose for a server

Short answer: the choice is determined not by speed, but by how ready you are to depend on a specific board model. The software array mdadm is assembled by the kernel on top of ordinary drives; the hardware one lives inside the controller with its own cache and logic.

Mirroring or parity is obtained in both cases, but they behave differently on failure, replacement, and migration.

Replacing a drive in a software array proceeds without stopping the service: the faulty disk is removed, a new one is introduced into the array, and data is restored under live load.

Rebuild speed is limited by synchronization parameters — otherwise it takes bandwidth from the application and stretches response time. With a hardware controller, this work goes to the board, but progress is visible only through its utility.

Array portability is the main divide. With software, the layout is stored on the drives themselves: move the disks to another server and bring up the array with a single assembly command, without searching for compatible hardware.

A hardware array is read by the same or a compatible controller — if the board has failed and there is no replacement of the same model, data will have to be recovered seriously.

The limitations of a software array include performance drop during rebuild, lack of non-volatile cache, and monitoring with your own tools: array state, temperature, and drive failure prediction are checked by separate services.

In return, you get transparency — you can see which block lies on which disk, and there is no firmware black box. In practice, we assemble most servers on a pass-through adapter with a software array, and leave hardware cache for tasks with dense writes.

Addressing common doubts: degradation, disk replacement, monitoring

Array degradation is not an incident but a standard mode: it continues to serve data, redundancy simply disappears until disk replacement. The danger is not the drive failure itself, but a second failure during the recovery window — then the entire set is lost. Below are answers to questions that most often come up when handing an array over to operations.

Can a disk be replaced without reinstalling and stopping services?

Yes, if the enclosure supports hot swap: the failed drive is excluded from the array, a new one is added as a spare, and recovery proceeds in the background.

The array remains in operation, but until synchronization finishes, redundancy is reduced — we try to shorten this interval by limiting rebuild speed so that it does not crowd out production load.

How can you be sure that the array is actually intact and not "just responding"?

A periodic check reads all blocks and verifies them against parity, showing the number of mismatches. We put it in a nighttime window and investigate each mismatch: a single one is usually resolved by rewriting, a systematic one points to a degrading drive or a faulty cable.

Is separate monitoring needed if the array recovers itself?

It does not recover itself — it survives. Without notifications, degradation can remain unnoticed for weeks until the second disk fails.

We configure notifications for degradation, failure, and rebuild completion events and supplement them with drive health monitoring so that the signal arrives before data loss, not after.

Let's discuss the task: we will return with an estimate and work plan

Setting up RAID for a task begins not with the array creation command, but with inventory: how many drives, where the system will be, where the data will be, whether there is a hot spare, and how disk replacement is serviced. The more accurate the inputs, the fewer revisions to the work plan after the start.

What we need from you to start:

• list of servers and their roles: where the system is, where the data is, where the backups are;
• drive composition by slot, including unused and those requiring replacement;
• required fault-tolerance level: mirror, parity, or a combination of mirrors and striping;
• availability requirements: maintenance window and acceptable time for the array to run degraded;
• access: administrator account, server management console, monitoring system;
• constraints: partition encryption, virtualization, cluster add-ons, regulatory requirements.

In response, we will return an array layout scheme with partitioning and boot entries on both drives, a list of RAID setup tasks for the task — from partitioning to disk replacement procedure, integrity checks, and alert thresholds, as well as a timeline estimate with downtime windows.

We will agree on the work plan before changes on production servers so that every step remains reversible.

Describe the task and current configuration — we will discuss it and return with an estimate and plan.