Hardware RAID Controller: Where Disk Reliability Is Lost
A hardware RAID controller is designed as insurance: a disk fails, the array keeps running, the service doesn't notice. In practice, a disk failure on a server is far from the most common downtime scenario.
Far more often, infrastructure goes down where the controller is configured "by default" and no one has checked what exactly it does with the data.
RAID array degradation develops unnoticed. One disk drops out, the array goes into degraded mode, but the service responds as usual — and the signal goes unheeded.
The next failure, now on a second disk, turns degradation into data loss: the rebuild doesn't finish in time, and the recovery window depends on policy and load, not on the duty engineer's wishes.
A separate category is controller cache errors: write-back enabled without protected power turns a power failure into lost transactions, and a battery or supercapacitor that has lost capacity quietly changes write behavior.
RAID write policies are where expectations and reality most often diverge. A controller with write-back speeds up writes but requires a healthy cache; write-through is safer but slower.
If the policy wasn't coordinated with the load profile and wasn't verified after equipment replacement, a server can run for years in a mode that no one consciously chose.
The cost of these failures is not abstract "instability" but hours of downtime and recovery from a backup where a hot-swap of a disk would have sufficed.
That's why array reliability is determined not by the controller model, but by how the cache, policies, and monitoring are configured. Let's break down how this is addressed in practice.
What Proper Configuration of a Hardware RAID Controller Delivers
RAID Levels: Choosing for the Workload Profile
The array level determines both fault tolerance and the cost in disks. RAID 10 survives the loss of one disk in each mirrored pair and delivers linear read scaling — suitable for databases and virtualization, where write operations matter.
RAID 6 survives the failure of two disks simultaneously, but pays for it with double parity: writes are slower, and rebuilding after a disk replacement takes longer.
RAID 5 on large drives is risky today: the vulnerability window during recovery is measured in hours, and a second failure during that period collapses the array.
Cache with Battery and Write Policies
The controller cache with a battery backup unit (BBU) is not an option but a prerequisite for correct operation.
With a healthy battery, the write policy switches to write-back through the cache: acknowledgment arrives after the data is committed to protected memory rather than to disk.
This eliminates a whole class of failures where data already acknowledged to the application is lost during a power outage. Without a battery, the controller is forced to operate in write-through mode — write performance drops by a factor of several.
Array Rebuild Time
Rebuild speed is governed by the rebuild priority and the disk size. If you set the priority to maximum, the array will rebuild faster, but the production workload will suffer.
A reasonable compromise is dynamic priority: the controller allocates spare bandwidth to the rebuild and throttles it during peaks. In practice, this shortens the vulnerability window from many hours to just a few without degrading the service.
| Option | When It Fits | Limitations |
|---|---|---|
| RAID 10 | Databases, virtualization, intensive writes | Half the capacity goes to mirrors |
| RAID 6 | Archives, file storage | Slow writes, long rebuild |
| RAID 5 | Only small disks, non-critical data | Risk of a second failure during recovery |
| Cache with BBU | Any workload with writes | Battery must be replaced as it wears out |
We handle array configuration and health monitoring via MegaCli — it reports battery status, rebuild progress, and controller events in machine-readable form, which lets us catch degradation before failure.
How we configure a RAID controller: stages of work
Configuring a hardware RAID controller is not "enabling mirroring in the firmware," but a sequence of decisions, each of which subsequently determines the server's behavior under load and in the event of a disk failure.
We start with an audit of the RAID configuration: we look at which disks are physically installed, how they are connected to the controller, what state the cache battery (BBU/CacheVault) is in, which arrays are already assembled and how the logical volumes are partitioned.
It often turns out that some disks are running in JBOD mode, the cache is disabled due to a dead battery, and the array level was chosen "by eye" — these are the hidden causes of drops in disk operations.
Next, we design the array scheme for the task: where RAID 10 is needed for write speed, where RAID 6 is sufficient for capacity, what stripe size to set for the load profile, whether to keep a hot spare, and how to configure the rebuild policy.
Separately, we determine what to do with the cache: write-back with battery protection or write-through if there is no battery.
- Audit of the current configuration — inventory of disks, controller, cache battery state, existing arrays, and firmware version.
- Designing the array scheme — selecting the RAID level, stripe size, cache policy, and presence of a hot spare for the specific workload.
- Server configuration — creating arrays and logical volumes, partitioning, data migration, or clean installation.
- RAID testing — bench testing: simulating a disk failure, measuring rebuild speed, verifying that the array reassembles and data is not lost.
- Load testing — running the real operation profile to make sure the chosen level and cache deliver the required performance.
- Handover of documentation — array scheme, controller parameters, disk replacement procedure, and instructions for monitoring the state.
- Ongoing support — we set up alerts for array degradation and battery state so that a disk failure does not come as a surprise.
The key stage is bench testing the RAID before commissioning. We deliberately pull out a disk and watch how the array behaves: how long the rebuild takes, whether performance drops to an unacceptable level, whether the hot spare works correctly. This is cheaper than catching array degradation in production at the worst possible moment.
How does hardware RAID differ from software RAID?
Hardware RAID is a separate controller with its own processor, cache, and battery module that takes I/O operations upon itself and presents the operating system with a ready-made logical volume.
Software RAID is assembled by the OS from ordinary disks: parity calculations and mirroring are performed by the central processor, and system memory serves as the cache.
The difference manifests not in "disk speed" but in where exactly the resource is spent and what happens during a failure.
For writes in arrays with parity, a hardware controller maintains a battery-backed cache: acknowledgment comes from the non-volatile buffer, and data lands on the disks later, in batches.
A software array is forced to either wait for the physical write or use its own journal, and the CPU calculates every parity block.
Under mixed workloads — database, virtualization, file storage — this is visible in response latency and in how many cores remain available to the application.
Reliability is decided not by "hardware-ness" itself but by behavior during failures. A controller supports hot swap, background rebuild, patrol read, and migrating the entire array to another server; a battery-backed cache survives power loss without losing acknowledged writes.
A software array depends on the bootloader and the integrity of the OS: if the system fails to boot, assembling and checking the array is more difficult, and recovery requires manual work with utilities.
Software RAID, however, is not "bad" — it is sufficient where the workload is sequential and latency requirements are soft: a department file server, backup storage, a development stand.
It does not require a separate board, moves along with the disks between identical machines, and is transparent for monitoring.
RAID comparison in this case comes down to the cost of failure: if downtime during a rebuild is acceptable and there is a surplus of CPU cores, a software array gets the job done.
The choice is determined by three questions: what share of operations are writes, whether there are response latency requirements, and what will happen if the server fails entirely.
The answers to them are the boundary between hardware raid and software raid — beyond that, it is already a matter of the specific configuration.
Case: array recovery after controller failure
A server at the customer's site ran for several years without complaints, until the controller started losing one disk after another: first the array went into degradation, then the firmware stopped responding to service requests.
Formally, the data was in place, but any reboot could have ended in an unreadable set of disks — and this node hosted the file service and the accounting database.
What was there before intervention
An array of six disks in a parity configuration, one drive in a failed state, the second with a growing number of reallocated sectors. A backup was taken once a day, the recovery window from it was about eight hours, and that was without accounting for manual reconciliation.
The controller, meanwhile, no longer allowed the array configuration to be exported: the service utility saw it as an unrecognized device.
What was done
As the first step, a full image of each disk was taken onto separate media — so that any further actions would be performed on copies, not on the originals.
Then the array metadata was transferred: the disk order, stripe size, parity type, and service area offset were recovered from the controller logs and the labels on the drives themselves.
The new controller accepted the configuration, the array assembled in recovery mode, and parity was recalculated onto the replacement disk.
What became
| Metric | Before | After |
|---|---|---|
| Service availability | downtime due to the failure | planned window for replacement |
| Recovery time | about 8 hours from a copy | parity recalculation without stopping the service |
| Data integrity | in question on every reboot | confirmed by checksum verification |
| Failure margin | one disk | a full set of healthy ones |
Monitoring of the state of the drives and the controller was set up separately: polling every few minutes, notification at the first reallocated sectors, rather than when the disk has already dropped out.
This is the class of tasks where MegaRAID and similar platforms provide an exportable event log — it was precisely from it that the original array configuration was recovered, rather than assembling it anew from scratch.
Compatibility, migration and rollback: what we clarify before the start
Before the start, we always check three things: RAID compatibility with the current hardware, the data migration scenario and the rollback plan.
In practice, the most common issue that comes up is incompatibility of the controller firmware with the motherboard or backplane — an externally suitable hardware raid controller refuses to initialize the disks, and the server goes into downtime at the stage when the data has already been partially transferred. That is why we check raid compatibility against the vendor matrix before purchase, not after.
Data migration is the second source of risk. If the array is already built on a software solution or on a controller from another manufacturer, a direct transfer of disks will not work: the metadata is incompatible.
In that case we build a plan through intermediate storage — take an image, build a new array, return the data and verify checksums for each volume.
What if something goes wrong after switching to the new array?
We record the rollback plan in writing before the work begins: we save the original controller configuration, we do not disassemble the old array until the data on the new one has been confirmed as verified, and we keep the boot media with the previous layout. Rollback takes minutes — we switch the boot back, and the array comes up in its original state.
What about access to the server? We need credentials to log in to the controller, access to the management console and physical access to the rack if a battery replacement or cable reconnection is required. Without this, some operations cannot be performed remotely.
What will be included in the raid documentation? The array diagram with levels and disk distribution, the controller parameters, the procedure for replacing a failed drive and instructions for recovery after a failure — so that your team can act without us.
You receive:
- a diagram of the assembled array with levels and hot-spare;
- the controller configuration in an exportable form;
- a rollback plan with step-by-step commands;
- a procedure for replacing a disk and rebuilding the array;
- a post-handover verification checklist;
- a compatibility matrix for your hardware;
- a migration log with checksums.
Describe the task — current hardware, data volume, availability requirements. We will come back with an estimate and a work plan.