Software RAID on Linux with mdadm: Practical Setup, Maintenance, and Recovery Guide

Table of Contents

Context & Why: When and Why Software RAID Makes Sense

Drives will fail. It is never a question of if, but when. When you build servers without dedicated hardware RAID cards—such as cloud VMs with attached block volumes, budget dedicated servers (like Hetzner or OVH bare metal), or home lab hypervisors—Linux software RAID via mdadm delivers rock-solid redundancy without vendor lock-in.

Hardware controllers create a fragile single point of failure. If a proprietary MegaRAID or Smart Array card fries, finding an identical replacement card or matching firmware version can take days. In contrast, mdadm (Multiple Device Administrator) writes its metadata directly to the disk headers. You can pull the drives from a dead server, plug them into any off-the-shelf Linux box, and bring the array online within minutes.

Selecting the Right RAID Level

Match your setup to your specific capacity, throughput, and uptime requirements:

RAID 1 (Mirroring): Requires at least 2 disks. Every block is mirrored across drives. Read speeds roughly scale with disk count, while writes match your slowest disk. You keep only 50% usable capacity (e.g., two 1TB SSDs yield 1TB usable). Ideal for OS boot drives, /boot partitions, and core databases.
RAID 5 (Distributed Parity): Requires at least 3 disks. Writes data and parity blocks across all drives. Can survive losing exactly one drive. Usable capacity equals (N – 1) * Disk Size, so three 4TB drives yield 8TB usable. A great fit for bulk storage and media pools, though parity calculations penalize random write performance.
RAID 10 (Stripe of Mirrors): Requires at least 4 disks (even count). Combines RAID 0 throughput with RAID 1 fault tolerance. Delivers 50% capacity. Can withstand multiple drive losses as long as both sides of any single mirror pair stay healthy. The gold standard for high-IOPS PostgreSQL, MySQL, and virtualization hosts.

Installation: Preparing the System and Tooling

Every major Linux distribution packages mdadm in its primary repositories. Grab it through your standard package manager:

# Debian / Ubuntu
sudo apt update && sudo apt install -y mdadm

# RHEL / AlmaLinux / Rocky Linux
sudo dnf install -y mdadm

List your connected block devices to make sure the target drives are unmounted and visible. For this guide, we use four identical 1TB disks: /dev/sdb, /dev/sdc, /dev/sdd, and /dev/sde.

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

Previous partition tables or filesystem superblocks can confuse array initialization. Wipe them clean before proceeding:

sudo wipefs -a /dev/sdb /dev/sdc /dev/sdd /dev/sde

Configuration: Assembling Arrays and Handling Filesystems

1. Building a RAID 1 Array (2 Drives)

To create a mirrored array assigned to /dev/md0 using two disks:

sudo mdadm --create --verbose /dev/md0 \
  --level=1 \
  --raid-devices=2 \
  /dev/sdb /dev/sdc

2. Building a RAID 5 Array (3 Drives)

To initialize distributed parity across three drives:

sudo mdadm --create --verbose /dev/md5 \
  --level=5 \
  --raid-devices=3 \
  /dev/sdb /dev/sdc /dev/sdd

3. Building a RAID 10 Array (4 Drives)

For high-throughput, low-latency workloads:

sudo mdadm --create --verbose /dev/md10 \
  --level=10 \
  --raid-devices=4 \
  /dev/sdb /dev/sdc /dev/sdd /dev/sde

Filesystem Creation and Persistent Mounts

Once assembled, treat the virtual /dev/mdX device like any ordinary drive:

sudo mkfs.ext4 -F /dev/md0
sudo mkdir -p /mnt/raid_storage
sudo mount /dev/md0 /mnt/raid_storage

Without an explicit config, the kernel might assemble arrays under arbitrary device names on reboot (like /dev/md127). Save the array layout to mdadm.conf and rebuild your initial ramdisk so the root environment detects them during early boot:

# Save current array configuration
sudo mkdir -p /etc/mdadm
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

# Refresh the initramfs to persist changes
sudo update-initramfs -u  # Debian / Ubuntu
# sudo dracut -f           # RHEL / CentOS / Rocky Linux

Mounting by kernel name (/dev/md0) in /etc/fstab is brittle. Always look up the unique filesystem UUID:

sudo blkid -s UUID -o value /dev/md0

Then add this line to /etc/fstab using the returned UUID (for example, e2f8c5b1-36a4-4df1-8e9a-24128f654b03):

UUID=e2f8c5b1-36a4-4df1-8e9a-24128f654b03 /mnt/raid_storage ext4 defaults,nofail 0 2

Verification & Monitoring: Health Checks and Replacing Failed Disks

Checking Array Health and Rebuild Status

To inspect active arrays and track background sync operations in real time:

cat /proc/mdstat

For detailed disk status, raid personality, and superblock timestamps, query the array directly:

sudo mdadm --detail /dev/md0

Check the State field—healthy arrays report clean or active. If it reads degraded, examine which disk failed under the active drive summary at the bottom.

Speeding Up Initial Sync and Resync Operations

The Linux kernel throttles background resyncs by default (often capping them at 1,000 KB/s min to 200,000 KB/s max) to protect desktop and user-space I/O. On a 4-drive RAID 5 array of 8TB disks, standard limits can stretch a resync to over 30 hours. On a test server with NVMe and SATA arrays, cranking these sysctl thresholds slashed rebuild time from 14 hours down to under 3 hours:

# Raise rebuild throughput bounds to 50 MB/s min and 500 MB/s max
echo 50000 | sudo tee /proc/sys/dev/raid/speed_limit_min
echo 500000 | sudo tee /proc/sys/dev/raid/speed_limit_max

# Boost stripe cache size for RAID 5/6 (default is 256 pages; set to 4096 or 8192)
echo 8192 | sudo tee /sys/block/md5/md/stripe_cache_size

Replacing a Failed Drive Safely

When dmesg reports I/O errors or /proc/mdstat lists a drive with (F), follow these steps to swap the physical disk without taking your storage offline.

Mark the disk as faulty: If the disk is spitting errors but hasn’t fully detached, tell the kernel to stop routing I/O to it:
```
sudo mdadm --manage /dev/md0 --fail /dev/sdb
```
Remove the disk from the array: Unbind the faulty drive from the software stack:
```
sudo mdadm --manage /dev/md0 --remove /dev/sdb
```
Swap the physical hardware: Hot-swap the tray or halt the machine if your backplane lacks hot-plug support. Ensure the new disk is at least as large as the original drive—even a single missing sector will cause mdadm to reject it.
Add the replacement drive: Once Linux enumerates the new disk (verify its identifier with lsblk), insert it into the degraded array:
```
sudo mdadm --manage /dev/md0 --add /dev/sdb
```
Track the rebuild: The kernel kicks off synchronization automatically. Watch the progress bar, estimated completion time, and throughput:
```
watch -n 2 cat /proc/mdstat
```

Automating Email Alerts for Degradation

Silent hardware failures turn into data loss if a second drive dies before you notice the first. Configure mdadm‘s monitoring daemon to alert your ops inbox immediately when an array degrades. Edit /etc/mdadm/mdadm.conf:

MAILADDR [email protected]

Send a dry-run notification to ensure your local mail transfer agent (like Postfix or msmtp) functions properly:

sudo mdadm --monitor --scan --test --oneshot

Check your inbox. Once the test alert arrives, start and enable the systemd daemon so health checks run non-stop:

sudo systemctl enable --now mdmonitor