Stop Guessing: Benchmarking Linux Disk Performance with fio

The 2 AM Performance Crisis

My Grafana dashboard was glowing red. It was 2 AM, and a high-traffic PostgreSQL database that had been stable for months was suddenly gasping for air. Query times had tripled. I/O wait was spiking at 40%. Despite this, the cloud provider’s status page was a sea of green, claiming their “Premium NVMe Storage” was healthy. The terminal told a different story.

Experience managing dozens of Linux instances has taught me one thing: never trust a spec sheet. A drive marketed as “High Performance” might boast impressive sequential speeds for 4K video, yet crumble under the weight of thousands of tiny, random database writes. To find the truth, I use fio (Flexible I/O Tester). It is the industry standard for simulating real-world pressure on storage hardware.

Why dd is a Terrible Benchmark

Many sysadmins reach for dd to test disk speed. You’ve likely seen the command: dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct. While dd is fine for a five-second sanity check, it is useless for production planning. It only tests sequential writes. It ignores the complexities of modern storage controllers and the chaos of concurrent access.

In production, your disk doesn’t do one thing at a time. It handles hundreds of simultaneous requests. One process reads a config file while another writes a 50MB log and a third updates a database index. To understand if your server can handle the heat, you must track three specific metrics:

• IOPS (Input/Output Operations Per Second): The number of small requests (typically 4KB) the disk completes every second. This is the lifeblood of database performance.
• Throughput (Bandwidth): The volume of data moved per second (MB/s). This is what matters for backups and large file transfers.
• Latency: The time it takes for a single I/O request to finish. Low latency makes a server feel snappy.

Installing fio

Fio is lightweight and available in almost every official repository. It won’t bloat your system or mess with your dependencies.

For Ubuntu or Debian users:

sudo apt update
sudo apt install fio -y

For those on RHEL, CentOS, or AlmaLinux:

sudo yum install epel-release -y
sudo yum install fio -y

The Database Test: Random Read/Write

Random I/O is the most frequent bottleneck for web servers. If your disk can’t handle random 4KB writes, your application will eventually hang. This command simulates a heavy database workload by forcing the disk to work hard on non-sequential data.

fio --name=random-rw --ioengine=libaio --rw=randrw --bs=4k --numjobs=1 --size=2G --runtime=60 --time_based --do_verify=0 --direct=1 --group_reporting --iodepth=64 --filename=fiotest.tmp

Key Flags Explained:

• --ioengine=libaio: Uses Linux native asynchronous I/O. This mimics how modern applications like MySQL or NGINX interact with the kernel.
• --rw=randrw: Executes a mix of random reads and writes simultaneously.
• --bs=4k: Sets the block size to 4KB, the standard page size for most filesystems.
• --direct=1: Bypasses the OS buffer cache. This ensures you are testing the actual hardware, not the server’s RAM.
• --iodepth=64: Keeps 64 requests in flight at once to test the drive’s queue management.
• --filename=fiotest.tmp: The file used for testing. Caution: Never point this to a raw device like /dev/nvme0n1 unless you intend to format the drive!

The Backup Test: Sequential Throughput

If you need to know how fast you can move 100GB log files or run system backups, you need a sequential test with a larger 1MB block size.

fio --name=sequential-write --ioengine=libaio --rw=write --bs=1m --numjobs=1 --size=2G --runtime=60 --time_based --do_verify=0 --direct=1 --group_reporting --iodepth=32 --filename=fiotest.tmp

In this scenario, look at the BW (Bandwidth) result. A modern Gen4 NVMe drive should easily clear 3,000 MB/s. A standard SATA SSD usually caps out around 540 MB/s. If you see numbers under 100 MB/s, you are likely dealing with a throttled cloud volume or an old mechanical hard drive.

Reading the Results Without the Headache

Fio generates a lot of text, but you only need to focus on a few lines. Here is what a typical result looks like for a mid-range SSD:

read: IOPS=15.4k, BW=60.2MiB/s (63.1MB/s)(3612MiB/60001msec)
  lat (usec) : min=412, max=15432, avg=1024.05, stdev=342.12

Is 15.4k IOPS good? For a basic cloud VPS (like an AWS gp3 volume with default settings), 3,000 IOPS is the baseline. High-end NVMe drives can exceed 500,000 IOPS. If your “Enterprise” storage is hitting less than 1,000 IOPS under load, something is wrong.

Check the Latency: Look at the avg. It is measured in microseconds (usec) or milliseconds (msec). For a production database, you want the average latency to stay under 1ms or 2ms. If the max latency frequently jumps above 100ms, your users will experience noticeable lag.

The Verdict

After running fio on that struggling 2 AM server, the results were undeniable. The IOPS were flatlining at 500—roughly the speed of a laptop drive from 2012. I sent the fio logs to the provider’s support team. Faced with hard data, they admitted the host node had a failing RAID controller. They migrated my instance to a new hypervisor immediately. Ten minutes later, I/O wait dropped to 0.5%, and the database was back to full speed.

Don’t wait for a crisis to understand your hardware. Run these benchmarks before you deploy. Data doesn’t lie, even when marketing does.