Linux

Linux Storage Performance: Measure Before You Tune

Storage performance problems often produce misleading symptoms. An application may report slow queries while the real problem is a saturated device, a queue full of requests, or a workload dominated by small random I/O.

The safest tuning rule is simple: measure first, change one thing, and measure again.

Start with capacity

Before investigating performance, check whether the filesystem is simply running out of space.

df -h

For inode exhaustion:

df -i

A filesystem can have free gigabytes and still fail because it has no inodes left.

Identify the devices

lsblk gives a useful overview.

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS

This helps connect an application path such as /var/lib/postgresql to the underlying block device.

Do not assume the filesystem name tells you the physical device. LVM, RAID, multipath, and virtual disks can add several layers.

Look at mount options

Use findmnt for a readable view.

findmnt /var/lib/postgresql

For all mounts:

findmnt -t xfs,ext4

Mount options can affect behavior, but changing them without understanding the workload is not a performance strategy.

Measure device activity

The classic tool is iostat, usually provided by the sysstat package.

iostat -xz 1

The -x option provides extended statistics and -z hides devices with no activity.

Watch utilization, average request size, queue depth, and latency-related fields rather than focusing on one number.

Watch CPU and I/O together

vmstat provides a quick system-wide view.

vmstat 1

The wa column represents time spent waiting for I/O. High I/O wait can be useful evidence, but it does not identify which process caused it.

For that, look at processes.

Find the process doing I/O

pidstat can show per-process I/O activity.

pidstat -d 1

This is often more useful than staring at aggregate device utilization.

If one database backend is responsible for most reads or writes, the next question is what operation that process is performing.

Use top for context

CPU saturation and storage latency can interact.

top

A single busy CPU can limit a workload even when the SSD itself is capable of much higher throughput.

This is especially important for workloads with compression, checksums, serialization, or other CPU-heavy processing around I/O.

Test the device carefully

For controlled testing, fio is the standard tool.

A simple sequential read example might look like:

fio --name=read-test --filename=/path/to/testfile --size=4G --rw=read --bs=1M --iodepth=16 --direct=1

Do not run destructive tests against a live database device.

A benchmark should use a dedicated test file or test volume and should reflect the access pattern you are actually trying to understand.

Small I/O and large I/O are different

A workload reading large sequential blocks can produce impressive MB/s numbers.

A workload doing many small random operations may care much more about latency and IOPS.

That is why “my SSD can do 7 GB/s” does not mean every database query can read at 7 GB/s.

The application decides what requests reach the device.

Check open files

When a service is active, lsof can help connect files to processes.

sudo lsof +D /var/lib/myapp

Be careful with very large directory trees because recursive scans can themselves be expensive.

For a quick process-level view:

sudo lsof -p 1234 | head

Look for kernel messages

Storage errors often appear in the kernel log.

dmesg -T | grep -Ei 'error|timeout|nvme|scsi|xfs|ext4'

On systemd systems, the journal is another useful source.

journalctl -k -b

Do not ignore repeated controller resets or filesystem warnings just because the application still works.

Filesystem choice is workload-dependent

XFS and ext4 are both mature filesystems, but they have different behavior and administration features.

The important question is not which filesystem wins a generic benchmark.

Ask what the workload does: large files, millions of small files, sequential streams, random updates, snapshots, quotas, or database storage.

Do not tune blindly

Changing I/O schedulers, read-ahead values, mount options, dirty ratios, or database settings can make one benchmark better while making the real workload worse.

Keep a record of the baseline.

uname -r
lsblk
findmnt
mount | column -t
iostat -xz 1 5

Then change one variable.

Build a repeatable test

A good storage investigation answers four questions: what is the application doing, what is the kernel doing, what is the device doing, and where is the time going?

Once those answers are visible, tuning becomes engineering instead of folklore.