How to validate NVMe performance with controlled file-based fio tests

When preparing a new server for databases or file services, the speed of a single large file copy is not enough to validate its storage. The file may be served from cache, writes may not yet be durable, and background tasks can affect the result. A more reliable approach is to keep the test file, block size, and concurrency settings fixed, measure throughput and latency separately, and retain the raw output so the tests can be repeated.

CloudUDP dedicated servers come with two 1TB NVMe drives, each directly connected, with no hardware RAID. The following procedure uses Debian 13 and fio—a configurable I/O workload testing tool—to test a specific filesystem. It does not write to raw devices, and the results are not guarantees of application performance.

1. Confirm the test scope and where data is actually stored

A file-based test measures the path from the filesystem to storage, not the drive’s maximum performance in isolation. Encryption, volume management, mirroring, and mount options can all affect the results. Having two directly connected drives also does not mean that a particular directory uses both of them.

Start by installing the tool and checking the target directory. Here, /srv is the proposed test location; if it is not on the intended filesystem, replace it with the actual mount directory.

sudo apt-get install fio
findmnt -T /srv -o TARGET,SOURCE,FSTYPE,OPTIONS
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS
df -h /srv
df -i /srv
fio --version

Check available space, mount status, and directory permissions. Do not rely solely on the drive’s advertised capacity: filesystem overhead and existing data consume space. Even after the test file has been fully written, enough free space must remain for application growth.

Do not pass a device path such as /dev/nvme… to a write test. A separate test file reduces the risk of overwriting existing data, but it does not eliminate resource contention. Prefer running these tests before putting applications into production. Stop immediately if application latency becomes abnormal, free space approaches the reserved threshold, or I/O errors occur.

2. Create an isolated directory and prepare the complete test file

Run all commands in the same Bash session. Have an administrator create a temporary directory owned exclusively by the current user account, and keep both the test file and results there to avoid touching existing application files.

TESTDIR=$(sudo mktemp -d /srv/fio-acceptance.XXXXXX)
sudo chown "$(id -u):$(id -g)" "$TESTDIR"
chmod 700 "$TESTDIR"
export TESTDIR
printf '%s\n' "$TESTDIR"
findmnt -T "$TESTDIR"

This example uses a 1GiB file as a starting point for validating the procedure. It does not represent the full storage capacity or long-term steady-state performance. fio interprets size=1G in binary units, so it must not be treated as equivalent to the units used for advertised device capacity. Once you have confirmed that sufficient space is available, write the entire file sequentially:

fio --name=prepare \
  --filename="$TESTDIR/test.bin" \
  --size=1G --rw=write --bs=1M \
  --ioengine=libaio --direct=1 \
  --iodepth=1 --numjobs=1 --end_fsync=1 \
  --group_reporting --output-format=json \
  --output="$TESTDIR/prepare.json"

Check both the exit status and the error fields in the JSON output. If direct I/O is not supported, investigate the filesystem and mount configuration first. Do not silently switch to buffered I/O and continue comparing results. Errors during preparation leave subsequent read tests without a trustworthy basis.

direct=1 normally bypasses the page cache for file data, but it does not bypass every device cache. end_fsync=1 requests synchronization at the end of the job; it does not mean that every write simulates a database commit.

3. Keep parameters fixed and test random reads and writes separately

Run a single-job random read test first, followed by a random write test, so a mixed workload does not obscure problems. Do not copy data or run other storage tests at the same time.

fio --name=randread \
  --filename="$TESTDIR/test.bin" \
  --size=1G --rw=randread --bs=4k \
  --ioengine=libaio --direct=1 \
  --iodepth=8 --numjobs=1 \
  --time_based --runtime=30 --ramp_time=5 \
  --group_reporting --output-format=json \
  --output="$TESTDIR/randread-q8.json"

fio --name=randwrite \
  --filename="$TESTDIR/test.bin" \
  --size=1G --rw=randwrite --bs=4k \
  --ioengine=libaio --direct=1 \
  --iodepth=8 --numjobs=1 \
  --time_based --runtime=30 --ramp_time=5 \
  --end_fsync=1 --group_reporting \
  --output-format=json \
  --output="$TESTDIR/randwrite-q8.json"

The runtime and warm-up period here are simply test parameters. Short tests can be affected by device caches, temperature, and background housekeeping; they cannot establish sustained write performance.

Add a low-queue-depth comparison

Change iodepth in the read test to 1 and the output filename to randread-q1.json, leaving all other parameters unchanged. Queue depth is the number of I/O requests allowed to remain outstanding at once. It is neither the CPU core count nor the number of concurrent application connections.

If increasing queue depth raises throughput but substantially worsens tail latency, that only demonstrates a trade-off between throughput and waiting time under this workload. Do not apply the setting directly to a database connection pool.

4. Read throughput, IOPS, and tail latency together

IOPS is the number of I/O operations completed per second. With a fixed block size, it has a direct relationship to throughput; results from different block sizes cannot be ranked by IOPS alone. In fio JSON output, bw is usually expressed in KiB/s. Check the version and field units when reading it.

Focus on iops, the bandwidth fields, and the mean and percentiles in clat_ns within the read or write section. Completion latency is not the same as application response time: application-side queuing and processing fall outside this measurement path.

P99 means that approximately 99% of requests have latency at or below that value. It is not the maximum latency. If the average remains similar while P99 rises, a small proportion of requests may be experiencing longer waits. Interpret this alongside the actual queue depth and repeated test results.

The acceptance record should include at least:

• The fio version, complete commands, file size, and target mount information.
• The read or write workload, block size, job count, and I/O queue depth.
• Throughput, IOPS, mean completion latency, and P99.
• Error fields, application load during testing, and any abnormal behavior.

Also examine fio’s actual I/O depth distribution. Configuring a depth of 8 does not mean that depth is maintained throughout the run. Differences in I/O engine or filesystem behavior can make results incomparable even when the nominal parameters match.

5. Complete validation with repeat tests, archiving, and cleanup

Repeat each workload, using a different output filename for every run rather than overwriting previous results. Compare the range of variation instead of selecting the highest number. Before enlarging the test file or extending the runtime, check free space again and account for the additional write volume and resource contention.

To validate the two drives separately, choose a directory on each drive’s filesystem and test them sequentially with identical parameters. If both directories actually reside on the same volume, changing directories does not constitute independent validation of the second drive.

If a read or write test reports errors, preserve the raw output and relevant kernel logs before investigating the filesystem and device path. If there are no errors but results vary substantially, first rule out other tasks on the server and changes in test conditions. Do not immediately assume a hardware fault.

When testing is complete, copy the JSON files, mount information, and command records to a controlled archive location. Confirm that the archived files are readable, then verify the test directory and delete only the files generated for this test:

printf '%s\n' "$TESTDIR"
ls -lh -- "$TESTDIR"
rm -- "$TESTDIR/test.bin"

After confirming that the remaining result files have been archived, remove them individually. Delete the directory with rmdir only once it is empty; do not recursively delete a device mount point. State the final conclusion as “these results were obtained under these parameters and workload conditions,” not “this drive always achieves a particular speed.” A file-based baseline helps identify obvious deviations, but validation with real application read and write patterns is still required before production deployment.

Frequently asked questions