Bartosz Fenski maintains an automatically re-run benchmark suite for multi-device copy-on-write filesystems: 26 configurations across ext4, XFS, ZFS, btrfs and bcachefs, from single devices to md/LVM RAID10, ZFS mirrors and RAIDZ, btrfs RAID6 with RAID1C3 metadata, and bcachefs replicas and erasure coding — several with LUKS or dm-integrity on top. It runs on GitHub-hosted VMs over four 16 GiB loop devices, one fresh VM per configuration, with a host-calibration anchor per run and the standing instruction to compare shapes and ratios rather than absolute throughput.

The point is not the throughput. The point is the phases a typical fio suite never runs:

  • Metadata work: a 20k-file source tree (create, cp -r cold, rm -rf), the same create with four concurrent workers to expose lock contention, and 100k files in one directory — create, cold enumerate, cold stat, warm stat, mass delete.
  • Sparse-file honesty: ftruncate to 1 GiB, timed and reported with its actual st_blocks allocation.
  • Snapshot mechanics: an aging curve, space reclaim after deleting every snapshot, write bandwidth while reclaim runs, and create/remount/delete at 500 snapshots.
  • Divergence penalties: overwrite bandwidth into a plain file versus a fresh reflink clone versus a freshly snapshotted file.
  • Failure behaviour: degraded-array IOPS, rebuild time onto a spare, and a scrub after damage is injected behind the filesystem’s back.
  • Near-full writes at 95% and 99%, plus delete-at-ENOSPC and writable-after-delete verdicts.
  • Interactivity: fsync p99/p99.9, and a “how long until my prompt comes back” probe that writes 4k plus fsync every 200 ms while a second job floods the disk.

The corruption probe is the most useful result, because it separates RAID from checksums. Two gigabytes of random garbage are written directly onto one member device, the page cache is dropped, and a full scrub runs:

  • Every checksumming CoW filesystem — ZFS, btrfs, bcachefs — came back SURVIVED, with the test file readable and byte-identical. btrfs/raid1 reported 264,104 records found and 264,104 repaired.
  • Plain md and LVM RAID on ext4 and XFS could only count mismatches: 4,194,304 found, 0 repaired, test file not intact. They have no checksums to identify the good copy, so the verdict is probabilistic — reads round-robin between legs. The dm-integrity XFS variant survived.

Other results worth carrying:

  • fsync p99 spread is enormous: 1.19 ms for zfs/mirror-8k and 3 ms for bcachefs/single, against 196 ms for zfs/mirror and 190 ms for its encrypted twin. zfs/mirror-8k then blows up to 142 ms at p99.9.
  • Under a streaming-write flood the interactive probe starved almost everywhere: 2 of ~145 possible ops on xfs/single, 8 on ext4/single, 27 on btrfs/raid1, 29 on bcachefs/single.
  • Cold-cache 4k random reads favour mirrors: 12k–16k IOPS for ZFS mirror/raidz1 versus 6.5k for ext4 on one device.
  • Rebuild separates the layers: CoW filesystems move only live data (19–61 s), while md resyncs the whole member regardless of contents (86–106 s).
  • btrfs hit its chunk-allocation wall before reaching 95% fill on devices this small, so its two near-full cards read 0 MiB/s.
  • Compression lands at ~3.6–3.9x on 75%-compressible data for the CoW filesystems, and sometimes makes writes faster by putting fewer bytes on disk.

Two caveats the author states up front, and they matter: the array members are loop devices sharing one ephemeral VM’s storage, and the composite “Overall Core” index is a project-specific score model, not an industry-standard suite. Read as a whole, though, the run measures the thing storage choices actually get judged on — what happens when the array is degraded, the disk is 99% full, and the file you care about is sitting under a corrupted member.