Remote Support Start download

Measuring TrueNAS Performance with fio and iozone: Methodology, Pitfalls and How to Get Honest Numbers

TrueNASPerformanceBenchmark
Measuring TrueNAS Performance with fio and iozone: Methodology, Pitfalls and How to Get Honest Numbers

Every TrueNAS admin who has ever had a performance discussion with a hardware vendor, a software maker, or a managing director knows the scene: one person reads out an impressive IOPS figure, another pulls a counter benchmark out of their pocket, and in the end everyone is arguing about synthetic numbers that never actually show up in the real environment. This article is an attempt to get out of that loop. Not with even more numbers, but with a clean methodology.

At DATAZONE we regularly measure TrueNAS systems — during commissioning, after upgrades, in response to performance complaints, and as part of our internal quality assurance. Over the years we have adopted a few hard rules that we are happy to share here. Do not expect concrete benchmark numbers from customer environments; for good reasons we do not publish those. Expect instead the toolbox you need to generate your own numbers in a way that makes them trustworthy.

Why measuring TrueNAS performance is harder than it looks

ZFS is a file system that actively works to distort benchmarks — not out of spite, but because for production workloads it does exactly the right thing. It caches aggressively in RAM through the ARC, it collects writes in transaction groups, it deduplicates and compresses on the fly, and it spreads I/O across VDEVs. Each of these mechanisms helps your real workload — and sabotages almost every naive benchmark.

Then there is the network side. If you measure on an SMB share from a Windows client, you are actually measuring the network stack, protocol overhead, client caching, server caching and the disks underneath, all rolled into one number. For root cause analysis that resolution is simply too low. For that reason our methodology strictly separates local pool performance from network performance.

And then there is the time axis. A pool tested on Monday morning with an empty ARC behaves differently from the same pool on Tuesday afternoon after two days of production. Every honest benchmark must address this time dependency explicitly.

The two tools in the toolbox: fio and iozone

Why fio is the main tool

fio — the Flexible I/O Tester — is the de facto standard tool for storage benchmarks on Linux. It is available on TrueNAS SCALE directly or can be added in a jail or a debug container. fio supports synchronous and asynchronous I/O, different engines (sync, libaio, io_uring, posixaio), direct I/O bypassing the pagecache, precise run lengths and detailed latency histograms. Almost everything you want to measure in practice can be expressed with fio.

The price is that fio is so flexible that it can be configured in almost any direction. A workload with rw=read, bs=4k, iodepth=1 on a 10 GB test file running on a system with 128 GB RAM gives you, after the first pass, a pure ARC benchmark. The result looks impressive and is technically not a lie — it just does not measure what you wanted to measure.

Why iozone still makes sense as a complement

iozone is older, less flexible, but has one decisive advantage: its automatic test mode with varying file sizes and block sizes produces a matrix that shows at a glance where in the ratio of file size to RAM the ARC effect tips over. If you run an iozone pass on a 128 GB system across file sizes from 1 GB to 256 GB, the result matrix shows very clearly from which size on the disks are actually loaded. That information is gold for the methodology.

In practice we use iozone as an exploration tool to find the cache-to-disk boundary, and fio for the actual measurements whose numbers we then report.

The pitfalls that distort almost every TrueNAS benchmark

Pitfall 1: the ARC cache effect

This is by far the most common mistake. ZFS caches read blocks in RAM, and it does so very aggressively. On a system with 128 GB RAM, easily 100 GB or more can be available for the ARC. If your test file is 10 GB in size, it sits entirely in RAM after the first pass. Every further read pass then measures the speed of RAM access, not of your disks.

The clean answer is to work with test files that are significantly larger than the available ARC — as a rule of thumb three times the RAM volume. On a 128 GB system that means test files around 400 GB. Alternatively you can temporarily limit the ARC for the duration of a benchmark, but that changes behaviour so dramatically that the results are no longer representative for production. We recommend going with large test files.

Pitfall 2: synchronous versus asynchronous I/O

This point is possibly the most important one for interpreting numbers. ZFS treats synchronous and asynchronous writes fundamentally differently. Asynchronous writes land in the ZFS Intent Log in RAM and are only written to disk later in batches — the result looks lightning fast. Synchronous writes, by contrast, must be written to stable storage before acknowledgement, either to the SLOG if one exists or directly to the pool disks. Latencies for synchronous workloads are typically orders of magnitude higher.

If you plan a database workload, measure synchronously. If you plan a backup workload, measure asynchronously. Anyone claiming to have a general “TrueNAS benchmark” has in reality only half the truth. In fio we control this with sync=1 or fdatasync=1, in iozone with the corresponding modes.

And very important: a VMware or Proxmox hypervisor usually talks to storage synchronously. Anyone benching their virtualisation back end only with async writes will be rudely surprised in production.

Pitfall 3: the wrong block size

4K random reads look spectacular on any halfway healthy pool. 1M sequential writes look similar everywhere. What matters is the block size that fits your real workload. Databases typically work with 8K to 16K, VMs on average with 4K to 64K, backup workloads with 128K to 1M, office file shares with a mix of everything.

In our methodology we always run at least three block sizes: 4K, 64K, 1M. That covers the practically relevant range and shows the curve, not just a single data point. If you have a specific application in mind, add the matching block size on top.

Pitfall 4: run times that are too short

A 30-second benchmark on an enterprise pool says very little. ZFS has transaction groups that typically flush to disk every 5 seconds, it has ARC warm-up times, and it has scrubs and other background activity that produces load in waves. We usually measure for 10 minutes per workload, longer for synchronous writes, and discard the first 60 seconds as warm-up. The variance over time is at least as interesting as the mean.

Pitfall 5: the single pass

Every individual benchmark run fluctuates. Background scrubs, replication jobs, other admin activities — all of that changes the numbers. We run each workload at least three times and report mean plus variance. Anything else is coincidence with decimal places.

A practical measurement kit

What we typically run in customer projects breaks down into four blocks. First: a local pool benchmark with fio, directly on the TrueNAS system, direct I/O bypassing the pagecache, test file three times the RAM volume, 10 minutes run time, three repetitions, three block sizes, once synchronous and once asynchronous. Second: a network benchmark with iperf3 between client and server to make sure the link can carry the numbers at all. Third: a protocol benchmark over SMB or NFS from the client, with the same block sizes, to make the overhead visible. Fourth: an application benchmark with the real workload — a database load simulation, a VM boot, a backup run. The results of the first three blocks are only interesting insofar as they help explain block four.

What benchmarks can tell you and what they cannot

Benchmarks can do three things very well: they can make hardware issues visible (a disk that behaves differently stands out immediately in a systematic run), they can make configuration decisions comparable (SLOG before and after, record size before and after), and they can establish a baseline over time against which complaints can be compared.

What benchmarks cannot do is predict the future. A pool that delivers 50,000 IOPS today may look entirely different tomorrow under a changed workload. A benchmark is a snapshot of a specific configuration under a specific load. No more, no less.

And they cannot replace the quality of a setup. A pool with a good IOPS number but no SLOG is still the wrong choice for synchronous workloads. A pool with impressive sequential throughput but only a single mirror VDEV does not scale under mixed load. Numbers without an understanding of architecture lead you astray. That is why in customer projects we always combine measurement with a review of the pool topology and ZFS settings.

How we apply this in DATAZONE projects

For every new TrueNAS system from our configurator we run a complete baseline pass according to exactly this methodology before handover. The results are documented and remain available to the customer as a reference. If performance complaints arise a year later, we have a yardstick. Without that baseline, every re-measurement is just speculation.

For existing systems under support we measure on demand — when a customer reports degradation, when an upgrade is coming up, when a workload changes. Then we compare against the baseline and can often pinpoint the cause clearly: a pool fill level that has grown too high, an ARC hit ratio drop caused by a new workload profile, or a disk that is starting to die slowly.

In environments with Proxmox and TrueNAS together the benchmarks are particularly instructive because network stack, virtualisation overhead and storage layer all overlap here. Cleanly separating the measurement layers is then the only way to reliably identify bottlenecks.

Bottom line

Measuring TrueNAS performance is not a five-minute job and not a discussion of IOPS numbers from data sheets. It is a craft with rules. If you set up fio and iozone cleanly, treat the ARC effect deliberately, separate synchronous and asynchronous, run multiple block sizes and measure over long enough intervals, you get trustworthy numbers. Anything else is marketing.

For you that means: next time, take the time for the methodology before you argue about the numbers. And if you are unsure whether your measurement environment is clean, get in touch with us. We are happy to measure alongside you — and we will tell you honestly if the pool everyone is arguing about is in fact delivering exactly what you should expect from this configuration.

FAQ

Which tools do I need at a minimum to measure TrueNAS performance?

fio as the main tool and iperf3 for the network part are enough for almost all questions. iozone is a good complement if you want to systematically make the cache boundary of your pool visible. All three tools are either directly available on TrueNAS or easy to install.

How large does my test file need to be so that the ARC cache does not distort the measurement?

As a rule of thumb, three times the RAM available to the system. On a system with 128 GB RAM that means around 400 GB. Smaller test files deliver pure ARC benchmarks that have nothing to do with disk performance.

Why are synchronous and asynchronous writes so different in speed?

Synchronous writes must be committed to stable storage before acknowledgement, either to the SLOG or directly to the pool disks. Asynchronous writes are gathered in RAM and flushed later. Virtualisation back ends and databases typically run synchronously, which is why only a synchronous measurement tells you how the pool behaves under such workloads.

How long should a benchmark run last at a minimum?

We recommend at least 10 minutes per workload, discarding the first 60 seconds as warm-up. Shorter runs sit too close to transaction group boundaries and background activities and vary too much.

Is a single benchmark pass enough?

No. Every individual measurement fluctuates due to background scrubs, replication jobs or other activities. At least three repetitions per workload, then report mean and variance.

Can synthetic benchmarks replace real workloads?

No, they are an aid, not a prediction. For the real validation we always recommend adding a real application workload — a database load simulation, a VM boot, a backup run. Only in combination do you get an honest picture.

What does DATAZONE do differently from a vendor data sheet?

We measure in your environment, with your configuration, on your disks, with your workload. Data sheet numbers apply to the vendor’s lab conditions. Both sets of numbers can coexist — but only your own numbers say something about your system. And those are exactly the numbers you get in a clean baseline measurement, as we run for every new system before handover.

More on these topics:

Plan your TrueNAS sizing live
Translate this article straight into a concrete configuration: pick model, drives, RAID level and networking — and get the resulting capacity in real time.
Configure now

Need IT consulting?

Contact us for a no-obligation consultation on Proxmox, OPNsense, TrueNAS and more.

Get in touch