Automatic Storage Tiering with ZFS: Hot/Cold, Special VDEV and What Actually Works in Practice

“Automatic storage tiering” is on every NetApp, Dell and HPE datasheet: data drifts between fast flash and cheap HDD on its own, the admin just watches. Anyone bringing the same expectation to TrueNAS or a ZFS pool under Proxmox will be disappointed. ZFS has no real automatic tiering in the sense of a policy engine that moves blocks between tiers. What ZFS offers instead — and which parts of it actually pay off in SMB environments — we lay out honestly in this article.

The myth of automatic ZFS tiering

The idea is appealing: one pool, three tiers (NVMe, SSD, HDD), ZFS shuffles blocks between layers based on access frequency. In reality that feature does not exist in OpenZFS and it is not on the roadmap. The reason is architectural: ZFS stripes writes evenly across all VDEVs of a pool. A mix of NVMe and HDD VDEVs in the same pool would randomly write to both — no policy, no rebalancing, with the performance of the slowest VDEV. That is exactly not what you want.

What ZFS does offer are caching layers (ARC, L2ARC) and specialised VDEVs (Special, SLOG, Dedup) inside a single pool. Those are not tiers in the classic sense — they accelerate access to hot data or offload specific block types onto faster storage. For real hot/cold tiering between a fast and a slow pool, only manual dataset tiering with zfs send | zfs receive remains.

This honest framing matters before we dive into the technical details. Running TrueNAS or Proxmox on ZFS does not get you FabricPool or PowerStore AITR — it gets you something better, just different.

What ZFS actually offers: caching instead of tiering

ARC: the hidden hot tier in RAM

The Adaptive Replacement Cache (ARC) is active on every ZFS system and by default claims up to 50 percent of system RAM. It caches metadata and data blocks in a two-list algorithm (MRU + MFU) and delivers hit ratios of 95 to 99 percent on typical SMB workloads. In that sense, RAM is the real hot tier of a ZFS pool — and the fastest one you will ever get. We cover ARC sizing in detail in Tuning ZFS ARC and L2ARC on Proxmox Hosts.

L2ARC: the read-side SSD in front of the HDDs

L2ARC is an SSD acting as a second cache tier. It buffers blocks that have been evicted from ARC before they would have to be read from HDD again. For pure HDD pools with large working sets and read-heavy workloads it can help a lot. The honest expectations: L2ARC is read-only, does nothing for writes, costs additional RAM for header tables (roughly 70 to 90 bytes per block), and brings nothing when the pool is already all SSD.

An L2ARC hit ratio below 20 percent is a strong signal in practice that those SSD slots are better used for a Special VDEV or a mirror. More context in our article ZFS ARC and L2ARC Cache Tuning.

SLOG: often confused, not a tiering solution

The SLOG (Separate Intent Log) is regularly confused with tiering. It is neither a cache nor a tier — it accelerates synchronous writes exclusively, by moving the ZIL to a fast, power-loss-protected SSD. For regular file access a SLOG brings nothing. If you want the clean separation, see ZFS SLOG, Special VDEV and Write Cache in Practice.

Special VDEV: metadata and small blocks on SSD

The Special VDEV is the ZFS feature closest to the term “automatic tiering” — for two block types:

1. Metadata (directory structures, inodes, checksums) lands automatically on the Special VDEV instead of the HDDs.
2. Small blocks up to a configurable threshold (special_small_blocks) also migrate to the Special VDEV.

In an HDD pool with a Special-VDEV mirror of two enterprise SSDs the user notices this immediately: ls, find, rsync --dry-run or SMB directory browsing on large shares feel much more responsive, because metadata reads no longer wait for HDD heads.

# HDD pool with a special VDEV mirror
zpool create tank \
  raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf \
  special mirror /dev/nvme0n1 /dev/nvme1n1

# Put small blocks up to 128K on the special VDEV
zfs set special_small_blocks=128K tank/dataset

Caution: if the Special VDEV fails and was not mirrored, the entire pool is lost. Always deploy Special VDEVs as mirrors — ideally triple-mirrored for critical data. And: once the Special VDEV fills up, small blocks silently fall back to the HDDs. VDEV capacity monitoring belongs to the basic hygiene.

We cover hybrid-pool design in more depth in TrueNAS Hybrid Storage with SSD and HDD.

Manual dataset tiering: zfs send and receive as a hot/cold workflow

Anyone who truly needs hot/cold tiering between two physically separate pools cannot avoid a scripted workflow in ZFS. That sounds like a step backward, but it has one advantage: it is transparent, auditable, and uses the same tools as replication and backup. The core idea:

• Hot pool: NVMe or SSD RAIDZ for active projects, databases, running VMs.
• Cold pool: HDD RAIDZ2 or RAIDZ3 for completed projects, archives, backup history.
• Policy: datasets whose content has not changed for a defined period are moved to the cold pool via zfs send -R | zfs receive -F; afterwards the source dataset is destroyed and optionally a symlink is placed.

The full policy framework, example scripts and a cron-based nightly job are documented in our article ZFS Dataset Tiering: automatic storage movement between SSD and HDD. For most SMBs a policy set for two to four data classes (active projects, completed projects, backup rotation, archive) is enough — more becomes complexity-killer rather than help.

A clear limitation: this workflow moves entire datasets or — in finer-grained variants — individual files. It does not operate block-wise like an enterprise SAN. For typical SMB workloads (file servers, project shares, backup repositories) that is perfectly sufficient; for high-frequency OLTP databases it is the wrong tool — they belong on an all-SSD pool, full stop.

When separate pools pay off — and when they do not

In customer projects we see two archetypes:

Setup A: one hybrid pool with Special VDEV and optionally L2ARC For file servers with mixed file sizes, SMB/NFS workloads and a moderate user count, a single HDD pool with a Special VDEV mirror is often the best answer. No datasets to move, no scripting, straightforward snapshot and replication logic. This covers the vast majority of the SMBs we work with.

Setup B: two separate pools (hot + cold) with dataset tiering For larger TrueNAS environments with a clear separation between active and archived data — media production, design offices or long statutory retention periods — the extra effort pays off. Active data stays on the fast pool, the cheaper HDD pool absorbs the bulk of the capacity. A TrueNAS M30 or M40 as hot tier and a TrueNAS Mini-R or Mini-X+ as cold target is a typical layout. The EOL models M50 and M60 are replaced during upgrades by the current M30 or M40 successors.

If you are unsure which setup fits, we build hardware recommendations and pool layouts via our TrueNAS configurator, and for complex cases through personal consulting under TrueNAS support.

The alternative: hybrid pool with Special VDEV and L2ARC

For many environments the hybrid pool is the sweet spot. It combines the things ZFS truly automates well:

• HDD RAIDZ2 as the capacity backbone, cheap per terabyte.
• Special VDEV mirror of two enterprise SSDs for metadata and small blocks.
• L2ARC optional, but only if ARC sizing and workload analysis justify it.
• ARC RAM generously sized, with a rule of thumb of at least 1 GB RAM per terabyte of HDD net capacity.

In daily operations this combination behaves very tier-like: directory listings, small files and database indexes see SSD latency, large streaming workloads run on HDDs — without any script ever moving data around.

Comparison: ZFS vs. NetApp FabricPool vs. Dell PowerStore AITR

ZFS does not replace an enterprise SAN with fully automatic tiering in the strict sense. But with Special VDEV, caching layers and a clean dataset-tiering workflow it covers 90 percent of SMB workloads with much more affordable hardware and comparable or better results.

What we actually deploy in TrueNAS projects

In our ongoing TrueNAS projects a typical tiering setup looks like this:

• Hot pool on TrueNAS M30/M40: NVMe RAIDZ or SSD mirror for VM backing store, databases, active projects.
• Cold pool on TrueNAS Mini-R or a second M30: HDD RAIDZ2 with a Special VDEV mirror for archives, backup history, completed projects.
• Replication: both pools are linked via ZFS replication, so the cold pool doubles as DR target.
• Dataset tiering: a scripted nightly job moves project datasets older than 180 days from hot to cold.
• Monitoring: ARC hit ratio, Special VDEV utilisation and pool fragmentation are observed via Checkmk; the setup is part of our IT service.

Hardware pricing varies by configuration; current builds are quoted individually via the TrueNAS configurator or on request.

Conclusion

ZFS does not offer automatic tiering like NetApp FabricPool. It offers something more pragmatic: a very good RAM cache, a metadata tier via Special VDEV, an optional read cache via L2ARC and a scripted dataset-tiering workflow for real hot/cold. In practice, a well-designed hybrid pool with a Special VDEV is more than enough for most SMBs. Only large, cleanly separated data lifecycles justify two separate pools and a tiering workflow.

Anyone who designs their ZFS setup around that reality — not around marketing expectations — ends up with stable, auditable and honestly affordable storage. Anything else is wishful thinking.

Frequently asked questions

Can I configure ZFS so data moves from SSD to HDD automatically, without a script?

No. OpenZFS has no policy engine for block-level tiering between pools. What runs automatically inside a pool is the Special VDEV (metadata, small blocks) and the ARC/L2ARC cache. For real dataset tiering between two pools you need a cron-driven zfs send workflow.

Is an L2ARC worthwhile on top of a Special VDEV?

Not always. The Special VDEV already keeps metadata and small blocks on SSD latency — that covers most latency-critical access. An additional L2ARC pays off only if the working set is clearly larger than RAM, the workload is read-heavy and the L2ARC hit ratio is above 30 percent. In many SMB setups, more RAM is a better investment than an L2ARC SSD.

What happens when the Special VDEV fills up?

ZFS silently writes new metadata and small blocks back to the HDDs. No error is raised, but the perceived performance drops to HDD levels. That is why Special VDEV utilisation needs active monitoring, with planned expansion or replacement when it reaches 70 to 80 percent.

Which TrueNAS models fit as hot and cold pool?

As the current hot tier we recommend TrueNAS M30 or M40, depending on capacity and performance requirements. As a cold tier a TrueNAS Mini-R or a second M30 with large HDDs and a Special VDEV mirror works well. The earlier M50 and M60 models are end of life and are replaced by the current M30 and M40 during upgrades.

Which TrueNAS version should I run for a tiering setup?

In production we currently deploy TrueNAS 25.10 Goldeye, which is the stable line. TrueNAS 27 is scheduled to reach GA in early December 2026; we recommend validating production migrations in your own staging window rather than cutting over on GA week. Details on the renaming and roadmap are in our TrueNAS 27 news post.

What does a hot/cold configuration cost at DATAZONE?

Prices for TrueNAS hardware fluctuate significantly right now — flash market, HDD capacities, DRAM pricing. We quote every configuration individually in the TrueNAS configurator or on request. As a rule of thumb: a hybrid pool with a Special VDEV is noticeably cheaper than two separate pools with replication and dataset tiering — and sufficient for most SMBs.

Planning a hot/cold setup or unsure whether Special VDEV, L2ARC or separate pools are the right choice? Get in touch — TrueNAS support at DATAZONE accompanies you as an authorised iXsystems partner from planning to day-two operations.