Proxmox Host: Tuning ZFS ARC and L2ARC the Right Way

ZFS is the default for local storage pools on Proxmox hosts — robust, with snapshots, replication and self-healing. What many administrators underestimate: the Adaptive Replacement Cache (ARC) is not a “nice to have” but the decisive performance factor. By default, the ARC on Proxmox VE 8.x claims up to 50 percent of system RAM. On a host with 256 GB RAM, that is 128 GB — a number that quickly conflicts with VM workloads, backups and L2ARC.

In this article we show how to set zfs_arc_max correctly, when an L2ARC actually makes sense, and which pitfalls we regularly see in customer projects. The tips apply to Proxmox VE 8.3 and 9.0 with OpenZFS 2.2.x.

Understanding ARC: What really lives in RAM

The ARC is an in-memory cache for data blocks (data) and metadata (metadata). Unlike the classic Linux page cache, it uses a two-list algorithm of MRU and MFU and delivers hit ratios of 95 to 99 percent on typical SMB workloads — provided it has enough room.

Check the current state with arc_summary from the zfsutils-linux package:

arc_summary | head -40
arcstat 1 10

The interesting fields are ARC size, Target size (adaptive), Hit ratio and Most Frequently Used (MFU). If the hit ratio sits below 90 percent, the ARC is too small — or your workload simply does not benefit from caching (e.g. linear backups, large media streams).

Setting zfs_arc_max cleanly

The ARC maximum is configured persistently via /etc/modprobe.d/zfs.conf. Values are given in bytes:

# /etc/modprobe.d/zfs.conf
# 32 GB ARC max, 8 GB ARC min
options zfs zfs_arc_max=34359738368
options zfs zfs_arc_min=8589934592

For the value to take effect at boot, regenerate the initramfs:

update-initramfs -u -k all
reboot

A live change is possible, but only as long as the new value is smaller than the current one — the ARC never grows above the configured ceiling on its own:

echo 34359738368 > /sys/module/zfs/parameters/zfs_arc_max

Rules of thumb for ARC sizing

Total RAM	VM workload	Recommended zfs_arc_max	Reserve VMs/host
64 GB	light	16 GB	~46 GB
128 GB	mixed	32 GB	~92 GB
256 GB	DB/mail VMs	64 GB	~188 GB
512 GB	heavy IO	96-128 GB	~380 GB

Important: these values are starting points. After two weeks in production, verify with arc_summary that the hit ratio is healthy. For VMs that maintain their own cache (e.g. PostgreSQL, MariaDB with a large innodb_buffer_pool) a smaller host ARC is fine.

L2ARC: When a cache SSD actually helps

L2ARC is an SSD acting as a second cache tier in front of the HDDs. Sounds tempting, but in practice it makes sense less often than people think. L2ARC helps when:

• the working set is substantially larger than RAM,
• the workload is read-heavy (at least 70 percent reads),
• many random reads hit a “warm” dataset,
• RAM is already maxed out and cannot be extended.

L2ARC does not help with:

• write-heavy workloads — L2ARC is a read-only cache,
• sequential streams (backups, video),
• pools that already consist of SSDs/NVMe.

An L2ARC costs additional RAM for header tables: around 70-90 bytes per data block. For 1 TB of L2ARC and 128k blocks, that is around 700 MB — with 16k blocks, over 5 GB. That memory is no longer available to the ARC.

Adding an L2ARC device:

zpool add rpool cache /dev/disk/by-id/nvme-Samsung_PM9A3_960GB
zpool status rpool

Enabling persistent L2ARC

Since OpenZFS 2.0, L2ARC survives a reboot. On Proxmox VE 8.x/9.x the feature ships, but is not always active:

echo 1 > /sys/module/zfs/parameters/l2arc_rebuild_enabled

Persistently in /etc/modprobe.d/zfs.conf:

options zfs l2arc_rebuild_enabled=1
options zfs l2arc_write_max=67108864      # 64 MB/s warm-up
options zfs l2arc_write_boost=134217728   # 128 MB/s during boot
options zfs l2arc_noprefetch=0            # cache prefetched data too

L2ARC status is visible under arc_summary -s l2arc. A hit ratio below 20 percent usually means: L2ARC brings nothing here, those SSD slots are better used for a special VDEV or mirror.

Common pitfalls

Prefetch: The ZFS prefetcher can cause cache pollution on database workloads. Check with arcstat. Disable for pure OLTP workloads:

echo 1 > /sys/module/zfs/parameters/zfs_prefetch_disable

For mixed workloads the default (0) is correct.

VM ballooning vs. ARC: If you have KVM ballooning enabled, the kernel can shrink the ARC under memory pressure — but slowly. The VM gets RAM, the ARC suddenly delivers only 70 percent hit ratio, pool latency spikes. Recommendation: on Proxmox hosts with ZFS, disable ballooning and assign RAM statically.

Swap on ZFS: A classic performance killer. If the host swaps and swap sits on a ZVOL, the ARC cannot keep up. Place swap on a separate small partition or set vm.swappiness=10 at minimum.

Backups: Proxmox Backup Server reads sequentially and fills the ARC with data that will not come back. If needed, set primarycache=metadata on datasets that only serve as backup targets.

Monitoring: What you should check regularly

We recommend feeding ARC metrics into your monitoring (Zabbix, Checkmk, Prometheus with node_exporter and zfs_collector). The most important values:

• arc_hit_ratio — target: > 95 percent
• arc_size / arc_target — should sit close together
• l2_hit_ratio — > 30 percent justifies the L2ARC
• pool_alloc and frag via zpool list — pool fragmentation > 50 percent is a warning sign

For our customers we build standardised Checkmk templates as part of our Proxmox consulting that alert on these metrics — including thresholds for ARC shrink events. For pure backup targets, a dedicated tuning strategy with reduced ARC and L2ARC often pays off.

Conclusion

ZFS ARC is the underrated performance lever on Proxmox hosts. With a cleanly sized zfs_arc_max, disabled ballooning and realistic expectations of L2ARC, you get noticeably more out of existing hardware. L2ARC is not a silver bullet — it costs RAM and only helps with specific workloads. When in doubt: more RAM first, then a special VDEV, and only then L2ARC.

DATAZONE supports SMB customers in the Neuburg, Ingolstadt and Augsburg region with the design of Proxmox clusters, ZFS pools and matching hardware. If you need a second opinion on your current configuration or are planning new hardware, get in touch — contact us.