Remote Support Start download

Importing and Exporting a ZFS Pool: Seven Rescue Steps When the System Is Fried

ZFSRescueTrueNAS
Importing and Exporting a ZFS Pool: Seven Rescue Steps When the System Is Fried

When a TrueNAS or Proxmox host no longer boots after a dead mainboard, a botched update or a lightning strike, the data on a healthy ZFS pool can almost always be rescued. ZFS keeps data strictly separate from the host: pool metadata lives on the drives themselves, not in a controller configuration and not in an operating system database. If you know the right steps, you can move the pool to another appliance before the customer has even noticed.

This article describes the workflow we use in the DATAZONE workshop to bring broken systems back to life. The seven steps are listed in the order they happen in practice: from a clean zpool export through a correct import using by-id paths, all the way to the scrub after the move. Every step includes the typical traps we have run into over the years.

Why ZFS pool rescue works at all

A ZFS pool is a self-contained construct. Every drive in a vdev carries a label with pool GUID, vdev GUID, hostid and membership information. The host system only needs access to the drives and a compatible ZFS module – the operating system around it is irrelevant to the pool itself. A pool from TrueNAS 25.10 can just as well be imported on a Debian live system or a fresh Proxmox install, as long as the ZFS version on the target supports the pool features.

Two things matter: the drives must be detected completely and stably, and the target system must know the pool features. The latter is the more common pitfall – an attempt to import a pool from OpenZFS 2.2 on an older OpenZFS 2.1, especially when newer features such as raidz_expansion were enabled, ends in a hard abort. The reverse direction – old pool on a new system – rarely causes problems.

Step 1: Clean zpool export on the old system

If the old system still runs somehow, a zpool export is always the cleanest path. ZFS flushes all pending transactions, removes the pool from /etc/zfs/zpool.cache and sets the “pool is exported” flag in the label. When the pool is later imported on the new host, there is no discussion about hostids or force flags.

# Stop all active shares, jails, VMs or containers
systemctl stop smbd nfs-kernel-server
zfs unmount -a

# Cleanly take the pool out of the system
zpool export tank

If the export aborts with pool is busy, some process is still holding a dataset open. lsof +D /mnt/tank or fuser -vm /mnt/tank reveals the culprit. On TrueNAS SCALE this is usually running apps or NFS clients that have not released their mounts. Only when zpool export finishes cleanly is it safe to physically move the drives. We skip this step only when the old system refuses to boot at all – then the import on the new host is done with -f (see step 4).

Step 2: Detecting the drives on the new system

Before the actual import you need to be sure that every drive is detected correctly by the target. If even one member of a mirror or RAID-Z vdev is missing, ZFS either refuses to import the pool or imports it degraded. In our workshop we work in a fixed order: insert drives, boot the system, check the HBA in the BIOS, then run lsblk, dmesg | grep -i ata and smartctl -a per disk.

The crucial detail is to work with /dev/disk/by-id/. These paths contain the drive serial and stay stable regardless of SATA port or chassis bay. The classic /dev/sdX paths are off limits during rescue work – they can shift between boots and leave you with pools where half the drives show up as UNAVAIL.

# List drives with serial and mapping
ls -l /dev/disk/by-id/ | grep -v part
lsblk -o NAME,SIZE,MODEL,SERIAL,WWN

If the move takes a pool from an HBA in IT mode to a new system that has a RAID controller, that controller must be switched to JBOD or HBA mode. ZFS needs direct block access, otherwise all checksum and self-healing features become useless.

Step 3: zpool import with /dev/disk/by-id as the search path

The import always starts with a read-only zpool import without arguments. This call scans the configured search paths for ZFS labels and lists all pools it finds. Nothing changes at this stage – the pool stays exported.

# Only show which pools can be found
zpool import -d /dev/disk/by-id/

# Example output:
#   pool: tank
#   id:   12345678901234567890
#   state: ONLINE
#   status: Some supported features are not enabled on the pool.
#   action: The pool can be imported using its name or numeric identifier.

If the output shows state: ONLINE, the pool is complete and healthy. DEGRADED means a member is missing and you need to go back to step 2. FAULTED indicates a real problem – a second drive failure in a RAID-Z1, a destroyed label or an incomplete mirror. In that case diagnosis comes first; a force import without diagnosis only makes things worse.

With -d /dev/disk/by-id/ you force ZFS to list drives by their stable identifiers. Later diagnostics become much easier because zpool status then shows serial numbers instead of sdX paths.

Step 4: Force import when the pool was never exported

If step 1 was skipped, ZFS replies with the well-known “pool was previously in use from another system” error on import. This is not data damage but a safety mechanism: ZFS remembers the hostid of the last importer in the label. A new system has a different hostid, so ZFS refuses the import to prevent a concurrent dual-mount, for example in a SAN setup.

If you are certain the old drives are no longer active anywhere, the force import does the job:

zpool import -d /dev/disk/by-id/ -f tank

After a forced import ZFS rewrites the hostid to the new system and the pool behaves normally again. Important: -f is not a tool for broken pools, only for missing exports. Forcing a FAULTED pool with -f solves no problem – it only hides the warning.

Step 5: Readonly import as rescue mode

Two situations call for a readonly import: a pool in a questionable state after a power loss when you want to secure data before mounting, and a pool with features missing on the rescue host. A readonly import writes no new label, creates no new pool cache and modifies no metadata.

# Readonly import with an alternative mount point
zpool import -d /dev/disk/by-id/ -o readonly=on -R /mnt/rescue tank

With -R /mnt/rescue you set an altroot. All datasets of the pool appear under /mnt/rescue/... instead of their original mount points. This is critical during a rescue, otherwise ZFS would try to mount the pool on /mnt/tank and could clobber the running configuration of a TrueNAS live system.

In readonly mode you can calmly use rsync, zfs send or restic to push a consistent copy to a second target before importing the pool read-write. This step has saved us more than once in cases where a customer had already tried to “fix something themselves”.

Step 6: Check the pool and adjust features

After the import, whether regular or forced, the first look is always at zpool status:

zpool status -v tank
zpool list -o name,size,alloc,free,fragmentation,health

Three things matter: the state of each vdev (ONLINE, DEGRADED, UNAVAIL), the number of checksum errors per drive and the pool status line. If it reads “Some supported features are not enabled”, the new host offers more features than the pool uses. zpool upgrade tank enables them – but only when you never plan to move the pool back to the old system. A pool upgrade is not reversible.

If the message is “features are not supported on this pool”, it is the other way round: the pool uses features the target system does not know. Then either the ZFS version on the target has to be updated, or a downgrade of the pool is impossible – and the only path is a long detour via zfs send to a compatible target. We see this especially on older TrueNAS CORE hosts facing pools from current SCALE systems.

Step 7: A scrub after the import is mandatory

Every rescued pool gets a full scrub right after the import at our shop. A scrub reads all data plus all parity or mirror copies from disk and verifies them against the ZFS checksums. Damaged blocks are repaired from the redundancy, and at the end you know exactly in what state the pool really is.

# Start the scrub and watch progress
zpool scrub tank
zpool status tank | grep scan

The scrub runs in the background and takes anywhere from a few hours to several days depending on pool size and drive IOPS. On a production system we deliberately plan the scrub for a low-load window – but there is no need to stop any service, the pool stays fully usable during the scrub. We have described the topic in more detail in ZFS scrub and SMART.

If the scrub finds errors, zpool status -v lists the affected files individually. That is the moment to restore from a second backup or an offsite replica – and the reason we recommend an offsite replica for every production TrueNAS.

A real case: Dead TrueNAS, disks into a new appliance

A typical case from our daily work: a customer reports that their TrueNAS Mini R no longer boots after a power outage. Diagnosis says the mainboard is dead, the disks are untouched. We move the drives into a spare appliance from our TrueNAS configurator line-up, boot the new system and walk through the seven steps. Result: after roughly two hours the pool is back online, the SMB shares are reachable, the scrub runs overnight. For the customer this means two hours of downtime instead of a week of data loss.

This kind of rescue work is part of our standard repertoire. Anyone who works regularly with ZFS knows that the pool can almost always be saved – it only becomes dangerous when someone without experience starts touching the metadata with -f or -FX and destroys the pool for good. If you are unsure, reach us directly via TrueNAS support or our general IT service. For the classic bootpool rescue cases on Linux we have written up the paths separately in Linux ZFS boot mirror rescue.

FAQ on ZFS pool rescue

Can I import a ZFS pool from TrueNAS CORE on TrueNAS SCALE?

Yes, this is the classic migration path at iXsystems. CORE is based on FreeBSD with OpenZFS, SCALE on Debian with OpenZFS. Pool features are compatible as long as the OpenZFS versions line up. The key points are a clean export on CORE before the move and a readonly import as the first test step on SCALE.

What do I do if zpool import does not find the pool at all?

The problem is usually below the ZFS layer. Check lsblk to confirm that all drives appear with the expected size, and dmesg for HBA or cable errors. If a drive is missing, the pool does not show up because no complete vdev can be assembled. In a RAID-Z2 or mirror one drive may be absent, yet ZFS still lists the pool – if it does not appear, essential metadata drives are missing or the HBA is not detecting the disks.

May I force a FAULTED pool with -f?

No, that is the opposite of the right reaction. -f only overrides the hostid check, not the pool integrity. A FAULTED pool has a real redundancy problem (such as two failed drives in RAID-Z1). Before an import the missing drive must be recovered or replaced. For critical cases there is zpool import -FX, but that is specialist territory – we regularly see customer pools that were permanently destroyed by a careless -FX.

How long does a scrub take after the import?

Rule of thumb: a scrub reaches roughly the raw sequential read rate of the slowest drive in the vdev, spread across all vdevs. On a pool of eight 20 TB HDDs in RAID-Z2 we plan for 24 to 36 hours, while a pure NVMe pool is usually done in under an hour. The scrub runs alongside production load.

Do I need to export the pool before shutting down the system?

No, a clean shutdown or reboot is enough – the operating system exports the pool automatically. zpool export only matters when you intend to physically remove the drives or move them to another host. An export also writes the “exported” flag to the label and makes a later import without -f possible.

What happens if the target system has an older ZFS version than the pool?

Then zpool import refuses access as soon as the pool uses at least one feature the target does not know. A readonly import may still work in some cases – ZFS can read the data without modifying the unknown features. For production use the target must be updated to a compatible OpenZFS version. In our DATAZONE lab we standardise on TrueNAS 25.10 Goldeye as the stable base for new appliances, which effectively eliminates this problem for moves between our systems.

More on these topics:

Plan your TrueNAS sizing live
Translate this article straight into a concrete configuration: pick model, drives, RAID level and networking — and get the resulting capacity in real time.
Configure now

Need IT consulting?

Contact us for a no-obligation consultation on Proxmox, OPNsense, TrueNAS and more.

Get in touch