Linux ZFS Boot Mirror Broken: 4 Rescue Approaches

A ZFS root mirror is one of the most robust boot configurations for Linux servers. In practice we still see Proxmox and Ubuntu systems that refuse to start after a power cut, a firmware update or the failure of one mirror half. The screen stays black, GRUB drops into the rescue shell or initramfs bluntly reports cannot import 'rpool': no such pool available.

This article describes four proven approaches we use at customer sites to bring a broken ZFS boot pool back online. They are ordered by effort and risk: from a simple chroot repair to a full reimport from snapshot. The commands are deliberately conservative and have been verified on Proxmox VE 8/9 and Ubuntu 22.04/24.04.

Why ZFS boot pools die in the first place

An rpool consists of several layers that all have to play together: the EFI system partition (ESP) with the bootloader, the ZFS partition with the pool membership label and finally the kernel plus initramfs inside a dataset. If any of these layers fails, the symptom looks the same, yet the cause lies elsewhere.

Typical causes we see:

Symptom	Likely cause	First check
GRUB rescue shell, no menu	ESP not found or GRUB module missing	Boot order in BIOS/UEFI
Initramfs prompt, no pool available	ZFS label corrupt or cache stale	zpool import from live ISO
Kernel panic on mount	Wrong kernel/initramfs version	zfs list -t snapshot
Only one drive boots	Second mirror member missing from ESP chain	efibootmgr -v or grub-install
Boot hangs at Loading initial ramdisk	Initramfs contains no ZFS module	update-initramfs -u -k all

Important: a ZFS mirror protects against data loss on the block layer, not against a broken bootloader. If you replaced a disk and ran grub-install only on the first drive, you effectively no longer have a boot mirror, even if zpool status cheerfully reports ONLINE.

Approach 1: chroot from the live ISO

The standard path for most cases. Boot a suitable live ISO, import the pool writable and switch into the installed system via chroot. From that point on every standard tool works exactly as in a normal session.

Prerequisite is a live environment with a matching ZFS module. For Proxmox we recommend the Proxmox VE ISO in debug mode, for Ubuntu a recent Live Server image with zfsutils-linux installed on top.

# 1. Find the pool and import it with an alternate mountpoint
zpool import
zpool import -f -R /mnt rpool

# 2. Bind kernel pseudo-filesystems into place
mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys

# 3. Mount the EFI partition (adjust path to your layout)
mount /dev/nvme0n1p2 /mnt/boot/efi

# 4. Enter the system
chroot /mnt /bin/bash

Inside the chroot you have the full system available: apt, dpkg, grub-install, update-initramfs, zpool status. The -R /mnt on the import is critical, otherwise ZFS will try to mount on / and destroy the live session.

Once finished, exit cleanly:

exit
umount -R /mnt/boot/efi /mnt/sys /mnt/proc /mnt/dev
zpool export rpool
reboot

The zpool export is not cosmetic. Without it, the pool membership label remains as “in use by another system” and the next boot will complain with pool was previously in use from another system.

Approach 2: rebuild the bootloader

If the pool imports cleanly but the system still refuses to boot, the bootloader is usually to blame. Typical triggers: a GRUB update that broke the ESP chain, a disk replacement without proxmox-boot-tool init, or a new kernel version whose initramfs never got ZFS support.

Inside the chroot from approach 1, run:

# Proxmox systems (systemd-boot or GRUB since PVE 7):
proxmox-boot-tool status
proxmox-boot-tool refresh

# When the ESP has to be initialised from scratch:
proxmox-boot-tool format /dev/nvme0n1p2
proxmox-boot-tool init /dev/nvme0n1p2

# Ubuntu / classic GRUB:
grub-install --target=x86_64-efi \
             --efi-directory=/boot/efi \
             --bootloader-id=ubuntu --recheck
update-grub
update-initramfs -u -k all

For mirrored setups grub-install or proxmox-boot-tool init has to run separately for every ESP partition. We regularly see systems where only the first drive has a working ESP, so the failure of the first SSD then takes down the entire server, even though the ZFS mirror itself keeps running fine.

For verification:

efibootmgr -v          # lists all EFI boot entries
proxmox-boot-tool status
zpool status -v rpool

All three outputs have to match: same kernel, same ESP UUID, same pool members. When they do not, the system probably only boots as long as the “right” drive sits in the “right” port.

Approach 3: emergency single-drive boot

Sometimes one mirror half is physically dead while the other is still on the controller. ZFS can happily boot from that, the bootloader often cannot. The goal of this approach: get the system back into production quickly before you install a replacement drive at your leisure.

Steps inside the chroot:

# 1. Mark the missing drive as detached in the pool
zpool status rpool
zpool detach rpool <missing-guid>

# 2. Make sure the surviving drive is bootable
proxmox-boot-tool status                     # Proxmox
grub-install --target=x86_64-efi \
             --efi-directory=/boot/efi ...   # Ubuntu

# 3. Reboot and verify

Order matters: first detach, then rewrite the bootloader. A zpool remove is not what you want for mirror pools, that is meant for log/cache vdevs. For mirror members zpool detach is the correct tool.

After the reboot the server is up again but has no redundancy. As soon as the replacement drive arrives:

# Partition the new drive with the same layout as the survivor
sgdisk --replicate=/dev/nvme1n1 /dev/nvme0n1
sgdisk --randomize-guids /dev/nvme1n1

# Attach the ZFS partition to the existing mirror
zpool attach rpool <surviving-part> /dev/nvme1n1p3

# Add the new drive's ESP to the boot canon
proxmox-boot-tool format /dev/nvme1n1p2
proxmox-boot-tool init /dev/nvme1n1p2

Resilvering runs in the background. zpool status rpool shows progress and estimated time remaining.

Approach 4: full reimport from snapshot

The last resort when the pool no longer imports cleanly or somebody accidentally ran zpool destroy. Prerequisite: you have zfs send snapshots on a separate target, ideally on a TrueNAS backup target or at a second site.

Rough sequence:

1. Fresh install on new drives, same ZFS version as the source system.
2. New rpool is created, system boots empty.
3. Boot the live ISO, try to import the old rpool (if still partially present) with zpool import -F -R /mnt rpool_old. The -F allows rollback to the last consistent transaction group, which is often enough after an abrupt power loss.
4. If that fails, restore snapshots from backup:

zfs receive -F rpool/ROOT/pve-1 < /mnt/backup/rpool_root_2026-07-29.zfs
zfs receive -F rpool/data       < /mnt/backup/rpool_data_2026-07-29.zfs

5. Rebuild the bootloader with the steps from approach 2.
6. Verify zfs set mountpoint=/ rpool/ROOT/pve-1 and zfs set bootfs=rpool/ROOT/pve-1 rpool.

Depending on data volume this takes hours to days. It is not something you improvise on the day, it is something you rehearse once a year as a restore drill.

UEFI vs. BIOS: the most common pitfalls

A big share of the rescue cases we get called into come down to UEFI/BIOS mix-ups. A system installed in UEFI mode that accidentally boots in Legacy mode will not find the pool even though technically nothing is broken.

Checklist when nothing else fits:

• Check [ -d /sys/firmware/efi ] in the live ISO: are you actually in UEFI?
• Secure Boot in UEFI: older unsigned ZFS modules leave the screen black.
• CSM/Legacy in UEFI must be disabled when the install was UEFI.
• On Proxmox: proxmox-boot-tool status shows whether systemd-boot or GRUB is active. Running both in parallel reliably ends in chaos.
• On Ubuntu: /boot/efi and /boot are separate partitions, please do not mix them up.

When in doubt, log lsblk -f, efibootmgr -v and zpool status -v to a file on a USB stick before any major change. That saves a lot of searching in an emergency.

Conclusion

A ZFS boot pool is robust, not magic. The four approaches described here cover the vast majority of cases we see in customer systems. What matters is picking the right path: not every boot problem is a pool problem, and not every pool problem can be solved with grub-install. Anyone who stores zfs send snapshots outside the system on a regular basis and runs an annual restore test will also stay calm during total failures.

DATAZONE helps you secure your Linux and Proxmox servers with ZFS root mirror, backup strategies on TrueNAS and emergency recovery. For design, rescue work or a backup concept that also works in reality, get in touch: contact us or book a session for our Linux consulting.