The 70,000-Hour Time Bomb: How We Brought a Dead Storage Cluster Back to Life Without Losing a Single Byte

We deliberately engaged two data-recovery labs to leave no option on the table — neither would flash the drives. The backup was six months old, an entire cluster was down — and in the end all data came back without loss. A case study with full technical documentation.

The Call

„Our RAID is broken. Two caching SSDs are said to be dead.” The patient: a Fujitsu PRIMERGY cluster, two nodes, shared SAS storage, running on Windows Server Storage Spaces in a failover cluster.

One thing up front, because it matters: This was not a customer under our Managed Service, but an external emergency request. The condition we found — unpatched SSD firmware and a mere six-monthly backup — is explicitly not what we expect of our own customers. In our Managed Backup service, a data set that old would be simply unthinkable.

Our approach is always the same: purely passive analysis first, touch nothing.

The Diagnosis

A classic cascade scenario: the storage pool was reported as Degraded. The 6 TB data volume — a parity space used as a Cluster Shared Volume — was offline and could not be mounted. Almost all VMs had failed, including the only domain controller. And because that DC lived as a VM on precisely this storage, there was no more Kerberos logon server — the nodes could no longer authenticate cleanly.

We decoded the storage layer passively with PowerShell:

Get-PhysicalDisk | Select FriendlyName,MediaType,Usage,HealthStatus,OperationalStatus,Size
Get-StoragePool -FriendlyName <Pool> | fl HealthStatus,OperationalStatus,IsReadOnly
Get-VirtualDisk  | fl FriendlyName,ResiliencySettingName,WriteCacheSize,OperationalStatus,DetachedReason
Get-ClusterSharedVolume | Select Name,State,OwnerNode

Two SSDs reported themselves as 0 bytes with Lost Communication and Usage: Retired. In the configuration they were the mirrored journal / write-back cache of the parity volume. Both gone at once = no copy of the journal = the volume refuses to attach. The actual payload data was sitting untouched on eight healthy HDDs.

The Root Cause: A Firmware Countdown Straight From the Factory

The suspects: Toshiba PX02SMF020, 200 GB SAS SSDs. This series — and many identical OEM variants from Dell, HPE and Fujitsu — carries a notorious firmware bug: after exactly 70,000 operating hours (~7 years, 11 months) the drive locks itself out on the next power cycle. It then reports 0 bytes and refuses read and write commands. The vendors have released preventive firmware; once the lock has triggered, it is widely considered a dead end.

That both SSDs died practically at the same time was no coincidence: identical operating hours, identical mirror load — the same countdown ran in lockstep.

Crucially: this is not a wear-out death. The NAND is intact — only the firmware has locked itself down. And that is reversible.

We Engaged Two Data-Recovery Labs — Both Declined

To leave no option unexplored, we engaged two specialised data-recovery companies in parallel and described the case. Both refused to flash the drives — they only offered classic data recovery: reading the SSDs out in the lab.

That would have accomplished nothing here. The SSDs held no payload data, only the mirrored journal of the parity volume. A raw image of these SSDs does not reconstruct a Storage Spaces volume — the actual data was on the HDDs anyway. The only lossless path was to bring the physical drives themselves back to life so their journal would return. Exactly what both labs ruled out.

And the pressure was real: the latest backup was six months old. A restore would have cost half a year of data. Repair rather than recovery was therefore not the more elegant but the only acceptable option — and we implemented it ourselves.

The Technical Path in Detail

1. Verify the Drive State

With the open-source openSeaChest the drives were addressable despite reading „0 bytes”:

openSeaChest_Firmware --scan
# TOSHIBA  PDx  PX02SMF020  <SN>  5203

openSeaChest_Firmware -d PDx -i
# Drive Capacity: 0.00 / 0.00     MaxLBA: 0
# SMART Status: Unknown or Not Supported
# Firmware Download Support: Not Supported   <- lock active

openSeaChest_Firmware -d PDx --fwdlInfo
# Modes Supported: Full, Segmented           <- a download path does exist

2. Obtain the Right Firmware

Generic community images were rejected by the OEM-branded drives with Sense 5h/26h — Invalid Field in Parameter List — a signature mismatch. The answer was the vendor-issued firmware package bound to the serial number — Fujitsu ships its „SSD Firmware update tool for PM2”, a bootable flash ISO.

Important: This firmware is tied to the server serial number and must not be redistributed publicly. Always obtain the image through your vendor support channel for your drives.

3. Extract the Firmware From the Boot Image

Only the pure firmware binary from the ISO — layer by layer via WSL, no root needed:

# a) unpack ISO -> INITRD.IMG, without mount, via pycdlib
python3 -c "import pycdlib;i=pycdlib.PyCdlib();i.open('FIXdisk.iso');\
i.get_file_from_iso('initrd.img',iso_path='/ISOLINUX/INITRD.IMG;1');i.close()"

# b) unpack initrd (gzip + cpio)
zcat initrd.img | cpio -idm

# c) inside lives the root FS as a bzip2 tarball
bunzip2 -k root.img.bz2 && tar -xf root.img

# d) result: the signature-correct firmware
#    Data/PM252050100.frm  (FW 5205)  Header "PX02DHWF ... PM2:52:05:01:00"

A look at the vendor flash scripts revealed the transfer mode: WRITE BUFFER, Mode 7 (segmented).

4. Flash Under Windows — The Decisive Trick

First attempts aborted mid-segment stream (26h, then 20h — Invalid Command Operation Code on the final segment). The breakthrough: the switch that ignores exactly this final-segment error.

openSeaChest_Firmware -d PDx --downloadFW PM252050100.frm ^
    --downloadMode segmented --forceSCSI --forceFWDLPassthrough ^
    --fwdlIgnoreFinalSegment

# Firmware Download successful
# New firmware version is 5205
# Exit code 32 = Firmware Download Complete

5. Verify the Result

openSeaChest_Firmware -d PDx -i
# Firmware Revision: 5205
# Drive Capacity: 200.05 GB / 186.31 GiB
# SMART Status: Good

From 0 bytes to 200 GB, from Unknown to Good — on both drives. Data intact.

The Finale: A Clean Restart

A controlled restart was enough for Windows to re-enumerate the revived SSDs. After that the cluster healed itself: the pool moved to Online / Healthy, quorum went green, the first VMs came back. Because the mirrored journal was back, the 6 TB parity volume could be re-attached without data loss — no „dirty” force-attach, no restore from a six-month-old backup.

„The RAID is dead” and the prospect of six months of data loss turned into full data rescue without a single lost record.

What You Should Take Away

1. Know the operating hours of your SSDs. Enterprise SSDs from ~2015–2017 are hitting the 70,000-hour mark right now.
2. Apply firmware fixes before they detonate. Once patched the drive is immune — before the fix it is a time bomb.
3. Redundancy does not protect against systematic faults. Two identical mirror SSDs of the same age die together.
4. Backups must be current and tested. A six-monthly backup like this external case is a nightmare — we back up our managed customers tightly, automatically and with verified restores.
5. Never virtualise the sole DC on the very storage whose availability depends on that DC. This deadlock amplifies every outage.
6. Data recovery is not the right tool for every problem. Where two labs would only „read out”, repairing the device was the only lossless path.

Facing a „Dead” Storage Yourself?

Before you pay an expensive lab or lose weeks of backups: talk to us. We analyse passively first, without risking anything — and often find a path that is cheaper, faster and lossless. Exactly like here.

DATAZONE is your partner for Storage, Virtualisation and Data Recovery — get in touch.