In a Proxmox machine I noticed some of the backups of some VM's were failing, so I wanted to test stuff. Whilst testing the whole host stopped responding and I forced a reboot. After the reboot I seem to have lost the whole data store. Almost every zfs command results in a freeze. zpool status,zpool list, you name it, it locks up and you can't even ctrl break out of it. I can still create a new SSH session and try other things though. In an attempt to see what is causing the commands to hang I thought about running
zpool set failmode=continue
hoping it will show me an error, but as you can guess, that command also hangs.
It's a pool created on two nvme drives. The original command to create the pool was
zpool create -f -o ashift=12 storage-vm /dev/nvme0n1 /dev/nvme1n1
First thing I thought was that one of the nvme's had gone bad so I checked the SMART status, but it shows both drives are perfectly healthy. Then before trying other stuff I decided to backup the drives to an NFS share with the dd command.
dd if=/dev/nvme0n1 of=/mnt/pve/recovery/nvme0n1
dd if=/dev/nvme1n1 of=/mnt/pve/recovery/nvme1n1
Both commands completed and on the NFS share I have 2 images of the exact same size (2TB each)
Then I tried to do a non destructive read/write test with dd on both the nvme's and got no errors. In order to rule out as much as possible I build another Proxmox machine using spare hardware (same brand and type etc.) and place the drives in there. On the new machine all zpool commands also hang. If i run zpool status with the drives removed from the motherboard, it does not hang, but obviously it has nothing to show. So I placed the nvme's back in the original machine.
zdb -l /dev/nvme0n1 gives
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3
which kind of worries me. It does the same for the other nvme.
And now I'm running out of ideas. I have little knowledge of the zfs system and don't know what is possible to save the data. Obviously, the drives are not really dead as the smart tells me it is healthy and I can dd an image from them. Things like faulty RAM or motherboard are pretty much ruled out also with the hardware swap. Is there a way to recover at least some VM's from that storage? Help/pointers wil be greatly appreciated.