Re-Enumerate and use PCIe SSD in Linux without shutdown

Viewed 6893

Good day,

I am currently working on a project where PCIe SSDs are constantly being swapped out and tested through benchmark programs such as VDBench and Iometer. The problem I face right now, which is only on the Linux side (got it working fine on windows), is that if the drives were not on at initial boot-up, they never appear under GParted or Disks. Here's what I have done:

Cold boot, PCIe Add-in-card SSD is off. It is then powered on through a pass through card that is logically controlled to make sure power and shorts are not an issue.

I turn the device on, then run:

sudo sh -c "echo 1 > /sys/bus/pci/rescan"

Performing a

lspci -tv

The device shows with no issues in the tree. When I check under Disks however, it is not there.

I have tried a bunch of different commands with none of them seeming to help me. I have tried

partprobe

Which did not do anything. and:

sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:82:00.0/remove"

Followed up another rescan:

sudo sh -c "echo 1 > /sys/bus/pci/rescan"

As well as:

sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:82:00.0/enable"

Still nothing. Also ran:

dmesg

Which shows, amongst other things:

[   68.128778] pci 0000:82:00.0: [8086:0953] type 00 class 0x010802
[   68.128797] pci 0000:82:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[   68.128820] pci 0000:82:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[   68.133398] pci 0000:84:00.0: [1c58:0003] type 00 class 0x010802
..............................

[   68.141751] nvme 0000:82:00.0: enabling device (0100 -> 0102)
..............................

I do see a lot of fails in dmesg for other addresses, such as:

[ 1264.718446] pcieport 0000:03:04.0: BAR 14: no space for [mem size     0x00400000]
[ 1264.718448] pcieport 0000:03:04.0: BAR 14: failed to assign [mem size 0x00400000]
[ 1264.718451] pcieport 0000:03:04.0: BAR 13: no space for [io  size 0x1000]
[ 1264.718453] pcieport 0000:03:04.0: BAR 13: failed to assign [io  size 0x1000]

Although I have a feeling that those are unrelated to what I am doing, although I'd be happy for someone to prove me wrong.

So, after all of these attempts, does anyone know if there is a way (or if it is even possible) to scan for this PCIe Add-in NVMe SSD and be able to use it without rebooting? I also took a look at some of the threads for other HDDs that reference a rescan for sata based drives, but this is NOT that, so referencing that won't help either.

Thanks in advance.

2 Answers

I ran into the same issue benchmarking nvme PCIE passthrough with QEMU / Proxmox.

First take note of the driver in use:

lspci -nnk -s '0000:82:00.0'

It should say

Kernel driver in use: vfio-pci

Now unbind the driver, then reprobe:

echo '0000:82:00.0' > /sys/bus/pci/drivers/vfio-pci/unbind
echo '0000:82:00.0' > /sys/bus/pci/drivers_probe

Check the driver again with:

lspci -nnk -s '0000:82:00.0'

Kernel driver in use: nvme

lsblk should now show the drive. Found the procedure here

Related