VFIO kernel module fails to capture PCI device

  • Open
  • quality assurance status badge
Details
3 participants
  • Nick Zalutskiy
  • Nikola Brković
  • Lars Rustand
Owner
unassigned
Submitted by
Nick Zalutskiy
Severity
normal

Debbugs page

Nick Zalutskiy wrote 3 years ago
(address . bug-guix@gnu.org)
ed5747f5-eaa6-4ae5-8434-d9db467079b5@www.fastmail.com
Hello all,

I am trying to capture my graphics card at initrd, using vfio, to later pass it through to a virtual machine. Judging by dmesg, the VFIO module does load early, however, the card is not captured at that point and the amdgpu driver is later loaded instead.

This is what I have in my `operating-system` config:

Toggle quote (3 lines)
> (kernel-arguments '("iommu=pt" "vfio-pci.ids=1002:73bf"))
> (initrd-modules (cons* "vfio_pci" "vfio" "vfio_iommu_type1" "vfio_virqfd" %base-initrd-modules))

There are two video cards in the system, both AMD, but different models. The video card of interest is in a separate IOMMU group and the <vendor id>:<device id> combination is correct for my machine.

Best I can tell, vfio-pci.ids argument is not propagated to the module by initramfs. See the following:

Searching online I came up against a GitHub issue for a different initramfs generator that exhibited the same symptoms: VFIO module was loaded, kernel arguments were correct, yet the card was not captured by the vfio driver. The maintainer there did a great job tracking down and fixing the issue and came up with this insight https://github.com/anatol/booster/issues/20#issuecomment-808956316

Toggle quote (2 lines)
> After reading kmod code I found that kernel does not use cmdline params for loadable modules. It was surprising for me. Instead it is expected that userspace handles cmdline parsing and provides required module params explicitly.

Another way to attach the correct driver to the gpu is to run a script at initrd, which I don't know how to accomplish with Guix. This approach has the advantage of working with two identical video cards (or disks, etc) See https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs

I tried following the kernel docs to rebind a different driver after boot, but I believe this doesn't work for video cards, and hasn't worked for me.

Any help is greatly appreciated!

Links:
Kernel docs for vfio

Arch guide for GPU passthrough

Thank you!

-Nick
Attachment: file
Lars Rustand wrote 2 years ago
(address . 55907@debbugs.gnu.org)
dc4ppksuoymmrnxpphcm4hsblqp5rqcr6mgcdlrkds63mrbxmm@ief6maxjt5af
Hello Nick,

Did you ever figure this out? I am struggling with the same problem.


Thank you,

- Lars
Nikola Brković wrote 8 months ago
(name . 55907@debbugs.gnu.org)(address . 55907@debbugs.gnu.org)
VuGXUGjGQnHxEiL7CJApZEAU1eqUAndvtqNH7YqDhdef982qR0yMCHqTd8DcAWFXTueEcXdK3sHbxtYvGAjW4g0L_3eERvp1mXX_RjZztSA=@protonmail.com
I have managed to get VFIO working by creating a service of boot-service-type which overrides the GPU driver with vfio-pci and binds the GPU to VFIO:

Toggle quote (10 lines)
>(simple-service 'vfio-override boot-service-type
>    '(and (call-with-output-file "/sys/bus/pci/devices/0000:04:00.0/driver_override"
>    (lambda (p)
>      (display "vfio-pci" p)))
>  (call-with-output-file "/sys/bus/pci/drivers/vfio-pci/new_id"
>    (lambda (p)
>      (display "1002 665f" p)))
>  )
>)

Sorry for the hard-coded IDs, you should replace them with your own. You might need to unbind the GPU's audio card from its driver as well, after you're fully booted. QEMU will refuse to pass-through the GPU if the audio card is in the same IOMMU group and not using vfio-pci.

In my case, the service runs early enough in the boot process where amdgpu has not initialized the GPU yet. There might be a better way to accomplish this, I'm still new to Guix and Scheme.

Thanks,
Nikola
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 55907@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 55907
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch
You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags
mumi command -t +confirmed -t +easy
Or, remove the moreinfo tag and set the help tag
mumi command -t -moreinfo -t +help