From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 11 09:40:49 2022 Received: (at submit) by debbugs.gnu.org; 11 Jun 2022 13:40:49 +0000 Received: from localhost ([127.0.0.1]:52206 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o01M8-00026T-C9 for submit@debbugs.gnu.org; Sat, 11 Jun 2022 09:40:48 -0400 Received: from lists.gnu.org ([209.51.188.17]:40738) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o01M5-00026K-Tv for submit@debbugs.gnu.org; Sat, 11 Jun 2022 09:40:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44846) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o01M4-00032v-Ox for bug-guix@gnu.org; Sat, 11 Jun 2022 09:40:45 -0400 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:45857) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o01M1-0001l0-LZ for bug-guix@gnu.org; Sat, 11 Jun 2022 09:40:43 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 75BD65C00EA for ; Sat, 11 Jun 2022 09:40:39 -0400 (EDT) Received: from imap43 ([10.202.2.93]) by compute2.internal (MEProxy); Sat, 11 Jun 2022 09:40:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=const.fun; h=cc :content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm1; t= 1654954839; x=1655041239; bh=OFPKtNqzSmK68K4uUHtg6L8C6NRQ2QV7m05 HA6cm9hY=; b=bk3wh5x0Tj3niWJPpN5j5x1nX46gb6JGdEGwYvR/Qm+xRHG+zYu OfF+4qY/hZ8rvisV0TfSiwlsFy6px+Sz1t5g64s1Ax7kHnSnTHpwCe6dNqCHbjZQ XTmmUIwvAfIihO53uoxxq3FG84379Rklmq48zTWR5tymK7qk2YARL2BwK5+FW+Oj GSFsi0MpmuoGwr6EoZsPhWh8T0eYgU9rX3cXL9IuAJdPlBYWcnQv/5FmpjcWnIvs NDlujdsC94ntkytr5NliFADKkUannQz5rI1LcjvD4r2SgP2fOV9qmUOlhZBWECKg LOK30L0ttZX7qJ6vlg8h6ZsN6ZSMmMY9mIQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:mime-version :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1654954839; x= 1655041239; bh=OFPKtNqzSmK68K4uUHtg6L8C6NRQ2QV7m05HA6cm9hY=; b=X tt6Kf8boA33/VuJfKwZQDuPqNT3jqSpBUe9f8sMsG3eCCiXPrleWF8XyVr1iFQi9 z2wqDVJY+AUdkfVjxZnHsDYBUm1NT8Ov9ZCFtSn7VebESQhsO4zjGRTSd2iHgjDE ccQnvzSfKU141U0uSL4GvrhNaS5FNyWyy8IpR7o10X2lEVf6jOFedm0+mctEurBl 4AFwrtT2nfurHIN7lQf/FJ5TnUakKgLM7j9qLtcj0O10URebLBO845i5Hu3hB0eb bBmvXu4Tu/ynE0mOH/76fQjpjMYAetz1YDmEOyOMSA8MEWAvzlkEq9EoKYcoLy8R pL5+fvD0ykjCEXUhcFOZw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddufedgieehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfffhffvufgtsegrtderre erredtnecuhfhrohhmpedfpfhitghkucgkrghluhhtshhkihihfdcuoehnihgtkhestgho nhhsthdrfhhunheqnecuggftrfgrthhtvghrnhepffethfdthefgkedugeetieekueevfe fhgfeiffetgfdvgeehleeitefhheehfeeinecuffhomhgrihhnpehgihhthhhusgdrtgho mhdprghrtghhlhhinhhugidrohhrghdpkhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepnhhitghksegtohhnshhtrdhf uhhn X-ME-Proxy: Feedback-ID: i25a94707:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 276A62D40070; Sat, 11 Jun 2022 09:40:39 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-692-gb287c361f5-fm-20220603.003-gb287c361 Mime-Version: 1.0 Message-Id: Date: Sat, 11 Jun 2022 09:40:18 -0400 From: "Nick Zalutskiy" To: bug-guix@gnu.org Subject: VFIO kernel module fails to capture PCI device Content-Type: multipart/alternative; boundary=88555654cab94e5396cdcfa3e24335ec Received-SPF: pass client-ip=66.111.4.26; envelope-from=nick@const.fun; helo=out2-smtp.messagingengine.com X-Spam_score_int: 17 X-Spam_score: 1.7 X-Spam_bar: + X-Spam_report: (1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_SUSPICIOUS_NTLD=0.499, FROM_SUSPICIOUS_NTLD_FP=1.997, HTML_MESSAGE=0.001, PDS_OTHER_BAD_TLD=1.997, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-Spam-Score: 2.5 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hello all, I am trying to capture my graphics card at initrd, using vfio, to later pass it through to a virtual machine. Judging by dmesg, the VFIO module does load early, however, the card is not captured at th [...] Content analysis details: (2.5 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [209.51.188.17 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.7 SPF_NEUTRAL SPF: sender does not match SPF record (neutral) 1.6 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: const.fun (fun)] 0.0 HTML_MESSAGE BODY: HTML included in message 2.0 FROM_SUSPICIOUS_NTLD_FP From abused NTLD 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD -0.0 T_SCC_BODY_TEXT_LINE No description available. X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) --88555654cab94e5396cdcfa3e24335ec Content-Type: text/plain Hello all, I am trying to capture my graphics card at initrd, using vfio, to later pass it through to a virtual machine. Judging by dmesg, the VFIO module does load early, however, the card is not captured at that point and the amdgpu driver is later loaded instead. This is what I have in my `operating-system` config: > (kernel-arguments '("iommu=pt" "vfio-pci.ids=1002:73bf")) > (initrd-modules (cons* "vfio_pci" "vfio" "vfio_iommu_type1" "vfio_virqfd" %base-initrd-modules)) There are two video cards in the system, both AMD, but different models. The video card of interest is in a separate IOMMU group and the : combination is correct for my machine. Best I can tell, vfio-pci.ids argument is not propagated to the module by initramfs. See the following: Searching online I came up against a GitHub issue for a different initramfs generator that exhibited the same symptoms: VFIO module was loaded, kernel arguments were correct, yet the card was not captured by the vfio driver. The maintainer there did a great job tracking down and fixing the issue and came up with this insight https://github.com/anatol/booster/issues/20#issuecomment-808956316 > After reading kmod code I found that kernel does not use cmdline params for loadable modules. It was surprising for me. Instead it is expected that userspace handles cmdline parsing and provides required module params explicitly. Another way to attach the correct driver to the gpu is to run a script at initrd, which I don't know how to accomplish with Guix. This approach has the advantage of working with two identical video cards (or disks, etc) See https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs I tried following the kernel docs to rebind a different driver after boot, but I believe this doesn't work for video cards, and hasn't worked for me. Any help is greatly appreciated! Links: Kernel docs for vfio https://www.kernel.org/doc/html/latest/driver-api/vfio.html Arch guide for GPU passthrough https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF Thank you! -Nick --88555654cab94e5396cdcfa3e24335ec Content-Type: text/html Content-Transfer-Encoding: quoted-printable
Hello all,
<= /div>

I am trying to capture my graphics card at init= rd, using vfio, to later pass it through to a virtual machine. Judging b= y dmesg, the VFIO module does load early, however, the card is not captu= red at that point and the amdgpu driver is later loaded instead.

This is what I have in my `operating-system` confi= g:

(kernel-argume= nts '("iommu=3Dpt" "vfio-pci.ids=3D1002:73bf"))
  (in= itrd-modules (cons* "vfio_pci" "vfio" "vfio_iommu_type1" "vfio_virqfd" %= base-initrd-modules))

There ar= e two video cards in the system, both AMD, but different models. The vid= eo card of interest is in a separate IOMMU group and the <vendor id&g= t;:<device id> combination is correct for my machine.

Best I can tell, vfio-pci.ids argument is not propagate= d to the module by initramfs. See the following:

Searching online I came up against a GitHub issue for a different = initramfs generator that exhibited the same symptoms: VFIO module was lo= aded, kernel arguments were correct, yet the card was not captured by th= e vfio driver. The maintainer there did a great job tracking down and fi= xing the issue and came up with this insight https://github.com/a= natol/booster/issues/20#issuecomment-808956316

After reading kmod code I found that k= ernel does not use cmdline params=20 for loadable modules. It was surprising for me. Instead it is expected=20 that userspace handles cmdline parsing and provides required module=20 params explicitly.

Another way= to attach the correct driver to the gpu is to run a script at initrd, w= hich I don't know how to accomplish with Guix. This approach has the adv= antage of working with two identical video cards (or disks, etc) See https://wiki.archlinux.org/title/PCI_pass= through_via_OVMF#Using_identical_guest_and_host_GPUs
<= br>
I tried following the kernel docs to rebind a different dr= iver after boot, but I believe this doesn't work for video cards, and ha= sn't worked for me.

Any help is greatly app= reciated!

Links:
Kernel docs = for vfio

Arch guide for GPU passthrou= gh

Thank you!

-Nick

--88555654cab94e5396cdcfa3e24335ec--