Something appears to disable linux kernel modules from loading.

Hello!

tl;dr

Something is causing linux 5.4.20 not to load kernel modules at boot.

Perhaps the culprit is the shepherd 0.7.0.

So I recently reconfigured guix on my ThinkPad T400.

#+BEGIN_SRC sh

$ guix describe

Generation 32 Feb 18 2020 16:53:46 (current)

jmacs 1a401c8

repository URL: https://notabug.org/jbranso/guix-packages.git

branch: master

commit: 1a401c81ed0617e2ca2fa37ec8caa9ca96032b7a

guix 97c6670

repository URL: https://git.savannah.gnu.org/git/guix.git

branch: master

commit: 97c6670ef79bbf5e82dece2e837ed73b6462da01

#+END_SRC

After I rebooted sway refused to start. The sway error message said

#+BEGIN_SRC sh

$ sway

[backend/backend.c:339] Failed to open any DRM device.

[sway/server.c] Unable to create backend.

#+END_SRC

So sway is not starting. I believe this is because my i915 intel

driver is not loaded. I know this is the case, because the text on my

virtual console is LARGE. $ sudo modprobe i915 loads the required

driver. And I can now log into sway.

However, my ethernet driver is not loaded.

#+BEGIN_SRC sh

$ lsmod | grep e1000e

#+END_SRC

#+BEGIN_SRC sh

$ sudo modprobe e1000e

#+END_SRC

Then I ran ifconfig, and indeed I now have an ethernet device.

#+BEGIN_SRC sh :results output :exports both

$ ifconfig

enp0s25 Link encap:Ethernet HWaddr 00:1C:25:9A:37:BA

inet addr:172.16.113.199 Bcast:172.16.115.255 Mask:255.255.252.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:505 errors:0 dropped:0 overruns:0 frame:0

TX packets:570 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:143838 TX bytes:60886

Interrupt:20 Memory:fc100000-fc120000

lo Link encap:Local Loopback

inet addr:127.0.0.1 Bcast:0.0.0.0 Mask:255.0.0.0

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 TX bytes:0

#+END_SRC

And I now have an internet connection.

#+BEGIN_SRC sh :results output :exports both

joshua@dobby ~/prog/org$ ping -c 3 gnu.org

PING gnu.org (209.51.188.148): 56 data bytes

64 bytes from 209.51.188.148: icmp_seq=0 ttl=56 time=35.610 ms

64 bytes from 209.51.188.148: icmp_seq=1 ttl=56 time=35.635 ms

64 bytes from 209.51.188.148: icmp_seq=2 ttl=56 time=35.501 ms

--- gnu.org ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max/stddev = 35.501/35.582/35.635/0.058 ms

#+END_SRC

But I do not have a working touchpad. I guess my touchpad driver is

not loaded. I'm not certain what driver my touchpad uses, so for now

I am doing with out.

I then went and asked for some tips on irc. HappyEnt claimed the

problem for him was commit 205c1e0.

janneke was also having issues with kernel modules not being

automatically booted. He said that this commit worked for him:

ef2b1d2c8b60776bf66573b0c810474e9e1a4abd. He then said that commit

d39885a8a9e0e03c2bf6277d475d384168bba642 also worked, which implied

the problem was not the linux version 5.4.20, but possibly the

shepherd.

I tried reconfiguring with commit

d39885a8a9e0e03c2bf6277d475d384168bba642, to see if that solved my issue.

#+BEGIN_SRC sh :results output :exports both

guix time-machine --commit=d39885a8a9e0e03c2bf6277d475d384168bba642 -- system reconfigure

#+END_SRC

#+BEGIN_SRC sh

$ uname -r

5.4.20-gnu

#+END_SRC

janneke's commit did solve my issue. After rebooting, sway starts, I

have internet, and my touchpad and sound work.

Man I love guix system. It really does work smoothly, even when you

have problems. So thanks for making a great GNU/Linux distro!

Thanks Joshua

Hello Guix!

After my system reconfigure, GDM didn't start. During boot, I saw "New

gdm session c1" followed by "Session c1 removed".

For me, the last known commit to work properly is "41884bf". Anything

after that breaks my system.

Regards,

RG.

Hi!

Joshua Branson <jbranso@dismail.de> skribis:

On IRC, tsmish posted this bisect log:

What does “cat /proc/sys/kernel/modprobe” return on the broken system?

I would expect 8b9cad01e9619f53dc5a65892ca6a09ca5de3447 to be suspect #1

because it potentially leads to ‘LINUX_MODULE_DIRECTORY’ being unset in

places where it used to be set before.

Heh. :-)

Ludo’.

On Wed, 19 Feb 2020, Ludovic Courtès wrote:

/gnu/store/daq5zs7ni529zh3xxgyhidna52wa17js-modprobe

The contents of that file are:

"""

#!/gnu/store/1mkkv2caiqbdbbd256c4dirfi4kwsacv-guile-2.2.6/bin/guile --no-auto-compile

!#

(begin (setenv "LINUX_MODULE_DIRECTORY" "/run/booted-system/kernel/lib/modules")

(apply execl "/run/current-system/profile/bin/modprobe"

(cons "/run/current-system/profile/bin/modprobe" (cdr (command-line)))))

"""

Based on the commit message, I would have expected that too, but

205c1e04e04b9a9338c7219ff82bd13f000fb8c8 is the first bad commit for me as

well. The above modprobe script is from a system at commit

8b9cad01e9619f53dc5a65892ca6a09ca5de3447.

Best,

Jack

severity 39671 important

quit

Jack Hill <jackhill@jackhill.us> skribis:

The kernel’s lazy module loading mechanism invokes this program, which

should just work.

Does /run/current-system/profile/bin/modprobe exist?

Any hints in “dmesg” or /var/log/messages?

Thanks,

Ludo’.

On Wed, 19 Feb 2020, Ludovic Courtès wrote:

Yes, it is a symlink to

/gnu/store/d064bv2b1hrb07j2zj78i608db7qldx9-kmod-26/bin/modprobe

Not that jump out at me. I do see

udevd[197]: no sender credentials received, message ignored

printed in red, but that message also appears when everything is working

correctly.

The full dmesg output is attached. The bit at the end about e1000e was

after I manually ran `modprobe e1000e`.

Best,

Jack

On Wed, Feb 19, 2020 at 04:50:06PM -0500, Jack Hill wrote:

I can confirm this commit broke xf86-video-fbdev (no DRI support), USB

ethernet and USB modem support for me. The output of running as root

out=~florian/environ-bad ; for f in $(ls /proc); do ls -l /proc/$f/exe; cat /proc/$f/environ; done > $out; chown florian:users $out

is attached as environ-bad.

Patching Shepherd to revert 501d62c4d235045e79e5aef684b5a9250c1cf8c4

(service: Export 'default-environment-variables') leads to a kernel

panic. Doing that and reverting Guix commit

8b9cad01e9619f53dc5a65892ca6a09ca5de3447 (after a rebuild of udisks

and dependencies — why?) leads to a working system again. It is

attached as environ-good.

Regards,

Florian

On Thu, Feb 20, 2020 at 11:18:29AM +0100, pelzflorian (Florian Pelz) wrote:

P.S. It is *not* sufficient to only revert

8b9cad01e9619f53dc5a65892ca6a09ca5de3447.

So I just ran guix pull & reconfigure.

joshua@dobby /run/booted-system/kernel/lib/modules$ guix describe

Generation 36 Feb 20 2020 10:11:40 (current)

jmacs 65b1737

repository URL: https://notabug.org/jbranso/guix-packages.git

branch: master

commit: 65b17377136095935ef6da3a836142d56fecce2b

guix bf8dfe3

repository URL: https://git.savannah.gnu.org/git/guix.git

branch: master

commit: bf8dfe3df025e4ac80cccb87497b4f072ba10e2a

So I feel like Jack beat me to it,

but the contents of /proc/sys/kernel/modprobe are:

#+BEGIN_SRC sh

$ cat /proc/sys/kernel/modprobe

/gnu/store/daq5zs7ni529zh3xxgyhidna52wa17js-modprobe

#+END_SRC

#+BEGIN_SRC sh

cat /gnu/store/daq5zs7ni529zh3xxgyhidna52wa17js-modprobe

#!/gnu/store/1mkkv2caiqbdbbd256c4dirfi4kwsacv-guile-2.2.6/bin/guile --no-auto-compile

!#

(begin (setenv "LINUX_MODULE_DIRECTORY" "/run/booted-system/kernel/lib/modules") (apply execl "/run/current-system/profile/bin/modprobe" (cons "/run/current-system/profile/bin/modprobe" (cdr (command-line)))))~ $

#+END_SRC

Then I was just poking around.

#+BEGIN_SRC sh

~ $ cat /run/booted-system/kernel/lib/modules

Read error: Is a directory, /run/booted-system/kernel/lib/modules

~ $ cd /run/booted-system/kernel/lib/modules

/run/booted-system/kernel/lib/modules $ ls

5.4.21-gnu

/run/booted-system/kernel/lib/modules $ cd 5.4.21-gnu/

/run/booted-system/kernel/lib/modules/5.4.21-gnu $ ls

build modules.alias modules.builtin modules.builtin.modinfo modules.dep.bin modules.order modules.symbols source

kernel modules.alias.bin modules.builtin.bin modules.dep modules.devname modules.softdep modules.symbols.bin

/run/booted-system/kernel/lib/modules/5.4.21-gnu $ cat modules.order

kernel/arch/x86/events/amd/power.ko

kernel/arch/x86/kernel/msr.ko

kernel/arch/x86/kernel/cpuid.ko

kernel/arch/x86/crypto/glue_helper.ko

kernel/arch/x86/crypto/des3_ede-x86_64.ko

kernel/arch/x86/crypto/camellia-x86_64.ko

kernel/arch/x86/crypto/blowfish-x86_64.ko

kernel/arch/x86/crypto/twofish-x86_64.ko

kernel/arch/x86/crypto/twofish-x86_64-3way.ko

... tons more modules...

kernel/drivers/gpu/drm/i915/i915.ko

kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko

#+END_SRC

You'll notice that my i915 graphics and e1000e (ethernet) driver is

listed in that directory.

On my working system, these were the modules that lspci -v showed I was

using:

#+BEGIN_SRC sh :results output :exports both

lspci -v | grep "Kernel driver in"

#+END_SRC

#+RESULTS:

#+begin_example

Kernel driver in use: i915

Kernel driver in use: e1000e

Kernel driver in use: uhci_hcd

Kernel driver in use: uhci_hcd

Kernel driver in use: uhci_hcd

Kernel driver in use: ehci-pci

Kernel driver in use: snd_hda_intel

Kernel driver in use: pcieport

Kernel driver in use: pcieport

Kernel driver in use: pcieport

Kernel driver in use: pcieport

Kernel driver in use: pcieport

Kernel driver in use: uhci_hcd

Kernel driver in use: uhci_hcd

Kernel driver in use: uhci_hcd

Kernel driver in use: ehci-pci

Kernel driver in use: lpc_ich

Kernel driver in use: ata_piix

Kernel driver in use: i801_smbus

Kernel driver in use: yenta_cardbus

Kernel driver in use: firewire_ohci

#+end_example

On the broken system, I am able to load my i915 and e1000e driver, but I

am not able to load yenta_cardbus.

#+BEGIN_SRC sh

$ sudo modprobe yenta_cardbus

modprobe: FATAL: Module yenta_cardbus not found in directory /run/booted-system/kernel/lib/modules/5.4.21-gnu

#+END_SRC

Sorry if I'm being repetitive. I am just wanting to help. :)

--

Joshua Branson

Sent from Emacs and Gnus

The attached patch fixes the issue for me. Can someone confirm?

It replaces or copies setenv calls for the udev service.

There are more service definitions that call setenv; I do not know if

they need to be moved too.

I will try if a smaller patch would suffice as well now.

Regards,

Florian

From 11509370049355680c815e06250afc09325c589c Mon Sep 17 00:00:00 2001

environment variables.

Fixes https://bugs.gnu.org/39671.

environment variables to 'fork+exec-command' instead of 'setenv'.

---

gnu/services/base.scm | 19 +++++++++++++------

1 file changed, 13 insertions(+), 6 deletions(-)

On Thu, Feb 20, 2020 at 08:59:25PM +0100, pelzflorian (Florian Pelz) wrote:

Thanks! Which commmit is this based on? I can't apply it with `git

am`...

On Thu, Feb 20, 2020 at 03:34:02PM -0500, Leo Famulari wrote:

:)

commit bfb6c393b91dc4a236678a0682348d5b7bd77193 (origin/master, origin/HEAD)

Author: Ricardo Wurmus <rekado@elephly.net>

gnu: gwl: Build with Guile 3.

(I wonder why it does not apply; there were no changes to

gnu/services/base.scm in a long time. Strange.)

Regards,

Florian

On Thu, Feb 20, 2020 at 09:43:45PM +0100, pelzflorian (Florian Pelz) wrote:

It did not apply for me either. Here it is in compressed form so its

encoding is not changed.

Thank you Leo!

Regards,

Florian

On Thu, Feb 20, 2020 at 09:43:45PM +0100, pelzflorian (Florian Pelz) wrote:

Oh well, I applied it "by hand" and everything seems to be working

again :) Thank you!

pelzflorian (Florian Pelz) writes:

Hi Florian,

It works for me; thank you! It did not apply -- I do not have this

From 11509370049355680c815e06250afc09325c589c Mon Sep 17 00:00:00 2001

commit.

Attached is the result of my manual application...

janneke

--

Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org

Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

Good catch, it makes perfect sense. Please push!

Ludo’, who, ahem, apologizes for the inconvenience.

Thank you Jan and Leo for testing and Ludo for confirming it makes

sense, and of course to all bug reporters. Closing this bug. I guess

I will switch e-mail programs so maybe the encoding of patches I send

is not broken.

And thank you to Ludo for making important (though breaking) changes

to Shepherd, making it more functional. Other than udev, the other

uses of setenv in (gnu services …) look benign.

Regards,

Florian