[PATCH 0/4] Load Linux module only when supported hardware is present.

OpenSubmitted by Danny Milosavljevic.
D
Danny Milosavljevic wrote on Sun Feb 25 12:45:57+0100 2018
(address . guix-patches@gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180225114557.816-1-dannym@scratchpost.org
Danny Milosavljevic (4):
gnu: kmod: Split off kmod-minimal.
gnu: Add kmod-minimal-static.
linux-initrd: Add kmod.
linux-boot: Load kernel modules only when the hardware is present.

gnu/build/linux-boot.scm | 24 ++++++++++--
gnu/build/linux-initrd.scm | 13 ++++++-
gnu/packages/linux.scm | 91 +++++++++++++++++++++++++++++++++++++--------
gnu/system/linux-initrd.scm | 52 ++++++++++++++++++++------
4 files changed, 148 insertions(+), 32 deletions(-)
D
Danny Milosavljevic wrote on Sun Feb 25 12:48:13+0100 2018
[PATCH 1/4] gnu: kmod: Split off kmod-minimal.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180225114816.869-1-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel. This includes commands such as
to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(license license:gpl2+)))

-(define-public kmod
+(define kmod-minimal
(package
+ (name "kmod-minimal")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:tests? #f)) ; FIXME: Investigate test failures
+ (home-page "https://www.kernel.org/")
+ (synopsis "Kernel module tools")
+ (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod. The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+ (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+ (package (inherit kmod-minimal)
(name "kmod")
(version "24")
(source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(base32
"15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
(patches (search-patches "kmod-module-directory.patch"))))
- (build-system gnu-build-system)
(native-inputs
`(("pkg-config" ,pkg-config)))
(inputs
`(("xz" ,xz)
("zlib" ,zlib)))
(arguments
- `(#:tests? #f ; FIXME: Investigate test failures
- #:configure-flags '("--with-xz" "--with-zlib")
+ `(#:configure-flags '("--with-xz" "--with-zlib")
+ #:tests? #f ; FIXME: Investigate test failures
#:phases (alist-cons-after
'install 'install-modprobe&co
(lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(symlink "kmod"
(string-append bin "/" tool)))
'("insmod" "rmmod" "lsmod" "modprobe"
- "modinfo" "depmod"))))
- %standard-phases)))
- (home-page "https://www.kernel.org/")
- (synopsis "Kernel module tools")
- (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod. The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
- (license license:gpl2+))) ; library under lgpl2.1+
+ "modinfo" "depmod"))
+ #t))
+ %standard-phases)))))

(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
D
Danny Milosavljevic wrote on Sun Feb 25 12:48:14+0100 2018
[PATCH 2/4] gnu: Add kmod-minimal-static.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180225114816.869-2-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
---
gnu/packages/linux.scm | 44 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 44 insertions(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
#t))
%standard-phases)))))

+(define-public kmod-minimal/static
+ (static-package
+ (package (inherit kmod-minimal)
+ (name "kmod-minimal-static")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (arguments
+ (substitute-keyword-arguments
+ (package-arguments (static-package kmod-minimal))
+ ((#:configure-flags flags ''())
+ `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+ ((#:make-flags flags ''())
+ `(cons* "LDFLAGS=-all-static" ,flags))
+ ((#:phases phases '%standard-phases)
+ `(modify-phases ,phases
+ (delete 'install-license-files)
+ (add-after 'unpack 'patch-kmod
+ (lambda _
+ ;; Reduce size by 200 kiB.
+ (substitute* "tools/kmod.c"
+ (("[&]kmod_cmd_compat_lsmod,") "")
+ (("[&]kmod_cmd_compat_rmmod,") "")
+ (("[&]kmod_cmd_compat_insmod,") "")
+ (("[&]kmod_cmd_compat_modinfo,") ""))
+ #t))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (bin (string-append out "/bin")))
+ (install-file "tools/kmod" bin)
+ (for-each
+ (lambda (tool)
+ (symlink "kmod" (string-append bin "/" tool)))
+ '("modprobe" "depmod"))
+ #t))))))))))
+
(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
(package
D
Danny Milosavljevic wrote on Sun Feb 25 12:48:15+0100 2018
[PATCH 3/4] linux-initrd: Add kmod.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180225114816.869-3-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
gnu/build/linux-initrd.scm | 13 +++++++++++-
gnu/system/linux-initrd.scm | 48 ++++++++++++++++++++++++++++++++++-----------
2 files changed, 49 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..52cc180b4 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init kmod linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (if kmod
+ (begin
+ (mkdir-p "sbin")
+ (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe")))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..93089a869 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ kmod
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
+ #:kmod #$kmod
+ #:linux-module-directory #$linux-module-directory
;; Copy everything INIT refers to into the initrd.
#:references-graphs '("closure")
#:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
(gexp->derivation name builder
#:references-graphs `(("closure" ,init))))

-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
"Return a flat directory containing the Linux kernel modules listed in
MODULES and taken from LINUX."
(define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex)
+ (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
(srfi srfi-1)
(guix build utils)
(gnu build linux-modules))
@@ -138,13 +142,30 @@ MODULES and taken from LINUX."
(recursive-module-dependencies modules
#:lookup-module lookup))))

- (mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
+ (define version
+ (car
+ (filter
+ (lambda (name)
+ (not (string-prefix? "." name)))
+ (scandir module-dir))))
+
+ (display "VERSION")
+ (display version)
+ (newline)
+
+ (let ((output (string-append #$output "/lib/modules/" version)))
+ (mkdir-p output)
+ (for-each (lambda (module)
+ (format #t "copying '~a'...~%" module)
+ (copy-file module
+ (string-append output "/"
+ (basename module))))
+ (delete-duplicates modules)))
+ (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+ ; -E
+ "-F" (string-append #$linux "/System.map")
+ version)
+ #t)))

(computed-file "linux-modules" build-exp))

@@ -152,6 +173,7 @@ MODULES and taken from LINUX."
#:key
(linux linux-libre)
(linux-modules '())
+ (kmod kmod-minimal/static)
(mapped-devices '())
(helper-packages '())
qemu-networking?
@@ -185,7 +207,7 @@ upon error."
mapped-devices))

(define kodir
- (flat-linux-module-directory linux linux-modules))
+ (flat-linux-module-directory linux linux-modules kmod))

(expression->initrd
(with-imported-modules (source-module-closure
@@ -223,6 +245,8 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:kmod kmod
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +269,7 @@ FILE-SYSTEMS."
(define* (base-initrd file-systems
#:key
(linux linux-libre)
+ (kmod kmod-minimal/static)
(mapped-devices '())
qemu-networking?
volatile-root?
@@ -322,8 +347,9 @@ loaded at boot time in the order in which they appear."
(raw-initrd file-systems
#:linux linux
#:linux-modules linux-modules
+ #:kmod kmod
#:mapped-devices mapped-devices
- #:helper-packages helper-packages
+ #:helper-packages (cons kmod helper-packages)
#:qemu-networking? qemu-networking?
#:volatile-root? volatile-root?
#:on-error on-error))
D
Danny Milosavljevic wrote on Sun Feb 25 12:48:16+0100 2018
[PATCH 4/4] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180225114816.869-4-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 24 ++++++++++++++++++++----
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 23 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..65c91c50f 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,25 @@ upon error."
(string-append linux-module-directory "/"
(ensure-dot-ko name)))

+ (define (load-kernel-modules)
+ (define enter?
+ (const #t))
+ (file-system-fold
+ enter?
+ ;; This is the Leaf handler. It tries to modprobe all the modaliases.
+ (lambda (path stat result) ; leaf
+ (let ((modalias-name (string-append path "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias (call-with-input-file modalias-name read-string)))
+ (if (not (string=? modalias ""))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))))
+ (const #t) ; down
+ (const #t) ; up
+ (const #f) ; skip
+ (const #f) ; error
+ #f ; init
+ "/sys/devices"))
+
(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")

@@ -486,10 +505,7 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 93089a869..573f4a324 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -228,7 +228,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
M
Mathieu Othacehe wrote on Sun Feb 25 15:05:24+0100 2018
Re: [bug#30604] [PATCH 3/4] linux-initrd: Add kmod.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87vaelf9ln.fsf@gmail.com
Hi Danny,

Here are some small remarks,

> + ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
> + (if kmod
> + (begin

You can use "when" to avoid "begin".

> + (define version
> + (car

Prefer match to car/cdr use.

> + (filter
> + (lambda (name)
> + (not (string-prefix? "." name)))
> + (scandir module-dir))))
> +
> + (display "VERSION")
> + (display version)
> + (newline)

(format #t "VERSION" ~a~%" version) would be shorter.

> + ; -E

Why is this commented ?

> + "-F" (string-append #$linux "/System.map")
> + version)
> + #t)))
>

Thanks,

Mathieu
D
Danny Milosavljevic wrote on Sun Feb 25 16:07:50+0100 2018
(name . Mathieu Othacehe)(address . m.othacehe@gmail.com)(address . 30604@debbugs.gnu.org)
20180225160750.46bf5b50@scratchpost.org
Hi Mathieu,

thanks for the review.

On Sun, 25 Feb 2018 15:05:24 +0100
Mathieu Othacehe <m.othacehe@gmail.com> wrote:

> > + ; -E
>
> Why is this commented ?

Because it's an option for specifying the location of "Module.symvers" - and
I don't know whether guix uses it (probably not). If one doesn't specify an
option, depmod will default to the running kernel - which is not what we want.

I should elaborate in the comment that, if we start using Module.symvers, we
MUST pass "-E" there. Maybe better to even just check for the file existence
and add it right now, otherwise we might forget later. What do you think?
M
Marius Bakke wrote on Mon Feb 26 02:10:14+0100 2018
Re: [bug#30604] [PATCH 1/4] gnu: kmod: Split off kmod-minimal.
87h8q4wo7d.fsf@fastmail.com
Danny Milosavljevic <dannym@scratchpost.org> writes:

> * gnu/packages/linux.scm (kmod-minimal): New variable.
> (kmod): Modify.
> ---
> gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
> 1 file changed, 33 insertions(+), 16 deletions(-)
>
> diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
> index 238398e84..1f8bf3050 100644
> --- a/gnu/packages/linux.scm
> +++ b/gnu/packages/linux.scm
> @@ -1933,8 +1933,35 @@ for systems using the Linux kernel. This includes commands such as
> to use Linux' inotify mechanism, which allows file accesses to be monitored.")
> (license license:gpl2+)))
>
> -(define-public kmod
> +(define kmod-minimal
> (package
> + (name "kmod-minimal")
> + (version "13")

Why this old version?

> + (source (origin
> + (method url-fetch)
> + (uri
> + (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
> + "kmod-" version ".tar.xz"))
> + (sha256
> + (base32
> + "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
> + (patches (search-patches "kmod-13-module-directory.patch"))))

This patch seems to be missing.

> + (build-system gnu-build-system)
> + (arguments
> + `(#:tests? #f)) ; FIXME: Investigate test failures
> + (home-page "https://www.kernel.org/")
> + (synopsis "Kernel module tools")
> + (description "Kmod is a set of tools to handle common tasks with Linux
> +kernel modules like insert, remove, list, check properties, resolve
> +dependencies and aliases.
> +
> +These tools are designed on top of libkmod, a library that is shipped with
> +kmod. The aim is to be compatible with tools, configurations and indices
> +from the module-init-tools project.")
> + (license license:gpl2+))) ; library under lgpl2.1+
> +
> +(define-public kmod
> + (package (inherit kmod-minimal)
> (name "kmod")
> (version "24")
> (source (origin
> @@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
> (base32
> "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
> (patches (search-patches "kmod-module-directory.patch"))))
> - (build-system gnu-build-system)
> (native-inputs
> `(("pkg-config" ,pkg-config)))
> (inputs
> `(("xz" ,xz)
> ("zlib" ,zlib)))
> (arguments
> - `(#:tests? #f ; FIXME: Investigate test failures
> - #:configure-flags '("--with-xz" "--with-zlib")
> + `(#:configure-flags '("--with-xz" "--with-zlib")
> + #:tests? #f ; FIXME: Investigate test failures
> #:phases (alist-cons-after
> 'install 'install-modprobe&co
> (lambda* (#:key outputs #:allow-other-keys)
> @@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
> (symlink "kmod"
> (string-append bin "/" tool)))
> '("insmod" "rmmod" "lsmod" "modprobe"
> - "modinfo" "depmod"))))
> - %standard-phases)))
> - (home-page "https://www.kernel.org/")
> - (synopsis "Kernel module tools")
> - (description "Kmod is a set of tools to handle common tasks with Linux
> -kernel modules like insert, remove, list, check properties, resolve
> -dependencies and aliases.
> -
> -These tools are designed on top of libkmod, a library that is shipped with
> -kmod. The aim is to be compatible with tools, configurations and indices
> -from the module-init-tools project.")
> - (license license:gpl2+))) ; library under lgpl2.1+
> + "modinfo" "depmod"))
> + #t))
> + %standard-phases)))))
>
> (define-public eudev
> ;; The post-systemd fork, maintained by Gentoo.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlqTXnYACgkQoqBt8qM6
VPpGtAf9HdKJjuRHFJxIT76qB3Yjub3mj5c3LRNhSA1vPnaPEMOXVDDTlq4t8XcR
K0oaixbKYcSvSmHJ+Sp17jy+SYpeA1WGGpCxKVsBl8FCKhEYh2ztbgRUtMwjXZyF
ZjO15dLd2SU40hWXVabKDeDTZyD9whMZMrbKAmu1OelG9lSPJw7fWfSU8UnZeXJg
TflXbAN+9Ikm5ey4kxY/v+IjE60aWd5CRlUqcWVzKzZ7bSmU81r+paUWSe+11V7n
7LoBFm/MwQM3JFkglmSkWsDv/9eQwfm6CI6yV15qChL+MXuSFKzuJNKeETd7TvV9
P8zks8IDY7CA9I3PwJXPEZRuwA7QYw==
=YgmH
-----END PGP SIGNATURE-----
D
Danny Milosavljevic wrote on Mon Feb 26 04:50:20+0100 2018
[PATCH v2 1/6] gnu: kmod: Split off kmod-minimal.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226035025.1698-2-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel. This includes commands such as
to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(license license:gpl2+)))

-(define-public kmod
+(define kmod-minimal
(package
+ (name "kmod-minimal")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:tests? #f)) ; FIXME: Investigate test failures
+ (home-page "https://www.kernel.org/")
+ (synopsis "Kernel module tools")
+ (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod. The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+ (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+ (package (inherit kmod-minimal)
(name "kmod")
(version "24")
(source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(base32
"15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
(patches (search-patches "kmod-module-directory.patch"))))
- (build-system gnu-build-system)
(native-inputs
`(("pkg-config" ,pkg-config)))
(inputs
`(("xz" ,xz)
("zlib" ,zlib)))
(arguments
- `(#:tests? #f ; FIXME: Investigate test failures
- #:configure-flags '("--with-xz" "--with-zlib")
+ `(#:configure-flags '("--with-xz" "--with-zlib")
+ #:tests? #f ; FIXME: Investigate test failures
#:phases (alist-cons-after
'install 'install-modprobe&co
(lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(symlink "kmod"
(string-append bin "/" tool)))
'("insmod" "rmmod" "lsmod" "modprobe"
- "modinfo" "depmod"))))
- %standard-phases)))
- (home-page "https://www.kernel.org/")
- (synopsis "Kernel module tools")
- (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod. The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
- (license license:gpl2+))) ; library under lgpl2.1+
+ "modinfo" "depmod"))
+ #t))
+ %standard-phases)))))

(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
D
Danny Milosavljevic wrote on Mon Feb 26 04:50:21+0100 2018
[PATCH v2 2/6] gnu: Add kmod-minimal-static.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226035025.1698-3-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
* gnu/packages/patches/kmod-13-module-directory.patch: New file.
* gnu/local.mk: Add it.
---
gnu/local.mk | 1 +
gnu/packages/linux.scm | 44 ++++++++++++++++++++++
.../patches/kmod-13-module-directory.patch | 33 ++++++++++++++++
3 files changed, 78 insertions(+)
create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 21195f8c1..b1e3c878d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -795,6 +795,7 @@ dist_patch_DATA = \
%D%/packages/patches/kiki-makefile.patch \
%D%/packages/patches/kiki-missing-includes.patch \
%D%/packages/patches/kiki-portability-64bit.patch \
+ %D%/packages/patches/kmod-13-module-directory.patch \
%D%/packages/patches/kmod-module-directory.patch \
%D%/packages/patches/kobodeluxe-paths.patch \
%D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch \
diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
#t))
%standard-phases)))))

+(define-public kmod-minimal/static
+ (static-package
+ (package (inherit kmod-minimal)
+ (name "kmod-minimal-static")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (arguments
+ (substitute-keyword-arguments
+ (package-arguments (static-package kmod-minimal))
+ ((#:configure-flags flags ''())
+ `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+ ((#:make-flags flags ''())
+ `(cons* "LDFLAGS=-all-static" ,flags))
+ ((#:phases phases '%standard-phases)
+ `(modify-phases ,phases
+ (delete 'install-license-files)
+ (add-after 'unpack 'patch-kmod
+ (lambda _
+ ;; Reduce size by 200 kiB.
+ (substitute* "tools/kmod.c"
+ (("[&]kmod_cmd_compat_lsmod,") "")
+ (("[&]kmod_cmd_compat_rmmod,") "")
+ (("[&]kmod_cmd_compat_insmod,") "")
+ (("[&]kmod_cmd_compat_modinfo,") ""))
+ #t))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (bin (string-append out "/bin")))
+ (install-file "tools/kmod" bin)
+ (for-each
+ (lambda (tool)
+ (symlink "kmod" (string-append bin "/" tool)))
+ '("modprobe" "depmod"))
+ #t))))))))))
+
(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
(package
diff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patch
new file mode 100644
index 000000000..5ff2f8a60
--- /dev/null
+++ b/gnu/packages/patches/kmod-13-module-directory.patch
@@ -0,0 +1,33 @@
+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'
+environment variable, rather than looking for modules exclusively in
+/lib/modules.
+
+Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to
+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable
+name.
+
+
+--- kmod-7/libkmod/libkmod.c 2012-03-15 08:19:16.750010226 -0400
++++ kmod-7/libkmod/libkmod.c 2012-04-04 15:21:29.532074313 -0400
+@@ -200,7 +200,7 @@
+ static char *get_kernel_release(const char *dirname)
+ {
+ struct utsname u;
+- char *p;
++ char *p, *dirname_prefix;
+
+ if (dirname != NULL)
+ return path_make_absolute_cwd(dirname);
+@@ -208,7 +208,10 @@
+ if (uname(&u) < 0)
+ return NULL;
+
+- if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)
++ if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)
++ dirname_prefix = dirname_default_prefix;
++
++ if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)
+ return NULL;
+
+ return p;
+
D
Danny Milosavljevic wrote on Mon Feb 26 04:50:19+0100 2018
[PATCH v2 0/6] Load Linux module only when supported hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226035025.1698-1-dannym@scratchpost.org
Danny Milosavljevic (6):
gnu: kmod: Split off kmod-minimal.
gnu: Add kmod-minimal-static.
linux-initrd: Add kmod.
linux-boot: Load kernel modules only when the hardware is present.
vm: Allow qemu-image builder to load Linux kernel modules.
vm: Make the virtio-blk is uniquely identifyable in /sys.

gnu/build/linux-boot.scm | 31 +++++++-
gnu/build/linux-initrd.scm | 12 ++-
gnu/build/vm.scm | 2 +-
gnu/local.mk | 1 +
gnu/packages/linux.scm | 91 ++++++++++++++++++----
.../patches/kmod-13-module-directory.patch | 33 ++++++++
gnu/system/linux-initrd.scm | 48 +++++++++---
gnu/system/vm.scm | 28 ++++++-
8 files changed, 210 insertions(+), 36 deletions(-)
create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch
D
Danny Milosavljevic wrote on Mon Feb 26 04:50:22+0100 2018
[PATCH v2 3/6] linux-initrd: Add kmod.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226035025.1698-4-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
gnu/build/linux-initrd.scm | 12 +++++++++++-
gnu/system/linux-initrd.scm | 44 +++++++++++++++++++++++++++++++++-----------
2 files changed, 44 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..6356007df 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init kmod linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (when kmod
+ (mkdir-p "sbin")
+ (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..91e498787 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ kmod
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
+ #:kmod #$kmod
+ #:linux-module-directory #$linux-module-directory
;; Copy everything INIT refers to into the initrd.
#:references-graphs '("closure")
#:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
(gexp->derivation name builder
#:references-graphs `(("closure" ,init))))

-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
"Return a flat directory containing the Linux kernel modules listed in
MODULES and taken from LINUX."
(define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex)
+ (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
(srfi srfi-1)
(guix build utils)
(gnu build linux-modules))
@@ -138,13 +142,26 @@ MODULES and taken from LINUX."
(recursive-module-dependencies modules
#:lookup-module lookup))))

- (mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
+ (define version
+ (car
+ (filter
+ (lambda (name)
+ (not (string-prefix? "." name)))
+ (scandir module-dir))))
+
+ (let ((output (string-append #$output "/lib/modules/" version)))
+ (mkdir-p output)
+ (for-each (lambda (module)
+ (format #t "copying '~a'...~%" module)
+ (copy-file module
+ (string-append output "/"
+ (basename module))))
+ (delete-duplicates modules)))
+ (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+ ; -E
+ "-F" (string-append #$linux "/System.map")
+ version)
+ #t)))

(computed-file "linux-modules" build-exp))

@@ -152,6 +169,7 @@ MODULES and taken from LINUX."
#:key
(linux linux-libre)
(linux-modules '())
+ (kmod kmod-minimal/static)
(mapped-devices '())
(helper-packages '())
qemu-networking?
@@ -185,7 +203,7 @@ upon error."
mapped-devices))

(define kodir
- (flat-linux-module-directory linux linux-modules))
+ (flat-linux-module-directory linux linux-modules kmod))

(expression->initrd
(with-imported-modules (source-module-closure
@@ -223,6 +241,8 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:kmod kmod
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +265,7 @@ FILE-SYSTEMS."
(define* (base-initrd file-systems
#:key
(linux linux-libre)
+ (kmod kmod-minimal/static)
(mapped-devices '())
qemu-networking?
volatile-root?
@@ -322,8 +343,9 @@ loaded at boot time in the order in which they appear."
(raw-initrd file-systems
#:linux linux
#:linux-modules linux-modules
+ #:kmod kmod
#:mapped-devices mapped-devices
- #:helper-packages helper-packages
+ #:helper-packages (cons kmod helper-packages)
#:qemu-networking? qemu-networking?
#:volatile-root? volatile-root?
#:on-error on-error))
D
Danny Milosavljevic wrote on Mon Feb 26 04:50:23+0100 2018
[PATCH v2 4/6] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226035025.1698-5-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 31 +++++++++++++++++++++++++++----
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..6d00ea9be 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,32 @@ upon error."
(string-append linux-module-directory "/"
(ensure-dot-ko name)))

+ (define (load-kernel-modules)
+ "Examine /sys/devices to find out which modules to load and load them."
+ (define enter?
+ (const #t))
+ (define (down! path stat result)
+ ;; Note: modprobe mutates the tree starting with path.
+ (let ((modalias-name (string-append path "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias
+ (string-trim-right (call-with-input-file modalias-name
+ read-string)
+ #\newline)))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))
+ #t)
+ (define up
+ (const #t))
+ (define skip
+ (const #t))
+ (define leaf
+ (const #t))
+ (define (error name stat errno result)
+ (format (current-error-port) "warning: ~a: ~a~%"
+ name (strerror errno))
+ result)
+ (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+
(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")

@@ -486,10 +512,7 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 91e498787..6d130ccac 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -224,7 +224,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
D
Danny Milosavljevic wrote on Mon Feb 26 04:50:24+0100 2018
[PATCH v2 5/6] vm: Allow qemu-image builder to load Linux kernel modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226035025.1698-6-dannym@scratchpost.org
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
gnu/system/vm.scm | 25 +++++++++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..c64c9678f 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
#:single-file-output? #t
#:references-graphs inputs))

+(define (%modprobe-wrapper modprobe linux-module-directory)
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ ;;
+ ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+ ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+ ;; environment variable is not set---hence the need for this wrapper.
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+ (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
@@ -275,6 +286,8 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+ (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
(expression->derivation-in-linux-vm
name
(with-imported-modules (source-module-closure '((gnu build bootloader)
@@ -288,7 +301,7 @@ the image."
(ice-9 binary-ports))

(let ((inputs
- '#$(append (list qemu parted e2fsprogs dosfstools)
+ '#$(append (list qemu parted e2fsprogs dosfstools kmod)
(map canonical-package
(list sed grep coreutils findutils gawk))
(if register-closures? (list guix) '())))
@@ -302,6 +315,14 @@ the image."
inputs)))

(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+ ;; It's possible that we need to load nls modules in order to
+ ;; mount the new partition.
+ (if (file-exists? #$modprobe-name)
+ (activate-modprobe #$(%modprobe-wrapper modprobe-name
+ linux-module-directory))
+ (format (current-error-port)
+ "WARNING: No modprobe found in ~s. \
+Loading kernel modules will be impossible.\n" #$modprobe-name))

(let* ((graphs '#$(match inputs
(((names . _) ...)
@@ -364,7 +385,7 @@ the image."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs))
+ #:references-graphs inputs)))


;;;
D
Danny Milosavljevic wrote on Mon Feb 26 04:50:25+0100 2018
[PATCH v2 6/6] vm: Make the virtio-blk is uniquely identifyable in /sys.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226035025.1698-7-dannym@scratchpost.org
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index c64c9678f..ad48999ee 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -704,7 +704,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
D
Danny Milosavljevic wrote on Mon Feb 26 05:06:04+0100 2018
[PATCH v3 1/6] gnu: kmod: Split off kmod-minimal.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226040609.3066-2-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel. This includes commands such as
to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(license license:gpl2+)))

-(define-public kmod
+(define kmod-minimal
(package
+ (name "kmod-minimal")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:tests? #f)) ; FIXME: Investigate test failures
+ (home-page "https://www.kernel.org/")
+ (synopsis "Kernel module tools")
+ (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod. The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+ (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+ (package (inherit kmod-minimal)
(name "kmod")
(version "24")
(source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(base32
"15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
(patches (search-patches "kmod-module-directory.patch"))))
- (build-system gnu-build-system)
(native-inputs
`(("pkg-config" ,pkg-config)))
(inputs
`(("xz" ,xz)
("zlib" ,zlib)))
(arguments
- `(#:tests? #f ; FIXME: Investigate test failures
- #:configure-flags '("--with-xz" "--with-zlib")
+ `(#:configure-flags '("--with-xz" "--with-zlib")
+ #:tests? #f ; FIXME: Investigate test failures
#:phases (alist-cons-after
'install 'install-modprobe&co
(lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(symlink "kmod"
(string-append bin "/" tool)))
'("insmod" "rmmod" "lsmod" "modprobe"
- "modinfo" "depmod"))))
- %standard-phases)))
- (home-page "https://www.kernel.org/")
- (synopsis "Kernel module tools")
- (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod. The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
- (license license:gpl2+))) ; library under lgpl2.1+
+ "modinfo" "depmod"))
+ #t))
+ %standard-phases)))))

(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
D
Danny Milosavljevic wrote on Mon Feb 26 05:06:03+0100 2018
[PATCH v3 0/6] Load Linux module only when supported hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226040609.3066-1-dannym@scratchpost.org
Danny Milosavljevic (6):
gnu: kmod: Split off kmod-minimal.
gnu: Add kmod-minimal-static.
linux-initrd: Add kmod.
linux-boot: Load kernel modules only when the hardware is present.
vm: Allow qemu-image builder to load Linux kernel modules.
vm: Make the virtio-blk is uniquely identifyable in /sys.

gnu/build/linux-boot.scm | 31 +++++++-
gnu/build/linux-initrd.scm | 12 ++-
gnu/build/vm.scm | 2 +-
gnu/local.mk | 1 +
gnu/packages/linux.scm | 91 ++++++++++++++++++----
.../patches/kmod-13-module-directory.patch | 33 ++++++++
gnu/system/linux-initrd.scm | 49 +++++++++---
gnu/system/vm.scm | 34 ++++++--
8 files changed, 215 insertions(+), 38 deletions(-)
create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch
D
Danny Milosavljevic wrote on Mon Feb 26 05:06:05+0100 2018
[PATCH v3 2/6] gnu: Add kmod-minimal-static.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226040609.3066-3-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
* gnu/packages/patches/kmod-13-module-directory.patch: New file.
* gnu/local.mk: Add it.
---
gnu/local.mk | 1 +
gnu/packages/linux.scm | 44 ++++++++++++++++++++++
.../patches/kmod-13-module-directory.patch | 33 ++++++++++++++++
3 files changed, 78 insertions(+)
create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 21195f8c1..b1e3c878d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -795,6 +795,7 @@ dist_patch_DATA = \
%D%/packages/patches/kiki-makefile.patch \
%D%/packages/patches/kiki-missing-includes.patch \
%D%/packages/patches/kiki-portability-64bit.patch \
+ %D%/packages/patches/kmod-13-module-directory.patch \
%D%/packages/patches/kmod-module-directory.patch \
%D%/packages/patches/kobodeluxe-paths.patch \
%D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch \
diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
#t))
%standard-phases)))))

+(define-public kmod-minimal/static
+ (static-package
+ (package (inherit kmod-minimal)
+ (name "kmod-minimal-static")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (arguments
+ (substitute-keyword-arguments
+ (package-arguments (static-package kmod-minimal))
+ ((#:configure-flags flags ''())
+ `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+ ((#:make-flags flags ''())
+ `(cons* "LDFLAGS=-all-static" ,flags))
+ ((#:phases phases '%standard-phases)
+ `(modify-phases ,phases
+ (delete 'install-license-files)
+ (add-after 'unpack 'patch-kmod
+ (lambda _
+ ;; Reduce size by 200 kiB.
+ (substitute* "tools/kmod.c"
+ (("[&]kmod_cmd_compat_lsmod,") "")
+ (("[&]kmod_cmd_compat_rmmod,") "")
+ (("[&]kmod_cmd_compat_insmod,") "")
+ (("[&]kmod_cmd_compat_modinfo,") ""))
+ #t))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (bin (string-append out "/bin")))
+ (install-file "tools/kmod" bin)
+ (for-each
+ (lambda (tool)
+ (symlink "kmod" (string-append bin "/" tool)))
+ '("modprobe" "depmod"))
+ #t))))))))))
+
(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
(package
diff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patch
new file mode 100644
index 000000000..5ff2f8a60
--- /dev/null
+++ b/gnu/packages/patches/kmod-13-module-directory.patch
@@ -0,0 +1,33 @@
+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'
+environment variable, rather than looking for modules exclusively in
+/lib/modules.
+
+Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to
+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable
+name.
+
+
+--- kmod-7/libkmod/libkmod.c 2012-03-15 08:19:16.750010226 -0400
++++ kmod-7/libkmod/libkmod.c 2012-04-04 15:21:29.532074313 -0400
+@@ -200,7 +200,7 @@
+ static char *get_kernel_release(const char *dirname)
+ {
+ struct utsname u;
+- char *p;
++ char *p, *dirname_prefix;
+
+ if (dirname != NULL)
+ return path_make_absolute_cwd(dirname);
+@@ -208,7 +208,10 @@
+ if (uname(&u) < 0)
+ return NULL;
+
+- if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)
++ if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)
++ dirname_prefix = dirname_default_prefix;
++
++ if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)
+ return NULL;
+
+ return p;
+
D
Danny Milosavljevic wrote on Mon Feb 26 05:06:06+0100 2018
[PATCH v3 3/6] linux-initrd: Add kmod.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226040609.3066-4-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
gnu/build/linux-initrd.scm | 12 +++++++++++-
gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++-----------
2 files changed, 45 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..6356007df 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init kmod linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (when kmod
+ (mkdir-p "sbin")
+ (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..46ef055f0 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ kmod
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
+ #:kmod #$kmod
+ #:linux-module-directory #$linux-module-directory
;; Copy everything INIT refers to into the initrd.
#:references-graphs '("closure")
#:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
(gexp->derivation name builder
#:references-graphs `(("closure" ,init))))

-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
"Return a flat directory containing the Linux kernel modules listed in
MODULES and taken from LINUX."
(define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex)
+ (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
(srfi srfi-1)
(guix build utils)
(gnu build linux-modules))
@@ -138,13 +142,27 @@ MODULES and taken from LINUX."
(recursive-module-dependencies modules
#:lookup-module lookup))))

- (mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
+ (define version
+ (match
+ (filter
+ (lambda (name)
+ (not (string-prefix? "." name)))
+ (scandir module-dir))
+ ((item) item)))
+
+ (let ((output (string-append #$output "/lib/modules/" version)))
+ (mkdir-p output)
+ (for-each (lambda (module)
+ (format #t "copying '~a'...~%" module)
+ (copy-file module
+ (string-append output "/"
+ (basename module))))
+ (delete-duplicates modules)))
+ (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+ ; -E
+ "-F" (string-append #$linux "/System.map")
+ version)
+ #t)))

(computed-file "linux-modules" build-exp))

@@ -152,6 +170,7 @@ MODULES and taken from LINUX."
#:key
(linux linux-libre)
(linux-modules '())
+ (kmod kmod-minimal/static)
(mapped-devices '())
(helper-packages '())
qemu-networking?
@@ -185,7 +204,7 @@ upon error."
mapped-devices))

(define kodir
- (flat-linux-module-directory linux linux-modules))
+ (flat-linux-module-directory linux linux-modules kmod))

(expression->initrd
(with-imported-modules (source-module-closure
@@ -223,6 +242,8 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:kmod kmod
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +266,7 @@ FILE-SYSTEMS."
(define* (base-initrd file-systems
#:key
(linux linux-libre)
+ (kmod kmod-minimal/static)
(mapped-devices '())
qemu-networking?
volatile-root?
@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."
(raw-initrd file-systems
#:linux linux
#:linux-modules linux-modules
+ #:kmod kmod
#:mapped-devices mapped-devices
- #:helper-packages helper-packages
+ #:helper-packages (cons kmod helper-packages)
#:qemu-networking? qemu-networking?
#:volatile-root? volatile-root?
#:on-error on-error))
D
Danny Milosavljevic wrote on Mon Feb 26 05:06:07+0100 2018
[PATCH v3 4/6] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226040609.3066-5-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 31 +++++++++++++++++++++++++++----
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..6d00ea9be 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,32 @@ upon error."
(string-append linux-module-directory "/"
(ensure-dot-ko name)))

+ (define (load-kernel-modules)
+ "Examine /sys/devices to find out which modules to load and load them."
+ (define enter?
+ (const #t))
+ (define (down! path stat result)
+ ;; Note: modprobe mutates the tree starting with path.
+ (let ((modalias-name (string-append path "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias
+ (string-trim-right (call-with-input-file modalias-name
+ read-string)
+ #\newline)))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))
+ #t)
+ (define up
+ (const #t))
+ (define skip
+ (const #t))
+ (define leaf
+ (const #t))
+ (define (error name stat errno result)
+ (format (current-error-port) "warning: ~a: ~a~%"
+ name (strerror errno))
+ result)
+ (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+
(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")

@@ -486,10 +512,7 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 46ef055f0..c8a9e4950 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -225,7 +225,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
D
Danny Milosavljevic wrote on Mon Feb 26 05:06:08+0100 2018
[PATCH v3 5/6] vm: Allow qemu-image builder to load Linux kernel modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226040609.3066-6-dannym@scratchpost.org
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
#:single-file-output? #t
#:references-graphs inputs))

+(define (%modprobe-wrapper modprobe linux-module-directory)
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ ;;
+ ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+ ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+ ;; environment variable is not set---hence the need for this wrapper.
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+ (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+ (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
(expression->derivation-in-linux-vm
name
- (with-imported-modules (source-module-closure '((gnu build bootloader)
+ (with-imported-modules (source-module-closure '((gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)))
#~(begin
- (use-modules (gnu build bootloader)
+ (use-modules (gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

(let ((inputs
- '#$(append (list qemu parted e2fsprogs dosfstools)
+ '#$(append (list qemu parted e2fsprogs dosfstools kmod)
(map canonical-package
(list sed grep coreutils findutils gawk))
(if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
inputs)))

(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+ ;; It's possible that we need to load nls modules in order to
+ ;; mount the new partition.
+ (if (file-exists? #$modprobe-name)
+ (activate-modprobe #$(%modprobe-wrapper modprobe-name
+ linux-module-directory))
+ (format (current-error-port)
+ "WARNING: No modprobe found in ~s. \
+Loading kernel modules will be impossible.\n" #$modprobe-name))

(let* ((graphs '#$(match inputs
(((names . _) ...)
@@ -364,7 +387,7 @@ the image."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs))
+ #:references-graphs inputs)))


;;;
D
Danny Milosavljevic wrote on Mon Feb 26 05:06:09+0100 2018
[PATCH v3 6/6] vm: Make the virtio-blk is uniquely identifyable in /sys.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180226040609.3066-7-dannym@scratchpost.org
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
M
Mathieu Othacehe wrote on Mon Feb 26 12:51:51+0100 2018
Re: [bug#30604] [PATCH 3/4] linux-initrd: Add kmod.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87h8q4j7e0.fsf@gmail.com
Hey Danny,

> Because it's an option for specifying the location of "Module.symvers" - and
> I don't know whether guix uses it (probably not). If one doesn't specify an
> option, depmod will default to the running kernel - which is not what we want.
>
> I should elaborate in the comment that, if we start using Module.symvers, we
> MUST pass "-E" there. Maybe better to even just check for the file existence
> and add it right now, otherwise we might forget later. What do you think?

Reading 'depmod' manpage, I understand that -E and -F are mutually
exclusive. Both options seem to have an interest only if -e is supplied
to "reports any symbols which a module needs which are not supplied by
other modules or the kernel".

So maybe something like "-F (string-append #$linux "/System.map") -e"
would make more sense ?

Mathieu
D
Danny Milosavljevic wrote on Tue Feb 27 12:26:13+0100 2018
[PATCH v4 1/7] gnu: kmod: Split off kmod-minimal.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227112619.5071-2-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel. This includes commands such as
to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(license license:gpl2+)))

-(define-public kmod
+(define kmod-minimal
(package
+ (name "kmod-minimal")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:tests? #f)) ; FIXME: Investigate test failures
+ (home-page "https://www.kernel.org/")
+ (synopsis "Kernel module tools")
+ (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod. The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+ (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+ (package (inherit kmod-minimal)
(name "kmod")
(version "24")
(source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(base32
"15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
(patches (search-patches "kmod-module-directory.patch"))))
- (build-system gnu-build-system)
(native-inputs
`(("pkg-config" ,pkg-config)))
(inputs
`(("xz" ,xz)
("zlib" ,zlib)))
(arguments
- `(#:tests? #f ; FIXME: Investigate test failures
- #:configure-flags '("--with-xz" "--with-zlib")
+ `(#:configure-flags '("--with-xz" "--with-zlib")
+ #:tests? #f ; FIXME: Investigate test failures
#:phases (alist-cons-after
'install 'install-modprobe&co
(lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(symlink "kmod"
(string-append bin "/" tool)))
'("insmod" "rmmod" "lsmod" "modprobe"
- "modinfo" "depmod"))))
- %standard-phases)))
- (home-page "https://www.kernel.org/")
- (synopsis "Kernel module tools")
- (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod. The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
- (license license:gpl2+))) ; library under lgpl2.1+
+ "modinfo" "depmod"))
+ #t))
+ %standard-phases)))))

(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
D
Danny Milosavljevic wrote on Tue Feb 27 12:26:14+0100 2018
[PATCH v4 2/7] gnu: Add kmod-minimal-static.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227112619.5071-3-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
* gnu/packages/patches/kmod-13-module-directory.patch: New file.
* gnu/local.mk: Add it.
---
gnu/local.mk | 1 +
gnu/packages/linux.scm | 44 ++++++++++++++++++++++
.../patches/kmod-13-module-directory.patch | 33 ++++++++++++++++
3 files changed, 78 insertions(+)
create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 21195f8c1..b1e3c878d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -795,6 +795,7 @@ dist_patch_DATA = \
%D%/packages/patches/kiki-makefile.patch \
%D%/packages/patches/kiki-missing-includes.patch \
%D%/packages/patches/kiki-portability-64bit.patch \
+ %D%/packages/patches/kmod-13-module-directory.patch \
%D%/packages/patches/kmod-module-directory.patch \
%D%/packages/patches/kobodeluxe-paths.patch \
%D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch \
diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
#t))
%standard-phases)))))

+(define-public kmod-minimal/static
+ (static-package
+ (package (inherit kmod-minimal)
+ (name "kmod-minimal-static")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (arguments
+ (substitute-keyword-arguments
+ (package-arguments (static-package kmod-minimal))
+ ((#:configure-flags flags ''())
+ `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+ ((#:make-flags flags ''())
+ `(cons* "LDFLAGS=-all-static" ,flags))
+ ((#:phases phases '%standard-phases)
+ `(modify-phases ,phases
+ (delete 'install-license-files)
+ (add-after 'unpack 'patch-kmod
+ (lambda _
+ ;; Reduce size by 200 kiB.
+ (substitute* "tools/kmod.c"
+ (("[&]kmod_cmd_compat_lsmod,") "")
+ (("[&]kmod_cmd_compat_rmmod,") "")
+ (("[&]kmod_cmd_compat_insmod,") "")
+ (("[&]kmod_cmd_compat_modinfo,") ""))
+ #t))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (bin (string-append out "/bin")))
+ (install-file "tools/kmod" bin)
+ (for-each
+ (lambda (tool)
+ (symlink "kmod" (string-append bin "/" tool)))
+ '("modprobe" "depmod"))
+ #t))))))))))
+
(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
(package
diff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patch
new file mode 100644
index 000000000..5ff2f8a60
--- /dev/null
+++ b/gnu/packages/patches/kmod-13-module-directory.patch
@@ -0,0 +1,33 @@
+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'
+environment variable, rather than looking for modules exclusively in
+/lib/modules.
+
+Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to
+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable
+name.
+
+
+--- kmod-7/libkmod/libkmod.c 2012-03-15 08:19:16.750010226 -0400
++++ kmod-7/libkmod/libkmod.c 2012-04-04 15:21:29.532074313 -0400
+@@ -200,7 +200,7 @@
+ static char *get_kernel_release(const char *dirname)
+ {
+ struct utsname u;
+- char *p;
++ char *p, *dirname_prefix;
+
+ if (dirname != NULL)
+ return path_make_absolute_cwd(dirname);
+@@ -208,7 +208,10 @@
+ if (uname(&u) < 0)
+ return NULL;
+
+- if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)
++ if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)
++ dirname_prefix = dirname_default_prefix;
++
++ if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)
+ return NULL;
+
+ return p;
+
D
Danny Milosavljevic wrote on Tue Feb 27 12:26:12+0100 2018
[PATCH v4 0/7] Load Linux module only when supported hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227112619.5071-1-dannym@scratchpost.org
Danny Milosavljevic (7):
gnu: kmod: Split off kmod-minimal.
gnu: Add kmod-minimal-static.
linux-initrd: Add kmod.
linux-boot: Load kernel modules only when the hardware is present.
vm: Allow qemu-image builder to load Linux kernel modules.
vm: Make the virtio-blk uniquely identifyable in /sys.
linux-boot: Call make-static-device-nodes much earlier.

gnu/build/linux-boot.scm | 43 ++++++++--
gnu/build/linux-initrd.scm | 12 ++-
gnu/build/vm.scm | 2 +-
gnu/local.mk | 1 +
gnu/packages/linux.scm | 91 ++++++++++++++++++----
.../patches/kmod-13-module-directory.patch | 33 ++++++++
gnu/services/base.scm | 11 ---
gnu/system/linux-initrd.scm | 50 +++++++++---
gnu/system/vm.scm | 34 ++++++--
9 files changed, 224 insertions(+), 53 deletions(-)
create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch
D
Danny Milosavljevic wrote on Tue Feb 27 12:26:15+0100 2018
[PATCH v4 3/7] linux-initrd: Add kmod.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227112619.5071-4-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
gnu/build/linux-initrd.scm | 12 +++++++++++-
gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++-----------
2 files changed, 45 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..6356007df 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init kmod linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (when kmod
+ (mkdir-p "sbin")
+ (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..46ef055f0 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ kmod
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
+ #:kmod #$kmod
+ #:linux-module-directory #$linux-module-directory
;; Copy everything INIT refers to into the initrd.
#:references-graphs '("closure")
#:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
(gexp->derivation name builder
#:references-graphs `(("closure" ,init))))

-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
"Return a flat directory containing the Linux kernel modules listed in
MODULES and taken from LINUX."
(define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex)
+ (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
(srfi srfi-1)
(guix build utils)
(gnu build linux-modules))
@@ -138,13 +142,27 @@ MODULES and taken from LINUX."
(recursive-module-dependencies modules
#:lookup-module lookup))))

- (mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
+ (define version
+ (match
+ (filter
+ (lambda (name)
+ (not (string-prefix? "." name)))
+ (scandir module-dir))
+ ((item) item)))
+
+ (let ((output (string-append #$output "/lib/modules/" version)))
+ (mkdir-p output)
+ (for-each (lambda (module)
+ (format #t "copying '~a'...~%" module)
+ (copy-file module
+ (string-append output "/"
+ (basename module))))
+ (delete-duplicates modules)))
+ (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+ "-e"
+ "-F" (string-append #$linux "/System.map")
+ version)
+ #t)))

(computed-file "linux-modules" build-exp))

@@ -152,6 +170,7 @@ MODULES and taken from LINUX."
#:key
(linux linux-libre)
(linux-modules '())
+ (kmod kmod-minimal/static)
(mapped-devices '())
(helper-packages '())
qemu-networking?
@@ -185,7 +204,7 @@ upon error."
mapped-devices))

(define kodir
- (flat-linux-module-directory linux linux-modules))
+ (flat-linux-module-directory linux linux-modules kmod))

(expression->initrd
(with-imported-modules (source-module-closure
@@ -223,6 +242,8 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:kmod kmod
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +266,7 @@ FILE-SYSTEMS."
(define* (base-initrd file-systems
#:key
(linux linux-libre)
+ (kmod kmod-minimal/static)
(mapped-devices '())
qemu-networking?
volatile-root?
@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."
(raw-initrd file-systems
#:linux linux
#:linux-modules linux-modules
+ #:kmod kmod
#:mapped-devices mapped-devices
- #:helper-packages helper-packages
+ #:helper-packages (cons kmod helper-packages)
#:qemu-networking? qemu-networking?
#:volatile-root? volatile-root?
#:on-error on-error))
D
Danny Milosavljevic wrote on Tue Feb 27 12:26:16+0100 2018
[PATCH v4 4/7] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227112619.5071-5-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 31 +++++++++++++++++++++++++++----
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..6d00ea9be 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,32 @@ upon error."
(string-append linux-module-directory "/"
(ensure-dot-ko name)))

+ (define (load-kernel-modules)
+ "Examine /sys/devices to find out which modules to load and load them."
+ (define enter?
+ (const #t))
+ (define (down! path stat result)
+ ;; Note: modprobe mutates the tree starting with path.
+ (let ((modalias-name (string-append path "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias
+ (string-trim-right (call-with-input-file modalias-name
+ read-string)
+ #\newline)))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))
+ #t)
+ (define up
+ (const #t))
+ (define skip
+ (const #t))
+ (define leaf
+ (const #t))
+ (define (error name stat errno result)
+ (format (current-error-port) "warning: ~a: ~a~%"
+ name (strerror errno))
+ result)
+ (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+
(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")

@@ -486,10 +512,7 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 46ef055f0..c8a9e4950 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -225,7 +225,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
D
Danny Milosavljevic wrote on Tue Feb 27 12:26:17+0100 2018
[PATCH v4 5/7] vm: Allow qemu-image builder to load Linux kernel modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227112619.5071-6-dannym@scratchpost.org
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
#:single-file-output? #t
#:references-graphs inputs))

+(define (%modprobe-wrapper modprobe linux-module-directory)
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ ;;
+ ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+ ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+ ;; environment variable is not set---hence the need for this wrapper.
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+ (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+ (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
(expression->derivation-in-linux-vm
name
- (with-imported-modules (source-module-closure '((gnu build bootloader)
+ (with-imported-modules (source-module-closure '((gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)))
#~(begin
- (use-modules (gnu build bootloader)
+ (use-modules (gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

(let ((inputs
- '#$(append (list qemu parted e2fsprogs dosfstools)
+ '#$(append (list qemu parted e2fsprogs dosfstools kmod)
(map canonical-package
(list sed grep coreutils findutils gawk))
(if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
inputs)))

(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+ ;; It's possible that we need to load nls modules in order to
+ ;; mount the new partition.
+ (if (file-exists? #$modprobe-name)
+ (activate-modprobe #$(%modprobe-wrapper modprobe-name
+ linux-module-directory))
+ (format (current-error-port)
+ "WARNING: No modprobe found in ~s. \
+Loading kernel modules will be impossible.\n" #$modprobe-name))

(let* ((graphs '#$(match inputs
(((names . _) ...)
@@ -364,7 +387,7 @@ the image."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs))
+ #:references-graphs inputs)))


;;;
D
Danny Milosavljevic wrote on Tue Feb 27 12:26:18+0100 2018
[PATCH v4 6/7] vm: Make the virtio-blk uniquely identifyable in /sys.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227112619.5071-7-dannym@scratchpost.org
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
D
Danny Milosavljevic wrote on Tue Feb 27 12:26:19+0100 2018
[PATCH v4 7/7] linux-boot: Call make-static-device-nodes much earlier.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227112619.5071-8-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (expression->initrd): Store data files for
make-static-device-nodes.
* gnu/build/linux-boot.scm (make-static-device-nodes): Unexport.
(boot-system): Call make-static-device-nodes. Delete lookup-module.
* gnu/services/base.scm (udev-shepherd-service): Delete
make-static-device-nodes call.
---
gnu/build/linux-boot.scm | 14 +++++++++-----
gnu/services/base.scm | 11 -----------
gnu/system/linux-initrd.scm | 3 ++-
3 files changed, 11 insertions(+), 17 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 6d00ea9be..f0ac755f8 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -39,7 +39,6 @@
find-long-option
find-long-options
make-essential-device-nodes
- make-static-device-nodes
configure-qemu-networking

device-number
@@ -469,10 +468,6 @@ upon error."
mounts)
"ext4"))

- (define (lookup-module name)
- (string-append linux-module-directory "/"
- (ensure-dot-ko name)))
-
(define (load-kernel-modules)
"Examine /sys/devices to find out which modules to load and load them."
(define enter?
@@ -512,6 +507,15 @@ upon error."
(when (member "--repl" args)
(start-repl))

+ (let* ((kernel-release
+ (utsname:release (uname)))
+ (directory
+ (string-append linux-module-directory "/lib/modules/"
+ kernel-release))
+ (old-umask (umask #o022)))
+ (make-static-device-nodes directory)
+ (umask old-umask))
+
(load-kernel-modules)

(when qemu-guest-networking?
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 69e211ffa..0cba1c66f 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1857,17 +1857,6 @@ item of @var{packages}."
(setenv "EUDEV_RULES_DIRECTORY"
#$(file-append rules "/lib/udev/rules.d"))

- (let* ((kernel-release
- (utsname:release (uname)))
- (linux-module-directory
- (getenv "LINUX_MODULE_DIRECTORY"))
- (directory
- (string-append linux-module-directory "/"
- kernel-release))
- (old-umask (umask #o022)))
- (make-static-device-nodes directory)
- (umask old-umask))
-
(let ((pid (primitive-fork)))
(case pid
((0)
L
Ludovic Courtès wrote on Tue Feb 27 15:26:06+0100 2018
Re: [bug#30604] [PATCH v3 4/6] linux-boot: Load kernel modules only when the hardware is present.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87muzu33wh.fsf@gnu.org
Hello,

Nice patch series!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> * gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
> the hardware is present.
> * gnu/system/linux-initrd.scm (raw-initrd): Add imports.

[...]

> + (define (load-kernel-modules)
> + "Examine /sys/devices to find out which modules to load and load them."
> + (define enter?
> + (const #t))
> + (define (down! path stat result)
> + ;; Note: modprobe mutates the tree starting with path.
> + (let ((modalias-name (string-append path "/modalias")))
> + (if (file-exists? modalias-name)
> + (let ((modalias
> + (string-trim-right (call-with-input-file modalias-name
> + read-string)
> + #\newline)))
> + (system* "/sbin/modprobe" "-q" "--" modalias))))

Can we build upon (gnu build linux-modules) to achieve this?

Hopefully the tools at
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30629#14> can help a bit.

Thanks,
Ludo’.
D
Danny Milosavljevic wrote on Tue Feb 27 16:50:45+0100 2018
[PATCH v5 1/7] gnu: kmod: Split off kmod-minimal.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227155051.1141-2-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel. This includes commands such as
to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(license license:gpl2+)))

-(define-public kmod
+(define kmod-minimal
(package
+ (name "kmod-minimal")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:tests? #f)) ; FIXME: Investigate test failures
+ (home-page "https://www.kernel.org/")
+ (synopsis "Kernel module tools")
+ (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod. The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+ (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+ (package (inherit kmod-minimal)
(name "kmod")
(version "24")
(source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(base32
"15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
(patches (search-patches "kmod-module-directory.patch"))))
- (build-system gnu-build-system)
(native-inputs
`(("pkg-config" ,pkg-config)))
(inputs
`(("xz" ,xz)
("zlib" ,zlib)))
(arguments
- `(#:tests? #f ; FIXME: Investigate test failures
- #:configure-flags '("--with-xz" "--with-zlib")
+ `(#:configure-flags '("--with-xz" "--with-zlib")
+ #:tests? #f ; FIXME: Investigate test failures
#:phases (alist-cons-after
'install 'install-modprobe&co
(lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
(symlink "kmod"
(string-append bin "/" tool)))
'("insmod" "rmmod" "lsmod" "modprobe"
- "modinfo" "depmod"))))
- %standard-phases)))
- (home-page "https://www.kernel.org/")
- (synopsis "Kernel module tools")
- (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod. The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
- (license license:gpl2+))) ; library under lgpl2.1+
+ "modinfo" "depmod"))
+ #t))
+ %standard-phases)))))

(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
D
Danny Milosavljevic wrote on Tue Feb 27 16:50:44+0100 2018
[PATCH v5 0/7] Load Linux module only when supported hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227155051.1141-1-dannym@scratchpost.org
Danny Milosavljevic (7):
gnu: kmod: Split off kmod-minimal.
gnu: Add kmod-minimal-static.
linux-initrd: Add kmod.
linux-boot: Load kernel modules only when the hardware is present.
vm: Allow qemu-image builder to load Linux kernel modules.
vm: Make the virtio-blk is uniquely identifyable in /sys.
linux-boot: Call make-static-device-nodes much earlier.

gnu/build/linux-boot.scm | 42 ++++++++--
gnu/build/linux-initrd.scm | 12 ++-
gnu/build/vm.scm | 2 +-
gnu/local.mk | 1 +
gnu/packages/linux.scm | 91 ++++++++++++++++++----
.../patches/kmod-13-module-directory.patch | 33 ++++++++
gnu/system/linux-initrd.scm | 50 +++++++++---
gnu/system/vm.scm | 34 ++++++--
8 files changed, 224 insertions(+), 41 deletions(-)
create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch
D
Danny Milosavljevic wrote on Tue Feb 27 16:50:46+0100 2018
[PATCH v5 2/7] gnu: Add kmod-minimal-static.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227155051.1141-3-dannym@scratchpost.org
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
* gnu/packages/patches/kmod-13-module-directory.patch: New file.
* gnu/local.mk: Add it.
---
gnu/local.mk | 1 +
gnu/packages/linux.scm | 44 ++++++++++++++++++++++
.../patches/kmod-13-module-directory.patch | 33 ++++++++++++++++
3 files changed, 78 insertions(+)
create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 21195f8c1..b1e3c878d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -795,6 +795,7 @@ dist_patch_DATA = \
%D%/packages/patches/kiki-makefile.patch \
%D%/packages/patches/kiki-missing-includes.patch \
%D%/packages/patches/kiki-portability-64bit.patch \
+ %D%/packages/patches/kmod-13-module-directory.patch \
%D%/packages/patches/kmod-module-directory.patch \
%D%/packages/patches/kobodeluxe-paths.patch \
%D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch \
diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
#t))
%standard-phases)))))

+(define-public kmod-minimal/static
+ (static-package
+ (package (inherit kmod-minimal)
+ (name "kmod-minimal-static")
+ (version "13")
+ (source (origin
+ (method url-fetch)
+ (uri
+ (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+ "kmod-" version ".tar.xz"))
+ (sha256
+ (base32
+ "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+ (patches (search-patches "kmod-13-module-directory.patch"))))
+ (arguments
+ (substitute-keyword-arguments
+ (package-arguments (static-package kmod-minimal))
+ ((#:configure-flags flags ''())
+ `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+ ((#:make-flags flags ''())
+ `(cons* "LDFLAGS=-all-static" ,flags))
+ ((#:phases phases '%standard-phases)
+ `(modify-phases ,phases
+ (delete 'install-license-files)
+ (add-after 'unpack 'patch-kmod
+ (lambda _
+ ;; Reduce size by 200 kiB.
+ (substitute* "tools/kmod.c"
+ (("[&]kmod_cmd_compat_lsmod,") "")
+ (("[&]kmod_cmd_compat_rmmod,") "")
+ (("[&]kmod_cmd_compat_insmod,") "")
+ (("[&]kmod_cmd_compat_modinfo,") ""))
+ #t))
+ (replace 'install
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (bin (string-append out "/bin")))
+ (install-file "tools/kmod" bin)
+ (for-each
+ (lambda (tool)
+ (symlink "kmod" (string-append bin "/" tool)))
+ '("modprobe" "depmod"))
+ #t))))))))))
+
(define-public eudev
;; The post-systemd fork, maintained by Gentoo.
(package
diff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patch
new file mode 100644
index 000000000..5ff2f8a60
--- /dev/null
+++ b/gnu/packages/patches/kmod-13-module-directory.patch
@@ -0,0 +1,33 @@
+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'
+environment variable, rather than looking for modules exclusively in
+/lib/modules.
+
+Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to
+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable
+name.
+
+
+--- kmod-7/libkmod/libkmod.c 2012-03-15 08:19:16.750010226 -0400
++++ kmod-7/libkmod/libkmod.c 2012-04-04 15:21:29.532074313 -0400
+@@ -200,7 +200,7 @@
+ static char *get_kernel_release(const char *dirname)
+ {
+ struct utsname u;
+- char *p;
++ char *p, *dirname_prefix;
+
+ if (dirname != NULL)
+ return path_make_absolute_cwd(dirname);
+@@ -208,7 +208,10 @@
+ if (uname(&u) < 0)
+ return NULL;
+
+- if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)
++ if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)
++ dirname_prefix = dirname_default_prefix;
++
++ if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)
+ return NULL;
+
+ return p;
+
D
Danny Milosavljevic wrote on Tue Feb 27 16:50:47+0100 2018
[PATCH v5 3/7] linux-initrd: Add kmod.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227155051.1141-4-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
gnu/build/linux-initrd.scm | 12 +++++++++++-
gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++-----------
2 files changed, 45 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..6356007df 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init kmod linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (when kmod
+ (mkdir-p "sbin")
+ (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..46ef055f0 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ kmod
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
+ #:kmod #$kmod
+ #:linux-module-directory #$linux-module-directory
;; Copy everything INIT refers to into the initrd.
#:references-graphs '("closure")
#:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
(gexp->derivation name builder
#:references-graphs `(("closure" ,init))))

-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
"Return a flat directory containing the Linux kernel modules listed in
MODULES and taken from LINUX."
(define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex)
+ (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
(srfi srfi-1)
(guix build utils)
(gnu build linux-modules))
@@ -138,13 +142,27 @@ MODULES and taken from LINUX."
(recursive-module-dependencies modules
#:lookup-module lookup))))

- (mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
+ (define version
+ (match
+ (filter
+ (lambda (name)
+ (not (string-prefix? "." name)))
+ (scandir module-dir))
+ ((item) item)))
+
+ (let ((output (string-append #$output "/lib/modules/" version)))
+ (mkdir-p output)
+ (for-each (lambda (module)
+ (format #t "copying '~a'...~%" module)
+ (copy-file module
+ (string-append output "/"
+ (basename module))))
+ (delete-duplicates modules)))
+ (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+ "-e"
+ "-F" (string-append #$linux "/System.map")
+ version)
+ #t)))

(computed-file "linux-modules" build-exp))

@@ -152,6 +170,7 @@ MODULES and taken from LINUX."
#:key
(linux linux-libre)
(linux-modules '())
+ (kmod kmod-minimal/static)
(mapped-devices '())
(helper-packages '())
qemu-networking?
@@ -185,7 +204,7 @@ upon error."
mapped-devices))

(define kodir
- (flat-linux-module-directory linux linux-modules))
+ (flat-linux-module-directory linux linux-modules kmod))

(expression->initrd
(with-imported-modules (source-module-closure
@@ -223,6 +242,8 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:kmod kmod
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +266,7 @@ FILE-SYSTEMS."
(define* (base-initrd file-systems
#:key
(linux linux-libre)
+ (kmod kmod-minimal/static)
(mapped-devices '())
qemu-networking?
volatile-root?
@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."
(raw-initrd file-systems
#:linux linux
#:linux-modules linux-modules
+ #:kmod kmod
#:mapped-devices mapped-devices
- #:helper-packages helper-packages
+ #:helper-packages (cons kmod helper-packages)
#:qemu-networking? qemu-networking?
#:volatile-root? volatile-root?
#:on-error on-error))
D
Danny Milosavljevic wrote on Tue Feb 27 16:50:48+0100 2018
[PATCH v5 4/7] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227155051.1141-5-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 31 +++++++++++++++++++++++++++----
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..6d00ea9be 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,32 @@ upon error."
(string-append linux-module-directory "/"
(ensure-dot-ko name)))

+ (define (load-kernel-modules)
+ "Examine /sys/devices to find out which modules to load and load them."
+ (define enter?
+ (const #t))
+ (define (down! path stat result)
+ ;; Note: modprobe mutates the tree starting with path.
+ (let ((modalias-name (string-append path "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias
+ (string-trim-right (call-with-input-file modalias-name
+ read-string)
+ #\newline)))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))
+ #t)
+ (define up
+ (const #t))
+ (define skip
+ (const #t))
+ (define leaf
+ (const #t))
+ (define (error name stat errno result)
+ (format (current-error-port) "warning: ~a: ~a~%"
+ name (strerror errno))
+ result)
+ (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+
(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")

@@ -486,10 +512,7 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 46ef055f0..c8a9e4950 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -225,7 +225,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
D
Danny Milosavljevic wrote on Tue Feb 27 16:50:49+0100 2018
[PATCH v5 5/7] vm: Allow qemu-image builder to load Linux kernel modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227155051.1141-6-dannym@scratchpost.org
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
#:single-file-output? #t
#:references-graphs inputs))

+(define (%modprobe-wrapper modprobe linux-module-directory)
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ ;;
+ ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+ ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+ ;; environment variable is not set---hence the need for this wrapper.
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+ (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+ (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
(expression->derivation-in-linux-vm
name
- (with-imported-modules (source-module-closure '((gnu build bootloader)
+ (with-imported-modules (source-module-closure '((gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)))
#~(begin
- (use-modules (gnu build bootloader)
+ (use-modules (gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

(let ((inputs
- '#$(append (list qemu parted e2fsprogs dosfstools)
+ '#$(append (list qemu parted e2fsprogs dosfstools kmod)
(map canonical-package
(list sed grep coreutils findutils gawk))
(if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
inputs)))

(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+ ;; It's possible that we need to load nls modules in order to
+ ;; mount the new partition.
+ (if (file-exists? #$modprobe-name)
+ (activate-modprobe #$(%modprobe-wrapper modprobe-name
+ linux-module-directory))
+ (format (current-error-port)
+ "WARNING: No modprobe found in ~s. \
+Loading kernel modules will be impossible.\n" #$modprobe-name))

(let* ((graphs '#$(match inputs
(((names . _) ...)
@@ -364,7 +387,7 @@ the image."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs))
+ #:references-graphs inputs)))


;;;
D
Danny Milosavljevic wrote on Tue Feb 27 16:50:50+0100 2018
[PATCH v5 6/7] vm: Make the virtio-blk is uniquely identifyable in /sys.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227155051.1141-7-dannym@scratchpost.org
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
D
Danny Milosavljevic wrote on Tue Feb 27 16:50:51+0100 2018
[PATCH v5 7/7] linux-boot: Call make-static-device-nodes much earlier.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180227155051.1141-8-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (expression->initrd): Store data files for
make-static-device-nodes.
* gnu/build/linux-boot.scm (boot-system): Call make-static-device-nodes.
Delete lookup-module.
---
gnu/build/linux-boot.scm | 13 +++++++++----
gnu/system/linux-initrd.scm | 3 ++-
2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 6d00ea9be..1b16f267a 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,10 +469,6 @@ upon error."
mounts)
"ext4"))

- (define (lookup-module name)
- (string-append linux-module-directory "/"
- (ensure-dot-ko name)))
-
(define (load-kernel-modules)
"Examine /sys/devices to find out which modules to load and load them."
(define enter?
@@ -512,6 +508,15 @@ upon error."
(when (member "--repl" args)
(start-repl))

+ (let* ((kernel-release
+ (utsname:release (uname)))
+ (directory
+ (string-append linux-module-directory "/lib/modules/"
+ kernel-release))
+ (old-umask (umask #o022)))
+ (make-static-device-nodes directory)
+ (umask old-umask))
+
(load-kernel-modules)

(when qemu-guest-networking?
D
Danny Milosavljevic wrote on Fri Mar 02 15:16:06+0100 2018
[PATCH v6 0/6] Load Linux module only when supported hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302141606.10669-1-dannym@scratchpost.org
Danny Milosavljevic (6):
linux-modules: Add module-aliases.
linux-modules: Add install-modules.
linux-boot: Load kernel modules only when the hardware is present.
vm: Allow qemu-image builder to load Linux kernel modules.
vm: Make the virtio-blk uniquely identifiable in /sys.
linux-initrd: Provide modprobe to the initrd.

gnu/build/linux-boot.scm | 42 ++++++++++++++++++----
gnu/build/linux-initrd.scm | 13 ++++++-
gnu/build/linux-modules.scm | 61 +++++++++++++++++++++++++++++++
gnu/build/vm.scm | 2 +-
gnu/system/linux-initrd.scm | 88 +++++++++++++++++++++++++++++++++++++++------
gnu/system/vm.scm | 34 +++++++++++++++---
6 files changed, 216 insertions(+), 24 deletions(-)
D
Danny Milosavljevic wrote on Fri Mar 02 15:17:15+0100 2018
[PATCH v6 1/6] linux-modules: Add module-aliases.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302141720.10720-1-dannym@scratchpost.org
* gnu/build/linux-modules.scm (module-aliases): New variable.
---
gnu/build/linux-modules.scm | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4a6d4ff08..364339df9 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
#:use-module (ice-9 rdelim)
#:export (dot-ko
ensure-dot-ko
+ module-aliases
module-dependencies
recursive-module-dependencies
modules-loaded
@@ -95,6 +96,14 @@ contains module names, not actual file names."
(('depends . what)
(string-tokenize what %not-comma)))))

+(define (module-aliases file)
+ "Return the list of aliases for FILE."
+ (let ((info (modinfo-section-contents file)))
+ (filter-map (match-lambda
+ (('alias . value)
+ value)
+ (_ #f)) (modinfo-section-contents file))))
+
(define dot-ko
(cut string-append <> ".ko"))
D
Danny Milosavljevic wrote on Fri Mar 02 15:17:16+0100 2018
[PATCH v6 2/6] linux-modules: Add install-modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302141720.10720-2-dannym@scratchpost.org
* gnu/build/linux-modules.scm (install-modules): New procedure.
(%not-dash): New variable.
---
gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 52 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 364339df9..af217c974 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -36,6 +36,7 @@
modules-loaded
module-loaded?
load-linux-module*
+ install-module-files

current-module-debugging-port

@@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
module)))
known-aliases))

+(define %not-dash
+ (char-set-complement (char-set #\-)))
+
+(define (install-module-files module-files output)
+ "Install MODULE-FILES to OUTPUT.
+Precondition: OUTPUT is an empty directory."
+ (let ((aliases
+ (map (lambda (module-file-name)
+ (format #t "copying '~a'...~%" module-file-name)
+ (copy-file module-file-name
+ (string-append output "/"
+ (basename module-file-name)))
+ `(,(file-name->module-name module-file-name) .
+ ,(module-aliases module-file-name)))
+ (sort module-files string<))))
+ (call-with-output-file (string-append output "/modules.alias")
+ (lambda (port)
+ (format port
+ "# Aliases extracted from modules themselves.\n")
+ (for-each (match-lambda ((module . aliases)
+ (for-each (lambda (alias)
+ (format port "alias ~a ~a\n" alias
+ module))
+ aliases)))
+ aliases)))
+ (call-with-output-file (string-append output "/modules.devname")
+ (lambda (port)
+ (format port
+ "# Device nodes to trigger on-demand module loading.\n")
+ (let* ((aliases (append-map (match-lambda
+ ((module . aliases) aliases))
+ aliases))
+ (devname #f))
+ ;; Note: there's only one devname and then only one (char-major|block-major).
+ (for-each
+ (match-lambda
+ (((? (cut string-prefix? "devname:" <>) alias) . value)
+ (set! devname (string-drop value (string-length "devname:"))))
+ (((? (cut string-prefix? "char-major-" <>) alias) . value)
+ (let ((parts (string-tokenize %not-dash)))
+ (match parts
+ ((a b major minor)
+ (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
+ (((? (cut string-prefix? "block-major-" <>) alias) . value)
+ (let ((parts (string-tokenize %not-dash)))
+ (match parts
+ ((a b major minor)
+ (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
+ (_ #f))
+ aliases))))))
+
;;; linux-modules.scm ends here
D
Danny Milosavljevic wrote on Fri Mar 02 15:17:17+0100 2018
[PATCH v6 3/6] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302141720.10720-3-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
(lookup-module): Delete procedure.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 42 +++++++++++++++++++++++++++++++++++-------
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..1b16f267a 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,9 +469,31 @@ upon error."
mounts)
"ext4"))

- (define (lookup-module name)
- (string-append linux-module-directory "/"
- (ensure-dot-ko name)))
+ (define (load-kernel-modules)
+ "Examine /sys/devices to find out which modules to load and load them."
+ (define enter?
+ (const #t))
+ (define (down! path stat result)
+ ;; Note: modprobe mutates the tree starting with path.
+ (let ((modalias-name (string-append path "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias
+ (string-trim-right (call-with-input-file modalias-name
+ read-string)
+ #\newline)))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))
+ #t)
+ (define up
+ (const #t))
+ (define skip
+ (const #t))
+ (define leaf
+ (const #t))
+ (define (error name stat errno result)
+ (format (current-error-port) "warning: ~a: ~a~%"
+ name (strerror errno))
+ result)
+ (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))

(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")
@@ -486,10 +508,16 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (let* ((kernel-release
+ (utsname:release (uname)))
+ (directory
+ (string-append linux-module-directory "/lib/modules/"
+ kernel-release))
+ (old-umask (umask #o022)))
+ (make-static-device-nodes directory)
+ (umask old-umask))
+
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index e7f97bb88..b50d3ff80 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -208,7 +208,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
D
Danny Milosavljevic wrote on Fri Mar 02 15:17:18+0100 2018
[PATCH v6 4/6] vm: Allow qemu-image builder to load Linux kernel modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302141720.10720-4-dannym@scratchpost.org
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
#:single-file-output? #t
#:references-graphs inputs))

+(define (%modprobe-wrapper modprobe linux-module-directory)
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ ;;
+ ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+ ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+ ;; environment variable is not set---hence the need for this wrapper.
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+ (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+ (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
(expression->derivation-in-linux-vm
name
- (with-imported-modules (source-module-closure '((gnu build bootloader)
+ (with-imported-modules (source-module-closure '((gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)))
#~(begin
- (use-modules (gnu build bootloader)
+ (use-modules (gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

(let ((inputs
- '#$(append (list qemu parted e2fsprogs dosfstools)
+ '#$(append (list qemu parted e2fsprogs dosfstools kmod)
(map canonical-package
(list sed grep coreutils findutils gawk))
(if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
inputs)))

(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+ ;; It's possible that we need to load nls modules in order to
+ ;; mount the new partition.
+ (if (file-exists? #$modprobe-name)
+ (activate-modprobe #$(%modprobe-wrapper modprobe-name
+ linux-module-directory))
+ (format (current-error-port)
+ "WARNING: No modprobe found in ~s. \
+Loading kernel modules will be impossible.\n" #$modprobe-name))

(let* ((graphs '#$(match inputs
(((names . _) ...)
@@ -364,7 +387,7 @@ the image."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs))
+ #:references-graphs inputs)))


;;;
D
Danny Milosavljevic wrote on Fri Mar 02 15:17:19+0100 2018
[PATCH v6 5/6] vm: Make the virtio-blk uniquely identifiable in /sys.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302141720.10720-5-dannym@scratchpost.org
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
D
Danny Milosavljevic wrote on Fri Mar 02 15:17:20+0100 2018
[PATCH v6 6/6] linux-initrd: Provide modprobe to the initrd.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302141720.10720-6-dannym@scratchpost.org
* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and the
linux modules to the initrd.
* gnu/system/linux-initrd.scm (%modprobe): New procedure.
(expression->initrd): Use it. Add linux-module-directory.
(raw-initrd): Pass linux-module-directory.
---
gnu/build/linux-initrd.scm | 13 ++++++-
gnu/system/linux-initrd.scm | 84 ++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 87 insertions(+), 10 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..d4cb5e2d8 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init modprobe linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (when modprobe
+ (mkdir-p "sbin")
+ (symlink modprobe "sbin/modprobe")
+ (compile-to-cache "sbin/modprobe"))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index b50d3ff80..a69497ff8 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -56,11 +56,73 @@
;;;
;;; Code:

+(define* (%modprobe linux-module-directory #:key
+ (guile %guile-static-stripped))
+ (program-file "modprobe"
+ (with-imported-modules (source-module-closure
+ '((gnu build linux-modules)))
+ #~(begin
+ (use-modules (gnu build linux-modules) (ice-9 getopt-long)
+ (ice-9 match) (srfi srfi-1) (ice-9 ftw))
+ (define (find-only-entry directory)
+ (match (scandir directory)
+ (("." ".." basename)
+ (string-append directory "/" basename))))
+ (define (resolve-alias alias)
+ (let* ((linux-release-module-directory
+ (find-only-entry (string-append "/lib/modules"))))
+ (match (delete-duplicates (matching-modules alias
+ (known-module-aliases
+ (string-append linux-release-module-directory
+ "/modules.alias"))))
+ (()
+ (error "no alias by that name" alias))
+ (items
+ items))))
+ (define (lookup-module module)
+ (let* ((linux-release-module-directory
+ (find-only-entry (string-append "/lib/modules")))
+ (file-name (string-append linux-release-module-directory
+ "/" (ensure-dot-ko module))))
+ (if (file-exists? file-name)
+ file-name
+ (error "no module file found for module" module))))
+ (define option-spec
+ '((quiet (single-char #\q) (value #f))))
+ (define options
+ (getopt-long (command-line) option-spec))
+ (when (option-ref options 'quiet #f)
+ (current-error-port (%make-void-port "w"))
+ (current-output-port (%make-void-port "w")))
+ (let ((exit-status 0))
+ (for-each (match-lambda
+ (('quiet . #t)
+ #f)
+ ((() modules ...)
+ (for-each (lambda (alias)
+ (catch #t
+ (lambda ()
+ (let ((modules (resolve-alias alias))) (for-each (lambda (module)
+ (load-linux-module*
+ (lookup-module module)
+ #:lookup-module
+ lookup-module))
+ modules)))
+ (lambda (key . args)
+ (display (cons* key args)
+ (current-error-port))
+ (newline (current-error-port))
+ (set! exit-status 1))))
+ modules)))
+ options)
+ (exit exit-status))))
+ #:guile guile))

(define* (expression->initrd exp
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."
(define init
(program-file "init" exp #:guile guile))

+ (define modprobe
+ (%modprobe linux-module-directory #:guile guile))
+
(define builder
(with-imported-modules (source-module-closure
'((gnu build linux-initrd)))
@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
- ;; Copy everything INIT refers to into the initrd.
- #:references-graphs '("closure")
+ #:modprobe #$modprobe
+ #:linux-module-directory #$linux-module-directory
+ ;; Copy everything INIT and MODPROBE refer to into the
+ ;; initrd.
+ #:references-graphs '("init-closure"
+ "modprobe-closure")
#:gzip (string-append #$gzip "/bin/gzip")))))

(gexp->derivation name builder
- #:references-graphs `(("closure" ,init))))
+ #:references-graphs `(("init-closure" ,init)
+ ("modprobe-closure" ,modprobe))))

(define (flat-linux-module-directory linux modules)
"Return a flat directory containing the Linux kernel modules listed in
@@ -141,12 +211,7 @@ MODULES and taken from LINUX."
#:lookup-module lookup))))

(mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
+ (install-module-files (delete-duplicates modules) #$output))))

(computed-file "linux-modules" build-exp))

@@ -227,6 +292,7 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
D
Danny Milosavljevic wrote on Fri Mar 02 16:34:02+0100 2018
[PATCH v7 0/6] Load Linux module only when supported hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302153408.14091-1-dannym@scratchpost.org
Danny Milosavljevic (6):
linux-modules: Add module-aliases.
linux-modules: Add install-modules.
linux-boot: Load kernel modules only when the hardware is present.
vm: Allow qemu-image builder to load Linux kernel modules.
vm: Make the virtio-blk uniquely identifiable in /sys.
linux-initrd: Provide modprobe to the initrd.

gnu/build/linux-boot.scm | 42 +++++++++++++++---
gnu/build/linux-initrd.scm | 13 +++++-
gnu/build/linux-modules.scm | 61 ++++++++++++++++++++++++++
gnu/build/vm.scm | 2 +-
gnu/system/linux-initrd.scm | 103 ++++++++++++++++++++++++++++++++++++++------
gnu/system/vm.scm | 34 ++++++++++++---
6 files changed, 228 insertions(+), 27 deletions(-)
D
Danny Milosavljevic wrote on Fri Mar 02 16:34:03+0100 2018
[PATCH v7 1/6] linux-modules: Add module-aliases.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302153408.14091-2-dannym@scratchpost.org
* gnu/build/linux-modules.scm (module-aliases): New variable.
---
gnu/build/linux-modules.scm | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4a6d4ff08..364339df9 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
#:use-module (ice-9 rdelim)
#:export (dot-ko
ensure-dot-ko
+ module-aliases
module-dependencies
recursive-module-dependencies
modules-loaded
@@ -95,6 +96,14 @@ contains module names, not actual file names."
(('depends . what)
(string-tokenize what %not-comma)))))

+(define (module-aliases file)
+ "Return the list of aliases for FILE."
+ (let ((info (modinfo-section-contents file)))
+ (filter-map (match-lambda
+ (('alias . value)
+ value)
+ (_ #f)) (modinfo-section-contents file))))
+
(define dot-ko
(cut string-append <> ".ko"))
D
Danny Milosavljevic wrote on Fri Mar 02 16:34:04+0100 2018
[PATCH v7 2/6] linux-modules: Add install-modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302153408.14091-3-dannym@scratchpost.org
* gnu/build/linux-modules.scm (install-modules): New procedure.
(%not-dash): New variable.
---
gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 52 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 364339df9..af217c974 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -36,6 +36,7 @@
modules-loaded
module-loaded?
load-linux-module*
+ install-module-files

current-module-debugging-port

@@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
module)))
known-aliases))

+(define %not-dash
+ (char-set-complement (char-set #\-)))
+
+(define (install-module-files module-files output)
+ "Install MODULE-FILES to OUTPUT.
+Precondition: OUTPUT is an empty directory."
+ (let ((aliases
+ (map (lambda (module-file-name)
+ (format #t "copying '~a'...~%" module-file-name)
+ (copy-file module-file-name
+ (string-append output "/"
+ (basename module-file-name)))
+ `(,(file-name->module-name module-file-name) .
+ ,(module-aliases module-file-name)))
+ (sort module-files string<))))
+ (call-with-output-file (string-append output "/modules.alias")
+ (lambda (port)
+ (format port
+ "# Aliases extracted from modules themselves.\n")
+ (for-each (match-lambda ((module . aliases)
+ (for-each (lambda (alias)
+ (format port "alias ~a ~a\n" alias
+ module))
+ aliases)))
+ aliases)))
+ (call-with-output-file (string-append output "/modules.devname")
+ (lambda (port)
+ (format port
+ "# Device nodes to trigger on-demand module loading.\n")
+ (let* ((aliases (append-map (match-lambda
+ ((module . aliases) aliases))
+ aliases))
+ (devname #f))
+ ;; Note: there's only one devname and then only one (char-major|block-major).
+ (for-each
+ (match-lambda
+ (((? (cut string-prefix? "devname:" <>) alias) . value)
+ (set! devname (string-drop value (string-length "devname:"))))
+ (((? (cut string-prefix? "char-major-" <>) alias) . value)
+ (let ((parts (string-tokenize %not-dash)))
+ (match parts
+ ((a b major minor)
+ (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
+ (((? (cut string-prefix? "block-major-" <>) alias) . value)
+ (let ((parts (string-tokenize %not-dash)))
+ (match parts
+ ((a b major minor)
+ (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
+ (_ #f))
+ aliases))))))
+
;;; linux-modules.scm ends here
D
Danny Milosavljevic wrote on Fri Mar 02 16:34:05+0100 2018
[PATCH v7 3/6] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302153408.14091-4-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
(lookup-module): Delete procedure.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 42 +++++++++++++++++++++++++++++++++++-------
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..1b16f267a 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,9 +469,31 @@ upon error."
mounts)
"ext4"))

- (define (lookup-module name)
- (string-append linux-module-directory "/"
- (ensure-dot-ko name)))
+ (define (load-kernel-modules)
+ "Examine /sys/devices to find out which modules to load and load them."
+ (define enter?
+ (const #t))
+ (define (down! path stat result)
+ ;; Note: modprobe mutates the tree starting with path.
+ (let ((modalias-name (string-append path "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias
+ (string-trim-right (call-with-input-file modalias-name
+ read-string)
+ #\newline)))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))
+ #t)
+ (define up
+ (const #t))
+ (define skip
+ (const #t))
+ (define leaf
+ (const #t))
+ (define (error name stat errno result)
+ (format (current-error-port) "warning: ~a: ~a~%"
+ name (strerror errno))
+ result)
+ (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))

(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")
@@ -486,10 +508,16 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (let* ((kernel-release
+ (utsname:release (uname)))
+ (directory
+ (string-append linux-module-directory "/lib/modules/"
+ kernel-release))
+ (old-umask (umask #o022)))
+ (make-static-device-nodes directory)
+ (umask old-umask))
+
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index e7f97bb88..b50d3ff80 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -208,7 +208,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
D
Danny Milosavljevic wrote on Fri Mar 02 16:34:06+0100 2018
[PATCH v7 4/6] vm: Allow qemu-image builder to load Linux kernel modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302153408.14091-5-dannym@scratchpost.org
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
#:single-file-output? #t
#:references-graphs inputs))

+(define (%modprobe-wrapper modprobe linux-module-directory)
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ ;;
+ ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+ ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+ ;; environment variable is not set---hence the need for this wrapper.
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+ (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+ (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
(expression->derivation-in-linux-vm
name
- (with-imported-modules (source-module-closure '((gnu build bootloader)
+ (with-imported-modules (source-module-closure '((gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)))
#~(begin
- (use-modules (gnu build bootloader)
+ (use-modules (gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

(let ((inputs
- '#$(append (list qemu parted e2fsprogs dosfstools)
+ '#$(append (list qemu parted e2fsprogs dosfstools kmod)
(map canonical-package
(list sed grep coreutils findutils gawk))
(if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
inputs)))

(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+ ;; It's possible that we need to load nls modules in order to
+ ;; mount the new partition.
+ (if (file-exists? #$modprobe-name)
+ (activate-modprobe #$(%modprobe-wrapper modprobe-name
+ linux-module-directory))
+ (format (current-error-port)
+ "WARNING: No modprobe found in ~s. \
+Loading kernel modules will be impossible.\n" #$modprobe-name))

(let* ((graphs '#$(match inputs
(((names . _) ...)
@@ -364,7 +387,7 @@ the image."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs))
+ #:references-graphs inputs)))


;;;
D
Danny Milosavljevic wrote on Fri Mar 02 16:34:07+0100 2018
[PATCH v7 5/6] vm: Make the virtio-blk uniquely identifiable in /sys.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302153408.14091-6-dannym@scratchpost.org
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
D
Danny Milosavljevic wrote on Fri Mar 02 16:34:08+0100 2018
[PATCH v7 6/6] linux-initrd: Provide modprobe to the initrd.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180302153408.14091-7-dannym@scratchpost.org
* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and the
linux modules to the initrd.
* gnu/system/linux-initrd.scm (%modprobe): New procedure.
(expression->initrd): Use it. Add linux-module-directory.
(raw-initrd): Pass linux-module-directory.
---
gnu/build/linux-initrd.scm | 13 +++++-
gnu/system/linux-initrd.scm | 99 +++++++++++++++++++++++++++++++++++++++------
2 files changed, 99 insertions(+), 13 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..d4cb5e2d8 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init modprobe linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (when modprobe
+ (mkdir-p "sbin")
+ (symlink modprobe "sbin/modprobe")
+ (compile-to-cache "sbin/modprobe"))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index b50d3ff80..8050ac47e 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -56,11 +56,73 @@
;;;
;;; Code:

+(define* (%modprobe linux-module-directory #:key
+ (guile %guile-static-stripped))
+ (program-file "modprobe"
+ (with-imported-modules (source-module-closure
+ '((gnu build linux-modules)))
+ #~(begin
+ (use-modules (gnu build linux-modules) (ice-9 getopt-long)
+ (ice-9 match) (srfi srfi-1) (ice-9 ftw))
+ (define (find-only-entry directory)
+ (match (scandir directory)
+ (("." ".." basename)
+ (string-append directory "/" basename))))
+ (define (resolve-alias alias)
+ (let* ((linux-release-module-directory
+ (find-only-entry (string-append "/lib/modules"))))
+ (match (delete-duplicates (matching-modules alias
+ (known-module-aliases
+ (string-append linux-release-module-directory
+ "/modules.alias"))))
+ (()
+ (error "no alias by that name" alias))
+ (items
+ items))))
+ (define (lookup-module module)
+ (let* ((linux-release-module-directory
+ (find-only-entry (string-append "/lib/modules")))
+ (file-name (string-append linux-release-module-directory
+ "/" (ensure-dot-ko module))))
+ (if (file-exists? file-name)
+ file-name
+ (error "no module file found for module" module))))
+ (define option-spec
+ '((quiet (single-char #\q) (value #f))))
+ (define options
+ (getopt-long (command-line) option-spec))
+ (when (option-ref options 'quiet #f)
+ (current-error-port (%make-void-port "w"))
+ (current-output-port (%make-void-port "w")))
+ (let ((exit-status 0))
+ (for-each (match-lambda
+ (('quiet . #t)
+ #f)
+ ((() modules ...)
+ (for-each (lambda (alias)
+ (catch #t
+ (lambda ()
+ (let ((modules (resolve-alias alias))) (for-each (lambda (module)
+ (load-linux-module*
+ (lookup-module module)
+ #:lookup-module
+ lookup-module))
+ modules)))
+ (lambda (key . args)
+ (display (cons* key args)
+ (current-error-port))
+ (newline (current-error-port))
+ (set! exit-status 1))))
+ modules)))
+ options)
+ (exit exit-status))))
+ #:guile guile))

(define* (expression->initrd exp
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."
(define init
(program-file "init" exp #:guile guile))

+ (define modprobe
+ (%modprobe linux-module-directory #:guile guile))
+
(define builder
(with-imported-modules (source-module-closure
'((gnu build linux-initrd)))
@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
- ;; Copy everything INIT refers to into the initrd.
- #:references-graphs '("closure")
+ #:modprobe #$modprobe
+ #:linux-module-directory #$linux-module-directory
+ ;; Copy everything INIT and MODPROBE refer to into the
+ ;; initrd.
+ #:references-graphs '("init-closure"
+ "modprobe-closure")
#:gzip (string-append #$gzip "/bin/gzip")))))

(gexp->derivation name builder
- #:references-graphs `(("closure" ,init))))
+ #:references-graphs `(("init-closure" ,init)
+ ("modprobe-closure" ,modprobe))))

(define (flat-linux-module-directory linux modules)
"Return a flat directory containing the Linux kernel modules listed in
@@ -111,7 +181,7 @@ MODULES and taken from LINUX."
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex)
+ (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
(srfi srfi-1)
(guix build utils)
(gnu build linux-modules))
@@ -140,14 +210,18 @@ MODULES and taken from LINUX."
(recursive-module-dependencies modules
#:lookup-module lookup))))

- (mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
-
+ (define version
+ (match
+ (filter
+ (lambda (name)
+ (not (string-prefix? "." name)))
+ (scandir module-dir))
+ ((item) item)))
+
+ (let ((output (string-append #$output "/lib/modules/" version)))
+ (mkdir-p output)
+ (install-module-files (delete-duplicates modules) output))
+ #t)))
(computed-file "linux-modules" build-exp))

(define* (raw-initrd file-systems
@@ -227,6 +301,7 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
D
Danny Milosavljevic wrote on Fri Mar 02 17:47:11+0100 2018
Re: [PATCH v7 3/6] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)
20180302174711.4018dc49@scratchpost.org
> + (define (load-kernel-modules)
> + "Examine /sys/devices to find out which modules to load and load them."
> + (define enter?
> + (const #t))
> + (define (down! path stat result)
> + ;; Note: modprobe mutates the tree starting with path.
> + (let ((modalias-name (string-append path "/modalias")))

I should rename "path" to "directory".
D
Danny Milosavljevic wrote on Fri Mar 02 17:47:51+0100 2018
Re: [PATCH v7 1/6] linux-modules: Add module-aliases.
(address . 30604@debbugs.gnu.org)
20180302174751.741d5dd3@scratchpost.org
> + "Return the list of aliases for FILE."
^^^ replace by "in"
D
Danny Milosavljevic wrote on Sat Mar 03 14:55:26+0100 2018
[PATCH v8 0/7] Load Linux module only when supported hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180303135533.6112-1-dannym@scratchpost.org
Danny Milosavljevic (7):
linux-modules: Add module-aliases.
linux-modules: Add install-modules.
linux-boot: Load kernel modules only when the hardware is present.
vm: Allow qemu-image builder to load Linux kernel modules.
vm: Make the virtio-blk uniquely identifiable in /sys.
linux-initrd: Provide modprobe to the initrd.
linux-initrd: Factorize %modprobe and flat-linux-module-directory.

gnu/build/linux-boot.scm | 42 ++++++++++++---
gnu/build/linux-initrd.scm | 13 ++++-
gnu/build/linux-modules.scm | 115 +++++++++++++++++++++++++++++++++++++++++
gnu/build/vm.scm | 2 +-
gnu/system/linux-initrd.scm | 121 ++++++++++++++++++++++++++++++++------------
gnu/system/vm.scm | 34 +++++++++++--
6 files changed, 281 insertions(+), 46 deletions(-)
D
Danny Milosavljevic wrote on Sat Mar 03 14:55:27+0100 2018
[PATCH v8 1/7] linux-modules: Add module-aliases.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180303135533.6112-2-dannym@scratchpost.org
* gnu/build/linux-modules.scm (module-aliases): New variable.
---
gnu/build/linux-modules.scm | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4a6d4ff08..364339df9 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
#:use-module (ice-9 rdelim)
#:export (dot-ko
ensure-dot-ko
+ module-aliases
module-dependencies
recursive-module-dependencies
modules-loaded
@@ -95,6 +96,14 @@ contains module names, not actual file names."
(('depends . what)
(string-tokenize what %not-comma)))))

+(define (module-aliases file)
+ "Return the list of aliases of module FILE."
+ (let ((info (modinfo-section-contents file)))
+ (filter-map (match-lambda
+ (('alias . value)
+ value)
+ (_ #f)) (modinfo-section-contents file))))
+
(define dot-ko
(cut string-append <> ".ko"))
D
Danny Milosavljevic wrote on Sat Mar 03 14:55:28+0100 2018
[PATCH v8 2/7] linux-modules: Add install-modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180303135533.6112-3-dannym@scratchpost.org
* gnu/build/linux-modules.scm (install-modules): New procedure.
(%not-dash): New variable.
---
gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 52 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 364339df9..af217c974 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -36,6 +36,7 @@
modules-loaded
module-loaded?
load-linux-module*
+ install-module-files

current-module-debugging-port

@@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
module)))
known-aliases))

+(define %not-dash
+ (char-set-complement (char-set #\-)))
+
+(define (install-module-files module-files output)
+ "Install MODULE-FILES to OUTPUT.
+Precondition: OUTPUT is an empty directory."
+ (let ((aliases
+ (map (lambda (module-file-name)
+ (format #t "copying '~a'...~%" module-file-name)
+ (copy-file module-file-name
+ (string-append output "/"
+ (basename module-file-name)))
+ `(,(file-name->module-name module-file-name) .
+ ,(module-aliases module-file-name)))
+ (sort module-files string<))))
+ (call-with-output-file (string-append output "/modules.alias")
+ (lambda (port)
+ (format port
+ "# Aliases extracted from modules themselves.\n")
+ (for-each (match-lambda ((module . aliases)
+ (for-each (lambda (alias)
+ (format port "alias ~a ~a\n" alias
+ module))
+ aliases)))
+ aliases)))
+ (call-with-output-file (string-append output "/modules.devname")
+ (lambda (port)
+ (format port
+ "# Device nodes to trigger on-demand module loading.\n")
+ (let* ((aliases (append-map (match-lambda
+ ((module . aliases) aliases))
+ aliases))
+ (devname #f))
+ ;; Note: there's only one devname and then only one (char-major|block-major).
+ (for-each
+ (match-lambda
+ (((? (cut string-prefix? "devname:" <>) alias) . value)
+ (set! devname (string-drop value (string-length "devname:"))))
+ (((? (cut string-prefix? "char-major-" <>) alias) . value)
+ (let ((parts (string-tokenize %not-dash)))
+ (match parts
+ ((a b major minor)
+ (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
+ (((? (cut string-prefix? "block-major-" <>) alias) . value)
+ (let ((parts (string-tokenize %not-dash)))
+ (match parts
+ ((a b major minor)
+ (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
+ (_ #f))
+ aliases))))))
+
;;; linux-modules.scm ends here
D
Danny Milosavljevic wrote on Sat Mar 03 14:55:29+0100 2018
[PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180303135533.6112-4-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
(lookup-module): Delete procedure.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 42 +++++++++++++++++++++++++++++++++++-------
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..1b16f267a 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,9 +469,31 @@ upon error."
mounts)
"ext4"))

- (define (lookup-module name)
- (string-append linux-module-directory "/"
- (ensure-dot-ko name)))
+ (define (load-kernel-modules)
+ "Examine /sys/devices to find out which modules to load and load them."
+ (define enter?
+ (const #t))
+ (define (down! directory stat result)
+ ;; Note: modprobe mutates the tree starting with DIRECTORY.
+ (let ((modalias-name (string-append directory "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias
+ (string-trim-right (call-with-input-file modalias-name
+ read-string)
+ #\newline)))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))
+ #t)
+ (define up
+ (const #t))
+ (define skip
+ (const #t))
+ (define leaf
+ (const #t))
+ (define (error name stat errno result)
+ (format (current-error-port) "warning: ~a: ~a~%"
+ name (strerror errno))
+ result)
+ (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))

(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")
@@ -486,10 +508,16 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (let* ((kernel-release
+ (utsname:release (uname)))
+ (directory
+ (string-append linux-module-directory "/lib/modules/"
+ kernel-release))
+ (old-umask (umask #o022)))
+ (make-static-device-nodes directory)
+ (umask old-umask))
+
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index e7f97bb88..b50d3ff80 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -208,7 +208,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
D
Danny Milosavljevic wrote on Sat Mar 03 14:55:30+0100 2018
[PATCH v8 4/7] vm: Allow qemu-image builder to load Linux kernel modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180303135533.6112-5-dannym@scratchpost.org
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
#:single-file-output? #t
#:references-graphs inputs))

+(define (%modprobe-wrapper modprobe linux-module-directory)
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ ;;
+ ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+ ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+ ;; environment variable is not set---hence the need for this wrapper.
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+ (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+ (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
(expression->derivation-in-linux-vm
name
- (with-imported-modules (source-module-closure '((gnu build bootloader)
+ (with-imported-modules (source-module-closure '((gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)))
#~(begin
- (use-modules (gnu build bootloader)
+ (use-modules (gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

(let ((inputs
- '#$(append (list qemu parted e2fsprogs dosfstools)
+ '#$(append (list qemu parted e2fsprogs dosfstools kmod)
(map canonical-package
(list sed grep coreutils findutils gawk))
(if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
inputs)))

(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+ ;; It's possible that we need to load nls modules in order to
+ ;; mount the new partition.
+ (if (file-exists? #$modprobe-name)
+ (activate-modprobe #$(%modprobe-wrapper modprobe-name
+ linux-module-directory))
+ (format (current-error-port)
+ "WARNING: No modprobe found in ~s. \
+Loading kernel modules will be impossible.\n" #$modprobe-name))

(let* ((graphs '#$(match inputs
(((names . _) ...)
@@ -364,7 +387,7 @@ the image."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs))
+ #:references-graphs inputs)))


;;;
D
Danny Milosavljevic wrote on Sat Mar 03 14:55:31+0100 2018
[PATCH v8 5/7] vm: Make the virtio-blk uniquely identifiable in /sys.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180303135533.6112-6-dannym@scratchpost.org
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
D
Danny Milosavljevic wrote on Sat Mar 03 14:55:32+0100 2018
[PATCH v8 6/7] linux-initrd: Provide modprobe to the initrd.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180303135533.6112-7-dannym@scratchpost.org
* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and the
linux modules to the initrd.
* gnu/system/linux-initrd.scm (%modprobe): New procedure.
(expression->initrd): Use it. Add linux-module-directory.
(raw-initrd): Pass linux-module-directory.
---
gnu/build/linux-initrd.scm | 13 +++++-
gnu/system/linux-initrd.scm | 99 +++++++++++++++++++++++++++++++++++++++------
2 files changed, 99 insertions(+), 13 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..d4cb5e2d8 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init modprobe linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (when modprobe
+ (mkdir-p "sbin")
+ (symlink modprobe "sbin/modprobe")
+ (compile-to-cache "sbin/modprobe"))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index b50d3ff80..8050ac47e 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -56,11 +56,73 @@
;;;
;;; Code:

+(define* (%modprobe linux-module-directory #:key
+ (guile %guile-static-stripped))
+ (program-file "modprobe"
+ (with-imported-modules (source-module-closure
+ '((gnu build linux-modules)))
+ #~(begin
+ (use-modules (gnu build linux-modules) (ice-9 getopt-long)
+ (ice-9 match) (srfi srfi-1) (ice-9 ftw))
+ (define (find-only-entry directory)
+ (match (scandir directory)
+ (("." ".." basename)
+ (string-append directory "/" basename))))
+ (define (resolve-alias alias)
+ (let* ((linux-release-module-directory
+ (find-only-entry (string-append "/lib/modules"))))
+ (match (delete-duplicates (matching-modules alias
+ (known-module-aliases
+ (string-append linux-release-module-directory
+ "/modules.alias"))))
+ (()
+ (error "no alias by that name" alias))
+ (items
+ items))))
+ (define (lookup-module module)
+ (let* ((linux-release-module-directory
+ (find-only-entry (string-append "/lib/modules")))
+ (file-name (string-append linux-release-module-directory
+ "/" (ensure-dot-ko module))))
+ (if (file-exists? file-name)
+ file-name
+ (error "no module file found for module" module))))
+ (define option-spec
+ '((quiet (single-char #\q) (value #f))))
+ (define options
+ (getopt-long (command-line) option-spec))
+ (when (option-ref options 'quiet #f)
+ (current-error-port (%make-void-port "w"))
+ (current-output-port (%make-void-port "w")))
+ (let ((exit-status 0))
+ (for-each (match-lambda
+ (('quiet . #t)
+ #f)
+ ((() modules ...)
+ (for-each (lambda (alias)
+ (catch #t
+ (lambda ()
+ (let ((modules (resolve-alias alias))) (for-each (lambda (module)
+ (load-linux-module*
+ (lookup-module module)
+ #:lookup-module
+ lookup-module))
+ modules)))
+ (lambda (key . args)
+ (display (cons* key args)
+ (current-error-port))
+ (newline (current-error-port))
+ (set! exit-status 1))))
+ modules)))
+ options)
+ (exit exit-status))))
+ #:guile guile))

(define* (expression->initrd exp
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."
(define init
(program-file "init" exp #:guile guile))

+ (define modprobe
+ (%modprobe linux-module-directory #:guile guile))
+
(define builder
(with-imported-modules (source-module-closure
'((gnu build linux-initrd)))
@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
- ;; Copy everything INIT refers to into the initrd.
- #:references-graphs '("closure")
+ #:modprobe #$modprobe
+ #:linux-module-directory #$linux-module-directory
+ ;; Copy everything INIT and MODPROBE refer to into the
+ ;; initrd.
+ #:references-graphs '("init-closure"
+ "modprobe-closure")
#:gzip (string-append #$gzip "/bin/gzip")))))

(gexp->derivation name builder
- #:references-graphs `(("closure" ,init))))
+ #:references-graphs `(("init-closure" ,init)
+ ("modprobe-closure" ,modprobe))))

(define (flat-linux-module-directory linux modules)
"Return a flat directory containing the Linux kernel modules listed in
@@ -111,7 +181,7 @@ MODULES and taken from LINUX."
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex)
+ (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
(srfi srfi-1)
(guix build utils)
(gnu build linux-modules))
@@ -140,14 +210,18 @@ MODULES and taken from LINUX."
(recursive-module-dependencies modules
#:lookup-module lookup))))

- (mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
-
+ (define version
+ (match
+ (filter
+ (lambda (name)
+ (not (string-prefix? "." name)))
+ (scandir module-dir))
+ ((item) item)))
+
+ (let ((output (string-append #$output "/lib/modules/" version)))
+ (mkdir-p output)
+ (install-module-files (delete-duplicates modules) output))
+ #t)))
(computed-file "linux-modules" build-exp))

(define* (raw-initrd file-systems
@@ -227,6 +301,7 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
D
Danny Milosavljevic wrote on Sat Mar 03 14:55:33+0100 2018
[PATCH v8 7/7] linux-initrd: Factorize %modprobe and flat-linux-module-directory.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180303135533.6112-8-dannym@scratchpost.org
* gnu/build/linux-modules.scm (module-aliases->module-file-names): New
procedure.
* gnu/system/linux-initrd.scm (%modprobe): Use
module-aliases->module-file-names.
(flat-linux-module-directory): Use module-aliases->module-file-names.
---
gnu/build/linux-modules.scm | 56 +++++++++++++++++++++-
gnu/system/linux-initrd.scm | 110 ++++++++++++++++++--------------------------
2 files changed, 100 insertions(+), 66 deletions(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index af217c974..44059ad93 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -21,6 +21,7 @@
#:use-module (guix elf)
#:use-module (guix glob)
#:use-module (guix build syscalls)
+ #:use-module (guix build utils) ; find-files
#:use-module (rnrs io ports)
#:use-module (rnrs bytevectors)
#:use-module (srfi srfi-1)
@@ -28,9 +29,12 @@
#:use-module (ice-9 vlist)
#:use-module (ice-9 match)
#:use-module (ice-9 rdelim)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 ftw)
#:export (dot-ko
ensure-dot-ko
module-aliases
+ module-aliases->module-file-names
module-dependencies
recursive-module-dependencies
modules-loaded
@@ -385,7 +389,7 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by

(define (install-module-files module-files output)
"Install MODULE-FILES to OUTPUT.
-Precondition: OUTPUT is an empty directory."
+Precondition: OUTPUT is an empty directory except for \"modules.builtin\"."
(let ((aliases
(map (lambda (module-file-name)
(format #t "copying '~a'...~%" module-file-name)
@@ -431,4 +435,54 @@ Precondition: OUTPUT is an empty directory."
(_ #f))
aliases))))))

+(define (module-aliases->module-file-names linux aliases)
+ "Resolve ALIASES to module file names, including their dependencies (which will appear
+first). Each alias will map to a list of module file names.
+LINUX is the directory containing \"lib\"."
+ (define (string->regexp str)
+ ;; Return a regexp that matches STR exactly.
+ (string-append "^" (regexp-quote str) "$"))
+
+ (define module-dir
+ (string-append linux "/lib/modules"))
+
+ (define (find-only-entry directory)
+ (match (scandir directory)
+ (("." ".." basename)
+ (string-append directory "/" basename))))
+
+ (define linux-release-module-directory
+ (find-only-entry module-dir))
+
+ (define known-module-aliases*
+ (known-module-aliases
+ (string-append linux-release-module-directory
+ "/modules.alias")))
+ (define (resolve-alias alias)
+ "If possible, resolve ALIAS to a list of module names.
+Otherwise return just ALIAS as possible module names."
+ (match (delete-duplicates (matching-modules alias
+ known-module-aliases*))
+ (()
+ (list alias))
+ (items
+ items)))
+
+ (define (lookup module)
+ (let ((name (ensure-dot-ko module)))
+ (match (find-files module-dir (string->regexp name))
+ ((file)
+ file)
+ (()
+ (error "module not found" name module-dir))
+ ((_ ...)
+ (error "several modules by that name"
+ name module-dir)))))
+ (append-map (lambda (alias)
+ (let ((modules (map lookup (resolve-alias alias))))
+ (append (recursive-module-dependencies modules
+ #:lookup-module
+ lookup) modules)))
+ aliases))
+
;;; linux-modules.scm ends here
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 8050ac47e..dc826c63e 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -58,35 +58,14 @@

(define* (%modprobe linux-module-directory #:key
(guile %guile-static-stripped))
+ "Minimal implementation of modprobe for our initrd.
+LINUX-MODULE-DIRECTORY is the directory that contains \"lib\"."
(program-file "modprobe"
(with-imported-modules (source-module-closure
'((gnu build linux-modules)))
#~(begin
(use-modules (gnu build linux-modules) (ice-9 getopt-long)
- (ice-9 match) (srfi srfi-1) (ice-9 ftw))
- (define (find-only-entry directory)
- (match (scandir directory)
- (("." ".." basename)
- (string-append directory "/" basename))))
- (define (resolve-alias alias)
- (let* ((linux-release-module-directory
- (find-only-entry (string-append "/lib/modules"))))
- (match (delete-duplicates (matching-modules alias
- (known-module-aliases
- (string-append linux-release-module-directory
- "/modules.alias"))))
- (()
- (error "no alias by that name" alias))
- (items
- items))))
- (define (lookup-module module)
- (let* ((linux-release-module-directory
- (find-only-entry (string-append "/lib/modules")))
- (file-name (string-append linux-release-module-directory
- "/" (ensure-dot-ko module))))
- (if (file-exists? file-name)
- file-name
- (error "no module file found for module" module))))
+ (ice-9 match) (srfi srfi-1))
(define option-spec
'((quiet (single-char #\q) (value #f))))
(define options
@@ -98,22 +77,31 @@
(for-each (match-lambda
(('quiet . #t)
#f)
- ((() modules ...)
- (for-each (lambda (alias)
- (catch #t
- (lambda ()
- (let ((modules (resolve-alias alias))) (for-each (lambda (module)
- (load-linux-module*
- (lookup-module module)
- #:lookup-module
- lookup-module))
- modules)))
- (lambda (key . args)
- (display (cons* key args)
- (current-error-port))
- (newline (current-error-port))
- (set! exit-status 1))))
- modules)))
+ ((() aliases ...)
+ (catch #t
+ (lambda ()
+ (let ((module-file-names
+ (module-aliases->module-file-names
+ #$linux-module-directory aliases)))
+ (for-each (lambda (name)
+ (catch 'system-error
+ (lambda ()
+ (when (not (load-linux-module* name
+ #:recursive?
+ #f))
+ (set! exit-status 1)))
+ (lambda (key . args)
+ (when (not (= EEXIST
+ (system-error-errno
+ (cons key args))))
+ (print-exception (current-error-port)
+ #f key args)
+ (set! exit-status 1)))))
+ module-file-names)))
+ (lambda (key . args)
+ (print-exception (current-error-port)
+ #f key args)
+ (set! exit-status 1)))))
options)
(exit exit-status))))
#:guile guile))
@@ -173,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
#:references-graphs `(("init-closure" ,init)
("modprobe-closure" ,modprobe))))

-(define (flat-linux-module-directory linux modules)
- "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+(define (flat-linux-module-directory linux aliases)
+ "Return a flat directory containing the Linux kernel modules resolved by
+ALIASES and taken from LINUX."
(define build-exp
(with-imported-modules (source-module-closure
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
+ (use-modules (ice-9 match) (ice-9 ftw)
(srfi srfi-1)
- (guix build utils)
+ (guix build utils) ; TODO: Remove
(gnu build linux-modules))

(define (string->regexp str)
@@ -193,33 +181,25 @@ MODULES and taken from LINUX."
(define module-dir
(string-append #$linux "/lib/modules"))

- (define (lookup module)
- (let ((name (ensure-dot-ko module)))
- (match (find-files module-dir (string->regexp name))
- ((file)
- file)
- (()
- (error "module not found" name module-dir))
- ((_ ...)
- (error "several modules by that name"
- name module-dir)))))
+ (define (find-only-entry directory)
+ (match (scandir directory)
+ (("." ".." basename)
+ (string-append directory "/" basename))))
+
+ (define linux-release-module-directory
+ (find-only-entry module-dir))

(define modules
- (let ((modules (map lookup '#$modules)))
- (append modules
- (recursive-module-dependencies modules
- #:lookup-module lookup))))
+ (module-aliases->module-file-names #$linux '#$aliases))

(define version
- (match
- (filter
- (lambda (name)
- (not (string-prefix? "." name)))
- (scandir module-dir))
- ((item) item)))
+ (basename linux-release-module-directory))

(let ((output (string-append #$output "/lib/modules/" version)))
(mkdir-p output)
+ (install-file
+ (string-append linux-release-module-directory "/modules.builtin")
+ output)
(install-module-files (delete-duplicates modules) output))
#t)))
(computed-file "linux-modules" build-exp))
D
Danny Milosavljevic wrote on Sat Mar 03 16:32:12+0100 2018
Re: [PATCH v8 2/7] linux-modules: Add install-modules.
(address . 30604@debbugs.gnu.org)
20180303163212.40f17905@scratchpost.org
> + ;; Note: there's only one devname and then only one (char-major|block-major).
> + (for-each
> + (match-lambda
> + (((? (cut string-prefix? "devname:" <>) alias) . value)
> + (set! devname (string-drop value (string-length "devname:"))))
> + (((? (cut string-prefix? "char-major-" <>) alias) . value)
> + (let ((parts (string-tokenize %not-dash)))
> + (match parts
> + ((a b major minor)
> + (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
> + (((? (cut string-prefix? "block-major-" <>) alias) . value)
> + (let ((parts (string-tokenize %not-dash)))
> + (match parts
> + ((a b major minor)
> + (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
> + (_ #f))
> + aliases))))))

Probably better to be more careful that devname is set early enough.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqav/wACgkQ5xo1VCww
uqV50wf/VBhDH1xsp4i365g/suHyV27f38/lFep7ohe11uMGlKxolmCCaGoo8/bC
lhVuRnGme8dWV3DaKW8Ih+DE82FkZr+EulBxK9MmPxuw3FRCZDqw058PPkjM+pAM
a09O8uj3qHTsWhNZzKA+Mf0o442+Pz4MwQE2JnX1eEuFltTzBrVAKjIsgbHgmRfh
OAfPnmmBK3lNRTDB/qLvc6U1iuZOP+JduRr05kr7cjOFDZ2Rr0Q1dgB034GXwlj0
3dCw2izbdKc6xhGP5wjDt6Hradg+ReglrDgLzh0mXY0hq9aUIvXxj9ngA3Q05gu1
5n9B45TPg7OOKL7hsHumczfWFXLBLA==
=EtMr
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Sat Mar 03 19:01:31+0100 2018
Re: [PATCH v8 7/7] linux-initrd: Factorize %modprobe and flat-linux-module-directory.
(address . 30604@debbugs.gnu.org)
20180303190131.5a400f7d@scratchpost.org
> + (guix build utils) ; TODO: Remove

Required for mkdir-p
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqa4vsACgkQ5xo1VCww
uqXOsgf/YAhsza+9v+prxTjzBWMjmdnk4mmMOMjJMoKQvMJlUDKzpVGXRBmmiAXc
gkDel+oaMwi59dhe9jCmMb6cxDETIrFWk7HTHnjpAW5TwS6giIDu4ka83Qk65hI0
cQYgntsHOLfoOrONKnZ9Aw6TnG19lZAsyga1efmFllvyuU+tXuKAde2YZhrbx9ri
vvKJTIVmeAd5tXRxhw44yniPigWS2MRJkEasgYOJPNbaBtoZ4JkfIaWkBIGxsDtF
uVw+JxlMqp/SlbnPPEdYasDWSijIv9ypqAFjNoU/oZd9NqjO8ORSdctRutKy///R
ANg1nn1m7F3x48yAenSc78etjezWfA==
=ILLM
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Sat Mar 03 22:58:36+0100 2018
Re: [bug#30604] [PATCH v8 1/7] linux-modules: Add module-aliases.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87371g6ctv.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

> * gnu/build/linux-modules.scm (module-aliases): New variable.

LGTM!

> +(define (module-aliases file)
> + "Return the list of aliases of module FILE."
> + (let ((info (modinfo-section-contents file)))
> + (filter-map (match-lambda
> + (('alias . value)
> + value)
> + (_ #f)) (modinfo-section-contents file))))

Nitpick: align like this:

(filter-map first
second)

when the first arg spans several lines.
L
Ludovic Courtès wrote on Sat Mar 03 23:07:57+0100 2018
Re: [bug#30604] [PATCH v8 2/7] linux-modules: Add install-modules.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87y3j84xtu.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

> * gnu/build/linux-modules.scm (install-modules): New procedure.
> (%not-dash): New variable.

We could reuse modules.alias and modules.devname from the ‘linux-libre’
package (right?), but I guess it doesn’t hurt to generate custom ones.

> +(define (install-module-files module-files output)
> + "Install MODULE-FILES to OUTPUT.
> +Precondition: OUTPUT is an empty directory."
> + (let ((aliases
> + (map (lambda (module-file-name)
> + (format #t "copying '~a'...~%" module-file-name)
> + (copy-file module-file-name
> + (string-append output "/"
> + (basename module-file-name)))
> + `(,(file-name->module-name module-file-name) .
> + ,(module-aliases module-file-name)))
> + (sort module-files string<))))
> + (call-with-output-file (string-append output "/modules.alias")
> + (lambda (port)
> + (format port
> + "# Aliases extracted from modules themselves.\n")
> + (for-each (match-lambda ((module . aliases)
> + (for-each (lambda (alias)
> + (format port "alias ~a ~a\n" alias
> + module))
> + aliases)))
> + aliases)))
> + (call-with-output-file (string-append output "/modules.devname")
> + (lambda (port)
> + (format port
> + "# Device nodes to trigger on-demand module loading.\n")
> + (let* ((aliases (append-map (match-lambda
> + ((module . aliases) aliases))
> + aliases))
> + (devname #f))
> + ;; Note: there's only one devname and then only one (char-major|block-major).
> + (for-each
> + (match-lambda
> + (((? (cut string-prefix? "devname:" <>) alias) . value)
> + (set! devname (string-drop value (string-length "devname:"))))
> + (((? (cut string-prefix? "char-major-" <>) alias) . value)
> + (let ((parts (string-tokenize %not-dash)))
> + (match parts
> + ((a b major minor)
> + (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
> + (((? (cut string-prefix? "block-major-" <>) alias) . value)
> + (let ((parts (string-tokenize %not-dash)))
> + (match parts
> + ((a b major minor)
> + (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
> + (_ #f))
> + aliases))))))

I think we need different procedures here:

(write-module-alias-database modules port) ;for “modules.alias”
(write-module-device-database modules port) ;for “modules.devname”

with docstrings.

I’m not sure we need ‘install-module-files’ itself. Perhaps we can
inline it at the call site?

For the devname code, please avoid ‘set!’. Instead you can thread the
current devname as the state of a loop:

(let loop ((devname #f)
(aliases aliases))
(match aliases
(() …)
(((? devname-alias? devname) . rest)
(loop devname rest))
…))

The indentation of ‘match’ forms is wrong. Would it be OK for you to
pass it through ./etc/indent-code.el? (It’s non interactive, you don’t
need to actually use Emacs.)

Thanks,
Ludo’.
L
Ludovic Courtès wrote on Sat Mar 03 23:48:38+0100 2018
Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87sh9g4vy1.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

> * gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
> the hardware is present.
> (lookup-module): Delete procedure.
> * gnu/system/linux-initrd.scm (raw-initrd): Add imports.

[...]

> + (define (load-kernel-modules)
> + "Examine /sys/devices to find out which modules to load and load them."
> + (define enter?
> + (const #t))
> + (define (down! directory stat result)
> + ;; Note: modprobe mutates the tree starting with DIRECTORY.
> + (let ((modalias-name (string-append directory "/modalias")))
> + (if (file-exists? modalias-name)
> + (let ((modalias
> + (string-trim-right (call-with-input-file modalias-name
> + read-string)
> + #\newline)))
> + (system* "/sbin/modprobe" "-q" "--" modalias))))

If we change ‘flat-linux-module-directory’ to produce a ‘modules.alias’
file, here we could read ‘modules.aliases’ directly and load the right
thing.

With the patch below, we get ‘needed-modules’, and we could simply do:

(for-each (catch-ENOENT load-linux-module*)
(needed-modules
(known-module-aliases (string-append linux-module-directory
"/modules.alias"))))

and we can do away with kmod’s modprobe.

Thoughts?

Ludo’.
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4a6d4ff08..251095072 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -20,6 +20,7 @@
(define-module (gnu build linux-modules)
#:use-module (guix elf)
#:use-module (guix glob)
+ #:use-module (guix build utils)
#:use-module (guix build syscalls)
#:use-module (rnrs io ports)
#:use-module (rnrs bytevectors)
@@ -40,7 +41,8 @@

device-module-aliases
known-module-aliases
- matching-modules))
+ matching-modules
+ needed-modules))

;;; Commentary:
;;;
@@ -370,4 +372,25 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
module)))
known-aliases))

+(define (system-device-aliases)
+ "Browse /sys/devices in search of \"modalias\" files and return the list of
+device aliases for the current system."
+ (let ((files (find-files "/sys/devices"
+ (lambda (file stat)
+ (and (eq? 'regular (stat:type stat))
+ (string=? "modalias" (basename file)))))))
+ (filter-map (lambda (file)
+ (match (string-trim-right
+ (call-with-input-file file get-string-all))
+ ("" #f)
+ (alias alias)))
+ files)))
+
+(define* (needed-modules #:optional (known-aliases (known-module-aliases)))
+ "Return the list of modules needed by devices on the current system. This
+is achieved by browsing /sys/devices and returning the maching modules from
+KNOWN-ALIASES."
+ (append-map (cut matching-modules <> known-aliases)
+ (system-device-aliases)))
+
;;; linux-modules.scm ends here
D
Danny Milosavljevic wrote on Sun Mar 04 02:06:43+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180304020643.6773e016@scratchpost.org
Hi Ludo,

On Sat, 03 Mar 2018 23:48:38 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> > + (system* "/sbin/modprobe" "-q" "--" modalias))))
>
> If we change ‘flat-linux-module-directory’ to produce a ‘modules.alias’
> file, here we could read ‘modules.aliases’ directly and load the right
> thing.
>
> With the patch below, we get ‘needed-modules’, and we could simply do:
>
> (for-each (catch-ENOENT load-linux-module*)
> (needed-modules
> (known-module-aliases (string-append linux-module-directory
> "/modules.alias"))))
>
> and we can do away with kmod’s modprobe.

It's not kmod's modprobe anymore. It's our pure-Guile implementation.

Linux lazy-invokes modprobe (for example when mounting stuff), so
/sbin/modprobe is never going away - but it can be our implementation.

I doubt it will take the modules from the correct directory with your patch.
(Of course otherwise it looks much nicer - but I think it won't pass the tests)

My newest version (v9) will use the same procedure for both computing the list
of modules for flat-linux-module-directory and the list of modules that are to
be modprobed - I think it's nice to be able to keep those in sync so we don't
get nasty surprises. Let's see how that goes...
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqbRqMACgkQ5xo1VCww
uqXDwQgAowI5zaIU+SBYbyZPcBgJoXZC0c5raPEdUouIjGTcIY3A0uIY9hLfuMDX
kX+Sg60ZEvxBtoJmsy9eMNKlvuzVduB8jNEouH1MQcUBLu6wSLvHhkQRHjamAHkN
oRKpLTj429itvdQ0jNewulqshW3xeeneepYDITKFD/sOAHt/ek0DFthYYxIJUEXY
PXLkJ+O2mCyspt4uoc+8L+4dtEO9gv3asmu4mhk8D5+bXZ0DgBVsNqmQrOe3XSkc
S0no3CafpSInmq+HuGAbFwnJcQbp1ISd3AoSMXEBAfaSVoaE/ZzKFKLgavNuvbpD
SQdW8dstK+ZyUiOP6KiaG3g5tn2AHg==
=Qtnk
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Sun Mar 04 02:09:07+0100 2018
[PATCH v9 0/7] Load Linux module only when supported hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180304010914.1322-1-dannym@scratchpost.org
Danny Milosavljevic (7):
linux-modules: Add "modules.devname" and "modules.alias" writer.
linux-modules: Add module-aliases->module-file-names.
linux-initrd: Provide pure-Guile modprobe.
linux-boot: Load kernel modules only when the hardware is present.
vm: Allow qemu-image builder to load Linux kernel modules.
vm: Make the virtio-blk uniquely identifiable in /sys.
linux-initrd: Use module-aliases->module-file-names, too.

gnu/build/linux-boot.scm | 42 +++++++++++---
gnu/build/linux-initrd.scm | 13 ++++-
gnu/build/linux-modules.scm | 114 +++++++++++++++++++++++++++++++++++++
gnu/build/vm.scm | 2 +-
gnu/system/linux-initrd.scm | 136 +++++++++++++++++++++++++++++++++-----------
gnu/system/vm.scm | 34 +++++++++--
6 files changed, 295 insertions(+), 46 deletions(-)
D
Danny Milosavljevic wrote on Sun Mar 04 02:09:08+0100 2018
[PATCH v9 1/7] linux-modules: Add "modules.devname" and "modules.alias" writer.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180304010914.1322-2-dannym@scratchpost.org
* gnu/build/linux-modules.scm (write-module-alias-database): New procedure.
(write-module-device-database): New procedure.
(%not-dash): New variable.
---
gnu/build/linux-modules.scm | 58 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4fe673cca..0aaf2ff6f 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -36,6 +36,8 @@
modules-loaded
module-loaded?
load-linux-module*
+ write-module-alias-database
+ write-module-device-database

current-module-debugging-port

@@ -380,4 +382,60 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
module)))
known-aliases))

+(define %not-dash
+ (char-set-complement (char-set #\-)))
+
+(define (write-module-alias-database aliases output)
+ "Install \"modules.alias\" for ALIASES to directory OUTPUT."
+ (call-with-output-file (string-append output "/modules.alias")
+ (lambda (port)
+ (format port
+ "# Aliases extracted from modules themselves.\n")
+ (for-each (match-lambda ((module . aliases)
+ (for-each (lambda (alias)
+ (format port "alias ~a ~a\n" alias
+ module))
+ aliases)))
+ aliases))))
+
+(define (write-module-device-database aliases output)
+ "Install \"modules.devname\" for ALIASES to directory OUTPUT."
+ (call-with-output-file (string-append output "/modules.devname")
+ (lambda (port)
+ (format port
+ "# Device nodes to trigger on-demand module loading.\n")
+ (for-each (match-lambda
+ ((module . aliases)
+ (let* ((interesting-aliases
+ ;; Note: there's only one devname and then only one
+ ;; (char-major|block-major).
+ (filter-map
+ (match-lambda
+ ((? (cut string-prefix? "devname:" <>) alias)
+ `(devname . ,(string-drop alias (string-length "devname:"))))
+ ((? (cut string-prefix? "char-major-" <>) alias)
+ `(char-major . ,(string-drop alias (string-length "char-major-"))))
+ ((? (cut string-prefix? "block-major-" <>) alias)
+ `(block-major . ,(string-drop alias (string-length "block-major-"))))
+ (_ #f))
+ aliases))
+ (devname (assq-ref interesting-aliases
+ 'devname))
+ (char-major (assq-ref interesting-aliases
+ 'char-major))
+ (block-major (assq-ref interesting-aliases
+ 'block-major)))
+ (when (and devname char-major)
+ (let ((parts (string-tokenize char-major %not-dash)))
+ (match parts
+ ((major minor)
+ (format port "~a ~a ~a~a:~a\n" module devname
+ "c" major minor)))))
+ (when (and devname block-major)
+ (let ((parts (string-tokenize block-major %not-dash)))
+ (match parts
+ ((major minor)
+ (format port "~a ~a ~a~a:~a\n" module devname
+ "b" major minor)))))))) aliases))))
+
;;; linux-modules.scm ends here
D
Danny Milosavljevic wrote on Sun Mar 04 02:09:09+0100 2018
[PATCH v9 2/7] linux-modules: Add module-aliases->module-file-names.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180304010914.1322-3-dannym@scratchpost.org
* gnu/build/linux-modules.scm (module-aliases->module-file-names): New
procedure.
---
gnu/build/linux-modules.scm | 55 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 55 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 0aaf2ff6f..f6bb0512b 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -21,6 +21,7 @@
#:use-module (guix elf)
#:use-module (guix glob)
#:use-module (guix build syscalls)
+ #:use-module (guix build utils) ; find-files
#:use-module (rnrs io ports)
#:use-module (rnrs bytevectors)
#:use-module (srfi srfi-1)
@@ -28,9 +29,12 @@
#:use-module (ice-9 vlist)
#:use-module (ice-9 match)
#:use-module (ice-9 rdelim)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 ftw)
#:export (dot-ko
ensure-dot-ko
module-aliases
+ module-aliases->module-file-names
module-dependencies
recursive-module-dependencies
modules-loaded
@@ -438,4 +442,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
(format port "~a ~a ~a~a:~a\n" module devname
"b" major minor)))))))) aliases))))

+(define (module-aliases->module-file-names linux aliases)
+ "Resolve ALIASES to module file names, including their dependencies (which will appear
+first). Each alias will map to a list of module file names.
+LINUX is the directory containing \"lib\"."
+ (define (string->regexp str)
+ ;; Return a regexp that matches STR exactly.
+ (string-append "^" (regexp-quote str) "$"))
+
+ (define module-dir
+ (string-append linux "/lib/modules"))
+
+ (define (find-only-entry directory)
+ (match (scandir directory)
+ (("." ".." basename)
+ (string-append directory "/" basename))))
+
+ (define linux-release-module-directory
+ (find-only-entry module-dir))
+
+ (define known-module-aliases*
+ (known-module-aliases
+ (string-append linux-release-module-directory
+ "/modules.alias")))
+ (define (resolve-alias alias)
+ "If possible, resolve ALIAS to a list of module names.
+Otherwise return just ALIAS as possible module names."
+ (match (delete-duplicates (matching-modules alias
+ known-module-aliases*))
+ (()
+ (list alias))
+ (items
+ items)))
+
+ (define (lookup module)
+ (let ((name (ensure-dot-ko module)))
+ (match (find-files module-dir (string->regexp name))
+ ((file)
+ file)
+ (()
+ (error "module not found" name module-dir))
+ ((_ ...)
+ (error "several modules by that name"
+ name module-dir)))))
+ (append-map (lambda (alias)
+ (let ((modules (map lookup (resolve-alias alias))))
+ (append (recursive-module-dependencies modules
+ #:lookup-module
+ lookup)
+ modules)))
+ aliases))
+
;;; linux-modules.scm ends here
D
Danny Milosavljevic wrote on Sun Mar 04 02:09:10+0100 2018
[PATCH v9 3/7] linux-initrd: Provide pure-Guile modprobe.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180304010914.1322-4-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (%modprobe): New variable.
(expression->initrd): Add modprobe, LINUX-MODULE-DIRECTORY.
(raw-initrd): Pass KODIR as LINUX-MODULE-DIRECTORY.
* gnu/build/linux-initrd.scm (build-initrd): Add modprobe.
---
gnu/build/linux-initrd.scm | 13 ++++++++-
gnu/system/linux-initrd.scm | 65 ++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 74 insertions(+), 4 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..d4cb5e2d8 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init modprobe linux-module-directory
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+ (when modprobe
+ (mkdir-p "sbin")
+ (symlink modprobe "sbin/modprobe")
+ (compile-to-cache "sbin/modprobe"))
+
+ ;; Make modules available as /lib/modules so modprobe finds them.
+ (mkdir-p "lib")
+ (symlink (string-append linux-module-directory "/lib/modules")
+ "lib/modules")
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index e0cb59c00..6ad6d75f7 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -46,7 +46,8 @@
%base-initrd-modules
raw-initrd
file-system-packages
- base-initrd))
+ base-initrd
+ %modprobe))


;;; Commentary:
@@ -56,11 +57,61 @@
;;;
;;; Code:

+(define* (%modprobe linux-module-directory #:key
+ (guile %guile-static-stripped))
+ "Minimal implementation of modprobe for our initrd.
+LINUX-MODULE-DIRECTORY is the directory that contains \"lib\"."
+ (program-file "modprobe"
+ (with-imported-modules (source-module-closure
+ '((gnu build linux-modules)))
+ #~(begin
+ (use-modules (gnu build linux-modules) (ice-9 getopt-long)
+ (ice-9 match) (srfi srfi-1))
+ (define option-spec
+ '((quiet (single-char #\q) (value #f))))
+ (define options
+ (getopt-long (command-line) option-spec))
+ (when (option-ref options 'quiet #f)
+ (current-error-port (%make-void-port "w"))
+ (current-output-port (%make-void-port "w")))
+ (let ((exit-status 0))
+ (for-each (match-lambda
+ (('quiet . #t)
+ #f)
+ ((() aliases ...)
+ (catch #t
+ (lambda ()
+ (let ((module-file-names
+ (module-aliases->module-file-names
+ #$linux-module-directory aliases)))
+ (for-each (lambda (name)
+ (catch 'system-error
+ (lambda ()
+ (when (not (load-linux-module* name
+ #:recursive?
+ #f))
+ (set! exit-status 1)))
+ (lambda (key . args)
+ (when (not (= EEXIST
+ (system-error-errno
+ (cons key args))))
+ (print-exception (current-error-port)
+ #f key args)
+ (set! exit-status 1)))))
+ module-file-names)))
+ (lambda (key . args)
+ (print-exception (current-error-port)
+ #f key args)
+ (set! exit-status 1)))))
+ options)
+ (exit exit-status))))
+ #:guile guile))

(define* (expression->initrd exp
#:key
(guile %guile-static-stripped)
(gzip gzip)
+ linux-module-directory
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -73,6 +124,9 @@ the derivations referenced by EXP are automatically copied to the initrd."
(define init
(program-file "init" exp #:guile guile))

+ (define modprobe
+ (%modprobe linux-module-directory #:guile guile))
+
(define builder
(with-imported-modules (source-module-closure
'((gnu build linux-initrd)))
@@ -96,12 +150,16 @@ the derivations referenced by EXP are automatically copied to the initrd."
(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
+ #:modprobe #$modprobe
+ #:linux-module-directory #$linux-module-directory
;; Copy everything INIT refers to into the initrd.
- #:references-graphs '("closure")
+ #:references-graphs '("init-closure"
+ "modprobe-closure")
#:gzip (string-append #$gzip "/bin/gzip")))))

(gexp->derivation name builder
- #:references-graphs `(("closure" ,init))))
+ #:references-graphs `(("init-closure" ,init)
+ ("modprobe-closure" ,modprobe))))

(define (flat-linux-module-directory linux modules)
"Return a flat directory containing the Linux kernel modules listed in
@@ -225,6 +283,7 @@ upon error."
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:linux-module-directory kodir
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
D
Danny Milosavljevic wrote on Sun Mar 04 02:09:11+0100 2018
[PATCH v9 4/7] linux-boot: Load kernel modules only when the hardware is present.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180304010914.1322-5-dannym@scratchpost.org
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
gnu/build/linux-boot.scm | 42 +++++++++++++++++++++++++++++++++++-------
gnu/system/linux-initrd.scm | 4 +++-
2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..2236d8971 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,9 +469,31 @@ upon error."
mounts)
"ext4"))

- (define (lookup-module name)
- (string-append linux-module-directory "/"
- (ensure-dot-ko name)))
+ (define (load-kernel-modules)
+ "Examine /sys/devices to find out which modules to load and load them."
+ (define enter?
+ (const #t))
+ (define (down! directory stat result)
+ ;; Note: modprobe mutates the tree starting with DIRECTORY.
+ (let ((modalias-name (string-append directory "/modalias")))
+ (if (file-exists? modalias-name)
+ (let ((modalias
+ (string-trim-right (call-with-input-file modalias-name
+ read-string)
+ #\newline)))
+ (system* "/sbin/modprobe" "-q" "--" modalias))))
+ #t)
+ (define up
+ (const #t))
+ (define skip
+ (const #t))
+ (define leaf
+ (const #t))
+ (define (error name stat errno result)
+ (format (current-error-port) "warning: ~a: ~a~%"
+ name (strerror errno))
+ result)
+ (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))

(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")
@@ -486,10 +508,16 @@ upon error."
(when (member "--repl" args)
(start-repl))

- (display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (let* ((kernel-release
+ (utsname:release (uname)))
+ (directory
+ (string-append linux-module-directory "/lib/modules/"
+ kernel-release))
+ (old-umask (umask #o022)))
+ (make-static-device-nodes directory)
+ (umask old-umask))
+
+ (load-kernel-modules)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 6ad6d75f7..339ecf754 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -266,7 +266,9 @@ upon error."
;; this info via gexps.
((gnu build file-systems)
#:select (find-partition-by-luks-uuid))
- (rnrs bytevectors))
+ (rnrs bytevectors)
+ (ice-9 ftw)
+ (ice-9 rdelim))

(with-output-to-port (%make-void-port "w")
(lambda ()
D
Danny Milosavljevic wrote on Sun Mar 04 02:09:12+0100 2018
[PATCH v9 5/7] vm: Allow qemu-image builder to load Linux kernel modules.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180304010914.1322-6-dannym@scratchpost.org
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 91ff32ce9..cf1ec651a 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
#:single-file-output? #t
#:references-graphs inputs))

+(define (%modprobe-wrapper modprobe linux-module-directory)
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ ;;
+ ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+ ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+ ;; environment variable is not set---hence the need for this wrapper.
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+ (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+ (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
(expression->derivation-in-linux-vm
name
- (with-imported-modules (source-module-closure '((gnu build bootloader)
+ (with-imported-modules (source-module-closure '((gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)))
#~(begin
- (use-modules (gnu build bootloader)
+ (use-modules (gnu build activation)
+ (gnu build bootloader)
(gnu build vm)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

(let ((inputs
- '#$(append (list qemu parted e2fsprogs dosfstools)
+ '#$(append (list qemu parted e2fsprogs dosfstools kmod)
(map canonical-package
(list sed grep coreutils findutils gawk))
(if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
inputs)))

(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+ ;; It's possible that we need to load nls modules in order to
+ ;; mount the new partition.
+ (if (file-exists? #$modprobe-name)
+ (activate-modprobe #$(%modprobe-wrapper modprobe-name
+ linux-module-directory))
+ (format (current-error-port)
+ "WARNING: No modprobe found in ~s. \
+Loading kernel modules will be impossible.\n" #$modprobe-name))

(let* ((graphs '#$(match inputs
(((names . _) ...)
@@ -364,7 +387,7 @@ the image."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs))
+ #:references-graphs inputs)))


;;;
D
Danny Milosavljevic wrote on Sun Mar 04 02:09:13+0100 2018
[PATCH v9 6/7] vm: Make the virtio-blk uniquely identifiable in /sys.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180304010914.1322-7-dannym@scratchpost.org
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index cf1ec651a..78cc8cad1 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -700,7 +700,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
D
Danny Milosavljevic wrote on Sun Mar 04 02:09:14+0100 2018
[PATCH v9 7/7] linux-initrd: Use module-aliases->module-file-names, too.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180304010914.1322-8-dannym@scratchpost.org
* gnu/system/linux-initrd.scm (flat-linux-module-directory): Use
module-aliases->module-file-names.
* gnu/build/linux-modules.scm (file-name->module-name): Export.
---
gnu/build/linux-modules.scm | 1 +
gnu/system/linux-initrd.scm | 67 ++++++++++++++++++++++++++-------------------
2 files changed, 40 insertions(+), 28 deletions(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index f6bb0512b..81a4b15b1 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -35,6 +35,7 @@
ensure-dot-ko
module-aliases
module-aliases->module-file-names
+ file-name->module-name
module-dependencies
recursive-module-dependencies
modules-loaded
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 339ecf754..0b976afad 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -161,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
#:references-graphs `(("init-closure" ,init)
("modprobe-closure" ,modprobe))))

-(define (flat-linux-module-directory linux modules)
- "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+(define (flat-linux-module-directory linux aliases)
+ "Return a flat directory containing the Linux kernel modules resolved by
+ALIASES and taken from LINUX."
(define build-exp
(with-imported-modules (source-module-closure
'((guix build utils)
(gnu build linux-modules)))
#~(begin
- (use-modules (ice-9 match) (ice-9 regex)
+ (use-modules (ice-9 match) (ice-9 ftw)
(srfi srfi-1)
- (guix build utils)
+ (guix build utils) ; mkdir-p
(gnu build linux-modules))

(define (string->regexp str)
@@ -181,31 +181,42 @@ MODULES and taken from LINUX."
(define module-dir
(string-append #$linux "/lib/modules"))

- (define (lookup module)
- (let ((name (ensure-dot-ko module)))
- (match (find-files module-dir (string->regexp name))
- ((file)
- file)
- (()
- (error "module not found" name module-dir))
- ((_ ...)
- (error "several modules by that name"
- name module-dir)))))
+ (define (find-only-entry directory)
+ (match (scandir directory)
+ (("." ".." basename)
+ (string-append directory "/" basename))))

- (define modules
- (let ((modules (map lookup '#$modules)))
- (append modules
- (recursive-module-dependencies modules
- #:lookup-module lookup))))
-
- (mkdir #$output)
- (for-each (lambda (module)
- (format #t "copying '~a'...~%" module)
- (copy-file module
- (string-append #$output "/"
- (basename module))))
- (delete-duplicates modules)))))
+ (define linux-release-module-directory
+ (find-only-entry module-dir))

+ (define modules
+ (module-aliases->module-file-names #$linux '#$aliases))
+
+ (define version
+ (basename linux-release-module-directory))
+
+ (define (install-module-files module-files output)
+ "Install MODULE-FILES to OUTPUT.
+Precondition: OUTPUT is an empty directory except for \"modules.builtin\"."
+ (let ((aliases
+ (map (lambda (module-file-name)
+ (format #t "copying '~a'...~%" module-file-name)
+ (copy-file module-file-name
+ (string-append output "/"
+ (basename module-file-name)))
+ `(,(file-name->module-name module-file-name) .
+ ,(module-aliases module-file-name)))
+ (sort module-files string<))))
+ (install-file (string-append linux-release-module-directory
+ "/modules.builtin")
+ output)
+ (write-module-alias-database aliases output)
+ (write-module-device-database aliases output)))
+
+ (let ((output (string-append #$output "/lib/modules/" version)))
+ (mkdir-p output)
+ (install-module-files (delete-duplicates modules) output)
+ #t))))
(computed-file "linux-modules" build-exp))

(define* (raw-initrd file-systems
D
Danny Milosavljevic wrote on Sun Mar 04 02:54:54+0100 2018
Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180304025454.791a2b87@scratchpost.org
Ah, your attached patch is fine, just the proposed usage

(for-each (catch-ENOENT load-linux-module*)
^^^^^^^^^^^^^^^^^^
(needed-modules
(known-module-aliases (string-append linux-module-directory
"/modules.alias"))))

is probably not working.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqbUe8ACgkQ5xo1VCww
uqXHyAf/ZSamYfxnytp+YzqCZ3ZmoZ/6vD+5H176T5CuMtS7ZtfrkFe4d/lvjB3R
QfAd5rBRzuJ5jO0Or75iGgUleXGO191Rqt/r/AoSeOCur+T3VUAOSrv689F9l/bU
bAOdnOJqNarX7oBrrqmySg0MINUdjaCpWW7B+HSCTuG81fSJl2vdJnUZvPjs25Zn
A+wLozvtzps+wOOrbiQNFspCvwTecMccsUbG+s022c/Ssr/hHxwE43SdUzApR1YZ
F3ugEM1G60KgfXgjTFS9DOKIMLVbfvTZy7J+cbbc9GTrbPhoLFEieUQj+MyjTMb6
oCT+/VZANgBc4YzfoMHEi3Pp1uGg6Q==
=54nz
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Sun Mar 04 13:34:44+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180304133444.4edceecd@scratchpost.org
Hi Ludo,

I've been thinking about it some more - I think it can be made to work "your" way.

I'm trying to integrate it right now.

But! The module-aliases->module-file-names way I've been pursuing (which also
returns the recursive dependencies) has the advantage of using exactly the same
procedure for both flat-linux-module-directory (which copies files) and for
our modprobe (which loads them later).

This means that it's very likely these two will match.

But if load-linux-module* recursively loads modules on its own it's easily
possible that these will diverge - what flat-linux-module-directory is doing
could (accidentially) differ in the future.

Maybe that's overly defensive programming. What do you think?

My flat-linux-module-directory basically just returns a subset of the directory
tree it gets as parameter (and flattens it for no reason).

Also, I've been trying to use find-files for /sys before, like you do.
It doesn't work correctly - probably because of the mutation that modprobe does
(find-files sorts - not sure whether that's the whole story).

Integration tests just finished - even the basic system test fails because of it
now.

I tried only this:

(define (load-kernel-modules)
"Examine /sys/devices to find out which modules to load and load them."
- (define enter?
- (const #t))
- (define (down! directory stat result)
- ;; Note: modprobe mutates the tree starting with DIRECTORY.
- (let ((modalias-name (string-append directory "/modalias")))
- (if (file-exists? modalias-name)
- (let ((modalias
- (string-trim-right (call-with-input-file modalias-name
- read-string)
- #\newline)))
- (system* "/sbin/modprobe" "-q" "--" modalias))))
- #t)
- (define up
- (const #t))
- (define skip
- (const #t))
- (define leaf
- (const #t))
- (define (error name stat errno result)
- (format (current-error-port) "warning: ~a: ~a~%"
- name (strerror errno))
- result)
- (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+ (for-each (lambda (modalias)
+ (system* "/sbin/modprobe" "-q" "--" modalias))
+ (system-device-aliases)))

Doesn't work anymore...

ERROR: In procedure network-interface-flags:
In procedure network-interface-flags: No such device

(I've exported system-device-aliases)
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqb5+QACgkQ5xo1VCww
uqXJ9wf/UFzvHkvHtXvKpynbANSL48kLwGHYSKetR6r353nTtySJZV0kVrRf6aEk
eLZnfbjOFX1Y7cQYKPgiC4/lg5JbqxWkWNR0R1zNQvUgXByL8hxfkuZYtwWE7z5G
Buw/dZBJNdRcG8ZlaOw6r88bhD3CtnceI+XhcqH6sOGud2Bef3cn8HO2gLeA5vi9
DOmeXM1wlP45eYxZ+FNKRJ7WXQwsnGo3WTwowpttZQdA0p80BKmjy+7Jv2DOW3D9
z8lfZ/yHeOQkW1PE7vi0TsjTKy1ryFSBfI37MMcOfmFgenkMKG4K/+c9rMUPltGi
0rBPw7P1HKqxQAXWYaISS3MpQRrvpQ==
=uxOr
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Sun Mar 04 13:45:04+0100 2018
Re: [bug#30604] [PATCH v8 2/7] linux-modules: Add install-modules.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180304134504.0a15b26d@scratchpost.org
Hi Ludo,

> (write-module-alias-database modules port) ;for “modules.alias”
> (write-module-device-database modules port) ;for “modules.devname”

I'd prefer "install-" because their interface is somewhat like install-file
(caller provides output directory and callee adds "modules."* automatically).

But I used the names "write-module-"* in the patchset v9.

> I’m not sure we need ‘install-module-files’ itself. Perhaps we can
> inline it at the call site?

Sure. Did so now.

> For the devname code, please avoid ‘set!’.

I found an even better way to avoid both.

> The indentation of ‘match’ forms is wrong. Would it be OK for you to
> pass it through ./etc/indent-code.el? (It’s non interactive, you don’t
> need to actually use Emacs.)

I'm actually trying to switch to emacs but when I press Tab it messes up
the stuff more. Is there some kind of configuration for guile indentation?

Also, every time I open gnu/build/linux-modules.scm I get

"The local variables list .. may not be safe

* bug-reference-bug-repexp
* eval: (modify-syntax-entry 126 "'")
* eval: (modify-syntax-entry 36 "'")
* eval: (modify-syntax-entry 43 "'")"

???
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqb6lAACgkQ5xo1VCww
uqVseAf/VW2PLCgcY8nImiTi4YlI/BODnYJVCkVIzmA5x/hS3+5OxKXh1LnUjECD
p50Bhz9u8i6v3P6PFKYa7Om3aowsyKOH9E2nochuk8tGKkbBWTPA6ou2TZcJBE2G
SjswuTE7nUBBIjg8vEEzZ9ufF536jDzrzzsyFaVTHlfCHr/eTJ29fg9fR0wMPGgB
PVHZKSI5J0zrZycmSlO8+I1m/NQGzvUMDE2V01DGGIhbLQhcOSy/YL8hCYwbBF+G
lMznbEgw+ZjvoZCeRJfooZeuRU/myeAbJ1mAVUlTeDLLdZzFKOTqTyRR/e7B9jUT
hCsEiOUOX4Pino5+64dzLsfv16jBcg==
=wMFl
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Sun Mar 04 13:53:27+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180304135327.0a103e57@scratchpost.org
Hi Ludo,

On Sat, 03 Mar 2018 23:07:57 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> We could reuse modules.alias and modules.devname from the ‘linux-libre’
> package (right?), but I guess it doesn’t hurt to generate custom ones.

Yes, but it would be somewhat dirty. Here's why:

Linux-libre contains a superset of modules and also their modules.alias
and modules.devname contain a superset of modules compared to the initrd.

That means

(1) modules.alias would contain aliases which map to modules which
don't exist. Since we don't for-each it, who cares.

(2) modules.devname would contain devnames for modules which don't
exist. That means the boot code would create files in /dev which,
when someone accesses them, would try to load modules which don't
exist - and then the access would fail.

Technically both are not bad, but (2) is bad "UX" practise nowadays - and
not done in the Linux world any more since udev exists.

I can still remember the days of 8540 entries static /dev filesystem (as in,
really on the hard disk), 7000 of which didn't work (or even valid for this
system type :P).

If something goes wrong, good luck finding the needle in the haystack
(which /dev file you can use to boot).
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqb7EcACgkQ5xo1VCww
uqXPwAf+P1CjrHIU4Ewpz6ucWfrmOOxqL3oR5zvmf1xtQGitXspjCtfP7OQhEok4
NS43aa4Rcurf32znHZvciYwiTIvkziXGJS3aMh7RbbJc2M8VG13gYTLYr9+YRs28
GSGLIFlHKAjozMeyOHOv9zzpMVgTH0zVE0B1dlV+0ttGUxSXu00v9QX4aoWTh97p
p7HNT/yHvdhfYFUJDjiEk63eSvSkLyvWMxqChGU2iTOv3EDCRFREzKMYWHGBfjpu
5wQRUW5uivoX116X0iA2ipwJ6mb8C31yqInEVAFxt6GFzP5+AmwIrdVrGfG4N/we
NkQd9yJol2LXAPV4qI8pfU6h7aFkPQ==
=P/RW
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Fri Mar 09 23:06:50+0100 2018
Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87muzgykcl.fsf@gnu.org
Hello!

I’ve reworked this patch series a bit, but now I’m stuck on this:

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> ERROR: In procedure network-interface-flags:
> In procedure network-interface-flags: No such device

When than happens, our modprobe gets called with the “netdev-eth0”
alias, which it cannot handle, leading to this error.

By explicitly loading “virtio_net” and sleeping a bit, we get past that
point, but fail later on:

--8<---------------cut here---------------start------------->8---
loading '/gnu/store/n2zvdxp25kvg7wy5xr1y5n9r4502fw80-linux-vm-loader'...
environment variable `PATH' set to `/gnu/store/vxl918zb9brnrgaipzsykpy3mzcafvcr-qemu-minimal-2.11.1/bin:/gnu/store/lvkaf3xlvy57bvap51xlzb209ilbkgcv-parted-3.2/sbin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/bin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/sbin:/gnu/store/0sq2nflm42x0znkv44add0gk82khkcb6-dosfstools-4.1/sbin:/gnu/store/0hl513mnpkhszm2hjai2w9cxmpxs0vgq-sed-4.4/bin:/gnu/store/1h44pkgdd7n6s3i2vjh54awsvfmc219j-grep-3.1/bin:/gnu/store/kgzvfby2ggi1xawsh5vjh4s93qk2dp9k-coreutils-8.28/bin:/gnu/store/k7r2m2wgj8x8jjhccwjsiimp0dlzxb7i-findutils-4.6.0/bin:/gnu/store/ahxc89r6npzf2bbl8yg5vdjicskzzjf3-gawk-4.1.4/bin'
creating partition table with 2 partitions (20.0 MiB, 40.0 MiB)...
Warning: The resulting partition is not properly aligned for best performance.
creating ext4 partition...
mke2fs 1.43.6 (29-Aug-2017)
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/vda1 is mounted.
Creating filesystem with 20480 1k blocks and 5136 inodes
Filesystem UUID: 81a538e9-3efe-406b-bf67-fa153fa3c6a6
Superblock backups stored on blocks:
8193

Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

[ 3.190801] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
populating...
clearing file timestamps...
creating FAT partition...
mkfs.fat 4.1 (2017-01-24)
[ 3.235493] FAT-fs (vda2): IO charset iso8859-1 not found
ERROR: In procedure mount:
In procedure mount: Invalid argument
--8<---------------cut here---------------end--------------->8---

Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t even
invoked, although my understanding is that it should (per ‘load_nls’ in
nls_base.c in the kernel, called from vfat/inode.c).

Similarly, it seems that virtio_blk has to be loaded explicitly: is not
reported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nor
do we get a modprobe query for it.

All in all on-demand loading seems more complicated that it seemed.

What are we missing here?

Ludo’.
D
Danny Milosavljevic wrote on Fri Mar 09 23:13:20+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180309231320.20a02822@scratchpost.org
Hi Ludo,

On Fri, 09 Mar 2018 23:06:50 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> When than happens, our modprobe gets called with the “netdev-eth0”
> alias, which it cannot handle, leading to this error.
>
> By explicitly loading “virtio_net” and


> sleeping a bit,

Uhhhhh no please don't.

> point, but fail later on:
...
> --8<---------------cut here---------------end--------------->8---
>
> Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t even
> invoked, although my understanding is that it should (per ‘load_nls’ in
> nls_base.c in the kernel, called from vfat/inode.c).
>
> Similarly, it seems that virtio_blk has to be loaded explicitly: is not
> reported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nor
> do we get a modprobe query for it.
>
> All in all on-demand loading seems more complicated that it seemed.
>
> What are we missing here?

As I said, modprobe mutates /sys - you cannot use find-files. I haven't used
ftw just to be contrarian. There's even a warning comment :-)

In order to find that out, try to print how /sys looked before modprobe - then
in the early guile recovery REPL print how /sys looks again - the alias it
was juuust complaining about will be there just fine.

My patch series works fine lazy-loading the IO charset, virtio_blk, virtio_net.

It took about 30 h to work out the correct minimal combination - just saying :-)
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqjBwAACgkQ5xo1VCww
uqV8/Qf+J555Zoy+CZAq38ezF4hHwS5YbLy//BfK3/3lu2RwSu4h53vmvUqcFZrl
vk+Y1Q7b7SJUmzUKCj/doQ+VgjJgpWlBia0PM0H0kNdKYrz4qbCH3KMDmyKKxlUS
m0uMD7xiSef1evuBRttunzQIwv6zCUJn8kqyalkFQZ01lDXv7Mex0vpG/QkTTVJZ
aTregruimbDWlMnNfUfN9aq5tKlvZfBrzysXGVpZDZlbZrUBkGkd4R8Skq4WdAwv
OQcLkclyTwJOzEO7+ZCkw1b2uzKilS0K6kdoSAylcaXkc9wqjaV1feE9HrQkUHb3
OTPBJQScuFZ7VApDRwn6xCL/DfTvhg==
=uf4c
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Fri Mar 09 23:19:42+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180309231942.023f6655@scratchpost.org
oops,

On Fri, 9 Mar 2018 23:13:20 +0100
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> As I said, modprobe mutates /sys - you cannot use find-files.

I guess technically it's the Linux kernel module that mutates it.

>I haven't used
> ftw just to be contrarian. There's even a warning comment :-)
>
> In order to find that out, try to print how /sys looked before modprobe - then
> in the early guile recovery REPL print how /sys looks again - the alias it
> was juuust complaining about will be there just fine.
>
> My patch series works fine lazy-loading the IO charset, virtio_blk, virtio_net.

(v9)
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqjCH4ACgkQ5xo1VCww
uqWLvwgAp6KezsUDCUjxB5czeSEyKuSBRy6gciJgiTDnedaerVSA4+yHafKUUKe1
1JY9TK5I6qJbSnkaBXFK1ajrYEcz69jODXPRbv7NS0NtK+PrJBpIeXXqpVfYhsll
KO2CKDWrZ9A7CXLURtSsRrf8vzhkuccoJs8xQnqqoUPYt6FTRYFm9G6XukWY5Wop
5nXX2it0JYFN2M47Eb99uMy3khSanLUY7iC6Ib9vMxBURVURXXFkteb4FRbYC24c
6P2SxkabRhOkc1SrWOXLaUvLl5FKozFO82elcohJCol4s/osEodG9vS8KNTFD0Ej
XkVhv/a5u7/+i/iqoyuXj4KEU9054A==
=ixMQ
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Fri Mar 09 23:26:50+0100 2018
Re: [PATCH v9 7/7] linux-initrd: Use module-aliases->module-file-names, too.
(address . 30604@debbugs.gnu.org)
20180309232650.0f37fb89@scratchpost.org
On Sun, 4 Mar 2018 02:09:14 +0100
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> --- a/gnu/system/linux-initrd.scm
> +++ b/gnu/system/linux-initrd.scm
> @@ -161,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
> #:references-graphs `(("init-closure" ,init)
> ("modprobe-closure" ,modprobe))))
>
> -(define (flat-linux-module-directory linux modules)
> - "Return a flat directory containing the Linux kernel modules listed in
> -MODULES and taken from LINUX."
> +(define (flat-linux-module-directory linux aliases)
> + "Return a flat directory containing the Linux kernel modules resolved by
> +ALIASES and taken from LINUX."
...
> + (define (find-only-entry directory)
> + (match (scandir directory)
> + (("." ".." basename)
> + (string-append directory "/" basename))))

Probably unsafe since the order is not guaranteed.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqjCioACgkQ5xo1VCww
uqXOZQgAkZgLnU86tFzMFoHikvaAZ41DEtYdjK+787F90a0+RSvTIMDMpFXvL40K
LjRBRv3k7Im8o/gw4v0h0Jx9DhfwKmL7zXOOFkaK4urXx80Gfp49kozLeAUJWutP
LRt3kn4hqw8PQzT/nDW46jH9GnNUyCAk2sANoQQWOdk+LpBc1EHs4Z571y6rQDb8
i6x9Ui3CNvDHYXjWsRDRS1+EIZaKTNpeTbD9/iyZOLltD6CXR37/6vgPyKP/kqE9
RHtHdwyrl6MVfPuomee2iiqC5z1HT+EXWjevXXVkIvP1azIGwM2mWxpJLynDO5Ld
0KYbk0UMDZ3/HFCSbcoODoMwvFeNEA==
=PIV0
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Fri Mar 09 23:36:44+0100 2018
Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180309233644.7dc53f8e@scratchpost.org
> Similarly, it seems that virtio_blk has to be loaded explicitly: is not
> reported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nor
> do we get a modprobe query for it.

qemu seems to have a bug that it doesn't assign pci ids properly.

Patch v9 6/7 works around it by fixing the virtio-blk-pci address.

Nicer fixes welcome.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqjDHwACgkQ5xo1VCww
uqXulgf+OpolkwmELThupRx/W0t2WQJeFm+Ho+G6OdSRoHPG7pIg+Tcb24jtRVTD
87fYf2IRRJe2Vx+O0ASGZd1NfDjyVX+fXnhnvfPRFg9SB2zGRkoLSMuACEelx3Ir
v8qpPnm+CmqUlTBldzTwJ/TSNWHVGB1SH3W/tgGGK0yljFvHIfnPYssrgMT84tDo
+G1yVVtSxSU+5Y0DxkCjTeo4ZYkJ5fn/0lsL96iUSb4vJNlUqDJUo3ALJUa2aa5Z
dmw1zzum71yBzSH2QjPyqqEoxgo/apAA/rqZt1J0o0Yayd+ZETBtDjg0PdSxXl0n
d1cB0c6ejWpFsSxhcDRnjOmXC2Q5ig==
=4rsy
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Fri Mar 09 23:44:34+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180309234434.2bf5e3ff@scratchpost.org
Hi Ludo,

sorry for the many e-mails - I'm recollecting all the workarounds :)

>--8<---------------cut here---------------start------------->8---
>loading '/gnu/store/n2zvdxp25kvg7wy5xr1y5n9r4502fw80-linux-vm-loader'...
>environment variable `PATH' set to `/gnu/store/vxl918zb9brnrgaipzsykpy3mzcafvcr-qemu-minimal-2.11.1/bin:/gnu/store/lvkaf3xlvy57bvap51xlzb209ilbkgcv-parted-3.2/sbin:/>gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/bin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/sbin:/gnu/>store/0sq2nflm42x0znkv44add0gk82khkcb6-dosfstools-4.1/sbin:/gnu/store/0hl513mnpkhszm2hjai2w9cxmpxs0vgq-sed-4.4/bin:/gnu/store/1h44pkgdd7n6s3i2vjh54awsvfmc219j->grep-3.1/bin:/gnu/store/kgzvfby2ggi1xawsh5vjh4s93qk2dp9k-coreutils-8.28/bin:/gnu/store/k7r2m2wgj8x8jjhccwjsiimp0dlzxb7i-findutils-4.6.0/bin:/gnu/store/>ahxc89r6npzf2bbl8yg5vdjicskzzjf3-gawk-4.1.4/bin'
>creating partition table with 2 partitions (20.0 MiB, 40.0 MiB)...
>Warning: The resulting partition is not properly aligned for best performance.
>creating ext4 partition...
...
[ 3.235493] FAT-fs (vda2): IO charset iso8859-1 not found
ERROR: In procedure mount:
In procedure mount: Invalid argument
--8<---------------cut here---------------end--------------->8---

>Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t even
>invoked, although my understanding is that it should (per ‘load_nls’ in
>nls_base.c in the kernel, called from vfat/inode.c).

Yeah, that was an evil problem. There are two different ways linux-boot
can boot:
* Either it loads a full system in which case /run/booted-system will
contain kernel module files which can and will be loaded
* Or it loads a "--load" file only - in which case there's no system
modprobe, but switch-root just switched our pure-Guile modprobe away,
so now we are out of modprobes :)
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqjDlIACgkQ5xo1VCww
uqWZ7Qf+N2/hBERWip1Lvyq6ZzPc425T5v1Jrkahu3TEoHiSJd6IxXG4A/OzqR0h
CehvMgm1sIznAECajIe/4X9UFTi42yOWJZB5aauCVn11kkllTbiEJbbtkhNnScQZ
FNCdyFF8Jese+aUXTpWxY0TucI6xvLwgvU6UYrRHOgND5Ug+G5ilWYTca8sFHena
InbXiYP4GNOGKcPltMn/i05TMOHcnHHUg67jta8VnqEROxHFWwukWtujGxrPu548
Rgcr/o5Lu9EjKGn1zFi+WR4W1Jh2sExFh9hvdHHeox3gAfq+tivXd7vZBYnQNJ88
6v6YlrVDX9OZ7H6eCrmZmxaK1yc9Iw==
=sXjo
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Sun Mar 11 21:24:30+0100 2018
Re: [PATCH v9 3/7] linux-initrd: Provide pure-Guile modprobe.
(address . 30604@debbugs.gnu.org)
20180311212430.707ef0ea@scratchpost.org
> + (define option-spec
> + '((quiet (single-char #\q) (value #f))))

Would make sense to also support "-a" (used when invoking modprobe to load multiple
kernel modules at once).
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqlkH4ACgkQ5xo1VCww
uqWFsggAiePy7PVn0EvxcvzawT0U/UuWoHTaiMJZDwj5nSBhWHpNLnMGfYe2emd4
q10BZdapDN4FI+4iwRdene+8NbH2y4rsUpM1VMKG12Iq+hNE2v/D1hiBMFgozX1b
jPbvb1NIZXZDT+iHwo2no1fvylSySl4SjE+b68RQpeieKyodums5B9Dg1H7aavcU
B/qo0pkotTl11ChmZ0wAgj3HyZfGndP1b6BBTsUvz1fKFvcYTaH0tryh9oK2Ug1h
lpRdSRtugfCb2BQ9ajzC937gXUTzZ8gNMF8nQm76HPe/WeB+DkXUhMPic1pndz8h
JHTXjkmP8NUywJjnGkmi0N0sruquoA==
=hAM5
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Mon Mar 12 13:38:51+0100 2018
Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87ina1qxic.fsf@gnu.org
Heya Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> As I said, modprobe mutates /sys - you cannot use find-files. I haven't used
> ftw just to be contrarian. There's even a warning comment :-)

I did see this comment, and I don’t doubt you’re acting in good faith!

> In order to find that out, try to print how /sys looked before modprobe - then
> in the early guile recovery REPL print how /sys looks again - the alias it
> was juuust complaining about will be there just fine.

It wasn’t clear to me what “modprobe mutates /sys” meant, and I guess I
didn’t think it could lead to not seeing relevant “modalias” files. Now
I’ve experienced first-hand that it does matter. :-)

> It took about 30 h to work out the correct minimal combination - just saying :-)

Yeah I can imagine (and it would have taken me much longer I guess.)

While reviewing this series I realized I didn’t understand everything,
as in the example above, some things seemed unnecessary, and I thought
we could improve the style of a few things.

The patches that follow build upon your work with a few changes:

• The “modules.alias” writer style is made more idiomatic and
hopefully simpler.

• The “modules.devname” writer is actually unnecessary. I changed it
to a more functional style and with smaller functions. It’s the
last patch of this series; it’s completely optional since we don’t
use it currently, but we might want to keep it around for later.

• The pure-Scheme modprobe uses SRFI-37 instead of (ice-9
getopt-long), as in the rest of the code base.

• /proc/sys/kernel/modprobe is set in ‘boot-system’ directly, so that
we don’t have to special-case the “modprobe” closure in
‘build-initrd’.

• ‘module-aliases->module-file-names’ is gone. Instead there’s
‘load-needed-linux-modules’, similar to the ‘load-kernel-modules’
you had, but a bit simpler.

• I added comments here and there, especially about the
virtio-blk-pci, which was far from obvious. :-)

I didn’t add the ‘needed-modules’ and ‘system-device-aliases’ procedures
I posted earlier because we don’t need them currently. If you think
they could be useful, we can always add them.

The following command succeeds:

make check-system TESTS="basic installed-os iso-image-installer"

WDYT?


I realize that makes for a not-so-efficient review process, though I
think the result is worth it. I’m open to suggestions though.

Overall, these changes are mostly about (1) making self-contained
commits, (2) commenting code, and (3) coding style. About #3, I think
the guidelines I applied were:

1. Avoid imperative style. When we do need side effects, try hard to
move them separately.

2. Keep functions short.

3. Decompose functions in a way that allows them to be combined
“nicely”.

Some of that is a bit subjective, but overall we should be able to
converge.

Thanks for the great work, and sorry for the back-and-forth and delays!

Ludo’.
L
Ludovic Courtès wrote on Mon Mar 12 13:39:13+0100 2018
[PATCH v10 1/6] linux-modules: Add "modules.alias" writer.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180312123918.22645-1-ludo@gnu.org
From: Danny Milosavljevic <dannym@scratchpost.org>

* gnu/build/linux-modules.scm (write-module-alias-database): New
procedure.
(%not-dash): New variable.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
---
gnu/build/linux-modules.scm | 26 +++++++++++++++++++++++++-
1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4fe673cca..6101b8dc3 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2014, 2016, 2018 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
+;;; Copyright © 2018 Danny Milosavljevic <dannym@scratchpost.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -21,6 +22,7 @@
#:use-module (guix elf)
#:use-module (guix glob)
#:use-module (guix build syscalls)
+ #:use-module ((guix build utils) #:select (find-files))
#:use-module (rnrs io ports)
#:use-module (rnrs bytevectors)
#:use-module (srfi srfi-1)
@@ -41,7 +43,8 @@

device-module-aliases
known-module-aliases
- matching-modules))
+ matching-modules
+ write-module-alias-database))

;;; Commentary:
;;;
@@ -380,4 +383,25 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
module)))
known-aliases))

+(define %not-dash
+ (char-set-complement (char-set #\-)))
+
+(define (write-module-alias-database directory)
+ "Traverse the '.ko' files in DIRECTORY and create the corresponding
+'modules.alias' file."
+ (define aliases
+ (map (lambda (file)
+ (cons (file-name->module-name file) (module-aliases file)))
+ (find-files directory "\\.ko$")))
+
+ (call-with-output-file (string-append directory "/modules.alias")
+ (lambda (port)
+ (display "# Aliases extracted from modules themselves.\n" port)
+ (for-each (match-lambda
+ ((module . aliases)
+ (for-each (lambda (alias)
+ (format port "alias ~a ~a\n" alias module))
+ aliases)))
+ aliases))))
+
;;; linux-modules.scm ends here
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 13:39:14+0100 2018
[PATCH v10 2/6] linux-modules: Add 'load-linux-modules-from-directory'.
(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180312123918.22645-2-ludo@gnu.org
* gnu/build/linux-modules.scm (load-linux-modules-from-directory): New
procedure.
* gnu/build/linux-boot.scm (boot-system)[lookup-module]: Remove.
Use 'load-linux-modules-from-directory' instead.
---
gnu/build/linux-boot.scm | 9 ++-------
gnu/build/linux-modules.scm | 12 ++++++++++++
2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..df0b2b2d1 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,10 +469,6 @@ upon error."
mounts)
"ext4"))

- (define (lookup-module name)
- (string-append linux-module-directory "/"
- (ensure-dot-ko name)))
-
(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")

@@ -487,9 +483,8 @@ upon error."
(start-repl))

(display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (load-linux-modules-from-directory linux-modules
+ linux-module-directory)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 6101b8dc3..1337b5e88 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -38,6 +38,7 @@
modules-loaded
module-loaded?
load-linux-module*
+ load-linux-modules-from-directory

current-module-debugging-port

@@ -232,6 +233,17 @@ appears in BLACK-LIST are not loaded."
(or (and recursive? (= EEXIST (system-error-errno args)))
(apply throw args)))))))

+(define (load-linux-modules-from-directory modules directory)
+ "Load MODULES and their dependencies from DIRECTORY, a directory containing
+the '.ko' files. The '.ko' suffix is automatically added to MODULES if
+needed."
+ (define (lookup-module name)
+ (string-append directory "/" (ensure-dot-ko name)))
+
+ (for-each (cut load-linux-module* <>
+ #:lookup-module lookup-module)
+ (map lookup-module modules)))
+

;;;
;;; Device modules.
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 13:39:15+0100 2018
[PATCH v10 3/6] linux-modules: Add 'load-needed-linux-modules'.
(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180312123918.22645-3-ludo@gnu.org
Based on a patch by Danny Milosavljevic <dannym@scratchpost.org>.

* gnu/build/linux-modules.scm (load-needed-linux-modules): New
procedure.
---
gnu/build/linux-modules.scm | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 1337b5e88..9aabe6d37 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
#:use-module (ice-9 vlist)
#:use-module (ice-9 match)
#:use-module (ice-9 rdelim)
+ #:use-module (ice-9 ftw)
#:export (dot-ko
ensure-dot-ko
module-aliases
@@ -45,7 +46,8 @@
device-module-aliases
known-module-aliases
matching-modules
- write-module-alias-database))
+ write-module-alias-database
+ load-needed-linux-modules))

;;; Commentary:
;;;
@@ -416,4 +418,34 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
aliases)))
aliases))))

+(define (load-needed-linux-modules module-directory)
+ "Examine /sys/devices to find out which modules are needed and load those we
+have in MODULE-DIRECTORY. Return the list of modules loaded, not including
+dependencies.
+
+Note: loading modules leads to the creation of new entries in /sys/devices,
+which is why we need traversal and loading to be interleaved. If we walked
+/sys/devices and *then* loaded modules, we'd miss the entries added as a
+side-effect and would thus need to traverse /sys/devices again."
+ (define aliases
+ (known-module-aliases
+ (string-append module-directory "/modules.alias")))
+
+ (define (enter? director stat result) result)
+ (define (down directory stat result) result)
+ (define (up directory stat result) result)
+ (define (skip entry stat result) result)
+ (define (error name stat errno result) result)
+ (define (leaf file stat result)
+ (if (string=? (basename file) "modalias")
+ (let* ((alias (string-trim-right
+ (call-with-input-file file get-string-all)))
+ (modules (matching-modules alias aliases)))
+ (load-linux-modules-from-directory modules
+ module-directory)
+ (append modules result))
+ result))
+
+ (file-system-fold enter? leaf down up skip error #t "/sys/devices"))
+
;;; linux-modules.scm ends here
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 13:39:16+0100 2018
[PATCH v10 4/6] vm: Make the virtio-blk uniquely identifiable in /sys.
(address . 30604@debbugs.gnu.org)
20180312123918.22645-4-ludo@gnu.org
From: Danny Milosavljevic <dannym@scratchpost.org>

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.

Signed-off-by: Ludovic Courtès <ludo@gnu.org>
---
gnu/build/vm.scm | 6 +++++-
gnu/system/vm.scm | 7 ++++++-
2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..773b738ae 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,11 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+
+ ;; QEMU seems to have a bug that it doesn't assign PCI
+ ;; IDs properly, so force the address of our virtio-blk
+ ;; device.
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 91ff32ce9..4360adf15 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -677,7 +677,12 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+
+ ;; QEMU seems to have a bug that it doesn't assign PCI IDs properly, so
+ ;; force the address of our virtio-blk device.
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 13:39:18+0100 2018
[PATCH v10 6/6] linux-modules: Add "modules.devname" writer.
(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180312123918.22645-6-ludo@gnu.org
* gnu/build/linux-modules.scm (aliases->device-tuple)
(write-module-device-database): New procedures.

Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>.
---
gnu/build/linux-modules.scm | 48 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 48 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 9aabe6d37..2ed04e5b4 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -47,6 +47,8 @@
known-module-aliases
matching-modules
write-module-alias-database
+ write-module-device-database
+
load-needed-linux-modules))

;;; Commentary:
@@ -418,6 +420,52 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
aliases)))
aliases))))

+(define (aliases->device-tuple aliases)
+ "Traverse ALIASES, a list of module aliases, and search for
+\"char-major-M-N\", \"block-major-M-N\", or \"devname:\" aliases. When they
+are found, return a tuple (DEVNAME TYPE MAJOR MINOR), otherwise return #f."
+ (define (char/block-major? alias)
+ (or (string-prefix? "char-major-" alias)
+ (string-prefix? "block-major-" alias)))
+
+ (define (char/block-major->tuple alias)
+ (match (string-tokenize alias %not-dash)
+ ((type "major" (= string->number major) (= string->number minor))
+ (list (match type
+ ("char" "c")
+ ("block" "b"))
+ major minor))))
+
+ (let* ((devname (any (lambda (alias)
+ (and (string-prefix? "devname:" alias)
+ (string-drop alias 8)))
+ aliases))
+ (major/minor (match (find char/block-major? aliases)
+ (#f #f)
+ (str (char/block-major->tuple str)))))
+ (and devname major/minor
+ (cons devname major/minor))))
+
+(define (write-module-device-database directory)
+ "Traverse the '.ko' files in DIRECTORY and create the corresponding
+'modules.devname' file. This file contains information about modules that can
+be loaded on-demand, such as file system modules."
+ (define aliases
+ (filter-map (lambda (file)
+ (match (aliases->device-tuple (module-aliases file))
+ (#f #f)
+ (tuple (cons (file-name->module-name file) tuple))))
+ (find-files directory "\\.ko$")))
+
+ (call-with-output-file (string-append "/tmp" "/modules.devname")
+ (lambda (port)
+ (display "# Device nodes to trigger on-demand module loading.\n" port)
+ (for-each (match-lambda
+ ((module devname type major minor)
+ (format port "~a ~a ~a~a:~a~%"
+ module devname type major minor)))
+ aliases))))
+
(define (load-needed-linux-modules module-directory)
"Examine /sys/devices to find out which modules are needed and load those we
have in MODULE-DIRECTORY. Return the list of modules loaded, not including
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 13:39:17+0100 2018
[PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180312123918.22645-5-ludo@gnu.org
This allows us to load modules on demand when the kernel asks for them.

* gnu/system/linux-initrd.scm (modprobe-program): New variable.
(flat-linux-module-directory): Call 'write-module-alias-database'.
(raw-initrd): Pass #:modprobe to 'boot-system'.
(expression->initrd): Copy "closure" to $out/references.
* gnu/build/linux-boot.scm (boot-system): Add #:modprobe and honor it.
Call 'load-needed-linux-modules'.
* gnu/system/vm.scm (qemu-image): Add #:linux parameter. Define
'modprobe-wrapper' and pass it to 'activate-modprobe'. Pass #:linux to
'expression->derivation-in-linux-vm'.

Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>
---
gnu/build/linux-boot.scm | 13 +++++--
gnu/system/linux-initrd.scm | 86 +++++++++++++++++++++++++++++++++++++++++----
gnu/system/vm.scm | 21 +++++++++++
3 files changed, 111 insertions(+), 9 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index df0b2b2d1..eedc4bb9d 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -435,6 +435,7 @@ bailing out.~%root contents: ~s~%" (scandir "/"))


(define* (boot-system #:key
+ modprobe
(linux-modules '())
linux-module-directory
qemu-guest-networking?
@@ -449,6 +450,9 @@ QEMU-GUEST-NETWORKING? is true, calling PRE-MOUNT, mounting the file systems
specified in MOUNTS, and finally booting into the new root if any. The initrd
supports kernel command-line options '--load', '--root', and '--repl'.

+MODPROBE must be #f or a program to install as the modprobe program that the
+kernel will invoke when it needs to load modules.
+
Mount the root file system, specified by the '--root' command-line argument,
if any.

@@ -482,9 +486,14 @@ upon error."
(when (member "--repl" args)
(start-repl))

+ (when modprobe
+ ;; Tell the kernel to invoke MODPROBE.
+ (call-with-output-file "/proc/sys/kernel/modprobe"
+ (lambda (port)
+ (display modprobe port))))
+
(display "loading kernel modules...\n")
- (load-linux-modules-from-directory linux-modules
- linux-module-directory)
+ (load-needed-linux-modules linux-module-directory)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 1eb5f5130..6b7883c2c 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -63,6 +63,75 @@
;;;
;;; Code:

+(define* (modprobe-program linux-module-directory #:key
+ (guile %guile-static-stripped))
+ "Return a minimal implementation of 'modprobe' for our initrd that looks up
+modules in LINUX-MODULE-DIRECTORY. This program will be invoked by the kernel
+when modules need to be loaded."
+ (define program
+ (with-imported-modules (source-module-closure
+ '((gnu build linux-modules)))
+ #~(begin
+ (use-modules (gnu build linux-modules)
+ (ice-9 match)
+ (srfi srfi-1)
+ (srfi srfi-26)
+ (srfi srfi-37))
+
+ (define option-spec
+ (list (option '(#\q "quiet") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'quiet? #t result)))))
+
+ (define options
+ ;; Alist of options and non-option arguments.
+ (args-fold (cdr (program-arguments))
+ option-spec
+ (lambda (opt name arg result)
+ (error "unrecognized option" name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ '()))
+
+ (define aliases
+ ;; The list of aliases we are asked to load.
+ (filter-map (match-lambda
+ (('argument . alias) alias)
+ (_ #f))
+ options))
+
+ (define linux-module-directory
+ ;; The module directory. Note: We expect a flat directory here.
+ #$linux-module-directory)
+
+ (define %known-aliases
+ ;; The alias database.
+ (known-module-aliases
+ (string-append linux-module-directory "/modules.alias")))
+
+ (define (device-aliases->module-names aliases)
+ ;; Return the list of module names for the subset of ALIASES that
+ ;; appears in %KNOWN-ALIASES.
+ (append-map (cut matching-modules <> %known-aliases)
+ aliases))
+
+ (when (assq-ref options 'quiet?)
+ (current-error-port (%make-void-port "w"))
+ (current-output-port (%make-void-port "w")))
+
+ (let ((modules (device-aliases->module-names aliases)))
+ (call-with-output-file "/dev/kmsg"
+ (lambda (port)
+ (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%"
+ (getpid) aliases modules (program-arguments))))
+
+ (when (< (length modules) (length aliases))
+ (error "alias resolution failed" aliases))
+
+ (load-linux-modules-from-directory (reverse modules)
+ linux-module-directory)))))
+
+ (program-file "modprobe" program #:guile guile))

(define* (expression->initrd exp
#:key
@@ -89,16 +158,16 @@ the derivations referenced by EXP are automatically copied to the initrd."
(mkdir #$output)

;; The guile used in the initrd must be present in the store, so
- ;; that module loading works once the root is switched.
+ ;; that module loading works once the root is switched. Similarly,
+ ;; the 'modprobe' program installed in /proc/sys/kernel/modprobe by
+ ;; the initrd, if any, must be present after switch root.
;;
;; To ensure that is the case, add an explicit reference to the
;; guile package used in the initrd to the output.
;;
- ;; This fixes guix-patches bug #28399, "Fix mysql activation, and
+ ;; This fixes <https://bugs.gnu.org/28399>, "Fix mysql activation, and
;; add a basic test".
- (call-with-output-file (string-append #$ output "/references")
- (lambda (port)
- (simple-format port "~A\n" #$guile)))
+ (copy-file "closure" (string-append #$output "/references"))

(build-initrd (string-append #$output "/initrd")
#:guile #$guile
@@ -153,7 +222,9 @@ MODULES and taken from LINUX."
(copy-file module
(string-append #$output "/"
(basename module))))
- (delete-duplicates modules)))))
+ (delete-duplicates modules))
+
+ (write-module-alias-database #$output))))

(computed-file "linux-modules" build-exp))

@@ -222,7 +293,8 @@ upon error."
(set-path-environment-variable "PATH" '("bin" "sbin")
'#$helper-packages)))

- (boot-system #:mounts
+ (boot-system #:modprobe #$(modprobe-program kodir)
+ #:mounts
(map spec->file-system
'#$(map file-system->spec file-systems))
#:pre-mount (lambda ()
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4360adf15..ecb7ed506 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -249,6 +249,7 @@ INPUTS is a list of inputs (as for packages)."
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
+ (linux linux-libre)
(qemu qemu-minimal)
(disk-image-size 'guess)
(disk-image-format "qcow2")
@@ -275,18 +276,37 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (define modprobe-wrapper
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ (let ((modprobe (file-append kmod "/bin/modprobe")))
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY"
+ #$(file-append linux "/lib/modules"))
+ (apply execl #$modprobe
+ (cons #$modprobe (cdr (command-line))))))))
+
+
(expression->derivation-in-linux-vm
name
(with-imported-modules (source-module-closure '((gnu build bootloader)
(gnu build vm)
+ (gnu build activation)
(guix build utils)))
#~(begin
(use-modules (gnu build bootloader)
(gnu build vm)
+ ((gnu build activation) #:select (activate-modprobe))
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

+ ;; We may need to lazy-load modules. The initrd installs a
+ ;; 'modprobe' that can only search through the modules available in
+ ;; the initrd, but here we want to be able to use all the modules of
+ ;; LINUX. Thus, install a real 'modprobe'.
+ (activate-modprobe #$modprobe-wrapper)
+
(let ((inputs
'#$(append (list qemu parted e2fsprogs dosfstools)
(map canonical-package
@@ -361,6 +381,7 @@ the image."
#$(bootloader-installer bootloader))
(reboot)))))
#:system system
+ #:linux linux
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 15:45:29+0100 2018
Re: [bug#30604] [PATCH v9 3/7] linux-initrd: Provide pure-Guile modprobe.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
878taxnyie.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> + (define option-spec
>> + '((quiet (single-char #\q) (value #f))))
>
> Would make sense to also support "-a" (used when invoking modprobe to load multiple
> kernel modules at once).

I suppose we can do that later because the kernel doesn’t use it in
‘call_modprobe’ and this thing is only ever invoked by the kernel.

WDYT?

Ludo’.
D
Danny Milosavljevic wrote on Mon Mar 12 18:51:55+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312183840.44e1b6c2@scratchpost.org
Hi Ludo,

On Mon, 12 Mar 2018 15:45:29 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> > Would make sense to also support "-a" (used when invoking modprobe to load multiple
> > kernel modules at once).
>
> I suppose we can do that later because the kernel doesn’t use it in
> ‘call_modprobe’ and this thing is only ever invoked by the kernel.

Sure.

I just wanted to comment because I noticed that kmod's

modprobe a b c

doesn't load three modules a, b, c. But my v9's pure-Guile modprobe was
implemented in a way that it did load three modules a, b, c - which it
definitely shouldn't be doing.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqmvjsACgkQ5xo1VCww
uqXeIwf/bWzHASgwuyDjDHnyky9v07N2HOiSCOKw02bU0cL1fq2XiqTMAOJ1aurg
yrijV9mfr6v27OKikXld/AoTPf4kSJR3DLRosyJeu1CoXHm4CdLnBpFm2Zbf4HSm
c+jBjo3x21qMmIj1O5KDtJQo22fbgTp29/2ndokOOrDLpShdKxCRNiihK9e3gRTM
eZdw10jhypvXZWLgfeWrOvLkxrBeLgR437UdWYXkal6SOZ1EY2jP3d1XqhxCi/BA
gGPAcEQYIc2SNVhf30MfaH6ssih+bcDeLxlBkTJ/U9vpXMJDPiKoC9kgpus7ncLN
Ph7AT+i7N1bp9JEbQfnDrNdGwqctKQ==
=bZwL
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Mon Mar 12 20:26:54+0100 2018
Re: [bug#30604] [PATCH v10 2/6] linux-modules: Add 'load-linux-modules-from-directory'.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312202654.43eaa5b1@scratchpost.org
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqm1H4ACgkQ5xo1VCww
uqUaDQgAoQZlq9m3ie+LFkr6iYXYrja7h9oMesSDWz5Prfd8ou8Y59JZnoGHPkMc
PucsWmIvytg4OgT7xqbJk7rmNFdwyM7EFqj/oizhKnwYBo0kObRfUXfA6jbE2GiG
JdLHMuiIbfJAe2S1B01PEMT2Wdkxb+2qTJJMCQaHR1DPQe6NgnSK2vI4567CYD/U
iU3ugZV6pTIWjoju//TvaiNbGFJiW65f1E3I3X79gIiY2UJJQiWeCDJjmBPAC24P
XzIGzhywgHcL/zItbRnp6LlVl8pjpg4soK+IIEu/Rlu/Pgfhx4RmSsHqE91c0V9h
pdzRqIfDXaqB0PbdF20hY3I1ZMfOHQ==
=8AKq
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Mon Mar 12 20:40:02+0100 2018
Re: [bug#30604] [PATCH v10 3/6] linux-modules: Add 'load-needed-linux-modules'.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312204002.09bfc72f@scratchpost.org
> + (let* ((alias (string-trim-right
> + (call-with-input-file file get-string-all)))

Hmm... doesn't this trim other things than newline?

A quick internet search shows that most people do that. Huh.

LGTM!
D
Danny Milosavljevic wrote on Mon Mar 12 20:54:44+0100 2018
Re: [PATCH v10 1/6] linux-modules: Add "modules.alias" writer.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312205444.07c05f4c@scratchpost.org
> +(define %not-dash
> + (char-set-complement (char-set #\-)))

Is that used?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqm2wQACgkQ5xo1VCww
uqWAEggAoQSbJS4ZokiOPU19tL7sruZCtY+obW6fPybblMPoEGu4lvtpYgGS2LKJ
WM/BqQ47Gl+YsaHN0rtubgE/x6UR1QSgNT4u8YzKgUUndYDCnTE/IZeZxJpInCHm
vX/zZrABiPijn2XScY+CPIZhDSqc2clk0+9SwG0HuZ95YGmsR46fgBk7pxkgtwHi
tF2KGNgAvTY9ras70li10hgd8yhddjYV7hDzlD5V4NPUO0zUHq/bnLMvQUwvL8Vf
DbeqIeZo2LuEmC8/ndD8cOYD9abRswWxwiV5X6W2Mv/sM1ihgP1qFxXZsfXffq7e
IUSeM6PeZH9zXjm7zhULRhbu5jBFlQ==
=5l1S
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Mon Mar 12 21:09:36+0100 2018
Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312210936.7f89a29c@scratchpost.org
Hi Ludo,

On Mon, 12 Mar 2018 13:39:17 +0100
Ludovic Courtès <ludo@gnu.org> wrote:

> + (when modprobe
> + ;; Tell the kernel to invoke MODPROBE.
> + (call-with-output-file "/proc/sys/kernel/modprobe"
> + (lambda (port)
> + (display modprobe port))))
> +
> (display "loading kernel modules...\n")

There's a race because Linux could just start loading modules
immediately before boot-system even starts / executed far enough.

That was why earlier I kept /sbin/modprobe as the name (it's the
default Linux uses) and made sure that the file /sbin/modprobe is already
in the initrd (as opposed to created by boot-system).

The race didn't happen to me yet (and I doubt that Linux has modules
fine-grained enough that that's a problem in practise) - but I would prefer
it not to be possible.

WDYT?

> + (define aliases
> + ;; The list of aliases we are asked to load.
> + (filter-map (match-lambda
> + (('argument . alias) alias)
> + (_ #f))
> + options))

Without "-a", it's not supposed to be a list of aliases. It's supposed to be
one alias and the remainder are parameters supposed to be passed to that
kernel module.

(I didn't know that before).

When Linux invokes /sbin/modprobe, it doesn't pass "-a" but it doesn't
pass any module parameters either - it just passes one module alias.

For clarity and in order that we don't have weird problems later (trying to
load a module named "verbose=1" or something :) ), let's just accept one alias
for now.

>+(define* (modprobe-program linux-module-directory #:key
>+ (guile %guile-static-stripped))
[...]
> + (call-with-output-file "/dev/kmsg"
> + (lambda (port)
> + (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%"
> + (getpid) aliases modules (program-arguments))))

Hmm, logging is nice!

But with /dev/kmsg one write(2) syscall needs to pass the entire message.

What do you think about additionally calling setvbuf with 1024 ? Because:

kernel/printk/printk.c: #define LOG_LINE_MAX (1024 - PREFIX_MAX)

> + (when (< (length modules) (length aliases))
> + (error "alias resolution failed" aliases))

The original modprobe loads as many modules as it can and possibly sets an
(always the same) error flag (and continues loading the other modules).

Then modprobe's exit status is that error flag.

This is moot because we shouldn't accept more than one alias - see above.

> + (write-module-alias-database #$output))))

One of the devnames created statically is the one for btrfs, so not writing or
using devnames is not going to end well.

If it did work without it I'd be even more on guard :)

(I'd also copy the modules.builtin (from Linux).
Also, what happens if we load a module which has as dependency a builtin?
Will we try to load the builtin as a .ko file and fail the entire thing?)

> + ;; We may need to lazy-load modules. The initrd installs a

Nitpick: "kernel modules" (to clarify).
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqm3oAACgkQ5xo1VCww
uqVxHQf/SAsE6KJ/6f6BdakZaXlVHzpjRCqHAsBqgjysduzvcrFV8UCjlL+oHwbx
wSTlb3pX/UlqNI6URpKE2lAvroThzv4nrohwxTlIHCUM/Y1eYanCrT0F6v6Qwg6z
fr+AF1KUyJHeLU7I0vgu+Sl2KWEJwRvQkUCPoF4UnpkQnwreJFatjBa5jkJeRPYP
GRQe2vfBmOQUQetL2JjeqvOJ6FSfKV51A//8HoJFHpVgkktBD1mADHR4yiB+CWuD
9BiPndNOCm8FgF5szqQK+htdfm15kMF9yapqCVib12xZzUEji6zXPSN201q3V0uY
O0fbTWR4qYQL/2nlcZU+Tm/RMPfs5g==
=D7DW
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Mon Mar 12 22:10:39+0100 2018
Re: [bug#30604] [PATCH v10 3/6] linux-modules: Add 'load-needed-linux-modules'.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87r2opknjk.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> + (let* ((alias (string-trim-right
>> + (call-with-input-file file get-string-all)))
>
> Hmm... doesn't this trim other things than newline?

Yes, it trims whitespace.

> A quick internet search shows that most people do that. Huh.

Yeah I’d think that whitespace is non-significant in these files, but in
practice they’re always generated anyway.

Ludo’.
D
Danny Milosavljevic wrote on Mon Mar 12 22:12:26+0100 2018
Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312221226.72eb6cb7@scratchpost.org
> That was why earlier I kept /sbin/modprobe as the name (it's the
> default Linux uses) and made sure that the file /sbin/modprobe is already
> in the initrd (as opposed to created by boot-system).

Note: The simplest change would be to keep this patch as-is but add

;; Make sure that Linux can modprobe even before boot-system started up.
(when modprobe
(mkdir-p "sbin")
(symlink modprobe "sbin/modprobe")
(compile-to-cache "sbin/modprobe")) ; Note: without this line booting is dead slow. [Maybe in this case that wouldn't be so bad - since boot-system switches away immediately anyway :)]

to gnu/build/linux-initrd.scm
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqm7ToACgkQ5xo1VCww
uqXvXgf/SnwfDNtEZL1XSdFOuX/41Dp/eKf3g0roct+Yxfv1fjvliPoGPFfttKv+
Zx+vSHS1R//outrcrcXifpJJaNpsASsol1HxtuoPtzQ7nLXDIZKB4bG9aZYeVW1H
OnWOKL1Si9ozts4KJxfBSyovFyDqIs/J85BrSaYhsCI5qkpH3pmKx03bTuPPOlXH
kI5S5j4977ATl5IXiUdZ0Y2+QOun98k+a5XM4ujlfaID+gu+9PrEp4R9bg6V1nGZ
M3efWovUJeewL+nk4gB1pLx3bo5ISLeoqdq1Bso2HzYkfmZtiI5Ke5+k37zPpOg1
tfBGWBNxzKJLnS40yu6tbFDweZ/1sg==
=5MQy
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Mon Mar 12 22:13:26+0100 2018
Re: [PATCH v10 1/6] linux-modules: Add "modules.alias" writer.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87muzdknex.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> +(define %not-dash
>> + (char-set-complement (char-set #\-)))
>
> Is that used?

Oops, not until patch 6/6. I’ve moved it there.

Ludo’.
L
Ludovic Courtès wrote on Mon Mar 12 23:14:59+0100 2018
Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87h8plkkkc.fsf@gnu.org
Howdy!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Mon, 12 Mar 2018 13:39:17 +0100
> Ludovic Courtès <ludo@gnu.org> wrote:
>
>> + (when modprobe
>> + ;; Tell the kernel to invoke MODPROBE.
>> + (call-with-output-file "/proc/sys/kernel/modprobe"
>> + (lambda (port)
>> + (display modprobe port))))
>> +
>> (display "loading kernel modules...\n")
>
> There's a race because Linux could just start loading modules
> immediately before boot-system even starts / executed far enough.
>
> That was why earlier I kept /sbin/modprobe as the name (it's the
> default Linux uses) and made sure that the file /sbin/modprobe is already
> in the initrd (as opposed to created by boot-system).
>
> The race didn't happen to me yet (and I doubt that Linux has modules
> fine-grained enough that that's a problem in practise) - but I would prefer
> it not to be possible.
>
> WDYT?

You’re right, I wondered what would happen. I’ve restored the
/sbin/modprobe symlink in v11.

>> + (define aliases
>> + ;; The list of aliases we are asked to load.
>> + (filter-map (match-lambda
>> + (('argument . alias) alias)
>> + (_ #f))
>> + options))
>
> Without "-a", it's not supposed to be a list of aliases. It's supposed to be
> one alias and the remainder are parameters supposed to be passed to that
> kernel module.
>
> (I didn't know that before).
>
> When Linux invokes /sbin/modprobe, it doesn't pass "-a" but it doesn't
> pass any module parameters either - it just passes one module alias.
>
> For clarity and in order that we don't have weird problems later (trying to
> load a module named "verbose=1" or something :) ), let's just accept one alias
> for now.

Indeed, done in v11.

>>+(define* (modprobe-program linux-module-directory #:key
>>+ (guile %guile-static-stripped))
> [...]
>> + (call-with-output-file "/dev/kmsg"
>> + (lambda (port)
>> + (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%"
>> + (getpid) aliases modules (program-arguments))))
>
> Hmm, logging is nice!

I didn’t mean to leave it, but after all, it’s pretty useful. :-)

> But with /dev/kmsg one write(2) syscall needs to pass the entire message.
>
> What do you think about additionally calling setvbuf with 1024 ? Because:

Sure.

>> + (write-module-alias-database #$output))))
>
> One of the devnames created statically is the one for btrfs, so not writing or
> using devnames is not going to end well.

We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,
“fs-9p”, etc. alias, which shows up in “modules.alias”. No need for
“modules.devname” AFAICS.

> (I'd also copy the modules.builtin (from Linux).
> Also, what happens if we load a module which has as dependency a builtin?
> Will we try to load the builtin as a .ko file and fail the entire thing?)

The dependency of a builtin is necessarily a builtin, and the kernel
won’t invoke modprobe for a builtin, will it? IOW, I think there’s no
problem here. :-)

>> + ;; We may need to lazy-load modules. The initrd installs a
>
> Nitpick: "kernel modules" (to clarify).

Fixed.

Thanks for the insightful review!

Ludo’.
L
Ludovic Courtès wrote on Mon Mar 12 23:15:36+0100 2018
[PATCH v11 1/6] linux-modules: Add "modules.alias" writer.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180312221541.1886-1-ludo@gnu.org
From: Danny Milosavljevic <dannym@scratchpost.org>

* gnu/build/linux-modules.scm (write-module-alias-database): New
procedure.
(%not-dash): New variable.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
---
gnu/build/linux-modules.scm | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4fe673cca..b444e6f90 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2014, 2016, 2018 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
+;;; Copyright © 2018 Danny Milosavljevic <dannym@scratchpost.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -21,6 +22,7 @@
#:use-module (guix elf)
#:use-module (guix glob)
#:use-module (guix build syscalls)
+ #:use-module ((guix build utils) #:select (find-files))
#:use-module (rnrs io ports)
#:use-module (rnrs bytevectors)
#:use-module (srfi srfi-1)
@@ -41,7 +43,8 @@

device-module-aliases
known-module-aliases
- matching-modules))
+ matching-modules
+ write-module-alias-database))

;;; Commentary:
;;;
@@ -380,4 +383,22 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
module)))
known-aliases))

+(define (write-module-alias-database directory)
+ "Traverse the '.ko' files in DIRECTORY and create the corresponding
+'modules.alias' file."
+ (define aliases
+ (map (lambda (file)
+ (cons (file-name->module-name file) (module-aliases file)))
+ (find-files directory "\\.ko$")))
+
+ (call-with-output-file (string-append directory "/modules.alias")
+ (lambda (port)
+ (display "# Aliases extracted from modules themselves.\n" port)
+ (for-each (match-lambda
+ ((module . aliases)
+ (for-each (lambda (alias)
+ (format port "alias ~a ~a\n" alias module))
+ aliases)))
+ aliases))))
+
;;; linux-modules.scm ends here
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 23:15:37+0100 2018
[PATCH v11 2/6] linux-modules: Add 'load-linux-modules-from-directory'.
(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180312221541.1886-2-ludo@gnu.org
* gnu/build/linux-modules.scm (load-linux-modules-from-directory): New
procedure.
* gnu/build/linux-boot.scm (boot-system)[lookup-module]: Remove.
Use 'load-linux-modules-from-directory' instead.
---
gnu/build/linux-boot.scm | 9 ++-------
gnu/build/linux-modules.scm | 12 ++++++++++++
2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..df0b2b2d1 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,10 +469,6 @@ upon error."
mounts)
"ext4"))

- (define (lookup-module name)
- (string-append linux-module-directory "/"
- (ensure-dot-ko name)))
-
(display "Welcome, this is GNU's early boot Guile.\n")
(display "Use '--repl' for an initrd REPL.\n\n")

@@ -487,9 +483,8 @@ upon error."
(start-repl))

(display "loading kernel modules...\n")
- (for-each (cut load-linux-module* <>
- #:lookup-module lookup-module)
- (map lookup-module linux-modules))
+ (load-linux-modules-from-directory linux-modules
+ linux-module-directory)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index b444e6f90..0084f972b 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -38,6 +38,7 @@
modules-loaded
module-loaded?
load-linux-module*
+ load-linux-modules-from-directory

current-module-debugging-port

@@ -232,6 +233,17 @@ appears in BLACK-LIST are not loaded."
(or (and recursive? (= EEXIST (system-error-errno args)))
(apply throw args)))))))

+(define (load-linux-modules-from-directory modules directory)
+ "Load MODULES and their dependencies from DIRECTORY, a directory containing
+the '.ko' files. The '.ko' suffix is automatically added to MODULES if
+needed."
+ (define (lookup-module name)
+ (string-append directory "/" (ensure-dot-ko name)))
+
+ (for-each (cut load-linux-module* <>
+ #:lookup-module lookup-module)
+ (map lookup-module modules)))
+

;;;
;;; Device modules.
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 23:15:38+0100 2018
[PATCH v11 3/6] linux-modules: Add 'load-needed-linux-modules'.
(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180312221541.1886-3-ludo@gnu.org
Based on a patch by Danny Milosavljevic <dannym@scratchpost.org>.

* gnu/build/linux-modules.scm (load-needed-linux-modules): New
procedure.
---
gnu/build/linux-modules.scm | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 0084f972b..55161e026 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
#:use-module (ice-9 vlist)
#:use-module (ice-9 match)
#:use-module (ice-9 rdelim)
+ #:use-module (ice-9 ftw)
#:export (dot-ko
ensure-dot-ko
module-aliases
@@ -45,7 +46,8 @@
device-module-aliases
known-module-aliases
matching-modules
- write-module-alias-database))
+ write-module-alias-database
+ load-needed-linux-modules))

;;; Commentary:
;;;
@@ -413,4 +415,34 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
aliases)))
aliases))))

+(define (load-needed-linux-modules module-directory)
+ "Examine /sys/devices to find out which modules are needed and load those we
+have in MODULE-DIRECTORY. Return the list of modules loaded, not including
+dependencies.
+
+Note: loading modules leads to the creation of new entries in /sys/devices,
+which is why we need traversal and loading to be interleaved. If we walked
+/sys/devices and *then* loaded modules, we'd miss the entries added as a
+side-effect and would thus need to traverse /sys/devices again."
+ (define aliases
+ (known-module-aliases
+ (string-append module-directory "/modules.alias")))
+
+ (define (enter? director stat result) result)
+ (define (down directory stat result) result)
+ (define (up directory stat result) result)
+ (define (skip entry stat result) result)
+ (define (error name stat errno result) result)
+ (define (leaf file stat result)
+ (if (string=? (basename file) "modalias")
+ (let* ((alias (string-trim-right
+ (call-with-input-file file get-string-all)))
+ (modules (matching-modules alias aliases)))
+ (load-linux-modules-from-directory modules
+ module-directory)
+ (append modules result))
+ result))
+
+ (file-system-fold enter? leaf down up skip error #t "/sys/devices"))
+
;;; linux-modules.scm ends here
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 23:15:39+0100 2018
[PATCH v11 4/6] vm: Make the virtio-blk uniquely identifiable in /sys.
(address . 30604@debbugs.gnu.org)
20180312221541.1886-4-ludo@gnu.org
From: Danny Milosavljevic <dannym@scratchpost.org>

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.

Signed-off-by: Ludovic Courtès <ludo@gnu.org>
---
gnu/build/vm.scm | 6 +++++-
gnu/system/vm.scm | 7 ++++++-
2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..773b738ae 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,11 @@ the #:references-graphs parameter of 'derivation'."
builder)
(append
(if make-disk-image?
- `("-device" "virtio-blk,drive=myhd"
+
+ ;; QEMU seems to have a bug that it doesn't assign PCI
+ ;; IDs properly, so force the address of our virtio-blk
+ ;; device.
+ `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
"-drive" ,(string-append "if=none,file=" output
",format=" disk-image-format
",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 91ff32ce9..4360adf15 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -677,7 +677,12 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."

#$@(map virtfs-option shared-fs)
"-vga std"
- (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+
+ ;; QEMU seems to have a bug that it doesn't assign PCI IDs properly, so
+ ;; force the address of our virtio-blk device.
+ "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+
+ (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
#$image)))

(define* (system-qemu-image/shared-store-script os
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 23:15:40+0100 2018
[PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20180312221541.1886-5-ludo@gnu.org
From: Danny Milosavljevic <dannym@scratchpost.org>

This allows us to load modules on demand when the kernel asks for them.

* gnu/system/linux-initrd.scm (modprobe-program): New variable.
(flat-linux-module-directory): Call 'write-module-alias-database'.
(raw-initrd): Pass #:modprobe to 'boot-system' and to 'expression->initrd'.
(expression->initrd): Copy "closure" to $out/references. Add #:modprobe
and pass it to 'build-initrd'.
* gnu/build/linux-initrd.scm (build-initrd): Add #:modprobe and honor
it.
* gnu/build/linux-boot.scm (boot-system): Add #:modprobe and honor it.
Call 'load-needed-linux-modules'.
* gnu/system/vm.scm (qemu-image): Add #:linux parameter. Define
'modprobe-wrapper' and pass it to 'activate-modprobe'. Pass #:linux to
'expression->derivation-in-linux-vm'.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
---
gnu/build/linux-boot.scm | 13 +++++-
gnu/build/linux-initrd.scm | 16 ++++++--
gnu/system/linux-initrd.scm | 97 ++++++++++++++++++++++++++++++++++++++++-----
gnu/system/vm.scm | 21 ++++++++++
4 files changed, 133 insertions(+), 14 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index df0b2b2d1..eedc4bb9d 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -435,6 +435,7 @@ bailing out.~%root contents: ~s~%" (scandir "/"))


(define* (boot-system #:key
+ modprobe
(linux-modules '())
linux-module-directory
qemu-guest-networking?
@@ -449,6 +450,9 @@ QEMU-GUEST-NETWORKING? is true, calling PRE-MOUNT, mounting the file systems
specified in MOUNTS, and finally booting into the new root if any. The initrd
supports kernel command-line options '--load', '--root', and '--repl'.

+MODPROBE must be #f or a program to install as the modprobe program that the
+kernel will invoke when it needs to load modules.
+
Mount the root file system, specified by the '--root' command-line argument,
if any.

@@ -482,9 +486,14 @@ upon error."
(when (member "--repl" args)
(start-repl))

+ (when modprobe
+ ;; Tell the kernel to invoke MODPROBE.
+ (call-with-output-file "/proc/sys/kernel/modprobe"
+ (lambda (port)
+ (display modprobe port))))
+
(display "loading kernel modules...\n")
- (load-linux-modules-from-directory linux-modules
- linux-module-directory)
+ (load-needed-linux-modules linux-module-directory)

(when qemu-guest-networking?
(unless (configure-qemu-networking)
diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..4fa2bee7d 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2013, 2014, 2015 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2013, 2014, 2015, 2018 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -107,12 +107,18 @@ This is similar to what 'compiled-file-name' in (system base compile) does."

(define* (build-initrd output
#:key
- guile init
+ guile init modprobe
(references-graphs '())
(gzip "gzip"))
"Write an initial RAM disk (initrd) to OUTPUT. The initrd starts the script
at INIT, running GUILE. It contains all the items referred to by
-REFERENCES-GRAPHS."
+REFERENCES-GRAPHS.
+
+When MODPROBE is true, make /sbin/modprobe a symlink to it. This is useful
+because Linux invokes 'modprobe' when it needs to load a module and its
+default file name is '/sbin/modprobe' (see 'call_modprobe' in kernel/kmod.c).
+Creating this symlink allows us to make sure there's no time window during
+which 'modprobe' is unavailable."
(mkdir "contents")

;; Copy the closures of all the items referenced in REFERENCES-GRAPHS.
@@ -131,6 +137,10 @@ REFERENCES-GRAPHS."
(symlink (string-append guile "/bin/guile") "proc/self/exe")
(readlink "proc/self/exe")

+ (when modprobe
+ (mkdir-p "sbin")
+ (symlink modprobe "sbin/modprobe"))
+
;; Reset the timestamps of all the files that will make it in the initrd.
(for-each (lambda (file)
(unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 1eb5f5130..7a167146f 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -63,16 +63,87 @@
;;;
;;; Code:

+(define* (modprobe-program linux-module-directory #:key
+ (guile %guile-static-stripped))
+ "Return a minimal implementation of 'modprobe' for our initrd that looks up
+modules in LINUX-MODULE-DIRECTORY. This program will be invoked by the kernel
+when modules need to be loaded."
+ (define program
+ (with-imported-modules (source-module-closure
+ '((gnu build linux-modules)))
+ #~(begin
+ (use-modules (gnu build linux-modules)
+ (ice-9 match)
+ (srfi srfi-1)
+ (srfi srfi-26)
+ (srfi srfi-37))
+
+ (define option-spec
+ (list (option '(#\q "quiet") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'quiet? #t result)))))
+
+ (define options
+ ;; Alist of options and non-option arguments.
+ (args-fold (cdr (program-arguments))
+ option-spec
+ (lambda (opt name arg result)
+ (error "unrecognized option" name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ '()))
+
+ (define alias
+ ;; The alias we are asked to load. The remaining arguments are
+ ;; module parameters. In practice the kernel doesn't pass module
+ ;; parameters so we ignore them here.
+ (any (match-lambda
+ (('argument . alias) alias)
+ (_ #f))
+ options))
+
+ (define linux-module-directory
+ ;; The module directory. Note: We expect a flat directory here.
+ #$linux-module-directory)
+
+ (define %known-aliases
+ ;; The alias database.
+ (known-module-aliases
+ (string-append linux-module-directory "/modules.alias")))
+
+ (when (assq-ref options 'quiet?)
+ (current-error-port (%make-void-port "w"))
+ (current-output-port (%make-void-port "w")))
+
+ (let ((modules (matching-modules alias %known-aliases)))
+ (call-with-output-file "/dev/kmsg"
+ (lambda (port)
+ (setvbuf port 'block 1024)
+ (format port "modprobe[~a]: alias ~s; modules ~s; args ~s~%"
+ (getpid) alias modules (program-arguments))))
+
+ (when (null? modules)
+ (error "alias resolution failed" alias))
+
+ (load-linux-modules-from-directory modules
+ linux-module-directory)))))
+
+ (program-file "modprobe" program #:guile guile))

(define* (expression->initrd exp
#:key
+ modprobe
(guile %guile-static-stripped)
(gzip gzip)
(name "guile-initrd")
(system (%current-system)))
"Return a derivation that builds a Linux initrd (a gzipped cpio archive)
containing GUILE and that evaluates EXP, a G-expression, upon booting. All
-the derivations referenced by EXP are automatically copied to the initrd."
+the derivations referenced by EXP are automatically copied to the initrd.
+
+When MODPROBE is true, '/sbin/modprobe' is created as a symlink pointing to
+it. This allows Linux to call out to MODPROBE as soon as it boots if it needs
+to load modules."

;; General Linux overview in `Documentation/early-userspace/README' and
;; `Documentation/filesystems/ramfs-rootfs-initramfs.txt'.
@@ -89,20 +160,21 @@ the derivations referenced by EXP are automatically copied to the initrd."
(mkdir #$output)

;; The guile used in the initrd must be present in the store, so
- ;; that module loading works once the root is switched.
+ ;; that module loading works once the root is switched. Similarly,
+ ;; the 'modprobe' program installed in /proc/sys/kernel/modprobe by
+ ;; the initrd, if any, must be present after switch root.
;;
- ;; To ensure that is the case, add an explicit reference to the
- ;; guile package used in the initrd to the output.
+ ;; To ensure that is the case, add an explicit reference to these in
+ ;; the output.
;;
- ;; This fixes guix-patches bug #28399, "Fix mysql activation, and
+ ;; This fixes <https://bugs.gnu.org/28399>, "Fix mysql activation, and
;; add a basic test".
- (call-with-output-file (string-append #$ output "/references")
- (lambda (port)
- (simple-format port "~A\n" #$guile)))
+ (copy-file "closure" (string-append #$output "/references"))

(build-initrd (string-append #$output "/initrd")
#:guile #$guile
#:init #$init
+ #:modprobe #$modprobe
;; Copy everything INIT refers to into the initrd.
#:references-graphs '("closure")
#:gzip (string-append #$gzip "/bin/gzip")))))
@@ -153,7 +225,9 @@ MODULES and taken from LINUX."
(copy-file module
(string-append #$output "/"
(basename module))))
- (delete-duplicates modules)))))
+ (delete-duplicates modules))
+
+ (write-module-alias-database #$output))))

(computed-file "linux-modules" build-exp))

@@ -196,6 +270,9 @@ upon error."
(define kodir
(flat-linux-module-directory linux linux-modules))

+ (define modprobe
+ (modprobe-program kodir))
+
(expression->initrd
(with-imported-modules (source-module-closure
'((gnu build linux-boot)
@@ -229,9 +306,11 @@ upon error."
(and #$@device-mapping-commands))
#:linux-modules '#$linux-modules
#:linux-module-directory '#$kodir
+ #:modprobe #$modprobe
#:qemu-guest-networking? #$qemu-networking?
#:volatile-root? '#$volatile-root?
#:on-error '#$on-error)))
+ #:modprobe modprobe
#:name "raw-initrd"))

(define* (file-system-packages file-systems #:key (volatile-root? #f))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4360adf15..7f552bef8 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -249,6 +249,7 @@ INPUTS is a list of inputs (as for packages)."
(define* (qemu-image #:key
(name "qemu-image")
(system (%current-system))
+ (linux linux-libre)
(qemu qemu-minimal)
(disk-image-size 'guess)
(disk-image-format "qcow2")
@@ -275,18 +276,37 @@ INPUTS is a list of inputs (as for packages). When COPY-INPUTS? is true, copy
all of INPUTS into the image being built. When REGISTER-CLOSURES? is true,
register INPUTS in the store database of the image so that Guix can be used in
the image."
+ (define modprobe-wrapper
+ ;; Wrapper for the 'modprobe' command that knows where modules live.
+ (let ((modprobe (file-append kmod "/bin/modprobe")))
+ (program-file "modprobe"
+ #~(begin
+ (setenv "LINUX_MODULE_DIRECTORY"
+ #$(file-append linux "/lib/modules"))
+ (apply execl #$modprobe
+ (cons #$modprobe (cdr (command-line))))))))
+
+
(expression->derivation-in-linux-vm
name
(with-imported-modules (source-module-closure '((gnu build bootloader)
(gnu build vm)
+ (gnu build activation)
(guix build utils)))
#~(begin
(use-modules (gnu build bootloader)
(gnu build vm)
+ ((gnu build activation) #:select (activate-modprobe))
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))

+ ;; We may need to lazy-load Linux modules. The initrd installs a
+ ;; 'modprobe' that can only search through the modules available in
+ ;; the initrd, but here we want to be able to use all the modules of
+ ;; LINUX. Thus, install a real 'modprobe'.
+ (activate-modprobe #$modprobe-wrapper)
+
(let ((inputs
'#$(append (list qemu parted e2fsprogs dosfstools)
(map canonical-package
@@ -361,6 +381,7 @@ the image."
#$(bootloader-installer bootloader))
(reboot)))))
#:system system
+ #:linux linux
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
--
2.16.2
L
Ludovic Courtès wrote on Mon Mar 12 23:15:41+0100 2018
[PATCH v11 6/6] linux-modules: Add "modules.devname" writer.
(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20180312221541.1886-6-ludo@gnu.org
* gnu/build/linux-modules.scm (aliases->device-tuple)
(write-module-device-database): New procedures.

Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>.
---
gnu/build/linux-modules.scm | 51 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 51 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 55161e026..d3ba2c60a 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -47,6 +47,8 @@
known-module-aliases
matching-modules
write-module-alias-database
+ write-module-device-database
+
load-needed-linux-modules))

;;; Commentary:
@@ -415,6 +417,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
aliases)))
aliases))))

+(define (aliases->device-tuple aliases)
+ "Traverse ALIASES, a list of module aliases, and search for
+\"char-major-M-N\", \"block-major-M-N\", or \"devname:\" aliases. When they
+are found, return a tuple (DEVNAME TYPE MAJOR MINOR), otherwise return #f."
+ (define (char/block-major? alias)
+ (or (string-prefix? "char-major-" alias)
+ (string-prefix? "block-major-" alias)))
+
+ (define (char/block-major->tuple alias)
+ (match (string-tokenize alias %not-dash)
+ ((type "major" (= string->number major) (= string->number minor))
+ (list (match type
+ ("char" "c")
+ ("block" "b"))
+ major minor))))
+
+ (let* ((devname (any (lambda (alias)
+ (and (string-prefix? "devname:" alias)
+ (string-drop alias 8)))
+ aliases))
+ (major/minor (match (find char/block-major? aliases)
+ (#f #f)
+ (str (char/block-major->tuple str)))))
+ (and devname major/minor
+ (cons devname major/minor))))
+
+(define %not-dash
+ (char-set-complement (char-set #\-)))
+
+(define (write-module-device-database directory)
+ "Traverse the '.ko' files in DIRECTORY and create the corresponding
+'modules.devname' file. This file contains information about modules that can
+be loaded on-demand, such as file system modules."
+ (define aliases
+ (filter-map (lambda (file)
+ (match (aliases->device-tuple (module-aliases file))
+ (#f #f)
+ (tuple (cons (file-name->module-name file) tuple))))
+ (find-files directory "\\.ko$")))
+
+ (call-with-output-file (string-append "/tmp" "/modules.devname")
+ (lambda (port)
+ (display "# Device nodes to trigger on-demand module loading.\n" port)
+ (for-each (match-lambda
+ ((module devname type major minor)
+ (format port "~a ~a ~a~a:~a~%"
+ module devname type major minor)))
+ aliases))))
+
(define (load-needed-linux-modules module-directory)
"Examine /sys/devices to find out which modules are needed and load those we
have in MODULE-DIRECTORY. Return the list of modules loaded, not including
--
2.16.2
D
Danny Milosavljevic wrote on Mon Mar 12 23:48:14+0100 2018
Re: [PATCH v11 1/6] linux-modules: Add "modules.alias" writer.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312234814.6b89aecc@scratchpost.org
Hi Ludo,

On Mon, 12 Mar 2018 23:15:36 +0100
Ludovic Courtès <ludo@gnu.org> wrote:

> (%not-dash): New variable.

Old commit message?

Otherwise LGTM!
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqnA64ACgkQ5xo1VCww
uqVHgwf7BoMySgvE89KaihiVsydHIYT6X+ewGMl4O+hJqVk/qLunMnIp9bEwSoOt
HzwIRt1iZwVPVkZUKkvnb3KDrL5OiDPGhTMOxntfdjHCnxU9WnCo/5K5SJEu5QJA
H0Bnr16fHa9O8e5Po+1fCbS8lASoNVRuod7/7MaHOy5Q8h/8k7smwXL5m4oNPxKL
Dc8CtM2bCOmr/WimkMqkdyWbEswZRQ/ZpX2qbKhGdaYkhvydQhxPmskSpkyWPga6
blmVj1z/7ylLux0YNGPB0Eg1Gatga+enCQlJmP8DqDKgCj1OXS2r58HQWKGCj4ZR
183Umm1tGLdgSgY49SGmEVEs9y2pXw==
=vuYP
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Mon Mar 12 23:48:36+0100 2018
Re: [bug#30604] [PATCH v11 2/6] linux-modules: Add 'load-linux-modules-from-directory'.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312234836.106e2eb0@scratchpost.org
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqnA8QACgkQ5xo1VCww
uqU5RAf/RcXh3olPJFZykEZMmUHATzkHioRJD3XilWmU0row0mm1LZR4PzN/tuRm
ASvxkAvXbTc/FeoFU22rLJAv5vY8WfXCFk3NS2PDI7eNyVaeW3wt/qZE4FYiIi9C
oHFzAh+q1v+AHyxly4FhNpGID4XGoCPPDU4/FcRHDvVjBaDR7wjebBluZAxz7/DC
z9CYLPGLTdaxHeNIvW+BRaKfAQrj3tEtKim3RIQbE8rSWFEtj0wYZ/zEwB1IQBRA
oV4X40vz1FG1JkGSNWO/mDD8yJQ8m4XJfOIhfWK2pLWTgknh1X0ShfYUPQY7PbHk
PvMjaWU1xpdu2mOtnivV5Dr5NqkbHA==
=4f2L
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Mon Mar 12 23:49:22+0100 2018
Re: [PATCH v11 4/6] vm: Make the virtio-blk uniquely identifiable in /sys.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312234922.14fc6425@scratchpost.org
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqnA/IACgkQ5xo1VCww
uqUC9Qf9GftT17RdFjms1zaPaJXnpZKbClxAvlZD1xLNy3muQXYcznwdEl+rZs6s
YSeDaUhblppXsORQBjcuNJkgp0/dx6hsdZGmjgChwxNyGxcBHQgGUmf6BuUYwt//
2q4EFmcA/rjXGUj9BdX6n5tdPFXHFZtXA0dUNiWx95j+/6gY+8BqLuovn7vlxrBR
32MpR9o5gapDgRABtI+OYxwuPofwIAnSscyxnVFHkm/PA5Bh62KDyLuJYP6eMqpi
xdCEWuzaGpZw3TarvjTuqwsAoPdWsJ15kHeEU6I5cfH8/W9bqt/kJQt/Z3wERaQN
4WLepIiJ917ijx54RtGtMuUyj/Pkww==
=4Olm
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Mon Mar 12 23:55:46+0100 2018
Re: [bug#30604] [PATCH v11 6/6] linux-modules: Add "modules.devname" writer.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180312235547.6ac0af0a@scratchpost.org
> + (call-with-output-file (string-append "/tmp" "/modules.devname")
^^^^^^ directory

Otherwise LGTM!
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqnBXMACgkQ5xo1VCww
uqUZ3Qf+MApqVj/sHxKgauy8+bnT5EbH1jEqXkqa/BGsD95wRG5WodT0AZi0iRBv
wv8mIC8vOl4VS9pcmQSPBvdkIX21RB29Bqrf8CtaThTDqYI8Z5N5izWJLqP2uFan
MJC6ckHcokY/WpTXB+kNZUCfyqvQgqLEaAWb/5JtuPNRQl8ogcX2ccpQosr16f7D
EUplZLt61zHZj90BJBmqmfUk7oWw/PMUw+A2+yNUk4PgQnDQ0R8it+tNDs20ahQX
6Gs2YkDOYtwteWXQnKQOL4cgy7lp9Xck5GnTDpJU21wSYxnePg1CwNjRdfJsGft0
kMS7X1UKiFxrZwg8TUNWxWGuMDLzGA==
=2nsk
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Tue Mar 13 00:00:31+0100 2018
Re: [bug#30604] [PATCH v11 3/6] linux-modules: Add 'load-needed-linux-modules'.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180313000031.06767edf@scratchpost.org
> +(define (load-needed-linux-modules module-directory)
[...]
> + (file-system-fold enter? leaf down up skip error #t "/sys/devices"))

Isn't #t a strange initial value? Shouldn't it be '() ?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqnBo8ACgkQ5xo1VCww
uqUgxAgAjg3Z/ieFvl0Up29vWcWFDgAx7wbEyTmyPlzbcJv6Y64smPzrWawa05vJ
OUi80fg9cnoANXuEdVKslegj48YbpsYtcPC08hUS+Ccjl8KJc7frPtZHUXZ8mPzA
v4V0wUxvaT+SW5BxCI6ancAZJdRutQVh9Nxztb2bLqnnagxAVXn/Aeqy5tLQUYjR
HulvVPHEKyI7RwDAjhm8IFSvuks3IUp5yOnh36GxhQaqdwvuhW5o6iZHjS7R8LER
BXo8NiAK3/RhOiUx7I/i72avNHqp4tnAPDpSRGm2O1fSS3WD+ZCf1vdEvdq5u+E4
oUVZalTFyual5SDiC4ZsvDxtrM/00A==
=tBne
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Tue Mar 13 09:50:40+0100 2018
Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
87fu54fjfj.fsf@gnu.org
Hello!

I’m using this on the bare metal. \o/

I had to make this change so that dm-crypt would be loaded once I’ve
entered my passphrase:
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7a167146f..ef81fe718 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -115,16 +115,19 @@ when modules need to be loaded."
(current-error-port (%make-void-port "w"))
(current-output-port (%make-void-port "w")))

- (let ((modules (matching-modules alias %known-aliases)))
+ ;; Look up ALIAS in %KNOWN-ALIASES first, and then, if it fails,
+ ;; assume that ALIAS is a module name (some modules such as
+ ;; 'dm-crypt' don't have an alias.) Note that
+ ;; 'kmod_module_new_from_lookup' uses a different search order.
+ (let ((modules (match (matching-modules alias %known-aliases)
+ (() (list alias))
+ (lst lst))))
(call-with-output-file "/dev/kmsg"
(lambda (port)
(setvbuf port 'block 1024)
(format port "modprobe[~a]: alias ~s; modules ~s; args ~s~%"
(getpid) alias modules (program-arguments))))

- (when (null? modules)
- (error "alias resolution failed" alias))
-
(load-linux-modules-from-directory modules
linux-module-directory)))))
FWIW I have:

--8<---------------cut here---------------start------------->8---
modprobe[146]: alias "net-pf-38"; modules ("net-pf-38"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "net-pf-38")
modprobe[152]: alias "dm-crypt"; modules ("dm-crypt"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "dm-crypt")
modprobe[157]: alias "crypto-xts(aes)"; modules ("crypto-xts(aes)"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts(aes)")
modprobe[162]: alias "crypto-xts(aes)-all"; modules ("crypto-xts(aes)-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts(aes)-all")
modprobe[168]: alias "crypto-xts"; modules ("xts"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts")
nfo> [1520929735.4127] manager[0x12950c0]: monitoring kernel firmware directory '/lib/firmware'.
modprobe[173]: alias "crypto-aes"; modules ("crypto-aes"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-aes")
modprobe[178]: alias "crypto-aes-all"; modules ("crypto-aes-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-aes-all")
modprobe[183]: alias "crypto-ecb(aes)"; modules ("crypto-ecb(aes)"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-ecb(aes)")
modprobe[188]: alias "crypto-ecb(aes)-all"; modules ("crypto-ecb(aes)-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-ecb(aes)-all")
--8<---------------cut here---------------end--------------->8---

Something annoying is that my external USB keyboard doesn’t work while
in the initrd (when I type my passphrase). I can see that it’s detected
early on, before I type my passphrase:

--8<---------------cut here---------------start------------->8---
[ 1.532237] usb 1-4.2: new low-speed USB device number 4 using xhci_hcd
[ 1.648836] usb 1-4.2: New USB device found, idVendor=413c, idProduct=2105
[ 1.649189] usb 1-4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1.649533] usb 1-4.2: Product: Dell USB Keyboard
[ 1.649882] usb 1-4.2: Manufacturer: Dell
--8<---------------cut here---------------end--------------->8---

but its modules don’t get loaded until after eudev has been started, and
there’s no trace of a modprobe invocation for it.

--8<---------------cut here---------------start------------->8---
[ 20.968938] input: Dell Dell USB Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4.2/1-4.2:1.0/0003:413C:2105.0001/input/input16
--8<---------------cut here---------------end--------------->8---

Any idea what we’re missing?

Thanks,
Ludo’.
L
Ludovic Courtès wrote on Tue Mar 13 09:54:55+0100 2018
Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87bmfsfj8g.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> That was why earlier I kept /sbin/modprobe as the name (it's the
>> default Linux uses) and made sure that the file /sbin/modprobe is already
>> in the initrd (as opposed to created by boot-system).
>
> Note: The simplest change would be to keep this patch as-is but add
>
> ;; Make sure that Linux can modprobe even before boot-system started up.
> (when modprobe
> (mkdir-p "sbin")
> (symlink modprobe "sbin/modprobe")
> (compile-to-cache "sbin/modprobe")) ; Note: without this line booting is dead slow. [Maybe in this case that wouldn't be so bad - since boot-system switches away immediately anyway :)]
>
> to gnu/build/linux-initrd.scm

Yeah, done. Ideally ‘program-file’ would take care of this. Future
work!

Ludo’.
L
Ludovic Courtès wrote on Tue Mar 13 09:55:54+0100 2018
Re: [bug#30604] [PATCH v11 3/6] linux-modules: Add 'load-needed-linux-modules'.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
877eqgfj6t.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> +(define (load-needed-linux-modules module-directory)
> [...]
>> + (file-system-fold enter? leaf down up skip error #t "/sys/devices"))
>
> Isn't #t a strange initial value? Shouldn't it be '() ?

Indeed! Fixed, thanks.
L
Ludovic Courtès wrote on Tue Mar 13 09:56:39+0100 2018
Re: [bug#30604] [PATCH v11 6/6] linux-modules: Add "modules.devname" writer.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
873714fj5k.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> + (call-with-output-file (string-append "/tmp" "/modules.devname")
> ^^^^^^ directory
>
> Otherwise LGTM!

Oops, fixed!
D
Danny Milosavljevic wrote on Tue Mar 13 10:27:06+0100 2018
Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180313100539.442c4aa9@scratchpost.org
Hi Ludo,

On Mon, 12 Mar 2018 23:14:59 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> > One of the devnames created statically is the one for btrfs, so not writing or
> > using devnames is not going to end well.
>
> We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,
> “fs-9p”, etc. alias, which shows up in “modules.alias”. No need for
> “modules.devname” AFAICS.

The other filesystems are not such a problem - but BTRFS has its own snapshotting/
multivolume functionality - and it's possible that someone accesses /dev/btrfs-control
"too early" for that.

I applied your patches to a fresh clone of guix master now, ran the btrfs-root-os
system check, and indeed I get (tried two rounds, happened both times):

$ make TESTS=btrfs-root-os check-system
[...]
Scanning for Btrfs filesystems
WARNING: failed to open /dev/btrfs-control, skipping device registration: No suy
ERROR: there are 1 errors while registering devices
File system check on /dev/vda2 failed; spawning Bourne-like REPL
GNU Guile 2.2.3
Copyright (C) 1995-2017 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
(can't evaluate anything here)

> > (I'd also copy the modules.builtin (from Linux).
> > Also, what happens if we load a module which has as dependency a builtin?
> > Will we try to load the builtin as a .ko file and fail the entire thing?)
>
> The dependency of a builtin is necessarily a builtin,

Yes.

>and the kernel won’t invoke modprobe for a builtin, will it?

I've read Linux's __request_module and I can't find where they
pre-check anything - neither whether there's already a builtin
nor whether it's loaded already.

They probably don't check. But I'll read it again, more carefully...

(request_module isn't that popular so it makes sense for them not to complicate
the kernel by these checks when there are like 8 callers in total - and all on init)

>Thanks for the insightful review!

You're welcome :)
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqnmWoACgkQ5xo1VCww
uqWYNQf+JldN8uxgw6yQsYw8jQ9KKV5ilkDItpc6iw+L5Sl3JsYovJ/ABc6JX5m6
nOuQHU4f1rjqJS5kqpFW/qOJbk+3tJSNfRuleJUdKjHfzp3qtTLLYOsN3fTF7OSI
fqN2zqS0y5o/C/T1Ienr7Qx+oC6u56smD1FyoLELJDmL8vGj0qUjNytilqavqxwR
cwtlGxaREEHmuIB2UjNCXUTUdgw5SDS8Dga/x53AvOEIBsc61Cjyd+j/jzoH3NYC
g/reldxXWMPq/DlSkil2293jbnK11+5K/6do+bf5a7uoEdE8E5gKZFqrze7ZiwRS
rfgNmo0u7rv146A2b1bNEw1XgZ9tPw==
=fti+
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Tue Mar 13 10:28:52+0100 2018
Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
876060qq7f.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

> Something annoying is that my external USB keyboard doesn’t work while
> in the initrd (when I type my passphrase). I can see that it’s detected
> early on, before I type my passphrase:

I tried looking up “modalias” right when entering a /sys/devices
directory, like you did in the initial patch, but that didn’t help:
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4515839e9..213a8bbb9 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -480,19 +480,22 @@ side-effect and would thus need to traverse /sys/devices again."
(string-append module-directory "/modules.alias")))

(define (enter? director stat result) result)
- (define (down directory stat result) result)
- (define (up directory stat result) result)
- (define (skip entry stat result) result)
- (define (error name stat errno result) result)
- (define (leaf file stat result)
- (if (string=? (basename file) "modalias")
+ (define (down directory stat result)
+ ;; Check for "modalias" right when entering DIRECTORY instead of waiting
+ ;; for LEAF to be called. XXX: Doesn't help!
+ (let ((modalias (string-append directory "/modalias")))
+ (if (file-exists? modalias)
(let* ((alias (string-trim-right
- (call-with-input-file file get-string-all)))
+ (call-with-input-file modalias get-string-all)))
(modules (matching-modules alias aliases)))
(load-linux-modules-from-directory modules
module-directory)
(append modules result))
- result))
+ result)))
+ (define (up directory stat result) result)
+ (define (skip entry stat result) result)
+ (define (error name stat errno result) result)
+ (define (leaf file stat result) file stat result)

(file-system-fold enter? leaf down up skip error '() "/sys/devices"))
Ludo’.
L
Ludovic Courtès wrote on Tue Mar 13 10:29:05+0100 2018
control message for bug #30604
(address . control@debbugs.gnu.org)
874llkqq72.fsf@gnu.org
severity 30604 important
L
Ludovic Courtès wrote on Tue Mar 13 11:51:42+0100 2018
Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87zi3cp7sx.fsf@gnu.org
Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Mon, 12 Mar 2018 23:14:59 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> > One of the devnames created statically is the one for btrfs, so not writing or
>> > using devnames is not going to end well.
>>
>> We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,
>> “fs-9p”, etc. alias, which shows up in “modules.alias”. No need for
>> “modules.devname” AFAICS.
>
> The other filesystems are not such a problem - but BTRFS has its own snapshotting/
> multivolume functionality - and it's possible that someone accesses /dev/btrfs-control
> "too early" for that.
>
> I applied your patches to a fresh clone of guix master now, ran the btrfs-root-os
> system check, and indeed I get (tried two rounds, happened both times):
>
> $ make TESTS=btrfs-root-os check-system
> [...]
> Scanning for Btrfs filesystems
> WARNING: failed to open /dev/btrfs-control, skipping device registration: No suy
> ERROR: there are 1 errors while registering devices
> File system check on /dev/vda2 failed; spawning Bourne-like REPL
> GNU Guile 2.2.3
> Copyright (C) 1995-2017 Free Software Foundation, Inc.
>
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
>
> Enter `,help' for help.

Do you see a modprobe invocation for “fs-btrfs” or “block-major-xxx”
before?

>> > (I'd also copy the modules.builtin (from Linux).
>> > Also, what happens if we load a module which has as dependency a builtin?
>> > Will we try to load the builtin as a .ko file and fail the entire thing?)
>>
>> The dependency of a builtin is necessarily a builtin,
>
> Yes.
>
>>and the kernel won’t invoke modprobe for a builtin, will it?
>
> I've read Linux's __request_module and I can't find where they
> pre-check anything - neither whether there's already a builtin
> nor whether it's loaded already.
>
> They probably don't check. But I'll read it again, more carefully...
>
> (request_module isn't that popular so it makes sense for them not to complicate
> the kernel by these checks when there are like 8 callers in total - and all on init)

Right, and the worst that can happen is that modprobe will fail, but
that’s fine because the functionality is already there anyway.

Ludo’.
D
Danny Milosavljevic wrote on Tue Mar 13 13:05:57+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180313130557.247b02c6@scratchpost.org
Hi Ludo,

On Tue, 13 Mar 2018 11:51:42 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Right, and the worst that can happen is that modprobe will fail, but
> that’s fine because the functionality is already there anyway.

Yeah - but what does the kernel do when we return a non-zero exit code for something he wants to use? :)

./fs/filesystems.c: if (!fs && (request_module("%.*s", len, name) == 0))
^^^^ uh oh
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqnvqUACgkQ5xo1VCww
uqWuFgf+K8xlYmtgpo26Gl7ucp1qTzN8oeZNOe778AXvoFD3KkBL0rTrd0+yWLJ+
f7Jmtm0H7lSwa8TvsRdm6deyXxcIggqJXyBStEG2tp13UB0ox1RLfpQJTX0atV6z
HRe8AR83nLQDf6GEWoFDvg0yzt2BXH1vuBQ27mkXrzvugzqNcw7a3ME7dqR1y2ub
ry7HYBmrXCy/3grvqJL8O4D4QsZ5iNaKAS9xwdNbt7DsjAw0ZzCkpivNl0zERR8/
sN9bCGBlT4Qei5b2vL97OCgn2Ts6ULo7jyVTuWWao68KG6GhOqX6ODPAl1SBa9Jn
Hq2JRrfrCqLMrQrBAbwqMnn7r97LJw==
=xN11
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Tue Mar 13 13:28:02+0100 2018
Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180313132154.212a1b92@scratchpost.org
> I’m using this on the bare metal. \o/

Hehe nice!

> - (let ((modules (matching-modules alias %known-aliases)))
> + ;; Look up ALIAS in %KNOWN-ALIASES first, and then, if it fails,
> + ;; assume that ALIAS is a module name (some modules such as
> + ;; 'dm-crypt' don't have an alias.) Note that
> + ;; 'kmod_module_new_from_lookup' uses a different search order.

I also wondered about the resolution order :)

At least module dependencies can't be aliases.

> + (let ((modules (match (matching-modules alias %known-aliases)
> + (() (list alias))
> + (lst lst))))

Yeah, same happened to me. Did the same (back then in "resolve-alias").

So should be fine.

> Something annoying is that my external USB keyboard doesn’t work while
> in the initrd (when I type my passphrase).

Did it work before? I remember that Andreas had a similar problem before all this.

Did you implement globs with [0-9] yet? I remember there were some USB devices
using those :)

What's the modalias of your usb keyboard?

Does your initrd's modules.alias contain any patterns that are supposed to match?

(You can look into those via
cat initrd | gunzip -cd |cpio -i

- it will unpack)
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqnw9IACgkQ5xo1VCww
uqUxygf+MRub2zk/V+bjGnAET899eTQrpa+rL2ailthjXeEgBXkGti9xump0cNde
6rgUm2sRFyzz1bnCbqyklaG5B3cefP7FHcQqQsr9rC+fAwBOcWLsCHLY1skv1OeC
6E8yvaZ60VzsUXGK6HzFWmTGSITquor6iwwkMtx1MnTBqKuELfqZG1DB3amVsAVz
4SN4b6ZQSdIicm1uI4JCrxYBYfNSEBHIEyE0mxtfCZUCSLN5lg/vRmAdN3zaERb9
WN9CCPLTIPMPWftAzX3Xd3eeHGRfcX3lumerD285mIX1w6k//CRJynxWPu90fCeP
f1EdHUu05t/7fUZm35j22zDQD8Hwkw==
=GJWH
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Tue Mar 13 20:17:02+0100 2018
Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180313201702.259bccd0@scratchpost.org
Hi Ludo,

> Do you see a modprobe invocation for “fs-btrfs” or “block-major-xxx”
> before?

I can't get the logfile to show any modprobe log message in this case
(which is "make TESTS=btrfs-root-os check-system").

I had that problem before and I think it's because guix is
using some special /tmp/repl pipe for qemu here[1] and so the message
vanishes somewhere where I can't see it (that is, the guest does
everything correctly but the host loses it somewhere).

The usual trick I use in this case which is "steal the cmdline and
the files from /proc/<qemu> and invoke it myself" doesn't work
either because of the pipe... hrmmmm...

I tried both /dev/kmsg and just not closing current-output-port etc.

I've double-checked that I've applied all v11 patches, including
the resolve-alias one (which I think is necessary anyway).

Help?

Log file attached.

[1] When it uses the VM to create the root fs.
Attachment: L.bz2
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlqoI64ACgkQ5xo1VCww
uqVqvAf/dHMacbdHNm0OuBqbRsEzH8FpFUGYcw84u9iSUJOAlSW3wrBc1kS/JVMD
m+ApWEbylvLt34ar3dEtBiKGIi4Mue6mGrsAcizqFlsYISJo7F+MKnZx3jDh0gb2
gqTtpp7p3YZGcQpMzzmz2KEF40NO8PMWxD8ESDHQKmzIks3NePfV9DAwBWUX/9qq
B0N7eO3cRYNxL7f4o8EFx9MfZQcZCGQqKoLnpIEEhWuBKRkC5M4uSEFDsqXf1RgQ
SSs7WA+IyeT4W8njEXLbnmmGD3dYg2OnOhBVL5bwwhqvzpbF7sjB60zxgnqtF8sz
01Swsb29wkfHaQMYIx9AJYukCoPG4Q==
=b/2o
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Sun Mar 18 16:03:06+0100 2018
Keyboard detection before ‘cryptsetup’ runs
(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
87o9jlo28l.fsf@inria.fr
Hi Danny,

ludo@gnu.org (Ludovic Courtès) skribis:

> Something annoying is that my external USB keyboard doesn’t work while
> in the initrd (when I type my passphrase). I can see that it’s detected
> early on, before I type my passphrase:

While experimenting with this on the bare metal, it became clear that
it’s a timing issue: the keyboard is detected right after the cryptsetup
is displayed, so ‘load-needed-linux-modules’ didn’t have a chance to
load the relevant modules.

Similarly, if I boot with ‘--repl’, and I manually type
(load-needed-linux-modules …) from there, then the keyboard’s module
gets loaded as expected. That’s because the device showed up in the
meantime and the kernel created a /sys entry for it.

In essence, we want a mini-udev and something akin to “udevadm settle”.
Merely calling ‘load-needed-linux-modules’ once isn’t enough.

One way to do that would be to have a separate thread that calls
‘load-needed-linux-modules’ as appropriate. Ideally it would use
inotify on /sys like udev does, but a poor programmer’s version could
simply call ‘load-needed-linux-modules’ every half a second or so.

Alternately, before passing control to user code (pre-mount actions,
etc.), we could do a “settle” kind of thing: call
‘load-linux-modules-from-directory’ every 0.5 seconds until its result
is the same as before. There’s still a risk of missing devices, and
those devices will never show up later because nobody’s monitoring /sys.
But then again, “udevadm settle” must have the same problem: it can’t
really know whether things have settled, I guess.

WDYT?

Thanks,
Ludo’.
D
Danny Milosavljevic wrote on Fri Mar 23 14:02:31+0100 2018
Re: Keyboard detection before ‘cryptsetup ’ runs
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180323140231.1c96615c@scratchpost.org
Hi Ludo,

I thought about it for a bit...

On Sun, 18 Mar 2018 16:03:06 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> One way to do that would be to have a separate thread that calls
> ‘load-needed-linux-modules’ as appropriate. Ideally it would use
> inotify on /sys like udev does, but a poor programmer’s version could
> simply call ‘load-needed-linux-modules’ every half a second or so.
>
> Alternately, before passing control to user code (pre-mount actions,
> etc.), we could do a “settle” kind of thing: call
> ‘load-linux-modules-from-directory’ every 0.5 seconds until its result
> is the same as before. There’s still a risk of missing devices, and
> those devices will never show up later because nobody’s monitoring /sys.
> But then again, “udevadm settle” must have the same problem: it can’t
> really know whether things have settled, I guess.

Yes, but udevd continues running and monitoring /sys anyway - so I guess
they don't really care (if I wrote it, I'd start up the monitoring first
and then traverse /sys for modalias - and make sure the modprobe is
idempotent).

I have to say what we have to do is much more complicated than I thought
it would be - and I kinda regret starting this patchset. I didn't
want to make it less likely for people to boot than before :-(

I suggest we find out how the other distributions do it - if they do it.

A small bit of research shows that:

- Busybox uses a loop with timeout to try to mount over and over again,
depending on bus type [2].
- Alpine coldplugs twice [1].
- Fatdog has a "waitdev" boot parameter that specifies how long it sleeps
until it tries to mount the root.
- Redhat mkinitrd has an mkinitrd option like "--with=scsi_wait_scan" which
they use after they saw a modalias starting with "scsi:" - or add the
kernel option "scsi_mod.scan=sync" to the command line [3] (but they say that
this is unreliable now and one is supposed to use udev inside the initrd).
Not sure what they do for USB.

The Linux kernel itself has a "rootdelay" parameter which
was specifically introduced because of USB devices that took 15 s (!)
to show up.

All in all, we have alternative ways to continue:

(a) Drop this patchset/feature entirely because it's too unreliable.
(b) Monitor /sys using inotify in an extra thread/process.
(c) Include udev into the initrd and have it do its thing.
(d) Monitor the netlink socket in our own thread and play udev ourselves.
(e) Find out how to, for each bus type, detect when enumeration is finished.
Find out which buses are there. Wait for them to finish enumerating.
(f) Always modprobe usb-hid first, no matter what. I think this would
become a whack-a-mole thing fast.

I'd prefer either (a) or (b) for reliability and simplicity. WDYT?

(inotify is Linux-only, but so is /sys/devices :P)

[1] https://git.alpinelinux.org/cgit/abuild/tree/initramfs-init?id=9154b0d9b7251cca900f9e834ac9af9ae2ae61d9 (scan_drivers)
[2] http://lists.busybox.net/pipermail/busybox/2015-March/082688.html
[3] https://bugzilla.redhat.com/show_bug.cgi?id=466607
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlq0+ucACgkQ5xo1VCww
uqWjAwf/ShXF09RtyD6sKDqunwq6f4RVl1YCwD+UXe4YpocIT2o3tmomD4Fo4ZI5
tXARLYRQLOl0K43+kXiwv2X7LHcFwkw8PSWMG6DoP/qoq9caOFUs5dS/XI/i/dUF
AUwck47T9kiEblAJi+rFL2Rkm2HSmAYOkG8nfCg9OmMFWOhIyuObA8YPsL1G+Qb7
uI+p+miA0m9l3QWCNS2eIO5EWKl6gWwHOHqWbyO/s+R6PeAC+vpdLkpCE/KlLgV3
P75p0mx+QMkCagcCfZsY3b4nnUOH3A63xEJ3nPqV7o6XrsND1XNVsnhFcot3bEc6
zdQZsljL1SHX4WcTLaFSBp2B3kEyoA==
=YjRi
-----END PGP SIGNATURE-----

D
Danny Milosavljevic wrote on Fri Mar 23 16:07:02+0100 2018
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)
20180323160702.45fa3fe5@scratchpost.org
> (b) Monitor /sys using inotify in an extra thread/process.

Oh - or loop { load-/sys ; sleep 1 }. That's what you meant.
-----BEGIN PGP SIGNATURE-----

iQEyBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlq1GBYACgkQ5xo1VCww
uqVK/gf4iqTdvsTHWIySk+mGwEXRxx4/kTsuwGqybxyXmjgb7qVbcRCa8+8InGkS
FhAfpyYGRf584fnU8iLjDmVgt4dHvg1dI/W+icTH3hmovn0G8/iX9fsTtp/IkkoY
tE9PQx1sCvZdZBqxSI8nT/s7BWWBVMwt9KUeF9bCdXS+kESU3f8YygWiaVX0DpdV
q2iJsv8tCn6g12pojA/IMDZ7j14kGitU1Vt0TwBPP44cJK8Ll9lN87CkNjOpDvaS
woAgte3ZHCV51iEcRGAiMJtW8WfurtXdn2lh5Zd3PX83/c/4sb2DrkMRUOy7kV0r
o6peRbYCRcW+svANv/u1r2Wvl1oM
=duha
-----END PGP SIGNATURE-----

L
Ludovic Courtès wrote on Sat Mar 24 18:07:44+0100 2018
Re: Keyboard detection before ‘cryptsetup’ runs
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87po3tjtb3.fsf@gnu.org
Hi Danny!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> (b) Monitor /sys using inotify in an extra thread/process.
>
> Oh - or loop { load-/sys ; sleep 1 }. That's what you meant.

Yeah, though we wouldn’t know when to stop, but if I read your message
correctly, nobody does. :-)

Ludo’.
L
Ludovic Courtès wrote on Sat Mar 24 18:12:52+0100 2018
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)
87h8p5jt2j.fsf@gnu.org
Heya,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> I have to say what we have to do is much more complicated than I thought
> it would be - and I kinda regret starting this patchset. I didn't
> want to make it less likely for people to boot than before :-(

Yeah, I had a nightmare where I dreamed of all the bug reports due to
devices not showing up quickly enough. :-)

> A small bit of research shows that:
>
> - Busybox uses a loop with timeout to try to mount over and over again,
> depending on bus type [2].
> - Alpine coldplugs twice [1].
> - Fatdog has a "waitdev" boot parameter that specifies how long it sleeps
> until it tries to mount the root.
> - Redhat mkinitrd has an mkinitrd option like "--with=scsi_wait_scan" which
> they use after they saw a modalias starting with "scsi:" - or add the
> kernel option "scsi_mod.scan=sync" to the command line [3] (but they say that
> this is unreliable now and one is supposed to use udev inside the initrd).
> Not sure what they do for USB.
>
> The Linux kernel itself has a "rootdelay" parameter which
> was specifically introduced because of USB devices that took 15 s (!)
> to show up.

It’s a kludge fest, which is reassuring, in a way.

> All in all, we have alternative ways to continue:
>
> (a) Drop this patchset/feature entirely because it's too unreliable.
> (b) Monitor /sys using inotify in an extra thread/process.
> (c) Include udev into the initrd and have it do its thing.
> (d) Monitor the netlink socket in our own thread and play udev ourselves.
> (e) Find out how to, for each bus type, detect when enumeration is finished.
> Find out which buses are there. Wait for them to finish enumerating.
> (f) Always modprobe usb-hid first, no matter what. I think this would
> become a whack-a-mole thing fast.
>
> I'd prefer either (a) or (b) for reliability and simplicity. WDYT?

Same here! Sounds like (b) is something we could easily try, though we
won’t know how well it works until people start running it on the bare
metal.

I can adjust the branch I have to do that and push it for people to try.

Thanks a lot of your feedback!

Ludo’.
L
Ludovic Courtès wrote on Mon Mar 26 15:37:01+0200 2018
Re: bug#30922: LUKS-encrypted root fails using device numbering, needs luksUUID
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87efk7dkle.fsf@gnu.org
Hello,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> After booting I get
>
> Device /dev/nvme0n1p1 doesn't exist or access denied

This message comes from ‘cryptsetup’.

The ‘init’ script for the config you posted does this:

--8<---------------cut here---------------start------------->8---
(boot-system #:mounts (map spec->file-system
(quote (("/dev/mapper/guix"
device "/" "ext4" ()
#f #t))))
#:pre-mount (lambda ()
(and (let ((source "/dev/nvme0n1p1"))
(use-modules (rnrs bytevectors)
((gnu build file-systems)
#:select
(find-partition-by-luks-uuid)))
(zero? (system*
"/gnu/store/v5r0dx5v08847rcy033kyish6m3b3621-cryptsetup-static-1.7.5/sbin/cryptsetup"
"open" "--type" "luks"
(if (bytevector? source)
(or (let loop ((tries-left 10))
(and (positive? tries-left)
(or (find-partition-by-luks-uuid source)
(begin
(sleep 1)
(loop (- tries-left 1))))))
(error "LUKS partition not found" source))
source)
"guix")))))

--8<---------------cut here---------------end--------------->8---

IOW, it does the right thing whether or not you’re using a UUID.

However, when using a UUID, it waits for the device to show up, whereas
otherwise it assumes the /dev node is already there and invokes
‘cryptsetup’ right away.

We should fix that and generalize the wait-for-device loop, which
already exists in a couple of instances.

There are connections with <https://bugs.gnu.org/30604>.

Ludo’.
L
Ludovic Courtès wrote on Fri Aug 16 23:10:15+0200 2019
Re: bug#34902: guix cannot find a module on boot
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
874l2gg6ko.fsf@gnu.org
Hello!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> Maybe I'm too paranoid but can we have "guix" in the file name "modules.name"
> somewhere? Otherwise I see it coming that upstream uses modules.name for an
> incompatible purpose and then we'd be with a guix interface that's broken
> and/or break their interface.
>
> (So much complexity for something so silly. Honestly, I feel like E-mailing
> the upstream author and telling him what I think. WTF :P)
>
> Should we warn when we use the fallback? I like the defensive programming
> but I feel we shouldn't have it *silently* fall back when the database is
> broken/missing.
>
> Otherwise LGTM!

So I went ahead and pushed these patches, derived from our beautiful
patch set at <https://issues.guix.gnu.org/issue/30604>:

c85ccf60bf linux-modules: Define and use a module name database.
e1a9a7f275 linux-modules: Add 'load-linux-modules-from-directory'.
2a693b69ca linux-modules: Add "modules.devname" writer.
4f8b9d1a6f linux-modules: Add "modules.alias" writer.

The actual fix for the hyphen/underscore mismatch that Julien reported
is commit c85ccf60bf. The “modules.devname” and “modules.alias” are
actually unused so far but (1) it was easier to preserve them, and (2)
that’ll give us an incentive to finish
<https://issues.guix.gnu.org/issue/30604>. :-)

I added an explicit comment that “modules.name” uses a Guix-specific
format. We can always rename it if the kernel folks decide to acquire
that file name.

Julien, could you please confirm that your initial issue is fixed?

Thanks,
Ludo’.
?
Your comment

Comments via the web interface are not currently supported. To comment on this conversation send email to 30604@debbugs.gnu.org

4 participants
  • Danny Milosavljevic
  • Ludovic Courtès
  • Mathieu Othacehe
  • Marius Bakke
Owner
unassigned
Severity
important
Status
Open