[PATCH 0/4] Load Linux module only when supported hardware is present.

OpenSubmitted by Danny Milosavljevic.

Details

4 participants

Danny Milosavljevic
Ludovic Courtès
Mathieu Othacehe
Marius Bakke

Owner: unassigned

Severity: important

Danny Milosavljevic wrote on 25 Feb 2018 12:45

Recipients:(address . guix-patches@gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180225114557.816-1-dannym@scratchpost.org

Danny Milosavljevic (4):  gnu: kmod: Split off kmod-minimal.  gnu: Add kmod-minimal-static.  linux-initrd: Add kmod.  linux-boot: Load kernel modules only when the hardware is present.
 gnu/build/linux-boot.scm    | 24 ++++++++++-- gnu/build/linux-initrd.scm  | 13 ++++++- gnu/packages/linux.scm      | 91 +++++++++++++++++++++++++++++++++++++-------- gnu/system/linux-initrd.scm | 52 ++++++++++++++++++++------ 4 files changed, 148 insertions(+), 32 deletions(-)

Danny Milosavljevic wrote on 25 Feb 2018 12:48

[PATCH 1/4] gnu: kmod: Split off kmod-minimal.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180225114816.869-1-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal): New variable.(kmod): Modify.--- gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-)

Toggle diff (81 lines)diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 238398e84..1f8bf3050 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as to use Linux' inotify mechanism, which allows file accesses to be monitored.")     (license license:gpl2+))) -(define-public kmod+(define kmod-minimal   (package+    (name "kmod-minimal")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (build-system gnu-build-system)+    (arguments+     `(#:tests? #f)) ; FIXME: Investigate test failures+    (home-page "https://www.kernel.org/")+    (synopsis "Kernel module tools")+    (description "Kmod is a set of tools to handle common tasks with Linux+kernel modules like insert, remove, list, check properties, resolve+dependencies and aliases.++These tools are designed on top of libkmod, a library that is shipped with+kmod.  The aim is to be compatible with tools, configurations and indices+from the module-init-tools project.")+    (license license:gpl2+))) ; library under lgpl2.1+++(define-public kmod+  (package (inherit kmod-minimal)     (name "kmod")     (version "24")     (source (origin@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                (base32                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))               (patches (search-patches "kmod-module-directory.patch"))))-    (build-system gnu-build-system)     (native-inputs      `(("pkg-config" ,pkg-config)))     (inputs      `(("xz" ,xz)        ("zlib" ,zlib)))     (arguments-     `(#:tests? #f ; FIXME: Investigate test failures-       #:configure-flags '("--with-xz" "--with-zlib")+     `(#:configure-flags '("--with-xz" "--with-zlib")+       #:tests? #f ; FIXME: Investigate test failures        #:phases (alist-cons-after                  'install 'install-modprobe&co                  (lambda* (#:key outputs #:allow-other-keys)@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                                  (symlink "kmod"                                           (string-append bin "/" tool)))                                '("insmod" "rmmod" "lsmod" "modprobe"-                                 "modinfo" "depmod"))))-                 %standard-phases)))-    (home-page "https://www.kernel.org/")-    (synopsis "Kernel module tools")-    (description "Kmod is a set of tools to handle common tasks with Linux-kernel modules like insert, remove, list, check properties, resolve-dependencies and aliases.--These tools are designed on top of libkmod, a library that is shipped with-kmod.  The aim is to be compatible with tools, configurations and indices-from the module-init-tools project.")-    (license license:gpl2+))) ; library under lgpl2.1++                                 "modinfo" "depmod"))+                     #t))+                 %standard-phases)))))  (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.

Danny Milosavljevic wrote on 25 Feb 2018 12:48

[PATCH 2/4] gnu: Add kmod-minimal-static.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180225114816.869-2-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal/static): New variable.--- gnu/packages/linux.scm | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)

Toggle diff (55 lines)diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 1f8bf3050..b2e47f79a 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1994,6 +1994,50 @@ from the module-init-tools project.")                      #t))                  %standard-phases))))) +(define-public kmod-minimal/static+  (static-package+   (package (inherit kmod-minimal)+    (name "kmod-minimal-static")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (arguments+     (substitute-keyword-arguments+       (package-arguments (static-package kmod-minimal))+       ((#:configure-flags flags ''())+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))+       ((#:make-flags flags ''())+        `(cons* "LDFLAGS=-all-static" ,flags))+       ((#:phases phases '%standard-phases)+        `(modify-phases ,phases+          (delete 'install-license-files)+          (add-after 'unpack 'patch-kmod+           (lambda _+             ;; Reduce size by 200 kiB.+             (substitute* "tools/kmod.c"+              (("[&]kmod_cmd_compat_lsmod,") "")+              (("[&]kmod_cmd_compat_rmmod,") "")+              (("[&]kmod_cmd_compat_insmod,") "")+              (("[&]kmod_cmd_compat_modinfo,") ""))+             #t))+          (replace 'install+            (lambda* (#:key outputs #:allow-other-keys)+              (let* ((out (assoc-ref outputs "out"))+                     (bin (string-append out "/bin")))+                (install-file "tools/kmod" bin)+                (for-each+                 (lambda (tool)+                   (symlink "kmod" (string-append bin "/" tool)))+                 '("modprobe" "depmod"))+                #t))))))))))+ (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.   (package

Danny Milosavljevic wrote on 25 Feb 2018 12:48

[PATCH 3/4] linux-initrd: Add kmod.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180225114816.869-3-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.(base-initrd): Add kmod.(expression->initrd): Add kmod, linux-module-directory.(flat-linux-module-directory): Add kmod; invoke depmod.* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.--- gnu/build/linux-initrd.scm  | 13 +++++++++++- gnu/system/linux-initrd.scm | 48 ++++++++++++++++++++++++++++++++++----------- 2 files changed, 49 insertions(+), 12 deletions(-)

Toggle diff (154 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..52cc180b4 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init kmod linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (if kmod+        (begin+          (mkdir-p "sbin")+          (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe")))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 7170d1c0e..93089a869 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -59,6 +59,8 @@                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             kmod+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init+                        #:kmod #$kmod+                        #:linux-module-directory #$linux-module-directory                         ;; Copy everything INIT refers to into the initrd.                         #:references-graphs '("closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."   (gexp->derivation name builder                     #:references-graphs `(("closure" ,init)))) -(define (flat-linux-module-directory linux modules)+(define (flat-linux-module-directory linux modules kmod)   "Return a flat directory containing the Linux kernel modules listed in MODULES and taken from LINUX."   (define build-exp@@ -109,7 +113,7 @@ MODULES and taken from LINUX."                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex)+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)                        (srfi srfi-1)                        (guix build utils)                        (gnu build linux-modules))@@ -138,13 +142,30 @@ MODULES and taken from LINUX."                       (recursive-module-dependencies modules                                                      #:lookup-module lookup)))) -          (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))+          (define version+            (car+             (filter+              (lambda (name)+                (not (string-prefix? "." name)))+              (scandir module-dir))))++          (display "VERSION")+          (display version)+          (newline)++          (let ((output (string-append #$output "/lib/modules/" version)))+            (mkdir-p output)+            (for-each (lambda (module)+                        (format #t "copying '~a'...~%" module)+                        (copy-file module+                                   (string-append output "/"+                                                  (basename module))))+                      (delete-duplicates modules)))+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output+                  ; -E+                  "-F" (string-append #$linux "/System.map")+                  version)+          #t)))    (computed-file "linux-modules" build-exp)) @@ -152,6 +173,7 @@ MODULES and taken from LINUX."                       #:key                       (linux linux-libre)                       (linux-modules '())+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       (helper-packages '())                       qemu-networking?@@ -185,7 +207,7 @@ upon error."          mapped-devices))    (define kodir-    (flat-linux-module-directory linux linux-modules))+    (flat-linux-module-directory linux linux-modules kmod))    (expression->initrd    (with-imported-modules (source-module-closure@@ -223,6 +245,8 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:kmod kmod+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))@@ -245,6 +269,7 @@ FILE-SYSTEMS." (define* (base-initrd file-systems                       #:key                       (linux linux-libre)+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       qemu-networking?                       volatile-root?@@ -322,8 +347,9 @@ loaded at boot time in the order in which they appear."   (raw-initrd file-systems               #:linux linux               #:linux-modules linux-modules+              #:kmod kmod               #:mapped-devices mapped-devices-              #:helper-packages helper-packages+              #:helper-packages (cons kmod helper-packages)               #:qemu-networking? qemu-networking?               #:volatile-root? volatile-root?               #:on-error on-error))

Danny Milosavljevic wrote on 25 Feb 2018 12:48

[PATCH 4/4] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180225114816.869-4-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 24 ++++++++++++++++++++---- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 23 insertions(+), 5 deletions(-)

Toggle diff (57 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..65c91c50f 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -473,6 +473,25 @@ upon error."     (string-append linux-module-directory "/"                    (ensure-dot-ko name))) +  (define (load-kernel-modules)+    (define enter?+      (const #t))+    (file-system-fold+     enter?+     ;; This is the Leaf handler.  It tries to modprobe all the modaliases.+     (lambda (path stat result) ; leaf+       (let ((modalias-name (string-append path "/modalias")))+         (if (file-exists? modalias-name)+             (let ((modalias (call-with-input-file modalias-name read-string)))+               (if (not (string=? modalias ""))+                   (system* "/sbin/modprobe" "-q" "--" modalias))))))+     (const #t) ; down+     (const #t) ; up+     (const #f) ; skip+     (const #f) ; error+     #f ; init+     "/sys/devices"))+   (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n") @@ -486,10 +505,7 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 93089a869..573f4a324 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -228,7 +228,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Mathieu Othacehe wrote on 25 Feb 2018 15:05

Re: [bug#30604] [PATCH 3/4] linux-initrd: Add kmod.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87vaelf9ln.fsf@gmail.com

Hi Danny,
Here are some small remarks,

Toggle quote (4 lines)

> + ;; Make modprobe available as /sbin/modprobe so the kernel finds it.> + (if kmod> + (begin

You can use "when" to avoid "begin".

Toggle quote (3 lines)

> + (define version> + (car

Prefer match to car/cdr use.

Toggle quote (9 lines)

> + (filter> + (lambda (name)> + (not (string-prefix? "." name)))> + (scandir module-dir))))> +> + (display "VERSION")> + (display version)> + (newline)

(format #t "VERSION" ~a~%" version) would be shorter.

Toggle quote (2 lines)

> + ; -E

Why is this commented ?

Toggle quote (5 lines)

> + "-F" (string-append #$linux "/System.map")> + version)> + #t)))>

Thanks,
Mathieu

Danny Milosavljevic wrote on 25 Feb 2018 16:07

Recipients:(name . Mathieu Othacehe)(address . m.othacehe@gmail.com)(address . 30604@debbugs.gnu.org)

Message-ID:20180225160750.46bf5b50@scratchpost.org

Hi Mathieu,
thanks for the review.
On Sun, 25 Feb 2018 15:05:24 +0100Mathieu Othacehe <m.othacehe@gmail.com> wrote:

Toggle quote (4 lines)

> > + ; -E > > Why is this commented ?

Because it's an option for specifying the location of "Module.symvers" - andI don't know whether guix uses it (probably not).  If one doesn't specify anoption, depmod will default to the running kernel - which is not what we want.
I should elaborate in the comment that, if we start using Module.symvers, weMUST pass "-E" there.  Maybe better to even just check for the file existenceand add it right now, otherwise we might forget later.  What do you think?

Marius Bakke wrote on 26 Feb 2018 02:10

Re: [bug#30604] [PATCH 1/4] gnu: kmod: Split off kmod-minimal.

Recipients:

Message-ID:87h8q4wo7d.fsf@fastmail.com

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (20 lines)> * gnu/packages/linux.scm (kmod-minimal): New variable.> (kmod): Modify.> --->  gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++---------------->  1 file changed, 33 insertions(+), 16 deletions(-)>> diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm> index 238398e84..1f8bf3050 100644> --- a/gnu/packages/linux.scm> +++ b/gnu/packages/linux.scm> @@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as>  to use Linux' inotify mechanism, which allows file accesses to be monitored.")>      (license license:gpl2+)))>  > -(define-public kmod> +(define kmod-minimal>    (package> +    (name "kmod-minimal")> +    (version "13")

Why this old version?

Toggle quote (10 lines)> +    (source (origin> +              (method url-fetch)> +              (uri> +               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"> +                              "kmod-" version ".tar.xz"))> +              (sha256> +               (base32> +                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))> +              (patches (search-patches "kmod-13-module-directory.patch"))))

This patch seems to be missing.

Toggle quote (59 lines)> +    (build-system gnu-build-system)> +    (arguments> +     `(#:tests? #f)) ; FIXME: Investigate test failures> +    (home-page "https://www.kernel.org/")> +    (synopsis "Kernel module tools")> +    (description "Kmod is a set of tools to handle common tasks with Linux> +kernel modules like insert, remove, list, check properties, resolve> +dependencies and aliases.> +> +These tools are designed on top of libkmod, a library that is shipped with> +kmod.  The aim is to be compatible with tools, configurations and indices> +from the module-init-tools project.")> +    (license license:gpl2+))) ; library under lgpl2.1+> +> +(define-public kmod> +  (package (inherit kmod-minimal)>      (name "kmod")>      (version "24")>      (source (origin> @@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")>                 (base32>                  "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))>                (patches (search-patches "kmod-module-directory.patch"))))> -    (build-system gnu-build-system)>      (native-inputs>       `(("pkg-config" ,pkg-config)))>      (inputs>       `(("xz" ,xz)>         ("zlib" ,zlib)))>      (arguments> -     `(#:tests? #f ; FIXME: Investigate test failures> -       #:configure-flags '("--with-xz" "--with-zlib")> +     `(#:configure-flags '("--with-xz" "--with-zlib")> +       #:tests? #f ; FIXME: Investigate test failures>         #:phases (alist-cons-after>                   'install 'install-modprobe&co>                   (lambda* (#:key outputs #:allow-other-keys)> @@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")>                                   (symlink "kmod">                                            (string-append bin "/" tool)))>                                 '("insmod" "rmmod" "lsmod" "modprobe"> -                                 "modinfo" "depmod"))))> -                 %standard-phases)))> -    (home-page "https://www.kernel.org/")> -    (synopsis "Kernel module tools")> -    (description "Kmod is a set of tools to handle common tasks with Linux> -kernel modules like insert, remove, list, check properties, resolve> -dependencies and aliases.> -> -These tools are designed on top of libkmod, a library that is shipped with> -kmod.  The aim is to be compatible with tools, configurations and indices> -from the module-init-tools project.")> -    (license license:gpl2+))) ; library under lgpl2.1+> +                                 "modinfo" "depmod"))> +                     #t))> +                 %standard-phases)))))>  >  (define-public eudev>    ;; The post-systemd fork, maintained by Gentoo.

Danny Milosavljevic wrote on 26 Feb 2018 04:50

[PATCH v2 1/6] gnu: kmod: Split off kmod-minimal.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226035025.1698-2-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal): New variable.(kmod): Modify.--- gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-)

Toggle diff (81 lines)diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 238398e84..1f8bf3050 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as to use Linux' inotify mechanism, which allows file accesses to be monitored.")     (license license:gpl2+))) -(define-public kmod+(define kmod-minimal   (package+    (name "kmod-minimal")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (build-system gnu-build-system)+    (arguments+     `(#:tests? #f)) ; FIXME: Investigate test failures+    (home-page "https://www.kernel.org/")+    (synopsis "Kernel module tools")+    (description "Kmod is a set of tools to handle common tasks with Linux+kernel modules like insert, remove, list, check properties, resolve+dependencies and aliases.++These tools are designed on top of libkmod, a library that is shipped with+kmod.  The aim is to be compatible with tools, configurations and indices+from the module-init-tools project.")+    (license license:gpl2+))) ; library under lgpl2.1+++(define-public kmod+  (package (inherit kmod-minimal)     (name "kmod")     (version "24")     (source (origin@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                (base32                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))               (patches (search-patches "kmod-module-directory.patch"))))-    (build-system gnu-build-system)     (native-inputs      `(("pkg-config" ,pkg-config)))     (inputs      `(("xz" ,xz)        ("zlib" ,zlib)))     (arguments-     `(#:tests? #f ; FIXME: Investigate test failures-       #:configure-flags '("--with-xz" "--with-zlib")+     `(#:configure-flags '("--with-xz" "--with-zlib")+       #:tests? #f ; FIXME: Investigate test failures        #:phases (alist-cons-after                  'install 'install-modprobe&co                  (lambda* (#:key outputs #:allow-other-keys)@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                                  (symlink "kmod"                                           (string-append bin "/" tool)))                                '("insmod" "rmmod" "lsmod" "modprobe"-                                 "modinfo" "depmod"))))-                 %standard-phases)))-    (home-page "https://www.kernel.org/")-    (synopsis "Kernel module tools")-    (description "Kmod is a set of tools to handle common tasks with Linux-kernel modules like insert, remove, list, check properties, resolve-dependencies and aliases.--These tools are designed on top of libkmod, a library that is shipped with-kmod.  The aim is to be compatible with tools, configurations and indices-from the module-init-tools project.")-    (license license:gpl2+))) ; library under lgpl2.1++                                 "modinfo" "depmod"))+                     #t))+                 %standard-phases)))))  (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.

Danny Milosavljevic wrote on 26 Feb 2018 04:50

[PATCH v2 2/6] gnu: Add kmod-minimal-static.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226035025.1698-3-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal/static): New variable.* gnu/packages/patches/kmod-13-module-directory.patch: New file.* gnu/local.mk: Add it.--- gnu/local.mk                                       |  1 + gnu/packages/linux.scm                             | 44 ++++++++++++++++++++++ .../patches/kmod-13-module-directory.patch         | 33 ++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

Toggle diff (106 lines)diff --git a/gnu/local.mk b/gnu/local.mkindex 21195f8c1..b1e3c878d 100644--- a/gnu/local.mk+++ b/gnu/local.mk@@ -795,6 +795,7 @@ dist_patch_DATA =						\   %D%/packages/patches/kiki-makefile.patch			\   %D%/packages/patches/kiki-missing-includes.patch		\   %D%/packages/patches/kiki-portability-64bit.patch		\+  %D%/packages/patches/kmod-13-module-directory.patch		\   %D%/packages/patches/kmod-module-directory.patch		\   %D%/packages/patches/kobodeluxe-paths.patch			\   %D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch	\diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 1f8bf3050..b2e47f79a 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1994,6 +1994,50 @@ from the module-init-tools project.")                      #t))                  %standard-phases))))) +(define-public kmod-minimal/static+  (static-package+   (package (inherit kmod-minimal)+    (name "kmod-minimal-static")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (arguments+     (substitute-keyword-arguments+       (package-arguments (static-package kmod-minimal))+       ((#:configure-flags flags ''())+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))+       ((#:make-flags flags ''())+        `(cons* "LDFLAGS=-all-static" ,flags))+       ((#:phases phases '%standard-phases)+        `(modify-phases ,phases+          (delete 'install-license-files)+          (add-after 'unpack 'patch-kmod+           (lambda _+             ;; Reduce size by 200 kiB.+             (substitute* "tools/kmod.c"+              (("[&]kmod_cmd_compat_lsmod,") "")+              (("[&]kmod_cmd_compat_rmmod,") "")+              (("[&]kmod_cmd_compat_insmod,") "")+              (("[&]kmod_cmd_compat_modinfo,") ""))+             #t))+          (replace 'install+            (lambda* (#:key outputs #:allow-other-keys)+              (let* ((out (assoc-ref outputs "out"))+                     (bin (string-append out "/bin")))+                (install-file "tools/kmod" bin)+                (for-each+                 (lambda (tool)+                   (symlink "kmod" (string-append bin "/" tool)))+                 '("modprobe" "depmod"))+                #t))))))))))+ (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.   (packagediff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patchnew file mode 100644index 000000000..5ff2f8a60--- /dev/null+++ b/gnu/packages/patches/kmod-13-module-directory.patch@@ -0,0 +1,33 @@+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'+environment variable, rather than looking for modules exclusively in+/lib/modules.++Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable+name.+++--- kmod-7/libkmod/libkmod.c	2012-03-15 08:19:16.750010226 -0400++++ kmod-7/libkmod/libkmod.c	2012-04-04 15:21:29.532074313 -0400+@@ -200,7 +200,7 @@+ static char *get_kernel_release(const char *dirname)+ {+ 	struct utsname u;+-	char *p;++	char *p, *dirname_prefix;+ + 	if (dirname != NULL)+ 		return path_make_absolute_cwd(dirname);+@@ -208,7 +208,10 @@+ 	if (uname(&u) < 0)+ 		return NULL;+ +-	if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)++	if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)++		dirname_prefix = dirname_default_prefix;++++	if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)+ 		return NULL;+ + 	return p;+

Danny Milosavljevic wrote on 26 Feb 2018 04:50

[PATCH v2 0/6] Load Linux module only when supported hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226035025.1698-1-dannym@scratchpost.org

Danny Milosavljevic (6):  gnu: kmod: Split off kmod-minimal.  gnu: Add kmod-minimal-static.  linux-initrd: Add kmod.  linux-boot: Load kernel modules only when the hardware is present.  vm: Allow qemu-image builder to load Linux kernel modules.  vm: Make the virtio-blk is uniquely identifyable in /sys.
 gnu/build/linux-boot.scm                           | 31 +++++++- gnu/build/linux-initrd.scm                         | 12 ++- gnu/build/vm.scm                                   |  2 +- gnu/local.mk                                       |  1 + gnu/packages/linux.scm                             | 91 ++++++++++++++++++---- .../patches/kmod-13-module-directory.patch         | 33 ++++++++ gnu/system/linux-initrd.scm                        | 48 +++++++++--- gnu/system/vm.scm                                  | 28 ++++++- 8 files changed, 210 insertions(+), 36 deletions(-) create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

Danny Milosavljevic wrote on 26 Feb 2018 04:50

[PATCH v2 3/6] linux-initrd: Add kmod.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226035025.1698-4-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.(base-initrd): Add kmod.(expression->initrd): Add kmod, linux-module-directory.(flat-linux-module-directory): Add kmod; invoke depmod.* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.--- gnu/build/linux-initrd.scm  | 12 +++++++++++- gnu/system/linux-initrd.scm | 44 +++++++++++++++++++++++++++++++++----------- 2 files changed, 44 insertions(+), 12 deletions(-)

Toggle diff (149 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..6356007df 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init kmod linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (when kmod+      (mkdir-p "sbin")+      (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 7170d1c0e..91e498787 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -59,6 +59,8 @@                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             kmod+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init+                        #:kmod #$kmod+                        #:linux-module-directory #$linux-module-directory                         ;; Copy everything INIT refers to into the initrd.                         #:references-graphs '("closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."   (gexp->derivation name builder                     #:references-graphs `(("closure" ,init)))) -(define (flat-linux-module-directory linux modules)+(define (flat-linux-module-directory linux modules kmod)   "Return a flat directory containing the Linux kernel modules listed in MODULES and taken from LINUX."   (define build-exp@@ -109,7 +113,7 @@ MODULES and taken from LINUX."                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex)+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)                        (srfi srfi-1)                        (guix build utils)                        (gnu build linux-modules))@@ -138,13 +142,26 @@ MODULES and taken from LINUX."                       (recursive-module-dependencies modules                                                      #:lookup-module lookup)))) -          (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))+          (define version+            (car+             (filter+              (lambda (name)+                (not (string-prefix? "." name)))+              (scandir module-dir))))++          (let ((output (string-append #$output "/lib/modules/" version)))+            (mkdir-p output)+            (for-each (lambda (module)+                        (format #t "copying '~a'...~%" module)+                        (copy-file module+                                   (string-append output "/"+                                                  (basename module))))+                      (delete-duplicates modules)))+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output+                  ; -E+                  "-F" (string-append #$linux "/System.map")+                  version)+          #t)))    (computed-file "linux-modules" build-exp)) @@ -152,6 +169,7 @@ MODULES and taken from LINUX."                       #:key                       (linux linux-libre)                       (linux-modules '())+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       (helper-packages '())                       qemu-networking?@@ -185,7 +203,7 @@ upon error."          mapped-devices))    (define kodir-    (flat-linux-module-directory linux linux-modules))+    (flat-linux-module-directory linux linux-modules kmod))    (expression->initrd    (with-imported-modules (source-module-closure@@ -223,6 +241,8 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:kmod kmod+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))@@ -245,6 +265,7 @@ FILE-SYSTEMS." (define* (base-initrd file-systems                       #:key                       (linux linux-libre)+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       qemu-networking?                       volatile-root?@@ -322,8 +343,9 @@ loaded at boot time in the order in which they appear."   (raw-initrd file-systems               #:linux linux               #:linux-modules linux-modules+              #:kmod kmod               #:mapped-devices mapped-devices-              #:helper-packages helper-packages+              #:helper-packages (cons kmod helper-packages)               #:qemu-networking? qemu-networking?               #:volatile-root? volatile-root?               #:on-error on-error))

Danny Milosavljevic wrote on 26 Feb 2018 04:50

[PATCH v2 4/6] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226035025.1698-5-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 31 +++++++++++++++++++++++++++---- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 30 insertions(+), 5 deletions(-)

Toggle diff (64 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..6d00ea9be 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -473,6 +473,32 @@ upon error."     (string-append linux-module-directory "/"                    (ensure-dot-ko name))) +  (define (load-kernel-modules)+    "Examine /sys/devices to find out which modules to load and load them."+    (define enter?+      (const #t))+    (define (down! path stat result)+     ;; Note: modprobe mutates the tree starting with path.+     (let ((modalias-name (string-append path "/modalias")))+       (if (file-exists? modalias-name)+           (let ((modalias+                 (string-trim-right (call-with-input-file modalias-name+                                                          read-string)+                                    #\newline)))+             (system* "/sbin/modprobe" "-q" "--" modalias))))+       #t)+    (define up+      (const #t))+    (define skip+      (const #t))+    (define leaf+      (const #t))+    (define (error name stat errno result)+      (format (current-error-port) "warning: ~a: ~a~%"+              name (strerror errno))+      result)+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))+   (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n") @@ -486,10 +512,7 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 91e498787..6d130ccac 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -224,7 +224,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Danny Milosavljevic wrote on 26 Feb 2018 04:50

[PATCH v2 5/6] vm: Allow qemu-image builder to load Linux kernel modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226035025.1698-6-dannym@scratchpost.org

* gnu/system/vm.scm (%modprobe-wrapper): New variable.(qemu-image): Modify.--- gnu/system/vm.scm | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)

Toggle diff (64 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 345cecedd..c64c9678f 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."    #:single-file-output? #t    #:references-graphs inputs)) +(define (%modprobe-wrapper modprobe linux-module-directory)+  ;; Wrapper for the 'modprobe' command that knows where modules live.+  ;;+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'+  ;; environment variable is not set---hence the need for this wrapper.+  (program-file "modprobe"+    #~(begin+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))+ (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))@@ -275,6 +286,8 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))   (expression->derivation-in-linux-vm    name    (with-imported-modules (source-module-closure '((gnu build bootloader)@@ -288,7 +301,7 @@ the image."                       (ice-9 binary-ports))           (let ((inputs-                '#$(append (list qemu parted e2fsprogs dosfstools)+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)                            (map canonical-package                                 (list sed grep coreutils findutils gawk))                            (if register-closures? (list guix) '())))@@ -302,6 +315,14 @@ the image."                         inputs)))             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)+           ;; It's possible that we need to load nls modules in order to+           ;; mount the new partition.+           (if (file-exists? #$modprobe-name)+               (activate-modprobe #$(%modprobe-wrapper modprobe-name+                                     linux-module-directory))+               (format (current-error-port)+                "WARNING: No modprobe found in ~s.  \+Loading kernel modules will be impossible.\n" #$modprobe-name))             (let* ((graphs     '#$(match inputs                                    (((names . _) ...)@@ -364,7 +385,7 @@ the image."    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-   #:references-graphs inputs))+   #:references-graphs inputs)))   ;;;

Danny Milosavljevic wrote on 26 Feb 2018 04:50

[PATCH v2 6/6] vm: Make the virtio-blk is uniquely identifyable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226035025.1698-7-dannym@scratchpost.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.--- gnu/build/vm.scm  | 2 +- gnu/system/vm.scm | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..ebf9e9f6e 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex c64c9678f..ad48999ee 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -704,7 +704,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os

Danny Milosavljevic wrote on 26 Feb 2018 05:06

[PATCH v3 1/6] gnu: kmod: Split off kmod-minimal.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226040609.3066-2-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal): New variable.(kmod): Modify.--- gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-)

Toggle diff (81 lines)diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 238398e84..1f8bf3050 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as to use Linux' inotify mechanism, which allows file accesses to be monitored.")     (license license:gpl2+))) -(define-public kmod+(define kmod-minimal   (package+    (name "kmod-minimal")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (build-system gnu-build-system)+    (arguments+     `(#:tests? #f)) ; FIXME: Investigate test failures+    (home-page "https://www.kernel.org/")+    (synopsis "Kernel module tools")+    (description "Kmod is a set of tools to handle common tasks with Linux+kernel modules like insert, remove, list, check properties, resolve+dependencies and aliases.++These tools are designed on top of libkmod, a library that is shipped with+kmod.  The aim is to be compatible with tools, configurations and indices+from the module-init-tools project.")+    (license license:gpl2+))) ; library under lgpl2.1+++(define-public kmod+  (package (inherit kmod-minimal)     (name "kmod")     (version "24")     (source (origin@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                (base32                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))               (patches (search-patches "kmod-module-directory.patch"))))-    (build-system gnu-build-system)     (native-inputs      `(("pkg-config" ,pkg-config)))     (inputs      `(("xz" ,xz)        ("zlib" ,zlib)))     (arguments-     `(#:tests? #f ; FIXME: Investigate test failures-       #:configure-flags '("--with-xz" "--with-zlib")+     `(#:configure-flags '("--with-xz" "--with-zlib")+       #:tests? #f ; FIXME: Investigate test failures        #:phases (alist-cons-after                  'install 'install-modprobe&co                  (lambda* (#:key outputs #:allow-other-keys)@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                                  (symlink "kmod"                                           (string-append bin "/" tool)))                                '("insmod" "rmmod" "lsmod" "modprobe"-                                 "modinfo" "depmod"))))-                 %standard-phases)))-    (home-page "https://www.kernel.org/")-    (synopsis "Kernel module tools")-    (description "Kmod is a set of tools to handle common tasks with Linux-kernel modules like insert, remove, list, check properties, resolve-dependencies and aliases.--These tools are designed on top of libkmod, a library that is shipped with-kmod.  The aim is to be compatible with tools, configurations and indices-from the module-init-tools project.")-    (license license:gpl2+))) ; library under lgpl2.1++                                 "modinfo" "depmod"))+                     #t))+                 %standard-phases)))))  (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.

Danny Milosavljevic wrote on 26 Feb 2018 05:06

[PATCH v3 0/6] Load Linux module only when supported hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226040609.3066-1-dannym@scratchpost.org

Danny Milosavljevic (6):  gnu: kmod: Split off kmod-minimal.  gnu: Add kmod-minimal-static.  linux-initrd: Add kmod.  linux-boot: Load kernel modules only when the hardware is present.  vm: Allow qemu-image builder to load Linux kernel modules.  vm: Make the virtio-blk is uniquely identifyable in /sys.
 gnu/build/linux-boot.scm                           | 31 +++++++- gnu/build/linux-initrd.scm                         | 12 ++- gnu/build/vm.scm                                   |  2 +- gnu/local.mk                                       |  1 + gnu/packages/linux.scm                             | 91 ++++++++++++++++++---- .../patches/kmod-13-module-directory.patch         | 33 ++++++++ gnu/system/linux-initrd.scm                        | 49 +++++++++--- gnu/system/vm.scm                                  | 34 ++++++-- 8 files changed, 215 insertions(+), 38 deletions(-) create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

Danny Milosavljevic wrote on 26 Feb 2018 05:06

[PATCH v3 2/6] gnu: Add kmod-minimal-static.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226040609.3066-3-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal/static): New variable.* gnu/packages/patches/kmod-13-module-directory.patch: New file.* gnu/local.mk: Add it.--- gnu/local.mk                                       |  1 + gnu/packages/linux.scm                             | 44 ++++++++++++++++++++++ .../patches/kmod-13-module-directory.patch         | 33 ++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

Toggle diff (106 lines)diff --git a/gnu/local.mk b/gnu/local.mkindex 21195f8c1..b1e3c878d 100644--- a/gnu/local.mk+++ b/gnu/local.mk@@ -795,6 +795,7 @@ dist_patch_DATA =						\   %D%/packages/patches/kiki-makefile.patch			\   %D%/packages/patches/kiki-missing-includes.patch		\   %D%/packages/patches/kiki-portability-64bit.patch		\+  %D%/packages/patches/kmod-13-module-directory.patch		\   %D%/packages/patches/kmod-module-directory.patch		\   %D%/packages/patches/kobodeluxe-paths.patch			\   %D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch	\diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 1f8bf3050..b2e47f79a 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1994,6 +1994,50 @@ from the module-init-tools project.")                      #t))                  %standard-phases))))) +(define-public kmod-minimal/static+  (static-package+   (package (inherit kmod-minimal)+    (name "kmod-minimal-static")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (arguments+     (substitute-keyword-arguments+       (package-arguments (static-package kmod-minimal))+       ((#:configure-flags flags ''())+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))+       ((#:make-flags flags ''())+        `(cons* "LDFLAGS=-all-static" ,flags))+       ((#:phases phases '%standard-phases)+        `(modify-phases ,phases+          (delete 'install-license-files)+          (add-after 'unpack 'patch-kmod+           (lambda _+             ;; Reduce size by 200 kiB.+             (substitute* "tools/kmod.c"+              (("[&]kmod_cmd_compat_lsmod,") "")+              (("[&]kmod_cmd_compat_rmmod,") "")+              (("[&]kmod_cmd_compat_insmod,") "")+              (("[&]kmod_cmd_compat_modinfo,") ""))+             #t))+          (replace 'install+            (lambda* (#:key outputs #:allow-other-keys)+              (let* ((out (assoc-ref outputs "out"))+                     (bin (string-append out "/bin")))+                (install-file "tools/kmod" bin)+                (for-each+                 (lambda (tool)+                   (symlink "kmod" (string-append bin "/" tool)))+                 '("modprobe" "depmod"))+                #t))))))))))+ (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.   (packagediff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patchnew file mode 100644index 000000000..5ff2f8a60--- /dev/null+++ b/gnu/packages/patches/kmod-13-module-directory.patch@@ -0,0 +1,33 @@+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'+environment variable, rather than looking for modules exclusively in+/lib/modules.++Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable+name.+++--- kmod-7/libkmod/libkmod.c	2012-03-15 08:19:16.750010226 -0400++++ kmod-7/libkmod/libkmod.c	2012-04-04 15:21:29.532074313 -0400+@@ -200,7 +200,7 @@+ static char *get_kernel_release(const char *dirname)+ {+ 	struct utsname u;+-	char *p;++	char *p, *dirname_prefix;+ + 	if (dirname != NULL)+ 		return path_make_absolute_cwd(dirname);+@@ -208,7 +208,10 @@+ 	if (uname(&u) < 0)+ 		return NULL;+ +-	if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)++	if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)++		dirname_prefix = dirname_default_prefix;++++	if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)+ 		return NULL;+ + 	return p;+

Danny Milosavljevic wrote on 26 Feb 2018 05:06

[PATCH v3 3/6] linux-initrd: Add kmod.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226040609.3066-4-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.(base-initrd): Add kmod.(expression->initrd): Add kmod, linux-module-directory.(flat-linux-module-directory): Add kmod; invoke depmod.* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.--- gnu/build/linux-initrd.scm  | 12 +++++++++++- gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++----------- 2 files changed, 45 insertions(+), 12 deletions(-)

Toggle diff (150 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..6356007df 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init kmod linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (when kmod+      (mkdir-p "sbin")+      (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 7170d1c0e..46ef055f0 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -59,6 +59,8 @@                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             kmod+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init+                        #:kmod #$kmod+                        #:linux-module-directory #$linux-module-directory                         ;; Copy everything INIT refers to into the initrd.                         #:references-graphs '("closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."   (gexp->derivation name builder                     #:references-graphs `(("closure" ,init)))) -(define (flat-linux-module-directory linux modules)+(define (flat-linux-module-directory linux modules kmod)   "Return a flat directory containing the Linux kernel modules listed in MODULES and taken from LINUX."   (define build-exp@@ -109,7 +113,7 @@ MODULES and taken from LINUX."                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex)+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)                        (srfi srfi-1)                        (guix build utils)                        (gnu build linux-modules))@@ -138,13 +142,27 @@ MODULES and taken from LINUX."                       (recursive-module-dependencies modules                                                      #:lookup-module lookup)))) -          (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))+          (define version+            (match+             (filter+              (lambda (name)+                (not (string-prefix? "." name)))+              (scandir module-dir))+             ((item) item)))++          (let ((output (string-append #$output "/lib/modules/" version)))+            (mkdir-p output)+            (for-each (lambda (module)+                        (format #t "copying '~a'...~%" module)+                        (copy-file module+                                   (string-append output "/"+                                                  (basename module))))+                      (delete-duplicates modules)))+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output+                  ; -E+                  "-F" (string-append #$linux "/System.map")+                  version)+          #t)))    (computed-file "linux-modules" build-exp)) @@ -152,6 +170,7 @@ MODULES and taken from LINUX."                       #:key                       (linux linux-libre)                       (linux-modules '())+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       (helper-packages '())                       qemu-networking?@@ -185,7 +204,7 @@ upon error."          mapped-devices))    (define kodir-    (flat-linux-module-directory linux linux-modules))+    (flat-linux-module-directory linux linux-modules kmod))    (expression->initrd    (with-imported-modules (source-module-closure@@ -223,6 +242,8 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:kmod kmod+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))@@ -245,6 +266,7 @@ FILE-SYSTEMS." (define* (base-initrd file-systems                       #:key                       (linux linux-libre)+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       qemu-networking?                       volatile-root?@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."   (raw-initrd file-systems               #:linux linux               #:linux-modules linux-modules+              #:kmod kmod               #:mapped-devices mapped-devices-              #:helper-packages helper-packages+              #:helper-packages (cons kmod helper-packages)               #:qemu-networking? qemu-networking?               #:volatile-root? volatile-root?               #:on-error on-error))

Danny Milosavljevic wrote on 26 Feb 2018 05:06

[PATCH v3 4/6] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226040609.3066-5-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 31 +++++++++++++++++++++++++++---- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 30 insertions(+), 5 deletions(-)

Toggle diff (64 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..6d00ea9be 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -473,6 +473,32 @@ upon error."     (string-append linux-module-directory "/"                    (ensure-dot-ko name))) +  (define (load-kernel-modules)+    "Examine /sys/devices to find out which modules to load and load them."+    (define enter?+      (const #t))+    (define (down! path stat result)+     ;; Note: modprobe mutates the tree starting with path.+     (let ((modalias-name (string-append path "/modalias")))+       (if (file-exists? modalias-name)+           (let ((modalias+                 (string-trim-right (call-with-input-file modalias-name+                                                          read-string)+                                    #\newline)))+             (system* "/sbin/modprobe" "-q" "--" modalias))))+       #t)+    (define up+      (const #t))+    (define skip+      (const #t))+    (define leaf+      (const #t))+    (define (error name stat errno result)+      (format (current-error-port) "warning: ~a: ~a~%"+              name (strerror errno))+      result)+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))+   (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n") @@ -486,10 +512,7 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 46ef055f0..c8a9e4950 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -225,7 +225,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Danny Milosavljevic wrote on 26 Feb 2018 05:06

[PATCH v3 5/6] vm: Allow qemu-image builder to load Linux kernel modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226040609.3066-6-dannym@scratchpost.org

* gnu/system/vm.scm (%modprobe-wrapper): New variable.(qemu-image): Modify.--- gnu/system/vm.scm | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)

Toggle diff (74 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 345cecedd..b5a559012 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."    #:single-file-output? #t    #:references-graphs inputs)) +(define (%modprobe-wrapper modprobe linux-module-directory)+  ;; Wrapper for the 'modprobe' command that knows where modules live.+  ;;+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'+  ;; environment variable is not set---hence the need for this wrapper.+  (program-file "modprobe"+    #~(begin+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))+ (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))   (expression->derivation-in-linux-vm    name-   (with-imported-modules (source-module-closure '((gnu build bootloader)+   (with-imported-modules (source-module-closure '((gnu build activation)+                                                   (gnu build bootloader)                                                    (gnu build vm)                                                    (guix build utils)))      #~(begin-         (use-modules (gnu build bootloader)+         (use-modules (gnu build activation)+                      (gnu build bootloader)                       (gnu build vm)                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports))           (let ((inputs-                '#$(append (list qemu parted e2fsprogs dosfstools)+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)                            (map canonical-package                                 (list sed grep coreutils findutils gawk))                            (if register-closures? (list guix) '())))@@ -302,6 +317,14 @@ the image."                         inputs)))             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)+           ;; It's possible that we need to load nls modules in order to+           ;; mount the new partition.+           (if (file-exists? #$modprobe-name)+               (activate-modprobe #$(%modprobe-wrapper modprobe-name+                                     linux-module-directory))+               (format (current-error-port)+                "WARNING: No modprobe found in ~s.  \+Loading kernel modules will be impossible.\n" #$modprobe-name))             (let* ((graphs     '#$(match inputs                                    (((names . _) ...)@@ -364,7 +387,7 @@ the image."    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-   #:references-graphs inputs))+   #:references-graphs inputs)))   ;;;

Danny Milosavljevic wrote on 26 Feb 2018 05:06

[PATCH v3 6/6] vm: Make the virtio-blk is uniquely identifyable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180226040609.3066-7-dannym@scratchpost.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.--- gnu/build/vm.scm  | 2 +- gnu/system/vm.scm | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..ebf9e9f6e 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex b5a559012..fdff64ed9 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os

Mathieu Othacehe wrote on 26 Feb 2018 12:51

Re: [bug#30604] [PATCH 3/4] linux-initrd: Add kmod.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87h8q4j7e0.fsf@gmail.com

Hey Danny,

Toggle quote (8 lines)

> Because it's an option for specifying the location of "Module.symvers" - and> I don't know whether guix uses it (probably not). If one doesn't specify an> option, depmod will default to the running kernel - which is not what we want.>> I should elaborate in the comment that, if we start using Module.symvers, we> MUST pass "-E" there. Maybe better to even just check for the file existence> and add it right now, otherwise we might forget later. What do you think?

Reading 'depmod' manpage, I understand that -E and -F are mutuallyexclusive. Both options seem to have an interest only if -e is suppliedto "reports any symbols which a module needs which are not supplied byother modules or the kernel".
So maybe something like "-F (string-append #$linux "/System.map") -e"would make more sense ?
Mathieu

Danny Milosavljevic wrote on 27 Feb 2018 12:26

[PATCH v4 1/7] gnu: kmod: Split off kmod-minimal.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227112619.5071-2-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal): New variable.(kmod): Modify.--- gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-)

Toggle diff (81 lines)diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 238398e84..1f8bf3050 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as to use Linux' inotify mechanism, which allows file accesses to be monitored.")     (license license:gpl2+))) -(define-public kmod+(define kmod-minimal   (package+    (name "kmod-minimal")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (build-system gnu-build-system)+    (arguments+     `(#:tests? #f)) ; FIXME: Investigate test failures+    (home-page "https://www.kernel.org/")+    (synopsis "Kernel module tools")+    (description "Kmod is a set of tools to handle common tasks with Linux+kernel modules like insert, remove, list, check properties, resolve+dependencies and aliases.++These tools are designed on top of libkmod, a library that is shipped with+kmod.  The aim is to be compatible with tools, configurations and indices+from the module-init-tools project.")+    (license license:gpl2+))) ; library under lgpl2.1+++(define-public kmod+  (package (inherit kmod-minimal)     (name "kmod")     (version "24")     (source (origin@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                (base32                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))               (patches (search-patches "kmod-module-directory.patch"))))-    (build-system gnu-build-system)     (native-inputs      `(("pkg-config" ,pkg-config)))     (inputs      `(("xz" ,xz)        ("zlib" ,zlib)))     (arguments-     `(#:tests? #f ; FIXME: Investigate test failures-       #:configure-flags '("--with-xz" "--with-zlib")+     `(#:configure-flags '("--with-xz" "--with-zlib")+       #:tests? #f ; FIXME: Investigate test failures        #:phases (alist-cons-after                  'install 'install-modprobe&co                  (lambda* (#:key outputs #:allow-other-keys)@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                                  (symlink "kmod"                                           (string-append bin "/" tool)))                                '("insmod" "rmmod" "lsmod" "modprobe"-                                 "modinfo" "depmod"))))-                 %standard-phases)))-    (home-page "https://www.kernel.org/")-    (synopsis "Kernel module tools")-    (description "Kmod is a set of tools to handle common tasks with Linux-kernel modules like insert, remove, list, check properties, resolve-dependencies and aliases.--These tools are designed on top of libkmod, a library that is shipped with-kmod.  The aim is to be compatible with tools, configurations and indices-from the module-init-tools project.")-    (license license:gpl2+))) ; library under lgpl2.1++                                 "modinfo" "depmod"))+                     #t))+                 %standard-phases)))))  (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.

Danny Milosavljevic wrote on 27 Feb 2018 12:26

[PATCH v4 2/7] gnu: Add kmod-minimal-static.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227112619.5071-3-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal/static): New variable.* gnu/packages/patches/kmod-13-module-directory.patch: New file.* gnu/local.mk: Add it.--- gnu/local.mk                                       |  1 + gnu/packages/linux.scm                             | 44 ++++++++++++++++++++++ .../patches/kmod-13-module-directory.patch         | 33 ++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

Toggle diff (106 lines)diff --git a/gnu/local.mk b/gnu/local.mkindex 21195f8c1..b1e3c878d 100644--- a/gnu/local.mk+++ b/gnu/local.mk@@ -795,6 +795,7 @@ dist_patch_DATA =						\   %D%/packages/patches/kiki-makefile.patch			\   %D%/packages/patches/kiki-missing-includes.patch		\   %D%/packages/patches/kiki-portability-64bit.patch		\+  %D%/packages/patches/kmod-13-module-directory.patch		\   %D%/packages/patches/kmod-module-directory.patch		\   %D%/packages/patches/kobodeluxe-paths.patch			\   %D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch	\diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 1f8bf3050..b2e47f79a 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1994,6 +1994,50 @@ from the module-init-tools project.")                      #t))                  %standard-phases))))) +(define-public kmod-minimal/static+  (static-package+   (package (inherit kmod-minimal)+    (name "kmod-minimal-static")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (arguments+     (substitute-keyword-arguments+       (package-arguments (static-package kmod-minimal))+       ((#:configure-flags flags ''())+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))+       ((#:make-flags flags ''())+        `(cons* "LDFLAGS=-all-static" ,flags))+       ((#:phases phases '%standard-phases)+        `(modify-phases ,phases+          (delete 'install-license-files)+          (add-after 'unpack 'patch-kmod+           (lambda _+             ;; Reduce size by 200 kiB.+             (substitute* "tools/kmod.c"+              (("[&]kmod_cmd_compat_lsmod,") "")+              (("[&]kmod_cmd_compat_rmmod,") "")+              (("[&]kmod_cmd_compat_insmod,") "")+              (("[&]kmod_cmd_compat_modinfo,") ""))+             #t))+          (replace 'install+            (lambda* (#:key outputs #:allow-other-keys)+              (let* ((out (assoc-ref outputs "out"))+                     (bin (string-append out "/bin")))+                (install-file "tools/kmod" bin)+                (for-each+                 (lambda (tool)+                   (symlink "kmod" (string-append bin "/" tool)))+                 '("modprobe" "depmod"))+                #t))))))))))+ (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.   (packagediff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patchnew file mode 100644index 000000000..5ff2f8a60--- /dev/null+++ b/gnu/packages/patches/kmod-13-module-directory.patch@@ -0,0 +1,33 @@+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'+environment variable, rather than looking for modules exclusively in+/lib/modules.++Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable+name.+++--- kmod-7/libkmod/libkmod.c	2012-03-15 08:19:16.750010226 -0400++++ kmod-7/libkmod/libkmod.c	2012-04-04 15:21:29.532074313 -0400+@@ -200,7 +200,7 @@+ static char *get_kernel_release(const char *dirname)+ {+ 	struct utsname u;+-	char *p;++	char *p, *dirname_prefix;+ + 	if (dirname != NULL)+ 		return path_make_absolute_cwd(dirname);+@@ -208,7 +208,10 @@+ 	if (uname(&u) < 0)+ 		return NULL;+ +-	if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)++	if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)++		dirname_prefix = dirname_default_prefix;++++	if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)+ 		return NULL;+ + 	return p;+

Danny Milosavljevic wrote on 27 Feb 2018 12:26

[PATCH v4 0/7] Load Linux module only when supported hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227112619.5071-1-dannym@scratchpost.org

Danny Milosavljevic (7):  gnu: kmod: Split off kmod-minimal.  gnu: Add kmod-minimal-static.  linux-initrd: Add kmod.  linux-boot: Load kernel modules only when the hardware is present.  vm: Allow qemu-image builder to load Linux kernel modules.  vm: Make the virtio-blk uniquely identifyable in /sys.  linux-boot: Call make-static-device-nodes much earlier.
 gnu/build/linux-boot.scm                           | 43 ++++++++-- gnu/build/linux-initrd.scm                         | 12 ++- gnu/build/vm.scm                                   |  2 +- gnu/local.mk                                       |  1 + gnu/packages/linux.scm                             | 91 ++++++++++++++++++---- .../patches/kmod-13-module-directory.patch         | 33 ++++++++ gnu/services/base.scm                              | 11 --- gnu/system/linux-initrd.scm                        | 50 +++++++++--- gnu/system/vm.scm                                  | 34 ++++++-- 9 files changed, 224 insertions(+), 53 deletions(-) create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

Danny Milosavljevic wrote on 27 Feb 2018 12:26

[PATCH v4 3/7] linux-initrd: Add kmod.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227112619.5071-4-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.(base-initrd): Add kmod.(expression->initrd): Add kmod, linux-module-directory.(flat-linux-module-directory): Add kmod; invoke depmod.* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.--- gnu/build/linux-initrd.scm  | 12 +++++++++++- gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++----------- 2 files changed, 45 insertions(+), 12 deletions(-)

Toggle diff (150 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..6356007df 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init kmod linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (when kmod+      (mkdir-p "sbin")+      (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 7170d1c0e..46ef055f0 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -59,6 +59,8 @@                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             kmod+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init+                        #:kmod #$kmod+                        #:linux-module-directory #$linux-module-directory                         ;; Copy everything INIT refers to into the initrd.                         #:references-graphs '("closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."   (gexp->derivation name builder                     #:references-graphs `(("closure" ,init)))) -(define (flat-linux-module-directory linux modules)+(define (flat-linux-module-directory linux modules kmod)   "Return a flat directory containing the Linux kernel modules listed in MODULES and taken from LINUX."   (define build-exp@@ -109,7 +113,7 @@ MODULES and taken from LINUX."                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex)+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)                        (srfi srfi-1)                        (guix build utils)                        (gnu build linux-modules))@@ -138,13 +142,27 @@ MODULES and taken from LINUX."                       (recursive-module-dependencies modules                                                      #:lookup-module lookup)))) -          (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))+          (define version+            (match+             (filter+              (lambda (name)+                (not (string-prefix? "." name)))+              (scandir module-dir))+             ((item) item)))++          (let ((output (string-append #$output "/lib/modules/" version)))+            (mkdir-p output)+            (for-each (lambda (module)+                        (format #t "copying '~a'...~%" module)+                        (copy-file module+                                   (string-append output "/"+                                                  (basename module))))+                      (delete-duplicates modules)))+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output+                  "-e"+                  "-F" (string-append #$linux "/System.map")+                  version)+          #t)))    (computed-file "linux-modules" build-exp)) @@ -152,6 +170,7 @@ MODULES and taken from LINUX."                       #:key                       (linux linux-libre)                       (linux-modules '())+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       (helper-packages '())                       qemu-networking?@@ -185,7 +204,7 @@ upon error."          mapped-devices))    (define kodir-    (flat-linux-module-directory linux linux-modules))+    (flat-linux-module-directory linux linux-modules kmod))    (expression->initrd    (with-imported-modules (source-module-closure@@ -223,6 +242,8 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:kmod kmod+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))@@ -245,6 +266,7 @@ FILE-SYSTEMS." (define* (base-initrd file-systems                       #:key                       (linux linux-libre)+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       qemu-networking?                       volatile-root?@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."   (raw-initrd file-systems               #:linux linux               #:linux-modules linux-modules+              #:kmod kmod               #:mapped-devices mapped-devices-              #:helper-packages helper-packages+              #:helper-packages (cons kmod helper-packages)               #:qemu-networking? qemu-networking?               #:volatile-root? volatile-root?               #:on-error on-error))

Danny Milosavljevic wrote on 27 Feb 2018 12:26

[PATCH v4 4/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227112619.5071-5-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 31 +++++++++++++++++++++++++++---- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 30 insertions(+), 5 deletions(-)

Toggle diff (64 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..6d00ea9be 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -473,6 +473,32 @@ upon error."     (string-append linux-module-directory "/"                    (ensure-dot-ko name))) +  (define (load-kernel-modules)+    "Examine /sys/devices to find out which modules to load and load them."+    (define enter?+      (const #t))+    (define (down! path stat result)+     ;; Note: modprobe mutates the tree starting with path.+     (let ((modalias-name (string-append path "/modalias")))+       (if (file-exists? modalias-name)+           (let ((modalias+                 (string-trim-right (call-with-input-file modalias-name+                                                          read-string)+                                    #\newline)))+             (system* "/sbin/modprobe" "-q" "--" modalias))))+       #t)+    (define up+      (const #t))+    (define skip+      (const #t))+    (define leaf+      (const #t))+    (define (error name stat errno result)+      (format (current-error-port) "warning: ~a: ~a~%"+              name (strerror errno))+      result)+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))+   (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n") @@ -486,10 +512,7 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 46ef055f0..c8a9e4950 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -225,7 +225,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Danny Milosavljevic wrote on 27 Feb 2018 12:26

[PATCH v4 5/7] vm: Allow qemu-image builder to load Linux kernel modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227112619.5071-6-dannym@scratchpost.org

* gnu/system/vm.scm (%modprobe-wrapper): New variable.(qemu-image): Modify.--- gnu/system/vm.scm | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)

Toggle diff (74 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 345cecedd..b5a559012 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."    #:single-file-output? #t    #:references-graphs inputs)) +(define (%modprobe-wrapper modprobe linux-module-directory)+  ;; Wrapper for the 'modprobe' command that knows where modules live.+  ;;+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'+  ;; environment variable is not set---hence the need for this wrapper.+  (program-file "modprobe"+    #~(begin+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))+ (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))   (expression->derivation-in-linux-vm    name-   (with-imported-modules (source-module-closure '((gnu build bootloader)+   (with-imported-modules (source-module-closure '((gnu build activation)+                                                   (gnu build bootloader)                                                    (gnu build vm)                                                    (guix build utils)))      #~(begin-         (use-modules (gnu build bootloader)+         (use-modules (gnu build activation)+                      (gnu build bootloader)                       (gnu build vm)                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports))           (let ((inputs-                '#$(append (list qemu parted e2fsprogs dosfstools)+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)                            (map canonical-package                                 (list sed grep coreutils findutils gawk))                            (if register-closures? (list guix) '())))@@ -302,6 +317,14 @@ the image."                         inputs)))             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)+           ;; It's possible that we need to load nls modules in order to+           ;; mount the new partition.+           (if (file-exists? #$modprobe-name)+               (activate-modprobe #$(%modprobe-wrapper modprobe-name+                                     linux-module-directory))+               (format (current-error-port)+                "WARNING: No modprobe found in ~s.  \+Loading kernel modules will be impossible.\n" #$modprobe-name))             (let* ((graphs     '#$(match inputs                                    (((names . _) ...)@@ -364,7 +387,7 @@ the image."    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-   #:references-graphs inputs))+   #:references-graphs inputs)))   ;;;

Danny Milosavljevic wrote on 27 Feb 2018 12:26

[PATCH v4 6/7] vm: Make the virtio-blk uniquely identifyable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227112619.5071-7-dannym@scratchpost.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.--- gnu/build/vm.scm  | 2 +- gnu/system/vm.scm | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..ebf9e9f6e 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex b5a559012..fdff64ed9 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os

Danny Milosavljevic wrote on 27 Feb 2018 12:26

[PATCH v4 7/7] linux-boot: Call make-static-device-nodes much earlier.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227112619.5071-8-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (expression->initrd): Store data files formake-static-device-nodes.* gnu/build/linux-boot.scm (make-static-device-nodes): Unexport.(boot-system): Call make-static-device-nodes.  Delete lookup-module.* gnu/services/base.scm (udev-shepherd-service): Deletemake-static-device-nodes call.--- gnu/build/linux-boot.scm    | 14 +++++++++----- gnu/services/base.scm       | 11 ----------- gnu/system/linux-initrd.scm |  3 ++- 3 files changed, 11 insertions(+), 17 deletions(-)

Toggle diff (61 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 6d00ea9be..f0ac755f8 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -39,7 +39,6 @@             find-long-option             find-long-options             make-essential-device-nodes-            make-static-device-nodes             configure-qemu-networking              device-number@@ -469,10 +468,6 @@ upon error."              mounts)         "ext4")) -  (define (lookup-module name)-    (string-append linux-module-directory "/"-                   (ensure-dot-ko name)))-   (define (load-kernel-modules)     "Examine /sys/devices to find out which modules to load and load them."     (define enter?@@ -512,6 +507,15 @@ upon error."        (when (member "--repl" args)          (start-repl)) +       (let* ((kernel-release+               (utsname:release (uname)))+              (directory+               (string-append linux-module-directory "/lib/modules/"+                              kernel-release))+              (old-umask (umask #o022)))+         (make-static-device-nodes directory)+         (umask old-umask))+        (load-kernel-modules)         (when qemu-guest-networking?diff --git a/gnu/services/base.scm b/gnu/services/base.scmindex 69e211ffa..0cba1c66f 100644--- a/gnu/services/base.scm+++ b/gnu/services/base.scm@@ -1857,17 +1857,6 @@ item of @var{packages}."                     (setenv "EUDEV_RULES_DIRECTORY"                             #$(file-append rules "/lib/udev/rules.d")) -                    (let* ((kernel-release-                            (utsname:release (uname)))-                           (linux-module-directory-                            (getenv "LINUX_MODULE_DIRECTORY"))-                           (directory-                            (string-append linux-module-directory "/"-                                           kernel-release))-                           (old-umask (umask #o022)))-                      (make-static-device-nodes directory)-                      (umask old-umask))-                     (let ((pid (primitive-fork)))                       (case pid                         ((0)

Ludovic Courtès wrote on 27 Feb 2018 15:26

Re: [bug#30604] [PATCH v3 4/6] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87muzu33wh.fsf@gnu.org

Hello,
Nice patch series!
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (4 lines)

> * gnu/build/linux-boot.scm (boot-system): Load kernel modules only when> the hardware is present.> * gnu/system/linux-initrd.scm (raw-initrd): Add imports.

[...]

Toggle quote (14 lines)> +  (define (load-kernel-modules)> +    "Examine /sys/devices to find out which modules to load and load them."> +    (define enter?> +      (const #t))> +    (define (down! path stat result)> +     ;; Note: modprobe mutates the tree starting with path.> +     (let ((modalias-name (string-append path "/modalias")))> +       (if (file-exists? modalias-name)> +           (let ((modalias> +                 (string-trim-right (call-with-input-file modalias-name> +                                                          read-string)> +                                    #\newline)))> +             (system* "/sbin/modprobe" "-q" "--" modalias))))

Can we build upon (gnu build linux-modules) to achieve this?
Hopefully the tools athttps://debbugs.gnu.org/cgi/bugreport.cgi?bug=30629#14 can help a bit.
Thanks,Ludo’.

Danny Milosavljevic wrote on 27 Feb 2018 16:50

[PATCH v5 1/7] gnu: kmod: Split off kmod-minimal.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227155051.1141-2-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal): New variable.(kmod): Modify.--- gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-)

Toggle diff (81 lines)diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 238398e84..1f8bf3050 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as to use Linux' inotify mechanism, which allows file accesses to be monitored.")     (license license:gpl2+))) -(define-public kmod+(define kmod-minimal   (package+    (name "kmod-minimal")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (build-system gnu-build-system)+    (arguments+     `(#:tests? #f)) ; FIXME: Investigate test failures+    (home-page "https://www.kernel.org/")+    (synopsis "Kernel module tools")+    (description "Kmod is a set of tools to handle common tasks with Linux+kernel modules like insert, remove, list, check properties, resolve+dependencies and aliases.++These tools are designed on top of libkmod, a library that is shipped with+kmod.  The aim is to be compatible with tools, configurations and indices+from the module-init-tools project.")+    (license license:gpl2+))) ; library under lgpl2.1+++(define-public kmod+  (package (inherit kmod-minimal)     (name "kmod")     (version "24")     (source (origin@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                (base32                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))               (patches (search-patches "kmod-module-directory.patch"))))-    (build-system gnu-build-system)     (native-inputs      `(("pkg-config" ,pkg-config)))     (inputs      `(("xz" ,xz)        ("zlib" ,zlib)))     (arguments-     `(#:tests? #f ; FIXME: Investigate test failures-       #:configure-flags '("--with-xz" "--with-zlib")+     `(#:configure-flags '("--with-xz" "--with-zlib")+       #:tests? #f ; FIXME: Investigate test failures        #:phases (alist-cons-after                  'install 'install-modprobe&co                  (lambda* (#:key outputs #:allow-other-keys)@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")                                  (symlink "kmod"                                           (string-append bin "/" tool)))                                '("insmod" "rmmod" "lsmod" "modprobe"-                                 "modinfo" "depmod"))))-                 %standard-phases)))-    (home-page "https://www.kernel.org/")-    (synopsis "Kernel module tools")-    (description "Kmod is a set of tools to handle common tasks with Linux-kernel modules like insert, remove, list, check properties, resolve-dependencies and aliases.--These tools are designed on top of libkmod, a library that is shipped with-kmod.  The aim is to be compatible with tools, configurations and indices-from the module-init-tools project.")-    (license license:gpl2+))) ; library under lgpl2.1++                                 "modinfo" "depmod"))+                     #t))+                 %standard-phases)))))  (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.

Danny Milosavljevic wrote on 27 Feb 2018 16:50

[PATCH v5 0/7] Load Linux module only when supported hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227155051.1141-1-dannym@scratchpost.org

Danny Milosavljevic (7):  gnu: kmod: Split off kmod-minimal.  gnu: Add kmod-minimal-static.  linux-initrd: Add kmod.  linux-boot: Load kernel modules only when the hardware is present.  vm: Allow qemu-image builder to load Linux kernel modules.  vm: Make the virtio-blk is uniquely identifyable in /sys.  linux-boot: Call make-static-device-nodes much earlier.
 gnu/build/linux-boot.scm                           | 42 ++++++++-- gnu/build/linux-initrd.scm                         | 12 ++- gnu/build/vm.scm                                   |  2 +- gnu/local.mk                                       |  1 + gnu/packages/linux.scm                             | 91 ++++++++++++++++++---- .../patches/kmod-13-module-directory.patch         | 33 ++++++++ gnu/system/linux-initrd.scm                        | 50 +++++++++--- gnu/system/vm.scm                                  | 34 ++++++-- 8 files changed, 224 insertions(+), 41 deletions(-) create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

Danny Milosavljevic wrote on 27 Feb 2018 16:50

[PATCH v5 2/7] gnu: Add kmod-minimal-static.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227155051.1141-3-dannym@scratchpost.org

* gnu/packages/linux.scm (kmod-minimal/static): New variable.* gnu/packages/patches/kmod-13-module-directory.patch: New file.* gnu/local.mk: Add it.--- gnu/local.mk                                       |  1 + gnu/packages/linux.scm                             | 44 ++++++++++++++++++++++ .../patches/kmod-13-module-directory.patch         | 33 ++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

Toggle diff (106 lines)diff --git a/gnu/local.mk b/gnu/local.mkindex 21195f8c1..b1e3c878d 100644--- a/gnu/local.mk+++ b/gnu/local.mk@@ -795,6 +795,7 @@ dist_patch_DATA =						\   %D%/packages/patches/kiki-makefile.patch			\   %D%/packages/patches/kiki-missing-includes.patch		\   %D%/packages/patches/kiki-portability-64bit.patch		\+  %D%/packages/patches/kmod-13-module-directory.patch		\   %D%/packages/patches/kmod-module-directory.patch		\   %D%/packages/patches/kobodeluxe-paths.patch			\   %D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch	\diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scmindex 1f8bf3050..b2e47f79a 100644--- a/gnu/packages/linux.scm+++ b/gnu/packages/linux.scm@@ -1994,6 +1994,50 @@ from the module-init-tools project.")                      #t))                  %standard-phases))))) +(define-public kmod-minimal/static+  (static-package+   (package (inherit kmod-minimal)+    (name "kmod-minimal-static")+    (version "13")+    (source (origin+              (method url-fetch)+              (uri+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"+                              "kmod-" version ".tar.xz"))+              (sha256+               (base32+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))+              (patches (search-patches "kmod-13-module-directory.patch"))))+    (arguments+     (substitute-keyword-arguments+       (package-arguments (static-package kmod-minimal))+       ((#:configure-flags flags ''())+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))+       ((#:make-flags flags ''())+        `(cons* "LDFLAGS=-all-static" ,flags))+       ((#:phases phases '%standard-phases)+        `(modify-phases ,phases+          (delete 'install-license-files)+          (add-after 'unpack 'patch-kmod+           (lambda _+             ;; Reduce size by 200 kiB.+             (substitute* "tools/kmod.c"+              (("[&]kmod_cmd_compat_lsmod,") "")+              (("[&]kmod_cmd_compat_rmmod,") "")+              (("[&]kmod_cmd_compat_insmod,") "")+              (("[&]kmod_cmd_compat_modinfo,") ""))+             #t))+          (replace 'install+            (lambda* (#:key outputs #:allow-other-keys)+              (let* ((out (assoc-ref outputs "out"))+                     (bin (string-append out "/bin")))+                (install-file "tools/kmod" bin)+                (for-each+                 (lambda (tool)+                   (symlink "kmod" (string-append bin "/" tool)))+                 '("modprobe" "depmod"))+                #t))))))))))+ (define-public eudev   ;; The post-systemd fork, maintained by Gentoo.   (packagediff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patchnew file mode 100644index 000000000..5ff2f8a60--- /dev/null+++ b/gnu/packages/patches/kmod-13-module-directory.patch@@ -0,0 +1,33 @@+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'+environment variable, rather than looking for modules exclusively in+/lib/modules.++Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable+name.+++--- kmod-7/libkmod/libkmod.c	2012-03-15 08:19:16.750010226 -0400++++ kmod-7/libkmod/libkmod.c	2012-04-04 15:21:29.532074313 -0400+@@ -200,7 +200,7 @@+ static char *get_kernel_release(const char *dirname)+ {+ 	struct utsname u;+-	char *p;++	char *p, *dirname_prefix;+ + 	if (dirname != NULL)+ 		return path_make_absolute_cwd(dirname);+@@ -208,7 +208,10 @@+ 	if (uname(&u) < 0)+ 		return NULL;+ +-	if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)++	if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)++		dirname_prefix = dirname_default_prefix;++++	if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)+ 		return NULL;+ + 	return p;+

Danny Milosavljevic wrote on 27 Feb 2018 16:50

[PATCH v5 3/7] linux-initrd: Add kmod.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227155051.1141-4-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.(base-initrd): Add kmod.(expression->initrd): Add kmod, linux-module-directory.(flat-linux-module-directory): Add kmod; invoke depmod.* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.--- gnu/build/linux-initrd.scm  | 12 +++++++++++- gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++----------- 2 files changed, 45 insertions(+), 12 deletions(-)

Toggle diff (150 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..6356007df 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init kmod linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (when kmod+      (mkdir-p "sbin")+      (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 7170d1c0e..46ef055f0 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -59,6 +59,8 @@                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             kmod+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init+                        #:kmod #$kmod+                        #:linux-module-directory #$linux-module-directory                         ;; Copy everything INIT refers to into the initrd.                         #:references-graphs '("closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."   (gexp->derivation name builder                     #:references-graphs `(("closure" ,init)))) -(define (flat-linux-module-directory linux modules)+(define (flat-linux-module-directory linux modules kmod)   "Return a flat directory containing the Linux kernel modules listed in MODULES and taken from LINUX."   (define build-exp@@ -109,7 +113,7 @@ MODULES and taken from LINUX."                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex)+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)                        (srfi srfi-1)                        (guix build utils)                        (gnu build linux-modules))@@ -138,13 +142,27 @@ MODULES and taken from LINUX."                       (recursive-module-dependencies modules                                                      #:lookup-module lookup)))) -          (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))+          (define version+            (match+             (filter+              (lambda (name)+                (not (string-prefix? "." name)))+              (scandir module-dir))+             ((item) item)))++          (let ((output (string-append #$output "/lib/modules/" version)))+            (mkdir-p output)+            (for-each (lambda (module)+                        (format #t "copying '~a'...~%" module)+                        (copy-file module+                                   (string-append output "/"+                                                  (basename module))))+                      (delete-duplicates modules)))+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output+                  "-e"+                  "-F" (string-append #$linux "/System.map")+                  version)+          #t)))    (computed-file "linux-modules" build-exp)) @@ -152,6 +170,7 @@ MODULES and taken from LINUX."                       #:key                       (linux linux-libre)                       (linux-modules '())+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       (helper-packages '())                       qemu-networking?@@ -185,7 +204,7 @@ upon error."          mapped-devices))    (define kodir-    (flat-linux-module-directory linux linux-modules))+    (flat-linux-module-directory linux linux-modules kmod))    (expression->initrd    (with-imported-modules (source-module-closure@@ -223,6 +242,8 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:kmod kmod+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))@@ -245,6 +266,7 @@ FILE-SYSTEMS." (define* (base-initrd file-systems                       #:key                       (linux linux-libre)+                      (kmod kmod-minimal/static)                       (mapped-devices '())                       qemu-networking?                       volatile-root?@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."   (raw-initrd file-systems               #:linux linux               #:linux-modules linux-modules+              #:kmod kmod               #:mapped-devices mapped-devices-              #:helper-packages helper-packages+              #:helper-packages (cons kmod helper-packages)               #:qemu-networking? qemu-networking?               #:volatile-root? volatile-root?               #:on-error on-error))

Danny Milosavljevic wrote on 27 Feb 2018 16:50

[PATCH v5 4/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227155051.1141-5-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 31 +++++++++++++++++++++++++++---- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 30 insertions(+), 5 deletions(-)

Toggle diff (64 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..6d00ea9be 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -473,6 +473,32 @@ upon error."     (string-append linux-module-directory "/"                    (ensure-dot-ko name))) +  (define (load-kernel-modules)+    "Examine /sys/devices to find out which modules to load and load them."+    (define enter?+      (const #t))+    (define (down! path stat result)+     ;; Note: modprobe mutates the tree starting with path.+     (let ((modalias-name (string-append path "/modalias")))+       (if (file-exists? modalias-name)+           (let ((modalias+                 (string-trim-right (call-with-input-file modalias-name+                                                          read-string)+                                    #\newline)))+             (system* "/sbin/modprobe" "-q" "--" modalias))))+       #t)+    (define up+      (const #t))+    (define skip+      (const #t))+    (define leaf+      (const #t))+    (define (error name stat errno result)+      (format (current-error-port) "warning: ~a: ~a~%"+              name (strerror errno))+      result)+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))+   (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n") @@ -486,10 +512,7 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 46ef055f0..c8a9e4950 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -225,7 +225,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Danny Milosavljevic wrote on 27 Feb 2018 16:50

[PATCH v5 5/7] vm: Allow qemu-image builder to load Linux kernel modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227155051.1141-6-dannym@scratchpost.org

* gnu/system/vm.scm (%modprobe-wrapper): New variable.(qemu-image): Modify.--- gnu/system/vm.scm | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)

Toggle diff (74 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 345cecedd..b5a559012 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."    #:single-file-output? #t    #:references-graphs inputs)) +(define (%modprobe-wrapper modprobe linux-module-directory)+  ;; Wrapper for the 'modprobe' command that knows where modules live.+  ;;+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'+  ;; environment variable is not set---hence the need for this wrapper.+  (program-file "modprobe"+    #~(begin+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))+ (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))   (expression->derivation-in-linux-vm    name-   (with-imported-modules (source-module-closure '((gnu build bootloader)+   (with-imported-modules (source-module-closure '((gnu build activation)+                                                   (gnu build bootloader)                                                    (gnu build vm)                                                    (guix build utils)))      #~(begin-         (use-modules (gnu build bootloader)+         (use-modules (gnu build activation)+                      (gnu build bootloader)                       (gnu build vm)                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports))           (let ((inputs-                '#$(append (list qemu parted e2fsprogs dosfstools)+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)                            (map canonical-package                                 (list sed grep coreutils findutils gawk))                            (if register-closures? (list guix) '())))@@ -302,6 +317,14 @@ the image."                         inputs)))             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)+           ;; It's possible that we need to load nls modules in order to+           ;; mount the new partition.+           (if (file-exists? #$modprobe-name)+               (activate-modprobe #$(%modprobe-wrapper modprobe-name+                                     linux-module-directory))+               (format (current-error-port)+                "WARNING: No modprobe found in ~s.  \+Loading kernel modules will be impossible.\n" #$modprobe-name))             (let* ((graphs     '#$(match inputs                                    (((names . _) ...)@@ -364,7 +387,7 @@ the image."    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-   #:references-graphs inputs))+   #:references-graphs inputs)))   ;;;

Danny Milosavljevic wrote on 27 Feb 2018 16:50

[PATCH v5 6/7] vm: Make the virtio-blk is uniquely identifyable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227155051.1141-7-dannym@scratchpost.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.--- gnu/build/vm.scm  | 2 +- gnu/system/vm.scm | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..ebf9e9f6e 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex b5a559012..fdff64ed9 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os

Danny Milosavljevic wrote on 27 Feb 2018 16:50

[PATCH v5 7/7] linux-boot: Call make-static-device-nodes much earlier.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180227155051.1141-8-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (expression->initrd): Store data files formake-static-device-nodes.* gnu/build/linux-boot.scm (boot-system): Call make-static-device-nodes.Delete lookup-module.--- gnu/build/linux-boot.scm    | 13 +++++++++---- gnu/system/linux-initrd.scm |  3 ++- 2 files changed, 11 insertions(+), 5 deletions(-)

Toggle diff (31 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 6d00ea9be..1b16f267a 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -469,10 +469,6 @@ upon error."              mounts)         "ext4")) -  (define (lookup-module name)-    (string-append linux-module-directory "/"-                   (ensure-dot-ko name)))-   (define (load-kernel-modules)     "Examine /sys/devices to find out which modules to load and load them."     (define enter?@@ -512,6 +508,15 @@ upon error."        (when (member "--repl" args)          (start-repl)) +       (let* ((kernel-release+               (utsname:release (uname)))+              (directory+               (string-append linux-module-directory "/lib/modules/"+                              kernel-release))+              (old-umask (umask #o022)))+         (make-static-device-nodes directory)+         (umask old-umask))+        (load-kernel-modules)         (when qemu-guest-networking?

Danny Milosavljevic wrote on 2 Mar 2018 15:16

[PATCH v6 0/6] Load Linux module only when supported hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302141606.10669-1-dannym@scratchpost.org

Danny Milosavljevic (6):  linux-modules: Add module-aliases.  linux-modules: Add install-modules.  linux-boot: Load kernel modules only when the hardware is present.  vm: Allow qemu-image builder to load Linux kernel modules.  vm: Make the virtio-blk uniquely identifiable in /sys.  linux-initrd: Provide modprobe to the initrd.
 gnu/build/linux-boot.scm    | 42 ++++++++++++++++++---- gnu/build/linux-initrd.scm  | 13 ++++++- gnu/build/linux-modules.scm | 61 +++++++++++++++++++++++++++++++ gnu/build/vm.scm            |  2 +- gnu/system/linux-initrd.scm | 88 +++++++++++++++++++++++++++++++++++++++------ gnu/system/vm.scm           | 34 +++++++++++++++--- 6 files changed, 216 insertions(+), 24 deletions(-)

Danny Milosavljevic wrote on 2 Mar 2018 15:17

[PATCH v6 1/6] linux-modules: Add module-aliases.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302141720.10720-1-dannym@scratchpost.org

* gnu/build/linux-modules.scm (module-aliases): New variable.--- gnu/build/linux-modules.scm | 9 +++++++++ 1 file changed, 9 insertions(+)

Toggle diff (26 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 4a6d4ff08..364339df9 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -30,6 +30,7 @@   #:use-module (ice-9 rdelim)   #:export (dot-ko             ensure-dot-ko+            module-aliases             module-dependencies             recursive-module-dependencies             modules-loaded@@ -95,6 +96,14 @@ contains module names, not actual file names."       (('depends . what)        (string-tokenize what %not-comma))))) +(define (module-aliases file)+  "Return the list of aliases for FILE."+  (let ((info (modinfo-section-contents file)))+    (filter-map (match-lambda+                 (('alias . value)+                  value)+                 (_ #f)) (modinfo-section-contents file))))+ (define dot-ko   (cut string-append <> ".ko"))

Danny Milosavljevic wrote on 2 Mar 2018 15:17

[PATCH v6 2/6] linux-modules: Add install-modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302141720.10720-2-dannym@scratchpost.org

* gnu/build/linux-modules.scm (install-modules): New procedure.(%not-dash): New variable.--- gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)

Toggle diff (68 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 364339df9..af217c974 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -36,6 +36,7 @@             modules-loaded             module-loaded?             load-linux-module*+            install-module-files              current-module-debugging-port @@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                       module)))               known-aliases)) +(define %not-dash+  (char-set-complement (char-set #\-)))++(define (install-module-files module-files output)+  "Install MODULE-FILES to OUTPUT.+Precondition: OUTPUT is an empty directory."+  (let ((aliases+         (map (lambda (module-file-name)+                (format #t "copying '~a'...~%" module-file-name)+                (copy-file module-file-name+                           (string-append output "/"+                                          (basename module-file-name)))+                `(,(file-name->module-name module-file-name) .+                  ,(module-aliases module-file-name)))+              (sort module-files string<))))+    (call-with-output-file (string-append output "/modules.alias")+      (lambda (port)+        (format port+                "# Aliases extracted from modules themselves.\n")+        (for-each (match-lambda ((module . aliases)+                                 (for-each (lambda (alias)+                                             (format port "alias ~a ~a\n" alias+                                                     module))+                                           aliases)))+                  aliases)))+    (call-with-output-file (string-append output "/modules.devname")+      (lambda (port)+        (format port+                "# Device nodes to trigger on-demand module loading.\n")+        (let* ((aliases (append-map (match-lambda+                                     ((module . aliases) aliases))+                                    aliases))+               (devname #f))+          ;; Note: there's only one devname and then only one (char-major|block-major).+          (for-each+           (match-lambda+            (((? (cut string-prefix? "devname:" <>) alias) . value)+             (set! devname (string-drop value (string-length "devname:"))))+            (((? (cut string-prefix? "char-major-" <>) alias) . value)+             (let ((parts (string-tokenize %not-dash)))+               (match parts+                      ((a b major minor)+                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))+            (((? (cut string-prefix? "block-major-" <>) alias) . value)+             (let ((parts (string-tokenize %not-dash)))+               (match parts+                      ((a b major minor)+                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))+            (_ #f))+           aliases))))))+ ;;; linux-modules.scm ends here

Danny Milosavljevic wrote on 2 Mar 2018 15:17

[PATCH v6 3/6] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302141720.10720-3-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.(lookup-module): Delete procedure.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 42 +++++++++++++++++++++++++++++++++++------- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 38 insertions(+), 8 deletions(-)

Toggle diff (75 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..1b16f267a 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -469,9 +469,31 @@ upon error."              mounts)         "ext4")) -  (define (lookup-module name)-    (string-append linux-module-directory "/"-                   (ensure-dot-ko name)))+  (define (load-kernel-modules)+    "Examine /sys/devices to find out which modules to load and load them."+    (define enter?+      (const #t))+    (define (down! path stat result)+     ;; Note: modprobe mutates the tree starting with path.+     (let ((modalias-name (string-append path "/modalias")))+       (if (file-exists? modalias-name)+           (let ((modalias+                 (string-trim-right (call-with-input-file modalias-name+                                                          read-string)+                                    #\newline)))+             (system* "/sbin/modprobe" "-q" "--" modalias))))+       #t)+    (define up+      (const #t))+    (define skip+      (const #t))+    (define leaf+      (const #t))+    (define (error name stat errno result)+      (format (current-error-port) "warning: ~a: ~a~%"+              name (strerror errno))+      result)+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))    (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n")@@ -486,10 +508,16 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (let* ((kernel-release+               (utsname:release (uname)))+              (directory+               (string-append linux-module-directory "/lib/modules/"+                              kernel-release))+              (old-umask (umask #o022)))+         (make-static-device-nodes directory)+         (umask old-umask))++       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex e7f97bb88..b50d3ff80 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -208,7 +208,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Danny Milosavljevic wrote on 2 Mar 2018 15:17

[PATCH v6 4/6] vm: Allow qemu-image builder to load Linux kernel modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302141720.10720-4-dannym@scratchpost.org

* gnu/system/vm.scm (%modprobe-wrapper): New variable.(qemu-image): Modify.--- gnu/system/vm.scm | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)

Toggle diff (74 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 345cecedd..b5a559012 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."    #:single-file-output? #t    #:references-graphs inputs)) +(define (%modprobe-wrapper modprobe linux-module-directory)+  ;; Wrapper for the 'modprobe' command that knows where modules live.+  ;;+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'+  ;; environment variable is not set---hence the need for this wrapper.+  (program-file "modprobe"+    #~(begin+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))+ (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))   (expression->derivation-in-linux-vm    name-   (with-imported-modules (source-module-closure '((gnu build bootloader)+   (with-imported-modules (source-module-closure '((gnu build activation)+                                                   (gnu build bootloader)                                                    (gnu build vm)                                                    (guix build utils)))      #~(begin-         (use-modules (gnu build bootloader)+         (use-modules (gnu build activation)+                      (gnu build bootloader)                       (gnu build vm)                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports))           (let ((inputs-                '#$(append (list qemu parted e2fsprogs dosfstools)+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)                            (map canonical-package                                 (list sed grep coreutils findutils gawk))                            (if register-closures? (list guix) '())))@@ -302,6 +317,14 @@ the image."                         inputs)))             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)+           ;; It's possible that we need to load nls modules in order to+           ;; mount the new partition.+           (if (file-exists? #$modprobe-name)+               (activate-modprobe #$(%modprobe-wrapper modprobe-name+                                     linux-module-directory))+               (format (current-error-port)+                "WARNING: No modprobe found in ~s.  \+Loading kernel modules will be impossible.\n" #$modprobe-name))             (let* ((graphs     '#$(match inputs                                    (((names . _) ...)@@ -364,7 +387,7 @@ the image."    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-   #:references-graphs inputs))+   #:references-graphs inputs)))   ;;;

Danny Milosavljevic wrote on 2 Mar 2018 15:17

[PATCH v6 5/6] vm: Make the virtio-blk uniquely identifiable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302141720.10720-5-dannym@scratchpost.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.--- gnu/build/vm.scm  | 2 +- gnu/system/vm.scm | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..ebf9e9f6e 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex b5a559012..fdff64ed9 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os

Danny Milosavljevic wrote on 2 Mar 2018 15:17

[PATCH v6 6/6] linux-initrd: Provide modprobe to the initrd.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302141720.10720-6-dannym@scratchpost.org

* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and thelinux modules to the initrd.* gnu/system/linux-initrd.scm (%modprobe): New procedure.(expression->initrd): Use it.  Add linux-module-directory.(raw-initrd): Pass linux-module-directory.--- gnu/build/linux-initrd.scm  | 13 ++++++- gnu/system/linux-initrd.scm | 84 ++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 87 insertions(+), 10 deletions(-)

Toggle diff (162 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..d4cb5e2d8 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init modprobe linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +     ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (when modprobe+      (mkdir-p "sbin")+      (symlink modprobe "sbin/modprobe")+      (compile-to-cache "sbin/modprobe"))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex b50d3ff80..a69497ff8 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -56,11 +56,73 @@ ;;; ;;; Code: +(define* (%modprobe linux-module-directory #:key+                    (guile %guile-static-stripped))+  (program-file "modprobe"+    (with-imported-modules (source-module-closure+                            '((gnu build linux-modules)))+      #~(begin+          (use-modules (gnu build linux-modules) (ice-9 getopt-long)+                       (ice-9 match) (srfi srfi-1) (ice-9 ftw))+          (define (find-only-entry directory)+            (match (scandir directory)+             (("." ".." basename)+              (string-append directory "/" basename))))+          (define (resolve-alias alias)+            (let* ((linux-release-module-directory+                    (find-only-entry (string-append "/lib/modules"))))+              (match (delete-duplicates (matching-modules alias+                      (known-module-aliases+                        (string-append linux-release-module-directory+                                       "/modules.alias"))))+               (()+                (error "no alias by that name" alias))+               (items+                items))))+          (define (lookup-module module)+            (let* ((linux-release-module-directory+                    (find-only-entry (string-append "/lib/modules")))+                   (file-name (string-append linux-release-module-directory+                                             "/" (ensure-dot-ko module))))+              (if (file-exists? file-name)+                  file-name+                  (error "no module file found for module" module))))+          (define option-spec+           '((quiet    (single-char #\q) (value #f))))+          (define options+            (getopt-long (command-line) option-spec))+          (when (option-ref options 'quiet #f)+            (current-error-port (%make-void-port "w"))+            (current-output-port (%make-void-port "w")))+          (let ((exit-status 0))+            (for-each (match-lambda+                        (('quiet . #t)+                         #f)+                        ((() modules ...)+                         (for-each (lambda (alias)+                                     (catch #t+                                       (lambda ()+                                         (let ((modules (resolve-alias alias)))                                           (for-each (lambda (module)+                                                       (load-linux-module*+                                                        (lookup-module module)+                                                        #:lookup-module+                                                        lookup-module))+                                                     modules)))+                                       (lambda (key . args)+                                         (display (cons* key args)+                                                  (current-error-port))+                                         (newline (current-error-port))+                                         (set! exit-status 1))))+                                   modules)))+                      options)+            (exit exit-status))))+  #:guile guile))  (define* (expression->initrd exp                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."   (define init     (program-file "init" exp #:guile guile)) +  (define modprobe+    (%modprobe linux-module-directory #:guile guile))+   (define builder     (with-imported-modules (source-module-closure                             '((gnu build linux-initrd)))@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init-                        ;; Copy everything INIT refers to into the initrd.-                        #:references-graphs '("closure")+                        #:modprobe #$modprobe+                        #:linux-module-directory #$linux-module-directory+                        ;; Copy everything INIT and MODPROBE refer to into the+                        ;; initrd.+                        #:references-graphs '("init-closure"+                                              "modprobe-closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))    (gexp->derivation name builder-                    #:references-graphs `(("closure" ,init))))+                    #:references-graphs `(("init-closure" ,init)+                                          ("modprobe-closure" ,modprobe))))  (define (flat-linux-module-directory linux modules)   "Return a flat directory containing the Linux kernel modules listed in@@ -141,12 +211,7 @@ MODULES and taken from LINUX."                                                      #:lookup-module lookup))))            (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))+          (install-module-files (delete-duplicates modules) #$output))))    (computed-file "linux-modules" build-exp)) @@ -227,6 +292,7 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))

Danny Milosavljevic wrote on 2 Mar 2018 16:34

[PATCH v7 0/6] Load Linux module only when supported hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302153408.14091-1-dannym@scratchpost.org

Danny Milosavljevic (6):  linux-modules: Add module-aliases.  linux-modules: Add install-modules.  linux-boot: Load kernel modules only when the hardware is present.  vm: Allow qemu-image builder to load Linux kernel modules.  vm: Make the virtio-blk uniquely identifiable in /sys.  linux-initrd: Provide modprobe to the initrd.
 gnu/build/linux-boot.scm    |  42 +++++++++++++++--- gnu/build/linux-initrd.scm  |  13 +++++- gnu/build/linux-modules.scm |  61 ++++++++++++++++++++++++++ gnu/build/vm.scm            |   2 +- gnu/system/linux-initrd.scm | 103 ++++++++++++++++++++++++++++++++++++++------ gnu/system/vm.scm           |  34 ++++++++++++--- 6 files changed, 228 insertions(+), 27 deletions(-)

Danny Milosavljevic wrote on 2 Mar 2018 16:34

[PATCH v7 1/6] linux-modules: Add module-aliases.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302153408.14091-2-dannym@scratchpost.org

* gnu/build/linux-modules.scm (module-aliases): New variable.--- gnu/build/linux-modules.scm | 9 +++++++++ 1 file changed, 9 insertions(+)

Toggle diff (26 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 4a6d4ff08..364339df9 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -30,6 +30,7 @@   #:use-module (ice-9 rdelim)   #:export (dot-ko             ensure-dot-ko+            module-aliases             module-dependencies             recursive-module-dependencies             modules-loaded@@ -95,6 +96,14 @@ contains module names, not actual file names."       (('depends . what)        (string-tokenize what %not-comma))))) +(define (module-aliases file)+  "Return the list of aliases for FILE."+  (let ((info (modinfo-section-contents file)))+    (filter-map (match-lambda+                 (('alias . value)+                  value)+                 (_ #f)) (modinfo-section-contents file))))+ (define dot-ko   (cut string-append <> ".ko"))

Danny Milosavljevic wrote on 2 Mar 2018 16:34

[PATCH v7 2/6] linux-modules: Add install-modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302153408.14091-3-dannym@scratchpost.org

* gnu/build/linux-modules.scm (install-modules): New procedure.(%not-dash): New variable.--- gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)

Toggle diff (68 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 364339df9..af217c974 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -36,6 +36,7 @@             modules-loaded             module-loaded?             load-linux-module*+            install-module-files              current-module-debugging-port @@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                       module)))               known-aliases)) +(define %not-dash+  (char-set-complement (char-set #\-)))++(define (install-module-files module-files output)+  "Install MODULE-FILES to OUTPUT.+Precondition: OUTPUT is an empty directory."+  (let ((aliases+         (map (lambda (module-file-name)+                (format #t "copying '~a'...~%" module-file-name)+                (copy-file module-file-name+                           (string-append output "/"+                                          (basename module-file-name)))+                `(,(file-name->module-name module-file-name) .+                  ,(module-aliases module-file-name)))+              (sort module-files string<))))+    (call-with-output-file (string-append output "/modules.alias")+      (lambda (port)+        (format port+                "# Aliases extracted from modules themselves.\n")+        (for-each (match-lambda ((module . aliases)+                                 (for-each (lambda (alias)+                                             (format port "alias ~a ~a\n" alias+                                                     module))+                                           aliases)))+                  aliases)))+    (call-with-output-file (string-append output "/modules.devname")+      (lambda (port)+        (format port+                "# Device nodes to trigger on-demand module loading.\n")+        (let* ((aliases (append-map (match-lambda+                                     ((module . aliases) aliases))+                                    aliases))+               (devname #f))+          ;; Note: there's only one devname and then only one (char-major|block-major).+          (for-each+           (match-lambda+            (((? (cut string-prefix? "devname:" <>) alias) . value)+             (set! devname (string-drop value (string-length "devname:"))))+            (((? (cut string-prefix? "char-major-" <>) alias) . value)+             (let ((parts (string-tokenize %not-dash)))+               (match parts+                      ((a b major minor)+                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))+            (((? (cut string-prefix? "block-major-" <>) alias) . value)+             (let ((parts (string-tokenize %not-dash)))+               (match parts+                      ((a b major minor)+                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))+            (_ #f))+           aliases))))))+ ;;; linux-modules.scm ends here

Danny Milosavljevic wrote on 2 Mar 2018 16:34

[PATCH v7 3/6] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302153408.14091-4-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.(lookup-module): Delete procedure.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 42 +++++++++++++++++++++++++++++++++++------- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 38 insertions(+), 8 deletions(-)

Toggle diff (75 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..1b16f267a 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -469,9 +469,31 @@ upon error."              mounts)         "ext4")) -  (define (lookup-module name)-    (string-append linux-module-directory "/"-                   (ensure-dot-ko name)))+  (define (load-kernel-modules)+    "Examine /sys/devices to find out which modules to load and load them."+    (define enter?+      (const #t))+    (define (down! path stat result)+     ;; Note: modprobe mutates the tree starting with path.+     (let ((modalias-name (string-append path "/modalias")))+       (if (file-exists? modalias-name)+           (let ((modalias+                 (string-trim-right (call-with-input-file modalias-name+                                                          read-string)+                                    #\newline)))+             (system* "/sbin/modprobe" "-q" "--" modalias))))+       #t)+    (define up+      (const #t))+    (define skip+      (const #t))+    (define leaf+      (const #t))+    (define (error name stat errno result)+      (format (current-error-port) "warning: ~a: ~a~%"+              name (strerror errno))+      result)+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))    (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n")@@ -486,10 +508,16 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (let* ((kernel-release+               (utsname:release (uname)))+              (directory+               (string-append linux-module-directory "/lib/modules/"+                              kernel-release))+              (old-umask (umask #o022)))+         (make-static-device-nodes directory)+         (umask old-umask))++       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex e7f97bb88..b50d3ff80 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -208,7 +208,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Danny Milosavljevic wrote on 2 Mar 2018 16:34

[PATCH v7 4/6] vm: Allow qemu-image builder to load Linux kernel modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302153408.14091-5-dannym@scratchpost.org

* gnu/system/vm.scm (%modprobe-wrapper): New variable.(qemu-image): Modify.--- gnu/system/vm.scm | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)

Toggle diff (74 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 345cecedd..b5a559012 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."    #:single-file-output? #t    #:references-graphs inputs)) +(define (%modprobe-wrapper modprobe linux-module-directory)+  ;; Wrapper for the 'modprobe' command that knows where modules live.+  ;;+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'+  ;; environment variable is not set---hence the need for this wrapper.+  (program-file "modprobe"+    #~(begin+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))+ (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))   (expression->derivation-in-linux-vm    name-   (with-imported-modules (source-module-closure '((gnu build bootloader)+   (with-imported-modules (source-module-closure '((gnu build activation)+                                                   (gnu build bootloader)                                                    (gnu build vm)                                                    (guix build utils)))      #~(begin-         (use-modules (gnu build bootloader)+         (use-modules (gnu build activation)+                      (gnu build bootloader)                       (gnu build vm)                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports))           (let ((inputs-                '#$(append (list qemu parted e2fsprogs dosfstools)+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)                            (map canonical-package                                 (list sed grep coreutils findutils gawk))                            (if register-closures? (list guix) '())))@@ -302,6 +317,14 @@ the image."                         inputs)))             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)+           ;; It's possible that we need to load nls modules in order to+           ;; mount the new partition.+           (if (file-exists? #$modprobe-name)+               (activate-modprobe #$(%modprobe-wrapper modprobe-name+                                     linux-module-directory))+               (format (current-error-port)+                "WARNING: No modprobe found in ~s.  \+Loading kernel modules will be impossible.\n" #$modprobe-name))             (let* ((graphs     '#$(match inputs                                    (((names . _) ...)@@ -364,7 +387,7 @@ the image."    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-   #:references-graphs inputs))+   #:references-graphs inputs)))   ;;;

Danny Milosavljevic wrote on 2 Mar 2018 16:34

[PATCH v7 5/6] vm: Make the virtio-blk uniquely identifiable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302153408.14091-6-dannym@scratchpost.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.--- gnu/build/vm.scm  | 2 +- gnu/system/vm.scm | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..ebf9e9f6e 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex b5a559012..fdff64ed9 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os

Danny Milosavljevic wrote on 2 Mar 2018 16:34

[PATCH v7 6/6] linux-initrd: Provide modprobe to the initrd.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180302153408.14091-7-dannym@scratchpost.org

* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and thelinux modules to the initrd.* gnu/system/linux-initrd.scm (%modprobe): New procedure.(expression->initrd): Use it.  Add linux-module-directory.(raw-initrd): Pass linux-module-directory.--- gnu/build/linux-initrd.scm  | 13 +++++- gnu/system/linux-initrd.scm | 99 +++++++++++++++++++++++++++++++++++++++------ 2 files changed, 99 insertions(+), 13 deletions(-)

Toggle diff (184 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..d4cb5e2d8 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init modprobe linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +     ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (when modprobe+      (mkdir-p "sbin")+      (symlink modprobe "sbin/modprobe")+      (compile-to-cache "sbin/modprobe"))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex b50d3ff80..8050ac47e 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -56,11 +56,73 @@ ;;; ;;; Code: +(define* (%modprobe linux-module-directory #:key+                    (guile %guile-static-stripped))+  (program-file "modprobe"+    (with-imported-modules (source-module-closure+                            '((gnu build linux-modules)))+      #~(begin+          (use-modules (gnu build linux-modules) (ice-9 getopt-long)+                       (ice-9 match) (srfi srfi-1) (ice-9 ftw))+          (define (find-only-entry directory)+            (match (scandir directory)+             (("." ".." basename)+              (string-append directory "/" basename))))+          (define (resolve-alias alias)+            (let* ((linux-release-module-directory+                    (find-only-entry (string-append "/lib/modules"))))+              (match (delete-duplicates (matching-modules alias+                      (known-module-aliases+                        (string-append linux-release-module-directory+                                       "/modules.alias"))))+               (()+                (error "no alias by that name" alias))+               (items+                items))))+          (define (lookup-module module)+            (let* ((linux-release-module-directory+                    (find-only-entry (string-append "/lib/modules")))+                   (file-name (string-append linux-release-module-directory+                                             "/" (ensure-dot-ko module))))+              (if (file-exists? file-name)+                  file-name+                  (error "no module file found for module" module))))+          (define option-spec+           '((quiet    (single-char #\q) (value #f))))+          (define options+            (getopt-long (command-line) option-spec))+          (when (option-ref options 'quiet #f)+            (current-error-port (%make-void-port "w"))+            (current-output-port (%make-void-port "w")))+          (let ((exit-status 0))+            (for-each (match-lambda+                        (('quiet . #t)+                         #f)+                        ((() modules ...)+                         (for-each (lambda (alias)+                                     (catch #t+                                       (lambda ()+                                         (let ((modules (resolve-alias alias)))                                           (for-each (lambda (module)+                                                       (load-linux-module*+                                                        (lookup-module module)+                                                        #:lookup-module+                                                        lookup-module))+                                                     modules)))+                                       (lambda (key . args)+                                         (display (cons* key args)+                                                  (current-error-port))+                                         (newline (current-error-port))+                                         (set! exit-status 1))))+                                   modules)))+                      options)+            (exit exit-status))))+  #:guile guile))  (define* (expression->initrd exp                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."   (define init     (program-file "init" exp #:guile guile)) +  (define modprobe+    (%modprobe linux-module-directory #:guile guile))+   (define builder     (with-imported-modules (source-module-closure                             '((gnu build linux-initrd)))@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init-                        ;; Copy everything INIT refers to into the initrd.-                        #:references-graphs '("closure")+                        #:modprobe #$modprobe+                        #:linux-module-directory #$linux-module-directory+                        ;; Copy everything INIT and MODPROBE refer to into the+                        ;; initrd.+                        #:references-graphs '("init-closure"+                                              "modprobe-closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))    (gexp->derivation name builder-                    #:references-graphs `(("closure" ,init))))+                    #:references-graphs `(("init-closure" ,init)+                                          ("modprobe-closure" ,modprobe))))  (define (flat-linux-module-directory linux modules)   "Return a flat directory containing the Linux kernel modules listed in@@ -111,7 +181,7 @@ MODULES and taken from LINUX."                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex)+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)                        (srfi srfi-1)                        (guix build utils)                        (gnu build linux-modules))@@ -140,14 +210,18 @@ MODULES and taken from LINUX."                       (recursive-module-dependencies modules                                                      #:lookup-module lookup)))) -          (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))-+          (define version+            (match+             (filter+              (lambda (name)+                (not (string-prefix? "." name)))+              (scandir module-dir))+             ((item) item)))++          (let ((output (string-append #$output "/lib/modules/" version)))+            (mkdir-p output)+            (install-module-files (delete-duplicates modules) output))+          #t)))   (computed-file "linux-modules" build-exp))  (define* (raw-initrd file-systems@@ -227,6 +301,7 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))

Danny Milosavljevic wrote on 2 Mar 2018 17:47

Re: [PATCH v7 3/6] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)

Message-ID:20180302174711.4018dc49@scratchpost.org

Toggle quote (8 lines)

> + (define (load-kernel-modules)> + "Examine /sys/devices to find out which modules to load and load them."> + (define enter?> + (const #t))> + (define (down! path stat result)> + ;; Note: modprobe mutates the tree starting with path.> + (let ((modalias-name (string-append path "/modalias")))

I should rename "path" to "directory".

Danny Milosavljevic wrote on 2 Mar 2018 17:47

Re: [PATCH v7 1/6] linux-modules: Add module-aliases.

Recipients:(address . 30604@debbugs.gnu.org)

Message-ID:20180302174751.741d5dd3@scratchpost.org

Toggle quote (1 lines)

> + "Return the list of aliases for FILE."

^^^ replace by "in"

Danny Milosavljevic wrote on 3 Mar 2018 14:55

[PATCH v8 0/7] Load Linux module only when supported hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180303135533.6112-1-dannym@scratchpost.org

Danny Milosavljevic (7):  linux-modules: Add module-aliases.  linux-modules: Add install-modules.  linux-boot: Load kernel modules only when the hardware is present.  vm: Allow qemu-image builder to load Linux kernel modules.  vm: Make the virtio-blk uniquely identifiable in /sys.  linux-initrd: Provide modprobe to the initrd.  linux-initrd: Factorize %modprobe and flat-linux-module-directory.
 gnu/build/linux-boot.scm    |  42 ++++++++++++--- gnu/build/linux-initrd.scm  |  13 ++++- gnu/build/linux-modules.scm | 115 +++++++++++++++++++++++++++++++++++++++++ gnu/build/vm.scm            |   2 +- gnu/system/linux-initrd.scm | 121 ++++++++++++++++++++++++++++++++------------ gnu/system/vm.scm           |  34 +++++++++++-- 6 files changed, 281 insertions(+), 46 deletions(-)

Danny Milosavljevic wrote on 3 Mar 2018 14:55

[PATCH v8 1/7] linux-modules: Add module-aliases.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180303135533.6112-2-dannym@scratchpost.org

* gnu/build/linux-modules.scm (module-aliases): New variable.--- gnu/build/linux-modules.scm | 9 +++++++++ 1 file changed, 9 insertions(+)

Toggle diff (26 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 4a6d4ff08..364339df9 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -30,6 +30,7 @@   #:use-module (ice-9 rdelim)   #:export (dot-ko             ensure-dot-ko+            module-aliases             module-dependencies             recursive-module-dependencies             modules-loaded@@ -95,6 +96,14 @@ contains module names, not actual file names."       (('depends . what)        (string-tokenize what %not-comma))))) +(define (module-aliases file)+  "Return the list of aliases of module FILE."+  (let ((info (modinfo-section-contents file)))+    (filter-map (match-lambda+                 (('alias . value)+                  value)+                 (_ #f)) (modinfo-section-contents file))))+ (define dot-ko   (cut string-append <> ".ko"))

Danny Milosavljevic wrote on 3 Mar 2018 14:55

[PATCH v8 2/7] linux-modules: Add install-modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180303135533.6112-3-dannym@scratchpost.org

* gnu/build/linux-modules.scm (install-modules): New procedure.(%not-dash): New variable.--- gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)

Toggle diff (68 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 364339df9..af217c974 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -36,6 +36,7 @@             modules-loaded             module-loaded?             load-linux-module*+            install-module-files              current-module-debugging-port @@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                       module)))               known-aliases)) +(define %not-dash+  (char-set-complement (char-set #\-)))++(define (install-module-files module-files output)+  "Install MODULE-FILES to OUTPUT.+Precondition: OUTPUT is an empty directory."+  (let ((aliases+         (map (lambda (module-file-name)+                (format #t "copying '~a'...~%" module-file-name)+                (copy-file module-file-name+                           (string-append output "/"+                                          (basename module-file-name)))+                `(,(file-name->module-name module-file-name) .+                  ,(module-aliases module-file-name)))+              (sort module-files string<))))+    (call-with-output-file (string-append output "/modules.alias")+      (lambda (port)+        (format port+                "# Aliases extracted from modules themselves.\n")+        (for-each (match-lambda ((module . aliases)+                                 (for-each (lambda (alias)+                                             (format port "alias ~a ~a\n" alias+                                                     module))+                                           aliases)))+                  aliases)))+    (call-with-output-file (string-append output "/modules.devname")+      (lambda (port)+        (format port+                "# Device nodes to trigger on-demand module loading.\n")+        (let* ((aliases (append-map (match-lambda+                                     ((module . aliases) aliases))+                                    aliases))+               (devname #f))+          ;; Note: there's only one devname and then only one (char-major|block-major).+          (for-each+           (match-lambda+            (((? (cut string-prefix? "devname:" <>) alias) . value)+             (set! devname (string-drop value (string-length "devname:"))))+            (((? (cut string-prefix? "char-major-" <>) alias) . value)+             (let ((parts (string-tokenize %not-dash)))+               (match parts+                      ((a b major minor)+                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))+            (((? (cut string-prefix? "block-major-" <>) alias) . value)+             (let ((parts (string-tokenize %not-dash)))+               (match parts+                      ((a b major minor)+                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))+            (_ #f))+           aliases))))))+ ;;; linux-modules.scm ends here

Danny Milosavljevic wrote on 3 Mar 2018 14:55

[PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180303135533.6112-4-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.(lookup-module): Delete procedure.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 42 +++++++++++++++++++++++++++++++++++------- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 38 insertions(+), 8 deletions(-)

Toggle diff (75 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..1b16f267a 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -469,9 +469,31 @@ upon error."              mounts)         "ext4")) -  (define (lookup-module name)-    (string-append linux-module-directory "/"-                   (ensure-dot-ko name)))+  (define (load-kernel-modules)+    "Examine /sys/devices to find out which modules to load and load them."+    (define enter?+      (const #t))+    (define (down! directory stat result)+     ;; Note: modprobe mutates the tree starting with DIRECTORY.+     (let ((modalias-name (string-append directory "/modalias")))+       (if (file-exists? modalias-name)+           (let ((modalias+                 (string-trim-right (call-with-input-file modalias-name+                                                          read-string)+                                    #\newline)))+             (system* "/sbin/modprobe" "-q" "--" modalias))))+       #t)+    (define up+      (const #t))+    (define skip+      (const #t))+    (define leaf+      (const #t))+    (define (error name stat errno result)+      (format (current-error-port) "warning: ~a: ~a~%"+              name (strerror errno))+      result)+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))    (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n")@@ -486,10 +508,16 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (let* ((kernel-release+               (utsname:release (uname)))+              (directory+               (string-append linux-module-directory "/lib/modules/"+                              kernel-release))+              (old-umask (umask #o022)))+         (make-static-device-nodes directory)+         (umask old-umask))++       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex e7f97bb88..b50d3ff80 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -208,7 +208,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Danny Milosavljevic wrote on 3 Mar 2018 14:55

[PATCH v8 4/7] vm: Allow qemu-image builder to load Linux kernel modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180303135533.6112-5-dannym@scratchpost.org

* gnu/system/vm.scm (%modprobe-wrapper): New variable.(qemu-image): Modify.--- gnu/system/vm.scm | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)

Toggle diff (74 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 345cecedd..b5a559012 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."    #:single-file-output? #t    #:references-graphs inputs)) +(define (%modprobe-wrapper modprobe linux-module-directory)+  ;; Wrapper for the 'modprobe' command that knows where modules live.+  ;;+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'+  ;; environment variable is not set---hence the need for this wrapper.+  (program-file "modprobe"+    #~(begin+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))+ (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))   (expression->derivation-in-linux-vm    name-   (with-imported-modules (source-module-closure '((gnu build bootloader)+   (with-imported-modules (source-module-closure '((gnu build activation)+                                                   (gnu build bootloader)                                                    (gnu build vm)                                                    (guix build utils)))      #~(begin-         (use-modules (gnu build bootloader)+         (use-modules (gnu build activation)+                      (gnu build bootloader)                       (gnu build vm)                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports))           (let ((inputs-                '#$(append (list qemu parted e2fsprogs dosfstools)+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)                            (map canonical-package                                 (list sed grep coreutils findutils gawk))                            (if register-closures? (list guix) '())))@@ -302,6 +317,14 @@ the image."                         inputs)))             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)+           ;; It's possible that we need to load nls modules in order to+           ;; mount the new partition.+           (if (file-exists? #$modprobe-name)+               (activate-modprobe #$(%modprobe-wrapper modprobe-name+                                     linux-module-directory))+               (format (current-error-port)+                "WARNING: No modprobe found in ~s.  \+Loading kernel modules will be impossible.\n" #$modprobe-name))             (let* ((graphs     '#$(match inputs                                    (((names . _) ...)@@ -364,7 +387,7 @@ the image."    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-   #:references-graphs inputs))+   #:references-graphs inputs)))   ;;;

Danny Milosavljevic wrote on 3 Mar 2018 14:55

[PATCH v8 5/7] vm: Make the virtio-blk uniquely identifiable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180303135533.6112-6-dannym@scratchpost.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.--- gnu/build/vm.scm  | 2 +- gnu/system/vm.scm | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..ebf9e9f6e 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex b5a559012..fdff64ed9 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os

Danny Milosavljevic wrote on 3 Mar 2018 14:55

[PATCH v8 6/7] linux-initrd: Provide modprobe to the initrd.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180303135533.6112-7-dannym@scratchpost.org

* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and thelinux modules to the initrd.* gnu/system/linux-initrd.scm (%modprobe): New procedure.(expression->initrd): Use it.  Add linux-module-directory.(raw-initrd): Pass linux-module-directory.--- gnu/build/linux-initrd.scm  | 13 +++++- gnu/system/linux-initrd.scm | 99 +++++++++++++++++++++++++++++++++++++++------ 2 files changed, 99 insertions(+), 13 deletions(-)

Toggle diff (184 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..d4cb5e2d8 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init modprobe linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +     ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (when modprobe+      (mkdir-p "sbin")+      (symlink modprobe "sbin/modprobe")+      (compile-to-cache "sbin/modprobe"))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex b50d3ff80..8050ac47e 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -56,11 +56,73 @@ ;;; ;;; Code: +(define* (%modprobe linux-module-directory #:key+                    (guile %guile-static-stripped))+  (program-file "modprobe"+    (with-imported-modules (source-module-closure+                            '((gnu build linux-modules)))+      #~(begin+          (use-modules (gnu build linux-modules) (ice-9 getopt-long)+                       (ice-9 match) (srfi srfi-1) (ice-9 ftw))+          (define (find-only-entry directory)+            (match (scandir directory)+             (("." ".." basename)+              (string-append directory "/" basename))))+          (define (resolve-alias alias)+            (let* ((linux-release-module-directory+                    (find-only-entry (string-append "/lib/modules"))))+              (match (delete-duplicates (matching-modules alias+                      (known-module-aliases+                        (string-append linux-release-module-directory+                                       "/modules.alias"))))+               (()+                (error "no alias by that name" alias))+               (items+                items))))+          (define (lookup-module module)+            (let* ((linux-release-module-directory+                    (find-only-entry (string-append "/lib/modules")))+                   (file-name (string-append linux-release-module-directory+                                             "/" (ensure-dot-ko module))))+              (if (file-exists? file-name)+                  file-name+                  (error "no module file found for module" module))))+          (define option-spec+           '((quiet    (single-char #\q) (value #f))))+          (define options+            (getopt-long (command-line) option-spec))+          (when (option-ref options 'quiet #f)+            (current-error-port (%make-void-port "w"))+            (current-output-port (%make-void-port "w")))+          (let ((exit-status 0))+            (for-each (match-lambda+                        (('quiet . #t)+                         #f)+                        ((() modules ...)+                         (for-each (lambda (alias)+                                     (catch #t+                                       (lambda ()+                                         (let ((modules (resolve-alias alias)))                                           (for-each (lambda (module)+                                                       (load-linux-module*+                                                        (lookup-module module)+                                                        #:lookup-module+                                                        lookup-module))+                                                     modules)))+                                       (lambda (key . args)+                                         (display (cons* key args)+                                                  (current-error-port))+                                         (newline (current-error-port))+                                         (set! exit-status 1))))+                                   modules)))+                      options)+            (exit exit-status))))+  #:guile guile))  (define* (expression->initrd exp                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."   (define init     (program-file "init" exp #:guile guile)) +  (define modprobe+    (%modprobe linux-module-directory #:guile guile))+   (define builder     (with-imported-modules (source-module-closure                             '((gnu build linux-initrd)))@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init-                        ;; Copy everything INIT refers to into the initrd.-                        #:references-graphs '("closure")+                        #:modprobe #$modprobe+                        #:linux-module-directory #$linux-module-directory+                        ;; Copy everything INIT and MODPROBE refer to into the+                        ;; initrd.+                        #:references-graphs '("init-closure"+                                              "modprobe-closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))    (gexp->derivation name builder-                    #:references-graphs `(("closure" ,init))))+                    #:references-graphs `(("init-closure" ,init)+                                          ("modprobe-closure" ,modprobe))))  (define (flat-linux-module-directory linux modules)   "Return a flat directory containing the Linux kernel modules listed in@@ -111,7 +181,7 @@ MODULES and taken from LINUX."                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex)+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)                        (srfi srfi-1)                        (guix build utils)                        (gnu build linux-modules))@@ -140,14 +210,18 @@ MODULES and taken from LINUX."                       (recursive-module-dependencies modules                                                      #:lookup-module lookup)))) -          (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))-+          (define version+            (match+             (filter+              (lambda (name)+                (not (string-prefix? "." name)))+              (scandir module-dir))+             ((item) item)))++          (let ((output (string-append #$output "/lib/modules/" version)))+            (mkdir-p output)+            (install-module-files (delete-duplicates modules) output))+          #t)))   (computed-file "linux-modules" build-exp))  (define* (raw-initrd file-systems@@ -227,6 +301,7 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))

Danny Milosavljevic wrote on 3 Mar 2018 14:55

[PATCH v8 7/7] linux-initrd: Factorize %modprobe and flat-linux-module-directory.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180303135533.6112-8-dannym@scratchpost.org

* gnu/build/linux-modules.scm (module-aliases->module-file-names): Newprocedure.* gnu/system/linux-initrd.scm (%modprobe): Usemodule-aliases->module-file-names.(flat-linux-module-directory): Use module-aliases->module-file-names.--- gnu/build/linux-modules.scm |  56 +++++++++++++++++++++- gnu/system/linux-initrd.scm | 110 ++++++++++++++++++-------------------------- 2 files changed, 100 insertions(+), 66 deletions(-)

Toggle diff (249 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex af217c974..44059ad93 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -21,6 +21,7 @@   #:use-module (guix elf)   #:use-module (guix glob)   #:use-module (guix build syscalls)+  #:use-module (guix build utils) ; find-files   #:use-module (rnrs io ports)   #:use-module (rnrs bytevectors)   #:use-module (srfi srfi-1)@@ -28,9 +29,12 @@   #:use-module (ice-9 vlist)   #:use-module (ice-9 match)   #:use-module (ice-9 rdelim)+  #:use-module (ice-9 regex)+  #:use-module (ice-9 ftw)   #:export (dot-ko             ensure-dot-ko             module-aliases+            module-aliases->module-file-names             module-dependencies             recursive-module-dependencies             modules-loaded@@ -385,7 +389,7 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by  (define (install-module-files module-files output)   "Install MODULE-FILES to OUTPUT.-Precondition: OUTPUT is an empty directory."+Precondition: OUTPUT is an empty directory except for \"modules.builtin\"."   (let ((aliases          (map (lambda (module-file-name)                 (format #t "copying '~a'...~%" module-file-name)@@ -431,4 +435,54 @@ Precondition: OUTPUT is an empty directory."             (_ #f))            aliases)))))) +(define (module-aliases->module-file-names linux aliases)+  "Resolve ALIASES to module file names, including their dependencies (which will appear+first).  Each alias will map to a list of module file names.+LINUX is the directory containing \"lib\"."+  (define (string->regexp str)+    ;; Return a regexp that matches STR exactly.+    (string-append "^" (regexp-quote str) "$"))++  (define module-dir+    (string-append linux "/lib/modules"))++  (define (find-only-entry directory)+    (match (scandir directory)+     (("." ".." basename)+      (string-append directory "/" basename))))++  (define linux-release-module-directory+    (find-only-entry module-dir))++  (define known-module-aliases*+    (known-module-aliases+     (string-append linux-release-module-directory+                    "/modules.alias")))+  (define (resolve-alias alias)+    "If possible, resolve ALIAS to a list of module names.+Otherwise return just ALIAS as possible module names."+    (match (delete-duplicates (matching-modules alias+                                                known-module-aliases*))+           (()+            (list alias))+           (items+            items)))++  (define (lookup module)+    (let ((name (ensure-dot-ko module)))+      (match (find-files module-dir (string->regexp name))+             ((file)+              file)+             (()+              (error "module not found" name module-dir))+             ((_ ...)+              (error "several modules by that name"+                     name module-dir)))))+  (append-map (lambda (alias)+                (let ((modules (map lookup (resolve-alias alias))))+                  (append (recursive-module-dependencies modules+                                                         #:lookup-module+                                                         lookup) modules)))+              aliases))+ ;;; linux-modules.scm ends herediff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 8050ac47e..dc826c63e 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -58,35 +58,14 @@  (define* (%modprobe linux-module-directory #:key                     (guile %guile-static-stripped))+  "Minimal implementation of modprobe for our initrd.+LINUX-MODULE-DIRECTORY is the directory that contains \"lib\"."   (program-file "modprobe"     (with-imported-modules (source-module-closure                             '((gnu build linux-modules)))       #~(begin           (use-modules (gnu build linux-modules) (ice-9 getopt-long)-                       (ice-9 match) (srfi srfi-1) (ice-9 ftw))-          (define (find-only-entry directory)-            (match (scandir directory)-             (("." ".." basename)-              (string-append directory "/" basename))))-          (define (resolve-alias alias)-            (let* ((linux-release-module-directory-                    (find-only-entry (string-append "/lib/modules"))))-              (match (delete-duplicates (matching-modules alias-                      (known-module-aliases-                        (string-append linux-release-module-directory-                                       "/modules.alias"))))-               (()-                (error "no alias by that name" alias))-               (items-                items))))-          (define (lookup-module module)-            (let* ((linux-release-module-directory-                    (find-only-entry (string-append "/lib/modules")))-                   (file-name (string-append linux-release-module-directory-                                             "/" (ensure-dot-ko module))))-              (if (file-exists? file-name)-                  file-name-                  (error "no module file found for module" module))))+                       (ice-9 match) (srfi srfi-1))           (define option-spec            '((quiet    (single-char #\q) (value #f))))           (define options@@ -98,22 +77,31 @@             (for-each (match-lambda                         (('quiet . #t)                          #f)-                        ((() modules ...)-                         (for-each (lambda (alias)-                                     (catch #t-                                       (lambda ()-                                         (let ((modules (resolve-alias alias)))                                           (for-each (lambda (module)-                                                       (load-linux-module*-                                                        (lookup-module module)-                                                        #:lookup-module-                                                        lookup-module))-                                                     modules)))-                                       (lambda (key . args)-                                         (display (cons* key args)-                                                  (current-error-port))-                                         (newline (current-error-port))-                                         (set! exit-status 1))))-                                   modules)))+                        ((() aliases ...)+                         (catch #t+                           (lambda ()+                             (let ((module-file-names+                                    (module-aliases->module-file-names+                                     #$linux-module-directory aliases)))+                               (for-each (lambda (name)+                                           (catch 'system-error+                                             (lambda ()+                                               (when (not (load-linux-module* name+                                                                              #:recursive?+                                                                              #f))+                                                 (set! exit-status 1)))+                                             (lambda (key . args)+                                               (when (not (= EEXIST+                                                             (system-error-errno+                                                              (cons key args))))+                                                 (print-exception (current-error-port)+                                                                  #f key args)+                                                 (set! exit-status 1)))))+                                         module-file-names)))+                           (lambda (key . args)+                             (print-exception (current-error-port)+                                              #f key args)+                             (set! exit-status 1)))))                       options)             (exit exit-status))))   #:guile guile))@@ -173,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."                     #:references-graphs `(("init-closure" ,init)                                           ("modprobe-closure" ,modprobe)))) -(define (flat-linux-module-directory linux modules)-  "Return a flat directory containing the Linux kernel modules listed in-MODULES and taken from LINUX."+(define (flat-linux-module-directory linux aliases)+  "Return a flat directory containing the Linux kernel modules resolved by+ALIASES and taken from LINUX."   (define build-exp     (with-imported-modules (source-module-closure                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)+          (use-modules (ice-9 match) (ice-9 ftw)                        (srfi srfi-1)-                       (guix build utils)+                       (guix build utils) ; TODO: Remove                        (gnu build linux-modules))            (define (string->regexp str)@@ -193,33 +181,25 @@ MODULES and taken from LINUX."           (define module-dir             (string-append #$linux "/lib/modules")) -          (define (lookup module)-            (let ((name (ensure-dot-ko module)))-              (match (find-files module-dir (string->regexp name))-                ((file)-                 file)-                (()-                 (error "module not found" name module-dir))-                ((_ ...)-                 (error "several modules by that name"-                        name module-dir)))))+          (define (find-only-entry directory)+            (match (scandir directory)+             (("." ".." basename)+              (string-append directory "/" basename))))++          (define linux-release-module-directory+            (find-only-entry module-dir))            (define modules-            (let ((modules (map lookup '#$modules)))-              (append modules-                      (recursive-module-dependencies modules-                                                     #:lookup-module lookup))))+            (module-aliases->module-file-names #$linux '#$aliases))            (define version-            (match-             (filter-              (lambda (name)-                (not (string-prefix? "." name)))-              (scandir module-dir))-             ((item) item)))+            (basename linux-release-module-directory))            (let ((output (string-append #$output "/lib/modules/" version)))             (mkdir-p output)+            (install-file+             (string-append linux-release-module-directory "/modules.builtin")+             output)             (install-module-files (delete-duplicates modules) output))           #t)))   (computed-file "linux-modules" build-exp))

Danny Milosavljevic wrote on 3 Mar 2018 16:32

Re: [PATCH v8 2/7] linux-modules: Add install-modules.

Recipients:(address . 30604@debbugs.gnu.org)

Message-ID:20180303163212.40f17905@scratchpost.org

Toggle quote (18 lines)> +          ;; Note: there's only one devname and then only one (char-major|block-major).> +          (for-each> +           (match-lambda> +            (((? (cut string-prefix? "devname:" <>) alias) . value)> +             (set! devname (string-drop value (string-length "devname:"))))> +            (((? (cut string-prefix? "char-major-" <>) alias) . value)> +             (let ((parts (string-tokenize %not-dash)))> +               (match parts> +                      ((a b major minor)> +                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))> +            (((? (cut string-prefix? "block-major-" <>) alias) . value)> +             (let ((parts (string-tokenize %not-dash)))> +               (match parts> +                      ((a b major minor)> +                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))> +            (_ #f))> +           aliases))))))

Probably better to be more careful that devname is set early enough.

Danny Milosavljevic wrote on 3 Mar 2018 19:01

Re: [PATCH v8 7/7] linux-initrd: Factorize %modprobe and flat-linux-module-directory.

Recipients:(address . 30604@debbugs.gnu.org)

Message-ID:20180303190131.5a400f7d@scratchpost.org

Toggle quote (2 lines)

> + (guix build utils) ; TODO: Remove

Required for mkdir-p

Ludovic Courtès wrote on 3 Mar 2018 22:58

Re: [bug#30604] [PATCH v8 1/7] linux-modules: Add module-aliases.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87371g6ctv.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (2 lines)

> * gnu/build/linux-modules.scm (module-aliases): New variable.

LGTM!

Toggle quote (8 lines)

> +(define (module-aliases file)> + "Return the list of aliases of module FILE."> + (let ((info (modinfo-section-contents file)))> + (filter-map (match-lambda> + (('alias . value)> + value)> + (_ #f)) (modinfo-section-contents file))))

Nitpick: align like this:
(filter-map first second)
when the first arg spans several lines.

Ludovic Courtès wrote on 3 Mar 2018 23:07

Re: [bug#30604] [PATCH v8 2/7] linux-modules: Add install-modules.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87y3j84xtu.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (3 lines)

> * gnu/build/linux-modules.scm (install-modules): New procedure.> (%not-dash): New variable.

We could reuse modules.alias and modules.devname from the ‘linux-libre’package (right?), but I guess it doesn’t hurt to generate custom ones.

Toggle quote (48 lines)> +(define (install-module-files module-files output)> +  "Install MODULE-FILES to OUTPUT.> +Precondition: OUTPUT is an empty directory."> +  (let ((aliases> +         (map (lambda (module-file-name)> +                (format #t "copying '~a'...~%" module-file-name)> +                (copy-file module-file-name> +                           (string-append output "/"> +                                          (basename module-file-name)))> +                `(,(file-name->module-name module-file-name) .> +                  ,(module-aliases module-file-name)))> +              (sort module-files string<))))> +    (call-with-output-file (string-append output "/modules.alias")> +      (lambda (port)> +        (format port> +                "# Aliases extracted from modules themselves.\n")> +        (for-each (match-lambda ((module . aliases)> +                                 (for-each (lambda (alias)> +                                             (format port "alias ~a ~a\n" alias> +                                                     module))> +                                           aliases)))> +                  aliases)))> +    (call-with-output-file (string-append output "/modules.devname")> +      (lambda (port)> +        (format port> +                "# Device nodes to trigger on-demand module loading.\n")> +        (let* ((aliases (append-map (match-lambda> +                                     ((module . aliases) aliases))> +                                    aliases))> +               (devname #f))> +          ;; Note: there's only one devname and then only one (char-major|block-major).> +          (for-each> +           (match-lambda> +            (((? (cut string-prefix? "devname:" <>) alias) . value)> +             (set! devname (string-drop value (string-length "devname:"))))> +            (((? (cut string-prefix? "char-major-" <>) alias) . value)> +             (let ((parts (string-tokenize %not-dash)))> +               (match parts> +                      ((a b major minor)> +                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))> +            (((? (cut string-prefix? "block-major-" <>) alias) . value)> +             (let ((parts (string-tokenize %not-dash)))> +               (match parts> +                      ((a b major minor)> +                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))> +            (_ #f))> +           aliases))))))

I think we need different procedures here:
(write-module-alias-database modules port) ;for “modules.alias” (write-module-device-database modules port) ;for “modules.devname”
with docstrings.
I’m not sure we need ‘install-module-files’ itself. Perhaps we caninline it at the call site?
For the devname code, please avoid ‘set!’. Instead you can thread thecurrent devname as the state of a loop:
(let loop ((devname #f) (aliases aliases)) (match aliases (() …) (((? devname-alias? devname) . rest) (loop devname rest)) …)) The indentation of ‘match’ forms is wrong. Would it be OK for you topass it through ./etc/indent-code.el? (It’s non interactive, you don’tneed to actually use Emacs.)
Thanks,Ludo’.

Ludovic Courtès wrote on 3 Mar 2018 23:48

Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87sh9g4vy1.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (5 lines)

> * gnu/build/linux-boot.scm (boot-system): Load kernel modules only when> the hardware is present.> (lookup-module): Delete procedure.> * gnu/system/linux-initrd.scm (raw-initrd): Add imports.

[...]

Toggle quote (14 lines)> +  (define (load-kernel-modules)> +    "Examine /sys/devices to find out which modules to load and load them."> +    (define enter?> +      (const #t))> +    (define (down! directory stat result)> +     ;; Note: modprobe mutates the tree starting with DIRECTORY.> +     (let ((modalias-name (string-append directory "/modalias")))> +       (if (file-exists? modalias-name)> +           (let ((modalias> +                 (string-trim-right (call-with-input-file modalias-name> +                                                          read-string)> +                                    #\newline)))> +             (system* "/sbin/modprobe" "-q" "--" modalias))))

If we change ‘flat-linux-module-directory’ to produce a ‘modules.alias’file, here we could read ‘modules.aliases’ directly and load the rightthing.
With the patch below, we get ‘needed-modules’, and we could simply do:
(for-each (catch-ENOENT load-linux-module*) (needed-modules (known-module-aliases (string-append linux-module-directory "/modules.alias"))))
and we can do away with kmod’s modprobe.
Thoughts?
Ludo’.

Toggle diff (48 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 4a6d4ff08..251095072 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -20,6 +20,7 @@ (define-module (gnu build linux-modules)   #:use-module (guix elf)   #:use-module (guix glob)+  #:use-module (guix build utils)   #:use-module (guix build syscalls)   #:use-module (rnrs io ports)   #:use-module (rnrs bytevectors)@@ -40,7 +41,8 @@              device-module-aliases             known-module-aliases-            matching-modules))+            matching-modules+            needed-modules))  ;;; Commentary: ;;;@@ -370,4 +372,25 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                       module)))               known-aliases)) +(define (system-device-aliases)+  "Browse /sys/devices in search of \"modalias\" files and return the list of+device aliases for the current system."+  (let ((files (find-files "/sys/devices"+                           (lambda (file stat)+                             (and (eq? 'regular (stat:type stat))+                                  (string=? "modalias" (basename file)))))))+    (filter-map (lambda (file)+                  (match (string-trim-right+                          (call-with-input-file file get-string-all))+                    ("" #f)+                    (alias alias)))+                files)))++(define* (needed-modules #:optional (known-aliases (known-module-aliases)))+  "Return the list of modules needed by devices on the current system.  This+is achieved by browsing /sys/devices and returning the maching modules from+KNOWN-ALIASES."+  (append-map (cut matching-modules <> known-aliases)+              (system-device-aliases)))+ ;;; linux-modules.scm ends here

Danny Milosavljevic wrote on 4 Mar 2018 02:06

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180304020643.6773e016@scratchpost.org

Hi Ludo,
On Sat, 03 Mar 2018 23:48:38 +0100ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (15 lines)> > +             (system* "/sbin/modprobe" "-q" "--" modalias))))  > > If we change ‘flat-linux-module-directory’ to produce a ‘modules.alias’> file, here we could read ‘modules.aliases’ directly and load the right> thing.> > With the patch below, we get ‘needed-modules’, and we could simply do:> >   (for-each (catch-ENOENT load-linux-module*)>             (needed-modules>              (known-module-aliases (string-append linux-module-directory>                                                   "/modules.alias"))))> > and we can do away with kmod’s modprobe.

It's not kmod's modprobe anymore. It's our pure-Guile implementation.
Linux lazy-invokes modprobe (for example when mounting stuff), so/sbin/modprobe is never going away - but it can be our implementation.
I doubt it will take the modules from the correct directory with your patch.(Of course otherwise it looks much nicer - but I think it won't pass the tests)
My newest version (v9) will use the same procedure for both computing the listof modules for flat-linux-module-directory and the list of modules that are tobe modprobed - I think it's nice to be able to keep those in sync so we don'tget nasty surprises. Let's see how that goes...

Danny Milosavljevic wrote on 4 Mar 2018 02:09

[PATCH v9 0/7] Load Linux module only when supported hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180304010914.1322-1-dannym@scratchpost.org

Danny Milosavljevic (7):  linux-modules: Add "modules.devname" and "modules.alias" writer.  linux-modules: Add module-aliases->module-file-names.  linux-initrd: Provide pure-Guile modprobe.  linux-boot: Load kernel modules only when the hardware is present.  vm: Allow qemu-image builder to load Linux kernel modules.  vm: Make the virtio-blk uniquely identifiable in /sys.  linux-initrd: Use module-aliases->module-file-names, too.
 gnu/build/linux-boot.scm    |  42 +++++++++++--- gnu/build/linux-initrd.scm  |  13 ++++- gnu/build/linux-modules.scm | 114 +++++++++++++++++++++++++++++++++++++ gnu/build/vm.scm            |   2 +- gnu/system/linux-initrd.scm | 136 +++++++++++++++++++++++++++++++++----------- gnu/system/vm.scm           |  34 +++++++++-- 6 files changed, 295 insertions(+), 46 deletions(-)

Danny Milosavljevic wrote on 4 Mar 2018 02:09

[PATCH v9 1/7] linux-modules: Add "modules.devname" and "modules.alias" writer.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180304010914.1322-2-dannym@scratchpost.org

* gnu/build/linux-modules.scm (write-module-alias-database): New procedure.(write-module-device-database): New procedure.(%not-dash): New variable.--- gnu/build/linux-modules.scm | 58 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)

Toggle diff (74 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 4fe673cca..0aaf2ff6f 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -36,6 +36,8 @@             modules-loaded             module-loaded?             load-linux-module*+            write-module-alias-database+            write-module-device-database              current-module-debugging-port @@ -380,4 +382,60 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                       module)))               known-aliases)) +(define %not-dash+  (char-set-complement (char-set #\-)))++(define (write-module-alias-database aliases output)+  "Install \"modules.alias\" for ALIASES to directory OUTPUT."+  (call-with-output-file (string-append output "/modules.alias")+    (lambda (port)+      (format port+              "# Aliases extracted from modules themselves.\n")+      (for-each (match-lambda ((module . aliases)+                               (for-each (lambda (alias)+                                           (format port "alias ~a ~a\n" alias+                                                   module))+                                         aliases)))+                aliases))))++(define (write-module-device-database aliases output)+  "Install \"modules.devname\" for ALIASES to directory OUTPUT."+  (call-with-output-file (string-append output "/modules.devname")+    (lambda (port)+      (format port+              "# Device nodes to trigger on-demand module loading.\n")+      (for-each (match-lambda+                 ((module . aliases)+                  (let* ((interesting-aliases+                          ;; Note: there's only one devname and then only one+                          ;; (char-major|block-major).+                          (filter-map+                           (match-lambda+                            ((? (cut string-prefix? "devname:" <>) alias)+                             `(devname . ,(string-drop alias (string-length "devname:"))))+                            ((? (cut string-prefix? "char-major-" <>) alias)+                             `(char-major . ,(string-drop alias (string-length "char-major-"))))+                            ((? (cut string-prefix? "block-major-" <>) alias)+                             `(block-major . ,(string-drop alias (string-length "block-major-"))))+                            (_ #f))+                           aliases))+                         (devname (assq-ref interesting-aliases+                                            'devname))+                         (char-major (assq-ref interesting-aliases+                                               'char-major))+                         (block-major (assq-ref interesting-aliases+                                               'block-major)))+                    (when (and devname char-major)+                      (let ((parts (string-tokenize char-major %not-dash)))+                        (match parts+                         ((major minor)+                          (format port "~a ~a ~a~a:~a\n" module devname+                                       "c" major minor)))))+                    (when (and devname block-major)+                      (let ((parts (string-tokenize block-major %not-dash)))+                        (match parts+                         ((major minor)+                          (format port "~a ~a ~a~a:~a\n" module devname+                                       "b" major minor)))))))) aliases))))+ ;;; linux-modules.scm ends here

Danny Milosavljevic wrote on 4 Mar 2018 02:09

[PATCH v9 2/7] linux-modules: Add module-aliases->module-file-names.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180304010914.1322-3-dannym@scratchpost.org

* gnu/build/linux-modules.scm (module-aliases->module-file-names): Newprocedure.--- gnu/build/linux-modules.scm | 55 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)

Toggle diff (81 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 0aaf2ff6f..f6bb0512b 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -21,6 +21,7 @@   #:use-module (guix elf)   #:use-module (guix glob)   #:use-module (guix build syscalls)+  #:use-module (guix build utils) ; find-files   #:use-module (rnrs io ports)   #:use-module (rnrs bytevectors)   #:use-module (srfi srfi-1)@@ -28,9 +29,12 @@   #:use-module (ice-9 vlist)   #:use-module (ice-9 match)   #:use-module (ice-9 rdelim)+  #:use-module (ice-9 regex)+  #:use-module (ice-9 ftw)   #:export (dot-ko             ensure-dot-ko             module-aliases+            module-aliases->module-file-names             module-dependencies             recursive-module-dependencies             modules-loaded@@ -438,4 +442,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                           (format port "~a ~a ~a~a:~a\n" module devname                                        "b" major minor)))))))) aliases)))) +(define (module-aliases->module-file-names linux aliases)+  "Resolve ALIASES to module file names, including their dependencies (which will appear+first).  Each alias will map to a list of module file names.+LINUX is the directory containing \"lib\"."+  (define (string->regexp str)+    ;; Return a regexp that matches STR exactly.+    (string-append "^" (regexp-quote str) "$"))++  (define module-dir+    (string-append linux "/lib/modules"))++  (define (find-only-entry directory)+    (match (scandir directory)+     (("." ".." basename)+      (string-append directory "/" basename))))++  (define linux-release-module-directory+    (find-only-entry module-dir))++  (define known-module-aliases*+    (known-module-aliases+     (string-append linux-release-module-directory+                    "/modules.alias")))+  (define (resolve-alias alias)+    "If possible, resolve ALIAS to a list of module names.+Otherwise return just ALIAS as possible module names."+    (match (delete-duplicates (matching-modules alias+                                                known-module-aliases*))+           (()+            (list alias))+           (items+            items)))++  (define (lookup module)+    (let ((name (ensure-dot-ko module)))+      (match (find-files module-dir (string->regexp name))+             ((file)+              file)+             (()+              (error "module not found" name module-dir))+             ((_ ...)+              (error "several modules by that name"+                     name module-dir)))))+  (append-map (lambda (alias)+                (let ((modules (map lookup (resolve-alias alias))))+                  (append (recursive-module-dependencies modules+                                                         #:lookup-module+                                                         lookup)+                          modules)))+              aliases))+ ;;; linux-modules.scm ends here

Danny Milosavljevic wrote on 4 Mar 2018 02:09

[PATCH v9 3/7] linux-initrd: Provide pure-Guile modprobe.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180304010914.1322-4-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (%modprobe): New variable.(expression->initrd): Add modprobe, LINUX-MODULE-DIRECTORY.(raw-initrd): Pass KODIR as LINUX-MODULE-DIRECTORY.* gnu/build/linux-initrd.scm (build-initrd): Add modprobe.--- gnu/build/linux-initrd.scm  | 13 ++++++++- gnu/system/linux-initrd.scm | 65 ++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 74 insertions(+), 4 deletions(-)

Toggle diff (144 lines)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..d4cb5e2d8 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init modprobe linux-module-directory                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +     ;; Make modprobe available as /sbin/modprobe so the kernel finds it.+    (when modprobe+      (mkdir-p "sbin")+      (symlink modprobe "sbin/modprobe")+      (compile-to-cache "sbin/modprobe"))++    ;; Make modules available as /lib/modules so modprobe finds them.+    (mkdir-p "lib")+    (symlink (string-append linux-module-directory "/lib/modules")+             "lib/modules")+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex e0cb59c00..6ad6d75f7 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -46,7 +46,8 @@             %base-initrd-modules             raw-initrd             file-system-packages-            base-initrd))+            base-initrd+            %modprobe))   ;;; Commentary:@@ -56,11 +57,61 @@ ;;; ;;; Code: +(define* (%modprobe linux-module-directory #:key+                    (guile %guile-static-stripped))+  "Minimal implementation of modprobe for our initrd.+LINUX-MODULE-DIRECTORY is the directory that contains \"lib\"."+  (program-file "modprobe"+    (with-imported-modules (source-module-closure+                            '((gnu build linux-modules)))+      #~(begin+          (use-modules (gnu build linux-modules) (ice-9 getopt-long)+                       (ice-9 match) (srfi srfi-1))+          (define option-spec+           '((quiet    (single-char #\q) (value #f))))+          (define options+            (getopt-long (command-line) option-spec))+          (when (option-ref options 'quiet #f)+            (current-error-port (%make-void-port "w"))+            (current-output-port (%make-void-port "w")))+          (let ((exit-status 0))+            (for-each (match-lambda+                        (('quiet . #t)+                         #f)+                        ((() aliases ...)+                         (catch #t+                           (lambda ()+                             (let ((module-file-names+                                    (module-aliases->module-file-names+                                     #$linux-module-directory aliases)))+                               (for-each (lambda (name)+                                           (catch 'system-error+                                             (lambda ()+                                               (when (not (load-linux-module* name+                                                                              #:recursive?+                                                                              #f))+                                                 (set! exit-status 1)))+                                             (lambda (key . args)+                                               (when (not (= EEXIST+                                                             (system-error-errno+                                                              (cons key args))))+                                                 (print-exception (current-error-port)+                                                                  #f key args)+                                                 (set! exit-status 1)))))+                                         module-file-names)))+                           (lambda (key . args)+                             (print-exception (current-error-port)+                                              #f key args)+                             (set! exit-status 1)))))+                      options)+            (exit exit-status))))+  #:guile guile))  (define* (expression->initrd exp                              #:key                              (guile %guile-static-stripped)                              (gzip gzip)+                             linux-module-directory                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)@@ -73,6 +124,9 @@ the derivations referenced by EXP are automatically copied to the initrd."   (define init     (program-file "init" exp #:guile guile)) +  (define modprobe+    (%modprobe linux-module-directory #:guile guile))+   (define builder     (with-imported-modules (source-module-closure                             '((gnu build linux-initrd)))@@ -96,12 +150,16 @@ the derivations referenced by EXP are automatically copied to the initrd."           (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init+                        #:modprobe #$modprobe+                        #:linux-module-directory #$linux-module-directory                         ;; Copy everything INIT refers to into the initrd.-                        #:references-graphs '("closure")+                        #:references-graphs '("init-closure"+                                              "modprobe-closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))    (gexp->derivation name builder-                    #:references-graphs `(("closure" ,init))))+                    #:references-graphs `(("init-closure" ,init)+                                          ("modprobe-closure" ,modprobe))))  (define (flat-linux-module-directory linux modules)   "Return a flat directory containing the Linux kernel modules listed in@@ -225,6 +283,7 @@ upon error."                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:linux-module-directory kodir    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))

Danny Milosavljevic wrote on 4 Mar 2018 02:09

[PATCH v9 4/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180304010914.1322-5-dannym@scratchpost.org

* gnu/build/linux-boot.scm (boot-system): Load kernel modules only whenthe hardware is present.* gnu/system/linux-initrd.scm (raw-initrd): Add imports.--- gnu/build/linux-boot.scm    | 42 +++++++++++++++++++++++++++++++++++------- gnu/system/linux-initrd.scm |  4 +++- 2 files changed, 38 insertions(+), 8 deletions(-)

Toggle diff (75 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..2236d8971 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -469,9 +469,31 @@ upon error."              mounts)         "ext4")) -  (define (lookup-module name)-    (string-append linux-module-directory "/"-                   (ensure-dot-ko name)))+  (define (load-kernel-modules)+    "Examine /sys/devices to find out which modules to load and load them."+    (define enter?+      (const #t))+    (define (down! directory stat result)+     ;; Note: modprobe mutates the tree starting with DIRECTORY.+     (let ((modalias-name (string-append directory "/modalias")))+       (if (file-exists? modalias-name)+           (let ((modalias+                 (string-trim-right (call-with-input-file modalias-name+                                                          read-string)+                                    #\newline)))+             (system* "/sbin/modprobe" "-q" "--" modalias))))+       #t)+    (define up+      (const #t))+    (define skip+      (const #t))+    (define leaf+      (const #t))+    (define (error name stat errno result)+      (format (current-error-port) "warning: ~a: ~a~%"+              name (strerror errno))+      result)+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))    (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n")@@ -486,10 +508,16 @@ upon error."        (when (member "--repl" args)          (start-repl)) -       (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (let* ((kernel-release+               (utsname:release (uname)))+              (directory+               (string-append linux-module-directory "/lib/modules/"+                              kernel-release))+              (old-umask (umask #o022)))+         (make-static-device-nodes directory)+         (umask old-umask))++       (load-kernel-modules)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 6ad6d75f7..339ecf754 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -266,7 +266,9 @@ upon error."                       ;; this info via gexps.                       ((gnu build file-systems)                        #:select (find-partition-by-luks-uuid))-                      (rnrs bytevectors))+                      (rnrs bytevectors)+                      (ice-9 ftw)+                      (ice-9 rdelim))           (with-output-to-port (%make-void-port "w")            (lambda ()

Danny Milosavljevic wrote on 4 Mar 2018 02:09

[PATCH v9 5/7] vm: Allow qemu-image builder to load Linux kernel modules.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180304010914.1322-6-dannym@scratchpost.org

* gnu/system/vm.scm (%modprobe-wrapper): New variable.(qemu-image): Modify.--- gnu/system/vm.scm | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-)

Toggle diff (74 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 91ff32ce9..cf1ec651a 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."    #:single-file-output? #t    #:references-graphs inputs)) +(define (%modprobe-wrapper modprobe linux-module-directory)+  ;; Wrapper for the 'modprobe' command that knows where modules live.+  ;;+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'+  ;; environment variable is not set---hence the need for this wrapper.+  (program-file "modprobe"+    #~(begin+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))+ (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))   (expression->derivation-in-linux-vm    name-   (with-imported-modules (source-module-closure '((gnu build bootloader)+   (with-imported-modules (source-module-closure '((gnu build activation)+                                                   (gnu build bootloader)                                                    (gnu build vm)                                                    (guix build utils)))      #~(begin-         (use-modules (gnu build bootloader)+         (use-modules (gnu build activation)+                      (gnu build bootloader)                       (gnu build vm)                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports))           (let ((inputs-                '#$(append (list qemu parted e2fsprogs dosfstools)+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)                            (map canonical-package                                 (list sed grep coreutils findutils gawk))                            (if register-closures? (list guix) '())))@@ -302,6 +317,14 @@ the image."                         inputs)))             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)+           ;; It's possible that we need to load nls modules in order to+           ;; mount the new partition.+           (if (file-exists? #$modprobe-name)+               (activate-modprobe #$(%modprobe-wrapper modprobe-name+                                     linux-module-directory))+               (format (current-error-port)+                "WARNING: No modprobe found in ~s.  \+Loading kernel modules will be impossible.\n" #$modprobe-name))             (let* ((graphs     '#$(match inputs                                    (((names . _) ...)@@ -364,7 +387,7 @@ the image."    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-   #:references-graphs inputs))+   #:references-graphs inputs)))   ;;;

Danny Milosavljevic wrote on 4 Mar 2018 02:09

[PATCH v9 6/7] vm: Make the virtio-blk uniquely identifiable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180304010914.1322-7-dannym@scratchpost.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.--- gnu/build/vm.scm  | 2 +- gnu/system/vm.scm | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..ebf9e9f6e 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex cf1ec651a..78cc8cad1 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -700,7 +700,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os

Danny Milosavljevic wrote on 4 Mar 2018 02:09

[PATCH v9 7/7] linux-initrd: Use module-aliases->module-file-names, too.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180304010914.1322-8-dannym@scratchpost.org

* gnu/system/linux-initrd.scm (flat-linux-module-directory): Usemodule-aliases->module-file-names.* gnu/build/linux-modules.scm (file-name->module-name): Export.--- gnu/build/linux-modules.scm |  1 + gnu/system/linux-initrd.scm | 67 ++++++++++++++++++++++++++------------------- 2 files changed, 40 insertions(+), 28 deletions(-)

Toggle diff (105 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex f6bb0512b..81a4b15b1 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -35,6 +35,7 @@             ensure-dot-ko             module-aliases             module-aliases->module-file-names+            file-name->module-name             module-dependencies             recursive-module-dependencies             modules-loadeddiff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 339ecf754..0b976afad 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -161,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."                     #:references-graphs `(("init-closure" ,init)                                           ("modprobe-closure" ,modprobe)))) -(define (flat-linux-module-directory linux modules)-  "Return a flat directory containing the Linux kernel modules listed in-MODULES and taken from LINUX."+(define (flat-linux-module-directory linux aliases)+  "Return a flat directory containing the Linux kernel modules resolved by+ALIASES and taken from LINUX."   (define build-exp     (with-imported-modules (source-module-closure                             '((guix build utils)                               (gnu build linux-modules)))       #~(begin-          (use-modules (ice-9 match) (ice-9 regex)+          (use-modules (ice-9 match) (ice-9 ftw)                        (srfi srfi-1)-                       (guix build utils)+                       (guix build utils) ; mkdir-p                        (gnu build linux-modules))            (define (string->regexp str)@@ -181,31 +181,42 @@ MODULES and taken from LINUX."           (define module-dir             (string-append #$linux "/lib/modules")) -          (define (lookup module)-            (let ((name (ensure-dot-ko module)))-              (match (find-files module-dir (string->regexp name))-                ((file)-                 file)-                (()-                 (error "module not found" name module-dir))-                ((_ ...)-                 (error "several modules by that name"-                        name module-dir)))))+          (define (find-only-entry directory)+            (match (scandir directory)+             (("." ".." basename)+              (string-append directory "/" basename)))) -          (define modules-            (let ((modules (map lookup '#$modules)))-              (append modules-                      (recursive-module-dependencies modules-                                                     #:lookup-module lookup))))--          (mkdir #$output)-          (for-each (lambda (module)-                      (format #t "copying '~a'...~%" module)-                      (copy-file module-                                 (string-append #$output "/"-                                                (basename module))))-                    (delete-duplicates modules)))))+          (define linux-release-module-directory+            (find-only-entry module-dir)) +          (define modules+            (module-aliases->module-file-names #$linux '#$aliases))++          (define version+            (basename linux-release-module-directory))++          (define (install-module-files module-files output)+            "Install MODULE-FILES to OUTPUT.+Precondition: OUTPUT is an empty directory except for \"modules.builtin\"."+            (let ((aliases+                   (map (lambda (module-file-name)+                          (format #t "copying '~a'...~%" module-file-name)+                          (copy-file module-file-name+                           (string-append output "/"+                                          (basename module-file-name)))+                         `(,(file-name->module-name module-file-name) .+                            ,(module-aliases module-file-name)))+                     (sort module-files string<))))+              (install-file (string-append linux-release-module-directory+                                           "/modules.builtin")+                            output)+              (write-module-alias-database aliases output)+              (write-module-device-database aliases output)))++          (let ((output (string-append #$output "/lib/modules/" version)))+            (mkdir-p output)+            (install-module-files (delete-duplicates modules) output)+            #t))))   (computed-file "linux-modules" build-exp))  (define* (raw-initrd file-systems

Danny Milosavljevic wrote on 4 Mar 2018 02:54

Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180304025454.791a2b87@scratchpost.org

Ah, your attached patch is fine, just the proposed usage
(for-each (catch-ENOENT load-linux-module*) ^^^^^^^^^^^^^^^^^^ (needed-modules (known-module-aliases (string-append linux-module-directory "/modules.alias"))))
is probably not working.

Danny Milosavljevic wrote on 4 Mar 2018 13:34

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180304133444.4edceecd@scratchpost.org

Hi Ludo,
I've been thinking about it some more - I think it can be made to work "your" way.
I'm trying to integrate it right now.
But! The module-aliases->module-file-names way I've been pursuing (which alsoreturns the recursive dependencies) has the advantage of using exactly the sameprocedure for both flat-linux-module-directory (which copies files) and forour modprobe (which loads them later).
This means that it's very likely these two will match.
But if load-linux-module* recursively loads modules on its own it's easilypossible that these will diverge - what flat-linux-module-directory is doingcould (accidentially) differ in the future.
Maybe that's overly defensive programming. What do you think?
My flat-linux-module-directory basically just returns a subset of the directorytree it gets as parameter (and flattens it for no reason).
Also, I've been trying to use find-files for /sys before, like you do.It doesn't work correctly - probably because of the mutation that modprobe does(find-files sorts - not sure whether that's the whole story).
Integration tests just finished - even the basic system test fails because of itnow.
I tried only this:
(define (load-kernel-modules) "Examine /sys/devices to find out which modules to load and load them."- (define enter?- (const #t))- (define (down! directory stat result)- ;; Note: modprobe mutates the tree starting with DIRECTORY.- (let ((modalias-name (string-append directory "/modalias")))- (if (file-exists? modalias-name)- (let ((modalias- (string-trim-right (call-with-input-file modalias-name- read-string)- #\newline)))- (system* "/sbin/modprobe" "-q" "--" modalias))))- #t)- (define up- (const #t))- (define skip- (const #t))- (define leaf- (const #t))- (define (error name stat errno result)- (format (current-error-port) "warning: ~a: ~a~%"- name (strerror errno))- result)- (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))+ (for-each (lambda (modalias)+ (system* "/sbin/modprobe" "-q" "--" modalias))+ (system-device-aliases)))
Doesn't work anymore...
ERROR: In procedure network-interface-flags:In procedure network-interface-flags: No such device
(I've exported system-device-aliases)

Danny Milosavljevic wrote on 4 Mar 2018 13:45

Re: [bug#30604] [PATCH v8 2/7] linux-modules: Add install-modules.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180304134504.0a15b26d@scratchpost.org

Hi Ludo,

Toggle quote (3 lines)

> (write-module-alias-database modules port) ;for “modules.alias”> (write-module-device-database modules port) ;for “modules.devname”

I'd prefer "install-" because their interface is somewhat like install-file(caller provides output directory and callee adds "modules."* automatically).
But I used the names "write-module-"* in the patchset v9.

Toggle quote (3 lines)

> I’m not sure we need ‘install-module-files’ itself. Perhaps we can> inline it at the call site?

Sure. Did so now.

Toggle quote (2 lines)

> For the devname code, please avoid ‘set!’.

I found an even better way to avoid both.

Toggle quote (4 lines)

> The indentation of ‘match’ forms is wrong. Would it be OK for you to> pass it through ./etc/indent-code.el? (It’s non interactive, you don’t> need to actually use Emacs.)

I'm actually trying to switch to emacs but when I press Tab it messes upthe stuff more. Is there some kind of configuration for guile indentation?
Also, every time I open gnu/build/linux-modules.scm I get
"The local variables list .. may not be safe
* bug-reference-bug-repexp* eval: (modify-syntax-entry 126 "'")* eval: (modify-syntax-entry 36 "'")* eval: (modify-syntax-entry 43 "'")"
???

Danny Milosavljevic wrote on 4 Mar 2018 13:53

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180304135327.0a103e57@scratchpost.org

Hi Ludo,
On Sat, 03 Mar 2018 23:07:57 +0100ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (3 lines)

> We could reuse modules.alias and modules.devname from the ‘linux-libre’> package (right?), but I guess it doesn’t hurt to generate custom ones.

Yes, but it would be somewhat dirty. Here's why:
Linux-libre contains a superset of modules and also their modules.aliasand modules.devname contain a superset of modules compared to the initrd.
That means
(1) modules.alias would contain aliases which map to modules whichdon't exist. Since we don't for-each it, who cares.
(2) modules.devname would contain devnames for modules which don'texist. That means the boot code would create files in /dev which,when someone accesses them, would try to load modules which don'texist - and then the access would fail.
Technically both are not bad, but (2) is bad "UX" practise nowadays - andnot done in the Linux world any more since udev exists.
I can still remember the days of 8540 entries static /dev filesystem (as in,really on the hard disk), 7000 of which didn't work (or even valid for thissystem type :P).
If something goes wrong, good luck finding the needle in the haystack(which /dev file you can use to boot).

Ludovic Courtès wrote on 9 Mar 2018 23:06

Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87muzgykcl.fsf@gnu.org

Hello!
I’ve reworked this patch series a bit, but now I’m stuck on this:
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (3 lines)

> ERROR: In procedure network-interface-flags:> In procedure network-interface-flags: No such device

When than happens, our modprobe gets called with the “netdev-eth0”alias, which it cannot handle, leading to this error.
By explicitly loading “virtio_net” and sleeping a bit, we get past thatpoint, but fail later on:

Toggle snippet (27 lines)loading '/gnu/store/n2zvdxp25kvg7wy5xr1y5n9r4502fw80-linux-vm-loader'...environment variable `PATH' set to `/gnu/store/vxl918zb9brnrgaipzsykpy3mzcafvcr-qemu-minimal-2.11.1/bin:/gnu/store/lvkaf3xlvy57bvap51xlzb209ilbkgcv-parted-3.2/sbin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/bin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/sbin:/gnu/store/0sq2nflm42x0znkv44add0gk82khkcb6-dosfstools-4.1/sbin:/gnu/store/0hl513mnpkhszm2hjai2w9cxmpxs0vgq-sed-4.4/bin:/gnu/store/1h44pkgdd7n6s3i2vjh54awsvfmc219j-grep-3.1/bin:/gnu/store/kgzvfby2ggi1xawsh5vjh4s93qk2dp9k-coreutils-8.28/bin:/gnu/store/k7r2m2wgj8x8jjhccwjsiimp0dlzxb7i-findutils-4.6.0/bin:/gnu/store/ahxc89r6npzf2bbl8yg5vdjicskzzjf3-gawk-4.1.4/bin'creating partition table with 2 partitions (20.0 MiB, 40.0 MiB)...Warning: The resulting partition is not properly aligned for best performance.creating ext4 partition...mke2fs 1.43.6 (29-Aug-2017)ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/vda1 is mounted.Creating filesystem with 20480 1k blocks and 5136 inodesFilesystem UUID: 81a538e9-3efe-406b-bf67-fa153fa3c6a6Superblock backups stored on blocks: 	8193
Allocating group tables: done                            Writing inode tables: done                            Creating journal (1024 blocks): doneWriting superblocks and filesystem accounting information: done
[    3.190801] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)populating...clearing file timestamps...creating FAT partition...mkfs.fat 4.1 (2017-01-24)[    3.235493] FAT-fs (vda2): IO charset iso8859-1 not foundERROR: In procedure mount:In procedure mount: Invalid argument

Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t eveninvoked, although my understanding is that it should (per ‘load_nls’ innls_base.c in the kernel, called from vfat/inode.c).
Similarly, it seems that virtio_blk has to be loaded explicitly: is notreported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nordo we get a modprobe query for it.
All in all on-demand loading seems more complicated that it seemed.
What are we missing here?
Ludo’.

Danny Milosavljevic wrote on 9 Mar 2018 23:13

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180309231320.20a02822@scratchpost.org

Hi Ludo,
On Fri, 09 Mar 2018 23:06:50 +0100ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (8 lines)

> When than happens, our modprobe gets called with the “netdev-eth0”> alias, which it cannot handle, leading to this error.> > By explicitly loading “virtio_net” and

> sleeping a bit,

Uhhhhh no please don't.

Toggle quote (1 lines)

> point, but fail later on:

...

Toggle quote (14 lines)> --8<---------------cut here---------------end--------------->8---> > Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t even> invoked, although my understanding is that it should (per ‘load_nls’ in> nls_base.c in the kernel, called from vfat/inode.c).> > Similarly, it seems that virtio_blk has to be loaded explicitly: is not> reported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nor> do we get a modprobe query for it.> > All in all on-demand loading seems more complicated that it seemed.> > What are we missing here?

As I said, modprobe mutates /sys - you cannot use find-files. I haven't usedftw just to be contrarian. There's even a warning comment :-)
In order to find that out, try to print how /sys looked before modprobe - thenin the early guile recovery REPL print how /sys looks again - the alias itwas juuust complaining about will be there just fine.
My patch series works fine lazy-loading the IO charset, virtio_blk, virtio_net.
It took about 30 h to work out the correct minimal combination - just saying :-)

Danny Milosavljevic wrote on 9 Mar 2018 23:19

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180309231942.023f6655@scratchpost.org

oops,
On Fri, 9 Mar 2018 23:13:20 +0100Danny Milosavljevic <dannym@scratchpost.org> wrote:

Toggle quote (2 lines)

> As I said, modprobe mutates /sys - you cannot use find-files.

I guess technically it's the Linux kernel module that mutates it.

Toggle quote (9 lines)

>I haven't used> ftw just to be contrarian. There's even a warning comment :-)> > In order to find that out, try to print how /sys looked before modprobe - then> in the early guile recovery REPL print how /sys looks again - the alias it> was juuust complaining about will be there just fine.> > My patch series works fine lazy-loading the IO charset, virtio_blk, virtio_net.

(v9)

Danny Milosavljevic wrote on 9 Mar 2018 23:26

Re: [PATCH v9 7/7] linux-initrd: Use module-aliases->module-file-names, too.

Recipients:(address . 30604@debbugs.gnu.org)

Message-ID:20180309232650.0f37fb89@scratchpost.org

On Sun, 4 Mar 2018 02:09:14 +0100Danny Milosavljevic <dannym@scratchpost.org> wrote:

Toggle quote (12 lines)> --- a/gnu/system/linux-initrd.scm> +++ b/gnu/system/linux-initrd.scm> @@ -161,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd.">                      #:references-graphs `(("init-closure" ,init)>                                            ("modprobe-closure" ,modprobe))))>  > -(define (flat-linux-module-directory linux modules)> -  "Return a flat directory containing the Linux kernel modules listed in> -MODULES and taken from LINUX."> +(define (flat-linux-module-directory linux aliases)> +  "Return a flat directory containing the Linux kernel modules resolved by> +ALIASES and taken from LINUX."

...

Toggle quote (5 lines)

> + (define (find-only-entry directory)> + (match (scandir directory)> + (("." ".." basename)> + (string-append directory "/" basename))))

Probably unsafe since the order is not guaranteed.

Danny Milosavljevic wrote on 9 Mar 2018 23:36

Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180309233644.7dc53f8e@scratchpost.org

Toggle quote (4 lines)

> Similarly, it seems that virtio_blk has to be loaded explicitly: is not> reported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nor> do we get a modprobe query for it.

qemu seems to have a bug that it doesn't assign pci ids properly.
Patch v9 6/7 works around it by fixing the virtio-blk-pci address.
Nicer fixes welcome.

Danny Milosavljevic wrote on 9 Mar 2018 23:44

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180309234434.2bf5e3ff@scratchpost.org

Hi Ludo,
sorry for the many e-mails - I'm recollecting all the workarounds :)

Toggle quote (6 lines)>--8<---------------cut here---------------start------------->8--->loading '/gnu/store/n2zvdxp25kvg7wy5xr1y5n9r4502fw80-linux-vm-loader'...>environment variable `PATH' set to `/gnu/store/vxl918zb9brnrgaipzsykpy3mzcafvcr-qemu-minimal-2.11.1/bin:/gnu/store/lvkaf3xlvy57bvap51xlzb209ilbkgcv-parted-3.2/sbin:/>gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/bin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/sbin:/gnu/>store/0sq2nflm42x0znkv44add0gk82khkcb6-dosfstools-4.1/sbin:/gnu/store/0hl513mnpkhszm2hjai2w9cxmpxs0vgq-sed-4.4/bin:/gnu/store/1h44pkgdd7n6s3i2vjh54awsvfmc219j->grep-3.1/bin:/gnu/store/kgzvfby2ggi1xawsh5vjh4s93qk2dp9k-coreutils-8.28/bin:/gnu/store/k7r2m2wgj8x8jjhccwjsiimp0dlzxb7i-findutils-4.6.0/bin:/gnu/store/>ahxc89r6npzf2bbl8yg5vdjicskzzjf3-gawk-4.1.4/bin'>creating partition table with 2 partitions (20.0 MiB, 40.0 MiB)...>Warning: The resulting partition is not properly aligned for best performance.>creating ext4 partition...

...[    3.235493] FAT-fs (vda2): IO charset iso8859-1 not foundERROR: In procedure mount:In procedure mount: Invalid argument

Toggle snippet (13 lines)

>Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t even>invoked, although my understanding is that it should (per ‘load_nls’ in>nls_base.c in the kernel, called from vfat/inode.c).
Yeah, that was an evil problem. There are two different ways linux-bootcan boot:* Either it loads a full system in which case /run/booted-system willcontain kernel module files which can and will be loaded* Or it loads a "--load" file only - in which case there's no systemmodprobe, but switch-root just switched our pure-Guile modprobe away,so now we are out of modprobes :)

Danny Milosavljevic wrote on 11 Mar 2018 21:24

Re: [PATCH v9 3/7] linux-initrd: Provide pure-Guile modprobe.

Recipients:(address . 30604@debbugs.gnu.org)

Message-ID:20180311212430.707ef0ea@scratchpost.org

Toggle quote (3 lines)

> + (define option-spec> + '((quiet (single-char #\q) (value #f))))

Would make sense to also support "-a" (used when invoking modprobe to load multiplekernel modules at once).

Ludovic Courtès wrote on 12 Mar 2018 13:38

Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware is present.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87ina1qxic.fsf@gnu.org

Heya Danny,
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (3 lines)

> As I said, modprobe mutates /sys - you cannot use find-files. I haven't used> ftw just to be contrarian. There's even a warning comment :-)

I did see this comment, and I don’t doubt you’re acting in good faith!

Toggle quote (4 lines)

> In order to find that out, try to print how /sys looked before modprobe - then> in the early guile recovery REPL print how /sys looks again - the alias it> was juuust complaining about will be there just fine.

It wasn’t clear to me what “modprobe mutates /sys” meant, and I guess Ididn’t think it could lead to not seeing relevant “modalias” files. NowI’ve experienced first-hand that it does matter. :-)

Toggle quote (2 lines)

> It took about 30 h to work out the correct minimal combination - just saying :-)

Yeah I can imagine (and it would have taken me much longer I guess.)
While reviewing this series I realized I didn’t understand everything,as in the example above, some things seemed unnecessary, and I thoughtwe could improve the style of a few things.
The patches that follow build upon your work with a few changes:
• The “modules.alias” writer style is made more idiomatic and hopefully simpler.
• The “modules.devname” writer is actually unnecessary. I changed it to a more functional style and with smaller functions. It’s the last patch of this series; it’s completely optional since we don’t use it currently, but we might want to keep it around for later.
• The pure-Scheme modprobe uses SRFI-37 instead of (ice-9 getopt-long), as in the rest of the code base.
• /proc/sys/kernel/modprobe is set in ‘boot-system’ directly, so that we don’t have to special-case the “modprobe” closure in ‘build-initrd’.
• ‘module-aliases->module-file-names’ is gone. Instead there’s ‘load-needed-linux-modules’, similar to the ‘load-kernel-modules’ you had, but a bit simpler.
• I added comments here and there, especially about the virtio-blk-pci, which was far from obvious. :-)
I didn’t add the ‘needed-modules’ and ‘system-device-aliases’ proceduresI posted earlier because we don’t need them currently. If you thinkthey could be useful, we can always add them.
The following command succeeds:
make check-system TESTS="basic installed-os iso-image-installer"
WDYT?

I realize that makes for a not-so-efficient review process, though Ithink the result is worth it. I’m open to suggestions though.
Overall, these changes are mostly about (1) making self-containedcommits, (2) commenting code, and (3) coding style. About #3, I thinkthe guidelines I applied were:
1. Avoid imperative style. When we do need side effects, try hard to move them separately.
2. Keep functions short.
3. Decompose functions in a way that allows them to be combined “nicely”.
Some of that is a bit subjective, but overall we should be able toconverge.
Thanks for the great work, and sorry for the back-and-forth and delays!
Ludo’.

Ludovic Courtès wrote on 12 Mar 2018 13:39

[PATCH v10 1/6] linux-modules: Add "modules.alias" writer.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180312123918.22645-1-ludo@gnu.org

* gnu/build/linux-modules.scm (write-module-alias-database): Newprocedure.(%not-dash): New variable.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>--- gnu/build/linux-modules.scm | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-)

Toggle diff (58 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 4fe673cca..6101b8dc3 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014, 2016, 2018 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>+;;; Copyright © 2018 Danny Milosavljevic <dannym@scratchpost.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -21,6 +22,7 @@   #:use-module (guix elf)   #:use-module (guix glob)   #:use-module (guix build syscalls)+  #:use-module ((guix build utils) #:select (find-files))   #:use-module (rnrs io ports)   #:use-module (rnrs bytevectors)   #:use-module (srfi srfi-1)@@ -41,7 +43,8 @@              device-module-aliases             known-module-aliases-            matching-modules))+            matching-modules+            write-module-alias-database))  ;;; Commentary: ;;;@@ -380,4 +383,25 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                       module)))               known-aliases)) +(define %not-dash+  (char-set-complement (char-set #\-)))++(define (write-module-alias-database directory)+  "Traverse the '.ko' files in DIRECTORY and create the corresponding+'modules.alias' file."+  (define aliases+    (map (lambda (file)+           (cons (file-name->module-name file) (module-aliases file)))+         (find-files directory "\\.ko$")))++  (call-with-output-file (string-append directory "/modules.alias")+    (lambda (port)+      (display "# Aliases extracted from modules themselves.\n" port)+      (for-each (match-lambda+                  ((module . aliases)+                   (for-each (lambda (alias)+                               (format port "alias ~a ~a\n" alias module))+                             aliases)))+                aliases))))+ ;;; linux-modules.scm ends here-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 13:39

[PATCH v10 2/6] linux-modules: Add 'load-linux-modules-from-directory'.

Recipients:(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20180312123918.22645-2-ludo@gnu.org

* gnu/build/linux-modules.scm (load-linux-modules-from-directory): Newprocedure.* gnu/build/linux-boot.scm (boot-system)[lookup-module]: Remove.Use 'load-linux-modules-from-directory' instead.--- gnu/build/linux-boot.scm    |  9 ++------- gnu/build/linux-modules.scm | 12 ++++++++++++ 2 files changed, 14 insertions(+), 7 deletions(-)

Toggle diff (59 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..df0b2b2d1 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -469,10 +469,6 @@ upon error."              mounts)         "ext4")) -  (define (lookup-module name)-    (string-append linux-module-directory "/"-                   (ensure-dot-ko name)))-   (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n") @@ -487,9 +483,8 @@ upon error."          (start-repl))         (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (load-linux-modules-from-directory linux-modules+                                          linux-module-directory)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 6101b8dc3..1337b5e88 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -38,6 +38,7 @@             modules-loaded             module-loaded?             load-linux-module*+            load-linux-modules-from-directory              current-module-debugging-port @@ -232,6 +233,17 @@ appears in BLACK-LIST are not loaded."              (or (and recursive? (= EEXIST (system-error-errno args)))                  (apply throw args))))))) +(define (load-linux-modules-from-directory modules directory)+  "Load MODULES and their dependencies from DIRECTORY, a directory containing+the '.ko' files.  The '.ko' suffix is automatically added to MODULES if+needed."+  (define (lookup-module name)+    (string-append directory "/" (ensure-dot-ko name)))++  (for-each (cut load-linux-module* <>+                 #:lookup-module lookup-module)+            (map lookup-module modules)))+  ;;; ;;; Device modules.-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 13:39

[PATCH v10 3/6] linux-modules: Add 'load-needed-linux-modules'.

Recipients:(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20180312123918.22645-3-ludo@gnu.org

Based on a patch by Danny Milosavljevic <dannym@scratchpost.org>.
* gnu/build/linux-modules.scm (load-needed-linux-modules): Newprocedure.--- gnu/build/linux-modules.scm | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)

Toggle diff (59 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 1337b5e88..9aabe6d37 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -30,6 +30,7 @@   #:use-module (ice-9 vlist)   #:use-module (ice-9 match)   #:use-module (ice-9 rdelim)+  #:use-module (ice-9 ftw)   #:export (dot-ko             ensure-dot-ko             module-aliases@@ -45,7 +46,8 @@             device-module-aliases             known-module-aliases             matching-modules-            write-module-alias-database))+            write-module-alias-database+            load-needed-linux-modules))  ;;; Commentary: ;;;@@ -416,4 +418,34 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                              aliases)))                 aliases)))) +(define (load-needed-linux-modules module-directory)+  "Examine /sys/devices to find out which modules are needed and load those we+have in MODULE-DIRECTORY.  Return the list of modules loaded, not including+dependencies.++Note: loading modules leads to the creation of new entries in /sys/devices,+which is why we need traversal and loading to be interleaved.  If we walked+/sys/devices and *then* loaded modules, we'd miss the entries added as a+side-effect and would thus need to traverse /sys/devices again."+  (define aliases+    (known-module-aliases+     (string-append module-directory "/modules.alias")))++  (define (enter? director stat result) result)+  (define (down directory stat result) result)+  (define (up directory stat result) result)+  (define (skip entry stat result) result)+  (define (error name stat errno result) result)+  (define (leaf file stat result)+    (if (string=? (basename file) "modalias")+        (let* ((alias   (string-trim-right+                         (call-with-input-file file get-string-all)))+               (modules (matching-modules alias aliases)))+          (load-linux-modules-from-directory modules+                                             module-directory)+          (append modules result))+        result))++  (file-system-fold enter? leaf down up skip error #t "/sys/devices"))+ ;;; linux-modules.scm ends here-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 13:39

[PATCH v10 4/6] vm: Make the virtio-blk uniquely identifiable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)

Message-ID:20180312123918.22645-4-ludo@gnu.org

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>--- gnu/build/vm.scm | 6 +++++- gnu/system/vm.scm | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-)

Toggle diff (37 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..773b738ae 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,11 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"++                       ;; QEMU seems to have a bug that it doesn't assign PCI+                       ;; IDs properly, so force the address of our virtio-blk+                       ;; device.+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 91ff32ce9..4360adf15 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -677,7 +677,12 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"++     ;; QEMU seems to have a bug that it doesn't assign PCI IDs properly, so+     ;; force the address of our virtio-blk device.+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"++     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 13:39

[PATCH v10 6/6] linux-modules: Add "modules.devname" writer.

Recipients:(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20180312123918.22645-6-ludo@gnu.org

* gnu/build/linux-modules.scm (aliases->device-tuple)(write-module-device-database): New procedures.
Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>.--- gnu/build/linux-modules.scm | 48 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)

Toggle diff (68 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 9aabe6d37..2ed04e5b4 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -47,6 +47,8 @@             known-module-aliases             matching-modules             write-module-alias-database+            write-module-device-database+             load-needed-linux-modules))  ;;; Commentary:@@ -418,6 +420,52 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                              aliases)))                 aliases)))) +(define (aliases->device-tuple aliases)+  "Traverse ALIASES, a list of module aliases, and search for+\"char-major-M-N\", \"block-major-M-N\", or \"devname:\" aliases.  When they+are found, return a tuple (DEVNAME TYPE MAJOR MINOR), otherwise return #f."+  (define (char/block-major? alias)+    (or (string-prefix? "char-major-" alias)+        (string-prefix? "block-major-" alias)))++  (define (char/block-major->tuple alias)+    (match (string-tokenize alias %not-dash)+      ((type "major" (= string->number major) (= string->number minor))+       (list (match type+               ("char" "c")+               ("block" "b"))+             major minor))))++  (let* ((devname     (any (lambda (alias)+                             (and (string-prefix? "devname:" alias)+                                  (string-drop alias 8)))+                           aliases))+         (major/minor (match (find char/block-major? aliases)+                        (#f #f)+                        (str (char/block-major->tuple str)))))+    (and devname major/minor+         (cons devname major/minor))))++(define (write-module-device-database directory)+  "Traverse the '.ko' files in DIRECTORY and create the corresponding+'modules.devname' file.  This file contains information about modules that can+be loaded on-demand, such as file system modules."+  (define aliases+    (filter-map (lambda (file)+                  (match (aliases->device-tuple (module-aliases file))+                    (#f #f)+                    (tuple (cons (file-name->module-name file) tuple))))+                (find-files directory "\\.ko$")))++  (call-with-output-file (string-append "/tmp" "/modules.devname")+    (lambda (port)+      (display "# Device nodes to trigger on-demand module loading.\n" port)+      (for-each (match-lambda+                  ((module devname type major minor)+                   (format port "~a ~a ~a~a:~a~%"+                           module devname type major minor)))+                aliases))))+ (define (load-needed-linux-modules module-directory)   "Examine /sys/devices to find out which modules are needed and load those we have in MODULE-DIRECTORY.  Return the list of modules loaded, not including-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 13:39

[PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20180312123918.22645-5-ludo@gnu.org

This allows us to load modules on demand when the kernel asks for them.
* gnu/system/linux-initrd.scm (modprobe-program): New variable.(flat-linux-module-directory): Call 'write-module-alias-database'.(raw-initrd): Pass #:modprobe to 'boot-system'.(expression->initrd): Copy "closure" to $out/references.* gnu/build/linux-boot.scm (boot-system): Add #:modprobe and honor it.Call 'load-needed-linux-modules'.* gnu/system/vm.scm (qemu-image): Add #:linux parameter.  Define'modprobe-wrapper' and pass it to 'activate-modprobe'.  Pass #:linux to'expression->derivation-in-linux-vm'.
Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>--- gnu/build/linux-boot.scm    | 13 +++++-- gnu/system/linux-initrd.scm | 86 +++++++++++++++++++++++++++++++++++++++++---- gnu/system/vm.scm           | 21 +++++++++++ 3 files changed, 111 insertions(+), 9 deletions(-)

Toggle diff (222 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex df0b2b2d1..eedc4bb9d 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -435,6 +435,7 @@ bailing out.~%root contents: ~s~%" (scandir "/"))   (define* (boot-system #:key+                      modprobe                       (linux-modules '())                       linux-module-directory                       qemu-guest-networking?@@ -449,6 +450,9 @@ QEMU-GUEST-NETWORKING? is true, calling PRE-MOUNT, mounting the file systems specified in MOUNTS, and finally booting into the new root if any.  The initrd supports kernel command-line options '--load', '--root', and '--repl'. +MODPROBE must be #f or a program to install as the modprobe program that the+kernel will invoke when it needs to load modules.+ Mount the root file system, specified by the '--root' command-line argument, if any. @@ -482,9 +486,14 @@ upon error."        (when (member "--repl" args)          (start-repl)) +       (when modprobe+         ;; Tell the kernel to invoke MODPROBE.+         (call-with-output-file "/proc/sys/kernel/modprobe"+           (lambda (port)+             (display modprobe port))))+        (display "loading kernel modules...\n")-       (load-linux-modules-from-directory linux-modules-                                          linux-module-directory)+       (load-needed-linux-modules linux-module-directory)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 1eb5f5130..6b7883c2c 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -63,6 +63,75 @@ ;;; ;;; Code: +(define* (modprobe-program linux-module-directory #:key+                           (guile %guile-static-stripped))+  "Return a minimal implementation of 'modprobe' for our initrd that looks up+modules in LINUX-MODULE-DIRECTORY.  This program will be invoked by the kernel+when modules need to be loaded."+  (define program+    (with-imported-modules (source-module-closure+                            '((gnu build linux-modules)))+      #~(begin+          (use-modules (gnu build linux-modules)+                       (ice-9 match)+                       (srfi srfi-1)+                       (srfi srfi-26)+                       (srfi srfi-37))++          (define option-spec+            (list (option '(#\q "quiet") #f #f+                          (lambda (opt name arg result)+                            (alist-cons 'quiet? #t result)))))++          (define options+            ;; Alist of options and non-option arguments.+            (args-fold (cdr (program-arguments))+                       option-spec+                       (lambda (opt name arg result)+                         (error "unrecognized option" name))+                       (lambda (arg result)+                         (alist-cons 'argument arg result))+                       '()))++          (define aliases+            ;; The list of aliases we are asked to load.+            (filter-map (match-lambda+                          (('argument . alias) alias)+                          (_ #f))+                        options))++          (define linux-module-directory+            ;; The module directory.  Note: We expect a flat directory here.+            #$linux-module-directory)++          (define %known-aliases+            ;; The alias database.+            (known-module-aliases+             (string-append linux-module-directory "/modules.alias")))++          (define (device-aliases->module-names aliases)+            ;; Return the list of module names for the subset of ALIASES that+            ;; appears in %KNOWN-ALIASES.+            (append-map (cut matching-modules <> %known-aliases)+                        aliases))++          (when (assq-ref options 'quiet?)+            (current-error-port (%make-void-port "w"))+            (current-output-port (%make-void-port "w")))++          (let ((modules (device-aliases->module-names aliases)))+            (call-with-output-file "/dev/kmsg"+              (lambda (port)+                (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%"+                        (getpid) aliases modules (program-arguments))))++            (when (< (length modules) (length aliases))+              (error "alias resolution failed" aliases))++            (load-linux-modules-from-directory (reverse modules)+                                               linux-module-directory)))))++  (program-file "modprobe" program #:guile guile))  (define* (expression->initrd exp                              #:key@@ -89,16 +158,16 @@ the derivations referenced by EXP are automatically copied to the initrd."           (mkdir #$output)            ;; The guile used in the initrd must be present in the store, so-          ;; that module loading works once the root is switched.+          ;; that module loading works once the root is switched.  Similarly,+          ;; the 'modprobe' program installed in /proc/sys/kernel/modprobe by+          ;; the initrd, if any, must be present after switch root.           ;;           ;; To ensure that is the case, add an explicit reference to the           ;; guile package used in the initrd to the output.           ;;-          ;; This fixes guix-patches bug #28399, "Fix mysql activation, and+          ;; This fixes <https://bugs.gnu.org/28399>, "Fix mysql activation, and           ;; add a basic test".-          (call-with-output-file (string-append #$ output "/references")-            (lambda (port)-              (simple-format port "~A\n" #$guile)))+          (copy-file "closure" (string-append #$output "/references"))            (build-initrd (string-append #$output "/initrd")                         #:guile #$guile@@ -153,7 +222,9 @@ MODULES and taken from LINUX."                       (copy-file module                                  (string-append #$output "/"                                                 (basename module))))-                    (delete-duplicates modules)))))+                    (delete-duplicates modules))++          (write-module-alias-database #$output))))    (computed-file "linux-modules" build-exp)) @@ -222,7 +293,8 @@ upon error."              (set-path-environment-variable "PATH" '("bin" "sbin")                                             '#$helper-packages))) -         (boot-system #:mounts+         (boot-system #:modprobe #$(modprobe-program kodir)+                      #:mounts                       (map spec->file-system                            '#$(map file-system->spec file-systems))                       #:pre-mount (lambda ()diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 4360adf15..ecb7ed506 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -249,6 +249,7 @@ INPUTS is a list of inputs (as for packages)." (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))+                     (linux linux-libre)                      (qemu qemu-minimal)                      (disk-image-size 'guess)                      (disk-image-format "qcow2")@@ -275,18 +276,37 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (define modprobe-wrapper+    ;; Wrapper for the 'modprobe' command that knows where modules live.+    (let ((modprobe (file-append kmod "/bin/modprobe")))+      (program-file "modprobe"+                    #~(begin+                        (setenv "LINUX_MODULE_DIRECTORY"+                                #$(file-append linux "/lib/modules"))+                        (apply execl #$modprobe+                               (cons #$modprobe (cdr (command-line))))))))++   (expression->derivation-in-linux-vm    name    (with-imported-modules (source-module-closure '((gnu build bootloader)                                                    (gnu build vm)+                                                   (gnu build activation)                                                    (guix build utils)))      #~(begin          (use-modules (gnu build bootloader)                       (gnu build vm)+                      ((gnu build activation) #:select (activate-modprobe))                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports)) +         ;; We may need to lazy-load modules.  The initrd installs a+         ;; 'modprobe' that can only search through the modules available in+         ;; the initrd, but here we want to be able to use all the modules of+         ;; LINUX.  Thus, install a real 'modprobe'.+         (activate-modprobe #$modprobe-wrapper)+          (let ((inputs                 '#$(append (list qemu parted e2fsprogs dosfstools)                            (map canonical-package@@ -361,6 +381,7 @@ the image."                                    #$(bootloader-installer bootloader))              (reboot)))))    #:system system+   #:linux linux    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 15:45

Re: [bug#30604] [PATCH v9 3/7] linux-initrd: Provide pure-Guile modprobe.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:878taxnyie.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (6 lines)

>> + (define option-spec>> + '((quiet (single-char #\q) (value #f))))>> Would make sense to also support "-a" (used when invoking modprobe to load multiple> kernel modules at once).

I suppose we can do that later because the kernel doesn’t use it in‘call_modprobe’ and this thing is only ever invoked by the kernel.
WDYT?
Ludo’.

Danny Milosavljevic wrote on 12 Mar 2018 18:51

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312183840.44e1b6c2@scratchpost.org

Hi Ludo,
On Mon, 12 Mar 2018 15:45:29 +0100ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (6 lines)

> > Would make sense to also support "-a" (used when invoking modprobe to load multiple> > kernel modules at once). > > I suppose we can do that later because the kernel doesn’t use it in> ‘call_modprobe’ and this thing is only ever invoked by the kernel.

Sure.
I just wanted to comment because I noticed that kmod's
modprobe a b c
doesn't load three modules a, b, c. But my v9's pure-Guile modprobe wasimplemented in a way that it did load three modules a, b, c - which itdefinitely shouldn't be doing.

Danny Milosavljevic wrote on 12 Mar 2018 20:26

Re: [bug#30604] [PATCH v10 2/6] linux-modules: Add 'load-linux-modules-from-directory'.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312202654.43eaa5b1@scratchpost.org

LGTM!

Danny Milosavljevic wrote on 12 Mar 2018 20:40

Re: [bug#30604] [PATCH v10 3/6] linux-modules: Add 'load-needed-linux-modules'.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312204002.09bfc72f@scratchpost.org

Toggle quote (3 lines)

> + (let* ((alias (string-trim-right> + (call-with-input-file file get-string-all)))

Hmm... doesn't this trim other things than newline?
A quick internet search shows that most people do that. Huh.
LGTM!

Danny Milosavljevic wrote on 12 Mar 2018 20:54

Re: [PATCH v10 1/6] linux-modules: Add "modules.alias" writer.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312205444.07c05f4c@scratchpost.org

Toggle quote (3 lines)

> +(define %not-dash> + (char-set-complement (char-set #\-)))

Is that used?

Danny Milosavljevic wrote on 12 Mar 2018 21:09

Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312210936.7f89a29c@scratchpost.org

Hi Ludo,
On Mon, 12 Mar 2018 13:39:17 +0100Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (8 lines)

> + (when modprobe> + ;; Tell the kernel to invoke MODPROBE.> + (call-with-output-file "/proc/sys/kernel/modprobe"> + (lambda (port)> + (display modprobe port))))> +> (display "loading kernel modules...\n")

There's a race because Linux could just start loading modulesimmediately before boot-system even starts / executed far enough.
That was why earlier I kept /sbin/modprobe as the name (it's thedefault Linux uses) and made sure that the file /sbin/modprobe is alreadyin the initrd (as opposed to created by boot-system).
The race didn't happen to me yet (and I doubt that Linux has modulesfine-grained enough that that's a problem in practise) - but I would preferit not to be possible.
WDYT?

Toggle quote (7 lines)

> + (define aliases> + ;; The list of aliases we are asked to load.> + (filter-map (match-lambda> + (('argument . alias) alias)> + (_ #f))> + options))

Without "-a", it's not supposed to be a list of aliases. It's supposed to beone alias and the remainder are parameters supposed to be passed to thatkernel module.
(I didn't know that before).
When Linux invokes /sbin/modprobe, it doesn't pass "-a" but it doesn'tpass any module parameters either - it just passes one module alias.
For clarity and in order that we don't have weird problems later (trying toload a module named "verbose=1" or something :) ), let's just accept one aliasfor now.

Toggle quote (2 lines)

>+(define* (modprobe-program linux-module-directory #:key>+ (guile %guile-static-stripped))

[...]

Toggle quote (5 lines)

> + (call-with-output-file "/dev/kmsg"> + (lambda (port)> + (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%"> + (getpid) aliases modules (program-arguments))))

Hmm, logging is nice!
But with /dev/kmsg one write(2) syscall needs to pass the entire message.
What do you think about additionally calling setvbuf with 1024 ? Because:
kernel/printk/printk.c: #define LOG_LINE_MAX (1024 - PREFIX_MAX)

Toggle quote (3 lines)

> + (when (< (length modules) (length aliases))> + (error "alias resolution failed" aliases))

The original modprobe loads as many modules as it can and possibly sets an(always the same) error flag (and continues loading the other modules).
Then modprobe's exit status is that error flag.
This is moot because we shouldn't accept more than one alias - see above.

Toggle quote (2 lines)

> + (write-module-alias-database #$output))))

One of the devnames created statically is the one for btrfs, so not writing orusing devnames is not going to end well.
If it did work without it I'd be even more on guard :)
(I'd also copy the modules.builtin (from Linux). Also, what happens if we load a module which has as dependency a builtin? Will we try to load the builtin as a .ko file and fail the entire thing?)

Toggle quote (2 lines)

> + ;; We may need to lazy-load modules. The initrd installs a

Nitpick: "kernel modules" (to clarify).

Ludovic Courtès wrote on 12 Mar 2018 22:10

Re: [bug#30604] [PATCH v10 3/6] linux-modules: Add 'load-needed-linux-modules'.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87r2opknjk.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (5 lines)

>> + (let* ((alias (string-trim-right>> + (call-with-input-file file get-string-all)))>> Hmm... doesn't this trim other things than newline?

Yes, it trims whitespace.

Toggle quote (2 lines)

> A quick internet search shows that most people do that. Huh.

Yeah I’d think that whitespace is non-significant in these files, but inpractice they’re always generated anyway.
Ludo’.

Danny Milosavljevic wrote on 12 Mar 2018 22:12

Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312221226.72eb6cb7@scratchpost.org

Toggle quote (4 lines)

> That was why earlier I kept /sbin/modprobe as the name (it's the> default Linux uses) and made sure that the file /sbin/modprobe is already> in the initrd (as opposed to created by boot-system).

Note: The simplest change would be to keep this patch as-is but add
;; Make sure that Linux can modprobe even before boot-system started up. (when modprobe (mkdir-p "sbin") (symlink modprobe "sbin/modprobe") (compile-to-cache "sbin/modprobe")) ; Note: without this line booting is dead slow. [Maybe in this case that wouldn't be so bad - since boot-system switches away immediately anyway :)]
to gnu/build/linux-initrd.scm

Ludovic Courtès wrote on 12 Mar 2018 22:13

Re: [PATCH v10 1/6] linux-modules: Add "modules.alias" writer.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87muzdknex.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (5 lines)

>> +(define %not-dash>> + (char-set-complement (char-set #\-)))>> Is that used?

Oops, not until patch 6/6. I’ve moved it there.
Ludo’.

Ludovic Courtès wrote on 12 Mar 2018 23:14

Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87h8plkkkc.fsf@gnu.org

Howdy!
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (24 lines)> On Mon, 12 Mar 2018 13:39:17 +0100> Ludovic Courtès <ludo@gnu.org> wrote:>>> +       (when modprobe>> +         ;; Tell the kernel to invoke MODPROBE.>> +         (call-with-output-file "/proc/sys/kernel/modprobe">> +           (lambda (port)>> +             (display modprobe port))))>> +>>         (display "loading kernel modules...\n")>> There's a race because Linux could just start loading modules> immediately before boot-system even starts / executed far enough.>> That was why earlier I kept /sbin/modprobe as the name (it's the> default Linux uses) and made sure that the file /sbin/modprobe is already> in the initrd (as opposed to created by boot-system).>> The race didn't happen to me yet (and I doubt that Linux has modules> fine-grained enough that that's a problem in practise) - but I would prefer> it not to be possible.>> WDYT?

You’re right, I wondered what would happen. I’ve restored the/sbin/modprobe symlink in v11.

Toggle quote (20 lines)>> +          (define aliases>> +            ;; The list of aliases we are asked to load.>> +            (filter-map (match-lambda>> +                          (('argument . alias) alias)>> +                          (_ #f))>> +                        options))>> Without "-a", it's not supposed to be a list of aliases.  It's supposed to be> one alias and the remainder are parameters supposed to be passed to that> kernel module.>> (I didn't know that before).>> When Linux invokes /sbin/modprobe, it doesn't pass "-a" but it doesn't> pass any module parameters either - it just passes one module alias.>> For clarity and in order that we don't have weird problems later (trying to> load a module named "verbose=1" or something :) ), let's just accept one alias> for now.

Indeed, done in v11.

Toggle quote (10 lines)>>+(define* (modprobe-program linux-module-directory #:key>>+                           (guile %guile-static-stripped))> [...]>> +            (call-with-output-file "/dev/kmsg">> +              (lambda (port)>> +                (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%">> +                        (getpid) aliases modules (program-arguments))))>> Hmm, logging is nice!

I didn’t mean to leave it, but after all, it’s pretty useful. :-)

Toggle quote (4 lines)

> But with /dev/kmsg one write(2) syscall needs to pass the entire message.>> What do you think about additionally calling setvbuf with 1024 ? Because:

Sure.

Toggle quote (5 lines)

>> + (write-module-alias-database #$output))))>> One of the devnames created statically is the one for btrfs, so not writing or> using devnames is not going to end well.

We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,“fs-9p”, etc. alias, which shows up in “modules.alias”. No need for“modules.devname” AFAICS.

Toggle quote (4 lines)

> (I'd also copy the modules.builtin (from Linux).> Also, what happens if we load a module which has as dependency a builtin?> Will we try to load the builtin as a .ko file and fail the entire thing?)

The dependency of a builtin is necessarily a builtin, and the kernelwon’t invoke modprobe for a builtin, will it? IOW, I think there’s noproblem here. :-)

Toggle quote (4 lines)

>> + ;; We may need to lazy-load modules. The initrd installs a>> Nitpick: "kernel modules" (to clarify).

Fixed.
Thanks for the insightful review!
Ludo’.

Ludovic Courtès wrote on 12 Mar 2018 23:15

[PATCH v11 1/6] linux-modules: Add "modules.alias" writer.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180312221541.1886-1-ludo@gnu.org

* gnu/build/linux-modules.scm (write-module-alias-database): Newprocedure.(%not-dash): New variable.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>--- gnu/build/linux-modules.scm | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)

Toggle diff (55 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 4fe673cca..b444e6f90 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014, 2016, 2018 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>+;;; Copyright © 2018 Danny Milosavljevic <dannym@scratchpost.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -21,6 +22,7 @@   #:use-module (guix elf)   #:use-module (guix glob)   #:use-module (guix build syscalls)+  #:use-module ((guix build utils) #:select (find-files))   #:use-module (rnrs io ports)   #:use-module (rnrs bytevectors)   #:use-module (srfi srfi-1)@@ -41,7 +43,8 @@              device-module-aliases             known-module-aliases-            matching-modules))+            matching-modules+            write-module-alias-database))  ;;; Commentary: ;;;@@ -380,4 +383,22 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                       module)))               known-aliases)) +(define (write-module-alias-database directory)+  "Traverse the '.ko' files in DIRECTORY and create the corresponding+'modules.alias' file."+  (define aliases+    (map (lambda (file)+           (cons (file-name->module-name file) (module-aliases file)))+         (find-files directory "\\.ko$")))++  (call-with-output-file (string-append directory "/modules.alias")+    (lambda (port)+      (display "# Aliases extracted from modules themselves.\n" port)+      (for-each (match-lambda+                  ((module . aliases)+                   (for-each (lambda (alias)+                               (format port "alias ~a ~a\n" alias module))+                             aliases)))+                aliases))))+ ;;; linux-modules.scm ends here-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 23:15

[PATCH v11 2/6] linux-modules: Add 'load-linux-modules-from-directory'.

Recipients:(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20180312221541.1886-2-ludo@gnu.org

* gnu/build/linux-modules.scm (load-linux-modules-from-directory): Newprocedure.* gnu/build/linux-boot.scm (boot-system)[lookup-module]: Remove.Use 'load-linux-modules-from-directory' instead.--- gnu/build/linux-boot.scm    |  9 ++------- gnu/build/linux-modules.scm | 12 ++++++++++++ 2 files changed, 14 insertions(+), 7 deletions(-)

Toggle diff (59 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex 18d87260a..df0b2b2d1 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -469,10 +469,6 @@ upon error."              mounts)         "ext4")) -  (define (lookup-module name)-    (string-append linux-module-directory "/"-                   (ensure-dot-ko name)))-   (display "Welcome, this is GNU's early boot Guile.\n")   (display "Use '--repl' for an initrd REPL.\n\n") @@ -487,9 +483,8 @@ upon error."          (start-repl))         (display "loading kernel modules...\n")-       (for-each (cut load-linux-module* <>-                      #:lookup-module lookup-module)-                 (map lookup-module linux-modules))+       (load-linux-modules-from-directory linux-modules+                                          linux-module-directory)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex b444e6f90..0084f972b 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -38,6 +38,7 @@             modules-loaded             module-loaded?             load-linux-module*+            load-linux-modules-from-directory              current-module-debugging-port @@ -232,6 +233,17 @@ appears in BLACK-LIST are not loaded."              (or (and recursive? (= EEXIST (system-error-errno args)))                  (apply throw args))))))) +(define (load-linux-modules-from-directory modules directory)+  "Load MODULES and their dependencies from DIRECTORY, a directory containing+the '.ko' files.  The '.ko' suffix is automatically added to MODULES if+needed."+  (define (lookup-module name)+    (string-append directory "/" (ensure-dot-ko name)))++  (for-each (cut load-linux-module* <>+                 #:lookup-module lookup-module)+            (map lookup-module modules)))+  ;;; ;;; Device modules.-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 23:15

[PATCH v11 3/6] linux-modules: Add 'load-needed-linux-modules'.

Recipients:(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20180312221541.1886-3-ludo@gnu.org

Toggle diff (59 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 0084f972b..55161e026 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -30,6 +30,7 @@   #:use-module (ice-9 vlist)   #:use-module (ice-9 match)   #:use-module (ice-9 rdelim)+  #:use-module (ice-9 ftw)   #:export (dot-ko             ensure-dot-ko             module-aliases@@ -45,7 +46,8 @@             device-module-aliases             known-module-aliases             matching-modules-            write-module-alias-database))+            write-module-alias-database+            load-needed-linux-modules))  ;;; Commentary: ;;;@@ -413,4 +415,34 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                              aliases)))                 aliases)))) +(define (load-needed-linux-modules module-directory)+  "Examine /sys/devices to find out which modules are needed and load those we+have in MODULE-DIRECTORY.  Return the list of modules loaded, not including+dependencies.++Note: loading modules leads to the creation of new entries in /sys/devices,+which is why we need traversal and loading to be interleaved.  If we walked+/sys/devices and *then* loaded modules, we'd miss the entries added as a+side-effect and would thus need to traverse /sys/devices again."+  (define aliases+    (known-module-aliases+     (string-append module-directory "/modules.alias")))++  (define (enter? director stat result) result)+  (define (down directory stat result) result)+  (define (up directory stat result) result)+  (define (skip entry stat result) result)+  (define (error name stat errno result) result)+  (define (leaf file stat result)+    (if (string=? (basename file) "modalias")+        (let* ((alias   (string-trim-right+                         (call-with-input-file file get-string-all)))+               (modules (matching-modules alias aliases)))+          (load-linux-modules-from-directory modules+                                             module-directory)+          (append modules result))+        result))++  (file-system-fold enter? leaf down up skip error #t "/sys/devices"))+ ;;; linux-modules.scm ends here-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 23:15

[PATCH v11 4/6] vm: Make the virtio-blk uniquely identifiable in /sys.

Recipients:(address . 30604@debbugs.gnu.org)

Message-ID:20180312221541.1886-4-ludo@gnu.org

Toggle diff (37 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scmindex fe003ea45..773b738ae 100644--- a/gnu/build/vm.scm+++ b/gnu/build/vm.scm@@ -154,7 +154,11 @@ the #:references-graphs parameter of 'derivation'."                                            builder)                   (append                    (if make-disk-image?-                       `("-device" "virtio-blk,drive=myhd"++                       ;; QEMU seems to have a bug that it doesn't assign PCI+                       ;; IDs properly, so force the address of our virtio-blk+                       ;; device.+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"                          "-drive" ,(string-append "if=none,file=" output                                                   ",format=" disk-image-format                                                   ",id=myhd"))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 91ff32ce9..4360adf15 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -677,7 +677,12 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."       #$@(map virtfs-option shared-fs)      "-vga std"-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"++     ;; QEMU seems to have a bug that it doesn't assign PCI IDs properly, so+     ;; force the address of our virtio-blk device.+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"++     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"              #$image)))  (define* (system-qemu-image/shared-store-script os-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 23:15

[PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20180312221541.1886-5-ludo@gnu.org

From: Danny Milosavljevic <dannym@scratchpost.org>
This allows us to load modules on demand when the kernel asks for them.
* gnu/system/linux-initrd.scm (modprobe-program): New variable.(flat-linux-module-directory): Call 'write-module-alias-database'.(raw-initrd): Pass #:modprobe to 'boot-system' and to 'expression->initrd'.(expression->initrd): Copy "closure" to $out/references.  Add #:modprobeand pass it to 'build-initrd'.* gnu/build/linux-initrd.scm (build-initrd): Add #:modprobe and honorit.* gnu/build/linux-boot.scm (boot-system): Add #:modprobe and honor it.Call 'load-needed-linux-modules'.* gnu/system/vm.scm (qemu-image): Add #:linux parameter.  Define'modprobe-wrapper' and pass it to 'activate-modprobe'.  Pass #:linux to'expression->derivation-in-linux-vm'.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>--- gnu/build/linux-boot.scm    | 13 +++++- gnu/build/linux-initrd.scm  | 16 ++++++-- gnu/system/linux-initrd.scm | 97 ++++++++++++++++++++++++++++++++++++++++----- gnu/system/vm.scm           | 21 ++++++++++ 4 files changed, 133 insertions(+), 14 deletions(-)

Toggle diff (297 lines)diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scmindex df0b2b2d1..eedc4bb9d 100644--- a/gnu/build/linux-boot.scm+++ b/gnu/build/linux-boot.scm@@ -435,6 +435,7 @@ bailing out.~%root contents: ~s~%" (scandir "/"))   (define* (boot-system #:key+                      modprobe                       (linux-modules '())                       linux-module-directory                       qemu-guest-networking?@@ -449,6 +450,9 @@ QEMU-GUEST-NETWORKING? is true, calling PRE-MOUNT, mounting the file systems specified in MOUNTS, and finally booting into the new root if any.  The initrd supports kernel command-line options '--load', '--root', and '--repl'. +MODPROBE must be #f or a program to install as the modprobe program that the+kernel will invoke when it needs to load modules.+ Mount the root file system, specified by the '--root' command-line argument, if any. @@ -482,9 +486,14 @@ upon error."        (when (member "--repl" args)          (start-repl)) +       (when modprobe+         ;; Tell the kernel to invoke MODPROBE.+         (call-with-output-file "/proc/sys/kernel/modprobe"+           (lambda (port)+             (display modprobe port))))+        (display "loading kernel modules...\n")-       (load-linux-modules-from-directory linux-modules-                                          linux-module-directory)+       (load-needed-linux-modules linux-module-directory)         (when qemu-guest-networking?          (unless (configure-qemu-networking)diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scmindex c65b5aacf..4fa2bee7d 100644--- a/gnu/build/linux-initrd.scm+++ b/gnu/build/linux-initrd.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2013, 2014, 2015 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2013, 2014, 2015, 2018 Ludovic Courtès <ludo@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -107,12 +107,18 @@ This is similar to what 'compiled-file-name' in (system base compile) does."  (define* (build-initrd output                        #:key-                       guile init+                       guile init modprobe                        (references-graphs '())                        (gzip "gzip"))   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script at INIT, running GUILE.  It contains all the items referred to by-REFERENCES-GRAPHS."+REFERENCES-GRAPHS.++When MODPROBE is true, make /sbin/modprobe a symlink to it.  This is useful+because Linux invokes 'modprobe' when it needs to load a module and its+default file name is '/sbin/modprobe' (see 'call_modprobe' in kernel/kmod.c).+Creating this symlink allows us to make sure there's no time window during+which 'modprobe' is unavailable."   (mkdir "contents")    ;; Copy the closures of all the items referenced in REFERENCES-GRAPHS.@@ -131,6 +137,10 @@ REFERENCES-GRAPHS."     (symlink (string-append guile "/bin/guile") "proc/self/exe")     (readlink "proc/self/exe") +    (when modprobe+      (mkdir-p "sbin")+      (symlink modprobe "sbin/modprobe"))+     ;; Reset the timestamps of all the files that will make it in the initrd.     (for-each (lambda (file)                 (unless (eq? 'symlink (stat:type (lstat file)))diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 1eb5f5130..7a167146f 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -63,16 +63,87 @@ ;;; ;;; Code: +(define* (modprobe-program linux-module-directory #:key+                           (guile %guile-static-stripped))+  "Return a minimal implementation of 'modprobe' for our initrd that looks up+modules in LINUX-MODULE-DIRECTORY.  This program will be invoked by the kernel+when modules need to be loaded."+  (define program+    (with-imported-modules (source-module-closure+                            '((gnu build linux-modules)))+      #~(begin+          (use-modules (gnu build linux-modules)+                       (ice-9 match)+                       (srfi srfi-1)+                       (srfi srfi-26)+                       (srfi srfi-37))++          (define option-spec+            (list (option '(#\q "quiet") #f #f+                          (lambda (opt name arg result)+                            (alist-cons 'quiet? #t result)))))++          (define options+            ;; Alist of options and non-option arguments.+            (args-fold (cdr (program-arguments))+                       option-spec+                       (lambda (opt name arg result)+                         (error "unrecognized option" name))+                       (lambda (arg result)+                         (alist-cons 'argument arg result))+                       '()))++          (define alias+            ;; The alias we are asked to load.  The remaining arguments are+            ;; module parameters.  In practice the kernel doesn't pass module+            ;; parameters so we ignore them here.+            (any (match-lambda+                   (('argument . alias) alias)+                   (_ #f))+                 options))++          (define linux-module-directory+            ;; The module directory.  Note: We expect a flat directory here.+            #$linux-module-directory)++          (define %known-aliases+            ;; The alias database.+            (known-module-aliases+             (string-append linux-module-directory "/modules.alias")))++          (when (assq-ref options 'quiet?)+            (current-error-port (%make-void-port "w"))+            (current-output-port (%make-void-port "w")))++          (let ((modules (matching-modules alias %known-aliases)))+            (call-with-output-file "/dev/kmsg"+              (lambda (port)+                (setvbuf port 'block 1024)+                (format port "modprobe[~a]: alias ~s; modules ~s; args ~s~%"+                        (getpid) alias modules (program-arguments))))++            (when (null? modules)+              (error "alias resolution failed" alias))++            (load-linux-modules-from-directory modules+                                               linux-module-directory)))))++  (program-file "modprobe" program #:guile guile))  (define* (expression->initrd exp                              #:key+                             modprobe                              (guile %guile-static-stripped)                              (gzip gzip)                              (name "guile-initrd")                              (system (%current-system)))   "Return a derivation that builds a Linux initrd (a gzipped cpio archive) containing GUILE and that evaluates EXP, a G-expression, upon booting.  All-the derivations referenced by EXP are automatically copied to the initrd."+the derivations referenced by EXP are automatically copied to the initrd.++When MODPROBE is true, '/sbin/modprobe' is created as a symlink pointing to+it.  This allows Linux to call out to MODPROBE as soon as it boots if it needs+to load modules."    ;; General Linux overview in `Documentation/early-userspace/README' and   ;; `Documentation/filesystems/ramfs-rootfs-initramfs.txt'.@@ -89,20 +160,21 @@ the derivations referenced by EXP are automatically copied to the initrd."           (mkdir #$output)            ;; The guile used in the initrd must be present in the store, so-          ;; that module loading works once the root is switched.+          ;; that module loading works once the root is switched.  Similarly,+          ;; the 'modprobe' program installed in /proc/sys/kernel/modprobe by+          ;; the initrd, if any, must be present after switch root.           ;;-          ;; To ensure that is the case, add an explicit reference to the-          ;; guile package used in the initrd to the output.+          ;; To ensure that is the case, add an explicit reference to these in+          ;; the output.           ;;-          ;; This fixes guix-patches bug #28399, "Fix mysql activation, and+          ;; This fixes <https://bugs.gnu.org/28399>, "Fix mysql activation, and           ;; add a basic test".-          (call-with-output-file (string-append #$ output "/references")-            (lambda (port)-              (simple-format port "~A\n" #$guile)))+          (copy-file "closure" (string-append #$output "/references"))            (build-initrd (string-append #$output "/initrd")                         #:guile #$guile                         #:init #$init+                        #:modprobe #$modprobe                         ;; Copy everything INIT refers to into the initrd.                         #:references-graphs '("closure")                         #:gzip (string-append #$gzip "/bin/gzip")))))@@ -153,7 +225,9 @@ MODULES and taken from LINUX."                       (copy-file module                                  (string-append #$output "/"                                                 (basename module))))-                    (delete-duplicates modules)))))+                    (delete-duplicates modules))++          (write-module-alias-database #$output))))    (computed-file "linux-modules" build-exp)) @@ -196,6 +270,9 @@ upon error."   (define kodir     (flat-linux-module-directory linux linux-modules)) +  (define modprobe+    (modprobe-program kodir))+   (expression->initrd    (with-imported-modules (source-module-closure                            '((gnu build linux-boot)@@ -229,9 +306,11 @@ upon error."                                     (and #$@device-mapping-commands))                       #:linux-modules '#$linux-modules                       #:linux-module-directory '#$kodir+                      #:modprobe #$modprobe                       #:qemu-guest-networking? #$qemu-networking?                       #:volatile-root? '#$volatile-root?                       #:on-error '#$on-error)))+   #:modprobe modprobe    #:name "raw-initrd"))  (define* (file-system-packages file-systems #:key (volatile-root? #f))diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 4360adf15..7f552bef8 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -249,6 +249,7 @@ INPUTS is a list of inputs (as for packages)." (define* (qemu-image #:key                      (name "qemu-image")                      (system (%current-system))+                     (linux linux-libre)                      (qemu qemu-minimal)                      (disk-image-size 'guess)                      (disk-image-format "qcow2")@@ -275,18 +276,37 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true, register INPUTS in the store database of the image so that Guix can be used in the image."+  (define modprobe-wrapper+    ;; Wrapper for the 'modprobe' command that knows where modules live.+    (let ((modprobe (file-append kmod "/bin/modprobe")))+      (program-file "modprobe"+                    #~(begin+                        (setenv "LINUX_MODULE_DIRECTORY"+                                #$(file-append linux "/lib/modules"))+                        (apply execl #$modprobe+                               (cons #$modprobe (cdr (command-line))))))))++   (expression->derivation-in-linux-vm    name    (with-imported-modules (source-module-closure '((gnu build bootloader)                                                    (gnu build vm)+                                                   (gnu build activation)                                                    (guix build utils)))      #~(begin          (use-modules (gnu build bootloader)                       (gnu build vm)+                      ((gnu build activation) #:select (activate-modprobe))                       (guix build utils)                       (srfi srfi-26)                       (ice-9 binary-ports)) +         ;; We may need to lazy-load Linux modules.  The initrd installs a+         ;; 'modprobe' that can only search through the modules available in+         ;; the initrd, but here we want to be able to use all the modules of+         ;; LINUX.  Thus, install a real 'modprobe'.+         (activate-modprobe #$modprobe-wrapper)+          (let ((inputs                 '#$(append (list qemu parted e2fsprogs dosfstools)                            (map canonical-package@@ -361,6 +381,7 @@ the image."                                    #$(bootloader-installer bootloader))              (reboot)))))    #:system system+   #:linux linux    #:make-disk-image? #t    #:disk-image-size disk-image-size    #:disk-image-format disk-image-format-- 2.16.2

Ludovic Courtès wrote on 12 Mar 2018 23:15

[PATCH v11 6/6] linux-modules: Add "modules.devname" writer.

Recipients:(address . 30604@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20180312221541.1886-6-ludo@gnu.org

* gnu/build/linux-modules.scm (aliases->device-tuple)(write-module-device-database): New procedures.
Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>.--- gnu/build/linux-modules.scm | 51 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+)

Toggle diff (71 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 55161e026..d3ba2c60a 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -47,6 +47,8 @@             known-module-aliases             matching-modules             write-module-alias-database+            write-module-device-database+             load-needed-linux-modules))  ;;; Commentary:@@ -415,6 +417,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by                              aliases)))                 aliases)))) +(define (aliases->device-tuple aliases)+  "Traverse ALIASES, a list of module aliases, and search for+\"char-major-M-N\", \"block-major-M-N\", or \"devname:\" aliases.  When they+are found, return a tuple (DEVNAME TYPE MAJOR MINOR), otherwise return #f."+  (define (char/block-major? alias)+    (or (string-prefix? "char-major-" alias)+        (string-prefix? "block-major-" alias)))++  (define (char/block-major->tuple alias)+    (match (string-tokenize alias %not-dash)+      ((type "major" (= string->number major) (= string->number minor))+       (list (match type+               ("char" "c")+               ("block" "b"))+             major minor))))++  (let* ((devname     (any (lambda (alias)+                             (and (string-prefix? "devname:" alias)+                                  (string-drop alias 8)))+                           aliases))+         (major/minor (match (find char/block-major? aliases)+                        (#f #f)+                        (str (char/block-major->tuple str)))))+    (and devname major/minor+         (cons devname major/minor))))++(define %not-dash+  (char-set-complement (char-set #\-)))++(define (write-module-device-database directory)+  "Traverse the '.ko' files in DIRECTORY and create the corresponding+'modules.devname' file.  This file contains information about modules that can+be loaded on-demand, such as file system modules."+  (define aliases+    (filter-map (lambda (file)+                  (match (aliases->device-tuple (module-aliases file))+                    (#f #f)+                    (tuple (cons (file-name->module-name file) tuple))))+                (find-files directory "\\.ko$")))++  (call-with-output-file (string-append "/tmp" "/modules.devname")+    (lambda (port)+      (display "# Device nodes to trigger on-demand module loading.\n" port)+      (for-each (match-lambda+                  ((module devname type major minor)+                   (format port "~a ~a ~a~a:~a~%"+                           module devname type major minor)))+                aliases))))+ (define (load-needed-linux-modules module-directory)   "Examine /sys/devices to find out which modules are needed and load those we have in MODULE-DIRECTORY.  Return the list of modules loaded, not including-- 2.16.2

Danny Milosavljevic wrote on 12 Mar 2018 23:48

Re: [PATCH v11 1/6] linux-modules: Add "modules.alias" writer.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312234814.6b89aecc@scratchpost.org

Hi Ludo,
On Mon, 12 Mar 2018 23:15:36 +0100Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)

> (%not-dash): New variable.

Old commit message?
Otherwise LGTM!

Danny Milosavljevic wrote on 12 Mar 2018 23:48

Re: [bug#30604] [PATCH v11 2/6] linux-modules: Add 'load-linux-modules-from-directory'.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312234836.106e2eb0@scratchpost.org

LGTM!

Danny Milosavljevic wrote on 12 Mar 2018 23:49

Re: [PATCH v11 4/6] vm: Make the virtio-blk uniquely identifiable in /sys.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312234922.14fc6425@scratchpost.org

LGTM!

Danny Milosavljevic wrote on 12 Mar 2018 23:55

Re: [bug#30604] [PATCH v11 6/6] linux-modules: Add "modules.devname" writer.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180312235547.6ac0af0a@scratchpost.org

Toggle quote (1 lines)

> + (call-with-output-file (string-append "/tmp" "/modules.devname")

^^^^^^ directory
Otherwise LGTM!

Danny Milosavljevic wrote on 13 Mar 2018 00:00

Re: [bug#30604] [PATCH v11 3/6] linux-modules: Add 'load-needed-linux-modules'.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180313000031.06767edf@scratchpost.org

Toggle quote (1 lines)

> +(define (load-needed-linux-modules module-directory)

[...]

Toggle quote (2 lines)

> + (file-system-fold enter? leaf down up skip error #t "/sys/devices"))

Isn't #t a strange initial value? Shouldn't it be '() ?

Ludovic Courtès wrote on 13 Mar 2018 09:50

Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:87fu54fjfj.fsf@gnu.org

Hello!
I’m using this on the bare metal. \o/
I had to make this change so that dm-crypt would be loaded once I’veentered my passphrase:

Toggle diff (27 lines)diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scmindex 7a167146f..ef81fe718 100644--- a/gnu/system/linux-initrd.scm+++ b/gnu/system/linux-initrd.scm@@ -115,16 +115,19 @@ when modules need to be loaded."             (current-error-port (%make-void-port "w"))             (current-output-port (%make-void-port "w"))) -          (let ((modules (matching-modules alias %known-aliases)))+          ;; Look up ALIAS in %KNOWN-ALIASES first, and then, if it fails,+          ;; assume that ALIAS is a module name (some modules such as+          ;; 'dm-crypt' don't have an alias.)  Note that+          ;; 'kmod_module_new_from_lookup' uses a different search order.+          (let ((modules (match (matching-modules alias %known-aliases)+                           (()  (list alias))+                           (lst lst))))             (call-with-output-file "/dev/kmsg"               (lambda (port)                 (setvbuf port 'block 1024)                 (format port "modprobe[~a]: alias ~s; modules ~s; args ~s~%"                         (getpid) alias modules (program-arguments)))) -            (when (null? modules)-              (error "alias resolution failed" alias))-             (load-linux-modules-from-directory modules                                                linux-module-directory)))))

FWIW I have:

Toggle snippet (12 lines)modprobe[146]: alias "net-pf-38"; modules ("net-pf-38"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "net-pf-38")modprobe[152]: alias "dm-crypt"; modules ("dm-crypt"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "dm-crypt")modprobe[157]: alias "crypto-xts(aes)"; modules ("crypto-xts(aes)"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts(aes)")modprobe[162]: alias "crypto-xts(aes)-all"; modules ("crypto-xts(aes)-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts(aes)-all")modprobe[168]: alias "crypto-xts"; modules ("xts"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts")nfo>  [1520929735.4127] manager[0x12950c0]: monitoring kernel firmware directory '/lib/firmware'.modprobe[173]: alias "crypto-aes"; modules ("crypto-aes"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-aes")modprobe[178]: alias "crypto-aes-all"; modules ("crypto-aes-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-aes-all")modprobe[183]: alias "crypto-ecb(aes)"; modules ("crypto-ecb(aes)"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-ecb(aes)")modprobe[188]: alias "crypto-ecb(aes)-all"; modules ("crypto-ecb(aes)-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-ecb(aes)-all")

Something annoying is that my external USB keyboard doesn’t work whilein the initrd (when I type my passphrase). I can see that it’s detectedearly on, before I type my passphrase:

Toggle snippet (7 lines)

[ 1.532237] usb 1-4.2: new low-speed USB device number 4 using xhci_hcd[ 1.648836] usb 1-4.2: New USB device found, idVendor=413c, idProduct=2105[ 1.649189] usb 1-4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0[ 1.649533] usb 1-4.2: Product: Dell USB Keyboard[ 1.649882] usb 1-4.2: Manufacturer: Dell

but its modules don’t get loaded until after eudev has been started, andthere’s no trace of a modprobe invocation for it.

Toggle snippet (3 lines)

[ 20.968938] input: Dell Dell USB Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4.2/1-4.2:1.0/0003:413C:2105.0001/input/input16

Any idea what we’re missing?
Thanks,Ludo’.

Ludovic Courtès wrote on 13 Mar 2018 09:54

Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87bmfsfj8g.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (14 lines)>> That was why earlier I kept /sbin/modprobe as the name (it's the>> default Linux uses) and made sure that the file /sbin/modprobe is already>> in the initrd (as opposed to created by boot-system).>> Note: The simplest change would be to keep this patch as-is but add >>     ;; Make sure that Linux can modprobe even before boot-system started up.>     (when modprobe>       (mkdir-p "sbin")>       (symlink modprobe "sbin/modprobe")>       (compile-to-cache "sbin/modprobe")) ; Note: without this line booting is dead slow. [Maybe in this case that wouldn't be so bad - since boot-system switches away immediately anyway :)]>> to gnu/build/linux-initrd.scm

Yeah, done. Ideally ‘program-file’ would take care of this. Futurework!
Ludo’.

Ludovic Courtès wrote on 13 Mar 2018 09:55

Re: [bug#30604] [PATCH v11 3/6] linux-modules: Add 'load-needed-linux-modules'.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:877eqgfj6t.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (6 lines)

>> +(define (load-needed-linux-modules module-directory)> [...]>> + (file-system-fold enter? leaf down up skip error #t "/sys/devices"))>> Isn't #t a strange initial value? Shouldn't it be '() ?

Indeed! Fixed, thanks.

Ludovic Courtès wrote on 13 Mar 2018 09:56

Re: [bug#30604] [PATCH v11 6/6] linux-modules: Add "modules.devname" writer.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:873714fj5k.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (5 lines)

>> + (call-with-output-file (string-append "/tmp" "/modules.devname")> ^^^^^^ directory>> Otherwise LGTM!

Oops, fixed!

Danny Milosavljevic wrote on 13 Mar 2018 10:27

Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180313100539.442c4aa9@scratchpost.org

Hi Ludo,
On Mon, 12 Mar 2018 23:14:59 +0100ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (7 lines)

> > One of the devnames created statically is the one for btrfs, so not writing or> > using devnames is not going to end well. > > We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,> “fs-9p”, etc. alias, which shows up in “modules.alias”. No need for> “modules.devname” AFAICS.

The other filesystems are not such a problem - but BTRFS has its own snapshotting/multivolume functionality - and it's possible that someone accesses /dev/btrfs-control"too early" for that.
I applied your patches to a fresh clone of guix master now, ran the btrfs-root-ossystem check, and indeed I get (tried two rounds, happened both times):
$ make TESTS=btrfs-root-os check-system[...]Scanning for Btrfs filesystemsWARNING: failed to open /dev/btrfs-control, skipping device registration: No suyERROR: there are 1 errors while registering devicesFile system check on /dev/vda2 failed; spawning Bourne-like REPLGNU Guile 2.2.3Copyright (C) 1995-2017 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.This program is free software, and you are welcome to redistribute itunder certain conditions; type `,show c' for details.
Enter `,help' for help.(can't evaluate anything here)

Toggle quote (6 lines)

> > (I'd also copy the modules.builtin (from Linux).> > Also, what happens if we load a module which has as dependency a builtin?> > Will we try to load the builtin as a .ko file and fail the entire thing?) > > The dependency of a builtin is necessarily a builtin,

Yes.

Toggle quote (2 lines)

>and the kernel won’t invoke modprobe for a builtin, will it?

I've read Linux's __request_module and I can't find where theypre-check anything - neither whether there's already a builtinnor whether it's loaded already.
They probably don't check. But I'll read it again, more carefully...
(request_module isn't that popular so it makes sense for them not to complicatethe kernel by these checks when there are like 8 callers in total - and all on init)

Toggle quote (2 lines)

>Thanks for the insightful review!

You're welcome :)

Ludovic Courtès wrote on 13 Mar 2018 10:28

Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:876060qq7f.fsf@gnu.org

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (4 lines)

> Something annoying is that my external USB keyboard doesn’t work while> in the initrd (when I type my passphrase). I can see that it’s detected> early on, before I type my passphrase:

I tried looking up “modalias” right when entering a /sys/devicesdirectory, like you did in the initial patch, but that didn’t help:

Toggle diff (34 lines)diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scmindex 4515839e9..213a8bbb9 100644--- a/gnu/build/linux-modules.scm+++ b/gnu/build/linux-modules.scm@@ -480,19 +480,22 @@ side-effect and would thus need to traverse /sys/devices again."      (string-append module-directory "/modules.alias")))    (define (enter? director stat result) result)-  (define (down directory stat result) result)-  (define (up directory stat result) result)-  (define (skip entry stat result) result)-  (define (error name stat errno result) result)-  (define (leaf file stat result)-    (if (string=? (basename file) "modalias")+  (define (down directory stat result)+    ;; Check for "modalias" right when entering DIRECTORY instead of waiting+    ;; for LEAF to be called.  XXX: Doesn't help!+    (let ((modalias (string-append directory "/modalias")))+      (if (file-exists? modalias)           (let* ((alias   (string-trim-right-                         (call-with-input-file file get-string-all)))+                           (call-with-input-file modalias get-string-all)))                  (modules (matching-modules alias aliases)))             (load-linux-modules-from-directory modules                                                module-directory)             (append modules result))-        result))+          result)))+  (define (up directory stat result) result)+  (define (skip entry stat result) result)+  (define (error name stat errno result) result)+  (define (leaf file stat result) file stat result)    (file-system-fold enter? leaf down up skip error '() "/sys/devices"))

Ludo’.

Ludovic Courtès wrote on 13 Mar 2018 10:29

control message for bug #30604

Recipients:(address . control@debbugs.gnu.org)

Message-ID:874llkqq72.fsf@gnu.org

severity 30604 important

Ludovic Courtès wrote on 13 Mar 2018 11:51

Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87zi3cp7sx.fsf@gnu.org

Hello,
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (32 lines)> On Mon, 12 Mar 2018 23:14:59 +0100> ludo@gnu.org (Ludovic Courtès) wrote:>>> > One of the devnames created statically is the one for btrfs, so not writing or>> > using devnames is not going to end well.  >> >> We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,>> “fs-9p”, etc. alias, which shows up in “modules.alias”.  No need for>> “modules.devname” AFAICS.>> The other filesystems are not such a problem - but BTRFS has its own snapshotting/> multivolume functionality - and it's possible that someone accesses /dev/btrfs-control> "too early" for that.>> I applied your patches to a fresh clone of guix master now, ran the btrfs-root-os> system check, and indeed I get (tried two rounds, happened both times):>> $ make TESTS=btrfs-root-os check-system> [...]> Scanning for Btrfs filesystems> WARNING: failed to open /dev/btrfs-control, skipping device registration: No suy> ERROR: there are 1 errors while registering devices> File system check on /dev/vda2 failed; spawning Bourne-like REPL> GNU Guile 2.2.3> Copyright (C) 1995-2017 Free Software Foundation, Inc.>> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.> This program is free software, and you are welcome to redistribute it> under certain conditions; type `,show c' for details.>> Enter `,help' for help.

Do you see a modprobe invocation for “fs-btrfs” or “block-major-xxx”before?

Toggle quote (19 lines)>> > (I'd also copy the modules.builtin (from Linux).>> >  Also, what happens if we load a module which has as dependency a builtin?>> >  Will we try to load the builtin as a .ko file and fail the entire thing?)  >> >> The dependency of a builtin is necessarily a builtin, >> Yes.>>>and the kernel won’t invoke modprobe for a builtin, will it?  >> I've read Linux's __request_module and I can't find where they> pre-check anything - neither whether there's already a builtin> nor whether it's loaded already.>> They probably don't check.  But I'll read it again, more carefully...>> (request_module isn't that popular so it makes sense for them not to complicate> the kernel by these checks when there are like 8 callers in total - and all on init)

Right, and the worst that can happen is that modprobe will fail, butthat’s fine because the functionality is already there anyway.
Ludo’.

Danny Milosavljevic wrote on 13 Mar 2018 13:05

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180313130557.247b02c6@scratchpost.org

Hi Ludo,
On Tue, 13 Mar 2018 11:51:42 +0100ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (3 lines)

> Right, and the worst that can happen is that modprobe will fail, but> that’s fine because the functionality is already there anyway.

Yeah - but what does the kernel do when we return a non-zero exit code for something he wants to use? :)
./fs/filesystems.c: if (!fs && (request_module("%.*s", len, name) == 0)) ^^^^ uh oh

Danny Milosavljevic wrote on 13 Mar 2018 13:28

Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180313132154.212a1b92@scratchpost.org

Toggle quote (2 lines)

> I’m using this on the bare metal. \o/

Hehe nice!

Toggle quote (6 lines)

> - (let ((modules (matching-modules alias %known-aliases)))> + ;; Look up ALIAS in %KNOWN-ALIASES first, and then, if it fails,> + ;; assume that ALIAS is a module name (some modules such as> + ;; 'dm-crypt' don't have an alias.) Note that> + ;; 'kmod_module_new_from_lookup' uses a different search order.

I also wondered about the resolution order :)
At least module dependencies can't be aliases.

Toggle quote (4 lines)

> + (let ((modules (match (matching-modules alias %known-aliases)> + (() (list alias))> + (lst lst))))

Yeah, same happened to me. Did the same (back then in "resolve-alias").
So should be fine.

Toggle quote (3 lines)

> Something annoying is that my external USB keyboard doesn’t work while> in the initrd (when I type my passphrase).

Did it work before? I remember that Andreas had a similar problem before all this.
Did you implement globs with [0-9] yet? I remember there were some USB devicesusing those :)
What's the modalias of your usb keyboard?
Does your initrd's modules.alias contain any patterns that are supposed to match?
(You can look into those via cat initrd | gunzip -cd |cpio -i
- it will unpack)

Danny Milosavljevic wrote on 13 Mar 2018 20:17

Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180313201702.259bccd0@scratchpost.org

Hi Ludo,

Toggle quote (3 lines)

> Do you see a modprobe invocation for “fs-btrfs” or “block-major-xxx”> before?

I can't get the logfile to show any modprobe log message in this case(which is "make TESTS=btrfs-root-os check-system").
I had that problem before and I think it's because guix isusing some special /tmp/repl pipe for qemu here[1] and so the messagevanishes somewhere where I can't see it (that is, the guest doeseverything correctly but the host loses it somewhere).
The usual trick I use in this case which is "steal the cmdline andthe files from /proc/<qemu> and invoke it myself" doesn't workeither because of the pipe... hrmmmm...
I tried both /dev/kmsg and just not closing current-output-port etc.
I've double-checked that I've applied all v11 patches, includingthe resolve-alias one (which I think is necessary anyway).
Help?
Log file attached.
[1] When it uses the VM to create the root fs.

Attachment: L.bz2

Ludovic Courtès wrote on 18 Mar 2018 16:03

Keyboard detection before ‘cryptsetup’ runs

Recipients:(address . 30604@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:87o9jlo28l.fsf@inria.fr

Hi Danny,
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (4 lines)

> Something annoying is that my external USB keyboard doesn’t work while> in the initrd (when I type my passphrase). I can see that it’s detected> early on, before I type my passphrase:

While experimenting with this on the bare metal, it became clear thatit’s a timing issue: the keyboard is detected right after the cryptsetupis displayed, so ‘load-needed-linux-modules’ didn’t have a chance toload the relevant modules.
Similarly, if I boot with ‘--repl’, and I manually type(load-needed-linux-modules …) from there, then the keyboard’s modulegets loaded as expected. That’s because the device showed up in themeantime and the kernel created a /sys entry for it.
In essence, we want a mini-udev and something akin to “udevadm settle”.Merely calling ‘load-needed-linux-modules’ once isn’t enough.
One way to do that would be to have a separate thread that calls‘load-needed-linux-modules’ as appropriate. Ideally it would useinotify on /sys like udev does, but a poor programmer’s version couldsimply call ‘load-needed-linux-modules’ every half a second or so.
Alternately, before passing control to user code (pre-mount actions,etc.), we could do a “settle” kind of thing: call‘load-linux-modules-from-directory’ every 0.5 seconds until its resultis the same as before. There’s still a risk of missing devices, andthose devices will never show up later because nobody’s monitoring /sys.But then again, “udevadm settle” must have the same problem: it can’treally know whether things have settled, I guess.
WDYT?
Thanks,Ludo’.

Danny Milosavljevic wrote on 23 Mar 2018 14:02

Re: Keyboard detection before ‘cryptsetup ’ runs

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180323140231.1c96615c@scratchpost.org

Hi Ludo,
I thought about it for a bit...
On Sun, 18 Mar 2018 16:03:06 +0100ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (13 lines)> One way to do that would be to have a separate thread that calls> ‘load-needed-linux-modules’ as appropriate.  Ideally it would use> inotify on /sys like udev does, but a poor programmer’s version could> simply call ‘load-needed-linux-modules’ every half a second or so.> > Alternately, before passing control to user code (pre-mount actions,> etc.), we could do a “settle” kind of thing: call> ‘load-linux-modules-from-directory’ every 0.5 seconds until its result> is the same as before.  There’s still a risk of missing devices, and> those devices will never show up later because nobody’s monitoring /sys.> But then again, “udevadm settle” must have the same problem: it can’t> really know whether things have settled, I guess.

Yes, but udevd continues running and monitoring /sys anyway - so I guessthey don't really care (if I wrote it, I'd start up the monitoring firstand then traverse /sys for modalias - and make sure the modprobe isidempotent).
I have to say what we have to do is much more complicated than I thoughtit would be - and I kinda regret starting this patchset. I didn'twant to make it less likely for people to boot than before :-(
I suggest we find out how the other distributions do it - if they do it.
A small bit of research shows that:
- Busybox uses a loop with timeout to try to mount over and over again,depending on bus type [2].- Alpine coldplugs twice [1].- Fatdog has a "waitdev" boot parameter that specifies how long it sleepsuntil it tries to mount the root.- Redhat mkinitrd has an mkinitrd option like "--with=scsi_wait_scan" whichthey use after they saw a modalias starting with "scsi:" - or add thekernel option "scsi_mod.scan=sync" to the command line [3] (but they say thatthis is unreliable now and one is supposed to use udev inside the initrd).Not sure what they do for USB.
The Linux kernel itself has a "rootdelay" parameter whichwas specifically introduced because of USB devices that took 15 s (!)to show up.
All in all, we have alternative ways to continue:
(a) Drop this patchset/feature entirely because it's too unreliable.(b) Monitor /sys using inotify in an extra thread/process.(c) Include udev into the initrd and have it do its thing.(d) Monitor the netlink socket in our own thread and play udev ourselves.(e) Find out how to, for each bus type, detect when enumeration is finished.Find out which buses are there. Wait for them to finish enumerating.(f) Always modprobe usb-hid first, no matter what. I think this wouldbecome a whack-a-mole thing fast.
I'd prefer either (a) or (b) for reliability and simplicity. WDYT?
(inotify is Linux-only, but so is /sys/devices :P)
[1] https://git.alpinelinux.org/cgit/abuild/tree/initramfs-init?id=9154b0d9b7251cca900f9e834ac9af9ae2ae61d9(scan_drivers)[2] http://lists.busybox.net/pipermail/busybox/2015-March/082688.html[3] https://bugzilla.redhat.com/show_bug.cgi?id=466607

Danny Milosavljevic wrote on 23 Mar 2018 16:07

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 30604@debbugs.gnu.org)

Message-ID:20180323160702.45fa3fe5@scratchpost.org

Toggle quote (2 lines)

> (b) Monitor /sys using inotify in an extra thread/process.

Oh - or loop { load-/sys ; sleep 1 }. That's what you meant.

Ludovic Courtès wrote on 24 Mar 2018 18:07

Re: Keyboard detection before ‘cryptsetup’ runs

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87po3tjtb3.fsf@gnu.org

Hi Danny!
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (4 lines)

>> (b) Monitor /sys using inotify in an extra thread/process.>> Oh - or loop { load-/sys ; sleep 1 }. That's what you meant.

Yeah, though we wouldn’t know when to stop, but if I read your messagecorrectly, nobody does. :-)
Ludo’.

Ludovic Courtès wrote on 24 Mar 2018 18:12

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 30604@debbugs.gnu.org)

Message-ID:87h8p5jt2j.fsf@gnu.org

Heya,
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (4 lines)

> I have to say what we have to do is much more complicated than I thought> it would be - and I kinda regret starting this patchset. I didn't> want to make it less likely for people to boot than before :-(

Yeah, I had a nightmare where I dreamed of all the bug reports due todevices not showing up quickly enough. :-)

Toggle quote (17 lines)> A small bit of research shows that:>> - Busybox uses a loop with timeout to try to mount over and over again,> depending on bus type [2].> - Alpine coldplugs twice [1].> - Fatdog has a "waitdev" boot parameter that specifies how long it sleeps> until it tries to mount the root.> - Redhat mkinitrd has an mkinitrd option like "--with=scsi_wait_scan" which> they use after they saw a modalias starting with "scsi:" - or add the> kernel option "scsi_mod.scan=sync" to the command line [3] (but they say that> this is unreliable now and one is supposed to use udev inside the initrd).> Not sure what they do for USB.>> The Linux kernel itself has a "rootdelay" parameter which> was specifically introduced because of USB devices that took 15 s (!)> to show up.

It’s a kludge fest, which is reassuring, in a way.

Toggle quote (13 lines)> All in all, we have alternative ways to continue:>> (a) Drop this patchset/feature entirely because it's too unreliable.> (b) Monitor /sys using inotify in an extra thread/process.> (c) Include udev into the initrd and have it do its thing.> (d) Monitor the netlink socket in our own thread and play udev ourselves.> (e) Find out how to, for each bus type, detect when enumeration is finished.> Find out which buses are there.  Wait for them to finish enumerating.> (f) Always modprobe usb-hid first, no matter what.  I think this would> become a whack-a-mole thing fast.>> I'd prefer either (a) or (b) for reliability and simplicity.  WDYT?

Same here! Sounds like (b) is something we could easily try, though wewon’t know how well it works until people start running it on the baremetal.
I can adjust the branch I have to do that and push it for people to try.
Thanks a lot of your feedback!
Ludo’.

Ludovic Courtès wrote on 26 Mar 2018 15:37

Re: bug#30922: LUKS-encrypted root fails using device numbering, needs luksUUID

Recipients:(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)

Message-ID:87efk7dkle.fsf@gnu.org

Hello,
Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (4 lines)

> After booting I get>> Device /dev/nvme0n1p1 doesn't exist or access denied

This message comes from ‘cryptsetup’.
The ‘init’ script for the config you posted does this:

Toggle snippet (26 lines)(boot-system #:mounts (map spec->file-system                           (quote (("/dev/mapper/guix"                                    device "/" "ext4" ()                                    #f #t))))             #:pre-mount (lambda ()                           (and (let ((source "/dev/nvme0n1p1"))                                  (use-modules (rnrs bytevectors)                                               ((gnu build file-systems)                                                #:select                                                (find-partition-by-luks-uuid)))                                  (zero? (system*                                          "/gnu/store/v5r0dx5v08847rcy033kyish6m3b3621-cryptsetup-static-1.7.5/sbin/cryptsetup"                                          "open" "--type" "luks"                                          (if (bytevector? source)                                              (or (let loop ((tries-left 10))                                                    (and (positive? tries-left)                                                         (or (find-partition-by-luks-uuid source)                                                             (begin                                                               (sleep 1)                                                               (loop (- tries-left 1))))))                                                  (error "LUKS partition not found" source))                                              source)                                          "guix")))))             …

IOW, it does the right thing whether or not you’re using a UUID.
However, when using a UUID, it waits for the device to show up, whereasotherwise it assumes the /dev node is already there and invokes‘cryptsetup’ right away.
We should fix that and generalize the wait-for-device loop, whichalready exists in a couple of instances.
There are connections with https://bugs.gnu.org/30604.
Ludo’.

Ludovic Courtès wrote on 16 Aug 2019 23:10

Re: bug#34902: guix cannot find a module on boot

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:874l2gg6ko.fsf@gnu.org

Hello!
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (14 lines)> Maybe I'm too paranoid but can we have "guix" in the file name "modules.name"> somewhere?  Otherwise I see it coming that upstream uses modules.name for an> incompatible purpose and then we'd be with a guix interface that's broken> and/or break their interface.>> (So much complexity for something so silly.  Honestly, I feel like E-mailing> the upstream author and telling him what I think.  WTF :P)>> Should we warn when we use the fallback?  I like the defensive programming> but I feel we shouldn't have it *silently* fall back when the database is> broken/missing.>> Otherwise LGTM!

So I went ahead and pushed these patches, derived from our beautifulpatch set at https://issues.guix.gnu.org/issue/30604:
c85ccf60bf linux-modules: Define and use a module name database. e1a9a7f275 linux-modules: Add 'load-linux-modules-from-directory'. 2a693b69ca linux-modules: Add "modules.devname" writer. 4f8b9d1a6f linux-modules: Add "modules.alias" writer.
The actual fix for the hyphen/underscore mismatch that Julien reportedis commit c85ccf60bf. The “modules.devname” and “modules.alias” areactually unused so far but (1) it was easier to preserve them, and (2)that’ll give us an incentive to finishhttps://issues.guix.gnu.org/issue/30604. :-)
I added an explicit comment that “modules.name” uses a Guix-specificformat. We can always rename it if the kernel folks decide to acquirethat file name.
Julien, could you please confirm that your initial issue is fixed?
Thanks,Ludo’.

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 30604@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date