[PATCH 0/3] Use native qemu to build vm-image.

  • Done
  • quality assurance status badge
Details
4 participants
  • Jan Nieuwenhuizen
  • Ludovic Courtès
  • Maxim Cournoyer
  • Mathieu Othacehe
Owner
unassigned
Submitted by
Jan Nieuwenhuizen
Severity
normal
J
J
Jan Nieuwenhuizen wrote on 17 May 2020 12:01
(address . guix-patches@gnu.org)
87mu66q3rt.fsf@gnu.org
Hi!

Cross-building a vm-image used to be done using a cross-qemu, e.g, qemu-ARM.
That does not work for the Hurd, as there is no qemu-HURD.

This patch switches to cross building vm-images using a native qemu vm. If
there a reason for using qemu-TARGET we may want to make this switch
conditional for cross building to the Hurd?

There is a tricky aspect to this: the "user-builder" includes all the
cross-built packages for TARGET, but we need to hard the native qemu a
native #+LOADER, loading native #+LINUX and #+INITRD.

To accomplish this NATIVE/CROSS mixt, the builder is then run with a native
guile, like so

Toggle snippet (5 lines)
(system* #+(file-append (default-guile) "/bin/guile")
"--no-auto-compile"
#$(preserve-target user-builder))

while the packages in USER-BUILDER will still be cross-built. However,
because the builder has

(with-extensions gcrypt-sqlite3&co
(use-modules (gnu store database))
...)

this makes the native guile "see" the cross-built .GO files. That would
"only" be a secondary problem except that loading (sqlite3) throws an
exception when dynamic-loading the .SO fails. Seeing it like this I am not so
sure anymore this is a bug, WDYT? ...so instead

...the problem is now avoided by removing the sqlite dependency when
cross-building by not registering closures and postponing the loading of (gnu
store database) and thus (sqlite3).

I have reset wip-hurd-vm onto these changes, so you can also look there.
Doing

./pre-inst-env guix system vm-image --target=i586-pc-gnu --no-grafts \
gnu/system/examples/bare-hurd.tmpl

now produces a pretty nice hurd VM :-)

Greetings,
Janneke

Jan (janneke) Nieuwenhuizen (3):
utils: Move 'reset-timestamps' out of database.
system: vm: Do not register-closures when cross-building.
system: vm: Build vm-image using native qemu.

gnu/bootloader/grub.scm | 4 +--
gnu/build/vm.scm | 3 ++-
gnu/system/vm.scm | 54 ++++++++++++++++++++++++++---------------
guix/store/database.scm | 41 +++----------------------------
guix/utils.scm | 41 ++++++++++++++++++++++++++++---
5 files changed, 79 insertions(+), 64 deletions(-)

--
2.26.0

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 17 May 2020 12:03
[PATCH 1/3] utils: Move 'reset-timestamps' out of database.
(address . 41350@debbugs.gnu.org)
20200517100343.26361-1-janneke@gnu.org
This supports calling reset-timestamps without loading sqlite3.

* guix/store/database.scm (reset-timestamps): Move to...
* guix/utils.scm (reset-timestamps): ... here.
* gnu/build/vm.scm: Include it.
---
gnu/build/vm.scm | 1 +
guix/store/database.scm | 41 +++--------------------------------------
guix/utils.scm | 41 ++++++++++++++++++++++++++++++++++++++---
3 files changed, 42 insertions(+), 41 deletions(-)

Toggle diff (146 lines)
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 433b5a7e8d..c751e6b0e2 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -26,6 +26,7 @@
#:use-module (guix build utils)
#:use-module (guix build store-copy)
#:use-module (guix build syscalls)
+ #:use-module ((guix utils) #:select (reset-timestamps))
#:use-module (guix store database)
#:use-module (gnu build bootloader)
#:use-module (gnu build linux-boot)
diff --git a/guix/store/database.scm b/guix/store/database.scm
index ef52036ede..b8fe313c3d 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -24,9 +24,8 @@
#:use-module (guix store deduplication)
#:use-module (guix base16)
#:use-module (guix progress)
- #:use-module (guix build syscalls)
- #:use-module ((guix build utils)
- #:select (mkdir-p executable-file?))
+ #:use-module ((guix build utils) #:select (mkdir-p))
+ #:use-module ((guix utils) #:select (reset-timestamps))
#:use-module (guix build store-copy)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
@@ -42,8 +41,7 @@
sqlite-register
register-path
register-items
- %epoch
- reset-timestamps))
+ %epoch))
;;; Code for working with the store database directly.
@@ -227,39 +225,6 @@ Every store item in REFERENCES must already be registered."
;;;
;;; High-level interface.
;;;
-
-(define* (reset-timestamps file #:key preserve-permissions?)
- "Reset the modification time on FILE and on all the files it contains, if
-it's a directory. Canonicalize file permissions unless PRESERVE-PERMISSIONS?
-is true."
- ;; Note: We're resetting to one second after the Epoch like 'guix-daemon'
- ;; has always done.
- (let loop ((file file)
- (type (stat:type (lstat file))))
- (case type
- ((directory)
- (unless preserve-permissions?
- (chmod file #o555))
- (utime file 1 1 0 0)
- (let ((parent file))
- (for-each (match-lambda
- (("." . _) #f)
- ((".." . _) #f)
- ((file . properties)
- (let ((file (string-append parent "/" file)))
- (loop file
- (match (assoc-ref properties 'type)
- ((or 'unknown #f)
- (stat:type (lstat file)))
- (type type))))))
- (scandir* parent))))
- ((symlink)
- (utime file 1 1 0 0 AT_SYMLINK_NOFOLLOW))
- (else
- (unless preserve-permissions?
- (chmod file (if (executable-file? file) #o555 #o444)))
- (utime file 1 1 0 0)))))
-
(define* (register-path path
#:key (references '()) deriver prefix
state-directory (deduplicate? #t)
diff --git a/guix/utils.scm b/guix/utils.scm
index d7b197fa44..812617dd61 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -35,8 +35,10 @@
#:use-module (rnrs io ports) ;need 'port-position' etc.
#:use-module ((rnrs bytevectors) #:select (bytevector-u8-set!))
#:use-module (guix memoization)
- #:use-module ((guix build utils) #:select (dump-port mkdir-p delete-file-recursively))
- #:use-module ((guix build syscalls) #:select (mkdtemp! fdatasync))
+ #:use-module ((guix build utils)
+ #:select (dump-port mkdir-p delete-file-recursively
+ executable-file?))
+ #:use-module ((guix build syscalls) #:select (mkdtemp! fdatasync scandir*))
#:use-module (ice-9 format)
#:use-module (ice-9 regex)
#:use-module (ice-9 match)
@@ -109,7 +111,8 @@
call-with-decompressed-port
compressed-output-port
call-with-compressed-output-port
- canonical-newline-port))
+ canonical-newline-port
+ reset-timestamps))
;;;
@@ -843,6 +846,38 @@ a location object."
fix-hint?
(hint condition-fix-hint)) ;string
+(define* (reset-timestamps file #:key preserve-permissions?)
+ "Reset the modification time on FILE and on all the files it contains, if
+it's a directory. Canonicalize file permissions unless PRESERVE-PERMISSIONS?
+is true."
+ ;; Note: We're resetting to one second after the Epoch like 'guix-daemon'
+ ;; has always done.
+ (let loop ((file file)
+ (type (stat:type (lstat file))))
+ (case type
+ ((directory)
+ (unless preserve-permissions?
+ (chmod file #o555))
+ (utime file 1 1 0 0)
+ (let ((parent file))
+ (for-each (match-lambda
+ (("." . _) #f)
+ ((".." . _) #f)
+ ((file . properties)
+ (let ((file (string-append parent "/" file)))
+ (loop file
+ (match (assoc-ref properties 'type)
+ ((or 'unknown #f)
+ (stat:type (lstat file)))
+ (type type))))))
+ (scandir* parent))))
+ ((symlink)
+ (utime file 1 1 0 0 AT_SYMLINK_NOFOLLOW))
+ (else
+ (unless preserve-permissions?
+ (chmod file (if (executable-file? file) #o555 #o444)))
+ (utime file 1 1 0 0)))))
+
;;; Local Variables:
;;; eval: (put 'call-with-progress-reporter 'scheme-indent-function 1)
;;; End:
--
2.26.0
J
J
Jan (janneke) Nieuwenhuizen wrote on 17 May 2020 12:03
[PATCH 2/3] system: vm: Do not register-closures when cross-building.
(address . 41350@debbugs.gnu.org)
20200517100343.26361-2-janneke@gnu.org
This supports cross-building building a vm-image, running a native qemu.

* gnu/system/vm.scm (qemu-image)[register-closures?]: Default to #f when
cross-compiling. Only create sql-schema when actually registering closures.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 8 +++++---
2 files changed, 6 insertions(+), 4 deletions(-)

Toggle diff (45 lines)
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index c751e6b0e2..2b36dd29ce 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -27,7 +27,7 @@
#:use-module (guix build store-copy)
#:use-module (guix build syscalls)
#:use-module ((guix utils) #:select (reset-timestamps))
- #:use-module (guix store database)
+ #:autoload (guix store database) (%epoch register-items sql-schema)
#:use-module (gnu build bootloader)
#:use-module (gnu build linux-boot)
#:use-module (gnu build install)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 3e483fd86c..97b0bf461b 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -286,7 +286,8 @@ substitutable."
os
bootcfg-drv
bootloader
- (register-closures? (has-guix-service-type? os))
+ (register-closures? (and (has-guix-service-type? os)
+ (not target)))
(inputs '())
copy-inputs?
(substitutable? #t))
@@ -333,12 +334,13 @@ system that is passed to 'populate-root-file-system'."
((gnu build linux-boot)
#:select (make-essential-device-nodes
make-hurd-device-nodes))
- (guix store database)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))
- (sql-schema #$schema)
+ (when #$register-closures?
+ (use-modules (guix store database))
+ (sql-schema #$schema))
;; Allow non-ASCII file names--e.g., 'nss-certs'--to be decoded.
(setenv "GUIX_LOCPATH"
--
2.26.0
J
J
Jan (janneke) Nieuwenhuizen wrote on 17 May 2020 12:03
[PATCH 3/3] system: vm: Build vm-image using native qemu.
(address . 41350@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200517100343.26361-3-janneke@gnu.org
Cross-building a vm-image used to be done using a cross-qemu, e.g, qemu-ARM.
That does not work for the Hurd, as there is no qemu-HURD.

This patch switches to cross building vm-images using a native qemu vm.

* gnu/system/vm.scm (expression->derivation-in-linux-vm): Run native
qemu-command; use native linux, initrd, bootloader-package and
bootloader-installer.
[preserve-target]: New helper to install cross-packages into the native vm.
* gnu/bootloader/grub.scm (eye-candy): Use native font.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
---
gnu/bootloader/grub.scm | 4 ++--
gnu/system/vm.scm | 46 ++++++++++++++++++++++++++---------------
2 files changed, 31 insertions(+), 19 deletions(-)

Toggle diff (117 lines)
diff --git a/gnu/bootloader/grub.scm b/gnu/bootloader/grub.scm
index 8c5b5eac0c..842592ccc9 100644
--- a/gnu/bootloader/grub.scm
+++ b/gnu/bootloader/grub.scm
@@ -211,8 +211,8 @@ else
set menu_color_highlight=white/blue
fi~%"
#$setup-gfxterm-body
- #$(grub-root-search store-device font-file)
- #$(setup-gfxterm config font-file)
+ #+(grub-root-search store-device font-file)
+ #+(setup-gfxterm config font-file)
#$(grub-setup-io config)
#$(strip-mount-point store-mount-point image)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 97b0bf461b..aa094b0a06 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -180,19 +181,30 @@ made available under the /xchg CIFS share.
SUBSTITUTABLE? determines whether the returned derivation should be marked as
substitutable."
(define user-builder
- (program-file "builder-in-linux-vm" exp))
+ (scheme-file "builder-in-linux-vm" exp))
+
+ (define (preserve-target obj)
+ (if target
+ (with-parameters ((%current-target-system target))
+ obj)
+ obj))
(define loader
- ;; Invoke USER-BUILDER instead using 'primitive-load'. The reason for
- ;; this is to allow USER-BUILDER to dlopen stuff by using a full-featured
- ;; Guile, which it couldn't do using the statically-linked guile used in
- ;; the initrd. See example at
+ ;; Instead of using 'primitive-load', evaluate USER-BUILDER in a
+ ;; full-featured Guile so it can use dlopen stuff, which it couldn't do
+ ;; using the statically-linked guile used in the initrd. See example at
;; <https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00233.html>.
(program-file "linux-vm-loader"
- ;; Communicate USER-BUILDER's exit status via /xchg so that
- ;; the host can distinguish between success, failure, and
- ;; kernel panic.
- #~(let ((status (system* #$user-builder)))
+ ;; When cross-compiling, USER-BUILDER refers to the target
+ ;; (cross-compiled) system. Preserve that, even though
+ ;; LOADER itself is executed as a native program.
+ #~(let ((status (system* #+(file-append (default-guile)
+ "/bin/guile")
+ "--no-auto-compile"
+ #$(preserve-target user-builder))))
+ ;; Communicate USER-BUILDER's exit status via /xchg so
+ ;; that the host can distinguish between success,
+ ;; failure, and kernel panic.
(call-with-output-file "/xchg/.exit-status"
(lambda (port)
(write status port)))
@@ -226,10 +238,10 @@ substitutable."
(let* ((native-inputs
'#+(list qemu (canonical-package coreutils)))
- (linux (string-append #$linux "/"
- #$(system-linux-image-file-name)))
- (initrd #$initrd)
- (loader #$loader)
+ (linux (string-append #+linux "/"
+ #+(system-linux-image-file-name)))
+ (initrd #+initrd)
+ (loader #+loader)
(graphs '#$(match references-graphs
(((graph-files . _) ...) graph-files)
(_ #f)))
@@ -245,7 +257,7 @@ substitutable."
(load-in-linux-vm loader
#:output #$output
#:linux linux #:initrd initrd
- #:qemu (qemu-command target)
+ #:qemu (qemu-command)
#:memory-size #$memory-size
#:make-disk-image? #$make-disk-image?
#:single-file-output? #$single-file-output?
@@ -348,7 +360,7 @@ system that is passed to 'populate-root-file-system'."
(setlocale LC_ALL "en_US.utf8")
(let ((inputs
- '#$(append (list parted e2fsprogs dosfstools)
+ '#+(append (list parted e2fsprogs dosfstools)
(map canonical-package
(list sed grep coreutils findutils gawk))))
@@ -425,12 +437,12 @@ system that is passed to 'populate-root-file-system'."
#:partitions partitions
#:grub-efi grub-efi
#:bootloader-package
- #$(bootloader-package bootloader)
+ #+(bootloader-package bootloader)
#:bootcfg #$bootcfg-drv
#:bootcfg-location
#$(bootloader-configuration-file bootloader)
#:bootloader-installer
- #$(bootloader-installer bootloader)))))))
+ #+(bootloader-installer bootloader)))))))
#:system system
#:target target
#:make-disk-image? #t
--
2.26.0
M
M
Mathieu Othacehe wrote on 18 May 2020 11:10
Re: [bug#41350] [PATCH 0/3] Use native qemu to build vm-image.
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 41350@debbugs.gnu.org)
87y2ppfw28.fsf@gnu.org
Hello Jan,

Toggle quote (3 lines)
> Cross-building a vm-image used to be done using a cross-qemu, e.g, qemu-ARM.
> That does not work for the Hurd, as there is no qemu-HURD.

Yes that's an issue.

Toggle quote (5 lines)
>
> This patch switches to cross building vm-images using a native qemu vm. If
> there a reason for using qemu-TARGET we may want to make this switch
> conditional for cross building to the Hurd?

Well first this 'vm-image' thing should be done in (gnu system image) as
soon as I have sorted out the bootloader issue. So the solution we will
find to overcome this Hurd issue will be temporary I hope.

About using qemu-TARGET, let say you are on an armhf machine and you
want to cross-build an x86-64 image. I fear that using a native, armhf
Grub to install an x86-64 Grub won't work.

So maybe, it would be better to do that only for the architectures
without an available qemu-TARGET (only the Hurd for now)?

Toggle quote (4 lines)
> ...the problem is now avoided by removing the sqlite dependency when
> cross-building by not registering closures and postponing the loading of (gnu
> store database) and thus (sqlite3).

When I produce a system without register closures, once booted, "guix
build" something does not work. I don't know if its possible for
guix-daemon to work without its database. An option could be to generate
this database if its absent, at first boot?

Toggle quote (5 lines)
> ./pre-inst-env guix system vm-image --target=i586-pc-gnu --no-grafts \
> gnu/system/examples/bare-hurd.tmpl
>
> now produces a pretty nice hurd VM :-)

Great to see you so close :)

Thanks,

Mathieu
J
J
Jan Nieuwenhuizen wrote on 19 May 2020 09:22
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 41350@debbugs.gnu.org)
87367wbd82.fsf@gnu.org
Mathieu Othacehe writes:

Hello Mathieu,

Thanks for chiming in, I meant to CC you... :-)

Toggle quote (5 lines)
>> Cross-building a vm-image used to be done using a cross-qemu, e.g, qemu-ARM.
>> That does not work for the Hurd, as there is no qemu-HURD.
>
> Yes that's an issue.

Yes...luckily I think we found a way to use qemu-NATIVE to build
packages for the Hurd. Now to see if this can be put into digestible
form.

Toggle quote (8 lines)
>> This patch switches to cross building vm-images using a native qemu vm. If
>> there a reason for using qemu-TARGET we may want to make this switch
>> conditional for cross building to the Hurd?
>
> Well first this 'vm-image' thing should be done in (gnu system image) as
> soon as I have sorted out the bootloader issue. So the solution we will
> find to overcome this Hurd issue will be temporary I hope.

Can you elaborate a bit on that? Do you think it makes sense to continue
this temporary path some more (as it starts to work right now), or can we
better abandon it and work the permanent solution? How could I contribute?

Toggle quote (4 lines)
> About using qemu-TARGET, let say you are on an armhf machine and you
> want to cross-build an x86-64 image. I fear that using a native, armhf
> Grub to install an x86-64 Grub won't work.

Ah...right, that's helpful information.

Toggle quote (3 lines)
> So maybe, it would be better to do that only for the architectures
> without an available qemu-TARGET (only the Hurd for now)?

OK -- I have changed patch 2 and 3, and am sending a new patch set.

Toggle quote (9 lines)
>> ...the problem is now avoided by removing the sqlite dependency when
>> cross-building by not registering closures and postponing the loading of (gnu
>> store database) and thus (sqlite3).
>
> When I produce a system without register closures, once booted, "guix
> build" something does not work. I don't know if its possible for
> guix-daemon to work without its database. An option could be to generate
> this database if its absent, at first boot?

Ah, OK. I'm using this "solution" to use #:register-closures? #f now
for the Hurd only and we'll hit the problem that causes later. We can
then, s discussed on IRC, sqlite3 databases are indeed platform
independent and we could use something like guix/scripts/pack.scm's
store-database. WDYT?

Toggle quote (7 lines)
>> ./pre-inst-env guix system vm-image --target=i586-pc-gnu --no-grafts \
>> gnu/system/examples/bare-hurd.tmpl
>>
>> now produces a pretty nice hurd VM :-)
>
> Great to see you so close :)

Yes...

Greetings,
janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 19 May 2020 09:23
[PATCH v2 2/3] system: vm: Do not register-closures when cross-building to the Hurd.
20200519072302.9202-2-janneke@gnu.org
This supports cross-building building a vm-image for the Hurd, running a
native qemu.

* gnu/system/vm.scm (qemu-image)[register-closures?]: Default to #f when
cross-compiling to the Hurd. Only create sql-schema when actually registering
closures.
---
gnu/build/vm.scm | 2 +-
gnu/system/vm.scm | 9 ++++++---
2 files changed, 7 insertions(+), 4 deletions(-)

Toggle diff (53 lines)
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index c751e6b0e2..2b36dd29ce 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -27,7 +27,7 @@
#:use-module (guix build store-copy)
#:use-module (guix build syscalls)
#:use-module ((guix utils) #:select (reset-timestamps))
- #:use-module (guix store database)
+ #:autoload (guix store database) (%epoch register-items sql-schema)
#:use-module (gnu build bootloader)
#:use-module (gnu build linux-boot)
#:use-module (gnu build install)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 3e483fd86c..b343141c18 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -51,6 +51,7 @@
#:use-module (gnu packages zile)
#:use-module (gnu packages linux)
#:use-module (gnu packages admin)
+ #:use-module (gnu packages hurd)
#:use-module (gnu bootloader)
#:use-module (gnu bootloader grub)
@@ -286,7 +287,8 @@ substitutable."
os
bootcfg-drv
bootloader
- (register-closures? (has-guix-service-type? os))
+ (register-closures? (and (has-guix-service-type? os)
+ (not (hurd-target?))))
(inputs '())
copy-inputs?
(substitutable? #t))
@@ -333,12 +335,13 @@ system that is passed to 'populate-root-file-system'."
((gnu build linux-boot)
#:select (make-essential-device-nodes
make-hurd-device-nodes))
- (guix store database)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))
- (sql-schema #$schema)
+ (when #$register-closures?
+ (use-modules (guix store database))
+ (sql-schema #$schema))
;; Allow non-ASCII file names--e.g., 'nss-certs'--to be decoded.
(setenv "GUIX_LOCPATH"
--
2.26.2
J
J
Jan (janneke) Nieuwenhuizen wrote on 19 May 2020 09:23
[PATCH v2 1/3] utils: Move 'reset-timestamps' out of database.
20200519072302.9202-1-janneke@gnu.org
This supports calling reset-timestamps without loading sqlite3.

* guix/store/database.scm (reset-timestamps): Move to...
* guix/utils.scm (reset-timestamps): ... here.
* gnu/build/vm.scm: Include it.
---
gnu/build/vm.scm | 1 +
guix/store/database.scm | 41 +++--------------------------------------
guix/utils.scm | 41 ++++++++++++++++++++++++++++++++++++++---
3 files changed, 42 insertions(+), 41 deletions(-)

Toggle diff (146 lines)
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 433b5a7e8d..c751e6b0e2 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -26,6 +26,7 @@
#:use-module (guix build utils)
#:use-module (guix build store-copy)
#:use-module (guix build syscalls)
+ #:use-module ((guix utils) #:select (reset-timestamps))
#:use-module (guix store database)
#:use-module (gnu build bootloader)
#:use-module (gnu build linux-boot)
diff --git a/guix/store/database.scm b/guix/store/database.scm
index ef52036ede..b8fe313c3d 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -24,9 +24,8 @@
#:use-module (guix store deduplication)
#:use-module (guix base16)
#:use-module (guix progress)
- #:use-module (guix build syscalls)
- #:use-module ((guix build utils)
- #:select (mkdir-p executable-file?))
+ #:use-module ((guix build utils) #:select (mkdir-p))
+ #:use-module ((guix utils) #:select (reset-timestamps))
#:use-module (guix build store-copy)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
@@ -42,8 +41,7 @@
sqlite-register
register-path
register-items
- %epoch
- reset-timestamps))
+ %epoch))
;;; Code for working with the store database directly.
@@ -227,39 +225,6 @@ Every store item in REFERENCES must already be registered."
;;;
;;; High-level interface.
;;;
-
-(define* (reset-timestamps file #:key preserve-permissions?)
- "Reset the modification time on FILE and on all the files it contains, if
-it's a directory. Canonicalize file permissions unless PRESERVE-PERMISSIONS?
-is true."
- ;; Note: We're resetting to one second after the Epoch like 'guix-daemon'
- ;; has always done.
- (let loop ((file file)
- (type (stat:type (lstat file))))
- (case type
- ((directory)
- (unless preserve-permissions?
- (chmod file #o555))
- (utime file 1 1 0 0)
- (let ((parent file))
- (for-each (match-lambda
- (("." . _) #f)
- ((".." . _) #f)
- ((file . properties)
- (let ((file (string-append parent "/" file)))
- (loop file
- (match (assoc-ref properties 'type)
- ((or 'unknown #f)
- (stat:type (lstat file)))
- (type type))))))
- (scandir* parent))))
- ((symlink)
- (utime file 1 1 0 0 AT_SYMLINK_NOFOLLOW))
- (else
- (unless preserve-permissions?
- (chmod file (if (executable-file? file) #o555 #o444)))
- (utime file 1 1 0 0)))))
-
(define* (register-path path
#:key (references '()) deriver prefix
state-directory (deduplicate? #t)
diff --git a/guix/utils.scm b/guix/utils.scm
index d7b197fa44..812617dd61 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -35,8 +35,10 @@
#:use-module (rnrs io ports) ;need 'port-position' etc.
#:use-module ((rnrs bytevectors) #:select (bytevector-u8-set!))
#:use-module (guix memoization)
- #:use-module ((guix build utils) #:select (dump-port mkdir-p delete-file-recursively))
- #:use-module ((guix build syscalls) #:select (mkdtemp! fdatasync))
+ #:use-module ((guix build utils)
+ #:select (dump-port mkdir-p delete-file-recursively
+ executable-file?))
+ #:use-module ((guix build syscalls) #:select (mkdtemp! fdatasync scandir*))
#:use-module (ice-9 format)
#:use-module (ice-9 regex)
#:use-module (ice-9 match)
@@ -109,7 +111,8 @@
call-with-decompressed-port
compressed-output-port
call-with-compressed-output-port
- canonical-newline-port))
+ canonical-newline-port
+ reset-timestamps))
;;;
@@ -843,6 +846,38 @@ a location object."
fix-hint?
(hint condition-fix-hint)) ;string
+(define* (reset-timestamps file #:key preserve-permissions?)
+ "Reset the modification time on FILE and on all the files it contains, if
+it's a directory. Canonicalize file permissions unless PRESERVE-PERMISSIONS?
+is true."
+ ;; Note: We're resetting to one second after the Epoch like 'guix-daemon'
+ ;; has always done.
+ (let loop ((file file)
+ (type (stat:type (lstat file))))
+ (case type
+ ((directory)
+ (unless preserve-permissions?
+ (chmod file #o555))
+ (utime file 1 1 0 0)
+ (let ((parent file))
+ (for-each (match-lambda
+ (("." . _) #f)
+ ((".." . _) #f)
+ ((file . properties)
+ (let ((file (string-append parent "/" file)))
+ (loop file
+ (match (assoc-ref properties 'type)
+ ((or 'unknown #f)
+ (stat:type (lstat file)))
+ (type type))))))
+ (scandir* parent))))
+ ((symlink)
+ (utime file 1 1 0 0 AT_SYMLINK_NOFOLLOW))
+ (else
+ (unless preserve-permissions?
+ (chmod file (if (executable-file? file) #o555 #o444)))
+ (utime file 1 1 0 0)))))
+
;;; Local Variables:
;;; eval: (put 'call-with-progress-reporter 'scheme-indent-function 1)
;;; End:
--
2.26.2
J
J
Jan (janneke) Nieuwenhuizen wrote on 19 May 2020 09:23
[PATCH v2 3/3] system: vm: Build vm-image using native qemu, for the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)
20200519072302.9202-3-janneke@gnu.org
Cross-building a vm-image is usually done using a cross-qemu, e.g, qemu-ARM,
because, e.g., a native, x86_64 Grub cannot install an armhf-Grub. That
solution does not work for the Hurd, as there is no qemu-HURD.

This patch enables cross building vm-images for the Hurd using a native qemu
vm.

* gnu/system/vm.scm (expression->derivation-in-linux-vm): Run native
qemu-command; use native linux, initrd, bootloader-package and
bootloader-installer, for the Hurd.
[preserve-target]: New helper to install cross-packages into the native vm.
* gnu/bootloader/grub.scm (eye-candy): Use native font, for the Hurd.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
---
gnu/bootloader/grub.scm | 9 +++--
gnu/system/vm.scm | 78 +++++++++++++++++++++++++++++++----------
2 files changed, 66 insertions(+), 21 deletions(-)

Toggle diff (169 lines)
diff --git a/gnu/bootloader/grub.scm b/gnu/bootloader/grub.scm
index 8c5b5eac0c..cee4c9f6c5 100644
--- a/gnu/bootloader/grub.scm
+++ b/gnu/bootloader/grub.scm
@@ -31,6 +31,7 @@
#:use-module (gnu system file-systems)
#:use-module (gnu system keyboard)
#:use-module (gnu packages bootloaders)
+ #:use-module (gnu packages hurd)
#:autoload (gnu packages gtk) (guile-cairo guile-rsvg)
#:autoload (gnu packages xorg) (xkeyboard-config)
#:use-module (ice-9 match)
@@ -211,8 +212,12 @@ else
set menu_color_highlight=white/blue
fi~%"
#$setup-gfxterm-body
- #$(grub-root-search store-device font-file)
- #$(setup-gfxterm config font-file)
+ #$(if (hurd-target?)
+ #~#+(grub-root-search store-device font-file)
+ (grub-root-search store-device font-file))
+ #$(if (hurd-target?)
+ #~#+(setup-gfxterm config font-file)
+ (setup-gfxterm config font-file))
#$(grub-setup-io config)
#$(strip-mount-point store-mount-point image)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b343141c18..527596421d 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -181,19 +182,36 @@ made available under the /xchg CIFS share.
SUBSTITUTABLE? determines whether the returned derivation should be marked as
substitutable."
(define user-builder
- (program-file "builder-in-linux-vm" exp))
+ (scheme-file "builder-in-linux-vm" exp))
+
+ (define (preserve-target obj)
+ (if target
+ (with-parameters ((%current-target-system target))
+ obj)
+ obj))
(define loader
- ;; Invoke USER-BUILDER instead using 'primitive-load'. The reason for
- ;; this is to allow USER-BUILDER to dlopen stuff by using a full-featured
- ;; Guile, which it couldn't do using the statically-linked guile used in
- ;; the initrd. See example at
+ ;; Instead of using 'primitive-load', evaluate USER-BUILDER in a
+ ;; full-featured Guile so it can use dlopen stuff, which it couldn't do
+ ;; using the statically-linked guile used in the initrd. See example at
;; <https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00233.html>.
(program-file "linux-vm-loader"
- ;; Communicate USER-BUILDER's exit status via /xchg so that
- ;; the host can distinguish between success, failure, and
- ;; kernel panic.
- #~(let ((status (system* #$user-builder)))
+ ;; When cross-compiling, USER-BUILDER refers to the target
+ ;; (cross-compiled) system. Preserve that, even though
+ ;; LOADER itself is executed as a native program.
+ #~(let* ((guile #$(if (hurd-target?)
+ #~#+(file-append (default-guile)
+ "/bin/guile")
+ (file-append (default-guile)
+ "/bin/guile")))
+ (status (system* guile "--no-auto-compile"
+ #$(if (hurd-target?)
+ (preserve-target user-builder)
+ user-builder))))
+
+ ;; Communicate USER-BUILDER's exit status via /xchg so
+ ;; that the host can distinguish between success,
+ ;; failure, and kernel panic.
(call-with-output-file "/xchg/.exit-status"
(lambda (port)
(write status port)))
@@ -227,10 +245,16 @@ substitutable."
(let* ((native-inputs
'#+(list qemu (canonical-package coreutils)))
- (linux (string-append #$linux "/"
- #$(system-linux-image-file-name)))
- (initrd #$initrd)
- (loader #$loader)
+
+ (loader #$(if (hurd-target?) #~#+loader loader))
+ (linux #$(if (hurd-target?)
+ #~(string-append
+ #+linux "/"
+ #+(system-linux-image-file-name))
+ #~(string-append
+ #$linux "/"
+ #$(system-linux-image-file-name))))
+ (initrd #$(if (hurd-target?) #~#+initrd initrd))
(graphs '#$(match references-graphs
(((graph-files . _) ...) graph-files)
(_ #f)))
@@ -246,7 +270,9 @@ substitutable."
(load-in-linux-vm loader
#:output #$output
#:linux linux #:initrd initrd
- #:qemu (qemu-command target)
+ #:qemu #$(if (hurd-target?)
+ (qemu-command)
+ (qemu-command target))
#:memory-size #$memory-size
#:make-disk-image? #$make-disk-image?
#:single-file-output? #$single-file-output?
@@ -288,7 +314,9 @@ substitutable."
bootcfg-drv
bootloader
(register-closures? (and (has-guix-service-type? os)
- (not (hurd-target?))))
+ (not target)
+ ;(not (hurd-target?))
+ ))
(inputs '())
copy-inputs?
(substitutable? #t))
@@ -349,9 +377,17 @@ system that is passed to 'populate-root-file-system'."
(setlocale LC_ALL "en_US.utf8")
(let ((inputs
- '#$(append (list parted e2fsprogs dosfstools)
+ '#+(append (list parted e2fsprogs dosfstools)
(map canonical-package
- (list sed grep coreutils findutils gawk))))
+ (list sed grep coreutils findutils gawk)))
+ ;; (if #$(hurd-target?)
+ ;; '#+(append (list parted e2fsprogs dosfstools)
+ ;; (map canonical-package
+ ;; (list sed grep coreutils findutils gawk)))
+ ;; '#$(append (list parted e2fsprogs dosfstools)
+ ;; (map canonical-package
+ ;; (list sed grep coreutils findutils gawk))))
+ )
;; This variable is unused but allows us to add INPUTS-TO-COPY
;; as inputs.
@@ -426,12 +462,16 @@ system that is passed to 'populate-root-file-system'."
#:partitions partitions
#:grub-efi grub-efi
#:bootloader-package
- #$(bootloader-package bootloader)
+ (if #$(hurd-target?)
+ #+(bootloader-package bootloader)
+ #$(bootloader-package bootloader))
#:bootcfg #$bootcfg-drv
#:bootcfg-location
#$(bootloader-configuration-file bootloader)
#:bootloader-installer
- #$(bootloader-installer bootloader)))))))
+ (if #$(hurd-target?)
+ #+(bootloader-installer bootloader)
+ #$(bootloader-installer bootloader))))))))
#:system system
#:target target
#:make-disk-image? #t
--
2.26.2
M
M
Mathieu Othacehe wrote on 19 May 2020 11:14
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)
87blmkthg0.fsf@gnu.org
Hello Janneke,

Toggle quote (7 lines)
> + #$(if (hurd-target?)
> + #~#+(grub-root-search store-device font-file)
> + (grub-root-search store-device font-file))
> + #$(if (hurd-target?)
> + #~#+(setup-gfxterm config font-file)
> + (setup-gfxterm config font-file))

I'm not certain that hurd-target? always return the expected answer
here. Now that we have let-system, the safer way to check for the target
system could be to write:

Toggle snippet (4 lines)
#$(let-system (system target)
(hurd-target? system target))

Ludo, is this correct? Furthermore, if you make sure that
"grub-root-search" and "setup-gfxterm" return #+, you could maybe drop
this part?

Toggle quote (10 lines)
> +
> + (loader #$(if (hurd-target?) #~#+loader loader))
> + (linux #$(if (hurd-target?)
> + #~(string-append
> + #+linux "/"
> + #+(system-linux-image-file-name))
> + #~(string-append
> + #$linux "/"
> + #$(system-linux-image-file-name))))

Same concern as above about "hurd-target?".

Toggle quote (9 lines)
> + ;; (if #$(hurd-target?)
> + ;; '#+(append (list parted e2fsprogs dosfstools)
> + ;; (map canonical-package
> + ;; (list sed grep coreutils findutils gawk)))
> + ;; '#$(append (list parted e2fsprogs dosfstools)
> + ;; (map canonical-package
> + ;; (list sed grep coreutils findutils gawk))))
> + )

Is this needed?

Thanks,

Mathieu
M
M
Mathieu Othacehe wrote on 19 May 2020 12:02
Re: [bug#41350] [PATCH 0/3] Use native qemu to build vm-image.
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 41350@debbugs.gnu.org)
874ksctf85.fsf@gnu.org
Hello Jan,

Toggle quote (8 lines)
>> Well first this 'vm-image' thing should be done in (gnu system image) as
>> soon as I have sorted out the bootloader issue. So the solution we will
>> find to overcome this Hurd issue will be temporary I hope.
>
> Can you elaborate a bit on that? Do you think it makes sense to continue
> this temporary path some more (as it starts to work right now), or can we
> better abandon it and work the permanent solution? How could I contribute?

Yes. Besides offering a more modular image creation API, the vocation of
(gnu system image) is to offer a way to generate all kind of Guix system
images (raw disk-images, ISO9660 images, Qemu images) without resorting
to VM for image creation.

Dropping VM, means that the image need to be build on the host, without
root permissions. This brings several limitations. If is no longer
possible to use "mount" for instance, or to call "grub-install".

To get around these limitations, I used the same strategy as Buildroot,
Yocto and OpenWrt that do not require root permissions to generate
disk-images.

For ISO9660 images:

* I first build the "image-root" derivation. It's a store directory that
contains the root file-system.

* Then, I call make-iso9660-image that, run GNU Xorriso on this
directory.

For raw disk-images:

* For each partition, I build the "partition-image-root"
derivation. This is very similar to "image-root" but for the specified
partition.

* For each partition root directory, I create the corresponding
partition image, using tools such as mke2fs or mkdosfs depending on
the partition file-system type.
* Then, I need to "assemble" the partitions in a disk-image. For that, I
use "genimage" that will roughly use 'dd' to create a final disk-image
with the partitions copied at the right offsets.
* The missing part here is the bootloader installation. As I mentioned,
grub-install refuses to take a disk-image as argument (it requires a
mounted partition). For EFI systems there's a work-around. The idea is
to call grub-mkstandalone to create a Grub binary in the ESP
partition. This Grub is configured to load the Grub configuration file
located on the root file-system at /boot/grub/grub.cfg path.

Now back to the Hurd. I see that Debian is producing Hurd ISO images. We
could try to call `guix system disk-image --target=i586-pc-gnu
--file-system-type=iso9660 hurd.scm` and see if it works.

Regarding raw disk-image, I think we could try to produce EFI compatible
Hurd images. We could set the bootloader to grub-efi-bootloader in
%hurd-default-operating-system.

Finally, to produce raw disk-images with grub-minimal-bootloader or
grub-bootloader (what you are trying to do), we need to find a way to
make grub-install work on disk-images (MBR installation and so on). That
true for the Hurd but that's also true for Linux.

Sorry for the long explanation. Please tell me if something is not
clear.

The conclusion here is that, I think that we very few adaptations to
your branch, we should be able to produce Hurd ISO images or Hurd EFI
compatible disk-images. Maybe it would be a first step.

Then, we could find a way to create "MBR compatible" Hurd and Linux
disk-images in (gnu system image).

WDYT?

Thanks,

Mathieu
M
M
Mathieu Othacehe wrote on 20 May 2020 16:03
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 41350@debbugs.gnu.org)
87blmivh36.fsf@gnu.org
Hey,

Toggle quote (7 lines)
> The conclusion here is that, I think that we very few adaptations to
> your branch, we should be able to produce Hurd ISO images or Hurd EFI
> compatible disk-images. Maybe it would be a first step.
>
> Then, we could find a way to create "MBR compatible" Hurd and Linux
> disk-images in (gnu system image).

Ok, I made further progress. Turns out the Hurd EFI solution was a
dead-end, because I have a "no console will be available to os" message
in Qemu, instead of Hurd console output. I don't feel like debugging
this.

So, back to the MBR solution. I had another look to what OpenWrt is
doing. They found a really nice work-around! As I stated, we cannot use
"grub-install", but this command is in reality a wrapper around
"grub-mkimage" and "grub-bios-setup".

Toggle snippet (7 lines)
#######################################
# # # #
# MBR # MBR-GAP # FIRST PARTITION #
# # # #
#######################################

grub-mkimage generates a Grub image, small enough so that it can fit in
the MBR-GAP (the space between the MBR and the first partition).

It cannot contain all Grub modules, but that's not an issue, because
Grub will be able to find missing modules from the first partition when
started.

They also patched grub-bios-setup[1] so that it can work on a raw
disk-image, and install the previously generated "grub.img".

So with the following commands:

Toggle snippet (5 lines)
grub-mkimage -O i386-pc -o core.img biosdisk part_msdos ext2
echo "(hd0) /tmp/my-disk-image" > device.map
grub-bios-setup -m device.map -r "hd0,msdos1" -d tmp /tmp/qemu-image2

I'm able to make a Guix system image bootable, without root
permissions. It then starts the Hurd kernel and crashes, but that's
another story :p

So, if it's ok for you, I can integrate this stuff cleanly, and we won't
need the vm-image stuff anymore.

WDYT?

Thanks,

Mathieu

[1]:
J
J
Jan Nieuwenhuizen wrote on 20 May 2020 17:09
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87lflm4p9o.fsf@gnu.org
Mathieu Othacehe writes:

Hello Mathieu!

Toggle quote (12 lines)
>> The conclusion here is that, I think that we very few adaptations to
>> your branch, we should be able to produce Hurd ISO images or Hurd EFI
>> compatible disk-images. Maybe it would be a first step.
>>
>> Then, we could find a way to create "MBR compatible" Hurd and Linux
>> disk-images in (gnu system image).
>
> Ok, I made further progress. Turns out the Hurd EFI solution was a
> dead-end, because I have a "no console will be available to os" message
> in Qemu, instead of Hurd console output. I don't feel like debugging
> this.

Ah, well that's the only result that I also came up with and wanted to
share. Maybe Ludo has an idea how to fix that; I could not find
anything helpful yet.

Toggle quote (5 lines)
> So, back to the MBR solution. I had another look to what OpenWrt is
> doing. They found a really nice work-around! As I stated, we cannot use
> "grub-install", but this command is in reality a wrapper around
> "grub-mkimage" and "grub-bios-setup".

Ah! That's good to know.

Toggle quote (10 lines)
> #######################################
> # # # #
> # MBR # MBR-GAP # FIRST PARTITION #
> # # # #
> #######################################
>
>
> grub-mkimage generates a Grub image, small enough so that it can fit in
> the MBR-GAP (the space between the MBR and the first partition).

Okay...

Toggle quote (4 lines)
> It cannot contain all Grub modules, but that's not an issue, because
> Grub will be able to find missing modules from the first partition when
> started.

aha...

Toggle quote (3 lines)
> They also patched grub-bios-setup[1] so that it can work on a raw
> disk-image, and install the previously generated "grub.img".

So, this is one aspect of what we're avoiding by running grub-install
in a qemu vm, right? Any idea on the upstream status of this patch?
(I don't think it matters all that much for us, let's use it :-)

Toggle quote (10 lines)
> So with the following commands:
>
> grub-mkimage -O i386-pc -o core.img biosdisk part_msdos ext2
> echo "(hd0) /tmp/my-disk-image" > device.map
> grub-bios-setup -m device.map -r "hd0,msdos1" -d tmp /tmp/qemu-image2
>
> I'm able to make a Guix system image bootable, without root
> permissions. It then starts the Hurd kernel and crashes, but that's
> another story :p

Thats *great*, no really!

Toggle quote (5 lines)
> So, if it's ok for you, I can integrate this stuff cleanly, and we won't
> need the vm-image stuff anymore.
>
> WDYT?

That seems a very good idea, I'm cc'ing Ludo to ping him about this.

I'm attaching some hacky work that I did on the patch that you paste'd
to me via IRC, adding som Hurd'y things. Maybe the pointers are
helpful, but it's all pretty obvious.

Greetings,
Janneke

Toggle quote (2 lines)
> [1]:
> https://github.com/openwrt/openwrt/blob/master/package/boot/grub2/patches/100-grub_setup_root.patch
Attachment: 0001-WIP-disk-image.patch
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 20 May 2020 23:49
Re: [bug#41350] [PATCH v2 3/3] system: vm: Build vm-image using native qemu, for the Hurd.
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87ftbue0pr.fsf@gnu.org
Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

Toggle quote (16 lines)
>> + #$(if (hurd-target?)
>> + #~#+(grub-root-search store-device font-file)
>> + (grub-root-search store-device font-file))
>> + #$(if (hurd-target?)
>> + #~#+(setup-gfxterm config font-file)
>> + (setup-gfxterm config font-file))
>
> I'm not certain that hurd-target? always return the expected answer
> here. Now that we have let-system, the safer way to check for the target
> system could be to write:
>
> #$(let-system (system target)
> (hurd-target? system target))
>
> Ludo, is this correct?

Definitely! Well, modulo indentation. ;-)

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 20 May 2020 23:58
Re: [PATCH v2 3/3] system: vm: Build vm-image using native qemu, for the Hurd.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)
87367ue09v.fsf@gnu.org
Howdy!

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (15 lines)
> Cross-building a vm-image is usually done using a cross-qemu, e.g, qemu-ARM,
> because, e.g., a native, x86_64 Grub cannot install an armhf-Grub. That
> solution does not work for the Hurd, as there is no qemu-HURD.
>
> This patch enables cross building vm-images for the Hurd using a native qemu
> vm.
>
> * gnu/system/vm.scm (expression->derivation-in-linux-vm): Run native
> qemu-command; use native linux, initrd, bootloader-package and
> bootloader-installer, for the Hurd.
> [preserve-target]: New helper to install cross-packages into the native vm.
> * gnu/bootloader/grub.scm (eye-candy): Use native font, for the Hurd.
>
> Co-authored-by: Ludovic Courtès <ludo@gnu.org>

I think the big question here, not Hurd-specific, is about the
abstractions:

1. When cross-compiling, can the ‘qemu-image’ procedure to its job by
running exclusively native software (in particular using a native
QEMU, native kernel, etc.)?

2. What should ‘expression->derivation-in-linux-vm’ do when
cross-compiling?

I think the answer to (1) is yes, right? That is, we can build a QEMU
image for AArch64/EFI from an x86_64 box. That means we’d run the
x86_64/EFI ‘grub-install’ in the final step, an x86_64 ‘mke2fs’ and
‘parted’ to create the partitions, and so on. Likewise, we can create a
QEMU GNU/Hurd image without entirely on GNU/Linux.

As for (2), I’d say that when cross-compiling, it should just run native
software but simply preserve references to cross-compiled software,
which is what janneke’s patch does.

How does that sound? Mathieu?

(I guess I’m rewording what janneke wrote, but my mind became fuzzy last
time I looked at it, so I thought it’s best to double-check. :-))

Thanks!

Ludo’.
M
M
Mathieu Othacehe wrote on 22 May 2020 21:24
Re: [bug#41350] [PATCH v2 3/3] system: vm: Build vm-image using native qemu, for the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)
87pnavu611.fsf@gnu.org
Hey Ludo!

Toggle quote (4 lines)
> 1. When cross-compiling, can the ‘qemu-image’ procedure to its job by
> running exclusively native software (in particular using a native
> QEMU, native kernel, etc.)?

I think the answer is yes, but I raised a concern about being able to
run grub-install from an ARM system to build a cross-compiled x86-64
system (for instance).

For now, running this command shows:

Toggle snippet (4 lines)
ls $(guix build --system=aarch64-linux grub)/lib/grub/
arm64-efi

So, the native aarch64-linux is only able to install itself on the same
system. I think can be fixed though (same as for grub-hybrid package).

Toggle quote (4 lines)
> As for (2), I’d say that when cross-compiling, it should just run native
> software but simply preserve references to cross-compiled software,
> which is what janneke’s patch does.

Yes, I agree.

However, I think I found a way to install Grub, without root
permissions, from the host system (see:

This should allow to deprecate the whole (gnu system vm) module.

Thanks for having a look :)

Mathieu
J
J
Jan Nieuwenhuizen wrote on 23 May 2020 11:28
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87367r57b4.fsf@gnu.org
Mathieu Othacehe writes:

Hello Mathieu,

Toggle quote (14 lines)
>> + #$(if (hurd-target?)
>> + #~#+(grub-root-search store-device font-file)
>> + (grub-root-search store-device font-file))
>> + #$(if (hurd-target?)
>> + #~#+(setup-gfxterm config font-file)
>> + (setup-gfxterm config font-file))
>
> I'm not certain that hurd-target? always return the expected answer
> here. Now that we have let-system, the safer way to check for the target
> system could be to write:
>
> #$(let-system (system target)
> (hurd-target? system target))

Okay...

Toggle quote (4 lines)
> Ludo, is this correct? Furthermore, if you make sure that
> "grub-root-search" and "setup-gfxterm" return #+, you could maybe drop
> this part?

...used #+ and dropped the IFs here, but used this insight below.

Toggle quote (12 lines)
>> +
>> + (loader #$(if (hurd-target?) #~#+loader loader))
>> + (linux #$(if (hurd-target?)
>> + #~(string-append
>> + #+linux "/"
>> + #+(system-linux-image-file-name))
>> + #~(string-append
>> + #$linux "/"
>> + #$(system-linux-image-file-name))))
>
> Same concern as above about "hurd-target?".

...Ah, that makes sense...that's of course why Ludo already introduced

(define-syntax-rule (check predicate)
(let-system (system target)
(predicate (or target system))))

here. So, using "(check hurd-triplet?)", as Ludo was already doing in
some other places here.

Toggle quote (11 lines)
>> + ;; (if #$(hurd-target?)
>> + ;; '#+(append (list parted e2fsprogs dosfstools)
>> + ;; (map canonical-package
>> + ;; (list sed grep coreutils findutils gawk)))
>> + ;; '#$(append (list parted e2fsprogs dosfstools)
>> + ;; (map canonical-package
>> + ;; (list sed grep coreutils findutils gawk))))
>> + )
>
> Is this needed?

Oops -- that was unfinished business; that IF did not work yet; possibly
because of not going through LET-SYSTEM.

This helps a lot, I'm now again able to do both

./pre-inst-env guix system vm-image --target=i586-pc-gnu --no-grafts gnu/system/examples/bare-bones.tmpl
./pre-inst-env guix system vm-image gnu/system/examples/bare-bones.tmpl

again! Phew... And that's also because I learned to include the full
list of exported symbols in the autoload of (gnu store database).

Mathieu, if it turns out that you are dropping qemu-image next week
altogether then great!, and I have no problem whatsoever dropping this
patch set as well.

For now, I have reset wip-hurd-vm and put your commits towards the
bottom of the wip-hurd-vm stack too.

Sending an updated v3 patch set just because I'm so pleased with this
result.

Greetings,
janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 23 May 2020 11:30
[PATCH v3 1/3] utils: Move 'reset-timestamps' out of database.
(address . 41350@debbugs.gnu.org)
20200523093017.12149-1-janneke@gnu.org
This supports calling reset-timestamps without loading sqlite3.

* guix/store/database.scm (reset-timestamps): Move to...
* guix/utils.scm (reset-timestamps): ... here.
* gnu/build/vm.scm: Include it.
---
gnu/build/vm.scm | 1 +
guix/store/database.scm | 41 +++--------------------------------------
guix/utils.scm | 41 ++++++++++++++++++++++++++++++++++++++---
3 files changed, 42 insertions(+), 41 deletions(-)

Toggle diff (146 lines)
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 433b5a7e8d..c751e6b0e2 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -26,6 +26,7 @@
#:use-module (guix build utils)
#:use-module (guix build store-copy)
#:use-module (guix build syscalls)
+ #:use-module ((guix utils) #:select (reset-timestamps))
#:use-module (guix store database)
#:use-module (gnu build bootloader)
#:use-module (gnu build linux-boot)
diff --git a/guix/store/database.scm b/guix/store/database.scm
index ef52036ede..b8fe313c3d 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -24,9 +24,8 @@
#:use-module (guix store deduplication)
#:use-module (guix base16)
#:use-module (guix progress)
- #:use-module (guix build syscalls)
- #:use-module ((guix build utils)
- #:select (mkdir-p executable-file?))
+ #:use-module ((guix build utils) #:select (mkdir-p))
+ #:use-module ((guix utils) #:select (reset-timestamps))
#:use-module (guix build store-copy)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
@@ -42,8 +41,7 @@
sqlite-register
register-path
register-items
- %epoch
- reset-timestamps))
+ %epoch))
;;; Code for working with the store database directly.
@@ -227,39 +225,6 @@ Every store item in REFERENCES must already be registered."
;;;
;;; High-level interface.
;;;
-
-(define* (reset-timestamps file #:key preserve-permissions?)
- "Reset the modification time on FILE and on all the files it contains, if
-it's a directory. Canonicalize file permissions unless PRESERVE-PERMISSIONS?
-is true."
- ;; Note: We're resetting to one second after the Epoch like 'guix-daemon'
- ;; has always done.
- (let loop ((file file)
- (type (stat:type (lstat file))))
- (case type
- ((directory)
- (unless preserve-permissions?
- (chmod file #o555))
- (utime file 1 1 0 0)
- (let ((parent file))
- (for-each (match-lambda
- (("." . _) #f)
- ((".." . _) #f)
- ((file . properties)
- (let ((file (string-append parent "/" file)))
- (loop file
- (match (assoc-ref properties 'type)
- ((or 'unknown #f)
- (stat:type (lstat file)))
- (type type))))))
- (scandir* parent))))
- ((symlink)
- (utime file 1 1 0 0 AT_SYMLINK_NOFOLLOW))
- (else
- (unless preserve-permissions?
- (chmod file (if (executable-file? file) #o555 #o444)))
- (utime file 1 1 0 0)))))
-
(define* (register-path path
#:key (references '()) deriver prefix
state-directory (deduplicate? #t)
diff --git a/guix/utils.scm b/guix/utils.scm
index d7b197fa44..812617dd61 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -35,8 +35,10 @@
#:use-module (rnrs io ports) ;need 'port-position' etc.
#:use-module ((rnrs bytevectors) #:select (bytevector-u8-set!))
#:use-module (guix memoization)
- #:use-module ((guix build utils) #:select (dump-port mkdir-p delete-file-recursively))
- #:use-module ((guix build syscalls) #:select (mkdtemp! fdatasync))
+ #:use-module ((guix build utils)
+ #:select (dump-port mkdir-p delete-file-recursively
+ executable-file?))
+ #:use-module ((guix build syscalls) #:select (mkdtemp! fdatasync scandir*))
#:use-module (ice-9 format)
#:use-module (ice-9 regex)
#:use-module (ice-9 match)
@@ -109,7 +111,8 @@
call-with-decompressed-port
compressed-output-port
call-with-compressed-output-port
- canonical-newline-port))
+ canonical-newline-port
+ reset-timestamps))
;;;
@@ -843,6 +846,38 @@ a location object."
fix-hint?
(hint condition-fix-hint)) ;string
+(define* (reset-timestamps file #:key preserve-permissions?)
+ "Reset the modification time on FILE and on all the files it contains, if
+it's a directory. Canonicalize file permissions unless PRESERVE-PERMISSIONS?
+is true."
+ ;; Note: We're resetting to one second after the Epoch like 'guix-daemon'
+ ;; has always done.
+ (let loop ((file file)
+ (type (stat:type (lstat file))))
+ (case type
+ ((directory)
+ (unless preserve-permissions?
+ (chmod file #o555))
+ (utime file 1 1 0 0)
+ (let ((parent file))
+ (for-each (match-lambda
+ (("." . _) #f)
+ ((".." . _) #f)
+ ((file . properties)
+ (let ((file (string-append parent "/" file)))
+ (loop file
+ (match (assoc-ref properties 'type)
+ ((or 'unknown #f)
+ (stat:type (lstat file)))
+ (type type))))))
+ (scandir* parent))))
+ ((symlink)
+ (utime file 1 1 0 0 AT_SYMLINK_NOFOLLOW))
+ (else
+ (unless preserve-permissions?
+ (chmod file (if (executable-file? file) #o555 #o444)))
+ (utime file 1 1 0 0)))))
+
;;; Local Variables:
;;; eval: (put 'call-with-progress-reporter 'scheme-indent-function 1)
;;; End:
--
2.26.2
J
J
Jan (janneke) Nieuwenhuizen wrote on 23 May 2020 11:30
[PATCH v3 2/3] system: vm: Do not register-closures when cross-building to the Hurd.
(address . 41350@debbugs.gnu.org)
20200523093017.12149-2-janneke@gnu.org
This supports cross-building building a vm-image for the Hurd, running a
native qemu.

* gnu/system/vm.scm (qemu-image)[register-closures?]: Default to #f when
cross-compiling to the Hurd. Only create sql-schema when actually registering
closures.
---
gnu/build/vm.scm | 9 ++++++++-
gnu/system/vm.scm | 9 ++++++---
2 files changed, 14 insertions(+), 4 deletions(-)

Toggle diff (60 lines)
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index c751e6b0e2..a1b193134a 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -27,7 +27,14 @@
#:use-module (guix build store-copy)
#:use-module (guix build syscalls)
#:use-module ((guix utils) #:select (reset-timestamps))
- #:use-module (guix store database)
+ #:autoload (guix store database) (sql-schema ;must list all exported symbols
+ %default-database-file
+ with-database
+ path-id
+ sqlite-register
+ register-path
+ register-items
+ %epoch)
#:use-module (gnu build bootloader)
#:use-module (gnu build linux-boot)
#:use-module (gnu build install)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 3e483fd86c..b343141c18 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -51,6 +51,7 @@
#:use-module (gnu packages zile)
#:use-module (gnu packages linux)
#:use-module (gnu packages admin)
+ #:use-module (gnu packages hurd)
#:use-module (gnu bootloader)
#:use-module (gnu bootloader grub)
@@ -286,7 +287,8 @@ substitutable."
os
bootcfg-drv
bootloader
- (register-closures? (has-guix-service-type? os))
+ (register-closures? (and (has-guix-service-type? os)
+ (not (hurd-target?))))
(inputs '())
copy-inputs?
(substitutable? #t))
@@ -333,12 +335,13 @@ system that is passed to 'populate-root-file-system'."
((gnu build linux-boot)
#:select (make-essential-device-nodes
make-hurd-device-nodes))
- (guix store database)
(guix build utils)
(srfi srfi-26)
(ice-9 binary-ports))
- (sql-schema #$schema)
+ (when #$register-closures?
+ (use-modules (guix store database))
+ (sql-schema #$schema))
;; Allow non-ASCII file names--e.g., 'nss-certs'--to be decoded.
(setenv "GUIX_LOCPATH"
--
2.26.2
J
J
Jan (janneke) Nieuwenhuizen wrote on 23 May 2020 11:30
[PATCH v3 3/3] system: vm: Build vm-image using native qemu, for the Hurd.
(address . 41350@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200523093017.12149-3-janneke@gnu.org
Cross-building a vm-image is usually done using a cross-qemu, e.g, qemu-ARM,
because, e.g., a native, x86_64 Grub cannot install an armhf-Grub. That
solution does not work for the Hurd, as there is no qemu-HURD.

This patch enables cross building vm-images for the Hurd using a native qemu
vm.

* gnu/system/vm.scm (expression->derivation-in-linux-vm): Run native
qemu-command; use native linux, initrd, bootloader-package and
bootloader-installer, for the Hurd.
[preserve-target]: New helper to install cross-packages into the native vm.
* gnu/bootloader/grub.scm (eye-candy): Use native font.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
---
gnu/bootloader/grub.scm | 4 +-
gnu/system/vm.scm | 81 +++++++++++++++++++++++++++++------------
2 files changed, 60 insertions(+), 25 deletions(-)

Toggle diff (158 lines)
diff --git a/gnu/bootloader/grub.scm b/gnu/bootloader/grub.scm
index bb40c551a7..ccf70b3785 100644
--- a/gnu/bootloader/grub.scm
+++ b/gnu/bootloader/grub.scm
@@ -207,8 +207,8 @@ else
set menu_color_highlight=white/blue
fi~%"
#$setup-gfxterm-body
- #$(grub-root-search store-device font-file)
- #$(setup-gfxterm config font-file)
+ #+(grub-root-search store-device font-file)
+ #+(setup-gfxterm config font-file)
#$(grub-setup-io config)
#$image
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b343141c18..245ecc73b3 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -181,29 +182,46 @@ made available under the /xchg CIFS share.
SUBSTITUTABLE? determines whether the returned derivation should be marked as
substitutable."
(define user-builder
- (program-file "builder-in-linux-vm" exp))
+ (scheme-file "builder-in-linux-vm" exp))
+
+ (define (preserve-target obj)
+ (if target
+ (with-parameters ((%current-target-system target))
+ obj)
+ obj))
+
+ (define-syntax-rule (check predicate)
+ (let-system (system target)
+ (predicate (or target system))))
(define loader
- ;; Invoke USER-BUILDER instead using 'primitive-load'. The reason for
- ;; this is to allow USER-BUILDER to dlopen stuff by using a full-featured
- ;; Guile, which it couldn't do using the statically-linked guile used in
- ;; the initrd. See example at
+ ;; Instead of using 'primitive-load', evaluate USER-BUILDER in a
+ ;; full-featured Guile so it can use dlopen stuff, which it couldn't do
+ ;; using the statically-linked guile used in the initrd. See example at
;; <https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00233.html>.
(program-file "linux-vm-loader"
- ;; Communicate USER-BUILDER's exit status via /xchg so that
- ;; the host can distinguish between success, failure, and
- ;; kernel panic.
- #~(let ((status (system* #$user-builder)))
+ ;; When cross-compiling, USER-BUILDER refers to the target
+ ;; (cross-compiled) system. Preserve that, even though
+ ;; LOADER itself is executed as a native program.
+ #~(let* ((guile #$(if (check hurd-triplet?)
+ #~#+(file-append (default-guile)
+ "/bin/guile")
+ (file-append (default-guile)
+ "/bin/guile")))
+ (status (system* guile "--no-auto-compile"
+ #$(if (check hurd-triplet?)
+ (preserve-target user-builder)
+ user-builder))))
+
+ ;; Communicate USER-BUILDER's exit status via /xchg so
+ ;; that the host can distinguish between success,
+ ;; failure, and kernel panic.
(call-with-output-file "/xchg/.exit-status"
(lambda (port)
(write status port)))
(sync)
(reboot))))
- (define-syntax-rule (check predicate)
- (let-system (system target)
- (predicate (or target system))))
-
(let ((initrd (or initrd
(base-initrd file-systems
#:on-error 'backtrace
@@ -227,10 +245,16 @@ substitutable."
(let* ((native-inputs
'#+(list qemu (canonical-package coreutils)))
- (linux (string-append #$linux "/"
- #$(system-linux-image-file-name)))
- (initrd #$initrd)
- (loader #$loader)
+
+ (loader #$(if (check hurd-triplet?) #~#+loader loader))
+ (linux #$(if (check hurd-triplet?)
+ #~(string-append
+ #+linux "/"
+ #+(system-linux-image-file-name))
+ #~(string-append
+ #$linux "/"
+ #$(system-linux-image-file-name))))
+ (initrd #$(if (check hurd-triplet?) #~#+initrd initrd))
(graphs '#$(match references-graphs
(((graph-files . _) ...) graph-files)
(_ #f)))
@@ -246,7 +270,10 @@ substitutable."
(load-in-linux-vm loader
#:output #$output
#:linux linux #:initrd initrd
- #:qemu (qemu-command target)
+ #:qemu #$(if (or (not target)
+ (check hurd-triplet?))
+ (qemu-command)
+ (qemu-command target))
#:memory-size #$memory-size
#:make-disk-image? #$make-disk-image?
#:single-file-output? #$single-file-output?
@@ -349,9 +376,13 @@ system that is passed to 'populate-root-file-system'."
(setlocale LC_ALL "en_US.utf8")
(let ((inputs
- '#$(append (list parted e2fsprogs dosfstools)
- (map canonical-package
- (list sed grep coreutils findutils gawk))))
+ '#$(if (hurd-target?)
+ #~#+(append (list parted e2fsprogs dosfstools)
+ (map canonical-package
+ (list sed grep coreutils findutils gawk)))
+ (append (list parted e2fsprogs dosfstools)
+ (map canonical-package
+ (list sed grep coreutils findutils gawk)))))
;; This variable is unused but allows us to add INPUTS-TO-COPY
;; as inputs.
@@ -426,12 +457,16 @@ system that is passed to 'populate-root-file-system'."
#:partitions partitions
#:grub-efi grub-efi
#:bootloader-package
- #$(bootloader-package bootloader)
+ #$(if (hurd-target?)
+ #~#+(bootloader-package bootloader)
+ (bootloader-package bootloader))
#:bootcfg #$bootcfg-drv
#:bootcfg-location
#$(bootloader-configuration-file bootloader)
#:bootloader-installer
- #$(bootloader-installer bootloader)))))))
+ #$(if (hurd-target?)
+ #~#+(bootloader-installer bootloader)
+ (bootloader-installer bootloader))))))))
#:system system
#:target target
#:make-disk-image? #t
--
2.26.2
M
M
Mathieu Othacehe wrote on 23 May 2020 19:45
Re: [bug#41350] [PATCH v2 3/3] system: vm: Build vm-image using native qemu, for the Hurd.
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87v9kmilze.fsf@gnu.org
Hello Jan,

Toggle quote (7 lines)
> Mathieu, if it turns out that you are dropping qemu-image next week
> altogether then great!, and I have no problem whatsoever dropping this
> patch set as well.
>
> For now, I have reset wip-hurd-vm and put your commits towards the
> bottom of the wip-hurd-vm stack too.

I just pushed a few commits to your branch. I'm now able to run:

Toggle snippet (3 lines)
guix system disk-image --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl

no VM inside.

We now have to backport some of your work to (gnu system image), but
the good news is that closure registration is now working, and the Hurd
boots.

Toggle quote (3 lines)
> Sending an updated v3 patch set just because I'm so pleased with this
> result.

I'll have a look :)

Thanks,

Mathieu
J
J
Jan Nieuwenhuizen wrote on 23 May 2020 21:07
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87k112qxly.fsf@gnu.org
Mathieu Othacehe writes:

Toggle quote (15 lines)
> Hello Jan,
>
>> Mathieu, if it turns out that you are dropping qemu-image next week
>> altogether then great!, and I have no problem whatsoever dropping this
>> patch set as well.
>>
>> For now, I have reset wip-hurd-vm and put your commits towards the
>> bottom of the wip-hurd-vm stack too.
>
> I just pushed a few commits to your branch. I'm now able to run:
>
> guix system disk-image --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl
>
> no VM inside.

Whoa, that's 10 commits -- great! I do need to use --no-grafts, if I
don't I get

guix system: error: gnu/packages/glib.scm:406:2: gobject-introspection@1.62.0: build system `meson' does not support cross builds

Do you see that too?

Toggle quote (2 lines)
> We now have to backport some of your work to (gnu system image), but

Just build and started it -- beautiful. I'll have a stab later
tomorrow, unless someone beats me to it ;)

Toggle quote (3 lines)
> the good news is that closure registration is now working, and the Hurd
> boots.

very nice, that's already one ahead!

Toggle quote (2 lines)
> I'll have a look :)

Thanks :-)

Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
M
M
Mathieu Othacehe wrote on 24 May 2020 11:18
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87r1v9itd0.fsf@gnu.org
Hey!

Toggle quote (4 lines)
> guix system: error: gnu/packages/glib.scm:406:2: gobject-introspection@1.62.0: build system `meson' does not support cross builds
>
> Do you see that too?

Yes, same issue. When grafting is enabled, we try to cross-built the
native extension "guile-rsvg" in (gnu bootloader grub).

Taking inspiration from f52fbf709, I tried:

Toggle snippet (24 lines)
diff --git a/guix/packages.scm b/guix/packages.scm
index 3d9988d836..e74ac882cb 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -1276,13 +1276,14 @@ to (see 'graft-derivation'.)"
(define target (bag-target bag))
(define native-grafts
- (let ((->graft (input-graft store system)))
- (fold-bag-dependencies (lambda (package grafts)
- (match (->graft package)
- (#f grafts)
- (graft (cons graft grafts))))
- '()
- bag)))
+ (parameterize ((%current-target-system target))
+ (let ((->graft (input-graft store system)))
+ (fold-bag-dependencies (lambda (package grafts)
+ (match (->graft package)
+ (#f grafts)
+ (graft (cons graft grafts))))
+ '()
+ bag))))
which, by pure luck, fixes the issue for me. Maybe, I should also do
that for "target-grafts". Ludo, WDYT?

Thanks,

Mathieu
J
J
Jan Nieuwenhuizen wrote on 24 May 2020 13:19
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87pnat37ia.fsf@gnu.org
Jan Nieuwenhuizen writes:

Hello Mathieu,

Toggle quote (6 lines)
> Mathieu Othacehe writes:
>> We now have to backport some of your work to (gnu system image), but
>
> Just build and started it -- beautiful. I'll have a stab later
> tomorrow, unless someone beats me to it ;)

I have just pushed a few commits to wip-hurd-vm.

b0df9b0c6b squash! bootloader: grub: Add support for '<hurd-menu-entry>'.

We were using a cross-built gnumach, which does not work. This did not
happen with "vm-image" because "qemu-image" of how qemu-image built
grub. Best to squash that later.

6169cb6396 Revert "system: hurd: Remove bash from boot process."

As mentioned in the commit, where we used to run patched linux in the
VM, the "Remove bash from boot process" patch would now rely on the
host's linux being patched. It's a bit early times/inconvenient to
demand this, I suppose...

c378a2beec linux-boot: Update 'make-hurd-device-nodes'.

Moved a part of the Hurd stuff here, and the rest

b605a36031 WIP hurd-directives

here. It does not work and I don't understand that I had to add a call
to make-device-nodes...that's probably not how you want it to work?

Anyway...no real luck yet. I'm stuck at:

start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec startup proc auth
/hurd/startup: Failed to bind to /servers/startup: Device or resource busy
.
/gnu/store/5m6m1ic2m4m43nv4wl7nfa5ab9p76b2g-hurd-0.9-1.91a5167/hurd/exec: main.c:358: S_exec_init: Unexpected error: (ipc/mig) bad request message ID.

Ideas? ... Ludo perhaps?

Greetings,
janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
M
M
Mathieu Othacehe wrote on 24 May 2020 14:07
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87k111ilj0.fsf@gnu.org
Hello Jan,

Toggle quote (3 lines)
> here. It does not work and I don't understand that I had to add a call
> to make-device-nodes...that's probably not how you want it to work?

No its more an omission, and for Linux, it seems that it works without
make-device-nodes call.

Toggle quote (10 lines)
>
> Anyway...no real luck yet. I'm stuck at:
>
> start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec startup proc auth
> /hurd/startup: Failed to bind to /servers/startup: Device or resource busy
> .
> /gnu/store/5m6m1ic2m4m43nv4wl7nfa5ab9p76b2g-hurd-0.9-1.91a5167/hurd/exec: main.c:358: S_exec_init: Unexpected error: (ipc/mig) bad request message ID.
>
> Ideas? ... Ludo perhaps?

Running the following commands:

Toggle snippet (16 lines)
./pre-inst-env guix system disk-image --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl
cp /gnu/store/the-image /tmp/img_ko

./pre-inst-env guix system vm-image --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl
cp /gnu/store/the-vm-image /tmp/img_ok.qcow2
qemu-img convert -f qcow2 -O raw /tmp/img_ok.qcow2 /tmp/img_ok.raw

sudo losetup -P /dev/loop1 /tmp/img_ko
sudo losetup -P /dev/loop2 /tmp/img_ok.raw

sudo mount /dev/loop1p1 /mnt/img/
sudo mount /dev/loop2p1 /mnt/img2/

diff -r /mnt/img /mnt/img

I get:

Toggle snippet (47 lines)
Only in /mnt/img2/dev: cons
Only in /mnt/img2/dev: console
Only in /mnt/img2/dev: fd
Only in /mnt/img2/dev: klog
Only in /mnt/img2/dev: kmsg
Only in /mnt/img2/dev: log
Only in /mnt/img2/dev: mem
Only in /mnt/img2/dev: ptyp0
Only in /mnt/img2/dev: ptyp1
Only in /mnt/img2/dev: ptyp2
Only in /mnt/img2/dev: shm
Only in /mnt/img2/dev: stderr
Only in /mnt/img2/dev: stdin
Only in /mnt/img2/dev: stdout
Only in /mnt/img2/dev: time
Only in /mnt/img2/dev: tty
Only in /mnt/img2/dev: tty1
Only in /mnt/img2/dev: tty12
Only in /mnt/img2/dev: tty2
Only in /mnt/img2/dev: tty3
Only in /mnt/img2/dev: tty4
Only in /mnt/img2/dev: tty5
Only in /mnt/img2/dev: tty6
Only in /mnt/img2/dev: ttyp0
Only in /mnt/img2/dev: ttyp1
Only in /mnt/img2/dev: ttyp2
diff: /mnt/img2/dev/urandom: Structure needs cleaning
Only in /mnt/img2/dev: vcs
Only in /mnt/img2/etc: fstab
Only in /mnt/img2/etc: group
Only in /mnt/img2/etc: hostname
Only in /mnt/img2/etc: login
Only in /mnt/img2/etc: motd
Only in /mnt/img2/etc: pam.d
Only in /mnt/img2/etc: passwd
Only in /mnt/img2/etc: profile
Only in /mnt/img2/etc: protocols
Only in /mnt/img2/etc: .pwd.lock
Only in /mnt/img2/etc: services
Only in /mnt/img2/etc: shadow
Only in /mnt/img2/etc: shells
Only in /mnt/img2/etc: skel
Only in /mnt/img2/etc: ssh
Only in /mnt/img2/etc: ssl
Only in /mnt/img2/etc: static

It seems that at least etc and dev directories are not correctly
populated using disk-image. To be continued!

Mathieu
J
J
Jan Nieuwenhuizen wrote on 24 May 2020 16:20
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87a71x8ldi.fsf@gnu.org
Mathieu Othacehe writes:

Helo Mathieu,

Toggle quote (11 lines)
>> Ideas? ... Ludo perhaps?
>
> Running the following commands:
>
> ./pre-inst-env guix system disk-image --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl
> cp /gnu/store/the-image /tmp/img_ko
>
> ./pre-inst-env guix system vm-image --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl
> cp /gnu/store/the-vm-image /tmp/img_ok.qcow2
> qemu-img convert -f qcow2 -O raw /tmp/img_ok.qcow2 /tmp/img_ok.raw

Ah, qemu-img convert -- nice. So, we get different type of images.
Wonder if that could play some role?

Toggle quote (17 lines)
> sudo losetup -P /dev/loop1 /tmp/img_ko
> sudo losetup -P /dev/loop2 /tmp/img_ok.raw
>
> sudo mount /dev/loop1p1 /mnt/img/
> sudo mount /dev/loop2p1 /mnt/img2/
>
> diff -r /mnt/img /mnt/img
>
>
> I get:
>
> Only in /mnt/img2/dev: cons
> Only in /mnt/img2/dev: console
> Only in /mnt/img2/dev: fd
> Only in /mnt/img2/dev: klog
> Only in /mnt/img2/dev: kmsg

That's weird; I only have

/mnt/dev/null
/mnt/dev/urandom
/mnt/dev/full
/mnt/dev/zero
/mnt/dev/random

on both...

Toggle quote (3 lines)
> It seems that at least etc and dev directories are not correctly
> populated using disk-image. To be continued!

Hmm. I compared a file listing between these

./pre-inst-env guix system disk-image --no-grafts --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl
./pre-inst-env guix system vm-image --no-grafts --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl

and the differences are pretty minimal (see attached). The vm-image
fully works, the disk-image shows "/servers/startup: Device or resource
busy"

I used something like

Toggle snippet (5 lines)
modprobe nbd max_part=63
qemu-nbd -c /dev/nbd0 vm-image.img
mount /dev/nbd0p1 /mnt

The most interesting differences I see are wrt Grub, e.g.

-/mnt/boot/grub/fonts
-/mnt/boot/grub/fonts/unicode.pf2

could that still play a role? I'm having a look why this could be missing.

Greetings,
janneke
Attachment: vm-to-disk.diff
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 24 May 2020 18:36
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87imgl47e3.fsf@gnu.org
Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (19 lines)
> Mathieu Othacehe writes:
>
>> Hello Jan,
>>
>>> Mathieu, if it turns out that you are dropping qemu-image next week
>>> altogether then great!, and I have no problem whatsoever dropping this
>>> patch set as well.
>>>
>>> For now, I have reset wip-hurd-vm and put your commits towards the
>>> bottom of the wip-hurd-vm stack too.
>>
>> I just pushed a few commits to your branch. I'm now able to run:
>>
>> guix system disk-image --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl
>>
>> no VM inside.
>
> Whoa, that's 10 commits -- great!

Woow, excellent! Thumbs up!

(I’ll take a look at the graft issue, I feel a repulsion for dynamic
binding growing every day…)

Ludo’.
M
M
Mathieu Othacehe wrote on 24 May 2020 20:11
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87ftbpfbjl.fsf@gnu.org
Toggle quote (3 lines)
> Ah, qemu-img convert -- nice. So, we get different type of images.
> Wonder if that could play some role?

Hope not!

Toggle quote (7 lines)
> The most interesting differences I see are wrt Grub, e.g.
>
> -/mnt/boot/grub/fonts
> -/mnt/boot/grub/fonts/unicode.pf2
>
> could that still play a role? I'm having a look why this could be missing.

It's because I just copy "lib/grub" folder in (gnu bootloader grub). I
should also do that for the fonts I guess. But I doubt the issue comes
from here.

Toggle quote (8 lines)
> --- vm-image.lst-s 2020-05-24 15:41:02.051314009 +0200
> +++ disk-image.lst-s 2020-05-24 15:41:08.407414141 +0200
> @@ -3,293 +3,577 @@
> /mnt/boot
> /mnt/boot/activation
> /mnt/boot/grub
> -/mnt/boot/grub/fonts

At some point, if we have the same files inside, it should work :p I
noticed that doing a cfdisk on the raw converted, vm-image root
partition reports:

Attributes: 80

which doesn't appear on the disk-image.

However, tune2fs does not show any noticeable difference (both ext_attr
and Hurd OS type are set).

Mathieu
J
J
Jan Nieuwenhuizen wrote on 24 May 2020 20:40
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
874ks52n3o.fsf@gnu.org
Mathieu Othacehe writes:

Toggle quote (5 lines)
>> Ah, qemu-img convert -- nice. So, we get different type of images.
>> Wonder if that could play some role?
>
> Hope not!

Okay...let's continue searching elsewhere then.

Toggle quote (11 lines)
>> The most interesting differences I see are wrt Grub, e.g.
>>
>> -/mnt/boot/grub/fonts
>> -/mnt/boot/grub/fonts/unicode.pf2
>>
>> could that still play a role? I'm having a look why this could be missing.
>
> It's because I just copy "lib/grub" folder in (gnu bootloader grub). I
> should also do that for the fonts I guess. But I doubt the issue comes
> from here.

Meanwhile I tried adding that by hand; indeed that's not it.

Toggle quote (16 lines)
>> --- vm-image.lst-s 2020-05-24 15:41:02.051314009 +0200
>> +++ disk-image.lst-s 2020-05-24 15:41:08.407414141 +0200
>> @@ -3,293 +3,577 @@
>> /mnt/boot
>> /mnt/boot/activation
>> /mnt/boot/grub
>> -/mnt/boot/grub/fonts
>
> At some point, if we have the same files inside, it should work :p I
> noticed that doing a cfdisk on the raw converted, vm-image root
> partition reports:
>
> Attributes: 80
>
> which doesn't appear on the disk-image.

I am not aware that we would be using attributes, at least I reverted
the xattr trick on wip-hurd-vm for the /servers. Can we find out where
and what they are?

Toggle quote (3 lines)
> However, tune2fs does not show any noticeable difference (both ext_attr
> and Hurd OS type are set).

Indeed, I looked at that too. This may be so obvious once we find it...

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 25 May 2020 17:46
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87d06sc90t.fsf@gnu.org
Jan Nieuwenhuizen writes:

Hello Mathieu, Ludo',

Toggle quote (13 lines)
>>> The most interesting differences I see are wrt Grub, e.g.
>>>
>>> -/mnt/boot/grub/fonts
>>> -/mnt/boot/grub/fonts/unicode.pf2
>>>
>>> could that still play a role? I'm having a look why this could be missing.
>>
>> It's because I just copy "lib/grub" folder in (gnu bootloader grub). I
>> should also do that for the fonts I guess. But I doubt the issue comes
>> from here.
>
> Meanwhile I tried adding that by hand; indeed that's not it.

As discussed on IRC (thanks!!), I bisected it down to a problem with the
file /servers/exec. I suspected translator magic...but luckily Ludo
insisted he thinks that /servers/exec only needs to exist.

So, using current wip-hurd-vm, I tried this script:

Toggle snippet (12 lines)
set -ex
cp -f $(./pre-inst-env guix system disk-image --no-grafts --target=i586-pc-gnu gnu/system/examples/bare-hurd.tmpl) /tmp/disk-image.img
sudo losetup -P /dev/loop0 /tmp/disk-image.img
sudo mount /dev/loop0p1 /mnt
ls -l /mnt/servers/exec
sudo rm -f /mnt/servers/exec
sudo touch /mnt/servers/exec
ls -l /mnt/servers/exec
sudo umount /mnt
sudo losetup -d /dev/loop0

...which indeed produces a working VM!

Turns out that creating /servers/exec in the store gets
hard-linked/deduplicated or something...look:

Toggle snippet (8 lines)
+ ls -l /mnt/servers/exec
-r--r--r-- 17 root root 0 Jan 1 1970 /mnt/servers/exec
+ sudo rm -f /mnt/servers/exec
+ sudo touch /mnt/servers/exec
+ ls -l /mnt/servers/exec
-rw-r--r-- 1 root root 0 May 25 17:16 /mnt/servers/exec

The "fix" is just going from 17 to 1 link. Doh'

So, another "fix" is the diff below but I did not want to commit and
push that yet. Can we prevent creation of hard links in another way?

Greetings,
Janneke


$ git diff
Toggle diff (25 lines)
diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 7dd509d0d9..dfad83aa05 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -337,6 +337,7 @@ one specific hardware device. These we have to create."
(for-each (lambda (file)
(call-with-output-file (scope file)
(lambda (port)
+ (display file port) ;hack to avoid hard-linking
(chmod port #o666))))
'("dev/null"
"dev/zero"
@@ -350,6 +351,7 @@ one specific hardware device. These we have to create."
(for-each (lambda (file)
(call-with-output-file (scope (string-append "servers/" file))
(lambda (port)
+ (display file port) ;hack to avoid hard-linking
(chmod port #o444))))
'("startup"
"exec"


--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 27 May 2020 10:43
Re: [bug#41350] [PATCH v3 1/3] utils: Move 'reset-timestamps' out of database.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 41350@debbugs.gnu.org)
87zh9tlqdh.fsf@gnu.org
Hi!

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (6 lines)
> This supports calling reset-timestamps without loading sqlite3.
>
> * guix/store/database.scm (reset-timestamps): Move to...
> * guix/utils.scm (reset-timestamps): ... here.
> * gnu/build/vm.scm: Include it.

Please open different issues for different patch series so that each one
is visible. :-)

Toggle quote (8 lines)
> --- a/gnu/build/vm.scm
> +++ b/gnu/build/vm.scm
> @@ -26,6 +26,7 @@
> #:use-module (guix build utils)
> #:use-module (guix build store-copy)
> #:use-module (guix build syscalls)
> + #:use-module ((guix utils) #:select (reset-timestamps))

We shouldn’t include (guix utils) on the build side because it pulls in
the host (guix config), which is bad because it’s user-specific:

Toggle snippet (10 lines)
scheme@(guile-user)> ,use(guix modules)
scheme@(guile-user)> ,pp (source-module-closure '((guix utils)))
$7 = ((guix utils)
(guix config)
(guix memoization)
(guix profiling)
(guix build utils)
(guix build syscalls))

Or we have to remember to do the ((guix config) => ,(make-config.scm))
dance.

What’s the problem with loading sqlite3?

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 27 May 2020 10:45
Re: [bug#41350] [PATCH v3 2/3] system: vm: Do not register-closures when cross-building to the Hurd.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 41350@debbugs.gnu.org)
87v9khlqa3.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (7 lines)
> This supports cross-building building a vm-image for the Hurd, running a
> native qemu.
>
> * gnu/system/vm.scm (qemu-image)[register-closures?]: Default to #f when
> cross-compiling to the Hurd. Only create sql-schema when actually registering
> closures.

[...]

Toggle quote (4 lines)
> - (register-closures? (has-guix-service-type? os))
> + (register-closures? (and (has-guix-service-type? os)
> + (not (hurd-target?))))

What’s the problem here? (Sorry if I missed earlier discussions!)

Intuitively, I think there shouldn’t be system-specific bits here:
registering closures has nothing to do with the OS we’re targeting.

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 27 May 2020 10:59
Re: [bug#41350] [PATCH v3 1/3] utils: Move 'reset-timestamps' out of database.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 41350@debbugs.gnu.org)
87imghlpmw.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (3 lines)
> Please open different issues for different patch series so that each one
> is visible. :-)

Silly me, forget about that, I was just confused. Apologies!

/me goes back to the beginning of the thread…
J
J
Jan Nieuwenhuizen wrote on 27 May 2020 11:10
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 41350@debbugs.gnu.org)
87r1v5agl5.fsf@gnu.org
Ludovic Courtès writes:

Hi!

Toggle quote (11 lines)
> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> This supports calling reset-timestamps without loading sqlite3.
>>
>> * guix/store/database.scm (reset-timestamps): Move to...
>> * guix/utils.scm (reset-timestamps): ... here.
>> * gnu/build/vm.scm: Include it.
>
> Please open different issues for different patch series so that each one
> is visible. :-)

Okay...yes this "grew" out of the wip-hurd-vm => qemu-cross => etc. :-/

Toggle quote (20 lines)
>> --- a/gnu/build/vm.scm
>> +++ b/gnu/build/vm.scm
>> @@ -26,6 +26,7 @@
>> #:use-module (guix build utils)
>> #:use-module (guix build store-copy)
>> #:use-module (guix build syscalls)
>> + #:use-module ((guix utils) #:select (reset-timestamps))
>
> We shouldn’t include (guix utils) on the build side because it pulls in
> the host (guix config), which is bad because it’s user-specific:
>
> scheme@(guile-user)> ,use(guix modules)
> scheme@(guile-user)> ,pp (source-module-closure '((guix utils)))
> $7 = ((guix utils)
> (guix config)
> (guix memoization)
> (guix profiling)
> (guix build utils)
> (guix build syscalls))

Oh, crap!

Toggle quote (5 lines)
> Or we have to remember to do the ((guix config) => ,(make-config.scm))
> dance.
>
> What’s the problem with loading sqlite3?

When running a native qemu-image with cross-built content, any .GO files
for TARGET are skipped and .SCM load fine, except for sqlite. It fails
trying to load the sqlite3.so

Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 27 May 2020 11:13
Re: [bug#41350] [PATCH v3 2/3] system: vm: Do not register-closures when cross-building to the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 41350@debbugs.gnu.org)
87mu5tagfw.fsf@gnu.org
Ludovic Courtès writes:

Toggle quote (17 lines)
> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> This supports cross-building building a vm-image for the Hurd, running a
>> native qemu.
>>
>> * gnu/system/vm.scm (qemu-image)[register-closures?]: Default to #f when
>> cross-compiling to the Hurd. Only create sql-schema when actually registering
>> closures.
>
> [...]
>
>> - (register-closures? (has-guix-service-type? os))
>> + (register-closures? (and (has-guix-service-type? os)
>> + (not (hurd-target?))))
>
> What’s the problem here? (Sorry if I missed earlier discussions!)

register-closures is being done inside the VM and needs sqlite, which we
don't have because it's for the wrong architecture. So as long as we
avoid sqlite, we can cross-build a working VM.

Toggle quote (3 lines)
> Intuitively, I think there shouldn’t be system-specific bits here:
> registering closures has nothing to do with the OS we’re targeting.

True...but currently Hurd is the only image that we are trying to build
in this weird way, i.e., using a native qemu! Other cross-builds of
vm-images are done using qemu-TARGET, only for the Hurd that does
not work.

Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 27 May 2020 11:30
Re: [bug#41350] [PATCH v2 3/3] system: vm: Build vm-image using native qemu, for the Hurd.
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
878shdlo7v.fsf@gnu.org
Hi!

Mathieu Othacehe <othacehe@gnu.org> skribis:

Toggle quote (37 lines)
>> guix system: error: gnu/packages/glib.scm:406:2: gobject-introspection@1.62.0: build system `meson' does not support cross builds
>>
>> Do you see that too?
>
> Yes, same issue. When grafting is enabled, we try to cross-built the
> native extension "guile-rsvg" in (gnu bootloader grub).
>
> Taking inspiration from f52fbf709, I tried:
>
> diff --git a/guix/packages.scm b/guix/packages.scm
> index 3d9988d836..e74ac882cb 100644
> --- a/guix/packages.scm
> +++ b/guix/packages.scm
> @@ -1276,13 +1276,14 @@ to (see 'graft-derivation'.)"
> (define target (bag-target bag))
>
> (define native-grafts
> - (let ((->graft (input-graft store system)))
> - (fold-bag-dependencies (lambda (package grafts)
> - (match (->graft package)
> - (#f grafts)
> - (graft (cons graft grafts))))
> - '()
> - bag)))
> + (parameterize ((%current-target-system target))
> + (let ((->graft (input-graft store system)))
> + (fold-bag-dependencies (lambda (package grafts)
> + (match (->graft package)
> + (#f grafts)
> + (graft (cons graft grafts))))
> + '()
> + bag))))
>
>
> which, by pure luck, fixes the issue for me. Maybe, I should also do
> that for "target-grafts". Ludo, WDYT?

Could you boil the problem down to a “guix build --target” command?

I’d like to make sure we understand the issue so we can add a test when
we fix it.

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 28 May 2020 00:54
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87zh9tf0qa.fsf@gnu.org
Hi!

Mathieu Othacehe <othacehe@gnu.org> skribis:

Toggle quote (22 lines)
>> 1. When cross-compiling, can the ‘qemu-image’ procedure to its job by
>> running exclusively native software (in particular using a native
>> QEMU, native kernel, etc.)?
>
> I think the answer is yes, but I raised a concern about being able to
> run grub-install from an ARM system to build a cross-compiled x86-64
> system (for instance).
>
> For now, running this command shows:
>
> ls $(guix build --system=aarch64-linux grub)/lib/grub/
> arm64-efi
>
> So, the native aarch64-linux is only able to install itself on the same
> system. I think can be fixed though (same as for grub-hybrid package).
>
>> As for (2), I’d say that when cross-compiling, it should just run native
>> software but simply preserve references to cross-compiled software,
>> which is what janneke’s patch does.
>
> Yes, I agree.

So it took me the whole day, but I ended up with the following patches
addressing these two points. With that on master, I can do:

guix system vm --target=arm-linux-gnueabihf --no-grafts \
gnu/system/examples/bare-bones.tmpl

and get a usable script that spawns an ARM VM (I still need to add ‘-M
virt -nographic’ and remove ‘-enable-kvm’, but that’s a secondary
issue.)

Likewise, this produces what looks like a valid image, except probably
for the bootloader (I used GRUB in bare-bones, which didn’t complain,
but the result doesn’t work):

guix system vm-image --target=arm-linux-gnueabihf …

Thoughts?

I’ll see if I can test it with ‘wip-hurd-vm’ on top but I’m not sure
I’ll do it before the week-end.

Toggle quote (4 lines)
> However, I think I found a way to install Grub, without root
> permissions, from the host system (see:
> https://lists.gnu.org/archive/html/guix-patches/2020-05/msg00988.html).

Yay!

Toggle quote (2 lines)
> This should allow to deprecate the whole (gnu system vm) module.

Yeah, using the new image API will be nicer. We still need to keep some
of the API in (guix system vm), at least for the ‘guix system vm’
command.

Thanks,
Ludo’.
From 1ccd6f8e853be17fa4ec244a660fa53ffcec979f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Wed, 27 May 2020 17:40:14 +0200
Subject: [PATCH 1/7] system: 'system-linux-image-file-name' takes an optional
parameter.

* gnu/system.scm (system-linux-image-file-name): Make 'target' an
optional parameter.
---
gnu/system.scm | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)

Toggle diff (31 lines)
diff --git a/gnu/system.scm b/gnu/system.scm
index d929187695..ab0a6dfc33 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -466,15 +466,15 @@ from the initrd."
"Return the list of swap services for OS."
(map swap-service (operating-system-swap-devices os)))
-(define* (system-linux-image-file-name)
- "Return the basename of the kernel image file for SYSTEM."
- ;; FIXME: Evaluate the conditional based on the actual current system.
- (let ((target (or (%current-target-system) (%current-system))))
- (cond
- ((string-prefix? "arm" target) "zImage")
- ((string-prefix? "mips" target) "vmlinuz")
- ((string-prefix? "aarch64" target) "Image")
- (else "bzImage"))))
+(define* (system-linux-image-file-name #:optional
+ (target (or (%current-target-system)
+ (%current-system))))
+ "Return the basename of the kernel image file for TARGET."
+ (cond
+ ((string-prefix? "arm" target) "zImage")
+ ((string-prefix? "mips" target) "vmlinuz")
+ ((string-prefix? "aarch64" target) "Image")
+ (else "bzImage")))
(define (operating-system-kernel-file os)
"Return an object representing the absolute file name of the kernel image of
--
2.26.2
From 56d8d921199bfdf151d899639052f579f24fdd00 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Wed, 27 May 2020 23:04:48 +0200
Subject: [PATCH 2/7] vm: 'expression->derivation-in-linux-vm' always returns a
native build.

* gnu/system/vm.scm (expression->derivation-in-linux-vm): Remove #:target.
[builder]: Use #+. Don't pass #:target-arm32? and #:target-aarch64? to
'load-in-linux-vm'.
Pass #:target #f to 'gexp->derivation'.
(qemu-image): Adjust accordingly.
* gnu/build/vm.scm (load-in-linux-vm): Remove #:target-aarch64?
and #:target-arm32?. Define them as local variables.
---
gnu/build/vm.scm | 11 ++++++++---
gnu/system/vm.scm | 16 +++++++---------
2 files changed, 15 insertions(+), 12 deletions(-)

Toggle diff (86 lines)
diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 433b5a7e8d..0f0ceae18f 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -84,8 +84,6 @@
linux initrd
make-disk-image?
single-file-output?
- target-arm32?
- target-aarch64?
(disk-image-size (* 100 (expt 2 20)))
(disk-image-format "qcow2")
(references-graphs '()))
@@ -101,7 +99,14 @@ access it via /dev/hda.
REFERENCES-GRAPHS can specify a list of reference-graph files as produced by
the #:references-graphs parameter of 'derivation'."
- (define target-arm? (or target-arm32? target-aarch64?))
+ (define target-arm32?
+ (string-prefix? "arm-" %host-type))
+
+ (define target-aarch64?
+ (string-prefix? "aarch64-" %host-type))
+
+ (define target-arm?
+ (or target-arm32? target-aarch64?))
(define arch-specific-flags
`(;; On ARM, a machine has to be specified. Use "virt" machine to avoid
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 3e483fd86c..d737a5e4ec 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -141,7 +141,7 @@
(define* (expression->derivation-in-linux-vm name exp
#:key
- (system (%current-system)) target
+ (system (%current-system))
(linux linux-libre)
initrd
(qemu qemu-minimal)
@@ -226,10 +226,11 @@ substitutable."
(let* ((native-inputs
'#+(list qemu (canonical-package coreutils)))
- (linux (string-append #$linux "/"
- #$(system-linux-image-file-name)))
- (initrd #$initrd)
- (loader #$loader)
+ (linux (string-append
+ #+linux "/"
+ #+(system-linux-image-file-name system)))
+ (initrd #+initrd)
+ (loader #+loader)
(graphs '#$(match references-graphs
(((graph-files . _) ...) graph-files)
(_ #f)))
@@ -249,8 +250,6 @@ substitutable."
#:memory-size #$memory-size
#:make-disk-image? #$make-disk-image?
#:single-file-output? #$single-file-output?
- #:target-arm32? #$(check target-arm32?)
- #:target-aarch64? #$(check target-aarch64?)
#:disk-image-format #$disk-image-format
#:disk-image-size size
#:references-graphs graphs))))))
@@ -258,7 +257,7 @@ substitutable."
(gexp->derivation name builder
;; TODO: Require the "kvm" feature.
#:system system
- #:target target
+ #:target #f ;EXP is always executed natively
#:env-vars env-vars
#:guile-for-build guile-for-build
#:references-graphs references-graphs
@@ -430,7 +429,6 @@ system that is passed to 'populate-root-file-system'."
#:bootloader-installer
#$(bootloader-installer bootloader)))))))
#:system system
- #:target target
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
--
2.26.2
From a2b70d97aa80f70cba56d419470ee8d2221dcc1e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Wed, 27 May 2020 23:08:15 +0200
Subject: [PATCH 3/7] vm: 'qemu-image' uses the native partitioning tools and
bootloader.

* gnu/system/vm.scm (qemu-image): Use #+ for Parted, the bootloader, etc.
---
gnu/system/vm.scm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

Toggle diff (30 lines)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index d737a5e4ec..c7767db9df 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -345,7 +345,7 @@ system that is passed to 'populate-root-file-system'."
(setlocale LC_ALL "en_US.utf8")
(let ((inputs
- '#$(append (list parted e2fsprogs dosfstools)
+ '#+(append (list parted e2fsprogs dosfstools)
(map canonical-package
(list sed grep coreutils findutils gawk))))
@@ -422,12 +422,12 @@ system that is passed to 'populate-root-file-system'."
#:partitions partitions
#:grub-efi grub-efi
#:bootloader-package
- #$(bootloader-package bootloader)
+ #+(bootloader-package bootloader)
#:bootcfg #$bootcfg-drv
#:bootcfg-location
#$(bootloader-configuration-file bootloader)
#:bootloader-installer
- #$(bootloader-installer bootloader)))))))
+ #+(bootloader-installer bootloader)))))))
#:system system
#:make-disk-image? #t
#:disk-image-size disk-image-size
--
2.26.2
From db7a12bf54585c8979602956a03fbb6502381dcd Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 28 May 2020 00:37:33 +0200
Subject: [PATCH 4/7] vm: 'qemu-image' preserves the cross-compilation target
of the OS.

* gnu/system/vm.scm (qemu-image)[preserve-target, inputs*]: New variables.
In gexp, use INPUTS* instead of INPUTS. Wrap OS and BOOTCFG-DRV in
'preserve-target'. Pass INPUTS* instead of INPUTS as the #:references-graphs.
---
gnu/system/vm.scm | 23 +++++++++++++++++++----
1 file changed, 19 insertions(+), 4 deletions(-)

Toggle diff (64 lines)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index c7767db9df..991ea2d837 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -317,6 +317,21 @@ system that is passed to 'populate-root-file-system'."
(local-file (search-path %load-path
"guix/store/schema.sql"))))
+ (define preserve-target
+ (if target
+ (lambda (obj)
+ (with-parameters ((%current-target-system target))
+ obj))
+ identity))
+
+ (define inputs*
+ (map (match-lambda
+ ((name thing)
+ `(,name ,(preserve-target thing)))
+ ((name thing output)
+ `(,name ,(preserve-target thing) ,output)))
+ inputs))
+
(expression->derivation-in-linux-vm
name
(with-extensions gcrypt-sqlite3&co
@@ -355,7 +370,7 @@ system that is passed to 'populate-root-file-system'."
'#$(map (match-lambda
((name thing) thing)
((name thing output) `(,thing ,output)))
- inputs)))
+ inputs*)))
(set-path-environment-variable "PATH" '("bin" "sbin") inputs)
@@ -367,7 +382,7 @@ system that is passed to 'populate-root-file-system'."
#:closures graphs
#:copy-closures? #$copy-inputs?
#:register-closures? #$register-closures?
- #:system-directory #$os
+ #:system-directory #$(preserve-target os)
#:make-device-nodes
#$(match device-nodes
@@ -423,7 +438,7 @@ system that is passed to 'populate-root-file-system'."
#:grub-efi grub-efi
#:bootloader-package
#+(bootloader-package bootloader)
- #:bootcfg #$bootcfg-drv
+ #:bootcfg #$(preserve-target bootcfg-drv)
#:bootcfg-location
#$(bootloader-configuration-file bootloader)
#:bootloader-installer
@@ -432,7 +447,7 @@ system that is passed to 'populate-root-file-system'."
#:make-disk-image? #t
#:disk-image-size disk-image-size
#:disk-image-format disk-image-format
- #:references-graphs inputs
+ #:references-graphs inputs*
#:substitutable? substitutable?))
(define* (system-docker-image os
--
2.26.2
From a9270c1261b747dc9a82b52d1ce38314862c51cd Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Wed, 27 May 2020 23:09:49 +0200
Subject: [PATCH 5/7] vm: <virtual-machine> compiler honors system and target.

* gnu/system/vm.scm (system-qemu-image/shared-store): Add #:system
and #:target. Pass it down.
(system-qemu-image/shared-store-script): Likewise.
(virtual-machine-compiler): Likewise.
---
gnu/system/vm.scm | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)

Toggle diff (64 lines)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 991ea2d837..05f3986aca 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -764,6 +764,8 @@ environment with the store shared with the host. MAPPINGS is a list of
(define* (system-qemu-image/shared-store
os
#:key
+ (system (%current-system))
+ (target (%current-target-system))
full-boot?
(disk-image-size (* (if full-boot? 500 30) (expt 2 20))))
"Return a derivation that builds a QEMU image of OS that shares its store
@@ -784,6 +786,8 @@ bootloader refers to: OS kernel, initrd, bootloader data, etc."
;; This is more than needed (we only need the kernel, initrd, GRUB for its
;; font, and the background image), but it's hard to filter that.
(qemu-image #:os os
+ #:system system
+ #:target target
#:bootcfg-drv bootcfg
#:bootloader (bootloader-configuration-bootloader
(operating-system-bootloader os))
@@ -824,6 +828,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
(define* (system-qemu-image/shared-store-script os
#:key
+ (system (%current-system))
+ (target (%current-target-system))
(qemu qemu)
(graphic? #t)
(memory-size 256)
@@ -847,6 +853,8 @@ it is mostly useful when FULL-BOOT? is true."
(mlet* %store-monad ((os -> (virtualized-operating-system os mappings full-boot?))
(image (system-qemu-image/shared-store
os
+ #:system system
+ #:target target
#:full-boot? full-boot?
#:disk-image-size disk-image-size)))
(define kernel-arguments
@@ -920,10 +928,11 @@ FORWARDINGS is a list of host-port/guest-port pairs."
(define-gexp-compiler (virtual-machine-compiler (vm <virtual-machine>)
system target)
- ;; XXX: SYSTEM and TARGET are ignored.
(match vm
(($ <virtual-machine> os qemu graphic? memory-size disk-image-size ())
(system-qemu-image/shared-store-script os
+ #:system system
+ #:target target
#:qemu qemu
#:graphic? graphic?
#:memory-size memory-size
@@ -936,6 +945,8 @@ FORWARDINGS is a list of host-port/guest-port pairs."
"user,model=virtio-net-pci,"
(port-forwardings->qemu-options forwardings)))))
(system-qemu-image/shared-store-script os
+ #:system system
+ #:target target
#:qemu qemu
#:graphic? graphic?
#:memory-size memory-size
--
2.26.2
From dc3f04a9f412045a85ac810f9436648368c36adc Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Wed, 27 May 2020 23:11:14 +0200
Subject: [PATCH 6/7] vm: Shared-store script runs that native QEMU and Bash.

* gnu/system/vm.scm (system-qemu-image/shared-store-script): Use #+ for
QEMU and BASH.
---
gnu/system/vm.scm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (25 lines)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 05f3986aca..038cce19b6 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -862,7 +862,8 @@ it is mostly useful when FULL-BOOT? is true."
#+@(operating-system-kernel-arguments os "/dev/vda1")))
(define qemu-exec
- #~(list (string-append #$qemu "/bin/" #$(qemu-command (%current-system)))
+ #~(list #+(file-append qemu "/bin/"
+ (qemu-command (or target system)))
#$@(if full-boot?
#~()
#~("-kernel" #$(operating-system-kernel-file os)
@@ -879,7 +880,7 @@ it is mostly useful when FULL-BOOT? is true."
#~(call-with-output-file #$output
(lambda (port)
(format port "#!~a~% exec ~a \"$@\"~%"
- #$(file-append bash "/bin/sh")
+ #+(file-append bash "/bin/sh")
(string-join #$qemu-exec " "))
(chmod port #o555))))
--
2.26.2
From 6e936131ca85aba24f82d35c4616afe835ac7da5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Wed, 27 May 2020 23:57:41 +0200
Subject: [PATCH 7/7] gnu: guile-static: Disable JIT on ARMv7.

* gnu/packages/make-bootstrap.scm (make-guile-static): Pass
"--disable-jit" when 'target-arm32?' is true.
---
gnu/packages/make-bootstrap.scm | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)

Toggle diff (20 lines)
diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index fe86f810bf..b2d3e2a326 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -706,7 +706,12 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
;; When `configure' checks for ltdl availability, it
;; doesn't try to link using libtool, and thus fails
;; because of a missing -ldl. Work around that.
- ''("LDFLAGS=-ldl"))
+
+ ;; XXX: On ARMv7, disable JIT: it causes crashes with 3.0.2,
+ ;; possibly related to <https://bugs.gnu.org/40737>.
+ (if (target-arm32?)
+ ''("LDFLAGS=-ldl" "--disable-jit")
+ ''("LDFLAGS=-ldl")))
((#:phases phases '%standard-phases)
`(modify-phases ,phases
--
2.26.2
M
M
Mathieu Othacehe wrote on 28 May 2020 08:36
(name . Ludovic Courtès)(address . ludo@gnu.org)
87d06o36sc.fsf@gnu.org
Toggle quote (2 lines)
> So it took me the whole day, but I ended up with the following patches

Hey Ludo,

Toggle quote (5 lines)
> addressing these two points. With that on master, I can do:
>
> guix system vm --target=arm-linux-gnueabihf --no-grafts \
> gnu/system/examples/bare-bones.tmpl

Nice! Many thanks for helping us on that :)

Toggle quote (8 lines)
> Likewise, this produces what looks like a valid image, except probably
> for the bootloader (I used GRUB in bare-bones, which didn’t complain,
> but the result doesn’t work):
>
> guix system vm-image --target=arm-linux-gnueabihf …
>
> Thoughts?

qemu-system-arm supports a whole range of machine. You can type:

Toggle snippet (3 lines)
qemu-system-arm -M machine help

for the complete list. Depending on the selected machine, the required
bootloader differs. For the "virt" machine, it seems that we could pass
the bootloader as Qemu "bios" argument[1], but I'm not sure we can do
much better.

Toggle quote (4 lines)
> Yeah, using the new image API will be nicer. We still need to keep some
> of the API in (guix system vm), at least for the ‘guix system vm’
> command.

Yes, you are right.

Toggle quote (5 lines)
> From 6e936131ca85aba24f82d35c4616afe835ac7da5 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Wed, 27 May 2020 23:57:41 +0200
> Subject: [PATCH 7/7] gnu: guile-static: Disable JIT on ARMv7.

The whole serie LGTM. It's nice to have cross-compilation working for
both disk-images and vm-images!

Mathieu

[1]:
M
M
Mathieu Othacehe wrote on 28 May 2020 09:00
(name . Ludovic Courtès)(address . ludo@gnu.org)
878shc35of.fsf@gnu.org
Hey Ludo,

Toggle quote (2 lines)
> Could you boil the problem down to a “guix build --target” command?

No, but I just found a small reproducer:

Toggle snippet (18 lines)
(use-modules (gnu)
(gnu packages gtk)
(guix)
(guix store))

(run-with-store (open-connection)
(mlet* %store-monad ((target (set-current-target "aarch64-linux-gnu"))
(drv (lower-object
(computed-file "test"
#~(begin
(mkdir #$output)
#+guile-rsvg))))
(out -> (derivation->output-path drv)))
(mbegin %store-monad
(built-derivations (list drv))
(return out))))

This should fail with:

Toggle snippet (13 lines)
In guix/packages.scm:
1072:16 5 (package-derivation _ #<package guile-rsvg@2.18.1-0.05…> …)
1393:16 4 (thunk)
1280:6 3 (bag-grafts #<store-connection 256.99 7f484d450780> #<<…>)
1260:45 2 (fold-bag-dependencies #<procedure 7f484b8bd5e0 at gui…> …)
1072:16 1 (package->bag _ _ _ #:graft? _)
1183:21 0 (thunk)

guix/packages.scm:1183:21: In procedure thunk:
ERROR:
1. &package-cross-build-system-error: #<package gobject-introspection@1.62.0 gnu/packages/glib.scm:406 7f48407ec460>

Thanks,

Mathieu
J
J
Jan Nieuwenhuizen wrote on 28 May 2020 14:29
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87o8q8gs3w.fsf@gnu.org
Mathieu Othacehe writes:

Hello!

Toggle quote (4 lines)
>> So it took me the whole day, but I ended up with the following patches
>
> Hey Ludo,

Thanks so much for helping out!

Toggle quote (7 lines)
>> addressing these two points. With that on master, I can do:
>>
>> guix system vm --target=arm-linux-gnueabihf --no-grafts \
>> gnu/system/examples/bare-bones.tmpl
>
> Nice! Many thanks for helping us on that :)

+1

Toggle quote (8 lines)
>> Likewise, this produces what looks like a valid image, except probably
>> for the bootloader (I used GRUB in bare-bones, which didn’t complain,
>> but the result doesn’t work):
>>
>> guix system vm-image --target=arm-linux-gnueabihf …
>>
>> Thoughts?

I have locally rebased wip-hurd-vm and wip-disk-image (working title for
wip-hurd-vm minus vm-image support) on it: it looks nice and works for
me building a disk-impage.

Building a vm-image still needs quite some work: I tried building a
vm-image without hurd-specific vm-image patches (bisecting a bit:
register-closures/grub/qemu-image preserve-target stuff)...but in the
end I still needed all these

Toggle snippet (8 lines)
0547e4c251 * wip-hurd-vm system: vm: Activate the Hurd.
8c28ab91a5 * system: vm: Initial vm-image support for the Hurd.
d9b84d03bc * system: vm: Add defaults for the Hurd.
29281d053f * system: vm: Build vm-image using native qemu, for the Hurd.
86edea90c4 * system: vm: Do not register-closures when cross-building to the Hurd.
a29437ad82 * utils: Move 'reset-timestamps' out of database.

(especially the "using native qemu" patch was greatly reduced now)
as well as --no-grafts to get a vm-image.

So..as discussed on IRC yesterday I'm voting to drop/pause the vm-image
work for the Hurd and reset wip-hurd-vm to my current gitlab
wip-hurd-disk (or possibly push wip-hurd-disk...) WDYT?

Toggle quote (8 lines)
>> From 6e936131ca85aba24f82d35c4616afe835ac7da5 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Wed, 27 May 2020 23:57:41 +0200
>> Subject: [PATCH 7/7] gnu: guile-static: Disable JIT on ARMv7.
>
> The whole serie LGTM. It's nice to have cross-compilation working for
> both disk-images and vm-images!

I agree...although as noted above: how much effort do we want to put in
for the Hurd?

Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 28 May 2020 17:39
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
878shccblp.fsf@gnu.org
Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (5 lines)
> Building a vm-image still needs quite some work: I tried building a
> vm-image without hurd-specific vm-image patches (bisecting a bit:
> register-closures/grub/qemu-image preserve-target stuff)...but in the
> end I still needed all these

How does it fail exactly? (I’ll give it a spin as time permits.)

Toggle quote (4 lines)
> So..as discussed on IRC yesterday I'm voting to drop/pause the vm-image
> work for the Hurd and reset wip-hurd-vm to my current gitlab
> wip-hurd-disk (or possibly push wip-hurd-disk...) WDYT?

Sure, whatever works best!

Toggle quote (11 lines)
>>> From 6e936131ca85aba24f82d35c4616afe835ac7da5 Mon Sep 17 00:00:00 2001
>>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>>> Date: Wed, 27 May 2020 23:57:41 +0200
>>> Subject: [PATCH 7/7] gnu: guile-static: Disable JIT on ARMv7.
>>
>> The whole serie LGTM. It's nice to have cross-compilation working for
>> both disk-images and vm-images!
>
> I agree...although as noted above: how much effort do we want to put in
> for the Hurd?

I was looking at what needs to be fixed there, independently of the
Hurd.

But maybe we can just rebase ‘system-qemu-image’ & co. on top of (gnu
image)? What prevents us from doing that, Mathieu?

If we can do that, then indeed, there’s no point in insisting on fixing
cross-compilation support in (gnu system vm).

Thanks for your feedback!

Ludo’.
J
J
Jan Nieuwenhuizen wrote on 28 May 2020 19:07
(name . Ludovic Courtès)(address . ludo@gnu.org)
87mu5s7ztn.fsf@gnu.org
Ludovic Courtès writes:

Hello!

Toggle quote (9 lines)
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Building a vm-image still needs quite some work: I tried building a
>> vm-image without hurd-specific vm-image patches (bisecting a bit:
>> register-closures/grub/qemu-image preserve-target stuff)...but in the
>> end I still needed all these
>
> How does it fail exactly? (I’ll give it a spin as time permits.)

As discussed on IRC: same errors as before, mostly (sqlite loading,
meson cross build using --no-grafts, linux build for i586-pc-gnu).

Toggle quote (6 lines)
>> So..as discussed on IRC yesterday I'm voting to drop/pause the vm-image
>> work for the Hurd and reset wip-hurd-vm to my current gitlab
>> wip-hurd-disk (or possibly push wip-hurd-disk...) WDYT?
>
> Sure, whatever works best!

Done! I have just reset wip-hurd-vm, let the review+merge begin \o/

Toggle quote (14 lines)
>>>> From 6e936131ca85aba24f82d35c4616afe835ac7da5 Mon Sep 17 00:00:00 2001
>>>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>>>> Date: Wed, 27 May 2020 23:57:41 +0200
>>>> Subject: [PATCH 7/7] gnu: guile-static: Disable JIT on ARMv7.
>>>
>>> The whole serie LGTM. It's nice to have cross-compilation working for
>>> both disk-images and vm-images!
>>
>> I agree...although as noted above: how much effort do we want to put in
>> for the Hurd?
>
> I was looking at what needs to be fixed there, independently of the
> Hurd.

Yes, so I kept your commits at the bottom of the new wip-hurd-vm, they
are clean and there's some value in them, I think.

Toggle quote (3 lines)
> But maybe we can just rebase ‘system-qemu-image’ & co. on top of (gnu
> image)? What prevents us from doing that, Mathieu?

+1

Toggle quote (3 lines)
> If we can do that, then indeed, there’s no point in insisting on fixing
> cross-compilation support in (gnu system vm).

possibly, modulo s/no/litte/ s/point/urgency/
but yeah. A bit sad, between us we put quite some effort into that...

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
M
M
Mathieu Othacehe wrote on 28 May 2020 19:10
(name . Ludovic Courtès)(address . ludo@gnu.org)
87o8q8yohx.fsf@gnu.org
Hey Ludo & Jan,

Toggle quote (6 lines)
> But maybe we can just rebase ‘system-qemu-image’ & co. on top of (gnu
> image)? What prevents us from doing that, Mathieu?
>
> If we can do that, then indeed, there’s no point in insisting on fixing
> cross-compilation support in (gnu system vm).

I think we could proceed that way:

* Merge Ludo's serie on master.

* Then we could review & merge Jan's wip-hurd-disk.

* Finally, there's the wip-hurd-vm. I believe that some of stuff there
is fixed in Ludo's serie. So we would need to rebase it on top of
everything & finally push it.

Do you think that makes sense?

Thanks,

Mathieu
J
J
Jan Nieuwenhuizen wrote on 28 May 2020 20:19
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87sgfj6hy3.fsf@gnu.org
Mathieu Othacehe writes:

Hi Mathieu, Ludo

Toggle quote (14 lines)
> Hey Ludo & Jan,
>
>> But maybe we can just rebase ‘system-qemu-image’ & co. on top of (gnu
>> image)? What prevents us from doing that, Mathieu?
>>
>> If we can do that, then indeed, there’s no point in insisting on fixing
>> cross-compilation support in (gnu system vm).
>
> I think we could proceed that way:
>
> * Merge Ludo's serie on master.
>
> * Then we could review & merge Jan's wip-hurd-disk.

Oops, you sent your mail just a few minutes before mine; just reading it
now :-)

I push a fresh wip-hurd-vm, which is the above: wip-hurd-disk rebase
onto Ludo's series...almost the same.

Toggle quote (6 lines)
> * Finally, there's the wip-hurd-vm. I believe that some of stuff there
> is fixed in Ludo's serie. So we would need to rebase it on top of
> everything & finally push it.
>
> Do you think that makes sense?

Yes, I think this is more or less my idea too and what I did...except
for leaving my hurd-specific qemu-image/vm-image work rest.

Hope that's OK!
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 29 May 2020 10:18
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87o8q75f2r.fsf@gnu.org
Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

Toggle quote (10 lines)
>> But maybe we can just rebase ‘system-qemu-image’ & co. on top of (gnu
>> image)? What prevents us from doing that, Mathieu?
>>
>> If we can do that, then indeed, there’s no point in insisting on fixing
>> cross-compilation support in (gnu system vm).
>
> I think we could proceed that way:
>
> * Merge Ludo's serie on master.

I think that can wait because on IRC Janneke explained that it doesn’t
fix anything for GNU/Hurd (to my surprise). So I’ll maybe check again
once the relevant Hurd bits are on master, instead of checking ARM
cross-compilation.

Anyway it’s much less important now that (gnu image) can be used for the
task!

Toggle quote (2 lines)
> * Then we could review & merge Jan's wip-hurd-disk.

Do I get it right that we first need

The ‘wip-hurd-vm’ branch contains many things:

1. (gnu system hurd) with the Hurd services etc.

2. The ‘hurd’ field of <operating-system>.

3. <menu-entry> with multiboot support.

4. Hacks to work around vm.scm defects: uses of ‘with-parameters’,
‘hurd-target?’, disabling sqlite3, and #~#$ tricks.

I think part of the reason this cycle has been so long is that it’s been
kind of a big bang; big bangs are great because they lead to something
new and exciting, but they’re also intimidating. :-) For me personally,
looking at all these aspects at once was just too much.

For merging, I think it’d be great to see #1 and #2 as a first step, and
then #3.

I do not want any of #4 :-), because I really think it could lead to
maintenance headaches down the road, which would make the kind of
changes we’re making today practically impossible in the future.

Thoughts?

Ludo’.
J
J
Jan Nieuwenhuizen wrote on 29 May 2020 11:06
(name . Ludovic Courtès)(address . ludo@gnu.org)
871rn3rtza.fsf@gnu.org
Ludovic Courtès writes:

Toggle quote (17 lines)
> Hi,
>
> Mathieu Othacehe <othacehe@gnu.org> skribis:
>
>>> But maybe we can just rebase ‘system-qemu-image’ & co. on top of (gnu
>>> image)? What prevents us from doing that, Mathieu?
>>>
>>> If we can do that, then indeed, there’s no point in insisting on fixing
>>> cross-compilation support in (gnu system vm).
>>
>> I think we could proceed that way:
>>
>> * Merge Ludo's serie on master.
>
> I think that can wait because on IRC Janneke explained that it doesn’t
> fix anything for GNU/Hurd (to my surprise).

Well, that's too black/white. Of course this:

- (linux (string-append #$linux "/"
- #$(system-linux-image-file-name)))
- (initrd #$initrd)
- (loader #$loader)
+ (linux (string-append
+ #+linux "/"
+ #+(system-linux-image-file-name system)))
+ (initrd #+initrd)
+ (loader #+loader)

is an essential fix. Only, that used to be on wip-hurd-vm -- so from
that perspective this fix only got reviewed and moved.

Pretty similarly for this

+ (define preserve-target
+ (if target
+ (lambda (obj)
+ (with-parameters ((%current-target-system target))
+ obj))
+ identity))

also an essential bit. So, seen from master it fixes quite some bits!
What I meant to say is that I haven't seen /additional/ fixes to what
we had on wip-hurd-vm...

Because your fixes look clean and don't break anything, I think we
should maybe apply them to master -- and that's why I kept them at
the bottom of "wip-hurd-vm" for now.

Toggle quote (6 lines)
> So I’ll maybe check again once the relevant Hurd bits are on master,
> instead of checking ARM cross-compilation.
>
> Anyway it’s much less important now that (gnu image) can be used for the
> task!

Yes, so from the Hurd perspective this patch set has shifted from
top-priority to probably not going to be used for the Hurd ;)

Toggle quote (13 lines)
>> * Then we could review & merge Jan's wip-hurd-disk.
>
> Do I get it right that we first need
> <https://issues.guix.gnu.org/41560>?
>
> The ‘wip-hurd-vm’ branch contains many things:
>
> 1. (gnu system hurd) with the Hurd services etc.
>
> 2. The ‘hurd’ field of <operating-system>.
>
> 3. <menu-entry> with multiboot support.

Yes.

Toggle quote (2 lines)
> 4. Hacks to work around vm.scm defects: uses of ‘with-parameters’,

Together with dropping qemu-image support for the Hurd, I have removed
most of this; what remains is in

Toggle snippet (14 lines)
system: Add 'multiboot-modules' field to <boot-parameters>.
+ (mach (if (%current-target-system)
+ ;; A cross-built GNUmach does not work
+ (with-parameters ((%current-system "i686-linux")
+ (%current-target-system #f))
+ mach)
+ mach)))
[..]
+ (libc (if target
+ (with-parameters ((%current-target-system #f))
+ (cross-libc target))
+ glibc))

Toggle quote (2 lines)
> ‘hurd-target?’, disabling sqlite3, and #~#$ tricks.

I think this is mostly gone, two instances of hurd-target? remain. I'll
look more sharply at that too!

Toggle quote (5 lines)
> I think part of the reason this cycle has been so long is that it’s been
> kind of a big bang; big bangs are great because they lead to something
> new and exciting, but they’re also intimidating. :-) For me personally,
> looking at all these aspects at once was just too much.

Yes...

Toggle quote (3 lines)
> For merging, I think it’d be great to see #1 and #2 as a first step, and
> then #3.

Yes, that would be an amazing feat.

Toggle quote (6 lines)
> I do not want any of #4 :-), because I really think it could lead to
> maintenance headaches down the road, which would make the kind of
> changes we’re making today practically impossible in the future.

> Thoughts?

I fully agree, thanks!

Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 30 May 2020 12:08
(name . Ludovic Courtès)(address . ludo@gnu.org)
87d06ld9bj.fsf@gnu.org
Jan Nieuwenhuizen writes:

Hi!

A quick headsup for /boot/activation; I have found how to do without
/boot/activation, using the regular (I think) path using kernel
arguments --system=, --load=, and system/boot.

I'll be pushing a couple of squash commits onto wip-hurd-vm for
that.

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 30 May 2020 15:54
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)
87r1v1v88j.fsf@gnu.org
Hi,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (4 lines)
> A quick headsup for /boot/activation; I have found how to do without
> /boot/activation, using the regular (I think) path using kernel
> arguments --system=, --load=, and system/boot.

Yay!

Ludo’.
M
M
Maxim Cournoyer wrote on 28 Sep 2022 22:18
Re: bug#41350: [PATCH 0/3] Use native qemu to build vm-image.
87leq3s163.fsf_-_@gmail.com
Hello,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (8 lines)
> Hi,
>
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> A quick headsup for /boot/activation; I have found how to do without
>> /boot/activation, using the regular (I think) path using kernel
>> arguments --system=, --load=, and system/boot.

I was trying to see if there were any pending changes not yet committed
to our tree in this interesting but long thread? Please help me out by
closing it if not, or pointing which changes remain :-).

Thanks,

Maxim
M
M
Mathieu Othacehe wrote on 29 Sep 2022 16:17
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87sfkagt93.fsf@gnu.org
Hey Maxim,

Building 'vm-image' does not involve QEMU anymore so we can close this
one.

Thanks,

Mathieu
Closed
?