[PATCH] Test and fix the ISO Image Installer

Done

Details

3 participants

Danny Milosavljevic
Ludovic Courtès
Christopher Baines

Owner: unassigned

Submitted by: Christopher Baines

Severity: important

Christopher Baines wrote on 30 Aug 2017 08:17

Recipients:(address . guix-patches@gnu.org)

Message-ID:20170830071714.75f88e47@cbaines.net

Hey,

So at the GNU Hackers Meeting, I attempted to build the installer system

as an ISO image, and test installing Guix on a Bytemark VM with this. I

encountered problems, and therefore set about writing a system test to

see if this could be reproduced.

The failures did seem reproducible locally, and Ludo helped with some

code changes. I'll send the work in progress patches to this bug.

Christopher Baines wrote on 30 Aug 2017 08:20

[PATCH 1/2] tests: Add test for installing from an ISO Image.

Recipients:(address . 28288@debbugs.gnu.org)

Message-ID:20170830062010.11902-1-mail@cbaines.net

* gnu/tests/install.scm (%test-iso-image-installer): New variable.
  (run-install): Add #:installation-disk-image-file-system-type as a keyword
  argument.
---
 gnu/tests/install.scm | 48 +++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 37 insertions(+), 11 deletions(-)

Toggle diff (89 lines)diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 866bf885c..7596347a3 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -37,6 +37,7 @@
   #:use-module (guix utils)
   #:export (%test-installed-os
             %test-installed-extlinux-os
+            %test-iso-image-installer
             %test-separate-store-os
             %test-separate-home-os
             %test-raid-root-os
@@ -196,6 +197,7 @@ reboot\n")
                              (kernel-arguments '("console=ttyS0")))
                            #:imported-modules '((gnu services herd)
                                                 (guix combinators))))
+                      (installation-disk-image-file-system-type "ext4")
                       (target-size (* 1200 MiB)))
   "Run SCRIPT (a shell script following the GuixSD installation procedure) in
 OS to install TARGET-OS.  Return a VM image of TARGET-SIZE bytes containing
@@ -213,7 +215,9 @@ packages defined in installation-os."
                        (image  (system-disk-image
                                 (operating-system-with-gc-roots
                                  os (list target))
-                                #:disk-image-size (* 1500 MiB))))
+                                #:disk-image-size (* 1500 MiB)
+                                #:file-system-type
+                                installation-disk-image-file-system-type)))
     (define install
       (with-imported-modules '((guix build utils)
                                (gnu build marionette))
@@ -229,16 +233,24 @@ packages defined in installation-os."
 
             (define marionette
               (make-marionette
-               (cons (which #$(qemu-command system))
-                     (cons* "-no-reboot" "-m" "800"
-                            "-drive"
-                            (string-append "file=" #$image
-                                           ",if=virtio,readonly")
-                            "-drive"
-                            (string-append "file=" #$output ",if=virtio")
-                            (if (file-exists? "/dev/kvm")
-                                '("-enable-kvm")
-                                '())))))
+               `(,(which #$(qemu-command system))
+                 "-no-reboot"
+                 "-m" "800"
+                 ,@(case installation-disk-image-file-system-type
+                     (("ext4")
+                      `("-drive"
+                        ,(string-append "file=" #$image
+                                        ",if=virtio,readonly")))
+                     (("iso9660") '("-cdrom" #$image))
+                     (else
+                      (error
+                       "unsupported installation-disk-image-file-system-type"
+                       installation-disk-image-file-system-type)))
+                 "-drive"
+                 ,(string-append "file=" #$output ",if=virtio")
+                 ,@(if (file-exists? "/dev/kvm")
+                       '("-enable-kvm")
+                       '()))))
 
             (pk 'uname (marionette-eval '(uname) marionette))
 
@@ -312,6 +324,20 @@ per %test-installed-os, this test is expensive in terms of CPU and storage.")
       (run-basic-test %minimal-extlinux-os command
                       "installed-extlinux-os")))))
 
+(define %test-iso-image-installer
+  (system-test
+   (name "iso-image-installer")
+   (description
+    "")
+   (value
+    (mlet* %store-monad ((image   (run-install
+                                   %minimal-os
+                                   %minimal-os-source
+                                   #:installation-disk-image-file-system-type
+                                   "iso9660"))
+                         (command (qemu-command/writable-image image)))
+      (run-basic-test %minimal-os command name)))))
+
 
 ;;;
 ;;; Separate /home.
-- 
2.14.1

Christopher Baines wrote on 30 Aug 2017 08:20

[PATCH 2/2] WIP

Recipients:(address . 28288@debbugs.gnu.org)

Message-ID:20170830062010.11902-2-mail@cbaines.net

---

gnu/build/vm.scm | 19 +++++++++++++++++--

gnu/system/vm.scm | 16 +++++++++++-----

2 files changed, 28 insertions(+), 7 deletions(-)

Toggle diff (92 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 727494ad9..e2659cb5e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -366,12 +366,27 @@ SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
       (error "failed to create GRUB EFI image"))))
 
 (define* (make-iso9660-image grub config-file os-drv target
-                             #:key (volume-id "GuixSD_image") (volume-uuid #f))
+                             #:key (volume-id "GuixSD_image") (volume-uuid #f)
+                             register-closures? (closures '()))
   "Given a GRUB package, creates an iso image as TARGET, using CONFIG-FILE as
 GRUB configuration and OS-DRV as the stuff in it."
-  (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue")))
+  (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue"))
+        (target-store  (string-append "/tmp/root" (%store-directory))))
     (mkdir-p "/tmp/root/var/run")
     (mkdir-p "/tmp/root/run")
+
+    ;; XXX: 'guix-register' wants to palpate the things it registers, so
+    ;; bind-mount the store on the target.
+    (mkdir-p target-store)
+    (mount (%store-directory) target-store "" MS_BIND)
+
+    (when register-closures?
+      (display "registering closures...\n")
+      (for-each (lambda (closure)
+                  (register-closure target
+                                    (string-append "/xchg/" closure)))
+                closures))
+
     (unless (zero? (apply system*
                           `(,grub-mkrescue "-o" ,target
                             ,(string-append "boot/grub/grub.cfg=" config-file)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4494af003..b106dff0a 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -192,6 +192,7 @@ made available under the /xchg CIFS share."
                         os-drv
                         bootcfg-drv
                         bootloader
+                        register-closures?
                         (inputs '()))
   "Return a bootable, stand-alone iso9660 image.
 
@@ -207,8 +208,13 @@ INPUTS is a list of inputs (as for packages)."
          (let ((inputs
                 '#$(append (list qemu parted e2fsprogs dosfstools xorriso)
                            (map canonical-package
-                                (list sed grep coreutils findutils gawk))))
+                                (list sed grep coreutils findutils gawk))
+                           (if register-closures? (list guix) '())))
 
+
+               (graphs     '#$(match inputs
+                                   (((names . _) ...)
+                                    names)))
                ;; This variable is unused but allows us to add INPUTS-TO-COPY
                ;; as inputs.
                (to-register
@@ -222,6 +228,8 @@ INPUTS is a list of inputs (as for packages)."
                                #$bootcfg-drv
                                #$os-drv
                                "/xchg/guixsd.iso"
+                               #:register-closures? #$register-closures?
+                               #:closures graphs
                                #:volume-id #$file-system-label
                                #:volume-uuid #$file-system-uuid)
            (reboot))))
@@ -391,6 +399,7 @@ to USB sticks meant to be read-only."
                          #:file-system-label root-label
                          #:file-system-uuid #f
                          #:os-drv os-drv
+                         #:register-closures? #t
                          #:bootcfg-drv bootcfg
                          #:bootloader (bootloader-configuration-bootloader
                                         (operating-system-bootloader os))
@@ -403,10 +412,7 @@ to USB sticks meant to be read-only."
                                     (operating-system-bootloader os))
                       #:disk-image-size disk-image-size
                       #:disk-image-format "raw"
-                      #:file-system-type (if (string=? "iso9660"
-                                                       file-system-type)
-                                             "ext4"
-                                             file-system-type)
+                      #:file-system-type file-system-type
                       #:file-system-label root-label
                       #:copy-inputs? #t
                       #:register-closures? #t
-- 
2.14.1

Christopher Baines wrote on 30 Aug 2017 08:31

Recipients:(address . 28288@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20170830073113.426bf8d3@cbaines.net

Currently, I get the following error when running this test.

registering closures...

error: initialising database schema: disk I/O error

ERROR: In procedure scm-error:

ERROR: failed to register store items "/xchg/system"

I also need the patch from

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28273otherwise I can't

even run the test.

Danny, I guess you have been looking at installing from an ISO image as

well, as I spotted you added it to the release target. Have you been

able to successfully use it yet?

When attempting to use it initially, I encountered a problem where a

glibc in the store was partially deleted. /var/guix/db/db.sqlite also

initially didn't exist, which the WIP patch here attempts to address.

Danny Milosavljevic wrote on 30 Aug 2017 10:38

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 28288@debbugs.gnu.org)

Message-ID:20170830103800.09ea6e31@scratchpost.org

Hi Christopher,

On Wed, 30 Aug 2017 07:31:13 +0100

Christopher Baines <mail@cbaines.net> wrote:

Toggle quote (16 lines)> Currently, I get the following error when running this test.
> 
> registering closures...
> error: initialising database schema: disk I/O error
> ERROR: In procedure scm-error:
> ERROR: failed to register store items "/xchg/system"
> 
> 
> I also need the patch from
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28273 otherwise I can't
> even run the test.
> 
> Danny, I guess you have been looking at installing from an ISO image as
> well, as I spotted you added it to the release target. Have you been
> able to successfully use it yet?

Does that mean with the wip-installer-2 branch?

Otherwise in master I always tested it using gnu/system/install.scm until it has booted up successfully, which it did.

I've asked about the installation part before but there was no response.

Toggle quote (4 lines)

> When attempting to use it initially, I encountered a problem where a

> glibc in the store was partially deleted. /var/guix/db/db.sqlite also

> initially didn't exist, which the WIP patch here attempts to address.

Hmm, do you have logs of the release creation process? Maybe xorriso complained?

Danny Milosavljevic wrote on 30 Aug 2017 20:26

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 28288@debbugs.gnu.org)

Message-ID:20170830202637.6b6ace8b@scratchpost.org

Toggle quote (5 lines)

> + ;; XXX: 'guix-register' wants to palpate the things it registers, so

> + ;; bind-mount the store on the target.

> + (mkdir-p target-store)

> + (mount (%store-directory) target-store "" MS_BIND)

What does this do?

gnu/build/vm.scm make-iso9660-image already provides /gnu/store a few lines down:

(unless (zero? (apply system*

`(,grub-mkrescue "-o" ,target

,(string-append "boot/grub/grub.cfg=" config-file)

,(string-append "gnu/store=" os-drv "/..")

^^^^

Maybe I'm missing something...

Otherwise LGTM!

Christopher Baines wrote on 31 Aug 2017 00:56

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:20170830235647.1def024c@cbaines.net

On Wed, 30 Aug 2017 10:38:00 +0200

Danny Milosavljevic <dannym@scratchpost.org> wrote:

Toggle quote (23 lines)> Hi Christopher,
> 
> On Wed, 30 Aug 2017 07:31:13 +0100
> Christopher Baines <mail@cbaines.net> wrote:
> 
> > Currently, I get the following error when running this test.
> > 
> > registering closures...
> > error: initialising database schema: disk I/O error
> > ERROR: In procedure scm-error:
> > ERROR: failed to register store items "/xchg/system"
> > 
> > 
> > I also need the patch from
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28273 otherwise I
> > can't even run the test.
> > 
> > Danny, I guess you have been looking at installing from an ISO
> > image as well, as I spotted you added it to the release target.
> > Have you been able to successfully use it yet?  
> 
> Does that mean with the wip-installer-2 branch?

I've just been testing with the master branch, and with the changes I

sent to this bug. Are there any changes on the wip-installer-2 that are

not solely related to the installer (which if I understand correctly is

something more complicated than running guix system init).

Toggle quote (6 lines)

> Otherwise in master I always tested it using gnu/system/install.scm

> until it has booted up successfully, which it did.

> I've asked about the installation part before but there was no

> response.

Ok, that's good to know. I would guess it's pretty close now, great

work on moving the ISO stuff forward!

Toggle quote (8 lines)

> > When attempting to use it initially, I encountered a problem where a

> > glibc in the store was partially deleted. /var/guix/db/db.sqlite

> > also initially didn't exist, which the WIP patch here attempts to

> > address.

> Hmm, do you have logs of the release creation process? Maybe xorriso

> complained?

From talking to Ludo at the GHM, I think there is some extra steps
needed to make installing from the ISO possible, but I don't yet
understand this. I'll CC Ludo, as he might be able to explain better.

Ludovic Courtès wrote on 31 Aug 2017 12:27

control message for bug #28288

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87val4uj4o.fsf@gnu.org

severity 28288 important

Danny Milosavljevic wrote on 31 Aug 2017 15:04

Re: [bug#28288] [PATCH 2/2] WIP

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:20170831150448.41b0f41e@scratchpost.org

On Wed, 30 Aug 2017 23:56:47 +0100

Christopher Baines <mail@cbaines.net> wrote:

Toggle quote (4 lines)

> I've just been testing with the master branch, and with the changes I

> sent to this bug. Are there any changes on the wip-installer-2 that are

> not solely related to the installer

Toggle quote (2 lines)

>(which if I understand correctly is something more complicated than running guix system init).

Yes

Christopher Baines wrote on 31 Aug 2017 23:47

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 28288@debbugs.gnu.org)

Message-ID:20170831224728.59e79444@cbaines.net

On Thu, 31 Aug 2017 15:04:48 +0200

Danny Milosavljevic <dannym@scratchpost.org> wrote:

Toggle quote (14 lines)> On Wed, 30 Aug 2017 23:56:47 +0100
> Christopher Baines <mail@cbaines.net> wrote:
> 
> > I've just been testing with the master branch, and with the changes
> > I sent to this bug. Are there any changes on the wip-installer-2
> > that are not solely related to the installer   
> 
> No
> 
> >(which if I understand correctly is something more complicated than
> >running guix system init).  
> 
> Yes

Great, ok, good to know I haven't missed anything.

I'll hopefully be able to make some time soonish to take a look at

getting the ISO image in to a state where running guix system init from

it works, unless someone beats me to it!

Christopher Baines wrote on 3 Sep 2017 12:50

[PATCH 3/5] vm: Add support for registering closures to iso9660-image.

Recipients:(address . 28288@debbugs.gnu.org)

Message-ID:20170903105041.2925-3-mail@cbaines.net

* gnu/system/vm.scm (iso9660-image): Add support for registering closures.

---

gnu/system/vm.scm | 10 +++++++++-

1 file changed, 9 insertions(+), 1 deletion(-)

Toggle diff (38 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4494af003..fc55935aa 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -192,6 +192,7 @@ made available under the /xchg CIFS share."
                         os-drv
                         bootcfg-drv
                         bootloader
+                        register-closures?
                         (inputs '()))
   "Return a bootable, stand-alone iso9660 image.
 
@@ -207,8 +208,13 @@ INPUTS is a list of inputs (as for packages)."
          (let ((inputs
                 '#$(append (list qemu parted e2fsprogs dosfstools xorriso)
                            (map canonical-package
-                                (list sed grep coreutils findutils gawk))))
+                                (list sed grep coreutils findutils gawk))
+                           (if register-closures? (list guix) '())))
 
+
+               (graphs     '#$(match inputs
+                                   (((names . _) ...)
+                                    names)))
                ;; This variable is unused but allows us to add INPUTS-TO-COPY
                ;; as inputs.
                (to-register
@@ -222,6 +228,8 @@ INPUTS is a list of inputs (as for packages)."
                                #$bootcfg-drv
                                #$os-drv
                                "/xchg/guixsd.iso"
+                               #:register-closures? #$register-closures?
+                               #:closures graphs
                                #:volume-id #$file-system-label
                                #:volume-uuid #$file-system-uuid)
            (reboot))))
-- 
2.14.1

Christopher Baines wrote on 3 Sep 2017 12:50

[PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

Recipients:(address . 28288@debbugs.gnu.org)

Message-ID:20170903105041.2925-1-mail@cbaines.net

This mimics the functionality in the root-partition-initializer used in

creating the QEMU image. This helps when trying to run guix system init from

the generated ISO image.

* gnu/build/vm.scm (make-iso9660-image): Add support for registering closures.

---

gnu/build/vm.scm | 19 +++++++++++++++++--

1 file changed, 17 insertions(+), 2 deletions(-)

Toggle diff (36 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 727494ad9..606257d8c 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -366,12 +366,27 @@ SYSTEM-DIRECTORY is the name of the directory of the 'system' derivation."
       (error "failed to create GRUB EFI image"))))
 
 (define* (make-iso9660-image grub config-file os-drv target
-                             #:key (volume-id "GuixSD_image") (volume-uuid #f))
+                             #:key (volume-id "GuixSD_image") (volume-uuid #f)
+                             register-closures? (closures '()))
   "Given a GRUB package, creates an iso image as TARGET, using CONFIG-FILE as
 GRUB configuration and OS-DRV as the stuff in it."
-  (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue")))
+  (let ((grub-mkrescue (string-append grub "/bin/grub-mkrescue"))
+        (target-store  (string-append "/tmp/root" (%store-directory))))
     (mkdir-p "/tmp/root/var/run")
     (mkdir-p "/tmp/root/run")
+
+    (mkdir-p target-store)
+    (mount (%store-directory) target-store "" MS_BIND)
+
+    (when register-closures?
+      (display "registering closures...\n")
+      (for-each (lambda (closure)
+                  (register-closure
+                   "/tmp/root"
+                   (string-append "/xchg/" closure)
+                   #:deduplicate? #f))
+                closures))
+
     (unless (zero? (apply system*
                           `(,grub-mkrescue "-o" ,target
                             ,(string-append "boot/grub/grub.cfg=" config-file)
-- 
2.14.1

Christopher Baines wrote on 3 Sep 2017 12:50

[PATCH 2/5] vm: Create /mnt in the generated ISO image in make-iso9660-image.

Recipients:(address . 28288@debbugs.gnu.org)

Message-ID:20170903105041.2925-2-mail@cbaines.net

This is used in the installation process, as the mountpoint for the target

filesystem.

* gnu/build/vm.scm (make-iso9660-image): Create /mnt within the generated ISO

image.

---

gnu/build/vm.scm | 5 +++++

1 file changed, 5 insertions(+)

Toggle diff (25 lines)diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index 606257d8c..f6228b40b 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -374,6 +374,7 @@ GRUB configuration and OS-DRV as the stuff in it."
         (target-store  (string-append "/tmp/root" (%store-directory))))
     (mkdir-p "/tmp/root/var/run")
     (mkdir-p "/tmp/root/run")
+    (mkdir-p "/tmp/root/mnt")
 
     (mkdir-p target-store)
     (mount (%store-directory) target-store "" MS_BIND)
@@ -393,6 +394,10 @@ GRUB configuration and OS-DRV as the stuff in it."
                             ,(string-append "gnu/store=" os-drv "/..")
                             "var=/tmp/root/var"
                             "run=/tmp/root/run"
+                            ;; /mnt is used as part of the installation
+                            ;; process, as the mount point for the target
+                            ;; filesystem, so create it.
+                            "mnt=/tmp/root/mnt"
                             "--"
                             ;; Store two copies of the headers.
                             ;; The resulting ISO-9660 image has a DOS MBR and
-- 
2.14.1

Christopher Baines wrote on 3 Sep 2017 12:50

[PATCH 5/5] tests: Add test for installing from an ISO Image.

Recipients:(address . 28288@debbugs.gnu.org)

Message-ID:20170903105041.2925-5-mail@cbaines.net

* gnu/tests/install.scm (%test-iso-image-installer): New variable.
  (run-install): Add #:installation-disk-image-file-system-type as a keyword
  argument.
---
 gnu/tests/install.scm | 110 +++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 99 insertions(+), 11 deletions(-)

Toggle diff (151 lines)diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 866bf885c..93c8a89cd 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -37,6 +37,7 @@
   #:use-module (guix utils)
   #:export (%test-installed-os
             %test-installed-extlinux-os
+            %test-iso-image-installer
             %test-separate-store-os
             %test-separate-home-os
             %test-raid-root-os
@@ -196,6 +197,7 @@ reboot\n")
                              (kernel-arguments '("console=ttyS0")))
                            #:imported-modules '((gnu services herd)
                                                 (guix combinators))))
+                      (installation-disk-image-file-system-type "ext4")
                       (target-size (* 1200 MiB)))
   "Run SCRIPT (a shell script following the GuixSD installation procedure) in
 OS to install TARGET-OS.  Return a VM image of TARGET-SIZE bytes containing
@@ -213,7 +215,9 @@ packages defined in installation-os."
                        (image  (system-disk-image
                                 (operating-system-with-gc-roots
                                  os (list target))
-                                #:disk-image-size (* 1500 MiB))))
+                                #:disk-image-size (* 1500 MiB)
+                                #:file-system-type
+                                installation-disk-image-file-system-type)))
     (define install
       (with-imported-modules '((guix build utils)
                                (gnu build marionette))
@@ -229,16 +233,25 @@ packages defined in installation-os."
 
             (define marionette
               (make-marionette
-               (cons (which #$(qemu-command system))
-                     (cons* "-no-reboot" "-m" "800"
-                            "-drive"
-                            (string-append "file=" #$image
-                                           ",if=virtio,readonly")
-                            "-drive"
-                            (string-append "file=" #$output ",if=virtio")
-                            (if (file-exists? "/dev/kvm")
-                                '("-enable-kvm")
-                                '())))))
+               `(,(which #$(qemu-command system))
+                 "-no-reboot"
+                 "-m" "800"
+                 #$@(cond
+                     ((string=? "ext4" installation-disk-image-file-system-type)
+                      `("-drive"
+                        ,(file-append "file=" image
+                                      ",if=virtio,readonly")))
+                     ((string=? "iso9660" installation-disk-image-file-system-type)
+                      `("-cdrom" ,image))
+                     (else
+                      (error
+                       "unsupported installation-disk-image-file-system-type:"
+                       installation-disk-image-file-system-type)))
+                 "-drive"
+                 ,(string-append "file=" #$output ",if=virtio")
+                 ,@(if (file-exists? "/dev/kvm")
+                       '("-enable-kvm")
+                       '()))))
 
             (pk 'uname (marionette-eval '(uname) marionette))
 
@@ -313,6 +326,81 @@ per %test-installed-os, this test is expensive in terms of CPU and storage.")
                       "installed-extlinux-os")))))
 
 
+;;;
+;;; Installation through an ISO image.
+;;;
+
+(define-os-with-source (%minimal-os-on-vda %minimal-os-on-vda-source)
+  ;; The OS we want to install.
+  (use-modules (gnu) (gnu tests) (srfi srfi-1))
+
+  (operating-system
+    (host-name "liberigilo")
+    (timezone "Europe/Paris")
+    (locale "en_US.UTF-8")
+
+    (bootloader (grub-configuration (target "/dev/vda")))
+    (kernel-arguments '("console=ttyS0"))
+    (file-systems (cons (file-system
+                          (device "my-root")
+                          (title 'label)
+                          (mount-point "/")
+                          (type "ext4"))
+                        %base-file-systems))
+    (users (cons (user-account
+                  (name "alice")
+                  (comment "Bob's sister")
+                  (group "users")
+                  (supplementary-groups '("wheel" "audio" "video"))
+                  (home-directory "/home/alice"))
+                 %base-user-accounts))
+    (services (cons (service marionette-service-type
+                             (marionette-configuration
+                              (imported-modules '((gnu services herd)
+                                                  (guix combinators)))))
+                    %base-services))))
+
+(define %simple-installation-script-for-/dev/vda
+  ;; Shell script of a simple installation.
+  "\
+. /etc/profile
+set -e -x
+guix --version
+
+export GUIX_BUILD_OPTIONS=--no-grafts
+guix build isc-dhcp
+parted --script /dev/vda mklabel gpt \\
+  mkpart primary ext2 1M 3M \\
+  mkpart primary ext2 3M 1G \\
+  set 1 boot on \\
+  set 1 bios_grub on
+mkfs.ext4 -L my-root /dev/vda2
+mount /dev/vda2 /mnt
+df -h /mnt
+herd start cow-store /mnt
+mkdir /mnt/etc
+cp /etc/target-config.scm /mnt/etc/config.scm
+guix system init /mnt/etc/config.scm /mnt --no-substitutes
+sync
+reboot\n")
+
+(define %test-iso-image-installer
+  (system-test
+   (name "iso-image-installer")
+   (description
+    "")
+   (value
+    (mlet* %store-monad ((image   (run-install
+                                   %minimal-os-on-vda
+                                   %minimal-os-on-vda-source
+                                   #:script
+                                   %simple-installation-script-for-/dev/vda
+                                   #:installation-disk-image-file-system-type
+                                   "iso9660"))
+                         (command (qemu-command/writable-image image)))
+      (run-basic-test %minimal-os-on-vda command name)))))
+
+
 ;;;
 ;;; Separate /home.
 ;;;
-- 
2.14.1

Christopher Baines wrote on 3 Sep 2017 12:50

[PATCH 4/5] vm: Call iso9660-image with #:register-closures? as #t.

Recipients:(address . 28288@debbugs.gnu.org)

Message-ID:20170903105041.2925-4-mail@cbaines.net

* gnu/system/vm.scm (system-disk-image): Call iso9660-image with
  #:register-closures? as #t.
---
 gnu/system/vm.scm | 1 +
 1 file changed, 1 insertion(+)

Toggle diff (14 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index fc55935aa..9cd973385 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -399,6 +399,7 @@ to USB sticks meant to be read-only."
                          #:file-system-label root-label
                          #:file-system-uuid #f
                          #:os-drv os-drv
+                         #:register-closures? #t
                          #:bootcfg-drv bootcfg
                          #:bootloader (bootloader-configuration-bootloader
                                         (operating-system-bootloader os))
-- 
2.14.1

Christopher Baines wrote on 3 Sep 2017 12:58

Re: [bug#28288] [PATCH 5/5] tests: Add test for installing from an ISO Image.

Recipients:(address . 28288@debbugs.gnu.org)

Message-ID:20170903115838.0bc74483@cbaines.net

On Sun, 3 Sep 2017 11:50:41 +0100

Christopher Baines <mail@cbaines.net> wrote:

Toggle quote (8 lines)

> * gnu/tests/install.scm (%test-iso-image-installer): New variable.

> (run-install): Add #:installation-disk-image-file-system-type as a

> keyword argument.

> ---

> gnu/tests/install.scm | 110

> +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 99

> insertions(+), 11 deletions(-)

This test now passes! I've also successfully generated an ISO image

that I used to install GuixSD to a Bytemark VM [1].

Most of the changes are educated guesses, mostly from copying the code

relating to QEMU images, so it would good to get some review of these.

1: https://www.bytemark.co.uk/cloud-hosting/

Danny Milosavljevic wrote on 5 Sep 2017 14:56

Re: [bug#28288] [PATCH 4/5] vm: Call iso9660-image with #:register-closures? as #t.

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 28288@debbugs.gnu.org)

Message-ID:20170905145654.507c1c51@scratchpost.org

LGTM!

Danny Milosavljevic wrote on 5 Sep 2017 14:57

Re: [bug#28288] [PATCH 3/5] vm: Add support for registering closures to iso9660-image.

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 28288@debbugs.gnu.org)

Message-ID:20170905145736.3956109f@scratchpost.org

LGTM!

Danny Milosavljevic wrote on 5 Sep 2017 14:58

Re: [bug#28288] [PATCH 2/5] vm: Create /mnt in the generated ISO image in make-iso9660-image.

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 28288@debbugs.gnu.org)

Message-ID:20170905145804.4a65b5bc@scratchpost.org

LGTM!

Ludovic Courtès wrote on 5 Sep 2017 15:17

Re: [bug#28288] [PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 28288@debbugs.gnu.org)

Message-ID:877exds2s3.fsf@gnu.org

Hi!

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (6 lines)

> This mimics the functionality in the root-partition-initializer used in

> creating the QEMU image. This helps when trying to run guix system init from

> the generated ISO image.

> * gnu/build/vm.scm (make-iso9660-image): Add support for registering closures.

[...]

Toggle quote (9 lines)

> + (when register-closures?

> + (display "registering closures...\n")

> + (for-each (lambda (closure)

> + (register-closure

> + "/tmp/root"

> + (string-append "/xchg/" closure)

> + #:deduplicate? #f))

> + closures))

Was there any specific reason for #:deduplicate? #f here?

Otherwise LGTM!

Ludo’.

Ludovic Courtès wrote on 5 Sep 2017 15:18

Re: [bug#28288] [PATCH 5/5] tests: Add test for installing from an ISO Image.

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 28288@debbugs.gnu.org)

Message-ID:873781s2qp.fsf@gnu.org

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (14 lines)> On Sun,  3 Sep 2017 11:50:41 +0100
> Christopher Baines <mail@cbaines.net> wrote:
>
>> * gnu/tests/install.scm (%test-iso-image-installer): New variable.
>>   (run-install): Add #:installation-disk-image-file-system-type as a
>> keyword argument.
>> ---
>>  gnu/tests/install.scm | 110
>> +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 99
>> insertions(+), 11 deletions(-)
>
> This test now passes! I've also successfully generated an ISO image
> that I used to install GuixSD to a Bytemark VM [1].

Woohoo, really nice! \o/

Thanks for persevering!

Ludo’.

Ludovic Courtès wrote on 5 Sep 2017 15:18

Re: [bug#28288] [PATCH 4/5] vm: Call iso9660-image with #:register-closures? as #t.

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:87y3ptqo5s.fsf@gnu.org

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (2 lines)

> LGTM!

+1!

Excellent.

Ludo'.

Christopher Baines wrote on 6 Sep 2017 09:05

Re: [bug#28288] [PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 28288@debbugs.gnu.org)

Message-ID:20170906080515.4764be7b@cbaines.net

On Tue, 05 Sep 2017 15:17:32 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (24 lines)> Hi!
> 
> Christopher Baines <mail@cbaines.net> skribis:
> 
> > This mimics the functionality in the root-partition-initializer
> > used in creating the QEMU image. This helps when trying to run guix
> > system init from the generated ISO image.
> >
> > * gnu/build/vm.scm (make-iso9660-image): Add support for
> > registering closures.  
> 
> [...]
> 
> > +    (when register-closures?
> > +      (display "registering closures...\n")
> > +      (for-each (lambda (closure)
> > +                  (register-closure
> > +                   "/tmp/root"
> > +                   (string-append "/xchg/" closure)
> > +                   #:deduplicate? #f))
> > +                closures))  
> 
> Was there any specific reason for #:deduplicate? #f here?

Yep, the image creation process fails if this is not specified.

error: cannot link

`/tmp/root/gnu/store/.links/1wj8bqv0ygz606a0day0gk7w37prsnbg5z2im2yf025r0y6lqfs0'

`/tmp/root/gnu/store/00lgsgxvv1hfhl22yqmzsa3lrs5qa9my-firmware/lib/firmware/b43-open':

Invalid cross-device link

There is probably another way around this though. I'm not sure what

impact skipping the deduplication has on the size of the image.

Toggle quote (3 lines)

> Otherwise LGTM!

> Ludo’.

Christopher Baines wrote on 6 Sep 2017 09:46

Re: [bug#28288] [PATCH 5/5] tests: Add test for installing from an ISO Image.

Recipients:(address . 28288-done@debbugs.gnu.org)

Message-ID:20170906084649.5835480e@cbaines.net

On Tue, 05 Sep 2017 15:18:22 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (20 lines)> Christopher Baines <mail@cbaines.net> skribis:
> 
> > On Sun,  3 Sep 2017 11:50:41 +0100
> > Christopher Baines <mail@cbaines.net> wrote:
> >  
> >> * gnu/tests/install.scm (%test-iso-image-installer): New variable.
> >>   (run-install): Add #:installation-disk-image-file-system-type as
> >> a keyword argument.
> >> ---
> >>  gnu/tests/install.scm | 110
> >> +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed,
> >> 99 insertions(+), 11 deletions(-)  
> >
> > This test now passes! I've also successfully generated an ISO image
> > that I used to install GuixSD to a Bytemark VM [1].  
> 
> Woohoo, really nice!  \o/
> 
> Thanks for persevering!

Danny, Ludo, thanks for reviewing :) I've now pushed these changes.

Closed

Ludovic Courtès wrote on 6 Sep 2017 15:20

Re: [bug#28288] [PATCH 1/5] vm: Add support for registering closures to make-iso9660-image.

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 28288@debbugs.gnu.org)

Message-ID:87wp5c0xqr.fsf@gnu.org

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (35 lines)> On Tue, 05 Sep 2017 15:17:32 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Hi!
>> 
>> Christopher Baines <mail@cbaines.net> skribis:
>> 
>> > This mimics the functionality in the root-partition-initializer
>> > used in creating the QEMU image. This helps when trying to run guix
>> > system init from the generated ISO image.
>> >
>> > * gnu/build/vm.scm (make-iso9660-image): Add support for
>> > registering closures.  
>> 
>> [...]
>> 
>> > +    (when register-closures?
>> > +      (display "registering closures...\n")
>> > +      (for-each (lambda (closure)
>> > +                  (register-closure
>> > +                   "/tmp/root"
>> > +                   (string-append "/xchg/" closure)
>> > +                   #:deduplicate? #f))
>> > +                closures))  
>> 
>> Was there any specific reason for #:deduplicate? #f here?
>
> Yep, the image creation process fails if this is not specified.
>
> error: cannot link
> `/tmp/root/gnu/store/.links/1wj8bqv0ygz606a0day0gk7w37prsnbg5z2im2yf025r0y6lqfs0'
> to
> `/tmp/root/gnu/store/00lgsgxvv1hfhl22yqmzsa3lrs5qa9my-firmware/lib/firmware/b43-open':
> Invalid cross-device link

Hmm, OK. Looks like it’s trying to deduplicate between /tmp/root and

some other store. (Or are we using a bind mount here?)

Toggle quote (3 lines)

> There is probably another way around this though. I'm not sure what

> impact skipping the deduplication has on the size of the image.

It probably doesn’t matter much, so it’s not a blocker IMO.

Maybe leave an XXX comment mentioning the cross-device link issue.

Thanks!

Ludo’.

Christopher Baines wrote on 10 Sep 2017 12:45

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 28288@debbugs.gnu.org)

Message-ID:20170910114532.58628283@cbaines.net

On Wed, 06 Sep 2017 15:20:44 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (40 lines)> Christopher Baines <mail@cbaines.net> skribis:
> 
> > On Tue, 05 Sep 2017 15:17:32 +0200
> > ludo@gnu.org (Ludovic Courtès) wrote:
> >  
> >> Hi!
> >> 
> >> Christopher Baines <mail@cbaines.net> skribis:
> >>   
> >> > This mimics the functionality in the root-partition-initializer
> >> > used in creating the QEMU image. This helps when trying to run
> >> > guix system init from the generated ISO image.
> >> >
> >> > * gnu/build/vm.scm (make-iso9660-image): Add support for
> >> > registering closures.    
> >> 
> >> [...]
> >>   
> >> > +    (when register-closures?
> >> > +      (display "registering closures...\n")
> >> > +      (for-each (lambda (closure)
> >> > +                  (register-closure
> >> > +                   "/tmp/root"
> >> > +                   (string-append "/xchg/" closure)
> >> > +                   #:deduplicate? #f))
> >> > +                closures))    
> >> 
> >> Was there any specific reason for #:deduplicate? #f here?  
> >
> > Yep, the image creation process fails if this is not specified.
> >
> > error: cannot link
> > `/tmp/root/gnu/store/.links/1wj8bqv0ygz606a0day0gk7w37prsnbg5z2im2yf025r0y6lqfs0'
> > to
> > `/tmp/root/gnu/store/00lgsgxvv1hfhl22yqmzsa3lrs5qa9my-firmware/lib/firmware/b43-open':
> > Invalid cross-device link  
> 
> Hmm, OK.  Looks like it’s trying to deduplicate between /tmp/root and
> some other store.  (Or are we using a bind mount here?)

I think there is a bind mount in use here, so maybe that has something

to do with it.

Toggle quote (7 lines)

> > There is probably another way around this though. I'm not sure what

> > impact skipping the deduplication has on the size of the image.

> It probably doesn’t matter much, so it’s not a blocker IMO.

> Maybe leave an XXX comment mentioning the cross-device link issue.

Yep, I'll put in a comment :)

Your comment

This issue is archived.

To comment on this conversation send an email to 28288@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date