[PATCH] build: Make ISO-9660 image bootable from USB flash drive.

drive.

---

gnu/build/vm.scm | 2 ++

1 file changed, 2 insertions(+)

Danny Milosavljevic <dannym@scratchpost.org> skribis:

I think you’ll have to expound in the comment because it doesn’t sound

trivial. :-)

Thanks,

Ludo’.

drive.

---

gnu/build/vm.scm | 13 +++++++++++++

1 file changed, 13 insertions(+)

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Now I understand. :-)

Perfect, thank you!

Ludo’.

close 27661

Thanks!

Pushed.

If we want we can (in the future) provide only the iso9660 image on the homepage (for x86_64).

It will work from CD and USB flash drive - that should cover all the options.

(Usually, ARM systems have a more involved boot process and will not work, neither from SD card nor from CD-ROM - and booting from USB flash drive is out of the question).

Of course it would be nice if people could test the iso9660 image a bit (especially on UEFI systems of which I have none).

To generate one, for example do this:

guix system disk-image -t iso9660 foo.scm # will print the name of the directory that contains "guixsd.iso" in the end.

where foo.scm contains:

(use-modules (gnu))

(use-package-modules linux bootloaders)

(use-service-modules base networking ssh)

(operating-system

(host-name "beagle-bone-black")

(timezone "Europe/Zurich")

(locale "en_US.UTF-8")

(bootloader (grub-configuration (device "/dev/vda")))

(kernel linux-libre)

(file-systems

(cons

(file-system

(mount-point "/")

(type "iso9660")

(title 'label)

(device "gnu-disk-image"))

%base-file-systems))

(users

(cons*

(user-account

(name "test")

(group "users")

(supplementary-groups '("wheel" "video"))

(home-directory "/home/test")

(password "sa5JEXtYx/rm6")) ; Password is pass.

%base-user-accounts))

(services

(cons*

(dropbear-service)

(dhcp-client-service)

%base-services)))

Danny Milosavljevic transcribed 1.4K bytes:

Woo! Thanks for your work on this :)

I'll test this tomorrow evening.

Exciting news.

--

ng0

GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588

GnuPG: https://n0is.noblogs.org/my-keys

https://www.infotropique.orghttps://krosos.org

Danny Milosavljevic <dannym@scratchpost.org> writes:

It does not work on UEFI. I tested in qemu like this:

qemu-system-x86_64 -bios $(guix build ovmf)/share/firmware/ovmf_x64.bin \

-m 256M -enable-kvm -boot c -cdrom /gnu/store/...-disk-image/guixsd.iso

There doesn't seem to be an EFI system partition? How did you dodge my

nasty partition hack :)

Also, the derivation creates two files alongside the ISO, are those

actually required or can we make it it return the ISO directly?

Impressive work nonetheless! Many users have asked for ISO support for

use with e.g. hosting providers. And it's really cool to be able to

create a CDROM/USB, BIOS/UEFI* multi-hybrid at a whim!

Hi Marius,

On Thu, 13 Jul 2017 20:53:50 +0200

Marius Bakke <mbakke@fastmail.com> wrote:

It's not using initialize-harddisk but rather grub-mkrescue and xorriso in order to create the filesystem.

That's because ISO-9660 filesystems are read-only and have to be created *including* the entire content in one go.

No.

I don't know how...

Bug #27689 fixes it a bit if the bootloader used is grub-efi.

However, that image then doesn't work with normal (non-EFI) BIOS and neither does it work with "qemu -hda".

Unfortunately I can't debug the "qemu -hda" problem properly because it seems that when UEFI is in use, "grub linux16" doesn't work any more which means I can't increase the number of lines on screen anymore (also, there's a huge penguin in the way) which means I can't see the entire error message that the guest prints.

How can I see the entire Linux kernel crash message in qemu (UEFI)?

Debugging:

$ qemu-system-x86_64 -bios $(guix build ovmf)/share/firmware/ovmf_x64.bin -m 1G -enable-kvm -hda Zuefi -serial stdio

Then add "console=ttyS0" to Linux cmdline in grub menu edit.

I get:

[ 0.923605] scsi 0:0:0:0: Direct-Access ATA QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5

[ 0.926558] sd 0:0:0:0: [sda] 2215580 512-byte logical blocks: (1.13 GB/1.06 GiB)

[ 0.928683] sd 0:0:0:0: [sda] Write Protect is off

[ 0.930155] sd 0:0:0:0: Attached scsi generic sg0 type 0

[ 0.931705] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

[ 0.934772] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5

[ 0.949039] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray

[ 0.952674] cdrom: Uniform CD-ROM driver Revision: 3.20

[ 0.955449] sda: sda1 sda2 sda3 sda4

[ 0.956995] sd 0:0:0:0: [sda] Attached SCSI disk

[ 0.959031] sr 1:0:0:0: Attached scsi generic sg1 type 5

GC Warning: Couldn't read /proc/stat

Welcome, this is GNU's early boot Guile.

Use '--repl' for an initrd REPL.

loading '/gnu/store/48d1w55iy2vpcm5ass5lsp8almghszv8-system/boot'...

[ 1.235733] attempt to access beyond end of device

[ 1.237066] sda1: rw=0, want=949216, limit=76376

[ 1.238565] attempt to access beyond end of device

[ 1.240115] sda1: rw=0, want=949216, limit=76376

[ 1.242095] attempt to access beyond end of device

[ 1.243545] sda1: rw=0, want=949216, limit=76376

ERROR: In procedure primitive-load:

ERROR: In procedure fport_fill_input: Input/output error

Partitions are:

Disk Zuefi: 1.1 GiB, 1134376960 bytes, 2215580 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt

Disk identifier: DB26D0AA-65F4-4CE1-87F8-85CAD51B5C94

Device Start End Sectors Size Type

Zuefi1 64 76439 76376 37.3M Microsoft basic data <----- here is a suspect

Zuefi2 76440 82199 5760 2.8M EFI System

Zuefi3 82200 2214931 2132732 1G Apple HFS/HFS+

Zuefi4 2214932 2215531 600 300K Microsoft basic data

Backtrace:

Impossible. It did not actually enter the prompt.

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

I also vote for returning the ISO directly, for consistency with what

‘guix system disk-image’ does for the other formats. WDYT, Danny? :-)

Ludo’.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

It looks like the partition detection code in (gnu build file-systems)

tries to read beyond the end of disk or something. Could you try adding

a few ‘pk’ or ‘format’ calls in there so see what’s going on?

Alternately, you could run the same QEMU command line that the

derivation spawns, so you would get a REPL (and backtrace). For that

you can “ps aux | grep qemu” while the derivation is building, and

copy/paste the QEMU command from there.

HTH,

Ludo’.

Hi Ludo,

On Mon, 17 Jul 2017 10:25:08 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Yeah, it tries to read beyond the end of the partition. That's because grub-mkrescue creates a tiny fake partition which happens to start where the ISO-9660 filesystem starts and GuixSD mounts it and tries to boot from it (because it has a valid volume id). As it is going on booting, the system boot wanders off the end of the partition and Linux tells it that it can't do that.

I've posted bug# 27705 which is the minimal set of patches to get it to work. I just read the root filesystem directly from the whole disk there.

By now, I've also tested the resulting image on the bare metal on a real DVD on a non-EFI, non-Libreboot system. Works fine :)

Marius, could you also test it on EFI ?

As for fixing the original partitioning problem, I didn't manage to find the place yet. As far as I can see, grub-mkrescue uses xorriso which uses mkisofs to actually generate the ISO image. There's a "protective" DOS MBR generated somewhere, but where. It seems that Xorriso_coordinate_system_area and make_isohybrid_mbr are at least related somehow. Also, it's questionable whether it's possible to fix it without creating overlapped partitions, see below.

The resulting partition table is:

Disk J: 1.1 GiB, 1137074176 bytes, 2220848 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt

Disk identifier: F86B53DD-3E7B-4B5E-A4C9-3A1A0AA56D22

Device Start End Sectors Size Type

J1 64 76871 76808 37.5M Microsoft basic data <--- GuixSD tried to use this one as root

J2 76872 82631 5760 2.8M EFI System <-- mounts and reads fine as FAT FS

J3 82632 2220199 2137568 1G Apple HFS/HFS+ <--- can't find valid HFS on there

J4 2220200 2220799 600 300K Microsoft basic data <-- can't mount it

P.S. grub-mkrescue (with EFI) has a surprising number of supported architectures. It might be that a grub-mkrescue-generated image could also boot on MIPS, Sparc, PowerPC, ARM64.

Thanks! Very useful.

Hi Ludo,

Sure, I just don't know how to do that. The other two files are "system" and "bootcfg" and contain some kind of list of gnu store paths and single-digit numbers. Anyone can go ahead and remove those - and make it somehow pick the guixsd.iso ... *shrugs*

iso9660-image uses expression->derivation-in-linux-vm to run make-iso9660-image to create /xchg/guixsd.iso - not sure how the other two files came into being.

I could make iso9660-image delete the two other files after the fact, I guess. But that wouldn't make system-disk-image use just one regular file as the output - without the directory.

load-in-linux-vm has a flag "make-disk-image?". When not set (and in this case, it's really not set), it does

(mkdir output)

(copy-recursively "xchg" output)

.

On the other hand, when set, it creates a huge qemu disk image we don't need.

We could special-case "if there's a guixsd.iso in there, take it instead" there, but doesn't seem so nice, does it?

I tried something like the following. But now it throws away the output entirely. That's not an improvement :)

But I think this is overcomplicating things - and the easiest way with only aesthetic downsides would be to punt and do nothing...

Or we could replace make-disk-image? and make-output-directory? by two parameters "initializer" and "serializer" which would be procedures that could do whatever they wanted. I think that would be overly generic, though :)

Also, what amount of rebuilding would any change cause? The procedures modified are already used quite often...

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

They are “references graphs” files, describing the closure of the given

store items.

I gave it a stab and ended up with the attached patch, which Works For

Me™ and is not unreasonably ugly to my eye. :-)

WDYT?

Ludo’.

Hi Ludo,

On Tue, 18 Jul 2017 15:08:49 +0200 ludo@gnu.org (Ludovic Courtès) wrote:

What happens when SINGLE-FILE-OUTPUT? is true? :)

I'd say something like,

If SINGLE-FILE-OUTPUT? is true, copy a single file from /xchg to OUTPUT.

Otherwise, copy the contents of /xchg to a new directory OUTPUT.

Maybe raise error when there's more than one possibility?

What if it's true?

[...]

If we called it "disk-image" in the iso9660 case as well we'd not have this special case. Do we want to?

Overall LGTM!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

All good points, indeed. Pushed with these changes as

8d033e3e1607d5722ef7288208551d0331c8a853.

I find it marginally nicer to have an explicit “.iso” suffix. I pushed

this part as 5058bf56843baf3c0d82fbf0addbb30f00572428.

Thank you!

Ludo’.

Hi,

Adam Van Ymeren wrote on Mon, 17 Jul 2017 22:34:34 -0400:

Hmm, was that with patches from bug# 27695 (grub-hybrid-bootloader) or with patches from bug# 27705 (grub-efi patched to include some grub-pc files) ?