[core-updates-frozen] Grub fails to decrypt LUKS partition (cryptomount not found)

  • Done
  • quality assurance status badge
Details
4 participants
  • Jonathan Brielmaier
  • Ludovic Courtès
  • Mathieu Othacehe
  • Mathieu Othacehe
Owner
unassigned
Submitted by
Jonathan Brielmaier
Severity
important
J
J
Jonathan Brielmaier wrote on 13 Sep 2021 13:54
(address . bug-guix@gnu.org)
aa357aca-baea-e28f-2dd4-e8516e5c1796@web.de
Hi folks,

today I finally got my system reconfiguring on core-updates-frozen with
almost all packages I use (minus telegram-desktop, josm and lilypond).
But when I rebooted my PC it got stuck in the grub rescue console :(

```
Welcome to GRUB!

Unkown command 'crpytomount'.
error: disk 'cryptouuid/ad06...' not found.
Entering rescue mode...
[...]
grub rescue> crpytomount -u ad06...
Unknown command 'cryptomount'.
[...]
grub rescue> set
cmdpath=(hd0)/EFI/Guix
prefix=(cryptouuid)/ad06...)/boot/grub
root=cryptouuid/ad06...
```

commit: d1732a007fff1549fd49bae595ef9541bea2a332 (pulled from)
arch: x86_66
config:
```
(bootloader
(bootloader-configuration
(bootloader grub-efi-bootloader)
(target "/boot/efi")
(keyboard-layout keyboard-layout)))
(mapped-devices
(list (mapped-device
(source (uuid "ad069897-6b43-4607-b5e9-3f00f9af74aa"))
(target "cryptroot")
(type luks-device-mapping))))
(file-systems
(append (list
(file-system
(mount-point "/boot/efi")
(device (uuid "2954-F644" 'fat32))
(type "vfat"))
(file-system
(device "/dev/mapper/cryptroot")
(mount-point "/")
(type "ext4")
(dependencies mapped-devices))
(file-system
(mount-point "/media/jonathan/arbeit")
(device (uuid "094e6082-fae5-4ba9-ad94-1a4b6d04a094"))
(type "ext4")))
%base-file-systems))
```

I have reconfigured the same system on master two or three days ago. The
reboot back then was successful.

I'll try to collect more information later today...

~Jonathan
L
L
Ludovic Courtès wrote on 13 Sep 2021 14:30
control message for bug #50566
(address . control@debbugs.gnu.org)
87pmtc8xpw.fsf@gnu.org
severity 50566 important
quit
J
J
Jonathan Brielmaier wrote on 15 Sep 2021 00:57
[core-updates-frozen] Grub fails to decrypt LUKS partition (cryptomount not found)
(address . 50566@debbugs.gnu.org)
2462d810-f92a-1a00-c6d6-d44597f51d27@web.de
I went ahead and did a trick Tobias (nckx) told me in IRC. I booted a
Guix installer from an USB stick and hit 'c' (command line) in its GRUB
menu. Then I gave in the following commands:

```
insmod luks
cryptomount (hd1,gpt2)
Enter passphrase ...:
set root=(crypto0)
configfile /boot/grub/grub.cfg
```

I can then choose between the Guix system generations: 1x
core-updates-frozen and 3x master. The latter did worked before my
reconfigure on c-u-f.

But non of them really worked. I have to enter the LUKS password a
second time and then shepherd fails to start all the services. The cpu
fan blows a bit loud:
```
failed to start service 'file-systems'
failed to start service 'file-system-/boot/efi'
failed to start service 'user-processes'
[...]
```

An Alt+SysReq+E (SIGTERM all apart INIT) gives:
```
shepherd[1]: Service udev has been disabled.
shepherd[1]: Respawning libvirtd.
shepherd[1]: Service libvirtd has been started.
```

And some audit log message. Can audit break the boot?

Has anyone an idea how to find out why those services start to fail? How
to obtain logs?

~Jonathan
J
J
Jonathan Brielmaier wrote on 17 Sep 2021 17:08
(address . 50566@debbugs.gnu.org)
8de36a06-188d-7a97-71d3-e0d3ef23f6e1@web.de
Hi,

I found today an easier reproducer. I built an installation image from
current core-updates-frozen (716470cda447e8cc3e73b09a8dfdcf1032aa37f2).
Booting from an USB stick leads to the same error that it comes only to
"grub rescue>" saying "Unknown filesystem".

I double checked and booted from an USB stick with an image from current
master -> just fine.

It would be interesting if others try to boot from an c-u-f image,
especially on UEFI.

I'll try to bisect the issue as I have no an "easy" reproducer :)
M
M
Mathieu Othacehe wrote on 18 Sep 2021 17:48
(name . Jonathan Brielmaier)(address . jonathan.brielmaier@web.de)(address . 50566@debbugs.gnu.org)
874kahanrg.fsf@gnu.org
Hello Jonathan,

Toggle quote (2 lines)
> I'll try to bisect the issue as I have no an "easy" reproducer :)

I experienced the exact same issue. To be able to boot again, I booted
from a Guix System USB drive, hitted 'c' in Grub, then typed the
following commands:

Toggle snippet (4 lines)
set root=(hd0,gpt2)
configfile /boot/grub/grub.cfg

Now regarding the cause of the issue, it appears that running the
following command from a Guix "master" Grub fixes the issue:

Toggle snippet (3 lines)
sudo /gnu/store/w8v5d1i6xfqlpj78w89jg1x7f8dchh4k-grub-efi-2.06/sbin/grub-install --boot-directory /boot --efi-directory=/boot/efi

So this means that we probably have an issue related to the Grub package
on core-updates-frozen itself. Both master and core-updates-frozen are
using Grub 2.06 release.

The issue might then be related to the GCC 10 switch?

Here are the log files of Grub installation of master and
core-updates-frozen branches. I edited the /gnu/store/xxx-grub-efi-2.06
strings to remove the noise.

It looks like the core-updates-frozen Grub has a new .note.grub.property
section but I cannot find any other noticeable differences.

To be continued,

Mathieu
Attachment: master.log
Attachment: core.log
M
M
Mathieu Othacehe wrote on 19 Sep 2021 17:02
(name . Jonathan Brielmaier)(address . jonathan.brielmaier@web.de)(address . 50566@debbugs.gnu.org)
87k0jc7gnc.fsf@gnu.org
Turns out stripping Grub modules was causing this issue. I'm not sure
why we do not experience this issue on master.

Anyway, 71aa29911cf3f4e6db5f9bff9237308b5f93283d fixes it for me.

I also discovered another issue related to image creation that I'll
report separately.

Thanks,

Mathieu
J
J
Jonathan Brielmaier wrote on 19 Sep 2021 23:58
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50566@debbugs.gnu.org)
1f38c8ca-f7e0-e6a1-bfb3-ecadc2f9de96@web.de
Am 19.09.21 um 17:02 schrieb Mathieu Othacehe:
Toggle quote (6 lines)
>
> Turns out stripping Grub modules was causing this issue. I'm not sure
> why we do not experience this issue on master.
>
> Anyway, 71aa29911cf3f4e6db5f9bff9237308b5f93283d fixes it for me.

Good catch and thanks for the fix, which works for me.

Toggle quote (3 lines)
> I also discovered another issue related to image creation that I'll
> report separately.

During boot I'll run into https://issues.guix.gnu.org/50566like you.

So I think we can close this bug and work on the next one :)
L
L
Ludovic Courtès wrote on 20 Sep 2021 00:47
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87r1dkqj1r.fsf@gnu.org
Hi!

Mathieu Othacehe <othacehe@gnu.org> skribis:

Toggle quote (3 lines)
> Turns out stripping Grub modules was causing this issue. I'm not sure
> why we do not experience this issue on master.

Woow, good catch.

Toggle quote (2 lines)
> Anyway, 71aa29911cf3f4e6db5f9bff9237308b5f93283d fixes it for me.

‘master’ uses ‘--strip-debug’ whereas ‘core-updates-frozen’ uses
‘--strip-unneeded’ (from commit
e0f31baacc6ad30096a332b69433c85f5830bb2c).

Instead of #:strip-binaries? #f, we could try:

#:strip-flags '("--strip-debug" "--enable-deterministic-archives")

Thanks,
Ludo’.
J
J
Jonathan Brielmaier wrote on 20 Sep 2021 10:04
(address . 50566@debbugs.gnu.org)
2a9a6029-3103-56f4-6b7d-81110dd98cbe@web.de
Am 20.09.21 um 00:47 schrieb Ludovic Courtès:
Toggle quote (8 lines)
> ‘master’ uses ‘--strip-debug’ whereas ‘core-updates-frozen’ uses
> ‘--strip-unneeded’ (from commit
> e0f31baacc6ad30096a332b69433c85f5830bb2c).
>
> Instead of #:strip-binaries? #f, we could try:
>
> #:strip-flags '("--strip-debug" "--enable-deterministic-archives")

That one works as well. I tested it :)
M
M
Mathieu Othacehe wrote on 20 Sep 2021 11:19
(name . Jonathan Brielmaier)(address . jonathan.brielmaier@web.de)
875yuvtxiq.fsf@gnu.org
Hey,

Toggle quote (2 lines)
>> #:strip-flags '("--strip-debug" "--enable-deterministic-archives")

Pushed as 650e85d85514c5fae06adf97ae615643a41bbbd8 :).

Toggle quote (2 lines)
> That one works as well. I tested it :)

Thanks for the suggestion Ludo and for the testing Jonathan.

Mathieu
M
M
Mathieu Othacehe wrote on 20 Sep 2021 11:21
control message for bug #50566
(address . control@debbugs.gnu.org)
874kaftxe8.fsf@meije.i-did-not-set--mail-host-address--so-tickle-me
close 50566
quit
?