[core-updates-frozen] Grub fails to decrypt LUKS partition (cryptomount not found)

  • Done
  • quality assurance status badge
Details
4 participants
  • Jonathan Brielmaier
  • Ludovic Courtès
  • Mathieu Othacehe
  • Mathieu Othacehe
Owner
unassigned
Submitted by
Jonathan Brielmaier
Severity
important
J
J
Jonathan Brielmaier wrote on 13 Sep 2021 13:54
(address . bug-guix@gnu.org)
aa357aca-baea-e28f-2dd4-e8516e5c1796@web.de
Hi folks,

today I finally got my system reconfiguring on core-updates-frozen with
almost all packages I use (minus telegram-desktop, josm and lilypond).
But when I rebooted my PC it got stuck in the grub rescue console :(

```
Welcome to GRUB!

Unkown command 'crpytomount'.
error: disk 'cryptouuid/ad06...' not found.
Entering rescue mode...
[...]
grub rescue> crpytomount -u ad06...
Unknown command 'cryptomount'.
[...]
grub rescue> set
cmdpath=(hd0)/EFI/Guix
prefix=(cryptouuid)/ad06...)/boot/grub
root=cryptouuid/ad06...
```

commit: d1732a007fff1549fd49bae595ef9541bea2a332 (pulled from)
arch: x86_66
config:
```
(bootloader
(bootloader-configuration
(bootloader grub-efi-bootloader)
(target "/boot/efi")
(keyboard-layout keyboard-layout)))
(mapped-devices
(list (mapped-device
(source (uuid "ad069897-6b43-4607-b5e9-3f00f9af74aa"))
(target "cryptroot")
(type luks-device-mapping))))
(file-systems
(append (list
(file-system
(mount-point "/boot/efi")
(device (uuid "2954-F644" 'fat32))
(type "vfat"))
(file-system
(device "/dev/mapper/cryptroot")
(mount-point "/")
(type "ext4")
(dependencies mapped-devices))
(file-system
(mount-point "/media/jonathan/arbeit")
(device (uuid "094e6082-fae5-4ba9-ad94-1a4b6d04a094"))
(type "ext4")))
%base-file-systems))
```

I have reconfigured the same system on master two or three days ago. The
reboot back then was successful.

I'll try to collect more information later today...

~Jonathan
L
L
Ludovic Courtès wrote on 13 Sep 2021 14:30
control message for bug #50566
(address . control@debbugs.gnu.org)
87pmtc8xpw.fsf@gnu.org
severity 50566 important
quit
J
J
Jonathan Brielmaier wrote on 15 Sep 2021 00:57
[core-updates-frozen] Grub fails to decrypt LUKS partition (cryptomount not found)
(address . 50566@debbugs.gnu.org)
2462d810-f92a-1a00-c6d6-d44597f51d27@web.de
I went ahead and did a trick Tobias (nckx) told me in IRC. I booted a
Guix installer from an USB stick and hit 'c' (command line) in its GRUB
menu. Then I gave in the following commands:

```
insmod luks
cryptomount (hd1,gpt2)
Enter passphrase ...:
set root=(crypto0)
configfile /boot/grub/grub.cfg
```

I can then choose between the Guix system generations: 1x
core-updates-frozen and 3x master. The latter did worked before my
reconfigure on c-u-f.

But non of them really worked. I have to enter the LUKS password a
second time and then shepherd fails to start all the services. The cpu
fan blows a bit loud:
```
failed to start service 'file-systems'
failed to start service 'file-system-/boot/efi'
failed to start service 'user-processes'
[...]
```

An Alt+SysReq+E (SIGTERM all apart INIT) gives:
```
shepherd[1]: Service udev has been disabled.
shepherd[1]: Respawning libvirtd.
shepherd[1]: Service libvirtd has been started.
```

And some audit log message. Can audit break the boot?

Has anyone an idea how to find out why those services start to fail? How
to obtain logs?

~Jonathan
J
J
Jonathan Brielmaier wrote on 17 Sep 2021 17:08
(address . 50566@debbugs.gnu.org)
8de36a06-188d-7a97-71d3-e0d3ef23f6e1@web.de
Hi,

I found today an easier reproducer. I built an installation image from
current core-updates-frozen (716470cda447e8cc3e73b09a8dfdcf1032aa37f2).
Booting from an USB stick leads to the same error that it comes only to
"grub rescue>" saying "Unknown filesystem".

I double checked and booted from an USB stick with an image from current
master -> just fine.

It would be interesting if others try to boot from an c-u-f image,
especially on UEFI.

I'll try to bisect the issue as I have no an "easy" reproducer :)
M
M
Mathieu Othacehe wrote on 18 Sep 2021 17:48
(name . Jonathan Brielmaier)(address . jonathan.brielmaier@web.de)(address . 50566@debbugs.gnu.org)
874kahanrg.fsf@gnu.org
Hello Jonathan,

Toggle quote (2 lines)
> I'll try to bisect the issue as I have no an "easy" reproducer :)

I experienced the exact same issue. To be able to boot again, I booted
from a Guix System USB drive, hitted 'c' in Grub, then typed the
following commands:

Toggle snippet (4 lines)
set root=(hd0,gpt2)
configfile /boot/grub/grub.cfg

Now regarding the cause of the issue, it appears that running the
following command from a Guix "master" Grub fixes the issue:

Toggle snippet (3 lines)
sudo /gnu/store/w8v5d1i6xfqlpj78w89jg1x7f8dchh4k-grub-efi-2.06/sbin/grub-install --boot-directory /boot --efi-directory=/boot/efi

So this means that we probably have an issue related to the Grub package
on core-updates-frozen itself. Both master and core-updates-frozen are
using Grub 2.06 release.

The issue might then be related to the GCC 10 switch?

Here are the log files of Grub installation of master and
core-updates-frozen branches. I edited the /gnu/store/xxx-grub-efi-2.06
strings to remove the noise.

It looks like the core-updates-frozen Grub has a new .note.grub.property
section but I cannot find any other noticeable differences.

To be continued,

Mathieu
Attachment: master.log
Attachment: core.log
M
M
Mathieu Othacehe wrote on 19 Sep 2021 17:02
(name . Jonathan Brielmaier)(address . jonathan.brielmaier@web.de)(address . 50566@debbugs.gnu.org)
87k0jc7gnc.fsf@gnu.org
Turns out stripping Grub modules was causing this issue. I'm not sure
why we do not experience this issue on master.

Anyway, 71aa29911cf3f4e6db5f9bff9237308b5f93283d fixes it for me.

I also discovered another issue related to image creation that I'll
report separately.

Thanks,

Mathieu
J
J
Jonathan Brielmaier wrote on 19 Sep 2021 23:58
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50566@debbugs.gnu.org)
1f38c8ca-f7e0-e6a1-bfb3-ecadc2f9de96@web.de
Am 19.09.21 um 17:02 schrieb Mathieu Othacehe:
Toggle quote (6 lines)
>
> Turns out stripping Grub modules was causing this issue. I'm not sure
> why we do not experience this issue on master.
>
> Anyway, 71aa29911cf3f4e6db5f9bff9237308b5f93283d fixes it for me.

Good catch and thanks for the fix, which works for me.

Toggle quote (3 lines)
> I also discovered another issue related to image creation that I'll
> report separately.

During boot I'll run into https://issues.guix.gnu.org/50566like you.

So I think we can close this bug and work on the next one :)
L
L
Ludovic Courtès wrote on 20 Sep 2021 00:47
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87r1dkqj1r.fsf@gnu.org
Hi!

Mathieu Othacehe <othacehe@gnu.org> skribis:

Toggle quote (3 lines)
> Turns out stripping Grub modules was causing this issue. I'm not sure
> why we do not experience this issue on master.

Woow, good catch.

Toggle quote (2 lines)
> Anyway, 71aa29911cf3f4e6db5f9bff9237308b5f93283d fixes it for me.

‘master’ uses ‘--strip-debug’ whereas ‘core-updates-frozen’ uses
‘--strip-unneeded’ (from commit
e0f31baacc6ad30096a332b69433c85f5830bb2c).

Instead of #:strip-binaries? #f, we could try:

#:strip-flags '("--strip-debug" "--enable-deterministic-archives")

Thanks,
Ludo’.
J
J
Jonathan Brielmaier wrote on 20 Sep 2021 10:04
(address . 50566@debbugs.gnu.org)
2a9a6029-3103-56f4-6b7d-81110dd98cbe@web.de
Am 20.09.21 um 00:47 schrieb Ludovic Courtès:
Toggle quote (8 lines)
> ‘master’ uses ‘--strip-debug’ whereas ‘core-updates-frozen’ uses
> ‘--strip-unneeded’ (from commit
> e0f31baacc6ad30096a332b69433c85f5830bb2c).
>
> Instead of #:strip-binaries? #f, we could try:
>
> #:strip-flags '("--strip-debug" "--enable-deterministic-archives")

That one works as well. I tested it :)
M
M
Mathieu Othacehe wrote on 20 Sep 2021 11:19
(name . Jonathan Brielmaier)(address . jonathan.brielmaier@web.de)
875yuvtxiq.fsf@gnu.org
Hey,

Toggle quote (2 lines)
>> #:strip-flags '("--strip-debug" "--enable-deterministic-archives")

Pushed as 650e85d85514c5fae06adf97ae615643a41bbbd8 :).

Toggle quote (2 lines)
> That one works as well. I tested it :)

Thanks for the suggestion Ludo and for the testing Jonathan.

Mathieu
M
M
Mathieu Othacehe wrote on 20 Sep 2021 11:21
control message for bug #50566
(address . control@debbugs.gnu.org)
874kaftxe8.fsf@meije.i-did-not-set--mail-host-address--so-tickle-me
close 50566
quit
?
Your comment

This issue is archived.

To comment on this conversation send an email to 50566@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 50566
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch