Empty Scheme files in system generation, breaks `guix system` and boot

  • Open
  • quality assurance status badge
Details
4 participants
  • Bas Alberts
  • Felix Lechner
  • Leo Famulari
  • Tomas Volf
Owner
unassigned
Submitted by
Leo Famulari
Severity
grave

Debbugs page

Leo Famulari wrote 16 hours ago
(address . bug-guix@gnu.org)
Z9CdNwX8vaD5UPtG@jasmine.lan
In recent days, I reconfigured my system in the same way that I always
do it.

The resulting system was malformed, preventing boot and the use of `guix
system`.

Specifically, the files that make up the system itself --- such as
'/var/guix/profiles/system-NNN-link/parameters' --- are empty! Zero
bytes.

IIRC, I was able to work around it by running the activation scripts
from an older generation "by hand", but I don't remember for sure if
this was sufficient or not. For example:

`guile -s /var/guix/profiles/system-NNN-link/activate`

I didn't report this to bug-guix (although I did complain on #guix),
because I thought it was just me, but now another user on #guix has the
same problem:


`guix gc --verify=contents` reported all the broken files. Repair was
not possible for most of them.

`btrfs scrub` was successful, and memtest86+ passed 9 times, so I'm
relatively confident in the filesystem and the hardware.

I don't have a handle on the relevant details, but here is some system
information:

------
# cat /etc/fstab
# This file was generated from your Guix configuration. Any changes
# will be lost upon reboot or reconfiguration.

UUID=287d4ed7-7fc8-4726-8882-fbe1f71fe7cd / btrfs defaults
UUID=3A45-1EB8 /boot/efi vfat defaults
# guix system describe
Generation 92 Mar 10 2025 14:02:20 (current)
file name: /var/guix/profiles/system-92-link
canonical file name: /gnu/store/ws8gp0ylxppqxbraac14r0crk1jam7n3-system
label: GNU with Linux 6.13.6
bootloader: grub-efi
root device: UUID: 287d4ed7-7fc8-4726-8882-fbe1f71fe7cd
kernel: /gnu/store/bk469qvagzgxjvxbvj7myplyxzqa9qk0-linux-6.13.6/bzImage
channels:
nonguix:
branch: master
commit: 944619c1947ba59823f4ad6d8b13842b7e162d76
guix:
branch: kernel-updates
commit: ccf4b131e489f6845a041d566bbd1ce929324faa
configuration file: /gnu/store/aa5dq0j4ag28zf2vlr3n0jv8p2wi5y4s-configuration.scm
------
Felix Lechner wrote 16 hours ago
May prevent reboots, ergo grave
(name . GNU bug tracker automated control server)(address . control@debbugs.gnu.org)
87y0xbxpyb.fsf@lease-up.com
severity 76959 grave
thanks
Tomas Volf wrote 16 hours ago
Re: bug#76959: Empty Scheme files in system generation, breaks `guix system` and boot
(name . Leo Famulari)(address . leo@famulari.name)(address . 76959@debbugs.gnu.org)
875xkfxpqd.fsf@wolfsden.cz
Leo Famulari <leo@famulari.name> writes:

Toggle quote (9 lines)
> I didn't report this to bug-guix (although I did complain on #guix),
> because I thought it was just me, but now another user on #guix has the
> same problem:
>
> https://logs.guix.gnu.org/guix/2025-03-11.log#205907
>
> `guix gc --verify=contents` reported all the broken files. Repair was
> not possible for most of them.

I do not really have any useful information to add, but this was
happening to me as well over the past *months*. Happens occasionally,
both in QEMU and on real HW. BTRFS scrub is always fine, and just (some
of) *new* items copied during the deploy are corrupted.

Since no one else reported anything, I assumed it was just me. I cannot
really reproduce it, so I have just assumed I keep "holding it wrong".

The trick with running the activation script by hand is much
appreciated, usually the recovery was fairly painful process.

Tomas

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.
-----BEGIN PGP SIGNATURE-----

iQJCBAEBCgAsFiEEt4NJs4wUfTYpiGikL7/ufbZ/wakFAmfQoHoOHH5Ad29sZnNk
ZW4uY3oACgkQL7/ufbZ/wakmKg/+IfwhE39BW+xLLwxvrB3EBoVqxhtpWd/f8xfi
NR8TkeevMNL/9y8kw+YWXm2SPE/fgIuxhxy9VhOGiad5kWY5aCydXkGKylVhan3U
04+QJge5aKghoJd08grp2e9LRviyWQweLcfs67QqPnQ0h56N6W2Q0MxLIj8Rm3ZG
dpLfnEsEqZMC2qBRmcH3gdMMRuOp1ifw2JKCWgYuYxEGlAm27P8GbSTt9GR3JYT6
uLtYdowSq9lKxDH1IysnsqEtVfHxv3KPMVmO5DCtw3DY5jOYkB6SKe3FVD1p65+u
egYOqywgKTAencmSNy5uprBFimJOC79JQpqmDRG6+AfbjUyaUyRol9IdaAfCp3ul
H897Nq9ndAM1NPeS1pQkE43HFkTrqMIiy5VbcevFfNGQS+D/5M2cnHGWIxBMq1+Y
ZtDqj2+Dx8Be4AWuu4uez+PicBxwVxvdIIt41N7hHnBaKY9tEvOSiRZ4EUWvQIXi
jcx789Q6ciqQJLtVmi4XppeKJQg+zDmlLd4Y0el1kZdct8Wvm6Ts6e2fcPn3X4VE
rMyX5/Xq8ryKaItpZTQ7vUtjOG2cK3AkASKu0x6wYMYJX8uwOCj7LMU6Y9OVc7GF
2B1l91w6QA8uA5mKs6n3pSTu4l4KQoJOB5ZnlZyWfaowz19T76/KRHpqA4r3H7k2
EsnJTWk=
=MBnk
-----END PGP SIGNATURE-----

Leo Famulari wrote 12 hours ago
(address . 76959@debbugs.gnu.org)
Z9DUEv-KPRpIRJ2D@jasmine.lan
IRC user 'anticomputer' wrote the following about their experience with
this problem:

I have also run into this bug. It manifested itself as an empty boot
parameters file causing guix system errors:

`guix system: error: unrecognized boot parameters at '/var/guix/profiles/system/parameters'"`

Roll backs and generation switches are no longer possible due to the current
system profile being in a broken state.

It is still possible to boot into an older generation.

Can confirm that manually activating an older generation per:

`sudo guile -s /var/guix/profiles/system-XX-link/activate`

With a rw remount of `/` to manually set the current system profile link to
that profile will get you back into a state that lets you delete the offending
generation using `guix system delete-generation`.

Curiously, my latest working generation is where the faulty system generation
was configured from, and I was still seeing intermittent generation of e.g.
empty `.drv` files on this generation:

guix system describe:
...
Generation 33 Mar 04 2025 23:02:24 (current)
file name: /var/guix/profiles/system-33-link
canonical file name: /gnu/store/qszg286n66l3gwjipvzg0nhswz7pv996-system
label: GNU with Linux 6.12.13
bootloader: grub-efi
root device: /dev/mapper/system-root
kernel: /gnu/store/gz7w72cj3icm58zj9vs6vvvs6khcqzil-linux-6.12.13/bzImage
channels:
nonguix:
branch: master
commit: 8c41304decb916892fa07ca86893e4cf926ee884
guix:
branch: master
commit: 8bc831325a905dbd9015739b58e3a5138d2217da
configuration file: /gnu/store/45gp6why4xgkxxxl1cs2v7nkf1fxrmag-configuration.scm
...
uname -a:
...
Linux thinkpad 6.12.13 #1 SMP PREEMPT_DYNAMIC 1 x86_64 GNU/Linux

This is a configuration that deploys cleanly on another system but from
this generation (33) I get (inconsistent/intermittent) empty drv files:

...
Reconfiguring system ...
guix system: error: error parsing derivation `/gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv': expected string `Derive(['
λ ~ › cat /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
λ ~ › ls -alrt /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
-r--r--r-- 1 root root 0 Dec 31 1969 /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
λ ~ ›
```

`guix gc --verify=contents` also then shows many empty files with unexpected
hashes.

To get back into a working state I followed the following steps:

- Boot into a known-working older generation XX to purge broken generation YY

- Activate the known-working generation manually using:
`sudo -E guile -s /var/guix/profiles/system-XX-link/activate`

Note: this seems redundant, but just including as the actual steps I took

- remount `/` rw and manually update the current profile link
to enable `guix system` commands to work again:
`sudo mount -o remount,rw /dev/your-root-dev /`
`sudo rm /var/guix/profiles/system`
`sudo ln -s /var/guix/profiles/system-XX-link /var/guix/profiles/system`

- delete the offending generation YY:
`sudo -E guix system delete-generations YY`

- collect garbage:
`guix gc`

Note: I had ~35GB of gc, but plenty of diskspace, not sure if related but
curious that a `guix gc` resulted in being able to reconfigure without
empty files appearing once more

- update guix:
`guix pull`

- reconfigure to intended current generation system config:
`sudo -E guix system -L . reconfigure dolores/systems/CONFIG.scm`

- reboot

Leaving me in the following current/working state:

...
Generation 34 Mar 11 2025 18:09:09 (current)
file name: /var/guix/profiles/system-34-link
canonical file name: /gnu/store/17z7fpnbwbmi78v211xfm5r9wxhjgv6b-system
label: GNU with Linux 6.13.5
bootloader: grub-efi
root device: /dev/mapper/system-root
kernel: /gnu/store/skdgsqhqdwif0adykjg867i1n3xwfz9i-linux-6.13.5/bzImage
channels:
...
guix:
branch: master
commit: d685a45edf0f89e5876ffc9d880068d8610e5f8a
configuration file: /gnu/store/vnp7mhfhfff802zsp2vw74a7v0cv54hj-configuration.scm
...

Kind regards,
Bas
Bas Alberts wrote 15 hours ago
(No Subject)
(name . 76959@debbugs.gnu.org)(address . 76959@debbugs.gnu.org)
ecc5sfZ8WC5rVuuUe66B01F8A5bZmE6L8g5hcCOiIXrH3TwrL08QlE-JMz_XMwFFs54nSw0mgFYvsSJeqDubwb447iB2-NrVRf8O6vodE0Q=@anti.computer
I have also run into this bug. It initially manifested itself as an empty boot
parameters file causing guix system errors:

`guix system: error: unrecognized boot parameters at '/var/guix/profiles/system/parameters'"`

Roll backs and generation switches were no longer possible due to the current
system profile being in a broken state.

It is still possible to boot into an older generation.

Can confirm that manually activating an older generation per:

`sudo guile -s /var/guix/profiles/system-XX-link/activate`

With a rw remount of `/` to manually set the current system profile link to
that profile will get you back into a state that lets you delete the offending
generation using `guix system delete-generation`.

Curiously, my latest working generation is where the faulty system generation
was configured from. I am still seeing intermittent generation of e.g. empty
`.drv` files on this generation:

guix system describe:
...
Generation 33 Mar 04 2025 23:02:24 (current)
file name: /var/guix/profiles/system-33-link
canonical file name: /gnu/store/qszg286n66l3gwjipvzg0nhswz7pv996-system
label: GNU with Linux 6.12.13
bootloader: grub-efi
root device: /dev/mapper/system-root
kernel: /gnu/store/gz7w72cj3icm58zj9vs6vvvs6khcqzil-linux-6.12.13/bzImage
channels:
...
guix:
branch: master
commit: 8bc831325a905dbd9015739b58e3a5138d2217da
configuration file: /gnu/store/45gp6why4xgkxxxl1cs2v7nkf1fxrmag-configuration.scm
...
uname -a:
...
Linux thinkpad 6.12.13 #1 SMP PREEMPT_DYNAMIC 1 x86_64 GNU/Linux


e.g. this is a configuration that deploys cleanly on another system but from
this generation I get (inconsistent/intermittent) empty drv files:

...
Reconfiguring system ...
guix system: error: error parsing derivation `/gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv': expected string `Derive(['
λ ~ › cat /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
λ ~ › ls -alrt /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
-r--r--r-- 1 root root 0 Dec 31 1969 /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
λ ~ ›
```

Kind regards,
Bas
Bas Alberts wrote 15 hours ago
Re: bug#76959: Empty Scheme files in system generation, breaks `guix system` and boot
(name . 76959@debbugs.gnu.org)(address . 76959@debbugs.gnu.org)
1IviqDJcGBJeGUr_7msKtfCgdVXE5meQRjhzq7_7quxHbGrpfVrunsTX2t7obFR53pqEK_VFyDWcpyO3bvKDKwKJMmKSY7k3i4xarqc766o=@anti.computer
I have also run into this bug. It manifested itself as an empty boot
parameters file causing guix system errors:

`guix system: error: unrecognized boot parameters at '/var/guix/profiles/system/parameters'"`

Roll backs and generation switches are no longer possible due to the current
system profile being in a broken state.

It is still possible to boot into an older generation.

Can confirm that manually activating an older generation per:

`sudo guile -s /var/guix/profiles/system-XX-link/activate`

With a rw remount of `/` to manually set the current system profile link to
that profile will get you back into a state that lets you delete the offending
generation using `guix system delete-generation`.

Curiously, my latest working generation is where the faulty system generation
was configured from. I am still seeing intermittent generation of e.g. empty
`.drv` files on this generation:

guix system describe:
...
Generation 33 Mar 04 2025 23:02:24 (current)
file name: /var/guix/profiles/system-33-link
canonical file name: /gnu/store/qszg286n66l3gwjipvzg0nhswz7pv996-system
label: GNU with Linux 6.12.13
bootloader: grub-efi
root device: /dev/mapper/system-root
kernel: /gnu/store/gz7w72cj3icm58zj9vs6vvvs6khcqzil-linux-6.12.13/bzImage
channels:
nonguix:
branch: master
commit: 8c41304decb916892fa07ca86893e4cf926ee884
guix:
branch: master
commit: 8bc831325a905dbd9015739b58e3a5138d2217da
configuration file: /gnu/store/45gp6why4xgkxxxl1cs2v7nkf1fxrmag-configuration.scm
...
uname -a:
...
Linux thinkpad 6.12.13 #1 SMP PREEMPT_DYNAMIC 1 x86_64 GNU/Linux


e.g. this is a configuration that deploys cleanly on another system but from
this generation I get (inconsistent/intermittent) empty drv files:

...
Reconfiguring system ...
guix system: error: error parsing derivation `/gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv': expected string `Derive(['
λ ~ › cat /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
λ ~ › ls -alrt /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
-r--r--r-- 1 root root 0 Dec 31 1969 /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
λ ~ ›
```

`guix gc --verify=contents` shows many unexpected hashes for files as well, which on closer examination are also empty files.

Kind regards,
Bas
Bas Alberts wrote 13 hours ago
Re: bug#76959: Empty Scheme files in system generation, breaks `guix system` and boot
(name . 76959@debbugs.gnu.org)(address . 76959@debbugs.gnu.org)
YnFMAK_wBC2hLe1T9337FNKoQmhBITHpvIQGWox-lnGKSUlg7jt7wzvuGMpIkR0qU9ZLaTQ_vB0Ypq87lfDvY-_HILpV1ls0C-qpfrDnc-c=@anti.computer

I have also run into this bug. It manifested itself as an empty boot
parameters file causing guix system errors:

`guix system: error: unrecognized boot parameters at '/var/guix/profiles/system/parameters'"`

Roll backs and generation switches are no longer possible due to the current
system profile being in a broken state.

It is still possible to boot into an older generation.

Can confirm that manually activating an older generation per:

`sudo guile -s /var/guix/profiles/system-XX-link/activate`

With a rw remount of `/` to manually set the current system profile link to
that profile will get you back into a state that lets you delete the offending
generation using `guix system delete-generation`.

Curiously, my latest working generation is where the faulty system generation
was configured from, and I was still seeing intermittent generation of e.g.
empty `.drv` files on this generation:

guix system describe:
...
Generation 33 Mar 04 2025 23:02:24 (current)
file name: /var/guix/profiles/system-33-link
canonical file name: /gnu/store/qszg286n66l3gwjipvzg0nhswz7pv996-system
label: GNU with Linux 6.12.13
bootloader: grub-efi
root device: /dev/mapper/system-root
kernel: /gnu/store/gz7w72cj3icm58zj9vs6vvvs6khcqzil-linux-6.12.13/bzImage
channels:
...
guix:
branch: master
commit: 8bc831325a905dbd9015739b58e3a5138d2217da
configuration file: /gnu/store/45gp6why4xgkxxxl1cs2v7nkf1fxrmag-configuration.scm
...
uname -a:
...
Linux thinkpad 6.12.13 #1 SMP PREEMPT_DYNAMIC 1 x86_64 GNU/Linux

This is a configuration that deploys cleanly on another system but from
this generation (33) I get (inconsistent/intermittent) empty drv files:

...
Reconfiguring system ...
guix system: error: error parsing derivation `/gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv': expected string `Derive(['
λ ~ › cat /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
λ ~ › ls -alrt /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
-r--r--r-- 1 root root 0 Dec 31 1969 /gnu/store/f2xi0b47yvgb4h8b92hlj7xmaqairidq-shepherd-nftables.go.drv
λ ~ ›
```

`guix gc --verify=contents` also then shows many empty files with unexpected
hashes.

To get back into a working state I followed the following steps:

- Boot into a known-working older generation XX to purge broken generation YY

- Activate the known-working generation manually using:
`sudo -E guile -s /var/guix/profiles/system-XX-link/activate`

Note: this seems redundant, but just including as the actual steps I took

- remount `/` rw and manually update the current profile link
to enable `guix system` commands to work again:
`sudo mount -o remount,rw /dev/your-root-dev /`
`sudo rm /var/guix/profiles/system`
`sudo ln -s /var/guix/profiles/system-XX-link /var/guix/profiles/system`

- delete the offending generation YY:
`sudo -E guix system delete-generations YY`

- collect garbage:
`guix gc`

Note: I had ~35GB of gc, but plenty of diskspace, not sure if related but
curious that a `guix gc` resulted in being able to reconfigure without
empty files appearing once more

- update guix:
`guix pull`

- reconfigure to intended current generation system config:
`sudo -E guix system -L . reconfigure CONFIG.scm`

- reboot

Leaving me in the following current/working state:

...
Generation 34 Mar 11 2025 18:09:09 (current)
file name: /var/guix/profiles/system-34-link
canonical file name: /gnu/store/17z7fpnbwbmi78v211xfm5r9wxhjgv6b-system
label: GNU with Linux 6.13.5
bootloader: grub-efi
root device: /dev/mapper/system-root
kernel: /gnu/store/skdgsqhqdwif0adykjg867i1n3xwfz9i-linux-6.13.5/bzImage
channels:
...
guix:
branch: master
commit: d685a45edf0f89e5876ffc9d880068d8610e5f8a
configuration file: /gnu/store/vnp7mhfhfff802zsp2vw74a7v0cv54hj-configuration.scm
...

Kind regards,
Bas
Leo Famulari wrote 9 hours ago
(name . Bas Alberts)(address . bas@anti.computer)(name . 76959@debbugs.gnu.org)(address . 76959@debbugs.gnu.org)
Z9EAwN41KL9l2F3P@jasmine.lan
FYI, I made a mistake and accidentally arrnaged for 3 copies of these
messages to be delivered. Of course, the Bas only meant to send a single
copy. So if you are annoyed by the overwhelming deluge of messages in
this ticket, blame me :)
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 76959@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 76959
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch
You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags
mumi command -t +confirmed -t +easy
Or, remove the moreinfo tag and set the help tag
mumi command -t -moreinfo -t +help