guix gc fails catastrophically if the disk has no space left

OpenSubmitted by Stefan.
Details
3 participants
  • Ludovic Courtès
  • raingloom
  • Stefan
Owner
unassigned
Severity
important
Merged with
S
S
Stefan wrote on 23 May 2020 16:49
(address . bug-guix@gnu.org)
A33D3EFC-90DD-4E78-AFE5-AB0BF146373A@vodafonemail.de
Hi!
While trying to build guix, my disk ran out of space.
building /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv...100% [#########################################################################]note: build failure may have been caused by lack of free disk spacebuilder for `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed with exit code 1build of /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv failedView build log at '/var/log/guix/drvs/w5/l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv.bz2'.guix system: error: build of `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed
stefan@guix ~$ df -h /Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf:/volume5/RaspberryPi/Betriebssysteme/guix-system 1,4T 1,4T 0 100% /
OK, well, there’s certainly a lot of garbage to collect. Due to bad experiences in the past with the same situation, I first try to collect only a little garbage.
stefan@guix ~$ guix gc -F 100Mguix gc: 100 MiB werden freigegebenfinding garbage collector roots...note: can't create trash directory: creating directory `/gnu/store/trash': No space left on devicedeleting garbage...[0%] deleting '/gnu/store/g30aknl5yxk9jbpx2w7r5d0ak6xa5dl2-grub.cfg.lock'[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'[0%] deleting '/gnu/store/9cpj3ir8fnsg5fiqs3x8alai713xzfz0-grub-keymap.de.lock'[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'[0%] deleting '/gnu/store/wv7iwd66q6n5775qa4hgaq9j9f2vyvl4-guix-1.1.0-4.bdc801e.lock'[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union.lock'[0%] deleting '/gnu/store/3ldyga22s6ym9g2pc9ycjwdwqb4i41v2-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'[0%] deleting '/gnu/store/nfxgqhjyzwngiadn0l180hsbfnw3h4zy-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union'[0%] deleting '/gnu/store/dahbiixsbi1xhjiim9phl1vginm83bbm-grub.cfg.lock'[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'[0%] deleting '/gnu/store/q829h36vnwhbdlnzkgdrx2b51kp2mvdv-grub.cfg.lock'[0%] deleting '/gnu/store/ragpi42x39sf5qwcmcmdn751fp4jfhmv-grub.cfg.lock'[0%] deleting '/gnu/store/ncdsjfl245k6r7zym6w4sr9fyxrwhb38-grub.cfg.lock'[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection'[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files.lock'[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files'error (ignored): aborting transaction: cannot rollback - no transaction is activeguix gc: error: committing transaction: disk I/O errorstefan@guix ~$
And that’s it. Repeating the guix gc command now results in this immediate error:
stefan@guix ~$ guix gc -F 100Mwarning: SQLite database is busy (SQLITE_PROTOCOL)guix gc: error: setting synchronous mode: locking protocolstefan@guix ~$ guix gc -F 100Mwarning: SQLite database is busy (SQLITE_PROTOCOL)guix gc: error: setting synchronous mode: locking protocol
A reboot is needed to solve this SQLite error. But retrying all this leads to the same situation again and again. It is kind of a deadlock.
Well, as my guix system boots from an NFS share, in the past I solved this by removing other stuff – not related to guix – from the same disk. But right now there isn't unrelated stuff left to be deleted.
I intend to remove the listed /gnu/store/… files by hand from the NFS server side (which is much easier than removing the read-only bind-mount.) Is this save at least?
This is all on a Raspberry Pi 3b aarch64 machine, if it matters.

Bye
Stefan
S
S
Stefan wrote on 23 May 2020 17:53
(address . 41484@debbugs.gnu.org)
1DFE09AD-D999-4CAC-8362-72772D791380@vodafonemail.de
Hi!
After this trouble with guix gc I did a sudo reboot, and it got worse. At startup I now see these errors:
GC Warning: pthread_getattr_up or pthread_attr_getstack failed for main threadGC Warning: Couldn't read /proc/statWelcome, this is GNU's early boot Guile.Use --repl for an initrd REPL.
loading kernel modules…loading '/gnu/store/…-system/boot'...making '/gnu/store/…-system' the current system...setting up setuid programs in '/run/setuid-programs'...warning: failed to make '/gnu/store/…-shadow-4.8.1/bin/newuidmap' setuid-root: No space left on devicewarning: failed to make '/gnu/store/…' setuid-root: No space left on devicepopulating /etc from /gnu/store/…-etc...ERROR: In procedure fsync:In procedure fsync: No space left on device
Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue.GNU Guile 2.2.6Copyright …
Enter `help' for help.scheme@(guile-user)> ,btIn gnu/build/linux-boot.scm: 561:14 10 (_)In unknown file: 9 (primitive-load "/gnu/store/…")In ice-9/eval.scm: 619:8 (_ #f)In unknown file: 7 (primitive-load "/gnu/store/…")In ice-9/boot-9.scm: 260:13 6 (for-each #<procedure primitive-load (_)> _)In unknown file: 5 (primitive-load "/gnu/store/_")In gnu/build/activation.scm: 136:2 4 (activate-users+groups _ _)In guix/build/syscalls.scm: 1089:4 3 (call-with-file-lock _ _)In gnu/build/activation.scm: 139:6 2 (_)In gnu/build/accounts.scm: 253:14 1 (_ (#<<group-entry> name: "root" password: #f gid: …> …) …)In unknown file: 0 (fsync #<input-output: /etc/group.bJpcke 18>)scheme@(guile-user)>

Now this is the very end. Even selecting an older system generation is obviously not an option any more.
I’m happy that all this is on my NFS share to which I have access via my server.

Bye
Stefan
R
R
raingloom wrote on 23 May 2020 21:57
(address . bug-guix@gnu.org)
20200523215733.2ed2f0de@riseup.net
On Sat, 23 May 2020 16:49:02 +0200Stefan <stefan-guix@vodafonemail.de> wrote:
Toggle quote (107 lines)> Hi!> > While trying to build guix, my disk ran out of space.> > building> /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv...> 100%> [#########################################################################]note:> build failure may have been caused by lack of free disk space builder> for> `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv'> failed with exit code 1 build of> /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv> failed View build log at> '/var/log/guix/drvs/w5/l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv.bz2'.> guix system: error: build of> `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv'> failed> > stefan@guix ~$ df -h /> Dateisystem Größe Benutzt Verf.> Verw% Eingehängt auf> :/volume5/RaspberryPi/Betriebssysteme/guix-system 1,4T 1,4T 0> 100% /> > OK, well, there’s certainly a lot of garbage to collect. Due to bad> experiences in the past with the same situation, I first try to> collect only a little garbage.> > stefan@guix ~$ guix gc -F 100M> guix gc: 100 MiB werden freigegeben> finding garbage collector roots...> note: can't create trash directory: creating directory> `/gnu/store/trash': No space left on device deleting garbage...> [0%] deleting> '/gnu/store/g30aknl5yxk9jbpx2w7r5d0ak6xa5dl2-grub.cfg.lock' [0%]> deleting> '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'> [0%] deleting> '/gnu/store/9cpj3ir8fnsg5fiqs3x8alai713xzfz0-grub-keymap.de.lock'> [0%] deleting> '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'> [0%] deleting> '/gnu/store/wv7iwd66q6n5775qa4hgaq9j9f2vyvl4-guix-1.1.0-4.bdc801e.lock'> [0%] deleting> '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union.lock'> [0%] deleting> '/gnu/store/3ldyga22s6ym9g2pc9ycjwdwqb4i41v2-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'> [0%] deleting> '/gnu/store/nfxgqhjyzwngiadn0l180hsbfnw3h4zy-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'> [0%] deleting> '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union'> [0%] deleting> '/gnu/store/dahbiixsbi1xhjiim9phl1vginm83bbm-grub.cfg.lock' [0%]> deleting> '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'> [0%] deleting> '/gnu/store/q829h36vnwhbdlnzkgdrx2b51kp2mvdv-grub.cfg.lock' [0%]> deleting '/gnu/store/ragpi42x39sf5qwcmcmdn751fp4jfhmv-grub.cfg.lock'> [0%] deleting> '/gnu/store/ncdsjfl245k6r7zym6w4sr9fyxrwhb38-grub.cfg.lock' [0%]> deleting> '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'> [0%] deleting> '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection'> [0%] deleting> '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'> [0%] deleting> '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files.lock'> [0%] deleting> '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files' error> (ignored): aborting transaction: cannot rollback - no transaction is> active guix gc: error: committing transaction: disk I/O error> stefan@guix ~$ > > And that’s it. Repeating the guix gc command now results in this> immediate error:> > stefan@guix ~$ guix gc -F 100M> warning: SQLite database is busy (SQLITE_PROTOCOL)> guix gc: error: setting synchronous mode: locking protocol> stefan@guix ~$ guix gc -F 100M> warning: SQLite database is busy (SQLITE_PROTOCOL)> guix gc: error: setting synchronous mode: locking protocol> > A reboot is needed to solve this SQLite error. But retrying all this> leads to the same situation again and again. It is kind of a deadlock.> > Well, as my guix system boots from an NFS share, in the past I solved> this by removing other stuff – not related to guix – from the same> disk. But right now there isn't unrelated stuff left to be deleted. > > I intend to remove the listed /gnu/store/… files by hand from the NFS> server side (which is much easier than removing the read-only> bind-mount.) Is this save at least?> > This is all on a Raspberry Pi 3b aarch64 machine, if it matters.> > > Bye> > Stefan> > > >
Try deleting /var/log. That's what I did on my tiny Pentium II desktopwith 4 gigs of HDD space.
L
L
Ludovic Courtès wrote on 16 Jan 14:51 +0100
control message for bug #41484
(address . control@debbugs.gnu.org)
87ft31yopu.fsf@gnu.org
severity 41484 importantquit
L
L
Ludovic Courtès wrote on 16 Jan 15:23 +0100
control message for bug #45266
(address . control@debbugs.gnu.org)
87czy5yn90.fsf@gnu.org
merge 45266 41484quit
?