guix incorrectly says "No space left on device"

  • Open
  • quality assurance status badge
Details
3 participants
  • Jesse Gibbons
  • Julien Lepiller
  • Tobias Geerinckx-Rice
Owner
unassigned
Submitted by
Jesse Gibbons
Severity
normal
J
J
Jesse Gibbons wrote on 25 Feb 2020 04:01
(address . bug-guix@gnu.org)
6c750a5f65242c052a75df812123b95f94592b70.camel@gmail.com
I have a laptop with two drives. A few days ago, when I ran `df -h` it
outputs:
Filesystem Size Used Avail Use% Mounted on
none 16G 0 16G 0% /dev
/dev/sdb1 229G 189G 29G 87% /
/dev/sda1 458G 136G 299G 32% /gnu/store
tmpfs 16G 0 16G 0% /dev/shm
none 16G 64K 16G 1% /run/systemd
none 16G 0 16G 0% /run/user
cgroup 16G 0 16G 0% /sys/fs/cgroup
tmpfs 3.2G 16K 3.2G 1% /run/user/983
tmpfs 3.2G 60K 3.2G 1% /run/user/1001

As you can see, /dev/sda1 is the drive mounted on /gnu/store.
Everything in the store is written to it, and it has plenty of space
available.

Guix sometimes says there is "No space left on device". This always
happens in particular when I try `guix gc --optimize`, but it sometimes
happens when I call `guix pull` or `guix upgrade`. When guix pull or
guix upgrade fails with this message, I can clear up more space by
deleting ~/.cache and emtpying my trash and it works.


Today I have also seen this happen when I'm trying to upgrade a large
profile. It said it could not build anything because there was no more
disk space, even after I cleaned up /dev/sdb1 to 40% use. It finally
recognized the empty disk space when I called guix gc and it deleted a
few of the dependencies needed for the upgrades. But it didn't take
long to trigger this bug again. Here's the new output of `df -h`:

Filesystem Size Used Avail Use% Mounted on
none 16G 0 16G 0% /dev
/dev/sdb1 229G 86G 131G 40% /
/dev/sda1 458G 182G 253G 42% /gnu/store
tmpfs 16G 0 16G 0% /dev/shm
none 16G 80K 16G 1% /run/systemd
none 16G 0 16G 0% /run/user
cgroup 16G 0 16G 0% /sys/fs/cgroup
tmpfs 3.2G 24K 3.2G 1% /run/user/983
tmpfs 3.2G 12K 3.2G 1% /run/user/1000
tmpfs 3.2G 60K 3.2G 1% /run/user/1001

Any clues why this happens and what can be done to fix it? Could it be
related to how /dev/sdb1 is 229G large, and the total used space in /
and /gnu/store is more than that?

-Jesse
T
T
Tobias Geerinckx-Rice wrote on 25 Feb 2020 07:15
Re: bug#39774: guix says "No space left on device"
(address . 39774@debbugs.gnu.org)
87wo8bry3w.fsf@nckx
Jesse,

Are you sure Guix is incorrect?

$ df -i / /gnu/store

If you're (almost) out of inodes on either, you'll have to delete
some files.

In which case you can thank Unix for -ENOSPC meaning ‘you're out
of something; I won't tell you what’, but perhaps Guix could work
around its flaws by doing our own stat() and displaying a hint.
We already warn about low storage space. It makes sense to do the
same about inodes.

Kind regards,

T G-R
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEfo+u0AlEeO9y5k0W2Imw8BjFSTwFAl5Uu5MACgkQ2Imw8BjF
STyPDw/+NGsqyILMMFJ5/AnNpCCbPSEuukdTRUfe086QnwCYHoDpl4QDi9SiOA3K
ohwqYivUEVkE/0Kffv5i4bUkwfYsYNiBavFHTXaRJc1hvZEXSQh8sY/1Bz+K9e/F
w3bmZbiJW5F3HtHjfMdspFzNL5eoYWhGJyBvuR4rEjttgs8ynMD4EzCukEZZ5fTH
J9ZzsGDgd/PKd1kI2a+29QZmpZ1xJ2EMPGIspqCnAcOjjkABKUPW1Kmm4DobjXPM
qCt7/ca7ldbmIV+nn63VGO9eO5FdeoI/NKt3woyd1lNem3Lw2fR5NmiTpGcrzDug
jsD4S23VlsiSfD1kwpq7GYCHMVGQqbKgn1cebcerTlXnplc8TTRq/8xZlFE9w2/1
qTGmv1Y0v6kukdRjDKQUBCRuZD3so7KxgbKrw73QMgjqKCfT/YYLOR2V9wr7vFdB
KLRTU+sQkOXa6HRl4AwiYR/8LzHnqpubw7kijnt8UCHheTb63JENAAPYPIZfXeSq
5yy6qg9xhc4/u1XKD/7GOSFwy1QbVmw1s9qoMZWKd/n2LHAQR7OyMP0ywfW8F/je
yNRAtDhExEDlD5aZDQ1X4eqWpHdvyoT3bRRChwlJMLWsoZK4BFaRUSb0IjNnSmOy
+781UiyRHRxFYLmSGfHLjatR/vEJ2BpDcjXQpTAxaBFcPNPDD1E=
=G+9t
-----END PGP SIGNATURE-----

J
J
Julien Lepiller wrote on 25 Feb 2020 04:15
Re: bug#39774: guix incorrectly says "No space left on device"
31D6F2BA-A08C-49C4-AD20-6076036A215C@lepiller.eu
Le 24 février 2020 22:01:45 GMT-05:00, Jesse Gibbons <jgibbons2357@gmail.com> a écrit :
Toggle quote (49 lines)
>I have a laptop with two drives. A few days ago, when I ran `df -h` it
>outputs:
>Filesystem Size Used Avail Use% Mounted on
>none 16G 0 16G 0% /dev
>/dev/sdb1 229G 189G 29G 87% /
>/dev/sda1 458G 136G 299G 32% /gnu/store
>tmpfs 16G 0 16G 0% /dev/shm
>none 16G 64K 16G 1% /run/systemd
>none 16G 0 16G 0% /run/user
>cgroup 16G 0 16G 0% /sys/fs/cgroup
>tmpfs 3.2G 16K 3.2G 1% /run/user/983
>tmpfs 3.2G 60K 3.2G 1% /run/user/1001
>
>As you can see, /dev/sda1 is the drive mounted on /gnu/store.
>Everything in the store is written to it, and it has plenty of space
>available.
>
>Guix sometimes says there is "No space left on device". This always
>happens in particular when I try `guix gc --optimize`, but it sometimes
>happens when I call `guix pull` or `guix upgrade`. When guix pull or
>guix upgrade fails with this message, I can clear up more space by
>deleting ~/.cache and emtpying my trash and it works.
>
>
>Today I have also seen this happen when I'm trying to upgrade a large
>profile. It said it could not build anything because there was no more
>disk space, even after I cleaned up /dev/sdb1 to 40% use. It finally
>recognized the empty disk space when I called guix gc and it deleted a
>few of the dependencies needed for the upgrades. But it didn't take
>long to trigger this bug again. Here's the new output of `df -h`:
>
>Filesystem Size Used Avail Use% Mounted on
>none 16G 0 16G 0% /dev
>/dev/sdb1 229G 86G 131G 40% /
>/dev/sda1 458G 182G 253G 42% /gnu/store
>tmpfs 16G 0 16G 0% /dev/shm
>none 16G 80K 16G 1% /run/systemd
>none 16G 0 16G 0% /run/user
>cgroup 16G 0 16G 0% /sys/fs/cgroup
>tmpfs 3.2G 24K 3.2G 1% /run/user/983
>tmpfs 3.2G 12K 3.2G 1% /run/user/1000
>tmpfs 3.2G 60K 3.2G 1% /run/user/1001
>
>Any clues why this happens and what can be done to fix it? Could it be
>related to how /dev/sdb1 is 229G large, and the total used space in /
>and /gnu/store is more than that?
>
>-Jesse

There could be two explanations: you've run out of inodes or the filesystem that was out of space is not the one you think (maybe it was during a build and your /tmp is a tmpfs?). Try `df -i`.
J
J
Jesse Gibbons wrote on 25 Feb 2020 15:59
4af517dc0f1203daabdf9b8053a8e302dbf811af.camel@gmail.com
On Mon, 2020-02-24 at 22:15 -0500, Julien Lepiller wrote:
Toggle quote (64 lines)
> Le 24 février 2020 22:01:45 GMT-05:00, Jesse Gibbons <
> jgibbons2357@gmail.com> a écrit :
> > I have a laptop with two drives. A few days ago, when I ran `df -h`
> > it
> > outputs:
> > Filesystem Size Used Avail Use% Mounted on
> > none 16G 0 16G 0% /dev
> > /dev/sdb1 229G 189G 29G 87% /
> > /dev/sda1 458G 136G 299G 32% /gnu/store
> > tmpfs 16G 0 16G 0% /dev/shm
> > none 16G 64K 16G 1% /run/systemd
> > none 16G 0 16G 0% /run/user
> > cgroup 16G 0 16G 0% /sys/fs/cgroup
> > tmpfs 3.2G 16K 3.2G 1% /run/user/983
> > tmpfs 3.2G 60K 3.2G 1% /run/user/1001
> >
> > As you can see, /dev/sda1 is the drive mounted on /gnu/store.
> > Everything in the store is written to it, and it has plenty of
> > space
> > available.
> >
> > Guix sometimes says there is "No space left on device". This always
> > happens in particular when I try `guix gc --optimize`, but it
> > sometimes
> > happens when I call `guix pull` or `guix upgrade`. When guix pull
> > or
> > guix upgrade fails with this message, I can clear up more space by
> > deleting ~/.cache and emtpying my trash and it works.
> >
> >
> > Today I have also seen this happen when I'm trying to upgrade a
> > large
> > profile. It said it could not build anything because there was no
> > more
> > disk space, even after I cleaned up /dev/sdb1 to 40% use. It
> > finally
> > recognized the empty disk space when I called guix gc and it
> > deleted a
> > few of the dependencies needed for the upgrades. But it didn't take
> > long to trigger this bug again. Here's the new output of `df -h`:
> >
> > Filesystem Size Used Avail Use% Mounted on
> > none 16G 0 16G 0% /dev
> > /dev/sdb1 229G 86G 131G 40% /
> > /dev/sda1 458G 182G 253G 42% /gnu/store
> > tmpfs 16G 0 16G 0% /dev/shm
> > none 16G 80K 16G 1% /run/systemd
> > none 16G 0 16G 0% /run/user
> > cgroup 16G 0 16G 0% /sys/fs/cgroup
> > tmpfs 3.2G 24K 3.2G 1% /run/user/983
> > tmpfs 3.2G 12K 3.2G 1% /run/user/1000
> > tmpfs 3.2G 60K 3.2G 1% /run/user/1001
> >
> > Any clues why this happens and what can be done to fix it? Could it
> > be
> > related to how /dev/sdb1 is 229G large, and the total used space in
> > /
> > and /gnu/store is more than that?
> >
> > -Jesse
>
> There could be two explanations: you've run out of inodes or the
> filesystem that was out of space is not the one you think (maybe it
> was during a build and your /tmp is a tmpfs?). Try `df -i`.
~$ df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
none 4.0M 525 4.0M 1% /dev
/dev/sdb1 15M 77K 15M 1% /
/dev/sda1 30M 29M 1015K 97% /gnu/store
tmpfs 4.0M 1 4.0M 1% /dev/shm
none 4.0M 47 4.0M 1% /run/systemd
none 4.0M 4 4.0M 1% /run/user
cgroup 4.0M 11 4.0M 1% /sys/fs/cgroup
tmpfs 4.0M 13 4.0M 1% /run/user/983
tmpfs 4.0M 24 4.0M 1% /run/user/1001
tmpfs 4.0M 1 4.0M 1% /run/user/1000

That makes sense now. /dev/sda1 (mounted on /gnu/store) was out of
inodes. Is there a way to increase the maximum number of inodes a
partition can use? Or perhaps divide the store among multiple
partitions?
?