GC test failure on Btrfs

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Marius Bakke
  • Rutger Helling
Owner
unassigned
Submitted by
Rutger Helling
Severity
normal
R
R
Rutger Helling wrote on 20 Nov 2017 11:08
Single test failure building Guix
(address . bug-guix@gnu.org)
89edb4cc307bfae5db7812abe3b4a37a@mykolab.com
Hi,

when building Guix with 'guix build guix' I keep running into a single
test failure. I've attached the test-suite.log.

My kernel:

$ uname -a
Linux guixsd 4.14.0-gnu #1 SMP 1 x86_64 GNU/Linux

This is the guix-daemon I'm building with:

$ guix gc --references
/gnu/store/mwvmdfksm9iwj1symfiinlikiz56s2nl-guix-0.13.0-9.ff23b47/bin/guix-daemon
/gnu/store/2b22079yrvs59j256z4scccq506csy7c-guile-git-0.0-4.951a32c
/gnu/store/3h31zsqxjjg52da5gp3qmhkh4x8klhah-glibc-2.25
/gnu/store/5jgpy83fga8jjj02m0ncyvgggmbwsfdy-gnutls-3.5.13
/gnu/store/5rk1b2riamhzk8s1hs83mb1i8vdif0vd-gzip-1.8
/gnu/store/6d4ihp7xbdh3a0ffbpm5n45q4v3w0l35-sqlite-3.19.3
/gnu/store/6wyjls0q2c9gjskkplsr1ad09p3d8gzg-gcc-5.4.0-lib
/gnu/store/b1dg82khbvr2abaa346vv7r93ryqrb3j-xz-5.2.2
/gnu/store/bcmf06k2n1pfwqkzpclvvc3w9jdfi71a-guile-json-0.6.0
/gnu/store/f8k940vy9gck66m9r4id5m098w3hxgka-bash-minimal-4.4.12
/gnu/store/mwvmdfksm9iwj1symfiinlikiz56s2nl-guix-0.13.0-9.ff23b47
/gnu/store/navpkpm1jf6zf8zmi54wl5w3b2ddv1sw-zlib-1.2.11
/gnu/store/qfzl5frp52wdz1vbdj958sz35yfl94xi-libgcrypt-1.8.1
/gnu/store/swyipr8smrd5bc72n92sdfxzx0p4cjpi-guile-2.2.2
/gnu/store/wd01nlc7apixl4kyvhpzasxprqymw32s-bzip2-1.0.6
/gnu/store/xfaqdvk060yz7ddc9isk3wkybqmcfj3w-guile-ssh-0.11.2

Does anyone have any idea about what's going wrong?
Attachment: test-suite.log (.42 MiB)
L
L
Ludovic Courtès wrote on 20 Nov 2017 16:55
(name . Rutger Helling)(address . rhelling@mykolab.com)(address . 29363@debbugs.gnu.org)
874lpp3pxw.fsf@gnu.org
Hi Rutger,

Rutger Helling <rhelling@mykolab.com> skribis:

Toggle quote (18 lines)
> test-name: dead path can be explicitly collected
> location: /tmp/guix-build-guix-0.13.0-10.0b4c385.drv-0/source/tests/store.scm:178
> source:
> + (test-assert
> + "dead path can be explicitly collected"
> + (let ((p (add-text-to-store
> + %store
> + "random-text"
> + (random-text)
> + '())))
> + (let-values
> + (((paths freed) (delete-paths %store (list p))))
> + (and (equal? paths (list p))
> + (> freed 0)
> + (not (file-exists? p))))))
> actual-value: #f
> result: FAIL

I didn’t experience this on my laptop. Is it reproducible if you run
“guix build guix” a second time?

Thanks,
Ludo’.
R
R
Rutger Helling wrote on 20 Nov 2017 18:49
(address . ludo@gnu.org)
3050db67ea7f97da2733f9aacfef687f@mykolab.com
Hi Ludo,

it is indeed reproducible. No matter how many times it always keeps
failing on this one test.

I've had this problem for a long time, which is a little bit annoying
since it means I have to wait until a substitute is available every
time.

On 2017-11-20 16:55, ludo@gnu.org wrote:

Toggle quote (27 lines)
> Hi Rutger,
>
> Rutger Helling <rhelling@mykolab.com> skribis:
>
>> test-name: dead path can be explicitly collected
>> location: /tmp/guix-build-guix-0.13.0-10.0b4c385.drv-0/source/tests/store.scm:178
>> source:
>> + (test-assert
>> + "dead path can be explicitly collected"
>> + (let ((p (add-text-to-store
>> + %store
>> + "random-text"
>> + (random-text)
>> + '())))
>> + (let-values
>> + (((paths freed) (delete-paths %store (list p))))
>> + (and (equal? paths (list p))
>> + (> freed 0)
>> + (not (file-exists? p))))))
>> actual-value: #f
>> result: FAIL
>
> I didn't experience this on my laptop. Is it reproducible if you run
> "guix build guix" a second time?
>
> Thanks,
> Ludo'.
Attachment: file
M
M
Marius Bakke wrote on 21 Nov 2017 01:31
87a7zgtquv.fsf@fastmail.com
Hi Rutger,

Rutger Helling <rhelling@mykolab.com> writes:

Toggle quote (3 lines)
> when building Guix with 'guix build guix' I keep running into a single
> test failure. I've attached the test-suite.log.

Is this a Btrfs system by any chance, possibly on an SSD?

Toggle quote (18 lines)
> test-name: dead path can be explicitly collected
> location: /tmp/guix-build-guix-0.13.0-10.0b4c385.drv-0/source/tests/store.scm:178
> source:
> + (test-assert
> + "dead path can be explicitly collected"
> + (let ((p (add-text-to-store
> + %store
> + "random-text"
> + (random-text)
> + '())))
> + (let-values
> + (((paths freed) (delete-paths %store (list p))))
> + (and (equal? paths (list p))
> + (> freed 0)
> + (not (file-exists? p))))))
> actual-value: #f
> result: FAIL

I can reproduce this error on two different systems that have
Btrfs+LUKS+SSD, and the problem is that freed == 0.

I suspect it's related to Btrfs' "lazy" reporting of disk space, but
haven't dug very far.

Until we figure out what's going on, I suggest applying the patch
below. Can you confirm that it works on your system?
From bdc7b5310111e21801529ea57e290f6eb72ac6ed Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Tue, 21 Nov 2017 00:27:08 +0100
Subject: [PATCH] gnu: guix: Disable test that fails on Btrfs.

Reported by Rutger Helling <rhelling@mykolab.com>.

* gnu/packages/package-management.scm (guix)[arguments]: Rename
'disable-container-tests' phase to 'disable-failing-tests' and add substitution
to disable "dead path can be explicitly collected" test.
---
gnu/packages/package-management.scm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

Toggle diff (28 lines)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 4f1f7f577..3321ab1eb 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -169,8 +169,7 @@
(copy "armhf")
(copy "aarch64")
#t))
- (add-after
- 'unpack 'disable-container-tests
+ (add-after 'unpack 'disable-failing-tests
;; XXX FIXME: These tests fail within the build container.
(lambda _
(substitute* "tests/syscalls.scm"
@@ -183,6 +182,11 @@
(substitute* "tests/guix-environment-container.sh"
(("guix environment --version")
"exit 77\n")))
+ ;; XXX: This test may fail on some file systems.
+ ;; See <https://bugs.gnu.org/29363>.
+ (substitute* "tests/store.scm"
+ (("^(.*dead path can be explicitly collected\")" all)
+ (string-append "(test-skip 1)\n" all)))
#t))
(add-before 'check 'set-SHELL
(lambda _
--
2.15.0
Ludo, WDYT?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAloTc8gACgkQoqBt8qM6
VPp3VggAyASN/948LEV659qQUmlVVDOVJWibXdyBPu92Vo0VLGMoDxgOt+qBgRM7
iFx8aeEIvIHE+Rq6V7/WRDfZvoXgNiTI9khHHbgVVMsLWbmnwLjBqH5Nv2Lxx0Je
yAQ94T2/r1PJCGwP4T3kDIKT6x6uCcvPkuVksfsIpZ9VBfhPljp9nVP+VqxBOL42
1tEwlXbB+12p+CklVHtomvBkE2SiPNW3IsamZT7LKiQg6UJmFUSzHSAuwWwVQ2Ax
6SSyXMshuFJc9G0Bgc2PS0iGL3YdECMNJfibGAY+dXnWGUDBegKQ3xHU2eZewnRE
V1c01x+iW6Vdh5ugh2gaRmfIlBCL/Q==
=05cl
-----END PGP SIGNATURE-----

R
R
Rutger Helling wrote on 21 Nov 2017 08:31
(name . Marius Bakke)(address . mbakke@fastmail.com)
265e7019513d69adc9be947d5a59fc13@mykolab.com
Hi Marius,

your patch did the trick, thanks!

I'm indeed on Btrfs (with LUKS), no SSD though.

On 2017-11-21 01:31, Marius Bakke wrote:

Toggle quote (37 lines)
> Hi Rutger,
>
> Rutger Helling <rhelling@mykolab.com> writes:
>
>> when building Guix with 'guix build guix' I keep running into a single
>> test failure. I've attached the test-suite.log.
>
> Is this a Btrfs system by any chance, possibly on an SSD?
>
>> test-name: dead path can be explicitly collected
>> location: /tmp/guix-build-guix-0.13.0-10.0b4c385.drv-0/source/tests/store.scm:178
>> source:
>> + (test-assert
>> + "dead path can be explicitly collected"
>> + (let ((p (add-text-to-store
>> + %store
>> + "random-text"
>> + (random-text)
>> + '())))
>> + (let-values
>> + (((paths freed) (delete-paths %store (list p))))
>> + (and (equal? paths (list p))
>> + (> freed 0)
>> + (not (file-exists? p))))))
>> actual-value: #f
>> result: FAIL
>
> I can reproduce this error on two different systems that have
> Btrfs+LUKS+SSD, and the problem is that freed == 0.
>
> I suspect it's related to Btrfs' "lazy" reporting of disk space, but
> haven't dug very far.
>
> Until we figure out what's going on, I suggest applying the patch
> below. Can you confirm that it works on your system?
>
> Ludo, WDYT?
Attachment: file
L
L
Ludovic Courtès wrote on 21 Nov 2017 08:47
(name . Marius Bakke)(address . mbakke@fastmail.com)
871sksoyxj.fsf@gnu.org
Hello,

Marius Bakke <mbakke@fastmail.com> skribis:

Toggle quote (28 lines)
> Rutger Helling <rhelling@mykolab.com> writes:
>
>> when building Guix with 'guix build guix' I keep running into a single
>> test failure. I've attached the test-suite.log.
>
> Is this a Btrfs system by any chance, possibly on an SSD?
>
>> test-name: dead path can be explicitly collected
>> location: /tmp/guix-build-guix-0.13.0-10.0b4c385.drv-0/source/tests/store.scm:178
>> source:
>> + (test-assert
>> + "dead path can be explicitly collected"
>> + (let ((p (add-text-to-store
>> + %store
>> + "random-text"
>> + (random-text)
>> + '())))
>> + (let-values
>> + (((paths freed) (delete-paths %store (list p))))
>> + (and (equal? paths (list p))
>> + (> freed 0)
>> + (not (file-exists? p))))))
>> actual-value: #f
>> result: FAIL
>
> I can reproduce this error on two different systems that have
> Btrfs+LUKS+SSD, and the problem is that freed == 0.

If you comment out (> freed 0), does the test pass?

Toggle quote (18 lines)
> I suspect it's related to Btrfs' "lazy" reporting of disk space, but
> haven't dug very far.
>
> Until we figure out what's going on, I suggest applying the patch
> below. Can you confirm that it works on your system?
>
> From bdc7b5310111e21801529ea57e290f6eb72ac6ed Mon Sep 17 00:00:00 2001
> From: Marius Bakke <mbakke@fastmail.com>
> Date: Tue, 21 Nov 2017 00:27:08 +0100
> Subject: [PATCH] gnu: guix: Disable test that fails on Btrfs.
>
> Works around <https://bugs.gnu.org/29363>.
> Reported by Rutger Helling <rhelling@mykolab.com>.
>
> * gnu/packages/package-management.scm (guix)[arguments]: Rename
> 'disable-container-tests' phase to 'disable-failing-tests' and add substitution
> to disable "dead path can be explicitly collected" test.

Alternately, we could comment out (> freed 0) if that’s enough, with a
comment explaining why, and do “make update-guix-package”. That way
we’d avoid the extra build phase.

WDYT?

Thanks for finding out the root cause!

Ludo’.
L
L
Ludovic Courtès wrote on 21 Nov 2017 08:48
control message for bug #29363
(address . control@debbugs.gnu.org)
87zi7gnkck.fsf@gnu.org
retitle 29363 GC test failure on Btrfs
R
R
Rutger Helling wrote on 21 Nov 2017 10:10
Re: bug#29363: Single test failure building Guix
(address . ludo@gnu.org)
4464438fc18e9ae461ab8d8a9959b04c@mykolab.com
Commenting out that line still made the test fail for me.

On 2017-11-21 08:47, ludo@gnu.org wrote:

Toggle quote (30 lines)
> Hello,
>
> Marius Bakke <mbakke@fastmail.com> skribis:
>
> Rutger Helling <rhelling@mykolab.com> writes:
>
> when building Guix with 'guix build guix' I keep running into a single
> test failure. I've attached the test-suite.log.
> Is this a Btrfs system by any chance, possibly on an SSD?
>
> test-name: dead path can be explicitly collected
> location: /tmp/guix-build-guix-0.13.0-10.0b4c385.drv-0/source/tests/store.scm:178
> source:
> + (test-assert
> + "dead path can be explicitly collected"
> + (let ((p (add-text-to-store
> + %store
> + "random-text"
> + (random-text)
> + '())))
> + (let-values
> + (((paths freed) (delete-paths %store (list p))))
> + (and (equal? paths (list p))
> + (> freed 0)
> + (not (file-exists? p))))))
> actual-value: #f
> result: FAIL
> I can reproduce this error on two different systems that have
> Btrfs+LUKS+SSD, and the problem is that freed == 0.

If you comment out (> freed 0), does the test pass?

Toggle quote (18 lines)
> I suspect it's related to Btrfs' "lazy" reporting of disk space, but
> haven't dug very far.
>
> Until we figure out what's going on, I suggest applying the patch
> below. Can you confirm that it works on your system?
>
> From bdc7b5310111e21801529ea57e290f6eb72ac6ed Mon Sep 17 00:00:00 2001
> From: Marius Bakke <mbakke@fastmail.com>
> Date: Tue, 21 Nov 2017 00:27:08 +0100
> Subject: [PATCH] gnu: guix: Disable test that fails on Btrfs.
>
> Works around <https://bugs.gnu.org/29363>.
> Reported by Rutger Helling <rhelling@mykolab.com>.
>
> * gnu/packages/package-management.scm (guix)[arguments]: Rename
> 'disable-container-tests' phase to 'disable-failing-tests' and add substitution
> to disable "dead path can be explicitly collected" test.

Alternately, we could comment out (> freed 0) if that's enough, with a
comment explaining why, and do "make update-guix-package". That way
we'd avoid the extra build phase.

WDYT?

Thanks for finding out the root cause!

Ludo'.
Attachment: file
L
L
Ludovic Courtès wrote on 21 Nov 2017 13:53
(name . Rutger Helling)(address . rhelling@mykolab.com)
87tvxnn67k.fsf@gnu.org
Rutger Helling <rhelling@mykolab.com> skribis:

Toggle quote (2 lines)
> Commenting out that line still made the test fail for me.

Could you figure out which of the conditions in the ‘and’ is failing?

Toggle quote (4 lines)
> + (and (equal? paths (list p))
> + (> freed 0)
> + (not (file-exists? p))))))

TIA,
Ludo’.
R
R
Rutger Helling wrote on 21 Nov 2017 15:50
(address . ludo@gnu.org)
18611c9bc29c707f3efbc602735857d0@mykolab.com
I think commenting out (> freed 0) does work after all. When I ran 'guix
environment --pure guix' and 'make check TESTS=tests/store.scm' against
my git checkout the test passed.

On 2017-11-21 13:53, ludo@gnu.org wrote:

Toggle quote (12 lines)
> Rutger Helling <rhelling@mykolab.com> skribis:
>
>> Commenting out that line still made the test fail for me.
>
> Could you figure out which of the conditions in the 'and' is failing?
>
>> + (and (equal? paths (list p))
>> + (> freed 0)
>> + (not (file-exists? p))))))
>
> TIA,
> Ludo'.
Attachment: file
M
M
Marius Bakke wrote on 21 Nov 2017 22:11
(address . 29363@debbugs.gnu.org)
878tezqqus.fsf@fastmail.com
Rutger Helling <rhelling@mykolab.com> writes:

Toggle quote (4 lines)
> I think commenting out (> freed 0) does work after all. When I ran 'guix
> environment --pure guix' and 'make check TESTS=tests/store.scm' against
> my git checkout the test passed.

Thanks for confirming. I pushed a revised patch that only removes the
test for freed space as per Ludos suggestion.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAloUlosACgkQoqBt8qM6
VPrg3Qf/VWRS4xqjxdU0x1WYGWYk3OTtpcuohmiAcEpxWEKJWC3aPF5c+OGPhJkL
5wiLLAj6FAGEa2yJlmINRl494w8Lc40KgpsHOmCRobCfbsTB70Tn7475rcfgwCZ+
O86Ymqll84XDzVs9pCTaCSwiMoW7f5NW3w1G+FBJYtHvSorK8nQXKLNHeF9DGDbU
ShNEfAFQy+xBCyQoY8Bl95wnaxso8rO/8gafaJi1j6EboJgPvvF2fCWjpN+zkx6z
/958tfjsalKxE0DDpko7cnkvVQxrZF1S5joJidanpA4nykg76OeK7jsqavbVfjs1
/APzHneCwAjBNeIuzQybt29BMnlOuw==
=8bJT
-----END PGP SIGNATURE-----

M
M
Marius Bakke wrote on 21 Nov 2017 22:39
(name . Ludovic Courtès)(address . ludo@gnu.org)
87zi7fpb0o.fsf@fastmail.com
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (6 lines)
> Alternately, we could comment out (> freed 0) if that’s enough, with a
> comment explaining why, and do “make update-guix-package”. That way
> we’d avoid the extra build phase.
>
> WDYT?

Oh, I just re-read this message and realized you mentioned doing this in
tests/store.scm rather than working around it in the "guix" package.

Sound good to me. I will do this and remove the workaround. Sorry for
the confusion.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAloUnPcACgkQoqBt8qM6
VPpw1Af/Ypq/inSRnCI9Lf4xCrU3+OGjuzvoy4/Ow3xngOiPO34VmZ/wy3O7tVHr
iTHITHjFUmTYBFz1zxIrhaiZlqbpjYD4FxFKrJViGnQjN1GCcdtGZ1A+V9LHvj87
EQ9BZX7PdMw7bRztD4QOuOH6CtzN/HXlvA5ahGx3NYg3pBCv5aHsuf35b5j3Sh4e
hITFSlFh0CBL1mxfaX0G2ZiO+CCdy5DxQURPocDKJv9W0OpTa+CZssIo4l35s7B+
met/LCwQ7F0tLI/bTlrJtcMoC1RPr1J5Z3ODF2Nm0s2zB2vC5AbISJfJmBcDONEz
sZfou2obPl8iQi9yjOwsER/n5H5InA==
=4ZRP
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 22 Nov 2017 16:55
(name . Marius Bakke)(address . mbakke@fastmail.com)
87tvxmwbnd.fsf@gnu.org
Heya!

Marius Bakke <mbakke@fastmail.com> skribis:

Toggle quote (13 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>> Alternately, we could comment out (> freed 0) if that’s enough, with a
>> comment explaining why, and do “make update-guix-package”. That way
>> we’d avoid the extra build phase.
>>
>> WDYT?
>
> Oh, I just re-read this message and realized you mentioned doing this in
> tests/store.scm rather than working around it in the "guix" package.
>
> Sound good to me. I will do this and remove the workaround. Sorry for
> the confusion.

I see you’ve done this now, so I’m closing the bug.

Thank you gentlefolks!

Ludo’.
Closed
?