Test of guix-1.1.0-4.bdc801e is failing on aarch64

Hi!

Building guix-1.1.0-4.bdc801e is failing on aarch64 since about two weeks on ci.guix.gnu.org.

There is always the same failure during the test. See for example the raw log from http://ci.guix.gnu.org/build/2794788/details:

test-name: network-interface-names

…

actual-error:

+ (wrong-type-arg

+ "list-tail"

+ "Wrong type argument in position ~A (expecting ~A): ~S"

+ (1 "pair" ())

+ (()))

result: FAIL

When building locally this particular test is passing (after about three days on an SBC with 1 GB RAM).

But unfortunately building locally I get two different failures instead:

FAIL: tests/store.scm

FAIL: tests/guix-package.sh

Here are some more details:

test-name: verify-store + check-contents

location: /tmp/guix-build-guix-1.1.0-4.bdc801e.drv-0/source/tests/store.scm:972

…

actual-error:

+ (%exception

+ #<&store-protocol-error message: "path `dtmp/guix-tests/store/bpl866rgsjyzdczlngfw8ss2lkld6bim-mirrors' is not in the store" status: 1>)

result: FAIL

Note the “dtmp/” instead of “/tmp/”. I saw the same error some weeks ago already, but shortly after that was happening, there was a substitute available. But it seems this error is reproducible in my case.

guix install: error: profile t-guix-package-541/profile is locked by another process

+ true

+ kill 1387

+ rm -f t-profile-541 t-profile-541.lock t-profile-541-1-link t-guix-package-file-541

+ rm -rf t-guix-package-541 t-home-541

rm: cannot remove 't-guix-package-541': Directory not empty

FAIL tests/guix-package.sh (exit status: 1)

This may be due to a deleted but still opened file on an NFS share. There may be an intermediate hidden .nfs… file, which may get created in such a case (“delete on last close”, “silly rename”), However, the RFC-5661 for NFS demands even if OPEN4_RESULT_PRESERVE_UNLINKED is supported, that the directory entry of an open file must not be removed in this case, thus preventing a directory removal.

Hi!

The two errors when testing guix are still reproducible, now with guix-1.2.0rc2-1.0d4b1af, see the log for details.

I’m really wondering about the “dtmp/” instead of “/tmp/” bug. May this be a bug in guile itself?

Bye

Stefan

succeed on an NFS share, but this should not fail the test.

---

tests/guix-package.sh | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

Hi!

The “dtmp/” issue is not new. It appeared already in these bugs:

In all cases it happens in tests/store.scm:

test-name: verify-store + check-contents

This bug happened on x86_64 and aarch64.

The issues 27032 and 27034 where reported when the installer used a unionfs. Probably it was hidden when switching to an overlayfs.

This issue 41654 and 29676 show the “dtmp/” error when running this test on an NFS share.

Bye

Stefan

Hi Ludo’!

Could you please take a look at the bug report https://issues.guix.gnu.org/41654?

There are two reproducible problems when building the guix-1.2.0 package.

There is a patch already for the first of the two problems (an incomplete cleanup) at https://issues.guix.gnu.org/41654#2.

For the other issue with a wrong path "dtmp/guix-tests/store/", there is a summary at https://issues.guix.gnu.org/41654#3. This problem is existing since years and seems to be depending on the underlying file-system. It seems to appear with a tmp-store on a unionfs-fuse or NFS. In the past it got hidden by switching to overlayfs or changes on the NFS server side.

As the CI for aarch64 is always lagging behind these issues are kind of blockers to me. Today they block the build of guix-1.2.0-8.7624ebb.

Bye

Stefan

Hi!

Today I thought that it should be possible to work around this bug on my aarch64 system by mounting /tmp as a tmpfs, backed by swap on a small SD card, to prevent any issues due to my NFS root. This worked for the trap-cleanup-issue. But to my surprise this didn’t work out for the “dtmp/” error.

I was wondering, if using /tmp as a tmpfs on my virtual x86_64 system would make the test fail there as well. So I did

sudo mount -t tmpfs -o size=768M tmpfs /tmp

guix build --check --no-grafts guix@1.2.0-8.7624ebb

But there it was passing.

This is the interesting part from the log of the aarch64 system:

test-name: verify-store + check-contents

location: /tmp/guix-build-guix-1.2.0-8.7624ebb.drv-0/source/tests/store.scm:1156

source:

+ (test-assert

+ "verify-store + check-contents"

+ (with-store

+ s

+ (let* ((text (random-text))

+ (drv (build-expression->derivation

+ s

+ "corrupt"

+ `(let ((out (assoc-ref %outputs "out")))

+ (call-with-output-file

+ out

+ (lambda (port) (display ,text port)))

+ #t)

+ #:guile-for-build

+ (package-derivation

+ s

+ %bootstrap-guile

+ (%current-system))))

+ (file (derivation->output-path drv)))

+ (with-derivation-substitute

+ drv

+ text

+ (and (build-derivations s (list drv))

+ (verify-store s #:check-contents? #t)

+ (begin

+ (chmod file 420)

+ (call-with-output-file

+ file

+ (lambda (port) (display "corrupt!" port)))

+ #t)

+ (not (verify-store s #:check-contents? #t))

+ (delete-paths s (list file)))))))

actual-value: #f

actual-error:

+ (%exception

+ #<&store-protocol-error message: "path `dtmp/guix-tests/store/rn6xq5kaipp8aanhcnz2hvyfmr3y2laa-mirrors' is not in the store" status: 1>)

result: FAIL

Bye

Stefan