qemu-minimal-5.2.0 fails to build during test phase on aarch64

  • Done
  • quality assurance status badge
Details
2 participants
  • Maxim Cournoyer
  • Stefan
Owner
unassigned
Submitted by
Stefan
Severity
normal
Merged with
S
S
Stefan wrote on 20 Sep 2020 17:43
qemu-minimal-5.1.0 fails to build during test phase on aarch64
(address . bug-guix@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
F3951E9F-690C-4AD5-915D-4E8E462030F1@vodafonemail.de
Hi!

Building qemu-minimal-5.1.0 based on commit 4d82e8e5c5f7e43dbaf5d009697915717d15b155 fails during testing on aarch64, allthough I don’t find a failing test in the log.


Bye

Stefan
M
M
Maxim Cournoyer wrote on 20 Sep 2020 20:12
(name . Stefan)(address . stefan-guix@vodafonemail.de)(address . 43534@debbugs.gnu.org)
87bli0e2qw.fsf@gmail.com
Hello Stefan!

Stefan <stefan-guix@vodafonemail.de> writes:

Toggle quote (4 lines)
> Hi!
>
> Building qemu-minimal-5.1.0 based on commit 4d82e8e5c5f7e43dbaf5d009697915717d15b155 fails during testing on aarch64, allthough I don’t find a failing test in the log.

Do you think this is a regression? I doubt so, because even before
pushing the update to 5.1.0, the test suite of qemu (not qemu-minimal)
was failing on master.

The error in the log you posted appear to be:

ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm: assertion failed: (resp_has_props(_resp))
ERROR arm-cpu-features - Bail out! ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm: assertion failed: (resp_has_props(_resp))

You could try commenting out the "check-qtest-aarch64-y +=
arm-cpu-features" line in tests/qtest/Makefile.include, as was done for
bios-tables-test in commit afeed71ae447ea18f7bc1b0b331e73e995308251.

Maxim
S
S
Stefan wrote on 20 Sep 2020 23:36
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 43534@debbugs.gnu.org)
56B769EB-B804-495F-9091-E4E95A5CBB68@vodafonemail.de
Hi Maxim!

Toggle quote (4 lines)
> Do you think this is a regression? I doubt so, because even before
> pushing the update to 5.1.0, the test suite of qemu (not qemu-minimal)
> was failing on master.

I don’t know. Until today I always got a substitute. There was at least a successful build in late August.


Toggle quote (5 lines)
> The error in the log you posted appear to be:
>
> ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm: assertion failed: (resp_has_props(_resp))
> ERROR arm-cpu-features - Bail out! ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm: assertion failed: (resp_has_props(_resp))

I only searched for “FAIL”. :-)

The same test was passing in August.


MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=arm-softmmu/qemu-system-arm QTEST_QEMU_IMG=qemu-img tests/qtest/arm-cpu-features -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="arm-cpu-features"
PASS 1 arm-cpu-features /arm/arm/query-cpu-model-expansion
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=aarch64-softmmu/qemu-system-aarch64 QTEST_QEMU_IMG=qemu-img tests/qtest/arm-cpu-features -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="arm-cpu-features"
PASS 1 arm-cpu-features /aarch64/arm/query-cpu-model-expansion
PASS 2 arm-cpu-features /aarch64/arm/kvm/query-cpu-model-expansion
PASS 3 arm-cpu-features /aarch64/arm/kvm/query-cpu-model-expansion/sve-off
PASS 4 arm-cpu-features /aarch64/arm/max/query-cpu-model-expansion/sve-max-vq-8
PASS 5 arm-cpu-features /aarch64/arm/max/query-cpu-model-expansion/sve-off


Bye

Stefan
M
M
Maxim Cournoyer wrote on 26 Sep 2020 08:05
(name . Stefan)(address . stefan-guix@vodafonemail.de)(address . 43534@debbugs.gnu.org)
87ft75t6nc.fsf@gmail.com
Hello,

Stefan <stefan-guix@vodafonemail.de> writes:

Toggle quote (21 lines)
> Hi Maxim!
>
>> Do you think this is a regression? I doubt so, because even before
>> pushing the update to 5.1.0, the test suite of qemu (not qemu-minimal)
>> was failing on master.
>
> I don’t know. Until today I always got a substitute. There was at least a successful build in late August.
>
> http://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+spec%3Aguix-master+qemu-minimal-5
>
>> The error in the log you posted appear to be:
>>
>> ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm: assertion failed: (resp_has_props(_resp))
>> ERROR arm-cpu-features - Bail out! ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm: assertion failed: (resp_has_props(_resp))
>
> I only searched for “FAIL”. :-)
>
> The same test was passing in August.
>
> http://ci.guix.gnu.org/log/391wav2qcfd1wkss2ssv408zhbgcxhq7-qemu-minimal-5.0.0

OK!

Toggle quote (16 lines)
> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
> QTEST_QEMU_BINARY=arm-softmmu/qemu-system-arm QTEST_QEMU_IMG=qemu-img
> tests/qtest/arm-cpu-features -m=quick -k --tap < /dev/null |
> ./scripts/tap-driver.pl --test-name="arm-cpu-features"
> PASS 1 arm-cpu-features /arm/arm/query-cpu-model-expansion
> …
> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
> QTEST_QEMU_BINARY=aarch64-softmmu/qemu-system-aarch64
> QTEST_QEMU_IMG=qemu-img tests/qtest/arm-cpu-features -m=quick -k --tap
> < /dev/null | ./scripts/tap-driver.pl --test-name="arm-cpu-features"
> PASS 1 arm-cpu-features /aarch64/arm/query-cpu-model-expansion
> PASS 2 arm-cpu-features /aarch64/arm/kvm/query-cpu-model-expansion
> PASS 3 arm-cpu-features /aarch64/arm/kvm/query-cpu-model-expansion/sve-off
> PASS 4 arm-cpu-features /aarch64/arm/max/query-cpu-model-expansion/sve-max-vq-8
> PASS 5 arm-cpu-features /aarch64/arm/max/query-cpu-model-expansion/sve-off

Is this reproducible on your side?

Here, I couldn't reproduce this exact failure, but I'm getting another
one trying to build with --linux-aarch64:

ERROR:tests/qtest/qos-test.c:186:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/virtio-net-device/virtio-net/virtio-net-tests/vhost-user/migrate/subprocess [22034]) failed unexpectedly
ERROR qos-test - Bail out! ERROR:tests/qtest/qos-test.c:186:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/virtio-net-device/virtio-net/virtio-net-tests/vhost-user/migrate/subprocess [22034]) failed unexpectedly
qemu: uncaught target signal 6 (Aborted) - core dumped
make: *** [/tmp/guix-build-qemu-minimal-5.1.0.drv-0/qemu-5.1.0/tests/Makefile.include:650: check-qtest-arm] Error

I'll go back to the previous version and see if I can reproduce or not.

Thanks,

Maxim
S
S
Stefan wrote on 26 Sep 2020 13:14
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 43534@debbugs.gnu.org)
0E0B468F-B491-42E9-885D-06C5B8CAE680@vodafonemail.de
Hi Maxim!

Toggle quote (2 lines)
> Is this reproducible on your side?

Retrying my previous command it found a substitute this time.

stefan@guix ~/development/guix$ sudo -E -b ./pre-inst-env guix system reconfigure -L /home/stefan/guix/ /home/stefan/guix-system.scm &> nohup.out && tail -f nohup.out
Passwort:
guix system: warning: cannot determine provenance for current system
substitute: Liste der Substitute von „https://ci.guix.gnu.org“ wird aktualisiert … 100.0%
The following derivations will be built:
/gnu/store/rgpllviv84cfhhd3bnnkmjpa1ndph8wh-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.drv
/gnu/store/12balhwfp2lzqyvy246j9zgh0zif16pn-grub-efi-2.04.drv

18,7 MB will be downloaded
qemu-minimal-5.1.0 17.8MiB 533KiB/s 00:34 [##################] 100.0%

building /gnu/store/12balhwfp2lzqyvy246j9zgh0zif16pn-grub-efi-2.04.drv…


Is it possible, that the test is somehow picky about the exact type of processor in use?


By the way, I also tried this command:

guix build --rounds=2 qemu-minimal --no-substitutes

But (probably because I got a substitute) it did not compile anything – I’m a bit clueless how to compile it again.


Bye

Stefan
M
M
Maxim Cournoyer wrote on 27 Sep 2020 02:31
(name . Stefan)(address . stefan-guix@vodafonemail.de)(address . 43534@debbugs.gnu.org)
873634t60n.fsf@gmail.com
Hi Stefan,

Stefan <stefan-guix@vodafonemail.de> writes:

Toggle quote (6 lines)
> Hi Maxim!
>
>> Is this reproducible on your side?
>
> Retrying my previous command it found a substitute this time.

[...]

Toggle quote (4 lines)
> By the way, I also tried this command:
>
> guix build --rounds=2 qemu-minimal --no-substitutes

Possibly, or it could be a non-deterministic failure.

Toggle quote (2 lines)
> But (probably because I got a substitute) it did not compile anything – I’m a bit clueless how to compile it again.

The following command:

Toggle snippet (3 lines)
$ guix build --no-grafts --check qemu-minimal

Should do it.

Thanks!

Maxim
S
S
Stefan wrote on 27 Sep 2020 14:38
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 43534@debbugs.gnu.org)
6D763A37-0BA7-48DF-81BB-5FEB35BBCAFB@vodafonemail.de
Hi Maxim!

Toggle quote (8 lines)
> The following command:
>
> --8<---------------cut here---------------start------------->8---
> $ guix build --no-grafts --check qemu-minimal
> --8<---------------cut here---------------end--------------->8---
>
> Should do it.

Yes, thanks, that was the trick.

It seems to be reproducible:

ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm: assertion failed: (resp_has_props(_resp))
ERROR arm-cpu-features - Bail out! ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm: assertion failed: (resp_has_props(_resp))


Bye

Stefan
M
M
Maxim Cournoyer wrote on 12 Oct 2020 06:28
control message for bug #43534
(address . control@debbugs.gnu.org)
87v9fgrrtt.fsf@gmail.com
merge 43534 43534
quit
M
M
Maxim Cournoyer wrote on 16 Mar 2021 02:36
Re: bug#43534: qemu-minimal-5.1.0 fails to build during test phase on aarch64
(name . Stefan)(address . stefan-guix@vodafonemail.de)(address . 43534@debbugs.gnu.org)
87zgz3ri9w.fsf@gmail.com
Hi Stefan,

Stefan <stefan-guix@vodafonemail.de> writes:

Toggle quote (20 lines)
> Hi Maxim!
>
>> The following command:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ guix build --no-grafts --check qemu-minimal
>> --8<---------------cut here---------------end--------------->8---
>>
>> Should do it.
>
> Yes, thanks, that was the trick.
>
> It seems to be reproducible:
>
> ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm:
> assertion failed: (resp_has_props(_resp))
> ERROR arm-cpu-features - Bail out!
> ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm:
> assertion failed: (resp_has_props(_resp))

Would you be able to try to reproduce, now that our QEMU package has
been bumped to version 5.2.0?

Thank you,

Maxim
M
M
Maxim Cournoyer wrote on 18 Mar 2021 03:16
control message for bug #43534
(address . control@debbugs.gnu.org)
874kh9qk9a.fsf@gmail.com
retitle 43534 qemu-minimal-5.2.0 fails to build during test phase on aarch64
quit
M
M
Maxim Cournoyer wrote on 18 Mar 2021 03:19
(address . control@debbugs.gnu.org)
8735wtqk37.fsf@gmail.com
merge 43534 43500
quit
M
M
Maxim Cournoyer wrote on 28 May 2021 20:08
Re: bug#43500: qemu-minimal test suite crashes on armhf-linux, aarch64-linux
(name . Stefan)(address . stefan-guix@vodafonemail.de)
878s3yivem.fsf_-_@gmail.com
Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (27 lines)
> Hi Stefan,
>
> Stefan <stefan-guix@vodafonemail.de> writes:
>
>> Hi Maxim!
>>
>>> The following command:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> $ guix build --no-grafts --check qemu-minimal
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> Should do it.
>>
>> Yes, thanks, that was the trick.
>>
>> It seems to be reproducible:
>>
>> ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm:
>> assertion failed: (resp_has_props(_resp))
>> ERROR arm-cpu-features - Bail out!
>> ERROR:tests/qtest/arm-cpu-features.c:508:test_query_cpu_model_expansion_kvm:
>> assertion failed: (resp_has_props(_resp))
>
> Would you be able to try to reproduce, now that our QEMU package has
> been bumped to version 5.2.0?

QEMU 5.2.0 minimal was built twice without any problem on real hardware (the Softiron
ARMv8 machines we have available -- overdrive1 and dover).

Closing. Feel free to reopen if you still encounter issues.

Thanks,

Maxim
Closed
?