python-minimal tests hang

  • Done
  • quality assurance status badge
Details
4 participants
  • Danny Milosavljevic
  • Leo Famulari
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Danny Milosavljevic
Severity
normal
D
D
Danny Milosavljevic wrote on 13 Apr 2021 22:38
(address . bug-guix@gnu.org)
20210413223817.13450547@scratchpost.org
On x86_64, python-minimal build hangs when running the tests:

$ guix build /gnu/store/gifx79qc77zk88z6gnabj81iksp1xaj9-python-minimal-3.8.2.drv
[...]
1:06:10 load avg: 1.31 running: test_multiprocessing_forkserver (14 min 42 sec)

It's reproducible every time.

$ guix describe
Generation 230 Apr 13 2021 12:15:27 (current)
guix 822eacc
branch: master
commit: 822eacc6bb0878323e6687d4460a7c53066545e1
$ uname -a
Linux dayas 5.11.4-gnu #1 SMP 1 x86_64 GNU/Linux
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAmB2ATkACgkQ5xo1VCww
uqV6Bgf+IkLYk8ot2A0DvQGRcTY0ie7CAxT1OLMsK20KdTcqH8I4nehaDMclWpj2
Q7lp4PDLqZRrve5UZBxEy/jAYuTi/tI9AUCxqFFmPFRYxZRMR5jkELNuLXbvcw6k
iiIcOoHDXelqm9CgUQgKOOJYbsl9wGtMm+iigpfKd75ZPaS4lM3aSsqy9z/4zQKT
LP3B4i9/qnl5t/RZDSIOLsp/ZSKgYHQwKTcp6Pb7SlhbSw1RArKSS1d6rmrzYDq9
+cdt/JbWehownqEs2CZB+Cv4YfK1LY9vhshxNP+77+dGlfNR9Qk8+ZcaaDcqEfOS
WndkdwwXGLbEh1BMAKzsivvdIuhfig==
=uts/
-----END PGP SIGNATURE-----


L
L
Leo Famulari wrote on 13 Apr 2021 23:30
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 47759@debbugs.gnu.org)
YHYNevMS+41BqxYF@jasmine.lan
On Tue, Apr 13, 2021 at 10:38:17PM +0200, Danny Milosavljevic wrote:
Toggle quote (6 lines)
> On x86_64, python-minimal build hangs when running the tests:
>
> $ guix build /gnu/store/gifx79qc77zk88z6gnabj81iksp1xaj9-python-minimal-3.8.2.drv
> [...]
> 1:06:10 load avg: 1.31 running: test_multiprocessing_forkserver (14 min 42 sec)

Where does this derivation come from? I can't seem to find it on the
machines I have access to.

Toggle quote (9 lines)
> $ guix describe
> Generation 230 Apr 13 2021 12:15:27 (current)
> guix 822eacc
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: 822eacc6bb0878323e6687d4460a7c53066545e1
> $ uname -a
> Linux dayas 5.11.4-gnu #1 SMP 1 x86_64 GNU/Linux

------
$ guix describe
Generation 15 Apr 13 2021 23:26:10 (current)
guix 822eacc
commit: 822eacc6bb0878323e6687d4460a7c53066545e1
$ guix build --derivations --no-grafts python-minimal
/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
$ guix build --derivations --no-grafts python-minimal-wrapper
/gnu/store/qvib4wzz542czxfsl7dw4bnlz0kdrm82-python-minimal-wrapper-3.8.2.drv
------
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAmB2DXcACgkQJkb6MLrK
fwhrkg//RwJuQhLP8zV3VgTy+TB4qTN0odw+U95u20Hp4CkKH8K4h+/cVE1feeko
mbLWtlkvoTOcKcEzL2PautnRmibLGa9E6X9KclzgkR2JKGwTd/63hfkVdJ4T9lDq
+Yxq4LLlaSW9o5XEnuR6DR6X/Zrvu1GfSMpdhLvKXDMYklPT5q0NudXDuv+T9vL5
lkTjur6ngje2zlmXfKwoDW1Ygu2aPyTgqKkNFtrtv0q4KvT3uHQVVo4eZCS69hC1
Vd1f5hk/9RXBIZQ6P/mB1mExaaSbRq5NSfLnoO5K7t0r0xzSrA0E+r++FycWoPU4
U+W6JHrz5esSWzMYq9qicjLiFWNPDZfJiD64O7z0dYDML6l5GjXZIJIGsDXbLPdV
Y8JPoonm1HcUGQG/vNe29zuh8gZ7TAgECzPb87okWZX3bDXDA/AEuB3fZZl//B+R
lS/8ftaS2Ir4YDpy2AjatGqZbRZfncCgJ7lHOGvwoj3AmRQxxUfxpO/GeW6yHhta
83xrJtS+Gbwt0EwfWdL9OhpMcaCyzKZQUztiNs8jCzoTyfD0D+tGJUlef9SCYu4R
jg+EWDfGoTZK+SwQEnVG+MyYaucZvbvhkxpAV1gpMQHXtboAoo2vg53BvY59gRmR
KAB9HzTjYmGEOLwAQIZwxV8sIu+OsvsBwhlDNcxFoh6Wm2acvQw=
=SGDw
-----END PGP SIGNATURE-----


M
M
Maxim Cournoyer wrote on 20 Apr 2021 14:05
(name . Leo Famulari)(address . leo@famulari.name)
874kg1gnz2.fsf@gmail.com
block 47297 by 47759
thanks

Hi!

Leo Famulari <leo@famulari.name> writes:

Toggle quote (31 lines)
> On Tue, Apr 13, 2021 at 10:38:17PM +0200, Danny Milosavljevic wrote:
>> On x86_64, python-minimal build hangs when running the tests:
>>
>> $ guix build /gnu/store/gifx79qc77zk88z6gnabj81iksp1xaj9-python-minimal-3.8.2.drv
>> [...]
>> 1:06:10 load avg: 1.31 running: test_multiprocessing_forkserver (14 min 42 sec)
>
> Where does this derivation come from? I can't seem to find it on the
> machines I have access to.
>
>> $ guix describe
>> Generation 230 Apr 13 2021 12:15:27 (current)
>> guix 822eacc
>> repository URL: https://git.savannah.gnu.org/git/guix.git
>> branch: master
>> commit: 822eacc6bb0878323e6687d4460a7c53066545e1
>> $ uname -a
>> Linux dayas 5.11.4-gnu #1 SMP 1 x86_64 GNU/Linux
>
> ------
> $ guix describe
> Generation 15 Apr 13 2021 23:26:10 (current)
> guix 822eacc
> repository URL: https://git.savannah.gnu.org/git/guix.git
> commit: 822eacc6bb0878323e6687d4460a7c53066545e1
> $ guix build --derivations --no-grafts python-minimal
> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
> $ guix build --derivations --no-grafts python-minimal-wrapper
> /gnu/store/qvib4wzz542czxfsl7dw4bnlz0kdrm82-python-minimal-wrapper-3.8.2.drv
> ------

I can also reproduce this on the version-1.3.0, building for
armhf-linux. Another release blocker :-/.

Thanks,

Maxim
L
L
Ludovic Courtès wrote on 22 Apr 2021 22:29
(name . Leo Famulari)(address . leo@famulari.name)
87mttqysff.fsf@gnu.org
Hi!

Leo Famulari <leo@famulari.name> skribis:

Toggle quote (10 lines)
> $ guix describe
> Generation 15 Apr 13 2021 23:26:10 (current)
> guix 822eacc
> repository URL: https://git.savannah.gnu.org/git/guix.git
> commit: 822eacc6bb0878323e6687d4460a7c53066545e1
> $ guix build --derivations --no-grafts python-minimal
> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
> $ guix build --derivations --no-grafts python-minimal-wrapper
> /gnu/store/qvib4wzz542czxfsl7dw4bnlz0kdrm82-python-minimal-wrapper-3.8.2.drv

FWIW, berlin has binaries for x86_64 and armhf (the two you mentioned):

Toggle snippet (25 lines)
$ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --no-grafts -s x86_64-linux -s armhf-linux -d
/gnu/store/y9g688q1ffk6gbyiwda3pw5wb29q0a2x-python-minimal-3.8.2.drv
/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
$ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --no-grafts -s x86_64-linux -s armhf-linux
/gnu/store/c7s4wslhi6idzprnnigabw8bz4lp7gn8-python-minimal-3.8.2
/gnu/store/z3zxs5pynvnq7mcpsr1nkh5hbb9hicf5-python-minimal-3.8.2
$ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- weather python-minimal -s x86_64-linux -s armhf-linux
computing 1 package derivations for armhf-linux...
computing 1 package derivations for x86_64-linux...
looking for 2 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
100.0% substitutes available (2 out of 2)
at least 67.2 MiB of nars (compressed)
153.0 MiB on disk (uncompressed)
0.085 seconds per request (0.2 seconds in total)
11.8 requests per second

at least 1,000 queued builds
aarch64-linux: 1000 (100.0%)
build rate: 36.16 builds per hour
x86_64-linux: 8.38 builds per hour
i686-linux: 20.25 builds per hour
aarch64-linux: 8.93 builds per hour

I also rebuilt it locally on my x86_64 laptop and it ran to completion
(although there’s a bitwise reproducibility issue):

Toggle snippet (12 lines)
$ time guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --check --no-grafts -v1
La jena derivo estos konstruata:
/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv

building /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv...
- 'compress-documentation' phaseguix build: error: derivation `/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv' may not be deterministic: output `/gnu/store/z3zxs5pynvnq7mcpsr1nkh5hbb9hicf5-python-minimal-3.8.2' differs

real 9m45.336s
user 0m3.490s
sys 0m0.289s

So… no problem? Or perhaps it’s a function of the number of cores or
something? (Here that’s with 2 cores/4 threads.)

Ludo’.
M
M
Maxim Cournoyer wrote on 22 Apr 2021 23:01
(name . Ludovic Courtès)(address . ludo@gnu.org)
87a6pq2fvz.fsf@gmail.com
Hi,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (58 lines)
> Hi!
>
> Leo Famulari <leo@famulari.name> skribis:
>
>> $ guix describe
>> Generation 15 Apr 13 2021 23:26:10 (current)
>> guix 822eacc
>> repository URL: https://git.savannah.gnu.org/git/guix.git
>> commit: 822eacc6bb0878323e6687d4460a7c53066545e1
>> $ guix build --derivations --no-grafts python-minimal
>> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
>> $ guix build --derivations --no-grafts python-minimal-wrapper
>> /gnu/store/qvib4wzz542czxfsl7dw4bnlz0kdrm82-python-minimal-wrapper-3.8.2.drv
>
> FWIW, berlin has binaries for x86_64 and armhf (the two you mentioned):
>
> $ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --no-grafts -s x86_64-linux -s armhf-linux -d
> /gnu/store/y9g688q1ffk6gbyiwda3pw5wb29q0a2x-python-minimal-3.8.2.drv
> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
> $ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --no-grafts -s x86_64-linux -s armhf-linux
> /gnu/store/c7s4wslhi6idzprnnigabw8bz4lp7gn8-python-minimal-3.8.2
> /gnu/store/z3zxs5pynvnq7mcpsr1nkh5hbb9hicf5-python-minimal-3.8.2
> $ guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- weather python-minimal -s x86_64-linux -s armhf-linux
> computing 1 package derivations for armhf-linux...
> computing 1 package derivations for x86_64-linux...
> looking for 2 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
> 100.0% substitutes available (2 out of 2)
> at least 67.2 MiB of nars (compressed)
> 153.0 MiB on disk (uncompressed)
> 0.085 seconds per request (0.2 seconds in total)
> 11.8 requests per second
>
> at least 1,000 queued builds
> aarch64-linux: 1000 (100.0%)
> build rate: 36.16 builds per hour
> x86_64-linux: 8.38 builds per hour
> i686-linux: 20.25 builds per hour
> aarch64-linux: 8.93 builds per hour
>
>
> I also rebuilt it locally on my x86_64 laptop and it ran to completion
> (although there’s a bitwise reproducibility issue):
>
> $ time guix time-machine --commit=822eacc6bb0878323e6687d4460a7c53066545e1 -- build python-minimal --check --no-grafts -v1
> La jena derivo estos konstruata:
> /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv
>
> building /gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv...
> - 'compress-documentation' phaseguix build: error: derivation `/gnu/store/qkggqs5pxr9fmczc6gn5rs3d51ykhh36-python-minimal-3.8.2.drv' may not be deterministic: output `/gnu/store/z3zxs5pynvnq7mcpsr1nkh5hbb9hicf5-python-minimal-3.8.2' differs
>
> real 9m45.336s
> user 0m3.490s
> sys 0m0.289s
>
> So… no problem? Or perhaps it’s a function of the number of cores or
> something? (Here that’s with 2 cores/4 threads.)

For me it seems to happen only when using QEMU transparent emulation,
e.g. to build it for armhf-linux on an amd64 machine.

Danny, are you still able to reproduce this problem from the master
branch, without using QEMU emulation?

Thank you,

Maxim
M
M
Maxim Cournoyer wrote on 1 May 2021 05:45
(name . Ludovic Courtès)(address . ludo@gnu.org)
87a6pfp15x.fsf@gmail.com
unblock 47297 by 47759
thanks

A first RC1 release candidate with the 5 supported architectures could
be produced without this bug being a blocker. The key was in using real
hardware rather than QEMU emulation.

Unblocking.

Maxim
M
M
Maxim Cournoyer wrote on 14 Jul 2022 06:08
(name . Ludovic Courtès)(address . ludo@gnu.org)
877d4gxroe.fsf@gmail.com
Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (9 lines)
> unblock 47297 by 47759
> thanks
>
> A first RC1 release candidate with the 5 supported architectures could
> be produced without this bug being a blocker. The key was in using real
> hardware rather than QEMU emulation.
>
> Unblocking.

Closing, as I don't think tracking QEMU-induced build problems is in
scope for our bug tracker.

Thanks,

Maxim
Closed
?