numpy CPU dispatch probably prevents builds of python-numpy from being reproducible

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Maxime Devos
  • Ricardo Wurmus
Owner
unassigned
Submitted by
Ricardo Wurmus
Severity
normal
R
R
Ricardo Wurmus wrote on 20 Nov 2021 16:36
(address . bug-guix@gnu.org)
87y25ihm3e.fsf@elephly.net
Numpy accelerates operations with native code. Early in the build
it probes the CPU / compiler for supported instruction extensions
and then proceeds to build extensions for different features. I’m
guessing that this won’t work when building on older CPUs, so the
results of two builds on different CPUs may differ.

It would be helpful if someone could confirm this by building
numpy on an older CPU and then compare the build result with that
provided by ci.guix.gnu.org.

--
Ricardo
M
M
Maxime Devos wrote on 20 Nov 2021 17:26
ae41e8c06e2433cf8484bff5a5eb5b39d577e6a2.camel@telenet.be
Ricardo Wurmus schreef op za 20-11-2021 om 15:36 [+0000]:
Toggle quote (11 lines)
> Numpy accelerates operations with native code.  Early in the build
> it probes the CPU / compiler for supported instruction extensions
> and then proceeds to build extensions for different features.  I’m
> guessing that this won’t work when building on older CPUs, so the
> results of two builds on different CPUs may differ.
>
> It would be helpful if someone could confirm this by building
> numpy on an older CPU and then compare the build result with that
> provided by ci.guix.gnu.org.
>

I built it locally, and have the same result as ci.guix.gnu.org.
However, the bordeaux substitute is different:

$ LANGUAGE=en guix challenge python-numpy
/gnu/store/k311v0843hgq5frcbscylwimqssl1h54-python-numpy-1.17.3
contents differ:
local hash: 07var0wn8fywxchldz5pjqpnlavrlbc8s81aqwsqyi0i7qlh6ka7
: 07var0wn8fywxchldz5pjqpnlavrlbc8s81aqwsqyi0i7qlh6ka7
: 0cbl3q19bshb6ddz8xkcrjzkcmillsqii4z852ybzixyp7rg40qa
bordeaux.guix.gnu.org 3.9MiB 6.7MiB/s
00:01 [##################] 100.0%
1 store items were analyzed:
- 0 (0.0%) were identical
- 1 (100.0%) differed
- 0 (0.0%) were inconclusive

Diffoscope output:

bordeaux.guix.gnu.org 3.9MiB 4.2MiB/s 00:01 [##################] 100.0%
[...]
? ? ? ? ? ? ? ? --- /tmp/guix-directory.6ivbVt/lib/python3.8/site-packages/numpy/distutils/fcompiler/__pycache__/vast.cpython-38.pyc
? ? ? ? ? ? ? ??? +++ /gnu/store/k311v0843hgq5frcbscylwimqssl1h54-python-numpy-1.17.3/lib/python3.8/site-packages/numpy/distutils/fcompiler/__pycache__/vast.cpython-38.pyc
? ? ? ? ? ? ? ? @@ -106,15 +106,15 @@
? ? ? ? ? ? ? ? 00000690: 0104 0106 0104 0104 0106 0104 f906 0904 ................
? ? ? ? ? ? ? ? 000006a0: 0104 0208 0308 06da 085f 5f6d 6169 6e5f .........__main_
? ? ? ? ? ? ? ? 000006b0: 5f29 01da 036c 6f67 e902 0000 0029 01da _)...log.....)..
? ? ? ? ? ? ? ? 000006c0: 1463 7573 746f 6d69 7a65 645f 6663 6f6d .customized_fcom
? ? ? ? ? ? ? ? 000006d0: 7069 6c65 7272 0700 0000 2901 da08 636f pilerr....)...co
? ? ? ? ? ? ? ? 000006e0: 6d70 696c 6572 2911 da0a 5f5f 6675 7475 mpiler)...__futu
? ? ? ? ? ? ? ? 000006f0: 7265 5f5f 7202 0000 0072 0300 0000 7204 re__r....r....r.
? ? ? ? ? ? ? ? -00000700: 0000 0072 1900 0000 da1d 6e75 6d70 792e ...r......numpy.
? ? ? ? ? ? ? ? +00000700: 0000 0072 1900 0000 5a1d 6e75 6d70 792e ...r....Z.numpy.
? ? ? ? ? ? ? ? 00000710: 6469 7374 7574 696c 732e 6663 6f6d 7069 distutils.fcompi
? ? ? ? ? ? ? ? 00000720: 6c65 722e 676e 7572 0500 0000 da09 636f ler.gnur......co
? ? ? ? ? ? ? ? 00000730: 6d70 696c 6572 7372 0600 0000 7227 0000 mpilersr....r'..
? ? ? ? ? ? ? ? 00000740: 00da 0964 6973 7475 7469 6c73 7233 0000 ...distutilsr3..
? ? ? ? ? ? ? ? 00000750: 00da 0d73 6574 5f76 6572 626f 7369 7479 ...set_verbosity
? ? ? ? ? ? ? ? 00000760: da0f 6e75 6d70 792e 6469 7374 7574 696c ..numpy.distutil
? ? ? ? ? ? ? ? 00000770: 7372 3500 0000 da05 7072 696e 7472 2000 sr5.....printr .

1 bit is different.

Greetings,
Maxime.
L
L
Ludovic Courtès wrote on 10 Dec 2021 23:42
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87y24s5bfi.fsf@gnu.org
Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

Toggle quote (6 lines)
> Numpy accelerates operations with native code. Early in the build it
> probes the CPU / compiler for supported instruction extensions and
> then proceeds to build extensions for different features. I’m
> guessing that this won’t work when building on older CPUs, so the
> results of two builds on different CPUs may differ.

I don’t see anything in the build log that suggests it might be building
for a specific ISA extension (on x86_64).

On the contrary, it seems to do the right thing by dispatching to the
right optimized implementation at run time, in numpy/core/src/umath.
Apparently this directory contains templates for home-made function
multi-versioning, with variants and dispatch code generated by
numpy/core/code_generators/generate_umath.py.

Besides, I see the same .pyc difference as Maxime, but I don’t think it
relates to ISA extensions.

Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 52001@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 52001
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch