openblas builds not reproducible on different x86_64 machines

  • Open
  • quality assurance status badge
Details
4 participants
  • Efraim Flashner
  • Ludovic Courtès
  • Maxim Cournoyer
  • zimoun
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
important
M
M
Maxim Cournoyer wrote on 1 Nov 2021 04:07
(name . bug-guix)(address . bug-guix@gnu.org)
87h7cw7ewb.fsf@gmail.com
Hello Guix,

Our OpenBLAS package uses DYNAMIC_ARCH=1 to provide optimizations for
all supported targets, at least of x86 and x86_64. In theory that seems
OK, but in practice the builds differ depending on the host CPU.

I've made a build on an old Core2 CPU (Q6700), and another one on
Berlin. I've run diffoscope on the result, and got tons of differences;
here's the tail of the diffoscope output:

Toggle snippet (270 lines)
? ? Hex dump of section '.data':
? ? ? - 0x01d8b700 00b7d801 00000000 00000010 00000000 ................
? ? ? - 0x01d8b710 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b720 00000000 01000000 01000080 56555555 ............VUUU
? ? ? - 0x01d8b730 01000040 34333333 abaaaa2a 25499224 ...@4333...*%I.$
? ? ? - 0x01d8b740 01000020 1dc7711c 9a999919 75d14517 ... ..q.....u.E.
? ? ? - 0x01d8b750 56555515 143bb113 93244912 12111111 VUU..;...$I.....
? ? ? - 0x01d8b760 01000010 100f0f0f 8fe3380e 5f43790d ..........8._Cy.
? ? ? - 0x01d8b770 cdcccc0c 0dc3300c bbe8a20b 2d64210b ......0.....-d!.
? ? ? - 0x01d8b780 abaaaa0a a4703d0a 8a9dd809 5f427b09 .....p=....._B{.
? ? ? - 0x01d8b790 4a922409 b1dcd308 89888808 85104208 J.$...........B.
? ? ? - 0x01d8b7a0 01000008 7df0c107 88878707 08755007 ....}........uP.
? ? ? - 0x01d8b7b0 c8711c07 463eeb06 b0a1bc06 07699006 .q..F>.......i..
? ? ? - 0x01d8b7c0 67666606 64703e06 87611806 d117f405 gff.dp>..a......
? ? ? - 0x01d8b7d0 5e74d105 065bb005 17b29005 0b627205 ^t...[.......br.
? ? ? - 0x01d8b7e0 56555505 2a783905 52b81e05 06050505 VUU.*x9.R.......
? ? ? - 0x01d8b7f0 c54eec04 3f87d404 30a1bd04 4b90a704 .N..?...0...K...
? ? ? - 0x01d8b800 25499204 20c17d04 59ee6904 98c75604 %I.. .}.Y.i...V.
? ? ? - 0x01d8b810 45444404 545c3204 43082104 05411004 EDD.T\2.C.!..A..
? ? ? - 0x01d8b820 01000004 00000000 00000000 00000000 ................
? ? ? - 0x01d8b830 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b840 40000000 00000000 00010000 ffff0000 @...............
? ? ? - 0x01d8b850 00000000 00000000 00000000 08000000 ................
? ? ? - 0x01d8b860 04000000 08000000 00000000 00000000 ................
? ? ? - 0x01d8b870 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b880 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b890 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b8a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b8b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b8c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b8d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b8e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b8f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b900 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b910 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b920 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b930 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b940 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b950 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b960 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b970 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b980 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b990 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b9a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b9b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b9c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b9d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b9e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8b9f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba00 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba10 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba20 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba30 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba40 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba50 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba60 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba70 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba80 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8ba90 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8baa0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bab0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bac0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bad0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bae0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8baf0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb00 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb10 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb20 00000000 04000000 04000000 04000000 ................
? ? ? - 0x01d8bb30 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb40 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb50 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb60 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb70 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb80 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bb90 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bba0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bbb0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bbc0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bbd0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bbe0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bbf0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc00 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc10 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc20 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc30 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc40 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc50 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc60 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc70 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc80 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bc90 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bca0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bcb0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bcc0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bcd0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bce0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bcf0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd00 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd10 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd20 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd30 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd40 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd50 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd60 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd70 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd80 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bd90 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bda0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bdb0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bdc0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bdd0 00000000 00000000 00000000 04000000 ................
? ? ? - 0x01d8bde0 02000000 04000000 00000000 00000000 ................
? ? ? - 0x01d8bdf0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be00 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be10 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be20 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be30 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be40 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be50 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be60 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be70 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be80 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8be90 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bea0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8beb0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bec0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bed0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bee0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bef0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf00 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf10 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf20 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf30 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf40 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf50 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf60 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf70 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf80 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bf90 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bfa0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bfb0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bfc0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bfd0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bfe0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8bff0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c000 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c010 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c020 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c030 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c040 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c050 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c060 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c070 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c080 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c090 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c0a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c0b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c0c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c0d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c0e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c0f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c100 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c110 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c120 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c130 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c140 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c150 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c160 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c170 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c180 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c190 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c1a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c1b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c1c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c1d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c1e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c1f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c200 00000000 00000000 00000000 08000000 ................
? ? ? - 0x01d8c210 04000000 08000000 00000000 00000000 ................
? ? ? - 0x01d8c220 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c230 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c240 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c250 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c260 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c270 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c280 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c290 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c2a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c2b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c2c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c2d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c2e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c2f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c300 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c310 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c320 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c330 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c340 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c350 00000000 00000000 00000000 02000000 ................
? ? ? - 0x01d8c360 02000000 02000000 00000000 00000000 ................
? ? ? - 0x01d8c370 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c380 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c390 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c3a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c3b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c3c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c3d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c3e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c3f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c400 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c410 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c420 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c430 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c440 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c450 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c460 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c470 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c480 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c490 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c4a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c4b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c4c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c4d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c4e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c4f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c500 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c510 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c520 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c530 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c540 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c550 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c560 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c570 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c580 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c590 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c5a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c5b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c5c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c5d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c5e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c5f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c600 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c610 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c620 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c630 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c640 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c650 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c660 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c670 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c680 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c690 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c6a0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c6b0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c6c0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c6d0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c6e0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c6f0 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c700 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c710 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c720 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c730 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c740 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c750 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c760 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c770 00000000 00000000 00000000 00000000 ................
? ? ? - 0x01d8c780 00000000 00000000 00000000 04000000 ................
? ? ? - 0x01d8c790 04000000 04000000 00000000 00000000 ................
? ? ? - 0x01d8c7a0 00000000 00000000 00000000 00000000 ................
? ?
This message was truncated. Download the full message here.
E
E
Efraim Flashner wrote on 1 Nov 2021 09:54
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 51536@debbugs.gnu.org)
YX+rP5R0uFQPj5ym@3900XT
Attachment: file
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmF/qz0ACgkQQarn3Mo9
g1Em+A/+PsbjPnIqtDQcXpUz86fqqlhXpb3hFZpKKTBbshnUQO6M0kHJTlD4ucRS
8Ic91+hGNgtqgRvRji0WIK6QK/IkOMfiA+I2+U1CdF5oh1qx1Bmb/jYb4KU/Lhfi
ZJK9vFbfVwE+W2XjwOowb3uyX+55RnG8A2nJt/NHY6W2i2CKE9dLsNzAkrAcLV3T
Q9FVUphLcumh1hpDiJeTkxQnZ8otK4zdja86aF+9oDNikLhIysmh+qyWyjo+hJib
dhnOLzvi7IDYgNAYHm2ME656g62tkoYMDMUp5ZaIikXB3NHV82Mcm54+nZjnIDRP
OH9wgocAVB5ttBVa0W7nf1Xu0U/+bg+za1fD6qJGFLEgjEUlH2BtnAzNOGYzOkeD
1HQqoajn0wy7WVUsmfsj+CMrE2hBHLNn7ws+ODTDds4lUiPIf00bbpiSyX03Ll02
0lSKd++buQQUhSV1FER9Iofg3OpkaU/ty5tAcPXt3aO8o1IyuE/wnsFGXGDMvxvE
Xnprd2rvW524oPjpcrDjny+HIAoBLcIcrRMd7XDSQJybJE2bns/1XdR4jeIVyOPm
tVN9BYE/9Tlv4iLjIJgncDcRUDPelzq6dpVlD8jqOMnrQhjmFdgrfmIdo0b9N6Dg
SFve+HXMAisdfFl+E1d+yJ588sBbKo9WFAK1BClr8P9Nb953ZS4=
=fkWp
-----END PGP SIGNATURE-----


Z
Z
zimoun wrote on 3 Nov 2021 16:03
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 51536@debbugs.gnu.org)
CAJ3okZ2H=_MuXqU4_NjqC0LSZqi9pLJBDr+xN6pZ25zWnpzvFQ@mail.gmail.com
Hi Maxim,

On Mon, 1 Nov 2021 at 04:08, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

Toggle quote (8 lines)
> Our OpenBLAS package uses DYNAMIC_ARCH=1 to provide optimizations for
> all supported targets, at least of x86 and x86_64. In theory that seems
> OK, but in practice the builds differ depending on the host CPU.
>
> I've made a build on an old Core2 CPU (Q6700), and another one on
> Berlin. I've run diffoscope on the result, and got tons of differences;
> here's the tail of the diffoscope output:

Well, it rings this bell [1] and this one [2] too. ;-)

Maybe I am wrong, but speaking about HPC, it seems expected that the
builds differ depending on the host CPU and I am not convinced we can
do better than DYNAMIC_ARCH=1 for performances which somehow sacrifice
reproducibility, IIUC.



Cheers,
simon
M
M
Maxim Cournoyer wrote on 5 Nov 2021 17:38
(name . Efraim Flashner)(address . efraim@flashner.co.il)(address . 51536@debbugs.gnu.org)
87v91636dz.fsf@gmail.com
Hi Efraim!

Efraim Flashner <efraim@flashner.co.il> writes:

[...]

Toggle quote (6 lines)
> I compared our build flags with Debian's¹ (which is also what I often do
> when working on other architectures) and it looks like they have
> DYNAMIC_OLDER=1 and TARGET=GENERIC. From looking through Makefile.system
> it looks like TARGET=GENERIC makes it more portable, and DYNAMIC_OLDER=1
> adds support for some of the older x86_64 sub-architectures.

DYNAMIC_OLDER is probably a good idea (it'll inflate the binary size a
bit), but I doubt it's the culprit here, as my problematic CPU is a
Q6700 from the CORE2 family, already supposed to be covered by
DYNAMIC_ARCH=1 on x86 (c.f. Makefile.system):

Toggle snippet (7 lines)
ifeq ($(DYNAMIC_ARCH), 1)
ifeq ($(ARCH), x86)
DYNAMIC_CORE = KATMAI COPPERMINE NORTHWOOD PRESCOTT BANIAS \
CORE2 PENRYN DUNNINGTON NEHALEM ATHLON OPTERON OPTERON_SSE3 BARCELONA BOBCAT ATOM NANO
endif

From that same Makefile, TARGET=GENERIC seems to only do:

Toggle snippet (12 lines)
ifdef TARGET
GETARCH_FLAGS := -DFORCE_$(TARGET)
GETARCH_FLAGS += -DUSER_TARGET
ifeq ($(TARGET), GENERIC)
ifeq ($(DYNAMIC_ARCH), 1)
override NO_EXPRECISION=1
export NO_EXPRECiSION
endif
endif
endif

I.e. it sets NO_EXPRECISION to 1 and exports it (with a typo!). Perhaps
good to fix that typo and try it!

Toggle quote (6 lines)
> If it weren't for the 2k+ packages which depend on openblas I think it'd
> be obvious to add the extra flags now. I wonder if we could
> unconditionally set DYNAMIC_ARCH=1 TARGET=GENERIC and then override it
> on some architectures, rather than now where each architecture needs its
> own flags.

I've rebuilt core-updates-frozen-batched-changes so many times now I'm
not too afraid of it anymore ;-).

Toggle quote (4 lines)
> I also wonder if it's worth it to add a minimal openblas-native for an
> easy transform option of --with-graft=openblas=openblas-native, where
> openblas-native just adds -march=native and turns off substitutability.

Seems this could be done easily by someone already rewriting their Guix
packages graph.

To be continued...

Maxim
M
M
Maxim Cournoyer wrote on 7 Nov 2021 03:33
(name . Efraim Flashner)(address . efraim@flashner.co.il)(address . 51536@debbugs.gnu.org)
87fss83dav.fsf@gmail.com
Hi again,

Efraim Flashner <efraim@flashner.co.il> writes:

Toggle quote (3 lines)
> On Sun, Oct 31, 2021 at 11:07:00PM -0400, Maxim Cournoyer wrote:
>> Hello Guix,

[...]

Toggle quote (19 lines)
> I compared our build flags with Debian's¹ (which is also what I often do
> when working on other architectures) and it looks like they have
> DYNAMIC_OLDER=1 and TARGET=GENERIC. From looking through Makefile.system
> it looks like TARGET=GENERIC makes it more portable, and DYNAMIC_OLDER=1
> adds support for some of the older x86_64 sub-architectures.


> If it weren't for the 2k+ packages which depend on openblas I think it'd
> be obvious to add the extra flags now. I wonder if we could
> unconditionally set DYNAMIC_ARCH=1 TARGET=GENERIC and then override it
> on some architectures, rather than now where each architecture needs its
> own flags.
>
> I also wonder if it's worth it to add a minimal openblas-native for an
> easy transform option of --with-graft=openblas=openblas-native, where
> openblas-native just adds -march=native and turns off substitutability.
>
> ¹ https://sources.debian.org/src/openblas/0.3.18+ds-2/debian/rules/#L19

I've tried this:

Toggle snippet (24 lines)
modified gnu/packages/maths.scm
@@ -4444,13 +4444,13 @@ (define-public openblas
,@(let ((system (or (%current-target-system) (%current-system))))
(cond
((or (string-prefix? "x86_64" system)
- (string-prefix? "i686" system))
- '("DYNAMIC_ARCH=1"))
- ;; On some of these architectures the CPU can't be detected.
- ((or (string-prefix? "powerpc64le" system)
+ (string-prefix? "i686" system)
+ (string-prefix? "powerpc64le" system)
(string-prefix? "aarch64" system))
- '("DYNAMIC_ARCH=1"
- "TARGET=GENERIC"))
+ ;; Dynamic older enables a few extra CPU architectures that
+ ;; were released before 2010.
+ '("DYNAMIC_ARCH=1" "DYNAMIC_OLDER=1" "TARGET=GENERIC"))
+ ;; On some of these architectures the CPU can't be detected.
;; On MIPS we force the "SICORTEX" TARGET, as for the other
;; two available MIPS targets special extended instructions
;; for Loongson cores are used.

And it fixed the illegal instruction problem on my Core 2 Duo CPU
(tested by using an offloaded openblas build and building numpy locally
with it).

But it didn't resolve the reproducibily issue. I'll create a ticket
upstream asking for guidance.

Thank you,

Maxim
L
L
Ludovic Courtès wrote on 3 Feb 2022 00:13
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87czk4rheq.fsf@gnu.org
Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (4 lines)
> Our OpenBLAS package uses DYNAMIC_ARCH=1 to provide optimizations for
> all supported targets, at least of x86 and x86_64. In theory that seems
> OK, but in practice the builds differ depending on the host CPU.

What follows is the log of an investigation that didn’t find the root
cause, but perhaps it’ll give us ideas…

Right now the build results of ci.guix and bordeaux.guix differ:

Toggle snippet (20 lines)
$ guix describe
Generacio 202 Jan 30 2022 23:57:03 (nuna)
guix 43dd34c
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 43dd34c7777a212c99a97da7a2c237158faa9a1b
ludo@ribbon ~/src/guix$ guix challenge openblas
/gnu/store/ras6dprsw3wm3swk23jjp8ww5dwxj333-openblas-0.3.18 contents differ:
no local build for '/gnu/store/ras6dprsw3wm3swk23jjp8ww5dwxj333-openblas-0.3.18'
https://ci.guix.gnu.org/nar/lzip/ras6dprsw3wm3swk23jjp8ww5dwxj333-openblas-0.3.18: 0m1jlc26yrwxn8gxwpj8452kw4g84ywclh0hnab93873ifz87s5c
https://bordeaux.guix.gnu.org/nar/lzip/ras6dprsw3wm3swk23jjp8ww5dwxj333-openblas-0.3.18: 1d0m9v3kpsqzplpl1law2lfhm6rrbhkkqsvh19dlg9wx45vbbvjb
differing file:
/lib/libopenblasp-r0.3.18.so

1 store items were analyzed:
- 0 (0.0%) were identical
- 1 (100.0%) differed
- 0 (0.0%) were inconclusive

To get an idea, I thought we could compare the two build logs:


(Protip: I found the second one via

The “ar -ru ../libopenblasp-r0.3.18.a …” are apparently the same in
both cases, which rules out the simple case of unsorted .o files.

The .so on ci.guix is slightly bigger:

Toggle snippet (7 lines)
$ wget -qO - https://ci.guix.gnu.org/nar/lzip/ras6dprsw3wm3swk23jjp8ww5dwxj333-openblas-0.3.18| lzip -d | guix archive -x /tmp/o1
$ wget -qO - https://bordeaux.guix.gnu.org/nar/lzip/ras6dprsw3wm3swk23jjp8ww5dwxj333-openblas-0.3.18| lzip -d | guix archive -x /tmp/o2
$ ls -l /tmp/{o1,o2}/lib/libopenblasp-r0.3.18.so
-r-xr-xr-x 1 ludo users 40538768 Jan 1 1970 /tmp/o1/lib/libopenblasp-r0.3.18.so
-r-xr-xr-x 1 ludo users 40436368 Jan 1 1970 /tmp/o2/lib/libopenblasp-r0.3.18.so

Both have the same symbols though, and in the same order:

Toggle snippet (5 lines)
$ diff -u <(objdump -T /tmp/o1/lib/libopenblasp-r0.3.18.so |cut -c 60- ) <(objdump -T /tmp/o2/lib/libopenblasp-r0.3.18.so |cut -c60- )
$ echo $?
0

… which suggests they include code optimized for the same
micro-architectures because symbols include the name of the
micro-architecture:

Toggle snippet (13 lines)
$ objdump -T /tmp/o1/lib/libopenblasp-r0.3.18.so |cut -c 60-|tail -10
csymm3m_RU
cgemv_c_BARCELONA
csymv_U_HASWELL
dtrmm_iltncopy_CORE2
LAPACKE_dsytrs2
openblas_num_threads_env
csycon_rook_
csytri_rook_



Some of the offsets differ though:
$ diff -u <(objdump -T /tmp/o1/lib/libopenblasp-r0.3.18.so ) <(objdump -T /tmp/o2/lib/libopenblasp-r0.3.18.so )
--- /dev/fd/63 2022-02-03 00:10:17.308357982 +0100
+++ /dev/fd/62 2022-02-03 00:10:17.276357923 +0100
@@ -1,5 +1,5 @@
-/tmp/o1/lib/libopenblasp-r0.3.18.so: format de fixer elf64-x86-64
+/tmp/o2/lib/libopenblasp-r0.3.18.so: format de fixer elf64-x86-64
DYNAMIC SYMBOL TABLE:
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.3.2 pthread_cond_signal
@@ -91,57 +91,57 @@
00000000013edb70 g DF .text 00000000000001be Base zgemm3m_incopyb_BULLDOZER
0000000000e6d200 g DF .text 0000000000002b06 Base strsm_kernel_RT_BOBCAT
0000000000512c00 g DF .text 0000000000000a0a Base zsymv_U_PRESCOTT
-00000000023c7530 g DF .text 0000000000000201 Base LAPACKE_dpttrs_work
+00000000023ae930 g DF .text 0000000000000201 Base LAPACKE_dpttrs_work
0000000000692000 g DF .text 0000000000000b89 Base srot_k_PENRYN
000000000179caa0 g DF .text 0000000000000200 Base dgemm_beta_HASWELL
0000000000a44690 g DF .text 00000000000004b4 Base dtrsm_iutucopy_OPTERON
-000000000231cfc0 g DF .text 000000000000021d Base LAPACKE_sstein_work
-0000000002327800 g DF .text 000000000000014b Base LAPACKE_ssytrd
-0000000001ad9100 g DF .text 00000000000002aa Base chemm_outcopy_SKYLAKEX
+00000000023043c0 g DF .text 000000000000021d Base LAPACKE_sstein_work
+000000000230ec00 g DF .text 000000000000014b Base LAPACKE_ssytrd
+0000000001acc900 g DF .text 00000000000002aa Base chemm_outcopy_SKYLAKEX
00000000017d6c10 g DF .text 0000000000000c38 Base cgemv_n_HASWELL
-0000000002327b70 g DF .text 0000000000000143 Base LAPACKE_ssytrf
+000000000230ef70 g DF .text 0000000000000143 Base LAPACKE_ssytrf
000000000018f010 g DF .text 000000000000025c Base cblas_stbmv
0000000000195a20 g DF .text 000000000000003b Base cblas_idamin
-0000000002328d40 g DF .text 0000000000000101 Base LAPACKE_ssytri
+0000000002310140 g DF .text 0000000000000101 Base LAPACKE_ssytri
000000000077be00 g DF .text 0000000000000e65 Base ztrsm_kernel_RN_PENRYN
0000000001583f20 g DF .text 0000000000001c22 Base dtrmm_iltucopy_STEAMROLLER
-00000000021bf830 g DF .text 0000000000000527 Base ztbcon_
-0000000001a70630 g DF .text 00000000000001c7 Base dsymm_oltcopy_SKYLAKEX
-000000000245a910 g DF .text 000000000000001b Base LAPACKE_zpp_nancheck
+00000000021a6c30 g DF .text 0000000000000527 Base ztbcon_
+0000000001a640c0 g DF .text 000000000000066d Base dsymm_oltcopy_SKYLAKEX
+0000000002441d10 g DF .text 000000000000001b Base LAPACKE_zpp_nancheck
000000000108ee20 g DF .text 000000000000014d Base zgemm3m_oncopyb_ATOM
-0000000002409df0 g DF .text 000000000000035c Base LAPACKE_zgtsvx_work
-0000000001e7d120 g DF .text 0000000000001743 Base dlatrs_
-0000000001e948a0 g DF .text 00000000000001d1 Base drscl_
+00000000023f11f0 g DF .text 000000000000035c Base LAPACKE_zgtsvx_work
+0000000001e64520 g DF .text 0000000000001743 Base dlatrs_
+0000000001e7bca0 g DF .text 00000000000001d1 Base drscl_
00000000019ac700 g DF .text 00000000000004bd Base zhemm3m_iucopyb_ZEN
00000000003c0f30 g DF .text 000000000000001e Base support_avx512_bf16
-0000000002329ac0 g DF .text 0000000000000107 Base LAPACKE_ssytrs
+0000000002310ec0 g DF .text 0000000000000107 Base LAPACKE_ssytrs
0000000000f94890 g DF .text 00000000000002d3 Base ztrmm_oltncopy_BOBCAT
On #guix-hpc Ricardo mentioned encountering this reproducibility issue
earlier.

Ludo’.
L
L
Ludovic Courtès wrote on 3 Feb 2022 00:14
control message for bug #51536
(address . control@debbugs.gnu.org)
87bkzorhdf.fsf@gnu.org
severity 51536 important
quit
?