Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible

  • Open
  • quality assurance status badge
Details
7 participants
  • Bengt Richter
  • Chris Marusich
  • Efraim Flashner
  • Jack Hill
  • Leo Le Bouter
  • Ludovic Courtès
  • Vincent Legoll
Owner
unassigned
Submitted by
Chris Marusich
Severity
normal
C
C
Chris Marusich wrote on 2 Jun 2020 20:59
(address . bug-guix@gnu.org)
874krtnvk8.fsf@gmail.com
Hi,

As demonstrated in the following email thread, the powerpc64-linux
bootstrap-tarballs are not reproducible when cross-compiled from an
x86_64-linux system:


Four people attempted to invoke the following command using Guix commit
8159ce1970d91567468cf1bacac313099a009d2a:

guix build --no-substitutes --target=powerpc64-linux-gnu bootstrap-tarballs

All of the bootstrap tarballs except for gcc-stripped were reproducible.
However, these attempts produced four different versions of
gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz. At least two of the
attempts were performed on two different x86_64-linux systems.

The derivation that produced the differing output was:

/gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv

You can build this derivation in a variety of ways, for example:

guix build --target=powerpc64-linux-gnu -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'

or just

guix build /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv

On my x86_64-linux system, twice I tried running "guix build --check" on
this derivation, but each time it surprisingly reported no difference.
Out of paranoia, I tried deleting its output with "guix gc --delete",
and then building it again. The new output was, indeed, identical to
the old output. This makes me think that the non-reproducibility is
coming from something outside the immediate build logic of the
gcc-stripped-tarball-5.5.0.drv derivation itself.

I will next try to rebuild everything from scratch again. I will also
try to get copies of the differing outputs from the people involved in
that email thread, in order to run diffoscope on them.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl7WoXcACgkQ3UCaFdgi
Rp0twQ//VDfFrWpdAkX4XpGke8c4sT6/W2LbR7N+xl126nbXhcAKjH13SCjgX3vp
/Yrik1IlnEhXwPpujU6GXLv7UbjZLBDERMOee0bFpreaq9TQMq2NJQkcoKE4kOSr
+ockDEE71tv4ZEjW3hY8funw9uRGoZBcyPYHyIjd0bNYvRAa1x8nuhrXYE5+rRZ2
h/2k5tynQrkMpxUEVQr0XL0OWXhY5VPD9Wwgd3anY/l643RIg2eLTDF1kvnRCdOq
OFN+88hWEx3QzPMZ852SSc99XkHRBaboFkrNIquxIU2lsOHiUSwOkT4/NrdOoeW7
+pXgJ2dcUNvI1JHER50NUjHBndqCke1bYUbCll0f9VE7hi1z/V/hut4t7a5uTUXp
QP+aLeS9WDDDyFHwYPf6hotjNwCyOE1aBo0LlNpR+nIX4WSDYCzSKIFQFAva84l2
TDRte0rIKcZ/B3xRraGjgFwJBezme8RkOiORNzFpcRzf0y5hCSPvL3cLc8XmsbAl
HT8pCdm0b3kSlBNQmUSV+XxnXvpIUDnxernYb7PEYhgWsopyHBgQtfxMqCj/IZtE
OXeh/oumsuKZWxDcgbUI81liQPNlrFlXgfsWiUmtBmv56g/jFko+e+IgIIZroGWj
iWHVBGz1EMaENBMXpga4g7XMRTXoXO3QB9xGQ34mHsZC007QPwQ=
=DG+o
-----END PGP SIGNATURE-----

C
C
Chris Marusich wrote on 3 Jun 2020 11:48
(address . 41669@debbugs.gnu.org)
87y2p4mqe2.fsf@gmail.com
Hi all,

Chris Marusich <cmmarusich@gmail.com> writes:

Toggle quote (7 lines)
> The derivation that produced the differing output was:
>
> /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
>
> On my x86_64-linux system, twice I tried running "guix build --check" on
> this derivation, but each time it surprisingly reported no difference.

This derivation corresponds to %gcc-bootstrap-tarball from (gnu packages
make-bootstrap), which just creates a tarball of the output of
%gcc-stripped. Therefore, it's not too surprising that it's
reproducible. Similarly, %gcc-stripped just strips some store
references from the output of %gcc-static. The %gcc-static package is
more interesting: it's where we actually build the statically linked
bootstrap GCC (which is then stripped and packed into a tarball as
mentioned above), so I thought that it might be where the
non-determinism is coming from. However, I haven't yet pinpointed the
problem.

If you examine the derivations and their inputs, you'll find that they
depend upon each other in the following order:

guix build --target=powerpc64-linux-gnu -d -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'
/gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv

guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-stripped)'
/gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv

guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
/gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv

In other words, gcc-static-5.5.0.drv is an input of
gcc-stripped-5.5.0.drv, which is an input of
gcc-stripped-tarball-5.5.0.drv. Above, I've included example guix
commands you can use to obtain each derivation. Using "guix build
--check", I confirmed that all three of these derivations build
reproducibly on my machine.

I hoped to find more information by invoking "guix build --check" on
every input of gcc-static-5.5.0.drv. When I tried that, what I found
was that all of its inputs build reproducibly except the following two:

/gnu/store/x32cnfkd50fnxs10xp1jdn24h7ai2gxr-guile-3.0.2.drv
/gnu/store/g9fpkg2qa27mka1znqsvx8vxqyabsj2y-gcc-7.5.0.drv

I haven't investigated guile-3.0.2.drv. However, gcc-7.5.0.drv felt
more suspicious to me, and it is actually the derivation that builds
gcc-final from (gnu packages commencement), which you can see via:

guix build -d -e '(@@ (gnu packages commencement) gcc-final)'

Using "guix gc", I deleted the outputs of
gcc-stripped-tarball-5.5.0.drv, gcc-stripped-5.5.0.drv,
gcc-static-5.5.0.drv, and gcc-7.5.0.drv. I then tried building these
four derivations again (without substitutes, the same as before).
Before doing this, I stored the SHA-512 hashes of their output files,
and after the build succeeded, I compared the hashes of the new files
with the previous values. I found that these derivations' newly rebuilt
outputs were identical to their original values, except for
gcc-7.5.0.drv, which produced some different files. Most significantly,
this means that gcc-stripped-tarball-5.5.0.drv produced the exact same
tarball for me as it did the first time I built it, even though some of
its inputs are not themselves reproducible.

At present, it seems possible that within the context of a single
machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a
different machine, it may (reproducibly) build a different output. I'm
a bit paranoid about making mistakes, so I'll perform another full GC
and then try yet again to build gcc-stripped-tarball-5.5.0.drv in order
to verify whether it truly produces the same output when all (or nearly
all) of its inputs are rebuilt from scratch.

Some people have also shared their differing copies of the binaries on
the email list. It could be productive to compare the contents with
diffoscope, although I suspect the diff might be too large to be useful.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl7XceUACgkQ3UCaFdgi
Rp0EgRAA0OmlaDC8TFlxSXsGjlbPVhV5kO7mplWnYNAbROMWrFWoRU4okIISdCeV
pklnsneNtg9UN+OagjGMNsgw6w/iaiftMduLS17AqWeUoaCi7HUTGVgwg6AZD2z3
nWFFgjPdYy8NO0uSA4wxoqGXHIXZsmy6m8HlNkQ5Wxv4ILrMLW1aAloAwLuicnD7
xszXKVfC38UyVwdjVyTbdVqUcl8QPY8jYdQtk+RtK6Yb1aWba6xyFW6FZhgNl4PW
DcEIBD6lKewXYT8saDJkkD1dBhkGiHjMPWkDPKiDuPR7dfDhrm4QHxRZHifv5Fu3
wGoya2yxyCtaB2O/xH8YLiQiLbWjGE+pZdbGN3hIwf8ie2lilK97895hTIhMRasi
1q6XMpRW27X8cZG7nv/spMyo3Zu/PudOghzOk/h16W0Ey24hLv0IO4xAtUxCprnB
4gtU/hKxbKYS8mN4K9AmkgnCDlibfAuEqRAwAcDpA3oxobjZLTih/BBlulUn3pxN
lf4/xIn9DCaZpi9pckjYnucByP0wv/JjA4aJGBpPM9rfau805E8XeHtBBX1djsL0
ekPaT9BkEYVxILT9eHkrIp3JO5P1H0dRqhtT6XUFShKwe9FC3BgxaJeW8v7DsLnv
bsKJ/0nXMjrcjJBqR5FhuAy/ZZXi+ILivSHQ5QlG/D1uBg8kEPM=
=kMiK
-----END PGP SIGNATURE-----

V
C
C
Chris Marusich wrote on 10 Jun 2020 08:15
(address . 41669@debbugs.gnu.org)
87imfzcuqi.fsf@gmail.com
Hi Vincent and everyone,

Vincent Legoll <vincent.legoll@gmail.com> writes:

Toggle quote (6 lines)
> Is that showing the same (or a similar) problem :
>
> https://data.guix-patches.cbaines.net/gnu/store/0lcbxpw1vrca02dzpzw2rxhad7pn4zw7-gcc-objc-5.5.0
>
> ?

Can you clarify what you mean? I'm not sure what you're referring to.

Chris Marusich <cmmarusich@gmail.com> writes:

Toggle quote (8 lines)
> At present, it seems possible that within the context of a single
> machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a
> different machine, it may (reproducibly) build a different output.
> I'm a bit paranoid about making mistakes, so I'll perform another full
> GC and then try yet again to build gcc-stripped-tarball-5.5.0.drv in
> order to verify whether it truly produces the same output when all (or
> nearly all) of its inputs are rebuilt from scratch.

I repeated the experiment on the same machine (it took a day or two to
build), and the result was the same: on my machine,
gcc-stripped-tarball-5.5.0.drv builds identical output to what it built
before. To be clear, using Guix 8159ce1970d91567468cf1bacac313099a009d2a
on an x86_64-linux machine, I tried (yet again) the following steps:

- I deleted as much as I could from the store using 'guix gc'.

- I explicitly verified that all outputs for the following derivations
no longer existed in the store:

/gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
/gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv
/gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv
/gnu/store/g9fpkg2qa27mka1znqsvx8vxqyabsj2y-gcc-7.5.0.drv

- I then built gcc-stripped-tarball-5.5.0.drv via the following command:

guix build --no-substitutes --check --target=powerpc64-linux-gnu \
-e '(@@ (gnu packages make-bootstrap) %gcc-static)'

This rebuilt basically everything, including gcc-7.5.0.drv and the other
derivations mentioned above. When I checked the built output of the
gcc-stripped-tarball-5.5.0.drv derivation, I found that its SHA-512 hash
was identical to the one I calculated previously, which was:

Toggle snippet (3 lines)
8aca7f332a1ba8e3c2225c161a7545b0a04ddd690d164dc97afee9c9ea067b0c49bc155e9f06d285c22e24cdd16d91e59730af5f1dd9efcda13a26bede5948a2 /gnu/store/rsmhiyplmbiqm1qwniiafi4ak76pd61v-gcc-stripped-tarball-5.5.0/gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz

Anyway, this confirms what we already knew: GCC builds reproducibly on
my machine, but it is different when other people build it on other
machines. I'm now quite convinced of this fact.

Jack and Vincent shared their build results on the email list. For
reference, you can get them here:


I also received a copy of Vincent's build results via private email. In
short, they all seem to differ in basically the same ways:

Toggle snippet (63 lines)
[1] marusich@garuda-lan:~/Documents/notes/guix/ppc/gcc-stripped-tarballs
$ diff -r chris jack
Binary files chris/bin/c++ and jack/bin/c++ differ
Binary files chris/bin/cpp and jack/bin/cpp differ
Binary files chris/bin/g++ and jack/bin/g++ differ
Binary files chris/bin/gcc and jack/bin/gcc differ
Binary files chris/bin/gcov and jack/bin/gcov differ
Binary files chris/bin/gcov-dump and jack/bin/gcov-dump differ
Binary files chris/bin/gcov-tool and jack/bin/gcov-tool differ
Binary files chris/bin/powerpc64-linux-gnu-c++ and jack/bin/powerpc64-linux-gnu-c++ differ
Binary files chris/bin/powerpc64-linux-gnu-g++ and jack/bin/powerpc64-linux-gnu-g++ differ
Binary files chris/bin/powerpc64-linux-gnu-gcc and jack/bin/powerpc64-linux-gnu-gcc differ
Binary files chris/bin/powerpc64-linux-gnu-gcc-5.5.0 and jack/bin/powerpc64-linux-gnu-gcc-5.5.0 differ
Binary files chris/lib/libstdc++.a and jack/lib/libstdc++.a differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 and jack/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus and jack/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 and jack/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper and jack/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper differ
[1] marusich@garuda-lan:~/Documents/notes/guix/ppc/gcc-stripped-tarballs
$ diff -r chris vincent
Binary files chris/bin/c++ and vincent/bin/c++ differ
Binary files chris/bin/cpp and vincent/bin/cpp differ
Binary files chris/bin/g++ and vincent/bin/g++ differ
Binary files chris/bin/gcc and vincent/bin/gcc differ
Binary files chris/bin/gcov and vincent/bin/gcov differ
Binary files chris/bin/gcov-dump and vincent/bin/gcov-dump differ
Binary files chris/bin/gcov-tool and vincent/bin/gcov-tool differ
Binary files chris/bin/powerpc64-linux-gnu-c++ and vincent/bin/powerpc64-linux-gnu-c++ differ
Binary files chris/bin/powerpc64-linux-gnu-g++ and vincent/bin/powerpc64-linux-gnu-g++ differ
Binary files chris/bin/powerpc64-linux-gnu-gcc and vincent/bin/powerpc64-linux-gnu-gcc differ
Binary files chris/bin/powerpc64-linux-gnu-gcc-5.5.0 and vincent/bin/powerpc64-linux-gnu-gcc-5.5.0 differ
Binary files chris/lib/libstdc++.a and vincent/lib/libstdc++.a differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 and vincent/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus and vincent/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 and vincent/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper and vincent/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper differ
[1] marusich@garuda-lan:~/Documents/notes/guix/ppc/gcc-stripped-tarballs
$ diff -r chris efraim
Binary files chris/bin/c++ and efraim/bin/c++ differ
Binary files chris/bin/cpp and efraim/bin/cpp differ
Binary files chris/bin/g++ and efraim/bin/g++ differ
Binary files chris/bin/gcc and efraim/bin/gcc differ
Binary files chris/bin/gcov and efraim/bin/gcov differ
Binary files chris/bin/gcov-dump and efraim/bin/gcov-dump differ
Binary files chris/bin/gcov-tool and efraim/bin/gcov-tool differ
Binary files chris/bin/powerpc64-linux-gnu-c++ and efraim/bin/powerpc64-linux-gnu-c++ differ
Binary files chris/bin/powerpc64-linux-gnu-g++ and efraim/bin/powerpc64-linux-gnu-g++ differ
Binary files chris/bin/powerpc64-linux-gnu-gcc and efraim/bin/powerpc64-linux-gnu-gcc differ
Binary files chris/bin/powerpc64-linux-gnu-gcc-5.5.0 and efraim/bin/powerpc64-linux-gnu-gcc-5.5.0 differ
diff -r chris/lib/gcc/powerpc64-linux-gnu/5.5.0/include-fixed/bits/statx.h efraim/lib/gcc/powerpc64-linux-gnu/5.5.0/include-fixed/bits/statx.h
5c5
< "/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/include/bits/statx.h"
---
> "/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/include/bits/statx.h"
Binary files chris/lib/libstdc++.a and efraim/lib/libstdc++.a differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 and efraim/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus and efraim/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 and efraim/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper and efraim/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper differ
[1] marusich@garuda-lan:~/Documents/notes/guix/ppc/gcc-stripped-tarballs
$

Efraim's diff looks a little different in statx.h, even though he used
the same Guix commit as me. Maybe this is because he cross-compiled on
an aarch64-linux machine, while I cross-compiled on an x86_64-linux
machine. In the other cases, it looks like the binary files differ in
basically the same ways. I will share some examples below.

Here is some diffoscope output between my c++ and Efraim's (many other
sections also differed in similarly cryptic ways):

Toggle snippet (85 lines)
diffoscope chris/bin/c++ efraim/bin/c++
...
├── /gnu/store/xakj5dgs1729297nv50s84sdmq2jiz64-binutils-2.34/bin/readelf --wide --symbols {}
│ @@ -328,18 +328,18 @@
│ 324: 00000000101aece0 152 FUNC LOCAL DEFAULT 19 check_free.isra.0
│ 325: 00000000101aed28 24 FUNC LOCAL DEFAULT 19 fini
│ 326: 00000000101b6008 8 OBJECT LOCAL DEFAULT 24 static_buf
│ 327: 00000000101b6010 32 OBJECT LOCAL DEFAULT 24 last_result
│ 328: 00000000101b6030 4 OBJECT LOCAL DEFAULT 24 key
│ 329: 00000000101b6034 4 OBJECT LOCAL DEFAULT 24 once
│ 330: 00000000101b23b8 104 OBJECT LOCAL DEFAULT 22 _dlfcn_hooks
│ - 331: 00000000101a98f8 524 FUNC LOCAL DEFAULT 19 _ZN12_GLOBAL__N_14pool4freeEPv.constprop.2
│ - 332: 00000000101a9910 524 FUNC LOCAL DEFAULT 19 _ZN12_GLOBAL__N_14pool8allocateEm.constprop.3
│ - 333: 00000000101a99a0 180 FUNC LOCAL DEFAULT 19 _GLOBAL__sub_I_eh_alloc.cc
│ - 334: 00000000101b4f68 64 OBJECT LOCAL DEFAULT 24 _ZN12_GLOBAL__N_114emergency_poolE
│ + 331: 00000000101a9880 524 FUNC LOCAL DEFAULT 19 _ZN12_GLOBAL__N_14pool4freeEPv.constprop.2
│ + 332: 00000000101a9898 524 FUNC LOCAL DEFAULT 19 _ZN12_GLOBAL__N_14pool8allocateEm.constprop.3
│ + 333: 00000000101a9928 180 FUNC LOCAL DEFAULT 19 _GLOBAL__sub_I_eh_alloc.cc
│ + 334: 00000000101b4f78 64 OBJECT LOCAL DEFAULT 24 _ZN12_GLOBAL__N_114emergency_poolE
│ 335: 000000001011c5a8 19 OBJECT LOCAL DEFAULT 7 _ZZ18ggc_internal_allocmPFvPvEmmE12__FUNCTION__
│ 336: 000000001011c5c0 27 OBJECT LOCAL DEFAULT 7 _ZZ26ggc_internal_cleared_allocmPFvPvEmmE12__FUNCTION__
│ 337: 000000001011c600 21 OBJECT LOCAL DEFAULT 7 _ZZ20lang_specific_driverPP17cl_decoded_optionPjPiE12__FUNCTION__
│ 338: 00000000101a42d0 228 FUNC LOCAL DEFAULT 19 _ZL12elf_platformv
│ 339: 00000000101a42e8 236 FUNC LOCAL DEFAULT 19 _ZL15elf_dcachebsizev
│ 340: 00000000101a4300 164 FUNC LOCAL DEFAULT 19 _ZL14describe_cachejjjj.isra.0.constprop.1
│ 341: 000000001011c6b8 752 OBJECT LOCAL DEFAULT 7 _ZL9asm_names
...
│ @@ -1379,25 +1379,25 @@
│ 1375: 00000000101b2448 0 SECTION LOCAL DEFAULT 23
│ 1376: 00000000101b2b80 0 SECTION LOCAL DEFAULT 24
│ 1377: 00000000101b6860 0 SECTION LOCAL DEFAULT 25
│ 1378: 0000000000000000 0 SECTION LOCAL DEFAULT 26
│ 1379: 0000000000000000 0 SECTION LOCAL DEFAULT 27
│ 1380: 00000000101a2a28 752 OBJECT GLOBAL DEFAULT 18 _nl_C_LC_CTYPE
│ 1381: 00000000101ace20 56 IFUNC GLOBAL DEFAULT 19 __libc_strcspn
│ - 1382: 00000000101a9958 100 FUNC GLOBAL DEFAULT 19 __cxa_free_exception
│ + 1382: 00000000101a98e0 100 FUNC GLOBAL DEFAULT 19 __cxa_free_exception
│ 1383: 00000000101acfe8 104 FUNC WEAK DEFAULT 19 stpcpy
│ 1384: 00000000101ab7e8 516 FUNC GLOBAL DEFAULT 19 putchar
│ - 1385: 00000000101a9598 20 FUNC GLOBAL DEFAULT 19 _ZNKSt9type_info15__is_function_pEv
│ + 1385: 00000000101a9778 20 FUNC GLOBAL DEFAULT 19 _ZNKSt9type_info15__is_function_pEv
│ 1386: 000000001016a3cc 76 OBJECT GLOBAL DEFAULT 7 _nl_C_LC_CTYPE_class_print
│ 1387: 00000000101adfc0 640 FUNC WEAK DEFAULT 19 tsearch
│ 1388: 00000000101af0a0 176 FUNC WEAK DEFAULT 19 clock_gettime
│ 1389: 00000000101a4150 4856 FUNC GLOBAL DEFAULT 19 _ZNK6driver12set_up_specsEv
│ 1390: 00000000101a9268 88 FUNC GLOBAL DEFAULT 19 xmalloc
│ 1391: 00000000101b1d68 8 OBJECT GLOBAL DEFAULT 22 __morecore
│ - 1392: 00000000101a9568 40 FUNC GLOBAL DEFAULT 19 _ZSt10unexpectedv
│ + 1392: 00000000101a94f0 40 FUNC GLOBAL DEFAULT 19 _ZSt10unexpectedv
│ 1393: 00000000101a8ef0 60 FUNC GLOBAL DEFAULT 19 lbasename
│ 1394: 00000000101aded0 96 FUNC GLOBAL DEFAULT 19 __getdtablesize
│ 1395: 00000000101a4348 752 FUNC GLOBAL DEFAULT 19 update_path
...
├── /gnu/store/xakj5dgs1729297nv50s84sdmq2jiz64-binutils-2.34/bin/readelf --wide --decompress --hex-dump=.data.rel.ro {}
│ @@ -112,27 +112,27 @@
│ 0x101a1718 00000000 10164560 00000009 00000001 ......E`........
│ 0x101a1728 00000000 10164570 00000000 10164578 ......Ep......Ex
│ 0x101a1738 0000000c 00000002 00000000 101499e8 ................
│ 0x101a1748 00000000 10116018 00000001 00000001 ......`.........
│ 0x101a1758 00000000 10164588 00000000 10164590 ......E.......E.
│ 0x101a1768 00000007 00000001 00000000 10119850 ...............P
│ 0x101a1778 00000000 1015e290 00000002 00000002 ................
│ - 0x101a1788 00000000 101641c0 00000000 101653e8 ......A.......S.
│ + 0x101a1788 00000000 101641c0 00000000 101653a8 ......A.......S.
│ 0x101a1798 00000001 00000002 00000000 10164598 ..............E.
│ 0x101a17a8 00000000 10117df0 00000001 00000002 ......}.........
│ 0x101a17b8 00000000 101645a0 00000000 1015e238 ......E........8
│ 0x101a17c8 00000002 00000002 00000000 101645a8 ..............E.
│ 0x101a17d8 00000000 101631d8 00000001 00000002 ......1.........
│ 0x101a17e8 00000000 101645b0 00000000 1015e1e0 ......E.........
│ - 0x101a17f8 00000002 00000002 00000000 101653c0 ..............S.
│ + 0x101a17f8 00000002 00000002 00000000 10165380 ..............S.
│ 0x101a1808 00000000 1015e268 00000003 00000003 .......h........
│ - 0x101a1818 00000000 101653c8 00000000 1015e268 ......S........h
│ - 0x101a1828 00000003 00000003 00000000 101653d0 ..............S.
│ + 0x101a1818 00000000 10165388 00000000 1015e268 ......S........h
│ + 0x101a1828 00000003 00000003 00000000 10165390 ..............S.
│ 0x101a1838 00000000 1015e268 00000003 00000002 .......h........
│ - 0x101a1848 00000000 101653d8 00000000 1015e268 ......S........h
│ + 0x101a1848 00000000 10165398 00000000 1015e268 ......S........h
│ 0x101a1858 00000003 00000002 00000000 10144ea8 ..............N.
│ 0x101a1868 00000000 1015e1f0 00000002 00000002 ................
...

Here is some diffoscope output betweeen my c++ and Jack's (many other
sections also differed in similarly cryptic ways):

Toggle snippet (72 lines)
diffoscope chris/bin/c++ jack/bin/c++
...
├── /gnu/store/xakj5dgs1729297nv50s84sdmq2jiz64-binutils-2.34/bin/readelf --wide --symbols {}
│ @@ -328,17 +328,17 @@
│ 324: 00000000101aece0 152 FUNC LOCAL DEFAULT 19 check_free.isra.0
│ 325: 00000000101aed28 24 FUNC LOCAL DEFAULT 19 fini
│ 326: 00000000101b6008 8 OBJECT LOCAL DEFAULT 24 static_buf
│ 327: 00000000101b6010 32 OBJECT LOCAL DEFAULT 24 last_result
│ 328: 00000000101b6030 4 OBJECT LOCAL DEFAULT 24 key
│ 329: 00000000101b6034 4 OBJECT LOCAL DEFAULT 24 once
│ 330: 00000000101b23b8 104 OBJECT LOCAL DEFAULT 22 _dlfcn_hooks
│ - 331: 00000000101a98f8 524 FUNC LOCAL DEFAULT 19 _ZN12_GLOBAL__N_14pool4freeEPv.constprop.2
│ - 332: 00000000101a9910 524 FUNC LOCAL DEFAULT 19 _ZN12_GLOBAL__N_14pool8allocateEm.constprop.3
│ - 333: 00000000101a99a0 180 FUNC LOCAL DEFAULT 19 _GLOBAL__sub_I_eh_alloc.cc
│ + 331: 00000000101a94c0 524 FUNC LOCAL DEFAULT 19 _ZN12_GLOBAL__N_14pool4freeEPv.constprop.2
│ + 332: 00000000101a94d8 524 FUNC LOCAL DEFAULT 19 _ZN12_GLOBAL__N_14pool8allocateEm.constprop.3
│ + 333: 00000000101a9568 180 FUNC LOCAL DEFAULT 19 _GLOBAL__sub_I_eh_alloc.cc
│ 334: 00000000101b4f68 64 OBJECT LOCAL DEFAULT 24 _ZN12_GLOBAL__N_114emergency_poolE
│ 335: 000000001011c5a8 19 OBJECT LOCAL DEFAULT 7 _ZZ18ggc_internal_allocmPFvPvEmmE12__FUNCTION__
│ 336: 000000001011c5c0 27 OBJECT LOCAL DEFAULT 7 _ZZ26ggc_internal_cleared_allocmPFvPvEmmE12__FUNCTION__
│ 337: 000000001011c600 21 OBJECT LOCAL DEFAULT 7 _ZZ20lang_specific_driverPP17cl_decoded_optionPjPiE12__FUNCTION__
│ 338: 00000000101a42d0 228 FUNC LOCAL DEFAULT 19 _ZL12elf_platformv
│ 339: 00000000101a42e8 236 FUNC LOCAL DEFAULT 19 _ZL15elf_dcachebsizev
│ 340: 00000000101a4300 164 FUNC LOCAL DEFAULT 19 _ZL14describe_cachejjjj.isra.0.constprop.1
...
├── /gnu/store/xakj5dgs1729297nv50s84sdmq2jiz64-binutils-2.34/bin/readelf --wide --decompress --hex-dump=.data.rel.ro {}
│ @@ -206,60 +206,60 @@
│ 0x101a1cf8 00000000 101a3b08 00000000 10118108 ......;.........
│ 0x101a1d08 00000000 101a4120 00000000 10118120 ......A .......
│ 0x101a1d18 00000000 101a4108 00000000 10115758 ......A.......WX
│ 0x101a1d28 00000000 101a3a60 00000000 10118138 ......:`.......8
│ 0x101a1d38 00000000 101a3a48 00000000 10115ae8 ......:H......Z.
│ 0x101a1d48 00000000 101a3c10 00000000 10118150 ......<........P
│ 0x101a1d58 00000000 101a4318 00000000 00000000 ......C.........
│ - 0x101a1d68 00000000 00000000 00000000 101a1dd8 ................
│ - 0x101a1d78 00000000 101650d0 00000000 101a1e70 ......P........p
│ - 0x101a1d88 00000000 00000000 00000000 101a1d70 ...............p
│ - 0x101a1d98 00000000 101a9388 00000000 101a93a0 ................
│ - 0x101a1da8 00000000 101a9370 00000000 101a1dd8 .......p........
│ - 0x101a1db8 00000000 101650f0 00000000 101a1fd8 ......P.........
│ - 0x101a1dc8 00000000 00000000 00000000 101a1db0 ................
│ - 0x101a1dd8 00000000 101a93d0 00000000 101a93e8 ................
│ - 0x101a1de8 00000000 101a9598 00000000 101a9598 ................
│ - 0x101a1df8 00000000 101a9b20 00000000 101a9a90 ....... ........
│ - 0x101a1e08 00000000 101a9430 00000000 101a9418 .......0........
│ - 0x101a1e18 00000000 101a9400 00000000 101a2000 .............. .
│ - 0x101a1e28 00000000 10165118 00000000 00000000 ......Q.........
│ - 0x101a1e38 00000000 101a1e20 00000000 101a9580 ....... ........
│ - 0x101a1e48 00000000 101a95c8 00000000 101a9598 ................
│ - 0x101a1e58 00000000 101a9598 00000000 101a95e0 ................
│ - 0x101a1e68 00000000 101a95b0 00000000 101a2000 .............. .
│ - 0x101a1e78 00000000 10165128 00000000 101a1dd8 ......Q(........
│ - 0x101a1e88 00000000 10165138 00000000 101a1e70 ......Q8.......p
│ - 0x101a1e98 00000000 101a2000 00000000 10165150 ...... .......QP
│ - 0x101a1ea8 00000000 101a2000 00000000 10165170 ...... .......Qp
│ - 0x101a1eb8 00000000 00000000 00000000 101a1e70 ...............p
│ - 0x101a1ec8 00000000 101a96e8 00000000 101a9748 ...............H
│ - 0x101a1ed8 00000000 101a9718 00000000 00000000 ................
│ - 0x101a1ee8 00000000 101a1e80 00000000 101a9700 ................
│ - 0x101a1ef8 00000000 101a9760 00000000 101a9730 .......`.......0
│ - 0x101a1f08 00000000 00000000 00000000 101a1e98 ................
│ - 0x101a1f18 00000000 00000000 00000000 00000000 ................
│ - 0x101a1f28 00000000 101a99b8 00000000 00000000 ................
│ - 0x101a1f38 00000000 101a1ea8 00000000 00000000 ................
│ - 0x101a1f48 00000000 00000000 00000000 101a99b8 ................
│ - 0x101a1f58 00000000 101a1dd8 00000000 101651c0 ..............Q.
│ - 0x101a1f68 00000000 101a1e70 00000000 101a1dd8 .......p........
│ - 0x101a1f78 00000000 101651e8 00000000 101a1e70 ......Q........p
│ - 0x101a1f88 00000000 00000000 00000000 101a1f58 ...............X
│ - 0x101a1f98 00000000 101a9880 00000000 101a98b0 ................
│ - 0x101a1fa8 00000000 101a9850 00000000 00000000 .
This message was truncated. Download the full message here.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl7gemUACgkQ3UCaFdgi
Rp3CJA/+I1u4W3TlKPLAWFq8mPWkE9iJGwWKcbSfVAMONQGqqNFv6nBfix2AhZDj
trMnElbu7sDXDsihJnopBeoKmSpfYl59O4eFbRcpKzVxSg4aJ78GFoqwX9KKTNTG
l0BJNnnrSwdxWQWZiZ3+ujexdSd3gpWk176WkcrDzUWLknBfGE+SRQf2j4JldnZ+
8dDhMRhT+2jkRQBWUi6hZ6FP0xAmrOP6fpBv+aIrjQXgx6OsERpFy+paqn4lLjnn
Lr0J943KlrpSDRDTX6VOZkpgNnZqq75QWoOWdVpcNrGtzEDusw6AoxUudQm1ZNZf
TZqq1LQiH4S4V7M5T//PftkDRoCoHyynaePNrkQpzcNKjeG0jGK4xRZss+2eAEVV
pZZAXMbe17JhkNjjxxlYI+CwhXAXV5Xy5llUM1DMd9oG9TCl+21xgboIaV9Nvyz8
8fYU4B0diXCXSGLc7odtDhZfukhag9geVuOh+7XfzTdItYEfC/vwrzC9JZZt0vbj
jI6iKR5yvG6oEqdJkMTUBDONphCq2H7R3s+hyr8W5CO/VZ2UonE7Q/lE610hfpaS
lagRuriOAkFtsyt15IsfJ/dHNJDnlzwrpe2YI/7Wr+cyUBGAnAzRHsGw0oYiQNUV
jYoht9mw3kpqvh9/C8TLAF9KZBDUDWAcMaOFLSPgN72oGufMCq4=
=7To8
-----END PGP SIGNATURE-----

B
B
Bengt Richter wrote on 11 Jun 2020 00:20
(name . Chris Marusich)(address . cmmarusich@gmail.com)
20200610222008.GB3238@LionPure
Hi Chris, et al,

On +2020-06-09 23:15:01 -0700, Chris Marusich wrote:
Toggle quote (28 lines)
> Hi Vincent and everyone,
>
> Vincent Legoll <vincent.legoll@gmail.com> writes:
>
> > Is that showing the same (or a similar) problem :
> >
> > https://data.guix-patches.cbaines.net/gnu/store/0lcbxpw1vrca02dzpzw2rxhad7pn4zw7-gcc-objc-5.5.0
> >
> > ?
>
> Can you clarify what you mean? I'm not sure what you're referring to.
>
> Chris Marusich <cmmarusich@gmail.com> writes:
>
> > At present, it seems possible that within the context of a single
> > machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a
> > different machine, it may (reproducibly) build a different output.
> > I'm a bit paranoid about making mistakes, so I'll perform another full
> > GC and then try yet again to build gcc-stripped-tarball-5.5.0.drv in
> > order to verify whether it truly produces the same output when all (or
> > nearly all) of its inputs are rebuilt from scratch.
>
> I repeated the experiment on the same machine (it took a day or two to
> build), and the result was the same: on my machine,
> gcc-stripped-tarball-5.5.0.drv builds identical output to what it built
> before. To be clear, using Guix 8159ce1970d91567468cf1bacac313099a009d2a
> on an x86_64-linux machine, I tried (yet again) the following steps:
>
[...]

Toggle quote (9 lines)
> Efraim's diff looks a little different in statx.h, even though he used
> the same Guix commit as me. Maybe this is because he cross-compiled on
> an aarch64-linux machine, while I cross-compiled on an x86_64-linux
> machine. In the other cases, it looks like the binary files differ in
> basically the same ways. I will share some examples below.
>
> Here is some diffoscope output between my c++ and Efraim's (many other
> sections also differed in similarly cryptic ways):
>
[...]

Toggle quote (15 lines)
>
> If I'm reading this correctly, one problem seems to be that our GCC
> toolchains are putting symbols at different locations. This issue (and
> maybe others) could be trickling down, causing other aspects of the
> binaries to differ (e.g., in length). Nothing really stands out, but
> when we discussed this on IRC, we thought perhaps factors like the
> following might contribute to the non-reproducibility:
>
> - Perhaps we are all running different Linux kernel versions? In some
> cases, the kernel version can unfortunately influence the build
> output, so this might be worth testing.
>
> - Perhaps the GCC Makefiles etc. are doing something non-deterministic?
>

Questions triggered in my mind:

Where are respective machines getting their rules for packing and
aligning structs and unions?

Is any struct or rule/flags source dynamically generated, where different
rules could come from different defaults, or .configs, or even invalid
memoizations jumping domains?

Could pointer arithmetic get done in one domain and the offset be
misused in another? Wrong C preprocessor?

Difference in sort key comparisons for canonicalization of ordering?

Hope that's not all red herrings :)
Sorry for the noise otherwise.

Toggle quote (2 lines)
> - Something else?

Hm, some race condition between processes that should be order-independent
but are not.

Then if different hardware components on different systems -- disks, memory,
processors -- cause different but repeatable patterns of waits (convoying?)
you could get repeatable but different builds.

I guess you'd have to figure out which order was really right, and force
the order of processing explicitly to that order, so all systems would
do it that way.

Toggle quote (27 lines)
>
> Avenues of investigation:
>
> - If anything obvious stands out from the diffoscope output, please
> leave a comment.
>
> - Try building with different kernel versions on the same machine, to
> see if they differ.
>
> - If somebody else could please confirm that running the following
> command reports no difference on their own machine (i.e., exit code
> 0), that would be good to know, since it would help further solidify
> the theory that on a single machine, the build of gcc-static-5.5.0.drv
> is reproducible, even if it is not reproducible across machines:
>
> guix build --no-substitutes --check --target=powerpc64-linux-gnu \
> -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
>
> - Try building two different versions of gcc-7.5.0 (maybe by hand?), and
> then use them to build a simple reproduction case and compare results.
> If we're lucky, maybe this will help us understand the problem better.
>
> We'll get there!
>
> --
> Chris

--
Regards,
Bengt Richter
J
J
Jack Hill wrote on 11 Jun 2020 23:09
(name . Chris Marusich)(address . cmmarusich@gmail.com)
alpine.DEB.2.20.2006111707180.5735@marsh.hcoop.net
On Tue, 9 Jun 2020, Chris Marusich wrote:

Toggle quote (3 lines)
> - Try building with different kernel versions on the same machine, to
> see if they differ.

I've done the rebuild after updated from Linux 5.4.41 to 5.4.45, and go
identical results to my previous build. I am using a different kernel
configuration than Guix's default kernel, but it was the same between the
builds.

Best,
Jack
C
C
Chris Marusich wrote on 13 Sep 2020 04:53
(address . 41669@debbugs.gnu.org)
87a6xu2xrj.fsf@gmail.com
Hi everyone,

Chris Marusich <cmmarusich@gmail.com> writes:

Toggle quote (19 lines)
> If you examine the derivations and their inputs, you'll find that they
> depend upon each other in the following order:
>
> guix build --target=powerpc64-linux-gnu -d -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'
> /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
>
> guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-stripped)'
> /gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv
>
> guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
> /gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv
>
> In other words, gcc-static-5.5.0.drv is an input of
> gcc-stripped-5.5.0.drv, which is an input of
> gcc-stripped-tarball-5.5.0.drv. Above, I've included example guix
> commands you can use to obtain each derivation. Using "guix build
> --check", I confirmed that all three of these derivations build
> reproducibly on my machine.

After further experimentation, I've discovered that %gcc-static, when
built as shown above (without the -d option, of course), produces
different output on Debian than on Fedora.

Specifically, the %gcc-static output contains a file named libstdc++.a.
This file is an archive file. Although its members are
content-identical in the case of Fedora and Debian, the order of the
members in the archive differs. Because the exact same inputs were
used, it seems very likely that a difference in the Guix build
environment caused the %gcc-static build logic to order the members of
libstdc++.a differently.

I built %gcc-static using Guix commit
a02b2f8b86c0227eb69aa24b4373aef456365334. Both Debian and Fedora were
x86_64-linux systems. I took the following steps to make absolutely
certain that the exact same inputs were used on Debian and Fedora:

- I provisioned two fresh EC2 instances (Debian and Fedora).

- I installed Guix on Debian.

- I did "guix pull" on Debian to get to the aforementioned commit.

- I built %gcc-static on Debian as indicated above.

- I manually copied the Guix store and the Guix database from Debian to
Fedora.

- I manually fixed up Fedora so it could run Guix (I created the guix
users, added a systemd unit file, disabled selinux, etc.).

- I manually verified the Guix version and the store contents were
identical on Fedora and Debian.

- I GC'd %gcc-static (and nothing else) on Fedora.

- I rebuilt %gcc-static on Fedora.

- I compared the Fedora %gcc-static output to the Debian %gcc-static
output.

The %gcc-static package uses GCC 5.5.0 as its source. I got a copy of
the GCC 5.5.0 source code, and I looked at it. However, it's complex.
I can't pinpoint where they actually build the libstdc++.a file. Can
anyone point me to the code that does this in the GCC 5.5.0 source? I
expected to find the logic hiding in a makefile or a configure script or
something, but I haven't found it yet.

Since this is an old GCC, it is possible that this was a known
reproducibility bug which has since been fixed. I haven't looked into
that possibility yet. If that's the case, though, it would be nice
because we could simply backport a fix.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl9diZAACgkQ3UCaFdgi
Rp0WHxAAxQnPB0Dg4GfaXrtRxOwZsqVuNlg14Qpp46YzroCJ66MyK3x4ddekFIkj
p1pyyzLVF9RSvsPspf9oc0Go12JndQMOqpsAUE0rJLbVPbTlrNrkh9Jhhfar95IX
p+cSNQoeuab1+nOeUH17J2TMdoxsXiwzB/4IuQHQtNLe3DVNKmIYB0tgL/L/Xasc
WN0qHyNwkW/Lobs4jWNzukD3DiCv+zN0r1eDwsk02EOegW+m5yeyGMKbQUa7QCba
g5vGeSi92AiqEVkuj62/IObMFVQP1rlNUBN9a3Yn8UTxW//PF9fwkJRIpSqdmGx7
VYpRJ/e4L6NT2GE91X+RPz0Pz0J9viV/lsM8esALudxQRv2/1loh4T6mubM394Cw
yrhv2TxFmtlrBAEGJmIulCSOKGTJYy2bXbq2vR0sfbXZs3iKvPdY0FVyy3NShTz0
d3+Yk1T1dXw0tKQBmr04mpq+2SFvKTrHJB3FmVQa8/v+q+Q7ca10/X7NhcIgMG5H
Nq7ttes7zV/zlIvWRChCKUHyefQBzN00ZQeBWfsJykko4m6W5vquOVs0Log54TrJ
rgQIZAickNIQSMcG0fOVmzaj7iGf1smiaeB3bIW91Ub6FJ9NuoJuyLMFd6xmezVJ
gv3wpnd6Yh9kzORQV79C14iIDUvYIOPJn3qBm1DPe5RQdlu3wn4=
=OFt6
-----END PGP SIGNATURE-----

E
E
Efraim Flashner wrote on 13 Sep 2020 08:28
(name . Chris Marusich)(address . cmmarusich@gmail.com)
20200913062858.GC1100@E5400
On Sat, Sep 12, 2020 at 07:53:04PM -0700, Chris Marusich wrote:
Toggle quote (79 lines)
> Hi everyone,
>
> Chris Marusich <cmmarusich@gmail.com> writes:
>
> > If you examine the derivations and their inputs, you'll find that they
> > depend upon each other in the following order:
> >
> > guix build --target=powerpc64-linux-gnu -d -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'
> > /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
> >
> > guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-stripped)'
> > /gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv
> >
> > guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
> > /gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv
> >
> > In other words, gcc-static-5.5.0.drv is an input of
> > gcc-stripped-5.5.0.drv, which is an input of
> > gcc-stripped-tarball-5.5.0.drv. Above, I've included example guix
> > commands you can use to obtain each derivation. Using "guix build
> > --check", I confirmed that all three of these derivations build
> > reproducibly on my machine.
>
> After further experimentation, I've discovered that %gcc-static, when
> built as shown above (without the -d option, of course), produces
> different output on Debian than on Fedora.
>
> Specifically, the %gcc-static output contains a file named libstdc++.a.
> This file is an archive file. Although its members are
> content-identical in the case of Fedora and Debian, the order of the
> members in the archive differs. Because the exact same inputs were
> used, it seems very likely that a difference in the Guix build
> environment caused the %gcc-static build logic to order the members of
> libstdc++.a differently.
>
> I built %gcc-static using Guix commit
> a02b2f8b86c0227eb69aa24b4373aef456365334. Both Debian and Fedora were
> x86_64-linux systems. I took the following steps to make absolutely
> certain that the exact same inputs were used on Debian and Fedora:
>
> - I provisioned two fresh EC2 instances (Debian and Fedora).
>
> - I installed Guix on Debian.
>
> - I did "guix pull" on Debian to get to the aforementioned commit.
>
> - I built %gcc-static on Debian as indicated above.
>
> - I manually copied the Guix store and the Guix database from Debian to
> Fedora.
>
> - I manually fixed up Fedora so it could run Guix (I created the guix
> users, added a systemd unit file, disabled selinux, etc.).
>
> - I manually verified the Guix version and the store contents were
> identical on Fedora and Debian.
>
> - I GC'd %gcc-static (and nothing else) on Fedora.
>
> - I rebuilt %gcc-static on Fedora.
>
> - I compared the Fedora %gcc-static output to the Debian %gcc-static
> output.
>
> The %gcc-static package uses GCC 5.5.0 as its source. I got a copy of
> the GCC 5.5.0 source code, and I looked at it. However, it's complex.
> I can't pinpoint where they actually build the libstdc++.a file. Can
> anyone point me to the code that does this in the GCC 5.5.0 source? I
> expected to find the logic hiding in a makefile or a configure script or
> something, but I haven't found it yet.
>
> Since this is an old GCC, it is possible that this was a known
> reproducibility bug which has since been fixed. I haven't looked into
> that possibility yet. If that's the case, though, it would be nice
> because we could simply backport a fix.
>
> --
> Chris

Is this a file we actually need during the bootstrap process? Can we
"work around it" by just deleting it?


--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl9dvCoACgkQQarn3Mo9
g1Ecbg//XfW6XzWsg5kVwdUW0cxO40TNaBumnuPBed5NGR92cCP1QibPEzbvpM2y
Vol26ZUjFatN5REkuWnn4JCuTy6S4BUmzhV5UGwupioEuq2LdMeWGT2rpgB/f9pI
E2RfT/PfAZSW2pEIASzVHzJyEL+fKX2LzByUWciIkmssYDRfbtYfpLNsgUGSxqLw
mRMecqUW52UalRx7vKdqQOwbuF235hmHRx6JTlnoauq7KkYgx1qLqzwbty4VhkDR
J3D8I40lVuEcaZ4rRdn4Ng33B3v6HR7VhMN+59f0amRc4D6c4HHfknxyvU8wXnI8
cT43E7k+ZlW9TsGyZXWiL1vxolcBnggPopJ6ZCIDhDllFpHVtCtrhTR0wDyqw3X8
TtpAXUFlqwI+ivPgzOrYs3cYqZp8VoHfp/COO5rEKOCdA4aWhh2XEpWPMBgUQ1oo
0EWJW7Fx+0RcOJd1Dd+QWDSWd9n4nUqp6NtTDtnU+LZGQvmS557v4q3ehBhvJJq+
ioGVpT1Z4HEHrwl0a6W9LQjKpj9k/KYD3nqhBoWaDAZKqYgikKlRR9vsgUwqszFr
BsGrS/MRt9qsCZtxz0jgE4vSz5yB4Jet/LhOBaDr+DANuGv6fEJ3aciC3217PgYe
yad0y9FRbHPuScohHrmk+b+SXKLxnZy68hDaUPXQDIb+H0kp6vU=
=qHSj
-----END PGP SIGNATURE-----


C
C
Chris Marusich wrote on 26 Sep 2020 08:52
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87wo0hqbb3.fsf@gmail.com
Hi everyone,

Efraim Flashner <efraim@flashner.co.il> writes:

Toggle quote (3 lines)
> Is this a file we actually need during the bootstrap process? Can we
> "work around it" by just deleting it?

That's a good idea. I tried building %gcc-static with the
--disable-libstdcxx configure flag (see attached patch), and that caused
the build of %gcc-static itself to become reproducible on Debian and
Fedora when using exactly the same inputs. To clarify, I have recently
run the following experiments:

+------+-------------+---------------------+--------------+----------------+---------------+
| Case | Build | libstdcxx | substitutes? | Inputs | Result |
+------+-------------+---------------------+--------------+----------------+---------------+
| | | | | Built fresh on | Output |
| 1 | %gcc-static | enabled | yes | Debian, copied | differs: |
| | | | | to Fedora | libstdc++.a |
+------+-------------+---------------------+--------------+----------------+---------------+
| | | | | Re-used inputs | Toy binary |
| 2 | toy program | n/a | yes | from above | does not |
| | | | | | differ |
+------+-------------+---------------------+--------------+----------------+---------------+
| | | | | Built fresh on | Output does |
| 3 | %gcc-static | disabled | yes | Debian, copied | not differ |
| | | | | to Fedora | |
+------+-------------+---------------------+--------------+----------------+---------------+
| | | | | Built fresh on | Output |
| 4 | %gcc-static | disabled | yes | Debian, fresh | differs: |
| | | | | on Fedora | various files |
+------+-------------+---------------------+--------------+----------------+---------------+
| | | | | Inputs failed | Build failed |
| 5 | %gcc-static | disabled | no | to build on | on both |
| | | | | both systems | systems |
+------+-------------+---------------------+--------------+----------------+---------------+

The "toy program" in case 2 was just this:

#include <stdio.h>
int main() {
printf("Hello");
return 0;
}

When I say I "copied" the inputs from Debian to Fedora, I mean just
that. I copied stuff like /gnu and /var/guix from Debian to Fedora
(while guix-daemon was stopped, of course), GC'd just %gcc-static on
Fedora, and then rebuilt %gcc-static on Fedora so it would use exactly
the same inputs as were used on Debian.

The most notable new findings are cases 3 and 4. Case 5 made me pretty
sad because I spent almost an entire weekend trying to build Guix from
source using various binary Guix releases, and in none of the attempts
was I successful in running "guix pull" or even just "guix environment
--pure guix" without substitutes, which really surprised me.

Case 3 confirms Efraim's suggestion: we can fix the libstdc++.a
reproducibility problem by simply not building libstdc++.a in the first
place. It also confirms that, when built with --disable-libstdcxx,
%gcc-static can be built reproducibly on different systems as long as
exactly the same inputs are used. Whether or not %gcc-static can be
used to successfully bootstrap packages on powerpc64-linux when built in
this way remains to be seen.

Case 4, unfortunately, demonstrates that there are still other
reproducibility issues that we have not yet resolved. Since the only
difference between cases 3 and 4 is how the inputs were realized, the
differing %gcc-static output must be caused by the inputs somehow.

In case 4, the %gcc-static output differed in the following files (the
hashes on the left are MD5 hashes):

Toggle snippet (48 lines)
--- /dev/fd/63 2020-09-25 20:35:33.386554595 -0700
+++ /dev/fd/62 2020-09-25 20:35:33.387554604 -0700
@@ -1,28 +1,28 @@
-c9b0dfcbad566c0b8b88df94bb993312 ./bin/c++
-092823145dc96b9eb81111362f7b4ced ./bin/cpp
-c9b0dfcbad566c0b8b88df94bb993312 ./bin/g++
-e4cc43b7790dcd25f31419bad606b36e ./bin/gcc
+8f02302b55643f1c711e472a42fea8bd ./bin/c++
+9f1fd993e4f2b796fcc56f0b2f8a47d2 ./bin/cpp
+8f02302b55643f1c711e472a42fea8bd ./bin/g++
+583d1b011a7ba009d7385117dd7a33c8 ./bin/gcc
f9d94f4bb61f70d14ea4b2ce73c9be9d ./bin/gcc-ar
01fc2184f99c558771aa8f2fe30b373d ./bin/gcc-nm
da5356ee09ccda4ca06758d056370f7e ./bin/gcc-ranlib
-98645f7b00ba185e713915099853fd37 ./bin/gcov
-37dd62589454703ae7f2eaac1668b66e ./bin/gcov-dump
-f3dbc7e0c84a40194af3aa2429444e87 ./bin/gcov-tool
-c9b0dfcbad566c0b8b88df94bb993312 ./bin/powerpc64-linux-gnu-c++
-c9b0dfcbad566c0b8b88df94bb993312 ./bin/powerpc64-linux-gnu-g++
-e4cc43b7790dcd25f31419bad606b36e ./bin/powerpc64-linux-gnu-gcc
-e4cc43b7790dcd25f31419bad606b36e ./bin/powerpc64-linux-gnu-gcc-5.5.0
+a208bedbfca9c7bd6c27d0d42f7096fe ./bin/gcov
+43330e8ae00976b4b3427d2f83b0725e ./bin/gcov-dump
+9f37da5e96f147d733eb7195350ae5d2 ./bin/gcov-tool
+8f02302b55643f1c711e472a42fea8bd ./bin/powerpc64-linux-gnu-c++
+8f02302b55643f1c711e472a42fea8bd ./bin/powerpc64-linux-gnu-g++
+583d1b011a7ba009d7385117dd7a33c8 ./bin/powerpc64-linux-gnu-gcc
+583d1b011a7ba009d7385117dd7a33c8 ./bin/powerpc64-linux-gnu-gcc-5.5.0
f9d94f4bb61f70d14ea4b2ce73c9be9d ./bin/powerpc64-linux-gnu-gcc-ar
01fc2184f99c558771aa8f2fe30b373d ./bin/powerpc64-linux-gnu-gcc-nm
da5356ee09ccda4ca06758d056370f7e ./bin/powerpc64-linux-gnu-gcc-ranlib
-6ed530d13e65c3500b7e7b7cc863afdc ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1
-24a83af179ca8849da8c64aa854ec8ed ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus
-0c05b45bb926a06c2b09acdb1db9aad0 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2
+22b72247a5706f090505341263ca1fc2 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1
+3be618d184038dd30011d6aa8198c0be ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus
+2f31e84c01cc087318d0c7f15b6e3f47 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2
417a5b42a26275b2c912db16b9abf73a ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/fixincl
fd6f80ec9089ddf51f9cac26299e45af ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/fixinc.sh
a585abbd6a9cdc474564b54fc72e4efa ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/mkheaders
5071acceb24c0c0e8a423286205ed54c ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/mkinstalldirs
-4e77b773ac45ce8f82a4d21a34063920 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper
+5267311e0ed8bb5358e5556dfa205ca6 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper
a90ab86f837913280f72beb5310714bc ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbegin.o
0b38aa831c40b6bc4fb22248746d60d8 ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbeginS.o
8f62d8795bebd8e87caa3640522ff93b ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbeginT.o

These appear to be the same diffs that I reported before - with the
notable exception that libstdc++.a now is missing from the list of
differing files (hooray!).

Going forward, I'm not sure how best to investigate the inputs to find
out what's causing the differences. I just know I really, really,
really don't want to rebuild everything multiple times, since it takes
hours/days. If you have any creative ideas for how to speed up the
investigation, I'm all ears.

--
Chris
From e3d1778a86dfd171d59d91eb01417faaf63dfa17 Mon Sep 17 00:00:00 2001
From: Chris Marusich <cmmarusich@gmail.com>
Date: Sat, 19 Sep 2020 14:25:43 -0700
Subject: [PATCH] gnu: Disable libstdc++ in bootstrap GCC.


* gnu/packages/make-bootstrap.scm (%gcc-static) [#:configure-flags]: Add
--disable-libstdcxx to disable building the libstdc++-v3 directory.
---
gnu/packages/make-bootstrap.scm | 4 ++++
1 file changed, 4 insertions(+)

Toggle diff (17 lines)
diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index b2d3e2a326..8632d63c21 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -487,6 +487,10 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
;; Make sure gcc-nm doesn't require liblto_plugin.so.
"--disable-lto"
+ ;; In this GCC version, libstdc++.a is not reproducible:
+ ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41669
+ "--disable-libstdcxx"
+
"--disable-shared"
"--disable-plugin"
"--disable-libmudflap"
--
2.26.2
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl9u5UAACgkQ3UCaFdgi
Rp2DMQ/9GkMFoMU1/yUgSUlHgkGyFqK+SNKuURi3+f+90P2suKbfQ8qYJeTIM8KD
gDg4mmDztWvORu5otyIq8yrpSKlTTH3OgSyuxLf+EZOd9/rmA5vO9mEjAI6sQzvQ
mJXCk3Z6GVAnEiCqO7FKDapBRQVd9MIn8ObjXQbJrqGqeJg5CHxsMCVCtQz99m95
L56G2B0Ph3KbDfDqhb6AhYTyLhZ3oy+mrTzwc7aE9Ott6rWA9TJ7xH0sul6RL6gA
hwYJT6yIsvNuT+en94TYKtaPpMkEVJklNOUp34ERaggpU4eGBwszti351QjLCect
sz9O2DqU2cUlG2zCGS9x4cwmjNDFDoWvKg+VDtqmKO5C+1344qVgubcvHYjMruYf
Goaige9PmvGs9htG4cPVeKoR80Hy8l5cNyjteIGqQm9lavVP+WUYiL9TTC+jQEo4
jzkhlWxvfGLdnIT4SZbkOike/bQN4ckzwdg1d1MfLDmhOlFfnCM1rZ3qFdCwQBzn
szIqVWdaO+BUMoYin1OVhaUnTpSKSB3pDbsY6SwY8aHqcH7Kw6XRCeYHnHaP1MQn
hGowVLo0R58Fo5zghHlulqC/t/PmB4SJ3G/ZjbxTRexHNqvhq2MwQoGFH0gNP6Un
mW97TFHwvkejUCKMkytJNB+i26td9gXhU+zmbvy7JYOjTa9mlg4=
=5H4x
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 5 Oct 2020 14:33
(name . Chris Marusich)(address . cmmarusich@gmail.com)
87pn5wzwcf.fsf@gnu.org
Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

Toggle quote (10 lines)
> From e3d1778a86dfd171d59d91eb01417faaf63dfa17 Mon Sep 17 00:00:00 2001
> From: Chris Marusich <cmmarusich@gmail.com>
> Date: Sat, 19 Sep 2020 14:25:43 -0700
> Subject: [PATCH] gnu: Disable libstdc++ in bootstrap GCC.
>
> Fixes part of: <https://bugs.gnu.org/41669>.
>
> * gnu/packages/make-bootstrap.scm (%gcc-static) [#:configure-flags]: Add
> --disable-libstdcxx to disable building the libstdc++-v3 directory.

[...]

Toggle quote (4 lines)
> + ;; In this GCC version, libstdc++.a is not reproducible:
> + ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41669
> + "--disable-libstdcxx"

Does it have any effect with GCC > 4.7? My understanding is that it
builds its libstdc++ no matter what.

Also, if it’s just libstdc++.a that’s problematic (ordering issue in the
.a archive?), perhaps we can use --disable-shared?

My 2¢ (I didn’t follow the whole discussion),
Ludo’.
E
E
Efraim Flashner wrote on 5 Oct 2020 14:47
(name . Chris Marusich)(address . cmmarusich@gmail.com)
20201005124710.GA21174@E5400
Attachment: file
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl97Fc4ACgkQQarn3Mo9
g1HKChAAnk7rWppLToPQdIgAyKKvKdoVN9iJBetlXuNqumDKsZJOv5ge2vNxCdz5
Yn3IiCB6VpwL/u2gjpimgwAdjI4LwO/pB2i86S8DVdQ+M+IgQ6mIEpuO6GCVy0K0
QB+GAzvU/0otmV/XMOB9yosLvPkdA1394pEzkLfuPjCzepe0NxznP8nIAoipeYK0
q0iDT2DWWrZX8jdV8JZEwVHKF1k5KjgNQNyuc3ubyK8CTIqxWJu+wLqZjgBxiF0z
dPBDH5eJmgkyqhkiQotlWN/5cqeNuS64gybf7j15NAK+v8yDfYtDZObyIKUCI75o
cSiiOIv/lIHo9Mk893aZ6OL6oCrC5u4loqbyjiiYJdT5n8QPSX5gkaq+os13XqW0
Dw4IHm7e4SrnakqTuHjaFu9IAyj/h5PJZMC5r7EvpD7Qqk/XeSFATskJDyto7EYI
H16iHM+YgUycFc9aWnzeWGPYvNJLYhEH3qqGUiOOXZXyv0e2XwdiVpWpJ9RZO8r9
wUd+OS5hPFmmtcPvaJuFSNBhTtts3RWcSKVf5VdBM/+cfHIQMe6RWlNsl94BQxX+
cjW25lz8e6j5dwH4fv1mz7x38jzZaTkUsMglMtSf7dZBhyFMSBplAhFlv+zXElpv
lr0dO2AiD4tddZSsemCemSaP3ZltzyNiGxOGVJI2YPEPLbEscu0=
=zIrx
-----END PGP SIGNATURE-----


C
C
Chris Marusich wrote on 14 Dec 2020 00:36
(name . Ludovic Courtès)(address . ludo@gnu.org)
87pn3dth0l.fsf_-_@gmail.com
Hi,

I tried to do some experiments to see if this problem happens with the
current GCC (version 10).

I built GCC 10 (not cross-compiling) on an x86_64 system using Guix with
substitutes on Debian. (I tried without substitutes, too, but some of
the dependencies failed to be built for unrelated reasons.) I then
manually copied the /gnu/store and related files (except for the GCC 10
output paths) from Debian onto a Fedora machine, and I rebuilt GCC 10
there using Guix (again, not cross-compiling). The output on Fedora was
identical to that of Debian. Of course, the configuration Guix uses to
build GCC 10 is a bit different from the one used to (cross-)build the
powerpc64-linux bootstrap GCC, but it's still an interesting data point.
In particular, GCC 10's libstdc++.a was identical on Debian and Fedora,
so I suppose maybe they've fixed that issue in the more recent versions.

I also tried to use Guix (the current version, from master branch - I
ran guix pull today) to cross-build gcc-10 for the powerpc64-linux-gnu
target on both Debian and Fedora x86_64 systems, starting from scratch
with substitutes enabled:

guix build --target=powerpc64-linux-gnu -e '(@ (gnu packages gcc) gcc-10)'

On both Debian and Fedora, the build of gcc-10.2.0.drv failed with the
following error:

checking for -fPIC -shared... yes
configure: error:
Building GCC with plugin support requires a host that supports
-fPIC, -shared, -ldl and -rdynamic.

This basically just means that we can't cross-build gcc-10 for
powerpc64-linux-gnu out of the box on x86_64 with current Guix. I was
hoping that the builds would succeed, and I would be able to find out if
cross-building gcc-10 in this way would create non-reproducible
artifacts. I was hoping maybe I could ask for help from the GCC
community if that were the case. But since it doesn't even build, the
results of that experiment were not very useful.

It's been almost half a year now, and we're not really any closer to
figuring out why the cross-built GCC bootstrap binary is
non-reproducible. It seems counter-productive to obsess about making
this specific binary reproducible, although I wish it could be so.

What do you think about using the bootstrap binaries we built half a
year ago, and proceed with bootstrapping efforts? To be totally honest,
I'm feeling pretty exhausted by this bug, since I have spent so many
days trying to unravel it, and I haven't made any significant progress.
With no clear end in sight, I would really prefer to move on instead of
blocking the entire bootstrapping effort on this reproducibility bug.
The reproducibility of the bootstrap binaries is important, but simply
having any bootstrap binaries at all is also important. I think I have
done my due diligence to try making them reproducible. Most of them
are, but I just can't figure out why GCC isn't. I think it would be
best to proceed with the binaries we have.

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (29 lines)
> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> From e3d1778a86dfd171d59d91eb01417faaf63dfa17 Mon Sep 17 00:00:00 2001
>> From: Chris Marusich <cmmarusich@gmail.com>
>> Date: Sat, 19 Sep 2020 14:25:43 -0700
>> Subject: [PATCH] gnu: Disable libstdc++ in bootstrap GCC.
>>
>> Fixes part of: <https://bugs.gnu.org/41669>.
>>
>> * gnu/packages/make-bootstrap.scm (%gcc-static) [#:configure-flags]: Add
>> --disable-libstdcxx to disable building the libstdc++-v3 directory.
>
> [...]
>
>> + ;; In this GCC version, libstdc++.a is not reproducible:
>> + ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41669
>> + "--disable-libstdcxx"
>
> Does it have any effect with GCC > 4.7? My understanding is that it
> builds its libstdc++ no matter what.
>
> Also, if it’s just libstdc++.a that’s problematic (ordering issue in the
> .a archive?), perhaps we can use --disable-shared?
>
> My 2¢ (I didn’t follow the whole discussion),
> Ludo’.

Actually, --disable-shared is already present in the configure options.
My understanding is that libstdc++.a is a statically linked library
(perhaps I am mistaken...?), so I don't see why the presence or absence
of --disable-shared would affect it. I thought that option was just
supposed to control whether or not to build shared libraries.

Efraim Flashner <efraim@flashner.co.il> writes:

Toggle quote (8 lines)
> On Fri, Sep 25, 2020 at 11:52:48PM -0700, Chris Marusich wrote:
>> Hi everyone,
>>
>> Efraim Flashner <efraim@flashner.co.il> writes:
>>
>> > Is this a file we actually need during the bootstrap process? Can we
>> > "work around it" by just deleting it?

I've spent all of my spare Guix time trying to debug this
reproducibility issue first, and half a year has passed without progress
as a result. I think we should use the bootstrap binaries we built half
a year ago, and move on with life.

At this point, it might even make more sense to try bootstrapping for
powerpc64le instead of powerpc64, since the rest of the world seems to
be gravitating toward the little-endian variant on POWER9 hardware, and
thus various programs out there are more likely to be better tested on
powerpc64le than powerpc64.

In any case, I don't think we should wait any longer.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl/WpZsACgkQ3UCaFdgi
Rp0gVhAAjuI5BRuZXxOJ27NV8ss5267kHg/PYGoiq+zU25h9baMLOce8nQhHptlc
IzMSD295le/3RofgCfXBVDIcorwmeVzFEKfCriaTYyM+yC0sbeMWyjGf30HzMKgY
Gpy39xY/4CB2GZznsF2bxBv6RRKmNdOeagg2ns7AgFpo8fzuA4gv/+YxNi2C/ZcK
hfdfjWmNQztAl10e0At4Kg3bSwL7S/w0cPLR3B3FfKh9lc14UNri2XlnBIFn1TDX
BWQKmPL8XqlZgYpG1nlNuL1XnUptTgieoC1Waxslu4bJgFMmBTCPSqvqlM6mD8tR
EPMOTKJtAjOOe8iDjYpz6PlfQgewJ7Mwp3wK5nPNmr+ynEn457YO3ws/J9hCWeAX
vGER2mJlf1cH06hs6mIwT7yP+NyZIGSMVwbFfU1/4lqk6oZ7ASXIPLRjhLb1hDto
a6/rCvrjAi8Lu2Hk+0RvqRf5fsmMOp5sUWtbpDqIQbOKMROGtg5x4/Kc3esg/RO5
OddvzIeOSYw6SMWFfi8Ql98Cbdw1hkJ6ALP9GG90lEq38zgboItVncVLRFV8p689
JQqTIYBgdIOdLSXit3WodLtEyHZ86ZorxXDFKDZ9iUPujOLDBicss+x3CoD50HU0
X5CH6e5+5lh2cg2IZLFP9et6xzjLetkiEITbASYoJ+ngCMxOQ3Q=
=2Fs0
-----END PGP SIGNATURE-----

E
E
Efraim Flashner wrote on 14 Dec 2020 09:17
(name . Chris Marusich)(address . cmmarusich@gmail.com)
X9cfkfqlZZ7/5y8Y@E5400
Attachment: file
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl/XH40ACgkQQarn3Mo9
g1F2OQ/+IhXOhx+T/vTIGuINNrtMsoKqdHdGDlzxZOpaLVp7mp/vb/O8S5EPsHD+
yryb8CuSo5FktYZ/qbRHLIO9TgKKzZe08nklB1ZZQTP1ZUFwwrHvBIbntWk1JQUm
s8/slZwZyZBOp/NjjM5+7ypBiW7ml2pA1vLVHvyrWoDCc2n5r1p1T0t2rwwUSbRO
zAzTzArMVHyWWeugzbB8MsjMLzSJquyZsyznB4yY31s+e8Kv6+pL+W+3u80bcfp0
h9BWS2fqvtck1vjVQu6T1KNyqbgGJicjqwpt+y6vJpIhxR7Lltg346saJxqmzOHy
zTKHFZJBkmKF9N3PSGlif2DHyuK/QtXT1GxMcCgvoThrzt+mx9JNm6MP8eVWGu5f
Typ/3ncskStf+ybf2n+vmV4FDMDwT9KhB+N3W1zo54jSC9hZ3LWHXfK11jnzhhyb
L50SbyokTVwqhGZOrtvgh3el9Zgzg0+scJikjvHYIoJppX8m3M+QX6vCwhXFqpFa
+UuZobypZwr2G+AGZk4EiUWUEDSdfXaFtnmOfXHMXj9D0Uyl/i4+DXDKNOMnburk
XD6HlRv5C+Ncb/c5AuOjZXtjt1vL2+929+Jt1T+4DIEqMQxBJe8RQdsRtGOTOWLT
Twn0lmkK/miuls1sMKmYqzkHmshvn8gG1ynvQr6TuCYkFR/lk2w=
=6SXH
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 14 Dec 2020 09:36
(name . Chris Marusich)(address . cmmarusich@gmail.com)
874kkoyebq.fsf@gnu.org
Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

Toggle quote (17 lines)
> It's been almost half a year now, and we're not really any closer to
> figuring out why the cross-built GCC bootstrap binary is
> non-reproducible. It seems counter-productive to obsess about making
> this specific binary reproducible, although I wish it could be so.
>
> What do you think about using the bootstrap binaries we built half a
> year ago, and proceed with bootstrapping efforts? To be totally honest,
> I'm feeling pretty exhausted by this bug, since I have spent so many
> days trying to unravel it, and I haven't made any significant progress.
> With no clear end in sight, I would really prefer to move on instead of
> blocking the entire bootstrapping effort on this reproducibility bug.
> The reproducibility of the bootstrap binaries is important, but simply
> having any bootstrap binaries at all is also important. I think I have
> done my due diligence to try making them reproducible. Most of them
> are, but I just can't figure out why GCC isn't. I think it would be
> best to proceed with the binaries we have.

I didn’t follow the whole discussion nor did I try to investigate
myself, but thanks a lot for going to great lengths trying to identify
the issue; this is an impressive amount of work, and I can only share
your disappointment.

Given this effort, I agree that it may be best at this point to move on
and start with these non-reproducible binaries. At least, the problem
is now documented.

Toggle quote (6 lines)
> At this point, it might even make more sense to try bootstrapping for
> powerpc64le instead of powerpc64, since the rest of the world seems to
> be gravitating toward the little-endian variant on POWER9 hardware, and
> thus various programs out there are more likely to be better tested on
> powerpc64le than powerpc64.

Yes, my understanding is that other people, in particular Tobias Platen
and dftxbs3e, were looking at powerpc64le, so perhaps it’s a good idea
to concentrate on that one?

Anyhow, please let me know if/when bootstrap binaries should be uploaded
to ftp.gnu.org (with a signed message). When updating bootstrap.scm to
refer to them, please include the commit ID used to build them in the
commit message.

Thanks,
Ludo’.
L
L
Leo Le Bouter wrote on 14 Dec 2020 10:22
bc239fe7c4eebfa0434be5c98520d3dc9313b367.camel@zaclys.net
Hello Chris, Ludo and Efraim,

In my experience, the bootstrap binaries are reproducible on
powerpc64le-linux-gnu for a specific GNU Guix System and GNU Guix
version and configuration. It's not perfect that the kernel version has
to be pinned for reproducibility but it's better than nothing.

The issue with powerpc64le-linux-gnu on GNU Guix core-updates now is
that it has been upgraded to Glibc 2.32 and there is other issues due
to that. Glibc 2.31 is otherwise working well. I wish we could push
changes to build bootstrap binaries to master where there is Glibc 2.31
(and ensure the changes don't affect other architectures) and get this
over with.

Leo
E
E
Efraim Flashner wrote on 14 Dec 2020 11:27
(name . Leo Le Bouter)(address . lle-bout@zaclys.net)
X9c+AvdN+/xH9A8x@E5400
On Mon, Dec 14, 2020 at 10:22:43AM +0100, Leo Le Bouter wrote:
Toggle quote (14 lines)
> Hello Chris, Ludo and Efraim,
>
> In my experience, the bootstrap binaries are reproducible on
> powerpc64le-linux-gnu for a specific GNU Guix System and GNU Guix
> version and configuration. It's not perfect that the kernel version has
> to be pinned for reproducibility but it's better than nothing.
>
> The issue with powerpc64le-linux-gnu on GNU Guix core-updates now is
> that it has been upgraded to Glibc 2.32 and there is other issues due
> to that. Glibc 2.31 is otherwise working well. I wish we could push
> changes to build bootstrap binaries to master where there is Glibc 2.31
> (and ensure the changes don't affect other architectures) and get this
> over with.

It is possible to create a "perfect setup" by editing the package
definitions in gnu/packages/bootstrap to fix certain issues which are
needed to make bootstrap binaries actually work. I suppose it would be
possible to downgrade glibc in bootstrap.scm, but I wouldn't count on
maintaining a separate glibc for powerpc64le vs the rest of the
architectures.

Do you have a preference big-endian vs little endian?

--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl/XPf8ACgkQQarn3Mo9
g1EqZA//al6eVmCrax75nQPknTyF9/acmLq9HZ2j0WNBc20SbZbElQ40eCK1I2BV
qRfnFlwm5hftKXbQho55qndPDfU9eSvba5+/Pw+SM3jjs5pxhx/mHEZ+OAHfwsFi
GQfNfXZFsRzjGyKN9TgtwcI4jtZ6y9GK7vJ8hR0jn/LXaMT8WbzsX7HfOyn7OhA5
D4fxLF2I9jcsbHbuP7Ky5S2VsulOTHTLUIVHJGdtZ5/d8zRZesFWPVS2On0yACZB
GFIuOovDW9ZWR6ziy+7EZl0CP4+2ALplk2UD9nUTxet1KdiIBnVTkOAly0U2mvTq
bOiuKT/Ve/igMsW7KgXaEXYurR3qbU0TROdPcM87QL9Pfk5+9+wN9Huz+JvMFh0I
8yVU19RY4bCV0EcYInzlng5/pN8xKwGfD5ytHaJ9FrUf10Prs0WU1jFBkPmC2Mve
ZenMKpSr4r6KrLt9B6y57qzEu43lAmKMi9/RRYFLdXpD7J9GRrjX1taLHF2Ll+fn
jz5u0p6WltgnoxPa7s+OvhPI3dzyiOsGsCNk5M/yZXV2WQtoGxXSr4Fk/wcAJlD/
BpRfFvalubwNcTjaCM9BfUAtF0OFtQ25r3wIOO/CpRZ183JSTN+e1THo5wOan9DC
PFspJ9YMDOoxFkEpwdJoqHiyc1fzHXmEbyjJXOaVBcbu0ThGNFM=
=feIi
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 14 Dec 2020 11:34
(name . Efraim Flashner)(address . efraim@flashner.co.il)
c0fcb13260c693139e121a24ee1bcc7360dfd83f.camel@zaclys.net
On Mon, 2020-12-14 at 12:27 +0200, Efraim Flashner wrote:

Toggle quote (4 lines)
> but I wouldn't count on
> maintaining a separate glibc for powerpc64le vs the rest of the
> architectures.

It doesnt need to be maintained, it only needs to work in one commit on
master and then one uses time-machine to rebuild the bootstrap binaries
if they wish to. The make-bootstrap code is already unmaintained for
every architecture anyway since we never rebuild bootstrap binaries
using later GNU Guix revisions ever.

Toggle quote (2 lines)
> Do you have a preference big-endian vs little endian?

I'd like both but little endian has the widest eco-system support
especially w.r.t. to Linux drivers. Many Linux drivers have endianness
bugs (lack of endian-safe serialization for DMA..), it's such a plague
that sticking to little endian is just better right now. One common
example being mpt3sas and amdgpu drivers required in some
configurations of the Talos II system.
E
E
Efraim Flashner wrote on 14 Dec 2020 11:38
(name . Leo Le Bouter)(address . lle-bout@zaclys.net)
X9dArDmqqsFyIAG7@E5400
On Mon, Dec 14, 2020 at 11:34:35AM +0100, Leo Le Bouter wrote:
Toggle quote (12 lines)
> On Mon, 2020-12-14 at 12:27 +0200, Efraim Flashner wrote:
>
> > but I wouldn't count on
> > maintaining a separate glibc for powerpc64le vs the rest of the
> > architectures.
>
> It doesnt need to be maintained, it only needs to work in one commit on
> master and then one uses time-machine to rebuild the bootstrap binaries
> if they wish to. The make-bootstrap code is already unmaintained for
> every architecture anyway since we never rebuild bootstrap binaries
> using later GNU Guix revisions ever.

It looks like I mispoke, I meant gnu/packages/make-bootstrap.scm. If we
change glibc-for-bootstrap to inherit glibc-2.31 then the rest of the
bootstrap binaries should use that one and everything else will use the
regular glibc.

Toggle quote (11 lines)
>
> > Do you have a preference big-endian vs little endian?
>
> I'd like both but little endian has the widest eco-system support
> especially w.r.t. to Linux drivers. Many Linux drivers have endianness
> bugs (lack of endian-safe serialization for DMA..), it's such a plague
> that sticking to little endian is just better right now. One common
> example being mpt3sas and amdgpu drivers required in some
> configurations of the Talos II system.
>

I remember you mentioning that.

--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl/XQKoACgkQQarn3Mo9
g1Gv/Q//fQK+B4hrAZgwOvlWIm6/EUVPZj7LJKTvZIaHXCYJDDZgdEA93Dh5kUvY
esua7YJphlUp+PJ4/LbzdmviHcycgHZS25ZtVJfo0g6ghbdhKQPioEGshQQ9Q1jO
UAmWvHGw/j+ck3M0W+uUCBlzxtV3QNW6EzXJ75e3gU5HSaZ7qCy5PK392N0ZQsh2
zDhmdJ/r7aWn6WXGVvMlKjub8JFypgHm6yqmT2evHceXXnSjNAI2HDiiVyK8U6Ia
umwjGjNPSGX00+YxJl1S8Fm0YxUORGglhO1rFUH+mYze7LeJ3UBTbDMNeCs5IsQ/
/uGgy777kfJRHFUkx9JzDVyboSLd+2COoT894gAjYJBP1uOCQVwkL7Rfo+QapHlZ
l16r/ze/CrgbZ2O9TqIDv5COgC5Wvc72OOC4s2sbmdGyUYpT7ltGouHmXpl6+0Ck
DjotXNTACXPhgKzK2/A+Hlu4tben1XXXquctmm9/ek0ouTgKe/5A1/877nE/l1yL
94tVf5149Bf52Gyo2n0znBsjMA72HOZBCAEi8ubSj0B1MzmSrWz7FZEgVtAOKdYy
b/lGQUNKYtMbcvyJJA46ShRWAPYistZ8Ud1+UejIFGLHkNz79SpsK60MbIh1D10i
BrXo9SZa8xVdNy6tRjoJvp5OI1HbV1sIkCLSlLKjFOcp9IlnzDs=
=RfOP
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 14 Dec 2020 11:44
(name . Efraim Flashner)(address . efraim@flashner.co.il)
7730dab2af9de1e47cf343ff5ed506d472efe591.camel@zaclys.net
On Mon, 2020-12-14 at 12:38 +0200, Efraim Flashner wrote:
Toggle quote (7 lines)
> It looks like I mispoke, I meant gnu/packages/make-bootstrap.scm. If
> we
> change glibc-for-bootstrap to inherit glibc-2.31 then the rest of the
> bootstrap binaries should use that one and everything else will use
> the
> regular glibc.

It would be great to do that at least temporarily on master. It will
not affect much since other architectures are bootstrapped already. We
could also make it conditional. Reproducibility instructions will have
to contain exact commit id and configuration for both GNU Guix System
(x86_64-linux) and GNU Guix which can cross-compile.
L
L
Ludovic Courtès wrote on 14 Dec 2020 23:24
(name . Leo Le Bouter)(address . lle-bout@zaclys.net)
87a6ugqb4h.fsf@gnu.org
Hi Leo,

Leo Le Bouter <lle-bout@zaclys.net> skribis:

Toggle quote (6 lines)
> It would be great to do that at least temporarily on master. It will
> not affect much since other architectures are bootstrapped already. We
> could also make it conditional. Reproducibility instructions will have
> to contain exact commit id and configuration for both GNU Guix System
> (x86_64-linux) and GNU Guix which can cross-compile.

Yes, I think we could just cross-build those bootstrap binaries from
current ‘master’ and be done with it, if it works for you.

Thanks,
Ludo’.
L
L
Leo Le Bouter wrote on 15 Dec 2020 08:34
(name . Ludovic Courtès)(address . ludo@gnu.org)
54153897349cc396d89a022dbce141bfea4804db.camel@zaclys.net
On Mon, 2020-12-14 at 23:24 +0100, Ludovic Courtès wrote:
Toggle quote (3 lines)
> Yes, I think we could just cross-build those bootstrap binaries from
> current ‘master’ and be done with it, if it works for you.

Yes that would be awesome! The master branch just needs the patch from
considering it affects lots of packages it needs to be rewritten the
same way Efraim has rewritten their PowerPC 32-bits patch. I'll look
into that. Correct?
L
L
Leo Le Bouter wrote on 15 Dec 2020 08:46
(name . Ludovic Courtès)(address . ludo@gnu.org)
7237255c50a761ceda8d76c583309d3dd59039db.camel@zaclys.net
For the record, I attached the failed build log for current core-
updates.

$ ./pre-inst-env guix describe
Git checkout:
repository: /home/lle-bout/src/guix
branch: core-updates
commit: cc6cb6e80a42355147809b4830053a34d1563994

Build command:

$ ./pre-inst-env guix build --target=powerpc64le-linux-gnu bootstrap-
tarballs

The most important bit in the build log is:

configure: error: *** The compiler must support -mabi=ieeelongdouble
and -mlongdouble simultaneously.

This particular error does not happen with the GCC 7.x and Glibc 2.31
combo.
Attachment: file
L
L
Leo Le Bouter wrote on 15 Dec 2020 10:35
(name . Ludovic Courtès)(address . ludo@gnu.org)
7dd3dc784839b0b456af8cf34ab5107a27ef0a68.camel@zaclys.net
On Tue, 2020-12-15 at 08:34 +0100, Leo Le Bouter wrote:
Toggle quote (12 lines)
> On Mon, 2020-12-14 at 23:24 +0100, Ludovic Courtès wrote:
> > Yes, I think we could just cross-build those bootstrap binaries
> > from
> > current ‘master’ and be done with it, if it works for you.
>
> Yes that would be awesome! The master branch just needs the patch
> from
> bug <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=44778> - I guess
> considering it affects lots of packages it needs to be rewritten the
> same way Efraim has rewritten their PowerPC 32-bits patch. I'll look
> into that. Correct?

Rewrote patch and submitted here: <

Thanks
C
C
Chris Marusich wrote on 20 Dec 2020 08:28
87blepaqd8.fsf_-_@gmail.com
Hi all,

It's great to see forward motion again!

Efraim Flashner <efraim@flashner.co.il> writes:

Toggle quote (11 lines)
> As far as powerpc64 vs powerpc64le, I'll let those with the hardware
> have more of a say, they'll be the ones using it. As far as the
> bootstrap binaries go, I don't remember having this much pushback with
> my binaries for aarch64 (just a request to rebuild with guile-2.0.14
> since it was reproducible), and I'm not sure how much Janneke had with
> the Hurd binaries but I don't think it was this much. The ultimate goal
> anyway is to replace them with artisanally crafted mes binaries, and I
> understand we want to have them as reproducible as possible, but I don't
> think it's fair to keep this architecture out when we've let other ones
> in with similar reproducible problems.

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (9 lines)
> I didn’t follow the whole discussion nor did I try to investigate
> myself, but thanks a lot for going to great lengths trying to identify
> the issue; this is an impressive amount of work, and I can only share
> your disappointment.
>
> Given this effort, I agree that it may be best at this point to move on
> and start with these non-reproducible binaries. At least, the problem
> is now documented.

I'm glad you agree. For powerpc64 (big-endian), should we just make the
changes for bootstrapping on the master branch, or should we use a
separate branch?

If we use a separate branch, we could use the name "wip-ppc64", since
there is already a "wip-ppc64le" for powerpc64le. Is it expected that
commits on these "wip" branches will never be modified (e.g., via
rebasing)?

Leo Le Bouter <lle-bout@zaclys.net> writes:

Toggle quote (9 lines)
>> Do you have a preference big-endian vs little endian?
>
> I'd like both but little endian has the widest eco-system support
> especially w.r.t. to Linux drivers. Many Linux drivers have endianness
> bugs (lack of endian-safe serialization for DMA..), it's such a plague
> that sticking to little endian is just better right now. One common
> example being mpt3sas and amdgpu drivers required in some
> configurations of the Talos II system.

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (10 lines)
>> At this point, it might even make more sense to try bootstrapping for
>> powerpc64le instead of powerpc64, since the rest of the world seems to
>> be gravitating toward the little-endian variant on POWER9 hardware, and
>> thus various programs out there are more likely to be better tested on
>> powerpc64le than powerpc64.
>
> Yes, my understanding is that other people, in particular Tobias Platen
> and dftxbs3e, were looking at powerpc64le, so perhaps it’s a good idea
> to concentrate on that one?

I agree. It's probably better to focus on little-endian. However, it
isn't clear yet which will be ultimately harder for us to bootstrap, so
I also think it's fine to work on both in parallel and see how it goes.

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (5 lines)
> Anyhow, please let me know if/when bootstrap binaries should be uploaded
> to ftp.gnu.org (with a signed message). When updating bootstrap.scm to
> refer to them, please include the commit ID used to build them in the
> commit message.

Over the last few days, Leo and I coordinated to try cross-compiling the
powerpc64 bootstrap tarballs one more time, using two Guix System VMs.
We did this because we thought that maybe if we took care to keep the
Guix Systems "the same" (e.g., same kernel), it would produce identical
results. Instead, the result was the same as before: all bootstrap
tarballs except for gcc-static were identical, and gcc-static differed
in ways similar to what has already been described earlier in this bug
report. In fact, with the exception of gcc-static, the bootstrap
tarballs were identical to the tarballs multiple people built 6 months
ago in June. This means that (1) with the exception of gcc-static, the
bootstrap tarballs build reproducibly even across systems and after
quite a bit of change has taken place on the master branch, and (2) even
when built from source on two separate, fresh, practically identical
Guix System machines, without using substitutes, the gcc-static binary
still differs.

Now that we have decided to use these powerpc64 bootstrap tarballs, what
are the next steps for uploading them to the GNU FTP server? I've never
done that before, and I don't think I have access. For now I've put a
signed copy of the powerpc64-linux (big endian) bootstrap tarballs, with
a SHA-512 hash, here:


For the record, these bootstrap tarballs were built via the following
steps:

- Use
to install Guix System 1.2.0 on an x86_64-linux machine.
- Run: guix pull --no-substitutes --commit=1ced8379c7641788fa607b19b7a66d18f045362b
- Run: guix build --no-substitutes --target=powerpc64-linux-gnu bootstrap-tarballs
- I didn't run "guix system reconfigure" after installing Guix System;
theoretically it shouldn't matter, but for the purpose of our
experiment, I just left the system in its default configuration in
order to ensure that the kernel etc. would be the same on both VMs.

Once we get these binaries into the GNU FTP server, I'll get started on
updating gnu/packages/bootstrap.scm and other files necessary to begin
bootstrapping powerpc64-linux. I'll mainly be adapting the work that
Leo already did, and following the lead of others like Efraim and his
work on the wip-ppc branch. I will probably have questions as I go, so
I'll ask on guix-devel.

Please let me know if you'd like this done on a special branch.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl/e/QMACgkQ3UCaFdgi
Rp0HtRAAjn883AZMDTqduot7bySWoCfuRQHFqmiA8+1i4RmIcD56E6XuCnL/YIWI
LZF0QVqyMaZdPCtf2ZjNyeZoQi0u//y33fVd1Ju+c/rynlMqysLh4Ka+OTLnQVk8
t7sGSK/JfpvKmpAFkLJpZd1q/z/hDasZklupl8CLBYDnZpKZeruvpzRn8NwyMux9
6iJ2bYtXRkvTLk39DFozWYT+9sSsy6cmz9SWtaz09g5aNE2mlscMxVQ0JuIy2v/D
EdVa286sEt481KqpoiAPJ08Bthtx2ZUmiRzR1vdaiN+WsckAWmJ8FJC5/Me+FkHB
xGRi025RFm9s863DSi9qf0NdzSApeJ3ixHErd2rR2nFxNgBTH9PRPqUfXdQ4KBFd
ja+2op3hKx9Y9NLlg8fWdI9Xbue1qLQMDmWxrSyKSzcvzn34IRgg9OK2BOg20d3W
LLYCfwO7p0f8Ozf+U7lE1nN9GDAhVqx3jNWnRmtKMN0jaFcjwECkNoiYcQfLFgo/
MVcETGCvwNtQYAsOAFwda3aJKQ8z8tjN+H6AGxzJOlB6qGmEq15adshG0b/EFuC6
4QuftjdxAkDlwl7/hPXVoH/vwX+hHwq77r1f0fys52hpNzjipnqsNRu9HjVcbZ4A
jC9dsZa5j3RHXy//lRMWBpHN8Y1t4p8dL3C4msj5pxmNiefWLQY=
=Ai5P
-----END PGP SIGNATURE-----

L
L
Leo Le Bouter wrote on 28 Dec 2020 03:25
972680c699cf9fce5ad6e609228aa6715dff85ff.camel@zaclys.net
Hello!

I want to have one last attempt at making the binaries reproducible.

Could anyone help adjusting this patch so the package definition's hash
does not change on other architectures? So it can be proposed for merge
in master..

Thank you
From e6931a7ebb9cc0681a3211ac38a1c58c7a176481 Mon Sep 17 00:00:00 2001
From: John Doe <dftxbs3e@free.fr>
Date: Mon, 28 Dec 2020 03:21:08 +0100
Subject: [PATCH] gnu: gcc-4.7: Disable parallel compilation on powerpc64*.

* gnu/packages/gcc.scm (gcc-4.7)[arguments]: Conditionally disable
parallel compilation on powerpc64*.
---
gnu/packages/gcc.scm | 2 ++
1 file changed, 2 insertions(+)

Toggle diff (15 lines)
diff --git a/gnu/packages/gcc.scm b/gnu/packages/gcc.scm
index 4d5aaa7070..6d32677144 100644
--- a/gnu/packages/gcc.scm
+++ b/gnu/packages/gcc.scm
@@ -204,6 +204,8 @@ where the OS part is overloaded to denote a specific ABI---into GCC
,(if stripped? "-g0" "-g")))))
#:tests? #f
+ #:parallel-build? ,(string-prefix? "powerpc64" (or (%current-target-system)
+ (%current-system)))
#:phases
(modify-phases %standard-phases
--
2.29.2
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAl/pQhcACgkQRaix6GvN
EKZ5DBAAvvwVVRAH5Vzc3AU0RLSTrrNhb4tdh/KSHoIy0SHBt1yFoRWh8VJMujvy
90L4K0B4VPhfaJg1KUC7EOkS0NvHuBwsQt+v0q3nf8IWenIfGu33CmBNrcjfCi+7
NViN87nNbHCY3O77LTlpcIkbHEr8IKTiD7FFMaYEoVyc3IgNd72YntTmIGta0fWn
DHv3j+NDCIQVP69zh7pckB08PWI7p4SLE6FSpqPqcPUYsmU++k5lmiBpdlv/As0b
P45EBLW/8+sN9Y68e9EAZR2A3A0CmKjROdx3XAwpJBlmcdwT68ameJqJU+tZII7t
F1x40g/jvNiqAvkuYHheM3GOoM/Q/uorEal5gEckNUt+53fPiJFrvTChQEkoaIWr
kq4RzdYtQkGwIuRbH2F8Cg3FySUvLg4aRcmKfec1AFFCdQsGdqD2KIGHYyB5BKHl
L1VAsHlhQhrqW2HJJiyFAF/kD2VIFqS9MlmYIJv+BCaFwYThpWguhBfHISIbQ6n7
f9vUJPGQ5sev0gZK6DBaQ20Vh0Evpk71tLVM3dZ6mibUy4p5Ehb0obE2k0aEbAj7
jDQTUXodzzOXV80eq96vHjigUBkQtzTw2dMWNSrwNljJXvxbtrW57Yf+KPexy/P1
NHwiM5nB48iDbn6wQAJl25uunUdCjbOsc9yCDQ+8bX2iojdxkTA=
=/qYB
-----END PGP SIGNATURE-----


C
C
Chris Marusich wrote on 28 Dec 2020 05:23
(name . Leo Le Bouter)(address . lle-bout@zaclys.net)
87zh1y1ruu.fsf@gmail.com
Leo Le Bouter <lle-bout@zaclys.net> writes:

Toggle quote (33 lines)
> I want to have one last attempt at making the binaries reproducible.
>
> Could anyone help adjusting this patch so the package definition's hash
> does not change on other architectures? So it can be proposed for merge
> in master..
>
> Thank you
>
> From e6931a7ebb9cc0681a3211ac38a1c58c7a176481 Mon Sep 17 00:00:00 2001
> From: John Doe <dftxbs3e@free.fr>
> Date: Mon, 28 Dec 2020 03:21:08 +0100
> Subject: [PATCH] gnu: gcc-4.7: Disable parallel compilation on powerpc64*.
>
> * gnu/packages/gcc.scm (gcc-4.7)[arguments]: Conditionally disable
> parallel compilation on powerpc64*.
> ---
> gnu/packages/gcc.scm | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/gnu/packages/gcc.scm b/gnu/packages/gcc.scm
> index 4d5aaa7070..6d32677144 100644
> --- a/gnu/packages/gcc.scm
> +++ b/gnu/packages/gcc.scm
> @@ -204,6 +204,8 @@ where the OS part is overloaded to denote a specific ABI---into GCC
> ,(if stripped? "-g0" "-g")))))
>
> #:tests? #f
> + #:parallel-build? ,(string-prefix? "powerpc64" (or (%current-target-system)
> + (%current-system)))
>
> #:phases
> (modify-phases %standard-phases

If it's just for the sake of trying one last time, we could just add
--cores=1 to the Guix invocations, or run everything in a single-core
VM. Wouldn't that have the same effect?

I think you'll probably agree, so I've proactively started another build
on two fresh single-core VMs (using the same procedure I described
earlier, starting from the 1.2.0 installation ISO image). It'll take a
few days to finish, I'm sure. Please let me know if you think we need
the patch to run this final experiment. Otherwise, I'll just report the
results of this latest experiment in a few days' time.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl/pXbkACgkQ3UCaFdgi
Rp1oqRAAt3SeMcvhgdz7wzd3FPMd5L2tI+fdw9jS9cOdS9YYX82rpFYRhDXmjzlg
T6doiaQaYi4gy8gYzfHWYgswz2LD7pRHI275erdBRbmEVl6BhtvXjpXhnD1pfF7N
+kK7Z+EME0zppdH8/4slWIfYQms+PbgR52aBTKtIhjQ+kcErbAxkegwJIHZSvJvw
jZMSwdB0OtdhkRR5Z1s3Z8I87CNI9dPOkJGlMNQPdI2VHl1nSDSEbM9YmPo460sz
tltJFRp5lEh8lm97YZmGG9wMn5RXHignwAq3AKGucZwrp0XH46ElKWs2uuWbyqve
wwO9a67JXeTjzqt5FwwGGsVS9Q6vVz9PQHXl/rwvKoYx8iCR5fHs34DtbADGFfb4
pTnYhCLJs7+yIDbAMQ1nkRMkdxu9IXJ4gHyAFkDRBpGuBCCv5mflaJPbugm2XeZZ
UWER09AlvEnGucOLns6ONrGrfO+Tpna8yP+O5mP38qI0DsMeVQ0jxct7y84Ts55L
NZ1Jqepm30iFqjso3VydbdK+S7G+gXEAnarvEE71CLf2OF9hdUtmv+atKUyoCwZn
yjkHtF1wT0S8zOYuTNxmHqWpQF+Rbzamzyb2ttHh/e8w3EFouPa4nu2XOXBcSDsR
Y+jyu51kjAWfe5lNHDoFHgQcMeCn7tGTZS8Q51yhWJ8eLZAOf0o=
=emb+
-----END PGP SIGNATURE-----

E
E
Efraim Flashner wrote on 28 Dec 2020 09:07
(name . Chris Marusich)(address . cmmarusich@gmail.com)
X+mSMQYRD+7hozMi@3900XT
On Sun, Dec 27, 2020 at 08:23:21PM -0800, Chris Marusich wrote:
Toggle quote (39 lines)
> Leo Le Bouter <lle-bout@zaclys.net> writes:
>
> > I want to have one last attempt at making the binaries reproducible.
> >
> > Could anyone help adjusting this patch so the package definition's hash
> > does not change on other architectures? So it can be proposed for merge
> > in master..
> >
> > Thank you
> >
> > From e6931a7ebb9cc0681a3211ac38a1c58c7a176481 Mon Sep 17 00:00:00 2001
> > From: John Doe <dftxbs3e@free.fr>
> > Date: Mon, 28 Dec 2020 03:21:08 +0100
> > Subject: [PATCH] gnu: gcc-4.7: Disable parallel compilation on powerpc64*.
> >
> > * gnu/packages/gcc.scm (gcc-4.7)[arguments]: Conditionally disable
> > parallel compilation on powerpc64*.
> > ---
> > gnu/packages/gcc.scm | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/gnu/packages/gcc.scm b/gnu/packages/gcc.scm
> > index 4d5aaa7070..6d32677144 100644
> > --- a/gnu/packages/gcc.scm
> > +++ b/gnu/packages/gcc.scm
> > @@ -204,6 +204,8 @@ where the OS part is overloaded to denote a specific ABI---into GCC
> > ,(if stripped? "-g0" "-g")))))
> >
> > #:tests? #f
> > + #:parallel-build? ,(string-prefix? "powerpc64" (or (%current-target-system)
> > + (%current-system)))
> >
> > #:phases
> > (modify-phases %standard-phases
>
> If it's just for the sake of trying one last time, we could just add
> --cores=1 to the Guix invocations, or run everything in a single-core
> VM. Wouldn't that have the same effect?

Close enough. We can also add it into the commit message, to build it
with --cores=1

Toggle quote (12 lines)
> I think you'll probably agree, so I've proactively started another build
> on two fresh single-core VMs (using the same procedure I described
> earlier, starting from the 1.2.0 installation ISO image). It'll take a
> few days to finish, I'm sure. Please let me know if you think we need
> the patch to run this final experiment. Otherwise, I'll just report the
> results of this latest experiment in a few days' time.
>
> --
> Chris



--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl/pkioACgkQQarn3Mo9
g1FSPhAAmT4SylnLS1yvrqGtc9ieBE7CcspOnS2jPxkOkn1QWQDXsS2u6unP5spE
kNk7P5CSvmiFL6r3dSOtr6UoE+IoXouYTRp7NVvmlG7hFWcd5IW2oBYrHyVh/LTe
cPPhECdMGTm3w2VPen+FU3DoPUofO/i9D30JGMSgbgmwRszk8DS70yJ6Ir5tePPS
s5zUh67NUdp2IuoWNiWGVL1ub47P9WFT3MdbrTfA6ghxDw/2wuDwERa4pPNf4H0n
j7Yn9ane2k5fw7Nncex1C2vdC7Xg57h03z8HwvHs9zjNad4k0Fl0UVvOHq63+9bu
MfRtb/ZCXKIAjbjp2ozOBPxUwV/3XZ4fL8ezKr5zz1+Ya8xL2LEFXXK43UeMZOB9
HyUE5Cxhu2SD01U21Dv/GX8yUFX+DmXf5pq/tR0YqIlH+y0FUXo/e6bJtH14kvye
9TMukES/cHKpCByiCSqD8+LlvHmYgNFFDgLiMCGZ41TNrknmb7+cYpolTFUjM/SM
EwfeYxrWVFu9LMe2ZGdCSjJRLisoxbJZGT7tS6OZVvKlftj4I5gXqRn/wY7bOY3S
d7NkXXmuPmqMv0t/PBNU/Ndv0GxTXWCgQLK4xW9ZXh4M1ytCrQD3vnXI60RDKshj
aWuytLsSOqWTuSveArv3wziR16G80ta+r0LLGS8OgtcOv5Y6VZQ=
=JxnR
-----END PGP SIGNATURE-----


E
E
Efraim Flashner wrote on 28 Dec 2020 09:07
(name . Leo Le Bouter)(address . lle-bout@zaclys.net)
X+mSXrDa+FwXl+QQ@3900XT
On Mon, Dec 28, 2020 at 03:25:27AM +0100, Leo Le Bouter wrote:
Toggle quote (10 lines)
> Hello!
>
> I want to have one last attempt at making the binaries reproducible.
>
> Could anyone help adjusting this patch so the package definition's hash
> does not change on other architectures? So it can be proposed for merge
> in master..
>
> Thank you

I think you'll find the proposed patch does exactly the opposite of what
you want :P





--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl/pkl4ACgkQQarn3Mo9
g1EqshAArsx5l2nY1P4PTKMIAyNa4Z6xRNo6QI/w+RO2N7YOuNQIlCouWjOe7ozh
t1FmjrASVU381BNKiWpsLJqrCTZlhxK1K3qRy106XPmjo6Q7nDdZrrN550TCmnv/
U8oow9qU7WWo8mbSqeo5GO+b6t0WsYe1vJGXnm6VH70k9cvrG/0BWmBlKeHp171D
H2/FNIJKI3byNwOIlbJ/9yLSCzn6vWL6YiZT2ic+xeKyUiL41QXzL0r2SsG2rcOz
Pnhre7vLkgSO3+cJV/TGiplqKk1qSmGZ5uUWfB9NQolRlg6aFFciHRTTPmr8Qz/h
Vi1CY0AXVkqP+LbRusPyiz0l7HdTRRwtzuTPetvQrHRScRfhmXr/SoFlNtHlh5KM
4+ROvryTQX3LydcWI495vCTWeC3t9jpaqHVK9iRbmEL3FtGxcyQUPzfqdwTJ1Uet
E7OlNSmo2OC8A270RoMN84m14SBP4BuAhxHIsWFUgWmCqDrteP/wLlsxVMnB6YCH
tVF0sQG+OMxXAJF/OdwaY74ILKhdBM8HzzYUKnzFAR96m6E8iVpKLfSng4GbXR96
a8iWQB8Rux8HZhH60c640k1zpmCbLNS/ptymFQtvUILXWHziLJpxCsS0kU3lv6QE
XRWFhgrvq4mTAfhLiat9VcpEJ/VUYL3g5GlxhkgalxzJZT2+TH8=
=ysIY
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 28 Dec 2020 13:39
(name . Chris Marusich)(address . cmmarusich@gmail.com)
8d0eba14899e1c4ee1a00f883e3a612f215dcb95.camel@zaclys.net
On Sun, 2020-12-27 at 20:23 -0800, Chris Marusich wrote:
Toggle quote (4 lines)
> If it's just for the sake of trying one last time, we could just add
> --cores=1 to the Guix invocations, or run everything in a single-core
> VM. Wouldn't that have the same effect?

My system has an offload set up and somehow --cores=1 does not work, so
I needed this patch to make sure, also to optimize run-time I tried to
disable parallel build only for gcc.

I also tried to build only the gcc-static binary with:

$ ./pre-inst-env guix build --cores=1 --target=powerpc64le-linux-gnu -e
'(@@ (gnu packages make-bootstrap) %gcc-static)'
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAl/p0fsACgkQRaix6GvN
EKalzQ//bKS+IeBjqKNu1OCKsvBFD9aMlaQ+W4hl1Hkk0mkVHgCXoRdw7o+nTOJm
SW8Ex247ZkBAf/mqD2KN1nQH9IiMso/U7NrS9FbKiJ+mysmgrvcOQOTPwwBkTaqW
qGJEpMUDtbAhV4fvnsV6evkDRdZ01f70o9PNu+pryrzepJNkjgsBUI8C0qq8rKQV
3oMqLS8MVZeswsej7lne9rOAmP+WNmcBTG25a/CLeY1AcKFxf1vDCfzLqWaysT0l
xusaKUaVLyfgMpnk+alL7jPF16qy4wMU3Yr6q7b/axS1WVH9TBWra2RL2wgACXNJ
xXBAv/HQGf3nwhIzRDwLzcg1sxbTpZDbMTaOmk2aEvbomq93EovFFV+G08hI1/tG
KrgVYvNowLrfUKfWB96oNTqQEZh/2DbKGQ5ZVHrLVpq0chQuJGXd4RWN9R2m0ZZB
TAUfsbjfpXnuZpJWMPqRfumneiszYMzSD0Tjv3eRh7d4OnqT3bwTBYKks96uX2GL
XORrOUyGxUnBc6iTG8kTIIzYMwSBYPul8RSSjJn/lCBoBO+LiXp5xXp85AS6Smji
auKnedJdt64fv9xgoIgM58lruBv5I7VkYNZDFe5kqoa5hvJtuu7vxCowp9Qc8h9L
TnSObPwQXT8hdLDTteM0LTkI9smOIcSdQEcva6MEdSM5WLnCBSk=
=uRmW
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 28 Dec 2020 13:55
(name . Chris Marusich)(address . cmmarusich@gmail.com)
cd8907dc06894dce86c24a4741bcb6e8b3a97621.camel@zaclys.net
On Sun, 2020-12-27 at 20:23 -0800, Chris Marusich wrote:
Toggle quote (12 lines)
> I think you'll probably agree, so I've proactively started another
> build
> on two fresh single-core VMs (using the same procedure I described
> earlier, starting from the 1.2.0 installation ISO image). It'll take
> a
> few days to finish, I'm sure. Please let me know if you think we
> need
> the patch to run this final experiment. Otherwise, I'll just report
> the
> results of this latest experiment in a few days' time.
>

I did the same, I ran on both my laptop and another VM in the server:

$ ./pre-inst-env guix build --no-offload --cores=1 --
target=powerpc64le-linux-gnu -e '(@@ (gnu packages make-bootstrap)
%gcc-static)'

With latest master.

Let's see!
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAl/p1cAACgkQRaix6GvN
EKbtqxAAuTfGQkqTGltCXOwiZVXY2V9S+eCqWZ3HtOV6GZEcE16k1+U1ff1Ksx5L
5aa0bCDFC2McB49n7TTlHh/1PXdF6gHlVMeSPrcMHu0+v1qGGUJRx2ppqDPZ0OT2
8iVM2Wl4eAjK3rKTlEObe5lMnrJXd+NtYSmQYKT1Wa+qQSI/zACLBU3YG3J7Ahbv
oqnx+smjWdWdoh9kiG4cCjiFXdQb5ka34WvPRt78emO9LELnyWEaVBBwCwr371B8
d+lpcmnkeIJkx91nrrs+OT+OO3St6pZOoKqUUyCbbhM4XbGp8RC49TjnpmRTS/u2
63AWAv9VkO6FooNvC40EN9/iQ2r3haSMS+XV41g4LQyclPt02NOmlPLbtiguZPX1
g3JcU2398QrQ/AGXlGowocB1RiPxhKy3gjWLMTXJcOK0IYG3kpgh2f6M91CRS8bS
ARdCDf+ClKfwzgtbpTQt+HzvhOX0112DzBZYh2w7U71irDUvBxKdBize7jcZuLGm
jC0P088MOyV7pL/JYPnANxlUPjUQBsUHnlDFjNgYBaa+nNDhC7C4CGIOCYk+DjUN
Fti4k1pbzlyRGceo5eUoJOuT6ZHx+NxDwrx8pH31Lct7vflSbrROAeGMv2ZCeMyN
iv/ukvrTnXM+u1ZGGt1uB5x47bupPpQurQ1o+M7Pwv/Ba3p2IfI=
=7Rpz
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 28 Dec 2020 16:31
(name . Chris Marusich)(address . cmmarusich@gmail.com)
200570903d8a95f7d9772a53b2573b318d623fd4.camel@zaclys.net
On Mon, 2020-12-28 at 13:55 +0100, Leo Le Bouter wrote:
Toggle quote (10 lines)
> I did the same, I ran on both my laptop and another VM in the server:
>
> $ ./pre-inst-env guix build --no-offload --cores=1 --
> target=powerpc64le-linux-gnu -e '(@@ (gnu packages make-bootstrap)
> %gcc-static)'
>
> With latest master.
>
> Let's see!

I modified my virtual machine to only have a single core because it
seems --cores=1 does not work properly with cross-compilation.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAl/p+jwACgkQRaix6GvN
EKbzExAAo8iQoMp7CoRdrUbBOeSJaR4hyWk2BOx8HghMkwmSMShRSucaPZJDX5ig
uuQPGcY61h5MtndC1PhW1U8WRhDfMY1imVhK96E2S9fjxJXCpUclG+sPMAvrS5WG
SZ0aqsmi0kekzsviKX+oE+mI25zAXnnQb+w1Sg9fd8V9KXE0eZsfDWmYycyTAuzI
3J7eiiFutgzv2Hcu/1TKaG7OOtJ8SW+kS6EO5SHvI/RsO99nfn+4pxOdLIw35App
xrsXZR+CnGOPsSG2DlFxvUyAu2Ny9QLquwZVNJRp6487nFn0iVPcVBk1AwGSsgv+
7kx1038Nyeo8UYbLsIlTabLxD4RSXeKJcu02ddLdI6F28+5UfW1cVczfciYMWnMu
sXXCYGP+LYp8KvUOo0ovgaYvKr1orY6ZKJhG8Mh/ttWYJ0uVQOJ1hMqdv1mPKA2M
AuOFxsWrBXsXzCIE9vsf9uWAdaSGKZe+D1msMnkgv7DlOAQ4Zi65XxelZFX+zewB
BC2LJFSvmx3iOe2ZH5UdiRwqzrjXkE3SkVGSmsXgl3bt4E53SfeAP3GlMjSHMGxT
IXyw3UqvadfTT2ID4wz1kKgWXKZRY12SVbmg67z7jX2k093GJ3bu/UjobkIFoN+7
yncWk6YRDj+ufNWg840ft6IONBUul7ApUgcUnoqvAYg4sD4bQ/I=
=h28J
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 28 Dec 2020 18:40
(name . Chris Marusich)(address . cmmarusich@gmail.com)
2f35e98480d308bc86c3eebdb9ae551d757110e6.camel@zaclys.net
On Mon, 2020-12-28 at 16:31 +0100, Leo Le Bouter wrote:
Toggle quote (3 lines)
> I modified my virtual machine to only have a single core because it
> seems --cores=1 does not work properly with cross-compilation.

Great news! I think using only 1 core fixes it. But since --cores=1 is
broken for cross-compilation it seems, I had to use a single-core VM.

Can you do the same? (Create a single-core x86_64 however you want VM,
run a command and report hashes)

$ guix gc && sha256sum "$(guix build -M 1 -c 1 --target=powerpc64le-linux-gnu -e '(@@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)')/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz"

Also try with powerpc64-linux-gnu:

$ guix gc && sha256sum "$(guix build -M 1 -c 1 --target=powerpc64-linux-gnu -e '(@@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)')/gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz"

Thanks a lot!
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAl/qGKcACgkQRaix6GvN
EKY+Gg//T5ty8D+No88zQyz3Ah7YAjR1mYO1XoeTmwrYxF1HJtWzyOHaht2bhF1B
dNYYaZfA/UsbyGIu4TraMpQCVVkAa1OTmMXuVluZ5ZX/A/Hi6caeRb1+iFJ4hKxY
bhUzSjaV9JEKeZ6x1zfghM5/cA32spBG9vWYhtYq8Z+3va/nnPmGt5/TyqNXmJLn
rL4o9xxzmV91Y9zpZpclrALc9cEekSWpgdmDzmfqr/kM80L6vZrhkGfHprGmn5wi
Brl+vLHYbiFDi2JlTmw5p6UAR6BTiVmcZ6OXFANMlj9f3rzKpe8piYF2O+nut7jN
Bmfu3t0jzUh1/Du6714jqBxFbtMJLcPej0pV4u7MuwqgQfJFinCru3TpF09x807Y
nnDaCyH908LU7NTPjgMao/eZWLjvgjM1X0mawj7xUem5rmvBCcKCrQ2vMrErnuwU
nEcMieoQV1btiaKj9nhlyV+2yp3xZKl0ZhNoS6F2XfVmjbb4zOnZrLBU7CO92kas
wSs7GKF7vBMm05gCO3zaxm22I+ssly9wBjTUhCQNcIlO5iDpjcB/jn8C0p3IKdRJ
gmS2aI0KNzK7KheU2ZT+NHFXFJ9k0xn9m8JK+DE3j6d4Xmyc//OOqa+Rq+tZQ6K2
zY+vEFpi1UKnQu8bz9mscxTNmk1YOTpiz3+zIRz97xj9muB2LoQ=
=H/2a
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 28 Dec 2020 20:01
(name . Chris Marusich)(address . cmmarusich@gmail.com)
db8e5decdc4c1569dab340f76ba59081ded9ea20.camel@zaclys.net
By the way here's my powerpc64le-linux-gnu hash:

$ sha256sum /gnu/store/cc4sj9kiw1vzhhhxawnqvjjdi4ygbxp5-gcc-stripped-tarball-5.5.0/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz
153f356a8f58102e7e57a006e836ec7450e69dee120ba83d9f7dacf07162537b /gnu/store/cc4sj9kiw1vzhhhxawnqvjjdi4ygbxp5-gcc-stripped-tarball-5.5.0/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAl/qK20ACgkQRaix6GvN
EKYxuA/8CL6YKKxKdlf+ZPdYPoaqsq6mK2SuhvXRvIrrag4rUJ3wLfKE0iHVgaWl
k0wQqHqZyZ3A22GYOUBUE/9QXV1X6KuXifFghYmJmwTsocfg+I+ielHMMWqsZBKY
3C73slGdOw0VQ4Ybwl1u2NShGiT/BtkYOK8jHs19Zv/liysfENfYmz3fqWyxUkHJ
1a1JI7dO4aTinNRhX9MNpkjEq0ElGwdgltl4U4Pv2tZqrKi8xtQKVkYbN7saLXmE
NraGKvxw/bIZP24xxCcWNMO4gv9mLefQr2gbIJZFZPf4DPfu88rczAmziZhUaYV/
NewF2bKsvK8eURhvzNbqf4As+kaKmcxJ28FBo3N3bL65keW6RZ+SpUJVRDfPD8pR
aLj1ttDP2989xPihvRBmGoWaXggNC0vTykXFQX0NdAGx4MPb7ZyQFRQLffGoce2i
l33W26PBmiOjyOkHrAb+y5J/34PsFo9rhSXvyq71rMuqKRQubEWRJ7uWbMaI3iLq
sO4gSzVl4bkxwfmzIt/ajSfvle8wUL/TsKxOjLzB6FDZLwBJjhjaI1eetui9Ddyo
hhV7AgNRdGU8iDCmuuivXULdLO7nIZWGFXzaXrJHlL9L0a1f0ey9nOTP4tSHf2+d
VizXzlzpa1d1Pj+gNfhbw+pzocv7HPiGrL23km2JOrKkihycqJ8=
=Tr2k
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 28 Dec 2020 21:59
(name . Chris Marusich)(address . cmmarusich@gmail.com)
0d60350583ac756768e41b3957384e6837b25813.camel@zaclys.net
On Mon, 2020-12-28 at 20:01 +0100, Leo Le Bouter wrote:
Toggle quote (8 lines)
> By the way here's my powerpc64le-linux-gnu hash:
>
> $ sha256sum /gnu/store/cc4sj9kiw1vzhhhxawnqvjjdi4ygbxp5-gcc-stripped-
> tarball-5.5.0/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz
> 153f356a8f58102e7e57a006e836ec7450e69dee120ba83d9f7dacf07162537b /gn
> u/store/cc4sj9kiw1vzhhhxawnqvjjdi4ygbxp5-gcc-stripped-tarball-
> 5.5.0/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz

Nevermind, tried in a brand new single-core VM on my laptop and it did
not reproduce...
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAl/qRy4ACgkQRaix6GvN
EKarRA//eoOC9KIdLtdReDoZtou7voruXB9d0wbZsPIBz+W0ICs4stjHawUNdvnu
G8XN2O94KU8p9giu5HJ5j53xX8G4Mu8Lpr9fsAXBWupdn69kIJpO9+j2yzN68j5X
Wh8xrlpxe35OHIawYzDNzr/2twABVRk7/kZXrwvaWtPE7u5f6aU++/mAeqZPLrxC
DPxnboBia1oIYbnauqkVpIwemMXVfTyEybExW/rxxYyMayeQri9BSAFwYRKFWWNa
bbSYK0QCG6HNGEONzangbyRT2LfUVLUgsYL+Z1oS/UTOJpXJQicmiSfbmNvybw90
f2AiXV0qYngGPEMRJ1vcSUp8R0qiV4YNIGuiZf66UD4uEeqmRRIYkSytO6/P66TC
D28mSjnIkkVzWmKv2P3p5+H+F/Fl71MCcO9X7QZBkWnhBmrbGATat9gMxR591cBi
MMeRYl0bvFt3H+KNgUV1A7aTIK2BOe9lZDMkDxxx870SYLAEdzaZN3m0MuRi0aOc
SVaASWBda/xUIQL85kAiQpGH1nQ7iooGX92gT4D1lB5at3rbKLJ9RjmCR0fgyWCS
Bs0XMonLKoZpALO36/Jj/9JAGqvyFJ4quFH4fpozTCw3CJj4AN0NEl8L7BZFZw8b
6XNo434U/9Mgip+1a6/FtJdg9Riox25qSM2htnGUvTYDQlI8NrA=
=OVRM
-----END PGP SIGNATURE-----


E
E
Efraim Flashner wrote on 29 Dec 2020 08:08
(name . Leo Le Bouter)(address . lle-bout@zaclys.net)
X+rWCP4YLSAuWzjS@3900XT
On Mon, Dec 28, 2020 at 09:59:26PM +0100, Leo Le Bouter wrote:
Toggle quote (12 lines)
> On Mon, 2020-12-28 at 20:01 +0100, Leo Le Bouter wrote:
> > By the way here's my powerpc64le-linux-gnu hash:
> >
> > $ sha256sum /gnu/store/cc4sj9kiw1vzhhhxawnqvjjdi4ygbxp5-gcc-stripped-
> > tarball-5.5.0/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz
> > 153f356a8f58102e7e57a006e836ec7450e69dee120ba83d9f7dacf07162537b /gn
> > u/store/cc4sj9kiw1vzhhhxawnqvjjdi4ygbxp5-gcc-stripped-tarball-
> > 5.5.0/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz
>
> Nevermind, tried in a brand new single-core VM on my laptop and it did
> not reproduce...

ab2fd297f65df5ab6fb5662dc3988f8421f2194026ac3df97eba3b53832796b9 /gnu/store/cc4sj9kiw1vzhhhxawnqvjjdi4ygbxp5-gcc-stripped-tarball-5.5.0/gcc-stripped-5.5.0-powerpc64le-linux-gnu.tar.xz
9bf3685309b68cc0a39b2dc783ada393e6ef889398dd21fd07c7893159c13cc0 /gnu/store/kjxwb8lwbgnp2np7ji8dvjmh290p1jnm-gcc-stripped-tarball-5.5.0/gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz

--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl/q1ggACgkQQarn3Mo9
g1GzVg//b0bukg6EgZb0xoZA2Q88hDFuxNm4fp1z4xxarcs0RdYEF1sd0gWQIA7n
y1ORxgSPiheY8X25QjLeN4z5f+1lS9sxDSYqaKzfSc12fv2V2Wo6i3yrxV/Zgn12
dmrjRH7EvS9BRaztj0gYGTV4rGfLv6FfmlfumLSw3qyBxVJIc57PpgcCIJO85EZn
T9aOt+JbAQwo1cQVwHjhhp0b5a5ZO/vCjZl7piuv2pKp5zhJP0EGVVKMbbcj0wwg
ogKy+86/ixYmgjmq4gTr1/ajYrxc/VquYQANGQj0d2cSY6Kq7/1bMF47dGg5YpMc
orHTcL/lfmMdZbCLUxhALxFGILV7DPZkvpvITDiu+BEX1SdW/OrN1rvT8tVlySx2
jraesMFaonU02jEmdXg0Z/5vzzmThgSY7Lp2Uj4UL8QFnv108J/F5QRuTxoAfN4T
uljKyF747PDr8JdRawKdoCQqLqS8kgNdckHerr4Y2fK2J4pkhc/Pivld1X17HPGb
LSXG5mEeIek5RZN+oxWXsZc0aTmKRnmTmKJKYR78Zym4UWCS90nLhRabiuZ3Rlcn
ind1SOV511fhr8kIc4mf+Tl0s8kou8kCn14IDpfiC6FMrDGbn5z0mMH9SYFuV+5G
ITfTa+JIPfqwPbmelHQAPhMVDLzw6vA68m+Gs0NvWeKvTOc6wLc=
=VFBT
-----END PGP SIGNATURE-----


L
L
Leo Le Bouter wrote on 30 Dec 2020 02:28
d6d80d2011b5a04b2dc491e258aac27f08137ddc.camel@zaclys.net
Hello Ludo!

On Sat, 2020-12-19 at 23:28 -0800, Chris Marusich wrote:
Toggle quote (33 lines)
> Now that we have decided to use these powerpc64 bootstrap tarballs,
> what
> are the next steps for uploading them to the GNU FTP server? I've
> never
> done that before, and I don't think I have access. For now I've put
> a
> signed copy of the powerpc64-linux (big endian) bootstrap tarballs,
> with
> a SHA-512 hash, here:
>
> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz
> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.asc
> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.sha512sum
>
> For the record, these bootstrap tarballs were built via the following
> steps:
>
> - Use
>
> https://ftp.gnu.org/gnu/guix/guix-system-install-1.2.0.x86_64-linux.iso.xz
> to install Guix System 1.2.0 on an x86_64-linux machine.
> - Run: guix pull --no-substitutes --
> commit=1ced8379c7641788fa607b19b7a66d18f045362b
> - Run: guix build --no-substitutes --target=powerpc64-linux-gnu
> bootstrap-tarballs
> - I didn't run "guix system reconfigure" after installing Guix
> System;
> theoretically it shouldn't matter, but for the purpose of our
> experiment, I just left the system in its default configuration in
> order to ensure that the kernel etc. would be the same on both VMs.
>
> Once we get these binaries into the GNU FTP server,

Can you do it? Upload the binaries on GNU FTP server? Little endian
binaries will come along shortly as well. We decided (Chris and me)
that obsessing on this problem any longer is not going to help us
complete this port, the amount of struggle is getting very
demotivating. We got reasonable proof that this non-reproducibility GCC
problem exists and is non-trivial to solve, that's what I wanted with
the last attempts, I had hope it could still be trivial but turns out
not.

Toggle quote (12 lines)
> I'll get started on
> updating gnu/packages/bootstrap.scm and other files necessary to
> begin
> bootstrapping powerpc64-linux. I'll mainly be adapting the work that
> Leo already did, and following the lead of others like Efraim and his
> work on the wip-ppc branch. I will probably have questions as I go,
> so
> I'll ask on guix-devel.
>
> Please let me know if you'd like this done on a special branch.
>

Thank you
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAl/r18kACgkQRaix6GvN
EKZPuA//bL1yVmDUVfkg0wVu0ZCUPmTI6tsboXf1E0I16Y0ya7eEfTimuYcus5F5
tJkk1TAMRZ6D/T++t1DixusdKtnHu6BuU1sxpV0XEDUWjDCXNRijWtYFSO+GoAe5
v6bVwm7jAdmPoJNoeocXWW033w77hRLgOap/3PzSbtA0cStLmZRsj9EkDx1FCMk2
vaJgZ/zQl+4/A8pj6/VaIdkm1gXgAqtmjfj4piCfhrTVy7IDkInLyougHUdRn7eU
1EcxgH9aGg6Dicr/oCQaITGArF2pl0ZQ95bRapS7YSdd0ndKu1cmPxWeZuFAus++
833nkWJsGU4m1aEPXQjt/pqVzHwGOCjxDsAp2od77J0VE24bCX4cP3+tr7kHHpR6
rAhyZz/z0hZOwC6CqIzQ3MZxudz3hnOw8+vuoFw3I6D6++ekryUsucseUlOVNFPC
TauGHBbp1P2pTLjIMP6N2BBHKOZp+/vUMMm+vzucMhRanPWrLMaRRMuFeTXhlvqW
Sx4CRksl+KHxJ1akGeNMyM0BMVUhWYWQzKaWOpxruc3pvKi2pB1Y7BsBpCA9oISE
HZ3+pcS7q8UrqVdEntJ8IsjupuT3Nhbn76AhHQ07R4KPWHJyMnaTrsOCAQCZZXhd
bWVHm7c1JAjZKS0HnHVHFrvAJfFSTV6OF0t2i4BUhf6czak/xmw=
=wxiE
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 4 Jan 2021 10:37
(name . Leo Le Bouter)(address . lle-bout@zaclys.net)
87sg7hxctr.fsf@gnu.org
Hi,

Leo Le Bouter <lle-bout@zaclys.net> skribis:

Toggle quote (15 lines)
> On Sat, 2020-12-19 at 23:28 -0800, Chris Marusich wrote:
>> Now that we have decided to use these powerpc64 bootstrap tarballs,
>> what
>> are the next steps for uploading them to the GNU FTP server? I've
>> never
>> done that before, and I don't think I have access. For now I've put
>> a
>> signed copy of the powerpc64-linux (big endian) bootstrap tarballs,
>> with
>> a SHA-512 hash, here:
>>
>> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz
>> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.asc
>> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.sha512sum

[...]

Toggle quote (3 lines)
> Can you do it? Upload the binaries on GNU FTP server? Little endian
> binaries will come along shortly as well.

Sure. IIUC, the tarballs at the URL above are for PPC64 (system type:
‘powerpc64-linux’); is this also going to work on PPC64LE hardware?

(As you know, we use i386 binaries for both i686-linux and x86_64-linux.
Likewise, if we can have a single set of binaries instead of having
PPC32, PPC64, and PPC64LE, that’s better.)

Thanks,
Ludo’.
E
E
Efraim Flashner wrote on 4 Jan 2021 12:16
(name . Ludovic Courtès)(address . ludo@gnu.org)
X/L5KJGdftk0VdKR@3900XT
On Mon, Jan 04, 2021 at 10:37:20AM +0100, Ludovic Courtès wrote:
Toggle quote (31 lines)
> Hi,
>
> Leo Le Bouter <lle-bout@zaclys.net> skribis:
>
> > On Sat, 2020-12-19 at 23:28 -0800, Chris Marusich wrote:
> >> Now that we have decided to use these powerpc64 bootstrap tarballs,
> >> what
> >> are the next steps for uploading them to the GNU FTP server? I've
> >> never
> >> done that before, and I don't think I have access. For now I've put
> >> a
> >> signed copy of the powerpc64-linux (big endian) bootstrap tarballs,
> >> with
> >> a SHA-512 hash, here:
> >>
> >> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz
> >> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.asc
> >> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.sha512sum
>
> [...]
>
> > Can you do it? Upload the binaries on GNU FTP server? Little endian
> > binaries will come along shortly as well.
>
> Sure. IIUC, the tarballs at the URL above are for PPC64 (system type:
> ‘powerpc64-linux’); is this also going to work on PPC64LE hardware?
>
> (As you know, we use i386 binaries for both i686-linux and x86_64-linux.
> Likewise, if we can have a single set of binaries instead of having
> PPC32, PPC64, and PPC64LE, that’s better.)

Last I checked the PPC32 binaries aren't working. Even cbmuser from
Debian thinks there may be a glibc bug with glibc-2.32 so I can't test
that much ATM¹.

Rather than effectively needing to wait to add ppc32 and ppc64 at the
same time it'd be better to add them individually and then do any
rebasing necessary to make them both work with ppc32 binaries later.

¹ If my patch fixes it I'll add glibc hacker to my résumé.

--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl/y+SUACgkQQarn3Mo9
g1FXBQ//TgHlPqjGlybtWw/Io1yp6ybNVDgbbx23bYaa94COLK3pT1Etyig7uN9w
+m3RrnGu5Y4DjM24VYDfIK8f4q/hA9nHAMKiS4aqZovcPmQpU9lU5hyBOfEZTlQc
OGXe7kKavL0PVh15RP9i4XwMvOZijHoe34Aoru4hgDFqSNfCL9DLRN7qneCtf9rd
7iskNkc5TmHPuvySt4Olyp22ES7KEmNZS6NJs3BSZBNNtKau2vU7SZQKvQLze4+L
pqIK7xcXLWQWT17SEnamAPRFXryjF8A10qon/epc8rlj/NqmfRw1nKywGC9B/w5t
CYmnRNXweClc2JBP1dZqvXCrVQYvllEcbJOcXtYl+oeKfQRYmQ+6Ka1Ar9jk8hRS
4XTdNcQGazhVCDGS1hsy3mSEDwRnLBqqJEsmaygJm6bagsQ0fM/hWUDi+Xr4sTJP
8fkVbKSTyr8hXH3E+p1tPxOgpEK9FwXoBB9ZjhGapaPDRdAySGgok2EfmTLeCpG+
pmS+oTVSi7L6HGzo1U92JE0Cmzmn+wPNCnoklWp/UpSunu7OS/U6i9BFGz47ekyb
rJWIiCj9yenWBw63ScpefgMdT1vpGf2AXffBC/TP8YbQWTV75zNn6ytXoo7QSJOV
FFtW5pGSdsY1dvSvzEuafjYJNgQf7g/Y6tP5E0aZvLw+jANF1mU=
=BPhx
-----END PGP SIGNATURE-----


C
C
Chris Marusich wrote on 5 Jan 2021 04:15
(name . Ludovic Courtès)(address . ludo@gnu.org)
8735zg13c9.fsf@gmail.com
Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (9 lines)
>>> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.asc
>>> https://media.marusich.info/guix-ppc64-bootstrap/bootstrap-tarballs-from-guix-1ced8379c7.tar.xz.sha512sum
>
> [...]
>
> IIUC, the tarballs at the URL above are for PPC64 (system type:
> ‘powerpc64-linux’); is this also going to work on PPC64LE hardware?

No, we need a separate tarball for LE. I have prepared that here:


This tarball, containing the little-endian bootstrap binaries, was
generated using the same setup that I used earlier for big-endian.
Specifically, to generate the little-endian bootstrap binaries, I took
the following steps on two separate VMs:

- Use
to install Guix System 1.2.0 on an x86_64-linux machine.
- Run: guix pull --no-substitutes --commit=1ced8379c7641788fa607b19b7a66d18f045362b
- Run: guix pull --no-substitutes --commit=662e7e28d576ada91fc9dec7d27c100666114f03
- Run: guix build --no-substitutes --target=powerpc64le-linux-gnu bootstrap-tarballs
- I didn't run "guix system reconfigure" after installing Guix System;
theoretically it shouldn't matter, but for the purpose of our
experiment, I just left the system in its default configuration in
order to ensure that the kernel etc. would be the same on both VMs.

Notice that there are two "guix pull" invocations. This is because I
first pulled to 1ced8379c7641788fa607b19b7a66d18f045362b in order to
build the big-endian bootstrap-tarballs as described earlier. After
that, I pulled to 662e7e28d576ada91fc9dec7d27c100666114f03 in order to
build the little-endian bootstrap-tarballs on the same machine. In
theory it shouldn't matter how you arrive at commit
662e7e28d576ada91fc9dec7d27c100666114f03, but for the sake of
completeness and reproducibility I've included both pull steps.

By the way, just as with the big endian bootstrap binaries, all the
little endian bootstrap binaries I built were identical on both VMs
except for gcc-static. The output of gcc-static contained binaries that
differed in ways similar to what has been described earlier in this
thread. So, the non-reproducibility of gcc-static is not specific to
one PPC architecture. I wonder if gcc-static can be cross-built
reproducibly for any architecture at all.

Toggle quote (4 lines)
> (As you know, we use i386 binaries for both i686-linux and x86_64-linux.
> Likewise, if we can have a single set of binaries instead of having
> PPC32, PPC64, and PPC64LE, that’s better.)

This is a fair question. I agree that if it were possible, it would be
a great improvement. I didn't know the answer to this question, so I
asked in #talos-workstation on Freenode. The users there said that
although in theory this should be possible, it isn't currently feasible
because the ability to do this is not currently implemented in Linux.

For the moment, I think our focus should be on finding out which of
these two architectures can be bootstrapped in Guix in the first place.
The first step in doing that is to try using these bootstrap binaries.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl/z2dYACgkQ3UCaFdgi
Rp0STRAAg0bC3F6xlp/OxckoesF0XY6Htdmgz/lGAEKjiDBFi4VnDx6qF1Hl4udQ
qeu3TlSiHijYuVJUhPndQhTbwxGnxUMLfCRhZDFRng2e/3Xt07bOga1NvkdG2Wjm
rTtTghMLLUhcOrzbdW/AMbwn7t6joFNI/5jMhtcFS5EMjjV68IpwvfWWHD4Tv6Pk
DCzbWGjiiXuMvtvS331Qmkp6SRf1N+xIWiedtFvjSLuNIY/aoKhyO1hI6dVPnhwX
xwDvozWQrduqOxW9ZRh+vo5CgIj6rRdz8/EQOdSdOeLDtMKTVbEB/95o2kV7mLby
+742OAaqPF1upaRdFze4C+2HCn8zeRy/UHbDNjPu0f0Q31gjZP6OVBu9tOKIRvpN
pyN87lGDuQM8WhkpACVeb4ul/QnjPptyynum2p9z5/t/9F0MaRBdvg9Chf6ZO5uO
6wHQCDqo0EJeZ/PneVIjx/MNAyi3qeX0xdAt0qawg8AScB+58tod+YnXL/O+LPGu
MbVDLffTGFHGwbxDpBpgQNh7U4o1wBXlTYotFo472WyXbuyjxAqO2Efl9RUFmnNa
i9DhiN1+D3HQfXELMa6N/LyFh+tHIhVZfWvG/Hz3LYBNXjQCHIGxcWXGXY8Ye+9v
Izppu+SfEqBQWMDManzLdO+Bsj72nixcqxkmqabajanlA3Obp/o=
=ep7a
-----END PGP SIGNATURE-----

C
C
Chris Marusich wrote on 5 Jan 2021 04:54
(name . Leo Le Bouter)(address . lle-bout@zaclys.net)
87y2h8yr6u.fsf@gmail.com
Chris Marusich <cmmarusich@gmail.com> writes:

Toggle quote (11 lines)
> If it's just for the sake of trying one last time, we could just add
> --cores=1 to the Guix invocations, or run everything in a single-core
> VM. Wouldn't that have the same effect?
>
> I think you'll probably agree, so I've proactively started another build
> on two fresh single-core VMs (using the same procedure I described
> earlier, starting from the 1.2.0 installation ISO image). It'll take a
> few days to finish, I'm sure. Please let me know if you think we need
> the patch to run this final experiment. Otherwise, I'll just report the
> results of this latest experiment in a few days' time.

The builds finished on both of my VMs (new ones) the other day. The
result was the same as before: Even when built from source using a
single core and with --cores=1, gcc-static differed, and all other
binaries were identical. This is more evidence to support the
conclusion that the non-reproducibility is not due to concurrency.

For the record, this is the summary of the final experiment I did:

- I created two new x86_64 VMs using QEMU.
- I used
to install Guix System 1.2.0 on these two VMs.
- I ran: guix pull --cores=1 --no-substitutes --commit=1ced8379c7641788fa607b19b7a66d18f045362b
- I ran: guix build --cores=1 --no-substitutes --target=powerpc64-linux-gnu bootstrap-tarballs
- I didn't run "guix system reconfigure" after installing Guix System;
theoretically it shouldn't matter, but for the purpose of our
experiment, I just left the system in its default configuration in
order to ensure that the kernel etc. would be the same on both VMs.

In reality, it didn't go so smoothly. While running "guix pull", the
build for guile failed many times before I got it to succeed, which
prompted me to submit this bug report:

"Guile 3 fails to build non-deterministically"

While running "guix pull", the build for gnutls failed many times and
never succeeded - so I actually did substitute gnutls. I submitted a
bug report for that one here, too:

"gnutls 3.6.12 can consistently fail to build"

In any case, this confirms Leo's findings. It's clear that concurrency
is not the reason why gcc-static builds non-reproducibly.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl/z4tkACgkQ3UCaFdgi
Rp0r/A/9GMi2kxAaooZurpUsFG2N+asDtRGcOb2vBjeGoGSr3cxcZilKeaW90As5
ITw6MN/5+O2JparHXGAzFTz52XztoeBkwt0Cec5sFbwlrvpcdrXF6+oC09lqyh3W
0V+7Y8Mh2Fk3QyZwE6s8jszDyeR0Zp0HevRArKseuaGmUG85t3CGFRZ3iTBDUCMb
iDayeG61HFEhKZ1+jg3uRC3cDO6qWQ/hui5cllq9pLKYUMVW06jY3HKU4Q1eTW+a
/AVy9jDx2UJYMjjP45GiJ7YAHqML6G64t5Bzs6HA6Nu0z+y+xsfFpqC+8FEBgpBK
ili/dLfb05wJxmp9kjc3IMxRMr1GZryQQlX5EFhDP7nvQ8lppYOCNWEHcvGjx+xA
bpWNDTa6qTOi2wNHoo3sz4tFQvxdXKU82o7Xa2oJU+60lW7UsY7xt4fTCECZ+7OU
QnwWvFiQI42zZlFlj8sYj5wgcmQODRBwji69qPp1F2yzo+A6X/PZmTDSPcd8goEC
Z49YMv2sP0h6b05+1Ftsni6BvGZkOe/zc2Lik9TxZFPYHxl7T5CL6I5YPN8hr1+p
6L7JMDTt37YtvRKtuqjM2Q57d5eQHhWfvvasEZVUcsv3m8g/zD0WlXbwNHCooOvo
8sUCVmL8srZDwFHByCK9rY6ecncnSFOrS7tTSkJHhQSKSKPkbL0=
=AoFk
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 6 Jan 2021 09:59
(name . Chris Marusich)(address . cmmarusich@gmail.com)
871reysaoq.fsf@gnu.org
Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

Toggle quote (6 lines)
> No, we need a separate tarball for LE. I have prepared that here:
>
> https://media.marusich.info/guix-ppc64le-bootstrap/powerpc64le-linux-gnu-bootstrap-tarballs-from-guix-662e7e28d576.tar.xz
> https://media.marusich.info/guix-ppc64le-bootstrap/powerpc64le-linux-gnu-bootstrap-tarballs-from-guix-662e7e28d576.tar.xz.sha512sum
> https://media.marusich.info/guix-ppc64le-bootstrap/powerpc64le-linux-gnu-bootstrap-tarballs-from-guix-662e7e28d576.tar.xz.asc

Thanks. I have uploaded them to
(I meant to send them to ftp.gnu.org but typed it wrong; I can upload
them there as well later.)

You can update (gnu packages bootstrap) accordingly.

Toggle quote (16 lines)
> This tarball, containing the little-endian bootstrap binaries, was
> generated using the same setup that I used earlier for big-endian.
> Specifically, to generate the little-endian bootstrap binaries, I took
> the following steps on two separate VMs:
>
> - Use
> https://ftp.gnu.org/gnu/guix/guix-system-install-1.2.0.x86_64-linux.iso.xz
> to install Guix System 1.2.0 on an x86_64-linux machine.
> - Run: guix pull --no-substitutes --commit=1ced8379c7641788fa607b19b7a66d18f045362b
> - Run: guix pull --no-substitutes --commit=662e7e28d576ada91fc9dec7d27c100666114f03
> - Run: guix build --no-substitutes --target=powerpc64le-linux-gnu bootstrap-tarballs
> - I didn't run "guix system reconfigure" after installing Guix System;
> theoretically it shouldn't matter, but for the purpose of our
> experiment, I just left the system in its default configuration in
> order to ensure that the kernel etc. would be the same on both VMs.

In the commit log that updates (gnu packages bootstrap), please mention
these commands so we later know how those binaries were obtained. (Only
the second ‘guix pull’ matters.)

Toggle quote (8 lines)
> By the way, just as with the big endian bootstrap binaries, all the
> little endian bootstrap binaries I built were identical on both VMs
> except for gcc-static. The output of gcc-static contained binaries that
> differed in ways similar to what has been described earlier in this
> thread. So, the non-reproducibility of gcc-static is not specific to
> one PPC architecture. I wonder if gcc-static can be cross-built
> reproducibly for any architecture at all.

Yeah, that remains a mystery, perhaps we’ll eventually find out!

Toggle quote (14 lines)
>> (As you know, we use i386 binaries for both i686-linux and x86_64-linux.
>> Likewise, if we can have a single set of binaries instead of having
>> PPC32, PPC64, and PPC64LE, that’s better.)
>
> This is a fair question. I agree that if it were possible, it would be
> a great improvement. I didn't know the answer to this question, so I
> asked in #talos-workstation on Freenode. The users there said that
> although in theory this should be possible, it isn't currently feasible
> because the ability to do this is not currently implemented in Linux.
>
> For the moment, I think our focus should be on finding out which of
> these two architectures can be bootstrapped in Guix in the first place.
> The first step in doing that is to try using these bootstrap binaries.

Yes, that makes sense to me.

Thank you!

Ludo’.
-----BEGIN PGP SIGNATURE-----

iQJBBAEBCgArFiEEPORkVYqE/cadtAz7CQsRmT2a67UFAl/1e+UNHGx1ZG9AZ251
Lm9yZwAKCRAJCxGZPZrrtX79D/40ndhr8TOycZUV8fzJyi1xJh+iGwTUapRxJFS0
7dsXLnPYh6RQrZ8HoPsZ7mqA7pSo9+edzExEhOoqwVAU7De1wrMJ8MKv9r7g8aCI
bXYK3ms1C6trU9Dqnv6wLUzl9wCqriUyzDQmJpIdyDs6yP591CHBhROZ8uur5A7D
8eMWEvB0lqHZm+iVlPYFStA3itnyuXV9rC90gpZxsgvCWt8n1qLTczjoqlYmuN8g
O21vm6Hx8xEQS9Vs2Wig43aVFBZk+k685i1O1rq1vLmZl0zk4TiKRs7sp3GL+ugc
1J0lIjNgUhAy9GjFEuPEQfIHYvThXwpWrQrGK/xphR2COTnDH4bfc6zI5FBjoQH1
BbmW14PsPlnL4HsUURubWkfLIxncQKqrGt0vsOp2qVQmzCWqx3+JxHlbZ0gKS1Hc
fUR56dALatIFJojW4toQ5ZF6DpVnN7whkENYOH7omlh4WDZtkc0y18/hqBqw2pvZ
pGSpGLS3Bt2Cj6LfsGzfS2FGNOO9uHMLDvSxFOe4vG61XP5T9vWfg3QYL+wBsITC
gTQGmlkdbgG6I4+EyHkIFGIeXy9n9tYpEHqC6fvyl8VALWh3cnbdikI5rVAMH0ZZ
7872hoN9cAPkcLR7XHodHb3MPgCw95dl3Cmt9CL0wxXEu+/ZKnK84RYbXsEvgqsV
zLLRkA==
=QI9n
-----END PGP SIGNATURE-----

C
C
Chris Marusich wrote on 11 Jan 2021 11:31
(name . Ludovic Courtès)(address . ludo@gnu.org)
87k0sjbw8w.fsf@gmail.com
Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (2 lines)
> You can update (gnu packages bootstrap) accordingly.

Thank you for uploading the little-endian bootstrap binaries! I've
downloaded them, and I confirm that they are identical to the ones I
built. Do you also plan to upload the big-endian bootstrap binaries? I
think we were hoping to try both system types.

I'm afraid I've hit a snag using what you've uploaded, though. It looks
like we'll need to extract bash, mkdir, tar, and xz from
static-binaries-0-powerpc64le-linux-gnu.tar.xz and place a copy of each
in the following locations:


Could you do that? The reason why it's necessary is described below.

I've started making changes locally to (gnu packages bootstrap). So
far, with the attached patch, I'm able to do the following things on my
POWER9 machine running ppc64le Debian (unstable).

I can successfully build Guix from source using Debian packages (and a
manually-built copy of guile-avahi and guile-gnutls). With my patch, I
still have to supply the "--with-courage" configure option. Although
"make" succeeded, "make check" failed on the following tests:

- tests/build-utils.scm
- test/challenge.scm
- tests/containers.scm
- tests/debug-link.scm

The failures fell into two categories:

- Some tests couldn't download the bootstrap bash.
- In tests/containers.scm, call-with-container evaluated to #f when it
wasn't supposed to.

Nevertheless, I created the necessary build users and started the
guix-daemon via pre-inst-env. I then tried building a simple package:

./pre-inst-env guix build -e '(@@ (gnu packages bootstrap) %bootstrap-coreutils&co)'

The result was promising, but it quickly failed like some of the tests
did - it couldn't download the bash bootstrap binary:

Toggle snippet (55 lines)
marusich@suzaku:~/repos/guix$ ./pre-inst-env guix build -e '(@@ (gnu packages bootstrap) %bootstrap-coreutils&co)'
substitute: guix substitute: warning: ACL for archive imports seems to be uninitialized, substitutes may be unavailable
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv
/gnu/store/2nwml4l272qzq034hwf5icv9vxi813ja-xz.drv
/gnu/store/c1v9lmsh0awbhpai72mzw4qv02rrbpw6-module-import-compiled.drv
/gnu/store/n5hf44bybvqmsybjvnn61pkpmcdcrlbd-guile-bootstrap-2.0.drv
/gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv
/gnu/store/jxh9xn77flxarwzcjga485pgrkjknrgb-tar.drv
/gnu/store/yd1mib8s1f38qwdn61zj16ijx8p0ryzm-guile-static-stripped-2.0.14-powerpc64le-linux-gnu.tar.xz.drv
/gnu/store/z66wc9z4qvffn60q4jdx7in6rxpswhx3-mkdir.drv
/gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv
building /gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv...

Starting download of /gnu/store/2phdifnfw6i989rqbav04zakxx7qb165-static-binaries-0-powerpc64le-linux-gnu.tar.xz
From https://ftp.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz...
download failed "https://ftp.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz" 404 "Not Found"

Starting download of /gnu/store/2phdifnfw6i989rqbav04zakxx7qb165-static-binaries-0-powerpc64le-linux-gnu.tar.xz
From https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz...
downloading from https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz ...
static-binaries-0-powerpc64le-linux-gnu.tar.xz 4.4MiB 2.8MiB/s 00:02 [##################] 100.0%
successfully built /gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv
building /gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv...

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash...
download failed "https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash" 404 "Not found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash...
download failed "http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash" 404 "Not Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From https://ci.guix.gnu.org/file/bash/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj...
download failed "https://ci.guix.gnu.org/file/bash/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj" 404 "Not Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From https://tarballs.nixos.org/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj...
download failed "https://tarballs.nixos.org/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj" 404 "Not Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From https://archive.softwareheritage.org/api/1/content/sha256:d29a88a51b507d45f38241739040ee5e45759635c49c6463eac2b7598d3d3cce/raw/...
download failed "https://archive.softwareheritage.org/api/1/content/sha256:d29a88a51b507d45f38241739040ee5e45759635c49c6463eac2b7598d3d3cce/raw/" 404 "Not Found"
failed to download "/gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash" from ("https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash" "http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash")
builder for `/gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv' failed to produce output path `/gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash'
build of /gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv failed
View build log at '/var/log/guix/drvs/ix/4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv.gz'.
cannot build derivation `/gnu/store/n5hf44bybvqmsybjvnn61pkpmcdcrlbd-guile-bootstrap-2.0.drv': 1 dependencies couldn't be built
building /gnu/store/jxh9xn77flxarwzcjga485pgrkjknrgb-tar.drv...
cannot build derivation `/gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv' failed

To resolve this issue, it seems that we must also extract the bootstrap
bash, mkdir, tar, and xz from
static-binaries-0-powerpc64le-linux-gnu.tar.xz and place a copy of each
in the following locations:


After you've uploaded these files, I'll also need to adjust my patch so
it adds alpha.gnu.org to %bootstrap-executable-base-urls in
gnu/packages/bootstrap.scm. That's how Guix finds these raw, unpackaged
binaries.

By the way, I've noticed that the other architectures don't seem to have
"raw" binaries at all on alpha.gnu.org. Maybe you already knew this,
but it seems that the "raw" binaries are actually downloaded from this
specific URL (see the bootstrap-executable-file-name procedure):


Is that intended? I was surprised to discover that we store these four
"raw" binaries in a totally separate place. That seems like it would
make it easy for someone to accidentally forget to update the "raw"
binaries when they update an architecture's bootstrap tarballs.

In any case, for powerpc64le-linux, the right thing to do is probably
just to add copies of these "raw" binaries like I mentioned above.
Could you do that when you have the time?

Thank you,

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl/8KP8ACgkQ3UCaFdgi
Rp0QXRAAs4dUvro3BRO1Mikubz9H9MQPJbHsA4EZaGVCqELvz+Vujdth0OK3yL3A
oq+L6jM+3srljQPKx2YHeMlpXFptJ6FGPoCrst+XHM08XcU1lzBrvVmG7SRDwn+j
4tJYAxYcuvauqzC9+4F+/dpxf4AZq1cFlMF8GS20ghWf4UD/k0CsspwouNBeqQxQ
vg3QK+7H+fZNNX2SLF3H2ddawM3rH3bllMcBF9xCkv8OplNbgVmK/mRGLs14PSjs
iResZBsy/w1BelceZDlenIpgECDbJusTptBvep82RYEF392fmSGBb0HD+JtUKY/y
K97NNrqrTH/AVgl/zjpghfqII7k0cOztFMMYT2sMxys3DKXXegLMet1XodEX2rxb
bsOpumyEiz8h5bs/dgSjdg2mAiOxlKtf6qXvQPKrUM6iTksF+31fMtVEg298wQ5M
2y82NFHsYLiJkYl4QxZhdQeCV+8m40c3rhCJ1/KqJCQgrwVRfK/U0q4JsTehw2hq
18uaEOiEDTEG7NjsdlyfZyLfEEydRvBqaaBbUHBfhoo+y09fSHZQ7C66OGBbKSBg
Kg3jVFUd8gYyHhUd336RPpHhVxYG58riJqechDEmMuBqxdyTcCOkSp88vvZF8Qs6
6l8j9o9zuosc43JLELp6UsqcIrUXXKp11h9/T1jOqfWmxveZ+tA=
=lLE3
-----END PGP SIGNATURE-----

C
C
Chris Marusich wrote on 21 Jan 2021 07:26
(name . Ludovic Courtès)(address . ludo@gnu.org)
87a6t2955r.fsf@gmail.com
Hi Ludo,

I hope you're doing well! If you can find the time to upload those
"raw" binaries, too, we can continue trying to bootstrap. For details,
please refer to my last message, here:


Thank you for your help!

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAmAJHqAACgkQ3UCaFdgi
Rp27ZA//X65VkcV86vXPaS6TnoLoY/s7AxpQ/9sobQNjOftr00ZyWvqa91/eWk0Q
lPMzugzcQRycdtcIxoSAmPv2V05N/fWwoOCuX/lmbc5+2rVxleZFmyxDMu6pL/Ec
Rry59AvweZSBRUhGtmLVy/761IStJZTevf753QITbKlW5nxqsKYNQD4EjbYCuJbz
85PZ79CKdMYwfXUicKcXX8hWRjcPmMbyC9fmr1XrdT6Rlc9wQDmXqTNhgniSv6Ig
gfWdzJ5u6ym2Qq1QAyZjOPKvVIV+MFv/rbCsAb46Bg8UIfyuZqxDwMatwc2KpJQm
rDgojHeD7zlDMIkgqLkMi7OIcf127KzBiADjtD2Oug9LjPxYrhGEeH+FP0OlE14n
JAmiPY7n7NUc1qqgDHQGDI6BN0XQyXvq1618uM6hMEUJ5sQbUjbJ/+UU1pCXzzx9
/VIzxIROB/zBSUo+Ng/3x7MRFiWk+61KjKf9BwZypnvZ7m1InfyqxbqRKYa59vxf
Lvzxa4Wqx3YfiYFcNntwiKilQ/HOyyRp2WQqpBvaGCm5/+7iNd0wKEtuouXWkBIe
+j2poRcsgtg4FCk4q26mdxAEFiUWT2g9YwpPvSq8r+Q7hEl5En3bzpLcMKnbzYzI
PT/MulWLyAe0qhl8n0Tzqj4Nr36GdCrpO6EsGNTOrom7304L6o4=
=M0jp
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 31 Jan 2021 22:35
(name . Chris Marusich)(address . cmmarusich@gmail.com)
87lfc8kcx9.fsf@gnu.org
Hi Chris & all,

Apologies for taking so long to reply! Pinging me on IRC was the right
move. :-)

Chris Marusich <cmmarusich@gmail.com> skribis:

Toggle quote (12 lines)
> I'm afraid I've hit a snag using what you've uploaded, though. It looks
> like we'll need to extract bash, mkdir, tar, and xz from
> static-binaries-0-powerpc64le-linux-gnu.tar.xz and place a copy of each
> in the following locations:
>
> - https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/bash
> - https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/mkdir
> - https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/tar
> - https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/xz
>
> Could you do that? The reason why it's necessary is described below.

I’ve uploaded these four files now.

Toggle quote (20 lines)
> I've started making changes locally to (gnu packages bootstrap). So
> far, with the attached patch, I'm able to do the following things on my
> POWER9 machine running ppc64le Debian (unstable).
>
> I can successfully build Guix from source using Debian packages (and a
> manually-built copy of guile-avahi and guile-gnutls). With my patch, I
> still have to supply the "--with-courage" configure option. Although
> "make" succeeded, "make check" failed on the following tests:
>
> - tests/build-utils.scm
> - test/challenge.scm
> - tests/containers.scm
> - tests/debug-link.scm
>
> The failures fell into two categories:
>
> - Some tests couldn't download the bootstrap bash.
> - In tests/containers.scm, call-with-container evaluated to #f when it
> wasn't supposed to.

Perhaps we can investigate them later.

Toggle quote (5 lines)
> Nevertheless, I created the necessary build users and started the
> guix-daemon via pre-inst-env. I then tried building a simple package:
>
> ./pre-inst-env guix build -e '(@@ (gnu packages bootstrap) %bootstrap-coreutils&co)'

[...]

Toggle quote (6 lines)
> Starting download of /gnu/store/2phdifnfw6i989rqbav04zakxx7qb165-static-binaries-0-powerpc64le-linux-gnu.tar.xz
> From https://ftp.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz...

> failed to download "/gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash" from ("https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash" "http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash")
> builder for `/gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv' failed to produce output path `/gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash'

Note that for testing purposes, you could always work around this by
adding these four files to the store with the ‘add-to-store’ RPC (I
don’t think ‘guix download’ would work because you need the executable
bit.)

Toggle quote (9 lines)
> By the way, I've noticed that the other architectures don't seem to have
> "raw" binaries at all on alpha.gnu.org. Maybe you already knew this,
> but it seems that the "raw" binaries are actually downloaded from this
> specific URL (see the bootstrap-executable-file-name procedure):
>
> https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/?id=44f07d1dc6806e97c4e9ee3e6be883cc59dc666e
>
> Is that intended?

Yes: these files used to be in version control, part of the Guix source
tarball, but now we just download them (commit
836a85da0e8609d40716581be00802ee43463038).

Toggle quote (5 lines)
> I was surprised to discover that we store these four "raw" binaries in
> a totally separate place. That seems like it would make it easy for
> someone to accidentally forget to update the "raw" binaries when they
> update an architecture's bootstrap tarballs.

They’re visible in the dependency graph though, and they don’t need to
ever change.

Ludo’.
C
C
Chris Marusich wrote on 27 Feb 2021 03:39
(address . 41669@debbugs.gnu.org)
87im6e5j4p.fsf@gmail.com
Hi,

Chris Marusich <cmmarusich@gmail.com> writes:

Toggle quote (6 lines)
> Going forward, I'm not sure how best to investigate the inputs to find
> out what's causing the differences. I just know I really, really,
> really don't want to rebuild everything multiple times, since it takes
> hours/days. If you have any creative ideas for how to speed up the
> investigation, I'm all ears.

I've analyzed the graph of derivations that produce differing output
across machines. Here's an image of the graph:
And here is the GraphViz source, which you can render however you want:
I generated the GraphViz source by manually massaging the output of the
following script, which walks the graph of derivations that produce an
output that differs across machines, starting with the derivation that
cross-compiles %gcc-static:
Attachment: analyze.scm
This graph shows the derivations that produce differing outputs across
two Guix System machines, when everything is built without substitutes,
walking the graph of inputs starting from the derivation that
cross-compiles %gcc-static for powerpc64-linux-gnu (from x86_64-linux)
using Guix at commit 1ced8379c7641788fa607b19b7a66d18f045362b. If the
non-reproducibility (across systems) of %gcc-static is caused by a
similarly non-reproducible input, it's probably caused by one or more of
the derivations in this graph. This graph is significantly smaller than
the graph of all derivations, but it still includes 21 derivations.

Some things I noticed while looking at the graph:

- It's suspicious that the module-import-compiled derivations are all at
the bottom. I spot-checked the output of one of the
module-import-compiled derivations on both machines and found that the
.go files differed.

- Every derivation except the root (gcc-static-5.5.0-drv) uses
/gnu/store/0dn61y4n8ig333b23hmc80hvlcy8gdli-guile-bootstrap-2.0.drv as
an input. The root uses
/gnu/store/b5nnbpgkvgdpzgvj67539ylcaqacj90l-guile-3.0.2.drv.

- Strangely,
/gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv is
non-reproducible, even though it just builds a tarball. The tarball
itself does differ on the two systems, but the contents, when
extracted, are identical.

- Some of the derivations in the graph are reproducible when built
locally via "guix build --check"; however, some are not reproducible.
For every derivation, if it was reproducible locally on gs1, it was
also reproducible on gs2, and if it was not reproducible on gs1, it
was also not reproducible on gs2. Here is the full report:

Toggle snippet (47 lines)
rm -f /tmp/mylog && for drv in $(</tmp/drvs); do if guix build --check $drv; then echo OK: $drv >> /tmp/mylog; else echo FAILED: $drv >> /tmp/mylog; fi; done 2>&1 | tee /tmp/mylog-toplevel

...

marusich@gs1 ~$ grep 'may not be deterministic' /tmp/mylog-toplevel
guix build: error: derivation `/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv' may not be deterministic: output `/gnu/store/wgb70ir475v4vhg4bcjzv4h2kaa93qj7-glibc-mesboot-2.16.0' differs
guix build: error: derivation `/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv' may not be deterministic: output `/gnu/store/r1zsxj7wlvw1aa1ifv3nyrrjag44pc9s-xz-mesboot-5.0.0' differs
guix build: error: derivation `/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv' may not be deterministic: output `/gnu/store/mza1k0mgr8ab407yzavwwz07ly7221wy-binutils-mesboot-2.20.1a' differs
guix build: error: derivation `/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv' may not be deterministic: output `/gnu/store/x2iy3ka299sk1x97an783jl4pfbn60pj-bootar-1a' differs
guix build: error: derivation `/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv' may not be deterministic: output `/gnu/store/q7f07gpd9sq9krc68gxa73vbqmk06y0a-gcc-mesboot1-4.6.4' differs
guix build: error: derivation `/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv' may not be deterministic: output `/gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3' differs
guix build: error: derivation `/gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv' may not be deterministic: output `/gnu/store/56km255bk1s880yn6rxw4wj3crm2pshk-gcc-4.9.4.tar.xz' differs
guix build: error: derivation `/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv' may not be deterministic: output `/gnu/store/i4p3mp4sghwbjnn6w9dffpswgc31jhm7-gcc-mesboot0-2.95.3' differs
guix build: error: derivation `/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv' may not be deterministic: output `/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib' differs
guix build: error: derivation `/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv' may not be deterministic: output `/gnu/store/6h5a5sn3bx3wlkw8140ksxymwlksm6iz-glibc-mesboot0-2.2.5' differs
guix build: error: derivation `/gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv' may not be deterministic: output `/gnu/store/9qrvpjwydmqfjm81frrdcn0n3gpn0bs3-gcc-7.5.0.tar.xz' differs
guix build: error: derivation `/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv' may not be deterministic: output `/gnu/store/zpqfz6hvg47xml0yrwfy491drfbgg19p-binutils-mesboot1-2.14' differs
guix build: error: derivation `/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv' may not be deterministic: output `/gnu/store/2601fgjs5h9cpvv5nxrcv52wix29ifrd-binutils-cross-boot0-2.34' differs
guix build: error: derivation `/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv' may not be deterministic: output `/gnu/store/zsz0mj4jdsz4l511d368hwx2d9wrg4my-gash-utils-boot-0.1.0' differs
guix build: error: derivation `/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv' may not be deterministic: output `/gnu/store/zz2blvhk3gdqppcwy089p56f57q7zj0q-gcc-mesboot-4.9.4' differs
guix build: error: derivation `/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv' may not be deterministic: output `/gnu/store/prkqai3zwh3shlqpll6xyncmmqpj49dd-gash-boot-0.2.0' differs
marusich@gs1 ~$ cat /tmp/mylog
FAILED: /gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv
FAILED: /gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv
FAILED: /gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv
FAILED: /gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv
FAILED: /gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv
FAILED: /gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv
FAILED: /gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv
FAILED: /gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv
FAILED: /gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv
FAILED: /gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv
OK: /gnu/store/i5wn3xl6p0zw1vglscgk0bs9dwc6hdh6-gcc-static-5.5.0.drv
FAILED: /gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv
OK: /gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv
FAILED: /gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv
FAILED: /gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv
FAILED: /gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv
FAILED: /gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv
FAILED: /gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv
marusich@gs1 ~$ cat /tmp/mylog | grep ^FAILED | wc -l
16
marusich@gs1 ~$ grep 'may not be deterministic' /tmp/mylog-toplevel | wc -l
16
marusich@gs1 ~$

Although this does not resovle the cross-system reproducibility issue,
it does seem to narrow down the possibilities.

--
Chris
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAmA5sOYACgkQ3UCaFdgi
Rp02jxAApTZbLE9rM+o71dmhWwWnKFYCLp46OFl3YxiiJm0XvLc4J9/yasxXJOnJ
Z+cYNaXiKsfVVO7f2jLxGYQW8bZzJbfjUFy7ds7JWZ93M6JZc1J6Ja8Q+WSL+F3X
LqGOY2jgvLiWSb8hNk+miVYmKRSaOea4wKkyMm0ctkgKqo8p9A6ozbSHqfAWv8RI
dzHliquVfJ5w7MGyE68+fF9gbjpVmtQBsyop70e8uH7giB7vPI3qRyH5v7RnnGJ7
9nqSwIxSQMHuNU5SRvJBfhTXzVtoifiOWLltvJvLAubvFv/nNMpXuBd7JmPIXT/e
wYM0WjLKk7Ka9OU3mJ/WXMwRUxMkjxocuQIxGrhvyOwjI4Zk8LOQRYANnsfFsQFZ
9ZS0ZYj1BEWTjB9APO2XOoVXS1VbzZ7QKC/OeX6Nyv86y5qUIaSOGTPXP+cH5AJg
5A4sXbfWYGHZbrpsjh2qgAEsn39PJSViwyXGvHyCNYUC4c+7zbtwnZY9Dmm+oa15
soHbIwY3dS45XCVl3i6VfhBW89KzWvsGXcymX3kJkXhN3XXklw7DFHtWGfgMrbde
KLhjwqHkJGeuY6cbAcW3UJLiEleU69fObl+lxuspiFQOIA0TjeLXgN3kqm+TNeUb
UEqdyLeRMMeEr6jpwq6NQm/C3rTPNHdvBCnMpUnPzVikumLybeM=
=ot38
-----END PGP SIGNATURE-----

?