Hi all, Chris Marusich writes: > The derivation that produced the differing output was: > > /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv > > On my x86_64-linux system, twice I tried running "guix build --check" on > this derivation, but each time it surprisingly reported no difference. This derivation corresponds to %gcc-bootstrap-tarball from (gnu packages make-bootstrap), which just creates a tarball of the output of %gcc-stripped. Therefore, it's not too surprising that it's reproducible. Similarly, %gcc-stripped just strips some store references from the output of %gcc-static. The %gcc-static package is more interesting: it's where we actually build the statically linked bootstrap GCC (which is then stripped and packed into a tarball as mentioned above), so I thought that it might be where the non-determinism is coming from. However, I haven't yet pinpointed the problem. If you examine the derivations and their inputs, you'll find that they depend upon each other in the following order: guix build --target=powerpc64-linux-gnu -d -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)' /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-stripped)' /gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-static)' /gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv In other words, gcc-static-5.5.0.drv is an input of gcc-stripped-5.5.0.drv, which is an input of gcc-stripped-tarball-5.5.0.drv. Above, I've included example guix commands you can use to obtain each derivation. Using "guix build --check", I confirmed that all three of these derivations build reproducibly on my machine. I hoped to find more information by invoking "guix build --check" on every input of gcc-static-5.5.0.drv. When I tried that, what I found was that all of its inputs build reproducibly except the following two: /gnu/store/x32cnfkd50fnxs10xp1jdn24h7ai2gxr-guile-3.0.2.drv /gnu/store/g9fpkg2qa27mka1znqsvx8vxqyabsj2y-gcc-7.5.0.drv I haven't investigated guile-3.0.2.drv. However, gcc-7.5.0.drv felt more suspicious to me, and it is actually the derivation that builds gcc-final from (gnu packages commencement), which you can see via: guix build -d -e '(@@ (gnu packages commencement) gcc-final)' Using "guix gc", I deleted the outputs of gcc-stripped-tarball-5.5.0.drv, gcc-stripped-5.5.0.drv, gcc-static-5.5.0.drv, and gcc-7.5.0.drv. I then tried building these four derivations again (without substitutes, the same as before). Before doing this, I stored the SHA-512 hashes of their output files, and after the build succeeded, I compared the hashes of the new files with the previous values. I found that these derivations' newly rebuilt outputs were identical to their original values, except for gcc-7.5.0.drv, which produced some different files. Most significantly, this means that gcc-stripped-tarball-5.5.0.drv produced the exact same tarball for me as it did the first time I built it, even though some of its inputs are not themselves reproducible. At present, it seems possible that within the context of a single machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a different machine, it may (reproducibly) build a different output. I'm a bit paranoid about making mistakes, so I'll perform another full GC and then try yet again to build gcc-stripped-tarball-5.5.0.drv in order to verify whether it truly produces the same output when all (or nearly all) of its inputs are rebuilt from scratch. Some people have also shared their differing copies of the binaries on the email list. It could be productive to compare the contents with diffoscope, although I suspect the diff might be too large to be useful. -- Chris