Build for Julia is not reproducible

Ricardo Wurmus wrote on 4 Jan 2016 17:28

Recipients:(address . bug-guix@gnu.org)

Message-ID:idj1t9xia11.fsf@bimsb-sys02.mdc-berlin.net

When building “julia” twice we get different binaries for
  $out/lib/julia/sys.so
and
  $out/lib/julia/sys-debug.so
I could not determine the exact differences, because instead of a diff Ijust see something like this:
--- /gnu/store/6njfyp46b22b0wb0r0ccg2wdzjx7066g-julia-0.4.2/lib/julia/sys.so+++ 6njfyp46b22b0wb0r0ccg2wdzjx7066g-julia-0.4.2/lib/julia/sys.so├── objdump --disassemble --full-contents {}│ @@ -99994,8 +99994,8 @@│   186a00 05a3d38a 01488985 38ffffff 488b0515  .....H..8...H...│   186a10 5b8b0148 898540ff ffff4c89 ad48ffff  [..H..@...L..H..│   186a20 ff31ffba 03000000 4c8db538 ffffff4c  .1......L..8...L│   186a30 89f64989 f4e8b653 f0ff4889 8538ffff  ..I....S..H..8..│   186a40 ff31ffba 03000000 488d9d28 ffffff49  .1......H..(...I│   186a50 89de4c89 f6e89653 f0ff4889 8528ffff  ..L....S..H..(..│   186a60 ff488b05 38d08a01 48898530 ffffff48  .H..8...H..0...H│ -[ Too much input for diff ]                     │ +[ Too much input for diff ]                         
...
--- /gnu/store/6njfyp46b22b0wb0r0ccg2wdzjx7066g-julia-0.4.2/lib/julia/sys-debug.so+++ 6njfyp46b22b0wb0r0ccg2wdzjx7066g-julia-0.4.2/lib/julia/sys-debug.so├── objdump --disassemble --full-contents {}│ @@ -99994,8 +99994,8 @@│   1869f0 01488b38 be370200 00e84257 f0ffe89d  .H.8.7....BW....│   186a00 56f0ff66 6666662e 0f1f8400 00000000  V..ffff.........│   186a10 554889e5 488b3e48 8b7608e8 e0fdffff  UH..H.>H.v......│   186a20 4889c7e8 2856f0ff 5dc3660f 1f440000  H...(V..].f..D..│   186a30 554889e5 4883ec10 64488b04 25280000  UH..H...dH..%(..│   186a40 00488945 f848833d fb4d8d01 00740948  .H.E.H.=.M...t.H│   186a50 8b0df24d 8d01eb1f 488d3591 ac190048  ...M....H.5....H│ -[ Too much input for diff ]                     │ +[ Too much input for diff ]                         ╵
~~ Ricardo

Ludovic Courtès wrote on 5 Jan 2016 23:30

Recipients:(name . Ricardo Wurmus)(address . ricardo.wurmus@mdc-berlin.de)(address . 22304@debbugs.gnu.org)

Message-ID:8760z7k6av.fsf@gnu.org

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

Toggle quote (8 lines)

> When building “julia” twice we get different binaries for>> $out/lib/julia/sys.so>> and>> $out/lib/julia/sys-debug.so

Apparently there’s no __DATE__ and __TIME__ in the C code.
I would check if the order of object files on the command line thatlinks those shared objects is always the same.
Ludo’.

Ricardo Wurmus wrote on 4 Feb 2016 15:51

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22304@debbugs.gnu.org)

Message-ID:idjzivg8r5k.fsf@bimsb-sys02.mdc-berlin.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (15 lines)> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:>>> When building “julia” twice we get different binaries for>>>>   $out/lib/julia/sys.so>>>> and>>>>   $out/lib/julia/sys-debug.so>> Apparently there’s no __DATE__ and __TIME__ in the C code.>> I would check if the order of object files on the command line that> links those shared objects is always the same.

As far as I could tell the order of shared objects in two consecutiveruns is the same.
~~ Ricardo

zimoun wrote on 28 Nov 2019 18:14

bug#22304: Julia not reproducible

Recipients:

Message-ID:CAJ3okZ3i90aWjZgZmvZSNad6XbWoRqBsk6KQnxZXaWO8Tq53eQ@mail.gmail.com

Hi,
This bug [1] is still present even with the version 1.1.1 of Julia.
[1] http://issues.guix.gnu.org/issue/22304

Toggle snippet (8 lines)

guix describeGeneration 57 Nov 25 2019 14:26:15 (current) guix b5d4d5b repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f

Toggle snippet (10 lines)

guix build julia # populate the store with dependenciesguix build julia --check -K --no-grafts[...]guix build: error: derivation`/gnu/store/xdy3jjz7dzg1lr231b8zbss8xn44ldjj-julia-1.1.1.drv' may notbe deterministic: output`/gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1' differs from‘/gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check’

Toggle snippet (23 lines)diff -r /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-checkBinary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/lib/julia/sys.soand /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/lib/julia/sys.sodifferBinary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Bar/HXSAn.jiand /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Bar/HXSAn.jidifferBinary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Baz/rONVA.jiand /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Baz/rONVA.jidifferBinary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Foo/MYb1d.jiand /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Foo/MYb1d.jidifferBinary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Foo/TeeT6.jiand /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Foo/TeeT6.jidifferBinary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.jiand /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.jidiffer

Well, then I try to explore with diffoscope but hum? I do not docorrectly because the resulting diff is 845MB ouch!

All the best,simon

Ludovic Courtès wrote on 29 Nov 2019 17:19

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:874kymsli5.fsf@gnu.org

Hi,
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (8 lines)

> Binary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.ji> and /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.ji> differ>>> Well, then I try to explore with diffoscope but hum? I do not do> correctly because the resulting diff is 845MB ouch!

Pro tip: just run
diffoscope \ /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1{,-check}/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.ji
That’ll already be enough info for a start!
Ludo’.

zimoun wrote on 2 Dec 2019 18:28

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:CAJ3okZ0ZZQVqLWWYg=M9sHCfyScy=S9hyraydUbrF2GQwN5hPg@mail.gmail.com

Hi Ludo,
On Fri, 29 Nov 2019 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)

> Pro tip: just run

I think that I have already did the pro tip. :-)

Toggle snippet (7 lines)

diffoscope /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1{,-check}/lib/julia/sys.so\ > /tmp/sys.diffoscopedu -sh /tmp/sys.diffoscope845M /tmp/sys.diffoscope

Maybe I am doing incorrectly...
If I understand well [1], the file .ji are "precompiled" (analogy: .pyc). Hum?!
[1] https://docs.julialang.org/en/v1/devdocs/sysimg/

Cheers,simon

zimoun wrote on 24 Jan 2020 19:30

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:CAJ3okZ0Yx8+dveO5ooQUBkJe0jAwZHZuwk5rj_c1LKXYXberhg@mail.gmail.com

Hi,
On Mon, 2 Dec 2019 at 18:28, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (2 lines)

> If I understand well [1], the file .ji are "precompiled" (analogy: .pyc). Hum?!

The non-determinism seems coming from these "precompiled" .ji filesand 2 bugs are opened upstream.
https://github.com/JuliaLang/julia/issues/25900https://github.com/JuliaLang/julia/issues/34115
Thanks to Nicolò to pointing out these [1] when working on the version update.
[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38546#80
With lengthy and clear details. :-)

Let start the container with everything useful.

Toggle snippet (8 lines)

mkdir -p /tmp/test-julia && cd /tmp/test-julia
guix time-machine --commit=b5d4d5b \ environment -u nixbld --no-cwd --container --network \ --expose=$(realpath determinism) --share=/tmp/test-julia \ --ad-hoc julia coreutils nss-certs curl tar gzip

Then, the path to reproduce is odd because there is no Julia packagesin Guix. So we need to use 'Pkg' from Julia which downloads fromelsewhere and the reproducibility of such is not well controlled butit gives an idea. :-)

Toggle snippet (18 lines)

export SSL_CERT_DIR="$GUIX_ENVIRONMENT/etc/ssl/certs"export SSL_CERT_FILE="$SSL_CERT_DIR/ca-certificates.crt"export GIT_SSL_CAINFO="$SSL_CERT_FILE"
# populate the Julia package managerjulia --startup-file=no -q -e 'using Pkg; Pkg.instantiate()'
# install somethingjulia --startup-file=no -q -e 'using Pkg; Pkg.add("Compat")'
# use something / precompiled itjulia --startup-file=no -q -e 'using Compat'
# check the .ji filemd5sum ~/.julia/compiled/v1.1/Compat/*

Then each time one runs this block, one gets different hash.

All the best,simon

zimoun wrote on 13 May 2020 20:44

Julia v1.4.1 still not reproducible

Recipients:

Message-ID:CAJ3okZ0_SORCsKtY3vvPnT3o+NOYwovHO+v9E6gcG=68ACEyfQ@mail.gmail.com

Hi Efraim,
On Wed, 13 May 2020 at 13:04, Efraim Flashner <efraim@flashner.co.il> wrote:

Toggle quote (5 lines)

> guix build: error: derivation `/gnu/store/z21c12ijspavaclcins5pja8z1s8wbm0-julia-1.4.1.drv' may not be deterministic: output `/gnu/store/ri931fk8hf5wvagk76zhmwbphq2pwgnp-julia-1.4.1' differs from ?/gnu/store/ri931fk8hf5wvagk76zhmwbphq2pwgnp-julia-1.4.1-check?>> I built it on bayfront twice. I'll get the diffoscope output once it> finally builds.

Thank you.
Cheers,simon

Nicolò Balzarotti wrote on 17 May 2020 15:55

Recipients:(address . 22304@debbugs.gnu.org)

Message-ID:87v9kuwtrp.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me

Ops, I sent it to 22304@debuugs.gnu.org instead of 22304@debbugs.gnu.org

To: Efraim Flashner <efraim@flashner.co.il>, zimoun <zimon.toutoune@gmail.com>Cc: 22304@debuugs.gnu.orgIn-Reply-To: <20200517134039.GF31833@E5400>References: <873688i57c.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me> <CAJ3okZ0mhtva5ePoYZLYKrUfQeYHvVaC3c5qTBs2WGjZucFT0w@mail.gmail.com> <20200513110416.GE918@E5400> <CAJ3okZ3hjN6umT7bapg1CU4k+pChEio1rk78M0uz5j18oucbNA@mail.gmail.com> <20200517134039.GF31833@E5400>Message-ID: <87y2pqwtyz.fsf@guixSD.i-did-not-set--mail-host-address--so-tickle-me>MIME-Version: 1.0Content-Type: multipart/mixed; boundary="====-=-="
--====-=-=Content-Type: text/plainContent-Disposition: inline
Hi Efraim, Simon,
The attached patches provide Simon's suggested julia-deterministicbuild.
With this, I get deterministic julia build, *but* only on a singlemachine. Building on a different machine I get a different output. Istill need to investigate why. Also, some of the patches are probablynot needed at all. In the near future, as discussed previously, I aimat submitting required patches upstream.
I disabled tests in julia-deterministic, so it get built quite fast(~10minutes). If you want to try to see if this is reproducible//knowhow to figure out what might be happening on different machines,feedback is wellcome!
My ouptut is (patches applied on 75741af9b2):guix hash -r /gnu/store/mp8hl7j09nfnsxbb3a6ka23ww44gq0z2-julia-deterministic-1.4.11k6mak38nkiaj0b9l6rcmxfcjwvbijmczxs703hgawwi6z0jikxl

--====-=-=Content-Type: text/x-patchContent-Disposition: attachment; filename=0001-gnu-julia-Specify-correct-llvm-version.patch
From baf3f3853079ae375342f626516945998c15f1ca Mon Sep 17 00:00:00 2001
--- gnu/packages/julia.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (145 lines)diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scmindex c636fa756b..8db5e66131 100644--- a/gnu/packages/julia.scm+++ b/gnu/packages/julia.scm@@ -416,7 +416,7 @@ libraries.  It is also a bit like @code{ldd} and @code{otool -L}.")                         (assoc-ref %build-inputs "utf8proc")                         "/include")          "USE_SYSTEM_LLVM=1"-         "LLVM_VER=6.0.1"+         "LLVM_VER=8.0.0"           "USE_LLVM_SHLIB=1"          "USE_SYSTEM_LIBUNWIND=1"-- 2.26.2

--====-=-=Content-Type: text/x-patchContent-Disposition: attachment; filename=0002-gnu-Add-julia-deterministic.patch
From fc0bd3f4e8d83898e64fe8cae8c4c61defa7ba4d Mon Sep 17 00:00:00 2001From: nixo <nicolo@nixo.xyz>Date: Sat, 16 May 2020 11:56:26 +0200Subject: [PATCH 2/2] gnu: Add julia-deterministic.
* gnu/packages/julia.scm (julia-deterministic): New variable.--- gnu/packages/julia.scm | 64 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+)
diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scmindex 8db5e66131..0b4cb82966 100644--- a/gnu/packages/julia.scm+++ b/gnu/packages/julia.scm@@ -493,3 +493,67 @@ computing environments.  It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library.")     (license license:expat)))++;; Deterministic build of the julia executable+(define-public julia-deterministic+  (package+    (inherit julia)+    (name "julia-deterministic")+    (source+     (origin+       (inherit (package-source julia))+       ;; I wrote this patches to get a deterministic build.+       ;; We need to discuss the best way to add them upstream.+       (patches+        (list+         (origin ; timefuncs: support SOURCE_DATE_EPOCH for reproducible builds+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/1c56fb0975c7701596b5bdc9ad4e716a3e2fcfc9.patch")+           (sha256 (base32 "0zw5554s0k38kgq8bgyxcjflrbjk6xqfrh8kidwsh6zvmpkmgnf5")))+         (origin                ; don't use jl_hrtime to store module build_id+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/c13691df6747849dec8f654180dabc55e8044a64.patch")+           (sha256 (base32 "0n7mnbwzr411j9z557jrcib5vr8v8mgqbj5xdrxv7b2n0mrfcp5d")))+         (origin ; contrib: generate_precompile: do not use random temporary files+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/ae2929a5d3d1f724b421a7f04b6c7b84c9987583.patch")+           (sha256 (base32 "00vpj4m9f4wzfkvd4flarki58jbfhi0zngw35zrhaph53mfi6y27")))+         ;; (origin                        ; base: Base: deterministic random+         ;;   (method url-fetch)+         ;;   (uri "https://github.com/nico202/julia/commit/49a4edc26e9db110586c590c17df276301c344a7.patch")+         ;;   (sha256 (base32 "16p4axwr3shf2jp6hq8dk19c9s1l667ki5a8bv3dbv5b0ji9w7ra")))+         (origin                        ; remove some use of time()+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/5561587256b29761fe3a7d8fac169b4305afa240.patch")+           (sha256 (base32 "0s9b90x61jjrc057myqdck52yj4lb4xlm7ypjn5qvpypnxrbkz6b")))+         (origin                  ; contrib: generate_precompile: bare minimum+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/5cd454695e7f6a7e448c19d8f9b2d41fcdeb812e.patch")+           (sha256 (base32 "0c9pz9ligd6wfhvra5wn7k2k2qr1zw3x67b2y9dhf108xf4z79sm")))+         (origin          ; src: codegen: don't dump time to precompile stream+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/5445b5dc2ea30f2f39acb0fb45ba552122a7e6f5.patch")+           (sha256 (base32 "0sa1a73ajk8cm3a5l25hyc2bg8kq6l16r2vvdn2qbcnpnjff9skx")))+         (origin                        ; do not import repl+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/9a45d44ba387d7386b14abfe76e18f1f000eecc2.patch")+           (sha256 (base32 "0k6yyxr8rd3dcj8rh32pjzlyc0izvi34xglmxsdhqq6q3j0rf0wb")))+         (origin                        ; no more interactive+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/e78f791e048ed5192d6cb3166fe089207e2e994d.patch")+           (sha256 (base32 "1bvgiishhc0csypymknff46cik5dnanyx4q5kgipjk0a47wi80x2")))+         (origin                        ; generate precompile: sort modules+           (method url-fetch)+           (uri "https://github.com/nico202/julia/commit/e5359e1ce383d549db87659ecd72dcb615eb649d.patch")+           (sha256 (base32 "0slw4jczba88hpqyd0gq9x7jnw75yvq8wmbrl7dd8f83jmci8lzz")))))))+    (arguments+     (substitute-keyword-arguments (package-arguments julia)+       ((#:phases phases '%standard-phases)+        `(modify-phases ,phases+           (delete 'check)+           (add-after 'unpack 'generic-target+             (lambda _+               ;; Not sure is needed/is the best target+               (substitute* "contrib/generate_precompile.jl"+                 (("native") "generic"))+               #t))))))))-- 2.26.2

--====-=-=Content-Type: text/plain; charset=utf-8Content-Disposition: inlineContent-Transfer-Encoding: quoted-printable

Thanks, Nicol=C3=B2
Efraim Flashner <efraim@flashner.co.il> writes:
> On Wed, May 13, 2020 at 08:43:08PM +0200, zimoun wrote:>> Hi Efraim,>>=20>> On Wed, 13 May 2020 at 13:04, Efraim Flashner <efraim@flashner.co.il> wr=ote:>>=20>> > guix build: error: derivation `/gnu/store/z21c12ijspavaclcins5pja8z1s8=wbm0-julia-1.4.1.drv' may not be deterministic: output `/gnu/store/ri931fk8=hf5wvagk76zhmwbphq2pwgnp-julia-1.4.1' differs from ?/gnu/store/ri931fk8hf5w=vagk76zhmwbphq2pwgnp-julia-1.4.1-check?>> >>> > I built it on bayfront twice. I'll get the diffoscope output once it>> > finally builds.>> I wasn't able to provide the output of diffoscope, it turns out that the> output was too large to attach.>> --=20> Efraim Flashner   <efraim@flashner.co.il>   =D7=90=D7=A4=D7=A8=D7=99=D7==9D =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8> GPG key =3D A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351> Confidentiality cannot be guaranteed on emails sent or received unencrypt=ed
--====-=-=--

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date