Update Julia to 1.3.1.

Hello Guix!

Changes:

This set of patches improve the state of julia-build-system.

1. set path JULIA_LOAD_PATH to enable installing packages, and

JULIA_DEPOT_PATH to allow guix-precompiled cache to be recognized by julia. To

do this, a small "hack" with libfaketime was required (julia decides if the

cache is valid by comparing source ".jl" mtime with cache ".ji" file, in which

julia stores a timestamp. We need to force this timestamp to be equal to the

mtime of guix-store files).

2. Enable package tests

3. Add a first package (julia-compat) definition. If this is fine, I'll start

pushing many many more (I have them ready, but I need to update many of them)

Thanks! :)

One note: with this PATH changes, julia will not be able to find packages

installed by Julia Pkg3.jl anymore. This is expected (it should happen for R,

python, emacs... too) but since we still need to merge the packages, this

patch renders julia pretty unusable. The workaround is for users to add to

JULIA_LOAD_PATH and JULIA_LOAD_PATH local user-writable folders, activate a

profile, and then install packages manually:

#+begin_src bash

JULIA_LOAD_PATH=/tmp/julia:$JULIA_LOAD_PATH JULIA_DEPOT_PATH=/tmp/julia:$JULIA_DEPOT_PATH julia

#+end_src

nixo (3):

gnu: julia: Set to JULIA_LOAD_PATH and JULIA_DEPOT_PATH variables.

gnu: julia-build-system: Enable tests.

gnu: julia-xyz: Add julia-compat.

#+begin_src julia

using Pkg

Pkg.activate("/tmp/julia")

Pkg.add("PACKAGE_NAME")

#+end_src

--

2.24.0

gnu/packages/julia-xyz.scm | 62 +++++++++++++

gnu/packages/julia.scm | 146 +++++++++++++++---------------

guix/build-system/julia.scm | 4 +-

guix/build/julia-build-system.scm | 28 ++++--

4 files changed, 159 insertions(+), 81 deletions(-)

create mode 100644 gnu/packages/julia-xyz.scm

gnu/packages/julia.scm | 146 +++++++++++++++++++++--------------------

1 file changed, 75 insertions(+), 71 deletions(-)

---

guix/build-system/julia.scm | 4 +++-

guix/build/julia-build-system.scm | 28 +++++++++++++++++++---------

2 files changed, 22 insertions(+), 10 deletions(-)

---

gnu/packages/julia-xyz.scm | 62 ++++++++++++++++++++++++++++++++++++++

1 file changed, 62 insertions(+)

create mode 100644 gnu/packages/julia-xyz.scm

Hi!

nixo <anothersms@gmail.com> skribis:

These two environment variables are supported upstream, right?

OK. This hack shouldn’t break “normal” Julia uses (outside of the

store), right?

Ludo’.

Nicolò Balzarotti <anothersms@gmail.com> skribis:

Please make sure to add a commit log listing all the entities changed.

Apparently there was some reindentation that makes this patch looks big

and makes the actual changes harder to spot. Could you resubmit it

without the reindentation?

Why is this extra entry needed? Doesn’t Julia find its own standard

library by default, even if JULIA_LOAD_PATH is unset?

What’s JULIA_DEPOT_PATH supposed to point to again? ‘share/julia’ seems

to be very much a catch-all thing. :-)

Thanks,

Ludo’.

Nicolò Balzarotti <anothersms@gmail.com> skribis:

With a commit log, please. :-)

Do you think it would be feasible to modify Julia to emit this fixed

timestamp, or to honor SOURCE_DATE_EPOCH, instead of using ‘faketime’?

The reason I’m asking is that (1) ‘faketime’ relies on LD_PRELOAD and so

it could introduce its own set of issues, and (2) I wouldn’t be

surprised to find ready-to-use SOURCE_DATE_EPOCH patches from fellow

Reproducible Builders floating around. :-)

Ludo’.

Nicolò Balzarotti <anothersms@gmail.com> skribis:

The second line is unnecessary.

However, please make sure to add the file to ‘gnu/local.mk’.

Otherwise LGTM.

Thanks!

Ludo’.

Ok I had time to look at this again.

1. Package installation can be done both the julia way and the guix way

2. SOURCE_DATE_EPOCH is respected in package precompile timestamp. Other fixes are needed to get reproducible builds, but for now that means that package cache is used.

3. This patch adds some first julia package (HTTP.jl and dependencies)

4. Updated julia to 1.3.1 (latest release). It's working great and the

build recipe is simplified a lot. However, some test is failing. The

"interesting" part is that when running tests alone, they are working.

It seems like the problem arise when tests are run on the _same core_

and not on different cores. I opened a issue upstream for this

(https://github.com/JuliaLang/julia/issues/34330),I'll have to look

further on it, but for now if it's fine for me this is ready to be merged.

I'm sending the patch series here.

Thanks, Nicolò

Ludovic Courtès <ludo@gnu.org> writes:

---

gnu/local.mk | 1 +

gnu/packages/julia.scm | 4 ++-

...ia-fake-mtime-with-SOURCE_DATE_EPOCH.patch | 29 +++++++++++++++++++

3 files changed, 33 insertions(+), 1 deletion(-)

create mode 100644 gnu/packages/patches/julia-fake-mtime-with-SOURCE_DATE_EPOCH.patch

---

gnu/packages/julia.scm | 216 ++++++++++++++++++++---------------------

1 file changed, 108 insertions(+), 108 deletions(-)

---

gnu/packages/julia.scm | 25 +++++++++++++++++++++----

1 file changed, 21 insertions(+), 4 deletions(-)

precompilation

---

guix/build/julia-build-system.scm | 51 +++++++++++++------------------

1 file changed, 22 insertions(+), 29 deletions(-)

---

gnu/local.mk | 1 +

gnu/packages/julia-xyz.scm | 27 +++++++++++++++++++++++++++

2 files changed, 28 insertions(+)

create mode 100644 gnu/packages/julia-xyz.scm

---

gnu/packages/julia-xyz.scm | 20 ++++++++++++++++++++

1 file changed, 20 insertions(+)

---

gnu/packages/julia-xyz.scm | 23 +++++++++++++++++++++++

1 file changed, 23 insertions(+)

---

gnu/packages/julia-xyz.scm | 33 +++++++++++++++++++++++++++++++++

1 file changed, 33 insertions(+)

---

gnu/packages/julia-xyz.scm | 49 +++++++++++++++++++++++++++++++++++++-

1 file changed, 48 insertions(+), 1 deletion(-)

---

gnu/packages/julia-xyz.scm | 24 ++++++++++++++++++++++++

1 file changed, 24 insertions(+)

---

gnu/packages/julia.scm | 278 +++++++++++++++++------------------------

1 file changed, 113 insertions(+), 165 deletions(-)

---

gnu/packages/julia.scm | 278 +++++++++++++++++------------------------

1 file changed, 113 insertions(+), 165 deletions(-)

Ok I had time to look at this again.

1. Package installation can be done both the julia way and the guix way

2. SOURCE_DATE_EPOCH is respected in package precompile timestamp. Other fixes are needed to get reproducible builds, but for now that means that package cache is used.

3. This patch adds some first julia package (HTTP.jl and dependencies)

4. Updated julia to 1.3.1 (latest release). It's working great and the

build recipe is simplified a lot. However, some test is failing. The

"interesting" part is that when running tests alone, they are working.

It seems like the problem arise when tests are run on the _same core_

and not on different cores. I opened a issue upstream for this

(https://github.com/JuliaLang/julia/issues/34330),I'll have to look

further on it, but for now if it's fine for me this is ready to be merged.

I'm sending the patch series here.

Thanks, Nicolò

Ludovic Courtès <ludo@gnu.org> writes:

I'm sending the patches again as attachments since I think some of them

was badly formatted.

Sorry again, I fixed some unrelated changes that slipped in

Thanks, Nicolò

Hi Nicolò,

Thank you for working on this.

Julia rocks! ;-)

However, I am lost with your patch set.

I have tried only the first patch you attached. I hit this error:

So something is missing. :-)

Then I notice that you change the list of 'patches' and you "forgot"

to send them too. :-)

Almost related, does this update fix the reproducibility bug [1]? For example,

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22304

All the best,

simon

Hi Simon,

Sorry, I forgot to send the dsfmt patch. Also, julia's

SOURCE_DATE_EPOCH patch was named differently. I've fixed this in theattached patches. You need to apply Add-dsfmt.patch, Update-to-1.3.1

and then julia-use-SOURCE_DATE_EPOCH.

About reproducibility: if I'm not wrong, sys.so contains Base library

precompiled ([1]). Precompilation is still non deterministic (here's

[2] an issue on github). Something I did to check precompilation:

Since I'm tired of this, I've started investigating better.

Loaded a guix container

#+begin_src bash

mkdir -p /tmp/jlprecompiled/

./pre-inst-env guix environment -u nixbld --no-cwd --container --network --expose=$(realpath determinism) --share=/tmp/jlprecompiled/ --ad-hoc julia coreutils nss-certs curl tar gzip libfaketime

#+end_src

from this container, run some time this:

#+begin_src bash

export SSL_CERT_DIR="$GUIX_ENVIRONMENT/etc/ssl/certs"

export SSL_CERT_FILE="$SSL_CERT_DIR/ca-certificates.crt"

export GIT_SSL_CAINFO="$SSL_CERT_FILE"

SOURCE_DATE_EPOCH=1 julia --startup-file=no -q -e 'using Pkg; Pkg.instantiate()'

SOURCE_DATE_EPOCH=1 faketime '2008-12-24 08:15:42' julia --startup-file=no -q -e 'using Random; using Pkg; Pkg.add("Compat"); Random.seed!(0); using Compat'

cp -r ~/.julia/compiled/v1.3/Compat/ /tmp/jlprecompiled/$RANDOM

sha256sum ~/.julia/compiled/v1.3/Compat/*

# 6c10d65f6cd9cdd5b98e4d5e2b82b7bd00508f0daccef0967dec92cb6468052e /home/nixbld/.julia/compiled/v1.3/Compat/GSFWK_Pua4I.ji

ls -lah ~/.julia/compiled/v1.3/Compat/*

# -rw------- 1 nixbld users 102K Jan 21 07:39 /home/nixbld/.julia/compiled/v1.3/Compat/GSFWK_Pua4I.ji

#+end_src

I could not get the same results twice (also, size differs). I'll work

on this on some spare time (for example, there are other places where

SOURCE_DATE_EPOCH can be used, but this [3] is a problem I need to

solve first).

Maybe 1.3.1 (when reviewed) can be merged, since we have

the same problem with julia 1.1, but we can wait for the

source-date-epoch and julia-xyz patches until we solve this.

What do you think?

Thanks, Nicolò

[1] https://docs.julialang.org/en/v1/devdocs/sysimg/

[2] https://github.com/JuliaLang/julia/issues/25900

[3] https://github.com/JuliaLang/julia/issues/34115

zimoun <zimon.toutoune@gmail.com> writes:

Hi Nicolò,

Cool that you figured out a source of non-reproducibility.

On Tue, 21 Jan 2020 at 14:45, Nicolò Balzarotti <anothersms@gmail.com> wrote:

This patch 'julia-SOURCE_DATE_EPOCH-mtime.patch' is the one you

mentioned here [#], right?

Could you send it as an upstream PR?

[#] https://github.com/JuliaLang/julia/issues/34115#issuecomment-568171025

I am not sure to well understand the source of non-determinism.

Does the patch about SOURCE_DATE_EPOCH fix the issue of [1] and [2]?

Or is it something else?

Is the problem [3] not solved by 'julia-SOURCE_DATE_EPOCH-mtime.patch'?

My opinion is: if a patch is floating around to fix the

source-date-epoch issue, let try to push it upstream. If it is

rejected, let talk later if Guix will include it or not. And in the

meantime, I will try to review the 1.3.1 because yes I agree that it

should be included even if we know it is not reproducible -- the

package Guitarix [@] is updated and not reproducible neither.

[@] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21803

Thank you for working on this.

All the best,

simon

Hi Simon,

zimoun <zimon.toutoune@gmail.com> writes:

The first patch (the one I mention in [#]) fixes _one source of_

non-determinism (the stored mtime inside the precompile cache). But

others are present (precompiling the same file 2 times lead to

differences even in file size, something I reported on [2]). What I'm

working on is trying to find out what other sources are and patch them,

too. I patched some c code (src/support/timefuncs.c) so that it follows

SOURCE_DATE_EPOCH too. This finally lead to same-size files. I'm now

working on parsing binary (.ji) files directly with julia built in

function to have a textual representation of the differences.

I can't send SOURCE_DATE_EPOCH patches upstream until [3] is merged, as

patching timefuncts leads to an endless test suite (as they were using

time() to check for passed time. If time() always returns 1, tests

sleeps forever). Yesterday evening I completed that patch (and has been

approved). My plans are on finding the hopefully last source of

non-determinism, patch it and send everything upstream (hoping they

care).

Thanks! I hope in a 1.4 release where everything is fixed ;)

Nicolò

Hi Nicolò,

On Wed, 22 Jan 2020 at 10:59, Nicolò Balzarotti <anothersms@gmail.com> wrote:

Thank you for the explanations. Cool!

Awesome if upstream are responsive.

I have seen that 1.4-rc1 is just released. Hope that everything about

reproducibility will be fixed. ;-)

I have not yet tried your patches. Sorry bit busy at work.

And I will not have any slot next week, so I will be back on the topic

after the Guix Days. Except if someone beats me. :-)

(see you there :-)

Thank you for improving the Julia ecosystem in Guix because Julia is

becoming quite popular.

Cheers,

simon

Hi Nicoló,

On Fri, 24 Jan 2020 at 18:26, zimoun <zimon.toutoune@gmail.com> wrote:

I have tried them. Ouch! The tests eat so much...

Do you have a special motivation to rename 'llvm-patch' to 'julia-patch'?

(I agree, it is better :-))

Well, only 2 inputs are missing. Do you think it is affordable to pack

them and so remove the commentary?

Otherwise, there is some issues about indentation -- I think you use

Emacs, so check that '.dir-locals.el' is correctly setup -- double

space for sentences in description,

'patch/julia-SOURCE_DATE_EPOCH-mtime.patch' should be added with the

same commit that "Update", etc.

I have half-corrected so I can send you these 2 v2-patches if you want to. :-)

And I have not tried yet to build with '--check'.

All the best,

simon

Hi Nicoló,

On Tue, 4 Feb 2020 at 19:03, zimoun <zimon.toutoune@gmail.com> wrote:

The tests fail on my desktop machine with 8GB of RAM.

Do they pass on your machine?

Still not reproducible as expected [1]. :-)

[1] http://issues.guix.gnu.org/issue/22304

Cheers,

simon

ps:

I think you are busy by your thesis. Cross the fingers. ;-)

Hi Nicoló,

I have seen your message on IRC.

Cool for the release 1.4.

Could we try to include first this 1.3 one? Then push the 1.4 if it

is already ready.

Because then, this release will be usable "forever" (or almost ;-)

even if it is not perfect and issues are fixed with the 1.4.

I mean, once a package is inside Guix, then it becomes "easily" usable

with 'guix time-machine' and saved on Software Heritage.

Therefore, considering the big picture of Scientific Reproducibility,

it matters -- for the future to be able to travel back -- to also have

1.3. ;-)

I think we are almost done. From my point of view, we just need to

address (or comment) these 3 points:

1.

2.

3.

What do you think?

Cheers,

simon

zimoun <zimon.toutoune@gmail.com> writes:

Hi,

Ok, let's do both

Yes, because now it's not only used by llvm-julia but also by the new

libunwind-julia. So calling llvm-patch a function that takes a patch

from the julia repo and that is applied to a "julia module" is measleading.

I indeed tried. The main problem is that in the Makefile there's no

USE_SYSTEM_{OBJCONV/LIBWHICH}, and I didn't want to mess with the it.

But yes, I think I could at least try again.

If you already fixed them, yes please send a v2 patch and I'll continue

working from there.

Fine for me, do we need also to debug a broken test right?

Nicolò

On Tue, 11 Feb 2020 at 16:10, Nicolò Balzarotti <anothersms@gmail.com> wrote:

Ok.

Let me check how it will be crawled by SWH to be sure that Guix will

be able to still build Julia 1.3 even if GitHub does not exist

anymore.

And I am in favour to let these 2 inputs.

Stay tuned. :-)

I try to send it today...

I do not know if the test are broken. I am not able to run them on my

desktop machine. Not enough memory.

Cheers,

simon

zimoun <zimon.toutoune@gmail.com> writes:

Hi

Don't worry, I'm still working on my PhD thesis so my spare time until

the end of the week is low :D

Ok, so I'll run tests a few times on my server then

Thanks, Nicolò

retitle 38546 Update Julia to 1.3.1.

thanks

Hi,

Please find attach the almost updated julia using 2 patches. The first

add a dependency and the second update and patch.

I hope that I do not screw up the indentation; sometimes my Emacs does

weird stuff. Anyway.

If it is fine with you, please ping Ludo or any committer. :-)

Cheers,

simon

Hi,

Friendly ping. :-)

On Tue, 11 Feb 2020 at 18:58, zimoun <zimon.toutoune@gmail.com> wrote:

zimoun <zimon.toutoune@gmail.com> writes:

Hi,

Here's a friendly 1.3.1 + HTTP.jl + dependencies patch submission.

Hopefully this time it's fine. Package precompilation should be working

now (except json, whose cache, for some reason, is not valid and gets

rebuilt).

As a reminder: determinism is still not addressed and will be fixed in

the 1.4.0 release (hopefully, upstream is still not that responsive).

However, there are dozens of julia packages that I want to package, so

if this first batch is finally fine, more will come.

Thanks, Nicolò

Hi Nicolò,

On Mon, 24 Feb 2020 at 21:59, Nicolò Balzarotti <anothersms@gmail.com> wrote:

Cool! Thank you.

I would like to first push 1.3.1, i.e., the patches v4.

Then let open another bug report with the patches attached here.

Otherwise, it seems a non-ending process. ;-)

About processing, if you use Magit to generate your patches, do not

forget to increment the reroll-count (W c C-m v) otherwise "git

format-patch -v5". It eases to know which version of the patches we

are talking about. For example, here it is the fifth.

Yes, hope soon. :-)

Cheers,

simon

zimoun <zimon.toutoune@gmail.com> writes:

OOk

Thanks! Didn't know about it. Should I re-submit them with the v5 prefix?

Nicolò

Hi,

v5 of the Julia 1.3.1 update patch.

This patch updates julia to 1.3.1. According to previous simon email,

fixes to julia build system and julia package submissions will be done

separately.

Thanks, Nicolò

Nicolò Balzarotti <anothersms@gmail.com> writes:

The upgrade of libgit2 to 0.99.0 seems to have broken this

julia bin/install_packages.jl

Activating environment at `~/workspace/LMGPU.jl/Project.toml`

Cloning default registries into `~/.julia`

Cloning registry from "https://github.com/JuliaRegistries/General.git"

ERROR: LoadError: failed to clone from https://github.com/JuliaRegistries/General.git,error: GitError(Code:ERROR, Class:Invalid, invalid version 0 on git_proxy_options)

Other bug reports from other libgit2 version bumps suggest using an

older version of libgit2

--

Efraim Flashner <efraim@flashner.co.il> ????? ?????

GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351

Confidentiality cannot be guaranteed on emails sent or received unencrypted

Hi Efraim, thanks for the report.

Yes, according to this report[1] we should downgrade libgit2 0.99. I'll

try to send an updated patch in the next days.

In the meanwhile, reding arch bug-tracker [2] I saw that libgit2 depends

on http-parser (USE_HTTP_PARSER). Should we unbundle it from git?

Thanks, Nicolò

[1] https://github.com/JuliaLang/julia/issues/33111

[2] https://bugs.archlinux.org/task/65540

Efraim Flashner <efraim@flashner.co.il> writes:

On Tue, Mar 17, 2020 at 08:51:13AM +0100, Nicolò Balzarotti wrote:

We can just add a libgit2-0.28 for now and remove it later.

I saw that there were patches for using llvm-8 as a base for llvm-julia

instead of llvm-6, but that can wait until later.

That sounds like a separate issue, but a good idea.

--

Efraim Flashner <efraim@flashner.co.il> ????? ?????

GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351

Confidentiality cannot be guaranteed on emails sent or received unencrypted