[PATCH] gnu: julia: Update to 1.9.3.

  • Open
  • quality assurance status badge
Details
5 participants
  • alexis
  • Efraim Flashner
  • Ludovic Courtès
  • Ludovic Courtès
  • Simon Tournier
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal

Debbugs page

Ludovic Courtès wrote 6 months ago
(address . guix-patches@gnu.org)
d9e54f64579b184f4715713101f13f01c809873a.1726147051.git.ludo@gnu.org
From: Ludovic Courtès <ludovic.courtes@inria.fr>

* gnu/packages/julia.scm (julia): Update to 1.9.3.
[arguments]: Use gexps.
[inputs, native-inputs]: Remove labels.
* gnu/packages/llvm.scm (llvm-julia): Rename to…
(llvm-14-julia): … this. Upgrade to 14.0.6-3.
* gnu/packages/patches/julia-1.9-build-tweaks.patch: New file.
* gnu/packages/patches/julia-Use-MPFR-4.2.patch: Remove.
* gnu/local.mk (dist_patch_DATA): Adjust accordingly.

Co-authored-by: Philippe Virouleau <philippe.virouleau@inria.fr>
Change-Id: I6ef35c6cd31741a7725b50364dd0f8fa6a0c6424
---
gnu/local.mk | 2 +-
gnu/packages/julia.scm | 862 ++++++++++--------
gnu/packages/llvm.scm | 38 +-
.../patches/julia-1.9-build-tweaks.patch | 152 +++
gnu/packages/patches/julia-Use-MPFR-4.2.patch | 228 -----
5 files changed, 651 insertions(+), 631 deletions(-)
create mode 100644 gnu/packages/patches/julia-1.9-build-tweaks.patch
delete mode 100644 gnu/packages/patches/julia-Use-MPFR-4.2.patch

Hello comrades!

Here is an update of Julia adapted from:


Version 1.9.3 has been in the guix-hpc channel for a while. It’s not
the latest version, not even the latest series, but it’s an improvement
over what we have.

Thoughts?

Ludo’.

Toggle diff (371 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index ed630041ff..76d58c8e38 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1586,8 +1586,8 @@ dist_patch_DATA = \
%D%/packages/patches/john-the-ripper-jumbo-with-gcc-11.patch \
%D%/packages/patches/json-c-0.13-CVE-2020-12762.patch \
%D%/packages/patches/json-c-0.12-CVE-2020-12762.patch \
+ %D%/packages/patches/julia-1.9-build-tweaks.patch \
%D%/packages/patches/julia-SOURCE_DATE_EPOCH-mtime.patch \
- %D%/packages/patches/julia-Use-MPFR-4.2.patch \
%D%/packages/patches/libcall-ui-make-it-installable.patch \
%D%/packages/patches/libcamera-ipa_manager-disable-signature-verification.patch \
%D%/packages/patches/libcss-check-format.patch \
diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index 9668b3fd1e..5c7d900a09 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -28,6 +28,7 @@ (define-module (gnu packages julia)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
#:use-module (guix download)
+ #:use-module (guix gexp)
#:use-module (guix utils)
#:use-module (guix git-download)
#:use-module (guix build-system gnu)
@@ -142,412 +143,493 @@ (define-public libwhich
(define-public julia
(package
(name "julia")
- (version "1.8.3")
- (source (origin
- (method url-fetch)
- (uri (string-append
- "https://github.com/JuliaLang/julia/releases/download/v"
- version "/julia-" version ".tar.gz"))
- (sha256
- (base32
- "0jf8dr5j7y8cjnr65kn38xps5h9m2qvi8g1yd8qgiip5r87ld3ad"))
- (patches (search-patches "julia-SOURCE_DATE_EPOCH-mtime.patch"
- "julia-Use-MPFR-4.2.patch"))))
+ (version "1.9.3")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/JuliaLang/julia/releases/download/v" version
+ "/julia-" version ".tar.gz"))
+ (sha256
+ (base32 "0788hbcr1v89crv2j3insy89lxs0zwjbxk9q73jpj4g7j26bszcd"))
+ (patches (search-patches "julia-SOURCE_DATE_EPOCH-mtime.patch"
+ "julia-1.9-build-tweaks.patch"))))
(build-system gnu-build-system)
(arguments
- `(#:test-target "test"
- #:modules ((ice-9 match)
- (guix build gnu-build-system)
- (guix build utils))
+ (list #:test-target "test"
+ #:modules `((ice-9 match)
+ (guix build gnu-build-system)
+ (guix build utils))
- ;; The test suite takes many times longer than building and
- ;; can easily fail on smaller machines when they run out of memory.
- #:tests? ,(not (or (%current-target-system)
- (target-aarch64?)))
+ ;; The test suite takes many times longer than building and
+ ;; can easily fail on smaller machines when they run out of memory.
+ #:tests? (not (or (%current-target-system)
+ (target-aarch64?)))
- ;; Do not strip binaries to keep support for full backtraces.
- ;; See https://github.com/JuliaLang/julia/issues/17831
- #:strip-binaries? #f
+ ;; Do not strip binaries to keep support for full backtraces.
+ ;; See https://github.com/JuliaLang/julia/issues/17831
+ #:strip-binaries? #f
- ;; The DSOs use $ORIGIN to refer to each other, but (guix build
- ;; gremlin) doesn't support it yet, so skip this phase.
- #:validate-runpath? #f
+ ;; The DSOs use $ORIGIN to refer to each other, but (guix build
+ ;; gremlin) doesn't support it yet, so skip this phase.
+ #:validate-runpath? #f
- #:phases
- (modify-phases %standard-phases
- (delete 'configure)
- (add-after 'unpack 'prepare-deps
- (lambda* (#:key inputs #:allow-other-keys)
- ;; needed by libwhich
- (setenv "LD_LIBRARY_PATH"
- (string-join (map (lambda (pkg)
- (string-append (assoc-ref inputs pkg)
- "/lib"))
- '("curl" "dsfmt"
- "gmp" "lapack"
- "libssh2" "libnghttp2" "libgit2"
- "libblastrampoline"
- "mbedtls" "mpfr"
- "openblas" "openlibm" "pcre2"
- "suitesparse" "gfortran:lib"))
- ":"))))
- ;; FIXME: Building the documentation requires Julia packages that
- ;; would be downloaded from the Internet. We should build them in a
- ;; separate build phase.
- (add-after 'unpack 'disable-documentation
- (lambda _
- (substitute* "Makefile"
- (("(install: .*) \\$\\(BUILDROOT\\)/doc/_build/html/en/index.html" _ line)
- (string-append line "\n"))
- (("src ui doc deps")
- "src ui deps"))))
- (add-after 'unpack 'activate-gnu-source-for-loader
- (lambda _
- (substitute* "cli/Makefile"
- (("LOADER_CFLAGS =") "LOADER_CFLAGS = -D_GNU_SOURCE"))))
- ;; libquadmath is not available on all architectures.
- ;; https://github.com/JuliaLang/julia/issues/41613
- (add-after 'unpack 'make-libquadmath-optional
- (lambda _
- (substitute* "base/Makefile"
- (("libquadmath,0") "libquadmath,0,ALLOW_FAILURE"))))
- (add-before 'check 'set-home
- ;; Some tests require a home directory to be set.
- (lambda _ (setenv "HOME" "/tmp")))
- (add-before 'build 'fix-include-and-link-paths
- (lambda* (#:key inputs #:allow-other-keys)
- ;; The REPL must be linked with libuv.
- (substitute* "cli/Makefile"
- (("JLDFLAGS \\+= ")
- (string-append "JLDFLAGS += "
- (assoc-ref inputs "libuv")
- "/lib/libuv.so ")))))
- (add-before 'build 'replace-default-shell
- (lambda* (#:key inputs #:allow-other-keys)
- (substitute* "base/client.jl"
- (("/bin/sh") (search-input-file inputs "/bin/sh")))))
- (add-before 'build 'shared-objects-paths
- (lambda* (#:key inputs #:allow-other-keys)
- (let ((jlpath
- (lambda (pkgname)
- (string-append
- "stdlib/" pkgname "_jll/src/" pkgname "_jll.jl")))
- (from
- (lambda (libname)
- (string-append "const " libname " = .*\\.so")))
- (to
- (lambda* (pkg libname #:optional libname_jl)
+ #:phases
+ #~(modify-phases %standard-phases
+ (delete 'configure)
+ (add-after 'unpack 'prepare-deps
+ (lambda* (#:key inputs #:allow-other-keys)
+ ;; needed by libwhich
+ (match inputs
+ (((labels . directories) ...)
+ (set-path-environment-variable
+ "LD_LIBRARY_PATH"
+ '("lib") directories)))))
+ (add-before 'check 'set-home
+ ;; Some tests require a home directory to be set.
+ (lambda _
+ (setenv "HOME" "/tmp")))
+ (add-before 'build 'fix-include-and-link-paths
+ (lambda* (#:key inputs #:allow-other-keys)
+ ;; The REPL must be linked with libuv.
+ (substitute* "cli/Makefile"
+ (("JLDFLAGS \\+= ")
+ (string-append "JLDFLAGS += "
+ (search-input-file inputs
+ "/lib/libuv.so") " ")))))
+ (add-before 'build 'fix-nested-dlopen
+ (lambda* (#:key inputs #:allow-other-keys)
+ (substitute* "Make.inc"
+ ;; See the patch comment for the explanation, *must* be updated
+ ;; when updating julia.
+ (("GUIX_LIBUTF8PROC")
+ (search-input-file inputs "/lib/libutf8proc.a")))
+ (substitute* "Makefile"
+ ;; See the patch comment for the explanation, *must* be updated
+ ;; when updating julia.
+ (("GUIX_LIBCHOLMOD")
+ (search-input-file inputs "/lib/libcholmod.so"))
+ (("GUIX_LIBSPQR")
+ (search-input-file inputs "/lib/libspqr.so"))
+ (("GUIX_LIBSC")
+ (search-input-file inputs
+ "/lib/libsuitesparseconfig.so"))
+ (("GUIX_LIBUMFPACK")
+ (search-input-file inputs "/lib/libumfpack.so")))))
+ (add-before 'build 'replace-default-shell
+ (lambda* (#:key inputs #:allow-other-keys)
+ (substitute* "base/client.jl"
+ (("/bin/sh")
+ (search-input-file inputs "/bin/sh")))))
+ (add-before 'build 'shared-objects-paths
+ (lambda* (#:key inputs #:allow-other-keys)
+ (let ((jlpath (lambda (pkgname)
+ (string-append "stdlib/" pkgname
+ "_jll/src/" pkgname
+ "_jll.jl")))
+ (jlbasepath (lambda (pkgname)
+ (string-append "base/" pkgname
+ ".jl")))
+ (tolib (lambda (libname)
+ (search-input-file inputs
+ (string-append
+ "/lib/" libname
+ ".so"))))
+ (toquotedlib (lambda (libname)
+ (define file
+ (string-append "/lib/"
+ libname
+ ".so"))
+ (string-append "\""
+ (search-input-file
+ inputs file)
+ "\"")))
+ (from (lambda (libname)
+ (string-append "const " libname
+ " = .*\\.so")))
+ (to (lambda* (libname #:optional libname_jl)
+ (define file
+ (string-append "/lib/" libname ".so"))
+
+ (string-append "const "
+ (or libname_jl libname)
+ " = \""
+ (search-input-file
+ inputs file)))))
+ (substitute* "src/jitlayers.cpp"
+ (("libatomic.so")
+ (search-input-file inputs "/lib/libatomic.so")))
+ (substitute* (list (jlbasepath "linking")
+ (jlpath "LLD"))
+ (("\"lld\"")
+ (string-append "\""
+ (search-input-file inputs
+ "/bin/lld")
+ "\"")))
+ (substitute* (jlbasepath "pcre")
+ (("libpcre2-8")
+ (tolib "libpcre2-8")))
+ (substitute* (jlbasepath "gmp")
+ (("libgmp.so.10")
+ (tolib "libgmp")))
+ (substitute* (jlbasepath "mpfr")
+ (("libmpfr.so.6")
+ (tolib "libmpfr")))
+ (substitute* (jlbasepath "irrationals")
+ ((":libmpfr")
+ (toquotedlib "libmpfr")))
+ (substitute* "stdlib/Random/src/generation.jl"
+ ((":libmpfr")
+ (toquotedlib "libmpfr")))
+ (substitute* "stdlib/Printf/src/Printf.jl"
+ (("\"libmpfr\"")
+ (toquotedlib "libmpfr")))
+ (substitute* "stdlib/Random/src/DSFMT.jl"
+ ((":libdSFMT")
+ (toquotedlib "libdSFMT")))
+ (for-each (lambda (file)
+ (substitute* file
+ ((":libgit2")
+ (toquotedlib "libgit2"))))
+ (find-files "stdlib/LibGit2/" "\\.jl$"))
+ (substitute* (jlpath "CompilerSupportLibraries")
+ (((from "libgomp"))
+ (to "libgomp"))
+ (((from "libgfortran"))
+ (string-append
+ "const libgfortran = string(\""
+ (search-input-file inputs
+ "/lib/libgfortran.so"))))
+ (substitute* (jlpath "dSFMT")
+ (((from "libdSFMT"))
+ (to "libdSFMT")))
+ (substitute* (jlpath "GMP")
+ (((from "libgmp"))
+ (to "libgmp"))
+ (((from "libgmpxx"))
+ (to "libgmpxx")))
+ (substitute* (jlpath "libLLVM")
+ (((from "libLLVM"))
+ (to "libLLVM")))
+ (substitute* (jlbasepath "binaryplatforms")
+ (("libLLVM-14jl")
+ (tolib "libLLVM-14jl")))
+ (substitute* (jlpath "LibCURL")
+ (((from "libcurl"))
+ (to "libcurl")))
+ (substitute* (jlpath "LibGit2")
+ (((from "libgit2"))
+ (to "libgit2")))
+ (substitute* (jlpath "LibSSH2")
+ (((from "libssh2"))
+ (to "libssh2")))
+ (substitute* (jlpath "LibUV")
+ (((from "libuv"))
+ (to "libuv")))
+ (substitute* (jlpath "LibUnwind")
+ (((from "libunwind"))
+ (to "libunwind")))
+ (substitute* (jlpath "MPFR")
+ (((from "libmpfr"))
+ (to "libmpfr")))
+ (substitute* (jlpath "MbedTLS")
+ (((from "libmbedcrypto"))
+ (to "libmbedcrypto"))
+ (((from "libmbedtls"))
+ (to "libmbedtls"))
+ (((from "libmbedx509"))
+ (to "libmbedx509")))
+ (substitute* (jlpath "nghttp2")
+ (((from "libnghttp2"))
+ (to "libnghttp2")))
+ (substitute* (jlpath "OpenBLAS")
+ (("libgfortran.so")
+ (tolib "libgfortran"))
+ (((from "libopenblas"))
+ #$@(if (target-x86-64?)
+ `((to "libopenblas64_" "libopenblas"))
+ `((to "libopenblas")))))
+ (substitute* (jlpath "OpenLibm")
+ (((from "libopenlibm"))
+ (to "libopenlibm")))
+ (substitute* (jlpath "PCRE2")
+ (((from "libpcre2_8"))
+ (to "libpcre2-8" "libpcre2_8")))
+ (substitute* (jlpath "SuiteSparse")
+ (((from "libamd"))
+ (to "libamd"))
+ (((from "libbtf"))
+ (to "libbtf"))
+ (((from "libcamd"))
+ (to "libcamd"))
+ (((from "libccolamd"))
+ (to "libccolamd"))
+ (((from "libcholmod"))
+ (to "libcholmod"))
+ (((from "libcolamd"))
+ (to "libcolamd"))
+ (((from "libklu"))
+ (to "libklu"))
+ (((from "libldl"))
+ (to "libldl"))
+ (((from "librbio"))
+ (to "librbio"))
+ (((from "libspqr"))
+ (to "libspqr"))
+ (((from "libsuitesparse"))
+ (to "libsuitesparse"))
+ (((from "libsuitesparseconfig"))
+ (to "libsuitesparseconfig"))
+ (((from "libumfpack"))
+ (to "libumfpack")))
+ (substitute* (jlpath "Zlib")
+ (((from "libz"))
+ (to "libz")))
+ (substitute* (jlpath "libblastrampoline")
+ (("libblastrampoline\\.so")
+ (search-input-file inputs
+ "/lib/libblastrampoline.so"))))))
+ (add-before 'build 'use-ssl-cert-file
+ (lambda _
+ ;; We must adapt MozillaCACerts to use SSL_CERT_FILE.
+ (substitute* "stdlib/MozillaCACerts_jll/src/MozillaCACerts_jll.jl"
+ (("global cacert = .*")
This message was truncated. Download the full message here.
Efraim Flashner wrote 6 months ago
(name . Ludovic Courtès)(address . ludo@gnu.org)
ZufjPjnj4aZbjxiV@3900XT
On Thu, Sep 12, 2024 at 03:22:50PM +0200, Ludovic Courtès wrote:
Toggle quote (37 lines)
> From: Ludovic Courtès <ludovic.courtes@inria.fr>
>
> * gnu/packages/julia.scm (julia): Update to 1.9.3.
> [arguments]: Use gexps.
> [inputs, native-inputs]: Remove labels.
> * gnu/packages/llvm.scm (llvm-julia): Rename to…
> (llvm-14-julia): … this. Upgrade to 14.0.6-3.
> * gnu/packages/patches/julia-1.9-build-tweaks.patch: New file.
> * gnu/packages/patches/julia-Use-MPFR-4.2.patch: Remove.
> * gnu/local.mk (dist_patch_DATA): Adjust accordingly.
>
> Co-authored-by: Philippe Virouleau <philippe.virouleau@inria.fr>
> Change-Id: I6ef35c6cd31741a7725b50364dd0f8fa6a0c6424
> ---
> gnu/local.mk | 2 +-
> gnu/packages/julia.scm | 862 ++++++++++--------
> gnu/packages/llvm.scm | 38 +-
> .../patches/julia-1.9-build-tweaks.patch | 152 +++
> gnu/packages/patches/julia-Use-MPFR-4.2.patch | 228 -----
> 5 files changed, 651 insertions(+), 631 deletions(-)
> create mode 100644 gnu/packages/patches/julia-1.9-build-tweaks.patch
> delete mode 100644 gnu/packages/patches/julia-Use-MPFR-4.2.patch
>
> Hello comrades!
>
> Here is an update of Julia adapted from:
>
> https://gitlab.inria.fr/guix-hpc/guix-hpc/-/commit/eb83e466fe0b9492216abc46da26ec3a89b0a946
>
> Version 1.9.3 has been in the guix-hpc channel for a while. It’s not
> the latest version, not even the latest series, but it’s an improvement
> over what we have.
>
> Thoughts?
>
> Ludo’.

I tested it out on my machines and it built without problems on x86_64
and aarch64. However, I was unable to compile the julia packages we have
when there were some julia inputs.

Do we want to add it in as julia-next, and then we can continue pushing
the version forward and working on the package build failures separately?

--
Efraim Flashner <efraim@flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmbn4zoACgkQQarn3Mo9
g1GwfA//XljHqoXqbjGgiK5fLu5sK2ZUZvRcj+iAMhk+EQUZFXYk3qL0W0sJC+Mj
J7ktRRyMz+6VxX68sXbWNhHUj3UOW+Uk6OdHdFAfeimYyXQmcG3bTAUo3k1858Ej
wIiAIKRGZxd18ptTI/dUbvEbHgZXyzu7VcTdteB4mQYS6n/P33HC4J1Znx64qjT+
PNlaunzV48m/gSp8xx4dGsBE1AfnOfJak5SCkKetSimM4zWRgckWQFFZqXksm/LF
5RC7U69q8o7WIBUxyHKxs1PTA2TQODNv+nyvuYY+xBFF2z9XtPibswV3nO5b1e1u
NirXNIKS70oaV3I2A3oq8phZBhjSIYdxgCIg0ZRjkNrWLVYVgwG70HaTOyCTVwRj
jRrxx7+7CnPQfmki4pUXu3/gf25Meuy9fE4SnNjAWt3Vm9UjN11KY+Crc/yJGyN8
fGdDwzD33RhTuosfDxGbS3XmrLzN4V+gDPtky7NF5JyKg5jrpU5iEUq2K9XtJJQv
MiOvtdRNUKzX/cLpnlGyKTXEYD7/sZ/h86bEHfEXeb1NuBkub/WHMEjjWhlU7o6d
DatD4CcPShQMXpXvjw4jwM5WGQFdiDl4jYTcfp3xkAqK7lExeQYIRS6Gm7UX680y
sZpX46IHZVZcj1qP4/MMsusQmd/aZ5C9zaBvy7hyxtQ/K++rYCI=
=m4GT
-----END PGP SIGNATURE-----


Ludovic Courtès wrote 6 months ago
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87ed5k7wx5.fsf@gnu.org
Hey Efraim,

Efraim Flashner <efraim@flashner.co.il> skribis:

Toggle quote (7 lines)
> I tested it out on my machines and it built without problems on x86_64
> and aarch64. However, I was unable to compile the julia packages we have
> when there were some julia inputs.
>
> Do we want to add it in as julia-next, and then we can continue pushing
> the version forward and working on the package build failures separately?

Oh, good question. I was hoping to get some testing from
ci.guix.gnu.org but it hasn’t happened yet (I added a jobset but
apparently that didn’t work).

I would hope we can migrate straight to 1.9, but if that’s too tricky, I
agree we should follow the plan you describe.

Let me take a look at those failure to see how bad it is.

Thanks,
Ludo’.
Simon Tournier wrote 6 months ago
87a5g2tkbe.fsf@gmail.com
Hi Ludo,

On lun., 16 sept. 2024 at 11:47, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (5 lines)
> I would hope we can migrate straight to 1.9, but if that’s too tricky, I
> agree we should follow the plan you describe.
>
> Let me take a look at those failure to see how bad it is.

I would suggest to have julia-next.

And some ’package-with-julia’ transformation similar as
package-with-python or package-with-ocaml.

Somehow, that’s always painful to upgrade Julia because of the Julia
world rebuild and some (more than some?) package breakages.

That way it would easier to have the CI following the branch team-julia
where regular package would go, as well as julia-next upgrade. And the
manifest could build both set of packages (julia and julia-next) using
the transformation.

Users willing stable just install ’julia’ and ’julia-’ packages.
Adventurous users install ’julia-next’ and ’julia-*-next’.

WDYT?

Cheers,
simon
Efraim Flashner wrote 6 months ago
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
Zu_GE2mleWP15xOO@3900XT
On Fri, Sep 20, 2024 at 05:29:41PM +0200, Simon Tournier wrote:
Toggle quote (27 lines)
> Hi Ludo,
>
> On lun., 16 sept. 2024 at 11:47, Ludovic Courtès <ludo@gnu.org> wrote:
>
> > I would hope we can migrate straight to 1.9, but if that’s too tricky, I
> > agree we should follow the plan you describe.
> >
> > Let me take a look at those failure to see how bad it is.
>
> I would suggest to have julia-next.
>
> And some ’package-with-julia’ transformation similar as
> package-with-python or package-with-ocaml.
>
> Somehow, that’s always painful to upgrade Julia because of the Julia
> world rebuild and some (more than some?) package breakages.
>
> That way it would easier to have the CI following the branch team-julia
> where regular package would go, as well as julia-next upgrade. And the
> manifest could build both set of packages (julia and julia-next) using
> the transformation.
>
> Users willing stable just install ’julia’ and ’julia-’ packages.
> Adventurous users install ’julia-next’ and ’julia-*-next’.
>
> WDYT?

The problem is that the julia-build-system needs to be adapted to the
newer versions of Julia. IMO the benefit of having Julia and Julia-next
is that the julia-* packages are packaged, but those needing a newer
version of Julia can still get it from Guix. Then when we adapt the
julia-build-system we can deprecate julia-next, or have it point to the
next release.

--
Efraim Flashner <efraim@flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmbvxg8ACgkQQarn3Mo9
g1FfbRAAkAFeZXujUPsuzrX4nT5GuBvupX+VXjyfkEX7GFQMAa8cYFmFepzISYkz
asN8hucYvBeqpqQAq99BCpwN0BCZ7japoaZg8Qtf6PS11VYOTrhzWKTqnUyIhG7d
tixTPu66sg4tDhoUMxhWvzC8JIP7jux0IPAMl9ffE0cnmCkUrEiRacY96YCoVIgH
m5I8UU5O+8yybfAIcT03T1tQJIP7MqlabEjQ3+9pTpOdFeiyXGoRGfXGwy14EmMo
w0UaFgUhCdjaT4r8mw1w7/Yr4tyulxz35M45jt+uNbojta83/YyNBaKIQBmhQOby
xdCNgooqdSgqpQXbdEGqz9CJ6j0HAM11WaiGEJKRKT/T4RSYfUp2Uuluy1mUxFAo
bM7QIbzp6FQbjsZofcuEZorXcnDsZETV0ab085zmpjPM4qW1fyjKpL81qgleClMd
tKLS56Qt4pUThLwggwzXVfCTwvEJ7lHWPfwigw0tbP96r8iEojss54wQxjnKPTM6
CrPDYi2jwtanwbPH5uARra5tbXvQr9e42ill0TF6P4yPfVcf4wDETin6xlsR8ieG
9aGWQ2ryJdw+oQCP7ySn5yTLigDTiQvplp2eooTtie4sz62BX2Hd3hrst7PusNBX
Vg0k3D6EjLQtWkB56/pJkXXbBIE9M3PMEYyta6hF/Z14Nzbe+gU=
=OmET
-----END PGP SIGNATURE-----


Simon Tournier wrote 2 months ago
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87frlw3q3o.fsf@gmail.com
Hi,

I am resuming this patch. And I would like to have it merge before the
end of this month. In other words, I would like to mention this update
in the Guix HPC annual report.

On Sun, 22 Sep 2024 at 10:24, Efraim Flashner <efraim@flashner.co.il> wrote:

Toggle quote (2 lines)
>> > Let me take a look at those failure to see how bad it is.

Could we have a julia-team branch and that branch built by CI? It would
help by saving some resources.

From my numbers, ~230 Julia packages fail. Over ~300 packages.


Toggle quote (2 lines)
>> I would suggest to have julia-next.

[...]

Toggle quote (6 lines)
>> Users willing stable just install ’julia’ and ’julia-’ packages.
>> Adventurous users install ’julia-next’ and ’julia-*-next’.

> The problem is that the julia-build-system needs to be adapted to the
> newer versions of Julia.

Yes, you have the same with all *-next “compilers” applied to their
packages: emacs-next, ghc-next, guile-next or python-next. Somehow,
there is no guarantee that the python-build-system works equally for
both python and python-next, and there is no guarantee neither that
python-* packages build with python-next as Python “compiler”.

Somehow, there is the stable and the adventurous. :-)

Well, let keep that aside and let rediscuss it again for Julia 1.10 or
1.11 and let focus on making Julia 1.9.3 the default.

Please note that we already have a Julia package that’s half packaged:
julia-precompiletools. The test suite fails because the
julia-build-system has not been adapted yet. :-)

Cheers,
simon
Ludovic Courtès wrote 2 months ago
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87cygxvb47.fsf@gnu.org
Hello,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (4 lines)
> I am resuming this patch. And I would like to have it merge before the
> end of this month. In other words, I would like to mention this update
> in the Guix HPC annual report.

Excellent. :-)

Toggle quote (3 lines)
> Could we have a julia-team branch and that branch built by CI? It would
> help by saving some resources.

This branch is built already:

Toggle quote (2 lines)
> From my numbers, ~230 Julia packages fail. Over ~300 packages.

[...]

Toggle quote (3 lines)
>> The problem is that the julia-build-system needs to be adapted to the
>> newer versions of Julia.

To be clear, the build failures here have to do with JULIA_LOAD_PATH or
JULIA_DEPOT_PATH somehow not being set to the right value. (That’s why
roughly packages with no dependency other than Julia build fine, while
those that depend on other julia-* packages fails to build.)

I’ve rebased the branch and added debugging helpers I was using.

Protip: when fiddling with the ‘julia’ package, use
‘--without-tests=julia’. :-)

HTH!

Ludo’.
Simon Tournier wrote 2 months ago
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
87bjwa4lcb.fsf@gmail.com
Hi Ludo,

On Wed, 08 Jan 2025 at 11:44, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (3 lines)
> This branch is built already:
> <https://ci.guix.gnu.org/jobset/julia-upgrade>.

Thanks, cool!

Toggle quote (8 lines)
>>> The problem is that the julia-build-system needs to be adapted to the
>>> newer versions of Julia.
>
> To be clear, the build failures here have to do with JULIA_LOAD_PATH or
> JULIA_DEPOT_PATH somehow not being set to the right value. (That’s why
> roughly packages with no dependency other than Julia build fine, while
> those that depend on other julia-* packages fails to build.)

It appears to me more complicated than only JULIA_LOAD_PATH and/or
JULIA_DEPOT_PATH. When I sent the email, I also was thinking something
in this area. Then it took me some time to understand what’s wrong. :-)

For the record, it’s about base/loading.jl:

Toggle snippet (10 lines)
try
toplevel_load[] = false
# perform the search operation to select the module file require intends to load
- path = locate_package(pkg)
+ path = locate_package(pkg, env)
if path === nothing
throw(ArgumentError("""
Package $pkg is required but does not seem to be installed:

My debugging session if someone has a good fix for that. :-) Well, I
instructed the ’precompile’ phase of the julia-build-system in order to
better see; here the trace:

Toggle snippet (21 lines)
starting phase `precompile'
(init)JULIA_DEPOT_PATH: "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/share/julia/:/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/"
(init)JULIA_LOAD_PATH: "/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/loadpath/"
JULIA_DEPOT_PATH: "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/share/julia/:/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/:/gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/"
JULIA_LOAD_PATH: "/gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/loadpath/:/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/loadpath/"
(before)DEPOT_PATH: ["/homeless-shelter/.julia", "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/local/share/julia", "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/share/julia", "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/share/julia/", "/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/", "/gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/"]
(before)LOAD_PATH: ["@", "@v#.#", "@stdlib", "/gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/loadpath/", "/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/loadpath/"]
IDENTIFY_PACKAGE_ENV: (ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9], "/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/loadpath/")
LOCATE_PACKAGE: /gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/loadpath/ConstructionBase/src/ConstructionBase.jl
DEPOT_PATH: ["/gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/", "/homeless-shelter/.julia", "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/local/share/julia", "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/share/julia", "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/share/julia/", "/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/"]
LOAD_PATH: ["/gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/environments/v1.9/Project.toml", "/gnu/store/acvkqdyipdrsa5jrzb185ch37iyliw13-julia-1.9.3/share/julia/stdlib/v1.9", "/gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/loadpath/", "/gnu/store/iiykv7r9ckygxlz0qb9aag5zv317ffz9-julia-constructionbase-1.3.0/share/julia/loadpath/"]
PKG: Unitful [1986cc42-f94f-5a68-af5c-568840ba703d]
ENV: /gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/loadpath/
PATH: /gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/loadpath/Unitful/src/Unitful.jl
PKG: ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9]
ENV: /gnu/store/p6iq6pkc04lj8c43n5a2s058s3ng6rkc-julia-unitful-1.12.2/share/julia/loadpath/
PATH: nothing
ERROR: LoadError: ArgumentError: Package ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.

Indeed, it cannot find ’ConstructionBase’ under the “environment” of
’Unitful’ although it clearly appears both via LOAD_PATH
(JULIA_LOAD_PATH) and DEPOT_PATH (JULIA_DEPOT_PATH). Well, Julia
package management is designed to have one central place (usually
~/.julia), arf!

I will try to be creative if no one beats me. :-)

To be continued…

Cheers,
simon
Simon Tournier wrote 4 weeks ago
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
87ikpc5z6g.fsf@gmail.com
Hi,

On Mon, 13 Jan 2025 at 19:31, Simon Tournier <zimon.toutoune@gmail.com> wrote:

Toggle quote (13 lines)
> For the record, it’s about base/loading.jl:
>
> --8<---------------cut here---------------start------------->8---
> try
> toplevel_load[] = false
> # perform the search operation to select the module file require intends to load
> - path = locate_package(pkg)
> + path = locate_package(pkg, env)
> if path === nothing
> throw(ArgumentError("""
> Package $pkg is required but does not seem to be installed:
> --8<---------------cut here---------------end--------------->8---

Welcome to a debugging session using the most advanced technique: print! :-)

Attached the base/loading.jl with many ’println’.

Well, that’s very interesting because it seems related to some Julia
internals. Somehow, the order of LOAD_PATH matters!

In all the dance, the code enters once and it founds it!

Toggle snippet (7 lines)
ENTRY _require: Call locate_package( ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9] , /gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/ )

[...]

RESULT _require: path = /gnu/store/yg4bz2kb7dqb6haaj0v7hj4mrd8rn27l-julia-constructionbase-1.3.0/share/julia/loadpath/ConstructionBase/src/ConstructionBase.jl

Then, it enters later with the same call… And bang!

Toggle snippet (9 lines)
ENTRY _require: Call locate_package( ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9] , /gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/ )

[...]

RESULT _require: path = nothing
ERROR: LoadError: ArgumentError: Package ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.

What’s different? The call loops over load_path(), something like:

Toggle snippet (15 lines)
locate_package_env: START LOOP; over load_path() = ["/gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/environments/v1.9/Project.toml", "/gnu/store/cp0fbzd4j1znnp4fjpsknap0ksnp2xk1-julia-1.9.3/share/julia/stdlib/v1.9", "/gnu/store/yg4bz2kb7dqb6haaj0v7hj4mrd8rn27l-julia-constructionbase-1.3.0/share/julia/loadpath/", "/gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/"]

locate_package_env: loop = 1
locate_package_env: env = /gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/environments/v1.9/Project.toml
locate_package_env: path = nothing

locate_package_env: loop = 2
locate_package_env: env = /gnu/store/cp0fbzd4j1znnp4fjpsknap0ksnp2xk1-julia-1.9.3/share/julia/stdlib/v1.9
locate_package_env: path = nothing

locate_package_env: loop = 3
locate_package_env: env = /gnu/store/yg4bz2kb7dqb6haaj0v7hj4mrd8rn27l-julia-constructionbase-1.3.0/share/julia/loadpath/
locate_package_env: path = /gnu/store/yg4bz2kb7dqb6haaj0v7hj4mrd8rn27l-julia-constructionbase-1.3.0/share/julia/loadpath/ConstructionBase/src/ConstructionBase.jl

And for the second case, tandam!

Toggle snippet (28 lines)
locate_package_env: START LOOP; over load_path() = ["/tmp/.julia/environments/v1.9/Project.toml", "/gnu/store/cp0fbzd4j1znnp4fjpsknap0ksnp2xk1-julia-1.9.3/share/julia/stdlib/v1.9", "/gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/", "/gnu/store/yg4bz2kb7dqb6haaj0v7hj4mrd8rn27l-julia-constructionbase-1.3.0/share/julia/loadpath/"]

locate_package_env: loop = 1
locate_package_env: env = /tmp/.julia/environments/v1.9/Project.toml
locate_package_env: path = nothing

locate_package_env: loop = 2
locate_package_env: env = /gnu/store/cp0fbzd4j1znnp4fjpsknap0ksnp2xk1-julia-1.9.3/share/julia/stdlib/v1.9
locate_package_env: path = nothing

locate_package_env: loop = 3
locate_package_env: env = /gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/
locate_package_env: path = nothing
locate_package_env: BREAK: /gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/ == /gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/
locate_package_env: : loading_extension = false
locate_package_env: : precompiling_extension = false
locate_package_env: END LOOP

locate_package_env: pkg = ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9]
locate_package_env: Sys.STDLIB = /gnu/store/cp0fbzd4j1znnp4fjpsknap0ksnp2xk1-julia-1.9.3/share/julia/stdlib/v1.9
locate_package_env: Call manifest_uuid_path( /gnu/store/cp0fbzd4j1znnp4fjpsknap0ksnp2xk1-julia-1.9.3/share/julia/stdlib/v1.9 , ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9] )
locate_package_env: mbypath = nothing String? false
locate_package_env: RETURN: path = nothing ; env = /gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/
locate_package: RETURN: nothing
_require: Post locate_package( ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9] , /gnu/store/garmz1dz382ib25p2ckl0kj9rjay4xd4-julia-unitful-1.12.2/share/julia/loadpath/ )
RESULT _require: path = nothing

The attentive reader notes that load_path() is a list of 4 elements –the
last contains what we want– but a test breaks the loop.

Well, I do not know the details about the reasons of all that. Maybe
something worth to ask upstream. :-)

Well, I’ll (force) push shortly a trivial fix to the branch
wip-julia-upgrade.

Cheers,
simon

PS: For the record, some elements:

1. Checkout wip-julia-upgrade
2. cp $(./pre-inst-env guix build julia -S) /tmp/
3. Uncompress the source and copy (e.g., /home/simon/tmp/julia-1.9.3)
4. Tweak the base/loading.jl file (and other as base/initdefs.jl)
5. Use this source with:

(source
(local-file "/home/simon/tmp/julia-1.9.3"
#:recursive? #t))

6. guix shell -CPNWD guix --expose=/home/simon/tmp/=/home/simon/tmp/
7. Build: ./pre-inst-env guix build --no-grafts --without-tests=julia julia
8. Tweak guix/build-system/julia-build-system.scm
9. Loop ./pre-inst-env guix build --no-grafts --without-tests=julia julia-unitful

Hope that helps.

--
Attachment: loading.diff
zimoun wrote 3 weeks ago
Status wip-julia-update (was Re: bug#73197: [PATCH] gnu: julia: Update to 1.9.3.)
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
865xlcgrm8.fsf_-_@gmail.com
Hi,

On Fri, 14 Feb 2025 at 16:22, Simon Tournier <zimon.toutoune@gmail.com> wrote:

Toggle quote (3 lines)
> Well, I’ll (force) push shortly a trivial fix to the branch
> wip-julia-upgrade.

I did. \o/

And waiting the rebuild by ci.guix, I’m rebuilding some on my own. And
I obverse weird behaviour…

For example, on the top of 5f31ce18b3.

Toggle snippet (16 lines)
$ ./pre-inst-env guix build --no-grafts \
--without-tests=julia-argcheck \
--without-tests=julia-documenter \
--without-tests=julia-genericschur \
--without-tests=julia-static \
--without-tests=julia-statsbase \
--without-tests=julia-infinity \
--without-tests=julia-pycall \
--without-tests=julia-http \
--without-tests=julia-staticarrays \
--without-tests=julia-finitedifferences \
julia-benchmarktools

/gnu/store/pdj7ilvyh4i96jd5av97vrlghpxc25ni-julia-benchmarktools-1.3.2

So far, so good! Here all ’without-tests’ are packages that need to be
updated or need a fix in their test suite–e.g., one test fails because
hardware or other as it happens time to time with Julia packages.

Now, let do the same but using the ’julia’ package without its own
tests. It should be pass, no?

Toggle snippet (19 lines)
$ ./pre-inst-env guix build --no-grafts \
--without-tests=julia-argcheck \
--without-tests=julia-documenter \
--without-tests=julia-genericschur \
--without-tests=julia-static \
--without-tests=julia-statsbase \
--without-tests=julia-infinity \
--without-tests=julia-pycall \
--without-tests=julia-http \
--without-tests=julia-staticarrays \
--without-tests=julia-finitedifferences \
--without-tests=julia \
julia-benchmarktools

starting phase `precompile'
ERROR: LoadError: ArgumentError: Package Parsers [69de0a69-1ddd-5017-9359-2bf0b02dc9f0] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.

Euh, I’m probably doing wrong something somewhere… But it does not jump
to my eyes what. Any idea?

For the record, here the path:

$ guix graph --path julia-benchmarktools julia-parsers
julia-benchmarktools@1.3.2
julia-json@0.21.3
julia-parsers@2.2.4

And note that julia-parsers is propagated by julia-json then
julia-benchmarktools propagates julia-json.

Therefore, the full ’julia’ package finds julia-parsers but not all the
same except running the ’check’ phase of julia’.

What do I miss about package transformation?

That’s said, it’s weird behaviour because I also observe the converse:
something that passes using --without-tests=julia but fails with full
’julia’.

Bah… my creativity starts to be lacking…

Cheers,
simon

PS: What about Nix?

1. I don’t clearly understand what they do.
2. So much Julia code… Hum, I’m not convinced that’s the way…

alexis wrote 3 weeks ago
(name . zimoun)(address . zimon.toutoune@gmail.com)
c444bad9f2886690f55a6a6b2093685f@mail.infomaniak.com
Dear Simon,

Thanks for working on this ! I've tried to follow the nix install
process. Here are some (incomplete) notes.
For brievity's sake, only the Zlib package will be shown in the files
below.

1. A list of dependencies is stored into a YAML:

8<-----------------------------------------------------------------------

cat
/nix/store/3c8lw0flmvvl5hlkc8c7xcwvwj1kbzp2-julia-package-closure.yml

- name: Zlib_jll

  uuid: 83775a58-1f1d-513f-b197-d71354ab007a

  version: 1.2.13+0

  depends_on:

    "Libdl": "8f399da3-3557-5675-b5ff-fb832c97cbdb"

8<-----------------------------------------------------------------------

2. Each dependency is a derivation and transformed into a git repo

8<-----------------------------------------------------------------------

building
'/nix/store/icap989di414ax9zpq1b07ca4szn1v6j-Zlib_jll.jl-866bc71.drv'...

866bc7103cfb6233c35d6f1b2c6bd23ef9f6fe5f) into
/nix/store/pi31a0w9xm3qych5yfdw3mg93sxl41rq-Zlib_jll.jl-866bc71

Initialized empty Git repository in
/nix/store/pi31a0w9xm3qych5yfdw3mg93sxl41rq-Zlib_jll.jl-866bc71/.git/

remote: Enumerating objects: 26, done.

remote: Counting objects: 100% (7/7), done.

remote: Compressing objects: 100% (6/6), done.

remote: Total 26 (delta 1), reused 1 (delta 1), pack-reused 19 (from
1)

Unpacking objects: 100% (26/26), 5.55 KiB | 1.85 MiB/s, done.


* branch           
866bc7103cfb6233c35d6f1b2c6bd23ef9f6fe5f -> FETCH_HEAD

Switched to a new branch 'fetchgit'

removing `.git'...

8<-----------------------------------------------------------------------

3. Package location are overriden, first into a JSON

8<-----------------------------------------------------------------------

$ cat /nix/store/nf666fq31xz8f7hx9qi9ygicf4hdfx4i-Overrides.json

    "uuid-83775a58-1f1d-513f-b197-d71354ab007a": {

        "Zlib": {

            "path":
"/nix/store/6d4hpjr4f04zq9090kd9kkffa9j2q7gj-Zlib",

            "sha1":
"b86177a36c8ba482120ab766b6670177dffd72f3"

        }

    },

8<-----------------------------------------------------------------------

4. ... then in a TOML

8<-----------------------------------------------------------------------

$ cat /nix/store/gnwl6k320mv0pnb8hpcdww40hypp3i7f-Overrides.toml

b86177a36c8ba482120ab766b6670177dffd72f3 =
"/nix/store/6d4hpjr4f04zq9090kd9kkffa9j2q7gj-Zlib"

8<-----------------------------------------------------------------------

5. A local registry is created. Somewhere, the following code is run

8<-----------------------------------------------------------------------

Pkg.Registry.add(Pkg.RegistrySpec(path=\"/nix/store/p5dy9z5869qnmmawsid5dcy1nqy1k5sn-minimal-julia-registry`.

8<-----------------------------------------------------------------------

The registry looks like this:

8<-----------------------------------------------------------------------

$ tree
/nix/store/p5dy9z5869qnmmawsid5dcy1nqy1k5sn-minimal-julia-registry

├── A

│   ├── ArrowTypes

│   │   ├── Compat.toml

│   │   ├── Deps.toml

│   │   ├── Package.toml

│   │   └── Versions.toml

8<-----------------------------------------------------------------------

with Compat.toml

8<-----------------------------------------------------------------------

["1-1.1"]

julia = "1.3.0-1"

["1.2-2"]

julia = "1"

8<-----------------------------------------------------------------------

with Deps.toml

8<-----------------------------------------------------------------------

[1-2]

UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

["2.1-2"]

Sockets = "6462fe0b-24de-5631-8697-dd941f90decc"

8<-----------------------------------------------------------------------

with Package.toml

8<-----------------------------------------------------------------------

name = "ArrowTypes"

uuid = "31f734f8-188a-4ce0-8406-c8a06bd891cd"

repo =
"file:///nix/store/ndqz9cikh67hqwxh4l7vji07764p4xpk-julia-ArrowTypes-2.3.0"

subdir = "src/ArrowTypes"

["2.3.0"]

git-tree-sha1 = "404265cd8128a2515a81d5eae16de90fdef05101"

8<-----------------------------------------------------------------------

6. I'm not sure how the registry is used at this step so here the log
output if you can make sense out of it

8<-----------------------------------------------------------------------

Copying registry from
`/nix/store/p5dy9z5869qnmmawsid5dcy1nqy1k5sn-minimal-julia-registry`

Copied registry `General` to
`/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/depot/registries/General`

Adding packages: CSV

Resolving package versions...

Cloning [e2d170a0-9d28-54be-80f0-106bbe20a464] DataValueInterfaces
from
file:///nix/store/0gdzfghqzs56ilzb9dx239fij4hny9i3-julia-DataValueInterfaces-1.0.0

Installed DataValueInterfaces ───────── v1.0.0

...

Updating
`/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/project/Project.toml`

[336ed68f] + CSV v0.10.14

Updating
`/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/project/Manifest.toml`

Removing registry `General` from
/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/depot/registries/General

building
'/nix/store/0k35snxa68lrkap77s376gqx8ppxr8yn-julia-1.9.4-env.drv'...

8<-----------------------------------------------------------------------

Hope that helps,

Alexis
Attachment: file
Simon Tournier wrote 3 weeks ago
(address . alexis@praga.dev)
87seo86frb.fsf@gmail.com
Hi Alexis,

On Tue, 18 Feb 2025 at 23:30, Alexis Praga via Guix-patches via <guix-patches@gnu.org> wrote:

Toggle quote (3 lines)
> I've tried to follow the nix install
> process. Here are some (incomplete) notes.

Thanks! It helps.


Toggle quote (2 lines)
> 5. A local registry is created. Somewhere, the following code is run

On Monday, I thought about something like that. However, I’m note sure
that one package can write to another outputs than its own. Something
to check…

My idea was to create a “local” registry per package containing all the
requirements. Relying heavily on symbolic links, as profile does.

Today, it works that way:

Toggle snippet (10 lines)
$ tree -L 2 $(guix build julia-zygote)/share/julia
/gnu/store/n64x6qrxy7b2lci9aar3pv6kikp1rgan-julia-zygote-0.6.41/share/julia
├── compiled
│   └── v1.8
├── loadpath
│   └── Zygote
└── packages
└── Zygote

where julia-zygote contains this definition:

(propagated-inputs
(list julia-abstractffts
julia-chainrules
julia-chainrulescore
julia-diffrules
julia-fillarrays
julia-forwarddiff
julia-irtools
julia-logexpfunctions
julia-macrotools
julia-nanmath
julia-requires
julia-specialfunctions
julia-zygoterules))

In other words, when running the ’check’ phase, we setup the LOAD_PATH
and DEPOT_PATH via the environment variable JULIA_{LOAD,DEPOT}_PATH.

However, stuff changed with v1.9 as explained here [1]. Maybe I’m
missing something with all that. My current understanding: tweaking
these environment variables will not be enough.

Now, let enter inside “guix shell julia-zygote”:

Toggle snippet (32 lines)
$ tree -L 2 $GUIX_ENVIRONMENT/share/julia
/gnu/store/asis32i2wf3jr9854nh1yxjf6fsipk62-profile/share/julia
├── compiled
│   └── v1.8
├── loadpath
│   ├── AbstractFFTs -> /gnu/store/hlnlr2xkd06mrcg72v7ilynqv36z9xqi-julia-abstractffts-1.0.1/share/julia/loadpath/AbstractFFTs
│   ├── Aqua -> /gnu/store/jb4vzsn12nbfqyr8inhz2a570fv56x4i-julia-aqua-0.5.5/share/julia/loadpath/Aqua
│   ├── Calculus -> /gnu/store/k2ghzcx087sssllkk1b3c8xxjwbr3n43-julia-calculus-0.5.1/share/julia/loadpath/Calculus
│   ├── ChainRules -> /gnu/store/sj99c9z4siq87qhmix0xr5yyzkcphg80-julia-chainrules-1.35.0/share/julia/loadpath/ChainRules
│   ├── ChainRulesCore -> /gnu/store/h09v7lvgahl7rx95aj0rz92hc34xpmpn-julia-chainrulescore-1.12.2/share/julia/loadpath/ChainRulesCore

[...]

│   ├── StaticArrays -> /gnu/store/xn3jfkssqa0lggnrr3qkaicvawyn4wda-julia-staticarrays-1.2.13/share/julia/loadpath/StaticArrays
│   ├── Zygote -> /gnu/store/n64x6qrxy7b2lci9aar3pv6kikp1rgan-julia-zygote-0.6.41/share/julia/loadpath/Zygote
│   └── ZygoteRules -> /gnu/store/ska4a8nxxl9qmv1hgb4ah5147lngq588-julia-zygoterules-0.2.2/share/julia/loadpath/ZygoteRules
└── packages
├── AbstractFFTs -> /gnu/store/hlnlr2xkd06mrcg72v7ilynqv36z9xqi-julia-abstractffts-1.0.1/share/julia/packages/AbstractFFTs
├── Aqua -> /gnu/store/jb4vzsn12nbfqyr8inhz2a570fv56x4i-julia-aqua-0.5.5/share/julia/packages/Aqua
├── Calculus -> /gnu/store/k2ghzcx087sssllkk1b3c8xxjwbr3n43-julia-calculus-0.5.1/share/julia/packages/Calculus
├── ChainRules -> /gnu/store/sj99c9z4siq87qhmix0xr5yyzkcphg80-julia-chainrules-1.35.0/share/julia/packages/ChainRules
├── ChainRulesCore -> /gnu/store/h09v7lvgahl7rx95aj0rz92hc34xpmpn-julia-chainrulescore-1.12.2/share/julia/packages/ChainRulesCore

[...]

├── StaticArrays -> /gnu/store/xn3jfkssqa0lggnrr3qkaicvawyn4wda-julia-staticarrays-1.2.13/share/julia/packages/StaticArrays
├── Zygote -> /gnu/store/n64x6qrxy7b2lci9aar3pv6kikp1rgan-julia-zygote-0.6.41/share/julia/packages/Zygote
└── ZygoteRules -> /gnu/store/ska4a8nxxl9qmv1hgb4ah5147lngq588-julia-zygoterules-0.2.2/share/julia/packages/ZygoteRules

65 directories, 0 files

My idea is to do the same at least for the ’check’ phase,


Thanks again for trying with Nix. It helps! I’ll read again all these
details, food for creativity. ;-)

Cheers,
simon

1: [bug#73197] [PATCH] gnu: julia: Update to 1.9.3.
Simon Tournier <zimon.toutoune@gmail.com>
Mon, 13 Jan 2025 19:31:16 +0100
id:87bjwa4lcb.fsf@gmail.com
alexis wrote 3 weeks ago
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
6bbad5dddc7aa2b1b2c390cf96eeec06@mail.infomaniak.com
Hi Simon,

As a follow-up, here's how nix uses the local depot.
It wraps julia by setting the following variable to subfolders of the "meta" derivation

1. JULIA_DEPOT_PATH to ~/.julia and the local depot
home/alex/.julia:/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/depot

2. JULIA_PROJECT_PATH to a custom project
/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/project

3. JULIA_LOAD_PATH to the Project.toml
@:/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/project/Project.toml:@v#.#:@stdlib
(not sure what the weird characters are about, like '@')

The depot itself has the following structure
- source code in clones/ with UUID
- compiled library in compiled
- source code in packages/ with package name (again !)

❯ tree /nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/depot/ -L 2
/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/depot/
├── artifacts
│   └── Overrides.toml
├── clones
    ├── 944b1d66-785c-5afd-91f1-9de20f533193
    │   ├── LICENSE.md
    │   ├── Project.toml
    │   ├── README.md
    │   ├── src
    │   │   ├── CodecZlib.jl
    │   │   ├── compression.jl
    │   │   ├── decompression.jl
    │   │   └── libz.jl
    │   └── test
    │       ├── abra.gz
    │       ├── foo.txt.gz
    │       └── runtests.jl
│  ── [...]
|── compiled
|   └── v1.9
|       ├── CodecZlib
|       │   ├── 1TI30_fZ84u.ji
|       │   └── 1TI30_fZ84u.so
|       ├── [...]
├── logs
│   └── manifest_usage.toml
├── packages
│   ├── CodecZlib
|      └── Ndt0H
|          ├── LICENSE.md
|          ├── Project.toml
|          ├── README.md
|          ├── src
|          │   ├── CodecZlib.jl
|          │   ├── compression.jl
|          │   ├── decompression.jl
|          │   └── libz.jl
|          └── test
|              ├── abra.gz
|              ├── foo.txt.gz
|              └── runtests.jl
|      └── [...]
|   ├── registries
└── scratchspaces
    └── 44cfe95a-1eb2-52ea-b672-e2afdf69b78f

Finally, the project itself has a TOML file defining the package we want to install (in my case CSV) and the Manifest.toml for this package (I think)

❯ tree /nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/project/
/nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/project/
├── Manifest.toml
└── Project.toml

❯ cat /nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/project/Project.toml
     [deps]
     CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"

❯ head -n 10 /nix/store/ap871sp56qx74k4hz2nlnjvpys8pdh7j-julia-depot/project/Manifest.toml
     # This file is machine-generated - editing it directly is not advised

     julia_version = "1.9.4"
     manifest_format = "2.0"
     project_hash = "de8e0bb32b50d890eb0519dc7725e6a1f422aa74"

     [[deps.Artifacts]]
     uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

     [[deps.Base64]]


Hope that helps,

Alexis
Simon Tournier wrote 6 days ago
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
871pvb60k1.fsf@gmail.com
Hi,

On Fri, 14 Feb 2025 at 22:10, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (2 lines)
> Bah… my creativity starts to be lacking…

Unexpected failures by Cuirass?!

For the record, if we consider the last evaluation [1] of the branch
wip-julia-upgrade on the top of 5f31c [2], Cuirass reports this failure
[3] about julia-abstractffts.

Toggle snippet (8 lines)
starting phase `check'
running tests from '/gnu/store/w5iqv2y9yg12plyks3ppywyvq2ffirl8-julia-abstractffts-1.0.1/share/julia/loadpath/AbstractFFTs/test/runtests.jl'
ERROR: LoadError: ArgumentError: Package ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:

Hum?! What do I miss?

Cuirass says the derivation and the output are:

/gnu/store/pyvn3lmxizbhyvh29h8ladx8ycpsxbi6-julia-abstractffts-1.0.1.drv
/gnu/store/w5iqv2y9yg12plyks3ppywyvq2ffirl8-julia-abstractffts-1.0.1

And on my local machine, using Guix 056910e, it just builds and the
’check’ phase just passes.

Toggle snippet (49 lines)
$ guix build --check \
/gnu/store/pyvn3lmxizbhyvh29h8ladx8ycpsxbi6-julia-abstractffts-1.0.1.drv

The following derivation will be built:
/gnu/store/pyvn3lmxizbhyvh29h8ladx8ycpsxbi6-julia-abstractffts-1.0.1.drv
building /gnu/store/pyvn3lmxizbhyvh29h8ladx8ycpsxbi6-julia-abstractffts-1.0.1.drv...
starting phase `separate-from-pid1'
build process now running as PID 18
phase `separate-from-pid1' succeeded after 0.0 seconds

[...]

starting phase `check'
running tests from '/gnu/store/w5iqv2y9yg12plyks3ppywyvq2ffirl8-julia-abstractffts-1.0.1/share/julia/loadpath/AbstractFFTs/test/runtests.jl'
Test Summary: | Pass Total Time
rfft sizes | 5 5 0.2s
Test Summary: | Pass Total Time
Custom Plan | 4 4 0.5s
Test Summary: | Pass Total Time
Shift functions | 14 14 0.3s
Test Summary: | Pass Total Time
FFT Frequencies | 67 67 0.7s
Test Summary: | Pass Total Time
normalization | 1 1 0.0s
phase `check' succeeded after 13.2 seconds
starting phase `patch-shebangs'
phase `patch-shebangs' succeeded after 0.0 seconds
starting phase `strip'
phase `strip' succeeded after 0.0 seconds
starting phase `validate-runpath'
phase `validate-runpath' succeeded after 0.0 seconds
starting phase `validate-documentation-location'
phase `validate-documentation-location' succeeded after 0.0 seconds
starting phase `delete-info-dir-file'
phase `delete-info-dir-file' succeeded after 0.0 seconds
starting phase `patch-dot-desktop-files'
phase `patch-dot-desktop-files' succeeded after 0.0 seconds
starting phase `make-dynamic-linker-cache'
phase `make-dynamic-linker-cache' succeeded after 0.0 seconds
starting phase `install-license-files'
installing 1 license files from '.'
phase `install-license-files' succeeded after 0.0 seconds
starting phase `reset-gzip-timestamps'
phase `reset-gzip-timestamps' succeeded after 0.0 seconds
starting phase `compress-documentation'
phase `compress-documentation' succeeded after 0.0 seconds
guix build: error: derivation `/gnu/store/pyvn3lmxizbhyvh29h8ladx8ycpsxbi6-julia-abstractffts-1.0.1.drv' may not be deterministic: output `/gnu/store/w5iqv2y9yg12plyks3ppywyvq2ffirl8-julia-abstractffts-1.0.1' differs

The non-deterministic is expected; another story.


What could be twisted? How is it possible?

Cheers,
simon

Ludovic Courtès wrote 3 days ago
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87ldtgv358.fsf@gnu.org
Hello,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (6 lines)
> And on my local machine, using Guix 056910e, it just builds and the
> ’check’ phase just passes.
>
> $ guix build --check \
> /gnu/store/pyvn3lmxizbhyvh29h8ladx8ycpsxbi6-julia-abstractffts-1.0.1.drv

It fails on my laptop just like

Toggle snippet (6 lines)
starting phase `check'
running tests from '/gnu/store/w5iqv2y9yg12plyks3ppywyvq2ffirl8-julia-abstractffts-1.0.1/share/julia/loadpath/AbstractFFTs/test/runtests.jl'
ERROR: LoadError: ArgumentError: Package ConstructionBase [187b0558-2788-49d3-abe0-74a17ed4e7c9] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.

If you don’t get that failure on your own machine, perhaps there’s
something non-deterministic here? I wouldn’t expect this type of error
to be non-deterministic, but who knows.

Tricky!

Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 73197@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 73197
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch
You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags
mumi command -t +confirmed -t +easy
Or, remove the moreinfo tag and set the help tag
mumi command -t -moreinfo -t +help