Incompatibilities between gcc-toolchain and R packages

  • Open
  • quality assurance status badge
Details
2 participants
  • Lars-Dominik Braun
  • zimoun
Owner
unassigned
Submitted by
Lars-Dominik Braun
Severity
normal
L
L
Lars-Dominik Braun wrote on 19 Dec 2022 13:00
(address . bug-guix@gnu.org)
Y6BSQpbK7BgW5Idk@noor.fritz.box
Hi,

I ran into issues with the package r-brms. Take the following reproducer as an example:

$ guix shell r-brms r make sed gcc-toolchain bash -C --no-cwd --share=/tmp
$ R
> library(brms)
> fit1 <- brm(count ~ zAge + zBase * Trt + (1|patient), data = epilepsy, family = poisson())
Compiling Stan program...
Error in dyn.load(libLFile) :
unable to load shared object '/tmp/RtmpKqzbYg/file3245e787c.so':
/gnu/store/vqhamsanmlm8v6f90a635zc6gmhwlphp-gfortran-10.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /tmp/RtmpKqzbYg/file3245e787c.so)
Error in sink(type = "output") : invalid connection

The same code works well with gcc-toolchain@10 instead of gcc-toolchain
(@12, which is the default). As we can see the generated shared library
above depends on GCC 12, 10 and GFortran 10 at the same time:

$ ldd /tmp/RtmpKqzbYg/file3245e787c.so | grep -e gcc -e fortran
libstdc++.so.6 => /gnu/store/4zvswpr2h3b7dvqpvjcdam8vfhyjrmgl-gcc-12.2.0-lib/lib/libstdc++.so.6 (0x00007f69bd9fc000)
libgcc_s.so.1 => /gnu/store/4zvswpr2h3b7dvqpvjcdam8vfhyjrmgl-gcc-12.2.0-lib/lib/libgcc_s.so.1 (0x00007f69bd898000)
libgfortran.so.5 => /gnu/store/vqhamsanmlm8v6f90a635zc6gmhwlphp-gfortran-10.3.0-lib/lib/libgfortran.so.5 (0x00007f69bd3e6000)
libquadmath.so.0 => /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libquadmath.so.0 (0x00007f69bd39b000)
libgomp.so.1 => /gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/libgomp.so.1 (0x00007f69bccf9000)

The command used to link that .so is revealed by strace to be (filenames
are random and may differ between runs):

["g++", "-std=gnu++14", "-shared", "-L/gnu/store/145dmr8drw3yzrdhzbsksi05p599hrs6-r-minimal-4.2.2/lib/R/lib", "-L/usr/local/lib", "-o", "file3373276d0.so", "file3373276d0.o", "/gnu/store/5rdg6sv1nrg1ikqxcykvdh9g4yd2xjvy-r-rstan-2.21.7/site-library/rstan/lib//libStanServices.a", "-L/gnu/store/y7rvp47mz3hcnpvnqsx42qz5yvs10pnl-r-stanheaders-2.21.0-7/site-library/StanHeaders/lib/", "-lStanHeaders", "-L/gnu/store/01lhjrrkvrnsjd8xh4ssdn3bgxc422jl-r-rcppparallel-5.1.5/site-library/RcppParallel/lib/", "-ltbb", "-L/gnu/store/145dmr8drw3yzrdhzbsksi05p599hrs6-r-minimal-4.2.2/lib/R/lib", "-lR"]

So it links against libStanServices.a, libStanHeaders(.a), libtbb(.so) and
libR(.so) of which only libR has a reference to gfortran:

$ ldd /gnu/store/145dmr8drw3yzrdhzbsksi05p599hrs6-r-minimal-4.2.2/lib/R/lib/libR.so | grep -i fortran
libgfortran.so.5 => /gnu/store/vqhamsanmlm8v6f90a635zc6gmhwlphp-gfortran-10.3.0-lib/lib/libgfortran.so.5 (0x00007f5be49db000)

That means every software linking against R is also incompatible with
the current default gcc-toolchain (when using the command line or
specifications; different story when writing package recipes).

Possible solutions:

- Make gcc-toolchain@10 the default and rename gcc-toolchain@12 to
gcc-toolchain-next@12, like we do for Haskell and (sometimes) Python.
- Update, both, the default GCC and GFortran to version 12.
- Explicitly depend on the correct gcc-toolchain in packages that need
a compiler.

Cheers,
Lars
Z
Z
zimoun wrote on 20 Dec 2022 00:32
86wn6nynp1.fsf@gmail.com
Hi Lars,

On Mon, 19 Dec 2022 at 13:00, Lars-Dominik Braun <lars@6xq.net> wrote:
Toggle quote (14 lines)
> $ guix shell r-brms r make sed gcc-toolchain bash -C --no-cwd --share=/tmp
> $ R
> > library(brms)
> > fit1 <- brm(count ~ zAge + zBase * Trt + (1|patient), data = epilepsy, family = poisson())
> Compiling Stan program...
> Error in dyn.load(libLFile) :
> unable to load shared object '/tmp/RtmpKqzbYg/file3245e787c.so':
> /gnu/store/vqhamsanmlm8v6f90a635zc6gmhwlphp-gfortran-10.3.0-lib/lib/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /tmp/RtmpKqzbYg/file3245e787c.so)
> Error in sink(type = "output") : invalid connection
>
> The same code works well with gcc-toolchain@10 instead of gcc-toolchain
> (@12, which is the default). As we can see the generated shared library
> above depends on GCC 12, 10 and GFortran 10 at the same time:

[...]

Toggle quote (8 lines)
> Possible solutions:
>
> - Make gcc-toolchain@10 the default and rename gcc-toolchain@12 to
> gcc-toolchain-next@12, like we do for Haskell and (sometimes) Python.
> - Update, both, the default GCC and GFortran to version 12.
> - Explicitly depend on the correct gcc-toolchain in packages that need
> a compiler.

It depends on what you are naming default. :-)

The default GCC toolchain for compiling is provided by the package
gcc-toolchain@10.

Without any specification about the version, if a package name is
defined at several versions, then the command-line uses the higher
version of this package. To avoid the kind of confusing error as you
are observing, usually, the suffix ’-next’ is applied to the package
name for higher versions. For instance, python-numpy-next.

To add a data point to the confusion, consider the package ’gcc’ instead
of ’gcc-toolchain’. As noticed earlier [1], there is a bug. Consider:

Toggle snippet (20 lines)
$ guix show gcc@6
guix show: error: gcc@6: package not found

$ guix show gcc@7
name: gcc
version: 7.5.0
outputs:
+ lib: shared libraries
+ debug: debug information
+ out: everything else
systems: x86_64-linux i686-linux
dependencies: gmp@6.2.1 isl@0.23 libelf@0.8.13 libstdc++-headers@10.3.0 mpc@1.2.1 mpfr@4.1.0 perl@5.34.0 texinfo@6.7 zlib@1.2.11
location: gnu/packages/gcc.scm:601:2
homepage: https://gcc.gnu.org/
license: GPL 3+
synopsis: GNU Compiler Collection
description: GCC is the GNU Compiler Collection. It provides compiler front-ends for several languages, including C, C++, Objective-C, Fortran, Ada, and
+ Go. It also includes runtime support libraries for these languages.

It is because the ’properties’ field of gcc@7 is rewritten,

Toggle snippet (12 lines)
(define-public gcc-7
(package
(inherit gcc-6)
(version "7.5.0")
[...]
(properties
`((compiler-cpu-architectures
("aarch64" ,@%gcc-7.5-aarch64-micro-architectures)
("armhf" ,@%gcc-7.5-armhf-micro-architectures)
("x86_64" ,@%gcc-7.5-x86_64-micro-architectures))))))

and thus, it looses ’hidden’, as shown here:

Toggle snippet (19 lines)
$ guix repl
GNU Guile 3.0.8
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,use(gnu packages gcc)
scheme@(guix-user)> (package-properties gcc-6)
$1 = ((hidden? . #t) (gcc-libc . #f))
scheme@(guix-user)> (package-properties gcc-7)
$2 = ((compiler-cpu-architectures ("aarch64" "armv8-a" "armv8.1-a" "armv8.2-a" "armv8.3-a") ("armhf" "armv7" "armv7-a" "armv7-m" "armv7-r" "armv7e-m" "armv7ve" "armv8-a" "armv8-a+crc" "armv8.1-a" "armv8.1-a+crc" "armv8-m.base"
"armv8-m.main" "armv8-m.main+dsp" "iwmmxt" "iwmmxt2") ("x86_64" "core2" "nehalem" "westmere" "sandybridge" "ivybridge" "haswell" "broadwell" "skylake" "bonnell" "silvermont" "knl" "skylake-avx512" "k8" "k8-sse3" "barcelona" "bdver1"
"bdver2" "bdver3" "bdver4" "znver1" "btver1" "btver2" "geode")))
scheme@(guix-user)>

Because the intent of,

Toggle snippet (6 lines)
(define-public gcc-toolchain-aka-gcc
;; It's natural for users to try "guix install gcc". This package
;; automatically "redirects" them to 'gcc-toolchain'.
(deprecated-package "gcc" gcc-toolchain-10))

is to return the default GCC toolchain when typing ’gcc’ at the command
line. And error for any other version than @10.



Update the default GCC toolchain from 10 to 12 is a core-updates change
and a (almost) full world rebuild. When the issue you are describing is
an issue about an inconsistency with the command-line.


From my point of view, an option would to rename the packages
gcc-toolchain@11 and gcc-toolchain@12 as gcc-toolchain-next@11 and
gcc-toolchain-next@12. Which would be consistent with the rest.

And also fix the bug with the package ’gcc’. :-)


Cheers,
simon
L
L
Lars-Dominik Braun wrote on 21 Dec 2022 10:24
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 60200@debbugs.gnu.org)
Y6LQs9+in964glaz@noor.fritz.box
Hi Simon,

Toggle quote (3 lines)
> Without any specification about the version, if a package name is
> defined at several versions, then the command-line uses the higher
> version of this package.
minor nit-pick: Not the command-line, but everything that uses
specifications. So manifests via SPECIFICATIONS->MANIFEST are also
affected, see:

(use-modules (gnu packages) (guix profiles))
(manifest-entries (specifications->manifest '("gcc-toolchain")))
$2 = (#<<manifest-entry> name: "gcc-toolchain" version: "12.2.0" …)

The -next suffix has the obvious disadvantage that specifications may
become invalid as we move -next to the “regular” package. So maybe
marking packages “default” like the attached patch does could improve
the current situation. Not just for gcc, but also Haskell and Python
come to mind.

Cheers,
Lars
Toggle diff (53 lines)
diff --git a/gnu/packages.scm b/gnu/packages.scm
index 61345f75a9..7e5a6d49c2 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -356,20 +356,24 @@ (define cache
(find-packages-by-name/direct name version))))
(define (find-best-packages-by-name name version)
- "If version is #f, return the list of packages named NAME with the highest
-version numbers; otherwise, return the list of packages named NAME and at
-VERSION."
+ "If version is #f, return the list of packages named NAME with only
+packages marked default? or, if none exist, the highest version numbers;
+otherwise, return the list of packages named NAME and at VERSION."
(if version
(find-packages-by-name name version)
(match (find-packages-by-name name)
(()
'())
((matches ...)
- ;; Return the subset of MATCHES with the higher version number.
- (let ((highest (package-version (first matches))))
- (take-while (lambda (p)
- (string=? (package-version p) highest))
- matches))))))
+ ;; Return the subset of MATCHES which are marked default or those with
+ ;; the higher version number.
+ (let ((highest (package-version (first matches)))
+ (default (filter (lambda (p) (assoc-ref (package-properties p) 'default?)) matches)))
+ (if (not (null? default))
+ default
+ (take-while (lambda (p)
+ (string=? (package-version p) highest))
+ matches)))))))
;; Prevent Guile 3 from inlining this procedure so we can mock it in tests.
(set! find-best-packages-by-name find-best-packages-by-name)
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index b4566b41cc..2d5e0add26 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -3855,7 +3855,10 @@ (define* (make-gcc-toolchain gcc
("libc-static" ,libc "static"))))))
(define-public gcc-toolchain
- (make-gcc-toolchain gcc-final))
+ (let ((parent (make-gcc-toolchain gcc-final)))
+ (package
+ (inherit parent)
+ (properties (alist-cons 'default? #t (package-properties parent))))))
(define-public gcc-toolchain-4.8
(make-gcc-toolchain gcc-4.8))
?