[PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.

OpenSubmitted by Pierre Neidhardt.
Details
4 participants
  • Danny Milosavljevic
  • Sarah Morgensen
  • Jakub Kądziołka
  • Pierre Neidhardt
Owner
unassigned
Severity
normal
P
P
Pierre Neidhardt wrote on 28 Jul 2020 11:58
(address . guix-patches@gnu.org)
20200728095822.28375-1-mail@ambrevar.xyz
* gnu/packages/llvm.scm (llvm)[arguments]: Set configure-flags to build
a dynamic library bundle in the "lib" output.
Add phases to move the /bin and /include directories to the "out" output.

The goal of this change is to reduce the closure size of LLVM dependents.

- The dynamic library bundles saves a few dozen MiB over the separate dynamic
libraries.

- Removing the /bin and the /include directories from the dependent input
saves about 35 MiB for LLVM 10.
---
gnu/packages/llvm.scm | 65 +++++++++++++++++++++++++++++++++++--------
1 file changed, 53 insertions(+), 12 deletions(-)

Toggle diff (97 lines)
diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index b7bc21ea6e..3e9d428b9f 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -99,7 +99,7 @@ as \"x86_64-linux\"."
        (base32
         "1pwgm6cr0xr5a0hrbqs1zvsvvjvy0yq1y47c96804wcs795s90yz"))))
     (build-system cmake-build-system)
-    (outputs '("out" "opt-viewer"))
+    (outputs '("out" "opt-viewer" "lib"))
     (native-inputs
      `(("python" ,python-2) ;bytes->str conversion in clang>=3.7 needs python-2
        ("perl"   ,perl)))
@@ -108,12 +108,18 @@ as \"x86_64-linux\"."
     (propagated-inputs
      `(("zlib" ,zlib)))                 ;to use output from llvm-config
     (arguments
-     `(#:configure-flags '("-DCMAKE_SKIP_BUILD_RPATH=FALSE"
-                           "-DCMAKE_BUILD_WITH_INSTALL_RPATH=FALSE"
-                           "-DBUILD_SHARED_LIBS:BOOL=TRUE"
-                           "-DLLVM_ENABLE_FFI:BOOL=TRUE"
-                           "-DLLVM_REQUIRES_RTTI=1" ; For some third-party utilities
-                           "-DLLVM_INSTALL_UTILS=ON") ; Needed for rustc.
+     `(#:configure-flags (list "-DCMAKE_SKIP_BUILD_RPATH=FALSE"
+                               "-DCMAKE_BUILD_WITH_INSTALL_RPATH=FALSE"
+                               ;; LLVM cannot enable BUILD_SHARED_LIBS with LLVM_LINK_LLVM_DYLIB.
+                               ;; "-DBUILD_SHARED_LIBS:BOOL=TRUE"
+                               "-DLLVM_BUILD_LLVM_DYLIB=ON"
+                               "-DLLVM_LINK_LLVM_DYLIB=ON"
+                               (string-append "-DCMAKE_INSTALL_PREFIX=" (assoc-ref %outputs "lib"))
+                               (string-append "-DCMAKE_INSTALL_RPATH=" (assoc-ref %outputs "lib")
+                                              "/lib")
+                               "-DLLVM_ENABLE_FFI:BOOL=TRUE"
+                               "-DLLVM_REQUIRES_RTTI=1" ; For some third-party utilities
+                               "-DLLVM_INSTALL_UTILS=ON") ; Needed for rustc.
 
        ;; Don't use '-g' during the build, to save space.
        #:build-type "Release"
@@ -128,14 +134,49 @@ as \"x86_64-linux\"."
              (setenv "LD_LIBRARY_PATH"
                      (string-append (getcwd) "/lib"))
              #t))
-         (add-after 'install 'install-opt-viewer
+         (add-after 'install 'install-bin
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((out (assoc-ref outputs "out"))
+                    (out-lib (string-append out "/lib"))
+                    (lib-output (assoc-ref outputs "lib"))
+                    (lib-bin (string-append lib-output "/bin")))
+               (mkdir-p out)
+               (rename-file (string-append lib-output "/bin")
+                            (string-append out "/bin"))
+               ;; llvm-config is required by most lib dependents.  It's only a
+               ;; few KiB, so it does not warrant a separate output.
+               (mkdir-p lib-bin)
+               (rename-file (string-append out "/bin/llvm-config")
+                            (string-append lib-bin "/llvm-config"))
+               (rename-file (string-append lib-output "/include")
+                            (string-append out "/include"))
+               (mkdir-p out-lib)
+               (if (file-exists? (string-append lib-output "/lib/cmake"))
+                   (rename-file (string-append lib-output "/lib/cmake")
+                                (string-append out-lib "/cmake"))
+                   ;; The cmake files change location in llvm 3.9.
+                   (begin
+                     (mkdir-p (string-append out "/share/llvm"))
+                     (rename-file (string-append lib-output "/share/llvm/cmake")
+                                  (string-append out "/share/llvm/cmake"))))
+               (for-each
+                (lambda (file)
+                  (rename-file file
+                               (string-append out-lib "/" (basename file))))
+                (find-files (string-append lib-output "/lib") "\\.a$"))
+               (for-each
+                (lambda (file)
+                  (symlink file
+                           (string-append out-lib "/" (basename file))))
+                (find-files (string-append lib-output "/lib") "\\.so")))
+             #t))
+         (add-after 'install 'install-opt-viewer
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((lib-output (assoc-ref outputs "lib"))
                     (opt-viewer-out (assoc-ref outputs "opt-viewer"))
-                    (opt-viewer-share-dir (string-append opt-viewer-out "/share"))
-                    (opt-viewer-dir (string-append opt-viewer-share-dir "/opt-viewer")))
-               (mkdir-p opt-viewer-share-dir)
-               (rename-file (string-append out "/share/opt-viewer")
+                    (opt-viewer-dir (string-append opt-viewer-out "/share/opt-viewer")))
+               (mkdir-p (dirname opt-viewer-dir))
+               (rename-file (string-append lib-output "/share/opt-viewer")
                             opt-viewer-dir))
              #t)))))
     (home-page "https://www.llvm.org")

base-commit: 0e1428ac5dc3a7f1aa68988dd88885009e9706a6
-- 
2.27.0
P
P
Pierre Neidhardt wrote on 28 Jul 2020 12:03
[PATCH 2/2] gnu: llvm-3.9.1: Move libraries to "lib" output.
(address . 42576@debbugs.gnu.org)
20200728100341.31365-1-mail@ambrevar.xyz
* gnu/packages/llvm.scm (llvm-3.9.1)[outputs]: Add "lib".
---
gnu/packages/llvm.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (15 lines)
diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 3e9d428b9f..1c099bbdd3 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -690,7 +690,7 @@ components which highly leverage existing libraries in the larger LLVM Project."
       (sha256
        (base32
         "1vi9sf7rx1q04wj479rsvxayb6z740iaz3qniwp266fgp5a07n8z"))))
-    (outputs '("out"))
+    (outputs '("out" "lib"))
     (arguments
      (substitute-keyword-arguments (package-arguments llvm)
        ((#:phases phases)
-- 
2.27.0
P
P
Pierre Neidhardt wrote on 28 Jul 2020 12:07
Re: bug#42576: Acknowledgement ([PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.)
(address . 42576@debbugs.gnu.org)
877duodkao.fsf@ambrevar.xyz
This patch is meant for core-updates since it rebuilds every LLVm
dependent, so more than 1000+ packages. I've only tested against master
though.

It's not ready yet.

Since LLVM@10 takes a long time to compile, it's cumbersome to iterate
against it.
Instead, I found out that LLVM@3.5 builds much faster and the `pure'
package is a great candidate for testing.

I've added ("llvm" llvm-3.5 "lib") to the dependencies of pure.
But ("llvm" llvm-3.5) must be included as a native input because that's
where the C headers (include) files are.

The end result still depends on llvm "out" because the .so files has
references to the headers.

We could move the headers to a separate output, but LLVM@10 headers are
17MiB big already, so I'd rather not include them in the closure of
every package.

Any idea how to remove them?

--
Pierre Neidhardt
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl8f+P8ACgkQm9z0l6S7
zH98QAf8DC6LmFtVJ/ccpo9G3VQFhMG4TQmY8Ebud7L8qkQnjp2Pb6b7GZoSHVGL
nQCoVGe7+qiqLZm+Ic9nZ8cw3Ooh4T9PXrVl460IVJQt/kjYOncUwqdZ4JmTBr9I
PY8vZS5MpF0EZYtrzzxGwxbJq2rhUT4weihKNf9lJIGHfYtH9zfE2LKSOhfIPSjq
RkqESBD5pnXO/fOjPeJnMiStc1SB2lmfFUo0/Ee70tLHblyf05XZw/hjY6d7K2g3
grXpcPyDmpf4mRbfyufwDfLywUJoH/KJGOq1a8iXztAt1Fn+Pg8KXr4LcHFFYnx7
AcXQ23BIL43dyVc5cHmI9+ggSM06lw==
=8yjU
-----END PGP SIGNATURE-----

D
D
Danny Milosavljevic wrote on 1 Aug 2020 12:59
Re: [bug#42576] [PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.
(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 42576@debbugs.gnu.org)
20200801125953.2e898f63@scratchpost.org
Hmm, glib for example moves the binaries to a "bin" output while retaining the
libraries in "out".

Any reason this patch moves the libraries and not the binaries? Especially
since there are no end-user facing programs in llvm anyway (programmer-facing
maybe). I mean we can...

Also, I would have thought that those llvm programs reference the llvm
libraries anyway, and thus in the end no space could be saved. Is this
not the case?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl8lSykACgkQ5xo1VCww
uqX76QgAqCR8KGBc1m2VWuH7SNr+d1XlIAf+dVBIBgicYVPKwGJnJObHRnxtOYiX
P6euUmxjgkp2NUuAP6ldPcUGs43qcX9o8glln1mlZOeauvYScFbKg4qW3iG5tbMS
kzoQ4Mb+7Z+c6mcAvkSEg8c9qxEECwKSOjs4tx0N0XMk+Z0ebEOkYmLjECrvKtkT
PZMcPb+0xrLrM4rtoeRB1IOn3WYvrtXtvJ7KehDowfQULp+7SLpoAlkJMqCdTy2X
k5hFtU3ZiKLqP9XWrJ/0e9JgZOi2Xubj2Km1t8Q+4gCcoKSJIRMNcMcGYUs96/wA
4QwP9MziuB/U1STZF3fexf83pxqGlw==
=IXaz
-----END PGP SIGNATURE-----


P
P
Pierre Neidhardt wrote on 1 Aug 2020 13:18
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
878seyaa38.fsf@ambrevar.xyz
Hi Danny!

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (5 lines)
> Hmm, glib for example moves the binaries to a "bin" output while retaining the
> libraries in "out".
>
> Any reason this patch moves the libraries and not the binaries?

Ludo suggested this way I think out of consistency with the rest.

Toggle quote (4 lines)
> Also, I would have thought that those llvm programs reference the llvm
> libraries anyway, and thus in the end no space could be saved. Is this
> not the case?

This patch not about saving space for LLVM programs, but packages that depend
on LLVM libraries and which don't need the programs.

Cheers!

--
Pierre Neidhardt
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl8lT2sACgkQm9z0l6S7
zH/z7QgAl/Rsvlvwt/n1hXtdF1xGZnPbuuhYxMvfuIfpZp1Kpln/vaB98OuIgidf
onowNIbi7AXHzkE1yRmE7Hjjiw80wlBeuv2kelSEFLnqkBos86rHRspEq4QIbYEz
iqiJsGefHFXCXOK02teItfckypIZ0OjiMGjzbaVleVkhu/QlLrfgN4xXcQ1SHfEB
Xs/BGb1fmB5MLdWcCG8yWg8w8+dWFeVvQ6eqgHPb/51G0qZPoSLAJ759pbN5beWS
L4NyH4Xvfyje+SD6WRsrgpAGYngYU7EhXLq9yBbmWPTYkpGWsNkSu0HK3YWwaMna
t2nZoE0583xBFinDs4t6ONW3U3yOUg==
=/bjc
-----END PGP SIGNATURE-----

J
J
Jakub Kądziołka wrote on 7 Aug 2020 20:09
Re: [PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.
(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . guix-patches@gnu.org)
20200807180944.ujbtmm3ljx6rntnt@gravity
On Tue, Jul 28, 2020 at 11:58:22AM +0200, Pierre Neidhardt wrote:
Toggle quote (12 lines)
> * gnu/packages/llvm.scm (llvm)[arguments]: Set configure-flags to build
> a dynamic library bundle in the "lib" output.
> Add phases to move the /bin and /include directories to the "out" output.
>
> The goal of this change is to reduce the closure size of LLVM dependents.
>
> - The dynamic library bundles saves a few dozen MiB over the separate dynamic
> libraries.
>
> - Removing the /bin and the /include directories from the dependent input
> saves about 35 MiB for LLVM 10.

Pierre,

please note that cmake seems to store a list of files installed by the
package, which broke cmake-using dependents of clang when we attempted
to change the set of files installed: http://issues.guix.gnu.org/41872

Did you try building something that depends on LLVM and uses
cmake-build-system?

Regards,
Jakub Kądziołka
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE5Xa/ss9usT31cTO54xWnWEYTFWQFAl8tmOgACgkQ4xWnWEYT
FWTn0BAAn6ceQ0t1AQtTJ6zqE7LRJVVBLIuErYwjbC7sraSUQXiGntt2qaNgcQBx
kVJ/dNtU78o3247v7pkos3C6LPfXDv5zDWel0w7mgmNVZG//ocwRd0lnvmVgDjqC
Myt1ywQ6UExbz16y+fRQl57R3eynsj1KlC/h1tiNORzGJYUhx/LrpNSb9OM7KyrH
s5AlsFKvcEmQJSn98UXR0JcSaxqJloTGV2JrelRxlSjLE3DRSX6Q9ZpgP4T/TWAL
an78bP6oMtdfma5lojkYFCJ/YQ0svYxoua7paijxw8iuvyKklJlLsuoIZCkE5AO6
GvIuzekzKMhAjocJChmot6+nrzVNSfOpLupR2vjDfkXEbAT8DJ60WhuqFaETv/zW
xiC9QwFIGWN66l4EfjOik2/UdLOT6HiScwCab+GsAN0mp9DPI2Nioly0vcKHTXDR
r+Xosf6kyZQF+PHKfYgd3Ewo8DishJmitrScKgAjhxIj7sfH0RH/+MQ+NH+W9+NG
f8rrBpsOkpQ60psp+7v2XY1/4opWJLpL7YNV9RXAyjX481ECVBwNMRjreLi3poKG
tOKk10FX1hH6ZM2vtCMglnONqm8BkvVxMl6cg/HA1mZMTHUHOkHsaYj/mJYUlvKD
LIZUviRuilDbCVyxiMsTx8dYLk4GmdAWPSWF+c/TTe83XUp5ybA=
=ng0R
-----END PGP SIGNATURE-----


P
P
Pierre Neidhardt wrote on 8 Aug 2020 10:49
(name . Jakub Kądziołka)(address . kuba@kadziolka.net)(address . guix-patches@gnu.org)
87mu35o73x.fsf@ambrevar.xyz
Hi Jakub,

Jakub Kądziołka <kuba@kadziolka.net> writes:

Toggle quote (9 lines)
> Pierre,
>
> please note that cmake seems to store a list of files installed by the
> package, which broke cmake-using dependents of clang when we attempted
> to change the set of files installed: http://issues.guix.gnu.org/41872
>
> Did you try building something that depends on LLVM and uses
> cmake-build-system?

It is broken indeed, this is one of the things that need to be fixed
before we can merge this patch.

In issue 41872 the problem is with missing .a files.
A similar issue occurs here since we move files around, but the CMake
files are not aware of the move.

One possible fix would be to patch the CMake files with the new locations.
This is rather inelegant though.

A better fix would be to configure CMake to produce the various files
directly to the right location, e.g. the binary files, the headers and
the libraries to their own respective outputs.

Any clue if we can do that?

--
Pierre Neidhardt
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl8uZwIACgkQm9z0l6S7
zH8NKQgArGqhte+LWqzcYAdY7jxi6bTmkcYuP9zfLtJ6KpWqKcRGMfXh5ghqbtX2
RaNeP40rK/pfkNq2GJ0PICv6V42MY5Hfxa5Xcm3t35NPEk5AJxew2NGpboUajby+
eV+PgQu+2t35IH4Q0JElSXLw4J+uiiDQW37OmUZGzyGONwlB71M6WJCUYLXsTtKK
TlGnrtgdS8uTmWjTvXbA827lMhRt4sSF1xHYqHsB+ykOUyRQBcBfNapfRUvg5zsb
0dqDJOQR36anQCssLnnza7Pc2rn8DsCWe2vVTUHUpEfpkUPWj+QqSxScWU2+yMgS
CNYC2qOa1Dq1Viv2D4TiJZ3lZ3oTdw==
=Kl6R
-----END PGP SIGNATURE-----

S
S
Sarah Morgensen wrote on 24 Sep 2021 02:41
Re: [bug#42576] [PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.
(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)
86wnn6iz48.fsf@mgsn.dev
Hello Pierre,

Thanks for your pioneering work on this!

Pierre Neidhardt <mail@ambrevar.xyz> writes:

Toggle quote (29 lines)
> Hi Jakub,
>
> Jakub Kądziołka <kuba@kadziolka.net> writes:
>
>> Pierre,
>>
>> please note that cmake seems to store a list of files installed by the
>> package, which broke cmake-using dependents of clang when we attempted
>> to change the set of files installed: http://issues.guix.gnu.org/41872
>>
>> Did you try building something that depends on LLVM and uses
>> cmake-build-system?
>
> It is broken indeed, this is one of the things that need to be fixed
> before we can merge this patch.
>
> In issue 41872 the problem is with missing .a files.
> A similar issue occurs here since we move files around, but the CMake
> files are not aware of the move.
>
> One possible fix would be to patch the CMake files with the new locations.
> This is rather inelegant though.
>
> A better fix would be to configure CMake to produce the various files
> directly to the right location, e.g. the binary files, the headers and
> the libraries to their own respective outputs.
>
> Any clue if we can do that?

There has been some recent work on this in the LLVM project [0] and in
Nix [1][2], based on the `GnuInstallDirs' CMake module. It looks like
this would be doable for us, especially if we move in the direction of
'dev' outputs [3].


Perhaps it's time to revive this effort (particularly for clang, which
is a behemoth)?

--
Sarah
P
P
Pierre Neidhardt wrote on 24 Sep 2021 09:10
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)
87o88iih35.fsf@ambrevar.xyz
Hello Sarah, thanks for the update.

Unfortunately I won't be able to commit much time to the Guix project at
the moment, but if you want to take over this initiative, you're more
than welcome :)

Good luck!
-----BEGIN PGP SIGNATURE-----

iQFGBAEBCAAwFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAmFNef4SHG1haWxAYW1i
cmV2YXIueHl6AAoJEJvc9Jeku8x/qK4H/jLisZ1Pk+nqtTkpfoDjMfD22DYm+weT
DBEuCybF1tDStj8OqNqxkgns8qAVsWcU6ZkQxRNm0jxRs7UQz/XkfSvH+byXdDSz
O0wI4VlhT6YWpgHpNPxZdRCqnzLkVd2r8quisNS7bLv1XNbqoGR3krMeiJ/DWScL
LFCt7tCV5IcnEcKl4bzkyoB40xIafv5H+CiVPmCG8NvqF2DK+GBemQ/Sz4Cpw6qs
j6xGnqPO3BQqQNnzGDuSdeQUbbaqo9BoY1pXWohUQZ4qKIRoAlYeMI4VwADJioQ5
Q8dioPUvwsYtBCy01/R3A8qUGOtDsCHlk1v95y4oOzfgSXGW3ztvpqE=
=u7DJ
-----END PGP SIGNATURE-----

?