[PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.

Open

Details

4 participants

Danny Milosavljevic
Sarah Morgensen
Jakub K?dzio?ka
Pierre Neidhardt

Owner: unassigned

Submitted by: Pierre Neidhardt

Severity: normal

Pierre Neidhardt wrote on 28 Jul 2020 11:58

Recipients:(address . guix-patches@gnu.org)

Message-ID:20200728095822.28375-1-mail@ambrevar.xyz

* gnu/packages/llvm.scm (llvm)[arguments]: Set configure-flags to build

a dynamic library bundle in the "lib" output.

Add phases to move the /bin and /include directories to the "out" output.

The goal of this change is to reduce the closure size of LLVM dependents.

- The dynamic library bundles saves a few dozen MiB over the separate dynamic

libraries.

- Removing the /bin and the /include directories from the dependent input

saves about 35 MiB for LLVM 10.

---

gnu/packages/llvm.scm | 65 +++++++++++++++++++++++++++++++++++--------

1 file changed, 53 insertions(+), 12 deletions(-)

Toggle diff (97 lines)diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index b7bc21ea6e..3e9d428b9f 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -99,7 +99,7 @@ as \"x86_64-linux\"."
        (base32
         "1pwgm6cr0xr5a0hrbqs1zvsvvjvy0yq1y47c96804wcs795s90yz"))))
     (build-system cmake-build-system)
-    (outputs '("out" "opt-viewer"))
+    (outputs '("out" "opt-viewer" "lib"))
     (native-inputs
      `(("python" ,python-2) ;bytes->str conversion in clang>=3.7 needs python-2
        ("perl"   ,perl)))
@@ -108,12 +108,18 @@ as \"x86_64-linux\"."
     (propagated-inputs
      `(("zlib" ,zlib)))                 ;to use output from llvm-config
     (arguments
-     `(#:configure-flags '("-DCMAKE_SKIP_BUILD_RPATH=FALSE"
-                           "-DCMAKE_BUILD_WITH_INSTALL_RPATH=FALSE"
-                           "-DBUILD_SHARED_LIBS:BOOL=TRUE"
-                           "-DLLVM_ENABLE_FFI:BOOL=TRUE"
-                           "-DLLVM_REQUIRES_RTTI=1" ; For some third-party utilities
-                           "-DLLVM_INSTALL_UTILS=ON") ; Needed for rustc.
+     `(#:configure-flags (list "-DCMAKE_SKIP_BUILD_RPATH=FALSE"
+                               "-DCMAKE_BUILD_WITH_INSTALL_RPATH=FALSE"
+                               ;; LLVM cannot enable BUILD_SHARED_LIBS with LLVM_LINK_LLVM_DYLIB.
+                               ;; "-DBUILD_SHARED_LIBS:BOOL=TRUE"
+                               "-DLLVM_BUILD_LLVM_DYLIB=ON"
+                               "-DLLVM_LINK_LLVM_DYLIB=ON"
+                               (string-append "-DCMAKE_INSTALL_PREFIX=" (assoc-ref %outputs "lib"))
+                               (string-append "-DCMAKE_INSTALL_RPATH=" (assoc-ref %outputs "lib")
+                                              "/lib")
+                               "-DLLVM_ENABLE_FFI:BOOL=TRUE"
+                               "-DLLVM_REQUIRES_RTTI=1" ; For some third-party utilities
+                               "-DLLVM_INSTALL_UTILS=ON") ; Needed for rustc.
 
        ;; Don't use '-g' during the build, to save space.
        #:build-type "Release"
@@ -128,14 +134,49 @@ as \"x86_64-linux\"."
              (setenv "LD_LIBRARY_PATH"
                      (string-append (getcwd) "/lib"))
              #t))
-         (add-after 'install 'install-opt-viewer
+         (add-after 'install 'install-bin
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((out (assoc-ref outputs "out"))
+                    (out-lib (string-append out "/lib"))
+                    (lib-output (assoc-ref outputs "lib"))
+                    (lib-bin (string-append lib-output "/bin")))
+               (mkdir-p out)
+               (rename-file (string-append lib-output "/bin")
+                            (string-append out "/bin"))
+               ;; llvm-config is required by most lib dependents.  It's only a
+               ;; few KiB, so it does not warrant a separate output.
+               (mkdir-p lib-bin)
+               (rename-file (string-append out "/bin/llvm-config")
+                            (string-append lib-bin "/llvm-config"))
+               (rename-file (string-append lib-output "/include")
+                            (string-append out "/include"))
+               (mkdir-p out-lib)
+               (if (file-exists? (string-append lib-output "/lib/cmake"))
+                   (rename-file (string-append lib-output "/lib/cmake")
+                                (string-append out-lib "/cmake"))
+                   ;; The cmake files change location in llvm 3.9.
+                   (begin
+                     (mkdir-p (string-append out "/share/llvm"))
+                     (rename-file (string-append lib-output "/share/llvm/cmake")
+                                  (string-append out "/share/llvm/cmake"))))
+               (for-each
+                (lambda (file)
+                  (rename-file file
+                               (string-append out-lib "/" (basename file))))
+                (find-files (string-append lib-output "/lib") "\\.a$"))
+               (for-each
+                (lambda (file)
+                  (symlink file
+                           (string-append out-lib "/" (basename file))))
+                (find-files (string-append lib-output "/lib") "\\.so")))
+             #t))
+         (add-after 'install 'install-opt-viewer
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((lib-output (assoc-ref outputs "lib"))
                     (opt-viewer-out (assoc-ref outputs "opt-viewer"))
-                    (opt-viewer-share-dir (string-append opt-viewer-out "/share"))
-                    (opt-viewer-dir (string-append opt-viewer-share-dir "/opt-viewer")))
-               (mkdir-p opt-viewer-share-dir)
-               (rename-file (string-append out "/share/opt-viewer")
+                    (opt-viewer-dir (string-append opt-viewer-out "/share/opt-viewer")))
+               (mkdir-p (dirname opt-viewer-dir))
+               (rename-file (string-append lib-output "/share/opt-viewer")
                             opt-viewer-dir))
              #t)))))
     (home-page "https://www.llvm.org")

base-commit: 0e1428ac5dc3a7f1aa68988dd88885009e9706a6
-- 
2.27.0

Pierre Neidhardt wrote on 28 Jul 2020 12:03

[PATCH 2/2] gnu: llvm-3.9.1: Move libraries to "lib" output.

Recipients:(address . 42576@debbugs.gnu.org)

Message-ID:20200728100341.31365-1-mail@ambrevar.xyz

* gnu/packages/llvm.scm (llvm-3.9.1)[outputs]: Add "lib".

---

gnu/packages/llvm.scm | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (15 lines)diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 3e9d428b9f..1c099bbdd3 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -690,7 +690,7 @@ components which highly leverage existing libraries in the larger LLVM Project."
       (sha256
        (base32
         "1vi9sf7rx1q04wj479rsvxayb6z740iaz3qniwp266fgp5a07n8z"))))
-    (outputs '("out"))
+    (outputs '("out" "lib"))
     (arguments
      (substitute-keyword-arguments (package-arguments llvm)
        ((#:phases phases)
-- 
2.27.0

Pierre Neidhardt wrote on 28 Jul 2020 12:07

Re: bug#42576: Acknowledgement ([PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.)

Recipients:(address . 42576@debbugs.gnu.org)

Message-ID:877duodkao.fsf@ambrevar.xyz

This patch is meant for core-updates since it rebuilds every LLVm

dependent, so more than 1000+ packages. I've only tested against master

though.

It's not ready yet.

Since LLVM@10 takes a long time to compile, it's cumbersome to iterate

against it.

Instead, I found out that LLVM@3.5 builds much faster and the `pure'

package is a great candidate for testing.

I've added ("llvm" llvm-3.5 "lib") to the dependencies of pure.

But ("llvm" llvm-3.5) must be included as a native input because that's

where the C headers (include) files are.

The end result still depends on llvm "out" because the .so files has

references to the headers.

We could move the headers to a separate output, but LLVM@10 headers are

17MiB big already, so I'd rather not include them in the closure of

every package.

Any idea how to remove them?

Pierre Neidhardt

https://ambrevar.xyz/

Danny Milosavljevic wrote on 1 Aug 2020 12:59

Re: [bug#42576] [PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 42576@debbugs.gnu.org)

Message-ID:20200801125953.2e898f63@scratchpost.org

Hmm, glib for example moves the binaries to a "bin" output while retaining the

libraries in "out".

Any reason this patch moves the libraries and not the binaries? Especially

since there are no end-user facing programs in llvm anyway (programmer-facing

maybe). I mean we can...

Also, I would have thought that those llvm programs reference the llvm

libraries anyway, and thus in the end no space could be saved. Is this

not the case?

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl8lSykACgkQ5xo1VCww

uqX76QgAqCR8KGBc1m2VWuH7SNr+d1XlIAf+dVBIBgicYVPKwGJnJObHRnxtOYiX

P6euUmxjgkp2NUuAP6ldPcUGs43qcX9o8glln1mlZOeauvYScFbKg4qW3iG5tbMS

kzoQ4Mb+7Z+c6mcAvkSEg8c9qxEECwKSOjs4tx0N0XMk+Z0ebEOkYmLjECrvKtkT

PZMcPb+0xrLrM4rtoeRB1IOn3WYvrtXtvJ7KehDowfQULp+7SLpoAlkJMqCdTy2X

k5hFtU3ZiKLqP9XWrJ/0e9JgZOi2Xubj2Km1t8Q+4gCcoKSJIRMNcMcGYUs96/wA

4QwP9MziuB/U1STZF3fexf83pxqGlw==

=IXaz

-----END PGP SIGNATURE-----

Pierre Neidhardt wrote on 1 Aug 2020 13:18

Recipients:(name . Danny Milosavljevic)(address . dannym@scratchpost.org)

Message-ID:878seyaa38.fsf@ambrevar.xyz

Hi Danny!

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (5 lines)

> Hmm, glib for example moves the binaries to a "bin" output while retaining the

> libraries in "out".

> Any reason this patch moves the libraries and not the binaries?

Ludo suggested this way I think out of consistency with the rest.

Toggle quote (4 lines)

> Also, I would have thought that those llvm programs reference the llvm

> libraries anyway, and thus in the end no space could be saved. Is this

> not the case?

This patch not about saving space for LLVM programs, but packages that depend

on LLVM libraries and which don't need the programs.

Cheers!

Pierre Neidhardt

https://ambrevar.xyz/

Jakub K?dzio?ka wrote on 7 Aug 2020 20:09

Re: [PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . guix-patches@gnu.org)

Message-ID:20200807180944.ujbtmm3ljx6rntnt@gravity

On Tue, Jul 28, 2020 at 11:58:22AM +0200, Pierre Neidhardt wrote:

Toggle quote (12 lines)> * gnu/packages/llvm.scm (llvm)[arguments]: Set configure-flags to build
> a dynamic library bundle in the "lib" output.
> Add phases to move the /bin and /include directories to the "out" output.
> 
> The goal of this change is to reduce the closure size of LLVM dependents.
> 
> - The dynamic library bundles saves a few dozen MiB over the separate dynamic
>   libraries.
> 
> - Removing the /bin and the /include directories from the dependent input
>   saves about 35 MiB for LLVM 10.

Pierre,

please note that cmake seems to store a list of files installed by the

package, which broke cmake-using dependents of clang when we attempted

to change the set of files installed: http://issues.guix.gnu.org/41872

Did you try building something that depends on LLVM and uses

cmake-build-system?

Regards,

Jakub K?dzio?ka

Pierre Neidhardt wrote on 8 Aug 2020 10:49

Recipients:(name . Jakub K?dzio?ka)(address . kuba@kadziolka.net)(address . guix-patches@gnu.org)

Message-ID:87mu35o73x.fsf@ambrevar.xyz

Hi Jakub,

Jakub K?dzio?ka <kuba@kadziolka.net> writes:

Toggle quote (9 lines)

> Pierre,

> please note that cmake seems to store a list of files installed by the

> package, which broke cmake-using dependents of clang when we attempted

> to change the set of files installed: http://issues.guix.gnu.org/41872

> Did you try building something that depends on LLVM and uses

> cmake-build-system?

It is broken indeed, this is one of the things that need to be fixed

before we can merge this patch.

In issue 41872 the problem is with missing .a files.

A similar issue occurs here since we move files around, but the CMake

files are not aware of the move.

One possible fix would be to patch the CMake files with the new locations.

This is rather inelegant though.

A better fix would be to configure CMake to produce the various files

directly to the right location, e.g. the binary files, the headers and

the libraries to their own respective outputs.

Any clue if we can do that?

Pierre Neidhardt

https://ambrevar.xyz/

Sarah Morgensen wrote on 24 Sep 2021 02:41

Re: [bug#42576] [PATCH 1/2] gnu: llvm: Move dynamic libraries to a separate "lib" output.

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)

Message-ID:86wnn6iz48.fsf@mgsn.dev

Hello Pierre,

Thanks for your pioneering work on this!

Pierre Neidhardt <mail@ambrevar.xyz> writes:

Toggle quote (29 lines)> Hi Jakub,
>
> Jakub K?dzio?ka <kuba@kadziolka.net> writes:
>
>> Pierre,
>>
>> please note that cmake seems to store a list of files installed by the
>> package, which broke cmake-using dependents of clang when we attempted
>> to change the set of files installed: http://issues.guix.gnu.org/41872
>>
>> Did you try building something that depends on LLVM and uses
>> cmake-build-system?
>
> It is broken indeed, this is one of the things that need to be fixed
> before we can merge this patch.
>
> In issue 41872 the problem is with missing .a files.
> A similar issue occurs here since we move files around, but the CMake
> files are not aware of the move.
>
> One possible fix would be to patch the CMake files with the new locations.
> This is rather inelegant though.
>
> A better fix would be to configure CMake to produce the various files
> directly to the right location, e.g. the binary files, the headers and
> the libraries to their own respective outputs.
>
> Any clue if we can do that?

There has been some recent work on this in the LLVM project [0] and in

Nix [1][2], based on the `GnuInstallDirs' CMake module. It looks like

this would be doable for us, especially if we move in the direction of

'dev' outputs [3].

[0] https://reviews.llvm.org/D99484

[1] https://github.com/NixOS/nikpkgs/pull/111487

[2] https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/compilers/llvm/12

[3] https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00107.html

Perhaps it's time to revive this effort (particularly for clang, which

is a behemoth)?

Sarah

Pierre Neidhardt wrote on 24 Sep 2021 09:10

Recipients:(name . Sarah Morgensen)(address . iskarian@mgsn.dev)

Message-ID:87o88iih35.fsf@ambrevar.xyz

Hello Sarah, thanks for the update.

Unfortunately I won't be able to commit much time to the Guix project at

the moment, but if you want to take over this initiative, you're more

than welcome :)

Good luck!

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 42576@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 42576

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date