[PATCH] transformations: Add multituned-package.

  • Open
  • quality assurance status badge
Details
2 participants
  • Efraim Flashner
  • Ludovic Courtès
Owner
unassigned
Submitted by
Efraim Flashner
Severity
normal
E
E
Efraim Flashner wrote on 8 Nov 11:44 +0100
(address . guix-patches@gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
d3571383007eb50c7156d1ef88f9fb159b3fbb3d.1731062591.git.efraim@flashner.co.il
* guix/transformations.scm (package-tuned-for-psabi,
multituned-package): New variables.

Change-Id: I09ac7ae9fc2bcd9aa712b3c30fef807bc7d55895
---

This allows wrapping a package definition in multituned-package, ie:

(define-public opus
(multituned-package
(package
...)))

I'm not sure where to go with this patch from here. This will provide
the psabi libraries for x86_64 and powerpc64le so they get most of the
benefits from tuning for the architecture but without needing to specify
which architecture to tune for. It should also provide a nice boost for
guix packs and docker images and the like.

The downside with using this by default is the larger package size due
to the extra versions of the libraries, and if it is used then the
regular --tune is disabled for that package.

I think adding it as a '--tune=generic' or '--tune=psabi' would be a
nice way to use it.

guix/transformations.scm | 75 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 75 insertions(+)

Toggle diff (99 lines)
diff --git a/guix/transformations.scm b/guix/transformations.scm
index ea8b7a08443..4787e016b5d 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -60,6 +60,7 @@ (define-module (guix transformations)
tunable-package?
tuned-package
+ multituned-package
show-transformation-options-help
transformation-option-key?
@@ -634,6 +635,80 @@ (define (tuned-package p micro-architecture)
;; call 'tuned-package' again on this one.
,@(alist-delete 'tunable? (package-properties p))))))
+(define (package-tuned-for-psabi p psabi)
+ (let ((base (tuned-package p psabi)))
+ (package/inherit base
+ (name (string-append (package-name base) "-" psabi))
+ (arguments
+ (substitute-keyword-arguments (package-arguments base)
+ ((#:configure-flags flags #~'())
+ #~(append
+ (list
+ #$@(if (eq? (build-system-name (package-build-system p)) ; not base
+ (quote cmake-build-system))
+ #~((string-append "-DCMAKE_INSTALL_LIBDIR=lib/glibc-hwcaps/"
+ #$psabi))
+ #~((string-append "--libdir=" #$output
+ "/lib/glibc-hwcaps/" #$psabi))))
+ #$flags))
+ ((#:phases phases #~%standard-phases)
+ #~(modify-phases #$phases
+ (add-after 'install 'remove-extra-files
+ (lambda _
+ (for-each (lambda (dir)
+ (when (file-exists? (string-append #$output dir))
+ (delete-file-recursively
+ (string-append #$output dir))))
+ (list (string-append "/lib/glibc-hwcaps/"
+ #$psabi "/cmake")
+ (string-append "/lib/glibc-hwcaps/"
+ #$psabi "/pkgconfig")
+ "/bin" "/etc" "/include" "/libexec"
+ "/sbin" "/share" "/var")))))))))))
+
+(define (multituned-package p)
+ (package/inherit p
+ (arguments
+ (substitute-keyword-arguments (package-arguments p)
+ ((#:phases phases #~%standard-phases)
+ (if (or (target-x86-64?)
+ (target-ppc64le?))
+ #~(modify-phases #$phases
+ (add-after 'install 'install-optimized-libraries
+ (lambda* (#:key inputs outputs #:allow-other-keys)
+ (let ((hwcaps "/lib/glibc-hwcaps/"))
+ (for-each
+ (lambda (psabi)
+ (copy-recursively
+ (string-append
+ (assoc-ref inputs (string-append
+ #$(package-name p) "-" psabi))
+ hwcaps psabi)
+ (string-append #$output hwcaps psabi)))
+ #$(cond ((target-x86-64?)
+ #~(list "x86-64-v2" "x86-64-v3" "x86-64-v4"))
+ ((target-ppc64le?)
+ #~(list "power9" "power10"))
+ (#t #~'())))))))
+ phases))))
+ (inputs
+ (cond ((target-x86-64?)
+ (modify-inputs (package-inputs p)
+ (append (package-tuned-for-psabi p "x86-64-v2")
+ (package-tuned-for-psabi p "x86-64-v3")
+ (package-tuned-for-psabi p "x86-64-v4"))))
+ ((target-ppc64le?)
+ (modify-inputs (package-inputs p)
+ (append (package-tuned-for-psabi p "power9")
+ (package-tuned-for-psabi p "power10"))))
+ (#t (package-inputs p))))
+ ;; With the addition of the psABIs this package should not be tuned.
+ (properties
+ (if (or (target-x86-64?)
+ (target-ppc64le?))
+ '((alist-delete 'tunable? (package-properties p)))
+ (package-properties p)))))
+
(define (tunable-package? package)
"Return true if package PACKAGE is \"tunable\"--i.e., if tuning it for the
host CPU is worthwhile."

base-commit: 2a6d96425eea57dc6dd48a2bec16743046e32e06
--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
L
L
Ludovic Courtès wrote on 12 Nov 11:31 +0100
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87serwu4iq.fsf@gnu.org
Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

Toggle quote (26 lines)
> * guix/transformations.scm (package-tuned-for-psabi,
> multituned-package): New variables.
>
> Change-Id: I09ac7ae9fc2bcd9aa712b3c30fef807bc7d55895
> ---
>
> This allows wrapping a package definition in multituned-package, ie:
>
> (define-public opus
> (multituned-package
> (package
> ...)))
>
> I'm not sure where to go with this patch from here. This will provide
> the psabi libraries for x86_64 and powerpc64le so they get most of the
> benefits from tuning for the architecture but without needing to specify
> which architecture to tune for. It should also provide a nice boost for
> guix packs and docker images and the like.
>
> The downside with using this by default is the larger package size due
> to the extra versions of the libraries, and if it is used then the
> regular --tune is disabled for that package.
>
> I think adding it as a '--tune=generic' or '--tune=psabi' would be a
> nice way to use it.

Should that be a package transformation though? Could we instead have a
build system trick or the ‘multituned-package’ procedure exposed so
build the package several times and fill in lib/glibc-hwcaps?

That way, packagers would explicitly choose this technique for select
packages, which would then no longer need the ‘tunable?’ property.

The question becomes: how would we choose which packages is eligible to
this technique as opposed to ‘--tune’? Intuitively, I would use that
for general-purpose packages like ‘opus’, but keep ‘--tune’ for more
niche/scientific packages.

WDYT?

Thanks,
Ludo’.
E
E
Efraim Flashner wrote on 23 Nov 19:35 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
Z0IgZDbD7fZM8mn4@3900XT
On Tue, Nov 12, 2024 at 11:31:57AM +0100, Ludovic Courtès wrote:
Toggle quote (34 lines)
> Hi,
>
> Efraim Flashner <efraim@flashner.co.il> skribis:
>
> > * guix/transformations.scm (package-tuned-for-psabi,
> > multituned-package): New variables.
> >
> > Change-Id: I09ac7ae9fc2bcd9aa712b3c30fef807bc7d55895
> > ---
> >
> > This allows wrapping a package definition in multituned-package, ie:
> >
> > (define-public opus
> > (multituned-package
> > (package
> > ...)))
> >
> > I'm not sure where to go with this patch from here. This will provide
> > the psabi libraries for x86_64 and powerpc64le so they get most of the
> > benefits from tuning for the architecture but without needing to specify
> > which architecture to tune for. It should also provide a nice boost for
> > guix packs and docker images and the like.
> >
> > The downside with using this by default is the larger package size due
> > to the extra versions of the libraries, and if it is used then the
> > regular --tune is disabled for that package.
> >
> > I think adding it as a '--tune=generic' or '--tune=psabi' would be a
> > nice way to use it.
>
> Should that be a package transformation though? Could we instead have a
> build system trick or the ‘multituned-package’ procedure exposed so
> build the package several times and fill in lib/glibc-hwcaps?

I figured (guix transformations) worked well since it was using an
existing transformation to generate the different variants.

I'm not sure what you mean by a build system trick, like build the
package multiple times in one like with x265?

This exposes the multituned-package procedure (looks like I mistyped in
the commit message) which takes a package and can be just added on top
of an existing package definition, like hidden-package can.

Toggle quote (3 lines)
> That way, packagers would explicitly choose this technique for select
> packages, which would then no longer need the ‘tunable?’ property.

Something like a flag in the build-system like #:tests? is? Or like a
package property?

Toggle quote (5 lines)
> The question becomes: how would we choose which packages is eligible to
> this technique as opposed to ‘--tune’? Intuitively, I would use that
> for general-purpose packages like ‘opus’, but keep ‘--tune’ for more
> niche/scientific packages.

Unfortunately I think just like with choosing what to do with tunable?
we end up in the same spot deciding somewhat arbitrarily what to provide
the psabi options for and what not to.

Toggle quote (5 lines)
> WDYT?
>
> Thanks,
> Ludo’.

Looking more at the patch I've remembered that I've only taken care of
the gnu- and cmake- build-systems. I was going to add about tuning the
package on other architectures, like aarch64, but it seems I fixed that
already.

If it were just the configure-flags and the post-installation removal of
files then I could see trying to make something that could be inserted
manually into specific packages, but the whole thing rests on the
tuned-package procedure actually producing libraries tuned for the
different psABIs.

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmdCIF0ACgkQQarn3Mo9
g1EJlBAAsDrZYDFCOHNOdJbvN0AZg6aMrTGoAKKTJ2ElwT300Y76STETfw0EZvKm
vriX82SWf8/FWmkvSlyI+xjKvVbWW3wYL9he7/TjSIt0Un2JJrox31t598ug/qsI
KQ55TczsdJlv4P7yA5rmNliMHB6ac1KC9c6ZFi8cuLHrk7KLZjXOgpCDK0a+bGkv
/ZZNKbCevC2pmQ34A8XxyLFuQtnMdkVPVOq0WS0hv9dsmu79v0xLcKPAQOBSY97w
FZkSu9ZqQ/EYxR2v2gxzAeQCYhAoAj+UnskXq2dtjDGKlOawfKwuCB5hvKDMQTpL
duKQcNM9qDL7u88M3WZ3MSvQAR+n7vrqq7kMb0etuzspJLbYY+rtTbuVFEtmBX3t
/jClJFfkemTyAnCp56D4w6c4m6OFcvl9d4USxlimRsOk03nF7tQ2oXMY+6tmdHWX
Eip1d/E+Gk043mZtXkLuP0PIVDZlkJjH+iuibq+Nuj0F4edORTH2i7a1MCTUzFpO
8+rHJs9ytexZ5kC8yz8DYrZlicgU9Of1a0RcRva9wHWW+R5EpwRhr3X0HG/EmqVi
VZbtbsvL1SLnUZVEMuO3wgiJ+yzqg6nzjTuRSjiAU6MlZ10DK39tKiKNzJUOojdl
9rtdioVxWsgU8njqcMdeTCHryDY+1mg66OZnLiGYENitP/chp5o=
=Qlmk
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 25 Nov 10:13 +0100
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87o723fzgd.fsf@inria.fr
Hello,

Efraim Flashner <efraim@flashner.co.il> skribis:

Toggle quote (2 lines)
> On Tue, Nov 12, 2024 at 11:31:57AM +0100, Ludovic Courtès wrote:

[...]

Toggle quote (10 lines)
>> Should that be a package transformation though? Could we instead have a
>> build system trick or the ‘multituned-package’ procedure exposed so
>> build the package several times and fill in lib/glibc-hwcaps?
>
> I figured (guix transformations) worked well since it was using an
> existing transformation to generate the different variants.
>
> I'm not sure what you mean by a build system trick, like build the
> package multiple times in one like with x265?

Yes, that sort of thing, or basically what you have here but provided as
a build phase.

Now, it’s probably easier to implement it the way you did than as a
build phase, because how exactly to rebuild a package with the right
optimizations is very dependent on details about its build system.

Toggle quote (4 lines)
> This exposes the multituned-package procedure (looks like I mistyped in
> the commit message) which takes a package and can be just added on top
> of an existing package definition, like hidden-package can.

Yes, but I think the key ideal with hwcaps is that, contrary to what
‘--tune’ does, we’d provide several optimized versions upfront. In that
sense, we’d statically choose to “multi-tune” certain packages.

Toggle quote (6 lines)
>> That way, packagers would explicitly choose this technique for select
>> packages, which would then no longer need the ‘tunable?’ property.
>
> Something like a flag in the build-system like #:tests? is? Or like a
> package property?

Either that, or write (define p' (multituned-package p)) etc.

The key point being: packagers would be the one deciding that.

So hmm, maybe the way forward is to keep the ‘multituned-package’, but
just not expose it as a package transformation. Also, we might want to
move tuning support to (guix cpu-tuning) or similar, which is the module
that packagers would import (rather than (guix transformations)).

WDYT?

Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74253@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74253
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch