[PATCH 0/6] ] Python bindings, reduced size for OpenPMIx/OpenMPI.

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
normal
M
M
Maxim Cournoyer wrote 7 days ago
(address . guix-patches@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
cover.1732774966.git.maxim.cournoyer@gmail.com
Maxim Cournoyer (6):
gnu: openmpi-5: Update to 5.0.6.
gnu: openpmix: Enable Python bindings.
gnu: openpmix: Fix indentation.
gnu: openpmix: Do not install HTML documentation.
gnu: openmpi-5: Reformat.
gnu: openmpi-5: Streamline source and do not install HTML doc.

gnu/packages/mpi.scm | 134 +++++++++++++++++++++++---------------
gnu/packages/parallel.scm | 105 ++++++++++++++++-------------
2 files changed, 139 insertions(+), 100 deletions(-)


base-commit: 5ff9e51e2c4dae0dc74f5c85ec553e89fe1824fd
--
2.46.0
M
M
Maxim Cournoyer wrote 7 days ago
[PATCH 1/6] gnu: openmpi-5: Update to 5.0.6.
(address . 74572@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
a3b7d86060a68178def41d9cf6be74ea15471673.1732774966.git.maxim.cournoyer@gmail.com
* gnu/packages/mpi.scm (openmpi-5): Update to 5.0.6.

Change-Id: I132b1aaeee4d7a760301582013575a5ea99f03a1
---
gnu/packages/mpi.scm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

Toggle diff (24 lines)
diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 20497242e5..41e57c3012 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -318,7 +318,7 @@ (define-public openmpi openmpi-4)
(define-public openmpi-5
(package
(inherit openmpi)
- (version "5.0.3")
+ (version "5.0.6")
(source
(origin
(method url-fetch)
@@ -326,7 +326,7 @@ (define-public openmpi-5
(version-major+minor version)
"/downloads/openmpi-" version ".tar.bz2"))
(sha256
- (base32 "02x9xmpggw77mdpikjjx83j6i4v3gkqbncda73lk5axk0vr841cr"))))
+ (base32 "0mw1z4ppnlvxngwd58kl5q26qmvf3bgjkd4r8wjpqis3pky86hdx"))))
(inputs (modify-inputs (package-inputs openmpi)
;; As of Open MPI 5.0.X, PMIx is used to communicate
--
2.46.0
M
M
Maxim Cournoyer wrote 7 days ago
[PATCH 4/6] gnu: openpmix: Do not install HTML documentation.
(address . 74572@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
f85c943020bc54f470dfd5c7511ede29b549c660.1732774966.git.maxim.cournoyer@gmail.com
This reduces the package size from 9 MiB to 5 MiB.

* gnu/packages/parallel.scm (openpmix) [source]: Add snippet.

Change-Id: I36e4a1e456225588df96f5eb70ed607505dae493
---
gnu/packages/parallel.scm | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

Toggle diff (19 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index 3e50a75047..a6119743ce 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -705,7 +705,11 @@ (define-public openpmix
version "/pmix-" version ".tar.bz2"))
(sha256
(base32
- "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
+ "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))
+ (snippet
+ '(begin (use-modules (guix build utils))
+ ;; Remove ~5 MiB of pre-built HTML doc.
+ (delete-file-recursively "docs/_build/html")))))
(build-system gnu-build-system)
(arguments
(list
--
2.46.0
M
M
Maxim Cournoyer wrote 7 days ago
[PATCH 2/6] gnu: openpmix: Enable Python bindings.
(address . 74572@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
9b4fe0b847bd842e41fd6dc4e3c3636bee2d1141.1732774966.git.maxim.cournoyer@gmail.com
* gnu/packages/parallel.scm (openpmix)
[arguments] <configure-flags>: Add '--enable-python-bindings'.
<phases>: Add set-LDFLAGS phase.
[native-inputs]: Add python-cython.

Change-Id: I6910ed9c3c7b47b3a76085b95b1f251286a48077
---
gnu/packages/parallel.scm | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)

Toggle diff (46 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index e508dc6493..6a16f0e4c0 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -16,6 +16,7 @@
;;; Copyright © 2024 David Elsing <david.elsing@posteo.net>
;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
;;; Copyright © 2024 Arun Isaac <arunisaac@systemreboot.net>
+;;; Copyright © 2024 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -709,14 +710,20 @@ (define-public openpmix
(arguments
(list #:configure-flags
#~(list (string-append "--with-hwloc="
- (ungexp (this-package-input "hwloc") "lib")))
+ (ungexp (this-package-input "hwloc") "lib"))
+ "--enable-python-bindings") ;disabled by default
;; Don't keep a reference to GCC.
#:disallowed-references (and (not (%current-target-system))
(list (canonical-package gcc)))
-
#:phases
#~(modify-phases %standard-phases
+ (add-after 'unpack 'set-LDFLAGS
+ (lambda _
+ ;; The Cython-compiled shared library would fail the
+ ;; validate-runpath phase otherwise.
+ (setenv "LDFLAGS"
+ (string-append "-Wl,-rpath=" #$output "/lib"))))
(add-before 'configure 'strip-pmix-cc-absolute
(lambda _
;; The 'pmix_info' program prints the 'configure' command
@@ -733,7 +740,7 @@ (define-public openpmix
(("#define PMIX_CONFIGURE_CLI .*")
"#define PMIX_CONFIGURE_CLI \"[scrubbed]\"\n")))))))
(inputs (list libevent `(,hwloc "lib") zlib))
- (native-inputs (list perl python))
+ (native-inputs (list perl python python-cython))
(synopsis "PMIx library")
(description
"PMIx is an application programming interface standard that provides
--
2.46.0
M
M
Maxim Cournoyer wrote 7 days ago
[PATCH 3/6] gnu: openpmix: Fix indentation.
(address . 74572@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
3fb3e35ff7c3ed0187e8b0d85fc90e1b8ee83179.1732774966.git.maxim.cournoyer@gmail.com
* gnu/packages/parallel.scm (openpmix): Fix indentation.

Change-Id: I05751408847343ac593e666afc5f1efbbef94a79
---
gnu/packages/parallel.scm | 106 +++++++++++++++++++-------------------
1 file changed, 53 insertions(+), 53 deletions(-)

Toggle diff (121 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index 6a16f0e4c0..3e50a75047 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -696,61 +696,61 @@ (define-public psimd
(define-public openpmix
(package
- (name "openpmix")
- (version "4.2.8")
- (source (origin
- (method url-fetch)
- (uri (string-append
- "https://github.com/openpmix/openpmix/releases/download/v"
- version "/pmix-" version ".tar.bz2"))
- (sha256
- (base32
- "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
- (build-system gnu-build-system)
- (arguments
- (list #:configure-flags
- #~(list (string-append "--with-hwloc="
- (ungexp (this-package-input "hwloc") "lib"))
- "--enable-python-bindings") ;disabled by default
-
- ;; Don't keep a reference to GCC.
- #:disallowed-references (and (not (%current-target-system))
- (list (canonical-package gcc)))
- #:phases
- #~(modify-phases %standard-phases
- (add-after 'unpack 'set-LDFLAGS
- (lambda _
- ;; The Cython-compiled shared library would fail the
- ;; validate-runpath phase otherwise.
- (setenv "LDFLAGS"
- (string-append "-Wl,-rpath=" #$output "/lib"))))
- (add-before 'configure 'strip-pmix-cc-absolute
- (lambda _
- ;; The 'pmix_info' program prints the 'configure' command
- ;; line, compiler absolute file name, etc., which causes it
- ;; to keep references to many build-time packages. Scrub
- ;; these.
- (substitute* "configure"
- (("PMIX_CC_ABSOLUTE=\"(.*)\"" _ cc)
- (string-append "PMIX_CC_ABSOLUTE=\"$(basename \""
- cc "\")\"\n")))))
- (add-after 'configure 'strip-pmix-config-header
- (lambda _
- (substitute* "src/include/pmix_config.h"
- (("#define PMIX_CONFIGURE_CLI .*")
- "#define PMIX_CONFIGURE_CLI \"[scrubbed]\"\n")))))))
- (inputs (list libevent `(,hwloc "lib") zlib))
- (native-inputs (list perl python python-cython))
- (synopsis "PMIx library")
- (description
- "PMIx is an application programming interface standard that provides
+ (name "openpmix")
+ (version "4.2.8")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/openpmix/openpmix/releases/download/v"
+ version "/pmix-" version ".tar.bz2"))
+ (sha256
+ (base32
+ "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list
+ #:configure-flags
+ #~(list (string-append "--with-hwloc="
+ (ungexp (this-package-input "hwloc") "lib"))
+ "--enable-python-bindings") ;disabled by default
+
+ ;; Don't keep a reference to GCC.
+ #:disallowed-references (and (not (%current-target-system))
+ (list (canonical-package gcc)))
+ #:phases
+ #~(modify-phases %standard-phases
+ (add-after 'unpack 'set-LDFLAGS
+ (lambda _
+ ;; The Cython-compiled shared library would fail the
+ ;; validate-runpath phase otherwise.
+ (setenv "LDFLAGS"
+ (string-append "-Wl,-rpath=" #$output "/lib"))))
+ (add-before 'configure 'strip-pmix-cc-absolute
+ (lambda _
+ ;; The 'pmix_info' program prints the 'configure' command line,
+ ;; compiler absolute file name, etc., which causes it to keep
+ ;; references to many build-time packages. Scrub these.
+ (substitute* "configure"
+ (("PMIX_CC_ABSOLUTE=\"(.*)\"" _ cc)
+ (string-append "PMIX_CC_ABSOLUTE=\"$(basename \""
+ cc "\")\"\n")))))
+ (add-after 'configure 'strip-pmix-config-header
+ (lambda _
+ (substitute* "src/include/pmix_config.h"
+ (("#define PMIX_CONFIGURE_CLI .*")
+ "#define PMIX_CONFIGURE_CLI \"[scrubbed]\"\n")))))))
+ (inputs (list libevent `(,hwloc "lib") zlib))
+ (native-inputs (list perl python python-cython))
+ (synopsis "PMIx library")
+ (description
+ "PMIx is an application programming interface standard that provides
libraries and programming models with portable and well-defined access to
commonly needed services in distributed and parallel computing systems.")
- (home-page "https://pmix.org/")
- ;; configure: WARNING: PMIx does not support 32 bit builds.
- (supported-systems %64bit-supported-systems)
- ;; The provided license is kind of BSD-style but specific.
- (license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
+ (home-page "https://pmix.org/")
+ ;; configure: WARNING: PMIx does not support 32 bit builds.
+ (supported-systems %64bit-supported-systems)
+ ;; The provided license is kind of BSD-style but specific.
+ (license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
(define-public prrte
(package
--
2.46.0
M
M
Maxim Cournoyer wrote 7 days ago
[PATCH 5/6] gnu: openmpi-5: Reformat.
(address . 74572@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
83f356d678647fca5d92abc24c8c8ac8cd5282c9.1732774966.git.maxim.cournoyer@gmail.com
Better comply to our 80 characters maximum width guideline.

* gnu/packages/mpi.scm (openmpi-5): Reformat.

Change-Id: Ib164c4d7dae7bece60738787fd950003fb75e057
---
gnu/packages/mpi.scm | 103 ++++++++++++++++++++++---------------------
1 file changed, 53 insertions(+), 50 deletions(-)

Toggle diff (119 lines)
diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 41e57c3012..67e12657db 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -328,59 +328,62 @@ (define-public openmpi-5
(sha256
(base32 "0mw1z4ppnlvxngwd58kl5q26qmvf3bgjkd4r8wjpqis3pky86hdx"))))
- (inputs (modify-inputs (package-inputs openmpi)
- ;; As of Open MPI 5.0.X, PMIx is used to communicate
- ;; with SLURM, so SLURM'S PMI is no longer needed.
- (delete "slurm")
- (append ucx) ;for Infiniband support
- (append openpmix) ;for PMI support (launching via "srun")
- (append prrte))) ;for PMI support (launching via "srun")
- (native-inputs (modify-inputs (package-native-inputs openmpi)
- (append python)))
+ (inputs
+ (modify-inputs (package-inputs openmpi)
+ ;; As of Open MPI 5.0.X, PMIx is used to communicate with SLURM, so
+ ;; SLURM'S PMI is no longer needed.
+ (delete "slurm")
+ (append ucx) ;for Infiniband support
+ (append openpmix) ;for PMI support (launching via "srun")
+ (append prrte))) ;for PMI support (launching via "srun")
+ (native-inputs
+ (modify-inputs (package-native-inputs openmpi)
+ (append python)))
(outputs '("out" "debug"))
(arguments
- (list #:configure-flags
- #~(list "--enable-mpi-ext=affinity" ;cr doesn't work
- "--with-sge"
- "--disable-static"
-
- #$@(if (package? (this-package-input "valgrind"))
- #~("--enable-memchecker"
- "--with-valgrind")
- #~("--without-valgrind"))
-
- "--with-hwloc=external"
- "--with-libevent"
-
- ;; This replaces --enable-mpirun-prefix-by-default wich is deprecated
- ;; since 5.x.
- "--enable-prte-prefix-by-default"
-
- ;; Enable support for the 'Process Management Interface for Exascale'
- ;; (PMIx) used e.g. by Slurm for the management communication and
- ;; coordination of MPI processes.
- (string-append "--with-pmix=" #$(this-package-input "openpmix"))
- (string-append "--with-prrte=" #$(this-package-input "prrte"))
-
- ;; Since 5.x, Infiniband support is provided by ucx.
- ;; See https://docs.open-mpi.org/en/main/release-notes/networks.html#miscellaneous-network-notes
- (string-append "--with-ucx=" #$(this-package-input "ucx")))
-
- #:phases
- #~(modify-phases %standard-phases
- (add-before 'build 'remove-absolute
- (lambda _
- ;; Remove compiler absolute file names (OPAL_FC_ABSOLUTE
- ;; etc.) to reduce the closure size. See
- ;; <https://lists.gnu.org/archive/html/guix-devel/2017-07/msg00388.html>
- ;; and
- ;; <https://www.mail-archive.com/users@lists.open-mpi.org//msg31397.html>.
- (substitute* '("oshmem/tools/oshmem_info/param.c"
- "ompi/tools/ompi_info/param.c")
- (("_ABSOLUTE") "")))))
-
- #:disallowed-references (list (canonical-package gcc))))))
+ (list
+ #:configure-flags
+ #~(list "--enable-mpi-ext=affinity" ;cr doesn't work
+ "--with-sge"
+ "--disable-static"
+
+ #$@(if (package? (this-package-input "valgrind"))
+ #~("--enable-memchecker"
+ "--with-valgrind")
+ #~("--without-valgrind"))
+
+ "--with-hwloc=external"
+ "--with-libevent"
+
+ ;; This replaces --enable-mpirun-prefix-by-default wich is
+ ;; deprecated since 5.x.
+ "--enable-prte-prefix-by-default"
+
+ ;; Enable support for the 'Process Management Interface for
+ ;; Exascale' (PMIx) used e.g. by Slurm for the management
+ ;; communication and coordination of MPI processes.
+ (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+ (string-append "--with-prrte=" #$(this-package-input "prrte"))
+
+ ;; Since 5.x, Infiniband support is provided by ucx. See
+ ;; https://docs.open-mpi.org/en/main/release-notes/networks.html#miscellaneous-network-notes
+ (string-append "--with-ucx=" #$(this-package-input "ucx")))
+
+ #:phases
+ #~(modify-phases %standard-phases
+ (add-before 'build 'remove-absolute
+ (lambda _
+ ;; Remove compiler absolute file names (OPAL_FC_ABSOLUTE etc.)
+ ;; to reduce the closure size. See
+ ;; <https://lists.gnu.org/archive/html/guix-devel/2017-07/msg00388.html>
+ ;; and
+ ;; <https://www.mail-archive.com/users@lists.open-mpi.org//msg31397.html>.
+ (substitute* '("oshmem/tools/oshmem_info/param.c"
+ "ompi/tools/ompi_info/param.c")
+ (("_ABSOLUTE") "")))))
+
+ #:disallowed-references (list (canonical-package gcc))))))
(define-public openmpi-c++
(package/inherit openmpi
--
2.46.0
M
M
Maxim Cournoyer wrote 7 days ago
[PATCH 6/6] gnu: openmpi-5: Streamline source and do not install HTML doc.
(address . 74572@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
040e8d813af84e53e68c61f747059c229b6b70f5.1732774966.git.maxim.cournoyer@gmail.com
This reduces the installed package size by 63 MiB.

* gnu/packages/mpi.scm (openmpi-5) [source]: Add modules, snippet fields.

Change-Id: Ie3dc365a2e4e48dc4c385dd447d3c64b7db065ca
---
gnu/packages/mpi.scm | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)

Toggle diff (38 lines)
diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 67e12657db..2f37364ef3 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -325,6 +325,31 @@ (define-public openmpi-5
(uri (string-append "https://www.open-mpi.org/software/ompi/v"
(version-major+minor version)
"/downloads/openmpi-" version ".tar.bz2"))
+ (modules '((guix build utils)
+ (ice-9 ftw)
+ (srfi srfi-26)))
+ (snippet
+ '(begin
+ ;; XXX: 'delete-all-but' is copied from the turbovnc package.
+ (define (delete-all-but directory . preserve)
+ (define (directory? x)
+ (and=> (stat x #f)
+ (compose (cut eq? 'directory <>) stat:type)))
+ (with-directory-excursion directory
+ (let* ((pred
+ (negate (cut member <> (append '("." "..") preserve))))
+ (items (scandir "." pred)))
+ (for-each (lambda (item)
+ (if (directory? item)
+ (delete-file-recursively item)
+ (delete-file item)))
+ items))))
+ ;; Delete as many bundled libraries as permitted by the build
+ ;; system.
+ (delete-all-but "3rd-party" "treematch" "Makefile.in" "Makefile.am")
+ ;; Do not install 64 MiB worth fo pre-generated HTML
+ ;; documentation.
+ (delete-file-recursively "docs/html")))
(sha256
(base32 "0mw1z4ppnlvxngwd58kl5q26qmvf3bgjkd4r8wjpqis3pky86hdx"))))
--
2.46.0
L
L
Ludovic Courtès wrote 3 days ago
Re: [bug#74572] [PATCH 0/6] ] Python bindings, reduced size for OpenPMIx/OpenMPI.
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87iks3qqnt.fsf@gnu.org
Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (7 lines)
> gnu: openmpi-5: Update to 5.0.6.
> gnu: openpmix: Enable Python bindings.
> gnu: openpmix: Fix indentation.
> gnu: openpmix: Do not install HTML documentation.
> gnu: openmpi-5: Reformat.
> gnu: openmpi-5: Streamline source and do not install HTML doc.

I’m afraid you’ll have to rebase: I just pushed
https://issues.guix.gnu.org/74546, which also updated openmpi@5. :-)

BTW, I was telling Efraim we should probably create an HPC team to
coordinate on critical components like MPI implementations and their
drivers (at work we usually look for performance regressions against
high-speed interconnects before submitting changes to openmpi & co.).
How does that sound?

Ludo’.
L
L
Ludovic Courtès wrote 3 days ago
Re: [bug#74572] [PATCH 2/6] gnu: openpmix: Enable Python bindings.
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 74572@debbugs.gnu.org)
87ed2rqqlk.fsf@gnu.org
Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (7 lines)
> * gnu/packages/parallel.scm (openpmix)
> [arguments] <configure-flags>: Add '--enable-python-bindings'.
> <phases>: Add set-LDFLAGS phase.
> [native-inputs]: Add python-cython.
>
> Change-Id: I6910ed9c3c7b47b3a76085b95b1f251286a48077

LGTM, provided this does not incur a significant hit on the closure size
of openmpi (we routinely use ‘guix pack’ to send package sets that
include openmpi to supercomputers).

Ludo’.
L
L
Ludovic Courtès wrote 3 days ago
Re: [bug#74572] [PATCH 6/6] gnu: openmpi-5: Streamline source and do not install HTML doc.
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 74572@debbugs.gnu.org)
87a5dfqqjq.fsf@gnu.org
Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (6 lines)
> This reduces the installed package size by 63 MiB.
>
> * gnu/packages/mpi.scm (openmpi-5) [source]: Add modules, snippet fields.
>
> Change-Id: Ie3dc365a2e4e48dc4c385dd447d3c64b7db065ca

This and the rest LGTM. Thanks!
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74572@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74572
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch