[PATCH 0/4] Add OpenMPI 5.x

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Romain GARBAGE
Owner
unassigned
Submitted by
Romain GARBAGE
Severity
normal
R
R
Romain GARBAGE wrote on 11 Jul 16:25 +0200
(address . guix-patches@gnu.org)
20240711143230.12467-1-romain.garbage@inria.fr
This patch series aims at bringing the 5.x branch of OpenMPI to Guix.

First two patches are dependencies of OpenMPI 5.x.

Third patch adds a new package for OpenMPI 5.x, keeping OpenMPI 4.x as
the default version in package dependencies.

Fourth patch adds PMIx support, which is required for OpenMPI 5.x
support in SLURM.

These patches have been applied on top of commit
0663668507377a849317b0fff3d72f609c3139c2, all packages depending on
slurm and openmpi@5 have been successfully rebuilt.

Romain GARBAGE (4):
gnu: Add openpmix.
gnu: Add prrte.
gnu: Add openmpi-5.
gnu: slurm: Add PMIx support.

gnu/packages/mpi.scm | 52 ++++++++++++++++++++++++++++++
gnu/packages/parallel.scm | 68 +++++++++++++++++++++++++++++++++++++++
2 files changed, 120 insertions(+)

--
2.45.1
R
R
Romain GARBAGE wrote on 11 Jul 16:35 +0200
[PATCH 1/4] gnu: Add openpmix.
(address . 72062@debbugs.gnu.org)
20240711143540.25601-1-romain.garbage@inria.fr
* gnu/packages/parallel.scm (openpmix): New variable.

Change-Id: Id23bc710c9b61ede0aebf7e2a18863fbe5fb8304
---
gnu/packages/parallel.scm | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)

Toggle diff (56 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index a16a0d8689..4a57e85e6c 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -14,6 +14,7 @@
;;; Copyright © 2021 Stefan Reichör <stefan@xsteve.at>
;;; Copyright © 2024 Zheng Junjie <873216071@qq.com>
;;; Copyright © 2024 David Elsing <david.elsing@posteo.net>
+;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -51,6 +52,7 @@ (define-module (gnu packages parallel)
#:use-module (gnu packages documentation)
#:use-module (gnu packages flex)
#:use-module (gnu packages freeipmi)
+ #:use-module (gnu packages libevent)
#:use-module (gnu packages linux)
#:use-module (gnu packages maths)
#:use-module (gnu packages mpi)
@@ -612,3 +614,33 @@ (define-public psimd
"This header-only C++ library provides a portable interface to
single-instruction multiple-data (SIMD) intrinsics.")
(license license:expat))))
+
+(define-public openpmix
+ (package
+ (name "openpmix")
+ (version "4.2.8")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/openpmix/openpmix/releases/download/v"
+ version "/pmix-" version ".tar.bz2"))
+ (sha256
+ (base32
+ "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list #:configure-flags #~(list (string-append "--with-hwloc="
+ (ungexp (this-package-input "hwloc") "lib")))))
+ (inputs (list libevent
+ `(,hwloc "lib")))
+ (native-inputs (list perl
+ python))
+ (synopsis "PMIx library")
+ (description
+ "PMIx is an application programming interface standard that provides
+libraries and programming models with portable and well-defined access
+to commonly needed services in distributed and parallel computing
+systems.")
+ (home-page "https://pmix.org/")
+ ;; The provided license is kind of BSD-style but specific.
+ (license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
--
2.45.1
R
R
Romain GARBAGE wrote on 11 Jul 16:35 +0200
[PATCH 2/4] gnu: Add prrte.
(address . 72062@debbugs.gnu.org)
20240711143540.25601-2-romain.garbage@inria.fr
* gnu/packages/parallel.scm (prrte): New variable.

Change-Id: I4427eec1a5c7d94ed5f9cbceb55f9372834a8971
---
gnu/packages/parallel.scm | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)

Toggle diff (42 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index 4a57e85e6c..f173d1b75e 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -644,3 +644,35 @@ (define-public openpmix
(home-page "https://pmix.org/")
;; The provided license is kind of BSD-style but specific.
(license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
+
+(define-public prrte
+ (package
+ (name "prrte")
+ (version "3.0.6")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/openpmix/prrte/releases/download/v"
+ version "/prrte-" version ".tar.bz2"))
+ (sha256
+ (base32
+ "0wiy0vk37v4db1jgxza8bci0cczcvj34dalzsrlz05dk45zb7dl3"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list #:configure-flags #~(list (string-append "--with-hwloc="
+ (assoc-ref %build-inputs "hwloc"))
+ (string-append "--with-pmix=" #$(this-package-input "openpmix")))))
+ (inputs (list libevent
+ `(,hwloc "lib")
+ openpmix))
+ (native-inputs (list perl))
+ (synopsis "PMIx Reference RunTime Environment (PRRTE)")
+ (description
+ "The PMIx Reference RunTime Environment is a runtime environment
+containing the reference implementation and capable of operating
+within a host SMS. The reference RTE therefore provides an easy way of
+exploring PMIx capabilities and testing PMIx-based applications
+outside of a PMIx-enabled environment.")
+ (home-page "https://openpmix.github.io/")
+ ;; The provided license is kind of BSD-style but specific.
+ (license (license:fsf-free "https://github.com/openpmix/prrte?tab=License-1-ov-file#License-1-ov-file"))))
--
2.45.1
R
R
Romain GARBAGE wrote on 11 Jul 16:35 +0200
[PATCH 3/4] gnu: Add openmpi-5.
(address . 72062@debbugs.gnu.org)
20240711143540.25601-3-romain.garbage@inria.fr
* gnu/packages/mpi.scm (openmpi-5): New variable.

Change-Id: I048692b2e928077c9cfa9fb8076ceb356251e1db
---
gnu/packages/mpi.scm | 52 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 52 insertions(+)

Toggle diff (79 lines)
diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 6270108b16..8cde8ef044 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2018–2022 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2018 Paul Garlick <pgarlick@tourbillion-technology.com>
;;; Copyright © 2019, 2021 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -49,6 +50,7 @@ (define-module (gnu packages mpi)
#:use-module (gnu packages ncurses)
#:use-module (gnu packages parallel)
#:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages python)
#:use-module (gnu packages valgrind)
#:use-module (srfi srfi-1)
#:use-module (ice-9 match))
@@ -307,6 +309,56 @@ (define-public openmpi
;; See file://LICENSE
(license license:bsd-2)))
+(define-public openmpi-5
+ (package/inherit openmpi
+ (version "5.0.3")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (string-append "https://www.open-mpi.org/software/ompi/v"
+ (version-major+minor version)
+ "/downloads/openmpi-" version ".tar.bz2"))
+ (sha256
+ (base32 "02x9xmpggw77mdpikjjx83j6i4v3gkqbncda73lk5axk0vr841cr"))))
+
+ (inputs (modify-inputs (package-inputs openmpi)
+ ;; As of Open MPI 5.0.X, PMIx is used to communicate
+ ;; with SLURM, so SLURM'S PMI is no longer needed.
+ (delete "slurm")
+ (append ucx) ;for Infiniband support
+ (append openpmix) ;for PMI support (launching via "srun")
+ (append prrte))) ;for PMI support (launching via "srun")
+ (native-inputs (modify-inputs (package-native-inputs openmpi)
+ (append python)))
+
+ (outputs '("out" "debug"))
+ (arguments
+ (substitute-keyword-arguments (package-arguments openmpi)
+ ((#:configure-flags flags)
+ #~(append (list "--enable-prte-prefix-by-default" ;replaces --enable-mpirun-prefix-by-default
+ ;; Enable support for the 'Process Management
+ ;; Interface for Exascale' (PMIx) used e.g. by
+ ;; Slurm for the management communication and
+ ;; coordination of MPI processes.
+ (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+ (string-append "--with-prrte=" #$(this-package-input "prrte"))
+
+ ;; Since 5.x, Infiniband support is provided by ucx.
+ (string-append "--with-ucx=" #$(this-package-input "ucx")))
+ (filter (lambda (e)
+ (and (not (string-contains e "pmi")) ;; Since 5.x, OpenMPI doesn't support PMI/PMI-2
+ ;; Deprecated since 5.x.
+ (not (string-contains e "enable-mpirun-prefix-by-default"))
+ ;; Infiniband support is deprecated and superseded by ucx
+ ;; See https://docs.open-mpi.org/en/main/release-notes/networks.html#miscellaneous-network-notes
+ (not (string-contains e "openib"))))
+ #$flags)))
+
+ ((#:phases phases)
+ #~(modify-phases #$phases
+ (delete 'remove-absolute)
+ (delete 'scrub-timestamps)))))))
+
(define-public openmpi-c++
(package/inherit openmpi
(name "openmpi-c++")
--
2.45.1
R
R
Romain GARBAGE wrote on 11 Jul 16:35 +0200
[PATCH 4/4] gnu: slurm: Add PMIx support.
(address . 72062@debbugs.gnu.org)
20240711143540.25601-4-romain.garbage@inria.fr
* gnu/packages/parallel.scm (slurm) [inputs]: Add dependency.
[arguments]: Update configure flags.

Change-Id: I6544be5eb44980191d0788738ab960263c56c0ba
---
gnu/packages/parallel.scm | 4 ++++
1 file changed, 4 insertions(+)

Toggle diff (24 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index f173d1b75e..694968b727 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -229,6 +229,7 @@ (define-public slurm
`(,hwloc-2 "lib")
json-c
linux-pam
+ openpmix
munge
numactl
readline))
@@ -245,6 +246,9 @@ (define-public slurm
(string-append "--with-json=" #$(this-package-input "json-c"))
(string-append "--with-munge=" #$(this-package-input "munge"))
+ ;; Use PMIx bundled with Open MPI (this is required for Open MPI 5.x).
+ (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+
;; 32-bit support is marked as deprecated and needs to be
;; explicitly enabled.
#$@(if (target-64bit?) '() '("--enable-deprecated")))
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 09:27 +0200
Re: [PATCH 0/4] Add OpenMPI 5.x
(address . 72062@debbugs.gnu.org)(address . ludovic.courtes@inria.fr)
D2NDNQMMEWTG.OXU5W16TPCB1@guix-A102
On Thu Jul 11, 2024 at 4:25 PM CEST, Romain GARBAGE wrote:
Toggle quote (1 lines)
> This patch series aims at bringing the 5.x branch of OpenMPI to Guix.
[...]
Toggle quote (6 lines)
> Third patch adds a new package for OpenMPI 5.x, keeping OpenMPI 4.x as
> the default version in package dependencies.
>
> Fourth patch adds PMIx support, which is required for OpenMPI 5.x
> support in SLURM.

After thinking about it, the last two patches should be applied in the
reverse order so the when OpenMPI5 is present in the tree, SLURM already
has the PMIx support needed by OpenMPI5.
If the need arises to submit a v2, I'll swap them at this occasion.

Toggle quote (4 lines)
> These patches have been applied on top of commit
> 0663668507377a849317b0fff3d72f609c3139c2, all packages depending on
> slurm and openmpi@5 have been successfully rebuilt.

To be more precise, I rebuilt the packages that depend directly on slurm
and openmpi@5 (for the latter, there should be none for now).

Romain
L
L
Ludovic Courtès wrote on 12 Jul 10:40 +0200
Re: [PATCH 3/4] gnu: Add openmpi-5.
(name . Romain GARBAGE)(address . romain.garbage@inria.fr)(address . 72062@debbugs.gnu.org)
87r0bzypd6.fsf@inria.fr
Hello!

Romain GARBAGE <romain.garbage@inria.fr> skribis:

Toggle quote (4 lines)
> * gnu/packages/mpi.scm (openmpi-5): New variable.
>
> Change-Id: I048692b2e928077c9cfa9fb8076ceb356251e1db

[...]

Toggle quote (3 lines)
> +(define-public openmpi-5
> + (package/inherit openmpi

I think it’s a case where (package (inherit openmpi) …) is more
appropriate: ‘package/inherit’ would cause the ‘replacement’ of
‘openmpi’ to be inherited, which is probably not a good idea since it’s
a different version.

Toggle quote (21 lines)
> + (substitute-keyword-arguments (package-arguments openmpi)
> + ((#:configure-flags flags)
> + #~(append (list "--enable-prte-prefix-by-default" ;replaces --enable-mpirun-prefix-by-default
> + ;; Enable support for the 'Process Management
> + ;; Interface for Exascale' (PMIx) used e.g. by
> + ;; Slurm for the management communication and
> + ;; coordination of MPI processes.
> + (string-append "--with-pmix=" #$(this-package-input "openpmix"))
> + (string-append "--with-prrte=" #$(this-package-input "prrte"))
> +
> + ;; Since 5.x, Infiniband support is provided by ucx.
> + (string-append "--with-ucx=" #$(this-package-input "ucx")))
> + (filter (lambda (e)
> + (and (not (string-contains e "pmi")) ;; Since 5.x, OpenMPI doesn't support PMI/PMI-2
> + ;; Deprecated since 5.x.
> + (not (string-contains e "enable-mpirun-prefix-by-default"))
> + ;; Infiniband support is deprecated and superseded by ucx
> + ;; See https://docs.open-mpi.org/en/main/release-notes/networks.html#miscellaneous-network-notes
> + (not (string-contains e "openib"))))
> + #$flags)))

How about listing all the configure flags explicitly rather than
relative to those of 4.x? I feel like this would be clearer and that
there’s little to be gained by factorizing anyway.

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 12 Jul 10:42 +0200
(name . Romain GARBAGE)(address . romain.garbage@inria.fr)(address . 72062@debbugs.gnu.org)
87msmnypa0.fsf@inria.fr
Romain GARBAGE <romain.garbage@inria.fr> skribis:

Toggle quote (2 lines)
> +(define-public openmpi-5

Also, probably as a separate patch, you could rename ‘openmpi’ to
‘openmpi-4’ and add:

(define-public openmpi openmpi-4)

That way, we’ll only have to touch that variable when we decide to
upgrade wholesale.

WDYT?

Ludo’.
L
L
Ludovic Courtès wrote on 12 Jul 10:44 +0200
Re: [PATCH 0/4] Add OpenMPI 5.x
(name . Romain GARBAGE)(address . romain.garbage@inria.fr)(address . 72062@debbugs.gnu.org)
87ikxbyp7t.fsf@inria.fr
Romain GARBAGE <romain.garbage@inria.fr> skribis:

Toggle quote (5 lines)
> gnu: Add openpmix.
> gnu: Add prrte.
> gnu: Add openmpi-5.
> gnu: slurm: Add PMIx support.

Apart from the minor issues I reported, it LGTM! Let’s see if qa.guix
can give it a try…

Ludo’.
R
R
Romain GARBAGE wrote on 12 Jul 13:47 +0200
[PATCH v2 0/5] Add OpenMPI 5.x.
(address . 72062@debbugs.gnu.org)
20240712115235.20217-1-romain.garbage@inria.fr
Changes from v1->v2:
* Swapped patches #3 and #4 so SLURM with PMIx support is present before
OpenMPI 5.x.
* (openmpi-5)[arguments]: configure flags are not derived from the
inherited package but fully listed.
* (openmpi-5): Switched from package/inherit to (inherit ...).
* Renamed openmpi to openmpi-4.
* Set openmpi-4 as default openmpi.

Romain GARBAGE (5):
gnu: Add openpmix.
gnu: Add prrte.
gnu: slurm: Add PMIx support.
gnu: Add openmpi-5.
gnu: openmpi: Set default version to 4.x.

gnu/packages/mpi.scm | 63 +++++++++++++++++++++++++++++++++++-
gnu/packages/parallel.scm | 68 +++++++++++++++++++++++++++++++++++++++
2 files changed, 130 insertions(+), 1 deletion(-)


base-commit: 0663668507377a849317b0fff3d72f609c3139c2
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 13:47 +0200
[PATCH v2 1/5] gnu: Add openpmix.
(address . 72062@debbugs.gnu.org)
20240712115235.20217-2-romain.garbage@inria.fr
* gnu/packages/parallel.scm (openpmix): New variable.

Change-Id: Id23bc710c9b61ede0aebf7e2a18863fbe5fb8304
---
gnu/packages/parallel.scm | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)

Toggle diff (56 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index a16a0d8689..4a57e85e6c 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -14,6 +14,7 @@
;;; Copyright © 2021 Stefan Reichör <stefan@xsteve.at>
;;; Copyright © 2024 Zheng Junjie <873216071@qq.com>
;;; Copyright © 2024 David Elsing <david.elsing@posteo.net>
+;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -51,6 +52,7 @@ (define-module (gnu packages parallel)
#:use-module (gnu packages documentation)
#:use-module (gnu packages flex)
#:use-module (gnu packages freeipmi)
+ #:use-module (gnu packages libevent)
#:use-module (gnu packages linux)
#:use-module (gnu packages maths)
#:use-module (gnu packages mpi)
@@ -612,3 +614,33 @@ (define-public psimd
"This header-only C++ library provides a portable interface to
single-instruction multiple-data (SIMD) intrinsics.")
(license license:expat))))
+
+(define-public openpmix
+ (package
+ (name "openpmix")
+ (version "4.2.8")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/openpmix/openpmix/releases/download/v"
+ version "/pmix-" version ".tar.bz2"))
+ (sha256
+ (base32
+ "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list #:configure-flags #~(list (string-append "--with-hwloc="
+ (ungexp (this-package-input "hwloc") "lib")))))
+ (inputs (list libevent
+ `(,hwloc "lib")))
+ (native-inputs (list perl
+ python))
+ (synopsis "PMIx library")
+ (description
+ "PMIx is an application programming interface standard that provides
+libraries and programming models with portable and well-defined access
+to commonly needed services in distributed and parallel computing
+systems.")
+ (home-page "https://pmix.org/")
+ ;; The provided license is kind of BSD-style but specific.
+ (license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 13:47 +0200
[PATCH v2 2/5] gnu: Add prrte.
(address . 72062@debbugs.gnu.org)
20240712115235.20217-3-romain.garbage@inria.fr
* gnu/packages/parallel.scm (prrte): New variable.

Change-Id: I4427eec1a5c7d94ed5f9cbceb55f9372834a8971
---
gnu/packages/parallel.scm | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)

Toggle diff (42 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index 4a57e85e6c..f173d1b75e 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -644,3 +644,35 @@ (define-public openpmix
(home-page "https://pmix.org/")
;; The provided license is kind of BSD-style but specific.
(license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
+
+(define-public prrte
+ (package
+ (name "prrte")
+ (version "3.0.6")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/openpmix/prrte/releases/download/v"
+ version "/prrte-" version ".tar.bz2"))
+ (sha256
+ (base32
+ "0wiy0vk37v4db1jgxza8bci0cczcvj34dalzsrlz05dk45zb7dl3"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list #:configure-flags #~(list (string-append "--with-hwloc="
+ (assoc-ref %build-inputs "hwloc"))
+ (string-append "--with-pmix=" #$(this-package-input "openpmix")))))
+ (inputs (list libevent
+ `(,hwloc "lib")
+ openpmix))
+ (native-inputs (list perl))
+ (synopsis "PMIx Reference RunTime Environment (PRRTE)")
+ (description
+ "The PMIx Reference RunTime Environment is a runtime environment
+containing the reference implementation and capable of operating
+within a host SMS. The reference RTE therefore provides an easy way of
+exploring PMIx capabilities and testing PMIx-based applications
+outside of a PMIx-enabled environment.")
+ (home-page "https://openpmix.github.io/")
+ ;; The provided license is kind of BSD-style but specific.
+ (license (license:fsf-free "https://github.com/openpmix/prrte?tab=License-1-ov-file#License-1-ov-file"))))
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 13:47 +0200
[PATCH v2 3/5] gnu: slurm: Add PMIx support.
(address . 72062@debbugs.gnu.org)
20240712115235.20217-4-romain.garbage@inria.fr
* gnu/packages/parallel.scm (slurm) [inputs]: Add dependency.
[arguments]: Update configure flags.

Change-Id: I6544be5eb44980191d0788738ab960263c56c0ba
---
gnu/packages/parallel.scm | 4 ++++
1 file changed, 4 insertions(+)

Toggle diff (24 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index f173d1b75e..694968b727 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -229,6 +229,7 @@ (define-public slurm
`(,hwloc-2 "lib")
json-c
linux-pam
+ openpmix
munge
numactl
readline))
@@ -245,6 +246,9 @@ (define-public slurm
(string-append "--with-json=" #$(this-package-input "json-c"))
(string-append "--with-munge=" #$(this-package-input "munge"))
+ ;; Use PMIx bundled with Open MPI (this is required for Open MPI 5.x).
+ (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+
;; 32-bit support is marked as deprecated and needs to be
;; explicitly enabled.
#$@(if (target-64bit?) '() '("--enable-deprecated")))
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 13:47 +0200
[PATCH v2 5/5] gnu: openmpi: Set default version to 4.x.
(address . 72062@debbugs.gnu.org)
20240712115235.20217-6-romain.garbage@inria.fr
* gnu/packages/mpi.scm (openmpi-4): New variable.
(openmpi): Redefine variable.

Change-Id: I899e6259c9bcc579622c57a1ad1e9665f0e5db7f
---
gnu/packages/mpi.scm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

Toggle diff (17 lines)
diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 01f07a0f7c..08cfbc3062 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -185,7 +185,9 @@ (define-public hwloc
;; The latest stable series of hwloc.
hwloc-2)
-(define-public openmpi
+(define-public openmpi openmpi-4)
+
+(define-public openmpi-4
(package
(name "openmpi")
(version "4.1.6")
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 13:47 +0200
[PATCH v2 4/5] gnu: Add openmpi-5.
(address . 72062@debbugs.gnu.org)
20240712115235.20217-5-romain.garbage@inria.fr
* gnu/packages/mpi.scm (openmpi-5): New variable.

Change-Id: I048692b2e928077c9cfa9fb8076ceb356251e1db
---
gnu/packages/mpi.scm | 59 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)

Toggle diff (86 lines)
diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 6270108b16..01f07a0f7c 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2018–2022 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2018 Paul Garlick <pgarlick@tourbillion-technology.com>
;;; Copyright © 2019, 2021 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -49,6 +50,7 @@ (define-module (gnu packages mpi)
#:use-module (gnu packages ncurses)
#:use-module (gnu packages parallel)
#:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages python)
#:use-module (gnu packages valgrind)
#:use-module (srfi srfi-1)
#:use-module (ice-9 match))
@@ -307,6 +309,63 @@ (define-public openmpi
;; See file://LICENSE
(license license:bsd-2)))
+(define-public openmpi-5
+ (package
+ (inherit openmpi)
+ (version "5.0.3")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (string-append "https://www.open-mpi.org/software/ompi/v"
+ (version-major+minor version)
+ "/downloads/openmpi-" version ".tar.bz2"))
+ (sha256
+ (base32 "02x9xmpggw77mdpikjjx83j6i4v3gkqbncda73lk5axk0vr841cr"))))
+
+ (inputs (modify-inputs (package-inputs openmpi)
+ ;; As of Open MPI 5.0.X, PMIx is used to communicate
+ ;; with SLURM, so SLURM'S PMI is no longer needed.
+ (delete "slurm")
+ (append ucx) ;for Infiniband support
+ (append openpmix) ;for PMI support (launching via "srun")
+ (append prrte))) ;for PMI support (launching via "srun")
+ (native-inputs (modify-inputs (package-native-inputs openmpi)
+ (append python)))
+
+ (outputs '("out" "debug"))
+ (arguments
+ (substitute-keyword-arguments (package-arguments openmpi)
+ ((#:configure-flags _)
+ #~(list "--enable-mpi-ext=affinity" ;cr doesn't work
+ "--with-sge"
+
+ #$@(if (package? (this-package-input "valgrind"))
+ #~("--enable-memchecker"
+ "--with-valgrind")
+ #~("--without-valgrind"))
+
+ "--with-hwloc=external"
+ "--with-libevent"
+
+ ;; This replaces --enable-mpirun-prefix-by-default wich is deprecated
+ ;; since 5.x.
+ "--enable-prte-prefix-by-default"
+
+ ;; Enable support for the 'Process Management Interface for Exascale'
+ ;; (PMIx) used e.g. by Slurm for the management communication and
+ ;; coordination of MPI processes.
+ (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+ (string-append "--with-prrte=" #$(this-package-input "prrte"))
+
+ ;; Since 5.x, Infiniband support is provided by ucx.
+ ;; See https://docs.open-mpi.org/en/main/release-notes/networks.html#miscellaneous-network-notes
+ (string-append "--with-ucx=" #$(this-package-input "ucx"))))
+
+ ((#:phases phases)
+ #~(modify-phases #$phases
+ (delete 'remove-absolute)
+ (delete 'scrub-timestamps)))))))
+
(define-public openmpi-c++
(package/inherit openmpi
(name "openmpi-c++")
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 14:00 +0200
[PATCH v3 0/5] Add OpenMPI 5.x.
(address . 72062@debbugs.gnu.org)
20240712120140.20391-1-romain.garbage@inria.fr
Changes from v2->v3
* Moved the definition of openmpi below openmpi-4 (fixes build breakage).

Romain GARBAGE (5):
gnu: Add openpmix.
gnu: Add prrte.
gnu: slurm: Add PMIx support.
gnu: Add openmpi-5.
gnu: openmpi: Set default version to 4.x.

gnu/packages/mpi.scm | 63 +++++++++++++++++++++++++++++++++++-
gnu/packages/parallel.scm | 68 +++++++++++++++++++++++++++++++++++++++
2 files changed, 130 insertions(+), 1 deletion(-)


base-commit: 0663668507377a849317b0fff3d72f609c3139c2
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 14:00 +0200
[PATCH v3 1/5] gnu: Add openpmix.
(address . 72062@debbugs.gnu.org)
20240712120140.20391-2-romain.garbage@inria.fr
* gnu/packages/parallel.scm (openpmix): New variable.

Change-Id: Id23bc710c9b61ede0aebf7e2a18863fbe5fb8304
---
gnu/packages/parallel.scm | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)

Toggle diff (56 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index a16a0d8689..4a57e85e6c 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -14,6 +14,7 @@
;;; Copyright © 2021 Stefan Reichör <stefan@xsteve.at>
;;; Copyright © 2024 Zheng Junjie <873216071@qq.com>
;;; Copyright © 2024 David Elsing <david.elsing@posteo.net>
+;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -51,6 +52,7 @@ (define-module (gnu packages parallel)
#:use-module (gnu packages documentation)
#:use-module (gnu packages flex)
#:use-module (gnu packages freeipmi)
+ #:use-module (gnu packages libevent)
#:use-module (gnu packages linux)
#:use-module (gnu packages maths)
#:use-module (gnu packages mpi)
@@ -612,3 +614,33 @@ (define-public psimd
"This header-only C++ library provides a portable interface to
single-instruction multiple-data (SIMD) intrinsics.")
(license license:expat))))
+
+(define-public openpmix
+ (package
+ (name "openpmix")
+ (version "4.2.8")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/openpmix/openpmix/releases/download/v"
+ version "/pmix-" version ".tar.bz2"))
+ (sha256
+ (base32
+ "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list #:configure-flags #~(list (string-append "--with-hwloc="
+ (ungexp (this-package-input "hwloc") "lib")))))
+ (inputs (list libevent
+ `(,hwloc "lib")))
+ (native-inputs (list perl
+ python))
+ (synopsis "PMIx library")
+ (description
+ "PMIx is an application programming interface standard that provides
+libraries and programming models with portable and well-defined access
+to commonly needed services in distributed and parallel computing
+systems.")
+ (home-page "https://pmix.org/")
+ ;; The provided license is kind of BSD-style but specific.
+ (license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 14:00 +0200
[PATCH v3 3/5] gnu: slurm: Add PMIx support.
(address . 72062@debbugs.gnu.org)
20240712120140.20391-4-romain.garbage@inria.fr
* gnu/packages/parallel.scm (slurm) [inputs]: Add dependency.
[arguments]: Update configure flags.

Change-Id: I6544be5eb44980191d0788738ab960263c56c0ba
---
gnu/packages/parallel.scm | 4 ++++
1 file changed, 4 insertions(+)

Toggle diff (24 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index f173d1b75e..694968b727 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -229,6 +229,7 @@ (define-public slurm
`(,hwloc-2 "lib")
json-c
linux-pam
+ openpmix
munge
numactl
readline))
@@ -245,6 +246,9 @@ (define-public slurm
(string-append "--with-json=" #$(this-package-input "json-c"))
(string-append "--with-munge=" #$(this-package-input "munge"))
+ ;; Use PMIx bundled with Open MPI (this is required for Open MPI 5.x).
+ (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+
;; 32-bit support is marked as deprecated and needs to be
;; explicitly enabled.
#$@(if (target-64bit?) '() '("--enable-deprecated")))
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 14:00 +0200
[PATCH v3 4/5] gnu: Add openmpi-5.
(address . 72062@debbugs.gnu.org)
20240712120140.20391-5-romain.garbage@inria.fr
* gnu/packages/mpi.scm (openmpi-5): New variable.

Change-Id: I048692b2e928077c9cfa9fb8076ceb356251e1db
---
gnu/packages/mpi.scm | 59 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)

Toggle diff (86 lines)
diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 6270108b16..01f07a0f7c 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2018–2022 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2018 Paul Garlick <pgarlick@tourbillion-technology.com>
;;; Copyright © 2019, 2021 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -49,6 +50,7 @@ (define-module (gnu packages mpi)
#:use-module (gnu packages ncurses)
#:use-module (gnu packages parallel)
#:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages python)
#:use-module (gnu packages valgrind)
#:use-module (srfi srfi-1)
#:use-module (ice-9 match))
@@ -307,6 +309,63 @@ (define-public openmpi
;; See file://LICENSE
(license license:bsd-2)))
+(define-public openmpi-5
+ (package
+ (inherit openmpi)
+ (version "5.0.3")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (string-append "https://www.open-mpi.org/software/ompi/v"
+ (version-major+minor version)
+ "/downloads/openmpi-" version ".tar.bz2"))
+ (sha256
+ (base32 "02x9xmpggw77mdpikjjx83j6i4v3gkqbncda73lk5axk0vr841cr"))))
+
+ (inputs (modify-inputs (package-inputs openmpi)
+ ;; As of Open MPI 5.0.X, PMIx is used to communicate
+ ;; with SLURM, so SLURM'S PMI is no longer needed.
+ (delete "slurm")
+ (append ucx) ;for Infiniband support
+ (append openpmix) ;for PMI support (launching via "srun")
+ (append prrte))) ;for PMI support (launching via "srun")
+ (native-inputs (modify-inputs (package-native-inputs openmpi)
+ (append python)))
+
+ (outputs '("out" "debug"))
+ (arguments
+ (substitute-keyword-arguments (package-arguments openmpi)
+ ((#:configure-flags _)
+ #~(list "--enable-mpi-ext=affinity" ;cr doesn't work
+ "--with-sge"
+
+ #$@(if (package? (this-package-input "valgrind"))
+ #~("--enable-memchecker"
+ "--with-valgrind")
+ #~("--without-valgrind"))
+
+ "--with-hwloc=external"
+ "--with-libevent"
+
+ ;; This replaces --enable-mpirun-prefix-by-default wich is deprecated
+ ;; since 5.x.
+ "--enable-prte-prefix-by-default"
+
+ ;; Enable support for the 'Process Management Interface for Exascale'
+ ;; (PMIx) used e.g. by Slurm for the management communication and
+ ;; coordination of MPI processes.
+ (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+ (string-append "--with-prrte=" #$(this-package-input "prrte"))
+
+ ;; Since 5.x, Infiniband support is provided by ucx.
+ ;; See https://docs.open-mpi.org/en/main/release-notes/networks.html#miscellaneous-network-notes
+ (string-append "--with-ucx=" #$(this-package-input "ucx"))))
+
+ ((#:phases phases)
+ #~(modify-phases #$phases
+ (delete 'remove-absolute)
+ (delete 'scrub-timestamps)))))))
+
(define-public openmpi-c++
(package/inherit openmpi
(name "openmpi-c++")
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 14:00 +0200
[PATCH v3 5/5] gnu: openmpi: Set default version to 4.x.
(address . 72062@debbugs.gnu.org)
20240712120140.20391-6-romain.garbage@inria.fr
* gnu/packages/mpi.scm (openmpi-4): New variable.
(openmpi): Redefine variable.

Change-Id: If5bb4429c935982ddaa1a5acbd728367ebbfd6fd
---
gnu/packages/mpi.scm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

Toggle diff (24 lines)
diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 01f07a0f7c..7b639b9033 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -185,7 +185,7 @@ (define-public hwloc
;; The latest stable series of hwloc.
hwloc-2)
-(define-public openmpi
+(define-public openmpi-4
(package
(name "openmpi")
(version "4.1.6")
@@ -309,6 +309,8 @@ (define-public openmpi
;; See file://LICENSE
(license license:bsd-2)))
+(define-public openmpi openmpi-4)
+
(define-public openmpi-5
(package
(inherit openmpi)
--
2.45.1
R
R
Romain GARBAGE wrote on 12 Jul 14:00 +0200
[PATCH v3 2/5] gnu: Add prrte.
(address . 72062@debbugs.gnu.org)
20240712120140.20391-3-romain.garbage@inria.fr
* gnu/packages/parallel.scm (prrte): New variable.

Change-Id: I4427eec1a5c7d94ed5f9cbceb55f9372834a8971
---
gnu/packages/parallel.scm | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)

Toggle diff (42 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index 4a57e85e6c..f173d1b75e 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -644,3 +644,35 @@ (define-public openpmix
(home-page "https://pmix.org/")
;; The provided license is kind of BSD-style but specific.
(license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
+
+(define-public prrte
+ (package
+ (name "prrte")
+ (version "3.0.6")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/openpmix/prrte/releases/download/v"
+ version "/prrte-" version ".tar.bz2"))
+ (sha256
+ (base32
+ "0wiy0vk37v4db1jgxza8bci0cczcvj34dalzsrlz05dk45zb7dl3"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list #:configure-flags #~(list (string-append "--with-hwloc="
+ (assoc-ref %build-inputs "hwloc"))
+ (string-append "--with-pmix=" #$(this-package-input "openpmix")))))
+ (inputs (list libevent
+ `(,hwloc "lib")
+ openpmix))
+ (native-inputs (list perl))
+ (synopsis "PMIx Reference RunTime Environment (PRRTE)")
+ (description
+ "The PMIx Reference RunTime Environment is a runtime environment
+containing the reference implementation and capable of operating
+within a host SMS. The reference RTE therefore provides an easy way of
+exploring PMIx capabilities and testing PMIx-based applications
+outside of a PMIx-enabled environment.")
+ (home-page "https://openpmix.github.io/")
+ ;; The provided license is kind of BSD-style but specific.
+ (license (license:fsf-free "https://github.com/openpmix/prrte?tab=License-1-ov-file#License-1-ov-file"))))
--
2.45.1
L
L
Ludovic Courtès wrote on 19 Aug 16:09 +0200
Re: [bug#72062] [PATCH v3 0/5] Add OpenMPI 5.x.
(name . Romain GARBAGE)(address . romain.garbage@inria.fr)(address . 72062-done@debbugs.gnu.org)
87le0svc2h.fsf@gnu.org
Hello,

Romain GARBAGE <romain.garbage@inria.fr> skribis:

Toggle quote (6 lines)
> gnu: Add openpmix.
> gnu: Add prrte.
> gnu: slurm: Add PMIx support.
> gnu: Add openmpi-5.
> gnu: openmpi: Set default version to 4.x.

I made the change below to ensure that all the versions of the ‘slurm’
package would still build. It took me a while :-) but in the end it’s
just about removing ‘openmpix’ from the inputs of older versions.

Will push shortly.

Thank you!

Ludo’.
Toggle diff (97 lines)
diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index c62402cf5c..440a808f68 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -247,7 +247,10 @@ (define-public slurm
(string-append "--with-munge=" #$(this-package-input "munge"))
;; Use PMIx bundled with Open MPI (this is required for Open MPI 5.x).
- (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+ #$(let ((openmpix (this-package-input "openpmix")))
+ (if openmpix
+ #~(string-append "--with-pmix=" #$openmpix)
+ "--without-pmix"))
;; 32-bit support is marked as deprecated and needs to be
;; explicitly enabled.
@@ -310,10 +313,10 @@ (define-public slurm-22.05
(define-public slurm-21.08
(package
- (inherit slurm)
+ (inherit slurm-22.05)
(version "21.08.8")
(source (origin
- (inherit (package-source slurm))
+ (inherit (package-source slurm-22.05))
(method url-fetch)
(uri (string-append
"https://download.schedmd.com/slurm/slurm-"
@@ -321,14 +324,19 @@ (define-public slurm-21.08
(patches '()) ;drop 'salloc' patch
(sha256
(base32
- "1sjln54idc9rhg8f2nvm38sgs6fncncyzslas8ixy65pqz2hphbf"))))))
+ "1sjln54idc9rhg8f2nvm38sgs6fncncyzslas8ixy65pqz2hphbf"))))
+
+ ;; This and older versions of slurm have PMIx support but they seem to
+ ;; require an older version of openpmix. Disable PMIx support.
+ (inputs (pk 'x (modify-inputs (package-inputs slurm-22.05)
+ (delete "openpmix"))))))
(define-public slurm-20.11
(package
- (inherit slurm)
+ (inherit slurm-21.08)
(version "20.11.9")
(source (origin
- (inherit (package-source slurm))
+ (inherit (package-source slurm-21.08))
(method url-fetch)
(uri (string-append
"https://download.schedmd.com/slurm/slurm-"
@@ -340,10 +348,10 @@ (define-public slurm-20.11
(define-public slurm-20.02
(package
- (inherit slurm)
+ (inherit slurm-20.11)
(version "20.02.6-1")
(source (origin
- (inherit (package-source slurm))
+ (inherit (package-source slurm-20.11))
(method url-fetch)
(uri (string-append
"https://download.schedmd.com/slurm/slurm-"
@@ -353,7 +361,7 @@ (define-public slurm-20.02
(base32
"0qj4blfymrd2ry2qmb58l3jbr4jwygc3adcfw7my27rippcijlyc"))))
(arguments
- (substitute-keyword-arguments (package-arguments slurm)
+ (substitute-keyword-arguments (package-arguments slurm-20.11)
((#:configure-flags flags ''())
#~(append '("CFLAGS=-O2 -g -fcommon" "LDFLAGS=-fcommon")
#$flags))))))
@@ -363,7 +371,7 @@ (define-public slurm-19.05
(inherit slurm-20.02)
(version "19.05.8")
(source (origin
- (inherit (package-source slurm))
+ (inherit (package-source slurm-20.02))
(method url-fetch)
(uri (string-append
"https://download.schedmd.com/slurm/slurm-"
@@ -376,11 +384,11 @@ (define-public slurm-19.05
;; Same as Debian 10
(define-public slurm-18.08
(package
- (inherit slurm-20.02)
+ (inherit slurm-19.05)
(version "18.08.9")
(source
(origin
- (inherit (package-source slurm))
+ (inherit (package-source slurm-20.02))
(uri (string-append
"https://download.schedmd.com/slurm/slurm-"
version ".tar.bz2"))
Closed
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 72062@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 72062
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch