[PATCH Added hdf5-1.12-parallel-openmpi] * gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on HDF5 1.12.0

OpenSubmitted by Gerd Heber.
Details
5 participants
  • Gerd Heber
  • Gerd Heber via web
  • Ludovic Courtès
  • Ludovic Courtès
  • zimoun
Owner
unassigned
Severity
normal
G
G
Gerd Heber wrote on 28 Feb 2021 14:33
(address . guix-patches@gnu.org)(name . Gerd Heber)(address . gheber@hdfgroup.org)
20210228133314.22935-1-gheber@hdfgroup.org
---
gnu/packages/maths.scm | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)

Toggle diff (54 lines)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 58433d4b0c..df101635ab 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -1451,7 +1451,7 @@ Swath).")
     (license (license:non-copyleft home-page))))
 
 (define-public hdf5-parallel-openmpi
-  (package/inherit hdf5-1.10                      ;use the latest
+  (package/inherit hdf5-1.10
     (name "hdf5-parallel-openmpi")
     (inputs
      `(("mpi" ,openmpi)
@@ -1482,6 +1482,38 @@ Swath).")
                #t))))))
     (synopsis "Management suite for data with parallel IO support")))
 
+(define-public hdf5-1.12-parallel-openmpi
+  (package/inherit hdf5-1.12
+    (name "hdf5-1.12-parallel-openmpi")
+    (inputs
+     `(("mpi" ,openmpi)
+       ,@(package-inputs hdf5)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments hdf5)
+       ((#:configure-flags flags)
+        ``("--enable-parallel"
+           ,@(delete "--enable-cxx"
+                     (delete "--enable-threadsafe" ,flags))))
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (add-after 'build 'mpi-setup
+             ,%openmpi-setup)
+           (add-before 'check 'patch-tests
+             (lambda _
+               ;; OpenMPI's mpirun will exit with non-zero status if it
+               ;; detects an "abnormal termination", i.e. any process not
+               ;; calling MPI_Finalize().  Since the test is explicitly
+               ;; avoiding MPI_Finalize so as not to have at_exit and thus
+               ;; H5C_flush_cache from being called, mpirun will always
+               ;; complain, so turn this test off.
+               (substitute* "testpar/Makefile"
+                 (("(^TEST_PROG_PARA.*)t_pflush1(.*)" front back)
+                  (string-append front back "\n")))
+               (substitute* "tools/test/h5diff/testph5diff.sh"
+                 (("/bin/sh") (which "sh")))
+               #t))))))
+    (synopsis "Management suite for data with parallel IO support")))
+
 (define-public hdf5-blosc
   (package
     (name "hdf5-blosc")
-- 
2.30.1
G
G
Gerd Heber via web wrote on 28 Feb 2021 23:42
(address . 46830@debbugs.gnu.org)
7f9bd991d490.1956b6a559d1e5c4@guile.gnu.org
This needs to be a separate package since (as with all HDF5 major releases) it's not 100% API compatible with HDF5 1.10. It would break other packages, e.g., `petsc-lite`
Z
Z
zimoun wrote on 1 Mar 2021 11:41
(name . Gerd Heber)(address . gheber@hdfgroup.org)
86im6b9mwy.fsf@gmail.com
Hi,

On Sun, 28 Feb 2021 at 07:33, Gerd Heber <gerd.heber@gmail.com> wrote:

Toggle quote (33 lines)
> +(define-public hdf5-1.12-parallel-openmpi
> + (package/inherit hdf5-1.12
> + (name "hdf5-1.12-parallel-openmpi")
> + (inputs
> + `(("mpi" ,openmpi)
> + ,@(package-inputs hdf5)))
> + (arguments
> + (substitute-keyword-arguments (package-arguments hdf5)
> + ((#:configure-flags flags)
> + ``("--enable-parallel"
> + ,@(delete "--enable-cxx"
> + (delete "--enable-threadsafe" ,flags))))
> + ((#:phases phases)
> + `(modify-phases ,phases
> + (add-after 'build 'mpi-setup
> + ,%openmpi-setup)
> + (add-before 'check 'patch-tests
> + (lambda _
> + ;; OpenMPI's mpirun will exit with non-zero status if it
> + ;; detects an "abnormal termination", i.e. any process not
> + ;; calling MPI_Finalize(). Since the test is explicitly
> + ;; avoiding MPI_Finalize so as not to have at_exit and thus
> + ;; H5C_flush_cache from being called, mpirun will always
> + ;; complain, so turn this test off.
> + (substitute* "testpar/Makefile"
> + (("(^TEST_PROG_PARA.*)t_pflush1(.*)" front back)
> + (string-append front back "\n")))
> + (substitute* "tools/test/h5diff/testph5diff.sh"
> + (("/bin/sh") (which "sh")))
> + #t))))))
> + (synopsis "Management suite for data with parallel IO support")))
> +

Why duplicates? Something like:

Toggle snippet (6 lines)
(define-public hdf5-1.12-parallel-openmpi
(package/inherit hdf5-parallel-openmpi
(version (package-version hdf5-1.12))
(source (package-source hdf5-1.12)))

seems simpler.


All the best,
simon
Z
Z
zimoun wrote on 2 Mar 2021 10:41
CAJ3okZ3gBvFzk_rnKTRZF7FL-X10cP0G763SJzwhk0ULyBXW0A@mail.gmail.com
Hi,

(Please CC the bug number.)

On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber@gmail.com> wrote:
Toggle quote (12 lines)
> On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune@gmail.com> wrote:

> > --8<---------------cut here---------------start------------->8---
> > (define-public hdf5-1.12-parallel-openmpi
> > (package/inherit hdf5-parallel-openmpi
> > (version (package-version hdf5-1.12))
> > (source (package-source hdf5-1.12)))
> > --8<---------------cut here---------------end--------------->8---

> Hey, thanks, very clever! Should I submit a new patch or is there a
> way to patch the patch?

The usual way is to send a v2 which is the patch to apply to the Git
tree (and *not* a patch to apply to a patch to apply to the tree).

However, personally I am not in favor to add to Guix proper too much
versions of variant packages; but that's just my humble opinion.
Instead, it could be submitted to specific channels, for instance
Guix-HPC.


Cheers,
simon
L
L
Ludovic Courtès wrote on 2 Mar 2021 20:39
Re: bug#46830: [PATCH Added hdf5-1.12-parallel-openmpi] * gnu/packages/maths.scm (hdf5-1.12-parallel-openmpi): New package based on HDF5 1.12.0
(name . zimoun)(address . zimon.toutoune@gmail.com)
874khtuyyt.fsf_-_@gnu.org
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (22 lines)
> On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber@gmail.com> wrote:
>> On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune@gmail.com> wrote:
>
>> > --8<---------------cut here---------------start------------->8---
>> > (define-public hdf5-1.12-parallel-openmpi
>> > (package/inherit hdf5-parallel-openmpi
>> > (version (package-version hdf5-1.12))
>> > (source (package-source hdf5-1.12)))
>> > --8<---------------cut here---------------end--------------->8---
>
>> Hey, thanks, very clever! Should I submit a new patch or is there a
>> way to patch the patch?
>
> The usual way is to send a v2 which is the patch to apply to the Git
> tree (and *not* a patch to apply to a patch to apply to the tree).
>
> However, personally I am not in favor to add to Guix proper too much
> versions of variant packages; but that's just my humble opinion.
> Instead, it could be submitted to specific channels, for instance
> <https://gitlab.inria.fr/guix-hpc/guix-hpc> or any other listed in
> Guix-HPC.

In this particular case, I think the proposed variant is fine; it’s
consistent with what we’re already doing with hdf5.

Ludo’.
G
G
Gerd Heber wrote on 3 Mar 2021 15:10
(name . Ludovic Courtès)(address . ludo@gnu.org)
CANnLdWHqJ+EhCgU_82Dn0PG=d4oUH-aGNM-FvsemCd3tOiZ1ug@mail.gmail.com
Ludovic, how are you? Thanks for taking the time to reply. This
conversation makes
me wonder what the Guix model for packages such as HDF5 might be.
On the one hand, there should be defaults for typical, i.e., non-HPC users, and
they probably belong into gnu/packages/maths.scm. Once you add MPI to the
mix, things get a little more complicated, and channels, such as
Guix-HPC might be
the better place. (?) Should we run our own channel?

In HDF5's case, we also use versions in a peculiar way, where the expectation is
that 1.8.x, 1.10.x, and 1.12.x, etc. version are not compatible for
end-user applications,
mostly because of API changes.

What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12
belong into maths.scm, plus the thread-safe builds, but not
hdf5-parallel-openmpi?
I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI)
atomicity tests fail
with the OpenMPI version used in the current hdf5-parallel-openmpi.
And then there is MPICH...

I would also like to see HDFView as a Guix package. We have a Spack package, but
the Java story on Guix (I don't blame you ;-) is a little confusing.

I'm sold on the merits of Guix and you are doing fantastic work.
What's your recommendation, and what can we (The HDF Group) do to help?

Best, G.

On Tue, Mar 2, 2021 at 1:39 PM Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (29 lines)
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > On Tue, 2 Mar 2021 at 01:32, Gerd Heber <gerd.heber@gmail.com> wrote:
> >> On Mon, Mar 1, 2021 at 4:45 AM zimoun <zimon.toutoune@gmail.com> wrote:
> >
> >> > --8<---------------cut here---------------start------------->8---
> >> > (define-public hdf5-1.12-parallel-openmpi
> >> > (package/inherit hdf5-parallel-openmpi
> >> > (version (package-version hdf5-1.12))
> >> > (source (package-source hdf5-1.12)))
> >> > --8<---------------cut here---------------end--------------->8---
> >
> >> Hey, thanks, very clever! Should I submit a new patch or is there a
> >> way to patch the patch?
> >
> > The usual way is to send a v2 which is the patch to apply to the Git
> > tree (and *not* a patch to apply to a patch to apply to the tree).
> >
> > However, personally I am not in favor to add to Guix proper too much
> > versions of variant packages; but that's just my humble opinion.
> > Instead, it could be submitted to specific channels, for instance
> > <https://gitlab.inria.fr/guix-hpc/guix-hpc> or any other listed in
> > Guix-HPC.
>
> In this particular case, I think the proposed variant is fine; it’s
> consistent with what we’re already doing with hdf5.
>
> Ludo’.
Z
Z
zimoun wrote on 5 Mar 2021 02:08
(address . 46830@debbugs.gnu.org)
86r1ku76fq.fsf@gmail.com
Hi Gerd,

On Wed, 03 Mar 2021 at 08:10, Gerd Heber <gerd.heber@gmail.com> wrote:

Toggle quote (4 lines)
> What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12
> belong into maths.scm, plus the thread-safe builds, but not
> hdf5-parallel-openmpi?

From my understanding, hdf5 at versions 1.8 and 1.10 are used by other
packages. When those very packages will not use at one of these
versions, I guess the very version will be removed.

hdf5-parallel-openmpi is used by petsc-openmpi for instance. And this
hdf5 variant is built with version 1.10. Since there is no package that
requires hdf5-parallel-openmpi at another version than 1.10, I do not
see the point to include it in Guix proper. Especially when the custom
variant is straightforward to locally create and buildable on a
reasonable amount of time.

However, these words are not totally acceptable. :-) If I take the shoes
of a regular scientist, then they only wants the package at hand and not
necessary RTFM how to do package transformations.

Toggle quote (4 lines)
> I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI)
> atomicity tests fail
> with the OpenMPI version used in the current hdf5-parallel-openmpi.

Yes, and we could imagine different versions of openmpi. And then
compiled with different compiler versions, etc…

Toggle quote (2 lines)
> And then there is MPICH...

…and the matrix of combinations is exploding. ;-)


One issue with the channel is to provide substitutes for that channel.
For example, the channel guix-science uses TravisCI to build the package
from GitHub.



That’s said, the cons about channels–and so the pros about include the
hdf5 variant in Guix proper–is to keep consistency and detect breakage:
Guix proper updates a package that becomes incompatible with one the
variant living a channel. All in Guix proper, then Guix CI will detect
it. Some dependencies in Guix proper and hdf variant in a channel, then
the channel CI probably not since nothing changed (from the channel
side) and so nothing triggered a build. I do not know.

Well, let stay pragmatic. :-)


Toggle quote (3 lines)
> I would also like to see HDFView as a Guix package. We have a Spack
> package, but

It would be really cool!


Toggle quote (3 lines)
> I'm sold on the merits of Guix and you are doing fantastic work.
> What's your recommendation, and what can we (The HDF Group) do to help?

Thanks the HDF Group for their interest on Guix.


Where the package definition ends (channel vs Guix proper) is one thing,
maybe a start should to have these hdf5 variant definitions. Then from
a pragmatic point of view, depending on these definitions (number,
length, etc.), they could ends in (gnu packages maths) or maybe its own
module (gnu packages hdf) or maybe in a channel. WDYT?


All the best,
simon
L
L
Ludovic Courtès wrote on 8 Mar 2021 15:01
(name . Gerd Heber)(address . gerd.heber@gmail.com)
87r1kpd9rt.fsf@gnu.org
Hi Gerd,

Gerd Heber <gerd.heber@gmail.com> skribis:

Toggle quote (9 lines)
> Ludovic, how are you? Thanks for taking the time to reply. This
> conversation makes
> me wonder what the Guix model for packages such as HDF5 might be.
> On the one hand, there should be defaults for typical, i.e., non-HPC users, and
> they probably belong into gnu/packages/maths.scm. Once you add MPI to the
> mix, things get a little more complicated, and channels, such as
> Guix-HPC might be
> the better place. (?) Should we run our own channel?

As you write, the goal for packages in Guix proper should be to have
“good defaults”, and possibly useful variants, as is currently the case
with HDF5.

We won’t keep all versions and variants in Guix proper because that’d be
too much of a maintenance burden. For “unusual” variants, I recommend
maintaining your own channel. (This is something we did at Inria for
example with an Open MPI variant that includes the MPI1 compatibility
layer.)

Additionally, you can use things like ‘--with-input=openmpi=mpich’. We
may eventually get support for “parameterized packages”, which will
allow users to choose between HDF5 variants from the command line:


Toggle quote (4 lines)
> What's your recommendation? Maybe (hdf5-1.6?), hdf5-1.8, hdf-1.10, and hdf5-1.12
> belong into maths.scm, plus the thread-safe builds, but not
> hdf5-parallel-openmpi?

For Guix proper, I think we should keep the number of HDF5 variants
stable. We should remove old versions once they’re no longer needed by
any in-tree package.

Toggle quote (8 lines)
> I tried to build hdf5-1.8.22-parallel-openmpi, but some of the (MPI)
> atomicity tests fail
> with the OpenMPI version used in the current hdf5-parallel-openmpi.
> And then there is MPICH...
>
> I would also like to see HDFView as a Guix package. We have a Spack package, but
> the Java story on Guix (I don't blame you ;-) is a little confusing.

That’s a different story :-) but please don’t hesitate to share your
experience, frustration, and questions regarding Java on the mailing
lists, I’m sure you’ll get some guidance.

Toggle quote (3 lines)
> I'm sold on the merits of Guix and you are doing fantastic work.
> What's your recommendation, and what can we (The HDF Group) do to help?

It’s great to get feedback from upstream; you can certainly help us make
wise(r) decisions regarding HDF5 packaging, in particular in deciding
which variants and versions make sense, what defaults are reasonable,
and how well they work in an HPC and non-HPC setting.

WDYT? What changes would you like to see?

Thanks for your interest!

Ludo’.
?