[PATCH 0/2] VTK packaging improvements for HPC

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Ludovic Courtès
  • Simon Tournier
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 14 Mar 2023 12:04
(address . guix-patches@gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20230314110422.8667-1-ludo@gnu.org
Hello,

Here are VTK packaging improvements for high-performance computing users,
plus a reduction of the package's closure size.

Feedback welcome!

Ludo'.

Ludovic Courtès (2):
gnu: vtk: Enable OpenMP, TBB, and Open MPI support.
gnu: vtk: Remove reference to GCC.

gnu/packages/image-processing.scm | 91 ++++++++++++++++++++-----------
1 file changed, 58 insertions(+), 33 deletions(-)


base-commit: 0bb0eeddf647e5d56afd4517b12919a36acac6ee
--
2.39.2
L
L
Ludovic Courtès wrote on 14 Mar 2023 12:06
[PATCH 1/2] gnu: vtk: Enable OpenMP, TBB, and Open MPI support.
(address . 62183@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
20230314110633.8729-1-ludo@gnu.org
From: Ludovic Courtès <ludovic.courtes@inria.fr>

* gnu/packages/image-processing.scm (vtk)[arguments]: Pass options to
enable OpenMP, TBB, and MPI.
[inputs]: Add TBB and OPENMPI.
---
gnu/packages/image-processing.scm | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)

Toggle diff (41 lines)
diff --git a/gnu/packages/image-processing.scm b/gnu/packages/image-processing.scm
index 6f51499142..a612544232 100644
--- a/gnu/packages/image-processing.scm
+++ b/gnu/packages/image-processing.scm
@@ -1,7 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017 John Darrington <jmd@gnu.org>
;;; Copyright © 2017, 2019, 2022 Ricardo Wurmus <rekado@elephly.net>
-;;; Copyright © 2014, 2021-2022 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2021-2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2014 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2016 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2018–2021 Tobias Geerinckx-Rice <me@tobias.gr>
@@ -380,6 +380,10 @@ (define-public vtk
"-DVTK_MODULE_ENABLE_VTK_RenderingExternal=YES" ; For F3D
"-DVTK_WRAP_PYTHON=ON"
"-DVTK_PYTHON_VERSION:STRING=3"
+
+ "-DVTK_SMP_ENABLE_OPENNMP=ON"
+ "-DVTK_SMP_ENABLE_TBB=ON"
+ "-DVTK_USE_MPI=ON"
)
#:tests? #f)) ;XXX: test data not included
(inputs
@@ -402,11 +406,13 @@ (define-public vtk
mesa
netcdf
libpng
+ libtiff
+ openmpi
proj-7
python
;("pugixml" ,pugixml)
sqlite
- libtiff
+ tbb
xorgproto
zlib))
(propagated-inputs
--
2.39.2
L
L
Ludovic Courtès wrote on 14 Mar 2023 12:06
[PATCH 2/2] gnu: vtk: Remove reference to GCC.
(address . 62183@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
20230314110633.8729-2-ludo@gnu.org
From: Ludovic Courtès <ludovic.courtes@inria.fr>

This removes GCC from the closure of VTK, reducing its size by 200 MiB (17%).

* gnu/packages/image-processing.scm (vtk)[arguments]: Change to gexps.
Add #:phases.
(vtk-7)[arguments]: Turn #:configure-flags into a gexp.
---
gnu/packages/image-processing.scm | 87 +++++++++++++++++++------------
1 file changed, 53 insertions(+), 34 deletions(-)

Toggle diff (108 lines)
diff --git a/gnu/packages/image-processing.scm b/gnu/packages/image-processing.scm
index a612544232..5c3560174c 100644
--- a/gnu/packages/image-processing.scm
+++ b/gnu/packages/image-processing.scm
@@ -352,40 +352,59 @@ (define-public vtk
(properties `((release-monitoring-url . "https://vtk.org/download/")))
(build-system cmake-build-system)
(arguments
- '(#:build-type "Release" ;Build without '-g' to save space.
- #:configure-flags '(;"-DBUILD_TESTING:BOOL=TRUE"
- ; ; not honored
- "-DVTK_USE_EXTERNAL=OFF" ;; default
- "-DVTK_MODULE_USE_EXTERNAL_VTK_doubleconversion=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_eigen=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_expat=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_freetype=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_gl2ps=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_glew=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_hdf5=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_jpeg=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_jsoncpp=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_libharu=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_libproj=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_libxml2=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_lz4=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_netcdf=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_ogg=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_png=ON"
- ;"-DVTK_MODULE_USE_EXTERNAL_VTK_pugixml=ON" ; breaks IO/CityGML
- "-DVTK_MODULE_USE_EXTERNAL_VTK_sqlite=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_theora=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_tiff=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_zlib=ON"
- "-DVTK_MODULE_ENABLE_VTK_RenderingExternal=YES" ; For F3D
- "-DVTK_WRAP_PYTHON=ON"
- "-DVTK_PYTHON_VERSION:STRING=3"
+ (list #:build-type "Release" ;Build without '-g' to save space.
+ #:configure-flags
+ #~'( ;;"-DBUILD_TESTING:BOOL=TRUE" ;not honored
+ "-DVTK_USE_EXTERNAL=OFF" ;default
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_doubleconversion=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_eigen=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_expat=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_freetype=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_gl2ps=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_glew=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_hdf5=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_jpeg=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_jsoncpp=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_libharu=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_libproj=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_libxml2=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_lz4=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_netcdf=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_ogg=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_png=ON"
+ ;;"-DVTK_MODULE_USE_EXTERNAL_VTK_pugixml=ON" ;breaks IO/CityGML
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_sqlite=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_theora=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_tiff=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_zlib=ON"
+ "-DVTK_MODULE_ENABLE_VTK_RenderingExternal=YES" ;for F3D
+ "-DVTK_WRAP_PYTHON=ON"
+ "-DVTK_PYTHON_VERSION:STRING=3"
- "-DVTK_SMP_ENABLE_OPENNMP=ON"
- "-DVTK_SMP_ENABLE_TBB=ON"
- "-DVTK_USE_MPI=ON"
- )
- #:tests? #f)) ;XXX: test data not included
+ "-DVTK_SMP_ENABLE_OPENNMP=ON"
+ "-DVTK_SMP_ENABLE_TBB=ON"
+ "-DVTK_USE_MPI=ON"
+ )
+
+ #:phases
+ #~(modify-phases %standard-phases
+ (add-after 'unpack 'clear-reference-to-compiler
+ (lambda _
+ (define (choose . files)
+ (let loop ((files files))
+ (if (null? files)
+ #f
+ (if (file-exists? (car files))
+ (car files)
+ (loop (cdr files))))))
+
+ ;; Do not retain a reference to GCC.
+ (substitute* (choose
+ "Common/Core/vtkConfigureDeprecated.h.in" ;v9.x
+ "Common/Core/vtkConfigure.h.in") ;v7.x
+ (("@CMAKE_CXX_COMPILER@") "c++")))))
+
+ #:tests? #f)) ;XXX: test data not included
(inputs
(list double-conversion
eigen
@@ -452,7 +471,7 @@ (define-public vtk-7
((#:configure-flags flags)
;; Otherwise, the build would fail with: "error: invalid conversion
;; from ‘const char*’ to ‘char*’ [-fpermissive]".
- `(cons "-DCMAKE_CXX_FLAGS=-fpermissive" ,flags))
+ #~(cons "-DCMAKE_CXX_FLAGS=-fpermissive" #$flags))
((#:phases phases)
#~(modify-phases #$phases
(add-after 'unpack 'remove-kernel-version
--
2.39.2
L
L
Ludovic Courtès wrote on 20 Mar 2023 23:09
[PATCH v2 1/2] gnu: vtk: Enable OpenMP, TBB, and Open MPI support.
(address . 62183@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
20230320220956.13920-1-ludo@gnu.org
From: Ludovic Courtès <ludovic.courtes@inria.fr>

* gnu/packages/image-processing.scm (vtk)[arguments]: Pass options to
enable OpenMP, TBB, and MPI.
[inputs]: Add OPENMPI.
[propagated-inputs]: Add TBB.
---
gnu/packages/image-processing.scm | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)

Change since v1: TBB is now propagated, fixing builds of mia and f3d.

Ludo'.

Toggle diff (52 lines)
diff --git a/gnu/packages/image-processing.scm b/gnu/packages/image-processing.scm
index 0562376291..c8786d8edd 100644
--- a/gnu/packages/image-processing.scm
+++ b/gnu/packages/image-processing.scm
@@ -1,7 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017 John Darrington <jmd@gnu.org>
;;; Copyright © 2017, 2019, 2022 Ricardo Wurmus <rekado@elephly.net>
-;;; Copyright © 2014, 2021-2022 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2021-2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2014 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2016 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2018–2021 Tobias Geerinckx-Rice <me@tobias.gr>
@@ -380,6 +380,10 @@ (define-public vtk
"-DVTK_MODULE_ENABLE_VTK_RenderingExternal=YES" ; For F3D
"-DVTK_WRAP_PYTHON=ON"
"-DVTK_PYTHON_VERSION:STRING=3"
+
+ "-DVTK_SMP_ENABLE_OPENNMP=ON"
+ "-DVTK_SMP_ENABLE_TBB=ON"
+ "-DVTK_USE_MPI=ON"
)
#:tests? #f)) ;XXX: test data not included
(inputs
@@ -402,17 +406,20 @@ (define-public vtk
mesa
netcdf
libpng
+ libtiff
+ openmpi
proj-7
python
;("pugixml" ,pugixml)
sqlite
- libtiff
xorgproto
zlib))
(propagated-inputs
;; VTK's 'VTK-vtk-module-find-packages.cmake' calls
- ;; 'find_package(THEORA)', which in turns looks for libogg.
- (list libogg))
+ ;; 'find_package(THEORA)', which in turns looks for libogg. Likewise for
+ ;; TBB.
+ (list libogg
+ tbb))
(home-page "https://vtk.org/")
(synopsis "Libraries for 3D computer graphics")
(description

base-commit: 99114eb63895712d2d3e62c5a83e3a1540438f15
--
2.39.2
L
L
Ludovic Courtès wrote on 20 Mar 2023 23:09
[PATCH v2 2/2] gnu: vtk: Remove reference to GCC.
(address . 62183@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
20230320220956.13920-2-ludo@gnu.org
From: Ludovic Courtès <ludovic.courtes@inria.fr>

This removes GCC from the closure of VTK, reducing its size by 200 MiB (17%).

* gnu/packages/image-processing.scm (vtk)[arguments]: Change to gexps.
Add #:phases.
(vtk-7)[arguments]: Turn #:configure-flags into a gexp.
---
gnu/packages/image-processing.scm | 87 +++++++++++++++++++------------
1 file changed, 53 insertions(+), 34 deletions(-)

Toggle diff (108 lines)
diff --git a/gnu/packages/image-processing.scm b/gnu/packages/image-processing.scm
index c8786d8edd..7652b24e1a 100644
--- a/gnu/packages/image-processing.scm
+++ b/gnu/packages/image-processing.scm
@@ -352,40 +352,59 @@ (define-public vtk
(properties `((release-monitoring-url . "https://vtk.org/download/")))
(build-system cmake-build-system)
(arguments
- '(#:build-type "Release" ;Build without '-g' to save space.
- #:configure-flags '(;"-DBUILD_TESTING:BOOL=TRUE"
- ; ; not honored
- "-DVTK_USE_EXTERNAL=OFF" ;; default
- "-DVTK_MODULE_USE_EXTERNAL_VTK_doubleconversion=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_eigen=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_expat=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_freetype=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_gl2ps=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_glew=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_hdf5=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_jpeg=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_jsoncpp=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_libharu=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_libproj=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_libxml2=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_lz4=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_netcdf=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_ogg=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_png=ON"
- ;"-DVTK_MODULE_USE_EXTERNAL_VTK_pugixml=ON" ; breaks IO/CityGML
- "-DVTK_MODULE_USE_EXTERNAL_VTK_sqlite=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_theora=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_tiff=ON"
- "-DVTK_MODULE_USE_EXTERNAL_VTK_zlib=ON"
- "-DVTK_MODULE_ENABLE_VTK_RenderingExternal=YES" ; For F3D
- "-DVTK_WRAP_PYTHON=ON"
- "-DVTK_PYTHON_VERSION:STRING=3"
+ (list #:build-type "Release" ;Build without '-g' to save space.
+ #:configure-flags
+ #~'( ;;"-DBUILD_TESTING:BOOL=TRUE" ;not honored
+ "-DVTK_USE_EXTERNAL=OFF" ;default
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_doubleconversion=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_eigen=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_expat=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_freetype=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_gl2ps=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_glew=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_hdf5=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_jpeg=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_jsoncpp=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_libharu=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_libproj=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_libxml2=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_lz4=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_netcdf=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_ogg=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_png=ON"
+ ;;"-DVTK_MODULE_USE_EXTERNAL_VTK_pugixml=ON" ;breaks IO/CityGML
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_sqlite=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_theora=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_tiff=ON"
+ "-DVTK_MODULE_USE_EXTERNAL_VTK_zlib=ON"
+ "-DVTK_MODULE_ENABLE_VTK_RenderingExternal=YES" ;for F3D
+ "-DVTK_WRAP_PYTHON=ON"
+ "-DVTK_PYTHON_VERSION:STRING=3"
- "-DVTK_SMP_ENABLE_OPENNMP=ON"
- "-DVTK_SMP_ENABLE_TBB=ON"
- "-DVTK_USE_MPI=ON"
- )
- #:tests? #f)) ;XXX: test data not included
+ "-DVTK_SMP_ENABLE_OPENNMP=ON"
+ "-DVTK_SMP_ENABLE_TBB=ON"
+ "-DVTK_USE_MPI=ON"
+ )
+
+ #:phases
+ #~(modify-phases %standard-phases
+ (add-after 'unpack 'clear-reference-to-compiler
+ (lambda _
+ (define (choose . files)
+ (let loop ((files files))
+ (if (null? files)
+ #f
+ (if (file-exists? (car files))
+ (car files)
+ (loop (cdr files))))))
+
+ ;; Do not retain a reference to GCC.
+ (substitute* (choose
+ "Common/Core/vtkConfigureDeprecated.h.in" ;v9.x
+ "Common/Core/vtkConfigure.h.in") ;v7.x
+ (("@CMAKE_CXX_COMPILER@") "c++")))))
+
+ #:tests? #f)) ;XXX: test data not included
(inputs
(list double-conversion
eigen
@@ -453,7 +472,7 @@ (define-public vtk-7
((#:configure-flags flags)
;; Otherwise, the build would fail with: "error: invalid conversion
;; from ‘const char*’ to ‘char*’ [-fpermissive]".
- `(cons "-DCMAKE_CXX_FLAGS=-fpermissive" ,flags))
+ #~(cons "-DCMAKE_CXX_FLAGS=-fpermissive" #$flags))
((#:phases phases)
#~(modify-phases #$phases
(add-after 'unpack 'remove-kernel-version
--
2.39.2
L
L
Ludovic Courtès wrote on 21 Mar 2023 19:12
Re: bug#62183: [PATCH 0/2] VTK packaging improvements for HPC
(address . 62183-done@debbugs.gnu.org)
874jqedld7.fsf_-_@gnu.org
Pushed as e646032aa212e84cfc8c8a5be59cb633ffc11c0b!

Ludo’.
Closed
S
S
Simon Tournier wrote on 7 Apr 2023 17:38
Re: [bug#62183] [PATCH v2 2/2] gnu: vtk: Remove reference to GCC.
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
874jprln2m.fsf@gmail.com
Hi,

On lun., 20 mars 2023 at 23:09, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (3 lines)
> This removes GCC from the closure of VTK, reducing its size by 200 MiB
> (17%).

Awesome! Well done! :-)

Cheers,
simon

PS: Following the meeting with Kitware folks, you did not tweak
Paraview, right?

+ PARAVIEW_ENABLE_CATALYST because catalyst is not packages yet
+ No dependencies to OSPRay, OpenVDB, OpenXR, ADIOS2, Fides, OpenTURNS,
TTK, etc.

And ’paraview’ is only packages in Guix proper and not in some guix-hpc
channels, right? Well, the only versions [1] I found are from Guix
proper.

L
L
Ludovic Courtès wrote on 17 Apr 2023 10:17
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 62183@debbugs.gnu.org)
87o7nmj51r.fsf@gnu.org
Hi!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (7 lines)
> PS: Following the meeting with Kitware folks, you did not tweak
> Paraview, right?
>
> + PARAVIEW_ENABLE_CATALYST because catalyst is not packages yet
> + No dependencies to OSPRay, OpenVDB, OpenXR, ADIOS2, Fides, OpenTURNS,
> TTK, etc.

Correct, this has yet to be done.

Toggle quote (4 lines)
> And ’paraview’ is only packages in Guix proper and not in some guix-hpc
> channels, right? Well, the only versions [1] I found are from Guix
> proper.

Right, I removed the ‘paraview’ package from the ‘guix-hpc’ as it had
become stale.

Ludo’.
?