PETSc is not reproducible

  • Done
  • quality assurance status badge
Details
3 participants
  • Andreas Enge
  • Lars Bilke
  • Ludovic Courtès
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 5 Aug 2022 16:51
(address . bug-guix@gnu.org)
87fsiag2vm.fsf@inria.fr
PETSc 3.16.1 is not reproducible as it captures the kernel version and
number of processors of the machine it’s built on:

Toggle snippet (50 lines)
$ guix describe
Generation 223 Aug 01 2022 00:30:24 (current)
guix ab59155
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: ab59155c5a38dda7efaceb47c7528578fcf0def4
$ guix challenge petsc --diff="$(type -P diff) -ru"
/gnu/store/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1 contents differ:
no local build for '/gnu/store/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1'
https://ci.guix.gnu.org/nar/lzip/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1: 1fqg24x0qq0vww8a21a4ifyxlwy0l7bg8zzrzvg024lk76n54kxh
https://bordeaux.guix.gnu.org/nar/lzip/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1: 162fp2wsxxjjl1pd42522jb3i37hpq8lnfs12qaipzhkzqda5h34
bordeaux.guix.gnu.org 10.3MiB 9.7MiB/s 00:01 [##################] 100.0%diff -ru /tmp/guix-directory.4kiUDI/include/petscmachineinfo.h /tmp/guix-directory.0i2wFi/include/petscmachineinfo.h
--- /tmp/guix-directory.4kiUDI/include/petscmachineinfo.h 1970-01-01 01:00:01.000000000 +0100
+++ /tmp/guix-directory.0i2wFi/include/petscmachineinfo.h 1970-01-01 01:00:01.000000000 +0100
@@ -1,6 +1,6 @@
static const char *petscmachineinfo = "\n"
"-----------------------------------------\n"
-"Machine characteristics: Linux-5.15.10-gnu-x86_64-with-glibc2.33\n"
+"Machine characteristics: Linux-5.15.11-gnu-x86_64-with-glibc2.33\n"
"Using PETSc directory: /gnu/store/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1\n"
"Using PETSc arch: \n"
"-----------------------------------------\n";
Binary files /tmp/guix-directory.4kiUDI/lib/libpetsc.so and /tmp/guix-directory.0i2wFi/lib/libpetsc.so differ
Binary files /tmp/guix-directory.4kiUDI/lib/libpetsc.so.3.16 and /tmp/guix-directory.0i2wFi/lib/libpetsc.so.3.16 differ
Binary files /tmp/guix-directory.4kiUDI/lib/libpetsc.so.3.16.1 and /tmp/guix-directory.0i2wFi/lib/libpetsc.so.3.16.1 differ
diff -ru /tmp/guix-directory.4kiUDI/lib/petsc/conf/petscvariables /tmp/guix-directory.0i2wFi/lib/petsc/conf/petscvariables
--- /tmp/guix-directory.4kiUDI/lib/petsc/conf/petscvariables 1970-01-01 01:00:01.000000000 +0100
+++ /tmp/guix-directory.0i2wFi/lib/petsc/conf/petscvariables 1970-01-01 01:00:01.000000000 +0100
@@ -107,10 +107,10 @@
OMAKE_PRINTDIR = /gnu/store/55cbpsi18mahg131nmiya6km5b4mscfa-make-4.3/bin/make --print-directory
OMAKE = /gnu/store/55cbpsi18mahg131nmiya6km5b4mscfa-make-4.3/bin/make --no-print-directory
MAKE_PAR_OUT_FLG = --output-sync=recurse
-MAKE_NP = 40
-MAKE_TEST_NP = 40
-MAKE_LOAD = 96.0
-NPMAX = 64
+MAKE_NP = 24
+MAKE_TEST_NP = 24
+MAKE_LOAD = 48.0
+NPMAX = 32
SONAME_FUNCTION = $(1).$(SL_LINKER_SUFFIX).$(2)
SL_LINKER_FUNCTION = -shared -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 -O0 -Wl,-soname,$(call SONAME_FUNCTION,$(notdir $(1)),$(2))
BUILDSHAREDLIB = yes

1 store items were analyzed:
- 0 (0.0%) were identical
- 1 (100.0%) differed
- 0 (0.0%) were inconclusive

Ludo’.
L
L
Lars Bilke wrote on 29 Feb 10:46 +0100
[PATCH v1] gnu: petsc: scrub build machine specifics
(address . 57005@debbugs.gnu.org)(name . Lars Bilke)(address . lars.bilke@ufz.de)
de4e8fff610a489bb8c14b1d260a5bd542dfc7d6.1709200010.git.lars.bilke@ufz.de
'Libraries compiled on' does not have to be removed since PETSc 3.9:

https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f

Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68

---
gnu/packages/maths.scm | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)

Toggle diff (35 lines)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 1f61ee0583..f09cbee48b 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -3455,18 +3455,21 @@ (define-public petsc
(substitute* "config/example_template.py"
(("#!/usr/bin/env bash")
(string-append "#!" (which "bash")))))))
- (add-after 'configure 'clean-local-references
+ (add-after 'install 'clean-local-references
(lambda* (#:key outputs #:allow-other-keys)
(let ((out (assoc-ref outputs "out")))
(substitute* (find-files "." "^petsc(conf|machineinfo).h$")
;; Prevent build directory from leaking into compiled code
(((getcwd)) out)
- ;; Scrub timestamp for reproducibility
- ((".*Libraries compiled on.*") ""))
+ ;; Scrub build machine characteristic
+ (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
+ "Machine characteristics: Linux-x.x.x"))
(substitute* (find-files "." "petscvariables")
;; Do not expose build machine characteristics, set to defaults.
- (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
- (("NPMAX = [:digit:]+") "NPMAX = 2")))))
+ (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
+ (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
+ (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
+ (("NPMAX = [[:digit:]]+") "NPMAX = 2")))))
(add-after 'install 'clean-install
;; Try to keep installed files from leaking build directory names.
(lambda* (#:key inputs outputs #:allow-other-keys)

base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
--
2.43.0
L
L
Lars Bilke wrote on 29 Feb 11:00 +0100
(address . 57005@debbugs.gnu.org)
44D266D4-7B4A-4486-AA7E-24CC9E4622B0@ufz.de
This patch makes these files deterministic:

- petscmachineinfo.h
- petscvariables

Unfortunately libpetsc.so.3.16.1 still differs between builds, see e.g. this diffoscope output:


Maybe someone has an idea how to also fix the library?

Sincerely,
Lars

On 29 Feb 2024, at 10:46, Lars Bilke wrote:

Toggle quote (45 lines)
> 'Libraries compiled on' does not have to be removed since PETSc 3.9:
>
> https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f
>
> Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68
>
> ---
> gnu/packages/maths.scm | 13 ++++++++-----
> 1 file changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index 1f61ee0583..f09cbee48b 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -3455,18 +3455,21 @@ (define-public petsc
> (substitute* "config/example_template.py"
> (("#!/usr/bin/env bash")
> (string-append "#!" (which "bash")))))))
> - (add-after 'configure 'clean-local-references
> + (add-after 'install 'clean-local-references
> (lambda* (#:key outputs #:allow-other-keys)
> (let ((out (assoc-ref outputs "out")))
> (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
> ;; Prevent build directory from leaking into compiled code
> (((getcwd)) out)
> - ;; Scrub timestamp for reproducibility
> - ((".*Libraries compiled on.*") ""))
> + ;; Scrub build machine characteristic
> + (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
> + "Machine characteristics: Linux-x.x.x"))
> (substitute* (find-files "." "petscvariables")
> ;; Do not expose build machine characteristics, set to defaults.
> - (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
> - (("NPMAX = [:digit:]+") "NPMAX = 2")))))
> + (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
> + (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
> + (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
> + (("NPMAX = [[:digit:]]+") "NPMAX = 2")))))
> (add-after 'install 'clean-install
> ;; Try to keep installed files from leaking build directory names.
> (lambda* (#:key inputs outputs #:allow-other-keys)
>
> base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
> --
> 2.43.0
Attachment: file
Attachment: smime.p7s
A
A
Andreas Enge wrote on 29 Feb 18:48 +0100
Suggestion
(address . 57005@debbugs.gnu.org)
ZeDDeFwO98E7pi1g@jurong
Hello Lars,

thanks for the patch! Concerning "Machine characteristics", I see it only
in petscmachineinfo.h, so we can probably drop looking for petscconf.h
in find-files.

There is already the cleaning phase 'clean-install after 'install.
So I would either suggest to keep the phase 'clean-local-references
where it is, or to merge its content with 'clean-install (I do not see
why one should be preferred over the other).

Andreas
L
L
Lars Bilke wrote on 1 Mar 09:27 +0100
[PATCH v2] gnu: petsc: scrub build machine specifics
(address . 57005@debbugs.gnu.org)(name . Lars Bilke)(address . lars.bilke@ufz.de)
18f29b40d908e9c965f49bc124a505a6278b2a50.1709281622.git.lars.bilke@ufz.de
'Libraries compiled on' does not have to be removed since PETSc 3.9:

https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f

Also merged all add-after 'install phases into `clean-install.

Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68
---
gnu/packages/maths.scm | 43 ++++++++++++++++++------------------------
1 file changed, 18 insertions(+), 25 deletions(-)

Toggle diff (75 lines)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 1f61ee0583..4b4d033506 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -3455,30 +3455,34 @@ (define-public petsc
(substitute* "config/example_template.py"
(("#!/usr/bin/env bash")
(string-append "#!" (which "bash")))))))
- (add-after 'configure 'clean-local-references
- (lambda* (#:key outputs #:allow-other-keys)
+ (add-after 'install 'clean-install
+ ;; Try to keep installed files from leaking build directory names.
+ (lambda* (#:key inputs outputs #:allow-other-keys)
(let ((out (assoc-ref outputs "out")))
- (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
+ (substitute* (find-files "." "petscmachineinfo.h")
;; Prevent build directory from leaking into compiled code
(((getcwd)) out)
- ;; Scrub timestamp for reproducibility
- ((".*Libraries compiled on.*") ""))
+ ;; Scrub build machine characteristic
+ (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
+ "Machine characteristics: Linux-x.x.x"))
(substitute* (find-files "." "petscvariables")
;; Do not expose build machine characteristics, set to defaults.
- (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
- (("NPMAX = [:digit:]+") "NPMAX = 2")))))
- (add-after 'install 'clean-install
- ;; Try to keep installed files from leaking build directory names.
- (lambda* (#:key inputs outputs #:allow-other-keys)
- (let ((out (assoc-ref outputs "out")))
+ (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
+ (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
+ (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
+ (("NPMAX = [[:digit:]]+") "NPMAX = 2"))
(substitute* (map (lambda (file)
(string-append out "/lib/petsc/conf/" file))
'("petscvariables"))
(((getcwd)) out))
- ;; Make compiler references point to the store
(substitute* (string-append out "/lib/petsc/conf/petscvariables")
- (("= (gcc|g\\+\\+|gfortran)" _ compiler)
- (string-append "= " (which compiler))))
+ ;; Do not retain a reference to GCC and other build only inputs.
+ (("([[:graph:]]+)/bin/gcc") "gcc")
+ (("([[:graph:]]+)/bin/g\\+\\+") "g++")
+ (("([[:graph:]]+)/bin/make") "make")
+ (("([[:graph:]]+)/bin/diff") "diff")
+ (("([[:graph:]]+)/bin/sed") "sed")
+ (("([[:graph:]]+)/bin/gfortran") "gfortran"))
;; PETSc installs some build logs, which aren't necessary.
(for-each (lambda (file)
(let ((f (string-append out "/lib/petsc/conf/" file)))
@@ -3490,17 +3494,6 @@ (define-public petsc
"configure-hash"
;; Once installed, should uninstall with Guix
"uninstall.py")))))
- (add-after 'clean-install 'clear-reference-to-compiler
- (lambda* (#:key inputs outputs #:allow-other-keys)
- ;; Do not retain a reference to GCC and other build only inputs.
- (let ((out (assoc-ref outputs "out")))
- (substitute* (string-append out "/lib/petsc/conf/petscvariables")
- (("([[:graph:]]+)/bin/gcc") "gcc")
- (("([[:graph:]]+)/bin/g\\+\\+") "g++")
- (("([[:graph:]]+)/bin/make") "make")
- (("([[:graph:]]+)/bin/diff") "diff")
- (("([[:graph:]]+)/bin/sed") "sed")
- (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
(add-after 'install 'move-examples
(lambda* (#:key outputs #:allow-other-keys)
(let* ((out (assoc-ref outputs "out"))

base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
--
2.43.0
L
L
Lars Bilke wrote on 5 Mar 15:58 +0100
[PATCH v3] gnu: petsc: scrub build machine specifics
(address . 57005@debbugs.gnu.org)(name . Lars Bilke)(address . lars.bilke@ufz.de)
65e5bfc8e0e0580eb9c86e1cc0dc8bb89b286a7a.1709650693.git.lars.bilke@ufz.de
'Libraries compiled on' does not have to be removed since PETSc 3.9:

https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f

Also merged all add-after 'install phases into `clean-install.

Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68

sort source files in generated makefiles for reproducibility

Change-Id: I86e5b9cd0d7184019617bd8ae9a23eaf459891d7
---
gnu/packages/maths.scm | 46 +++++++++++++++++++-----------------------
1 file changed, 21 insertions(+), 25 deletions(-)

Toggle diff (85 lines)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 1f61ee0583..b828d5d5cb 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -3447,6 +3447,9 @@ (define-public petsc
(let* ((prefix (assoc-ref outputs "out"))
(flags `(,(string-append "--prefix=" prefix)
,@configure-flags)))
+ ;; sort source files in configure (for reproducibility)
+ (substitute* "config/gmakegen.py"
+ (("join\\(srcs\\[lang\\]\\)") "join(sorted(srcs[lang]))"))
(format #t "build directory: ~s~%" (getcwd))
(format #t "configure flags: ~s~%" flags)
(apply invoke "./configure" flags)
@@ -3455,30 +3458,34 @@ (define-public petsc
(substitute* "config/example_template.py"
(("#!/usr/bin/env bash")
(string-append "#!" (which "bash")))))))
- (add-after 'configure 'clean-local-references
- (lambda* (#:key outputs #:allow-other-keys)
+ (add-after 'install 'clean-install
+ ;; Try to keep installed files from leaking build directory names.
+ (lambda* (#:key inputs outputs #:allow-other-keys)
(let ((out (assoc-ref outputs "out")))
- (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
+ (substitute* (find-files "." "petscmachineinfo.h")
;; Prevent build directory from leaking into compiled code
(((getcwd)) out)
- ;; Scrub timestamp for reproducibility
- ((".*Libraries compiled on.*") ""))
+ ;; Scrub build machine characteristic
+ (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
+ "Machine characteristics: Linux-x.x.x"))
(substitute* (find-files "." "petscvariables")
;; Do not expose build machine characteristics, set to defaults.
- (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
- (("NPMAX = [:digit:]+") "NPMAX = 2")))))
- (add-after 'install 'clean-install
- ;; Try to keep installed files from leaking build directory names.
- (lambda* (#:key inputs outputs #:allow-other-keys)
- (let ((out (assoc-ref outputs "out")))
+ (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
+ (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
+ (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
+ (("NPMAX = [[:digit:]]+") "NPMAX = 2"))
(substitute* (map (lambda (file)
(string-append out "/lib/petsc/conf/" file))
'("petscvariables"))
(((getcwd)) out))
- ;; Make compiler references point to the store
(substitute* (string-append out "/lib/petsc/conf/petscvariables")
- (("= (gcc|g\\+\\+|gfortran)" _ compiler)
- (string-append "= " (which compiler))))
+ ;; Do not retain a reference to GCC and other build only inputs.
+ (("([[:graph:]]+)/bin/gcc") "gcc")
+ (("([[:graph:]]+)/bin/g\\+\\+") "g++")
+ (("([[:graph:]]+)/bin/make") "make")
+ (("([[:graph:]]+)/bin/diff") "diff")
+ (("([[:graph:]]+)/bin/sed") "sed")
+ (("([[:graph:]]+)/bin/gfortran") "gfortran"))
;; PETSc installs some build logs, which aren't necessary.
(for-each (lambda (file)
(let ((f (string-append out "/lib/petsc/conf/" file)))
@@ -3490,17 +3497,6 @@ (define-public petsc
"configure-hash"
;; Once installed, should uninstall with Guix
"uninstall.py")))))
- (add-after 'clean-install 'clear-reference-to-compiler
- (lambda* (#:key inputs outputs #:allow-other-keys)
- ;; Do not retain a reference to GCC and other build only inputs.
- (let ((out (assoc-ref outputs "out")))
- (substitute* (string-append out "/lib/petsc/conf/petscvariables")
- (("([[:graph:]]+)/bin/gcc") "gcc")
- (("([[:graph:]]+)/bin/g\\+\\+") "g++")
- (("([[:graph:]]+)/bin/make") "make")
- (("([[:graph:]]+)/bin/diff") "diff")
- (("([[:graph:]]+)/bin/sed") "sed")
- (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
(add-after 'install 'move-examples
(lambda* (#:key outputs #:allow-other-keys)
(let* ((out (assoc-ref outputs "out"))

base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
--
2.43.0
L
L
Lars Bilke wrote on 5 Mar 16:01 +0100
(address . 57005@debbugs.gnu.org)
AB996BF0-5CDD-4BE5-9715-16C586C24784@ufz.de
v3 of this patch finally makes PETSc fully reproducible!

On 5 Mar 2024, at 15:58, Lars Bilke wrote:

Toggle quote (100 lines)
> 'Libraries compiled on' does not have to be removed since PETSc 3.9:
>
> https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f
>
> Also merged all add-after 'install phases into `clean-install.
>
> Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68
>
> sort source files in generated makefiles for reproducibility
>
> Change-Id: I86e5b9cd0d7184019617bd8ae9a23eaf459891d7
> ---
> gnu/packages/maths.scm | 46 +++++++++++++++++++-----------------------
> 1 file changed, 21 insertions(+), 25 deletions(-)
>
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index 1f61ee0583..b828d5d5cb 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -3447,6 +3447,9 @@ (define-public petsc
> (let* ((prefix (assoc-ref outputs "out"))
> (flags `(,(string-append "--prefix=" prefix)
> ,@configure-flags)))
> + ;; sort source files in configure (for reproducibility)
> + (substitute* "config/gmakegen.py"
> + (("join\\(srcs\\[lang\\]\\)") "join(sorted(srcs[lang]))"))
> (format #t "build directory: ~s~%" (getcwd))
> (format #t "configure flags: ~s~%" flags)
> (apply invoke "./configure" flags)
> @@ -3455,30 +3458,34 @@ (define-public petsc
> (substitute* "config/example_template.py"
> (("#!/usr/bin/env bash")
> (string-append "#!" (which "bash")))))))
> - (add-after 'configure 'clean-local-references
> - (lambda* (#:key outputs #:allow-other-keys)
> + (add-after 'install 'clean-install
> + ;; Try to keep installed files from leaking build directory names.
> + (lambda* (#:key inputs outputs #:allow-other-keys)
> (let ((out (assoc-ref outputs "out")))
> - (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
> + (substitute* (find-files "." "petscmachineinfo.h")
> ;; Prevent build directory from leaking into compiled code
> (((getcwd)) out)
> - ;; Scrub timestamp for reproducibility
> - ((".*Libraries compiled on.*") ""))
> + ;; Scrub build machine characteristic
> + (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
> + "Machine characteristics: Linux-x.x.x"))
> (substitute* (find-files "." "petscvariables")
> ;; Do not expose build machine characteristics, set to defaults.
> - (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
> - (("NPMAX = [:digit:]+") "NPMAX = 2")))))
> - (add-after 'install 'clean-install
> - ;; Try to keep installed files from leaking build directory names.
> - (lambda* (#:key inputs outputs #:allow-other-keys)
> - (let ((out (assoc-ref outputs "out")))
> + (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
> + (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
> + (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
> + (("NPMAX = [[:digit:]]+") "NPMAX = 2"))
> (substitute* (map (lambda (file)
> (string-append out "/lib/petsc/conf/" file))
> '("petscvariables"))
> (((getcwd)) out))
> - ;; Make compiler references point to the store
> (substitute* (string-append out "/lib/petsc/conf/petscvariables")
> - (("= (gcc|g\\+\\+|gfortran)" _ compiler)
> - (string-append "= " (which compiler))))
> + ;; Do not retain a reference to GCC and other build only inputs.
> + (("([[:graph:]]+)/bin/gcc") "gcc")
> + (("([[:graph:]]+)/bin/g\\+\\+") "g++")
> + (("([[:graph:]]+)/bin/make") "make")
> + (("([[:graph:]]+)/bin/diff") "diff")
> + (("([[:graph:]]+)/bin/sed") "sed")
> + (("([[:graph:]]+)/bin/gfortran") "gfortran"))
> ;; PETSc installs some build logs, which aren't necessary.
> (for-each (lambda (file)
> (let ((f (string-append out "/lib/petsc/conf/" file)))
> @@ -3490,17 +3497,6 @@ (define-public petsc
> "configure-hash"
> ;; Once installed, should uninstall with Guix
> "uninstall.py")))))
> - (add-after 'clean-install 'clear-reference-to-compiler
> - (lambda* (#:key inputs outputs #:allow-other-keys)
> - ;; Do not retain a reference to GCC and other build only inputs.
> - (let ((out (assoc-ref outputs "out")))
> - (substitute* (string-append out "/lib/petsc/conf/petscvariables")
> - (("([[:graph:]]+)/bin/gcc") "gcc")
> - (("([[:graph:]]+)/bin/g\\+\\+") "g++")
> - (("([[:graph:]]+)/bin/make") "make")
> - (("([[:graph:]]+)/bin/diff") "diff")
> - (("([[:graph:]]+)/bin/sed") "sed")
> - (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
> (add-after 'install 'move-examples
> (lambda* (#:key outputs #:allow-other-keys)
> (let* ((out (assoc-ref outputs "out"))
>
> base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
> --
> 2.43.0
Attachment: smime.p7s
L
L
Lars Bilke wrote on 6 Mar 12:51 +0100
[PATCH v4] gnu: petsc: scrub build machine specifics
(address . 57005@debbugs.gnu.org)(name . Lars Bilke)(address . lars.bilke@ufz.de)
8d1edc3769b3994fc2a7be34470509e7f45aee31.1709725874.git.lars.bilke@ufz.de
'Libraries compiled on' does not have to be removed since PETSc 3.9:

https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f

Also merged all add-after 'install phases into `clean-install.

Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68

sort source files in generated makefiles for reproducibility

Change-Id: I86e5b9cd0d7184019617bd8ae9a23eaf459891d7
---
gnu/packages/maths.scm | 56 ++++++++++++++++++------------------------
1 file changed, 24 insertions(+), 32 deletions(-)

Toggle diff (89 lines)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 1f61ee0583..1ea5ccee74 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -3447,38 +3447,41 @@ (define-public petsc
(let* ((prefix (assoc-ref outputs "out"))
(flags `(,(string-append "--prefix=" prefix)
,@configure-flags)))
+ ;; sort source files in configure (for reproducibility)
+ (substitute* "config/gmakegen.py"
+ (("join\\(srcs\\[lang\\]\\)") "join(sorted(srcs[lang]))"))
(format #t "build directory: ~s~%" (getcwd))
(format #t "configure flags: ~s~%" flags)
(apply invoke "./configure" flags)
-
+ ;; Prevent build directory Linux version from leaking into
+ ;; compiled code
+ (substitute* (find-files "." "petscmachineinfo.h")
+ (((getcwd)) out)
+ ;; Scrub build machine characteristic
+ (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
+ "Machine characteristics: Linux-x.x.x"))
;; Generate test scripts with the right shebang.
(substitute* "config/example_template.py"
(("#!/usr/bin/env bash")
(string-append "#!" (which "bash")))))))
- (add-after 'configure 'clean-local-references
- (lambda* (#:key outputs #:allow-other-keys)
- (let ((out (assoc-ref outputs "out")))
- (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
- ;; Prevent build directory from leaking into compiled code
- (((getcwd)) out)
- ;; Scrub timestamp for reproducibility
- ((".*Libraries compiled on.*") ""))
- (substitute* (find-files "." "petscvariables")
- ;; Do not expose build machine characteristics, set to defaults.
- (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
- (("NPMAX = [:digit:]+") "NPMAX = 2")))))
(add-after 'install 'clean-install
- ;; Try to keep installed files from leaking build directory names.
(lambda* (#:key inputs outputs #:allow-other-keys)
(let ((out (assoc-ref outputs "out")))
- (substitute* (map (lambda (file)
- (string-append out "/lib/petsc/conf/" file))
- '("petscvariables"))
- (((getcwd)) out))
- ;; Make compiler references point to the store
(substitute* (string-append out "/lib/petsc/conf/petscvariables")
- (("= (gcc|g\\+\\+|gfortran)" _ compiler)
- (string-append "= " (which compiler))))
+ ;; Try to keep installed files from leaking build directory names.
+ (((getcwd)) out)
+ ;; Do not expose build machine characteristics, set to defaults.
+ (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
+ (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
+ (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
+ (("NPMAX = [[:digit:]]+") "NPMAX = 2")
+ ;; Do not retain a reference to GCC and other build only inputs.
+ (("([[:graph:]]+)/bin/gcc") "gcc")
+ (("([[:graph:]]+)/bin/g\\+\\+") "g++")
+ (("([[:graph:]]+)/bin/make") "make")
+ (("([[:graph:]]+)/bin/diff") "diff")
+ (("([[:graph:]]+)/bin/sed") "sed")
+ (("([[:graph:]]+)/bin/gfortran") "gfortran"))
;; PETSc installs some build logs, which aren't necessary.
(for-each (lambda (file)
(let ((f (string-append out "/lib/petsc/conf/" file)))
@@ -3490,17 +3493,6 @@ (define-public petsc
"configure-hash"
;; Once installed, should uninstall with Guix
"uninstall.py")))))
- (add-after 'clean-install 'clear-reference-to-compiler
- (lambda* (#:key inputs outputs #:allow-other-keys)
- ;; Do not retain a reference to GCC and other build only inputs.
- (let ((out (assoc-ref outputs "out")))
- (substitute* (string-append out "/lib/petsc/conf/petscvariables")
- (("([[:graph:]]+)/bin/gcc") "gcc")
- (("([[:graph:]]+)/bin/g\\+\\+") "g++")
- (("([[:graph:]]+)/bin/make") "make")
- (("([[:graph:]]+)/bin/diff") "diff")
- (("([[:graph:]]+)/bin/sed") "sed")
- (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
(add-after 'install 'move-examples
(lambda* (#:key outputs #:allow-other-keys)
(let* ((out (assoc-ref outputs "out"))

base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
--
2.43.0
L
L
Lars Bilke wrote on 6 Mar 12:54 +0100
(address . 57005@debbugs.gnu.org)
8412A210-E386-46B9-A3E1-10D9240AA484@ufz.de
v4 should (hopefully) fix all issues regarding reprocuibility of petsc:

I have tested v4 on multiple machines with exact identical output (ran `guix hash -S nar ...` on the packages) and also tested on one machine with `guix build --check`

On 6 Mar 2024, at 12:51, Lars Bilke wrote:

Toggle quote (104 lines)
> 'Libraries compiled on' does not have to be removed since PETSc 3.9:
>
> https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f
>
> Also merged all add-after 'install phases into `clean-install.
>
> Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68
>
> sort source files in generated makefiles for reproducibility
>
> Change-Id: I86e5b9cd0d7184019617bd8ae9a23eaf459891d7
> ---
> gnu/packages/maths.scm | 56 ++++++++++++++++++------------------------
> 1 file changed, 24 insertions(+), 32 deletions(-)
>
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index 1f61ee0583..1ea5ccee74 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -3447,38 +3447,41 @@ (define-public petsc
> (let* ((prefix (assoc-ref outputs "out"))
> (flags `(,(string-append "--prefix=" prefix)
> ,@configure-flags)))
> + ;; sort source files in configure (for reproducibility)
> + (substitute* "config/gmakegen.py"
> + (("join\\(srcs\\[lang\\]\\)") "join(sorted(srcs[lang]))"))
> (format #t "build directory: ~s~%" (getcwd))
> (format #t "configure flags: ~s~%" flags)
> (apply invoke "./configure" flags)
> -
> + ;; Prevent build directory Linux version from leaking into
> + ;; compiled code
> + (substitute* (find-files "." "petscmachineinfo.h")
> + (((getcwd)) out)
> + ;; Scrub build machine characteristic
> + (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
> + "Machine characteristics: Linux-x.x.x"))
> ;; Generate test scripts with the right shebang.
> (substitute* "config/example_template.py"
> (("#!/usr/bin/env bash")
> (string-append "#!" (which "bash")))))))
> - (add-after 'configure 'clean-local-references
> - (lambda* (#:key outputs #:allow-other-keys)
> - (let ((out (assoc-ref outputs "out")))
> - (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
> - ;; Prevent build directory from leaking into compiled code
> - (((getcwd)) out)
> - ;; Scrub timestamp for reproducibility
> - ((".*Libraries compiled on.*") ""))
> - (substitute* (find-files "." "petscvariables")
> - ;; Do not expose build machine characteristics, set to defaults.
> - (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
> - (("NPMAX = [:digit:]+") "NPMAX = 2")))))
> (add-after 'install 'clean-install
> - ;; Try to keep installed files from leaking build directory names.
> (lambda* (#:key inputs outputs #:allow-other-keys)
> (let ((out (assoc-ref outputs "out")))
> - (substitute* (map (lambda (file)
> - (string-append out "/lib/petsc/conf/" file))
> - '("petscvariables"))
> - (((getcwd)) out))
> - ;; Make compiler references point to the store
> (substitute* (string-append out "/lib/petsc/conf/petscvariables")
> - (("= (gcc|g\\+\\+|gfortran)" _ compiler)
> - (string-append "= " (which compiler))))
> + ;; Try to keep installed files from leaking build directory names.
> + (((getcwd)) out)
> + ;; Do not expose build machine characteristics, set to defaults.
> + (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
> + (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
> + (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
> + (("NPMAX = [[:digit:]]+") "NPMAX = 2")
> + ;; Do not retain a reference to GCC and other build only inputs.
> + (("([[:graph:]]+)/bin/gcc") "gcc")
> + (("([[:graph:]]+)/bin/g\\+\\+") "g++")
> + (("([[:graph:]]+)/bin/make") "make")
> + (("([[:graph:]]+)/bin/diff") "diff")
> + (("([[:graph:]]+)/bin/sed") "sed")
> + (("([[:graph:]]+)/bin/gfortran") "gfortran"))
> ;; PETSc installs some build logs, which aren't necessary.
> (for-each (lambda (file)
> (let ((f (string-append out "/lib/petsc/conf/" file)))
> @@ -3490,17 +3493,6 @@ (define-public petsc
> "configure-hash"
> ;; Once installed, should uninstall with Guix
> "uninstall.py")))))
> - (add-after 'clean-install 'clear-reference-to-compiler
> - (lambda* (#:key inputs outputs #:allow-other-keys)
> - ;; Do not retain a reference to GCC and other build only inputs.
> - (let ((out (assoc-ref outputs "out")))
> - (substitute* (string-append out "/lib/petsc/conf/petscvariables")
> - (("([[:graph:]]+)/bin/gcc") "gcc")
> - (("([[:graph:]]+)/bin/g\\+\\+") "g++")
> - (("([[:graph:]]+)/bin/make") "make")
> - (("([[:graph:]]+)/bin/diff") "diff")
> - (("([[:graph:]]+)/bin/sed") "sed")
> - (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
> (add-after 'install 'move-examples
> (lambda* (#:key outputs #:allow-other-keys)
> (let* ((out (assoc-ref outputs "out"))
>
> base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
> --
> 2.43.0
Attachment: file
Attachment: smime.p7s
A
A
Andreas Enge wrote on 5 Sep 14:59 +0200
Close
(address . 57005-done@debbugs.gnu.org)
ZtmrRjUPj7_N619d@jurong
Hello Lars,

apologies for my lack of engagement.

If I understood things correctly, Ludovic's recent patch from
https://issues.guix.gnu.org/72877solves the reproducibility issue.
So I am closing this bug. If I am mistaken, please feel free to reopen it
with an updated patch.

Thanks,

Andreas
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 57005@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 57005
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch