[PATCH] gnu: nomad-optimizer: Disable failing test.

  • Done
  • quality assurance status badge
Details
3 participants
  • Maxime Devos
  • Mathieu Othacehe
  • Paul A. Patience
Owner
unassigned
Submitted by
Paul A. Patience
Severity
normal
P
P
Paul A. Patience wrote on 22 Apr 2022 18:20
(address . guix-patches@gnu.org)(name . Paul A. Patience)(address . paul@apatience.com)
20220422162024.91287-1-paul@apatience.com
* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
It turns out the PSDMads test fails randomly.
I've disabled it.

gnu/packages/maths.scm | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)

Toggle diff (25 lines)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..26488a614c 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -2008,6 +2008,9 @@ (define-public nomad-optimizer
(add-after 'fix-sources-for-build 'fix-sources-for-tests
(lambda _
(substitute* "examples/CMakeLists.txt"
+ ;; This test passes only sometimes.
+ (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+ "")
;; examples/basic/batch/example3 is accidentally omitted.
(("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
_ prefix suffix)
@@ -2052,10 +2055,6 @@ (define-public nomad-optimizer
"examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

(let ((builddir (string-append (getcwd) "/../build")))
- ;; For some reason, omitting this fix causes the
- ;; examples/advanced/library/PSDMads test to fail, even though
- ;; it doesn't seem to reference any part of the FixedVariable
- ;; test.
(let ((dir "examples/advanced/library/FixedVariable"))
(substitute* (string-append dir "/fixedVariable.cpp")
(("^( +std::string sExe = ).*" _ prefix)
--
2.35.1
P
P
Paul A. Patience wrote on 22 Apr 2022 18:30
[PATCH v2] gnu: nomad-optimizer: Disable failing test.
(address . 55067@debbugs.gnu.org)(name . Paul A. Patience)(address . paul@apatience.com)
20220422163021.92454-1-paul@apatience.com
* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
Just updating the copyright header too.

gnu/packages/maths.scm | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)

Toggle diff (34 lines)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..190b62fca1 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -45,7 +45,7 @@
;;; Copyright © 2021 Gerd Heber <gerd.heber@gmail.com>
;;; Copyright © 2021 Franck Pérignon <franck.perignon@univ-grenoble-alpes.fr>
;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
-;;; Copyright © 2021 Paul A. Patience <paul@apatience.com>
+;;; Copyright © 2021-2022 Paul A. Patience <paul@apatience.com>
;;; Copyright © 2021 Ivan Gankevich <i.gankevich@spbu.ru>
;;; Copyright © 2021 Jean-Baptiste Volatier <jbv@pm.me>
;;; Copyright © 2021 Guillaume Le Vaillant <glv@posteo.net>
@@ -2008,6 +2008,9 @@ (define-public nomad-optimizer
(add-after 'fix-sources-for-build 'fix-sources-for-tests
(lambda _
(substitute* "examples/CMakeLists.txt"
+ ;; This test passes only sometimes.
+ (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+ "")
;; examples/basic/batch/example3 is accidentally omitted.
(("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
_ prefix suffix)
@@ -2052,10 +2055,6 @@ (define-public nomad-optimizer
"examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

(let ((builddir (string-append (getcwd) "/../build")))
- ;; For some reason, omitting this fix causes the
- ;; examples/advanced/library/PSDMads test to fail, even though
- ;; it doesn't seem to reference any part of the FixedVariable
- ;; test.
(let ((dir "examples/advanced/library/FixedVariable"))
(substitute* (string-append dir "/fixedVariable.cpp")
(("^( +std::string sExe = ).*" _ prefix)
--
2.35.1
M
M
Maxime Devos wrote on 22 Apr 2022 19:25
Re: [bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.
3fca716b728e196be715326f84c51d2e90e29eef.camel@telenet.be
Paul A. Patience schreef op vr 22-04-2022 om 16:20 [+0000]:
Toggle quote (3 lines)
> ---
> It turns out the PSDMads test fails randomly.

Sounds like a bug, preferably it would 100% reliably report the
underlying problem, whatever it is.

Toggle quote (2 lines)
> I've disabled it.

I don't think this is a good reason to disable the test.
Disabling tests whenever they fail defeats the point of tests.

Cf. with, say, clinical trials for testing safety of new medical
treatments. If some drug fails one of the tests (sometimes or always),
then one cannot simply ignore the test.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYmLlABccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hwiAQDwX5SoHl+wjcDbNijThzY3OQ96
qAOIGOXiQXf0puSCBwEAjPVDiqUo1xBiAHZUgg4ykSnzvXk/mwuiyCFDtJkfLgw=
=xbPW
-----END PGP SIGNATURE-----


P
P
Paul A. Patience wrote on 22 Apr 2022 21:02
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 55067@debbugs.gnu.org)
87ee1pne56.fsf@apatience.com
On 2022-04-22 13:25:20-04:00, Maxime Devos wrote:
Toggle quote (5 lines)
>> It turns out the PSDMads test fails randomly.
>
> Sounds like a bug, preferably it would 100% reliably report the
> underlying problem, whatever it is.

I agree with you, and I have already reported the issue to the
developers (along with some other issues that I addressed in the package
definition).

Toggle quote (5 lines)
>> I've disabled it.
>
> I don't think this is a good reason to disable the test.
> Disabling tests whenever they fail defeats the point of tests.

This test failure wasn't getting detected in the past due to a missing
"set -o pipefail" in the test runner, which could explain why it was
missed by the developers.
I don't know enough to fix the bug myself, but I have a suspicion it is
a bug in the test rather than in the library.

Best regards,
Paul
M
M
Maxime Devos wrote on 22 Apr 2022 23:37
(name . Paul A. Patience)(address . paul@apatience.com)(address . 55067@debbugs.gnu.org)
4ca2b6316466f8d2c64b5f5892cc70fcd5533130.camel@telenet.be
Paul A. Patience schreef op vr 22-04-2022 om 19:02 [+0000]:
Toggle quote (10 lines)
> On 2022-04-22 13:25:20-04:00, Maxime Devos wrote:
> > > It turns out the PSDMads test fails randomly.
> >
> > Sounds like a bug, preferably it would 100% reliably report the
> > underlying problem, whatever it is.
>
> I agree with you, and I have already reported the issue to the
> developers (along with some other issues that I addressed in the package
> definition).

Then I recommend adding a link to the upstream report (in a comment),
such that at the next update we know where to check if it has been
resolved, and to avoid giving the impression that just skipping tests
is acceptable.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYmMgERccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7v0ZAP9NqvGlqBERPkhuSCvo5l1FHave
Pw1y7Hf+93yJnuxEogEA+Vdkh3AG71PjK9dG1Yuav0E9Dz6OSnWg18WYW6ImLgg=
=Bm64
-----END PGP SIGNATURE-----


P
P
Paul A. Patience wrote on 25 Apr 2022 21:10
[PATCH v3] gnu: nomad-optimizer: Disable failing test.
(address . 55067@debbugs.gnu.org)(name . Paul A. Patience)(address . paul@apatience.com)
20220425191002.58325-1-paul@apatience.com
* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
I've added a link to the issue I opened upstream regarding this failing test,
as suggested by Maxime Devos.

gnu/packages/maths.scm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

Toggle diff (35 lines)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..448cdb4d73 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -45,7 +45,7 @@
;;; Copyright © 2021 Gerd Heber <gerd.heber@gmail.com>
;;; Copyright © 2021 Franck Pérignon <franck.perignon@univ-grenoble-alpes.fr>
;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
-;;; Copyright © 2021 Paul A. Patience <paul@apatience.com>
+;;; Copyright © 2021-2022 Paul A. Patience <paul@apatience.com>
;;; Copyright © 2021 Ivan Gankevich <i.gankevich@spbu.ru>
;;; Copyright © 2021 Jean-Baptiste Volatier <jbv@pm.me>
;;; Copyright © 2021 Guillaume Le Vaillant <glv@posteo.net>
@@ -2008,6 +2008,10 @@ (define-public nomad-optimizer
(add-after 'fix-sources-for-build 'fix-sources-for-tests
(lambda _
(substitute* "examples/CMakeLists.txt"
+ ;; This test passes only sometimes.
+ ;; See https://github.com/bbopt/nomad/issues/72.
+ (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+ "")
;; examples/basic/batch/example3 is accidentally omitted.
(("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
_ prefix suffix)
@@ -2052,10 +2056,6 @@ (define-public nomad-optimizer
"examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

(let ((builddir (string-append (getcwd) "/../build")))
- ;; For some reason, omitting this fix causes the
- ;; examples/advanced/library/PSDMads test to fail, even though
- ;; it doesn't seem to reference any part of the FixedVariable
- ;; test.
(let ((dir "examples/advanced/library/FixedVariable"))
(substitute* (string-append dir "/fixedVariable.cpp")
(("^( +std::string sExe = ).*" _ prefix)
--
2.35.1
M
M
Mathieu Othacehe wrote on 26 Apr 2022 10:45
Re: bug#55067: [PATCH] gnu: nomad-optimizer: Disable failing test.
(name . Paul A. Patience)(address . paul@apatience.com)(address . 55067-done@debbugs.gnu.org)
878rrsw7v8.fsf_-_@gnu.org
Toggle quote (3 lines)
> * gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
> examples/advanced/library/PSDMads test.

Pushed, thanks Paul & Maxime!

Mathieu
Closed
?