(address . bug-guix@gnu.org)
ZpWWxKkFQnVhybBF@ws
Hello,
currently there are two issues with build-aux/test-driver.scm.
1. :test-global-result: is mistyped as :global-test-result:.
2. Summarizing metadata are outputted on end of each test-group, meaning they
will be present multiple times, if there is more than one test group.
First is somewhat fine, since, per the manual page, unknown keys are (for now)
ignored. However second is explicitly stated to be undefined behavior, so it
should be fixed.
I am CC-ing both authors listed in the test-driver.scm, since I have no idea
where the upstream is (it seems to be just copied around between projects?).
You can find fixes for the first (0001) and second (0002) issues attached to
this email.
I am also including two additional patches (0003 refining :test-global-result:
for XPASS as XXX mentioned, 0005 improving support for test group since it was
somewhat lacking) which I consider useful and I have applied them to my local
copy of test-driver.scm file. Feel free to apply them, or ignore them, as you
see fit.
Have a nice day,
Tomas Volf
--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.
From ff4a7a69e7f35ddf80287b230ac1b377e54b43e9 Mon Sep 17 00:00:00 2001
From: Tomas Volf <~@wolfsden.cz>
Date: Sun, 14 Jul 2024 01:08:49 +0200
Subject: [PATCH 1/5] build: test-driver.scm: Fix :test-global-result: .trs
metadata.
According to the manual page, the correct metadata key is :test-global-result:
instead of the current :global-test-result:.
* build-aux/test-driver.scm (test-on-group-end-gnu): Fix :test-global-result:
metadata output.
---
build-aux/test-driver.scm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
Toggle diff (24 lines)
diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm
index 9a0dcde..471333b 100755
--- a/build-aux/test-driver.scm
+++ b/build-aux/test-driver.scm
@@ -3,7 +3,7 @@ exec guile --no-auto-compile -e main -s "$0" "$@"
!#
;;;; test-driver.scm - Guile test driver for Automake testsuite harness
-(define script-version "2023-12-08.16") ;UTC
+(define script-version "2024-07-14.10") ;UTC
;;; Copyright © 2015, 2016 Mathieu Lirzin <mthl@gnu.org>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
@@ -178,7 +178,7 @@ cases based on their names."
(skip (or (positive? (test-runner-skip-count runner))
(positive? (test-runner-xfail-count runner)))))
;; XXX: The global results need some refinements for XPASS.
- (format trs-port ":global-test-result: ~A~%"
+ (format trs-port ":test-global-result: ~A~%"
(if fail "FAIL" (if skip "SKIP" "PASS")))
(format trs-port ":recheck: ~A~%"
(if fail "yes" "no"))
--
2.45.2
From 250cf5af8324bc09591f178c868f409574692de0 Mon Sep 17 00:00:00 2001
From: Tomas Volf <~@wolfsden.cz>
Date: Sun, 14 Jul 2024 22:45:16 +0200
Subject: [PATCH 3/5] build: test-driver.scm: Refine the global test result.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The :test-global-result: .trs metadata contained just either FAIL, SKIP or
PASS with a comment that further refinements are required for XPASS. The
description of :test-global-result: is in the manual is as follows:
This is used to declare the "global result" of the script.
Currently, the value of this field is needed only to be reported
(more or less verbatim) in the generated global log file
‘$(TEST_SUITE_LOG)’, so it’s quite free-form. For example, a test
script which runs 10 test cases, 6 of which pass and 4 of which are
skipped, could reasonably have a ‘PASS/SKIP’ value for this field,
while a test script which runs 19 successful tests and one failed
test could have an ‘ALMOST PASSED’ value.
As we can see, the examples as `PASS/SKIP' and `ALMOST PASSED', so there is no
need to stick to strict model. Hence this commit changes the resulting value
to be comma-separated list of PASS, FAIL, XPASS, XFAIL and SKIP. The
respective elements are present only when the count of tests with such a
result is positive.
In practice, that should usually produce lines such as
:test-global-result: PASS,FAIL
or
:test-global-result: PASS
* build-aux/test-driver.scm (test-runner-gnu)[finalize]: Refine the output of
:test-global-result:.
---
build-aux/test-driver.scm | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
Toggle diff (39 lines)
diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm
index 71bca73..9d412e1 100755
--- a/build-aux/test-driver.scm
+++ b/build-aux/test-driver.scm
@@ -3,7 +3,7 @@ exec guile --no-auto-compile -e main -s "$0" "$@"
!#
;;;; test-driver.scm - Guile test driver for Automake testsuite harness
-(define script-version "2024-07-14.11") ;UTC
+(define script-version "2024-07-14.20") ;UTC
;;; Copyright © 2015, 2016 Mathieu Lirzin <mthl@gnu.org>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
@@ -182,9 +182,20 @@ do the final reporting."
(positive? (test-runner-xpass-count runner))))
(skip (or (positive? (test-runner-skip-count runner))
(positive? (test-runner-xfail-count runner)))))
- ;; XXX: The global results need some refinements for XPASS.
- (format trs-port ":test-global-result: ~A~%"
- (if fail "FAIL" (if skip "SKIP" "PASS")))
+ (format trs-port ":test-global-result: ~{~A~^,~}~%"
+ (filter-map (λ (proc str)
+ (let ((n (proc runner)))
+ (if (positive? n) str #f)))
+ (list test-runner-pass-count
+ test-runner-fail-count
+ test-runner-xpass-count
+ test-runner-xfail-count
+ test-runner-skip-count)
+ (list "PASS"
+ "FAIL"
+ "XPASS"
+ "XFAIL"
+ "SKIP")))
(format trs-port ":recheck: ~A~%"
(if fail "yes" "no"))
(format trs-port ":copy-in-global-log: ~A~%"
--
2.45.2
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEt4NJs4wUfTYpiGikL7/ufbZ/wakFAmaVlxoACgkQL7/ufbZ/
wan6dRAAhLhHKzlOxDVQn5DG3JVQcp/co0uLsOQoB/KNrIxL/wqeDwJJRLKKCl0V
D6IMpJLFq7arst9zGLyPUkDbJXnHP4WqdLm+VHwSbJVe9JPm96hJkiTyiN/cYUp7
yThZlGHWe9LCXnmSNrbu2F4KxD9kraWGUh5K4lX1JN0m7NGQCYSjdGlXa6yQsAEr
vuQSCJj1Cqqe9MtiTp1hQLuGQvhJghZ/Infy3t0OddZz6K3OId1Wffm+Qfa5Tr7/
IC1wN/bsX7HgFR3+eS6B461tHXC+rgn1CEnxeCWDvqRbH6sRVsdm134OnH7J8BK4
xhpfYy/bmtDXlM67aBUUeMii5vaTOKN4HrhoXdcagAolP7mDuiL7R3yTX4znNyzJ
1mB85ULlLuLWV4/Fgrs6FmnpJaedj4gigTZJiTpBAYwozGuNN8uOxHlfcO+ys9a8
/An7POSPjmfuVzKPfnkdvEkb/hrjS0S3W2/yxooubdjtzzqc6lsEAezDh1iEFw07
GziNpnmzxS7kFBfYekrrN26OnVc0oQdTFoCrWnwqgOdNMBJ7TK+hWzP88oU8ESOp
JJgAno9vT4ITQP1fKgneoerPbjeDaUE2oJiJYycJUL9HkdNQDINtDELhdETiyIqW
6+brSGgWr85pMei60PHauOTyTbf3II9XfRL1WgutBVjDX/Y1xoI=
=VcZs
-----END PGP SIGNATURE-----