[PATCH 0/7] Add anonip system test.

  • Open
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
normal
M
M
Maxim Cournoyer wrote on 1 Nov 07:20 +0100
(address . guix-patches@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
cover.1730441684.git.maxim.cournoyer@gmail.com
This was developed as part of investigating the source of bug #59181 ([berlin]
web services fail to start on reboot due to anonip).

There is no issue having anonip run and the system reboot (as should be
expected as /var/run is supposed to be cleared thus the named pipes should be
recreated every boot). This points to a race with nginx that would "win"
creating its log files under /var/run/anonip/* before the anonip service does.

It'll need further investigating in that direction.

Maxim Cournoyer (7):
doc: Use @table @code for anonip-configuration doc.
services: anonip: Add 'debug?' configuration field.
system/vm: Fix virtual-machine bug.
tests: web: Have the retry-on-error throw on exhausted attempts.
services: herd: Export 'eval-there' in API.
build: marionette: Make it possible to reboot VM during tests.
tests: Add anonip system test.

doc/guix.texi | 21 ++++---
gnu/build/marionette.scm | 2 +-
gnu/services/herd.scm | 1 +
gnu/services/web.scm | 8 ++-
gnu/system/vm.scm | 9 ++-
gnu/tests/web.scm | 117 ++++++++++++++++++++++++++++++++++++++-
6 files changed, 142 insertions(+), 16 deletions(-)


base-commit: 4009d1de954d694cb11af391d4113d29c5c1379d
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 08:11 +0100
[PATCH 1/7] doc: Use @table @code for anonip-configuration doc.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
281a4773768a6c271ff464f473fdbc333a58c348.1730441684.git.maxim.cournoyer@gmail.com
* doc/guix.texi (Log Rotation): Use @table @code for anonip-configuration doc.

Change-Id: Ib6f8136715ecabc81d53dc13a14f4b111302c5c9
---
doc/guix.texi | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)

Toggle diff (46 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 5768e195b0..b519b483fe 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20788,30 +20788,30 @@ Log Rotation
The following optional settings may be provided:
-@table @asis
-@item @code{skip-private?}
+@table @code
+@item skip-private?
When @code{#true} do not mask addresses in private ranges.
-@item @code{column}
+@item column
A 1-based indexed column number. Assume IP address is in the specified
column (default is 1).
-@item @code{replacement}
+@item replacement
Replacement string in case address parsing fails, e.g. @code{"0.0.0.0"}.
-@item @code{ipv4mask}
+@item ipv4mask
Number of bits to mask in IPv4 addresses.
-@item @code{ipv6mask}
+@item ipv6mask
Number of bits to mask in IPv6 addresses.
-@item @code{increment}
+@item increment
Increment the IP address by the given number. By default this is zero.
-@item @code{delimiter}
+@item delimiter
Log delimiter string.
-@item @code{regex}
+@item regex
Regular expression for detecting IP addresses. Use this instead of @code{column}.
@end table
@end deftp
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 08:11 +0100
[PATCH 2/7] services: anonip: Add 'debug?' configuration field.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
12708a075e13fc3acfb4f2685d561fd9ee481b87.1730441684.git.maxim.cournoyer@gmail.com
* gnu/services/web.scm (<anonip-configuration>) [debug?]: New field.
(anonip-shepherd-service): Honor it.
* doc/guix.texi (Log Rotation) <anonip-configuration>: Document it.

Change-Id: Iaf57b5992808374b069a55c34a9adfdfe52b046c
---
doc/guix.texi | 3 +++
gnu/services/web.scm | 8 ++++++--
2 files changed, 9 insertions(+), 2 deletions(-)

Toggle diff (55 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index b519b483fe..51a543d690 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20789,6 +20789,9 @@ Log Rotation
The following optional settings may be provided:
@table @code
+@item debug?
+Print debug messages when @code{#true}.
+
@item skip-private?
When @code{#true} do not mask addresses in private ranges.
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cc6f4e6d9b..cf3515bf70 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -220,6 +220,7 @@ (define-module (gnu services web)
anonip-configuration-anonip
anonip-configuration-input
anonip-configuration-output
+ anonip-configuration-debug?
anonip-configuration-skip-private?
anonip-configuration-column
anonip-configuration-replacement
@@ -1448,6 +1449,8 @@ (define-record-type* <anonip-configuration>
(default anonip))
(input anonip-configuration-input) ;string
(output anonip-configuration-output) ;string
+ (debug? anonip-configuration-debug? ;boolean
+ (default #f))
(skip-private? anonip-configuration-skip-private? ;boolean
(default #f))
(column anonip-configuration-column ;number
@@ -1503,14 +1506,15 @@ (define (anonip-shepherd-service config)
"/bin/anonip")
(string-append "--input=" #$input)
(string-append "--output=" #$output))
+ (if #$(anonip-configuration-debug? config)
+ '("--debug") (list))
(if #$(anonip-configuration-skip-private? config)
'("--skip-private") (list))
'#$(optional anonip-configuration-column "--column")
'#$(optional anonip-configuration-ipv4mask "--ipv4mask")
'#$(optional anonip-configuration-ipv6mask "--ipv6mask")
'#$(optional anonip-configuration-increment "--increment")
- '#$(optional anonip-configuration-replacement
- "--replacement")
+ '#$(optional anonip-configuration-replacement "--replacement")
'#$(optional anonip-configuration-delimiter "--delimiter")
'#$(optional anonip-configuration-regex "--regex"))
;; Run in a UTF-8 locale
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 08:11 +0100
[PATCH 3/7] system/vm: Fix virtual-machine bug.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
397067fb8b2b50efef8092d77c93c04e0f28225a.1730441684.git.maxim.cournoyer@gmail.com
The virtual-machine syntax would not accept a single operating-system field,
which was puzzling.

* gnu/system/vm.scm (virtual-machine): Add a pattern matching a single literal
'operating-system' field and value.

Change-Id: If207fd71df3a3f763b2e63229eafa82f63e80773
---
gnu/system/vm.scm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

Toggle diff (23 lines)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index a2743453e7..4a400056d7 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -422,10 +422,14 @@ (define-record-type* <virtual-machine> %virtual-machine
(default #f)))
(define-syntax virtual-machine
- (syntax-rules ()
+ (syntax-rules (operating-system)
"Declare a virtual machine running the specified OS, with the given
options."
- ((_ os) ;shortcut
+ ((_ (operating-system os))
+ ;; Also accept the long form (virtual-machine (operating-system os)), for
+ ;; correctness.
+ (%virtual-machine (operating-system os)))
+ ((_ os) ;shortcut
(%virtual-machine (operating-system os)))
((_ fields ...)
(%virtual-machine fields ...))))
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 08:11 +0100
[PATCH 4/7] tests: web: Have the retry-on-error throw on exhausted attempts.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
369fbb0b1eb904bf152356be15441aaed872c44a.1730441684.git.maxim.cournoyer@gmail.com
Previously, it'd simply return the error arguments, which would not always
fail a 'test-assert' test, for example.

* gnu/tests/web.scm (retry-on-error): Re-throw the error when the attempts
have been exhausted.

Change-Id: Ic1468d8ba23f0f78377e90d19bccb2878dc56f0e
---
gnu/tests/web.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (15 lines)
diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index a071e05e1d..5c50322cef 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -83,7 +83,7 @@ (define retry-on-error
return-value)
((#f . error-args)
(if (>= attempt times)
- error-args
+ (apply throw error-args)
(begin
(sleep delay)
(loop (+ 1 attempt)))))))))
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 08:11 +0100
[PATCH 5/7] services: herd: Export 'eval-there' in API.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
8c8215ff6140a98c412bb2df5faf19767635ea6b.1730441684.git.maxim.cournoyer@gmail.com
* gnu/services/herd.scm (gnu): Export 'eval-there', which is useful to
evaluate Scheme expressions.

Change-Id: Iff1db56e0847fc5886ac9fe594d677b363e7e6e7
---
gnu/services/herd.scm | 1 +
1 file changed, 1 insertion(+)

Toggle diff (14 lines)
diff --git a/gnu/services/herd.scm b/gnu/services/herd.scm
index 4b47acf72a..c185fc2cec 100644
--- a/gnu/services/herd.scm
+++ b/gnu/services/herd.scm
@@ -54,6 +54,7 @@ (define-module (gnu services herd)
with-shepherd-action
current-service
current-services
+ eval-there
unload-services
unload-service
load-services
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 08:11 +0100
[PATCH 6/7] build: marionette: Make it possible to reboot VM during tests.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
5d9173bd000c54075dc8b666dc0fcc8aa262f13a.1730441684.git.maxim.cournoyer@gmail.com
* gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
parameter.
* gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.

Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73
---
gnu/build/marionette.scm | 2 +-
gnu/system/vm.scm | 1 -
2 files changed, 1 insertion(+), 2 deletions(-)

Toggle diff (27 lines)
diff --git a/gnu/build/marionette.scm b/gnu/build/marionette.scm
index 0b0a8a70d8..af211eff43 100644
--- a/gnu/build/marionette.scm
+++ b/gnu/build/marionette.scm
@@ -108,7 +108,7 @@ (define* (make-marionette command
(list "-nographic"
"-monitor" (string-append "unix:" socket-directory "/monitor")
"-chardev" (string-append "socket,id=repl,path=" socket-directory
- "/repl")
+ "/repl,reconnect=1")
"-chardev" (string-append "socket,id=qga,server=on,wait=off,path="
socket-directory "/qemu-ga")
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4a400056d7..1e3f72c7b2 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
'("-enable-kvm")
'())
- "-no-reboot"
"-object" "rng-random,filename=/dev/urandom,id=guix-vm-rng"
"-device" "virtio-rng-pci,rng=guix-vm-rng"
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 08:11 +0100
[PATCH 7/7] tests: Add anonip system test.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
7c762086c432d21da2c664d4ee26c051fb54a481.1730441684.git.maxim.cournoyer@gmail.com
* gnu/tests/web.scm (%test-anonip): New test.
(%anonip-os): New variables.
(run-anonip-test): New procedure.

Change-Id: Ieed210a784dbdeee8a498e74b6c0e31cb72cd9b8
---
gnu/tests/web.scm | 115 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 115 insertions(+)

Toggle diff (149 lines)
diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 5c50322cef..f7a5659dc5 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2017, 2018 Clément Lassieur <clement@lassieur.org>
;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
;;; Copyright © 2018 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2024 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -33,6 +34,7 @@ (define-module (gnu tests web)
#:use-module (gnu services networking)
#:use-module (gnu services shepherd)
#:use-module (gnu services mail)
+ #:use-module (gnu packages base)
#:use-module (gnu packages databases)
#:use-module (gnu packages guile-xyz)
#:use-module (gnu packages patchutils)
@@ -52,6 +54,7 @@ (define-module (gnu tests web)
%test-php-fpm
%test-hpcguix-web
%test-tailon
+ %test-anonip
%test-patchwork
%test-agate))
@@ -509,6 +512,118 @@ (define %test-tailon
(description "Connect to a running Tailon server.")
(value (run-tailon-test))))
+
+;;;
+;;; Anonip
+;;;
+(define %anonip-os
+ ;; Operating system under test.
+ (simple-operating-system
+ (service anonip-service-type
+ (anonip-configuration
+ (input "/var/run/anonip/http.access.log")
+ (output "/var/log/anonip/http.access.log")
+ (debug? #t)))))
+
+(define (run-anonip-test)
+ (define os
+ (marionette-operating-system
+ %anonip-os
+ #:imported-modules '((gnu services herd)
+ (guix combinators))))
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ ;; We are interested in verifying if anonip still launches following a
+ ;; reboot; thus make the base image writable.
+ (volatile? #f)))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (ice-9 match)
+ (srfi srfi-64)
+ (gnu build marionette))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (test-runner-current (system-test-runner #$output))
+ (test-begin "anonip")
+
+ (test-assert "service is running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+ marionette))
+
+ (test-assert "service can be restarted"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (restart-service 'anonip-/var/log/anonip/http.access.log)
+ (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+ marionette))
+
+ (test-assert "ip addresses are anonymized"
+ (marionette-eval
+ '(begin
+ (use-modules (ice-9 textual-ports))
+ (call-with-output-file "/var/run/anonip/http.access.log"
+ (lambda (port)
+ (display "192.168.100.200 - - \
+[30/Oct/2024:14:57:44 +0100] GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)
+ (display "2001:0db8:85a3:0000:0000:8a2e:0370:7334 - - \
+[30/Oct/2024:14:57:44 +0100] \"GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)))
+ (#$retry-on-error
+ (lambda ()
+ (call-with-input-file "/var/log/anonip/http.access.log"
+ (lambda (port)
+ (let ((content (get-string-all port)))
+ ;; The expected values are taken from anonip's test
+ ;; suite (see its test_module.py file).
+ (or (and (string-contains content "192.168.96.0")
+ (string-contains content "2001:db8:85a0::"))
+ (error "could not find expected anonymized IPs"
+ content))))))
+ #:times 20
+ #:delay 1))
+ marionette))
+
+ (test-assert "service is running after reboot"
+ (begin
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (eval-there '(begin
+ (use-modules (shepherd system))
+ (sync) ;ensure the log is fully written
+ (reboot))))
+ marionette)
+ ;; Note: a distinct marionette-eval call is needed here; if
+ ;; included in the previous one issuing the reboot,
+ ;; 'wait-for-service' would apparently run before the system had
+ ;; rebooted (and succeed), which would defeat the test.
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+ marionette)))
+
+ (test-end))))
+
+ (gexp->derivation "anonip-test" test))
+
+(define %test-anonip
+ (system-test
+ (name "anonip")
+ (description "Anonymize logs via Anonip")
+ (value (run-anonip-test))))
+
;;;
;;; Patchwork
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 1/9] doc: Use @table @code for anonip-configuration doc.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
281a4773768a6c271ff464f473fdbc333a58c348.1730464675.git.maxim.cournoyer@gmail.com
* doc/guix.texi (Log Rotation): Use @table @code for anonip-configuration doc.

Change-Id: Ib6f8136715ecabc81d53dc13a14f4b111302c5c9
---
doc/guix.texi | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)

Toggle diff (48 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 5768e195b0..b519b483fe 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20788,30 +20788,30 @@ Log Rotation
The following optional settings may be provided:
-@table @asis
-@item @code{skip-private?}
+@table @code
+@item skip-private?
When @code{#true} do not mask addresses in private ranges.
-@item @code{column}
+@item column
A 1-based indexed column number. Assume IP address is in the specified
column (default is 1).
-@item @code{replacement}
+@item replacement
Replacement string in case address parsing fails, e.g. @code{"0.0.0.0"}.
-@item @code{ipv4mask}
+@item ipv4mask
Number of bits to mask in IPv4 addresses.
-@item @code{ipv6mask}
+@item ipv6mask
Number of bits to mask in IPv6 addresses.
-@item @code{increment}
+@item increment
Increment the IP address by the given number. By default this is zero.
-@item @code{delimiter}
+@item delimiter
Log delimiter string.
-@item @code{regex}
+@item regex
Regular expression for detecting IP addresses. Use this instead of @code{column}.
@end table
@end deftp

base-commit: 4009d1de954d694cb11af391d4113d29c5c1379d
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 2/9] services: anonip: Add 'debug?' configuration field.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
12708a075e13fc3acfb4f2685d561fd9ee481b87.1730464675.git.maxim.cournoyer@gmail.com
* gnu/services/web.scm (<anonip-configuration>) [debug?]: New field.
(anonip-shepherd-service): Honor it.
* doc/guix.texi (Log Rotation) <anonip-configuration>: Document it.

Change-Id: Iaf57b5992808374b069a55c34a9adfdfe52b046c
---
doc/guix.texi | 3 +++
gnu/services/web.scm | 8 ++++++--
2 files changed, 9 insertions(+), 2 deletions(-)

Toggle diff (55 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index b519b483fe..51a543d690 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20789,6 +20789,9 @@ Log Rotation
The following optional settings may be provided:
@table @code
+@item debug?
+Print debug messages when @code{#true}.
+
@item skip-private?
When @code{#true} do not mask addresses in private ranges.
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cc6f4e6d9b..cf3515bf70 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -220,6 +220,7 @@ (define-module (gnu services web)
anonip-configuration-anonip
anonip-configuration-input
anonip-configuration-output
+ anonip-configuration-debug?
anonip-configuration-skip-private?
anonip-configuration-column
anonip-configuration-replacement
@@ -1448,6 +1449,8 @@ (define-record-type* <anonip-configuration>
(default anonip))
(input anonip-configuration-input) ;string
(output anonip-configuration-output) ;string
+ (debug? anonip-configuration-debug? ;boolean
+ (default #f))
(skip-private? anonip-configuration-skip-private? ;boolean
(default #f))
(column anonip-configuration-column ;number
@@ -1503,14 +1506,15 @@ (define (anonip-shepherd-service config)
"/bin/anonip")
(string-append "--input=" #$input)
(string-append "--output=" #$output))
+ (if #$(anonip-configuration-debug? config)
+ '("--debug") (list))
(if #$(anonip-configuration-skip-private? config)
'("--skip-private") (list))
'#$(optional anonip-configuration-column "--column")
'#$(optional anonip-configuration-ipv4mask "--ipv4mask")
'#$(optional anonip-configuration-ipv6mask "--ipv6mask")
'#$(optional anonip-configuration-increment "--increment")
- '#$(optional anonip-configuration-replacement
- "--replacement")
+ '#$(optional anonip-configuration-replacement "--replacement")
'#$(optional anonip-configuration-delimiter "--delimiter")
'#$(optional anonip-configuration-regex "--regex"))
;; Run in a UTF-8 locale
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 3/9] system/vm: Fix virtual-machine bug.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
397067fb8b2b50efef8092d77c93c04e0f28225a.1730464675.git.maxim.cournoyer@gmail.com
The virtual-machine syntax would not accept a single operating-system field,
which was puzzling.

* gnu/system/vm.scm (virtual-machine): Add a pattern matching a single literal
'operating-system' field and value.

Change-Id: If207fd71df3a3f763b2e63229eafa82f63e80773
---
gnu/system/vm.scm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

Toggle diff (23 lines)
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index a2743453e7..4a400056d7 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -422,10 +422,14 @@ (define-record-type* <virtual-machine> %virtual-machine
(default #f)))
(define-syntax virtual-machine
- (syntax-rules ()
+ (syntax-rules (operating-system)
"Declare a virtual machine running the specified OS, with the given
options."
- ((_ os) ;shortcut
+ ((_ (operating-system os))
+ ;; Also accept the long form (virtual-machine (operating-system os)), for
+ ;; correctness.
+ (%virtual-machine (operating-system os)))
+ ((_ os) ;shortcut
(%virtual-machine (operating-system os)))
((_ fields ...)
(%virtual-machine fields ...))))
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 4/9] tests: web: Have the retry-on-error throw on exhausted attempts.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
369fbb0b1eb904bf152356be15441aaed872c44a.1730464675.git.maxim.cournoyer@gmail.com
Previously, it'd simply return the error arguments, which would not always
fail a 'test-assert' test, for example.

* gnu/tests/web.scm (retry-on-error): Re-throw the error when the attempts
have been exhausted.

Change-Id: Ic1468d8ba23f0f78377e90d19bccb2878dc56f0e
---
gnu/tests/web.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (15 lines)
diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index a071e05e1d..5c50322cef 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -83,7 +83,7 @@ (define retry-on-error
return-value)
((#f . error-args)
(if (>= attempt times)
- error-args
+ (apply throw error-args)
(begin
(sleep delay)
(loop (+ 1 attempt)))))))))
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 6/9] build: marionette: Make it possible to reboot VM during tests.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
5d9173bd000c54075dc8b666dc0fcc8aa262f13a.1730464675.git.maxim.cournoyer@gmail.com
* gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
parameter.
* gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.

Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73
---
gnu/build/marionette.scm | 2 +-
gnu/system/vm.scm | 1 -
2 files changed, 1 insertion(+), 2 deletions(-)

Toggle diff (27 lines)
diff --git a/gnu/build/marionette.scm b/gnu/build/marionette.scm
index 0b0a8a70d8..af211eff43 100644
--- a/gnu/build/marionette.scm
+++ b/gnu/build/marionette.scm
@@ -108,7 +108,7 @@ (define* (make-marionette command
(list "-nographic"
"-monitor" (string-append "unix:" socket-directory "/monitor")
"-chardev" (string-append "socket,id=repl,path=" socket-directory
- "/repl")
+ "/repl,reconnect=1")
"-chardev" (string-append "socket,id=qga,server=on,wait=off,path="
socket-directory "/qemu-ga")
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4a400056d7..1e3f72c7b2 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
'("-enable-kvm")
'())
- "-no-reboot"
"-object" "rng-random,filename=/dev/urandom,id=guix-vm-rng"
"-device" "virtio-rng-pci,rng=guix-vm-rng"
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 5/9] services: herd: Export 'eval-there' in API.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
8c8215ff6140a98c412bb2df5faf19767635ea6b.1730464675.git.maxim.cournoyer@gmail.com
* gnu/services/herd.scm (gnu): Export 'eval-there', which is useful to
evaluate Scheme expressions.

Change-Id: Iff1db56e0847fc5886ac9fe594d677b363e7e6e7
---
gnu/services/herd.scm | 1 +
1 file changed, 1 insertion(+)

Toggle diff (14 lines)
diff --git a/gnu/services/herd.scm b/gnu/services/herd.scm
index 4b47acf72a..c185fc2cec 100644
--- a/gnu/services/herd.scm
+++ b/gnu/services/herd.scm
@@ -54,6 +54,7 @@ (define-module (gnu services herd)
with-shepherd-action
current-service
current-services
+ eval-there
unload-services
unload-service
load-services
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 7/9] tests: Add anonip system test.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
f34e7fbf1b537ce923f9be089d9e5e86552f3e63.1730464675.git.maxim.cournoyer@gmail.com
* gnu/tests/web.scm (%test-anonip): New test.
(%anonip-os): New variables.
(run-anonip-test): New procedure.

Change-Id: Ieed210a784dbdeee8a498e74b6c0e31cb72cd9b8
---
gnu/tests/web.scm | 122 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 122 insertions(+)

Toggle diff (156 lines)
diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 5c50322cef..6ef32138ed 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2017, 2018 Clément Lassieur <clement@lassieur.org>
;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby@inria.fr>
;;; Copyright © 2018 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2024 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -33,6 +34,7 @@ (define-module (gnu tests web)
#:use-module (gnu services networking)
#:use-module (gnu services shepherd)
#:use-module (gnu services mail)
+ #:use-module (gnu packages base)
#:use-module (gnu packages databases)
#:use-module (gnu packages guile-xyz)
#:use-module (gnu packages patchutils)
@@ -52,6 +54,7 @@ (define-module (gnu tests web)
%test-php-fpm
%test-hpcguix-web
%test-tailon
+ %test-anonip
%test-patchwork
%test-agate))
@@ -509,6 +512,125 @@ (define %test-tailon
(description "Connect to a running Tailon server.")
(value (run-tailon-test))))
+
+;;;
+;;; Anonip
+;;;
+(define %anonip-os
+ ;; Operating system under test.
+ (simple-operating-system
+ (service anonip-service-type
+ (anonip-configuration
+ (input "/var/run/anonip/access.log")
+ (output "/var/log/anonip/access.log")
+ (debug? #t)))))
+
+(define (run-anonip-test)
+ (define os
+ (marionette-operating-system
+ %anonip-os
+ #:imported-modules '((gnu services herd)
+ (guix combinators))))
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ ;; We are interested in verifying if anonip still launches following a
+ ;; reboot; thus make the base image writable.
+ (volatile? #f)))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (ice-9 match)
+ (srfi srfi-64)
+ (gnu build marionette))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (test-runner-current (system-test-runner #$output))
+ (test-begin "anonip")
+
+ (test-assert "service is running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (wait-for-service 'anonip-/var/log/anonip/access.log))
+ marionette))
+
+ (test-assert "service can be restarted"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (restart-service 'anonip-/var/log/anonip/access.log)
+ (wait-for-service 'anonip-/var/log/anonip/access.log))
+ marionette))
+
+ (test-assert "ip addresses are anonymized"
+ (marionette-eval
+ '(begin
+ (use-modules (ice-9 textual-ports))
+ (call-with-output-file "/var/run/anonip/access.log"
+ (lambda (port)
+ (display "192.168.100.200 - - \
+[30/Oct/2024:14:57:44 +0100] GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)
+ (display "2001:0db8:85a3:0000:0000:8a2e:0370:7334 - - \
+[30/Oct/2024:14:57:44 +0100] \"GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)))
+ (#$retry-on-error
+ (lambda ()
+ (call-with-input-file "/var/log/anonip/access.log"
+ (lambda (port)
+ (let ((content (get-string-all port)))
+ ;; The expected values are taken from anonip's test
+ ;; suite (see its test_module.py file).
+ (or (and (string-contains content "192.168.96.0")
+ (string-contains content "2001:db8:85a0::"))
+ (error "could not find expected anonymized IPs"
+ content))))))
+ #:times 20
+ #:delay 1))
+ marionette))
+
+ (test-assert "service is running after reboot"
+ (begin
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (eval-there '(begin
+ (use-modules (shepherd system))
+ (sync) ;ensure the log is fully written
+ (reboot))))
+ marionette)
+ ;; Note: a distinct marionette-eval call is needed here; if
+ ;; included in the previous one issuing the reboot,
+ ;; 'wait-for-service' would apparently run before the system had
+ ;; rebooted (and succeed), which would defeat the test.
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (wait-for-service 'anonip-/var/log/anonip/access.log))
+ marionette)))
+
+ (test-assert "service can be stopped"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (stop-service 'anonip-/var/log/anonip/access.log))
+ marionette))
+
+ (test-end))))
+
+ (gexp->derivation "anonip-test" test))
+
+(define %test-anonip
+ (system-test
+ (name "anonip")
+ (description "Anonymize logs via Anonip")
+ (value (run-anonip-test))))
+
;;;
;;; Patchwork
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 8/9] tests: web: Add nginx+anonip test.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
6099a6852bb8bdc693b273f3a7129767d8aabffc.1730464675.git.maxim.cournoyer@gmail.com
This test reproduces the problem reported in bug #59181 ("[berlin] web
services fail to start on reboot due to anonip").

* gnu/tests/web.scm (run-webserver-test) <extra-args>: New keyword argument.
(%nginx+anonip-os, nginx-anonip-tests, %test-nginx+anonip): New variables.

Change-Id: I7bbc8fb9f2ab33ce89bf1c0945d7ddbddf26a7ea
---
New commit in v2

gnu/tests/web.scm | 43 ++++++++++++++++++++++++++++++++++++++++---
1 file changed, 40 insertions(+), 3 deletions(-)

Toggle diff (77 lines)
diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 6ef32138ed..33e0a697a8 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -50,6 +50,7 @@ (define-module (gnu tests web)
#:use-module (ice-9 match)
#:export (%test-httpd
%test-nginx
+ %test-nginx+anonip
%test-varnish
%test-php-fpm
%test-hpcguix-web
@@ -91,9 +92,11 @@ (define retry-on-error
(sleep delay)
(loop (+ 1 attempt)))))))))
-(define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080))
- "Run tests in %NGINX-OS, which has nginx running and listening on
-HTTP-PORT."
+(define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080)
+ extra-tests)
+ "Run tests in %NGINX-OS, which has nginx running and listening on HTTP-PORT.
+EXTRA-TESTS should be a sexp of gexp containing extra code to run as part of
+the tests."
(define os
(marionette-operating-system
test-os
@@ -154,6 +157,8 @@ (define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080))
marionette)))
'())
+ #$extra-tests
+
(test-end))))
(gexp->derivation (string-append name "-test") test))
@@ -209,6 +214,38 @@ (define %test-nginx
(value (run-webserver-test name %nginx-os
#:log-file "/var/log/nginx/access.log"))))
+(define %nginx+anonip-os
+ (simple-operating-system
+ (service dhcp-client-service-type)
+ (service anonip-service-type
+ (anonip-configuration
+ (input "/var/run/anonip/access.log")
+ (output "/var/log/anonip/access.log")
+ (debug? #t)))
+ (service nginx-service-type
+ (nginx-configuration
+ (log-directory "/var/run/anonip/")
+ (server-blocks %nginx-servers)
+ (shepherd-requirement '(anonip-/var/log/anonip/access.log))))
+ (simple-service 'make-http-root activation-service-type
+ %make-http-root)))
+
+(define nginx-anonip-tests
+ #~(test-assert "anonip service is running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (wait-for-service 'anonip-/var/log/anonip/access.log))
+ marionette)))
+
+(define %test-nginx+anonip
+ (system-test
+ (name "nginx+anonip")
+ (description "Run a NGINX server with logs anonymized by Anonip")
+ (value (run-webserver-test "nginx" %nginx+anonip-os
+ #:log-file "/var/log/anonip/access.log"
+ #:extra-tests nginx-anonip-tests))))
+
;;;
;;; Varnish
--
2.46.0
M
M
Maxim Cournoyer wrote on 1 Nov 13:39 +0100
[PATCH v2 9/9] services: web: Fix race between nginx activation and anonip.
(address . 74151@debbugs.gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
1d6c7e71f2a80a38f801965b7fd27f0b982016ec.1730464675.git.maxim.cournoyer@gmail.com
* gnu/services/web.scm (anonip-shepherd-service): Recreate the input file when
it's not a FIFO.

Change-Id: I8ba87f9fc48ecfd515e34bdee9e2949a2a559f9c
---
New commit in v2

gnu/services/web.scm | 64 ++++++++++++++++++++------------------------
1 file changed, 29 insertions(+), 35 deletions(-)

Toggle diff (77 lines)
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cf3515bf70..4cf7c68997 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -1499,41 +1499,35 @@ (define (anonip-shepherd-service config)
"Anonimyze the given log file location with anonip.")
(start
#~(lambda ()
- (define (spawn)
- (fork+exec-command
- (append
- (list #$(file-append (anonip-configuration-anonip config)
- "/bin/anonip")
- (string-append "--input=" #$input)
- (string-append "--output=" #$output))
- (if #$(anonip-configuration-debug? config)
- '("--debug") (list))
- (if #$(anonip-configuration-skip-private? config)
- '("--skip-private") (list))
- '#$(optional anonip-configuration-column "--column")
- '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
- '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
- '#$(optional anonip-configuration-increment "--increment")
- '#$(optional anonip-configuration-replacement "--replacement")
- '#$(optional anonip-configuration-delimiter "--delimiter")
- '#$(optional anonip-configuration-regex "--regex"))
- ;; Run in a UTF-8 locale
- #:environment-variables
- (list (string-append "GUIX_LOCPATH="
- #$(libc-utf8-locales-for-target)
- "/lib/locale")
- "LC_ALL=en_US.utf8")))
-
- (let ((stat (stat #$input #f)))
- (cond ((not stat)
- (mknod #$input 'fifo #o600 0)
- (spawn))
- ((eq? 'fifo (stat:type stat))
- (spawn))
- (else
- (format #t "'~a' is not a FIFO; bailing out~%"
- #$input)
- #f)))))
+ ;; Always attempt to recreate the named pipe, as activation scripts
+ ;; such as that of nginx may have created plain files in its place
+ ;; (see: https://issues.guix.gnu.org/59181).
+ (false-if-exception (delete-file #$input))
+ (mknod #$input 'fifo #o600 0)
+
+ (fork+exec-command
+ (append
+ (list #$(file-append (anonip-configuration-anonip config)
+ "/bin/anonip")
+ (string-append "--input=" #$input)
+ (string-append "--output=" #$output))
+ (if #$(anonip-configuration-debug? config)
+ '("--debug") (list))
+ (if #$(anonip-configuration-skip-private? config)
+ '("--skip-private") (list))
+ '#$(optional anonip-configuration-column "--column")
+ '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
+ '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
+ '#$(optional anonip-configuration-increment "--increment")
+ '#$(optional anonip-configuration-replacement "--replacement")
+ '#$(optional anonip-configuration-delimiter "--delimiter")
+ '#$(optional anonip-configuration-regex "--regex"))
+ ;; Run in a UTF-8 locale
+ #:environment-variables
+ (list (string-append "GUIX_LOCPATH="
+ #$(libc-utf8-locales-for-target)
+ "/lib/locale")
+ "LC_ALL=en_US.utf8"))))
(stop #~(make-kill-destructor))))))
(define anonip-service-type
--
2.46.0
L
L
Ludovic Courtès wrote on 20 Nov 10:29 +0100
Re: [bug#74151] [PATCH v2 6/9] build: marionette: Make it possible to reboot VM during tests.
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 74151@debbugs.gnu.org)
874j422qz3.fsf@gnu.org
Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (6 lines)
> * gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
> parameter.
> * gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.
>
> Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73

[...]

Toggle quote (7 lines)
> +++ b/gnu/system/vm.scm
> @@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
> '("-enable-kvm")
> '())
>
> - "-no-reboot"

I believe the reason we had ‘-no-reboot’ is because otherwise QEMU would
restart the VM when we halt it (?). I’m not sure actually what the
exact scenario was.

Anyway, if you confirm that other system tests are unaffected by this
change, this is probably good.

Ludo’.
L
L
Ludovic Courtès wrote on 20 Nov 10:31 +0100
Re: [bug#74151] [PATCH 0/7] Add anonip system test.
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 74151@debbugs.gnu.org)
87zflu1cbg.fsf@gnu.org
Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (10 lines)
> This was developed as part of investigating the source of bug #59181 ([berlin]
> web services fail to start on reboot due to anonip).
>
> There is no issue having anonip run and the system reboot (as should be
> expected as /var/run is supposed to be cleared thus the named pipes should be
> recreated every boot). This points to a race with nginx that would "win"
> creating its log files under /var/run/anonip/* before the anonip service does.
>
> It'll need further investigating in that direction.

This is nice work, LGTM!

This did not elucidate the cause of #59181, right?

Thanks,
Ludo’.
M
M
Maxim Cournoyer wrote on 24 Nov 13:07 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 74151@debbugs.gnu.org)
87y118q1ht.fsf@gmail.com
Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (18 lines)
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> This was developed as part of investigating the source of bug #59181 ([berlin]
>> web services fail to start on reboot due to anonip).
>>
>> There is no issue having anonip run and the system reboot (as should be
>> expected as /var/run is supposed to be cleared thus the named pipes should be
>> recreated every boot). This points to a race with nginx that would "win"
>> creating its log files under /var/run/anonip/* before the anonip service does.
>>
>> It'll need further investigating in that direction.
>
> This is nice work, LGTM!
>
> This did not elucidate the cause of #59181, right?

It did! The cause was the activation snippet of nginx, which runs the
nginx check command, which creates every file involved to check if it
has sufficient permissions.

The files would then exist before anonip had a chance to create them.

The fix has already been merged, so a reboot following a recent
reconfigure on berlin should show all our services up and running.

I'll run the system tests again and push it this doesn't break any.

--
Thanks,
Maxim
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74151@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74151
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch