Prometheus (and Alertmanager)

OpenSubmitted by Christopher Baines.
Details
2 participants
  • Ludovic Courtès
  • Christopher Baines
Owner
unassigned
Severity
normal
C
C
Christopher Baines wrote on 20 Apr 2020 23:11
Download
(address . guix-patches@gnu.org)
87v9lt50d4.fsf@cbaines.net
Download
I've had a go at packaging Prometheus and Alertmanager, as well aswriting Guix services for both of them.
The packages build, but still need quite a bit of work. They're bothwritten in Go, so the Git repositories not only include the relevantsource code, but the source code of the entire dependency tree (hencethe packages have no inputs). That'll need addressing.
I've tested the Prometheus service, and it seems to work, although thereisn't any record types for the Prometheus configuration yet. I haven'tyet tested the Alertmanager service and the system test doesn't pass, Ibelieve more configuration is required than Prometheus.
Having a Prometheus service would be a good complement for the existingPrometheus Node exporter service in Guix.
I'll send the patches following this email.
Thanks,
Chris
Download
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl6eD+dfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9XdmgBAArT9rLKl4NX8hZCBq4mD8/S4IhbNi4cBswFedk+sJetXvkRJcbd4Gk46uNWEQawFBugmj6P30hhd53l6iIAu5GalDrt1VvX9UyQ+2YG0H+afSttAe8k0IgFKaIBVCWBbNefAJHbClTw/VpmTVYG/hrS/Pgaq6wlkmvtFTL5APkDkOAV1IHgXsATANj5WlBbVu5Yf5vBg+tBXN809iRlDwBiuh4pCx41c92ccg8Fy5qwLHx38tAgqJVb48V9tll+e6vHHLddGGYx0Bo/aJICzNsa/Y3ygbsnIAqYuZEzE7uPjjjdCGcFkyWC7IN96nFq2ls4dtWzX8RzMuzsPHRmWoyKI8lPgl5JtEeyU71OjiWBQgReIM8ITqs+BV6uR7eGbFAsIPEaDY94ThbxFCcCd3wavlROq0aWIkU4R0Mp6BaySr1olehY2YJte6MBONPaijN8fAebmtqC8Sr99/x8EuAcMHpYuo03LaazKfP9SAr1ZaPwn4jsfkvZhHQizF5IXH/fbUF/r765lPmic94BXi5ryTx9Oq/iAebV4vWFz5wacrWxMeNxEtcVV1hrMvU2KMqm5wQ4/0IpiutmwYa8K3w52AcrUYPKgFkSVkywe38hkRPl6mFO/WCRpZ+X69CkRHR37++OyGiLbbsOtT1w++OEAdyD9sQRtiSkzMVa0EeOE==qpP+-----END PGP SIGNATURE-----
C
C
Christopher Baines wrote on 20 Apr 2020 23:15
Download
(address . control@debbugs.gnu.org)
87tv1d505n.fsf@cbaines.net
Download
tag 40738 moreinfothanks
Download
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl6eEPRfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9XeIyBAAjqhb3FOowSE1QVHAA+mDNoqy0lUCu8aG5YP86zF5YApkY9be3kTwIOA4mQ46/at/HGUGMxEUK/pRq2R1i6JblzWvEUTtcvAdYA5ZE/k8988vQV5hMz/C+OB7qFDB43cHOc8nsy4hyL1GhFnKTjpJrcuiv99YA6JlPgGg5SW20fl6AnBiSP9AObUe/BxQWf/NTlfeUo2sDf/XOJcq9DBlhMGCY9FTJv/iB6GejocKKlw+qgVPC92fFK9K/pmvLfXnpcmO7LNe/7x/oCNEr7U4fYAPmS3kDN0QVtthJxcXI5UtQBc1UohcCRVAAT+CicUh3BudkdDQG1k1izMXl2dzYtv4P7OLjlmNU9zvk0uCG2u6+c5/0Mx8WbuDbbj6MZjS6yBFMx8p4Es4pKafbaQdVQdR2NPkLRkeemv2JJnDth3VuHjWFAUCplHuSOKkL8N+gOQ9wvSVF+y4BQxDDXGlQ0CjbUMeHxxAtoeaFAX6vRyj5dW0tGYevxKoYk4ze8Cca+VlZ3tiyOD+a7hXkWSJyRteBp7XPm9AHjtQGEmnAM3zfodvNE23h65XNi6AEaYM9iJCNBa/UFgcjVU/hOE3UsFnLNAt0hfsTAVY3XhmKW3olT038/JY7k2fHR2ZSbP9oYImee7UtFFSpiSxYNsf47L1atGfyIUQURZxs5kVTvo==M7+a-----END PGP SIGNATURE-----
C
C
Christopher Baines wrote on 20 Apr 2020 23:17
Download
(address . 40738@debbugs.gnu.org)
20200420211743.23476-1-mail@cbaines.net
Download
* gnu/packages/monitoring.scm (prometheus): New variable.--- gnu/packages/monitoring.scm | 49 +++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
Toggle diff (62 lines)diff --git a/gnu/packages/monitoring.scm b/gnu/packages/monitoring.scmindex 8da31d6a84..345c2c16c5 100644--- a/gnu/packages/monitoring.scm+++ b/gnu/packages/monitoring.scm@@ -422,6 +422,55 @@ written in Go with pluggable metric collectors.") (home-page "https://github.com/prometheus/node_exporter") (license license:asl2.0))) +(define-public prometheus+ (package+ (name "prometheus")+ (version "2.17.1")+ (source (origin+ (method git-fetch)+ (uri (git-reference+ (url "https://github.com/prometheus/prometheus.git")+ (commit (string-append "v" version))))+ (file-name (git-file-name name version))+ (sha256+ (base32+ "1r7zpq6647lrm7cmid6nnf2xnljqh1i9g0fxvs0qrfd2sxxgj0c7"))))+ (build-system go-build-system)+ (arguments+ '(#:unpack-path "github.com/prometheus/prometheus"+ #:import-path "github.com/prometheus/prometheus/cmd/prometheus"+ #:install-source? #f+ #:phases+ (modify-phases %standard-phases+ (add-after 'unpack 'patch+ (lambda* (#:key outputs #:allow-other-keys)+ (let* ((out (assoc-ref outputs "out"))+ (assets-prefix+ (string-append out "/var/lib/prometheus/assets")))+ (substitute* "src/github.com/prometheus/prometheus/web/ui/ui.go"+ (("var assetsPrefix string")+ (string-append "var assetsPrefix string = \""+ assets-prefix+ "\""))))+ #t))+ (add-after 'install 'install-assets+ (lambda* (#:key outputs #:allow-other-keys)+ (let* ((out (assoc-ref outputs "out"))+ (assets-prefix+ (string-append out "/var/lib/prometheus/assets")))+ (for-each (lambda (directory)+ (copy-recursively+ (string-append "src/github.com/prometheus/prometheus"+ "/web/ui/" directory)+ (string-append assets-prefix+ "/" directory)))+ '("static" "templates")))+ #t)))))+ (home-page "https://prometheus.io/")+ (synopsis "")+ (description "")+ (license "")))+ (define-public fswatch (package (name "fswatch")-- 2.26.0
C
C
Christopher Baines wrote on 20 Apr 2020 23:17
Download
[PATCH 2/4] services: Add a Prometheus service.
(address . 40738@debbugs.gnu.org)
20200420211743.23476-2-mail@cbaines.net
Download
--- gnu/services/monitoring.scm | 82 +++++++++++++++++++++++++++++++++++++ gnu/tests/monitoring.scm | 73 ++++++++++++++++++++++++++++++++- 2 files changed, 154 insertions(+), 1 deletion(-)
Toggle diff (188 lines)diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scmindex 511f4fb2fe..a37dfd80d8 100644--- a/gnu/services/monitoring.scm+++ b/gnu/services/monitoring.scm@@ -40,6 +40,17 @@ darkstat-service-type prometheus-node-exporter-service-type + prometheus-service-type+ <prometheus-configuration>+ prometheus-configuration+ prometheus-configuration-package+ prometheus-configuration-user+ prometheus-configuration-group+ prometheus-configuration-config-file+ prometheus-configuration-web-listen-address+ prometheus-configuration-storage-tsdb-path+ prometheus-configuration-extra-options+ zabbix-server-configuration zabbix-server-service-type zabbix-agent-configuration@@ -110,6 +121,77 @@ HTTP.") (service-extension shepherd-root-service-type (compose list darkstat-shepherd-service)))))) + +;;;+;;; Prometheus+;;;++(define-record-type* <prometheus-configuration>+ prometheus-configuration+ make-prometheus-configuration+ prometheus-configuration?+ (package prometheus-configuration-package+ (default prometheus))+ (user prometheus-configuration-user+ (default "prometheus"))+ (group prometheusconfiguration-group+ (default "prometheus"))+ (config-file prometheus-config-file+ (default (plain-file "prometheus.yml" "")))+ (web-listen-address prometheus-web-listen-address+ (default "0.0.0.0:9090"))+ (storage-tsdb-path prometheus-storage-tsdb-path+ (default "/var/lib/prometheus/data/"))+ (extra-options prometheus-configuration-extra-options+ (default '())))++(define prometheus-shepherd-service+ (match-lambda+ (($ <prometheus-configuration> package user group+ config-file web-listen-address+ storage-tsdb-path extra-options)+ (shepherd-service+ (documentation "Prometheus monitoring system and time series database.")+ (provision '(prometheus))+ (requirement '(networking))+ (start #~(make-forkexec-constructor+ (list #$(file-append package "/bin/prometheus")+ "--config.file" #$config-file+ "--web.listen-address" #$web-listen-address+ "--storage.tsdb.path" #$storage-tsdb-path+ #$@extra-options)+ #:user #$user+ #:group #$group+ #:log-file "/var/log/prometheus.log"))+ (stop #~(make-kill-destructor))))))++(define (prometheus-account config)+ (match-record config <prometheus-configuration>+ (user group)+ (list (user-group+ (name group)+ (system? #t))+ (user-account+ (name user)+ (group group)+ (system? #t)+ (comment "Prometheus user")+ (home-directory "/var/lib/prometheus")+ (shell (file-append shadow "/sbin/nologin"))))))++(define prometheus-service-type+ (service-type+ (name 'prometheus)+ (description+ "Run @command{prometheus} to scrape targets for mertrics and provide the+web interface.")+ (extensions+ (list (service-extension shepherd-root-service-type+ (compose list prometheus-shepherd-service))+ (service-extension account-service-type+ prometheus-account)))+ (default-value (prometheus-configuration))))+ (define-record-type* <prometheus-node-exporter-configuration> prometheus-node-exporter-configuration make-prometheus-node-exporter-configurationdiff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scmindex 732fbc54d7..e8c0847229 100644--- a/gnu/tests/monitoring.scm+++ b/gnu/tests/monitoring.scm@@ -31,9 +31,80 @@ #:use-module (gnu system) #:use-module (gnu tests) #:use-module (guix gexp)- #:export (%test-prometheus-node-exporter+ #:export (%test-prometheus+ %test-prometheus-node-exporter %test-zabbix)) + +;;;+;;; Prometheus+;;;++(define* (run-prometheus-test name test-os)+ "Run tests in %TEST-OS, which has Prometheus running."+ (define os+ (marionette-operating-system+ test-os+ #:imported-modules '((gnu services herd))))++ (define vm+ (virtual-machine+ (operating-system os)+ (port-forwardings '((8080 . 9090)))))++ (define test+ (with-imported-modules '((gnu build marionette))+ #~(begin+ (use-modules (srfi srfi-11)+ (srfi srfi-64)+ (gnu build marionette)+ (web client)+ (web response))++ (define marionette+ (make-marionette (list #$vm)))++ (mkdir #$output)+ (chdir #$output)++ (test-begin #$name)++ (test-assert "prometheus running"+ (marionette-eval+ '(begin+ (use-modules (gnu services herd))+ (match (start-service 'prometheus)+ (#f #f)+ (('service response-parts ...)+ (match (assq-ref response-parts 'running)+ ((pid) (number? pid))))))+ marionette))++ (test-equal "prometheus healthy"+ 200+ (begin+ (wait-for-tcp-port 9090 marionette)+ (let-values (((response text)+ (http-get "http://localhost:8080/-/healthy")))+ (response-code response))))++ (test-end)+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))++ (gexp->derivation (string-append name "-test") test))++(define %prometheus-test-os+ (simple-operating-system+ (service dhcp-client-service-type)+ (service prometheus-service-type)))++(define %test-prometheus+ (system-test+ (name "prometheus")+ (description "Connect to a running Prometheus service.")+ (value (run-prometheus-test name+ %prometheus-test-os))))+ ;;; ;;; Prometheus Node Exporter-- 2.26.0
C
C
Christopher Baines wrote on 20 Apr 2020 23:17
Download
[PATCH 3/4] gnu: Add alertmanager.
(address . 40738@debbugs.gnu.org)
20200420211743.23476-3-mail@cbaines.net
Download
* gnu/packages/monitoring.scm (alertmanager): New variable.--- gnu/packages/monitoring.scm | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
Toggle diff (36 lines)diff --git a/gnu/packages/monitoring.scm b/gnu/packages/monitoring.scmindex 345c2c16c5..e0e345d259 100644--- a/gnu/packages/monitoring.scm+++ b/gnu/packages/monitoring.scm@@ -471,6 +471,29 @@ written in Go with pluggable metric collectors.") (description "") (license ""))) +(define-public alertmanager+ (package+ (name "alertmanager")+ (version "0.20.0")+ (source (origin+ (method git-fetch)+ (uri (git-reference+ (url "https://github.com/prometheus/alertmanager.git")+ (commit (string-append "v" version))))+ (file-name (git-file-name name version))+ (sha256+ (base32+ "1bq6vbpy25k7apvs2ga3fbp1cbnv9j0y1g1khvz2qgz6a2zvhgg3"))))+ (build-system go-build-system)+ (arguments+ '(#:unpack-path "github.com/prometheus/alertmanager"+ #:import-path "github.com/prometheus/alertmanager/cmd/alertmanager"+ #:install-source? #f))+ (home-page "https://prometheus.io/")+ (synopsis "")+ (description "")+ (license "")))+ (define-public fswatch (package (name "fswatch")-- 2.26.0
C
C
Christopher Baines wrote on 20 Apr 2020 23:17
Download
[PATCH 4/4] services: Add a service for Alertmanager.
(address . 40738@debbugs.gnu.org)
20200420211743.23476-4-mail@cbaines.net
Download
--- gnu/services/monitoring.scm | 82 +++++++++++++++++++++++++++++++++++++ gnu/tests/monitoring.scm | 71 ++++++++++++++++++++++++++++++++ 2 files changed, 153 insertions(+)
Toggle diff (191 lines)diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scmindex a37dfd80d8..50a4b7302c 100644--- a/gnu/services/monitoring.scm+++ b/gnu/services/monitoring.scm@@ -51,6 +51,17 @@ prometheus-configuration-storage-tsdb-path prometheus-configuration-extra-options + alertmanager-service-type+ <alertmanager-configuration>+ alertmanager-configuration+ alertmanager-configuration-package+ alertmanager-configuration-user+ alertmanager-configuration-group+ alertmanager-configuration-config-file+ alertmanager-configuration-web-listen-address+ alertmanager-configuration-storage-path+ alertmanager-configuration-extra-options+ zabbix-server-configuration zabbix-server-service-type zabbix-agent-configuration@@ -226,6 +237,77 @@ prometheus.") (compose list prometheus-node-exporter-shepherd-service)))) (default-value (prometheus-node-exporter-configuration)))) + +;;;+;;; Alertmanager+;;;++(define-record-type* <alertmanager-configuration>+ alertmanager-configuration+ make-alertmanager-configuration+ alertmanager-configuration?+ (package alertmanager-configuration-package+ (default alertmanager))+ (user alertmanager-configuration-user+ (default "alertmanager"))+ (group alertmanagerconfiguration-group+ (default "alertmanager"))+ (config-file alertmanager-config-file+ (default (plain-file "alertmanager.yml" "")))+ (web-listen-address alertmanager-web-listen-address+ (default ":9093"))+ (storage-tsdb-path alertmanager-storage-tsdb-path+ (default "/var/lib/alertmanager/data/"))+ (extra-options alertmanager-configuration-extra-options+ (default '())))++(define alertmanager-shepherd-service+ (match-lambda+ (($ <alertmanager-configuration> package user group+ config-file web-listen-address+ storage-tsdb-path extra-options)+ (shepherd-service+ (documentation "Alertmanager monitoring system and time series database.")+ (provision '(alertmanager))+ (requirement '(networking))+ (start #~(make-forkexec-constructor+ (list #$(file-append package "/bin/alertmanager")+ "--config.file" #$config-file+ "--web.listen-address" #$web-listen-address+ "--storage.path" #$storage-tsdb-path+ #$@extra-options)+ #:user #$user+ #:group #$group+ #:log-file "/var/log/alertmanager.log"))+ (stop #~(make-kill-destructor))))))++(define (alertmanager-account config)+ (match-record config <alertmanager-configuration>+ (user group)+ (list (user-group+ (name group)+ (system? #t))+ (user-account+ (name user)+ (group group)+ (system? #t)+ (comment "Alertmanager user")+ (home-directory "/var/lib/alertmanager")+ (shell (file-append shadow "/sbin/nologin"))))))++(define alertmanager-service-type+ (service-type+ (name 'alertmanager)+ (description+ "Run @command{alertmanager} to scrape targets for mertrics and provide the+web interface.")+ (extensions+ (list (service-extension shepherd-root-service-type+ (compose list alertmanager-shepherd-service))+ (service-extension account-service-type+ alertmanager-account)))+ (default-value (alertmanager-configuration))))+ ;;; ;;; Zabbix serverdiff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scmindex e8c0847229..b77b654abc 100644--- a/gnu/tests/monitoring.scm+++ b/gnu/tests/monitoring.scm@@ -33,6 +33,7 @@ #:use-module (guix gexp) #:export (%test-prometheus %test-prometheus-node-exporter+ %test-alertmanager %test-zabbix)) @@ -176,6 +177,76 @@ (value (run-prometheus-node-exporter-server-test name %prometheus-node-exporter-os)))) + +;;;+;;; Alertmanager+;;;++(define* (run-alertmanager-test name test-os)+ "Run tests in %TEST-OS, which has Alertmanager running."+ (define os+ (marionette-operating-system+ test-os+ #:imported-modules '((gnu services herd))))++ (define vm+ (virtual-machine+ (operating-system os)+ (port-forwardings '((8080 . 9093)))))++ (define test+ (with-imported-modules '((gnu build marionette))+ #~(begin+ (use-modules (srfi srfi-11)+ (srfi srfi-64)+ (gnu build marionette)+ (web client)+ (web response))++ (define marionette+ (make-marionette (list #$vm)))++ (mkdir #$output)+ (chdir #$output)++ (test-begin #$name)++ (test-assert "alertmanager running"+ (marionette-eval+ '(begin+ (use-modules (gnu services herd))+ (match (start-service 'alertmanager)+ (#f #f)+ (('service response-parts ...)+ (match (assq-ref response-parts 'running)+ ((pid) (number? pid))))))+ marionette))++ (test-equal "alertmanager healthy"+ 200+ (begin+ (wait-for-tcp-port 9090 marionette)+ (let-values (((response text)+ (http-get "http://localhost:8080/-/healthy")))+ (response-code response))))++ (test-end)+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))++ (gexp->derivation (string-append name "-test") test))++(define %alertmanager-test-os+ (simple-operating-system+ (service dhcp-client-service-type)+ (service alertmanager-service-type)))++(define %test-alertmanager+ (system-test+ (name "alertmanager")+ (description "Connect to a running Alertmanager service.")+ (value (run-alertmanager-test name+ %alertmanager-test-os))))+ ;;; ;;; Zabbix-- 2.26.0
L
L
Ludovic Courtès wrote on 3 May 2020 12:56
Download
Re: [bug#40738] Prometheus (and Alertmanager)
(name . Christopher Baines)(address . mail@cbaines.net)(address . 40738@debbugs.gnu.org)
87d07l1e3w.fsf@gnu.org
Download
Hello!
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (3 lines)> I've had a go at packaging Prometheus and Alertmanager, as well as> writing Guix services for both of them.
Nice! I’m not an expert, but it sounds like it could advantageouslyreplace Zabbix on berlin.
Toggle quote (5 lines)> The packages build, but still need quite a bit of work. They're both> written in Go, so the Git repositories not only include the relevant> source code, but the source code of the entire dependency tree (hence> the packages have no inputs). That'll need addressing.
Would the gopkg importer athttps://lists.gnu.org/archive/html/guix-devel/2018-04/msg00310.html beof any help? (I think I posted an updated version later on but I can’tfind it.)
Toggle quote (8 lines)> I've tested the Prometheus service, and it seems to work, although there> isn't any record types for the Prometheus configuration yet. I haven't> yet tested the Alertmanager service and the system test doesn't pass, I> believe more configuration is required than Prometheus.>> Having a Prometheus service would be a good complement for the existing> Prometheus Node exporter service in Guix.
Yup!
Thanks,Ludo’.
C
C
Christopher Baines wrote on 3 May 2020 13:17
Download
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 40738@debbugs.gnu.org)
874ksxmfmk.fsf@cbaines.net
Download
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (10 lines)>> The packages build, but still need quite a bit of work. They're both>> written in Go, so the Git repositories not only include the relevant>> source code, but the source code of the entire dependency tree (hence>> the packages have no inputs). That'll need addressing.>> Would the gopkg importer at> <https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00310.html> be> of any help? (I think I posted an updated version later on but I can’t> find it.)
Yeah, I think getting a working importer is going to be the way toaddress this.
Download
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl6uqGNfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9Xeu+w/+PYEZRUQ8vhNNxb9xsVpH10ZnjLvnU8W7hCgPaMBXYyvxVXa99VWl811TW7ZVjX1smgNEoUb3Yq81o3ib7DkkkH8n2T+GSnQNhw2n9LCo5hRa+cWB0W0qY23NTyu7alCiPpZOUI1aMbidpHULZlcyTVGMDda8croyy0Md+u0Yzh0Rr9SZ0G8yJGx7w/m4jNOCm+K6JYVSSb+1NyNXENxZ3/4DbSZLNKjWnEOxQ4zNCNLLTaIxeralh1OGfxmEeFAa7ZwNcA8bMA3Irvoxi+4lpeAL3G9I4QO7EEiMGC/fc+A2MM6PlcnkT/1ErdoRK0vPJGoSI7GvmQzmbx06d/WY1HfEvDyDyM9t87YZGWkCJ6A7DmsCd87McnfDJagWBIYx9PrdqHDkcs4pbjTQfpicGjjy59pEoab3AFlXLjJC4LtwJ5dqssZDC45erxjENCaSe51BGoXsU+1Iawi4VPuSyIePraV+WrqNeWrVUhGA7/HBQ98OiNH7I3wg3+7KtJM6To+FpW7HDuxYhqzXVAi4cEOCmHngza9eKNxqZ6+xS+yTxr2ySUgMWWzEy5AQcEobG3IEenBI0ckUt70+iQaonipaSIAv46tmO6S04Pto9N7ihDoeIcsYDc9TzmrfCLWL7e0k5L/9Lkk839CeNIJ03JrofIRnpYLp8qqoU6ewMKc==cbv7-----END PGP SIGNATURE-----
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 40738@debbugs.gnu.org