Prometheus (and Alertmanager)

  • Open
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Christopher Baines
Owner
unassigned
Submitted by
Christopher Baines
Severity
normal
C
C
Christopher Baines wrote on 20 Apr 2020 23:11
(address . guix-patches@gnu.org)
87v9lt50d4.fsf@cbaines.net
I've had a go at packaging Prometheus and Alertmanager, as well as
writing Guix services for both of them.

The packages build, but still need quite a bit of work. They're both
written in Go, so the Git repositories not only include the relevant
source code, but the source code of the entire dependency tree (hence
the packages have no inputs). That'll need addressing.

I've tested the Prometheus service, and it seems to work, although there
isn't any record types for the Prometheus configuration yet. I haven't
yet tested the Alertmanager service and the system test doesn't pass, I
believe more configuration is required than Prometheus.

Having a Prometheus service would be a good complement for the existing
Prometheus Node exporter service in Guix.

I'll send the patches following this email.

Thanks,

Chris
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl6eD+dfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9XdmgBAArT9rLKl4NX8hZCBq4mD8/S4IhbNi4cBswFedk+sJetXvkRJcbd4Gk46u
NWEQawFBugmj6P30hhd53l6iIAu5GalDrt1VvX9UyQ+2YG0H+afSttAe8k0IgFKa
IBVCWBbNefAJHbClTw/VpmTVYG/hrS/Pgaq6wlkmvtFTL5APkDkOAV1IHgXsATAN
j5WlBbVu5Yf5vBg+tBXN809iRlDwBiuh4pCx41c92ccg8Fy5qwLHx38tAgqJVb48
V9tll+e6vHHLddGGYx0Bo/aJICzNsa/Y3ygbsnIAqYuZEzE7uPjjjdCGcFkyWC7I
N96nFq2ls4dtWzX8RzMuzsPHRmWoyKI8lPgl5JtEeyU71OjiWBQgReIM8ITqs+BV
6uR7eGbFAsIPEaDY94ThbxFCcCd3wavlROq0aWIkU4R0Mp6BaySr1olehY2YJte6
MBONPaijN8fAebmtqC8Sr99/x8EuAcMHpYuo03LaazKfP9SAr1ZaPwn4jsfkvZhH
QizF5IXH/fbUF/r765lPmic94BXi5ryTx9Oq/iAebV4vWFz5wacrWxMeNxEtcVV1
hrMvU2KMqm5wQ4/0IpiutmwYa8K3w52AcrUYPKgFkSVkywe38hkRPl6mFO/WCRpZ
+X69CkRHR37++OyGiLbbsOtT1w++OEAdyD9sQRtiSkzMVa0EeOE=
=qpP+
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 20 Apr 2020 23:15
(address . control@debbugs.gnu.org)
87tv1d505n.fsf@cbaines.net
tag 40738 moreinfo
thanks
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl6eEPRfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9XeIyBAAjqhb3FOowSE1QVHAA+mDNoqy0lUCu8aG5YP86zF5YApkY9be3kTwIOA4
mQ46/at/HGUGMxEUK/pRq2R1i6JblzWvEUTtcvAdYA5ZE/k8988vQV5hMz/C+OB7
qFDB43cHOc8nsy4hyL1GhFnKTjpJrcuiv99YA6JlPgGg5SW20fl6AnBiSP9AObUe
/BxQWf/NTlfeUo2sDf/XOJcq9DBlhMGCY9FTJv/iB6GejocKKlw+qgVPC92fFK9K
/pmvLfXnpcmO7LNe/7x/oCNEr7U4fYAPmS3kDN0QVtthJxcXI5UtQBc1UohcCRVA
AT+CicUh3BudkdDQG1k1izMXl2dzYtv4P7OLjlmNU9zvk0uCG2u6+c5/0Mx8WbuD
bbj6MZjS6yBFMx8p4Es4pKafbaQdVQdR2NPkLRkeemv2JJnDth3VuHjWFAUCplHu
SOKkL8N+gOQ9wvSVF+y4BQxDDXGlQ0CjbUMeHxxAtoeaFAX6vRyj5dW0tGYevxKo
Yk4ze8Cca+VlZ3tiyOD+a7hXkWSJyRteBp7XPm9AHjtQGEmnAM3zfodvNE23h65X
Ni6AEaYM9iJCNBa/UFgcjVU/hOE3UsFnLNAt0hfsTAVY3XhmKW3olT038/JY7k2f
HR2ZSbP9oYImee7UtFFSpiSxYNsf47L1atGfyIUQURZxs5kVTvo=
=M7+a
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 20 Apr 2020 23:17
(address . 40738@debbugs.gnu.org)
20200420211743.23476-1-mail@cbaines.net
* gnu/packages/monitoring.scm (prometheus): New variable.
---
gnu/packages/monitoring.scm | 49 +++++++++++++++++++++++++++++++++++++
1 file changed, 49 insertions(+)

Toggle diff (62 lines)
diff --git a/gnu/packages/monitoring.scm b/gnu/packages/monitoring.scm
index 8da31d6a84..345c2c16c5 100644
--- a/gnu/packages/monitoring.scm
+++ b/gnu/packages/monitoring.scm
@@ -422,6 +422,55 @@ written in Go with pluggable metric collectors.")
(home-page "https://github.com/prometheus/node_exporter")
(license license:asl2.0)))
+(define-public prometheus
+ (package
+ (name "prometheus")
+ (version "2.17.1")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/prometheus/prometheus.git")
+ (commit (string-append "v" version))))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ "1r7zpq6647lrm7cmid6nnf2xnljqh1i9g0fxvs0qrfd2sxxgj0c7"))))
+ (build-system go-build-system)
+ (arguments
+ '(#:unpack-path "github.com/prometheus/prometheus"
+ #:import-path "github.com/prometheus/prometheus/cmd/prometheus"
+ #:install-source? #f
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'patch
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (assets-prefix
+ (string-append out "/var/lib/prometheus/assets")))
+ (substitute* "src/github.com/prometheus/prometheus/web/ui/ui.go"
+ (("var assetsPrefix string")
+ (string-append "var assetsPrefix string = \""
+ assets-prefix
+ "\""))))
+ #t))
+ (add-after 'install 'install-assets
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (assets-prefix
+ (string-append out "/var/lib/prometheus/assets")))
+ (for-each (lambda (directory)
+ (copy-recursively
+ (string-append "src/github.com/prometheus/prometheus"
+ "/web/ui/" directory)
+ (string-append assets-prefix
+ "/" directory)))
+ '("static" "templates")))
+ #t)))))
+ (home-page "https://prometheus.io/")
+ (synopsis "")
+ (description "")
+ (license "")))
+
(define-public fswatch
(package
(name "fswatch")
--
2.26.0
C
C
Christopher Baines wrote on 20 Apr 2020 23:17
[PATCH 2/4] services: Add a Prometheus service.
(address . 40738@debbugs.gnu.org)
20200420211743.23476-2-mail@cbaines.net
---
gnu/services/monitoring.scm | 82 +++++++++++++++++++++++++++++++++++++
gnu/tests/monitoring.scm | 73 ++++++++++++++++++++++++++++++++-
2 files changed, 154 insertions(+), 1 deletion(-)

Toggle diff (188 lines)
diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index 511f4fb2fe..a37dfd80d8 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -40,6 +40,17 @@
darkstat-service-type
prometheus-node-exporter-service-type
+ prometheus-service-type
+ <prometheus-configuration>
+ prometheus-configuration
+ prometheus-configuration-package
+ prometheus-configuration-user
+ prometheus-configuration-group
+ prometheus-configuration-config-file
+ prometheus-configuration-web-listen-address
+ prometheus-configuration-storage-tsdb-path
+ prometheus-configuration-extra-options
+
zabbix-server-configuration
zabbix-server-service-type
zabbix-agent-configuration
@@ -110,6 +121,77 @@ HTTP.")
(service-extension shepherd-root-service-type
(compose list darkstat-shepherd-service))))))
+
+;;;
+;;; Prometheus
+;;;
+
+(define-record-type* <prometheus-configuration>
+ prometheus-configuration
+ make-prometheus-configuration
+ prometheus-configuration?
+ (package prometheus-configuration-package
+ (default prometheus))
+ (user prometheus-configuration-user
+ (default "prometheus"))
+ (group prometheusconfiguration-group
+ (default "prometheus"))
+ (config-file prometheus-config-file
+ (default (plain-file "prometheus.yml" "")))
+ (web-listen-address prometheus-web-listen-address
+ (default "0.0.0.0:9090"))
+ (storage-tsdb-path prometheus-storage-tsdb-path
+ (default "/var/lib/prometheus/data/"))
+ (extra-options prometheus-configuration-extra-options
+ (default '())))
+
+(define prometheus-shepherd-service
+ (match-lambda
+ (($ <prometheus-configuration> package user group
+ config-file web-listen-address
+ storage-tsdb-path extra-options)
+ (shepherd-service
+ (documentation "Prometheus monitoring system and time series database.")
+ (provision '(prometheus))
+ (requirement '(networking))
+ (start #~(make-forkexec-constructor
+ (list #$(file-append package "/bin/prometheus")
+ "--config.file" #$config-file
+ "--web.listen-address" #$web-listen-address
+ "--storage.tsdb.path" #$storage-tsdb-path
+ #$@extra-options)
+ #:user #$user
+ #:group #$group
+ #:log-file "/var/log/prometheus.log"))
+ (stop #~(make-kill-destructor))))))
+
+(define (prometheus-account config)
+ (match-record config <prometheus-configuration>
+ (user group)
+ (list (user-group
+ (name group)
+ (system? #t))
+ (user-account
+ (name user)
+ (group group)
+ (system? #t)
+ (comment "Prometheus user")
+ (home-directory "/var/lib/prometheus")
+ (shell (file-append shadow "/sbin/nologin"))))))
+
+(define prometheus-service-type
+ (service-type
+ (name 'prometheus)
+ (description
+ "Run @command{prometheus} to scrape targets for mertrics and provide the
+web interface.")
+ (extensions
+ (list (service-extension shepherd-root-service-type
+ (compose list prometheus-shepherd-service))
+ (service-extension account-service-type
+ prometheus-account)))
+ (default-value (prometheus-configuration))))
+
(define-record-type* <prometheus-node-exporter-configuration>
prometheus-node-exporter-configuration
make-prometheus-node-exporter-configuration
diff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scm
index 732fbc54d7..e8c0847229 100644
--- a/gnu/tests/monitoring.scm
+++ b/gnu/tests/monitoring.scm
@@ -31,9 +31,80 @@
#:use-module (gnu system)
#:use-module (gnu tests)
#:use-module (guix gexp)
- #:export (%test-prometheus-node-exporter
+ #:export (%test-prometheus
+ %test-prometheus-node-exporter
%test-zabbix))
+
+;;;
+;;; Prometheus
+;;;
+
+(define* (run-prometheus-test name test-os)
+ "Run tests in %TEST-OS, which has Prometheus running."
+ (define os
+ (marionette-operating-system
+ test-os
+ #:imported-modules '((gnu services herd))))
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ (port-forwardings '((8080 . 9090)))))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (srfi srfi-11)
+ (srfi srfi-64)
+ (gnu build marionette)
+ (web client)
+ (web response))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin #$name)
+
+ (test-assert "prometheus running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (match (start-service 'prometheus)
+ (#f #f)
+ (('service response-parts ...)
+ (match (assq-ref response-parts 'running)
+ ((pid) (number? pid))))))
+ marionette))
+
+ (test-equal "prometheus healthy"
+ 200
+ (begin
+ (wait-for-tcp-port 9090 marionette)
+ (let-values (((response text)
+ (http-get "http://localhost:8080/-/healthy")))
+ (response-code response))))
+
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+ (gexp->derivation (string-append name "-test") test))
+
+(define %prometheus-test-os
+ (simple-operating-system
+ (service dhcp-client-service-type)
+ (service prometheus-service-type)))
+
+(define %test-prometheus
+ (system-test
+ (name "prometheus")
+ (description "Connect to a running Prometheus service.")
+ (value (run-prometheus-test name
+ %prometheus-test-os))))
+
;;;
;;; Prometheus Node Exporter
--
2.26.0
C
C
Christopher Baines wrote on 20 Apr 2020 23:17
[PATCH 3/4] gnu: Add alertmanager.
(address . 40738@debbugs.gnu.org)
20200420211743.23476-3-mail@cbaines.net
* gnu/packages/monitoring.scm (alertmanager): New variable.
---
gnu/packages/monitoring.scm | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)

Toggle diff (36 lines)
diff --git a/gnu/packages/monitoring.scm b/gnu/packages/monitoring.scm
index 345c2c16c5..e0e345d259 100644
--- a/gnu/packages/monitoring.scm
+++ b/gnu/packages/monitoring.scm
@@ -471,6 +471,29 @@ written in Go with pluggable metric collectors.")
(description "")
(license "")))
+(define-public alertmanager
+ (package
+ (name "alertmanager")
+ (version "0.20.0")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/prometheus/alertmanager.git")
+ (commit (string-append "v" version))))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ "1bq6vbpy25k7apvs2ga3fbp1cbnv9j0y1g1khvz2qgz6a2zvhgg3"))))
+ (build-system go-build-system)
+ (arguments
+ '(#:unpack-path "github.com/prometheus/alertmanager"
+ #:import-path "github.com/prometheus/alertmanager/cmd/alertmanager"
+ #:install-source? #f))
+ (home-page "https://prometheus.io/")
+ (synopsis "")
+ (description "")
+ (license "")))
+
(define-public fswatch
(package
(name "fswatch")
--
2.26.0
C
C
Christopher Baines wrote on 20 Apr 2020 23:17
[PATCH 4/4] services: Add a service for Alertmanager.
(address . 40738@debbugs.gnu.org)
20200420211743.23476-4-mail@cbaines.net
---
gnu/services/monitoring.scm | 82 +++++++++++++++++++++++++++++++++++++
gnu/tests/monitoring.scm | 71 ++++++++++++++++++++++++++++++++
2 files changed, 153 insertions(+)

Toggle diff (191 lines)
diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index a37dfd80d8..50a4b7302c 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -51,6 +51,17 @@
prometheus-configuration-storage-tsdb-path
prometheus-configuration-extra-options
+ alertmanager-service-type
+ <alertmanager-configuration>
+ alertmanager-configuration
+ alertmanager-configuration-package
+ alertmanager-configuration-user
+ alertmanager-configuration-group
+ alertmanager-configuration-config-file
+ alertmanager-configuration-web-listen-address
+ alertmanager-configuration-storage-path
+ alertmanager-configuration-extra-options
+
zabbix-server-configuration
zabbix-server-service-type
zabbix-agent-configuration
@@ -226,6 +237,77 @@ prometheus.")
(compose list prometheus-node-exporter-shepherd-service))))
(default-value (prometheus-node-exporter-configuration))))
+
+;;;
+;;; Alertmanager
+;;;
+
+(define-record-type* <alertmanager-configuration>
+ alertmanager-configuration
+ make-alertmanager-configuration
+ alertmanager-configuration?
+ (package alertmanager-configuration-package
+ (default alertmanager))
+ (user alertmanager-configuration-user
+ (default "alertmanager"))
+ (group alertmanagerconfiguration-group
+ (default "alertmanager"))
+ (config-file alertmanager-config-file
+ (default (plain-file "alertmanager.yml" "")))
+ (web-listen-address alertmanager-web-listen-address
+ (default ":9093"))
+ (storage-tsdb-path alertmanager-storage-tsdb-path
+ (default "/var/lib/alertmanager/data/"))
+ (extra-options alertmanager-configuration-extra-options
+ (default '())))
+
+(define alertmanager-shepherd-service
+ (match-lambda
+ (($ <alertmanager-configuration> package user group
+ config-file web-listen-address
+ storage-tsdb-path extra-options)
+ (shepherd-service
+ (documentation "Alertmanager monitoring system and time series database.")
+ (provision '(alertmanager))
+ (requirement '(networking))
+ (start #~(make-forkexec-constructor
+ (list #$(file-append package "/bin/alertmanager")
+ "--config.file" #$config-file
+ "--web.listen-address" #$web-listen-address
+ "--storage.path" #$storage-tsdb-path
+ #$@extra-options)
+ #:user #$user
+ #:group #$group
+ #:log-file "/var/log/alertmanager.log"))
+ (stop #~(make-kill-destructor))))))
+
+(define (alertmanager-account config)
+ (match-record config <alertmanager-configuration>
+ (user group)
+ (list (user-group
+ (name group)
+ (system? #t))
+ (user-account
+ (name user)
+ (group group)
+ (system? #t)
+ (comment "Alertmanager user")
+ (home-directory "/var/lib/alertmanager")
+ (shell (file-append shadow "/sbin/nologin"))))))
+
+(define alertmanager-service-type
+ (service-type
+ (name 'alertmanager)
+ (description
+ "Run @command{alertmanager} to scrape targets for mertrics and provide the
+web interface.")
+ (extensions
+ (list (service-extension shepherd-root-service-type
+ (compose list alertmanager-shepherd-service))
+ (service-extension account-service-type
+ alertmanager-account)))
+ (default-value (alertmanager-configuration))))
+
;;;
;;; Zabbix server
diff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scm
index e8c0847229..b77b654abc 100644
--- a/gnu/tests/monitoring.scm
+++ b/gnu/tests/monitoring.scm
@@ -33,6 +33,7 @@
#:use-module (guix gexp)
#:export (%test-prometheus
%test-prometheus-node-exporter
+ %test-alertmanager
%test-zabbix))
@@ -176,6 +177,76 @@
(value (run-prometheus-node-exporter-server-test
name %prometheus-node-exporter-os))))
+
+;;;
+;;; Alertmanager
+;;;
+
+(define* (run-alertmanager-test name test-os)
+ "Run tests in %TEST-OS, which has Alertmanager running."
+ (define os
+ (marionette-operating-system
+ test-os
+ #:imported-modules '((gnu services herd))))
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ (port-forwardings '((8080 . 9093)))))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (srfi srfi-11)
+ (srfi srfi-64)
+ (gnu build marionette)
+ (web client)
+ (web response))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin #$name)
+
+ (test-assert "alertmanager running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (match (start-service 'alertmanager)
+ (#f #f)
+ (('service response-parts ...)
+ (match (assq-ref response-parts 'running)
+ ((pid) (number? pid))))))
+ marionette))
+
+ (test-equal "alertmanager healthy"
+ 200
+ (begin
+ (wait-for-tcp-port 9090 marionette)
+ (let-values (((response text)
+ (http-get "http://localhost:8080/-/healthy")))
+ (response-code response))))
+
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+ (gexp->derivation (string-append name "-test") test))
+
+(define %alertmanager-test-os
+ (simple-operating-system
+ (service dhcp-client-service-type)
+ (service alertmanager-service-type)))
+
+(define %test-alertmanager
+ (system-test
+ (name "alertmanager")
+ (description "Connect to a running Alertmanager service.")
+ (value (run-alertmanager-test name
+ %alertmanager-test-os))))
+
;;;
;;; Zabbix
--
2.26.0
L
L
Ludovic Courtès wrote on 3 May 2020 12:56
Re: [bug#40738] Prometheus (and Alertmanager)
(name . Christopher Baines)(address . mail@cbaines.net)(address . 40738@debbugs.gnu.org)
87d07l1e3w.fsf@gnu.org
Hello!

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (3 lines)
> I've had a go at packaging Prometheus and Alertmanager, as well as
> writing Guix services for both of them.

Nice! I’m not an expert, but it sounds like it could advantageously
replace Zabbix on berlin.

Toggle quote (5 lines)
> The packages build, but still need quite a bit of work. They're both
> written in Go, so the Git repositories not only include the relevant
> source code, but the source code of the entire dependency tree (hence
> the packages have no inputs). That'll need addressing.

Would the gopkg importer at
of any help? (I think I posted an updated version later on but I can’t
find it.)

Toggle quote (8 lines)
> I've tested the Prometheus service, and it seems to work, although there
> isn't any record types for the Prometheus configuration yet. I haven't
> yet tested the Alertmanager service and the system test doesn't pass, I
> believe more configuration is required than Prometheus.
>
> Having a Prometheus service would be a good complement for the existing
> Prometheus Node exporter service in Guix.

Yup!

Thanks,
Ludo’.
C
C
Christopher Baines wrote on 3 May 2020 13:17
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 40738@debbugs.gnu.org)
874ksxmfmk.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (10 lines)
>> The packages build, but still need quite a bit of work. They're both
>> written in Go, so the Git repositories not only include the relevant
>> source code, but the source code of the entire dependency tree (hence
>> the packages have no inputs). That'll need addressing.
>
> Would the gopkg importer at
> <https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00310.html> be
> of any help? (I think I posted an updated version later on but I can’t
> find it.)

Yeah, I think getting a working importer is going to be the way to
address this.
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl6uqGNfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9Xeu+w/+PYEZRUQ8vhNNxb9xsVpH10ZnjLvnU8W7hCgPaMBXYyvxVXa99VWl811T
W7ZVjX1smgNEoUb3Yq81o3ib7DkkkH8n2T+GSnQNhw2n9LCo5hRa+cWB0W0qY23N
Tyu7alCiPpZOUI1aMbidpHULZlcyTVGMDda8croyy0Md+u0Yzh0Rr9SZ0G8yJGx7
w/m4jNOCm+K6JYVSSb+1NyNXENxZ3/4DbSZLNKjWnEOxQ4zNCNLLTaIxeralh1OG
fxmEeFAa7ZwNcA8bMA3Irvoxi+4lpeAL3G9I4QO7EEiMGC/fc+A2MM6PlcnkT/1E
rdoRK0vPJGoSI7GvmQzmbx06d/WY1HfEvDyDyM9t87YZGWkCJ6A7DmsCd87McnfD
JagWBIYx9PrdqHDkcs4pbjTQfpicGjjy59pEoab3AFlXLjJC4LtwJ5dqssZDC45e
rxjENCaSe51BGoXsU+1Iawi4VPuSyIePraV+WrqNeWrVUhGA7/HBQ98OiNH7I3wg
3+7KtJM6To+FpW7HDuxYhqzXVAi4cEOCmHngza9eKNxqZ6+xS+yTxr2ySUgMWWzE
y5AQcEobG3IEenBI0ckUt70+iQaonipaSIAv46tmO6S04Pto9N7ihDoeIcsYDc9T
zmrfCLWL7e0k5L/9Lkk839CeNIJ03JrofIRnpYLp8qqoU6ewMKc=
=cbv7
-----END PGP SIGNATURE-----

?