Prometheus (and Alertmanager)

Christopher Baines wrote on 20 Apr 2020 23:11

Recipients:(address . guix-patches@gnu.org)

Message-ID:87v9lt50d4.fsf@cbaines.net

I've had a go at packaging Prometheus and Alertmanager, as well as

writing Guix services for both of them.

The packages build, but still need quite a bit of work. They're both

written in Go, so the Git repositories not only include the relevant

source code, but the source code of the entire dependency tree (hence

the packages have no inputs). That'll need addressing.

I've tested the Prometheus service, and it seems to work, although there

isn't any record types for the Prometheus configuration yet. I haven't

yet tested the Alertmanager service and the system test doesn't pass, I

believe more configuration is required than Prometheus.

Having a Prometheus service would be a good complement for the existing

Prometheus Node exporter service in Guix.

I'll send the patches following this email.

Thanks,

Chris

Christopher Baines wrote on 20 Apr 2020 23:15

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87tv1d505n.fsf@cbaines.net

tag 40738 moreinfo

thanks

Christopher Baines wrote on 20 Apr 2020 23:17

Recipients:(address . 40738@debbugs.gnu.org)

Message-ID:20200420211743.23476-1-mail@cbaines.net

* gnu/packages/monitoring.scm (prometheus): New variable.

---

gnu/packages/monitoring.scm | 49 +++++++++++++++++++++++++++++++++++++

1 file changed, 49 insertions(+)

Toggle diff (62 lines)diff --git a/gnu/packages/monitoring.scm b/gnu/packages/monitoring.scm
index 8da31d6a84..345c2c16c5 100644
--- a/gnu/packages/monitoring.scm
+++ b/gnu/packages/monitoring.scm
@@ -422,6 +422,55 @@ written in Go with pluggable metric collectors.")
     (home-page "https://github.com/prometheus/node_exporter")
     (license license:asl2.0)))
 
+(define-public prometheus
+  (package
+    (name "prometheus")
+    (version "2.17.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/prometheus/prometheus.git")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1r7zpq6647lrm7cmid6nnf2xnljqh1i9g0fxvs0qrfd2sxxgj0c7"))))
+    (build-system go-build-system)
+    (arguments
+     '(#:unpack-path "github.com/prometheus/prometheus"
+       #:import-path "github.com/prometheus/prometheus/cmd/prometheus"
+       #:install-source? #f
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (assets-prefix
+                     (string-append out "/var/lib/prometheus/assets")))
+               (substitute* "src/github.com/prometheus/prometheus/web/ui/ui.go"
+                 (("var assetsPrefix string")
+                  (string-append "var assetsPrefix string = \""
+                                 assets-prefix
+                                 "\""))))
+             #t))
+         (add-after 'install 'install-assets
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (assets-prefix
+                     (string-append out "/var/lib/prometheus/assets")))
+               (for-each (lambda (directory)
+                           (copy-recursively
+                            (string-append "src/github.com/prometheus/prometheus"
+                                           "/web/ui/" directory)
+                            (string-append assets-prefix
+                                           "/" directory)))
+                         '("static" "templates")))
+             #t)))))
+    (home-page "https://prometheus.io/")
+    (synopsis "")
+    (description "")
+    (license "")))
+
 (define-public fswatch
   (package
     (name "fswatch")
-- 
2.26.0

Christopher Baines wrote on 20 Apr 2020 23:17

[PATCH 2/4] services: Add a Prometheus service.

Recipients:(address . 40738@debbugs.gnu.org)

Message-ID:20200420211743.23476-2-mail@cbaines.net

---

gnu/services/monitoring.scm | 82 +++++++++++++++++++++++++++++++++++++

gnu/tests/monitoring.scm | 73 ++++++++++++++++++++++++++++++++-

2 files changed, 154 insertions(+), 1 deletion(-)

Toggle diff (188 lines)diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index 511f4fb2fe..a37dfd80d8 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -40,6 +40,17 @@
             darkstat-service-type
             prometheus-node-exporter-service-type
 
+            prometheus-service-type
+            <prometheus-configuration>
+            prometheus-configuration
+            prometheus-configuration-package
+            prometheus-configuration-user
+            prometheus-configuration-group
+            prometheus-configuration-config-file
+            prometheus-configuration-web-listen-address
+            prometheus-configuration-storage-tsdb-path
+            prometheus-configuration-extra-options
+
             zabbix-server-configuration
             zabbix-server-service-type
             zabbix-agent-configuration
@@ -110,6 +121,77 @@ HTTP.")
           (service-extension shepherd-root-service-type
                              (compose list darkstat-shepherd-service))))))
 
+
+;;;
+;;; Prometheus
+;;;
+
+(define-record-type* <prometheus-configuration>
+  prometheus-configuration
+  make-prometheus-configuration
+  prometheus-configuration?
+  (package            prometheus-configuration-package
+                      (default prometheus))
+  (user               prometheus-configuration-user
+                      (default "prometheus"))
+  (group              prometheusconfiguration-group
+                      (default "prometheus"))
+  (config-file        prometheus-config-file
+                      (default (plain-file "prometheus.yml" "")))
+  (web-listen-address prometheus-web-listen-address
+                      (default "0.0.0.0:9090"))
+  (storage-tsdb-path  prometheus-storage-tsdb-path
+                      (default "/var/lib/prometheus/data/"))
+  (extra-options      prometheus-configuration-extra-options
+                      (default '())))
+
+(define prometheus-shepherd-service
+  (match-lambda
+    (($ <prometheus-configuration> package user group
+                                   config-file web-listen-address
+                                   storage-tsdb-path extra-options)
+     (shepherd-service
+      (documentation "Prometheus monitoring system and time series database.")
+      (provision '(prometheus))
+      (requirement '(networking))
+      (start #~(make-forkexec-constructor
+                (list #$(file-append package "/bin/prometheus")
+                      "--config.file" #$config-file
+                      "--web.listen-address" #$web-listen-address
+                      "--storage.tsdb.path" #$storage-tsdb-path
+                      #$@extra-options)
+                #:user #$user
+                #:group #$group
+                #:log-file "/var/log/prometheus.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (prometheus-account config)
+  (match-record config <prometheus-configuration>
+    (user group)
+    (list (user-group
+           (name group)
+           (system? #t))
+          (user-account
+           (name user)
+           (group group)
+           (system? #t)
+           (comment "Prometheus user")
+           (home-directory "/var/lib/prometheus")
+           (shell (file-append shadow "/sbin/nologin"))))))
+
+(define prometheus-service-type
+  (service-type
+   (name 'prometheus)
+   (description
+    "Run @command{prometheus} to scrape targets for mertrics and provide the
+web interface.")
+   (extensions
+    (list (service-extension shepherd-root-service-type
+                             (compose list prometheus-shepherd-service))
+          (service-extension account-service-type
+                             prometheus-account)))
+   (default-value (prometheus-configuration))))
+
 (define-record-type* <prometheus-node-exporter-configuration>
   prometheus-node-exporter-configuration
   make-prometheus-node-exporter-configuration
diff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scm
index 732fbc54d7..e8c0847229 100644
--- a/gnu/tests/monitoring.scm
+++ b/gnu/tests/monitoring.scm
@@ -31,9 +31,80 @@
   #:use-module (gnu system)
   #:use-module (gnu tests)
   #:use-module (guix gexp)
-  #:export (%test-prometheus-node-exporter
+  #:export (%test-prometheus
+            %test-prometheus-node-exporter
             %test-zabbix))
 
+
+;;;
+;;; Prometheus
+;;;
+
+(define* (run-prometheus-test name test-os)
+  "Run tests in %TEST-OS, which has Prometheus running."
+  (define os
+    (marionette-operating-system
+     test-os
+     #:imported-modules '((gnu services herd))))
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     (port-forwardings '((8080 . 9090)))))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (srfi srfi-11)
+                       (srfi srfi-64)
+                       (gnu build marionette)
+                       (web client)
+                       (web response))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (mkdir #$output)
+          (chdir #$output)
+
+          (test-begin #$name)
+
+          (test-assert "prometheus running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (match (start-service 'prometheus)
+                  (#f #f)
+                  (('service response-parts ...)
+                   (match (assq-ref response-parts 'running)
+                     ((pid) (number? pid))))))
+             marionette))
+
+          (test-equal "prometheus healthy"
+            200
+            (begin
+              (wait-for-tcp-port 9090 marionette)
+              (let-values (((response text)
+                            (http-get "http://localhost:8080/-/healthy")))
+                (response-code response))))
+
+          (test-end)
+          (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+  (gexp->derivation (string-append name "-test") test))
+
+(define %prometheus-test-os
+  (simple-operating-system
+   (service dhcp-client-service-type)
+   (service prometheus-service-type)))
+
+(define %test-prometheus
+  (system-test
+   (name "prometheus")
+   (description "Connect to a running Prometheus service.")
+   (value (run-prometheus-test name
+                               %prometheus-test-os))))
+
 
 ;;;
 ;;; Prometheus Node Exporter
-- 
2.26.0

Christopher Baines wrote on 20 Apr 2020 23:17

[PATCH 3/4] gnu: Add alertmanager.

Recipients:(address . 40738@debbugs.gnu.org)

Message-ID:20200420211743.23476-3-mail@cbaines.net

* gnu/packages/monitoring.scm (alertmanager): New variable.

---

gnu/packages/monitoring.scm | 23 +++++++++++++++++++++++

1 file changed, 23 insertions(+)

Toggle diff (36 lines)diff --git a/gnu/packages/monitoring.scm b/gnu/packages/monitoring.scm
index 345c2c16c5..e0e345d259 100644
--- a/gnu/packages/monitoring.scm
+++ b/gnu/packages/monitoring.scm
@@ -471,6 +471,29 @@ written in Go with pluggable metric collectors.")
     (description "")
     (license "")))
 
+(define-public alertmanager
+  (package
+    (name "alertmanager")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/prometheus/alertmanager.git")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1bq6vbpy25k7apvs2ga3fbp1cbnv9j0y1g1khvz2qgz6a2zvhgg3"))))
+    (build-system go-build-system)
+    (arguments
+     '(#:unpack-path "github.com/prometheus/alertmanager"
+       #:import-path "github.com/prometheus/alertmanager/cmd/alertmanager"
+       #:install-source? #f))
+    (home-page "https://prometheus.io/")
+    (synopsis "")
+    (description "")
+    (license "")))
+
 (define-public fswatch
   (package
     (name "fswatch")
-- 
2.26.0

Christopher Baines wrote on 20 Apr 2020 23:17

[PATCH 4/4] services: Add a service for Alertmanager.

Recipients:(address . 40738@debbugs.gnu.org)

Message-ID:20200420211743.23476-4-mail@cbaines.net

---

gnu/services/monitoring.scm | 82 +++++++++++++++++++++++++++++++++++++

gnu/tests/monitoring.scm | 71 ++++++++++++++++++++++++++++++++

2 files changed, 153 insertions(+)

Toggle diff (191 lines)diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index a37dfd80d8..50a4b7302c 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -51,6 +51,17 @@
             prometheus-configuration-storage-tsdb-path
             prometheus-configuration-extra-options
 
+            alertmanager-service-type
+            <alertmanager-configuration>
+            alertmanager-configuration
+            alertmanager-configuration-package
+            alertmanager-configuration-user
+            alertmanager-configuration-group
+            alertmanager-configuration-config-file
+            alertmanager-configuration-web-listen-address
+            alertmanager-configuration-storage-path
+            alertmanager-configuration-extra-options
+
             zabbix-server-configuration
             zabbix-server-service-type
             zabbix-agent-configuration
@@ -226,6 +237,77 @@ prometheus.")
            (compose list prometheus-node-exporter-shepherd-service))))
    (default-value (prometheus-node-exporter-configuration))))
 
+
+;;;
+;;; Alertmanager
+;;;
+
+(define-record-type* <alertmanager-configuration>
+  alertmanager-configuration
+  make-alertmanager-configuration
+  alertmanager-configuration?
+  (package            alertmanager-configuration-package
+                      (default alertmanager))
+  (user               alertmanager-configuration-user
+                      (default "alertmanager"))
+  (group              alertmanagerconfiguration-group
+                      (default "alertmanager"))
+  (config-file        alertmanager-config-file
+                      (default (plain-file "alertmanager.yml" "")))
+  (web-listen-address alertmanager-web-listen-address
+                      (default ":9093"))
+  (storage-tsdb-path  alertmanager-storage-tsdb-path
+                      (default "/var/lib/alertmanager/data/"))
+  (extra-options      alertmanager-configuration-extra-options
+                      (default '())))
+
+(define alertmanager-shepherd-service
+  (match-lambda
+    (($ <alertmanager-configuration> package user group
+                                   config-file web-listen-address
+                                   storage-tsdb-path extra-options)
+     (shepherd-service
+      (documentation "Alertmanager monitoring system and time series database.")
+      (provision '(alertmanager))
+      (requirement '(networking))
+      (start #~(make-forkexec-constructor
+                (list #$(file-append package "/bin/alertmanager")
+                      "--config.file" #$config-file
+                      "--web.listen-address" #$web-listen-address
+                      "--storage.path" #$storage-tsdb-path
+                      #$@extra-options)
+                #:user #$user
+                #:group #$group
+                #:log-file "/var/log/alertmanager.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (alertmanager-account config)
+  (match-record config <alertmanager-configuration>
+    (user group)
+    (list (user-group
+           (name group)
+           (system? #t))
+          (user-account
+           (name user)
+           (group group)
+           (system? #t)
+           (comment "Alertmanager user")
+           (home-directory "/var/lib/alertmanager")
+           (shell (file-append shadow "/sbin/nologin"))))))
+
+(define alertmanager-service-type
+  (service-type
+   (name 'alertmanager)
+   (description
+    "Run @command{alertmanager} to scrape targets for mertrics and provide the
+web interface.")
+   (extensions
+    (list (service-extension shepherd-root-service-type
+                             (compose list alertmanager-shepherd-service))
+          (service-extension account-service-type
+                             alertmanager-account)))
+   (default-value (alertmanager-configuration))))
+
 
 ;;;
 ;;; Zabbix server
diff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scm
index e8c0847229..b77b654abc 100644
--- a/gnu/tests/monitoring.scm
+++ b/gnu/tests/monitoring.scm
@@ -33,6 +33,7 @@
   #:use-module (guix gexp)
   #:export (%test-prometheus
             %test-prometheus-node-exporter
+            %test-alertmanager
             %test-zabbix))
 
 
@@ -176,6 +177,76 @@
    (value (run-prometheus-node-exporter-server-test
            name %prometheus-node-exporter-os))))
 
+
+;;;
+;;; Alertmanager
+;;;
+
+(define* (run-alertmanager-test name test-os)
+  "Run tests in %TEST-OS, which has Alertmanager running."
+  (define os
+    (marionette-operating-system
+     test-os
+     #:imported-modules '((gnu services herd))))
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     (port-forwardings '((8080 . 9093)))))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (srfi srfi-11)
+                       (srfi srfi-64)
+                       (gnu build marionette)
+                       (web client)
+                       (web response))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (mkdir #$output)
+          (chdir #$output)
+
+          (test-begin #$name)
+
+          (test-assert "alertmanager running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (match (start-service 'alertmanager)
+                  (#f #f)
+                  (('service response-parts ...)
+                   (match (assq-ref response-parts 'running)
+                     ((pid) (number? pid))))))
+             marionette))
+
+          (test-equal "alertmanager healthy"
+            200
+            (begin
+              (wait-for-tcp-port 9090 marionette)
+              (let-values (((response text)
+                            (http-get "http://localhost:8080/-/healthy")))
+                (response-code response))))
+
+          (test-end)
+          (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+  (gexp->derivation (string-append name "-test") test))
+
+(define %alertmanager-test-os
+  (simple-operating-system
+   (service dhcp-client-service-type)
+   (service alertmanager-service-type)))
+
+(define %test-alertmanager
+  (system-test
+   (name "alertmanager")
+   (description "Connect to a running Alertmanager service.")
+   (value (run-alertmanager-test name
+                               %alertmanager-test-os))))
+
 
 ;;;
 ;;; Zabbix
-- 
2.26.0

Ludovic Courtès wrote on 3 May 2020 12:56

Re: [bug#40738] Prometheus (and Alertmanager)

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 40738@debbugs.gnu.org)

Message-ID:87d07l1e3w.fsf@gnu.org

Hello!

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (3 lines)

> I've had a go at packaging Prometheus and Alertmanager, as well as

> writing Guix services for both of them.

Nice! I’m not an expert, but it sounds like it could advantageously

replace Zabbix on berlin.

Toggle quote (5 lines)

> The packages build, but still need quite a bit of work. They're both

> written in Go, so the Git repositories not only include the relevant

> source code, but the source code of the entire dependency tree (hence

> the packages have no inputs). That'll need addressing.

Would the gopkg importer at

https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00310.html be

of any help? (I think I posted an updated version later on but I can’t

find it.)

Toggle quote (8 lines)

> I've tested the Prometheus service, and it seems to work, although there

> isn't any record types for the Prometheus configuration yet. I haven't

> yet tested the Alertmanager service and the system test doesn't pass, I

> believe more configuration is required than Prometheus.

> Having a Prometheus service would be a good complement for the existing

> Prometheus Node exporter service in Guix.

Yup!

Thanks,

Ludo’.

Christopher Baines wrote on 3 May 2020 13:17

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 40738@debbugs.gnu.org)

Message-ID:874ksxmfmk.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (10 lines)>> The packages build, but still need quite a bit of work. They're both
>> written in Go, so the Git repositories not only include the relevant
>> source code, but the source code of the entire dependency tree (hence
>> the packages have no inputs). That'll need addressing.
>
> Would the gopkg importer at
> <https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00310.html> be
> of any help?  (I think I posted an updated version later on but I can’t
> find it.)

Yeah, I think getting a working importer is going to be the way to

address this.

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date