[PATCH 0/4] Add package and services for the Guix Build Coordinator

DoneSubmitted by Christopher Baines.
Details
3 participants
  • Jonathan Brielmaier
  • Ludovic Courtès
  • Christopher Baines
Owner
unassigned
Severity
normal
C
C
Christopher Baines wrote on 18 Sep 2020 20:34
(address . guix-patches@gnu.org)
87o8m39bnc.fsf@cbaines.net
This patch series is not particularly well tested, like the Guix Build
Coordinator itself, but it should hopefully still work at least in some
configurations.


Christopher Baines (4):
gnu: Add guix-build-coordinator.
services: guix: Add guix-build-coordinator-service-type.
services: guix: Add guix-build-coordinator-agent-service-type.
services: guix: Add guix-build-coordinator-queue-builds-service-type.

doc/guix.texi | 143 +++++++++
gnu/packages/package-management.scm | 89 ++++++
gnu/services/guix.scm | 433 +++++++++++++++++++++++++++-
gnu/tests/guix.scm | 75 ++++-
4 files changed, 736 insertions(+), 4 deletions(-)
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9k/cdfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9Xfrig//b1nSvUfhfTzlMrAc5stuOAEBdd7hrG82aj8exISjDZa2UfzFN0ox828z
ggjWbz69KjKP41OJfQgNGJ9GXqk582CvUaFloQBPQXH+3tBNJR+DlZHYEPhO5KlU
K0ZuatnpqCC704DQOsqTPAxKsaKuqSHYy9j7XFMieqilznvv0YMp8qpepTRyza6F
Y/H0qG4Ohfk3EhV5c99c4Yyvxx6x16TB+Gr93HBClVh3AYNbiEW2xuN7xSS03ljl
yr26rpna8fY4sXH3hUgbyOu7B1LNeiAveLRnEW2E3OtRBhXFB/nBc/L17Uz2tje4
zRD86Bl4nWZ26v9FKaTwQAKaCl0S5Fi5DSPZviO8FySVRXfDHYy3PaZks3HHpDDQ
/vgMbLba9BbBJ0Zq7/uoGgk9rWLUCuUDkfIigxlvyMHUAuNCnh3PnvjwJ3zenK6B
HGSJ2Xi2JDE0pHrt4Tzs4uBG2JC+JyDZAhgZjJSkFfWP4YdJaLKq7J8BkFJV5c/E
x42nxLdf9LaqeR7zfCqS8u0mBocmrvl9bfOY7aafo4ezE6ktDDp1ecwnxzpVlgIE
Pn3ItRAIrs/w1O8i0ITHIMArpUdS+Yk+rfyEs400QB4kSXnphaBQdUKL0arRIFXr
nqzi+I857L7ZqKWR7AKGSmWAJQay3KzzuENVho+dLJtC4wZxwcg=
=uTsG
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 18 Sep 2020 20:40
[PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(address . 43494@debbugs.gnu.org)
20200918184042.22660-4-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
New record type.
(guix-build-coordinator-queue-builds-configuration,
guix-build-coordinator-queue-builds-configuration?,
guix-build-coordinator-queue-builds-configuration-package,
guix-build-coordinator-queue-builds-configuration-user,
guix-build-coordinator-queue-builds-coordinator,
guix-build-coordinator-queue-builds-configuration-systems,
guix-build-coordinator-queue-builds-configuration-system-and-targets,
guix-build-coordinator-queue-builds-configuration-guix-data-service,
guix-build-coordinator-queue-builds-configuration-processed-commits-file,
guix-build-coordinator-queue-builds-shepherd-services,
guix-build-coordinator-queue-builds-activation,
guix-build-coordinator-queue-builds-account): New procedures.
(guix-build-coordinator-queue-builds-service-type): New variable.
---
doc/guix.texi | 38 +++++++++++++++
gnu/services/guix.scm | 108 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 146 insertions(+)

Toggle diff (184 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index db91831197..1b47a01217 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27590,6 +27590,44 @@ input store items aren't already available.
 @end table
 @end deftp
 
+@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
+Service type for the
+guix-build-coordinator-queue-builds-from-guix-data-service script.  Its
+value must be a @code{guix-build-coordinator-queue-builds-configuration}
+object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-queue-builds-configuration
+Data type representing the options to the queue builds from guix data
+service script.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"})
+The system user to run the service as.
+
+@item @code{coordinator} (default: @code{"http://localhost:8745"})
+The URI to use when connecting to the coordinator.
+
+@item @code{systems} (default: @code{#f})
+The systems for which to fetch derivations to build.
+
+@item @code{systems-and-targets} (default: @code{#f})
+An association list of system and target pairs for which to fetch
+derivations to build.
+
+@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})
+The Guix Data Service instance from which to query to find out about
+derivations to build.
+
+@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})
+A file to record which commits have been processed, to avoid needlessly
+processing them again if the service is restarted.
+
+@end table
+@end deftp
 
 @subsubheading Guix Data Service
 The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 71e58237da..975febebd1 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -65,6 +65,18 @@
 
             guix-build-coordinator-agent-service-type
 
+            guix-build-coordinator-queue-builds-configuration
+            guix-build-coordinator-queue-builds-configuration?
+            guix-build-coordinator-queue-builds-configuration-package
+            guix-build-coordinator-queue-builds-configuration-user
+            guix-build-coordinator-queue-builds-coordinator
+            guix-build-coordinator-queue-builds-configuration-systems
+            guix-build-coordinator-queue-builds-configuration-system-and-targets
+            guix-build-coordinator-queue-builds-configuration-guix-data-service
+            guix-build-coordinator-queue-builds-configuration-processed-commits-file
+
+            guix-build-coordinator-queue-builds-service-type
+
             <guix-data-service-configuration>
             guix-data-service-configuration
             guix-data-service-configuration?
@@ -137,6 +149,28 @@
    guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
    (default #f)))
 
+(define-record-type* <guix-build-coordinator-queue-builds-configuration>
+  guix-build-coordinator-queue-builds-configuration
+  make-guix-build-coordinator-queue-builds-configuration
+  guix-build-coordinator-queue-builds-configuration?
+  (package              guix-build-coordinator-queue-builds-configuration-package
+                        (default guix-build-coordinator))
+  (user                 guix-build-coordinator-queue-builds-configuration-user
+                        (default "guix-build-coordinator-queue-builds"))
+  (coordinator          guix-build-coordinator-queue-builds-coordinator
+                        (default "http://localhost:8745"))
+  (systems              guix-build-coordinator-queue-builds-configuration-systems
+                        (default #f))
+  (systems-and-targets
+   guix-build-coordinator-queue-builds-configuration-system-and-targets
+   (default #f))
+  (guix-data-service
+   guix-build-coordinator-queue-builds-configuration-guix-data-service
+   (default "https://data.guix.gnu.org"))
+  (processed-commits-file
+   guix-build-coordinator-queue-builds-configuration-processed-commits-file
+   (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))
+
 (define* (make-guix-build-coordinator-start-script database-uri-string
                                                    allocation-strategy
                                                    pid-file
@@ -364,6 +398,80 @@
    (description
     "Run an instance of the Guix Build Coordinator.")))
 
+(define (guix-build-coordinator-queue-builds-shepherd-services config)
+  (match-record config <guix-build-coordinator-queue-builds-configuration>
+    (package user coordinator systems systems-and-targets
+             guix-data-service processed-commits-file)
+    (list
+     (shepherd-service
+      (documentation "Guix Build Coordinator queue builds from Guix Data Service")
+      (provision '(guix-build-coordinator-queue-builds))
+      (requirement '(networking))
+      (start
+       #~(make-forkexec-constructor
+          (list
+           #$(file-append
+              package
+              "/bin/guix-build-coordinator-queue-builds-from-guix-data-service")
+           #$(string-append "--coordinator=" coordinator)
+           #$@(map (lambda (system)
+                     (string-append "--system=" system))
+                   (or systems '()))
+           #$@(map (match-lambda
+                     ((system . target)
+                      (string-append "--system-and-target=" system "=" target)))
+                   (or systems-and-targets '()))
+           #$@(if guix-data-service
+                  #~(#$(string-append "--guix-data-service=" guix-data-service))
+                  #~())
+           #$@(if processed-commits-file
+                  #~(#$(string-append "--processed-commits-file="
+                                      processed-commits-file))
+                  #~()))
+          #:user #$user
+          #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid"
+          #:environment-variables
+          `(,(string-append
+              "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+            "LC_ALL=en_US.utf8")
+          #:log-file "/var/log/guix-build-coordinator/queue-builds.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-queue-builds-activation config)
+  #~(begin
+      (use-modules (guix build utils))
+
+      (mkdir-p "/var/log/guix-build-coordinator")
+
+      ;; Allow writing the PID file
+      (mkdir-p "/var/run/guix-build-coordinator-queue-builds")
+      (chown "/var/run/guix-build-coordinator-queue-builds"
+             (passwd:uid %user)
+             (passwd:gid %user))))
+
+(define (guix-build-coordinator-queue-builds-account config)
+  (list (user-account
+         (name (guix-build-coordinator-queue-builds-configuration-user config))
+         (group "nogroup")
+         (system? #t)
+         (comment "Guix Build Coordinator queue-builds user")
+         (home-directory "/var/empty")
+         (shell (file-append shadow "/sbin/nologin")))))
+
+(define guix-build-coordinator-queue-builds-service-type
+  (service-type
+   (name 'guix-build-coordinator-queue-builds)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        guix-build-coordinator-queue-builds-shepherd-services)
+     (service-extension activation-service-type
+                        guix-build-coordinator-queue-builds-activation)
+     (service-extension account-service-type
+                        guix-build-coordinator-queue-builds-account)))
+   (description
+    "Run the Guix Build Coordinator queue builds script.")))
+
 
 ;;;
 ;;; Guix Data Service
-- 
2.28.0
C
C
Christopher Baines wrote on 18 Sep 2020 20:40
[PATCH 1/4] gnu: Add guix-build-coordinator.
(address . 43494@debbugs.gnu.org)
20200918184042.22660-1-mail@cbaines.net
* gnu/packages/package-management.scm (guix-build-coordinator): New variable.
---
gnu/packages/package-management.scm | 89 +++++++++++++++++++++++++++++
1 file changed, 89 insertions(+)

Toggle diff (109 lines)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index ec87226197..4272d3134d 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -50,6 +50,7 @@
   #:use-module (gnu packages cpio)
   #:use-module (gnu packages crypto)
   #:use-module (gnu packages curl)
+  #:use-module (gnu packages databases)
   #:use-module (gnu packages dbm)
   #:use-module (gnu packages docbook)
   #:use-module (gnu packages file)
@@ -981,6 +982,94 @@ environments.")
     ;; and the fonts included in this package are licensed OFL1.1.
     (license (list license:gpl3+ license:agpl3+ license:silofl1.1))))
 
+(define-public guix-build-coordinator
+  (let ((commit "5e8408c833e209efddfa0159114b90400c1aaf4d")
+        (revision "0"))
+    (package
+    (name "guix-build-coordinator")
+    (version (git-version "0" revision commit))
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://git.cbaines.net/git/guix/build-coordinator")
+                    (commit commit)))
+              (sha256
+               (base32
+                "0f7m1zg9mlb2m22qyblglaa36h8f49b810jc9j5b0hsb42ijwh4b"))
+              (file-name (string-append name "-" version "-checkout"))))
+    (build-system gnu-build-system)
+    (arguments
+     '(#:modules ((guix build utils)
+                  (guix build gnu-build-system)
+                  (ice-9 ftw)
+                  (ice-9 match)
+                  (ice-9 rdelim)
+                  (ice-9 popen))
+       #:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'set-GUILE_AUTO_COMPILE
+           (lambda _
+             ;; To avoid warnings relating to 'guild'.
+             (setenv "GUILE_AUTO_COMPILE" "0")
+             #t))
+         (add-after 'install 'wrap-executable
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (guile (assoc-ref inputs "guile"))
+                    (guile-effective-version
+                     (read-line
+                      (open-pipe* OPEN_READ
+                                  (string-append guile "/bin/guile")
+                                  "-c" "(display (effective-version))")))
+                    (scm (string-append out "/share/guile/site/"
+                                        guile-effective-version))
+                    (go  (string-append out "/lib/guile/"
+                                        guile-effective-version
+                                        "/site-ccache")))
+               (for-each
+                (lambda (file)
+                  (simple-format (current-error-port)
+                                 "wrapping: ~A\n"
+                                 (string-append bin "/" file))
+                  (wrap-program (string-append bin "/" file)
+                    `("PATH" ":" prefix
+                      (,bin ,(assoc-ref inputs "sqitch")))
+                    `("PERL5LIB" ":" prefix
+                      (,(getenv "PERL5LIB")))
+                    `("GUILE_LOAD_PATH" ":" prefix
+                      (,scm ,(getenv "GUILE_LOAD_PATH")))
+                    `("GUILE_LOAD_COMPILED_PATH" ":" prefix
+                      (,go ,(getenv "GUILE_LOAD_COMPILED_PATH")))))
+                (scandir bin
+                         (match-lambda
+                           ((or "." "..") #f)
+                           (_ #t))))
+               #t)))
+         (delete 'strip))))             ; As the .go files aren't compatible
+    (native-inputs
+     `(("pkg-config" ,pkg-config)
+       ("autoconf" ,autoconf)
+       ("automake" ,automake)
+       ("perl" ,perl)))
+    (inputs
+     `(("guile" ,guile-3.0-latest)
+       ("guile-fibers" ,guile-fibers)
+       ("guile-prometheus" ,guile-prometheus)
+       ("guile-gcrypt" ,guile-gcrypt)
+       ("guile-json" ,guile-json-3)
+       ("guile-lzlib" ,guile-lzlib)
+       ("guile-sqlite3" ,guile-sqlite3)
+       ("guix" ,guix)
+       ("sqlite" ,sqlite)
+       ("sqitch" ,sqitch)
+       ("perl-dbd-sqlite" ,perl-dbd-sqlite)))
+    (home-page "https://git.cbaines.net/guile/guix/build-coordinator")
+    (synopsis "")
+    (description
+     "")
+    (license license:gpl3+))))
+
 (define-public guix-jupyter
   (package
     (name "guix-jupyter")
-- 
2.28.0
C
C
Christopher Baines wrote on 18 Sep 2020 20:40
[PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
(address . 43494@debbugs.gnu.org)
20200918184042.22660-3-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New
record type.
(guix-build-coordinator-agent-configuration,
guix-build-coordinator-agent-configuration?,
guix-build-coordinator-agent-configuration-package,
guix-build-coordinator-agent-configuration-user,
guix-build-coordinator-agent-configuration-coordinator,
guix-build-coordinator-agent-configuration-uuid),
guix-build-coordinator-agent-configuration-password,
guix-build-coordinator-agent-configuration-password-file,
guix-build-coordinator-agent-configuration-systems,
guix-build-coordinator-agent-configuration-max-parallel-builds,
guix-build-coordinator-agent-configuration-derivation-substitute-urls,
guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,
guix-build-coordinator-agent-shepherd-services,
guix-build-coordinator-agent-activation,
guix-build-coordinator-agent-account): New procedures.
(guix-build-coordinator-agent-service-type): New variable.
* doc/guix.texi (Guix Services): Document it.
---
doc/guix.texi | 50 +++++++++++++++++
gnu/services/guix.scm | 121 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 171 insertions(+)

Toggle diff (209 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index e83008c177..db91831197 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27541,6 +27541,56 @@ The Guile package with which to run the Guix Build Coordinator.
 @end table
 @end deftp
 
+@defvar {Scheme Variable} guix-build-coordinator-agent-service-type
+Service type for a Guix Build Coordinator agent.  Its value must be a
+@code{guix-build-coordinator-agent-configuration} object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-agent-configuration
+Data type representing the configuration a Guix Build Coordinator agent.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator-agent"})
+The system user to run the service as.
+
+@item @code{coordinator} (default: @code{"http://localhost:8745"})
+The URI to use when connecting to the coordinator.
+
+@item @code{uuid}
+The UUID of the agent.  This should be generated by the coordinator
+process, stored in the coordinator database, and used by the intended
+agent.
+
+@item @code{password} (default: @code{#f})
+The password to use when connecting to the coordinator.  A file to read
+the password from can also be specified, and this is more secure.
+
+@item @code{password-file} (default: @code{#f})
+A file containing the password to use when connecting to the
+coordinator.
+
+@item @code{systems} (default: @var{#f})
+The systems for which this agent should fetch builds.  The agent process
+will use the current system it's running on as the default.
+
+@item @code{max-parallel-builds} (default: @code{1})
+The number of builds to perform in parallel.
+
+@item @code{derivation-substitute-urls} (default: @code{1})
+URLs from which to attempt to fetch substitutes for derivations, if the
+derivations aren't already available.
+
+@item @code{non-derivation-substitute-urls} (default: @code{1})
+URLs from which to attempt to fetch substitutes for build inputs, if the
+input store items aren't already available.
+
+@end table
+@end deftp
+
+
 @subsubheading Guix Data Service
 The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
 and provides data about GNU Guix.  This includes information about
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index de14fcadb2..71e58237da 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -50,6 +50,21 @@
 
             guix-build-coordinator-service-type
 
+            guix-build-coordinator-agent-configuration
+            guix-build-coordinator-agent-configuration?
+            guix-build-coordinator-agent-configuration-package
+            guix-build-coordinator-agent-configuration-user
+            guix-build-coordinator-agent-configuration-coordinator
+            guix-build-coordinator-agent-configuration-uuid)
+            guix-build-coordinator-agent-configuration-password
+            guix-build-coordinator-agent-configuration-password-file
+            guix-build-coordinator-agent-configuration-systems
+            guix-build-coordinator-agent-configuration-max-parallel-builds
+            guix-build-coordinator-agent-configuration-derivation-substitute-urls
+            guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
+
+            guix-build-coordinator-agent-service-type
+
             <guix-data-service-configuration>
             guix-data-service-configuration
             guix-data-service-configuration?
@@ -95,6 +110,33 @@
   (guile                           guix-build-coordinator-configuration-guile
                                    (default guile-3.0-latest)))
 
+(define-record-type* <guix-build-coordinator-agent-configuration>
+  guix-build-coordinator-agent-configuration
+  make-guix-build-coordinator-agent-configuration
+  guix-build-coordinator-agent-configuration?
+  (package             guix-build-coordinator-agent-configuration-package
+                       (default guix-build-coordinator))
+  (user                guix-build-coordinator-agent-configuration-user
+                       (default "guix-build-coordinator-agent"))
+  (coordinator         guix-build-coordinator-agent-configuration-coordinator
+                       (default "http://localhost:8745"))
+  (uuid                guix-build-coordinator-agent-configuration-uuid)
+  (password            guix-build-coordinator-agent-configuration-password
+                       (default #f))
+  (password-file       guix-build-coordinator-agent-configuration-password-file
+                       (default #f))
+  (systems             guix-build-coordinator-agent-configuration-systems
+                       (default #f))
+  (max-parallel-builds
+   guix-build-coordinator-agent-configuration-max-parallel-builds
+   (default 1))
+  (derivation-substitute-urls
+   guix-build-coordinator-agent-configuration-derivation-substitute-urls
+   (default #f))
+  (non-derivation-substitute-urls
+   guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
+   (default #f)))
+
 (define* (make-guix-build-coordinator-start-script database-uri-string
                                                    allocation-strategy
                                                    pid-file
@@ -243,6 +285,85 @@
    (description
     "Run an instance of the Guix Build Coordinator.")))
 
+(define (guix-build-coordinator-agent-shepherd-services config)
+  (match-record config <guix-build-coordinator-agent-configuration>
+    (package user coordinator uuid password password-file max-parallel-builds
+             derivation-substitute-urls non-derivation-substitute-urls
+             systems)
+    (list
+     (shepherd-service
+      (documentation "Guix Build Coordinator Agent")
+      (provision '(guix-build-coordinator-agent))
+      (requirement '(networking))
+      (start #~(make-forkexec-constructor
+                (list #$(file-append package "/bin/guix-build-coordinator-agent")
+                      #$(string-append "--coordinator=" coordinator)
+                      #$(string-append "--uuid=" uuid)
+                      #$@(if password
+                             #~(#$(string-append "--password=" password))
+                             #~())
+                      #$@(if password-file
+                             #~(#$(string-append "--password-file=" password-file))
+                             #~())
+                      #$(simple-format #f "--max-parallel-builds=~A"
+                                       max-parallel-builds)
+                      #$@(if derivation-substitute-urls
+                             #~(#$(string-append
+                                   "--derivation-substitute-urls="
+                                 (string-join derivation-substitute-urls " ")))
+                             #~())
+                      #$@(if non-derivation-substitute-urls
+                             #~(#$(string-append
+                                   "--non-derivation-substitute-urls="
+                                   (string-join derivation-substitute-urls " ")))
+                             #~())
+                      #$@(map (lambda (system)
+                                (string-append "--system=" system))
+                              (or systems '())))
+                #:user #$user
+                #:pid-file "/var/run/guix-build-coordinator-agent/pid"
+                #:environment-variables
+                `(,(string-append
+                    "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+                  "LC_ALL=en_US.utf8")
+                #:log-file "/var/log/guix-build-coordinator/agent.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-agent-activation config)
+  #~(begin
+      (use-modules (guix build utils))
+
+      (mkdir-p "/var/log/guix-build-coordinator")
+
+      ;; Allow writing the PID file
+      (mkdir-p "/var/run/guix-build-coordinator-agent")
+      (chown "/var/run/guix-build-coordinator-agent"
+             (passwd:uid %user)
+             (passwd:gid %user))))
+
+(define (guix-build-coordinator-agent-account config)
+  (list (user-account
+         (name (guix-build-coordinator-agent-configuration-user config))
+         (group "nogroup")
+         (system? #t)
+         (comment "Guix Build Coordinator agent user")
+         (home-directory "/var/empty")
+         (shell (file-append shadow "/sbin/nologin")))))
+
+(define guix-build-coordinator-agent-service-type
+  (service-type
+   (name 'guix-build-coordinator-agent)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        guix-build-coordinator-agent-shepherd-services)
+     (service-extension activation-service-type
+                        guix-build-coordinator-agent-activation)
+     (service-extension account-service-type
+                        guix-build-coordinator-agent-account)))
+   (description
+    "Run an instance of the Guix Build Coordinator.")))
+
 
 ;;;
 ;;; Guix Data Service
-- 
2.28.0
C
C
Christopher Baines wrote on 18 Sep 2020 20:40
[PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
(address . 43494@debbugs.gnu.org)
20200918184042.22660-2-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record
type.
(guix-build-coordinator-configuration, guix-build-coordinator-configuration?,
guix-build-coordinator-configuration-package,
guix-build-coordinator-configuration-user,
guix-build-coordinator-configuration-group,
guix-build-coordinator-configuration-datastore-uri-string,
guix-build-coordinator-configuration-agent-communication-uri-string,
guix-build-coordinator-configuration-client-communication-uri-string,
guix-build-coordinator-configuration-allocation-strategy,
guix-build-coordinator-configuration-hooks,
guix-build-coordinator-configuration-guile,
make-guix-build-coordinator-start-script,
guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,
guix-build-coordinator-account): New procedures.
(guix-build-coordinator-service-type): New variable.
* gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.
* doc/guix.texi (Guix Services): Document it.
---
doc/guix.texi | 55 ++++++++++++
gnu/services/guix.scm | 204 +++++++++++++++++++++++++++++++++++++++++-
gnu/tests/guix.scm | 75 +++++++++++++++-
3 files changed, 330 insertions(+), 4 deletions(-)

Toggle diff (392 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 88128a4b3a..e83008c177 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27486,6 +27486,61 @@ The complete list of possible options can be found in the man page for
 @node Guix Services
 @subsection Guix Services
 
+@subsubheading Guix Build Coordinator
+The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
+Coordinator} aids in building derivations.  The Guix Daemon is still
+used to build the derivations, but the Guix Build Coordinator provides
+ways of performing builds across many machines and working with the
+results of builds.
+
+There is a script to run the coordinator component of the Guix Build
+Coordinator, but the Guix service uses a custom Guile script instead, to
+provide better integration with G-expressions used in the configuration.
+
+@defvar {Scheme Variable} guix-build-coordinator-service-type
+Service type for the Guix Build Coordinator.  Its value must be a
+@code{guix-build-coordinator-configuration} object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-configuration
+Data type representing the configuration of the Guix Build Coordinator.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator"})
+The system user to run the service as.
+
+@item @code{group} (default: @code{"guix-build-coordinator"})
+The system group to run the service as.
+
+@item @code{database-uri-string} (default: @code{"sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"})
+The URI to use for the database.
+
+@item @code{agent-communication-uri} (default: @code{"http://0.0.0.0:8745"})
+The URI describing how to listen to requests from agent processes.
+
+@item @code{client-communication-uri} (default: @code{"http://127.0.0.1:8746"})
+The URI describing how to listen to requests from clients.  The client
+API allows submitting builds and currently isn't authenticated, so take
+care when configuring this value.
+
+@item @code{allocation-strategy} (default: @code{#~basic-build-allocation-strategy})
+A G-expression for the allocation strategy to be used.  This is a
+procedure that takes the datastore as an argument and populates the
+allocation plan in the database.
+
+@item @code{hooks} (default: @var{'()})
+An association list of hooks.  These provide a way to execute arbitrary
+code upon certian events, like a build result being processed.
+
+@item @code{guile} (default: @code{guile-3.0-latest})
+The Guile package with which to run the Guix Build Coordinator.
+
+@end table
+@end deftp
+
 @subsubheading Guix Data Service
 The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
 and provides data about GNU Guix.  This includes information about
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 10a8581a62..de14fcadb2 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -17,20 +17,40 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (gnu services guix)
+  #:use-module (srfi srfi-1)
   #:use-module (ice-9 match)
   #:use-module (guix gexp)
   #:use-module (guix records)
+  #:use-module (guix packages)
   #:use-module ((gnu packages base)
                 #:select (glibc-utf8-locales))
   #:use-module (gnu packages admin)
+  #:use-module (gnu packages databases)
   #:use-module (gnu packages web)
+  #:use-module (gnu packages guile)
+  #:use-module (gnu packages guile-xyz)
+  #:use-module (gnu packages package-management)
   #:use-module (gnu services)
   #:use-module (gnu services base)
   #:use-module (gnu services admin)
   #:use-module (gnu services shepherd)
   #:use-module (gnu services getmail)
   #:use-module (gnu system shadow)
-  #:export (<guix-data-service-configuration>
+  #:export (guix-build-coordinator-configuration
+            guix-build-coordinator-configuration?
+            guix-build-coordinator-configuration-package
+            guix-build-coordinator-configuration-user
+            guix-build-coordinator-configuration-group
+            guix-build-coordinator-configuration-datastore-uri-string
+            guix-build-coordinator-configuration-agent-communication-uri-string
+            guix-build-coordinator-configuration-client-communication-uri-string
+            guix-build-coordinator-configuration-allocation-strategy
+            guix-build-coordinator-configuration-hooks
+            guix-build-coordinator-configuration-guile
+
+            guix-build-coordinator-service-type
+
+            <guix-data-service-configuration>
             guix-data-service-configuration
             guix-data-service-configuration?
             guix-data-service-package
@@ -45,11 +65,189 @@
 
 ;;;; Commentary:
 ;;;
-;;; This module implements a service that to run instances of the Guix Data
-;;; Service, which provides data about Guix over time.
+;;; Services specifically related to GNU Guix.
 ;;;
 ;;;; Code:
 
+(define-record-type* <guix-build-coordinator-configuration>
+  guix-build-coordinator-configuration make-guix-build-coordinator-configuration
+  guix-build-coordinator-configuration?
+  (package                         guix-build-coordinator-configuration-package
+                                   (default guix-build-coordinator))
+  (user                            guix-build-coordinator-configuration-user
+                                   (default "guix-build-coordinator"))
+  (group                           guix-build-coordinator-configuration-group
+                                   (default "guix-build-coordinator"))
+  (database-uri-string
+   guix-build-coordinator-configuration-datastore-uri-string
+   (default "sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"))
+  (agent-communication-uri-string
+   guix-build-coordinator-configuration-agent-communication-uri-string
+   (default "http://0.0.0.0:8745"))
+  (client-communication-uri-string
+   guix-build-coordinator-configuration-client-communication-uri-string
+   (default "http://127.0.0.1:8746"))
+  (allocation-strategy
+   guix-build-coordinator-configuration-allocation-strategy
+   (default #~basic-build-allocation-strategy))
+  (hooks                           guix-build-coordinator-configuration-hooks
+                                   (default '()))
+  (guile                           guix-build-coordinator-configuration-guile
+                                   (default guile-3.0-latest)))
+
+(define* (make-guix-build-coordinator-start-script database-uri-string
+                                                   allocation-strategy
+                                                   pid-file
+                                                   guix-build-coordinator-package
+                                                   #:key
+                                                   agent-communication-uri-string
+                                                   client-communication-uri-string
+                                                   (hooks '())
+                                                   (guile guile-3.0))
+  (program-file
+   "start-guix-build-coordinator"
+   (with-extensions (cons guix-build-coordinator-package
+                          ;; This is a poorly constructed Guile load path,
+                          ;; since it contains things that aren't Guile
+                          ;; libraries, but it means that the Guile libraries
+                          ;; needed for the Guix Build Coordinator don't need
+                          ;; to be individually specified here.
+                          (map second (package-inputs
+                                       guix-build-coordinator-package)))
+     #~(begin
+         (use-modules (srfi srfi-1)
+                      (ice-9 match)
+                      (web uri)
+                      (prometheus)
+                      (guix-build-coordinator hooks)
+                      (guix-build-coordinator datastore)
+                      (guix-build-coordinator build-allocator)
+                      (guix-build-coordinator coordinator))
+
+         ;; Otherwise sqitch can't use SQlite
+         (setenv "PERL5LIB"
+                 #$(file-append perl-dbd-sqlite "/lib/perl5/site_perl"))
+
+         (let* ((metrics-registry (make-metrics-registry
+                                   #:namespace
+                                   "guixbuildcoordinator_"))
+                (datastore (database-uri->datastore
+                            #$database-uri-string
+                            #:metrics-registry metrics-registry))
+                (hooks
+                 (list #$@(map (match-lambda
+                                 ((name . hook-gexp)
+                                  #~(cons name #$hook-gexp)))
+                               hooks)))
+                (hooks-with-defaults
+                 `(,@hooks
+                   ,@(remove (match-lambda
+                               ((name . _) (assq-ref hooks name)))
+                             %default-hooks)))
+                (build-coordinator (make-build-coordinator
+                                    #:datastore datastore
+                                    #:hooks hooks-with-defaults
+                                    #:metrics-registry metrics-registry
+                                    #:allocation-strategy #$allocation-strategy)))
+
+           (run-coordinator-service
+            build-coordinator
+            #:update-datastore? #t
+            #:pid-file #$pid-file
+            #:agent-communication-uri (string->uri
+                                       #$agent-communication-uri-string)
+            #:client-communication-uri (string->uri
+                                        #$client-communication-uri-string)))))
+   #:guile guile))
+
+(define (guix-build-coordinator-shepherd-services config)
+  (match-record config <guix-build-coordinator-configuration>
+    (package user group database-uri-string
+             agent-communication-uri-string
+             client-communication-uri-string
+             allocation-strategy
+             hooks
+             guile)
+    (list
+     (shepherd-service
+      (documentation "Guix Build Coordinator")
+      (provision '(guix-build-coordinator))
+      (requirement '(networking))
+      (start #~(make-forkexec-constructor
+                (list #$(make-guix-build-coordinator-start-script
+                         database-uri-string
+                         allocation-strategy
+                         "/var/run/guix-build-coordinator/pid"
+                         package
+                         #:agent-communication-uri-string
+                         agent-communication-uri-string
+                         #:client-communication-uri-string
+                         client-communication-uri-string
+                         #:hooks hooks
+                         #:guile guile))
+                #:user #$user
+                #:group #$group
+                #:pid-file "/var/run/guix-build-coordinator/pid"
+                ;; Allow time for migrations to run
+                #:pid-file-timeout 60
+                #:environment-variables
+                `(,(string-append
+                    "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+                  "LC_ALL=en_US.utf8")
+                #:log-file "/var/log/guix-build-coordinator/coordinator.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-activation config)
+  #~(begin
+      (use-modules (guix build utils))
+
+      (define %user (getpw "guix-build-coordinator"))
+
+      (chmod "/var/lib/guix-build-coordinator" #o755)
+
+      (mkdir-p "/var/log/guix-build-coordinator")
+
+      ;; Allow writing the PID file
+      (mkdir-p "/var/run/guix-build-coordinator")
+      (chown "/var/run/guix-build-coordinator"
+             (passwd:uid %user)
+             (passwd:gid %user))))
+
+(define (guix-build-coordinator-account config)
+  (match-record config <guix-build-coordinator-configuration>
+    (user group)
+    (list (user-group
+           (name group)
+           (system? #t))
+          (user-account
+           (name user)
+           (group group)
+           (system? #t)
+           (comment "Guix Build Coordinator user")
+           (home-directory "/var/lib/guix-build-coordinator")
+           (shell (file-append shadow "/sbin/nologin"))))))
+
+(define guix-build-coordinator-service-type
+  (service-type
+   (name 'guix-build-coordinator)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        guix-build-coordinator-shepherd-services)
+     (service-extension activation-service-type
+                        guix-build-coordinator-activation)
+     (service-extension account-service-type
+                        guix-build-coordinator-account)))
+   (default-value
+     (guix-build-coordinator-configuration))
+   (description
+    "Run an instance of the Guix Build Coordinator.")))
+
+
+;;;
+;;; Guix Data Service
+;;;
+
 (define-record-type* <guix-data-service-configuration>
   guix-data-service-configuration make-guix-data-service-configuration
   guix-data-service-configuration?
diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm
index 6139e31cf0..20b67d55d3 100644
--- a/gnu/tests/guix.scm
+++ b/gnu/tests/guix.scm
@@ -35,7 +35,80 @@
   #:use-module (guix store)
   #:use-module (guix utils)
   #:use-module (ice-9 match)
-  #:export (%test-guix-data-service))
+  #:export (%test-guix-build-coordinator
+            %test-guix-data-service))
+
+;;;
+;;; Guix Build Coordinator
+;;;
+
+(define %guix-build-coordinator-os
+  (simple-operating-system
+   (service dhcp-client-service-type)
+   (service guix-build-coordinator-service-type)))
+
+(define (run-guix-build-coordinator-test)
+  (define os
+    (marionette-operating-system
+     %guix-build-coordinator-os
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define forwarded-port 8745)
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     (memory-size 1024)
+     (port-forwardings `((,forwarded-port . 8745)))))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (srfi srfi-11) (srfi srfi-64)
+                       (gnu build marionette)
+                       (web uri)
+                       (web client)
+                       (web response))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (mkdir #$output)
+          (chdir #$output)
+
+          (test-begin "guix-build-coordinator")
+
+          (test-assert "service running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (match (start-service 'guix-build-coordinator)
+                  (#f #f)
+                  (('service response-parts ...)
+                   (match (assq-ref response-parts 'running)
+                     ((pid) (number? pid))))))
+             marionette))
+
+          (test-equal "http-get"
+            200
+            (let-values
+                (((response text)
+                  (http-get #$(simple-format
+                               #f "http://localhost:~A/metrics" forwarded-port)
+                            #:decode-body? #t)))
+              (response-code response)))
+
+          (test-end)
+          (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+  (gexp->derivation "guix-build-coordinator-test" test))
+
+(define %test-guix-build-coordinator
+  (system-test
+   (name "guix-build-coordinator")
+   (description "Connect to a running Guix Build Coordinator.")
+   (value (run-guix-build-coordinator-test))))
 
 
 ;;;
-- 
2.28.0
L
L
Ludovic Courtès wrote on 18 Sep 2020 22:08
Re: [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87lfh6vodj.fsf@gnu.org
Hi!

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (2 lines)
> * gnu/packages/package-management.scm (guix-build-coordinator): New variable.

[...]

Toggle quote (10 lines)
> + (lambda* (#:key inputs outputs #:allow-other-keys)
> + (let* ((out (assoc-ref outputs "out"))
> + (bin (string-append out "/bin"))
> + (guile (assoc-ref inputs "guile"))
> + (guile-effective-version
> + (read-line
> + (open-pipe* OPEN_READ
> + (string-append guile "/bin/guile")
> + "-c" "(display (effective-version))")))

Maybe use ‘target-guile-effective-version’ from (guix build
guile-build-system).

Toggle quote (6 lines)
> + (wrap-program (string-append bin "/" file)
> + `("PATH" ":" prefix
> + (,bin ,(assoc-ref inputs "sqitch")))
> + `("PERL5LIB" ":" prefix
> + (,(getenv "PERL5LIB")))

Do we really need PERL5LIB here? Shouldn’t it be done in the wrapper of
‘sqitch’ directly?

Toggle quote (5 lines)
> + (scandir bin
> + (match-lambda
> + ((or "." "..") #f)
> + (_ #t))))

I think you could just use (find-files bin).

Toggle quote (3 lines)
> + (inputs
> + `(("guile" ,guile-3.0-latest)

Rather ,@(assoc-ref (package-native-inputs guix) "guile").

Toggle quote (3 lines)
> + ("sqitch" ,sqitch)
> + ("perl-dbd-sqlite" ,perl-dbd-sqlite)))

Shouldn’t perl-dbd-sqlite be a dependency of sqitch?

Toggle quote (5 lines)
> + (synopsis "")
> + (description
> + "")

Missing! :-)

LGTM with changes along these lines!

Ludo’.
L
L
Ludovic Courtès wrote on 18 Sep 2020 22:20
Re: [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87a6xmvnth.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (19 lines)
> * gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record
> type.
> (guix-build-coordinator-configuration, guix-build-coordinator-configuration?,
> guix-build-coordinator-configuration-package,
> guix-build-coordinator-configuration-user,
> guix-build-coordinator-configuration-group,
> guix-build-coordinator-configuration-datastore-uri-string,
> guix-build-coordinator-configuration-agent-communication-uri-string,
> guix-build-coordinator-configuration-client-communication-uri-string,
> guix-build-coordinator-configuration-allocation-strategy,
> guix-build-coordinator-configuration-hooks,
> guix-build-coordinator-configuration-guile,
> make-guix-build-coordinator-start-script,
> guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,
> guix-build-coordinator-account): New procedures.
> (guix-build-coordinator-service-type): New variable.
> * gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.
> * doc/guix.texi (Guix Services): Document it.

Yay!

Toggle quote (3 lines)
> +@subsubheading Guix Build Coordinator
> +The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
> +Coordinator} aids in building derivations. The Guix Daemon is still
^
Maybe something like: “in distributing derivation builds among machines
running an @dfn{agent}”.

Also, s/Guix Daemon/build daemon/ or similar.

(In general I’m in favor of avoiding “brands” in documentation and
concept names.)

Maybe add a “@quotation Note” stating that it’s work in progress and
subject to change.

Toggle quote (20 lines)
> +(define* (make-guix-build-coordinator-start-script database-uri-string
> + allocation-strategy
> + pid-file
> + guix-build-coordinator-package
> + #:key
> + agent-communication-uri-string
> + client-communication-uri-string
> + (hooks '())
> + (guile guile-3.0))
> + (program-file
> + "start-guix-build-coordinator"
> + (with-extensions (cons guix-build-coordinator-package
> + ;; This is a poorly constructed Guile load path,
> + ;; since it contains things that aren't Guile
> + ;; libraries, but it means that the Guile libraries
> + ;; needed for the Guix Build Coordinator don't need
> + ;; to be individually specified here.
> + (map second (package-inputs
> + guix-build-coordinator-package)))

Perhaps there should eventually be a ‘guix-build-coordinator’ command in
the package itself?

Otherwise LGTM, thanks!

One thing we discussed on IRC is the name and ways to set up one’s
monitor so that
guix-build-coordinator-configuration-client-communication-uri-string
fits on one line. :-)

An idea that came to mind was “(Guix) dispatch(er)”, which is
synonymous. You could use the (guix dispatch …) name space.
Food for thought! (No rush, of course!)

Ludo’.
L
L
Ludovic Courtès wrote on 18 Sep 2020 22:25
Re: [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87363evnmm.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (20 lines)
> * gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New
> record type.
> (guix-build-coordinator-agent-configuration,
> guix-build-coordinator-agent-configuration?,
> guix-build-coordinator-agent-configuration-package,
> guix-build-coordinator-agent-configuration-user,
> guix-build-coordinator-agent-configuration-coordinator,
> guix-build-coordinator-agent-configuration-uuid),
> guix-build-coordinator-agent-configuration-password,
> guix-build-coordinator-agent-configuration-password-file,
> guix-build-coordinator-agent-configuration-systems,
> guix-build-coordinator-agent-configuration-max-parallel-builds,
> guix-build-coordinator-agent-configuration-derivation-substitute-urls,
> guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,
> guix-build-coordinator-agent-shepherd-services,
> guix-build-coordinator-agent-activation,
> guix-build-coordinator-agent-account): New procedures.
> (guix-build-coordinator-agent-service-type): New variable.
> * doc/guix.texi (Guix Services): Document it.

[...]

Toggle quote (3 lines)
> +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type
> +Service type for a Guix Build Coordinator agent. Its value must be a

Perhaps “coordinator” and “agent” should be defined in a few sentences
above to clarify what this is about.

Toggle quote (8 lines)
> +@item @code{derivation-substitute-urls} (default: @code{1})
> +URLs from which to attempt to fetch substitutes for derivations, if the
> +derivations aren't already available.
> +
> +@item @code{non-derivation-substitute-urls} (default: @code{1})
> +URLs from which to attempt to fetch substitutes for build inputs, if the
> +input store items aren't already available.

This is interesting, I wonder how you can distinguish between the two in
code. You have to open different sessions, right? Or to call
‘set-build-options’ again?

Toggle quote (3 lines)
> + (description
> + "Run an instance of the Guix Build Coordinator.")))

+ “agent”

Otherwise LGTM!

Ludo’.
L
L
Ludovic Courtès wrote on 18 Sep 2020 22:31
Re: [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87pn6iu8rx.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (16 lines)
> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
> New record type.
> (guix-build-coordinator-queue-builds-configuration,
> guix-build-coordinator-queue-builds-configuration?,
> guix-build-coordinator-queue-builds-configuration-package,
> guix-build-coordinator-queue-builds-configuration-user,
> guix-build-coordinator-queue-builds-coordinator,
> guix-build-coordinator-queue-builds-configuration-systems,
> guix-build-coordinator-queue-builds-configuration-system-and-targets,
> guix-build-coordinator-queue-builds-configuration-guix-data-service,
> guix-build-coordinator-queue-builds-configuration-processed-commits-file,
> guix-build-coordinator-queue-builds-shepherd-services,
> guix-build-coordinator-queue-builds-activation,
> guix-build-coordinator-queue-builds-account): New procedures.
> (guix-build-coordinator-queue-builds-service-type): New variable.

[...]

Toggle quote (4 lines)
> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
> +Service type for the
> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its

Oh! :-)

‘guix-build-queue’ maybe? Because after all, it can queue builds coming
from anywhere, not just from the Data Service, right?

It’d be nice to add a sentence or two explaining how the “queue” fits in
the big picture (I’m not quite sure :-)).

Toggle quote (4 lines)
> +@item @code{systems-and-targets} (default: @code{#f})
> +An association list of system and target pairs for which to fetch
> +derivations to build.

Cross-compilation target triplets are a notion that’s not available at
the derivation level; it only exists for packages.

Are we mixing things here?

Toggle quote (4 lines)
> +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})
> +The Guix Data Service instance from which to query to find out about
> +derivations to build.

Ah so the queue pulls from the Data Service. Got it!

Toggle quote (4 lines)
> +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})
> +A file to record which commits have been processed, to avoid needlessly
> +processing them again if the service is restarted.

Maybe in /var/cache by default, no?

Toggle quote (4 lines)
> + (processed-commits-file
> + guix-build-coordinator-queue-builds-configuration-processed-commits-file
> + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))

Maybe “state-file”? In addition to saving space :-), it’d leave room
for additional bits of state.

That’s all, thank you!

Ludo’.
J
J
Jonathan Brielmaier wrote on 19 Sep 2020 01:34
Re: [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
(address . 43494@debbugs.gnu.org)
4332de6e-8002-1967-4838-3448df888242@web.de
On 18.09.20 22:08, Ludovic Courtès wrote:
Toggle quote (2 lines)
C
C
Christopher Baines wrote on 19 Sep 2020 11:00
(name . Jonathan Brielmaier)(address . jonathan.brielmaier@web.de)
87ft7e9m49.fsf@cbaines.net
Jonathan Brielmaier <jonathan.brielmaier@web.de> writes:

Toggle quote (5 lines)
> On 18.09.20 22:08, Ludovic Courtès wrote:
>>> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")
>
> It should be https://git.cbaines.net/guix/build-coordinator/

Thanks Jonathan, I've fixed this now :)
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9lyMZfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9XcELBAApxwnjfJu/B3H9p8cXiZCTl99ejaJXvKfT8Utx4aB4YpFg+ZR7KqlqiJ0
dUr1hXpMzRX31IoUrcIk2mc2DUw5GQjM2lq3mUxESsW8A7ODEU8WfW5KcwLfes4g
7M+jZlPMEJYpfzV2ryvFyBrWErv76u/9d4vcAXIvjIznhEPZbrMyEaW5T0CMUCdz
ULZuBB6PuN7NlV6atYh0Jy7FL83LY5hN+EX0T8ecJ+3pOrlZQWcTJitRsaoCtbvn
nYi3pSQjhuERfb6c4n+lK3NpX4StyMJWW86hF65CsYZhpVsG5xT/kI2R3xf2Ux/h
H4oVrhVxzLIRKPHZEFu2+48O6Pno2QkPam+VYl2Jc6pM5qJuRWrASbXPCu5lhleP
ZLoZPtVqkqIdAZDc3ItUW47hzihpEzsn2a7/rw5c6yh3L53q/edF8jFDYFYF2/xa
j46BG2UKYFpFSvnwUcbJJvNiVpZECcSzFpOYN1kb4z21TuuILL/BTzdEyUcPlH/Q
jVz1lP4ElGAVJPnOltTQYW8syElS0BCc0yhz9KOGeFGJmn3u5TL32ViBJ29p9q2l
R2OitOhLwRZxJwFm64i7+smjkps3DxYEaDkZJnHdA5bGYlk4ojxk7MsDFD42NYh6
kRoBQF7l1kjHXI9PGxFRMI8T9Gri1HZESEo4bVCTnS6BW6TqPc4=
=+xCv
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 19 Sep 2020 11:05
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
87d02i9lx9.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (21 lines)
> Hi!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/packages/package-management.scm (guix-build-coordinator): New variable.
>
> [...]
>
>> + (lambda* (#:key inputs outputs #:allow-other-keys)
>> + (let* ((out (assoc-ref outputs "out"))
>> + (bin (string-append out "/bin"))
>> + (guile (assoc-ref inputs "guile"))
>> + (guile-effective-version
>> + (read-line
>> + (open-pipe* OPEN_READ
>> + (string-append guile "/bin/guile")
>> + "-c" "(display (effective-version))")))
>
> Maybe use ‘target-guile-effective-version’ from (guix build
> guile-build-system).

I've changed this. I'll also try and remember and update the
guix-data-service package accordingly, as that's where I copied this
from.

Toggle quote (9 lines)
>> + (wrap-program (string-append bin "/" file)
>> + `("PATH" ":" prefix
>> + (,bin ,(assoc-ref inputs "sqitch")))
>> + `("PERL5LIB" ":" prefix
>> + (,(getenv "PERL5LIB")))
>
> Do we really need PERL5LIB here? Shouldn’t it be done in the wrapper of
> ‘sqitch’ directly?

Hmm, when I packaged sqitch, I didn't add all the database drivers which
are optional dependencies, but perl-dbd-pg did end up in there somehow
(probably because that's needed by the guix-data-service).

I've now gone ahead and added the database drivers which Guix has
packages for [1], it does increase the closure size a bit, but it was
quite big anyway.

1:

All that means I can remove the wrapping here.

Toggle quote (7 lines)
>> + (scandir bin
>> + (match-lambda
>> + ((or "." "..") #f)
>> + (_ #t))))
>
> I think you could just use (find-files bin).

Done.

Toggle quote (5 lines)
>> + (inputs
>> + `(("guile" ,guile-3.0-latest)
>
> Rather ,@(assoc-ref (package-native-inputs guix) "guile").

Done.

Toggle quote (5 lines)
>> + ("sqitch" ,sqitch)
>> + ("perl-dbd-sqlite" ,perl-dbd-sqlite)))
>
> Shouldn’t perl-dbd-sqlite be a dependency of sqitch?

See above.

Toggle quote (7 lines)
>> + (synopsis "")
>> + (description
>> + "")
>
> Missing! :-)

Ah, yeah, I should have remembered to lint the package. I've attempted
to write something now, I'm still working on trying to explain what the
Guix Build Coordinator is meant to be.

Toggle quote (2 lines)
> LGTM with changes along these lines!

Thanks!
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9lycJfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9XdAdQ/8CgG11hrFpUwT8jxkToEexip5q3Gxt9MwtUK7nEmN6YmiRKO+5aEiWbGl
rbLvKuxgtg673rb5+Th3LTUcprxlgh3SGFIXINHHqy3ybzyGm2vOFBDlZ1I8rbsS
Hr195bVVHsirWQJzqPk0c4ukfaxEKc9V/QFa+VllJs/LlTobbuo8oqwkQ5wC6wUJ
O+ZTJrXZpjDclFXGNGDEGGXJ2KnhxG7CMLt/SrgTrMy/0tpLyHsPYP1OXOOe7YQC
dDXAx5zN3EkNSOCGWygi2QGiocRunVKJzhAWJr5WE62Vzn9/XxbuMxECs30kQuza
AVKHA5q5hCL2kAFm80goM8nmdYzxZ/tMENpgySCb5FF1pcGQVpLjlsodao4AHPl9
c2x3cgr4DB8wiNx26O5ENEuR1vfInwlQVQz8zvd9tseD8BRh4bk+3f28ZXWs/VDz
ZGCQ67Vp83+3poAV12+ScHvFu9MNX3jwICEHtISNnhtYlw8GS+icOpe1pmFCf4D6
4kxyXmj595OcZjv2xfVUso32XccTmTl9+c52Djj+SLlu91mJ20jJt/flxvv7uK6u
T5SwfpHX+RThnJMFHjjwoFPV7CsyuTb9sfKDjgrf7hMepHYuX7jGqDjjrMMoXvco
4/0YlIIKP+gR3OvjTyg+YZVP/Fulbjm7xkgkHNZPxS6bXQJrKho=
=hrmM
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 19 Sep 2020 11:13
Re: [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
87a6xm9lih.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (38 lines)
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record
>> type.
>> (guix-build-coordinator-configuration, guix-build-coordinator-configuration?,
>> guix-build-coordinator-configuration-package,
>> guix-build-coordinator-configuration-user,
>> guix-build-coordinator-configuration-group,
>> guix-build-coordinator-configuration-datastore-uri-string,
>> guix-build-coordinator-configuration-agent-communication-uri-string,
>> guix-build-coordinator-configuration-client-communication-uri-string,
>> guix-build-coordinator-configuration-allocation-strategy,
>> guix-build-coordinator-configuration-hooks,
>> guix-build-coordinator-configuration-guile,
>> make-guix-build-coordinator-start-script,
>> guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,
>> guix-build-coordinator-account): New procedures.
>> (guix-build-coordinator-service-type): New variable.
>> * gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.
>> * doc/guix.texi (Guix Services): Document it.
>
> Yay!
>
>> +@subsubheading Guix Build Coordinator
>> +The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
>> +Coordinator} aids in building derivations. The Guix Daemon is still
> ^
> Maybe something like: “in distributing derivation builds among machines
> running an @dfn{agent}”.
>
> Also, s/Guix Daemon/build daemon/ or similar.
>
> (In general I’m in favor of avoiding “brands” in documentation and
> concept names.)
>
> Maybe add a “@quotation Note” stating that it’s work in progress and
> subject to change.

I've made these changes now.

Toggle quote (23 lines)
>> +(define* (make-guix-build-coordinator-start-script database-uri-string
>> + allocation-strategy
>> + pid-file
>> + guix-build-coordinator-package
>> + #:key
>> + agent-communication-uri-string
>> + client-communication-uri-string
>> + (hooks '())
>> + (guile guile-3.0))
>> + (program-file
>> + "start-guix-build-coordinator"
>> + (with-extensions (cons guix-build-coordinator-package
>> + ;; This is a poorly constructed Guile load path,
>> + ;; since it contains things that aren't Guile
>> + ;; libraries, but it means that the Guile libraries
>> + ;; needed for the Guix Build Coordinator don't need
>> + ;; to be individually specified here.
>> + (map second (package-inputs
>> + guix-build-coordinator-package)))
>
> Perhaps there should eventually be a ‘guix-build-coordinator’ command in
> the package itself?

There actually is, one thing I've had in mind for a while now though is
to use a scheme script constructed by the Guix service to run the
coordinator.

For guix.cbaines.net, I'm using the script, but with the hooks passed in
on the command line, the command is rather long, and it means that
backtraces don't work well with the hooks.

I'm unsure how well this has worked out, I didn't anticipate the issues
with the Guile load path, and because of the formatting stripping, both
the code and backtraces will be unreadable anyway... but it is useful in
allowing G-expressions to be used as part of the configuration.

Toggle quote (9 lines)
> One thing we discussed on IRC is the name and ways to set up one’s
> monitor so that
> guix-build-coordinator-configuration-client-communication-uri-string
> fits on one line. :-)
>
> An idea that came to mind was “(Guix) dispatch(er)”, which is
> synonymous. You could use the (guix dispatch …) name space.
> Food for thought! (No rush, of course!)

Indeed, I think that's a strong candidate :)
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9ly9ZfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9XdVWg/+KZ3LZDolesoD7M43Bg2HBcI4SB2nou7+rT5iiYXgcX17kZjX/PG6SWbS
DgdfJ+su/n59Hc9galUyZTLkbr52Yl8gByj35CQmWJJ0SJPzzvEdml6Ng7/KPOMr
LN1YHvFxVfhXn+tZbExxV98LBGqWIualpoUd2I7qJvrA+11QAUXyLOybkhCk/lJz
rrJRCjKoXP4mjxVBj6EgFB1s6CNXPI97kmJJvEHEp1V4kC8rym3kOkfYiuaZAmnZ
aZil9CslNLOtFBhiheXaZWgGzvzx7ifFe/7oV63zfghg/AzoPwHVjvIdRrly7b1t
eFJ5O3E3OOawXMAMV/K1+rPx/eTHaVyufex27wZcZpEv1UkX/7oMv6/XSVp0gOOD
UpmDCNwSAmrvApE+rh1LowGIDBHYBa+P03RZ+4DYH68jT87C/O3Vz9WbDDTbPow9
KanFLJbR80sYaD/xO5Ni8cZ+o6DmxrjLPv55EdLqds1TQ2Bp7x2AG4Ki3VoMrFol
H94YCS99hck5gXghZJZpsrUaclSDBUDET541OrBFtctfG8WksstGovRMnWO4u2wO
3JjQJTTdfr4yAqOVWUi3PE1JdZB1aZENE4XgZeNPBZCKVNt5noAV0OH/SN66noDk
od5DslxrE/KUmgagTiEWOAl6noCmIRH+jSBrgqUJrGMSL93HwTM=
=tSTq
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 19 Sep 2020 11:49
Re: [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
875z8a9jvz.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (30 lines)
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New
>> record type.
>> (guix-build-coordinator-agent-configuration,
>> guix-build-coordinator-agent-configuration?,
>> guix-build-coordinator-agent-configuration-package,
>> guix-build-coordinator-agent-configuration-user,
>> guix-build-coordinator-agent-configuration-coordinator,
>> guix-build-coordinator-agent-configuration-uuid),
>> guix-build-coordinator-agent-configuration-password,
>> guix-build-coordinator-agent-configuration-password-file,
>> guix-build-coordinator-agent-configuration-systems,
>> guix-build-coordinator-agent-configuration-max-parallel-builds,
>> guix-build-coordinator-agent-configuration-derivation-substitute-urls,
>> guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,
>> guix-build-coordinator-agent-shepherd-services,
>> guix-build-coordinator-agent-activation,
>> guix-build-coordinator-agent-account): New procedures.
>> (guix-build-coordinator-agent-service-type): New variable.
>> * doc/guix.texi (Guix Services): Document it.
>
> [...]
>
>> +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type
>> +Service type for a Guix Build Coordinator agent. Its value must be a
>
> Perhaps “coordinator” and “agent” should be defined in a few sentences
> above to clarify what this is about.

I've added a bit more information, I want to at some point actually give
a workable minimal example configuration, but I need to do some more
thinking about that.

Toggle quote (12 lines)
>> +@item @code{derivation-substitute-urls} (default: @code{1})
>> +URLs from which to attempt to fetch substitutes for derivations, if the
>> +derivations aren't already available.
>> +
>> +@item @code{non-derivation-substitute-urls} (default: @code{1})
>> +URLs from which to attempt to fetch substitutes for build inputs, if the
>> +input store items aren't already available.
>
> This is interesting, I wonder how you can distinguish between the two in
> code. You have to open different sessions, right? Or to call
> ‘set-build-options’ again?

Yeah, it's not strictly necessary to have some separation. In fact, I
can't remember exactly, but I think it should be possible to deploy the
coordinator and agents without explicit configuration of the substitute
URLs, but I need to check that.

These options were added mostly because it useful if you have two
seperate sources of substitutes, one for just derivations (say an
instance of the Guix Data Service), and one just for build outputs (say
populated by one of the hooks) [1].


Toggle quote (7 lines)
>> + (description
>> + "Run an instance of the Guix Build Coordinator.")))
>
> + “agent”
>
> Otherwise LGTM!

Thanks!
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9l1BBfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9XdqNA/+PAeN8zL9P608PFjf1IDPZZWx76WSAMZ6BfCStsRvGNR/X+ZOgzabH6+K
ZiJoAbvC0RIzFgGh90CfYI2GabG6SYTkQMAi4DN/Cs3iKFVG+DDvdpprmk9xDTFb
k+MnarQzw0sSidtZb8OL9E9u+c3evfDO6AMl9y2vKhfGDV6A5IvIcNJy1t8TM8OL
S/qcd8O2kIlLX2LBKoGJuhiszXt1MFYrXmbSLDEd+iZ0wcFXHaG2TV/y5UbqydP3
0MaCKAym2YN5NNK98u+z94TImZeppCeQGGY5OnN0Cq29TaIn8egDX1NJjFkV4OzT
+WwM2jY14oFvOZHblS5uqd0gd3I5yi/NX41K6CFjAOX43ADlpQSeLF6CtPZAJbWS
wNn+3pmBdtyuRJibe0GbtaAn5+0CvJ1L58gHzXIp22Vaz+yW5qoAVEOMuEx0a6ft
uMJ6RzjW9FGtvNzuyzcvCbeMmB5Wc3IuoTiPA6ZHBuj1KvdDBwD32zskPwUKk3Cu
1G7KBkHmAMOF28Z/Ci0Lr1vewKW6iR+6pIiXD6gxRiaGLZaz/8Ck3eweYa+llidU
YBkz/UzGMZggWegnABCMv+6bTRgDTwVFTH9QNlBmx1gJRYayx8+xclGNR/njcHH5
VR93AvkYiHmXPnpEGeXncGJwN8jTFoW4UP5Mb1BPT1geWj8xmzQ=
=RwqV
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 19 Sep 2020 12:05
Re: [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
87363e9j3u.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (32 lines)
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
>> New record type.
>> (guix-build-coordinator-queue-builds-configuration,
>> guix-build-coordinator-queue-builds-configuration?,
>> guix-build-coordinator-queue-builds-configuration-package,
>> guix-build-coordinator-queue-builds-configuration-user,
>> guix-build-coordinator-queue-builds-coordinator,
>> guix-build-coordinator-queue-builds-configuration-systems,
>> guix-build-coordinator-queue-builds-configuration-system-and-targets,
>> guix-build-coordinator-queue-builds-configuration-guix-data-service,
>> guix-build-coordinator-queue-builds-configuration-processed-commits-file,
>> guix-build-coordinator-queue-builds-shepherd-services,
>> guix-build-coordinator-queue-builds-activation,
>> guix-build-coordinator-queue-builds-account): New procedures.
>> (guix-build-coordinator-queue-builds-service-type): New variable.
>
> [...]
>
>> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
>> +Service type for the
>> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its
>
> Oh! :-)
>
> ‘guix-build-queue’ maybe? Because after all, it can queue builds coming
> from anywhere, not just from the Data Service, right?
>
> It’d be nice to add a sentence or two explaining how the “queue” fits in
> the big picture (I’m not quite sure :-)).

So, this got me thinking a bit, maybe this script should actually exist
outside of the guix-build-coordinator repository...

You can run `guix-build-coordinator build ...` to build a derivation,
and what this does is just make the relevant HTTP request to the
coordinator.

What this script does is just fetch derivation names from an instance of
the Guix Data Service, and ask the Guix Build Coordinator to build
them.

Toggle quote (9 lines)
>> +@item @code{systems-and-targets} (default: @code{#f})
>> +An association list of system and target pairs for which to fetch
>> +derivations to build.
>
> Cross-compilation target triplets are a notion that’s not available at
> the derivation level; it only exists for packages.
>
> Are we mixing things here?

This setting is basically configuing the query parameters to use for
this page for example [1]. You're right that it's not available at the
derivation level, but you can ask the Guix Data Service for derivations
for packages which were computed for a specific system and target.


Toggle quote (6 lines)
>> +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})
>> +The Guix Data Service instance from which to query to find out about
>> +derivations to build.
>
> Ah so the queue pulls from the Data Service. Got it!

I wouldn't read too much in to "queue" in the name here, "submit" might
be a better word to use, since that's the term used on the Guix Build
Coordinator side. There's also nothing resemling a queue anywhere...

Toggle quote (6 lines)
>> +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})
>> +A file to record which commits have been processed, to avoid needlessly
>> +processing them again if the service is restarted.
>
> Maybe in /var/cache by default, no?

Ah, yeah, I've changed it to be in /var/cache, as that's fine.

Toggle quote (7 lines)
>> + (processed-commits-file
>> + guix-build-coordinator-queue-builds-configuration-processed-commits-file
>> + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))
>
> Maybe “state-file”? In addition to saving space :-), it’d leave room
> for additional bits of state.

Currently, it's just a file with each commit on its own line. Given
that, if there's a need to store more state, it might just end up in
different files. If that happens, this setting could become a directory,
rather than a specific file to avoid adding extra configuration options.
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9l2AVfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9XdCjQ/8DIg3yovjZeLdZteOJnB4JMRyFhx+VTZPbTVEDJcT8BL8gwqiZrVKsiLW
Cz+vu62Bwiohk0zrXKLTuqTfsvq/NGnqWm4lt+cz6OiH8guX5CfvuXeLNXWh0t6S
PA6O8ieRry/ar+DFQBUEF20XNUIb/hCGhykcdjKC1vLFac4npBaYxRyIwCwiyMLG
PrugkmFveh93Xe9E7qqByT6DEPa22zw44zDrzRFWUP3Zbpv+fFuA+MwDg/BjS7lf
xGuc0tfI8XVBWwsbnmdMUUNuHneBf5mYRhMDBFqjP+ZrWDq8Z2PHyIUFvie9iaC1
phgwQvjUwXe5w2H5k0rav+izwyjYe2Lg8GgBZsn2j9DV0FlRq3AhM0Ex96wCs6js
xbDcAKeuKqaLKwbBoRd40uo+0Aoo5FBo7lVije0M3S4JaQjqGPxqR1HSCPz8VMxo
XoKzq36kmZQEly8XLgSf/lWFLkVvEgLvBR4g5bFMVLKxKEK2lHSEa1jfxDc1Lz6V
6/4qVF/bMN8SUqU7ZvRr6klUUbQ2OaTV5Csl5fG1jlZNWDXOTlEKXcgUq+jbU0QE
CoM1OED+rppjSXe1zbaubWZKKXk1Vq3Ix7uAIg3G7fwd0rD5FPjTXEdqtAZ4KLOO
REYJt6fooebFrPdxqPOGEJQ8IHXBvlOK305U3/GplY2iP70kxrg=
=d1v4
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 19 Sep 2020 12:10
[PATCH v2 1/4] gnu: Add guix-build-coordinator.
(address . 43494@debbugs.gnu.org)
20200919101059.8930-1-mail@cbaines.net
* gnu/packages/package-management.scm (guix-build-coordinator): New variable.
---
gnu/packages/package-management.scm | 74 +++++++++++++++++++++++++++++
1 file changed, 74 insertions(+)

Toggle diff (94 lines)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index ec87226197..afa45d6789 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -50,6 +50,7 @@
   #:use-module (gnu packages cpio)
   #:use-module (gnu packages crypto)
   #:use-module (gnu packages curl)
+  #:use-module (gnu packages databases)
   #:use-module (gnu packages dbm)
   #:use-module (gnu packages docbook)
   #:use-module (gnu packages file)
@@ -981,6 +982,79 @@ environments.")
     ;; and the fonts included in this package are licensed OFL1.1.
     (license (list license:gpl3+ license:agpl3+ license:silofl1.1))))
 
+(define-public guix-build-coordinator
+  (let ((commit "5e8408c833e209efddfa0159114b90400c1aaf4d")
+        (revision "0"))
+    (package
+    (name "guix-build-coordinator")
+    (version (git-version "0" revision commit))
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://git.cbaines.net/git/guix/build-coordinator")
+                    (commit commit)))
+              (sha256
+               (base32
+                "0f7m1zg9mlb2m22qyblglaa36h8f49b810jc9j5b0hsb42ijwh4b"))
+              (file-name (string-append name "-" version "-checkout"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:modules (((guix build guile-build-system)
+                   #:select (target-guile-effective-version))
+                  ,@%gnu-build-system-modules)
+       #:imported-modules ((guix build guile-build-system)
+                           ,@%gnu-build-system-modules)
+       #:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'set-GUILE_AUTO_COMPILE
+           (lambda _
+             ;; To avoid warnings relating to 'guild'.
+             (setenv "GUILE_AUTO_COMPILE" "0")
+             #t))
+         (add-after 'install 'wrap-executable
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (guile (assoc-ref inputs "guile"))
+                    (version (target-guile-effective-version))
+                    (scm (string-append out "/share/guile/site/" version))
+                    (go  (string-append out "/lib/guile/" version "/site-ccache")))
+               (for-each
+                (lambda (file)
+                  (simple-format (current-error-port) "wrapping: ~A\n" file)
+                  (wrap-program file
+                    `("PATH" ":" prefix
+                      (,bin ,(assoc-ref inputs "sqitch")))
+                    `("GUILE_LOAD_PATH" ":" prefix
+                      (,scm ,(getenv "GUILE_LOAD_PATH")))
+                    `("GUILE_LOAD_COMPILED_PATH" ":" prefix
+                      (,go ,(getenv "GUILE_LOAD_COMPILED_PATH")))))
+                (find-files bin)))
+             #t))
+         (delete 'strip))))             ; As the .go files aren't compatible
+    (native-inputs
+     `(("pkg-config" ,pkg-config)
+       ("autoconf" ,autoconf)
+       ("automake" ,automake)))
+    (inputs
+     `(("guile" ,@(assoc-ref (package-native-inputs guix) "guile"))
+       ("guile-fibers" ,guile-fibers)
+       ("guile-prometheus" ,guile-prometheus)
+       ("guile-gcrypt" ,guile-gcrypt)
+       ("guile-json" ,guile-json-3)
+       ("guile-lzlib" ,guile-lzlib)
+       ("guile-sqlite3" ,guile-sqlite3)
+       ("guix" ,guix)
+       ("sqlite" ,sqlite)
+       ("sqitch" ,sqitch)))
+    (home-page "https://git.cbaines.net/guix/build-coordinator/")
+    (synopsis "Tool to help build derivations")
+    (description
+     "The Guix Build Coordinator helps with performing lots of builds across
+potentially many machines, and with doing something with the results and
+outputs of those builds.")
+    (license license:gpl3+))))
+
 (define-public guix-jupyter
   (package
     (name "guix-jupyter")
-- 
2.28.0
C
C
Christopher Baines wrote on 19 Sep 2020 12:10
[PATCH v2 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
(address . 43494@debbugs.gnu.org)
20200919101059.8930-3-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New
record type.
(guix-build-coordinator-agent-configuration,
guix-build-coordinator-agent-configuration?,
guix-build-coordinator-agent-configuration-package,
guix-build-coordinator-agent-configuration-user,
guix-build-coordinator-agent-configuration-coordinator,
guix-build-coordinator-agent-configuration-uuid),
guix-build-coordinator-agent-configuration-password,
guix-build-coordinator-agent-configuration-password-file,
guix-build-coordinator-agent-configuration-systems,
guix-build-coordinator-agent-configuration-max-parallel-builds,
guix-build-coordinator-agent-configuration-derivation-substitute-urls,
guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,
guix-build-coordinator-agent-shepherd-services,
guix-build-coordinator-agent-activation,
guix-build-coordinator-agent-account): New procedures.
(guix-build-coordinator-agent-service-type): New variable.
* doc/guix.texi (Guix Services): Document it.
---
doc/guix.texi | 50 +++++++++++++++++
gnu/services/guix.scm | 121 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 171 insertions(+)

Toggle diff (209 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 4f3f0e851f..b2dde30ddb 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27554,6 +27554,56 @@ The Guile package with which to run the Guix Build Coordinator.
 @end table
 @end deftp
 
+@defvar {Scheme Variable} guix-build-coordinator-agent-service-type
+Service type for a Guix Build Coordinator agent.  Its value must be a
+@code{guix-build-coordinator-agent-configuration} object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-agent-configuration
+Data type representing the configuration a Guix Build Coordinator agent.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator-agent"})
+The system user to run the service as.
+
+@item @code{coordinator} (default: @code{"http://localhost:8745"})
+The URI to use when connecting to the coordinator.
+
+@item @code{uuid}
+The UUID of the agent.  This should be generated by the coordinator
+process, stored in the coordinator database, and used by the intended
+agent.
+
+@item @code{password} (default: @code{#f})
+The password to use when connecting to the coordinator.  A file to read
+the password from can also be specified, and this is more secure.
+
+@item @code{password-file} (default: @code{#f})
+A file containing the password to use when connecting to the
+coordinator.
+
+@item @code{systems} (default: @var{#f})
+The systems for which this agent should fetch builds.  The agent process
+will use the current system it's running on as the default.
+
+@item @code{max-parallel-builds} (default: @code{1})
+The number of builds to perform in parallel.
+
+@item @code{derivation-substitute-urls} (default: @code{1})
+URLs from which to attempt to fetch substitutes for derivations, if the
+derivations aren't already available.
+
+@item @code{non-derivation-substitute-urls} (default: @code{1})
+URLs from which to attempt to fetch substitutes for build inputs, if the
+input store items aren't already available.
+
+@end table
+@end deftp
+
+
 @subsubheading Guix Data Service
 The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
 and provides data about GNU Guix.  This includes information about
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 1bacd61190..e4f5cf57b9 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -50,6 +50,21 @@
 
             guix-build-coordinator-service-type
 
+            guix-build-coordinator-agent-configuration
+            guix-build-coordinator-agent-configuration?
+            guix-build-coordinator-agent-configuration-package
+            guix-build-coordinator-agent-configuration-user
+            guix-build-coordinator-agent-configuration-coordinator
+            guix-build-coordinator-agent-configuration-uuid
+            guix-build-coordinator-agent-configuration-password
+            guix-build-coordinator-agent-configuration-password-file
+            guix-build-coordinator-agent-configuration-systems
+            guix-build-coordinator-agent-configuration-max-parallel-builds
+            guix-build-coordinator-agent-configuration-derivation-substitute-urls
+            guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
+
+            guix-build-coordinator-agent-service-type
+
             <guix-data-service-configuration>
             guix-data-service-configuration
             guix-data-service-configuration?
@@ -95,6 +110,33 @@
   (guile                           guix-build-coordinator-configuration-guile
                                    (default guile-3.0-latest)))
 
+(define-record-type* <guix-build-coordinator-agent-configuration>
+  guix-build-coordinator-agent-configuration
+  make-guix-build-coordinator-agent-configuration
+  guix-build-coordinator-agent-configuration?
+  (package             guix-build-coordinator-agent-configuration-package
+                       (default guix-build-coordinator))
+  (user                guix-build-coordinator-agent-configuration-user
+                       (default "guix-build-coordinator-agent"))
+  (coordinator         guix-build-coordinator-agent-configuration-coordinator
+                       (default "http://localhost:8745"))
+  (uuid                guix-build-coordinator-agent-configuration-uuid)
+  (password            guix-build-coordinator-agent-configuration-password
+                       (default #f))
+  (password-file       guix-build-coordinator-agent-configuration-password-file
+                       (default #f))
+  (systems             guix-build-coordinator-agent-configuration-systems
+                       (default #f))
+  (max-parallel-builds
+   guix-build-coordinator-agent-configuration-max-parallel-builds
+   (default 1))
+  (derivation-substitute-urls
+   guix-build-coordinator-agent-configuration-derivation-substitute-urls
+   (default #f))
+  (non-derivation-substitute-urls
+   guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
+   (default #f)))
+
 (define* (make-guix-build-coordinator-start-script database-uri-string
                                                    allocation-strategy
                                                    pid-file
@@ -239,6 +281,85 @@
    (description
     "Run an instance of the Guix Build Coordinator.")))
 
+(define (guix-build-coordinator-agent-shepherd-services config)
+  (match-record config <guix-build-coordinator-agent-configuration>
+    (package user coordinator uuid password password-file max-parallel-builds
+             derivation-substitute-urls non-derivation-substitute-urls
+             systems)
+    (list
+     (shepherd-service
+      (documentation "Guix Build Coordinator Agent")
+      (provision '(guix-build-coordinator-agent))
+      (requirement '(networking))
+      (start #~(make-forkexec-constructor
+                (list #$(file-append package "/bin/guix-build-coordinator-agent")
+                      #$(string-append "--coordinator=" coordinator)
+                      #$(string-append "--uuid=" uuid)
+                      #$@(if password
+                             #~(#$(string-append "--password=" password))
+                             #~())
+                      #$@(if password-file
+                             #~(#$(string-append "--password-file=" password-file))
+                             #~())
+                      #$(simple-format #f "--max-parallel-builds=~A"
+                                       max-parallel-builds)
+                      #$@(if derivation-substitute-urls
+                             #~(#$(string-append
+                                   "--derivation-substitute-urls="
+                                 (string-join derivation-substitute-urls " ")))
+                             #~())
+                      #$@(if non-derivation-substitute-urls
+                             #~(#$(string-append
+                                   "--non-derivation-substitute-urls="
+                                   (string-join derivation-substitute-urls " ")))
+                             #~())
+                      #$@(map (lambda (system)
+                                (string-append "--system=" system))
+                              (or systems '())))
+                #:user #$user
+                #:pid-file "/var/run/guix-build-coordinator-agent/pid"
+                #:environment-variables
+                `(,(string-append
+                    "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+                  "LC_ALL=en_US.utf8")
+                #:log-file "/var/log/guix-build-coordinator/agent.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-agent-activation config)
+  #~(begin
+      (use-modules (guix build utils))
+
+      (mkdir-p "/var/log/guix-build-coordinator")
+
+      ;; Allow writing the PID file
+      (mkdir-p "/var/run/guix-build-coordinator-agent")
+      (chown "/var/run/guix-build-coordinator-agent"
+             (passwd:uid %user)
+             (passwd:gid %user))))
+
+(define (guix-build-coordinator-agent-account config)
+  (list (user-account
+         (name (guix-build-coordinator-agent-configuration-user config))
+         (group "nogroup")
+         (system? #t)
+         (comment "Guix Build Coordinator agent user")
+         (home-directory "/var/empty")
+         (shell (file-append shadow "/sbin/nologin")))))
+
+(define guix-build-coordinator-agent-service-type
+  (service-type
+   (name 'guix-build-coordinator-agent)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        guix-build-coordinator-agent-shepherd-services)
+     (service-extension activation-service-type
+                        guix-build-coordinator-agent-activation)
+     (service-extension account-service-type
+                        guix-build-coordinator-agent-account)))
+   (description
+    "Run a Guix Build Coordinator agent.")))
+
 
 ;;;
 ;;; Guix Data Service
-- 
2.28.0
C
C
Christopher Baines wrote on 19 Sep 2020 12:10
[PATCH v2 2/4] services: guix: Add guix-build-coordinator-service-type.
(address . 43494@debbugs.gnu.org)
20200919101059.8930-2-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record
type.
(guix-build-coordinator-configuration, guix-build-coordinator-configuration?,
guix-build-coordinator-configuration-package,
guix-build-coordinator-configuration-user,
guix-build-coordinator-configuration-group,
guix-build-coordinator-configuration-datastore-uri-string,
guix-build-coordinator-configuration-agent-communication-uri-string,
guix-build-coordinator-configuration-client-communication-uri-string,
guix-build-coordinator-configuration-allocation-strategy,
guix-build-coordinator-configuration-hooks,
guix-build-coordinator-configuration-guile,
make-guix-build-coordinator-start-script,
guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,
guix-build-coordinator-account): New procedures.
(guix-build-coordinator-service-type): New variable.
* gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.
* doc/guix.texi (Guix Services): Document it.
---
doc/guix.texi | 67 ++++++++++++++
gnu/services/guix.scm | 200 +++++++++++++++++++++++++++++++++++++++++-
gnu/tests/guix.scm | 75 +++++++++++++++-
3 files changed, 338 insertions(+), 4 deletions(-)

Toggle diff (400 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index f7e2204b53..4f3f0e851f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27487,6 +27487,73 @@ The complete list of possible options can be found in the man page for
 @node Guix Services
 @subsection Guix Services
 
+@subsubheading Guix Build Coordinator
+The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
+Coordinator} aids in distributing derivation builds among machines
+running an @dfn{agent}.  The build daemon is still used to build the
+derivations, but the Guix Build Coordinator manages allocating builds
+and working with the results.
+
+@quotation Note
+This service is considered experimental.  Configuration options may be
+changed in a backwards-incompatible manner, and not all features have
+been thorougly tested.
+@end quotation
+
+The Guix Build Coordinator consists of one @dfn{coordinator}, and one or
+more connected @dfn{agent} processes. The coordinator process handles
+clients submitting builds, and allocating builds to agents. The agent
+processes talk to a build daemon to actually perform the builds, then
+send the results back to the coordinator.
+
+There is a script to run the coordinator component of the Guix Build
+Coordinator, but the Guix service uses a custom Guile script instead, to
+provide better integration with G-expressions used in the configuration.
+
+@defvar {Scheme Variable} guix-build-coordinator-service-type
+Service type for the Guix Build Coordinator.  Its value must be a
+@code{guix-build-coordinator-configuration} object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-configuration
+Data type representing the configuration of the Guix Build Coordinator.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator"})
+The system user to run the service as.
+
+@item @code{group} (default: @code{"guix-build-coordinator"})
+The system group to run the service as.
+
+@item @code{database-uri-string} (default: @code{"sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"})
+The URI to use for the database.
+
+@item @code{agent-communication-uri} (default: @code{"http://0.0.0.0:8745"})
+The URI describing how to listen to requests from agent processes.
+
+@item @code{client-communication-uri} (default: @code{"http://127.0.0.1:8746"})
+The URI describing how to listen to requests from clients.  The client
+API allows submitting builds and currently isn't authenticated, so take
+care when configuring this value.
+
+@item @code{allocation-strategy} (default: @code{#~basic-build-allocation-strategy})
+A G-expression for the allocation strategy to be used.  This is a
+procedure that takes the datastore as an argument and populates the
+allocation plan in the database.
+
+@item @code{hooks} (default: @var{'()})
+An association list of hooks.  These provide a way to execute arbitrary
+code upon certian events, like a build result being processed.
+
+@item @code{guile} (default: @code{guile-3.0-latest})
+The Guile package with which to run the Guix Build Coordinator.
+
+@end table
+@end deftp
+
 @subsubheading Guix Data Service
 The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
 and provides data about GNU Guix.  This includes information about
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 10a8581a62..1bacd61190 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -17,20 +17,40 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (gnu services guix)
+  #:use-module (srfi srfi-1)
   #:use-module (ice-9 match)
   #:use-module (guix gexp)
   #:use-module (guix records)
+  #:use-module (guix packages)
   #:use-module ((gnu packages base)
                 #:select (glibc-utf8-locales))
   #:use-module (gnu packages admin)
+  #:use-module (gnu packages databases)
   #:use-module (gnu packages web)
+  #:use-module (gnu packages guile)
+  #:use-module (gnu packages guile-xyz)
+  #:use-module (gnu packages package-management)
   #:use-module (gnu services)
   #:use-module (gnu services base)
   #:use-module (gnu services admin)
   #:use-module (gnu services shepherd)
   #:use-module (gnu services getmail)
   #:use-module (gnu system shadow)
-  #:export (<guix-data-service-configuration>
+  #:export (guix-build-coordinator-configuration
+            guix-build-coordinator-configuration?
+            guix-build-coordinator-configuration-package
+            guix-build-coordinator-configuration-user
+            guix-build-coordinator-configuration-group
+            guix-build-coordinator-configuration-datastore-uri-string
+            guix-build-coordinator-configuration-agent-communication-uri-string
+            guix-build-coordinator-configuration-client-communication-uri-string
+            guix-build-coordinator-configuration-allocation-strategy
+            guix-build-coordinator-configuration-hooks
+            guix-build-coordinator-configuration-guile
+
+            guix-build-coordinator-service-type
+
+            <guix-data-service-configuration>
             guix-data-service-configuration
             guix-data-service-configuration?
             guix-data-service-package
@@ -45,11 +65,185 @@
 
 ;;;; Commentary:
 ;;;
-;;; This module implements a service that to run instances of the Guix Data
-;;; Service, which provides data about Guix over time.
+;;; Services specifically related to GNU Guix.
 ;;;
 ;;;; Code:
 
+(define-record-type* <guix-build-coordinator-configuration>
+  guix-build-coordinator-configuration make-guix-build-coordinator-configuration
+  guix-build-coordinator-configuration?
+  (package                         guix-build-coordinator-configuration-package
+                                   (default guix-build-coordinator))
+  (user                            guix-build-coordinator-configuration-user
+                                   (default "guix-build-coordinator"))
+  (group                           guix-build-coordinator-configuration-group
+                                   (default "guix-build-coordinator"))
+  (database-uri-string
+   guix-build-coordinator-configuration-datastore-uri-string
+   (default "sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"))
+  (agent-communication-uri-string
+   guix-build-coordinator-configuration-agent-communication-uri-string
+   (default "http://0.0.0.0:8745"))
+  (client-communication-uri-string
+   guix-build-coordinator-configuration-client-communication-uri-string
+   (default "http://127.0.0.1:8746"))
+  (allocation-strategy
+   guix-build-coordinator-configuration-allocation-strategy
+   (default #~basic-build-allocation-strategy))
+  (hooks                           guix-build-coordinator-configuration-hooks
+                                   (default '()))
+  (guile                           guix-build-coordinator-configuration-guile
+                                   (default guile-3.0-latest)))
+
+(define* (make-guix-build-coordinator-start-script database-uri-string
+                                                   allocation-strategy
+                                                   pid-file
+                                                   guix-build-coordinator-package
+                                                   #:key
+                                                   agent-communication-uri-string
+                                                   client-communication-uri-string
+                                                   (hooks '())
+                                                   (guile guile-3.0))
+  (program-file
+   "start-guix-build-coordinator"
+   (with-extensions (cons guix-build-coordinator-package
+                          ;; This is a poorly constructed Guile load path,
+                          ;; since it contains things that aren't Guile
+                          ;; libraries, but it means that the Guile libraries
+                          ;; needed for the Guix Build Coordinator don't need
+                          ;; to be individually specified here.
+                          (map second (package-inputs
+                                       guix-build-coordinator-package)))
+     #~(begin
+         (use-modules (srfi srfi-1)
+                      (ice-9 match)
+                      (web uri)
+                      (prometheus)
+                      (guix-build-coordinator hooks)
+                      (guix-build-coordinator datastore)
+                      (guix-build-coordinator build-allocator)
+                      (guix-build-coordinator coordinator))
+
+         (let* ((metrics-registry (make-metrics-registry
+                                   #:namespace
+                                   "guixbuildcoordinator_"))
+                (datastore (database-uri->datastore
+                            #$database-uri-string
+                            #:metrics-registry metrics-registry))
+                (hooks
+                 (list #$@(map (match-lambda
+                                 ((name . hook-gexp)
+                                  #~(cons name #$hook-gexp)))
+                               hooks)))
+                (hooks-with-defaults
+                 `(,@hooks
+                   ,@(remove (match-lambda
+                               ((name . _) (assq-ref hooks name)))
+                             %default-hooks)))
+                (build-coordinator (make-build-coordinator
+                                    #:datastore datastore
+                                    #:hooks hooks-with-defaults
+                                    #:metrics-registry metrics-registry
+                                    #:allocation-strategy #$allocation-strategy)))
+
+           (run-coordinator-service
+            build-coordinator
+            #:update-datastore? #t
+            #:pid-file #$pid-file
+            #:agent-communication-uri (string->uri
+                                       #$agent-communication-uri-string)
+            #:client-communication-uri (string->uri
+                                        #$client-communication-uri-string)))))
+   #:guile guile))
+
+(define (guix-build-coordinator-shepherd-services config)
+  (match-record config <guix-build-coordinator-configuration>
+    (package user group database-uri-string
+             agent-communication-uri-string
+             client-communication-uri-string
+             allocation-strategy
+             hooks
+             guile)
+    (list
+     (shepherd-service
+      (documentation "Guix Build Coordinator")
+      (provision '(guix-build-coordinator))
+      (requirement '(networking))
+      (start #~(make-forkexec-constructor
+                (list #$(make-guix-build-coordinator-start-script
+                         database-uri-string
+                         allocation-strategy
+                         "/var/run/guix-build-coordinator/pid"
+                         package
+                         #:agent-communication-uri-string
+                         agent-communication-uri-string
+                         #:client-communication-uri-string
+                         client-communication-uri-string
+                         #:hooks hooks
+                         #:guile guile))
+                #:user #$user
+                #:group #$group
+                #:pid-file "/var/run/guix-build-coordinator/pid"
+                ;; Allow time for migrations to run
+                #:pid-file-timeout 60
+                #:environment-variables
+                `(,(string-append
+                    "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+                  "LC_ALL=en_US.utf8")
+                #:log-file "/var/log/guix-build-coordinator/coordinator.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-activation config)
+  #~(begin
+      (use-modules (guix build utils))
+
+      (define %user (getpw "guix-build-coordinator"))
+
+      (chmod "/var/lib/guix-build-coordinator" #o755)
+
+      (mkdir-p "/var/log/guix-build-coordinator")
+
+      ;; Allow writing the PID file
+      (mkdir-p "/var/run/guix-build-coordinator")
+      (chown "/var/run/guix-build-coordinator"
+             (passwd:uid %user)
+             (passwd:gid %user))))
+
+(define (guix-build-coordinator-account config)
+  (match-record config <guix-build-coordinator-configuration>
+    (user group)
+    (list (user-group
+           (name group)
+           (system? #t))
+          (user-account
+           (name user)
+           (group group)
+           (system? #t)
+           (comment "Guix Build Coordinator user")
+           (home-directory "/var/lib/guix-build-coordinator")
+           (shell (file-append shadow "/sbin/nologin"))))))
+
+(define guix-build-coordinator-service-type
+  (service-type
+   (name 'guix-build-coordinator)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        guix-build-coordinator-shepherd-services)
+     (service-extension activation-service-type
+                        guix-build-coordinator-activation)
+     (service-extension account-service-type
+                        guix-build-coordinator-account)))
+   (default-value
+     (guix-build-coordinator-configuration))
+   (description
+    "Run an instance of the Guix Build Coordinator.")))
+
+
+;;;
+;;; Guix Data Service
+;;;
+
 (define-record-type* <guix-data-service-configuration>
   guix-data-service-configuration make-guix-data-service-configuration
   guix-data-service-configuration?
diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm
index 6139e31cf0..20b67d55d3 100644
--- a/gnu/tests/guix.scm
+++ b/gnu/tests/guix.scm
@@ -35,7 +35,80 @@
   #:use-module (guix store)
   #:use-module (guix utils)
   #:use-module (ice-9 match)
-  #:export (%test-guix-data-service))
+  #:export (%test-guix-build-coordinator
+            %test-guix-data-service))
+
+;;;
+;;; Guix Build Coordinator
+;;;
+
+(define %guix-build-coordinator-os
+  (simple-operating-system
+   (service dhcp-client-service-type)
+   (service guix-build-coordinator-service-type)))
+
+(define (run-guix-build-coordinator-test)
+  (define os
+    (marionette-operating-system
+     %guix-build-coordinator-os
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define forwarded-port 8745)
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     (memory-size 1024)
+     (port-forwardings `((,forwarded-port . 8745)))))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (srfi srfi-11) (srfi srfi-64)
+                       (gnu build marionette)
+                       (web uri)
+                       (web client)
+                       (web response))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (mkdir #$output)
+          (chdir #$output)
+
+          (test-begin "guix-build-coordinator")
+
+          (test-assert "service running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (match (start-service 'guix-build-coordinator)
+                  (#f #f)
+                  (('service response-parts ...)
+                   (match (assq-ref response-parts 'running)
+                     ((pid) (number? pid))))))
+             marionette))
+
+          (test-equal "http-get"
+            200
+            (let-values
+                (((response text)
+                  (http-get #$(simple-format
+                               #f "http://localhost:~A/metrics" forwarded-port)
+                            #:decode-body? #t)))
+              (response-code response)))
+
+          (test-end)
+          (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+  (gexp->derivation "guix-build-coordinator-test" test))
+
+(define %test-guix-build-coordinator
+  (system-test
+   (name "guix-build-coordinator")
+   (description "Connect to a running Guix Build Coordinator.")
+   (value (run-guix-build-coordinator-test))))
 
 
 ;;;
-- 
2.28.0
C
C
Christopher Baines wrote on 19 Sep 2020 12:10
[PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(address . 43494@debbugs.gnu.org)
20200919101059.8930-4-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
New record type.
(guix-build-coordinator-queue-builds-configuration,
guix-build-coordinator-queue-builds-configuration?,
guix-build-coordinator-queue-builds-configuration-package,
guix-build-coordinator-queue-builds-configuration-user,
guix-build-coordinator-queue-builds-coordinator,
guix-build-coordinator-queue-builds-configuration-systems,
guix-build-coordinator-queue-builds-configuration-system-and-targets,
guix-build-coordinator-queue-builds-configuration-guix-data-service,
guix-build-coordinator-queue-builds-configuration-processed-commits-file,
guix-build-coordinator-queue-builds-shepherd-services,
guix-build-coordinator-queue-builds-activation,
guix-build-coordinator-queue-builds-account): New procedures.
(guix-build-coordinator-queue-builds-service-type): New variable.
---
doc/guix.texi | 38 +++++++++++++++
gnu/services/guix.scm | 108 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 146 insertions(+)

Toggle diff (184 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index b2dde30ddb..767c99c5cb 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27603,6 +27603,44 @@ input store items aren't already available.
 @end table
 @end deftp
 
+@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
+Service type for the
+guix-build-coordinator-queue-builds-from-guix-data-service script.  Its
+value must be a @code{guix-build-coordinator-queue-builds-configuration}
+object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-queue-builds-configuration
+Data type representing the options to the queue builds from guix data
+service script.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"})
+The system user to run the service as.
+
+@item @code{coordinator} (default: @code{"http://localhost:8745"})
+The URI to use when connecting to the coordinator.
+
+@item @code{systems} (default: @code{#f})
+The systems for which to fetch derivations to build.
+
+@item @code{systems-and-targets} (default: @code{#f})
+An association list of system and target pairs for which to fetch
+derivations to build.
+
+@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})
+The Guix Data Service instance from which to query to find out about
+derivations to build.
+
+@item @code{processed-commits-file} (default: @code{"/var/cache/guix-build-coordinator-queue-builds/processed-commits"})
+A file to record which commits have been processed, to avoid needlessly
+processing them again if the service is restarted.
+
+@end table
+@end deftp
 
 @subsubheading Guix Data Service
 The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index e4f5cf57b9..1493a24be5 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -65,6 +65,18 @@
 
             guix-build-coordinator-agent-service-type
 
+            guix-build-coordinator-queue-builds-configuration
+            guix-build-coordinator-queue-builds-configuration?
+            guix-build-coordinator-queue-builds-configuration-package
+            guix-build-coordinator-queue-builds-configuration-user
+            guix-build-coordinator-queue-builds-coordinator
+            guix-build-coordinator-queue-builds-configuration-systems
+            guix-build-coordinator-queue-builds-configuration-system-and-targets
+            guix-build-coordinator-queue-builds-configuration-guix-data-service
+            guix-build-coordinator-queue-builds-configuration-processed-commits-file
+
+            guix-build-coordinator-queue-builds-service-type
+
             <guix-data-service-configuration>
             guix-data-service-configuration
             guix-data-service-configuration?
@@ -137,6 +149,28 @@
    guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
    (default #f)))
 
+(define-record-type* <guix-build-coordinator-queue-builds-configuration>
+  guix-build-coordinator-queue-builds-configuration
+  make-guix-build-coordinator-queue-builds-configuration
+  guix-build-coordinator-queue-builds-configuration?
+  (package              guix-build-coordinator-queue-builds-configuration-package
+                        (default guix-build-coordinator))
+  (user                 guix-build-coordinator-queue-builds-configuration-user
+                        (default "guix-build-coordinator-queue-builds"))
+  (coordinator          guix-build-coordinator-queue-builds-coordinator
+                        (default "http://localhost:8745"))
+  (systems              guix-build-coordinator-queue-builds-configuration-systems
+                        (default #f))
+  (systems-and-targets
+   guix-build-coordinator-queue-builds-configuration-system-and-targets
+   (default #f))
+  (guix-data-service
+   guix-build-coordinator-queue-builds-configuration-guix-data-service
+   (default "https://data.guix.gnu.org"))
+  (processed-commits-file
+   guix-build-coordinator-queue-builds-configuration-processed-commits-file
+   (default "/var/cache/guix-build-coordinator-queue-builds/processed-commits")))
+
 (define* (make-guix-build-coordinator-start-script database-uri-string
                                                    allocation-strategy
                                                    pid-file
@@ -360,6 +394,80 @@
    (description
     "Run a Guix Build Coordinator agent.")))
 
+(define (guix-build-coordinator-queue-builds-shepherd-services config)
+  (match-record config <guix-build-coordinator-queue-builds-configuration>
+    (package user coordinator systems systems-and-targets
+             guix-data-service processed-commits-file)
+    (list
+     (shepherd-service
+      (documentation "Guix Build Coordinator queue builds from Guix Data Service")
+      (provision '(guix-build-coordinator-queue-builds))
+      (requirement '(networking))
+      (start
+       #~(make-forkexec-constructor
+          (list
+           #$(file-append
+              package
+              "/bin/guix-build-coordinator-queue-builds-from-guix-data-service")
+           #$(string-append "--coordinator=" coordinator)
+           #$@(map (lambda (system)
+                     (string-append "--system=" system))
+                   (or systems '()))
+           #$@(map (match-lambda
+                     ((system . target)
+                      (string-append "--system-and-target=" system "=" target)))
+                   (or systems-and-targets '()))
+           #$@(if guix-data-service
+                  #~(#$(string-append "--guix-data-service=" guix-data-service))
+                  #~())
+           #$@(if processed-commits-file
+                  #~(#$(string-append "--processed-commits-file="
+                                      processed-commits-file))
+                  #~()))
+          #:user #$user
+          #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid"
+          #:environment-variables
+          `(,(string-append
+              "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+            "LC_ALL=en_US.utf8")
+          #:log-file "/var/log/guix-build-coordinator/queue-builds.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-queue-builds-activation config)
+  #~(begin
+      (use-modules (guix build utils))
+
+      (mkdir-p "/var/log/guix-build-coordinator")
+
+      ;; Allow writing the PID file
+      (mkdir-p "/var/run/guix-build-coordinator-queue-builds")
+      (chown "/var/run/guix-build-coordinator-queue-builds"
+             (passwd:uid %user)
+             (passwd:gid %user))))
+
+(define (guix-build-coordinator-queue-builds-account config)
+  (list (user-account
+         (name (guix-build-coordinator-queue-builds-configuration-user config))
+         (group "nogroup")
+         (system? #t)
+         (comment "Guix Build Coordinator queue-builds user")
+         (home-directory "/var/empty")
+         (shell (file-append shadow "/sbin/nologin")))))
+
+(define guix-build-coordinator-queue-builds-service-type
+  (service-type
+   (name 'guix-build-coordinator-queue-builds)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        guix-build-coordinator-queue-builds-shepherd-services)
+     (service-extension activation-service-type
+                        guix-build-coordinator-queue-builds-activation)
+     (service-extension account-service-type
+                        guix-build-coordinator-queue-builds-account)))
+   (description
+    "Run the Guix Build Coordinator queue builds script.")))
+
 
 ;;;
 ;;; Guix Data Service
-- 
2.28.0
L
L
Ludovic Courtès wrote on 25 Sep 2020 11:42
Re: [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87imc22nwm.fsf@gnu.org
Hi!

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (2 lines)
> Ludovic Courtès <ludo@gnu.org> writes:

[...]

Toggle quote (22 lines)
>>> + (program-file
>>> + "start-guix-build-coordinator"
>>> + (with-extensions (cons guix-build-coordinator-package
>>> + ;; This is a poorly constructed Guile load path,
>>> + ;; since it contains things that aren't Guile
>>> + ;; libraries, but it means that the Guile libraries
>>> + ;; needed for the Guix Build Coordinator don't need
>>> + ;; to be individually specified here.
>>> + (map second (package-inputs
>>> + guix-build-coordinator-package)))
>>
>> Perhaps there should eventually be a ‘guix-build-coordinator’ command in
>> the package itself?
>
> There actually is, one thing I've had in mind for a while now though is
> to use a scheme script constructed by the Guix service to run the
> coordinator.
>
> For guix.cbaines.net, I'm using the script, but with the hooks passed in
> on the command line, the command is rather long, and it means that
> backtraces don't work well with the hooks.

You mean because the hooks are interpreted, and so all you see in the
backtrace is a bunch of ‘eval’ calls?

Toggle quote (5 lines)
> I'm unsure how well this has worked out, I didn't anticipate the issues
> with the Guile load path, and because of the formatting stripping, both
> the code and backtraces will be unreadable anyway... but it is useful in
> allowing G-expressions to be used as part of the configuration.

OK.

Thanks,
Ludo’.
C
C
Christopher Baines wrote on 26 Sep 2020 10:43
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
87mu1dnd1o.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (33 lines)
> Hi!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>>>> + (program-file
>>>> + "start-guix-build-coordinator"
>>>> + (with-extensions (cons guix-build-coordinator-package
>>>> + ;; This is a poorly constructed Guile load path,
>>>> + ;; since it contains things that aren't Guile
>>>> + ;; libraries, but it means that the Guile libraries
>>>> + ;; needed for the Guix Build Coordinator don't need
>>>> + ;; to be individually specified here.
>>>> + (map second (package-inputs
>>>> + guix-build-coordinator-package)))
>>>
>>> Perhaps there should eventually be a ‘guix-build-coordinator’ command in
>>> the package itself?
>>
>> There actually is, one thing I've had in mind for a while now though is
>> to use a scheme script constructed by the Guix service to run the
>> coordinator.
>>
>> For guix.cbaines.net, I'm using the script, but with the hooks passed in
>> on the command line, the command is rather long, and it means that
>> backtraces don't work well with the hooks.
>
> You mean because the hooks are interpreted, and so all you see in the
> backtrace is a bunch of ‘eval’ calls?

Yeah, I haven't done much testing of this, but that's my assumption.
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9u/zNfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE
9Xdr5Q//QN7KCG9MfVAAMx7d9lor18eZMdbw5JJNRXwbPrTIhTd+rWadYU0w/8fm
y59oFNhv3MFFHtOYtKt8r/axRDey57CIRnzRPcOfd9Gr31hJ8wq5vqgDPl1xmOsG
4W9rhxi6MAdAfr0qcz4yxq9MHGujAuSmwsj5gP8S+UWVw5YbS3IemhAWg7lkWHlB
brakM/A2A6ZD4GBwbJuYi5ZnIK1Af8gyPuc4VSCjTPmrutFzaEkOFy5Mdlx/Ct49
v/VKSzmcHkkRYP3jZZv3kFZn9gsePFtK30uh5iPQg2MwRUdbq5p5DIBN6PSNuyAN
MMKKtyeXGnVM9iOwDFPli+s4B577W5Bo/Heixg+4RpituHKe5HP5eSg0jalQJkhK
uGaiTOCRZbDSTo9ZsXEvTwCK6v0rdlE3Er/oEATqcatkUSgOYU2qhxkVNwipHf0O
PmWjwDFMD3eYNjGxnCPePNHm2xxP7LjsSDh5r1ez6ck8zdKOxu5Ifv+3OQTl54bp
MimxV5gxd0tV8XmjE1U1cNlyV9VTReZKsrBEiXypz/5aYRbie6AdfR7ZPi4Faved
dm2le0sJTxnH1xkB4/k3zelqF2NCI9RztVvOnXZsn/AH8TpTW9hsMq8acgHvT2H0
A0SAZEhTsXYoznJOaEoT10PBH9xSmfXxiggCNNiOU6DlvxAIYcs=
=9BXV
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 5 Oct 2020 10:00
Re: [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87y2kl5cg8.fsf@gnu.org
Hi Chris,

Sorry for the delay. Patches 1–3 LGTM! One minor comment on patch 4:

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (16 lines)
> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
> New record type.
> (guix-build-coordinator-queue-builds-configuration,
> guix-build-coordinator-queue-builds-configuration?,
> guix-build-coordinator-queue-builds-configuration-package,
> guix-build-coordinator-queue-builds-configuration-user,
> guix-build-coordinator-queue-builds-coordinator,
> guix-build-coordinator-queue-builds-configuration-systems,
> guix-build-coordinator-queue-builds-configuration-system-and-targets,
> guix-build-coordinator-queue-builds-configuration-guix-data-service,
> guix-build-coordinator-queue-builds-configuration-processed-commits-file,
> guix-build-coordinator-queue-builds-shepherd-services,
> guix-build-coordinator-queue-builds-activation,
> guix-build-coordinator-queue-builds-account): New procedures.
> (guix-build-coordinator-queue-builds-service-type): New variable.

[...]

Toggle quote (8 lines)
>
> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
> +Service type for the
> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its
> +value must be a @code{guix-build-coordinator-queue-builds-configuration}
> +object.
> +@end defvar

Could you add a paragraph right above this explaining what this service
is about?

Toggle quote (14 lines)
> +(define guix-build-coordinator-queue-builds-service-type
> + (service-type
> + (name 'guix-build-coordinator-queue-builds)
> + (extensions
> + (list
> + (service-extension shepherd-root-service-type
> + guix-build-coordinator-queue-builds-shepherd-services)
> + (service-extension activation-service-type
> + guix-build-coordinator-queue-builds-activation)
> + (service-extension account-service-type
> + guix-build-coordinator-queue-builds-account)))
> + (description
> + "Run the Guix Build Coordinator queue builds script.")))

Likewise it’d be great if you could add a few works here.

OK to push with these changes, thank you!

Ludo’.
C
C
Christopher Baines wrote on 5 Oct 2020 19:16
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494-done@debbugs.gnu.org)
87a6x01tll.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (53 lines)
> Hi Chris,
>
> Sorry for the delay. Patches 1–3 LGTM! One minor comment on patch 4:
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
>> New record type.
>> (guix-build-coordinator-queue-builds-configuration,
>> guix-build-coordinator-queue-builds-configuration?,
>> guix-build-coordinator-queue-builds-configuration-package,
>> guix-build-coordinator-queue-builds-configuration-user,
>> guix-build-coordinator-queue-builds-coordinator,
>> guix-build-coordinator-queue-builds-configuration-systems,
>> guix-build-coordinator-queue-builds-configuration-system-and-targets,
>> guix-build-coordinator-queue-builds-configuration-guix-data-service,
>> guix-build-coordinator-queue-builds-configuration-processed-commits-file,
>> guix-build-coordinator-queue-builds-shepherd-services,
>> guix-build-coordinator-queue-builds-activation,
>> guix-build-coordinator-queue-builds-account): New procedures.
>> (guix-build-coordinator-queue-builds-service-type): New variable.
>
> [...]
>
>>
>> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
>> +Service type for the
>> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its
>> +value must be a @code{guix-build-coordinator-queue-builds-configuration}
>> +object.
>> +@end defvar
>
> Could you add a paragraph right above this explaining what this service
> is about?
>
>> +(define guix-build-coordinator-queue-builds-service-type
>> + (service-type
>> + (name 'guix-build-coordinator-queue-builds)
>> + (extensions
>> + (list
>> + (service-extension shepherd-root-service-type
>> + guix-build-coordinator-queue-builds-shepherd-services)
>> + (service-extension activation-service-type
>> + guix-build-coordinator-queue-builds-activation)
>> + (service-extension account-service-type
>> + guix-build-coordinator-queue-builds-account)))
>> + (description
>> + "Run the Guix Build Coordinator queue builds script.")))
>
> Likewise it’d be great if you could add a few works here.
>
> OK to push with these changes, thank you!

I've tried to add some clarity in the places you mention. I'm still very
tempted to make a separate repository/package for the queue builds from
Guix Data Service script... I'm not quite decided yet though.

Anyway, thanks for taking another look. I've gone ahead and pushed these
patches as c14714cbbfe9239410aec2ed73282192a64fbbdc. I did tweak the
package definition a bit though to move closer to being able to
cross-compile it (you can now if you tweak the inputs).

Thanks again,

Chris
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl97VOZfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XelCRAAi3Ihs57Jxpd29ZfQX/6QqK8LTMWGs8dQ
juPk6/Yl0DzaxcEeuzf+TPR0t/zuDCzucv9g0+bx4ohPTN08MPoGT6y+tdY1zDNh
7MxSYiLiM8IOLxDwPoB6LoFTyUcsV3EJNbfrdf3fN3IEwTd8y9HOBrRuCT46lRNc
03ku7yL44JhiHotzg1YWY+si2y/OAAPb4s0iP7Chx9ppLdvyqANcV0leuVJrtP3y
itb23Gb9BCmSfFobEeI3xo4se6LsXqpYj8fq3kGipC4DCoWkZiA99gW4UiY3C4+h
d2zmElR4+x2lcAQD2Z8SLWK/5bej1iWeJ7q8OvuAgRJS6LlDTkxxI73m8frqnui0
EGZGonG+UrXmsM9h/Pp7aIQU+C5rDu7u3NqmG+WeeDjgJTD5vorkpKhMwDfeH6sL
BSsMWiUvEpwWMx2MqYOZSo3qn8wIb/HfUFAZNNrWhasCygOTWaKOgm5IdfVbSENO
wxve8VPF6UG26diVc/1OaDBmMdao7+L9eU+CHD/J/U9xVcRBGPdZ8ATFJfKTOUnT
myusPwsdTYNM1AAdZFVev0sOrRhmW6lWh117n8ideJ68w4ldpK9s70Ln+IfTPbUx
jKlSiKn0dYOui61oag0EFwJrm3uWJEBggLd7TUC9Gp+tZwpDuFsYJCIULHkJeXC9
sWHkqe4RtGo=
=pTf2
-----END PGP SIGNATURE-----

Closed
?
Your comment

This issue is archived.

To comment on this conversation send email to 43494@debbugs.gnu.org