[PATCH 0/4] Add package and services for the Guix Build Coordinator

DoneSubmitted by Christopher Baines.
Details
3 participants
  • Jonathan Brielmaier
  • Ludovic Courtès
  • Christopher Baines
Owner
unassigned
Severity
normal
C
C
Christopher Baines wrote on 18 Sep 2020 20:34
(address . guix-patches@gnu.org)
87o8m39bnc.fsf@cbaines.net
This patch series is not particularly well tested, like the Guix BuildCoordinator itself, but it should hopefully still work at least in someconfigurations.

Christopher Baines (4): gnu: Add guix-build-coordinator. services: guix: Add guix-build-coordinator-service-type. services: guix: Add guix-build-coordinator-agent-service-type. services: guix: Add guix-build-coordinator-queue-builds-service-type.
doc/guix.texi | 143 +++++++++ gnu/packages/package-management.scm | 89 ++++++ gnu/services/guix.scm | 433 +++++++++++++++++++++++++++- gnu/tests/guix.scm | 75 ++++- 4 files changed, 736 insertions(+), 4 deletions(-)
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9k/cdfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9Xfrig//b1nSvUfhfTzlMrAc5stuOAEBdd7hrG82aj8exISjDZa2UfzFN0ox828zggjWbz69KjKP41OJfQgNGJ9GXqk582CvUaFloQBPQXH+3tBNJR+DlZHYEPhO5KlUK0ZuatnpqCC704DQOsqTPAxKsaKuqSHYy9j7XFMieqilznvv0YMp8qpepTRyza6FY/H0qG4Ohfk3EhV5c99c4Yyvxx6x16TB+Gr93HBClVh3AYNbiEW2xuN7xSS03ljlyr26rpna8fY4sXH3hUgbyOu7B1LNeiAveLRnEW2E3OtRBhXFB/nBc/L17Uz2tje4zRD86Bl4nWZ26v9FKaTwQAKaCl0S5Fi5DSPZviO8FySVRXfDHYy3PaZks3HHpDDQ/vgMbLba9BbBJ0Zq7/uoGgk9rWLUCuUDkfIigxlvyMHUAuNCnh3PnvjwJ3zenK6BHGSJ2Xi2JDE0pHrt4Tzs4uBG2JC+JyDZAhgZjJSkFfWP4YdJaLKq7J8BkFJV5c/Ex42nxLdf9LaqeR7zfCqS8u0mBocmrvl9bfOY7aafo4ezE6ktDDp1ecwnxzpVlgIEPn3ItRAIrs/w1O8i0ITHIMArpUdS+Yk+rfyEs400QB4kSXnphaBQdUKL0arRIFXrnqzi+I857L7ZqKWR7AKGSmWAJQay3KzzuENVho+dLJtC4wZxwcg==uTsG-----END PGP SIGNATURE-----
C
C
Christopher Baines wrote on 18 Sep 2020 20:40
[PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(address . 43494@debbugs.gnu.org)
20200918184042.22660-4-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):New record type.(guix-build-coordinator-queue-builds-configuration,guix-build-coordinator-queue-builds-configuration?,guix-build-coordinator-queue-builds-configuration-package,guix-build-coordinator-queue-builds-configuration-user,guix-build-coordinator-queue-builds-coordinator,guix-build-coordinator-queue-builds-configuration-systems,guix-build-coordinator-queue-builds-configuration-system-and-targets,guix-build-coordinator-queue-builds-configuration-guix-data-service,guix-build-coordinator-queue-builds-configuration-processed-commits-file,guix-build-coordinator-queue-builds-shepherd-services,guix-build-coordinator-queue-builds-activation,guix-build-coordinator-queue-builds-account): New procedures.(guix-build-coordinator-queue-builds-service-type): New variable.--- doc/guix.texi | 38 +++++++++++++++ gnu/services/guix.scm | 108 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+)
Toggle diff (184 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex db91831197..1b47a01217 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -27590,6 +27590,44 @@ input store items aren't already available. @end table @end deftp +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type+Service type for the+guix-build-coordinator-queue-builds-from-guix-data-service script. Its+value must be a @code{guix-build-coordinator-queue-builds-configuration}+object.+@end defvar++@deftp {Data Type} guix-build-coordinator-queue-builds-configuration+Data type representing the options to the queue builds from guix data+service script.++@table @asis+@item @code{package} (default: @code{guix-build-coordinator})+The Guix Build Coordinator package to use.++@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"})+The system user to run the service as.++@item @code{coordinator} (default: @code{"http://localhost:8745"})+The URI to use when connecting to the coordinator.++@item @code{systems} (default: @code{#f})+The systems for which to fetch derivations to build.++@item @code{systems-and-targets} (default: @code{#f})+An association list of system and target pairs for which to fetch+derivations to build.++@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})+The Guix Data Service instance from which to query to find out about+derivations to build.++@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})+A file to record which commits have been processed, to avoid needlessly+processing them again if the service is restarted.++@end table+@end deftp @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, storesdiff --git a/gnu/services/guix.scm b/gnu/services/guix.scmindex 71e58237da..975febebd1 100644--- a/gnu/services/guix.scm+++ b/gnu/services/guix.scm@@ -65,6 +65,18 @@ guix-build-coordinator-agent-service-type + guix-build-coordinator-queue-builds-configuration+ guix-build-coordinator-queue-builds-configuration?+ guix-build-coordinator-queue-builds-configuration-package+ guix-build-coordinator-queue-builds-configuration-user+ guix-build-coordinator-queue-builds-coordinator+ guix-build-coordinator-queue-builds-configuration-systems+ guix-build-coordinator-queue-builds-configuration-system-and-targets+ guix-build-coordinator-queue-builds-configuration-guix-data-service+ guix-build-coordinator-queue-builds-configuration-processed-commits-file++ guix-build-coordinator-queue-builds-service-type+ <guix-data-service-configuration> guix-data-service-configuration guix-data-service-configuration?@@ -137,6 +149,28 @@ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls (default #f))) +(define-record-type* <guix-build-coordinator-queue-builds-configuration>+ guix-build-coordinator-queue-builds-configuration+ make-guix-build-coordinator-queue-builds-configuration+ guix-build-coordinator-queue-builds-configuration?+ (package guix-build-coordinator-queue-builds-configuration-package+ (default guix-build-coordinator))+ (user guix-build-coordinator-queue-builds-configuration-user+ (default "guix-build-coordinator-queue-builds"))+ (coordinator guix-build-coordinator-queue-builds-coordinator+ (default "http://localhost:8745"))+ (systems guix-build-coordinator-queue-builds-configuration-systems+ (default #f))+ (systems-and-targets+ guix-build-coordinator-queue-builds-configuration-system-and-targets+ (default #f))+ (guix-data-service+ guix-build-coordinator-queue-builds-configuration-guix-data-service+ (default "https://data.guix.gnu.org"))+ (processed-commits-file+ guix-build-coordinator-queue-builds-configuration-processed-commits-file+ (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))+ (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file@@ -364,6 +398,80 @@ (description "Run an instance of the Guix Build Coordinator."))) +(define (guix-build-coordinator-queue-builds-shepherd-services config)+ (match-record config <guix-build-coordinator-queue-builds-configuration>+ (package user coordinator systems systems-and-targets+ guix-data-service processed-commits-file)+ (list+ (shepherd-service+ (documentation "Guix Build Coordinator queue builds from Guix Data Service")+ (provision '(guix-build-coordinator-queue-builds))+ (requirement '(networking))+ (start+ #~(make-forkexec-constructor+ (list+ #$(file-append+ package+ "/bin/guix-build-coordinator-queue-builds-from-guix-data-service")+ #$(string-append "--coordinator=" coordinator)+ #$@(map (lambda (system)+ (string-append "--system=" system))+ (or systems '()))+ #$@(map (match-lambda+ ((system . target)+ (string-append "--system-and-target=" system "=" target)))+ (or systems-and-targets '()))+ #$@(if guix-data-service+ #~(#$(string-append "--guix-data-service=" guix-data-service))+ #~())+ #$@(if processed-commits-file+ #~(#$(string-append "--processed-commits-file="+ processed-commits-file))+ #~()))+ #:user #$user+ #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid"+ #:environment-variables+ `(,(string-append+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")+ "LC_ALL=en_US.utf8")+ #:log-file "/var/log/guix-build-coordinator/queue-builds.log"))+ (stop #~(make-kill-destructor))))))++(define (guix-build-coordinator-queue-builds-activation config)+ #~(begin+ (use-modules (guix build utils))++ (mkdir-p "/var/log/guix-build-coordinator")++ ;; Allow writing the PID file+ (mkdir-p "/var/run/guix-build-coordinator-queue-builds")+ (chown "/var/run/guix-build-coordinator-queue-builds"+ (passwd:uid %user)+ (passwd:gid %user))))++(define (guix-build-coordinator-queue-builds-account config)+ (list (user-account+ (name (guix-build-coordinator-queue-builds-configuration-user config))+ (group "nogroup")+ (system? #t)+ (comment "Guix Build Coordinator queue-builds user")+ (home-directory "/var/empty")+ (shell (file-append shadow "/sbin/nologin")))))++(define guix-build-coordinator-queue-builds-service-type+ (service-type+ (name 'guix-build-coordinator-queue-builds)+ (extensions+ (list+ (service-extension shepherd-root-service-type+ guix-build-coordinator-queue-builds-shepherd-services)+ (service-extension activation-service-type+ guix-build-coordinator-queue-builds-activation)+ (service-extension account-service-type+ guix-build-coordinator-queue-builds-account)))+ (description+ "Run the Guix Build Coordinator queue builds script.")))+ ;;; ;;; Guix Data Service-- 2.28.0
C
C
Christopher Baines wrote on 18 Sep 2020 20:40
[PATCH 1/4] gnu: Add guix-build-coordinator.
(address . 43494@debbugs.gnu.org)
20200918184042.22660-1-mail@cbaines.net
* gnu/packages/package-management.scm (guix-build-coordinator): New variable.--- gnu/packages/package-management.scm | 89 +++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)
Toggle diff (109 lines)diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scmindex ec87226197..4272d3134d 100644--- a/gnu/packages/package-management.scm+++ b/gnu/packages/package-management.scm@@ -50,6 +50,7 @@ #:use-module (gnu packages cpio) #:use-module (gnu packages crypto) #:use-module (gnu packages curl)+ #:use-module (gnu packages databases) #:use-module (gnu packages dbm) #:use-module (gnu packages docbook) #:use-module (gnu packages file)@@ -981,6 +982,94 @@ environments.") ;; and the fonts included in this package are licensed OFL1.1. (license (list license:gpl3+ license:agpl3+ license:silofl1.1)))) +(define-public guix-build-coordinator+ (let ((commit "5e8408c833e209efddfa0159114b90400c1aaf4d")+ (revision "0"))+ (package+ (name "guix-build-coordinator")+ (version (git-version "0" revision commit))+ (source (origin+ (method git-fetch)+ (uri (git-reference+ (url "https://git.cbaines.net/git/guix/build-coordinator")+ (commit commit)))+ (sha256+ (base32+ "0f7m1zg9mlb2m22qyblglaa36h8f49b810jc9j5b0hsb42ijwh4b"))+ (file-name (string-append name "-" version "-checkout"))))+ (build-system gnu-build-system)+ (arguments+ '(#:modules ((guix build utils)+ (guix build gnu-build-system)+ (ice-9 ftw)+ (ice-9 match)+ (ice-9 rdelim)+ (ice-9 popen))+ #:phases+ (modify-phases %standard-phases+ (add-before 'build 'set-GUILE_AUTO_COMPILE+ (lambda _+ ;; To avoid warnings relating to 'guild'.+ (setenv "GUILE_AUTO_COMPILE" "0")+ #t))+ (add-after 'install 'wrap-executable+ (lambda* (#:key inputs outputs #:allow-other-keys)+ (let* ((out (assoc-ref outputs "out"))+ (bin (string-append out "/bin"))+ (guile (assoc-ref inputs "guile"))+ (guile-effective-version+ (read-line+ (open-pipe* OPEN_READ+ (string-append guile "/bin/guile")+ "-c" "(display (effective-version))")))+ (scm (string-append out "/share/guile/site/"+ guile-effective-version))+ (go (string-append out "/lib/guile/"+ guile-effective-version+ "/site-ccache")))+ (for-each+ (lambda (file)+ (simple-format (current-error-port)+ "wrapping: ~A\n"+ (string-append bin "/" file))+ (wrap-program (string-append bin "/" file)+ `("PATH" ":" prefix+ (,bin ,(assoc-ref inputs "sqitch")))+ `("PERL5LIB" ":" prefix+ (,(getenv "PERL5LIB")))+ `("GUILE_LOAD_PATH" ":" prefix+ (,scm ,(getenv "GUILE_LOAD_PATH")))+ `("GUILE_LOAD_COMPILED_PATH" ":" prefix+ (,go ,(getenv "GUILE_LOAD_COMPILED_PATH")))))+ (scandir bin+ (match-lambda+ ((or "." "..") #f)+ (_ #t))))+ #t)))+ (delete 'strip)))) ; As the .go files aren't compatible+ (native-inputs+ `(("pkg-config" ,pkg-config)+ ("autoconf" ,autoconf)+ ("automake" ,automake)+ ("perl" ,perl)))+ (inputs+ `(("guile" ,guile-3.0-latest)+ ("guile-fibers" ,guile-fibers)+ ("guile-prometheus" ,guile-prometheus)+ ("guile-gcrypt" ,guile-gcrypt)+ ("guile-json" ,guile-json-3)+ ("guile-lzlib" ,guile-lzlib)+ ("guile-sqlite3" ,guile-sqlite3)+ ("guix" ,guix)+ ("sqlite" ,sqlite)+ ("sqitch" ,sqitch)+ ("perl-dbd-sqlite" ,perl-dbd-sqlite)))+ (home-page "https://git.cbaines.net/guile/guix/build-coordinator")+ (synopsis "")+ (description+ "")+ (license license:gpl3+))))+ (define-public guix-jupyter (package (name "guix-jupyter")-- 2.28.0
C
C
Christopher Baines wrote on 18 Sep 2020 20:40
[PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
(address . 43494@debbugs.gnu.org)
20200918184042.22660-3-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): Newrecord type.(guix-build-coordinator-agent-configuration,guix-build-coordinator-agent-configuration?,guix-build-coordinator-agent-configuration-package,guix-build-coordinator-agent-configuration-user,guix-build-coordinator-agent-configuration-coordinator,guix-build-coordinator-agent-configuration-uuid),guix-build-coordinator-agent-configuration-password,guix-build-coordinator-agent-configuration-password-file,guix-build-coordinator-agent-configuration-systems,guix-build-coordinator-agent-configuration-max-parallel-builds,guix-build-coordinator-agent-configuration-derivation-substitute-urls,guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,guix-build-coordinator-agent-shepherd-services,guix-build-coordinator-agent-activation,guix-build-coordinator-agent-account): New procedures.(guix-build-coordinator-agent-service-type): New variable.* doc/guix.texi (Guix Services): Document it.--- doc/guix.texi | 50 +++++++++++++++++ gnu/services/guix.scm | 121 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+)
Toggle diff (209 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex e83008c177..db91831197 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -27541,6 +27541,56 @@ The Guile package with which to run the Guix Build Coordinator. @end table @end deftp +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type+Service type for a Guix Build Coordinator agent. Its value must be a+@code{guix-build-coordinator-agent-configuration} object.+@end defvar++@deftp {Data Type} guix-build-coordinator-agent-configuration+Data type representing the configuration a Guix Build Coordinator agent.++@table @asis+@item @code{package} (default: @code{guix-build-coordinator})+The Guix Build Coordinator package to use.++@item @code{user} (default: @code{"guix-build-coordinator-agent"})+The system user to run the service as.++@item @code{coordinator} (default: @code{"http://localhost:8745"})+The URI to use when connecting to the coordinator.++@item @code{uuid}+The UUID of the agent. This should be generated by the coordinator+process, stored in the coordinator database, and used by the intended+agent.++@item @code{password} (default: @code{#f})+The password to use when connecting to the coordinator. A file to read+the password from can also be specified, and this is more secure.++@item @code{password-file} (default: @code{#f})+A file containing the password to use when connecting to the+coordinator.++@item @code{systems} (default: @var{#f})+The systems for which this agent should fetch builds. The agent process+will use the current system it's running on as the default.++@item @code{max-parallel-builds} (default: @code{1})+The number of builds to perform in parallel.++@item @code{derivation-substitute-urls} (default: @code{1})+URLs from which to attempt to fetch substitutes for derivations, if the+derivations aren't already available.++@item @code{non-derivation-substitute-urls} (default: @code{1})+URLs from which to attempt to fetch substitutes for build inputs, if the+input store items aren't already available.++@end table+@end deftp++ @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores and provides data about GNU Guix. This includes information aboutdiff --git a/gnu/services/guix.scm b/gnu/services/guix.scmindex de14fcadb2..71e58237da 100644--- a/gnu/services/guix.scm+++ b/gnu/services/guix.scm@@ -50,6 +50,21 @@ guix-build-coordinator-service-type + guix-build-coordinator-agent-configuration+ guix-build-coordinator-agent-configuration?+ guix-build-coordinator-agent-configuration-package+ guix-build-coordinator-agent-configuration-user+ guix-build-coordinator-agent-configuration-coordinator+ guix-build-coordinator-agent-configuration-uuid)+ guix-build-coordinator-agent-configuration-password+ guix-build-coordinator-agent-configuration-password-file+ guix-build-coordinator-agent-configuration-systems+ guix-build-coordinator-agent-configuration-max-parallel-builds+ guix-build-coordinator-agent-configuration-derivation-substitute-urls+ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls++ guix-build-coordinator-agent-service-type+ <guix-data-service-configuration> guix-data-service-configuration guix-data-service-configuration?@@ -95,6 +110,33 @@ (guile guix-build-coordinator-configuration-guile (default guile-3.0-latest))) +(define-record-type* <guix-build-coordinator-agent-configuration>+ guix-build-coordinator-agent-configuration+ make-guix-build-coordinator-agent-configuration+ guix-build-coordinator-agent-configuration?+ (package guix-build-coordinator-agent-configuration-package+ (default guix-build-coordinator))+ (user guix-build-coordinator-agent-configuration-user+ (default "guix-build-coordinator-agent"))+ (coordinator guix-build-coordinator-agent-configuration-coordinator+ (default "http://localhost:8745"))+ (uuid guix-build-coordinator-agent-configuration-uuid)+ (password guix-build-coordinator-agent-configuration-password+ (default #f))+ (password-file guix-build-coordinator-agent-configuration-password-file+ (default #f))+ (systems guix-build-coordinator-agent-configuration-systems+ (default #f))+ (max-parallel-builds+ guix-build-coordinator-agent-configuration-max-parallel-builds+ (default 1))+ (derivation-substitute-urls+ guix-build-coordinator-agent-configuration-derivation-substitute-urls+ (default #f))+ (non-derivation-substitute-urls+ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls+ (default #f)))+ (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file@@ -243,6 +285,85 @@ (description "Run an instance of the Guix Build Coordinator."))) +(define (guix-build-coordinator-agent-shepherd-services config)+ (match-record config <guix-build-coordinator-agent-configuration>+ (package user coordinator uuid password password-file max-parallel-builds+ derivation-substitute-urls non-derivation-substitute-urls+ systems)+ (list+ (shepherd-service+ (documentation "Guix Build Coordinator Agent")+ (provision '(guix-build-coordinator-agent))+ (requirement '(networking))+ (start #~(make-forkexec-constructor+ (list #$(file-append package "/bin/guix-build-coordinator-agent")+ #$(string-append "--coordinator=" coordinator)+ #$(string-append "--uuid=" uuid)+ #$@(if password+ #~(#$(string-append "--password=" password))+ #~())+ #$@(if password-file+ #~(#$(string-append "--password-file=" password-file))+ #~())+ #$(simple-format #f "--max-parallel-builds=~A"+ max-parallel-builds)+ #$@(if derivation-substitute-urls+ #~(#$(string-append+ "--derivation-substitute-urls="+ (string-join derivation-substitute-urls " ")))+ #~())+ #$@(if non-derivation-substitute-urls+ #~(#$(string-append+ "--non-derivation-substitute-urls="+ (string-join derivation-substitute-urls " ")))+ #~())+ #$@(map (lambda (system)+ (string-append "--system=" system))+ (or systems '())))+ #:user #$user+ #:pid-file "/var/run/guix-build-coordinator-agent/pid"+ #:environment-variables+ `(,(string-append+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")+ "LC_ALL=en_US.utf8")+ #:log-file "/var/log/guix-build-coordinator/agent.log"))+ (stop #~(make-kill-destructor))))))++(define (guix-build-coordinator-agent-activation config)+ #~(begin+ (use-modules (guix build utils))++ (mkdir-p "/var/log/guix-build-coordinator")++ ;; Allow writing the PID file+ (mkdir-p "/var/run/guix-build-coordinator-agent")+ (chown "/var/run/guix-build-coordinator-agent"+ (passwd:uid %user)+ (passwd:gid %user))))++(define (guix-build-coordinator-agent-account config)+ (list (user-account+ (name (guix-build-coordinator-agent-configuration-user config))+ (group "nogroup")+ (system? #t)+ (comment "Guix Build Coordinator agent user")+ (home-directory "/var/empty")+ (shell (file-append shadow "/sbin/nologin")))))++(define guix-build-coordinator-agent-service-type+ (service-type+ (name 'guix-build-coordinator-agent)+ (extensions+ (list+ (service-extension shepherd-root-service-type+ guix-build-coordinator-agent-shepherd-services)+ (service-extension activation-service-type+ guix-build-coordinator-agent-activation)+ (service-extension account-service-type+ guix-build-coordinator-agent-account)))+ (description+ "Run an instance of the Guix Build Coordinator.")))+ ;;; ;;; Guix Data Service-- 2.28.0
C
C
Christopher Baines wrote on 18 Sep 2020 20:40
[PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
(address . 43494@debbugs.gnu.org)
20200918184042.22660-2-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-configuration>): New recordtype.(guix-build-coordinator-configuration, guix-build-coordinator-configuration?,guix-build-coordinator-configuration-package,guix-build-coordinator-configuration-user,guix-build-coordinator-configuration-group,guix-build-coordinator-configuration-datastore-uri-string,guix-build-coordinator-configuration-agent-communication-uri-string,guix-build-coordinator-configuration-client-communication-uri-string,guix-build-coordinator-configuration-allocation-strategy,guix-build-coordinator-configuration-hooks,guix-build-coordinator-configuration-guile,make-guix-build-coordinator-start-script,guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,guix-build-coordinator-account): New procedures.(guix-build-coordinator-service-type): New variable.* gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.* doc/guix.texi (Guix Services): Document it.--- doc/guix.texi | 55 ++++++++++++ gnu/services/guix.scm | 204 +++++++++++++++++++++++++++++++++++++++++- gnu/tests/guix.scm | 75 +++++++++++++++- 3 files changed, 330 insertions(+), 4 deletions(-)
Toggle diff (392 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex 88128a4b3a..e83008c177 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -27486,6 +27486,61 @@ The complete list of possible options can be found in the man page for @node Guix Services @subsection Guix Services +@subsubheading Guix Build Coordinator+The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build+Coordinator} aids in building derivations. The Guix Daemon is still+used to build the derivations, but the Guix Build Coordinator provides+ways of performing builds across many machines and working with the+results of builds.++There is a script to run the coordinator component of the Guix Build+Coordinator, but the Guix service uses a custom Guile script instead, to+provide better integration with G-expressions used in the configuration.++@defvar {Scheme Variable} guix-build-coordinator-service-type+Service type for the Guix Build Coordinator. Its value must be a+@code{guix-build-coordinator-configuration} object.+@end defvar++@deftp {Data Type} guix-build-coordinator-configuration+Data type representing the configuration of the Guix Build Coordinator.++@table @asis+@item @code{package} (default: @code{guix-build-coordinator})+The Guix Build Coordinator package to use.++@item @code{user} (default: @code{"guix-build-coordinator"})+The system user to run the service as.++@item @code{group} (default: @code{"guix-build-coordinator"})+The system group to run the service as.++@item @code{database-uri-string} (default: @code{"sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"})+The URI to use for the database.++@item @code{agent-communication-uri} (default: @code{"http://0.0.0.0:8745"})+The URI describing how to listen to requests from agent processes.++@item @code{client-communication-uri} (default: @code{"http://127.0.0.1:8746"})+The URI describing how to listen to requests from clients. The client+API allows submitting builds and currently isn't authenticated, so take+care when configuring this value.++@item @code{allocation-strategy} (default: @code{#~basic-build-allocation-strategy})+A G-expression for the allocation strategy to be used. This is a+procedure that takes the datastore as an argument and populates the+allocation plan in the database.++@item @code{hooks} (default: @var{'()})+An association list of hooks. These provide a way to execute arbitrary+code upon certian events, like a build result being processed.++@item @code{guile} (default: @code{guile-3.0-latest})+The Guile package with which to run the Guix Build Coordinator.++@end table+@end deftp+ @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores and provides data about GNU Guix. This includes information aboutdiff --git a/gnu/services/guix.scm b/gnu/services/guix.scmindex 10a8581a62..de14fcadb2 100644--- a/gnu/services/guix.scm+++ b/gnu/services/guix.scm@@ -17,20 +17,40 @@ ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. (define-module (gnu services guix)+ #:use-module (srfi srfi-1) #:use-module (ice-9 match) #:use-module (guix gexp) #:use-module (guix records)+ #:use-module (guix packages) #:use-module ((gnu packages base) #:select (glibc-utf8-locales)) #:use-module (gnu packages admin)+ #:use-module (gnu packages databases) #:use-module (gnu packages web)+ #:use-module (gnu packages guile)+ #:use-module (gnu packages guile-xyz)+ #:use-module (gnu packages package-management) #:use-module (gnu services) #:use-module (gnu services base) #:use-module (gnu services admin) #:use-module (gnu services shepherd) #:use-module (gnu services getmail) #:use-module (gnu system shadow)- #:export (<guix-data-service-configuration>+ #:export (guix-build-coordinator-configuration+ guix-build-coordinator-configuration?+ guix-build-coordinator-configuration-package+ guix-build-coordinator-configuration-user+ guix-build-coordinator-configuration-group+ guix-build-coordinator-configuration-datastore-uri-string+ guix-build-coordinator-configuration-agent-communication-uri-string+ guix-build-coordinator-configuration-client-communication-uri-string+ guix-build-coordinator-configuration-allocation-strategy+ guix-build-coordinator-configuration-hooks+ guix-build-coordinator-configuration-guile++ guix-build-coordinator-service-type++ <guix-data-service-configuration> guix-data-service-configuration guix-data-service-configuration? guix-data-service-package@@ -45,11 +65,189 @@ ;;;; Commentary: ;;;-;;; This module implements a service that to run instances of the Guix Data-;;; Service, which provides data about Guix over time.+;;; Services specifically related to GNU Guix. ;;; ;;;; Code: +(define-record-type* <guix-build-coordinator-configuration>+ guix-build-coordinator-configuration make-guix-build-coordinator-configuration+ guix-build-coordinator-configuration?+ (package guix-build-coordinator-configuration-package+ (default guix-build-coordinator))+ (user guix-build-coordinator-configuration-user+ (default "guix-build-coordinator"))+ (group guix-build-coordinator-configuration-group+ (default "guix-build-coordinator"))+ (database-uri-string+ guix-build-coordinator-configuration-datastore-uri-string+ (default "sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"))+ (agent-communication-uri-string+ guix-build-coordinator-configuration-agent-communication-uri-string+ (default "http://0.0.0.0:8745"))+ (client-communication-uri-string+ guix-build-coordinator-configuration-client-communication-uri-string+ (default "http://127.0.0.1:8746"))+ (allocation-strategy+ guix-build-coordinator-configuration-allocation-strategy+ (default #~basic-build-allocation-strategy))+ (hooks guix-build-coordinator-configuration-hooks+ (default '()))+ (guile guix-build-coordinator-configuration-guile+ (default guile-3.0-latest)))++(define* (make-guix-build-coordinator-start-script database-uri-string+ allocation-strategy+ pid-file+ guix-build-coordinator-package+ #:key+ agent-communication-uri-string+ client-communication-uri-string+ (hooks '())+ (guile guile-3.0))+ (program-file+ "start-guix-build-coordinator"+ (with-extensions (cons guix-build-coordinator-package+ ;; This is a poorly constructed Guile load path,+ ;; since it contains things that aren't Guile+ ;; libraries, but it means that the Guile libraries+ ;; needed for the Guix Build Coordinator don't need+ ;; to be individually specified here.+ (map second (package-inputs+ guix-build-coordinator-package)))+ #~(begin+ (use-modules (srfi srfi-1)+ (ice-9 match)+ (web uri)+ (prometheus)+ (guix-build-coordinator hooks)+ (guix-build-coordinator datastore)+ (guix-build-coordinator build-allocator)+ (guix-build-coordinator coordinator))++ ;; Otherwise sqitch can't use SQlite+ (setenv "PERL5LIB"+ #$(file-append perl-dbd-sqlite "/lib/perl5/site_perl"))++ (let* ((metrics-registry (make-metrics-registry+ #:namespace+ "guixbuildcoordinator_"))+ (datastore (database-uri->datastore+ #$database-uri-string+ #:metrics-registry metrics-registry))+ (hooks+ (list #$@(map (match-lambda+ ((name . hook-gexp)+ #~(cons name #$hook-gexp)))+ hooks)))+ (hooks-with-defaults+ `(,@hooks+ ,@(remove (match-lambda+ ((name . _) (assq-ref hooks name)))+ %default-hooks)))+ (build-coordinator (make-build-coordinator+ #:datastore datastore+ #:hooks hooks-with-defaults+ #:metrics-registry metrics-registry+ #:allocation-strategy #$allocation-strategy)))++ (run-coordinator-service+ build-coordinator+ #:update-datastore? #t+ #:pid-file #$pid-file+ #:agent-communication-uri (string->uri+ #$agent-communication-uri-string)+ #:client-communication-uri (string->uri+ #$client-communication-uri-string)))))+ #:guile guile))++(define (guix-build-coordinator-shepherd-services config)+ (match-record config <guix-build-coordinator-configuration>+ (package user group database-uri-string+ agent-communication-uri-string+ client-communication-uri-string+ allocation-strategy+ hooks+ guile)+ (list+ (shepherd-service+ (documentation "Guix Build Coordinator")+ (provision '(guix-build-coordinator))+ (requirement '(networking))+ (start #~(make-forkexec-constructor+ (list #$(make-guix-build-coordinator-start-script+ database-uri-string+ allocation-strategy+ "/var/run/guix-build-coordinator/pid"+ package+ #:agent-communication-uri-string+ agent-communication-uri-string+ #:client-communication-uri-string+ client-communication-uri-string+ #:hooks hooks+ #:guile guile))+ #:user #$user+ #:group #$group+ #:pid-file "/var/run/guix-build-coordinator/pid"+ ;; Allow time for migrations to run+ #:pid-file-timeout 60+ #:environment-variables+ `(,(string-append+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")+ "LC_ALL=en_US.utf8")+ #:log-file "/var/log/guix-build-coordinator/coordinator.log"))+ (stop #~(make-kill-destructor))))))++(define (guix-build-coordinator-activation config)+ #~(begin+ (use-modules (guix build utils))++ (define %user (getpw "guix-build-coordinator"))++ (chmod "/var/lib/guix-build-coordinator" #o755)++ (mkdir-p "/var/log/guix-build-coordinator")++ ;; Allow writing the PID file+ (mkdir-p "/var/run/guix-build-coordinator")+ (chown "/var/run/guix-build-coordinator"+ (passwd:uid %user)+ (passwd:gid %user))))++(define (guix-build-coordinator-account config)+ (match-record config <guix-build-coordinator-configuration>+ (user group)+ (list (user-group+ (name group)+ (system? #t))+ (user-account+ (name user)+ (group group)+ (system? #t)+ (comment "Guix Build Coordinator user")+ (home-directory "/var/lib/guix-build-coordinator")+ (shell (file-append shadow "/sbin/nologin"))))))++(define guix-build-coordinator-service-type+ (service-type+ (name 'guix-build-coordinator)+ (extensions+ (list+ (service-extension shepherd-root-service-type+ guix-build-coordinator-shepherd-services)+ (service-extension activation-service-type+ guix-build-coordinator-activation)+ (service-extension account-service-type+ guix-build-coordinator-account)))+ (default-value+ (guix-build-coordinator-configuration))+ (description+ "Run an instance of the Guix Build Coordinator.")))++ +;;;+;;; Guix Data Service+;;;+ (define-record-type* <guix-data-service-configuration> guix-data-service-configuration make-guix-data-service-configuration guix-data-service-configuration?diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scmindex 6139e31cf0..20b67d55d3 100644--- a/gnu/tests/guix.scm+++ b/gnu/tests/guix.scm@@ -35,7 +35,80 @@ #:use-module (guix store) #:use-module (guix utils) #:use-module (ice-9 match)- #:export (%test-guix-data-service))+ #:export (%test-guix-build-coordinator+ %test-guix-data-service))++;;;+;;; Guix Build Coordinator+;;;++(define %guix-build-coordinator-os+ (simple-operating-system+ (service dhcp-client-service-type)+ (service guix-build-coordinator-service-type)))++(define (run-guix-build-coordinator-test)+ (define os+ (marionette-operating-system+ %guix-build-coordinator-os+ #:imported-modules '((gnu services herd)+ (guix combinators))))++ (define forwarded-port 8745)++ (define vm+ (virtual-machine+ (operating-system os)+ (memory-size 1024)+ (port-forwardings `((,forwarded-port . 8745)))))++ (define test+ (with-imported-modules '((gnu build marionette))+ #~(begin+ (use-modules (srfi srfi-11) (srfi srfi-64)+ (gnu build marionette)+ (web uri)+ (web client)+ (web response))++ (define marionette+ (make-marionette (list #$vm)))++ (mkdir #$output)+ (chdir #$output)++ (test-begin "guix-build-coordinator")++ (test-assert "service running"+ (marionette-eval+ '(begin+ (use-modules (gnu services herd))+ (match (start-service 'guix-build-coordinator)+ (#f #f)+ (('service response-parts ...)+ (match (assq-ref response-parts 'running)+ ((pid) (number? pid))))))+ marionette))++ (test-equal "http-get"+ 200+ (let-values+ (((response text)+ (http-get #$(simple-format+ #f "http://localhost:~A/metrics" forwarded-port)+ #:decode-body? #t)))+ (response-code response)))++ (test-end)+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))++ (gexp->derivation "guix-build-coordinator-test" test))++(define %test-guix-build-coordinator+ (system-test+ (name "guix-build-coordinator")+ (description "Connect to a running Guix Build Coordinator.")+ (value (run-guix-build-coordinator-test)))) ;;;-- 2.28.0
L
L
Ludovic Courtès wrote on 18 Sep 2020 22:08
Re: [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87lfh6vodj.fsf@gnu.org
Hi!
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (2 lines)> * gnu/packages/package-management.scm (guix-build-coordinator): New variable.
[...]
Toggle quote (10 lines)> + (lambda* (#:key inputs outputs #:allow-other-keys)> + (let* ((out (assoc-ref outputs "out"))> + (bin (string-append out "/bin"))> + (guile (assoc-ref inputs "guile"))> + (guile-effective-version> + (read-line> + (open-pipe* OPEN_READ> + (string-append guile "/bin/guile")> + "-c" "(display (effective-version))")))
Maybe use ‘target-guile-effective-version’ from (guix buildguile-build-system).
Toggle quote (6 lines)> + (wrap-program (string-append bin "/" file)> + `("PATH" ":" prefix> + (,bin ,(assoc-ref inputs "sqitch")))> + `("PERL5LIB" ":" prefix> + (,(getenv "PERL5LIB")))
Do we really need PERL5LIB here? Shouldn’t it be done in the wrapper of‘sqitch’ directly?
Toggle quote (5 lines)> + (scandir bin> + (match-lambda> + ((or "." "..") #f)> + (_ #t))))
I think you could just use (find-files bin).
Toggle quote (3 lines)> + (inputs> + `(("guile" ,guile-3.0-latest)
Rather ,@(assoc-ref (package-native-inputs guix) "guile").
Toggle quote (3 lines)> + ("sqitch" ,sqitch)> + ("perl-dbd-sqlite" ,perl-dbd-sqlite)))
Shouldn’t perl-dbd-sqlite be a dependency of sqitch?
Toggle quote (5 lines)> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")> + (synopsis "")> + (description> + "")
Missing! :-)
LGTM with changes along these lines!
Ludo’.
L
L
Ludovic Courtès wrote on 18 Sep 2020 22:20
Re: [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87a6xmvnth.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (19 lines)> * gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record> type.> (guix-build-coordinator-configuration, guix-build-coordinator-configuration?,> guix-build-coordinator-configuration-package,> guix-build-coordinator-configuration-user,> guix-build-coordinator-configuration-group,> guix-build-coordinator-configuration-datastore-uri-string,> guix-build-coordinator-configuration-agent-communication-uri-string,> guix-build-coordinator-configuration-client-communication-uri-string,> guix-build-coordinator-configuration-allocation-strategy,> guix-build-coordinator-configuration-hooks,> guix-build-coordinator-configuration-guile,> make-guix-build-coordinator-start-script,> guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,> guix-build-coordinator-account): New procedures.> (guix-build-coordinator-service-type): New variable.> * gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.> * doc/guix.texi (Guix Services): Document it.
Yay!
Toggle quote (3 lines)> +@subsubheading Guix Build Coordinator> +The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build> +Coordinator} aids in building derivations. The Guix Daemon is still
^Maybe something like: “in distributing derivation builds among machinesrunning an @dfn{agent}”.
Also, s/Guix Daemon/build daemon/ or similar.
(In general I’m in favor of avoiding “brands” in documentation andconcept names.)
Maybe add a “@quotation Note” stating that it’s work in progress andsubject to change.
Toggle quote (20 lines)> +(define* (make-guix-build-coordinator-start-script database-uri-string> + allocation-strategy> + pid-file> + guix-build-coordinator-package> + #:key> + agent-communication-uri-string> + client-communication-uri-string> + (hooks '())> + (guile guile-3.0))> + (program-file> + "start-guix-build-coordinator"> + (with-extensions (cons guix-build-coordinator-package> + ;; This is a poorly constructed Guile load path,> + ;; since it contains things that aren't Guile> + ;; libraries, but it means that the Guile libraries> + ;; needed for the Guix Build Coordinator don't need> + ;; to be individually specified here.> + (map second (package-inputs> + guix-build-coordinator-package)))
Perhaps there should eventually be a ‘guix-build-coordinator’ command inthe package itself?
Otherwise LGTM, thanks!
One thing we discussed on IRC is the name and ways to set up one’smonitor so thatguix-build-coordinator-configuration-client-communication-uri-stringfits on one line. :-)
An idea that came to mind was “(Guix) dispatch(er)”, which issynonymous. You could use the (guix dispatch …) name space.Food for thought! (No rush, of course!)
Ludo’.
L
L
Ludovic Courtès wrote on 18 Sep 2020 22:25
Re: [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87363evnmm.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (20 lines)> * gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New> record type.> (guix-build-coordinator-agent-configuration,> guix-build-coordinator-agent-configuration?,> guix-build-coordinator-agent-configuration-package,> guix-build-coordinator-agent-configuration-user,> guix-build-coordinator-agent-configuration-coordinator,> guix-build-coordinator-agent-configuration-uuid),> guix-build-coordinator-agent-configuration-password,> guix-build-coordinator-agent-configuration-password-file,> guix-build-coordinator-agent-configuration-systems,> guix-build-coordinator-agent-configuration-max-parallel-builds,> guix-build-coordinator-agent-configuration-derivation-substitute-urls,> guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,> guix-build-coordinator-agent-shepherd-services,> guix-build-coordinator-agent-activation,> guix-build-coordinator-agent-account): New procedures.> (guix-build-coordinator-agent-service-type): New variable.> * doc/guix.texi (Guix Services): Document it.
[...]
Toggle quote (3 lines)> +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type> +Service type for a Guix Build Coordinator agent. Its value must be a
Perhaps “coordinator” and “agent” should be defined in a few sentencesabove to clarify what this is about.
Toggle quote (8 lines)> +@item @code{derivation-substitute-urls} (default: @code{1})> +URLs from which to attempt to fetch substitutes for derivations, if the> +derivations aren't already available.> +> +@item @code{non-derivation-substitute-urls} (default: @code{1})> +URLs from which to attempt to fetch substitutes for build inputs, if the> +input store items aren't already available.
This is interesting, I wonder how you can distinguish between the two incode. You have to open different sessions, right? Or to call‘set-build-options’ again?
Toggle quote (3 lines)> + (description> + "Run an instance of the Guix Build Coordinator.")))
+ “agent”
Otherwise LGTM!
Ludo’.
L
L
Ludovic Courtès wrote on 18 Sep 2020 22:31
Re: [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87pn6iu8rx.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (16 lines)> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):> New record type.> (guix-build-coordinator-queue-builds-configuration,> guix-build-coordinator-queue-builds-configuration?,> guix-build-coordinator-queue-builds-configuration-package,> guix-build-coordinator-queue-builds-configuration-user,> guix-build-coordinator-queue-builds-coordinator,> guix-build-coordinator-queue-builds-configuration-systems,> guix-build-coordinator-queue-builds-configuration-system-and-targets,> guix-build-coordinator-queue-builds-configuration-guix-data-service,> guix-build-coordinator-queue-builds-configuration-processed-commits-file,> guix-build-coordinator-queue-builds-shepherd-services,> guix-build-coordinator-queue-builds-activation,> guix-build-coordinator-queue-builds-account): New procedures.> (guix-build-coordinator-queue-builds-service-type): New variable.
[...]
Toggle quote (4 lines)> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type> +Service type for the> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its
Oh! :-)
‘guix-build-queue’ maybe? Because after all, it can queue builds comingfrom anywhere, not just from the Data Service, right?
It’d be nice to add a sentence or two explaining how the “queue” fits inthe big picture (I’m not quite sure :-)).
Toggle quote (4 lines)> +@item @code{systems-and-targets} (default: @code{#f})> +An association list of system and target pairs for which to fetch> +derivations to build.
Cross-compilation target triplets are a notion that’s not available atthe derivation level; it only exists for packages.
Are we mixing things here?
Toggle quote (4 lines)> +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})> +The Guix Data Service instance from which to query to find out about> +derivations to build.
Ah so the queue pulls from the Data Service. Got it!
Toggle quote (4 lines)> +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})> +A file to record which commits have been processed, to avoid needlessly> +processing them again if the service is restarted.
Maybe in /var/cache by default, no?
Toggle quote (4 lines)> + (processed-commits-file> + guix-build-coordinator-queue-builds-configuration-processed-commits-file> + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))
Maybe “state-file”? In addition to saving space :-), it’d leave roomfor additional bits of state.
That’s all, thank you!
Ludo’.
J
J
Jonathan Brielmaier wrote on 19 Sep 2020 01:34
Re: [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
(address . 43494@debbugs.gnu.org)
4332de6e-8002-1967-4838-3448df888242@web.de
On 18.09.20 22:08, Ludovic Courtès wrote:
Toggle quote (2 lines)>> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")
C
C
Christopher Baines wrote on 19 Sep 2020 11:00
(name . Jonathan Brielmaier)(address . jonathan.brielmaier@web.de)
87ft7e9m49.fsf@cbaines.net
Jonathan Brielmaier <jonathan.brielmaier@web.de> writes:
Toggle quote (5 lines)> On 18.09.20 22:08, Ludovic Courtès wrote:>>> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")>> It should be https://git.cbaines.net/guix/build-coordinator/
Thanks Jonathan, I've fixed this now :)
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9lyMZfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9XcELBAApxwnjfJu/B3H9p8cXiZCTl99ejaJXvKfT8Utx4aB4YpFg+ZR7KqlqiJ0dUr1hXpMzRX31IoUrcIk2mc2DUw5GQjM2lq3mUxESsW8A7ODEU8WfW5KcwLfes4g7M+jZlPMEJYpfzV2ryvFyBrWErv76u/9d4vcAXIvjIznhEPZbrMyEaW5T0CMUCdzULZuBB6PuN7NlV6atYh0Jy7FL83LY5hN+EX0T8ecJ+3pOrlZQWcTJitRsaoCtbvnnYi3pSQjhuERfb6c4n+lK3NpX4StyMJWW86hF65CsYZhpVsG5xT/kI2R3xf2Ux/hH4oVrhVxzLIRKPHZEFu2+48O6Pno2QkPam+VYl2Jc6pM5qJuRWrASbXPCu5lhlePZLoZPtVqkqIdAZDc3ItUW47hzihpEzsn2a7/rw5c6yh3L53q/edF8jFDYFYF2/xaj46BG2UKYFpFSvnwUcbJJvNiVpZECcSzFpOYN1kb4z21TuuILL/BTzdEyUcPlH/QjVz1lP4ElGAVJPnOltTQYW8syElS0BCc0yhz9KOGeFGJmn3u5TL32ViBJ29p9q2lR2OitOhLwRZxJwFm64i7+smjkps3DxYEaDkZJnHdA5bGYlk4ojxk7MsDFD42NYh6kRoBQF7l1kjHXI9PGxFRMI8T9Gri1HZESEo4bVCTnS6BW6TqPc4==+xCv-----END PGP SIGNATURE-----
C
C
Christopher Baines wrote on 19 Sep 2020 11:05
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
87d02i9lx9.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (21 lines)> Hi!>> Christopher Baines <mail@cbaines.net> skribis:>>> * gnu/packages/package-management.scm (guix-build-coordinator): New variable.>> [...]>>> + (lambda* (#:key inputs outputs #:allow-other-keys)>> + (let* ((out (assoc-ref outputs "out"))>> + (bin (string-append out "/bin"))>> + (guile (assoc-ref inputs "guile"))>> + (guile-effective-version>> + (read-line>> + (open-pipe* OPEN_READ>> + (string-append guile "/bin/guile")>> + "-c" "(display (effective-version))")))>> Maybe use ‘target-guile-effective-version’ from (guix build> guile-build-system).
I've changed this. I'll also try and remember and update theguix-data-service package accordingly, as that's where I copied thisfrom.
Toggle quote (9 lines)>> + (wrap-program (string-append bin "/" file)>> + `("PATH" ":" prefix>> + (,bin ,(assoc-ref inputs "sqitch")))>> + `("PERL5LIB" ":" prefix>> + (,(getenv "PERL5LIB")))>> Do we really need PERL5LIB here? Shouldn’t it be done in the wrapper of> ‘sqitch’ directly?
Hmm, when I packaged sqitch, I didn't add all the database drivers whichare optional dependencies, but perl-dbd-pg did end up in there somehow(probably because that's needed by the guix-data-service).
I've now gone ahead and added the database drivers which Guix haspackages for [1], it does increase the closure size a bit, but it wasquite big anyway.
1:http://git.savannah.gnu.org/cgit/guix.git/commit/?id=6a1c3a908c8aadfde5b5fe405132839683a01776
All that means I can remove the wrapping here.
Toggle quote (7 lines)>> + (scandir bin>> + (match-lambda>> + ((or "." "..") #f)>> + (_ #t))))>> I think you could just use (find-files bin).
Done.
Toggle quote (5 lines)>> + (inputs>> + `(("guile" ,guile-3.0-latest)>> Rather ,@(assoc-ref (package-native-inputs guix) "guile").
Done.
Toggle quote (5 lines)>> + ("sqitch" ,sqitch)>> + ("perl-dbd-sqlite" ,perl-dbd-sqlite)))>> Shouldn’t perl-dbd-sqlite be a dependency of sqitch?
See above.
Toggle quote (7 lines)>> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")>> + (synopsis "")>> + (description>> + "")>> Missing! :-)
Ah, yeah, I should have remembered to lint the package. I've attemptedto write something now, I'm still working on trying to explain what theGuix Build Coordinator is meant to be.
Toggle quote (2 lines)> LGTM with changes along these lines!
Thanks!
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9lycJfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9XdAdQ/8CgG11hrFpUwT8jxkToEexip5q3Gxt9MwtUK7nEmN6YmiRKO+5aEiWbGlrbLvKuxgtg673rb5+Th3LTUcprxlgh3SGFIXINHHqy3ybzyGm2vOFBDlZ1I8rbsSHr195bVVHsirWQJzqPk0c4ukfaxEKc9V/QFa+VllJs/LlTobbuo8oqwkQ5wC6wUJO+ZTJrXZpjDclFXGNGDEGGXJ2KnhxG7CMLt/SrgTrMy/0tpLyHsPYP1OXOOe7YQCdDXAx5zN3EkNSOCGWygi2QGiocRunVKJzhAWJr5WE62Vzn9/XxbuMxECs30kQuzaAVKHA5q5hCL2kAFm80goM8nmdYzxZ/tMENpgySCb5FF1pcGQVpLjlsodao4AHPl9c2x3cgr4DB8wiNx26O5ENEuR1vfInwlQVQz8zvd9tseD8BRh4bk+3f28ZXWs/VDzZGCQ67Vp83+3poAV12+ScHvFu9MNX3jwICEHtISNnhtYlw8GS+icOpe1pmFCf4D64kxyXmj595OcZjv2xfVUso32XccTmTl9+c52Djj+SLlu91mJ20jJt/flxvv7uK6uT5SwfpHX+RThnJMFHjjwoFPV7CsyuTb9sfKDjgrf7hMepHYuX7jGqDjjrMMoXvco4/0YlIIKP+gR3OvjTyg+YZVP/Fulbjm7xkgkHNZPxS6bXQJrKho==hrmM-----END PGP SIGNATURE-----
C
C
Christopher Baines wrote on 19 Sep 2020 11:13
Re: [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
87a6xm9lih.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (38 lines)> Christopher Baines <mail@cbaines.net> skribis:>>> * gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record>> type.>> (guix-build-coordinator-configuration, guix-build-coordinator-configuration?,>> guix-build-coordinator-configuration-package,>> guix-build-coordinator-configuration-user,>> guix-build-coordinator-configuration-group,>> guix-build-coordinator-configuration-datastore-uri-string,>> guix-build-coordinator-configuration-agent-communication-uri-string,>> guix-build-coordinator-configuration-client-communication-uri-string,>> guix-build-coordinator-configuration-allocation-strategy,>> guix-build-coordinator-configuration-hooks,>> guix-build-coordinator-configuration-guile,>> make-guix-build-coordinator-start-script,>> guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,>> guix-build-coordinator-account): New procedures.>> (guix-build-coordinator-service-type): New variable.>> * gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.>> * doc/guix.texi (Guix Services): Document it.>> Yay!>>> +@subsubheading Guix Build Coordinator>> +The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build>> +Coordinator} aids in building derivations. The Guix Daemon is still> ^> Maybe something like: “in distributing derivation builds among machines> running an @dfn{agent}”.>> Also, s/Guix Daemon/build daemon/ or similar.>> (In general I’m in favor of avoiding “brands” in documentation and> concept names.)>> Maybe add a “@quotation Note” stating that it’s work in progress and> subject to change.
I've made these changes now.
Toggle quote (23 lines)>> +(define* (make-guix-build-coordinator-start-script database-uri-string>> + allocation-strategy>> + pid-file>> + guix-build-coordinator-package>> + #:key>> + agent-communication-uri-string>> + client-communication-uri-string>> + (hooks '())>> + (guile guile-3.0))>> + (program-file>> + "start-guix-build-coordinator">> + (with-extensions (cons guix-build-coordinator-package>> + ;; This is a poorly constructed Guile load path,>> + ;; since it contains things that aren't Guile>> + ;; libraries, but it means that the Guile libraries>> + ;; needed for the Guix Build Coordinator don't need>> + ;; to be individually specified here.>> + (map second (package-inputs>> + guix-build-coordinator-package)))>> Perhaps there should eventually be a ‘guix-build-coordinator’ command in> the package itself?
There actually is, one thing I've had in mind for a while now though isto use a scheme script constructed by the Guix service to run thecoordinator.
For guix.cbaines.net, I'm using the script, but with the hooks passed inon the command line, the command is rather long, and it means thatbacktraces don't work well with the hooks.
I'm unsure how well this has worked out, I didn't anticipate the issueswith the Guile load path, and because of the formatting stripping, boththe code and backtraces will be unreadable anyway... but it is useful inallowing G-expressions to be used as part of the configuration.
Toggle quote (9 lines)> One thing we discussed on IRC is the name and ways to set up one’s> monitor so that> guix-build-coordinator-configuration-client-communication-uri-string> fits on one line. :-)>> An idea that came to mind was “(Guix) dispatch(er)”, which is> synonymous. You could use the (guix dispatch …) name space.> Food for thought! (No rush, of course!)
Indeed, I think that's a strong candidate :)
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9ly9ZfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9XdVWg/+KZ3LZDolesoD7M43Bg2HBcI4SB2nou7+rT5iiYXgcX17kZjX/PG6SWbSDgdfJ+su/n59Hc9galUyZTLkbr52Yl8gByj35CQmWJJ0SJPzzvEdml6Ng7/KPOMrLN1YHvFxVfhXn+tZbExxV98LBGqWIualpoUd2I7qJvrA+11QAUXyLOybkhCk/lJzrrJRCjKoXP4mjxVBj6EgFB1s6CNXPI97kmJJvEHEp1V4kC8rym3kOkfYiuaZAmnZaZil9CslNLOtFBhiheXaZWgGzvzx7ifFe/7oV63zfghg/AzoPwHVjvIdRrly7b1teFJ5O3E3OOawXMAMV/K1+rPx/eTHaVyufex27wZcZpEv1UkX/7oMv6/XSVp0gOODUpmDCNwSAmrvApE+rh1LowGIDBHYBa+P03RZ+4DYH68jT87C/O3Vz9WbDDTbPow9KanFLJbR80sYaD/xO5Ni8cZ+o6DmxrjLPv55EdLqds1TQ2Bp7x2AG4Ki3VoMrFolH94YCS99hck5gXghZJZpsrUaclSDBUDET541OrBFtctfG8WksstGovRMnWO4u2wO3JjQJTTdfr4yAqOVWUi3PE1JdZB1aZENE4XgZeNPBZCKVNt5noAV0OH/SN66noDkod5DslxrE/KUmgagTiEWOAl6noCmIRH+jSBrgqUJrGMSL93HwTM==tSTq-----END PGP SIGNATURE-----
C
C
Christopher Baines wrote on 19 Sep 2020 11:49
Re: [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
875z8a9jvz.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (30 lines)> Christopher Baines <mail@cbaines.net> skribis:>>> * gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New>> record type.>> (guix-build-coordinator-agent-configuration,>> guix-build-coordinator-agent-configuration?,>> guix-build-coordinator-agent-configuration-package,>> guix-build-coordinator-agent-configuration-user,>> guix-build-coordinator-agent-configuration-coordinator,>> guix-build-coordinator-agent-configuration-uuid),>> guix-build-coordinator-agent-configuration-password,>> guix-build-coordinator-agent-configuration-password-file,>> guix-build-coordinator-agent-configuration-systems,>> guix-build-coordinator-agent-configuration-max-parallel-builds,>> guix-build-coordinator-agent-configuration-derivation-substitute-urls,>> guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,>> guix-build-coordinator-agent-shepherd-services,>> guix-build-coordinator-agent-activation,>> guix-build-coordinator-agent-account): New procedures.>> (guix-build-coordinator-agent-service-type): New variable.>> * doc/guix.texi (Guix Services): Document it.>> [...]>>> +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type>> +Service type for a Guix Build Coordinator agent. Its value must be a>> Perhaps “coordinator” and “agent” should be defined in a few sentences> above to clarify what this is about.
I've added a bit more information, I want to at some point actually givea workable minimal example configuration, but I need to do some morethinking about that.
Toggle quote (12 lines)>> +@item @code{derivation-substitute-urls} (default: @code{1})>> +URLs from which to attempt to fetch substitutes for derivations, if the>> +derivations aren't already available.>> +>> +@item @code{non-derivation-substitute-urls} (default: @code{1})>> +URLs from which to attempt to fetch substitutes for build inputs, if the>> +input store items aren't already available.>> This is interesting, I wonder how you can distinguish between the two in> code. You have to open different sessions, right? Or to call> ‘set-build-options’ again?
Yeah, it's not strictly necessary to have some separation. In fact, Ican't remember exactly, but I think it should be possible to deploy thecoordinator and agents without explicit configuration of the substituteURLs, but I need to check that.
These options were added mostly because it useful if you have twoseperate sources of substitutes, one for just derivations (say aninstance of the Guix Data Service), and one just for build outputs (saypopulated by one of the hooks) [1].
1: https://git.cbaines.net/guix/build-coordinator/commit/?id=bda7d58853ed4fba976cac92a70c4dc68db263aa
Toggle quote (7 lines)>> + (description>> + "Run an instance of the Guix Build Coordinator.")))>> + “agent”>> Otherwise LGTM!
Thanks!
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9l1BBfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9XdqNA/+PAeN8zL9P608PFjf1IDPZZWx76WSAMZ6BfCStsRvGNR/X+ZOgzabH6+KZiJoAbvC0RIzFgGh90CfYI2GabG6SYTkQMAi4DN/Cs3iKFVG+DDvdpprmk9xDTFbk+MnarQzw0sSidtZb8OL9E9u+c3evfDO6AMl9y2vKhfGDV6A5IvIcNJy1t8TM8OLS/qcd8O2kIlLX2LBKoGJuhiszXt1MFYrXmbSLDEd+iZ0wcFXHaG2TV/y5UbqydP30MaCKAym2YN5NNK98u+z94TImZeppCeQGGY5OnN0Cq29TaIn8egDX1NJjFkV4OzT+WwM2jY14oFvOZHblS5uqd0gd3I5yi/NX41K6CFjAOX43ADlpQSeLF6CtPZAJbWSwNn+3pmBdtyuRJibe0GbtaAn5+0CvJ1L58gHzXIp22Vaz+yW5qoAVEOMuEx0a6ftuMJ6RzjW9FGtvNzuyzcvCbeMmB5Wc3IuoTiPA6ZHBuj1KvdDBwD32zskPwUKk3Cu1G7KBkHmAMOF28Z/Ci0Lr1vewKW6iR+6pIiXD6gxRiaGLZaz/8Ck3eweYa+llidUYBkz/UzGMZggWegnABCMv+6bTRgDTwVFTH9QNlBmx1gJRYayx8+xclGNR/njcHH5VR93AvkYiHmXPnpEGeXncGJwN8jTFoW4UP5Mb1BPT1geWj8xmzQ==RwqV-----END PGP SIGNATURE-----
C
C
Christopher Baines wrote on 19 Sep 2020 12:05
Re: [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
87363e9j3u.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (32 lines)> Christopher Baines <mail@cbaines.net> skribis:>>> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):>> New record type.>> (guix-build-coordinator-queue-builds-configuration,>> guix-build-coordinator-queue-builds-configuration?,>> guix-build-coordinator-queue-builds-configuration-package,>> guix-build-coordinator-queue-builds-configuration-user,>> guix-build-coordinator-queue-builds-coordinator,>> guix-build-coordinator-queue-builds-configuration-systems,>> guix-build-coordinator-queue-builds-configuration-system-and-targets,>> guix-build-coordinator-queue-builds-configuration-guix-data-service,>> guix-build-coordinator-queue-builds-configuration-processed-commits-file,>> guix-build-coordinator-queue-builds-shepherd-services,>> guix-build-coordinator-queue-builds-activation,>> guix-build-coordinator-queue-builds-account): New procedures.>> (guix-build-coordinator-queue-builds-service-type): New variable.>> [...]>>> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type>> +Service type for the>> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its>> Oh! :-)>> ‘guix-build-queue’ maybe? Because after all, it can queue builds coming> from anywhere, not just from the Data Service, right?>> It’d be nice to add a sentence or two explaining how the “queue” fits in> the big picture (I’m not quite sure :-)).
So, this got me thinking a bit, maybe this script should actually existoutside of the guix-build-coordinator repository...
You can run `guix-build-coordinator build ...` to build a derivation,and what this does is just make the relevant HTTP request to thecoordinator.
What this script does is just fetch derivation names from an instance ofthe Guix Data Service, and ask the Guix Build Coordinator to buildthem.
Toggle quote (9 lines)>> +@item @code{systems-and-targets} (default: @code{#f})>> +An association list of system and target pairs for which to fetch>> +derivations to build.>> Cross-compilation target triplets are a notion that’s not available at> the derivation level; it only exists for packages.>> Are we mixing things here?
This setting is basically configuing the query parameters to use forthis page for example [1]. You're right that it's not available at thederivation level, but you can ask the Guix Data Service for derivationsfor packages which were computed for a specific system and target.
1: http://data.guix.gnu.org/revision/70ef8b24550c54cc8e9f20026bfd24b8680499b4/package-derivations
Toggle quote (6 lines)>> +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})>> +The Guix Data Service instance from which to query to find out about>> +derivations to build.>> Ah so the queue pulls from the Data Service. Got it!
I wouldn't read too much in to "queue" in the name here, "submit" mightbe a better word to use, since that's the term used on the Guix BuildCoordinator side. There's also nothing resemling a queue anywhere...
Toggle quote (6 lines)>> +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})>> +A file to record which commits have been processed, to avoid needlessly>> +processing them again if the service is restarted.>> Maybe in /var/cache by default, no?
Ah, yeah, I've changed it to be in /var/cache, as that's fine.
Toggle quote (7 lines)>> + (processed-commits-file>> + guix-build-coordinator-queue-builds-configuration-processed-commits-file>> + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))>> Maybe “state-file”? In addition to saving space :-), it’d leave room> for additional bits of state.
Currently, it's just a file with each commit on its own line. Giventhat, if there's a need to store more state, it might just end up indifferent files. If that happens, this setting could become a directory,rather than a specific file to avoid adding extra configuration options.
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9l2AVfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9XdCjQ/8DIg3yovjZeLdZteOJnB4JMRyFhx+VTZPbTVEDJcT8BL8gwqiZrVKsiLWCz+vu62Bwiohk0zrXKLTuqTfsvq/NGnqWm4lt+cz6OiH8guX5CfvuXeLNXWh0t6SPA6O8ieRry/ar+DFQBUEF20XNUIb/hCGhykcdjKC1vLFac4npBaYxRyIwCwiyMLGPrugkmFveh93Xe9E7qqByT6DEPa22zw44zDrzRFWUP3Zbpv+fFuA+MwDg/BjS7lfxGuc0tfI8XVBWwsbnmdMUUNuHneBf5mYRhMDBFqjP+ZrWDq8Z2PHyIUFvie9iaC1phgwQvjUwXe5w2H5k0rav+izwyjYe2Lg8GgBZsn2j9DV0FlRq3AhM0Ex96wCs6jsxbDcAKeuKqaLKwbBoRd40uo+0Aoo5FBo7lVije0M3S4JaQjqGPxqR1HSCPz8VMxoXoKzq36kmZQEly8XLgSf/lWFLkVvEgLvBR4g5bFMVLKxKEK2lHSEa1jfxDc1Lz6V6/4qVF/bMN8SUqU7ZvRr6klUUbQ2OaTV5Csl5fG1jlZNWDXOTlEKXcgUq+jbU0QECoM1OED+rppjSXe1zbaubWZKKXk1Vq3Ix7uAIg3G7fwd0rD5FPjTXEdqtAZ4KLOOREYJt6fooebFrPdxqPOGEJQ8IHXBvlOK305U3/GplY2iP70kxrg==d1v4-----END PGP SIGNATURE-----
C
C
Christopher Baines wrote on 19 Sep 2020 12:10
[PATCH v2 1/4] gnu: Add guix-build-coordinator.
(address . 43494@debbugs.gnu.org)
20200919101059.8930-1-mail@cbaines.net
* gnu/packages/package-management.scm (guix-build-coordinator): New variable.--- gnu/packages/package-management.scm | 74 +++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+)
Toggle diff (94 lines)diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scmindex ec87226197..afa45d6789 100644--- a/gnu/packages/package-management.scm+++ b/gnu/packages/package-management.scm@@ -50,6 +50,7 @@ #:use-module (gnu packages cpio) #:use-module (gnu packages crypto) #:use-module (gnu packages curl)+ #:use-module (gnu packages databases) #:use-module (gnu packages dbm) #:use-module (gnu packages docbook) #:use-module (gnu packages file)@@ -981,6 +982,79 @@ environments.") ;; and the fonts included in this package are licensed OFL1.1. (license (list license:gpl3+ license:agpl3+ license:silofl1.1)))) +(define-public guix-build-coordinator+ (let ((commit "5e8408c833e209efddfa0159114b90400c1aaf4d")+ (revision "0"))+ (package+ (name "guix-build-coordinator")+ (version (git-version "0" revision commit))+ (source (origin+ (method git-fetch)+ (uri (git-reference+ (url "https://git.cbaines.net/git/guix/build-coordinator")+ (commit commit)))+ (sha256+ (base32+ "0f7m1zg9mlb2m22qyblglaa36h8f49b810jc9j5b0hsb42ijwh4b"))+ (file-name (string-append name "-" version "-checkout"))))+ (build-system gnu-build-system)+ (arguments+ `(#:modules (((guix build guile-build-system)+ #:select (target-guile-effective-version))+ ,@%gnu-build-system-modules)+ #:imported-modules ((guix build guile-build-system)+ ,@%gnu-build-system-modules)+ #:phases+ (modify-phases %standard-phases+ (add-before 'build 'set-GUILE_AUTO_COMPILE+ (lambda _+ ;; To avoid warnings relating to 'guild'.+ (setenv "GUILE_AUTO_COMPILE" "0")+ #t))+ (add-after 'install 'wrap-executable+ (lambda* (#:key inputs outputs #:allow-other-keys)+ (let* ((out (assoc-ref outputs "out"))+ (bin (string-append out "/bin"))+ (guile (assoc-ref inputs "guile"))+ (version (target-guile-effective-version))+ (scm (string-append out "/share/guile/site/" version))+ (go (string-append out "/lib/guile/" version "/site-ccache")))+ (for-each+ (lambda (file)+ (simple-format (current-error-port) "wrapping: ~A\n" file)+ (wrap-program file+ `("PATH" ":" prefix+ (,bin ,(assoc-ref inputs "sqitch")))+ `("GUILE_LOAD_PATH" ":" prefix+ (,scm ,(getenv "GUILE_LOAD_PATH")))+ `("GUILE_LOAD_COMPILED_PATH" ":" prefix+ (,go ,(getenv "GUILE_LOAD_COMPILED_PATH")))))+ (find-files bin)))+ #t))+ (delete 'strip)))) ; As the .go files aren't compatible+ (native-inputs+ `(("pkg-config" ,pkg-config)+ ("autoconf" ,autoconf)+ ("automake" ,automake)))+ (inputs+ `(("guile" ,@(assoc-ref (package-native-inputs guix) "guile"))+ ("guile-fibers" ,guile-fibers)+ ("guile-prometheus" ,guile-prometheus)+ ("guile-gcrypt" ,guile-gcrypt)+ ("guile-json" ,guile-json-3)+ ("guile-lzlib" ,guile-lzlib)+ ("guile-sqlite3" ,guile-sqlite3)+ ("guix" ,guix)+ ("sqlite" ,sqlite)+ ("sqitch" ,sqitch)))+ (home-page "https://git.cbaines.net/guix/build-coordinator/")+ (synopsis "Tool to help build derivations")+ (description+ "The Guix Build Coordinator helps with performing lots of builds across+potentially many machines, and with doing something with the results and+outputs of those builds.")+ (license license:gpl3+))))+ (define-public guix-jupyter (package (name "guix-jupyter")-- 2.28.0
C
C
Christopher Baines wrote on 19 Sep 2020 12:10
[PATCH v2 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
(address . 43494@debbugs.gnu.org)
20200919101059.8930-3-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): Newrecord type.(guix-build-coordinator-agent-configuration,guix-build-coordinator-agent-configuration?,guix-build-coordinator-agent-configuration-package,guix-build-coordinator-agent-configuration-user,guix-build-coordinator-agent-configuration-coordinator,guix-build-coordinator-agent-configuration-uuid),guix-build-coordinator-agent-configuration-password,guix-build-coordinator-agent-configuration-password-file,guix-build-coordinator-agent-configuration-systems,guix-build-coordinator-agent-configuration-max-parallel-builds,guix-build-coordinator-agent-configuration-derivation-substitute-urls,guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,guix-build-coordinator-agent-shepherd-services,guix-build-coordinator-agent-activation,guix-build-coordinator-agent-account): New procedures.(guix-build-coordinator-agent-service-type): New variable.* doc/guix.texi (Guix Services): Document it.--- doc/guix.texi | 50 +++++++++++++++++ gnu/services/guix.scm | 121 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+)
Toggle diff (209 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex 4f3f0e851f..b2dde30ddb 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -27554,6 +27554,56 @@ The Guile package with which to run the Guix Build Coordinator. @end table @end deftp +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type+Service type for a Guix Build Coordinator agent. Its value must be a+@code{guix-build-coordinator-agent-configuration} object.+@end defvar++@deftp {Data Type} guix-build-coordinator-agent-configuration+Data type representing the configuration a Guix Build Coordinator agent.++@table @asis+@item @code{package} (default: @code{guix-build-coordinator})+The Guix Build Coordinator package to use.++@item @code{user} (default: @code{"guix-build-coordinator-agent"})+The system user to run the service as.++@item @code{coordinator} (default: @code{"http://localhost:8745"})+The URI to use when connecting to the coordinator.++@item @code{uuid}+The UUID of the agent. This should be generated by the coordinator+process, stored in the coordinator database, and used by the intended+agent.++@item @code{password} (default: @code{#f})+The password to use when connecting to the coordinator. A file to read+the password from can also be specified, and this is more secure.++@item @code{password-file} (default: @code{#f})+A file containing the password to use when connecting to the+coordinator.++@item @code{systems} (default: @var{#f})+The systems for which this agent should fetch builds. The agent process+will use the current system it's running on as the default.++@item @code{max-parallel-builds} (default: @code{1})+The number of builds to perform in parallel.++@item @code{derivation-substitute-urls} (default: @code{1})+URLs from which to attempt to fetch substitutes for derivations, if the+derivations aren't already available.++@item @code{non-derivation-substitute-urls} (default: @code{1})+URLs from which to attempt to fetch substitutes for build inputs, if the+input store items aren't already available.++@end table+@end deftp++ @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores and provides data about GNU Guix. This includes information aboutdiff --git a/gnu/services/guix.scm b/gnu/services/guix.scmindex 1bacd61190..e4f5cf57b9 100644--- a/gnu/services/guix.scm+++ b/gnu/services/guix.scm@@ -50,6 +50,21 @@ guix-build-coordinator-service-type + guix-build-coordinator-agent-configuration+ guix-build-coordinator-agent-configuration?+ guix-build-coordinator-agent-configuration-package+ guix-build-coordinator-agent-configuration-user+ guix-build-coordinator-agent-configuration-coordinator+ guix-build-coordinator-agent-configuration-uuid+ guix-build-coordinator-agent-configuration-password+ guix-build-coordinator-agent-configuration-password-file+ guix-build-coordinator-agent-configuration-systems+ guix-build-coordinator-agent-configuration-max-parallel-builds+ guix-build-coordinator-agent-configuration-derivation-substitute-urls+ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls++ guix-build-coordinator-agent-service-type+ <guix-data-service-configuration> guix-data-service-configuration guix-data-service-configuration?@@ -95,6 +110,33 @@ (guile guix-build-coordinator-configuration-guile (default guile-3.0-latest))) +(define-record-type* <guix-build-coordinator-agent-configuration>+ guix-build-coordinator-agent-configuration+ make-guix-build-coordinator-agent-configuration+ guix-build-coordinator-agent-configuration?+ (package guix-build-coordinator-agent-configuration-package+ (default guix-build-coordinator))+ (user guix-build-coordinator-agent-configuration-user+ (default "guix-build-coordinator-agent"))+ (coordinator guix-build-coordinator-agent-configuration-coordinator+ (default "http://localhost:8745"))+ (uuid guix-build-coordinator-agent-configuration-uuid)+ (password guix-build-coordinator-agent-configuration-password+ (default #f))+ (password-file guix-build-coordinator-agent-configuration-password-file+ (default #f))+ (systems guix-build-coordinator-agent-configuration-systems+ (default #f))+ (max-parallel-builds+ guix-build-coordinator-agent-configuration-max-parallel-builds+ (default 1))+ (derivation-substitute-urls+ guix-build-coordinator-agent-configuration-derivation-substitute-urls+ (default #f))+ (non-derivation-substitute-urls+ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls+ (default #f)))+ (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file@@ -239,6 +281,85 @@ (description "Run an instance of the Guix Build Coordinator."))) +(define (guix-build-coordinator-agent-shepherd-services config)+ (match-record config <guix-build-coordinator-agent-configuration>+ (package user coordinator uuid password password-file max-parallel-builds+ derivation-substitute-urls non-derivation-substitute-urls+ systems)+ (list+ (shepherd-service+ (documentation "Guix Build Coordinator Agent")+ (provision '(guix-build-coordinator-agent))+ (requirement '(networking))+ (start #~(make-forkexec-constructor+ (list #$(file-append package "/bin/guix-build-coordinator-agent")+ #$(string-append "--coordinator=" coordinator)+ #$(string-append "--uuid=" uuid)+ #$@(if password+ #~(#$(string-append "--password=" password))+ #~())+ #$@(if password-file+ #~(#$(string-append "--password-file=" password-file))+ #~())+ #$(simple-format #f "--max-parallel-builds=~A"+ max-parallel-builds)+ #$@(if derivation-substitute-urls+ #~(#$(string-append+ "--derivation-substitute-urls="+ (string-join derivation-substitute-urls " ")))+ #~())+ #$@(if non-derivation-substitute-urls+ #~(#$(string-append+ "--non-derivation-substitute-urls="+ (string-join derivation-substitute-urls " ")))+ #~())+ #$@(map (lambda (system)+ (string-append "--system=" system))+ (or systems '())))+ #:user #$user+ #:pid-file "/var/run/guix-build-coordinator-agent/pid"+ #:environment-variables+ `(,(string-append+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")+ "LC_ALL=en_US.utf8")+ #:log-file "/var/log/guix-build-coordinator/agent.log"))+ (stop #~(make-kill-destructor))))))++(define (guix-build-coordinator-agent-activation config)+ #~(begin+ (use-modules (guix build utils))++ (mkdir-p "/var/log/guix-build-coordinator")++ ;; Allow writing the PID file+ (mkdir-p "/var/run/guix-build-coordinator-agent")+ (chown "/var/run/guix-build-coordinator-agent"+ (passwd:uid %user)+ (passwd:gid %user))))++(define (guix-build-coordinator-agent-account config)+ (list (user-account+ (name (guix-build-coordinator-agent-configuration-user config))+ (group "nogroup")+ (system? #t)+ (comment "Guix Build Coordinator agent user")+ (home-directory "/var/empty")+ (shell (file-append shadow "/sbin/nologin")))))++(define guix-build-coordinator-agent-service-type+ (service-type+ (name 'guix-build-coordinator-agent)+ (extensions+ (list+ (service-extension shepherd-root-service-type+ guix-build-coordinator-agent-shepherd-services)+ (service-extension activation-service-type+ guix-build-coordinator-agent-activation)+ (service-extension account-service-type+ guix-build-coordinator-agent-account)))+ (description+ "Run a Guix Build Coordinator agent.")))+ ;;; ;;; Guix Data Service-- 2.28.0
C
C
Christopher Baines wrote on 19 Sep 2020 12:10
[PATCH v2 2/4] services: guix: Add guix-build-coordinator-service-type.
(address . 43494@debbugs.gnu.org)
20200919101059.8930-2-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-configuration>): New recordtype.(guix-build-coordinator-configuration, guix-build-coordinator-configuration?,guix-build-coordinator-configuration-package,guix-build-coordinator-configuration-user,guix-build-coordinator-configuration-group,guix-build-coordinator-configuration-datastore-uri-string,guix-build-coordinator-configuration-agent-communication-uri-string,guix-build-coordinator-configuration-client-communication-uri-string,guix-build-coordinator-configuration-allocation-strategy,guix-build-coordinator-configuration-hooks,guix-build-coordinator-configuration-guile,make-guix-build-coordinator-start-script,guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,guix-build-coordinator-account): New procedures.(guix-build-coordinator-service-type): New variable.* gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.* doc/guix.texi (Guix Services): Document it.--- doc/guix.texi | 67 ++++++++++++++ gnu/services/guix.scm | 200 +++++++++++++++++++++++++++++++++++++++++- gnu/tests/guix.scm | 75 +++++++++++++++- 3 files changed, 338 insertions(+), 4 deletions(-)
Toggle diff (400 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex f7e2204b53..4f3f0e851f 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -27487,6 +27487,73 @@ The complete list of possible options can be found in the man page for @node Guix Services @subsection Guix Services +@subsubheading Guix Build Coordinator+The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build+Coordinator} aids in distributing derivation builds among machines+running an @dfn{agent}. The build daemon is still used to build the+derivations, but the Guix Build Coordinator manages allocating builds+and working with the results.++@quotation Note+This service is considered experimental. Configuration options may be+changed in a backwards-incompatible manner, and not all features have+been thorougly tested.+@end quotation++The Guix Build Coordinator consists of one @dfn{coordinator}, and one or+more connected @dfn{agent} processes. The coordinator process handles+clients submitting builds, and allocating builds to agents. The agent+processes talk to a build daemon to actually perform the builds, then+send the results back to the coordinator.++There is a script to run the coordinator component of the Guix Build+Coordinator, but the Guix service uses a custom Guile script instead, to+provide better integration with G-expressions used in the configuration.++@defvar {Scheme Variable} guix-build-coordinator-service-type+Service type for the Guix Build Coordinator. Its value must be a+@code{guix-build-coordinator-configuration} object.+@end defvar++@deftp {Data Type} guix-build-coordinator-configuration+Data type representing the configuration of the Guix Build Coordinator.++@table @asis+@item @code{package} (default: @code{guix-build-coordinator})+The Guix Build Coordinator package to use.++@item @code{user} (default: @code{"guix-build-coordinator"})+The system user to run the service as.++@item @code{group} (default: @code{"guix-build-coordinator"})+The system group to run the service as.++@item @code{database-uri-string} (default: @code{"sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"})+The URI to use for the database.++@item @code{agent-communication-uri} (default: @code{"http://0.0.0.0:8745"})+The URI describing how to listen to requests from agent processes.++@item @code{client-communication-uri} (default: @code{"http://127.0.0.1:8746"})+The URI describing how to listen to requests from clients. The client+API allows submitting builds and currently isn't authenticated, so take+care when configuring this value.++@item @code{allocation-strategy} (default: @code{#~basic-build-allocation-strategy})+A G-expression for the allocation strategy to be used. This is a+procedure that takes the datastore as an argument and populates the+allocation plan in the database.++@item @code{hooks} (default: @var{'()})+An association list of hooks. These provide a way to execute arbitrary+code upon certian events, like a build result being processed.++@item @code{guile} (default: @code{guile-3.0-latest})+The Guile package with which to run the Guix Build Coordinator.++@end table+@end deftp+ @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores and provides data about GNU Guix. This includes information aboutdiff --git a/gnu/services/guix.scm b/gnu/services/guix.scmindex 10a8581a62..1bacd61190 100644--- a/gnu/services/guix.scm+++ b/gnu/services/guix.scm@@ -17,20 +17,40 @@ ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. (define-module (gnu services guix)+ #:use-module (srfi srfi-1) #:use-module (ice-9 match) #:use-module (guix gexp) #:use-module (guix records)+ #:use-module (guix packages) #:use-module ((gnu packages base) #:select (glibc-utf8-locales)) #:use-module (gnu packages admin)+ #:use-module (gnu packages databases) #:use-module (gnu packages web)+ #:use-module (gnu packages guile)+ #:use-module (gnu packages guile-xyz)+ #:use-module (gnu packages package-management) #:use-module (gnu services) #:use-module (gnu services base) #:use-module (gnu services admin) #:use-module (gnu services shepherd) #:use-module (gnu services getmail) #:use-module (gnu system shadow)- #:export (<guix-data-service-configuration>+ #:export (guix-build-coordinator-configuration+ guix-build-coordinator-configuration?+ guix-build-coordinator-configuration-package+ guix-build-coordinator-configuration-user+ guix-build-coordinator-configuration-group+ guix-build-coordinator-configuration-datastore-uri-string+ guix-build-coordinator-configuration-agent-communication-uri-string+ guix-build-coordinator-configuration-client-communication-uri-string+ guix-build-coordinator-configuration-allocation-strategy+ guix-build-coordinator-configuration-hooks+ guix-build-coordinator-configuration-guile++ guix-build-coordinator-service-type++ <guix-data-service-configuration> guix-data-service-configuration guix-data-service-configuration? guix-data-service-package@@ -45,11 +65,185 @@ ;;;; Commentary: ;;;-;;; This module implements a service that to run instances of the Guix Data-;;; Service, which provides data about Guix over time.+;;; Services specifically related to GNU Guix. ;;; ;;;; Code: +(define-record-type* <guix-build-coordinator-configuration>+ guix-build-coordinator-configuration make-guix-build-coordinator-configuration+ guix-build-coordinator-configuration?+ (package guix-build-coordinator-configuration-package+ (default guix-build-coordinator))+ (user guix-build-coordinator-configuration-user+ (default "guix-build-coordinator"))+ (group guix-build-coordinator-configuration-group+ (default "guix-build-coordinator"))+ (database-uri-string+ guix-build-coordinator-configuration-datastore-uri-string+ (default "sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"))+ (agent-communication-uri-string+ guix-build-coordinator-configuration-agent-communication-uri-string+ (default "http://0.0.0.0:8745"))+ (client-communication-uri-string+ guix-build-coordinator-configuration-client-communication-uri-string+ (default "http://127.0.0.1:8746"))+ (allocation-strategy+ guix-build-coordinator-configuration-allocation-strategy+ (default #~basic-build-allocation-strategy))+ (hooks guix-build-coordinator-configuration-hooks+ (default '()))+ (guile guix-build-coordinator-configuration-guile+ (default guile-3.0-latest)))++(define* (make-guix-build-coordinator-start-script database-uri-string+ allocation-strategy+ pid-file+ guix-build-coordinator-package+ #:key+ agent-communication-uri-string+ client-communication-uri-string+ (hooks '())+ (guile guile-3.0))+ (program-file+ "start-guix-build-coordinator"+ (with-extensions (cons guix-build-coordinator-package+ ;; This is a poorly constructed Guile load path,+ ;; since it contains things that aren't Guile+ ;; libraries, but it means that the Guile libraries+ ;; needed for the Guix Build Coordinator don't need+ ;; to be individually specified here.+ (map second (package-inputs+ guix-build-coordinator-package)))+ #~(begin+ (use-modules (srfi srfi-1)+ (ice-9 match)+ (web uri)+ (prometheus)+ (guix-build-coordinator hooks)+ (guix-build-coordinator datastore)+ (guix-build-coordinator build-allocator)+ (guix-build-coordinator coordinator))++ (let* ((metrics-registry (make-metrics-registry+ #:namespace+ "guixbuildcoordinator_"))+ (datastore (database-uri->datastore+ #$database-uri-string+ #:metrics-registry metrics-registry))+ (hooks+ (list #$@(map (match-lambda+ ((name . hook-gexp)+ #~(cons name #$hook-gexp)))+ hooks)))+ (hooks-with-defaults+ `(,@hooks+ ,@(remove (match-lambda+ ((name . _) (assq-ref hooks name)))+ %default-hooks)))+ (build-coordinator (make-build-coordinator+ #:datastore datastore+ #:hooks hooks-with-defaults+ #:metrics-registry metrics-registry+ #:allocation-strategy #$allocation-strategy)))++ (run-coordinator-service+ build-coordinator+ #:update-datastore? #t+ #:pid-file #$pid-file+ #:agent-communication-uri (string->uri+ #$agent-communication-uri-string)+ #:client-communication-uri (string->uri+ #$client-communication-uri-string)))))+ #:guile guile))++(define (guix-build-coordinator-shepherd-services config)+ (match-record config <guix-build-coordinator-configuration>+ (package user group database-uri-string+ agent-communication-uri-string+ client-communication-uri-string+ allocation-strategy+ hooks+ guile)+ (list+ (shepherd-service+ (documentation "Guix Build Coordinator")+ (provision '(guix-build-coordinator))+ (requirement '(networking))+ (start #~(make-forkexec-constructor+ (list #$(make-guix-build-coordinator-start-script+ database-uri-string+ allocation-strategy+ "/var/run/guix-build-coordinator/pid"+ package+ #:agent-communication-uri-string+ agent-communication-uri-string+ #:client-communication-uri-string+ client-communication-uri-string+ #:hooks hooks+ #:guile guile))+ #:user #$user+ #:group #$group+ #:pid-file "/var/run/guix-build-coordinator/pid"+ ;; Allow time for migrations to run+ #:pid-file-timeout 60+ #:environment-variables+ `(,(string-append+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")+ "LC_ALL=en_US.utf8")+ #:log-file "/var/log/guix-build-coordinator/coordinator.log"))+ (stop #~(make-kill-destructor))))))++(define (guix-build-coordinator-activation config)+ #~(begin+ (use-modules (guix build utils))++ (define %user (getpw "guix-build-coordinator"))++ (chmod "/var/lib/guix-build-coordinator" #o755)++ (mkdir-p "/var/log/guix-build-coordinator")++ ;; Allow writing the PID file+ (mkdir-p "/var/run/guix-build-coordinator")+ (chown "/var/run/guix-build-coordinator"+ (passwd:uid %user)+ (passwd:gid %user))))++(define (guix-build-coordinator-account config)+ (match-record config <guix-build-coordinator-configuration>+ (user group)+ (list (user-group+ (name group)+ (system? #t))+ (user-account+ (name user)+ (group group)+ (system? #t)+ (comment "Guix Build Coordinator user")+ (home-directory "/var/lib/guix-build-coordinator")+ (shell (file-append shadow "/sbin/nologin"))))))++(define guix-build-coordinator-service-type+ (service-type+ (name 'guix-build-coordinator)+ (extensions+ (list+ (service-extension shepherd-root-service-type+ guix-build-coordinator-shepherd-services)+ (service-extension activation-service-type+ guix-build-coordinator-activation)+ (service-extension account-service-type+ guix-build-coordinator-account)))+ (default-value+ (guix-build-coordinator-configuration))+ (description+ "Run an instance of the Guix Build Coordinator.")))++ +;;;+;;; Guix Data Service+;;;+ (define-record-type* <guix-data-service-configuration> guix-data-service-configuration make-guix-data-service-configuration guix-data-service-configuration?diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scmindex 6139e31cf0..20b67d55d3 100644--- a/gnu/tests/guix.scm+++ b/gnu/tests/guix.scm@@ -35,7 +35,80 @@ #:use-module (guix store) #:use-module (guix utils) #:use-module (ice-9 match)- #:export (%test-guix-data-service))+ #:export (%test-guix-build-coordinator+ %test-guix-data-service))++;;;+;;; Guix Build Coordinator+;;;++(define %guix-build-coordinator-os+ (simple-operating-system+ (service dhcp-client-service-type)+ (service guix-build-coordinator-service-type)))++(define (run-guix-build-coordinator-test)+ (define os+ (marionette-operating-system+ %guix-build-coordinator-os+ #:imported-modules '((gnu services herd)+ (guix combinators))))++ (define forwarded-port 8745)++ (define vm+ (virtual-machine+ (operating-system os)+ (memory-size 1024)+ (port-forwardings `((,forwarded-port . 8745)))))++ (define test+ (with-imported-modules '((gnu build marionette))+ #~(begin+ (use-modules (srfi srfi-11) (srfi srfi-64)+ (gnu build marionette)+ (web uri)+ (web client)+ (web response))++ (define marionette+ (make-marionette (list #$vm)))++ (mkdir #$output)+ (chdir #$output)++ (test-begin "guix-build-coordinator")++ (test-assert "service running"+ (marionette-eval+ '(begin+ (use-modules (gnu services herd))+ (match (start-service 'guix-build-coordinator)+ (#f #f)+ (('service response-parts ...)+ (match (assq-ref response-parts 'running)+ ((pid) (number? pid))))))+ marionette))++ (test-equal "http-get"+ 200+ (let-values+ (((response text)+ (http-get #$(simple-format+ #f "http://localhost:~A/metrics" forwarded-port)+ #:decode-body? #t)))+ (response-code response)))++ (test-end)+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))++ (gexp->derivation "guix-build-coordinator-test" test))++(define %test-guix-build-coordinator+ (system-test+ (name "guix-build-coordinator")+ (description "Connect to a running Guix Build Coordinator.")+ (value (run-guix-build-coordinator-test)))) ;;;-- 2.28.0
C
C
Christopher Baines wrote on 19 Sep 2020 12:10
[PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(address . 43494@debbugs.gnu.org)
20200919101059.8930-4-mail@cbaines.net
* gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):New record type.(guix-build-coordinator-queue-builds-configuration,guix-build-coordinator-queue-builds-configuration?,guix-build-coordinator-queue-builds-configuration-package,guix-build-coordinator-queue-builds-configuration-user,guix-build-coordinator-queue-builds-coordinator,guix-build-coordinator-queue-builds-configuration-systems,guix-build-coordinator-queue-builds-configuration-system-and-targets,guix-build-coordinator-queue-builds-configuration-guix-data-service,guix-build-coordinator-queue-builds-configuration-processed-commits-file,guix-build-coordinator-queue-builds-shepherd-services,guix-build-coordinator-queue-builds-activation,guix-build-coordinator-queue-builds-account): New procedures.(guix-build-coordinator-queue-builds-service-type): New variable.--- doc/guix.texi | 38 +++++++++++++++ gnu/services/guix.scm | 108 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+)
Toggle diff (184 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex b2dde30ddb..767c99c5cb 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -27603,6 +27603,44 @@ input store items aren't already available. @end table @end deftp +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type+Service type for the+guix-build-coordinator-queue-builds-from-guix-data-service script. Its+value must be a @code{guix-build-coordinator-queue-builds-configuration}+object.+@end defvar++@deftp {Data Type} guix-build-coordinator-queue-builds-configuration+Data type representing the options to the queue builds from guix data+service script.++@table @asis+@item @code{package} (default: @code{guix-build-coordinator})+The Guix Build Coordinator package to use.++@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"})+The system user to run the service as.++@item @code{coordinator} (default: @code{"http://localhost:8745"})+The URI to use when connecting to the coordinator.++@item @code{systems} (default: @code{#f})+The systems for which to fetch derivations to build.++@item @code{systems-and-targets} (default: @code{#f})+An association list of system and target pairs for which to fetch+derivations to build.++@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})+The Guix Data Service instance from which to query to find out about+derivations to build.++@item @code{processed-commits-file} (default: @code{"/var/cache/guix-build-coordinator-queue-builds/processed-commits"})+A file to record which commits have been processed, to avoid needlessly+processing them again if the service is restarted.++@end table+@end deftp @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, storesdiff --git a/gnu/services/guix.scm b/gnu/services/guix.scmindex e4f5cf57b9..1493a24be5 100644--- a/gnu/services/guix.scm+++ b/gnu/services/guix.scm@@ -65,6 +65,18 @@ guix-build-coordinator-agent-service-type + guix-build-coordinator-queue-builds-configuration+ guix-build-coordinator-queue-builds-configuration?+ guix-build-coordinator-queue-builds-configuration-package+ guix-build-coordinator-queue-builds-configuration-user+ guix-build-coordinator-queue-builds-coordinator+ guix-build-coordinator-queue-builds-configuration-systems+ guix-build-coordinator-queue-builds-configuration-system-and-targets+ guix-build-coordinator-queue-builds-configuration-guix-data-service+ guix-build-coordinator-queue-builds-configuration-processed-commits-file++ guix-build-coordinator-queue-builds-service-type+ <guix-data-service-configuration> guix-data-service-configuration guix-data-service-configuration?@@ -137,6 +149,28 @@ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls (default #f))) +(define-record-type* <guix-build-coordinator-queue-builds-configuration>+ guix-build-coordinator-queue-builds-configuration+ make-guix-build-coordinator-queue-builds-configuration+ guix-build-coordinator-queue-builds-configuration?+ (package guix-build-coordinator-queue-builds-configuration-package+ (default guix-build-coordinator))+ (user guix-build-coordinator-queue-builds-configuration-user+ (default "guix-build-coordinator-queue-builds"))+ (coordinator guix-build-coordinator-queue-builds-coordinator+ (default "http://localhost:8745"))+ (systems guix-build-coordinator-queue-builds-configuration-systems+ (default #f))+ (systems-and-targets+ guix-build-coordinator-queue-builds-configuration-system-and-targets+ (default #f))+ (guix-data-service+ guix-build-coordinator-queue-builds-configuration-guix-data-service+ (default "https://data.guix.gnu.org"))+ (processed-commits-file+ guix-build-coordinator-queue-builds-configuration-processed-commits-file+ (default "/var/cache/guix-build-coordinator-queue-builds/processed-commits")))+ (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file@@ -360,6 +394,80 @@ (description "Run a Guix Build Coordinator agent."))) +(define (guix-build-coordinator-queue-builds-shepherd-services config)+ (match-record config <guix-build-coordinator-queue-builds-configuration>+ (package user coordinator systems systems-and-targets+ guix-data-service processed-commits-file)+ (list+ (shepherd-service+ (documentation "Guix Build Coordinator queue builds from Guix Data Service")+ (provision '(guix-build-coordinator-queue-builds))+ (requirement '(networking))+ (start+ #~(make-forkexec-constructor+ (list+ #$(file-append+ package+ "/bin/guix-build-coordinator-queue-builds-from-guix-data-service")+ #$(string-append "--coordinator=" coordinator)+ #$@(map (lambda (system)+ (string-append "--system=" system))+ (or systems '()))+ #$@(map (match-lambda+ ((system . target)+ (string-append "--system-and-target=" system "=" target)))+ (or systems-and-targets '()))+ #$@(if guix-data-service+ #~(#$(string-append "--guix-data-service=" guix-data-service))+ #~())+ #$@(if processed-commits-file+ #~(#$(string-append "--processed-commits-file="+ processed-commits-file))+ #~()))+ #:user #$user+ #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid"+ #:environment-variables+ `(,(string-append+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")+ "LC_ALL=en_US.utf8")+ #:log-file "/var/log/guix-build-coordinator/queue-builds.log"))+ (stop #~(make-kill-destructor))))))++(define (guix-build-coordinator-queue-builds-activation config)+ #~(begin+ (use-modules (guix build utils))++ (mkdir-p "/var/log/guix-build-coordinator")++ ;; Allow writing the PID file+ (mkdir-p "/var/run/guix-build-coordinator-queue-builds")+ (chown "/var/run/guix-build-coordinator-queue-builds"+ (passwd:uid %user)+ (passwd:gid %user))))++(define (guix-build-coordinator-queue-builds-account config)+ (list (user-account+ (name (guix-build-coordinator-queue-builds-configuration-user config))+ (group "nogroup")+ (system? #t)+ (comment "Guix Build Coordinator queue-builds user")+ (home-directory "/var/empty")+ (shell (file-append shadow "/sbin/nologin")))))++(define guix-build-coordinator-queue-builds-service-type+ (service-type+ (name 'guix-build-coordinator-queue-builds)+ (extensions+ (list+ (service-extension shepherd-root-service-type+ guix-build-coordinator-queue-builds-shepherd-services)+ (service-extension activation-service-type+ guix-build-coordinator-queue-builds-activation)+ (service-extension account-service-type+ guix-build-coordinator-queue-builds-account)))+ (description+ "Run the Guix Build Coordinator queue builds script.")))+ ;;; ;;; Guix Data Service-- 2.28.0
L
L
Ludovic Courtès wrote on 25 Sep 2020 11:42
Re: [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87imc22nwm.fsf@gnu.org
Hi!
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (2 lines)> Ludovic Courtès <ludo@gnu.org> writes:
[...]
Toggle quote (22 lines)>>> + (program-file>>> + "start-guix-build-coordinator">>> + (with-extensions (cons guix-build-coordinator-package>>> + ;; This is a poorly constructed Guile load path,>>> + ;; since it contains things that aren't Guile>>> + ;; libraries, but it means that the Guile libraries>>> + ;; needed for the Guix Build Coordinator don't need>>> + ;; to be individually specified here.>>> + (map second (package-inputs>>> + guix-build-coordinator-package)))>>>> Perhaps there should eventually be a ‘guix-build-coordinator’ command in>> the package itself?>> There actually is, one thing I've had in mind for a while now though is> to use a scheme script constructed by the Guix service to run the> coordinator.>> For guix.cbaines.net, I'm using the script, but with the hooks passed in> on the command line, the command is rather long, and it means that> backtraces don't work well with the hooks.
You mean because the hooks are interpreted, and so all you see in thebacktrace is a bunch of ‘eval’ calls?
Toggle quote (5 lines)> I'm unsure how well this has worked out, I didn't anticipate the issues> with the Guile load path, and because of the formatting stripping, both> the code and backtraces will be unreadable anyway... but it is useful in> allowing G-expressions to be used as part of the configuration.
OK.
Thanks,Ludo’.
C
C
Christopher Baines wrote on 26 Sep 2020 10:43
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494@debbugs.gnu.org)
87mu1dnd1o.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (33 lines)> Hi!>> Christopher Baines <mail@cbaines.net> skribis:>>> Ludovic Courtès <ludo@gnu.org> writes:>> [...]>>>>> + (program-file>>>> + "start-guix-build-coordinator">>>> + (with-extensions (cons guix-build-coordinator-package>>>> + ;; This is a poorly constructed Guile load path,>>>> + ;; since it contains things that aren't Guile>>>> + ;; libraries, but it means that the Guile libraries>>>> + ;; needed for the Guix Build Coordinator don't need>>>> + ;; to be individually specified here.>>>> + (map second (package-inputs>>>> + guix-build-coordinator-package)))>>>>>> Perhaps there should eventually be a ‘guix-build-coordinator’ command in>>> the package itself?>>>> There actually is, one thing I've had in mind for a while now though is>> to use a scheme script constructed by the Guix service to run the>> coordinator.>>>> For guix.cbaines.net, I'm using the script, but with the hooks passed in>> on the command line, the command is rather long, and it means that>> backtraces don't work well with the hooks.>> You mean because the hooks are interpreted, and so all you see in the> backtrace is a bunch of ‘eval’ calls?
Yeah, I haven't done much testing of this, but that's my assumption.
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl9u/zNfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE9Xdr5Q//QN7KCG9MfVAAMx7d9lor18eZMdbw5JJNRXwbPrTIhTd+rWadYU0w/8fmy59oFNhv3MFFHtOYtKt8r/axRDey57CIRnzRPcOfd9Gr31hJ8wq5vqgDPl1xmOsG4W9rhxi6MAdAfr0qcz4yxq9MHGujAuSmwsj5gP8S+UWVw5YbS3IemhAWg7lkWHlBbrakM/A2A6ZD4GBwbJuYi5ZnIK1Af8gyPuc4VSCjTPmrutFzaEkOFy5Mdlx/Ct49v/VKSzmcHkkRYP3jZZv3kFZn9gsePFtK30uh5iPQg2MwRUdbq5p5DIBN6PSNuyANMMKKtyeXGnVM9iOwDFPli+s4B577W5Bo/Heixg+4RpituHKe5HP5eSg0jalQJkhKuGaiTOCRZbDSTo9ZsXEvTwCK6v0rdlE3Er/oEATqcatkUSgOYU2qhxkVNwipHf0OPmWjwDFMD3eYNjGxnCPePNHm2xxP7LjsSDh5r1ez6ck8zdKOxu5Ifv+3OQTl54bpMimxV5gxd0tV8XmjE1U1cNlyV9VTReZKsrBEiXypz/5aYRbie6AdfR7ZPi4Faveddm2le0sJTxnH1xkB4/k3zelqF2NCI9RztVvOnXZsn/AH8TpTW9hsMq8acgHvT2H0A0SAZEhTsXYoznJOaEoT10PBH9xSmfXxiggCNNiOU6DlvxAIYcs==9BXV-----END PGP SIGNATURE-----
L
L
Ludovic Courtès wrote on 5 Oct 2020 10:00
Re: [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 43494@debbugs.gnu.org)
87y2kl5cg8.fsf@gnu.org
Hi Chris,
Sorry for the delay. Patches 1–3 LGTM! One minor comment on patch 4:
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (16 lines)> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):> New record type.> (guix-build-coordinator-queue-builds-configuration,> guix-build-coordinator-queue-builds-configuration?,> guix-build-coordinator-queue-builds-configuration-package,> guix-build-coordinator-queue-builds-configuration-user,> guix-build-coordinator-queue-builds-coordinator,> guix-build-coordinator-queue-builds-configuration-systems,> guix-build-coordinator-queue-builds-configuration-system-and-targets,> guix-build-coordinator-queue-builds-configuration-guix-data-service,> guix-build-coordinator-queue-builds-configuration-processed-commits-file,> guix-build-coordinator-queue-builds-shepherd-services,> guix-build-coordinator-queue-builds-activation,> guix-build-coordinator-queue-builds-account): New procedures.> (guix-build-coordinator-queue-builds-service-type): New variable.
[...]
Toggle quote (8 lines)> > +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type> +Service type for the> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its> +value must be a @code{guix-build-coordinator-queue-builds-configuration}> +object.> +@end defvar
Could you add a paragraph right above this explaining what this serviceis about?
Toggle quote (14 lines)> +(define guix-build-coordinator-queue-builds-service-type> + (service-type> + (name 'guix-build-coordinator-queue-builds)> + (extensions> + (list> + (service-extension shepherd-root-service-type> + guix-build-coordinator-queue-builds-shepherd-services)> + (service-extension activation-service-type> + guix-build-coordinator-queue-builds-activation)> + (service-extension account-service-type> + guix-build-coordinator-queue-builds-account)))> + (description> + "Run the Guix Build Coordinator queue builds script.")))
Likewise it’d be great if you could add a few works here.
OK to push with these changes, thank you!
Ludo’.
C
C
Christopher Baines wrote on 5 Oct 2020 19:16
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43494-done@debbugs.gnu.org)
87a6x01tll.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (53 lines)> Hi Chris,>> Sorry for the delay. Patches 1–3 LGTM! One minor comment on patch 4:>> Christopher Baines <mail@cbaines.net> skribis:>>> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):>> New record type.>> (guix-build-coordinator-queue-builds-configuration,>> guix-build-coordinator-queue-builds-configuration?,>> guix-build-coordinator-queue-builds-configuration-package,>> guix-build-coordinator-queue-builds-configuration-user,>> guix-build-coordinator-queue-builds-coordinator,>> guix-build-coordinator-queue-builds-configuration-systems,>> guix-build-coordinator-queue-builds-configuration-system-and-targets,>> guix-build-coordinator-queue-builds-configuration-guix-data-service,>> guix-build-coordinator-queue-builds-configuration-processed-commits-file,>> guix-build-coordinator-queue-builds-shepherd-services,>> guix-build-coordinator-queue-builds-activation,>> guix-build-coordinator-queue-builds-account): New procedures.>> (guix-build-coordinator-queue-builds-service-type): New variable.>> [...]>>> >> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type>> +Service type for the>> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its>> +value must be a @code{guix-build-coordinator-queue-builds-configuration}>> +object.>> +@end defvar>> Could you add a paragraph right above this explaining what this service> is about?>>> +(define guix-build-coordinator-queue-builds-service-type>> + (service-type>> + (name 'guix-build-coordinator-queue-builds)>> + (extensions>> + (list>> + (service-extension shepherd-root-service-type>> + guix-build-coordinator-queue-builds-shepherd-services)>> + (service-extension activation-service-type>> + guix-build-coordinator-queue-builds-activation)>> + (service-extension account-service-type>> + guix-build-coordinator-queue-builds-account)))>> + (description>> + "Run the Guix Build Coordinator queue builds script.")))>> Likewise it’d be great if you could add a few works here.>> OK to push with these changes, thank you!
I've tried to add some clarity in the places you mention. I'm still verytempted to make a separate repository/package for the queue builds fromGuix Data Service script... I'm not quite decided yet though.
Anyway, thanks for taking another look. I've gone ahead and pushed thesepatches as c14714cbbfe9239410aec2ed73282192a64fbbdc. I did tweak thepackage definition a bit though to move closer to being able tocross-compile it (you can now if you tweak the inputs).
Thanks again,
Chris
-----BEGIN PGP SIGNATURE-----
iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl97VOZfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2JhaW5lcy5uZXQACgkQXiijOwuE9XelCRAAi3Ihs57Jxpd29ZfQX/6QqK8LTMWGs8dQjuPk6/Yl0DzaxcEeuzf+TPR0t/zuDCzucv9g0+bx4ohPTN08MPoGT6y+tdY1zDNh7MxSYiLiM8IOLxDwPoB6LoFTyUcsV3EJNbfrdf3fN3IEwTd8y9HOBrRuCT46lRNc03ku7yL44JhiHotzg1YWY+si2y/OAAPb4s0iP7Chx9ppLdvyqANcV0leuVJrtP3yitb23Gb9BCmSfFobEeI3xo4se6LsXqpYj8fq3kGipC4DCoWkZiA99gW4UiY3C4+hd2zmElR4+x2lcAQD2Z8SLWK/5bej1iWeJ7q8OvuAgRJS6LlDTkxxI73m8frqnui0EGZGonG+UrXmsM9h/Pp7aIQU+C5rDu7u3NqmG+WeeDjgJTD5vorkpKhMwDfeH6sLBSsMWiUvEpwWMx2MqYOZSo3qn8wIb/HfUFAZNNrWhasCygOTWaKOgm5IdfVbSENOwxve8VPF6UG26diVc/1OaDBmMdao7+L9eU+CHD/J/U9xVcRBGPdZ8ATFJfKTOUnTmyusPwsdTYNM1AAdZFVev0sOrRhmW6lWh117n8ideJ68w4ldpK9s70Ln+IfTPbUxjKlSiKn0dYOui61oag0EFwJrm3uWJEBggLd7TUC9Gp+tZwpDuFsYJCIULHkJeXC9sWHkqe4RtGo==pTf2-----END PGP SIGNATURE-----
Closed
?