[PATCH 0/5] Start making substitute code less coupled

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Christopher Baines
Owner
unassigned
Submitted by
Christopher Baines
Severity
normal
C
C
Christopher Baines wrote on 20 Feb 20:05 +0100
(address . guix-patches@gnu.org)
87le7flz89.fsf@cbaines.net
These changes should help with using the substitute code in a Guile
implementation of the Guix daemon.


Christopher Baines (5):
scripts: substitute: Remove side effect warning from network-error?.
scripts: substitute: Allow not using with-timeout in download-nar.
scripts: substitute: Replace some leave calls with raise.
scripts: substitute: Untangle selecting fast vs small compressions.
scripts: substitute: Extract script specific output from download-nar.

guix/scripts/substitute.scm | 207 +++++++++++++++++++++---------------
1 file changed, 123 insertions(+), 84 deletions(-)


base-commit: 3d061d9677027be7651f8e5a3a02e19daacd9a85
--
2.41.0
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmXU+2ZfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xfoag//Uoq9twKRF3tAV9Iuv4PmvPQwLwLXUR++
pA1dqG1dQnu3KhMmi0QBdA2MZUIfTgbtuBvqQwPHdXtV7sQZW7smREY85Q/EUThb
qJs3GJGvVEnXjGe2N00w0zol1VMIz1itXcFb+JFLQNSaBnPYDMzgdn04RhO+WpqT
lLT4ccwKEzFCh3TAroUHx9m3lgZ4A1tg6MkH9kKlMEw+UQl6cgMZCNSgLqrJYRTx
kwIEMpcVbJvmSePx+rU0SHwIUS5JBLaAKMRMb53fr6iNstW4WrH9ophrHRJf2N9B
GTWmaKnj1uzYUQOSF2D0Vq7ZFEBeKNfUkwF0vyspWFBjTdZxkshHHRL2B/wDCLAz
WIwhKe4ydkN4hpDDz67o8hIAeDce3K3d9LjXN2GYdFeEFYQWq36cyGG1I5o8sEZv
yEH6WihzLvjC1Mlso4KhiB4ZQubqkCfRWRIOdxVEhw211FOFubtTGF1JHN7ZNKG2
LZnhuJa6vfnEgdmvG46Gx5O2udqgG1IyN0xSsXEl033B1ZnrNTTy/Wos1KuGjBFz
uQg+xoTMKgokm9nqMoXdFmRViR3WILiNMhuY8eG2z8fqQpxBM+q3iMZNVuWsI2Nq
QqiW7/jHcWKEiHj/TM3RHUVKLv4hOSXbzxJVyD9skZETG4yYNqpSOcqoNU/LInXO
qWtukYWSwIc=
=xIzK
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 20 Feb 20:42 +0100
[PATCH 2/5] scripts: substitute: Allow not using with-timeout in download-nar.
(address . 69291@debbugs.gnu.org)
2bcc2278b5aefcd81fbc53b791d08622b6d93c7c.1708458147.git.mail@cbaines.net
I don't think the approach of using SIGALARM here for the timeout will work
well in all cases (e.g. when using Guile Fibers), so make it possible to avoid
this.

* guix/scripts/substitute.scm (download-nar): Pass the fetch timeout in as an
option.

Change-Id: I8cbe6cdfa10cdaa7d41974cbea56a95f5efecfe6
---
guix/scripts/substitute.scm | 37 ++++++++++++++++++++++---------------
1 file changed, 22 insertions(+), 15 deletions(-)

Toggle diff (57 lines)
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index f3eed0eb44..575fa2a0b3 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -452,7 +452,8 @@ (define-syntax-rule (catch-system-error exp)
(define* (download-nar narinfo destination
#:key status-port
- deduplicate? print-build-trace?)
+ deduplicate? print-build-trace?
+ (fetch-timeout %fetch-timeout))
"Download the nar prescribed in NARINFO, which is assumed to be authentic
and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and
if DESTINATION is in the store, deduplicate its files. Print a status line to
@@ -473,20 +474,26 @@ (define* (download-nar narinfo destination
(let ((port (open-file (uri-path uri) "r0b")))
(values port (stat:size (stat port)))))
((http https)
- ;; Test this with:
- ;; sudo tc qdisc add dev eth0 root netem delay 1500ms
- ;; and then cancel with:
- ;; sudo tc qdisc del dev eth0 root
- (with-timeout %fetch-timeout
- (begin
- (warning (G_ "while fetching ~a: server is somewhat slow~%")
- (uri->string uri))
- (warning (G_ "try `--no-substitutes' if the problem persists~%")))
- (with-cached-connection uri port
- (http-fetch uri #:text? #f
- #:port port
- #:keep-alive? #t
- #:buffered? #f))))
+ (if fetch-timeout
+ ;; Test this with:
+ ;; sudo tc qdisc add dev eth0 root netem delay 1500ms
+ ;; and then cancel with:
+ ;; sudo tc qdisc del dev eth0 root
+ (with-timeout %fetch-timeout
+ (begin
+ (warning (G_ "while fetching ~a: server is somewhat slow~%")
+ (uri->string uri))
+ (warning (G_ "try `--no-substitutes' if the problem persists~%")))
+ (with-cached-connection uri port
+ (http-fetch uri #:text? #f
+ #:port port
+ #:keep-alive? #t
+ #:buffered? #f)))
+ (with-cached-connection uri port
+ (http-fetch uri #:text? #f
+ #:port port
+ #:keep-alive? #t
+ #:buffered? #f))))
(else
(leave (G_ "unsupported substitute URI scheme: ~a~%")
(uri->string uri)))))
--
2.41.0
C
C
Christopher Baines wrote on 20 Feb 20:42 +0100
[PATCH 1/5] scripts: substitute: Remove side effect warning from network-error?.
(address . 69291@debbugs.gnu.org)
c689a478b1b123eb607a89b2ec295b491d79f39f.1708458147.git.mail@cbaines.net
Instead, display the warning from process-substitution and
process-substitution/fallback in the relevant places.

I'm looking at this because I want to make the substitute code less tied to
the script and usable in the Guile guix-daemon.

* guix/scripts/substitute.scm (network-error?): Move warning to…
(process-substitution/fallback, process-substitution): here.

Change-Id: I082b482c0e6ec7e02a8d437ba22dcefca5c40787
---
guix/scripts/substitute.scm | 21 +++++++++++++--------
1 file changed, 13 insertions(+), 8 deletions(-)

Toggle diff (50 lines)
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 37cd08e289..f3eed0eb44 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -613,13 +613,7 @@ (define network-error?
(and (kind-and-args? exception)
(memq (exception-kind exception)
'(gnutls-error getaddrinfo-error)))
- (and (http-get-error? exception)
- (begin
- (warning (G_ "download from '~a' failed: ~a, ~s~%")
- (uri->string (http-get-error-uri exception))
- (http-get-error-code exception)
- (http-get-error-reason exception))
- #t))))))
+ (http-get-error? exception)))))
(define* (process-substitution/fallback port narinfo destination
#:key cache-urls acl
@@ -647,7 +641,13 @@ (define* (process-substitution/fallback port narinfo destination
(if (or (equivalent-narinfo? narinfo alternate)
(valid-narinfo? alternate acl)
(%allow-unauthenticated-substitutes?))
- (guard (c ((network-error? c) (loop rest)))
+ (guard (c ((network-error? c)
+ (when (http-get-error? c)
+ (warning (G_ "download from '~a' failed: ~a, ~s~%")
+ (uri->string (http-get-error-uri c))
+ (http-get-error-code c)
+ (http-get-error-reason c)))
+ (loop rest)))
(download-nar alternate destination
#:status-port port
#:deduplicate? deduplicate?
@@ -675,6 +675,11 @@ (define* (process-substitution port store-item destination
store-item))
(guard (c ((network-error? c)
+ (when (http-get-error? c)
+ (warning (G_ "download from '~a' failed: ~a, ~s~%")
+ (uri->string (http-get-error-uri c))
+ (http-get-error-code c)
+ (http-get-error-reason c)))
(format (current-error-port)
(G_ "retrying download of '~a' with other substitute URLs...~%")
store-item)

base-commit: 3d061d9677027be7651f8e5a3a02e19daacd9a85
--
2.41.0
C
C
Christopher Baines wrote on 20 Feb 20:42 +0100
[PATCH 3/5] scripts: substitute: Replace some leave calls with raise.
(address . 69291@debbugs.gnu.org)
3a88dc8a8369278f86f3a51a99aa62d991ba997d.1708458147.git.mail@cbaines.net
These calls happen inside of with-error-handling, so the effect should be the
same, but this opens up the possibility of using this code in a program that
doesn't want to exit when one of these error conditions is met.

Change-Id: I15d963615d85d419559fa0f4333fa4dc1dfbfd3b

* guix/scripts/substitute.scm (download-nar, process-substitution): Use raise
formatted-message rather than leave.

Change-Id: Idd0880206b69e3903e19e0536b87d65a52c200d5
---
guix/scripts/substitute.scm | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)

Toggle diff (47 lines)
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 575fa2a0b3..1875a4332d 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -495,8 +495,10 @@ (define* (download-nar narinfo destination
#:keep-alive? #t
#:buffered? #f))))
(else
- (leave (G_ "unsupported substitute URI scheme: ~a~%")
- (uri->string uri)))))
+ (raise
+ (formatted-message
+ (G_ "unsupported substitute URI scheme: ~a~%")
+ (uri->string uri))))))
(define (try-fetch choices)
(match choices
@@ -511,9 +513,11 @@ (define* (download-nar narinfo destination
(G_ "Downloading ~a...~%") (uri->string uri)))
(values port uri compression download-size))))
(()
- (leave (G_ "no valid nar URLs for ~a at ~a~%")
- (narinfo-path narinfo)
- (narinfo-uri-base narinfo)))))
+ (raise
+ (formatted-message
+ (G_ "no valid nar URLs for ~a at ~a~%")
+ (narinfo-path narinfo)
+ (narinfo-uri-base narinfo))))))
;; Delete DESTINATION first--necessary when starting over after a failed
;; download.
@@ -678,8 +682,10 @@ (define* (process-substitution port store-item destination
(cut valid-narinfo? <> acl))))
(unless narinfo
- (leave (G_ "no valid substitute for '~a'~%")
- store-item))
+ (raise
+ (formatted-message
+ (G_ "no valid substitute for '~a'~%")
+ store-item)))
(guard (c ((network-error? c)
(when (http-get-error? c)
--
2.41.0
C
C
Christopher Baines wrote on 20 Feb 20:42 +0100
[PATCH 4/5] scripts: substitute: Untangle selecting fast vs small compressions.
(address . 69291@debbugs.gnu.org)
2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@cbaines.net
Pulling the logic up to the script makes this code more portable and not
reliant on setting a global variable.

* guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to…
(%default-prefer-fast-decompression?): this.
(display-narinfo-data): Update accordingly.
(download-nar): Add prefer-fast-decompression? as a keyword argument, remove
code to set! it and return the cpu-usage recorded.
(process-substitution, process-substitution/fallback): Accept and pass through
prefer-fast-decompression? to download-nar.
(guix-substitute): Move the prefer fast decompression switching logic here.

Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb
---
guix/scripts/substitute.scm | 90 +++++++++++++++++++++----------------
1 file changed, 52 insertions(+), 38 deletions(-)

Toggle diff (179 lines)
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 1875a4332d..61e16b22db 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -261,12 +261,7 @@ (define (show-help)
;;; Daemon/substituter protocol.
;;;
-(define %prefer-fast-decompression?
- ;; Whether to prefer fast decompression over good compression ratios. This
- ;; serves in particular to choose between lzip (high compression ratio but
- ;; low decompression throughput) and zstd (lower compression ratio but high
- ;; decompression throughput).
- #f)
+(define %default-prefer-fast-decompression? #f)
(define (call-with-cpu-usage-monitoring proc)
(let ((before (times)))
@@ -297,7 +292,7 @@ (define (display-narinfo-data port narinfo)
(let ((uri compression file-size
(narinfo-best-uri narinfo
#:fast-decompression?
- %prefer-fast-decompression?)))
+ %default-prefer-fast-decompression?)))
(format port "~a\n~a\n"
(or file-size 0)
(or (narinfo-size narinfo) 0))))
@@ -453,7 +448,8 @@ (define-syntax-rule (catch-system-error exp)
(define* (download-nar narinfo destination
#:key status-port
deduplicate? print-build-trace?
- (fetch-timeout %fetch-timeout))
+ (fetch-timeout %fetch-timeout)
+ prefer-fast-decompression?)
"Download the nar prescribed in NARINFO, which is assumed to be authentic
and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and
if DESTINATION is in the store, deduplicate its files. Print a status line to
@@ -525,7 +521,7 @@ (define* (download-nar narinfo destination
(let ((choices (narinfo-preferred-uris narinfo
#:fast-decompression?
- %prefer-fast-decompression?)))
+ prefer-fast-decompression?)))
;; 'guix publish' without '--cache' doesn't specify a Content-Length, so
;; DOWNLOAD-SIZE is #f in this case.
(let* ((raw uri compression download-size (try-fetch choices))
@@ -566,21 +562,6 @@ (define* (download-nar narinfo destination
deduplicate?)
dump-file/deduplicate*
dump-file))))
-
- ;; Create a hysteresis: depending on CPU usage, favor compression
- ;; methods with faster decompression (like ztsd) or methods with better
- ;; compression ratios (like lzip). This stems from the observation that
- ;; substitution can be CPU-bound when high-speed networks are used:
- ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
- ;; To simulate "slow" networking or changing conditions, run:
- ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency 50ms burst 1540
- ;; and then cancel with:
- ;; sudo tc qdisc del dev eno1 root
- (when (> cpu-usage .8)
- (set! %prefer-fast-decompression? #t))
- (when (< cpu-usage .2)
- (set! %prefer-fast-decompression? #f))
-
(close-port hashed)
(close-port input)
@@ -604,7 +585,9 @@ (define* (download-nar narinfo destination
(format status-port "hash-mismatch ~a ~a ~a~%"
(hash-algorithm-name algorithm)
(bytevector->nix-base32-string expected)
- (bytevector->nix-base32-string actual)))))))
+ (bytevector->nix-base32-string actual))))
+
+ cpu-usage)))
(define (system-error? exception)
"Return true if EXCEPTION is a Guile 'system-error exception."
@@ -628,7 +611,8 @@ (define network-error?
(define* (process-substitution/fallback port narinfo destination
#:key cache-urls acl
- deduplicate? print-build-trace?)
+ deduplicate? print-build-trace?
+ prefer-fast-decompression?)
"Attempt to substitute NARINFO, which is assumed to be authorized or
equivalent, by trying to download its nar from each entry in CACHE-URLS.
@@ -662,14 +646,17 @@ (define* (process-substitution/fallback port narinfo destination
(download-nar alternate destination
#:status-port port
#:deduplicate? deduplicate?
- #:print-build-trace? print-build-trace?))
+ #:print-build-trace? print-build-trace?
+ #:prefer-fast-decompression?
+ prefer-fast-decompression?))
(loop rest)))
(()
(loop rest)))))))
(define* (process-substitution port store-item destination
#:key cache-urls acl
- deduplicate? print-build-trace?)
+ deduplicate? print-build-trace?
+ prefer-fast-decompression?)
"Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to
DESTINATION as a nar file. Verify the substitute against ACL, and verify its
hash against what appears in the narinfo. When DEDUPLICATE? is true, and if
@@ -701,11 +688,14 @@ (define* (process-substitution port store-item destination
#:acl acl
#:deduplicate? deduplicate?
#:print-build-trace?
- print-build-trace?)))
+ print-build-trace?
+ #:prefer-fast-decompression?
+ prefer-fast-decompression?)))
(download-nar narinfo destination
#:status-port port
#:deduplicate? deduplicate?
- #:print-build-trace? print-build-trace?)))
+ #:print-build-trace? print-build-trace?
+ #:prefer-fast-decompression? prefer-fast-decompression?)))
;;;
@@ -895,18 +885,42 @@ (define-command (guix-substitute . args)
;; Specify the number of columns of the terminal so the progress
;; report displays nicely.
(parameterize ((current-terminal-columns (client-terminal-columns)))
- (let loop ()
+ (let loop ((prefer-fast-decompression?
+ %default-prefer-fast-decompression?))
(match (read-line)
((? eof-object?)
#t)
((= string-tokenize ("substitute" store-path destination))
- (process-substitution reply-port store-path destination
- #:cache-urls (substitute-urls)
- #:acl (current-acl)
- #:deduplicate? deduplicate?
- #:print-build-trace?
- print-build-trace?)
- (loop))))))
+ (let ((cpu-usage
+ (process-substitution reply-port store-path destination
+ #:cache-urls (substitute-urls)
+ #:acl (current-acl)
+ #:deduplicate? deduplicate?
+ #:print-build-trace?
+ print-build-trace?
+ #:prefer-fast-decompression?
+ prefer-fast-decompression?)))
+
+ ;; Create a hysteresis: depending on CPU usage, favor
+ ;; compression methods with faster decompression (like ztsd)
+ ;; or methods with better compression ratios (like lzip).
+ ;; This stems from the observation that substitution can be
+ ;; CPU-bound when high-speed networks are used:
+ ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
+ ;; To simulate "slow" networking or changing conditions, run:
+ ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency
+ ;; 50ms burst 1540 and then cancel with: sudo tc qdisc del
+ ;; dev eno1 root
+ (loop (cond
+ ;; Whether to prefer fast decompression over good
+ ;; compression ratios. This serves in particular to
+ ;; choose between lzip (high compression ratio but low
+ ;; decompression throughput) and zstd (lower
+ ;; compression ratio but high decompression
+ ;; throughput).
+ ((> cpu-usage .8) #t)
+ ((< cpu-usage .2) #f)
+ (else prefer-fast-decompression?)))))))))
(opts
(leave (G_ "~a: unrecognized options~%") opts))))))
--
2.41.0
C
C
Christopher Baines wrote on 20 Feb 20:42 +0100
[PATCH 5/5] scripts: substitute: Extract script specific output from download-nar.
(address . 69291@debbugs.gnu.org)
f4af19b037826cad90bbcfe400ad864f028cc7d8.1708458147.git.mail@cbaines.net
As this moves download-nar in a direction where it could be used outside the
substitute script.

* guix/scripts/substitute.scm (download-nar): Return more information and move
status-port output to…
(guix-substitute): here.

Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0
---
guix/scripts/substitute.scm | 49 +++++++++++++++++++++----------------
1 file changed, 28 insertions(+), 21 deletions(-)

Toggle diff (76 lines)
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 61e16b22db..94eb6d2f71 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -568,26 +568,10 @@ (define* (download-nar narinfo destination
;; Wait for the reporter to finish.
(every (compose zero? cdr waitpid) pids)
- ;; Skip a line after what 'progress-reporter/file' printed, and another
- ;; one to visually separate substitutions. When PRINT-BUILD-TRACE? is
- ;; true, leave it up to (guix status) to prettify things.
- (newline (current-error-port))
- (unless print-build-trace?
- (newline (current-error-port)))
-
- ;; Check whether we got the data announced in NARINFO.
- (let ((actual (get-hash)))
- (if (bytevector=? actual expected)
- ;; Tell the daemon that we're done.
- (format status-port "success ~a ~a~%"
- (narinfo-hash narinfo) (narinfo-size narinfo))
- ;; The actual data has a different hash than that in NARINFO.
- (format status-port "hash-mismatch ~a ~a ~a~%"
- (hash-algorithm-name algorithm)
- (bytevector->nix-base32-string expected)
- (bytevector->nix-base32-string actual))))
-
- cpu-usage)))
+ (values narinfo
+ expected
+ (get-hash)
+ cpu-usage))))
(define (system-error? exception)
"Return true if EXCEPTION is a Guile 'system-error exception."
@@ -891,7 +875,10 @@ (define-command (guix-substitute . args)
((? eof-object?)
#t)
((= string-tokenize ("substitute" store-path destination))
- (let ((cpu-usage
+ (let ((narinfo
+ expected-hash
+ actual-hash
+ cpu-usage
(process-substitution reply-port store-path destination
#:cache-urls (substitute-urls)
#:acl (current-acl)
@@ -901,6 +888,26 @@ (define-command (guix-substitute . args)
#:prefer-fast-decompression?
prefer-fast-decompression?)))
+ ;; Skip a line after what 'progress-reporter/file' printed,
+ ;; and another one to visually separate substitutions. When
+ ;; PRINT-BUILD-TRACE? is true, leave it up to (guix status)
+ ;; to prettify things.
+ (newline (current-error-port))
+ (unless print-build-trace?
+ (newline (current-error-port)))
+
+ ;; Check whether we got the data announced in NARINFO.
+ (if (bytevector=? actual-hash expected-hash)
+ ;; Tell the daemon that we're done.
+ (format reply-port "success ~a ~a~%"
+ (narinfo-hash narinfo) (narinfo-size narinfo))
+ ;; The actual data has a different hash than that in NARINFO.
+ (format reply-port "hash-mismatch ~a ~a ~a~%"
+ (hash-algorithm-name
+ (narinfo-hash-algorithm+value narinfo))
+ (bytevector->nix-base32-string expected-hash)
+ (bytevector->nix-base32-string actual-hash)))
+
;; Create a hysteresis: depending on CPU usage, favor
;; compression methods with faster decompression (like ztsd)
;; or methods with better compression ratios (like lzip).
--
2.41.0
L
L
Ludovic Courtès wrote on 23 Feb 17:16 +0100
Re: [bug#69291] [PATCH 1/5] scripts: substitute: Remove side effect warning from network-error?.
(name . Christopher Baines)(address . mail@cbaines.net)
877civgnr3.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (11 lines)
> Instead, display the warning from process-substitution and
> process-substitution/fallback in the relevant places.
>
> I'm looking at this because I want to make the substitute code less tied to
> the script and usable in the Guile guix-daemon.
>
> * guix/scripts/substitute.scm (network-error?): Move warning to…
> (process-substitution/fallback, process-substitution): here.
>
> Change-Id: I082b482c0e6ec7e02a8d437ba22dcefca5c40787

LGTM.
L
L
Ludovic Courtès wrote on 23 Feb 17:19 +0100
Re: [bug#69291] [PATCH 2/5] scripts: substitute: Allow not using with-timeout in download-nar.
(name . Christopher Baines)(address . mail@cbaines.net)
8734tjgnkt.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (9 lines)
> I don't think the approach of using SIGALARM here for the timeout will work
> well in all cases (e.g. when using Guile Fibers), so make it possible to avoid
> this.
>
> * guix/scripts/substitute.scm (download-nar): Pass the fetch timeout in as an
> option.
>
> Change-Id: I8cbe6cdfa10cdaa7d41974cbea56a95f5efecfe6

The patch LGTM.

That said, maybe we should just pass #:timeout to ‘http-fetch’? It’s
not strictly equivalent because it only controls the timeout on
connection establishment, but in practice it should have the same
effect.

Ludo’.
L
L
Ludovic Courtès wrote on 23 Feb 17:27 +0100
Re: [bug#69291] [PATCH 5/5] scripts: substitute: Extract script specific output from download-nar.
(name . Christopher Baines)(address . mail@cbaines.net)
87o7c7f8nb.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (9 lines)
> As this moves download-nar in a direction where it could be used outside the
> substitute script.
>
> * guix/scripts/substitute.scm (download-nar): Return more information and move
> status-port output to…
> (guix-substitute): here.
>
> Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0

LGTM.

Thanks for this series!
L
L
Ludovic Courtès wrote on 23 Feb 17:26 +0100
Re: [bug#69291] [PATCH 4/5] scripts: substitute: Untangle selecting fast vs small compressions.
(name . Christopher Baines)(address . mail@cbaines.net)
87sf1jf8os.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (14 lines)
> Pulling the logic up to the script makes this code more portable and not
> reliant on setting a global variable.
>
> * guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to…
> (%default-prefer-fast-decompression?): this.
> (display-narinfo-data): Update accordingly.
> (download-nar): Add prefer-fast-decompression? as a keyword argument, remove
> code to set! it and return the cpu-usage recorded.
> (process-substitution, process-substitution/fallback): Accept and pass through
> prefer-fast-decompression? to download-nar.
> (guix-substitute): Move the prefer fast decompression switching logic here.
>
> Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb

[...]

Toggle quote (8 lines)
> -(define %prefer-fast-decompression?
> - ;; Whether to prefer fast decompression over good compression ratios. This
> - ;; serves in particular to choose between lzip (high compression ratio but
> - ;; low decompression throughput) and zstd (lower compression ratio but high
> - ;; decompression throughput).
> - #f)
> +(define %default-prefer-fast-decompression? #f)

I would either remove this variable or add a comment describing it (we
should do that for all top-level variables).

Toggle quote (9 lines)
> @@ -604,7 +585,9 @@ (define* (download-nar narinfo destination
> (format status-port "hash-mismatch ~a ~a ~a~%"
> (hash-algorithm-name algorithm)
> (bytevector->nix-base32-string expected)
> - (bytevector->nix-base32-string actual)))))))
> + (bytevector->nix-base32-string actual))))
> +
> + cpu-usage)))

[...]

Toggle quote (32 lines)
> + (let ((cpu-usage
> + (process-substitution reply-port store-path destination
> + #:cache-urls (substitute-urls)
> + #:acl (current-acl)
> + #:deduplicate? deduplicate?
> + #:print-build-trace?
> + print-build-trace?
> + #:prefer-fast-decompression?
> + prefer-fast-decompression?)))
> +
> + ;; Create a hysteresis: depending on CPU usage, favor
> + ;; compression methods with faster decompression (like ztsd)
> + ;; or methods with better compression ratios (like lzip).
> + ;; This stems from the observation that substitution can be
> + ;; CPU-bound when high-speed networks are used:
> + ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
> + ;; To simulate "slow" networking or changing conditions, run:
> + ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency
> + ;; 50ms burst 1540 and then cancel with: sudo tc qdisc del
> + ;; dev eno1 root
> + (loop (cond
> + ;; Whether to prefer fast decompression over good
> + ;; compression ratios. This serves in particular to
> + ;; choose between lzip (high compression ratio but low
> + ;; decompression throughput) and zstd (lower
> + ;; compression ratio but high decompression
> + ;; throughput).
> + ((> cpu-usage .8) #t)
> + ((< cpu-usage .2) #f)
> + (else prefer-fast-decompression?)))))))))


Instead of having ‘download-nar’ return its CPU usage, which is
surprising, maybe should wrap the ‘process-substitution’ call in
‘guix-substitute’ in ‘with-cpu-usage-monitoring’ and keep all the logic
in ‘guix-substitute’?

Ludo’.
L
L
Ludovic Courtès wrote on 23 Feb 17:20 +0100
Re: [bug#69291] [PATCH 3/5] scripts: substitute: Replace some leave calls with raise.
(name . Christopher Baines)(address . mail@cbaines.net)
87y1bbf8zs.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (11 lines)
> These calls happen inside of with-error-handling, so the effect should be the
> same, but this opens up the possibility of using this code in a program that
> doesn't want to exit when one of these error conditions is met.
>
> Change-Id: I15d963615d85d419559fa0f4333fa4dc1dfbfd3b
>
> * guix/scripts/substitute.scm (download-nar, process-substitution): Use raise
> formatted-message rather than leave.
>
> Change-Id: Idd0880206b69e3903e19e0536b87d65a52c200d5

LGTM.
C
C
Christopher Baines wrote on 3 Apr 19:26 +0200
Re: [bug#69291] [PATCH 2/5] scripts: substitute: Allow not using with-timeout in download-nar.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 69291@debbugs.gnu.org)
87wmpejr2n.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (18 lines)
> Christopher Baines <mail@cbaines.net> skribis:
>
>> I don't think the approach of using SIGALARM here for the timeout will work
>> well in all cases (e.g. when using Guile Fibers), so make it possible to avoid
>> this.
>>
>> * guix/scripts/substitute.scm (download-nar): Pass the fetch timeout in as an
>> option.
>>
>> Change-Id: I8cbe6cdfa10cdaa7d41974cbea56a95f5efecfe6
>
> The patch LGTM.
>
> That said, maybe we should just pass #:timeout to ‘http-fetch’? It’s
> not strictly equivalent because it only controls the timeout on
> connection establishment, but in practice it should have the same
> effect.

I haven't done that yet, but longer term I do want to make more changes
here.

In particular, I think the way to go regarding timeouts is to use Guile
suspendable ports and have the read/write waiters handle the
timeout. The build coordinator does this [1], it's quite similar to what
is happening with the http-fetch timeout in connect*, but it's
compatible with fibers.

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYNkTBfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xe5Jg//fpPob6caE6VA3e0war4+32itKvSmN/vj
ze7PjwiJLK/5t1ZnpUHCrbGLbYux4uKhYET2eKDaCeshF/Wwccd160/UC5OuJ0vr
Dxp49jR37AzC2sOGNTasWF14VItVpXLS6+fRGV1SOYuZYmX64rEkQo812VHHZgiJ
bgHwtdSgchkjgiQRjq63xOjEY8smePVCDwdRZPA1XRv1OAT8tgcK3Q4xyvQbN65m
KItU+rfz7ObcAn4Xo6oe/qU0dSlcozRGAsxCaEU3RSI7YHAElDG2qcOyhHogYRQ2
UyyhvxTDSrG4F/AKpPsnx6NM64jAAzbH0GEIe0nmeaOnPKLQMxfiBhlp1y/x3z3o
/MwokDXQJaXEq/v/D9eYraVr1vkVcqzWSyc11KGFZsyGXw2viLvX9td1AiXZ/+VI
wDHGJQ4X0gJRdIS4v7Ou9RfFv9ORZERVt95vL4gqtST2KHlf8419hn7ehURDIQIV
8nSPl6JU5A1VREG7aETEBX2QQQsRKdj7Fhkln/uPJsMPupNGsXh91S3j723nXLWB
xKyQJr7K88jN1ODfiUbTM8DoxlZkpBIaJ4qKiymZjKgbKvHAvbjxVgjeZcZ7jVLN
3EGgwTGSgo0+y4OiHu3S2CrKHWEj7i/P59AUPVUbLI/YHXUZqPILQVWt13g9GqhW
QEpePhJmp0k=
=Ug3X
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 3 Apr 19:28 +0200
Re: [bug#69291] [PATCH 4/5] scripts: substitute: Untangle selecting fast vs small compressions.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 69291@debbugs.gnu.org)
87o7aqjqz4.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (29 lines)
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Pulling the logic up to the script makes this code more portable and not
>> reliant on setting a global variable.
>>
>> * guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to…
>> (%default-prefer-fast-decompression?): this.
>> (display-narinfo-data): Update accordingly.
>> (download-nar): Add prefer-fast-decompression? as a keyword argument, remove
>> code to set! it and return the cpu-usage recorded.
>> (process-substitution, process-substitution/fallback): Accept and pass through
>> prefer-fast-decompression? to download-nar.
>> (guix-substitute): Move the prefer fast decompression switching logic here.
>>
>> Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb
>
> [...]
>
>> -(define %prefer-fast-decompression?
>> - ;; Whether to prefer fast decompression over good compression ratios. This
>> - ;; serves in particular to choose between lzip (high compression ratio but
>> - ;; low decompression throughput) and zstd (lower compression ratio but high
>> - ;; decompression throughput).
>> - #f)
>> +(define %default-prefer-fast-decompression? #f)
>
> I would either remove this variable or add a comment describing it (we
> should do that for all top-level variables).

I've added a comment now, and I'll sent an updated patch.

Toggle quote (48 lines)
>> @@ -604,7 +585,9 @@ (define* (download-nar narinfo destination
>> (format status-port "hash-mismatch ~a ~a ~a~%"
>> (hash-algorithm-name algorithm)
>> (bytevector->nix-base32-string expected)
>> - (bytevector->nix-base32-string actual)))))))
>> + (bytevector->nix-base32-string actual))))
>> +
>> + cpu-usage)))
>
> [...]
>
>> + (let ((cpu-usage
>> + (process-substitution reply-port store-path destination
>> + #:cache-urls (substitute-urls)
>> + #:acl (current-acl)
>> + #:deduplicate? deduplicate?
>> + #:print-build-trace?
>> + print-build-trace?
>> + #:prefer-fast-decompression?
>> + prefer-fast-decompression?)))
>> +
>> + ;; Create a hysteresis: depending on CPU usage, favor
>> + ;; compression methods with faster decompression (like ztsd)
>> + ;; or methods with better compression ratios (like lzip).
>> + ;; This stems from the observation that substitution can be
>> + ;; CPU-bound when high-speed networks are used:
>> + ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
>> + ;; To simulate "slow" networking or changing conditions, run:
>> + ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency
>> + ;; 50ms burst 1540 and then cancel with: sudo tc qdisc del
>> + ;; dev eno1 root
>> + (loop (cond
>> + ;; Whether to prefer fast decompression over good
>> + ;; compression ratios. This serves in particular to
>> + ;; choose between lzip (high compression ratio but low
>> + ;; decompression throughput) and zstd (lower
>> + ;; compression ratio but high decompression
>> + ;; throughput).
>> + ((> cpu-usage .8) #t)
>> + ((< cpu-usage .2) #f)
>> + (else prefer-fast-decompression?)))))))))
>
>
> Instead of having ‘download-nar’ return its CPU usage, which is
> surprising, maybe should wrap the ‘process-substitution’ call in
> ‘guix-substitute’ in ‘with-cpu-usage-monitoring’ and keep all the logic
> in ‘guix-substitute’?

Yeah, that makes sense. I'll send an updated patch shortly.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYNka9fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XdEWA//XHWY7mjjHWzznRmKks7rlBVFWjgjW60g
2/UVJp6YGz2DPunTKMcVPPF3+PF47JbuPZbPf4fPli17oh861QKJ30LFXtFRFomo
hmOAdW52yGjIZOBBE77wDmxgOTQctzuTDXl+xfdVMHptDt7HmsMmeFWs6mvGnxI+
1pUM4UjN0rDoEnjgX0pw3SUMlxao2Hfh77oOvgmb6AkVArTctcyOiAKQ6FojVqt8
QKb19eCNPudJhNR0fgnX1CEBQkif8S8xgHzj3nG+YKQ3RRxtexebh9oTBA1Uc9xE
apNpwRwOcLaJE2IblAM2CJ5EMHnLDFSh6jyKW2M6R/G7Of8MtWX9xvQQy3hdAc6L
KWjQhdyQYWazoOVOmxq/AnAQnAJo+ZaOVraM07ZTPpjBlCnG1vCYrlTRbrFQKosP
qSDqz3jn7lPNTXNWP3RNtNBZU94eXf3ODlHmeHv6v12jWVj91qG3Xm4I7kXQTvS9
2/ux3InIitJJs3YopC1TtjJkTSCPSyl28kVEvAeHn6nNQWd0hVJG5Ik8VKi67ePK
qjDD923+8WoC5P9lrYfuF6NFmzhxTX2UHIUEtqDauE4m3FQsidrbK8x3/fLR6hGc
qPDcWKvNwL8dNv7a2dOY1SZDh6dK5q+/QMSM9jNSCeS6WeyKazVdFtnP4uj3T9xd
AuUqofni73E=
=iiwd
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 3 Apr 19:30 +0200
Re: [bug#69291] [PATCH 5/5] scripts: substitute: Extract script specific output from download-nar.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 69291@debbugs.gnu.org)
87frw2jqv6.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (13 lines)
> Christopher Baines <mail@cbaines.net> skribis:
>
>> As this moves download-nar in a direction where it could be used outside the
>> substitute script.
>>
>> * guix/scripts/substitute.scm (download-nar): Return more information and move
>> status-port output to…
>> (guix-substitute): here.
>>
>> Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0
>
> LGTM.

Looking at this patch some more, I missed the failure case in
process-substitution/fallback, so I'll send an updated patch (for this
and the previous patch).

The other patches (1 to 3) have been pushed to master as
ecbab97f0732d6979642078a7164d4032b2102b8.

Chris
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYNkj1fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xf4DQ//aHxQVHxASQ+8EM31F4gEfI7+gfcdGRrU
Ba0kGYO5KlMTHI07MBU4UD5ptiAO4kkAcMxnhKKSzleNrsSEddevtuoT1y/+rRmQ
zDwjB2caAxFiQngL1YdTZJ0cl/b8Wey9JF5xuGZQ94KHjkQv0Dp/G/DdiGgeE++h
ph/rKclaXyzmHyao9lQgcjnhMYA/Bct8QLHsVaWLu+T7y/Iv4vaYbel0zvhiwvnL
wfrR64L6TlzWA9lDhdPHW77P9X23MkEdVqbKXSujxvQCgJAzP3HDd1EmZi1XP2z8
H9nXcuAfZ4uaDqHvlzhaUeuLMZlkGzq1S2+zSkUanF/HH/ECZgHyFWSC1d/tITFM
MPe5bK3iMqN/8dfqfz/HmQenacOqLXHEHibSyYwfOo2GeCzLKTAjLbFfu64/QIFW
HbEvPWaB6rN4fgQKT24SXqZbu5CnqsOL0kUGdVL9ZCUXJzDwHTNundVcgWJrETkm
1IRw/6w1YivStZ/d71/cyDAu1CJ3jWaJZolfptr5m+8JnZvgW8vH0A89cqABietG
MbRfBraDG3q9qHJ5hhVM0LrxUaBfM8HBQXRqe2Ozjrif/pseo38Kr7EiXOSuEXmJ
Y3U+E2B16ZBXiDjhhuEmXqKR/aEhsR5/SnI5MOkdlAH++aO7r1mYsGYWD/BLCQhW
OtPqWEOt2cE=
=n8G9
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 4 Apr 16:06 +0200
[PATCH v2 1/2] scripts: substitute: Untangle selecting fast vs small compressions.
(address . 69291@debbugs.gnu.org)
6673d6ef271b684628ddbc30c7b198b72b91ad46.1712239589.git.mail@cbaines.net
Pulling the logic up to the script makes this code more portable and not
reliant on setting a global variable.

* guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to…
(%default-prefer-fast-decompression?): this.
(call-with-cpu-usage-monitoring): Use multiple values to return the results
from the thunk as well as the cpu usage.
(display-narinfo-data): Update accordingly.
(download-nar): Add prefer-fast-decompression? as a keyword argument, remove
code to set! it and monitor the cpu-usage.
(process-substitution, process-substitution/fallback): Accept and pass through
prefer-fast-decompression? to download-nar.
(guix-substitute): Move the cpu usage monitoring and prefer fast decompression
switching logic here.

Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb
---
guix/scripts/substitute.scm | 126 +++++++++++++++++++++---------------
1 file changed, 73 insertions(+), 53 deletions(-)

Toggle diff (212 lines)
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index a7ad56dbcd..0d0fd0e73b 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -261,22 +261,24 @@ (define (show-help)
;;; Daemon/substituter protocol.
;;;
-(define %prefer-fast-decompression?
- ;; Whether to prefer fast decompression over good compression ratios. This
- ;; serves in particular to choose between lzip (high compression ratio but
- ;; low decompression throughput) and zstd (lower compression ratio but high
- ;; decompression throughput).
- #f)
-
-(define (call-with-cpu-usage-monitoring proc)
+;; Whether to initially prefer fast decompression or not
+(define %default-prefer-fast-decompression? #f)
+
+(define (call-with-cpu-usage-monitoring thunk)
(let ((before (times)))
- (proc)
- (let ((after (times)))
- (if (= (tms:clock after) (tms:clock before))
- 0
- (/ (- (tms:utime after) (tms:utime before))
- (- (tms:clock after) (tms:clock before))
- 1.)))))
+ (call-with-values thunk
+ (lambda vals
+ (let ((after (times)))
+ (apply
+ values
+ (append
+ (or vals '())
+ (list
+ (if (= (tms:clock after) (tms:clock before))
+ 0
+ (/ (- (tms:utime after) (tms:utime before))
+ (- (tms:clock after) (tms:clock before))
+ 1.))))))))))
(define-syntax-rule (with-cpu-usage-monitoring exp ...)
"Evaluate EXP... Return its CPU usage as a fraction between 0 and 1."
@@ -297,7 +299,7 @@ (define (display-narinfo-data port narinfo)
(let ((uri compression file-size
(narinfo-best-uri narinfo
#:fast-decompression?
- %prefer-fast-decompression?)))
+ %default-prefer-fast-decompression?)))
(format port "~a\n~a\n"
(or file-size 0)
(or (narinfo-size narinfo) 0))))
@@ -453,7 +455,8 @@ (define-syntax-rule (catch-system-error exp)
(define* (download-nar narinfo destination
#:key status-port
deduplicate? print-build-trace?
- (fetch-timeout %fetch-timeout))
+ (fetch-timeout %fetch-timeout)
+ prefer-fast-decompression?)
"Download the nar prescribed in NARINFO, which is assumed to be authentic
and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and
if DESTINATION is in the store, deduplicate its files. Print a status line to
@@ -527,7 +530,7 @@ (define* (download-nar narinfo destination
(let ((choices (narinfo-preferred-uris narinfo
#:fast-decompression?
- %prefer-fast-decompression?)))
+ prefer-fast-decompression?)))
;; 'guix publish' without '--cache' doesn't specify a Content-Length, so
;; DOWNLOAD-SIZE is #f in this case.
(let* ((raw uri compression download-size (try-fetch choices))
@@ -560,29 +563,13 @@ (define* (download-nar narinfo destination
;; Compute the actual nar hash as we read it.
(algorithm expected (narinfo-hash-algorithm+value narinfo))
(hashed get-hash (open-hash-input-port algorithm input)))
- ;; Unpack the Nar at INPUT into DESTINATION.
- (define cpu-usage
- (with-cpu-usage-monitoring
- (restore-file hashed destination
- #:dump-file (if (and destination-in-store?
- deduplicate?)
- dump-file/deduplicate*
- dump-file))))
-
- ;; Create a hysteresis: depending on CPU usage, favor compression
- ;; methods with faster decompression (like ztsd) or methods with better
- ;; compression ratios (like lzip). This stems from the observation that
- ;; substitution can be CPU-bound when high-speed networks are used:
- ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
- ;; To simulate "slow" networking or changing conditions, run:
- ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency 50ms burst 1540
- ;; and then cancel with:
- ;; sudo tc qdisc del dev eno1 root
- (when (> cpu-usage .8)
- (set! %prefer-fast-decompression? #t))
- (when (< cpu-usage .2)
- (set! %prefer-fast-decompression? #f))
+ ;; Unpack the Nar at INPUT into DESTINATION.
+ (restore-file hashed destination
+ #:dump-file (if (and destination-in-store?
+ deduplicate?)
+ dump-file/deduplicate*
+ dump-file))
(close-port hashed)
(close-port input)
@@ -630,7 +617,8 @@ (define network-error?
(define* (process-substitution/fallback port narinfo destination
#:key cache-urls acl
- deduplicate? print-build-trace?)
+ deduplicate? print-build-trace?
+ prefer-fast-decompression?)
"Attempt to substitute NARINFO, which is assumed to be authorized or
equivalent, by trying to download its nar from each entry in CACHE-URLS.
@@ -664,14 +652,17 @@ (define* (process-substitution/fallback port narinfo destination
(download-nar alternate destination
#:status-port port
#:deduplicate? deduplicate?
- #:print-build-trace? print-build-trace?))
+ #:print-build-trace? print-build-trace?
+ #:prefer-fast-decompression?
+ prefer-fast-decompression?))
(loop rest)))
(()
(loop rest)))))))
(define* (process-substitution port store-item destination
#:key cache-urls acl
- deduplicate? print-build-trace?)
+ deduplicate? print-build-trace?
+ prefer-fast-decompression?)
"Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to
DESTINATION as a nar file. Verify the substitute against ACL, and verify its
hash against what appears in the narinfo. When DEDUPLICATE? is true, and if
@@ -703,11 +694,14 @@ (define* (process-substitution port store-item destination
#:acl acl
#:deduplicate? deduplicate?
#:print-build-trace?
- print-build-trace?)))
+ print-build-trace?
+ #:prefer-fast-decompression?
+ prefer-fast-decompression?)))
(download-nar narinfo destination
#:status-port port
#:deduplicate? deduplicate?
- #:print-build-trace? print-build-trace?)))
+ #:print-build-trace? print-build-trace?
+ #:prefer-fast-decompression? prefer-fast-decompression?)))
;;;
@@ -897,18 +891,44 @@ (define-command (guix-substitute . args)
;; Specify the number of columns of the terminal so the progress
;; report displays nicely.
(parameterize ((current-terminal-columns (client-terminal-columns)))
- (let loop ()
+ (let loop ((prefer-fast-decompression?
+ %default-prefer-fast-decompression?))
(match (read-line)
((? eof-object?)
#t)
((= string-tokenize ("substitute" store-path destination))
- (process-substitution reply-port store-path destination
- #:cache-urls (substitute-urls)
- #:acl (current-acl)
- #:deduplicate? deduplicate?
- #:print-build-trace?
- print-build-trace?)
- (loop))))))
+ (let ((cpu-usage
+ (with-cpu-usage-monitoring
+ (process-substitution
+ reply-port store-path destination
+ #:cache-urls (substitute-urls)
+ #:acl (current-acl)
+ #:deduplicate? deduplicate?
+ #:print-build-trace?
+ print-build-trace?
+ #:prefer-fast-decompression?
+ prefer-fast-decompression?))))
+
+ ;; Create a hysteresis: depending on CPU usage, favor
+ ;; compression methods with faster decompression (like ztsd)
+ ;; or methods with better compression ratios (like lzip).
+ ;; This stems from the observation that substitution can be
+ ;; CPU-bound when high-speed networks are used:
+ ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
+ ;; To simulate "slow" networking or changing conditions, run:
+ ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency
+ ;; 50ms burst 1540 and then cancel with: sudo tc qdisc del
+ ;; dev eno1 root
+ (loop (cond
+ ;; Whether to prefer fast decompression over good
+ ;; compression ratios. This serves in particular to
+ ;; choose between lzip (high compression ratio but low
+ ;; decompression throughput) and zstd (lower
+ ;; compression ratio but high decompression
+ ;; throughput).
+ ((> cpu-usage .8) #t)
+ ((< cpu-usage .2) #f)
+ (else prefer-fast-decompression?)))))))))
(opts
(leave (G_ "~a: unrecognized options~%") opts))))))

base-commit: c9cd16c630ccba655b93ff32fd9a99570b4f5373
--
2.41.0
C
C
Christopher Baines wrote on 4 Apr 16:06 +0200
[PATCH v2 2/2] scripts: substitute: Extract script specific output from download-nar.
(address . 69291@debbugs.gnu.org)
eb3a8bba107ddf9bca1cf12af3ed39eefe13c872.1712239589.git.mail@cbaines.net
As this moves download-nar in a direction where it could be used outside the
substitute script.

* guix/scripts/substitute.scm (download-nar): Return expected and actual
hashes and move status-port output to guix-substitute.
(process-substitution/fallback): Remove port argument, and move output to port
to guix-substitute.
(process-substitution): Return hashes from download-nar or
process-substitution/fallback, plus the narinfo.
(guix-substitute): Don't pass the reply-port in to process-substitution and
implement the messages to the reply-port here.

Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0
---
guix/scripts/substitute.scm | 162 ++++++++++++++++++++----------------
1 file changed, 90 insertions(+), 72 deletions(-)

Toggle diff (237 lines)
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 0d0fd0e73b..c2bc16085d 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -453,14 +453,12 @@ (define-syntax-rule (catch-system-error exp)
(const #f)))
(define* (download-nar narinfo destination
- #:key status-port
- deduplicate? print-build-trace?
+ #:key deduplicate? print-build-trace?
(fetch-timeout %fetch-timeout)
prefer-fast-decompression?)
"Download the nar prescribed in NARINFO, which is assumed to be authentic
and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and
-if DESTINATION is in the store, deduplicate its files. Print a status line to
-STATUS-PORT."
+if DESTINATION is in the store, deduplicate its files."
(define destination-in-store?
(string-prefix? (string-append (%store-prefix) "/")
destination))
@@ -576,24 +574,8 @@ (define* (download-nar narinfo destination
;; Wait for the reporter to finish.
(every (compose zero? cdr waitpid) pids)
- ;; Skip a line after what 'progress-reporter/file' printed, and another
- ;; one to visually separate substitutions. When PRINT-BUILD-TRACE? is
- ;; true, leave it up to (guix status) to prettify things.
- (newline (current-error-port))
- (unless print-build-trace?
- (newline (current-error-port)))
-
- ;; Check whether we got the data announced in NARINFO.
- (let ((actual (get-hash)))
- (if (bytevector=? actual expected)
- ;; Tell the daemon that we're done.
- (format status-port "success ~a ~a~%"
- (narinfo-hash narinfo) (narinfo-size narinfo))
- ;; The actual data has a different hash than that in NARINFO.
- (format status-port "hash-mismatch ~a ~a ~a~%"
- (hash-algorithm-name algorithm)
- (bytevector->nix-base32-string expected)
- (bytevector->nix-base32-string actual)))))))
+ (values expected
+ (get-hash)))))
(define (system-error? exception)
"Return true if EXCEPTION is a Guile 'system-error exception."
@@ -615,7 +597,7 @@ (define network-error?
'(gnutls-error getaddrinfo-error)))
(http-get-error? exception)))))
-(define* (process-substitution/fallback port narinfo destination
+(define* (process-substitution/fallback narinfo destination
#:key cache-urls acl
deduplicate? print-build-trace?
prefer-fast-decompression?)
@@ -630,9 +612,8 @@ (define* (process-substitution/fallback port narinfo destination
(let loop ((cache-urls cache-urls))
(match cache-urls
(()
- (report-error (G_ "failed to find alternative substitute for '~a'~%")
- (narinfo-path narinfo))
- (display "not-found\n" port))
+ ;; Failure, so return two values like download-nar
+ (values #f #f))
((cache-url rest ...)
(match (lookup-narinfos cache-url
(list (narinfo-path narinfo))
@@ -650,7 +631,6 @@ (define* (process-substitution/fallback port narinfo destination
(http-get-error-reason c)))
(loop rest)))
(download-nar alternate destination
- #:status-port port
#:deduplicate? deduplicate?
#:print-build-trace? print-build-trace?
#:prefer-fast-decompression?
@@ -659,7 +639,7 @@ (define* (process-substitution/fallback port narinfo destination
(()
(loop rest)))))))
-(define* (process-substitution port store-item destination
+(define* (process-substitution store-item destination
#:key cache-urls acl
deduplicate? print-build-trace?
prefer-fast-decompression?)
@@ -680,28 +660,34 @@ (define* (process-substitution port store-item destination
(G_ "no valid substitute for '~a'~%")
store-item)))
- (guard (c ((network-error? c)
- (when (http-get-error? c)
- (warning (G_ "download from '~a' failed: ~a, ~s~%")
- (uri->string (http-get-error-uri c))
- (http-get-error-code c)
- (http-get-error-reason c)))
- (format (current-error-port)
- (G_ "retrying download of '~a' with other substitute URLs...~%")
- store-item)
- (process-substitution/fallback port narinfo destination
- #:cache-urls cache-urls
- #:acl acl
- #:deduplicate? deduplicate?
- #:print-build-trace?
- print-build-trace?
- #:prefer-fast-decompression?
- prefer-fast-decompression?)))
- (download-nar narinfo destination
- #:status-port port
- #:deduplicate? deduplicate?
- #:print-build-trace? print-build-trace?
- #:prefer-fast-decompression? prefer-fast-decompression?)))
+ (let ((expected-hash
+ actual-hash
+ (guard
+ (c ((network-error? c)
+ (when (http-get-error? c)
+ (warning (G_ "download from '~a' failed: ~a, ~s~%")
+ (uri->string (http-get-error-uri c))
+ (http-get-error-code c)
+ (http-get-error-reason c)))
+ (format
+ (current-error-port)
+ (G_ "retrying download of '~a' with other substitute URLs...~%")
+ store-item)
+ (process-substitution/fallback narinfo destination
+ #:cache-urls cache-urls
+ #:acl acl
+ #:deduplicate? deduplicate?
+ #:print-build-trace?
+ print-build-trace?
+ #:prefer-fast-decompression?
+ prefer-fast-decompression?)))
+ (download-nar narinfo destination
+ #:deduplicate? deduplicate?
+ #:print-build-trace? print-build-trace?
+ #:prefer-fast-decompression? prefer-fast-decompression?))))
+ (values narinfo
+ expected-hash
+ actual-hash)))
;;;
@@ -897,10 +883,13 @@ (define-command (guix-substitute . args)
((? eof-object?)
#t)
((= string-tokenize ("substitute" store-path destination))
- (let ((cpu-usage
+ (let ((narinfo
+ expected-hash
+ actual-hash
+ cpu-usage
(with-cpu-usage-monitoring
(process-substitution
- reply-port store-path destination
+ store-path destination
#:cache-urls (substitute-urls)
#:acl (current-acl)
#:deduplicate? deduplicate?
@@ -909,26 +898,55 @@ (define-command (guix-substitute . args)
#:prefer-fast-decompression?
prefer-fast-decompression?))))
- ;; Create a hysteresis: depending on CPU usage, favor
- ;; compression methods with faster decompression (like ztsd)
- ;; or methods with better compression ratios (like lzip).
- ;; This stems from the observation that substitution can be
- ;; CPU-bound when high-speed networks are used:
- ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
- ;; To simulate "slow" networking or changing conditions, run:
- ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit latency
- ;; 50ms burst 1540 and then cancel with: sudo tc qdisc del
- ;; dev eno1 root
- (loop (cond
- ;; Whether to prefer fast decompression over good
- ;; compression ratios. This serves in particular to
- ;; choose between lzip (high compression ratio but low
- ;; decompression throughput) and zstd (lower
- ;; compression ratio but high decompression
- ;; throughput).
- ((> cpu-usage .8) #t)
- ((< cpu-usage .2) #f)
- (else prefer-fast-decompression?)))))))))
+ (if expected-hash
+ (begin
+ ;; Skip a line after what 'progress-reporter/file'
+ ;; printed, and another one to visually separate
+ ;; substitutions. When PRINT-BUILD-TRACE? is true,
+ ;; leave it up to (guix status) to prettify things.
+ (newline (current-error-port))
+ (unless print-build-trace?
+ (newline (current-error-port)))
+
+ ;; Check whether we got the data announced in NARINFO.
+ (if (bytevector=? actual-hash expected-hash)
+ ;; Tell the daemon that we're done.
+ (format reply-port "success ~a ~a~%"
+ (narinfo-hash narinfo) (narinfo-size narinfo))
+ ;; The actual data has a different hash than that in NARINFO.
+ (format reply-port "hash-mismatch ~a ~a ~a~%"
+ (hash-algorithm-name
+ (narinfo-hash-algorithm+value narinfo))
+ (bytevector->nix-base32-string expected-hash)
+ (bytevector->nix-base32-string actual-hash)))
+
+ ;; Create a hysteresis: depending on CPU usage, favor
+ ;; compression methods with faster decompression (like
+ ;; ztsd) or methods with better compression ratios
+ ;; (like lzip). This stems from the observation that
+ ;; substitution can be CPU-bound when high-speed
+ ;; networks are used:
+ ;; <https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>.
+ ;; To simulate "slow" networking or changing
+ ;; conditions, run: sudo tc qdisc add dev eno1 root tbf
+ ;; rate 512kbit latency 50ms burst 1540 and then cancel
+ ;; with: sudo tc qdisc del dev eno1 root
+ (loop (cond
+ ;; Whether to prefer fast decompression over
+ ;; good compression ratios. This serves in
+ ;; particular to choose between lzip (high
+ ;; compression ratio but low decompression
+ ;; throughput) and zstd (lower compression ratio
+ ;; but high decompression throughput).
+ ((> cpu-usage .8) #t)
+ ((< cpu-usage .2) #f)
+ (else prefer-fast-decompression?))))
+ (begin
+ (report-error (G_ "failed to find alternative substitute for '~a'~%")
+ (narinfo-path narinfo))
+ (display "not-found\n" reply-port)
+
+ (loop prefer-fast-decompression?)))))))))
(opts
(leave (G_ "~a: unrecognized options~%") opts))))))
--
2.41.0
C
C
Christopher Baines wrote on 21 Apr 13:24 +0200
Re: [bug#69291] [PATCH 5/5] scripts: substitute: Extract script specific output from download-nar.
(address . 69291-done@debbugs.gnu.org)
875xwb2c1j.fsf@cbaines.net
Christopher Baines <mail@cbaines.net> writes:

Toggle quote (19 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Christopher Baines <mail@cbaines.net> skribis:
>>
>>> As this moves download-nar in a direction where it could be used outside the
>>> substitute script.
>>>
>>> * guix/scripts/substitute.scm (download-nar): Return more information and move
>>> status-port output to…
>>> (guix-substitute): here.
>>>
>>> Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0
>>
>> LGTM.
>
> Looking at this patch some more, I missed the failure case in
> process-substitution/fallback, so I'll send an updated patch (for this
> and the previous patch).

I've sent a couple of updated patches as part of the series to
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYk94hfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xfv8BAAgHSzx9+C9FKLugUD3CoDIUo85Fh/N8D3
xlQpbm9yj5U4mTy0auwUBS0KRS+fpZ6FrsmSLXiKIfRYnWKDPh7/AEnhtgn4wHen
4Ja1c/2E3SCfca5ei6Z2jj8u1nuFpxQbce5fpN1XWE5OTbtcF1Yr9z5FZQSIWJ35
x8gL8eEUjMZ/9iwSWzL44AKEh6xuvhxObd4m2a1AEj6qVxtoLRw2IKa3vR9Y19Yy
/lLSHKm8F4EQoUjRcvpqIiRcrJ+SLlykDoU7Vv6EOSr2rG0MxI/7oFEvJXWWpQkS
Cysk1u76ltcT/KfEg5Gwc2uR/lvfiA13RHUb/A2lY705zNGpLbSvJfHYb4FsY7md
N23kjg/lDsxuEQrYtTYvbtVcgNz7yfDueq0s6fiEjeTq47lK7q8+AjUr63Y2UcnV
AdZi9PSd8BQD4pxTr9lmLHEzpax1EmudzkeiMeZLdq3g4h4HBL6gyp58Jh5G9MNp
DfmI58H0/9aP829pKES+KDbs10Xyy3gpyyTgFE1dbXNw5SwRHcOhMCB9XZ7mao5D
/fstv5DNT3CrDPsEpaxcNtivo1N+gMYVQVyDTDpztA0+cvPbj1E/mPH1tNLmwAkG
zanZ2glgjy1oLEXoSzDp1etDEppogZlg9KgCk0SsZAcCHSkIASp7zkvXsSDGp+ki
0rrTeyUMTDY=
=Yb0I
-----END PGP SIGNATURE-----

Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 69291@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 69291
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch