[PATCH] guix: http-client: Tweak http-multiple-get error handling.

DoneSubmitted by Christopher Baines.
Details
3 participants
  • Ludovic Courtès
  • Christopher Baines
  • Maxime Devos
Owner
unassigned
Severity
normal
C
C
Christopher Baines wrote on 21 Mar 01:43 +0100
(address . guix-patches@gnu.org)
20210321004338.31867-1-mail@cbaines.net
This isn't meant to change the way errors are handled, and arguably makes thecode harder to read, but it's a uninformed attempt to improve theperformance (following on from a performance regression in205833b72c5517915a47a50dbe28e7024dc74e57).
I'm guessing something about Guile internals makes calling (loop ...) withinthe catch bit less performant than avoiding this and calling (loop ...) afterthe catch bit has finished. Since this happens lots, this seems to besufficient to make guix weather a lot slower than it was before.
Anecdotal testing of guix weather suggests this change might work.
* guix/http-client.scm (http-multiple-get): Tweak how the second catchstatement works.--- guix/http-client.scm | 77 +++++++++++++++++++++++--------------------- 1 file changed, 41 insertions(+), 36 deletions(-)
Toggle diff (90 lines)diff --git a/guix/http-client.scm b/guix/http-client.scmindex 4b4c14ed0b..a189cceecf 100644--- a/guix/http-client.scm+++ b/guix/http-client.scm@@ -219,42 +219,47 @@ returning." (remainder (connect p remainder result)))) ((head tail ...)- (catch #t- (lambda ()- (let* ((resp (read-response p))- (body (response-body-port resp))- (result (proc head resp body result)))- ;; The server can choose to stop responding at any time,- ;; in which case we have to try again. Check whether- ;; that is the case. Note that even upon "Connection:- ;; close", we can read from BODY.- (match (assq 'connection (response-headers resp))- (('connection 'close)- (close-port p)- (connect #f ;try again- (drop requests (+ 1 processed))- result))- (_- (loop tail (+ 1 processed) result))))) ;keep going- (lambda (key . args)- ;; If PORT was cached and the server closed the connection- ;; in the meantime, we get EPIPE. In that case, open a- ;; fresh connection and retry. We might also get- ;; 'bad-response or a similar exception from (web response)- ;; later on, once we've sent the request, or a- ;; ERROR/INVALID-SESSION from GnuTLS.- (if (or (and (eq? key 'system-error)- (= EPIPE (system-error-errno `(,key ,@args))))- (and (eq? key 'gnutls-error)- (eq? (first args) error/invalid-session))- (memq key- '(bad-response bad-header bad-header-component)))- (begin- (close-port p)- (connect #f ; try again- (drop requests (+ 1 processed))- result))- (apply throw key args))))))))))+ (match+ (catch #t+ (lambda ()+ (let* ((resp (read-response p))+ (body (response-body-port resp))+ (result (proc head resp body result)))+ ;; The server can choose to stop responding at any time,+ ;; in which case we have to try again. Check whether+ ;; that is the case. Note that even upon "Connection:+ ;; close", we can read from BODY.+ (match (assq 'connection (response-headers resp))+ (('connection 'close)+ (close-port p)+ 'try-again-with-new-connection)+ (_+ result))))+ (lambda (key . args)+ ;; If PORT was cached and the server closed the connection+ ;; in the meantime, we get EPIPE. In that case, open a+ ;; fresh connection and retry. We might also get+ ;; 'bad-response or a similar exception from (web response)+ ;; later on, once we've sent the request, or a+ ;; ERROR/INVALID-SESSION from GnuTLS.+ (if (or (and (eq? key 'system-error)+ (= EPIPE (system-error-errno `(,key ,@args))))+ (and (eq? key 'gnutls-error)+ (eq? (first args) error/invalid-session))1+ (memq key+ '(bad-response+ bad-header+ bad-header-component)))+ (begin+ (close-port p)+ 'try-again-with-new-connection)+ (apply throw key args))))+ ('try-again-with-new-connection+ (connect #f+ (drop requests (+ 1 processed))+ result))+ (result+ (loop tail (+ 1 processed) result))))))))) ;;;-- 2.30.1
C
C
Christopher Baines wrote on 21 Mar 01:56 +0100
[PATCH v2] guix: http-client: Tweak http-multiple-get error handling.
(address . 47288@debbugs.gnu.org)
20210321005600.12551-1-mail@cbaines.net
This isn't meant to change the way errors are handled, and arguably makes thecode harder to read, but it's a uninformed attempt to improve theperformance (following on from a performance regression in205833b72c5517915a47a50dbe28e7024dc74e57).
I'm guessing something about Guile internals makes calling (loop ...) withinthe catch bit less performant than avoiding this and calling (loop ...) afterthe catch bit has finished. Since this happens lots, this seems to besufficient to make guix weather a lot slower than it was before.
Anecdotal testing of guix weather suggests this change might work.
* guix/http-client.scm (http-multiple-get): Tweak how the second catchstatement works.--- guix/http-client.scm | 77 +++++++++++++++++++++++++------------------- 1 file changed, 43 insertions(+), 34 deletions(-)
Toggle diff (92 lines)diff --git a/guix/http-client.scm b/guix/http-client.scmindex 4b4c14ed0b..a9609445c8 100644--- a/guix/http-client.scm+++ b/guix/http-client.scm@@ -219,42 +219,51 @@ returning." (remainder (connect p remainder result)))) ((head tail ...)- (catch #t- (lambda ()- (let* ((resp (read-response p))- (body (response-body-port resp))- (result (proc head resp body result)))- ;; The server can choose to stop responding at any time,- ;; in which case we have to try again. Check whether- ;; that is the case. Note that even upon "Connection:- ;; close", we can read from BODY.- (match (assq 'connection (response-headers resp))- (('connection 'close)- (close-port p)- (connect #f ;try again- (drop requests (+ 1 processed))- result))- (_- (loop tail (+ 1 processed) result))))) ;keep going- (lambda (key . args)- ;; If PORT was cached and the server closed the connection- ;; in the meantime, we get EPIPE. In that case, open a- ;; fresh connection and retry. We might also get- ;; 'bad-response or a similar exception from (web response)- ;; later on, once we've sent the request, or a- ;; ERROR/INVALID-SESSION from GnuTLS.- (if (or (and (eq? key 'system-error)- (= EPIPE (system-error-errno `(,key ,@args))))- (and (eq? key 'gnutls-error)- (eq? (first args) error/invalid-session))- (memq key- '(bad-response bad-header bad-header-component)))- (begin- (close-port p)- (connect #f ; try again+ (match+ (catch #t+ (lambda ()+ (let* ((resp (read-response p))+ (body (response-body-port resp))+ (result (proc head resp body result)))+ ;; The server can choose to stop responding at any time,+ ;; in which case we have to try again. Check whether+ ;; that is the case. Note that even upon "Connection:+ ;; close", we can read from BODY.+ (match (assq 'connection (response-headers resp))+ (('connection 'close)+ (close-port p)+ (list 'connect+ #f (drop requests (+ 1 processed)) result))- (apply throw key args))))))))))+ (_+ (list 'loop tail (+ 1 processed) result)))))+ (lambda (key . args)+ ;; If PORT was cached and the server closed the connection+ ;; in the meantime, we get EPIPE. In that case, open a+ ;; fresh connection and retry. We might also get+ ;; 'bad-response or a similar exception from (web response)+ ;; later on, once we've sent the request, or a+ ;; ERROR/INVALID-SESSION from GnuTLS.+ (if (or (and (eq? key 'system-error)+ (= EPIPE (system-error-errno `(,key ,@args))))+ (and (eq? key 'gnutls-error)+ (eq? (first args) error/invalid-session))1+ (memq key+ '(bad-response+ bad-header+ bad-header-component)))+ (begin+ (close-port p)+ (list 'connect+ #f+ requests+ result))+ (apply throw key args))))+ (('connect . args)+ (apply connect args))+ (('loop . args)+ (apply loop args))))))))) ;;;-- 2.30.1
M
M
Maxime Devos wrote on 21 Mar 09:36 +0100
Re: [bug#47288] [PATCH] guix: http-client: Tweak http-multiple-get error handling.
70ae9918cfe06c3dcce6764f170a96b55e275d4d.camel@telenet.be
On Sun, 2021-03-21 at 00:43 +0000, Christopher Baines wrote:
Toggle quote (10 lines)> This isn't meant to change the way errors are handled, and arguably makes the> code harder to read, but it's a uninformed attempt to improve the> performance (following on from a performance regression in> 205833b72c5517915a47a50dbe28e7024dc74e57).> > I'm guessing something about Guile internals makes calling (loop ...) within> the catch bit less performant than avoiding this and calling (loop ...) after> the catch bit has finished. Since this happens lots, this seems to be> sufficient to make guix weather a lot slower than it was before.
I took a look at the code, and it seems we did somthing like:
(let loop VARS (match sent NON-LOOPING-CASES (STUFF (catch #t THUNK-THAT-MIGHT-CALL-LOOP-IN-TAIL-POSITION SOME-HANDLER-THAT-DOES-NOT-CALL-LOOP)
A small demonstration of what could go wrong with such a construction(run this in the Guile REPL):
(let loop ((attempts-todo 20)) (catch 'oops (lambda () (if (<= 0 attempts-todo) (loop (- attempts-todo 1)) (throw 'oops))) (lambda _ (display 'too-bad!) (newline) (backtrace))))
Output:too-bad!
Backtrace:In ice-9/boot-9.scm: 1731:15 19 (with-exception-handler #<procedure 7f73c7e615a0 at ice-9/boot-9.scm:1815:7 (exn)> _ # _ …) [The previous line repeated 17 times] 1731:15 1 (with-exception-handler #<procedure 7f73c7e61240 at ice-9/boot-9.scm:1815:7 (exn)> _ # _ …)In unknown file: 0 (backtrace #<undefined>)
With this construction, we were nesting exception handlers within exception handlers... So in hindsight it doesn't seem surprising this isn't very performant.(THUNK-THAT-MIGHT-CALL-LOOP-IN-TAIL-POSITION itself is not called in the tail-positionof the '(let loop ...)' form.)
Hope that sheds some light on the matter,Maxime
-----BEGIN PGP SIGNATURE-----
iI0EABYIADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYFcFmhccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7uELAPwLbkh01qJqniid0xELzUrLgGhzJqe9NbVxRRcY42SmygEAgz/0DZPmVAoBbxJVUXxGl/J+FVtGrSEmH2atCjL5gQA==8t1s-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 24 Mar 15:55 +0100
Re: bug#47288: [PATCH] guix: http-client: Tweak http-multiple-get error handling.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 47288@debbugs.gnu.org)
87pmzoeh3p.fsf_-_@gnu.org
As discussed at https://issues.guix.gnu.org/47283, I’m still unsurethese exceptions need to be caught within ‘http-multiple-get’ and ateach iteration.
Just minor comments:
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (46 lines)> + (catch #t> + (lambda ()> + (let* ((resp (read-response p))> + (body (response-body-port resp))> + (result (proc head resp body result)))> + ;; The server can choose to stop responding at any time,> + ;; in which case we have to try again. Check whether> + ;; that is the case. Note that even upon "Connection:> + ;; close", we can read from BODY.> + (match (assq 'connection (response-headers resp))> + (('connection 'close)> + (close-port p)> + (list 'connect> + #f> (drop requests (+ 1 processed))> result))> - (apply throw key args))))))))))> + (_> + (list 'loop tail (+ 1 processed) result)))))> + (lambda (key . args)> + ;; If PORT was cached and the server closed the connection> + ;; in the meantime, we get EPIPE. In that case, open a> + ;; fresh connection and retry. We might also get> + ;; 'bad-response or a similar exception from (web response)> + ;; later on, once we've sent the request, or a> + ;; ERROR/INVALID-SESSION from GnuTLS.> + (if (or (and (eq? key 'system-error)> + (= EPIPE (system-error-errno `(,key ,@args))))> + (and (eq? key 'gnutls-error)> + (eq? (first args) error/invalid-session))1> + (memq key> + '(bad-response> + bad-header> + bad-header-component)))> + (begin> + (close-port p)> + (list 'connect> + #f> + requests> + result))> + (apply throw key args))))> + (('connect . args)> + (apply connect args))> + (('loop . args)> + (apply loop args)))))))))
This is not new to this patch, but I think the whole exception handlingbit should be factorized and written in such a way that‘http-multiple-get’ still first on a horizontal screen (even though mineis actually vertical :-)). Otherwise one might easily overlook the corelogic of the function.
Ludo’.
L
L
Ludovic Courtès wrote on 24 Mar 15:55 +0100
(name . Christopher Baines)(address . mail@cbaines.net)(address . 47288@debbugs.gnu.org)
87o8f8eh3c.fsf_-_@gnu.org
As discussed at https://issues.guix.gnu.org/47283, I’m still unsurethese exceptions need to be caught within ‘http-multiple-get’ and ateach iteration.
Just minor comments:
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (46 lines)> + (catch #t> + (lambda ()> + (let* ((resp (read-response p))> + (body (response-body-port resp))> + (result (proc head resp body result)))> + ;; The server can choose to stop responding at any time,> + ;; in which case we have to try again. Check whether> + ;; that is the case. Note that even upon "Connection:> + ;; close", we can read from BODY.> + (match (assq 'connection (response-headers resp))> + (('connection 'close)> + (close-port p)> + (list 'connect> + #f> (drop requests (+ 1 processed))> result))> - (apply throw key args))))))))))> + (_> + (list 'loop tail (+ 1 processed) result)))))> + (lambda (key . args)> + ;; If PORT was cached and the server closed the connection> + ;; in the meantime, we get EPIPE. In that case, open a> + ;; fresh connection and retry. We might also get> + ;; 'bad-response or a similar exception from (web response)> + ;; later on, once we've sent the request, or a> + ;; ERROR/INVALID-SESSION from GnuTLS.> + (if (or (and (eq? key 'system-error)> + (= EPIPE (system-error-errno `(,key ,@args))))> + (and (eq? key 'gnutls-error)> + (eq? (first args) error/invalid-session))1> + (memq key> + '(bad-response> + bad-header> + bad-header-component)))> + (begin> + (close-port p)> + (list 'connect> + #f> + requests> + result))> + (apply throw key args))))> + (('connect . args)> + (apply connect args))> + (('loop . args)> + (apply loop args)))))))))
This is not new to this patch, but I think the whole exception handlingbit should be factorized and written in such a way that‘http-multiple-get’ still fits on a horizontal screen (even though mineis actually vertical :-)). Otherwise one might easily overlook the corelogic of the function.
Ludo’.
C
C
Christopher Baines wrote on 25 Mar 12:03 +0100
[PATCH v3 1/2] guix: http-client: Tweak http-multiple-get error handling.
(address . 47288@debbugs.gnu.org)
20210325110316.862-1-mail@cbaines.net
This isn't meant to change the way errors are handled, and arguably makes thecode harder to read, but it's a uninformed attempt to improve theperformance (following on from a performance regression in205833b72c5517915a47a50dbe28e7024dc74e57).
I'm guessing something about Guile internals makes calling (loop ...) withinthe catch bit less performant than avoiding this and calling (loop ...) afterthe catch bit has finished. Since this happens lots, this seems to besufficient to make guix weather a lot slower than it was before.
Anecdotal testing of guix weather suggests this change might work.
* guix/http-client.scm (http-multiple-get): Tweak how the second catchstatement works.--- guix/http-client.scm | 77 +++++++++++++++++++++++++------------------- 1 file changed, 43 insertions(+), 34 deletions(-)
Toggle diff (92 lines)diff --git a/guix/http-client.scm b/guix/http-client.scmindex 4b4c14ed0b..adbfbc0d6e 100644--- a/guix/http-client.scm+++ b/guix/http-client.scm@@ -219,42 +219,51 @@ returning." (remainder (connect p remainder result)))) ((head tail ...)- (catch #t- (lambda ()- (let* ((resp (read-response p))- (body (response-body-port resp))- (result (proc head resp body result)))- ;; The server can choose to stop responding at any time,- ;; in which case we have to try again. Check whether- ;; that is the case. Note that even upon "Connection:- ;; close", we can read from BODY.- (match (assq 'connection (response-headers resp))- (('connection 'close)- (close-port p)- (connect #f ;try again- (drop requests (+ 1 processed))- result))- (_- (loop tail (+ 1 processed) result))))) ;keep going- (lambda (key . args)- ;; If PORT was cached and the server closed the connection- ;; in the meantime, we get EPIPE. In that case, open a- ;; fresh connection and retry. We might also get- ;; 'bad-response or a similar exception from (web response)- ;; later on, once we've sent the request, or a- ;; ERROR/INVALID-SESSION from GnuTLS.- (if (or (and (eq? key 'system-error)- (= EPIPE (system-error-errno `(,key ,@args))))- (and (eq? key 'gnutls-error)- (eq? (first args) error/invalid-session))- (memq key- '(bad-response bad-header bad-header-component)))- (begin- (close-port p)- (connect #f ; try again+ (match+ (catch #t+ (lambda ()+ (let* ((resp (read-response p))+ (body (response-body-port resp))+ (result (proc head resp body result)))+ ;; The server can choose to stop responding at any time,+ ;; in which case we have to try again. Check whether+ ;; that is the case. Note that even upon "Connection:+ ;; close", we can read from BODY.+ (match (assq 'connection (response-headers resp))+ (('connection 'close)+ (close-port p)+ (list 'connect+ #f (drop requests (+ 1 processed)) result))- (apply throw key args))))))))))+ (_+ (list 'loop tail (+ 1 processed) result)))))+ (lambda (key . args)+ ;; If PORT was cached and the server closed the connection+ ;; in the meantime, we get EPIPE. In that case, open a+ ;; fresh connection and retry. We might also get+ ;; 'bad-response or a similar exception from (web response)+ ;; later on, once we've sent the request, or a+ ;; ERROR/INVALID-SESSION from GnuTLS.+ (if (or (and (eq? key 'system-error)+ (= EPIPE (system-error-errno `(,key ,@args))))+ (and (eq? key 'gnutls-error)+ (eq? (first args) error/invalid-session))+ (memq key+ '(bad-response+ bad-header+ bad-header-component)))+ (begin+ (close-port p)+ (list 'connect+ #f+ (drop requests processed)+ result))+ (apply throw key args))))+ (('connect . args)+ (apply connect args))+ (('loop . args)+ (apply loop args))))))))) ;;;-- 2.30.1
C
C
Christopher Baines wrote on 25 Mar 12:03 +0100
[PATCH v3 2/2] guix: http-client: Refactor http-multiple-get.
(address . 47288@debbugs.gnu.org)
20210325110316.862-2-mail@cbaines.net
Split the procedure in to three smaller procedures, rather than using twolonger let statements. This might make it easier to read.
* guix/http-client.scm (http-multiple-get): Refactor.--- guix/http-client.scm | 195 ++++++++++++++++++++++--------------------- 1 file changed, 101 insertions(+), 94 deletions(-)
Toggle diff (225 lines)diff --git a/guix/http-client.scm b/guix/http-client.scmindex adbfbc0d6e..b584feba5d 100644--- a/guix/http-client.scm+++ b/guix/http-client.scm@@ -147,7 +147,7 @@ Raise an '&http-get-error' condition if downloading fails." (uri->string uri) code (response-reason-phrase resp)))))))))))) -(define* (http-multiple-get base-uri proc seed requests+(define* (http-multiple-get base-uri proc seed all-requests #:key port (verify-certificate? #t) (open-connection guix:open-connection-for-uri) (keep-alive? #t)@@ -161,16 +161,90 @@ When PORT is specified, use it as the initial connection on which HTTP requests are sent; otherwise call OPEN-CONNECTION to open a new connection for a URI. When KEEP-ALIVE? is false, close the connection port before returning."- (let connect ((port port)- (requests requests)- (result seed))+ (define (send-batch-of-requests p batch)+ ;; Send BATCH in a row.+ ;; XXX: Do our own caching to work around inefficiencies when+ ;; communicating over TLS: <http://bugs.gnu.org/22966>.+ (let-values (((buffer get) (open-bytevector-output-port)))+ ;; Inherit the HTTP proxying property from P.+ (set-http-proxy-port?! buffer (http-proxy-port? p))++ (for-each (cut write-request <> buffer)+ batch)+ (put-bytevector p (get))+ (force-output p)))++ (define (process-batch-of-responses p+ all-remaining-requests+ batch-remaining-requests+ processed+ result)+ (if (null? batch-remaining-requests)+ (match (drop all-remaining-requests processed)+ (()+ (unless keep-alive?+ (close-port p))+ (reverse result))+ (remainder+ (connect-and-make-requests p remainder result)))+ (match+ (catch #t+ (lambda ()+ (let* ((request (car batch-remaining-requests))+ (resp (read-response p))+ (body (response-body-port resp))+ (result (proc request resp body result)))+ ;; The server can choose to stop responding at any time, in+ ;; which case we have to try again. Check whether that is+ ;; the case. Note that even upon "Connection: close", we can+ ;; read from BODY.+ (match (assq 'connection (response-headers resp))+ (('connection 'close)+ (close-port p)+ (list 'connect-and-make-requests+ #f+ (drop all-remaining-requests (+ 1 processed))+ result))+ (_+ (list 'process-batch-of-responses+ p+ all-remaining-requests+ (cdr batch-remaining-requests)+ (+ 1 processed)+ result)))))+ (lambda (key . args)+ ;; If PORT was cached and the server closed the connection in+ ;; the meantime, we get EPIPE. In that case, open a fresh+ ;; connection and retry. We might also get 'bad-response or a+ ;; similar exception from (web response) later on, once we've+ ;; sent the request, or a ERROR/INVALID-SESSION from GnuTLS.+ (if (or (and (eq? key 'system-error)+ (= EPIPE (system-error-errno `(,key ,@args))))+ (and (eq? key 'gnutls-error)+ (eq? (first args) error/invalid-session))+ (memq key+ '(bad-response+ bad-header+ bad-header-component)))+ (begin+ (close-port p)+ (list 'connect-and-make-requests+ #f+ (drop all-remaining-requests processed)+ result))+ (apply throw key args))))++ (('connect-and-make-requests . args)+ (apply connect-and-make-requests args))+ (('process-batch-of-responses . args)+ (apply process-batch-of-responses args)))))++ (define (connect-and-make-requests port remaining-requests result) (define batch- (if (>= batch-size (length requests))- requests- (take requests batch-size)))+ (if (>= batch-size (length remaining-requests))+ remaining-requests+ (take remaining-requests batch-size))) - ;; (format (current-error-port) "connecting (~a requests left)..."- ;; (length requests)) (let ((p (or port (open-connection base-uri #:verify-certificate? verify-certificate?))))@@ -178,92 +252,25 @@ returning." (when (file-port? p) (setvbuf p 'block (expt 2 16))) - ;; Send BATCH in a row.- ;; XXX: Do our own caching to work around inefficiencies when- ;; communicating over TLS: <http://bugs.gnu.org/22966>.- (let-values (((buffer get) (open-bytevector-output-port)))- ;; Inherit the HTTP proxying property from P.- (set-http-proxy-port?! buffer (http-proxy-port? p))-- (catch #t- (lambda ()- (for-each (cut write-request <> buffer)- batch)- (put-bytevector p (get))- (force-output p))- (lambda (key . args)- ;; If PORT becomes unusable, open a fresh connection and- ;; retry.- (if (or (and (eq? key 'system-error)- (= EPIPE (system-error-errno `(,key ,@args))))- (and (eq? key 'gnutls-error)- (eq? (first args) error/invalid-session)))- (begin- (close-port p) ; close the broken port- (connect #f- requests- result))- (apply throw key args)))))+ (catch #t+ (lambda ()+ (send-batch-of-requests p batch))+ (lambda (key . args)+ ;; If PORT becomes unusable, open a fresh connection and retry.+ (if (or (and (eq? key 'system-error)+ (= EPIPE (system-error-errno `(,key ,@args))))+ (and (eq? key 'gnutls-error)+ (eq? (first args) error/invalid-session)))+ (begin+ (close-port p) ; close the broken port+ (connect-and-make-requests #f+ remaining-requests+ result))+ (apply throw key args)))) - ;; Now start processing responses.- (let loop ((sent batch)- (processed 0)- (result result))- (match sent- (()- (match (drop requests processed)- (()- (unless keep-alive?- (close-port p))- (reverse result))- (remainder- (connect p remainder result))))- ((head tail ...)- (match- (catch #t- (lambda ()- (let* ((resp (read-response p))- (body (response-body-port resp))- (result (proc head resp body result)))- ;; The server can choose to stop responding at any time,- ;; in which case we have to try again. Check whether- ;; that is the case. Note that even upon "Connection:- ;; close", we can read from BODY.- (match (assq 'connection (response-headers resp))- (('connection 'close)- (close-port p)- (list 'connect- #f- (drop requests (+ 1 processed))- result))- (_- (list 'loop tail (+ 1 processed) result)))))- (lambda (key . args)- ;; If PORT was cached and the server closed the connection- ;; in the meantime, we get EPIPE. In that case, open a- ;; fresh connection and retry. We might also get- ;; 'bad-response or a similar exception from (web response)- ;; later on, once we've sent the request, or a- ;; ERROR/INVALID-SESSION from GnuTLS.- (if (or (and (eq? key 'system-error)- (= EPIPE (system-error-errno `(,key ,@args))))- (and (eq? key 'gnutls-error)- (eq? (first args) error/invalid-session))- (memq key- '(bad-response- bad-header- bad-header-component)))- (begin- (close-port p)- (list 'connect- #f- (drop requests processed)- result))- (apply throw key args))))- (('connect . args)- (apply connect args))- (('loop . args)- (apply loop args)))))))))+ (process-batch-of-responses p remaining-requests batch 0 result)))++ (connect-and-make-requests port all-requests seed)) ;;;-- 2.30.1
C
C
Christopher Baines wrote on 25 Mar 12:09 +0100
Re: bug#47288: [PATCH] guix: http-client: Tweak http-multiple-get error handling.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47288@debbugs.gnu.org)
87zgyra3sg.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (60 lines)> As discussed at https://issues.guix.gnu.org/47283, I’m still unsure> these exceptions need to be caught within ‘http-multiple-get’ and at> each iteration.>> Just minor comments:>> Christopher Baines <mail@cbaines.net> skribis:>>> + (catch #t>> + (lambda ()>> + (let* ((resp (read-response p))>> + (body (response-body-port resp))>> + (result (proc head resp body result)))>> + ;; The server can choose to stop responding at any time,>> + ;; in which case we have to try again. Check whether>> + ;; that is the case. Note that even upon "Connection:>> + ;; close", we can read from BODY.>> + (match (assq 'connection (response-headers resp))>> + (('connection 'close)>> + (close-port p)>> + (list 'connect>> + #f>> (drop requests (+ 1 processed))>> result))>> - (apply throw key args))))))))))>> + (_>> + (list 'loop tail (+ 1 processed) result)))))>> + (lambda (key . args)>> + ;; If PORT was cached and the server closed the connection>> + ;; in the meantime, we get EPIPE. In that case, open a>> + ;; fresh connection and retry. We might also get>> + ;; 'bad-response or a similar exception from (web response)>> + ;; later on, once we've sent the request, or a>> + ;; ERROR/INVALID-SESSION from GnuTLS.>> + (if (or (and (eq? key 'system-error)>> + (= EPIPE (system-error-errno `(,key ,@args))))>> + (and (eq? key 'gnutls-error)>> + (eq? (first args) error/invalid-session))1>> + (memq key>> + '(bad-response>> + bad-header>> + bad-header-component)))>> + (begin>> + (close-port p)>> + (list 'connect>> + #f>> + requests>> + result))>> + (apply throw key args))))>> + (('connect . args)>> + (apply connect args))>> + (('loop . args)>> + (apply loop args)))))))))>> This is not new to this patch, but I think the whole exception handling> bit should be factorized and written in such a way that> ‘http-multiple-get’ still first on a horizontal screen (even though mine> is actually vertical :-)). Otherwise one might easily overlook the core> logic of the function.
I've sent a v3 now, which makes a few changes to the original patch, andincludes a second patch for a potential refactoring.
I tested three variants for performance, http-multiple-get with no errorhandling, the first v3 patch and the first and second v3 patches, and atleast with the test I'm using, the performance seems similar. Assumingthe performance is lower with error handling, the impact seems to bewithin the margin of error, at least for test I was using.
(use-modules (ice-9 binary-ports) (srfi srfi-19) (web uri) (web request) (web response) (guix http-client)) (define (call-with-time-logging requests thunk) (let ((start (current-time time-utc))) (call-with-values thunk (lambda vals (let* ((end (current-time time-utc)) (elapsed (time-difference end start))) (display (format #f "~f seconds (~f microseconds per request)~%" (+ (time-second elapsed) (/ (time-nanosecond elapsed) 1e9)) (* (/ (+ (time-second elapsed) (/ (time-nanosecond elapsed) 1e9)) requests) 1e6))) (apply values vals)))))) (define requests (map (lambda _ (build-request (string->uri "http://localhost/does-not-exist") #:method 'GET)) (iota 200000))) (call-with-time-logging (length requests) (lambda () (http-multiple-get (string->uri "http://localhost/") (lambda (request response port result) (get-bytevector-n port (response-content-length response)) '()) '() requests)))
-----BEGIN PGP SIGNATURE-----
iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBcb1BfFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2JhaW5lcy5uZXQACgkQXiijOwuE9XcMYg//ZHjTZpEIBy3zvkUytf7DqxlMj6/e/fkRG5B8VLkRysYDJUckziRqN/bBykRRrjYR7mCt4qX1XHROZdUAkxZ++K+xkrtUxSPqk9ezSw+akd4+geCpfyw0/R8aiqHPMUVlrvuSheUEsb2jHRD/oikkoS9N7xRrRsQN3O4+alydMmKd24SKBjAHY375JE+Q0ad69Rry8mXw8rcXPuX4wGGsMfwamI2n38pZLghc9sgH82RXP4aZmdGyS2jWt6m/Qmydu0Par159cG9Ru0CCbio0HoMZd4Vpnvy7VcMzp8UM2pAffD3CfzpOgKgIKKkb4wWJ25EgPD2+sDbCNBInz3XbUX3qCKHhngmi6SDFzBFe59G5D/VnLGcOACUBg6a2IhFBjKuGcvWSELUDRCuQp2Z91KcdDb5kxPl1Aznk2IwetBRUjXJUs8MhKsoit7E7KYFihN9zXsvXRlbnRcsyGoatTV48BXZHQOpvpXFngQYroYnp4/UIFgTdx4Dm6Oxs+VWdSZgM6/bSUmVHQmTxw++83Ia+L8zRTgp6TCtycdC7tcJxlA7fQDfp99VPki5p3QGDZkShDIm3TqHwvlf1VG504HnNTRcr7yY+2feCfA3kcA5EJNYl5iDlhHT4dWM4LxGVUUCO51vWxVLKbsxCeRlTfl+GM8ePRzzS1ErFEpvTEKI==Wa1I-----END PGP SIGNATURE-----
L
L
Ludovic Courtès wrote on 25 Mar 23:20 +0100
(name . Christopher Baines)(address . mail@cbaines.net)(address . 47288@debbugs.gnu.org)
87r1k250za.fsf_-_@gnu.org
Hi!
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (10 lines)> This isn't meant to change the way errors are handled, and arguably makes the> code harder to read, but it's a uninformed attempt to improve the> performance (following on from a performance regression in> 205833b72c5517915a47a50dbe28e7024dc74e57).>> I'm guessing something about Guile internals makes calling (loop ...) within> the catch bit less performant than avoiding this and calling (loop ...) after> the catch bit has finished. Since this happens lots, this seems to be> sufficient to make guix weather a lot slower than it was before.
As Maxime wrote, the problem is that we were making non-tail calls,thereby consuming stack space as well as accumulating exceptionhandlers. As discussed earlier, ‘raise-exception’ exhibits quadraticbehavior in the number of exception handlers, which is okay in normalsituations, but not so much when there are thousands of handlers, as isthe case when asking for many substitutes.
Toggle quote (2 lines)> Anecdotal testing of guix weather suggests this change might work.
Don’t leave this last sentence in the actual commit. :-)
Please mention https://bugs.gnu.org/47283.
Toggle quote (15 lines)> * guix/http-client.scm (http-multiple-get): Tweak how the second catch> statement works.> ---> guix/http-client.scm | 77 +++++++++++++++++++++++++-------------------> 1 file changed, 43 insertions(+), 34 deletions(-)>> diff --git a/guix/http-client.scm b/guix/http-client.scm> index 4b4c14ed0b..adbfbc0d6e 100644> --- a/guix/http-client.scm> +++ b/guix/http-client.scm> @@ -219,42 +219,51 @@ returning."> (remainder> (connect p remainder result))))> ((head tail ...)
[...]
Toggle quote (47 lines)> + (match> + (catch #t> + (lambda ()> + (let* ((resp (read-response p))> + (body (response-body-port resp))> + (result (proc head resp body result)))> + ;; The server can choose to stop responding at any time,> + ;; in which case we have to try again. Check whether> + ;; that is the case. Note that even upon "Connection:> + ;; close", we can read from BODY.> + (match (assq 'connection (response-headers resp))> + (('connection 'close)> + (close-port p)> + (list 'connect> + #f> (drop requests (+ 1 processed))> result))> - (apply throw key args))))))))))> + (_> + (list 'loop tail (+ 1 processed) result)))))> + (lambda (key . args)> + ;; If PORT was cached and the server closed the connection> + ;; in the meantime, we get EPIPE. In that case, open a> + ;; fresh connection and retry. We might also get> + ;; 'bad-response or a similar exception from (web response)> + ;; later on, once we've sent the request, or a> + ;; ERROR/INVALID-SESSION from GnuTLS.> + (if (or (and (eq? key 'system-error)> + (= EPIPE (system-error-errno `(,key ,@args))))> + (and (eq? key 'gnutls-error)> + (eq? (first args) error/invalid-session))> + (memq key> + '(bad-response> + bad-header> + bad-header-component)))> + (begin> + (close-port p)> + (list 'connect> + #f> + (drop requests processed)> + result))> + (apply throw key args))))> + (('connect . args)> + (apply connect args))> + (('loop . args)> + (apply loop args)))))))))
OK to write it this way as the first commit, to ease review.
What about the approach below:
Toggle diff (103 lines)diff --git a/guix/http-client.scm b/guix/http-client.scmindex 4b4c14ed0b..6351e2d051 100644--- a/guix/http-client.scm+++ b/guix/http-client.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2015 Mark H Weaver <mhw@netris.org> ;;; Copyright © 2012, 2015 Free Software Foundation, Inc. ;;; Copyright © 2017 Tobias Geerinckx-Rice <me@tobias.gr>@@ -147,6 +147,27 @@ Raise an '&http-get-error' condition if downloading fails." (uri->string uri) code (response-reason-phrase resp)))))))))))) +(define-syntax-rule (false-if-networking-error exp)+ "Return #f if EXP triggers a network related exception."+ ;; FIXME: Duplicated from 'with-cached-connection'.+ (catch #t+ (lambda ()+ exp)+ (lambda (key . args)+ ;; If PORT was cached and the server closed the connection in the+ ;; meantime, we get EPIPE. In that case, open a fresh connection and+ ;; retry. We might also get 'bad-response or a similar exception from+ ;; (web response) later on, once we've sent the request, or a+ ;; ERROR/INVALID-SESSION from GnuTLS.+ (if (or (and (eq? key 'system-error)+ (= EPIPE (system-error-errno `(,key ,@args))))+ (and (eq? key 'gnutls-error)+ (eq? (first args) error/invalid-session))+ (memq key+ '(bad-response bad-header bad-header-component)))+ #f+ (apply throw key args)))))+ (define* (http-multiple-get base-uri proc seed requests #:key port (verify-certificate? #t) (open-connection guix:open-connection-for-uri)@@ -219,42 +240,27 @@ returning." (remainder (connect p remainder result)))) ((head tail ...)- (catch #t- (lambda ()- (let* ((resp (read-response p))- (body (response-body-port resp))- (result (proc head resp body result)))- ;; The server can choose to stop responding at any time,- ;; in which case we have to try again. Check whether- ;; that is the case. Note that even upon "Connection:- ;; close", we can read from BODY.- (match (assq 'connection (response-headers resp))- (('connection 'close)- (close-port p)- (connect #f ;try again- (drop requests (+ 1 processed))- result))- (_- (loop tail (+ 1 processed) result))))) ;keep going- (lambda (key . args)- ;; If PORT was cached and the server closed the connection- ;; in the meantime, we get EPIPE. In that case, open a- ;; fresh connection and retry. We might also get- ;; 'bad-response or a similar exception from (web response)- ;; later on, once we've sent the request, or a- ;; ERROR/INVALID-SESSION from GnuTLS.- (if (or (and (eq? key 'system-error)- (= EPIPE (system-error-errno `(,key ,@args))))- (and (eq? key 'gnutls-error)- (eq? (first args) error/invalid-session))- (memq key- '(bad-response bad-header bad-header-component)))- (begin- (close-port p)- (connect #f ; try again- (drop requests (+ 1 processed))- result))- (apply throw key args))))))))))+ (match (false-if-networking-error (read-response p))+ ((? response? resp)+ (let* ((body (response-body-port resp))+ (result (proc head resp body result)))+ ;; The server can choose to stop responding at any time,+ ;; in which case we have to try again. Check whether+ ;; that is the case. Note that even upon "Connection:+ ;; close", we can read from BODY.+ (match (assq 'connection (response-headers resp))+ (('connection 'close)+ (close-port p)+ (connect #f ;try again+ (drop requests (+ 1 processed))+ result))+ (_+ (loop tail (+ 1 processed) result)))))+ (#f+ (close-port p)+ (connect #f ; try again+ (drop requests (+ 1 processed))+ result))))))))) ;;;
I believe it’s a bit more readable because it moves ‘catch’ out of sightand avoids the sort of “mini DSL” where we return lists of arguments.
WDYT?
Thanks,Ludo’.
C
C
Christopher Baines wrote on 26 Mar 09:39 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47288@debbugs.gnu.org)
87h7ky9ulu.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (86 lines)> What about the approach below:>> diff --git a/guix/http-client.scm b/guix/http-client.scm> index 4b4c14ed0b..6351e2d051 100644> --- a/guix/http-client.scm> +++ b/guix/http-client.scm> @@ -1,5 +1,5 @@> ;;; GNU Guix --- Functional package management for GNU> -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2020 Ludovic Courtès <ludo@gnu.org>> +;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2020, 2021 Ludovic Courtès <ludo@gnu.org>> ;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>> ;;; Copyright © 2012, 2015 Free Software Foundation, Inc.> ;;; Copyright © 2017 Tobias Geerinckx-Rice <me@tobias.gr>> @@ -147,6 +147,27 @@ Raise an '&http-get-error' condition if downloading fails."> (uri->string uri) code> (response-reason-phrase resp))))))))))))> > +(define-syntax-rule (false-if-networking-error exp)> + "Return #f if EXP triggers a network related exception."> + ;; FIXME: Duplicated from 'with-cached-connection'.> + (catch #t> + (lambda ()> + exp)> + (lambda (key . args)> + ;; If PORT was cached and the server closed the connection in the> + ;; meantime, we get EPIPE. In that case, open a fresh connection and> + ;; retry. We might also get 'bad-response or a similar exception from> + ;; (web response) later on, once we've sent the request, or a> + ;; ERROR/INVALID-SESSION from GnuTLS.> + (if (or (and (eq? key 'system-error)> + (= EPIPE (system-error-errno `(,key ,@args))))> + (and (eq? key 'gnutls-error)> + (eq? (first args) error/invalid-session))> + (memq key> + '(bad-response bad-header bad-header-component)))> + #f> + (apply throw key args)))))> +> (define* (http-multiple-get base-uri proc seed requests> #:key port (verify-certificate? #t)> (open-connection guix:open-connection-for-uri)> @@ -219,42 +240,27 @@ returning."> (remainder> (connect p remainder result))))> ((head tail ...)> - (catch #t> - (lambda ()> - (let* ((resp (read-response p))> - (body (response-body-port resp))> - (result (proc head resp body result)))> - ;; The server can choose to stop responding at any time,> - ;; in which case we have to try again. Check whether> - ;; that is the case. Note that even upon "Connection:> - ;; close", we can read from BODY.> - (match (assq 'connection (response-headers resp))> - (('connection 'close)> - (close-port p)> - (connect #f ;try again> - (drop requests (+ 1 processed))> - result))> - (_> - (loop tail (+ 1 processed) result))))) ;keep going> - (lambda (key . args)> - ;; If PORT was cached and the server closed the connection> - ;; in the meantime, we get EPIPE. In that case, open a> - ;; fresh connection and retry. We might also get> - ;; 'bad-response or a similar exception from (web response)> - ;; later on, once we've sent the request, or a> - ;; ERROR/INVALID-SESSION from GnuTLS.> - (if (or (and (eq? key 'system-error)> - (= EPIPE (system-error-errno `(,key ,@args))))> - (and (eq? key 'gnutls-error)> - (eq? (first args) error/invalid-session))> - (memq key> - '(bad-response bad-header bad-header-component)))> - (begin> - (close-port p)> - (connect #f ; try again> - (drop requests (+ 1 processed))> - result))> - (apply throw key args))))))))))> + (match (false-if-networking-error (read-response p))> + ((? response? resp)> + (let* ((body (response-body-port resp))> + (result (proc head resp body result)))
Given body is a port, and that port is passed to proc, I'm guessing it'spossible for networking things to go wrong inside proc.
Toggle quote (17 lines)> + ;; The server can choose to stop responding at any time,> + ;; in which case we have to try again. Check whether> + ;; that is the case. Note that even upon "Connection:> + ;; close", we can read from BODY.> + (match (assq 'connection (response-headers resp))> + (('connection 'close)> + (close-port p)> + (connect #f ;try again> + (drop requests (+ 1 processed))> + result))> + (_> + (loop tail (+ 1 processed) result)))))> + (#f> + (close-port p)> + (connect #f ; try again> + (drop requests (+ 1 processed))
I realised earlier in this series of patches that this should actuallybe processed, rather than (+ 1 processed) since proc can't have been runfor the current response.
Toggle quote (10 lines)> + result)))))))))> > > ;;;>> I believe it’s a bit more readable because it moves ‘catch’ out of sight> and avoids the sort of “mini DSL” where we return lists of arguments.>> WDYT?
It looks OK to me, I think the only thing to figure out for sure iswhether it's safe to not include the activity on the body port in theerror handling.
Thanks,
Chris
-----BEGIN PGP SIGNATURE-----
iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBdnc1fFIAAAAAALgAoaXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNFODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2JhaW5lcy5uZXQACgkQXiijOwuE9XcABBAAl7UOhAkQGxDeOVP3ZAUoECJ2tb5FFhwBa1RCXVWCVWlFnOnikiA0h84zI9bM8yVDPuctdpKCEvUZRTUoaGDxSzK0GkC4i0C0MpCXSo3uX+cFGjgaZ38WWUX8WbRizpdnnuE+VCAByzsVvmge6LyZqcn0w6GeGno8TGYfXVsworVAxxjQi1VA7ZyEqfiBvCCJ9YkZwnfmGmrb6Qu1qSDT0z4GGrPWgEwWkkGtXxno0H1D1PKkwt9YrvqpiMXx1lHW7W6QGmQnuBK/kuIGhMq7fCgS50COxvAUpKOqOheOuBtiDC6eAYBG7tOlvwUNX4dTW2212wsEENx1qUqGl/imDu5AlLEDE5bteZDBaxuOEYS9bPw/oUHeGgS5ys2ds5Enp61PSmQef/1nthPGrhEaxBBV6Yt4+zx9Y8oRZnviA9bDGvcybI4UGidicAoAcP2Z4ri8HLS46aynEVzZlfATMiuZeihqnmixTXoGmY9+cqpJMyvaYIGuJye2AoRhkQ0BB7Ua9UqfUCuVeihSrWSa4KGWVMeqbvSJM1r3O1xvaThV1ckN3+umTxfOtKb7/H8qPCEYQ7Srfmdg0wDcn/wvTsdM0iIebICRXXK/1mQyFARWUt2MPTrgOFbciDc06hNLnlcvQbHe63HIfw1zF7cBN+36g0IF0ty6r8AwmsnO/oE==ihh+-----END PGP SIGNATURE-----
L
L
Ludovic Courtès wrote on 27 Mar 18:15 +0100
(name . Christopher Baines)(address . mail@cbaines.net)(address . 47288@debbugs.gnu.org)
87mtuozfep.fsf_-_@gnu.org
Hi,
Christopher Baines <mail@cbaines.net> skribis:
Toggle quote (2 lines)> Ludovic Courtès <ludo@gnu.org> writes:
[...]
Toggle quote (8 lines)>> + (match (false-if-networking-error (read-response p))>> + ((? response? resp)>> + (let* ((body (response-body-port resp))>> + (result (proc head resp body result)))>> Given body is a port, and that port is passed to proc, I'm guessing it's> possible for networking things to go wrong inside proc.
Yes, but how is this different from a regular read(2) call as made by‘get-bytevector-n’ or whatever? We wouldn’t write every read(2) call in‘catch’ because in general any error there is indeed exceptional.
I think the only bit that’s “less exceptional” here is that, becausewe’re reusing cached connection, we know that the first read(2) or thefirst write(2) to that port can trigger one of these errors—which, weknow are not “exceptional”. Errors in subsequent read(2) or write(2)calls remain exceptional/unrecoverable and should be treated as suchIMO.
Does that make sense?
(In that sense, I think wrapping every ‘read-response’ call rather thanjust the first one is already too much, but that’s okay.)
Toggle quote (21 lines)>> + ;; The server can choose to stop responding at any time,>> + ;; in which case we have to try again. Check whether>> + ;; that is the case. Note that even upon "Connection:>> + ;; close", we can read from BODY.>> + (match (assq 'connection (response-headers resp))>> + (('connection 'close)>> + (close-port p)>> + (connect #f ;try again>> + (drop requests (+ 1 processed))>> + result))>> + (_>> + (loop tail (+ 1 processed) result)))))>> + (#f>> + (close-port p)>> + (connect #f ; try again>> + (drop requests (+ 1 processed))>> I realised earlier in this series of patches that this should actually> be processed, rather than (+ 1 processed) since proc can't have been run> for the current response.
Oh, something to fix in a subsequent commit, then.
All in all, I propose to go with this patch if that’s fine with you.
Thanks!
Ludo’.
L
L
Ludovic Courtès wrote on 27 Mar 22:58 +0100
(name . Christopher Baines)(address . mail@cbaines.net)
87mtuoxnqo.fsf_-_@gnu.org
Hi!
I went ahead and pushed 45fce38fb0b6c6796906149ade145b8d3594c1c6 alongthese lines.
‘guix weather’ runs to completion and things seem to work fine. Let meknow if you notice anything wrong.
Thank you for your work and for your patience on this issue!
Ludo’.
Closed
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 47288@debbugs.gnu.org