From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 23 02:20:27 2022 Received: (at 57978) by debbugs.gnu.org; 23 Sep 2022 06:20:27 +0000 Received: from localhost ([127.0.0.1]:39158 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1obc30-0007H7-Fv for submit@debbugs.gnu.org; Fri, 23 Sep 2022 02:20:27 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49968) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1obc2y-0007Gi-64; Fri, 23 Sep 2022 02:20:24 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49284) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1obc2q-0007uy-5K; Fri, 23 Sep 2022 02:20:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=v8zk/MAZ5CmI+AWHnZJ9I77Ii2NvFYgBrf8bJ9F0cik=; b=q4Ut3ZolX4f1wa VAy+0BKigkFKNSD8nkjTppza1q7UGU8m2FymhjgmBq7lLwWLjjxnOsVCQ0gGKurNLiasaVaP8SLo7 eiIdoCf7F4fs5MrgfPnHobpNo1NJZUMQpw+Yn+cNXSA4GB3m7b24pyNVU3VkCdRV34MBUUXqkGqKe 6D5bxoqqVkfsguyuitdjTACqnXUUrqIHk+AwtHdJD/CUpl24PSzyEtmR6FtMcpjBRaSTOFDazX8kM ZRnDukiED9/IOZFfNF0c86gvWD2+kcNwzDd6sPY1J8bdF4QImMcGHIr0kJcvx4l2IWlwCxjb6/2/P 3J6xWL5ddlxYxs2DBOjQ==; Received: from [89.207.171.75] (port=38064 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1obc2o-0007Hi-Uw; Fri, 23 Sep 2022 02:20:15 -0400 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 58017@debbugs.gnu.org Subject: [PATCH 1/2] substitute: Split nar download. Date: Fri, 23 Sep 2022 08:19:56 +0200 Message-Id: <20220923061957.5658-1-ludo@gnu.org> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/scripts/substitute.scm (download-nar): New procedure, with most of the code moved from... (process-substitution): ... here. Call it. --- guix/scripts/substitute.scm | 52 +++++++++++++++++++++++ [...] Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [209.51.188.92 listed in list.dnswl.org] 3.6 RCVD_IN_SBL_CSS RBL: Received via a relay in Spamhaus SBL-CSS [89.207.171.75 listed in zen.spamhaus.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record X-Debbugs-Envelope-To: 57978 Cc: Attila Lendvai , 57978@debbugs.gnu.org, =?UTF-8?q?Ludovic=20Court=C3=A8s?= , zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) * guix/scripts/substitute.scm (download-nar): New procedure, with most of the code moved from... (process-substitution): ... here. Call it. --- guix/scripts/substitute.scm | 52 +++++++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 20 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index cdf591ac4d..e3b382d0d8 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -437,20 +437,13 @@ (define-syntax-rule (with-cached-connection uri port exp ...) "Bind PORT with EXP... to a socket connected to URI." (call-with-cached-connection uri (lambda (port) exp ...))) -(define* (process-substitution port store-item destination - #:key cache-urls acl - deduplicate? print-build-trace?) - "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to -DESTINATION as a nar file. Verify the substitute against ACL, and verify its -hash against what appears in the narinfo. When DEDUPLICATE? is true, and if -DESTINATION is in the store, deduplicate its files. Print a status line to -PORT." - (define narinfo - (lookup-narinfo cache-urls store-item - (if (%allow-unauthenticated-substitutes?) - (const #t) - (cut valid-narinfo? <> acl)))) - +(define* (download-nar narinfo destination + #:key status-port + deduplicate? print-build-trace?) + "Download the nar prescribed in NARINFO, which is assumed to be authentic +and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and +if DESTINATION is in the store, deduplicate its files. Print a status line to +STATUS-PORT." (define destination-in-store? (string-prefix? (string-append (%store-prefix) "/") destination)) @@ -490,10 +483,6 @@ (define (fetch uri) (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) - (unless narinfo - (leave (G_ "no valid substitute for '~a'~%") - store-item)) - (let ((uri compression file-size (narinfo-best-uri narinfo #:fast-decompression? @@ -575,14 +564,37 @@ (define cpu-usage (let ((actual (get-hash))) (if (bytevector=? actual expected) ;; Tell the daemon that we're done. - (format port "success ~a ~a~%" + (format status-port "success ~a ~a~%" (narinfo-hash narinfo) (narinfo-size narinfo)) ;; The actual data has a different hash than that in NARINFO. - (format port "hash-mismatch ~a ~a ~a~%" + (format status-port "hash-mismatch ~a ~a ~a~%" (hash-algorithm-name algorithm) (bytevector->nix-base32-string expected) (bytevector->nix-base32-string actual))))))) +(define* (process-substitution port store-item destination + #:key cache-urls acl + deduplicate? print-build-trace?) + "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to +DESTINATION as a nar file. Verify the substitute against ACL, and verify its +hash against what appears in the narinfo. When DEDUPLICATE? is true, and if +DESTINATION is in the store, deduplicate its files. Print a status line to +PORT." + (define narinfo + (lookup-narinfo cache-urls store-item + (if (%allow-unauthenticated-substitutes?) + (const #t) + (cut valid-narinfo? <> acl)))) + + (unless narinfo + (leave (G_ "no valid substitute for '~a'~%") + store-item)) + + (download-nar narinfo destination + #:status-port port + #:deduplicate? deduplicate? + #:print-build-trace? print-build-trace?)) + ;;; ;;; Entry point. -- 2.37.3