From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 19 11:04:45 2020 Received: (at 39530) by debbugs.gnu.org; 19 Feb 2020 16:04:45 +0000 Received: from localhost ([127.0.0.1]:43304 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j4Rq9-00056m-A4 for submit@debbugs.gnu.org; Wed, 19 Feb 2020 11:04:45 -0500 Received: from eggs.gnu.org ([209.51.188.92]:49312) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j4Rq6-00056X-Uw for 39530@debbugs.gnu.org; Wed, 19 Feb 2020 11:04:43 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40228) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1j4Rq1-0006KE-Bj; Wed, 19 Feb 2020 11:04:37 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=48422 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1j4Rpy-0007tb-9Z; Wed, 19 Feb 2020 11:04:35 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Julien Lepiller Subject: Re: [bug#39530] [PATCH] guix: Support partial download References: <20200209202358.17bb4a39@tachikoma.lepiller.eu> Date: Wed, 19 Feb 2020 17:04:32 +0100 In-Reply-To: <20200209202358.17bb4a39@tachikoma.lepiller.eu> (Julien Lepiller's message of "Sun, 9 Feb 2020 20:23:58 +0100") Message-ID: <874kvmmulr.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39530 Cc: 39530@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi, Julien Lepiller skribis: > First, I make sure that the guix daemon will not remove previously > failed attempts when trying to build something again, when that is a > fixed-output derivation. Then, I add a Range HTTP header when > performing an HTTP fetch; this ensures that we only query for the part > we don't already have, and append it to the target file. > > If a partial download fails, the same mirror/url is tried again, but > the partial file is removed first, ensuring we do a complete fetch this > time around. If that failed too, we try with the following url. If we > only perform a complete fetch, we proceed as usual. The next url will > be a partial fetch if there is already something locally. Nice! > However, with that daemon there was a lot of new builds required to run > guix environment guix as my user (and nothing was substituted, which > is weird), whereas with the system's daemon, there was nothing to > build. Maybe there's something fishy in that patch... Hmm, that sounds really weird. Could you clarify what you did? >>From 332793b7f29ea68ac9a1af22e3d1c4745200da7e Mon Sep 17 00:00:00 2001 > From: Julien Lepiller > Date: Sun, 9 Feb 2020 19:47:27 +0100 > Subject: [PATCH] guix: download: Add partial download support. Nitpick: you can remove =E2=80=9Cguix:=E2=80=9D from the subject. > * nix/libstore/build.cc (tryToBuild): Do not remove invalid fixed-output > derivations. > * guix/build/download.scm (http-fetch): Add a range argument. > (url-fetch): Performa partial download if a file already exists. [...] > -(define* (http-fetch uri #:key timeout (verify-certificate? #t)) > +(define* (http-fetch uri #:key timeout (verify-certificate? #t) range) > "Return an input port containing the data at URI, and the expected num= ber of > bytes available or #f. When TIMEOUT is true, bail out if the connection= could > not be established in less than TIMEOUT seconds. When VERIFY-CERTIFICAT= E? is > -true, verify HTTPS certificates; otherwise simply ignore them." > +true, verify HTTPS certificates; otherwise simply ignore them. When RAN= GE is > +a number, it is the number of bytes we want to skip from the data at URI; > +otherwise the full document is requested." I=E2=80=99d suggest to rename #:range to #:offset because it denotes the st= art offset. What response do we get if the server doesn=E2=80=99t support =E2=80=9CRang= e=E2=80=9D? Can servers silently ignore =E2=80=9CRange=E2=80=9D? > + (if (file-exists? file) > + (http-fetch uri > + #:verify-certificate? verify-certifi= cate? > + #:timeout timeout > + #:range (stat:size (stat file))) > + (http-fetch uri > + #:verify-certificate? verify-certifi= cate? > + #:timeout timeout)))) I=E2=80=99d remove the =E2=80=98if=E2=80=99: (http-fetch =E2=80=A6 #:offset (and=3D> (stat file #f) stat:size)) > --- a/nix/libstore/build.cc > +++ b/nix/libstore/build.cc > @@ -1320,6 +1320,7 @@ void DerivationGoal::tryToBuild() > Path path =3D i->second.path; > if (worker.store.isValidPath(path)) continue; > if (!pathExists(path)) continue; > + if (fixedOutput) continue; Please add a comment above explaining why fixed outputs are not deleted. Also please: not tabs. :-) Thanks! Ludo=E2=80=99.