From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 23 13:27:03 2021 Received: (at 50384) by debbugs.gnu.org; 23 Sep 2021 17:27:03 +0000 Received: from localhost ([127.0.0.1]:55771 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mTSUw-0008HM-Pm for submit@debbugs.gnu.org; Thu, 23 Sep 2021 13:27:03 -0400 Received: from laurent.telenet-ops.be ([195.130.137.89]:38974) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mTSUu-0008Gr-DR for 50384@debbugs.gnu.org; Thu, 23 Sep 2021 13:27:01 -0400 Received: from ptr-bvsjgyjmffd7q9timvx.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:aaf1:9810:a0b8:a55d]) by laurent.telenet-ops.be with bizsmtp id xVSy250040mfAB401VSyEa; Thu, 23 Sep 2021 19:26:58 +0200 Message-ID: <3563a99b37c8ef3b226b7fef25a5afa273c08b6e.camel@telenet.be> Subject: Re: [PATCH v4] Optimise search-patch (reducing I/O) From: Maxime Devos To: Ludovic =?ISO-8859-1?Q?Court=E8s?= Date: Thu, 23 Sep 2021 19:26:51 +0200 In-Reply-To: <87r1dhj2bk.fsf_-_@gnu.org> References: <8900fa8c8eef7f72fc97adc2408be26c88de7803.camel@telenet.be> <87tuj0xaja.fsf@gnu.org> <0ec7f0270fcccec730808f9210f074cd5339961f.camel@telenet.be> <87ee9xerac.fsf_-_@gnu.org> <87r1dhj2bk.fsf_-_@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-STReeQE9TXFr506Wlmil" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1632418018; bh=R+Shn0SMHkGBfThsKyNabrVQpWZuQE4iWaxtQkzwny0=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=iRqYRboQW1s5928GcGx6dStkwFnDPiOUemXzCwJMvCQrN0QsJ2khqvZvk16eRkqW8 pZ1Ntnm7B6YHsUzLBdk0kBG+RWuBoQsuYn6TnKybn3hX3RjssgUCh06MsH7pYzZdz3 FP/WB3QnZo72MyRRFW+pLpDEsk5TaUiWHuZtLjHcM1PFbGwJPKTH4MwaMM/7DQIwC1 Ab1UTe1xMmq27sOWjk8dlCUEb4jRny8c3Ra9P6PfVEaYNziYDbE2eztgGnjDn3sfnK uv5jUbKQclbSmD8NsGvbxx7GehATu62nLby7tQDUhFGbNwFYfc6I4oMYoNDsCJSjs4 zwWJfaRmWLJZQ== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 50384 Cc: 50384@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-STReeQE9TXFr506Wlmil Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op di 21-09-2021 om 18:55 [+0200]: > Hi! >=20 > I took the liberty to reopen this patch because there were good ideas > IMO. I=E2=80=99m sorry if my many questions and lack of responsiveness c= ame out > as a suggestion that this approach wasn=E2=80=99t good. I reordered your mail a little. > Looking at the big picture, what I=E2=80=99d like to have is a package > derivation cache designed in such a way that =E2=80=9Cguix install foo=E2= =80=9D wouldn=E2=80=99t > even need to load any package module on a cache hit. That=E2=80=99d make= a > noticeable difference performance-wise, that=E2=80=99s another level of > complexity=E2=80=A6 (I have a rough design in mind that we could discuss= .) This =E2=80=98package derivation cache=E2=80=99 seems an interesting idea t= o pursue, though I wonder what could be used as =E2=80=98keys=E2=80=99 in the derivat= ion cache. Package names aren't sufficient because packages can have multiple versions= , package names + package versions aren't sufficient because packages can hav= e multiple variants. Grafts might need some care. Having multiple versions = of guix can be addressed by including the commits of every channel in the key. Even if =E2=80=98foo=E2=80=99 isn't in the cache, the cache can still be us= eful if the inputs =E2=80=98bar=E2=80=99 and =E2=80=98baz=E2=80=99 of foo are in the ca= che. > Ludovic Court=C3=A8s skribis: >=20 > > > +;; repeated 'stat' calls. Allow computing the hash of the file in a= dvance, > > > +;; to avoid having to send the file to the daemon when it is already= interned > > > +;; in the store. > > > (define-record-type > > > - (%%local-file file absolute name recursive? select?) > > > + (%%local-file file absolute name sha256 recursive? select?) > > > local-file? > > > (file local-file-file) ;string > > > (absolute %local-file-absolute-file-name) ;promise string > > > (name local-file-name) ;string > > > + (sha256 local-file-sha256) ;sha256 bytevector= | #f > >=20 > > Could we store the result of =E2=80=98fixed-output-path=E2=80=99 rather= than the SHA256, > > while we=E2=80=99re at it? Embedding the result of =E2=80=98fixed-output-path=E2=80=99 in the .go migh= t be problematic from a closure size perspective, as that would create additional references= in the store items of guix. > I tried that with the patch below, roughly taking the same approach as > your patch series, but somewhat simplified, mostly so I could > experiment. [...] >=20 > We can estimate the performance of that strategy by commenting out the > =E2=80=98add-temp-root*=E2=80=99 call (thus getting a single RPC) in > =E2=80=98local-file-compiler=E2=80=99: this time it=E2=80=99s slightly fa= ster, but we=E2=80=99re in the > 1% range on the wall-clock time of =E2=80=98guix build pigx -d --no-graft= s=E2=80=99: > [...] > Perhaps the gains would be a bit higher if we change all the package > files to use =E2=80=98local-patches=E2=80=99, but we probably can=E2=80= =99t expect a lot more > anyway since that process is CPU-bound. >=20 > So I don=E2=80=99t know. It feels like a worthy optimization, and one th= at=E2=80=99s > manageable from a maintenance viewpoint, but it buys us very little. >=20 > Thoughts? As it is only <1%, I would prefer trying the =E2=80=98package derivation ca= che=E2=80=99 first, as it seems to have more potential. > Ludo=E2=80=99. >=20 --=-STReeQE9TXFr506Wlmil Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYUy42xccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7o0YAP92vmojOgaiI7vtCjLc3x/akVtC Gx8kiPOKLtr/2bgDIQEAt7YyiFMiVPXJwFRjL4D80kJ4sR/P7yISSYev/7IUjwA= =26Wc -----END PGP SIGNATURE----- --=-STReeQE9TXFr506Wlmil--