From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 05 16:59:08 2022 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 21:59:08 +0000 Received: from localhost ([127.0.0.1]:49233 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcQe-00025e-9O for submit@debbugs.gnu.org; Sat, 05 Mar 2022 16:59:08 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40164) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcQc-00025P-VJ for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 16:59:07 -0500 Received: from [2001:470:142:3::e] (port=36538 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcQW-000247-DE; Sat, 05 Mar 2022 16:59:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=GDMwIVgxPyksL0XlUsK8J0fS8EFz1dD5co4Un69j4uo=; b=Z0BYC784Oq+V0Wfxq/5o DYGb92tnb3Cbkd8zKGiRVkYRwRmBu74ELSVMm63PQ0vsmH917uLxMY5s/dTSO3pwXxi6iOjLbSvKj n4q6UCgrkIbjUG67qdU2LoemwHDiPEHywTLiBWYBfkSqmG5aDTGpjULmV6EzDZeKI2Q/fbGzbZRyq Wx/fx0Ajx17rkf7C5rwxzkClgiYzfiUVfIfTJi35+ddRmWKezG3g9Z9ODESfd5v2QDnHroziUXxoZ NsgN2qsO9QDuErHXOqCnG9ztAVqExhmevV+B1OYSf5egw3sMowFJ1xUWgbJCtCAmxVSozEIBDLwm3 pWFolq/LRKjqrQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50112 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcQV-0000hx-EL; Sat, 05 Mar 2022 16:59:00 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxime Devos Subject: Re: bug#54241: [PATCH 0/4] 'github' importer gracefully handles rate limiting References: <20220303211326.19884-1-ludo@gnu.org> <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> <87tucdqwen.fsf_-_@gnu.org> <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> Date: Sat, 05 Mar 2022 22:58:56 +0100 In-Reply-To: <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> (Maxime Devos's message of "Sat, 05 Mar 2022 10:44:48 +0100") Message-ID: <87a6e4njrz.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 54241 Cc: 54241@debbugs.gnu.org, Nicolas Goaziou X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op vr 04-03-2022 om 21:45 [+0100]: >> My concern is that we=E2=80=99d end up caching one or two little files in >> ~/.cache for each candidate package, and (rate limit aside) the overhead >> of dealing with the cache might outweigh the benefits.=C2=A0 I=E2=80=99d= rather use >> =E2=80=98http-fetch/cached=E2=80=99 for bigger files, like in (guix cve). >>=20 >> WDYT? > > If the overhead of caching little files is a concern, then perhaps a > SQLite (or GDBM) database could be used instead of the filesystem-based > cache? The number of packages in Guix was about 150 000 IIRC, if we > assume something around the magnitude of 200 bytes per package, then > we end up with about 29 MiB for the entirity of Guix. And there might > be some opportunities for compression, reducing this number. I think this would be going overboard in terms of complexity :-), and it wouldn=E2=80=99t radically change the run-time overhead (you still potentia= lly have to do an HTTP round trip with =E2=80=98If-Modified-Since=E2=80=99, you= =E2=80=99re just saving a few hundred bytes on the response in the best case.) > Something like this could be left for later though. Yup! Ludo=E2=80=99.