From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 05 04:44:52 2022 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 09:44:52 +0000 Received: from localhost ([127.0.0.1]:47304 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQQy4-0001cC-Cv for submit@debbugs.gnu.org; Sat, 05 Mar 2022 04:44:52 -0500 Received: from albert.telenet-ops.be ([195.130.137.90]:52322) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQQy2-0001c4-SF for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 04:44:51 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by albert.telenet-ops.be with bizsmtp id 2Zko2700U4UW6Th06ZkoPS; Sat, 05 Mar 2022 10:44:49 +0100 Message-ID: <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> Subject: Re: bug#54241: [PATCH 0/4] 'github' importer gracefully handles rate limiting From: Maxime Devos To: Ludovic =?ISO-8859-1?Q?Court=E8s?= Date: Sat, 05 Mar 2022 10:44:48 +0100 In-Reply-To: <87tucdqwen.fsf_-_@gnu.org> References: <20220303211326.19884-1-ludo@gnu.org> <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> <87tucdqwen.fsf_-_@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-tlTNn3P+kAqf4bC74gxw" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646473489; bh=GzRpajpEHHcqqsKMOdnfCcGmlyied3+zJ5NUA2Yrfn4=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=XLESYN8VGvIvk8pQNwo0GXnIAl7t+MARUam5SIB5KIDWo3Klu4QNFsQH3vfxWYF/e z4DSmeSf4tA5OHhB6W1iqHPdWXkIgAQ/g+yiEc/7l2zdQ9x8fR/1ch/K8ZJ1x5itK8 h5REGLWT7wXCNrbgtKSDd8QqSV8PHYGS4FUjSoAPU0YwbNUwE08a+i15HW7Tua4pDl thpNBADIRwhcwQloX4ZFjVzJe5mGrIZKDKiI8Wyu2mhr70M72TpBvFe64e+FG4fNcn ySPQtBcAnGcYHc9NZFmiBfx1bHlxQ1r3Wo2mrjjP9iGXqwkbIJA3Pa9NYsh1+hVGI+ KcddTMPPOBQyw== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 54241 Cc: 54241@debbugs.gnu.org, Nicolas Goaziou X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-tlTNn3P+kAqf4bC74gxw Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op vr 04-03-2022 om 21:45 [+0100]: > My concern is that we=E2=80=99d end up caching one or two little files in > ~/.cache for each candidate package, and (rate limit aside) the overhead > of dealing with the cache might outweigh the benefits.=C2=A0 I=E2=80=99d = rather use > =E2=80=98http-fetch/cached=E2=80=99 for bigger files, like in (guix cve). >=20 > WDYT? If the overhead of caching little files is a concern, then perhaps a SQLite (or GDBM) database could be used instead of the filesystem-based cache? The number of packages in Guix was about 150 000 IIRC, if we assume something around the magnitude of 200 bytes per package, then we end up with about 29 MiB for the entirity of Guix. And there might be some opportunities for compression, reducing this number. Something like this could be left for later though. Greetings, Maxime. --=-tlTNn3P+kAqf4bC74gxw Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiMxEBccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rlrAQCEUL9l/X7qI4RTPnAN4QTORipl HbeWWKeUBjBFdritsAEAx5BXSS1xnYcyioOiI845303n1EX1gat09Nm7oVWT2QY= =KduL -----END PGP SIGNATURE----- --=-tlTNn3P+kAqf4bC74gxw--