From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 30 06:46:50 2022 Received: (at 52555) by debbugs.gnu.org; 30 Jan 2022 11:46:50 +0000 Received: from localhost ([127.0.0.1]:35689 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nE8fS-0007PQ-13 for submit@debbugs.gnu.org; Sun, 30 Jan 2022 06:46:50 -0500 Received: from michel.telenet-ops.be ([195.130.137.88]:44058) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nE8fO-0007PC-6h for 52555@debbugs.gnu.org; Sun, 30 Jan 2022 06:46:48 -0500 Received: from [172.20.10.5] ([5.23.227.239]) by michel.telenet-ops.be with bizsmtp id ozmh260065AYamV06zmiXb; Sun, 30 Jan 2022 12:46:44 +0100 Message-ID: Subject: Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS From: Maxime Devos To: pukkamustard , 52555@debbugs.gnu.org Date: Sun, 30 Jan 2022 12:46:23 +0100 In-Reply-To: <20220125192201.7582-1-pukkamustard@posteo.net> References: <20211216161724.547-1-pukkamustard@posteo.net> <20220125192201.7582-1-pukkamustard@posteo.net> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-pmPqROV9QLMWjJsSQXtO" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1643543204; bh=/mEX7Ex4Zj3x4I4x3VUqJnt9SI1nWoiWC4yhuNVD5PM=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=lCV8q6vmLstF+lYAkYlg52n65UwLSkBIedOOkxrwZ4LfMoA46q8yBqjIPxL9uMQwX NC4iwQnajmo9Njk4SvA71T3MANvSacSdk5+z6Bd1AYdVZEUoEI8v1reqYXa2H7OS6E l7lxSDq9XAUfA2971kCfncXjjfMJ2u2YnSKptU5wARuq45WEEgurFpovAPhxf86wNH /V4SUPjFqjwHA37scY/ubWwqcK5EEiF75Bqmpcbdgdr/eaA9pRx2XcOvF5M00MoOP1 YDhqng6+MSIwCD1fU1xv+tySl1z1aLAZ3rGIfQ89adwmFK56bPdRBGiqmm/r1KXB6G eYcTFNYzJMSTA== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 52555 Cc: ~pukkamustard/eris@lists.sr.ht X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-pmPqROV9QLMWjJsSQXtO Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]: > I have only tested this for fairly small packages (up to a few MB). >=20 > One issue with IPFS might be that we have to create a new HTTP connection= to > the IPFS daemon for every single block (32KiB). The IPFS daemon does not= seem > to support HTTP connection re-use According to , the IPFS daemon supports connection reuse according to some people and doesn't according to other people. > and neither does the Guile (web client). Guix supports connection reuse, see 'call-with-cached-connection' in (guix scripts substitute). > I fear this might become a performance issue. IIUC, the performance problem primarily lies in the round-tripping between the client and the server. If the client and the server are on the same machine, then this round trip time is presumably small compared to, say, localhost contacting ci.guix.gnu.org. Still, connection reuse would be nice. > It seems possible to use IPFS more directly by exposing the Go code as a > C library and then using that with the Guile FFI [1]. This is however a b= it > complicated and adds a lot of dependencies. In particular, this should no= t become > a dependency of Guix itself. The performance of IPFS itself also needs to= be > evaluated, maybe the IPFS HTTP API will not be the bottle-neck. Security-wise, libipfs doesn't seem great: libipfs starts the IPFS daemon inside the process and guix/scripts/substitute.scm is run as root. > As mentioned in previous mail a simple HTTP transport for blocks would be= a > good fallback. This would allow users to get missing blocks (things that > somehow got dropped from IPFS) directly from a substitute server. [...] Seems a good idea to me -- DHTs can be unreliable. I presume this will be implemented with some kind of timeout: if no block is received within N seconds, fallback to HTTP? Also, don't forget to insert this missing block back into IPFS/GNUnet/BitTorrent/..., otherwise less and less blocks will be available until nothing is available anymore. > In any case, it would be necessary for the substitute server to store enc= oded > blocks of the NAR. For this I think it makes sense to use a small databas= e. We > have bindings to use ERIS with GDBM [2]. It might also make sense to use > SQLite, especially if there are other use-cases for such a database. Wouldn't this be a huge database? IIRC, according to logs.guix.gnu.org the size of the nars of the substitute servers are somewhere in the 200G-2T range or something like that. To reduce the size of the database, perhaps you could let the database be a mapping from block ids to the name of the nar + the position in the nar, and encode the block on-demand? The database doesn't seem necessary, the substitute server could have some end-point /publish-this-nar-again-into-IPFS/name-of-the-nar which, when contacted, inserts the nar again into IPFS. Then when a block was unavailable, the client contacts this end-point and retries. Greetings, Maxime. --=-pmPqROV9QLMWjJsSQXtO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfZ6jxccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rANAP9328I61zxzX3OGMvtg/yulC2fN hA2FYGhhmzeZUgJfWAD+Nz3A9TmanG1T0/QvW6bgstsy7kILYqCvXo7xF8onkAM= =psyK -----END PGP SIGNATURE----- --=-pmPqROV9QLMWjJsSQXtO--