From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 14 08:17:44 2019 Received: (at 33899) by debbugs.gnu.org; 14 Jan 2019 13:17:44 +0000 Received: from localhost ([127.0.0.1]:58369 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gj27a-0003gz-RH for submit@debbugs.gnu.org; Mon, 14 Jan 2019 08:17:44 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:58392) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gj27Y-0003gp-BJ for 33899@debbugs.gnu.org; Mon, 14 Jan 2019 08:17:41 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id A4FB7D1F; Mon, 14 Jan 2019 14:17:38 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fiz-E9TzlM6x; Mon, 14 Jan 2019 14:17:37 +0100 (CET) Received: from ribbon (unknown [IPv6:2001:660:6102:320:e120:2c8f:8909:cdfe]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 9A1E2AFB; Mon, 14 Jan 2019 14:17:37 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Hector Sanjuan Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS References: <20181228231205.8068-1-ludo@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 25 =?utf-8?Q?Niv=C3=B4se?= an 227 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 14 Jan 2019 14:17:36 +0100 In-Reply-To: (Hector Sanjuan's message of "Mon, 07 Jan 2019 14:43:56 +0000") Message-ID: <87r2dfv0nj.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 33899 Cc: "go-ipfs-wg@ipfs.io" , Pierre Neidhardt , "33899@debbugs.gnu.org" <33899@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi Hector, Happy new year to you too! :-) Hector Sanjuan skribis: > 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs > has a read/write API (on :5001) and a read-only API that we call "gateway" > and which runs on :8080. The gateway, apart from handling most of the > read-only methods from the HTTP API, also handles paths like "/ipfs/" > or "/ipns/" gracefully, and returns an autogenerated webpage for > directory-type CIDs. The gateway does not allow to "publish". Therefore I= think > the doc strings should say "IPFS daemon API" rather than "GATEWAY". Indeed, I=E2=80=99ll change that. > 2) I'm not proficient enough in schema to grasp the details of the > "directory" format. If I understand it right, you keep a separate manifest > object listing the directory structure, the contents and the executable b= it > for each. Thus, when adding a store item you add all the files separately= and > this manifest. And when retrieving a store item you fetch the manifest and > reconstruct the tree by fetching the contents in it (and applying the > executable flag). Is this correct? This works, but it can be improved: That=E2=80=99s correct. > You can add all the files/folders in a single request. If I'm > reading it right, now each files is added separately (and gets pinned > separately). It would probably make sense to add it all in a single reque= st, > letting IPFS to store the directory structure as "unixfs". You can > additionally add the sexp file with the dir-structure and executable flags > as an extra file to the root folder. This would allow to fetch the whole = thing > with a single request too /api/v0/get?arg=3D. And to pin a single h= ash > recursively (and not each separately). After getting the whole thing, you > will need to chmod +x things accordingly. Yes, I=E2=80=99m well aware of =E2=80=9Cunixfs=E2=80=9D. The problems, as = I see it, is that it stores =E2=80=9Ctoo much=E2=80=9D in a way (we don=E2=80=99t need to store = the mtimes or permissions; we could ignore them upon reconstruction though), and =E2=80= =9Cnot enough=E2=80=9D in another way (the executable bit is lost, IIUC.) > It will probably need some trial an error to get the multi-part right > to upload all in a single request. The Go code HTTP Clients doing > this can be found at: > > https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96 > > As you see, a directory part in the multipart will have the content-type = Header > set to "application/x-directory". The best way to see how "abspath" etc i= s set > is probably to sniff an `ipfs add -r ` operation (localhost:5= 001). > > Once UnixFSv2 lands, you will be in a position to just drop the sexp file > altogether. Yes, that makes sense. In the meantime, I guess we have to keep using our own format. What are the performance implications of adding and retrieving files one by one like I did? I understand we=E2=80=99re doing N HTTP requests to the local IPFS daemon where =E2=80=9Cipfs add -r=E2=80=9D makes a single reques= t, but this alone can=E2=80=99t be much of a problem since communication is happening locally. Does pinning each file separately somehow incur additional overhead? Thanks for your feedback! Ludo=E2=80=99.