From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 14 18:32:03 2019 Received: (at 33899) by debbugs.gnu.org; 14 Jul 2019 22:32:03 +0000 Received: from localhost ([127.0.0.1]:46174 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hmn2I-0007qp-GE for submit@debbugs.gnu.org; Sun, 14 Jul 2019 18:32:03 -0400 Received: from mail1.protonmail.ch ([185.70.40.18]:47974) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hmn2E-0007jh-W4 for 33899@debbugs.gnu.org; Sun, 14 Jul 2019 18:32:00 -0400 Date: Sun, 14 Jul 2019 22:31:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hector.link; s=protonmail; t=1563143510; bh=A1ca9BbPvEBnUQhGhk34WkOtYemz98SjsNpEYJUXbZk=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References: Feedback-ID:From; b=m29RboLIL4fAf+SslxgfY4xuNdk/Zig/YTJ4VGq3Z5KIZOEBBYS/iSYo5/w5WO1E3 Z9ZUk57jK2C/+D4fBCR+/inlRzz+Dw13Kv3ZRRxQg/giSQrWAhTynXnP3gTZg+bEXP tTJ8iZ+YMz1DiKKomfydSB35KRVLTEjaKvohRxbA= To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= From: Hector Sanjuan Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS Message-ID: In-Reply-To: <87ftnbf1rt.fsf@gnu.org> References: <20181228231205.8068-1-ludo@gnu.org> <87r2dfv0nj.fsf@gnu.org> <8736pqthqm.fsf@gnu.org> <87zhlxe8t9.fsf@ambrevar.xyz> <87ftnbf1rt.fsf@gnu.org> Feedback-ID: omu4N36cG9Zn6VL5xT9it_nicCrTOu6Y4hug0yDo4Tl9cKTzl2pTjvygerP9BSgqCuiI9HYyV2c14cyriTJmCg==:Ext:ProtonMail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.2 required=7.0 tests=ALL_TRUSTED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.protonmail.ch X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33899 Cc: Antoine Eiche , "go-ipfs-wg\\@ipfs.io" , Pierre Neidhardt , "33899\\@debbugs.gnu.org" <33899@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Hector Sanjuan Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hey! Thanks for reviving this discussion! =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Original Me= ssage =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 On Friday, July 12, 2019 10:15 PM, Ludovic Court=C3=A8s wrot= e: > Hello! > > Pierre Neidhardt mail@ambrevar.xyz skribis: > > > A little update/recap after many months! :) > > Thank you, and apologies for the delay! > > > - Extra metadata: IPFS stores files on UnixFSv1 which does not > > include the executable bit. > > - Right now we store a s-exp manifest with a list of files and a > > list of executable bits. But maybe we don't have to roll out ou= r own. > > > > - UnixFSv1 has some metadata field, but H=C3=A9ctor and Alex did = not > > recommend using it (not sure why though). > > > > - We could use UnixFSv2 but it's not released yet and it's unclea= r when > > it's going to be released. So we can't really count on it right= now. > > > > UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2 > would be appropriate, though it stores timestamps that we don=E2=80=99t n= eed > (not necessarily a problem). > > > - Flat storage vs. tree storage: Right now we are storing the files > > separately, but this has some shortcomings, namely we need multiple > > "get" requests instead of just one, and that IPFS does > > not "know" that those files are related. (We lose the web view of > > the tree, etc.) Storing them as tree could be better. > > I don't understand if that would work with the "IPLD manifest" > > suggested above. H=C3=A9ctor? > > > > I don=E2=80=99t consider the web view a strong argument :-) since we coul= d write > tools to deal with whatever format we use. > > Regarding multiple GET requests: we could pipeline them, and it seems > more like an implementation detail to me. The real question is whether > making separate GET requests prevents some optimization in IPFS. > > > - Pinning: Pinning all files separately incurs an overhead. It's > > enough to just pin the IPLD object since it propagates recursively. > > When adding a tree, then it's no problem since pinning is only done= once. > > > > Where=E2=80=99s the overhead exactly? There are reasons why we are proposing to create a single DAG with an IPLD object at the root. Pinning has a big overhead because it involves locking, reading, parsing, and writing an internal pin-DAG. This is specially relevant when the pinset is very large. Doing multiple GET requests also has overhead, like being unable to use a single bitswap session (which, when downloading something new means a big overhead since every request will have to find providers). And it's not just the web view, it's the ability to walk/traverse all the object related to a given root natively, which allows also to compare multiple trees and to be more efficient for some things ("pin update" for example). Your original idea is to create a manifest with references to different parts. I'm just asking you to create that manifest in a format where those references are understood not only by you, the file creator, but by IPFS and any tool that can read IPLD, by making this a IPLD object (which is just a json). The process of adding "something" to ipfs is as follows. ---- 1. Add to IPFS: multipart upload equivalent to "ipfs add -r": ~/ipfstest $ ipfs add -r -q . QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP 2. Add manifest as IPLD object. dag/put a json file like: cat <