‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Saturday, December 29, 2018 12:12 AM, Ludovic Courtès wrote: > Hello Guix! > > Here is a first draft adding support to distribute and retrieve substitutes > over IPFS. This builds on discussions at the R-B Summit with Héctor Sanjuan > of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian > Paul Schmidt posted on guix-devel last month. > > The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides > bindings to a subset of that API. This module also implements a custom > “directory” format to store directory trees in IPFS (IPFS already provides > “UnixFS” and “tar” but they store too many or too few file attributes.) > > ‘guix publish’ and ‘guix substitute’ use (guix ipfs) to > store and retrieve store items. Complete directory trees are stored in > IPFS “as is”, rather than as compressed archives (nars). This allows for > deduplication in IPFS. ‘guix publish’ adds a new “IPFS” field in > narinfos and ‘guix substitute’ can then query those objects over IPFS. > So the idea is that you still get narinfos over HTTP(S), and then you > have the option of downloading substitutes over IPFS. > > I’ve pushed these patches in ‘wip-ipfs-substitutes’. This is rough on the > edges and probably buggy, but the adventurous among us might want to give > it a spin. :-) > > Thanks, > Ludo’. Hey! Happy new year! This is great news. I'm very glad to see this. I haven't tried this yet but looking at the code there are a couple of things to point out. 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs has a read/write API (on :5001) and a read-only API that we call "gateway" and which runs on :8080. The gateway, apart from handling most of the read-only methods from the HTTP API, also handles paths like "/ipfs/" or "/ipns/" gracefully, and returns an autogenerated webpage for directory-type CIDs. The gateway does not allow to "publish". Therefore I think the doc strings should say "IPFS daemon API" rather than "GATEWAY". 2) I'm not proficient enough in schema to grasp the details of the "directory" format. If I understand it right, you keep a separate manifest object listing the directory structure, the contents and the executable bit for each. Thus, when adding a store item you add all the files separately and this manifest. And when retrieving a store item you fetch the manifest and reconstruct the tree by fetching the contents in it (and applying the executable flag). Is this correct? This works, but it can be improved: You can add all the files/folders in a single request. If I'm reading it right, now each files is added separately (and gets pinned separately). It would probably make sense to add it all in a single request, letting IPFS to store the directory structure as "unixfs". You can additionally add the sexp file with the dir-structure and executable flags as an extra file to the root folder. This would allow to fetch the whole thing with a single request too /api/v0/get?arg=. And to pin a single hash recursively (and not each separately). After getting the whole thing, you will need to chmod +x things accordingly. It will probably need some trial an error to get the multi-part right to upload all in a single request. The Go code HTTP Clients doing this can be found at: https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96 As you see, a directory part in the multipart will have the content-type Header set to "application/x-directory". The best way to see how "abspath" etc is set is probably to sniff an `ipfs add -r ` operation (localhost:5001). Once UnixFSv2 lands, you will be in a position to just drop the sexp file altogether. Let me know if you have any doubts, I'll make my best to answer them. In the meantime I'll try to get more familiar with Guix. Cheers, Hector PS. There is a place where it says "ifps" instead of "ipfs". A very common typo.