[RFC PATCH 0/3] Decentralized substitute distribution with ERIS

OpenSubmitted by pukkamustard.
Details
2 participants
  • Ludovic Courtès
  • pukkamustard
Owner
unassigned
Severity
important
P
P
pukkamustard wrote on 16 Dec 2021 17:17
(address . guix-patches@gnu.org)
20211216161724.547-1-pukkamustard@posteo.net
Hello Guix,

This is an initial patch and proposal towards decentralizing substitute
distribution with ERIS.

ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
uniformly sized, encryped and content-addressed blocks. The original content
can be reconstructed only with access to a read capability, which can be
encoded as an URN.

One key advantage of ERIS is that the encoding is protocol agnostic. Any
protocol that can transfer small (32KiB) sized blocks referenced by the hash of
their content will do. This can be done with things such as GNUNet, IPFS,
OpenDHT, HTTP or a USB stick on a bicycle.

The following patch allows substitutes to be published over IPFS using ERIS.
This is inspired and very similar to previous work on distributing substitutes
over IPFS [2].

The narinfos served by `guix publish` look like this:

Toggle snippet (16 lines)
StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: gzip
FileSize: 67363
ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: zstd
FileSize: 64917
ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
NarHash: sha256:1sagsz1mnlqkr8r8s6gwkzvvhq619rlzhpbxl3h0b111n5hn2w9w
NarSize: 220704
References: 2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33 81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib
Deriver: 260bk0ch4np4h2yz5yqhf8hjbsyhwpmr-hello-2.10.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNDk4ODkwODZDNTY4MzQyRENFQzk3QzA3NDE4NEQ1RkRCOTNCNDA2MUNCRDM4MUExRjVBQzVDODI0MTgwMTU3OSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMEU2NDlFODE4QzRFNjNGNEY2OUQ5QTAwRjUwNjRDMzQ3QjY3RDM0RTM0NTg2MkI4NTc3RTg5MUY5Q0Q3NDhBQiMpCiAgIChzICMwMTZGRjA1MDdCQjZGMzA2NUEzMjYzRDA2MTAyRDc5MTBEOEZGODc5RTdENjREODRFODBENDBGMTJFMTBBOTQ1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

For every compressed nar the ERIS URN is computed and added.

If the `--ipfs` is used for `guix publish` then the encoded blocks are also
uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
this:

Toggle snippet (8 lines)
(use-modules (eris)
(eris blocks ipfs))

(eris-decode->bytevector
"urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
eris-blocks-ipfs-ref)

These patches do not yet retrieve content from IPFS (TODO). But in principle,
anybody connected to IPFS can get the nar with the ERIS URN. This could be used
to reduce load on substitute server as they would only need to publish the ERIS
URN directly - substitutes could be delivered much more peer-to-peer.

Other transports that I have been looking in to and am pretty sure will work
include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
identifiers (URN) are abstracted away from specific transports (and also
applications). ERIS is almost exactly the same encoding as used in GNUNet
(ECRS).

Blocks can be stored in any kind of databases (see for example the GDBM
bindings [4]).

A tricky things is figuring out how to multiplex all these different
transports and storages...

The ERIS specification is still considered "experimental". However we feel
confident to stabilize it and intend to do so around February/March 2022 with a
release 1.0.0 of the specification. This will ensure that the identifiers
remain stable for the forseeable future (until the crypto breaks). Before that
there is also a small external security audit of the specification planned
(thanks to NGI0/NLnet!).

This is just a little demo of the idea and some food for thought and
discussion. Give it a try and let me know what you think!

I've also pushed the patches to my personal Guix mirror if you want to check it
out from there:


Also CCing ~pukkamustard/eris@lists.sr.ht where there is some general ERIS
related discussion.

Thanks,
-pukkamustard



pukkamustard (3):
publish: Add ERIS URN to narinfo
WIP: gnu: guile-eris: Update to unreleased git version.
publish: Add IPFS support.

configure.ac | 5 ++
gnu/packages/guile-xyz.scm | 10 ++--
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 10 ++--
guix/scripts/publish.scm | 79 ++++++++++++++++++++++-------
5 files changed, 79 insertions(+), 26 deletions(-)

--
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 17:20
[RFC PATCH 1/3] publish: Add ERIS URN to narinfo
(address . 52555@debbugs.gnu.org)
20211216162036.694-1-pukkamustard@posteo.net
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urns parameter and honor it.
(store-item->recutils): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-urns]: New field.
(narinfo-maker): Handle ERIS URN.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
configure.ac | 5 +++++
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 10 ++++++----
guix/scripts/publish.scm | 31 +++++++++++++++++++++--------
4 files changed, 35 insertions(+), 12 deletions(-)

Toggle diff (168 lines)
diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
 AM_CONDITIONAL([HAVE_GUILE_AVAHI],
   [test "x$have_guile_avahi" = "xyes"])
 
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+  [test "x$have_guile_eris" = "xyes"])
+
 dnl Guile-newt is used by the graphical installer.
 GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
 
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 9496499850..5c49167782 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -394,6 +394,7 @@ (define code
                        ("guile-zstd" ,guile-zstd)
                        ("guile-ssh" ,guile-ssh)
                        ("guile-git" ,guile-git)
+                       ("guile-eris" ,guile-eris)
 
                        ;; XXX: Keep the development inputs here even though
                        ;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..0972ede3c1 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -67,13 +67,14 @@ (define-module (guix narinfo)
             equivalent-narinfo?))
 
 (define-record-type <narinfo>
-  (%make-narinfo path uri-base uris compressions file-sizes file-hashes
-                 nar-hash nar-size references deriver system
+  (%make-narinfo path uri-base uris eris-urns compressions file-sizes
+                 file-hashes nar-hash nar-size references deriver system
                  signature contents)
   narinfo?
   (path         narinfo-path)
   (uri-base     narinfo-uri-base)        ;URI of the cache it originates from
   (uris         narinfo-uris)            ;list of strings
+  (eris-urns    narinfo-eris-urns)       ;list of (strings | #f)
   (compressions narinfo-compressions)    ;list of strings
   (file-sizes   narinfo-file-sizes)      ;list of (integers | #f)
   (file-hashes  narinfo-file-hashes)
@@ -134,7 +135,7 @@ (define (narinfo-signature->canonical-sexp str)
 (define (narinfo-maker str cache-url)
   "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
 must contain the original contents of a narinfo file."
-  (lambda (path urls compressions file-hashes file-sizes
+  (lambda (path urls eris-urns compressions file-hashes file-sizes
                 nar-hash nar-size references deriver system
                 signature)
     "Return a new <narinfo> object."
@@ -148,6 +149,7 @@ (define len (length urls))
                                    (string-append cache-url url)
                                    (string-append cache-url "/" url)))))
                         urls)
+                   eris-urns
                    compressions
                    (match file-sizes
                      (()        (make-list len #f))
@@ -186,7 +188,7 @@ (define* (read-narinfo port #:optional url
                      "FileHash" "FileSize" "NarHash" "NarSize"
                      "References" "Deriver" "System"
                      "Signature")
-                   '("URL" "Compression" "FileSize" "FileHash"))))
+                   '("URL" "ERIS" "Compression" "FileSize" "FileHash"))))
 
 (define (narinfo-sha256 narinfo)
   "Return the sha256 hash of NARINFO as a bytevector, or #f if NARINFO lacks a
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..8e4b90789b 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -308,7 +309,7 @@ (define* (store-item->recutils store-item
                                #:key
                                (nar-path "nar")
                                (compression %no-compression)
-                               file-size)
+                               file-size eris-urn)
   "Return the 'Compression' and 'URL' fields of the narinfo for STORE-ITEM,
 with COMPRESSION, starting at NAR-PATH."
   (let ((url (encode-and-join-uri-path
@@ -319,19 +320,22 @@ (define* (store-item->recutils store-item
                     (($ <compression> type)
                      (list (symbol->string type))))
                 ,(basename store-item)))))
-    (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]"
-            url (compression-type compression) file-size)))
+    (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]~@[ERIS: ~a~%~]"
+            url (compression-type compression) file-size eris-urn)))
 
 (define* (narinfo-string store store-path
                          #:key (compressions (list %no-compression))
-                         (nar-path "nar") (file-sizes '()))
+                         (nar-path "nar") (file-sizes '()) (eris-urns '()))
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
 
 Optionally, FILE-SIZES is a list of compression/integer pairs, where the
 integer is size in bytes of the compressed NAR; it informs the client of how
-much needs to be downloaded."
+much needs to be downloaded.
+
+Optionally, ERIS-URNS is a list of compression/string pairs, where the
+string is the ERIS URN of the compressed NAR."
   (let* ((path-info  (query-path-info store store-path))
          (compressions (actual-compressions store-path compressions))
          (hash       (bytevector->nix-base32-string
@@ -352,9 +356,12 @@ (define* (narinfo-string store store-path
                              store-path
                              (map (lambda (compression)
                                     (let ((size (assoc-ref file-sizes
-                                                           compression)))
+                                                           compression))
+                                          (eris-urn (assoc-ref eris-urns
+                                                               compression)))
                                       (store-item->recutils store-path
                                                             #:file-size size
+                                                            #:eris-urn eris-urn
                                                             #:nar-path nar-path
                                                             #:compression
                                                             compression)))
@@ -632,6 +639,12 @@ (define (compressed-nar-size compression)
       (and stat
            (cons compression (stat:size stat)))))
 
+  (define (compressed-eris-urn compression)
+    (let* ((nar (nar-cache-file cache item #:compression compression))
+           (stat (stat nar #f)))
+      (and stat
+           (cons compression (call-with-input-file nar eris-encode->urn)))))
+
   (let ((compression (actual-compressions item compressions)))
 
     (for-each (cut compress-nar cache item <>) compressions)
@@ -646,11 +659,13 @@ (define (compressed-nar-size compression)
              ;; thread's connection to the store since we would end up sending
              ;; stuff concurrently on the same channel.
              (with-store store
-               (let ((sizes (filter-map compressed-nar-size compression)))
+               (let ((sizes (filter-map compressed-nar-size compression))
+                     (eris-urns (filter-map compressed-eris-urn compression)))
                  (display (narinfo-string store item
                                           #:nar-path nar-path
                                           #:compressions compressions
-                                          #:file-sizes sizes)
+                                          #:file-sizes sizes
+                                          #:eris-urns eris-urns)
                           port)))
 
              ;; Make the cached narinfo world-readable, contrary to what
-- 
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 17:20
[RFC PATCH 2/3] WIP: gnu: guile-eris: Update to unreleased git version.
(address . 52555@debbugs.gnu.org)
20211216162036.694-2-pukkamustard@posteo.net
* gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version.
[source]: Update source URI.
[propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi.
---
gnu/packages/guile-xyz.scm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

Toggle diff (35 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index 8346d99996..5e6f31a6e2 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -4325,15 +4325,15 @@ (define-public guile-sodium
 (define-public guile-eris
   (package
     (name "guile-eris")
-    (version "0.2.0")
+    (version "f1e4dd87988f9a80b05a8051d7f5ba3daf79dcc1")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
-             (url "https://inqlab.net/git/eris.git")
-             (commit (string-append "v" version))))
+             (url "https://inqlab.net/git/guile-eris.git")
+             (commit version)))
        (file-name (git-file-name name version))
-       (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka"))))
+       (sha256 (base32 "0kgm4b4qn2s74wjvxy273gdi1l1m81i2k4kkk1zc6vlcg3np7p06"))))
     (build-system gnu-build-system)
     (arguments '())
     (native-inputs
@@ -4345,7 +4345,7 @@ (define-public guile-eris
            guile-srfi-180))
     (inputs (list guile-3.0))
     (propagated-inputs
-     (list guile-sodium))
+     (list guile-sodium guile-json-4 guile-gdbm-ffi))
     (synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
     (description
      "Guile-ERIS is the reference implementation of the Encoding for Robust
-- 
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 17:20
[RFC PATCH 3/3] publish: Add IPFS support.
(address . 52555@debbugs.gnu.org)
20211216162036.694-3-pukkamustard@posteo.net
* guix/scripts/publish.scm: (show-help, %options): Add '--ipfs'.
(render-narinfo/cached, bake-narinfo+nar, make-request-handler, run-publish-server): Add #:ipfs? and honor it.
(guix-publish): Honor '--ipfs' and parameterize %ipfs-base-url.
---
guix/scripts/publish.scm | 52 +++++++++++++++++++++++++++++++---------
1 file changed, 41 insertions(+), 11 deletions(-)

Toggle diff (178 lines)
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 8e4b90789b..8e7fb47b9e 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -41,6 +41,8 @@ (define-module (guix scripts publish)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
+  #:use-module (srfi srfi-171)
   #:use-module (web http)
   #:use-module (web request)
   #:use-module (web response)
@@ -52,6 +54,7 @@ (define-module (guix scripts publish)
   #:use-module (guix base64)
   #:use-module (guix config)
   #:use-module (guix derivations)
+  #:use-module ((guix ipfs) #:prefix ipfs:)
   #:use-module (gcrypt hash)
   #:use-module (guix pki)
   #:use-module (gcrypt pk-crypto)
@@ -59,6 +62,8 @@ (define-module (guix scripts publish)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
   #:use-module (eris)
+  #:use-module (eris read-capability)
+  #:use-module (eris blocks ipfs)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -83,6 +88,7 @@ (define-module (guix scripts publish)
             run-publish-server
             guix-publish))
 
+
 (define (show-help)
   (format #t (G_ "Usage: guix publish [OPTION]...
 Publish ~a over HTTP.\n") %store-directory)
@@ -102,6 +108,8 @@ (define (show-help)
   (display (G_ "
       --cache-bypass-threshold=SIZE
                          serve store items below SIZE even when not cached"))
+  (display (G_ "
+      --ipfs[=GATEWAY]   publish items over IPFS via GATEWAY"))
   (display (G_ "
       --workers=N        use N workers to bake items"))
   (display (G_ "
@@ -220,6 +228,10 @@ (define %options
                 (lambda (opt name arg result)
                   (alist-cons 'cache-bypass-threshold (size->number arg)
                               result)))
+        (option '("ipfs") #f #t
+                (lambda (opt name arg result)
+                  (alist-cons 'ipfs (or arg (ipfs:%ipfs-base-url))
+                              result)))
         (option '("workers") #t #f
                 (lambda (opt name arg result)
                   (alist-cons 'workers (string->number* arg)
@@ -526,7 +538,7 @@ (define (bypass-cache? store item)
 (define* (render-narinfo/cached store request hash
                                 #:key ttl (compressions (list %no-compression))
                                 (nar-path "nar") negative-ttl
-                                cache pool)
+                                cache pool ipfs?)
   "Respond to the narinfo request for REQUEST.  If the narinfo is available in
 CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo
 requested using POOL."
@@ -571,7 +583,8 @@ (define (delete-entry narinfo)
                  (bake-narinfo+nar cache item
                                    #:ttl ttl
                                    #:compressions compressions
-                                   #:nar-path nar-path)))
+                                   #:nar-path nar-path
+                                   #:ipfs? ipfs?)))
 
              (when ttl
                (single-baker 'cache-cleanup
@@ -631,7 +644,7 @@ (define (write-compressed-file call-with-compressed-output-port)
 
 (define* (bake-narinfo+nar cache item
                            #:key ttl (compressions (list %no-compression))
-                           (nar-path "/nar"))
+                           (nar-path "/nar") ipfs?)
   "Write the narinfo and nar for ITEM to CACHE."
   (define (compressed-nar-size compression)
     (let* ((nar  (nar-cache-file cache item #:compression compression))
@@ -641,9 +654,19 @@ (define (compressed-nar-size compression)
 
   (define (compressed-eris-urn compression)
     (let* ((nar (nar-cache-file cache item #:compression compression))
-           (stat (stat nar #f)))
+           (stat (stat nar #f))
+           (block-reducer (if ipfs?
+                              (eris-blocks-ipfs-reducer
+                               #:ipfs-base-url (ipfs:%ipfs-base-url))
+                              rcount)))
       (and stat
-           (cons compression (call-with-input-file nar eris-encode->urn)))))
+           (cons compression
+                 (call-with-input-file nar
+                   (lambda (port)
+                     (let ((read-cap _
+                                     (eris-encode port #:block-reducer
+                                                  block-reducer)))
+                       (read-capability->string read-cap))))))))
 
   (let ((compression (actual-compressions item compressions)))
 
@@ -1115,7 +1138,8 @@ (define* (make-request-handler store
                                cache pool
                                narinfo-ttl narinfo-negative-ttl
                                (nar-path "nar")
-                               (compressions (list %no-compression)))
+                               (compressions (list %no-compression))
+                               ipfs?)
   (define compression-type?
     string->compression-type)
 
@@ -1147,7 +1171,8 @@ (define (handle request body)
                                       #:ttl narinfo-ttl
                                       #:negative-ttl narinfo-negative-ttl
                                       #:nar-path nar-path
-                                      #:compressions compressions)
+                                      #:compressions compressions
+                                      #:ipfs? ipfs?)
                (render-narinfo store request hash
                                #:ttl narinfo-ttl
                                #:negative-ttl narinfo-negative-ttl
@@ -1218,7 +1243,7 @@ (define* (run-publish-server socket store
                              advertise? port
                              (compressions (list %no-compression))
                              (nar-path "nar") narinfo-ttl narinfo-negative-ttl
-                             cache pool)
+                             cache pool ipfs?)
   (when advertise?
     (let ((name (service-name)))
       ;; XXX: Use a callback from Guile-Avahi here, as Avahi can pick a
@@ -1234,7 +1259,8 @@ (define* (run-publish-server socket store
                                     #:nar-path nar-path
                                     #:narinfo-ttl narinfo-ttl
                                     #:narinfo-negative-ttl narinfo-negative-ttl
-                                    #:compressions compressions)
+                                    #:compressions compressions
+                                    #:ipfs? ipfs?)
               concurrent-http-server
               `(#:socket ,socket)))
 
@@ -1296,6 +1322,8 @@ (define-command (guix-publish . args)
            (repl-port (assoc-ref opts 'repl))
            (cache     (assoc-ref opts 'cache))
            (workers   (assoc-ref opts 'workers))
+           (ipfs      (assoc-ref opts 'ipfs))
+           (ipfs?     (if ipfs #t #f))
 
            ;; Read the key right away so that (1) we fail early on if we can't
            ;; access them, and (2) we can then drop privileges.
@@ -1315,7 +1343,8 @@ (define-command (guix-publish . args)
                      (%private-key private-key)
                      (cache-bypass-threshold
                       (or (assoc-ref opts 'cache-bypass-threshold)
-                          (cache-bypass-threshold))))
+                          (cache-bypass-threshold)))
+                     (ipfs:%ipfs-base-url ipfs))
         (info (G_ "publishing ~a on ~a, port ~d~%")
               %store-directory
               (inet-ntop (sockaddr:fam address) (sockaddr:addr address))
@@ -1344,7 +1373,8 @@ (define-command (guix-publish . args)
                               #:nar-path nar-path
                               #:compressions compressions
                               #:narinfo-negative-ttl negative-ttl
-                              #:narinfo-ttl ttl))))))
+                              #:narinfo-ttl ttl
+                              #:ipfs? ipfs?))))))
 
 ;;; Local Variables:
 ;;; eval: (put 'single-baker 'scheme-indent-function 1)
-- 
2.34.0
L
L
Ludovic Courtès wrote on 18 Dec 2021 21:01
control message for bug #52555
(address . control@debbugs.gnu.org)
87czltof7d.fsf@gnu.org
severity 52555 important
quit
L
L
Ludovic Courtès wrote on 20 Dec 2021 17:25
Re: bug#52555: [RFC PATCH 0/3] Decentralized substitute distribution with ERIS
(name . pukkamustard)(address . pukkamustard@posteo.net)
87h7b3gs64.fsf@gnu.org
Hi pukkamustard,

pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (3 lines)
> This is an initial patch and proposal towards decentralizing substitute
> distribution with ERIS.

Woohoo, sounds exciting!

Toggle quote (10 lines)
> ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
> uniformly sized, encryped and content-addressed blocks. The original content
> can be reconstructed only with access to a read capability, which can be
> encoded as an URN.
>
> One key advantage of ERIS is that the encoding is protocol agnostic. Any
> protocol that can transfer small (32KiB) sized blocks referenced by the hash of
> their content will do. This can be done with things such as GNUNet, IPFS,
> OpenDHT, HTTP or a USB stick on a bicycle.

Yes, that’s nice.

Toggle quote (16 lines)
> The following patch allows substitutes to be published over IPFS using ERIS.
> This is inspired and very similar to previous work on distributing substitutes
> over IPFS [2].
>
> The narinfos served by `guix publish` look like this:
>
> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: gzip
> FileSize: 67363
> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: zstd
> FileSize: 64917
> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM

Do we really need one URN per compression method? Couldn’t we leave
compression (of individual chunks, possibly) as a “detail” handled by
the encoding or the transport layer?

Toggle quote (16 lines)
> If the `--ipfs` is used for `guix publish` then the encoded blocks are also
> uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
> this:
>
> (use-modules (eris)
> (eris blocks ipfs))
>
> (eris-decode->bytevector
> "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
> eris-blocks-ipfs-ref)
>
> These patches do not yet retrieve content from IPFS (TODO). But in principle,
> anybody connected to IPFS can get the nar with the ERIS URN. This could be used
> to reduce load on substitute server as they would only need to publish the ERIS
> URN directly - substitutes could be delivered much more peer-to-peer.

Nice. So adjusting ‘guix substitute’ should be relatively easy?

Toggle quote (7 lines)
> Other transports that I have been looking in to and am pretty sure will work
> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
> identifiers (URN) are abstracted away from specific transports (and also
> applications). ERIS is almost exactly the same encoding as used in GNUNet
> (ECRS).

As a first step, ‘guix publish’ could implement RFC 2169, too.

I gather implementing the HTTP and IPFS backends in ‘guix substitute’
should be relatively easy, right?

Toggle quote (6 lines)
> Blocks can be stored in any kind of databases (see for example the GDBM
> bindings [4]).
>
> A tricky things is figuring out how to multiplex all these different
> transports and storages...

Yes. We don’t know yet what performance and data availability will be
like on IPFS, for instance, so it’s important for users to be able to
set priorities. It’s also important to gracefully fall back to direct
HTTP downloads when fancier p2p methods fail, regardless of how they
fail.

Toggle quote (7 lines)
> The ERIS specification is still considered "experimental". However we feel
> confident to stabilize it and intend to do so around February/March 2022 with a
> release 1.0.0 of the specification. This will ensure that the identifiers
> remain stable for the forseeable future (until the crypto breaks). Before that
> there is also a small external security audit of the specification planned
> (thanks to NGI0/NLnet!).

Neat.

This is all very exciting. I look forward to playing around with it!

Ludo’.
P
P
pukkamustard wrote on 23 Dec 2021 12:42
(name . Ludovic Courtès)(address . ludo@gnu.org)
86bl17ms56.fsf@posteo.net
Hi Ludo,

Thanks for your comments!

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (15 lines)
>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> Compression: gzip
>> FileSize: 67363
>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> Compression: zstd
>> FileSize: 64917
>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>
> Do we really need one URN per compression method? Couldn’t we leave
> compression (of individual chunks, possibly) as a “detail” handled by
> the encoding or the transport layer?
>

I agree that it would be nice to leave this to the encoding layer as
that would allow certain optimizations (e.g. de-duplication).

Unfortunately, we haven't figured out yet what the most suitable
compression/format would be. Something like EROSFS seems good (as it
aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
just an archive format and there do not seem to be any libraries that we
could use to neatly integrate. It seems possible to block-align a Tar
archive, but that seems a bit hackey [2]. Other things to look into
might be Tarlz [3] and ZPAQ [4].

To get started I suggest just using one of the compressions/formats
already in Guix. zstd seems to be a reasonable choice (for the same
reasons why it makes sense to use zstd with `--discover` [5]).

Does that sound like a plan?


Toggle quote (18 lines)
>> If the `--ipfs` is used for `guix publish` then the encoded blocks are also
>> uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
>> this:
>>
>> (use-modules (eris)
>> (eris blocks ipfs))
>>
>> (eris-decode->bytevector
>> "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
>> eris-blocks-ipfs-ref)
>>
>> These patches do not yet retrieve content from IPFS (TODO). But in principle,
>> anybody connected to IPFS can get the nar with the ERIS URN. This could be used
>> to reduce load on substitute server as they would only need to publish the ERIS
>> URN directly - substitutes could be delivered much more peer-to-peer.
>
> Nice. So adjusting ‘guix substitute’ should be relatively easy?

Yes, relatively! :)

I meant to send in a V2 that does this before going on holidays, but I'm
afraid I won't make it. V2 will come in early January!

Toggle quote (12 lines)
>> Other transports that I have been looking in to and am pretty sure will work
>> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
>> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
>> identifiers (URN) are abstracted away from specific transports (and also
>> applications). ERIS is almost exactly the same encoding as used in GNUNet
>> (ECRS).
>
> As a first step, ‘guix publish’ could implement RFC 2169, too.
>
> I gather implementing the HTTP and IPFS backends in ‘guix substitute’
> should be relatively easy, right?

Yes, those seem to be the two easiest backends to implement.

Toggle quote (9 lines)
>> A tricky things is figuring out how to multiplex all these different
>> transports and storages...
>
> Yes. We don’t know yet what performance and data availability will be
> like on IPFS, for instance, so it’s important for users to be able to
> set priorities. It’s also important to gracefully fall back to direct
> HTTP downloads when fancier p2p methods fail, regardless of how they
> fail.

Agree.

Thanks,
-pukkamustard
L
L
Ludovic Courtès wrote on 24 Dec 2021 15:48
(name . pukkamustard)(address . pukkamustard@posteo.net)
87ilveukhr.fsf@gnu.org
Hi!

pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (28 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: gzip
>>> FileSize: 67363
>>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: zstd
>>> FileSize: 64917
>>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>>
>> Do we really need one URN per compression method? Couldn’t we leave
>> compression (of individual chunks, possibly) as a “detail” handled by
>> the encoding or the transport layer?
>>
>
> I agree that it would be nice to leave this to the encoding layer as
> that would allow certain optimizations (e.g. de-duplication).
>
> Unfortunately, we haven't figured out yet what the most suitable
> compression/format would be. Something like EROSFS seems good (as it
> aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
> just an archive format and there do not seem to be any libraries that we
> could use to neatly integrate. It seems possible to block-align a Tar
> archive, but that seems a bit hackey [2]. Other things to look into
> might be Tarlz [3] and ZPAQ [4].

Yeah. Though it may be that deduplication at the block level doesn’t
buy us much. That was the conclusion I reached a long time ago[a], and
also seems to be supported by the recent guix-daemon deduplication
improvements[b].


Toggle quote (6 lines)
> To get started I suggest just using one of the compressions/formats
> already in Guix. zstd seems to be a reasonable choice (for the same
> reasons why it makes sense to use zstd with `--discover` [5]).
>
> Does that sound like a plan?

Sure!

Toggle quote (3 lines)
> I meant to send in a V2 that does this before going on holidays, but I'm
> afraid I won't make it. V2 will come in early January!

Alright, we’ll see! :-)

Until then, enjoy your holidays!

Ludo’.
P
P
pukkamustard wrote 4 hours ago
[RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(address . 52555@debbugs.gnu.org)
20220125192201.7582-1-pukkamustard@posteo.net
Hello Guix,

Here comes the V2 of a proposal towards decentralizing substitute distribution
with ERIS.

A quick summary (as this has become quite long):

- This adds support for publishing and getting substitutes over IPFS.
- By using the ERIS encoding we are not limited to using IPFS as transport. We
can also use GNUNet, Named Data Networking (possibly) or just plain old HTTP. Support
for these can be added in (guix eris).
- These patches are still very rough and we need better logic for when to use
IPFS et. al. and when to fallback to HTTP.
- There might be performance issues when using IPFS via the IPFS daemon HTTP
API.

I found the setup for testing this a bit tricky. I will try and describe how I
have been testing it. Please let me know how this can be improved!

** Authorize local substitutes

We will be running a local substitute server so we need to add the local
signing key to the list of authorized keys. In the system configurations:

#+BEGIN_SRC scheme
(modify-services %base-services
(guix-service-type
config =>
(guix-configuration
(inherit config)
(authorized-keys
(cons*
;; allow substitutes from ourselves for testing purposes
(local-file "/etc/signing-key.pub")
%default-authorized-guix-keys)))))
#+END_SRC

** Configure the local Guix checkout

#+BEGIN_SRC shell
./bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc
#+END_SRC

The ~--sysconfdir~ is required so that guix will use the ACL in ~/etc/guix/acl~.

** Start the IPFS daemon

#+BEGIN_SRC shell
guix shell go-ipfs -- ipfs daemon
#+END_SRC

Start a local substitute server:

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix publish --public-key=/etc/guix/signing-key.pub --private-key=/etc/guix/signing-key.sec --cache=/tmp/guix-publish-cache/ --port=8081 --compression=zstd:19
#+END_SRC

We use port 8081 as IPFS is running on 8080.

We use the temporary cache directory ~/tmp/guix-publish-cache~.

** Build some package locally

First we build some package:

#+BEGIN_SRC shell
./pre-inst-env guix build hello --no-substitutes --no-offload
#+END_SRC

#+RESULTS:
: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11

** Trigger the substitute server to "bake" a susbtitute

#+BEGIN_SRC shell
#+END_SRC
Toggle snippet (10 lines)
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

If you do this again after a few seconds you will get a different response that
has the ERIS URN and the FileSizes. The reason for this is that Guix publish
bakes the nars asyncrhonisly in the background:

#+BEGIN_SRC shell
#+END_SRC

#+RESULTS:
Toggle snippet (13 lines)
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
FileSize: 57691
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
ERISFormat: application/x-nix-archive+zstd-19
ERIS: urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

These patches have added the ERIS and ERISFormat fields. Eventually we would
have figured out what the best format for use over ERIS is, for now we encode
it in the ERISFormat field.

** Removing a package from the store

This is necessary in order to make guix look for a substitute.

#+BEGIN_SRC shell
./pre-inst-env guix gc -D /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
#+END_SRC

** Start the Guix daemon from the repository

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild --debug --substitute-urls=http://localhost:8081/
#+END_SRC

Note this will probably stop your system Guix daemon. Run ~sudo herd restart
guix-daemon~ to restart it.

#+BEGIN_SRC shell
./pre-inst-env guix build hello
#+END_SRC
Toggle snippet (7 lines)
substituting /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11...
downloading from urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q ...
urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q 502KiB/s 00:00 | 56KiB transferred

/gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11

We have just retreived the substitute for the hello package from IPFS. Hello
decentralized substitutes!

I have only tested this for fairly small packages (up to a few MB).

One issue with IPFS might be that we have to create a new HTTP connection to
the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem
to support HTTP connection re-use and neither does the Guile (web client). I
fear this might become a performance issue. It seems possible to use IPFS more
directly by exposing the Go code as a C library and then using that with the
Guile FFI [1]. This is however a bit complicated and adds a lot of
dependencies. In particular, this should not become a dependency of Guix
itself. The performance of IPFS itself also needs to be evaluated, maybe the
IPFS HTTP API will not be the bottle-neck.

As mentioned in previous mail a simple HTTP transport for blocks would be a
good fallback. This would allow users to get missing blocks (things that
somehow got dropped from IPFS) directly from a substitute server. This is
different then getting the entire NAR from a substitute server. A user might be
missing a single 32KiB block and should be able to get only that. However, such
a HTTP fallback would also suffer from the one-connection-per-block issue. As
part of general ERIS research we are investigating CoAP as a better fallback
transport.

In any case, it would be necessary for the substitute server to store encoded
blocks of the NAR. For this I think it makes sense to use a small database. We
have bindings to use ERIS with GDBM [2]. It might also make sense to use
SQLite, especially if there are other use-cases for such a database.

I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
as transports.

Thanks for making it so far and happy hacking!
-pukkamustard



pukkamustard (5):
WIP: gnu: guile-eris: Update to unreleased git version.
publish: Add ERIS URN to narinfo
Add (guix eris).
publish: Add support for storing ERIS encoded blocks to IPFS.
substitute: Fetch substitutes using ERIS.

Makefile.am | 1 +
configure.ac | 5 +++
gnu/packages/guile-xyz.scm | 10 ++---
gnu/packages/package-management.scm | 1 +
guix/eris.scm | 60 +++++++++++++++++++++++++++++
guix/narinfo.scm | 14 +++++--
guix/scripts/publish.scm | 32 +++++++++++++--
guix/scripts/substitute.scm | 21 +++++++---
8 files changed, 126 insertions(+), 18 deletions(-)
create mode 100644 guix/eris.scm

--
2.34.0
P
P
pukkamustard wrote 4 hours ago
[RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo
(address . 52555@debbugs.gnu.org)
20220125192201.7582-3-pukkamustard@posteo.net
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
(narinfo-maker): Handle ERIS URN and ERIS format.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
configure.ac | 5 +++++
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 14 ++++++++++----
guix/scripts/publish.scm | 24 ++++++++++++++++++++----
4 files changed, 36 insertions(+), 8 deletions(-)

Toggle diff (167 lines)
diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
 AM_CONDITIONAL([HAVE_GUILE_AVAHI],
   [test "x$have_guile_avahi" = "xyes"])
 
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+  [test "x$have_guile_eris" = "xyes"])
+
 dnl Guile-newt is used by the graphical installer.
 GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
 
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 05795824b5..a9094b8b7f 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -404,6 +404,7 @@ (define code
                        ("guile-zstd" ,guile-zstd)
                        ("guile-ssh" ,guile-ssh)
                        ("guile-git" ,guile-git)
+                       ("guile-eris" ,guile-eris)
 
                        ;; XXX: Keep the development inputs here even though
                        ;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..a6a5d3b84b 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -45,6 +45,8 @@ (define-module (guix narinfo)
             narinfo-file-sizes
             narinfo-hash
             narinfo-size
+            narinfo-eris-format
+            narinfo-eris-urn
             narinfo-references
             narinfo-deriver
             narinfo-system
@@ -68,8 +70,8 @@ (define-module (guix narinfo)
 
 (define-record-type <narinfo>
   (%make-narinfo path uri-base uris compressions file-sizes file-hashes
-                 nar-hash nar-size references deriver system
-                 signature contents)
+                 nar-hash nar-size eris-format eris-urn references deriver
+                 system signature contents)
   narinfo?
   (path         narinfo-path)
   (uri-base     narinfo-uri-base)        ;URI of the cache it originates from
@@ -79,6 +81,8 @@ (define-record-type <narinfo>
   (file-hashes  narinfo-file-hashes)
   (nar-hash     narinfo-hash)
   (nar-size     narinfo-size)
+  (eris-format  narinfo-eris-format)
+  (eris-urn     narinfo-eris-urn)
   (references   narinfo-references)
   (deriver      narinfo-deriver)
   (system       narinfo-system)
@@ -135,7 +139,7 @@ (define (narinfo-maker str cache-url)
   "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
 must contain the original contents of a narinfo file."
   (lambda (path urls compressions file-hashes file-sizes
-                nar-hash nar-size references deriver system
+                nar-hash nar-size eris-format eris-urn references deriver system
                 signature)
     "Return a new <narinfo> object."
     (define len (length urls))
@@ -157,6 +161,8 @@ (define len (length urls))
                      ((lst ...) (map string->number lst)))
                    nar-hash
                    (and=> nar-size string->number)
+                   eris-format
+                   (if eris-urn (string->uri eris-urn) #f)
                    (string-tokenize references)
                    (match deriver
                      ((or #f "") #f)
@@ -184,7 +190,7 @@ (define* (read-narinfo port #:optional url
                    (narinfo-maker str url)
                    '("StorePath" "URL" "Compression"
                      "FileHash" "FileSize" "NarHash" "NarSize"
-                     "References" "Deriver" "System"
+                     "ERISFormat" "ERIS" "References" "Deriver" "System"
                      "Signature")
                    '("URL" "Compression" "FileSize" "FileHash"))))
 
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..9c83f5183d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -146,6 +147,9 @@ (define %default-gzip-compression
   ;; Since we compress on the fly, default to fast compression.
   (compression 'gzip 3))
 
+(define %eris-zstd-compression
+  (compression 'zstd 19))
+
 (define (default-compression type)
   (compression type 3))
 
@@ -324,7 +328,8 @@ (define* (store-item->recutils store-item
 
 (define* (narinfo-string store store-path
                          #:key (compressions (list %no-compression))
-                         (nar-path "nar") (file-sizes '()))
+                         (nar-path "nar") (file-sizes '())
+                         eris-urn)
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
@@ -347,7 +352,7 @@ (define* (narinfo-string store store-path
 StorePath: ~a
 ~{~a~}\
 NarHash: sha256:~a
-NarSize: ~d
+NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~]
 References: ~a~%"
                              store-path
                              (map (lambda (compression)
@@ -359,7 +364,7 @@ (define* (narinfo-string store store-path
                                                             #:compression
                                                             compression)))
                                   compressions)
-                             hash size references))
+                             hash size eris-urn references))
          ;; Do not render a "Deriver" line if we are rendering info for a
          ;; derivation.  Also do not render a "System" line that would be
          ;; expensive to compute and is currently unused.
@@ -632,6 +637,16 @@ (define (compressed-nar-size compression)
       (and stat
            (cons compression (stat:size stat)))))
 
+  (define (eris-encode-nar compressions)
+    (and (member %eris-zstd-compression compressions)
+         (let* ((nar (nar-cache-file cache item
+                                     #:compression %eris-zstd-compression))
+                (stat (stat nar #f)))
+           (and stat
+                (call-with-input-file nar
+                  (cut eris-encode->string <>
+                       #:block-size %eris-block-size-large))))))
+
   (let ((compression (actual-compressions item compressions)))
 
     (for-each (cut compress-nar cache item <>) compressions)
@@ -650,7 +665,8 @@ (define (compressed-nar-size compression)
                  (display (narinfo-string store item
                                           #:nar-path nar-path
                                           #:compressions compressions
-                                          #:file-sizes sizes)
+                                          #:file-sizes sizes
+                                          #:eris-urn (eris-encode-nar compression))
                           port)))
 
              ;; Make the cached narinfo world-readable, contrary to what
-- 
2.34.0
P
P
pukkamustard wrote 4 hours ago
[RFC PATCH v2 3/5] Add (guix eris).
(address . 52555@debbugs.gnu.org)
20220125192201.7582-4-pukkamustard@posteo.net
* guix/ipfs.scm: New file.
* Makefile.am (MODULES): Add it.
---
Makefile.am | 1 +
guix/eris.scm | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+)
create mode 100644 guix/eris.scm

Toggle diff (75 lines)
diff --git a/Makefile.am b/Makefile.am
index a10aeb817b..7219386361 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -131,6 +131,7 @@ MODULES =					\
   guix/cve.scm					\
   guix/workers.scm				\
   guix/ipfs.scm					\
+  guix/eris.scm					\
   guix/build-system.scm				\
   guix/build-system/android-ndk.scm		\
   guix/build-system/ant.scm			\
diff --git a/guix/eris.scm b/guix/eris.scm
new file mode 100644
index 0000000000..163bbe05ac
--- /dev/null
+++ b/guix/eris.scm
@@ -0,0 +1,55 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix eris)
+  #:use-module (eris)
+  #:use-module (eris blocks ipfs)
+  #:use-module (web client)
+  #:use-module (web response)
+  #:use-module (srfi srfi-71)
+
+  #:export (guix-eris-block-reducer))
+
+(define (ipfs-daemon-alive?)
+  "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
+and #f else."
+  (with-exception-handler
+      (const #f)
+    (lambda _
+      (let ((response _
+                      (http-post (string-append (%ipfs-base-url)
+                                                "/api/v0/version"))))
+        (equal? 200 (response-code response))))
+    #:unwind? #t))
+
+(define guix-eris-block-reducer
+  (case-lambda
+
+    ;; Check if IPFS Daemon is running.
+    (() (if (ipfs-daemon-alive?)
+            (eris-blocks-ipfs-reducer)
+            #f))
+
+    ;; Completion. Nothing to do.
+    ((_) #t)
+
+    ((ipfs ref-block)
+     ;; If IPFS has been initialized store block there
+     (if ipfs
+         (eris-blocks-ipfs-reducer ipfs ref-block)
+         ipfs))))
-- 
2.34.0
P
P
pukkamustard wrote 4 hours ago
[RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-5-pukkamustard@posteo.net
* guix/scripts/publish.scm (bake-narinfo+nar): Use guix-eris-block-reducer
from (guix eris).
---
guix/scripts/publish.scm | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)

Toggle diff (39 lines)
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 9c83f5183d..556107ab7d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -41,6 +41,7 @@ (define-module (guix scripts publish)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
   #:use-module (web http)
   #:use-module (web request)
   #:use-module (web response)
@@ -58,6 +59,7 @@ (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (guix eris)
   #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
@@ -644,8 +646,14 @@ (define (eris-encode-nar compressions)
                 (stat (stat nar #f)))
            (and stat
                 (call-with-input-file nar
-                  (cut eris-encode->string <>
-                       #:block-size %eris-block-size-large))))))
+                  (lambda (port)
+                    (let ((eris-urn _
+                                    (eris-encode port
+                                                 #:block-size
+                                                 %eris-block-size-large
+                                                 #:block-reducer
+                                                 guix-eris-block-reducer)))
+                      (uri->string eris-urn))))))))
 
   (let ((compression (actual-compressions item compressions)))
 
-- 
2.34.0
P
P
pukkamustard wrote 4 hours ago
[RFC PATCH v2 1/5] WIP: gnu: guile-eris: Update to unreleased git version.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-2-pukkamustard@posteo.net
* gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version.
[source]: Update source URI.
[propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi.
---
gnu/packages/guile-xyz.scm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

Toggle diff (35 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index e2cf793acc..66ac486a74 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -4374,15 +4374,15 @@ (define-public guile-sodium
 (define-public guile-eris
   (package
     (name "guile-eris")
-    (version "0.2.0")
+    (version "bcbbcbc88f3ec1f2fafcd034ce5b620516bff105")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
-             (url "https://inqlab.net/git/eris.git")
-             (commit (string-append "v" version))))
+             (url "https://codeberg.org/eris/guile-eris")
+             (commit version)))
        (file-name (git-file-name name version))
-       (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka"))))
+       (sha256 (base32 "17v3h2hqx080739rl57nfradp5vlmy24fgqdxry1zal5z9d3i8sr"))))
     (build-system gnu-build-system)
     (arguments '())
     (native-inputs
@@ -4394,7 +4394,7 @@ (define-public guile-eris
            guile-srfi-180))
     (inputs (list guile-3.0))
     (propagated-inputs
-     (list guile-sodium))
+     (list guile-sodium guile-json-4 guile-gdbm-ffi))
     (synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
     (description
      "Guile-ERIS is the reference implementation of the Encoding for Robust
-- 
2.34.0
P
P
pukkamustard wrote 4 hours ago
[RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-6-pukkamustard@posteo.net
* guix/scripts/substitute.scm (process-substitution): Fetch substitutes using ERIS.
* guix/eris.scm (guix-eris-block-ref): New procedure.
---
guix/eris.scm | 7 ++++++-
guix/scripts/substitute.scm | 21 ++++++++++++++++-----
2 files changed, 22 insertions(+), 6 deletions(-)

Toggle diff (70 lines)
diff --git a/guix/eris.scm b/guix/eris.scm
index 163bbe05ac..0999564c1f 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -23,7 +23,8 @@ (define-module (guix eris)
   #:use-module (web response)
   #:use-module (srfi srfi-71)
 
-  #:export (guix-eris-block-reducer))
+  #:export (guix-eris-block-reducer
+            guix-eris-block-ref))
 
 (define (ipfs-daemon-alive?)
   "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
@@ -53,3 +54,7 @@ (define guix-eris-block-reducer
      (if ipfs
          (eris-blocks-ipfs-reducer ipfs ref-block)
          ipfs))))
+
+(define (guix-eris-block-ref ref)
+  "Dereference a block for decoding ERIS content"
+  (eris-blocks-ipfs-ref ref))
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 908a8334a8..852264976e 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -62,6 +62,8 @@ (define-module (guix scripts substitute)
   #:use-module (srfi srfi-35)
   #:use-module (web uri)
   #:use-module (guix http-client)
+  #:use-module (guix eris)
+  #:use-module (eris)
   #:export (%allow-unauthenticated-substitutes?
             %reply-file-descriptor
 
@@ -486,18 +488,27 @@ (define (fetch uri)
                          #:port port
                          #:keep-alive? #t
                          #:buffered? #f)))))
+
       (else
-       (leave (G_ "unsupported substitute URI scheme: ~a~%")
-              (uri->string uri)))))
+       (if (and (eris-read-capability? uri))
+           (values (eris-decode->port uri
+                                      #:block-ref
+                                      guix-eris-block-ref) #f)
+           (leave (G_ "unsupported substitute URI scheme: ~a~%")
+                  (uri->string uri))))))
+
+  (define* (best-uri narinfo #:key (eris? #f))
+    (if (and eris? (narinfo-eris-urn narinfo))
+        (values (narinfo-eris-urn narinfo) "zstd" #f)
+        (narinfo-best-uri narinfo #:fast-decompression?
+                          %prefer-fast-decompression?)))
 
   (unless narinfo
     (leave (G_ "no valid substitute for '~a'~%")
            store-item))
 
   (let-values (((uri compression file-size)
-                (narinfo-best-uri narinfo
-                                  #:fast-decompression?
-                                  %prefer-fast-decompression?)))
+                (best-uri narinfo #:eris? #t)))
     (unless print-build-trace?
       (format (current-error-port)
               (G_ "Downloading ~a...~%") (uri->string uri)))
-- 
2.34.0
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 52555@debbugs.gnu.org