[RFC PATCH 0/3] Decentralized substitute distribution with ERIS

OpenSubmitted by pukkamustard.
Details
3 participants
  • Ludovic Courtès
  • Maxime Devos
  • pukkamustard
Owner
unassigned
Severity
important
P
P
pukkamustard wrote on 16 Dec 2021 17:17
(address . guix-patches@gnu.org)
20211216161724.547-1-pukkamustard@posteo.net
Hello Guix,

This is an initial patch and proposal towards decentralizing substitute
distribution with ERIS.

ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
uniformly sized, encryped and content-addressed blocks. The original content
can be reconstructed only with access to a read capability, which can be
encoded as an URN.

One key advantage of ERIS is that the encoding is protocol agnostic. Any
protocol that can transfer small (32KiB) sized blocks referenced by the hash of
their content will do. This can be done with things such as GNUNet, IPFS,
OpenDHT, HTTP or a USB stick on a bicycle.

The following patch allows substitutes to be published over IPFS using ERIS.
This is inspired and very similar to previous work on distributing substitutes
over IPFS [2].

The narinfos served by `guix publish` look like this:

Toggle snippet (16 lines)
StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: gzip
FileSize: 67363
ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: zstd
FileSize: 64917
ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
NarHash: sha256:1sagsz1mnlqkr8r8s6gwkzvvhq619rlzhpbxl3h0b111n5hn2w9w
NarSize: 220704
References: 2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33 81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib
Deriver: 260bk0ch4np4h2yz5yqhf8hjbsyhwpmr-hello-2.10.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNDk4ODkwODZDNTY4MzQyRENFQzk3QzA3NDE4NEQ1RkRCOTNCNDA2MUNCRDM4MUExRjVBQzVDODI0MTgwMTU3OSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMEU2NDlFODE4QzRFNjNGNEY2OUQ5QTAwRjUwNjRDMzQ3QjY3RDM0RTM0NTg2MkI4NTc3RTg5MUY5Q0Q3NDhBQiMpCiAgIChzICMwMTZGRjA1MDdCQjZGMzA2NUEzMjYzRDA2MTAyRDc5MTBEOEZGODc5RTdENjREODRFODBENDBGMTJFMTBBOTQ1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

For every compressed nar the ERIS URN is computed and added.

If the `--ipfs` is used for `guix publish` then the encoded blocks are also
uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
this:

Toggle snippet (8 lines)
(use-modules (eris)
(eris blocks ipfs))

(eris-decode->bytevector
"urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
eris-blocks-ipfs-ref)

These patches do not yet retrieve content from IPFS (TODO). But in principle,
anybody connected to IPFS can get the nar with the ERIS URN. This could be used
to reduce load on substitute server as they would only need to publish the ERIS
URN directly - substitutes could be delivered much more peer-to-peer.

Other transports that I have been looking in to and am pretty sure will work
include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
identifiers (URN) are abstracted away from specific transports (and also
applications). ERIS is almost exactly the same encoding as used in GNUNet
(ECRS).

Blocks can be stored in any kind of databases (see for example the GDBM
bindings [4]).

A tricky things is figuring out how to multiplex all these different
transports and storages...

The ERIS specification is still considered "experimental". However we feel
confident to stabilize it and intend to do so around February/March 2022 with a
release 1.0.0 of the specification. This will ensure that the identifiers
remain stable for the forseeable future (until the crypto breaks). Before that
there is also a small external security audit of the specification planned
(thanks to NGI0/NLnet!).

This is just a little demo of the idea and some food for thought and
discussion. Give it a try and let me know what you think!

I've also pushed the patches to my personal Guix mirror if you want to check it
out from there:


Also CCing ~pukkamustard/eris@lists.sr.ht where there is some general ERIS
related discussion.

Thanks,
-pukkamustard



pukkamustard (3):
publish: Add ERIS URN to narinfo
WIP: gnu: guile-eris: Update to unreleased git version.
publish: Add IPFS support.

configure.ac | 5 ++
gnu/packages/guile-xyz.scm | 10 ++--
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 10 ++--
guix/scripts/publish.scm | 79 ++++++++++++++++++++++-------
5 files changed, 79 insertions(+), 26 deletions(-)

--
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 17:20
[RFC PATCH 1/3] publish: Add ERIS URN to narinfo
(address . 52555@debbugs.gnu.org)
20211216162036.694-1-pukkamustard@posteo.net
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urns parameter and honor it.
(store-item->recutils): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-urns]: New field.
(narinfo-maker): Handle ERIS URN.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
configure.ac | 5 +++++
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 10 ++++++----
guix/scripts/publish.scm | 31 +++++++++++++++++++++--------
4 files changed, 35 insertions(+), 12 deletions(-)

Toggle diff (168 lines)
diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
 AM_CONDITIONAL([HAVE_GUILE_AVAHI],
   [test "x$have_guile_avahi" = "xyes"])
 
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+  [test "x$have_guile_eris" = "xyes"])
+
 dnl Guile-newt is used by the graphical installer.
 GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
 
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 9496499850..5c49167782 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -394,6 +394,7 @@ (define code
                        ("guile-zstd" ,guile-zstd)
                        ("guile-ssh" ,guile-ssh)
                        ("guile-git" ,guile-git)
+                       ("guile-eris" ,guile-eris)
 
                        ;; XXX: Keep the development inputs here even though
                        ;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..0972ede3c1 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -67,13 +67,14 @@ (define-module (guix narinfo)
             equivalent-narinfo?))
 
 (define-record-type <narinfo>
-  (%make-narinfo path uri-base uris compressions file-sizes file-hashes
-                 nar-hash nar-size references deriver system
+  (%make-narinfo path uri-base uris eris-urns compressions file-sizes
+                 file-hashes nar-hash nar-size references deriver system
                  signature contents)
   narinfo?
   (path         narinfo-path)
   (uri-base     narinfo-uri-base)        ;URI of the cache it originates from
   (uris         narinfo-uris)            ;list of strings
+  (eris-urns    narinfo-eris-urns)       ;list of (strings | #f)
   (compressions narinfo-compressions)    ;list of strings
   (file-sizes   narinfo-file-sizes)      ;list of (integers | #f)
   (file-hashes  narinfo-file-hashes)
@@ -134,7 +135,7 @@ (define (narinfo-signature->canonical-sexp str)
 (define (narinfo-maker str cache-url)
   "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
 must contain the original contents of a narinfo file."
-  (lambda (path urls compressions file-hashes file-sizes
+  (lambda (path urls eris-urns compressions file-hashes file-sizes
                 nar-hash nar-size references deriver system
                 signature)
     "Return a new <narinfo> object."
@@ -148,6 +149,7 @@ (define len (length urls))
                                    (string-append cache-url url)
                                    (string-append cache-url "/" url)))))
                         urls)
+                   eris-urns
                    compressions
                    (match file-sizes
                      (()        (make-list len #f))
@@ -186,7 +188,7 @@ (define* (read-narinfo port #:optional url
                      "FileHash" "FileSize" "NarHash" "NarSize"
                      "References" "Deriver" "System"
                      "Signature")
-                   '("URL" "Compression" "FileSize" "FileHash"))))
+                   '("URL" "ERIS" "Compression" "FileSize" "FileHash"))))
 
 (define (narinfo-sha256 narinfo)
   "Return the sha256 hash of NARINFO as a bytevector, or #f if NARINFO lacks a
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..8e4b90789b 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -308,7 +309,7 @@ (define* (store-item->recutils store-item
                                #:key
                                (nar-path "nar")
                                (compression %no-compression)
-                               file-size)
+                               file-size eris-urn)
   "Return the 'Compression' and 'URL' fields of the narinfo for STORE-ITEM,
 with COMPRESSION, starting at NAR-PATH."
   (let ((url (encode-and-join-uri-path
@@ -319,19 +320,22 @@ (define* (store-item->recutils store-item
                     (($ <compression> type)
                      (list (symbol->string type))))
                 ,(basename store-item)))))
-    (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]"
-            url (compression-type compression) file-size)))
+    (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]~@[ERIS: ~a~%~]"
+            url (compression-type compression) file-size eris-urn)))
 
 (define* (narinfo-string store store-path
                          #:key (compressions (list %no-compression))
-                         (nar-path "nar") (file-sizes '()))
+                         (nar-path "nar") (file-sizes '()) (eris-urns '()))
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
 
 Optionally, FILE-SIZES is a list of compression/integer pairs, where the
 integer is size in bytes of the compressed NAR; it informs the client of how
-much needs to be downloaded."
+much needs to be downloaded.
+
+Optionally, ERIS-URNS is a list of compression/string pairs, where the
+string is the ERIS URN of the compressed NAR."
   (let* ((path-info  (query-path-info store store-path))
          (compressions (actual-compressions store-path compressions))
          (hash       (bytevector->nix-base32-string
@@ -352,9 +356,12 @@ (define* (narinfo-string store store-path
                              store-path
                              (map (lambda (compression)
                                     (let ((size (assoc-ref file-sizes
-                                                           compression)))
+                                                           compression))
+                                          (eris-urn (assoc-ref eris-urns
+                                                               compression)))
                                       (store-item->recutils store-path
                                                             #:file-size size
+                                                            #:eris-urn eris-urn
                                                             #:nar-path nar-path
                                                             #:compression
                                                             compression)))
@@ -632,6 +639,12 @@ (define (compressed-nar-size compression)
       (and stat
            (cons compression (stat:size stat)))))
 
+  (define (compressed-eris-urn compression)
+    (let* ((nar (nar-cache-file cache item #:compression compression))
+           (stat (stat nar #f)))
+      (and stat
+           (cons compression (call-with-input-file nar eris-encode->urn)))))
+
   (let ((compression (actual-compressions item compressions)))
 
     (for-each (cut compress-nar cache item <>) compressions)
@@ -646,11 +659,13 @@ (define (compressed-nar-size compression)
              ;; thread's connection to the store since we would end up sending
              ;; stuff concurrently on the same channel.
              (with-store store
-               (let ((sizes (filter-map compressed-nar-size compression)))
+               (let ((sizes (filter-map compressed-nar-size compression))
+                     (eris-urns (filter-map compressed-eris-urn compression)))
                  (display (narinfo-string store item
                                           #:nar-path nar-path
                                           #:compressions compressions
-                                          #:file-sizes sizes)
+                                          #:file-sizes sizes
+                                          #:eris-urns eris-urns)
                           port)))
 
              ;; Make the cached narinfo world-readable, contrary to what
-- 
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 17:20
[RFC PATCH 2/3] WIP: gnu: guile-eris: Update to unreleased git version.
(address . 52555@debbugs.gnu.org)
20211216162036.694-2-pukkamustard@posteo.net
* gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version.
[source]: Update source URI.
[propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi.
---
gnu/packages/guile-xyz.scm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

Toggle diff (35 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index 8346d99996..5e6f31a6e2 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -4325,15 +4325,15 @@ (define-public guile-sodium
 (define-public guile-eris
   (package
     (name "guile-eris")
-    (version "0.2.0")
+    (version "f1e4dd87988f9a80b05a8051d7f5ba3daf79dcc1")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
-             (url "https://inqlab.net/git/eris.git")
-             (commit (string-append "v" version))))
+             (url "https://inqlab.net/git/guile-eris.git")
+             (commit version)))
        (file-name (git-file-name name version))
-       (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka"))))
+       (sha256 (base32 "0kgm4b4qn2s74wjvxy273gdi1l1m81i2k4kkk1zc6vlcg3np7p06"))))
     (build-system gnu-build-system)
     (arguments '())
     (native-inputs
@@ -4345,7 +4345,7 @@ (define-public guile-eris
            guile-srfi-180))
     (inputs (list guile-3.0))
     (propagated-inputs
-     (list guile-sodium))
+     (list guile-sodium guile-json-4 guile-gdbm-ffi))
     (synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
     (description
      "Guile-ERIS is the reference implementation of the Encoding for Robust
-- 
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 17:20
[RFC PATCH 3/3] publish: Add IPFS support.
(address . 52555@debbugs.gnu.org)
20211216162036.694-3-pukkamustard@posteo.net
* guix/scripts/publish.scm: (show-help, %options): Add '--ipfs'.
(render-narinfo/cached, bake-narinfo+nar, make-request-handler, run-publish-server): Add #:ipfs? and honor it.
(guix-publish): Honor '--ipfs' and parameterize %ipfs-base-url.
---
guix/scripts/publish.scm | 52 +++++++++++++++++++++++++++++++---------
1 file changed, 41 insertions(+), 11 deletions(-)

Toggle diff (178 lines)
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 8e4b90789b..8e7fb47b9e 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -41,6 +41,8 @@ (define-module (guix scripts publish)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
+  #:use-module (srfi srfi-171)
   #:use-module (web http)
   #:use-module (web request)
   #:use-module (web response)
@@ -52,6 +54,7 @@ (define-module (guix scripts publish)
   #:use-module (guix base64)
   #:use-module (guix config)
   #:use-module (guix derivations)
+  #:use-module ((guix ipfs) #:prefix ipfs:)
   #:use-module (gcrypt hash)
   #:use-module (guix pki)
   #:use-module (gcrypt pk-crypto)
@@ -59,6 +62,8 @@ (define-module (guix scripts publish)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
   #:use-module (eris)
+  #:use-module (eris read-capability)
+  #:use-module (eris blocks ipfs)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -83,6 +88,7 @@ (define-module (guix scripts publish)
             run-publish-server
             guix-publish))
 
+
 (define (show-help)
   (format #t (G_ "Usage: guix publish [OPTION]...
 Publish ~a over HTTP.\n") %store-directory)
@@ -102,6 +108,8 @@ (define (show-help)
   (display (G_ "
       --cache-bypass-threshold=SIZE
                          serve store items below SIZE even when not cached"))
+  (display (G_ "
+      --ipfs[=GATEWAY]   publish items over IPFS via GATEWAY"))
   (display (G_ "
       --workers=N        use N workers to bake items"))
   (display (G_ "
@@ -220,6 +228,10 @@ (define %options
                 (lambda (opt name arg result)
                   (alist-cons 'cache-bypass-threshold (size->number arg)
                               result)))
+        (option '("ipfs") #f #t
+                (lambda (opt name arg result)
+                  (alist-cons 'ipfs (or arg (ipfs:%ipfs-base-url))
+                              result)))
         (option '("workers") #t #f
                 (lambda (opt name arg result)
                   (alist-cons 'workers (string->number* arg)
@@ -526,7 +538,7 @@ (define (bypass-cache? store item)
 (define* (render-narinfo/cached store request hash
                                 #:key ttl (compressions (list %no-compression))
                                 (nar-path "nar") negative-ttl
-                                cache pool)
+                                cache pool ipfs?)
   "Respond to the narinfo request for REQUEST.  If the narinfo is available in
 CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo
 requested using POOL."
@@ -571,7 +583,8 @@ (define (delete-entry narinfo)
                  (bake-narinfo+nar cache item
                                    #:ttl ttl
                                    #:compressions compressions
-                                   #:nar-path nar-path)))
+                                   #:nar-path nar-path
+                                   #:ipfs? ipfs?)))
 
              (when ttl
                (single-baker 'cache-cleanup
@@ -631,7 +644,7 @@ (define (write-compressed-file call-with-compressed-output-port)
 
 (define* (bake-narinfo+nar cache item
                            #:key ttl (compressions (list %no-compression))
-                           (nar-path "/nar"))
+                           (nar-path "/nar") ipfs?)
   "Write the narinfo and nar for ITEM to CACHE."
   (define (compressed-nar-size compression)
     (let* ((nar  (nar-cache-file cache item #:compression compression))
@@ -641,9 +654,19 @@ (define (compressed-nar-size compression)
 
   (define (compressed-eris-urn compression)
     (let* ((nar (nar-cache-file cache item #:compression compression))
-           (stat (stat nar #f)))
+           (stat (stat nar #f))
+           (block-reducer (if ipfs?
+                              (eris-blocks-ipfs-reducer
+                               #:ipfs-base-url (ipfs:%ipfs-base-url))
+                              rcount)))
       (and stat
-           (cons compression (call-with-input-file nar eris-encode->urn)))))
+           (cons compression
+                 (call-with-input-file nar
+                   (lambda (port)
+                     (let ((read-cap _
+                                     (eris-encode port #:block-reducer
+                                                  block-reducer)))
+                       (read-capability->string read-cap))))))))
 
   (let ((compression (actual-compressions item compressions)))
 
@@ -1115,7 +1138,8 @@ (define* (make-request-handler store
                                cache pool
                                narinfo-ttl narinfo-negative-ttl
                                (nar-path "nar")
-                               (compressions (list %no-compression)))
+                               (compressions (list %no-compression))
+                               ipfs?)
   (define compression-type?
     string->compression-type)
 
@@ -1147,7 +1171,8 @@ (define (handle request body)
                                       #:ttl narinfo-ttl
                                       #:negative-ttl narinfo-negative-ttl
                                       #:nar-path nar-path
-                                      #:compressions compressions)
+                                      #:compressions compressions
+                                      #:ipfs? ipfs?)
                (render-narinfo store request hash
                                #:ttl narinfo-ttl
                                #:negative-ttl narinfo-negative-ttl
@@ -1218,7 +1243,7 @@ (define* (run-publish-server socket store
                              advertise? port
                              (compressions (list %no-compression))
                              (nar-path "nar") narinfo-ttl narinfo-negative-ttl
-                             cache pool)
+                             cache pool ipfs?)
   (when advertise?
     (let ((name (service-name)))
       ;; XXX: Use a callback from Guile-Avahi here, as Avahi can pick a
@@ -1234,7 +1259,8 @@ (define* (run-publish-server socket store
                                     #:nar-path nar-path
                                     #:narinfo-ttl narinfo-ttl
                                     #:narinfo-negative-ttl narinfo-negative-ttl
-                                    #:compressions compressions)
+                                    #:compressions compressions
+                                    #:ipfs? ipfs?)
               concurrent-http-server
               `(#:socket ,socket)))
 
@@ -1296,6 +1322,8 @@ (define-command (guix-publish . args)
            (repl-port (assoc-ref opts 'repl))
            (cache     (assoc-ref opts 'cache))
            (workers   (assoc-ref opts 'workers))
+           (ipfs      (assoc-ref opts 'ipfs))
+           (ipfs?     (if ipfs #t #f))
 
            ;; Read the key right away so that (1) we fail early on if we can't
            ;; access them, and (2) we can then drop privileges.
@@ -1315,7 +1343,8 @@ (define-command (guix-publish . args)
                      (%private-key private-key)
                      (cache-bypass-threshold
                       (or (assoc-ref opts 'cache-bypass-threshold)
-                          (cache-bypass-threshold))))
+                          (cache-bypass-threshold)))
+                     (ipfs:%ipfs-base-url ipfs))
         (info (G_ "publishing ~a on ~a, port ~d~%")
               %store-directory
               (inet-ntop (sockaddr:fam address) (sockaddr:addr address))
@@ -1344,7 +1373,8 @@ (define-command (guix-publish . args)
                               #:nar-path nar-path
                               #:compressions compressions
                               #:narinfo-negative-ttl negative-ttl
-                              #:narinfo-ttl ttl))))))
+                              #:narinfo-ttl ttl
+                              #:ipfs? ipfs?))))))
 
 ;;; Local Variables:
 ;;; eval: (put 'single-baker 'scheme-indent-function 1)
-- 
2.34.0
L
L
Ludovic Courtès wrote on 18 Dec 2021 21:01
control message for bug #52555
(address . control@debbugs.gnu.org)
87czltof7d.fsf@gnu.org
severity 52555 important
quit
L
L
Ludovic Courtès wrote on 20 Dec 2021 17:25
Re: bug#52555: [RFC PATCH 0/3] Decentralized substitute distribution with ERIS
(name . pukkamustard)(address . pukkamustard@posteo.net)
87h7b3gs64.fsf@gnu.org
Hi pukkamustard,

pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (3 lines)
> This is an initial patch and proposal towards decentralizing substitute
> distribution with ERIS.

Woohoo, sounds exciting!

Toggle quote (10 lines)
> ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
> uniformly sized, encryped and content-addressed blocks. The original content
> can be reconstructed only with access to a read capability, which can be
> encoded as an URN.
>
> One key advantage of ERIS is that the encoding is protocol agnostic. Any
> protocol that can transfer small (32KiB) sized blocks referenced by the hash of
> their content will do. This can be done with things such as GNUNet, IPFS,
> OpenDHT, HTTP or a USB stick on a bicycle.

Yes, that’s nice.

Toggle quote (16 lines)
> The following patch allows substitutes to be published over IPFS using ERIS.
> This is inspired and very similar to previous work on distributing substitutes
> over IPFS [2].
>
> The narinfos served by `guix publish` look like this:
>
> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: gzip
> FileSize: 67363
> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: zstd
> FileSize: 64917
> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM

Do we really need one URN per compression method? Couldn’t we leave
compression (of individual chunks, possibly) as a “detail” handled by
the encoding or the transport layer?

Toggle quote (16 lines)
> If the `--ipfs` is used for `guix publish` then the encoded blocks are also
> uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
> this:
>
> (use-modules (eris)
> (eris blocks ipfs))
>
> (eris-decode->bytevector
> "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
> eris-blocks-ipfs-ref)
>
> These patches do not yet retrieve content from IPFS (TODO). But in principle,
> anybody connected to IPFS can get the nar with the ERIS URN. This could be used
> to reduce load on substitute server as they would only need to publish the ERIS
> URN directly - substitutes could be delivered much more peer-to-peer.

Nice. So adjusting ‘guix substitute’ should be relatively easy?

Toggle quote (7 lines)
> Other transports that I have been looking in to and am pretty sure will work
> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
> identifiers (URN) are abstracted away from specific transports (and also
> applications). ERIS is almost exactly the same encoding as used in GNUNet
> (ECRS).

As a first step, ‘guix publish’ could implement RFC 2169, too.

I gather implementing the HTTP and IPFS backends in ‘guix substitute’
should be relatively easy, right?

Toggle quote (6 lines)
> Blocks can be stored in any kind of databases (see for example the GDBM
> bindings [4]).
>
> A tricky things is figuring out how to multiplex all these different
> transports and storages...

Yes. We don’t know yet what performance and data availability will be
like on IPFS, for instance, so it’s important for users to be able to
set priorities. It’s also important to gracefully fall back to direct
HTTP downloads when fancier p2p methods fail, regardless of how they
fail.

Toggle quote (7 lines)
> The ERIS specification is still considered "experimental". However we feel
> confident to stabilize it and intend to do so around February/March 2022 with a
> release 1.0.0 of the specification. This will ensure that the identifiers
> remain stable for the forseeable future (until the crypto breaks). Before that
> there is also a small external security audit of the specification planned
> (thanks to NGI0/NLnet!).

Neat.

This is all very exciting. I look forward to playing around with it!

Ludo’.
P
P
pukkamustard wrote on 23 Dec 2021 12:42
(name . Ludovic Courtès)(address . ludo@gnu.org)
86bl17ms56.fsf@posteo.net
Hi Ludo,

Thanks for your comments!

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (15 lines)
>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> Compression: gzip
>> FileSize: 67363
>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> Compression: zstd
>> FileSize: 64917
>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>
> Do we really need one URN per compression method? Couldn’t we leave
> compression (of individual chunks, possibly) as a “detail” handled by
> the encoding or the transport layer?
>

I agree that it would be nice to leave this to the encoding layer as
that would allow certain optimizations (e.g. de-duplication).

Unfortunately, we haven't figured out yet what the most suitable
compression/format would be. Something like EROSFS seems good (as it
aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
just an archive format and there do not seem to be any libraries that we
could use to neatly integrate. It seems possible to block-align a Tar
archive, but that seems a bit hackey [2]. Other things to look into
might be Tarlz [3] and ZPAQ [4].

To get started I suggest just using one of the compressions/formats
already in Guix. zstd seems to be a reasonable choice (for the same
reasons why it makes sense to use zstd with `--discover` [5]).

Does that sound like a plan?


Toggle quote (18 lines)
>> If the `--ipfs` is used for `guix publish` then the encoded blocks are also
>> uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
>> this:
>>
>> (use-modules (eris)
>> (eris blocks ipfs))
>>
>> (eris-decode->bytevector
>> "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
>> eris-blocks-ipfs-ref)
>>
>> These patches do not yet retrieve content from IPFS (TODO). But in principle,
>> anybody connected to IPFS can get the nar with the ERIS URN. This could be used
>> to reduce load on substitute server as they would only need to publish the ERIS
>> URN directly - substitutes could be delivered much more peer-to-peer.
>
> Nice. So adjusting ‘guix substitute’ should be relatively easy?

Yes, relatively! :)

I meant to send in a V2 that does this before going on holidays, but I'm
afraid I won't make it. V2 will come in early January!

Toggle quote (12 lines)
>> Other transports that I have been looking in to and am pretty sure will work
>> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
>> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
>> identifiers (URN) are abstracted away from specific transports (and also
>> applications). ERIS is almost exactly the same encoding as used in GNUNet
>> (ECRS).
>
> As a first step, ‘guix publish’ could implement RFC 2169, too.
>
> I gather implementing the HTTP and IPFS backends in ‘guix substitute’
> should be relatively easy, right?

Yes, those seem to be the two easiest backends to implement.

Toggle quote (9 lines)
>> A tricky things is figuring out how to multiplex all these different
>> transports and storages...
>
> Yes. We don’t know yet what performance and data availability will be
> like on IPFS, for instance, so it’s important for users to be able to
> set priorities. It’s also important to gracefully fall back to direct
> HTTP downloads when fancier p2p methods fail, regardless of how they
> fail.

Agree.

Thanks,
-pukkamustard
L
L
Ludovic Courtès wrote on 24 Dec 2021 15:48
(name . pukkamustard)(address . pukkamustard@posteo.net)
87ilveukhr.fsf@gnu.org
Hi!

pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (28 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: gzip
>>> FileSize: 67363
>>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: zstd
>>> FileSize: 64917
>>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>>
>> Do we really need one URN per compression method? Couldn’t we leave
>> compression (of individual chunks, possibly) as a “detail” handled by
>> the encoding or the transport layer?
>>
>
> I agree that it would be nice to leave this to the encoding layer as
> that would allow certain optimizations (e.g. de-duplication).
>
> Unfortunately, we haven't figured out yet what the most suitable
> compression/format would be. Something like EROSFS seems good (as it
> aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
> just an archive format and there do not seem to be any libraries that we
> could use to neatly integrate. It seems possible to block-align a Tar
> archive, but that seems a bit hackey [2]. Other things to look into
> might be Tarlz [3] and ZPAQ [4].

Yeah. Though it may be that deduplication at the block level doesn’t
buy us much. That was the conclusion I reached a long time ago[a], and
also seems to be supported by the recent guix-daemon deduplication
improvements[b].


Toggle quote (6 lines)
> To get started I suggest just using one of the compressions/formats
> already in Guix. zstd seems to be a reasonable choice (for the same
> reasons why it makes sense to use zstd with `--discover` [5]).
>
> Does that sound like a plan?

Sure!

Toggle quote (3 lines)
> I meant to send in a V2 that does this before going on holidays, but I'm
> afraid I won't make it. V2 will come in early January!

Alright, we’ll see! :-)

Until then, enjoy your holidays!

Ludo’.
P
P
pukkamustard wrote on 25 Jan 20:21 +0100
[RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(address . 52555@debbugs.gnu.org)
20220125192201.7582-1-pukkamustard@posteo.net
Hello Guix,

Here comes the V2 of a proposal towards decentralizing substitute distribution
with ERIS.

A quick summary (as this has become quite long):

- This adds support for publishing and getting substitutes over IPFS.
- By using the ERIS encoding we are not limited to using IPFS as transport. We
can also use GNUNet, Named Data Networking (possibly) or just plain old HTTP. Support
for these can be added in (guix eris).
- These patches are still very rough and we need better logic for when to use
IPFS et. al. and when to fallback to HTTP.
- There might be performance issues when using IPFS via the IPFS daemon HTTP
API.

I found the setup for testing this a bit tricky. I will try and describe how I
have been testing it. Please let me know how this can be improved!

** Authorize local substitutes

We will be running a local substitute server so we need to add the local
signing key to the list of authorized keys. In the system configurations:

#+BEGIN_SRC scheme
(modify-services %base-services
(guix-service-type
config =>
(guix-configuration
(inherit config)
(authorized-keys
(cons*
;; allow substitutes from ourselves for testing purposes
(local-file "/etc/signing-key.pub")
%default-authorized-guix-keys)))))
#+END_SRC

** Configure the local Guix checkout

#+BEGIN_SRC shell
./bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc
#+END_SRC

The ~--sysconfdir~ is required so that guix will use the ACL in ~/etc/guix/acl~.

** Start the IPFS daemon

#+BEGIN_SRC shell
guix shell go-ipfs -- ipfs daemon
#+END_SRC

Start a local substitute server:

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix publish --public-key=/etc/guix/signing-key.pub --private-key=/etc/guix/signing-key.sec --cache=/tmp/guix-publish-cache/ --port=8081 --compression=zstd:19
#+END_SRC

We use port 8081 as IPFS is running on 8080.

We use the temporary cache directory ~/tmp/guix-publish-cache~.

** Build some package locally

First we build some package:

#+BEGIN_SRC shell
./pre-inst-env guix build hello --no-substitutes --no-offload
#+END_SRC

#+RESULTS:
: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11

** Trigger the substitute server to "bake" a susbtitute

#+BEGIN_SRC shell
#+END_SRC
Toggle snippet (10 lines)
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

If you do this again after a few seconds you will get a different response that
has the ERIS URN and the FileSizes. The reason for this is that Guix publish
bakes the nars asyncrhonisly in the background:

#+BEGIN_SRC shell
#+END_SRC

#+RESULTS:
Toggle snippet (13 lines)
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
FileSize: 57691
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
ERISFormat: application/x-nix-archive+zstd-19
ERIS: urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

These patches have added the ERIS and ERISFormat fields. Eventually we would
have figured out what the best format for use over ERIS is, for now we encode
it in the ERISFormat field.

** Removing a package from the store

This is necessary in order to make guix look for a substitute.

#+BEGIN_SRC shell
./pre-inst-env guix gc -D /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
#+END_SRC

** Start the Guix daemon from the repository

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild --debug --substitute-urls=http://localhost:8081/
#+END_SRC

Note this will probably stop your system Guix daemon. Run ~sudo herd restart
guix-daemon~ to restart it.

#+BEGIN_SRC shell
./pre-inst-env guix build hello
#+END_SRC
Toggle snippet (7 lines)
substituting /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11...
downloading from urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q ...
urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q 502KiB/s 00:00 | 56KiB transferred

/gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11

We have just retreived the substitute for the hello package from IPFS. Hello
decentralized substitutes!

I have only tested this for fairly small packages (up to a few MB).

One issue with IPFS might be that we have to create a new HTTP connection to
the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem
to support HTTP connection re-use and neither does the Guile (web client). I
fear this might become a performance issue. It seems possible to use IPFS more
directly by exposing the Go code as a C library and then using that with the
Guile FFI [1]. This is however a bit complicated and adds a lot of
dependencies. In particular, this should not become a dependency of Guix
itself. The performance of IPFS itself also needs to be evaluated, maybe the
IPFS HTTP API will not be the bottle-neck.

As mentioned in previous mail a simple HTTP transport for blocks would be a
good fallback. This would allow users to get missing blocks (things that
somehow got dropped from IPFS) directly from a substitute server. This is
different then getting the entire NAR from a substitute server. A user might be
missing a single 32KiB block and should be able to get only that. However, such
a HTTP fallback would also suffer from the one-connection-per-block issue. As
part of general ERIS research we are investigating CoAP as a better fallback
transport.

In any case, it would be necessary for the substitute server to store encoded
blocks of the NAR. For this I think it makes sense to use a small database. We
have bindings to use ERIS with GDBM [2]. It might also make sense to use
SQLite, especially if there are other use-cases for such a database.

I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
as transports.

Thanks for making it so far and happy hacking!
-pukkamustard



pukkamustard (5):
WIP: gnu: guile-eris: Update to unreleased git version.
publish: Add ERIS URN to narinfo
Add (guix eris).
publish: Add support for storing ERIS encoded blocks to IPFS.
substitute: Fetch substitutes using ERIS.

Makefile.am | 1 +
configure.ac | 5 +++
gnu/packages/guile-xyz.scm | 10 ++---
gnu/packages/package-management.scm | 1 +
guix/eris.scm | 60 +++++++++++++++++++++++++++++
guix/narinfo.scm | 14 +++++--
guix/scripts/publish.scm | 32 +++++++++++++--
guix/scripts/substitute.scm | 21 +++++++---
8 files changed, 126 insertions(+), 18 deletions(-)
create mode 100644 guix/eris.scm

--
2.34.0
P
P
pukkamustard wrote on 25 Jan 20:21 +0100
[RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo
(address . 52555@debbugs.gnu.org)
20220125192201.7582-3-pukkamustard@posteo.net
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
(narinfo-maker): Handle ERIS URN and ERIS format.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
configure.ac | 5 +++++
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 14 ++++++++++----
guix/scripts/publish.scm | 24 ++++++++++++++++++++----
4 files changed, 36 insertions(+), 8 deletions(-)

Toggle diff (167 lines)
diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
 AM_CONDITIONAL([HAVE_GUILE_AVAHI],
   [test "x$have_guile_avahi" = "xyes"])
 
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+  [test "x$have_guile_eris" = "xyes"])
+
 dnl Guile-newt is used by the graphical installer.
 GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
 
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 05795824b5..a9094b8b7f 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -404,6 +404,7 @@ (define code
                        ("guile-zstd" ,guile-zstd)
                        ("guile-ssh" ,guile-ssh)
                        ("guile-git" ,guile-git)
+                       ("guile-eris" ,guile-eris)
 
                        ;; XXX: Keep the development inputs here even though
                        ;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..a6a5d3b84b 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -45,6 +45,8 @@ (define-module (guix narinfo)
             narinfo-file-sizes
             narinfo-hash
             narinfo-size
+            narinfo-eris-format
+            narinfo-eris-urn
             narinfo-references
             narinfo-deriver
             narinfo-system
@@ -68,8 +70,8 @@ (define-module (guix narinfo)
 
 (define-record-type <narinfo>
   (%make-narinfo path uri-base uris compressions file-sizes file-hashes
-                 nar-hash nar-size references deriver system
-                 signature contents)
+                 nar-hash nar-size eris-format eris-urn references deriver
+                 system signature contents)
   narinfo?
   (path         narinfo-path)
   (uri-base     narinfo-uri-base)        ;URI of the cache it originates from
@@ -79,6 +81,8 @@ (define-record-type <narinfo>
   (file-hashes  narinfo-file-hashes)
   (nar-hash     narinfo-hash)
   (nar-size     narinfo-size)
+  (eris-format  narinfo-eris-format)
+  (eris-urn     narinfo-eris-urn)
   (references   narinfo-references)
   (deriver      narinfo-deriver)
   (system       narinfo-system)
@@ -135,7 +139,7 @@ (define (narinfo-maker str cache-url)
   "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
 must contain the original contents of a narinfo file."
   (lambda (path urls compressions file-hashes file-sizes
-                nar-hash nar-size references deriver system
+                nar-hash nar-size eris-format eris-urn references deriver system
                 signature)
     "Return a new <narinfo> object."
     (define len (length urls))
@@ -157,6 +161,8 @@ (define len (length urls))
                      ((lst ...) (map string->number lst)))
                    nar-hash
                    (and=> nar-size string->number)
+                   eris-format
+                   (if eris-urn (string->uri eris-urn) #f)
                    (string-tokenize references)
                    (match deriver
                      ((or #f "") #f)
@@ -184,7 +190,7 @@ (define* (read-narinfo port #:optional url
                    (narinfo-maker str url)
                    '("StorePath" "URL" "Compression"
                      "FileHash" "FileSize" "NarHash" "NarSize"
-                     "References" "Deriver" "System"
+                     "ERISFormat" "ERIS" "References" "Deriver" "System"
                      "Signature")
                    '("URL" "Compression" "FileSize" "FileHash"))))
 
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..9c83f5183d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -146,6 +147,9 @@ (define %default-gzip-compression
   ;; Since we compress on the fly, default to fast compression.
   (compression 'gzip 3))
 
+(define %eris-zstd-compression
+  (compression 'zstd 19))
+
 (define (default-compression type)
   (compression type 3))
 
@@ -324,7 +328,8 @@ (define* (store-item->recutils store-item
 
 (define* (narinfo-string store store-path
                          #:key (compressions (list %no-compression))
-                         (nar-path "nar") (file-sizes '()))
+                         (nar-path "nar") (file-sizes '())
+                         eris-urn)
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
@@ -347,7 +352,7 @@ (define* (narinfo-string store store-path
 StorePath: ~a
 ~{~a~}\
 NarHash: sha256:~a
-NarSize: ~d
+NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~]
 References: ~a~%"
                              store-path
                              (map (lambda (compression)
@@ -359,7 +364,7 @@ (define* (narinfo-string store store-path
                                                             #:compression
                                                             compression)))
                                   compressions)
-                             hash size references))
+                             hash size eris-urn references))
          ;; Do not render a "Deriver" line if we are rendering info for a
          ;; derivation.  Also do not render a "System" line that would be
          ;; expensive to compute and is currently unused.
@@ -632,6 +637,16 @@ (define (compressed-nar-size compression)
       (and stat
            (cons compression (stat:size stat)))))
 
+  (define (eris-encode-nar compressions)
+    (and (member %eris-zstd-compression compressions)
+         (let* ((nar (nar-cache-file cache item
+                                     #:compression %eris-zstd-compression))
+                (stat (stat nar #f)))
+           (and stat
+                (call-with-input-file nar
+                  (cut eris-encode->string <>
+                       #:block-size %eris-block-size-large))))))
+
   (let ((compression (actual-compressions item compressions)))
 
     (for-each (cut compress-nar cache item <>) compressions)
@@ -650,7 +665,8 @@ (define (compressed-nar-size compression)
                  (display (narinfo-string store item
                                           #:nar-path nar-path
                                           #:compressions compressions
-                                          #:file-sizes sizes)
+                                          #:file-sizes sizes
+                                          #:eris-urn (eris-encode-nar compression))
                           port)))
 
              ;; Make the cached narinfo world-readable, contrary to what
-- 
2.34.0
P
P
pukkamustard wrote on 25 Jan 20:21 +0100
[RFC PATCH v2 3/5] Add (guix eris).
(address . 52555@debbugs.gnu.org)
20220125192201.7582-4-pukkamustard@posteo.net
* guix/ipfs.scm: New file.
* Makefile.am (MODULES): Add it.
---
Makefile.am | 1 +
guix/eris.scm | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+)
create mode 100644 guix/eris.scm

Toggle diff (75 lines)
diff --git a/Makefile.am b/Makefile.am
index a10aeb817b..7219386361 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -131,6 +131,7 @@ MODULES =					\
   guix/cve.scm					\
   guix/workers.scm				\
   guix/ipfs.scm					\
+  guix/eris.scm					\
   guix/build-system.scm				\
   guix/build-system/android-ndk.scm		\
   guix/build-system/ant.scm			\
diff --git a/guix/eris.scm b/guix/eris.scm
new file mode 100644
index 0000000000..163bbe05ac
--- /dev/null
+++ b/guix/eris.scm
@@ -0,0 +1,55 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix eris)
+  #:use-module (eris)
+  #:use-module (eris blocks ipfs)
+  #:use-module (web client)
+  #:use-module (web response)
+  #:use-module (srfi srfi-71)
+
+  #:export (guix-eris-block-reducer))
+
+(define (ipfs-daemon-alive?)
+  "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
+and #f else."
+  (with-exception-handler
+      (const #f)
+    (lambda _
+      (let ((response _
+                      (http-post (string-append (%ipfs-base-url)
+                                                "/api/v0/version"))))
+        (equal? 200 (response-code response))))
+    #:unwind? #t))
+
+(define guix-eris-block-reducer
+  (case-lambda
+
+    ;; Check if IPFS Daemon is running.
+    (() (if (ipfs-daemon-alive?)
+            (eris-blocks-ipfs-reducer)
+            #f))
+
+    ;; Completion. Nothing to do.
+    ((_) #t)
+
+    ((ipfs ref-block)
+     ;; If IPFS has been initialized store block there
+     (if ipfs
+         (eris-blocks-ipfs-reducer ipfs ref-block)
+         ipfs))))
-- 
2.34.0
P
P
pukkamustard wrote on 25 Jan 20:22 +0100
[RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-5-pukkamustard@posteo.net
* guix/scripts/publish.scm (bake-narinfo+nar): Use guix-eris-block-reducer
from (guix eris).
---
guix/scripts/publish.scm | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)

Toggle diff (39 lines)
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 9c83f5183d..556107ab7d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -41,6 +41,7 @@ (define-module (guix scripts publish)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-71)
   #:use-module (web http)
   #:use-module (web request)
   #:use-module (web response)
@@ -58,6 +59,7 @@ (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (guix eris)
   #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
@@ -644,8 +646,14 @@ (define (eris-encode-nar compressions)
                 (stat (stat nar #f)))
            (and stat
                 (call-with-input-file nar
-                  (cut eris-encode->string <>
-                       #:block-size %eris-block-size-large))))))
+                  (lambda (port)
+                    (let ((eris-urn _
+                                    (eris-encode port
+                                                 #:block-size
+                                                 %eris-block-size-large
+                                                 #:block-reducer
+                                                 guix-eris-block-reducer)))
+                      (uri->string eris-urn))))))))
 
   (let ((compression (actual-compressions item compressions)))
 
-- 
2.34.0
P
P
pukkamustard wrote on 25 Jan 20:21 +0100
[RFC PATCH v2 1/5] WIP: gnu: guile-eris: Update to unreleased git version.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-2-pukkamustard@posteo.net
* gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version.
[source]: Update source URI.
[propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi.
---
gnu/packages/guile-xyz.scm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

Toggle diff (35 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index e2cf793acc..66ac486a74 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -4374,15 +4374,15 @@ (define-public guile-sodium
 (define-public guile-eris
   (package
     (name "guile-eris")
-    (version "0.2.0")
+    (version "bcbbcbc88f3ec1f2fafcd034ce5b620516bff105")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
-             (url "https://inqlab.net/git/eris.git")
-             (commit (string-append "v" version))))
+             (url "https://codeberg.org/eris/guile-eris")
+             (commit version)))
        (file-name (git-file-name name version))
-       (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka"))))
+       (sha256 (base32 "17v3h2hqx080739rl57nfradp5vlmy24fgqdxry1zal5z9d3i8sr"))))
     (build-system gnu-build-system)
     (arguments '())
     (native-inputs
@@ -4394,7 +4394,7 @@ (define-public guile-eris
            guile-srfi-180))
     (inputs (list guile-3.0))
     (propagated-inputs
-     (list guile-sodium))
+     (list guile-sodium guile-json-4 guile-gdbm-ffi))
     (synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
     (description
      "Guile-ERIS is the reference implementation of the Encoding for Robust
-- 
2.34.0
P
P
pukkamustard wrote on 25 Jan 20:22 +0100
[RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-6-pukkamustard@posteo.net
* guix/scripts/substitute.scm (process-substitution): Fetch substitutes using ERIS.
* guix/eris.scm (guix-eris-block-ref): New procedure.
---
guix/eris.scm | 7 ++++++-
guix/scripts/substitute.scm | 21 ++++++++++++++++-----
2 files changed, 22 insertions(+), 6 deletions(-)

Toggle diff (70 lines)
diff --git a/guix/eris.scm b/guix/eris.scm
index 163bbe05ac..0999564c1f 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -23,7 +23,8 @@ (define-module (guix eris)
   #:use-module (web response)
   #:use-module (srfi srfi-71)
 
-  #:export (guix-eris-block-reducer))
+  #:export (guix-eris-block-reducer
+            guix-eris-block-ref))
 
 (define (ipfs-daemon-alive?)
   "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
@@ -53,3 +54,7 @@ (define guix-eris-block-reducer
      (if ipfs
          (eris-blocks-ipfs-reducer ipfs ref-block)
          ipfs))))
+
+(define (guix-eris-block-ref ref)
+  "Dereference a block for decoding ERIS content"
+  (eris-blocks-ipfs-ref ref))
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 908a8334a8..852264976e 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -62,6 +62,8 @@ (define-module (guix scripts substitute)
   #:use-module (srfi srfi-35)
   #:use-module (web uri)
   #:use-module (guix http-client)
+  #:use-module (guix eris)
+  #:use-module (eris)
   #:export (%allow-unauthenticated-substitutes?
             %reply-file-descriptor
 
@@ -486,18 +488,27 @@ (define (fetch uri)
                          #:port port
                          #:keep-alive? #t
                          #:buffered? #f)))))
+
       (else
-       (leave (G_ "unsupported substitute URI scheme: ~a~%")
-              (uri->string uri)))))
+       (if (and (eris-read-capability? uri))
+           (values (eris-decode->port uri
+                                      #:block-ref
+                                      guix-eris-block-ref) #f)
+           (leave (G_ "unsupported substitute URI scheme: ~a~%")
+                  (uri->string uri))))))
+
+  (define* (best-uri narinfo #:key (eris? #f))
+    (if (and eris? (narinfo-eris-urn narinfo))
+        (values (narinfo-eris-urn narinfo) "zstd" #f)
+        (narinfo-best-uri narinfo #:fast-decompression?
+                          %prefer-fast-decompression?)))
 
   (unless narinfo
     (leave (G_ "no valid substitute for '~a'~%")
            store-item))
 
   (let-values (((uri compression file-size)
-                (narinfo-best-uri narinfo
-                                  #:fast-decompression?
-                                  %prefer-fast-decompression?)))
+                (best-uri narinfo #:eris? #t)))
     (unless print-build-trace?
       (format (current-error-port)
               (G_ "Downloading ~a...~%") (uri->string uri)))
-- 
2.34.0
M
M
Maxime Devos wrote on 29 Jan 22:00 +0100
Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(address . ~pukkamustard/eris@lists.sr.ht)
cc7ec17cded7fda6689d23a163df2ba72058a8aa.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (18 lines)
> ** Authorize local substitutes
>
> We will be running a local substitute server so we need to add the
> local
> signing key to the list of authorized keys. In the system
> configurations:
>
> #+BEGIN_SRC scheme
> (modify-services %base-services
> (guix-service-type config => [...]))
> #+END_SRC
> [...]
> ** Start the IPFS daemon
>
> #+BEGIN_SRC shell
> guix shell go-ipfs -- ipfs daemon
> #+END_SRC

There's an ipfs-service-type nowadays, so starting the daemon manually
isn't required (if using Guix System).

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWq7xccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7kWHAQD8B6ufMY7MntcR+Q44j+roW/MG
/b3y8wn1dPnDxb27cQEA+fgcZyuRGc458oFJKqyMP2xsoTGUYQmMC1Vc2aiYRwA=
=/y6d
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:08 +0100
(address . ~pukkamustard/eris@lists.sr.ht)
37651a21b0699b78b0282284b7d7322cb1a8c320.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (3 lines)
> I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
> as transports.

I have been writing a (Guile) Scheme port of GNUnet's client libraries
supported, but I'm working on DHT. DHT search/put already works to a
degree (see examples/web.scm), but there are plenty of sharp edges
(see TODOs about disconnecting, reconnecting and stopping fibers,
and see guix.scm for Guile bugs that are patched out and extra guile-
fibers features).

Tests are being written a edge cases will be addressed.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWs1BccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7iUHAP9GaXu5iS0pWHmlWrCS/xPjChSf
5aDwgq8ex1CbWgGslwEA2yz95+6uNajWeX9xbu6TV6b+AcAPwVUC4p2HgSvxBwA=
=tbfK
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:09 +0100
Re: [bug#52555] [RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo
(address . ~pukkamustard/eris@lists.sr.ht)
d0882404dc5153c7cf75846bb5cc70cb90138b78.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (5 lines)
> +dnl Check for Guile-eris.
> +GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
> +AM_CONDITIONAL([HAVE_GUILE_ERIS],
> +  [test "x$have_guile_eris" = "xyes"])

This could to be documented in (guix)Requirements.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWtJxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7qqUAQCEB1li7ZBR4r0xjFIQwDM+1Z2d
DmdnE6ogz16oO4WWjAD9HT5dzf21xMKPiGXXFNsOLwJKtc+2i4ZLOTTOvlCnTgk=
=ZFnB
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:15 +0100
(address . ~pukkamustard/eris@lists.sr.ht)
46558e7148b91f12cdaa92f32adb91bbcb3080bd.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (10 lines)
> +  (define (eris-encode-nar compressions)
> +    (and (member %eris-zstd-compression compressions)
> +         (let* ((nar (nar-cache-file cache item
> +                                     #:compression %eris-zstd-compression))
> +                (stat (stat nar #f)))
> +           (and stat
> +                (call-with-input-file nar
> +                  (cut eris-encode->string <>
> +                       #:block-size %eris-block-size-large))))))

Why are exceptions turned into #f (in (stat nar #f))?
Should this be done for all I/O errors, including, say, EOVERFLOW,
ENOMEM or ENAMETOOLONG, or only for ENOENT?

Is a race condition possible here? If so, maybe consider doing
something like

(catch 'system-error
(lambda () (call-with-input-file ...))
(lambda exception
(and it-is-a-ENOENT
(apply throw exception))))

to avoid it?

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWudxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7ka/AP0QTYuG8PJyBF3QVw+5ZnP0029l
Ax9ryDs/xHGvJ9rfgwEA40JgFHgUhz0M23qld+IUbNztFjshJjFr+Frzf69aow8=
=Imy3
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:23 +0100
Re: [bug#52555] [RFC PATCH v2 3/5] Add (guix eris).
(address . ~pukkamustard/eris@lists.sr.ht)
0b977ae15a49b051c04922d234149a8e4762404f.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (12 lines)
> +(define (ipfs-daemon-alive?)
> +  "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
> +and #f else."
> +  (with-exception-handler
> +      (const #f)
> +    (lambda _
> +      (let ((response _
> +                      (http-post (string-append (%ipfs-base-url)
> +                                                "/api/v0/version"))))
> +        (equal? 200 (response-code response))))
> +    #:unwind? #t))

This should preferably only be catching exceptions indicating that
the daemon is down (exceptions indicating 404s, or system-errors
indicating network errors, ...).

Toggle quote (18 lines)
> +
> +(define guix-eris-block-reducer
> +  (case-lambda
> +
> +    ;; Check if IPFS Daemon is running.
> +    (() (if (ipfs-daemon-alive?)
> +            (eris-blocks-ipfs-reducer)
> +            #f))
> +
> +    ;; Completion. Nothing to do.
> +    ((_) #t)
> +
> +    ((ipfs ref-block)
> +     ;; If IPFS has been initialized store block there
> +     (if ipfs
> +         (eris-blocks-ipfs-reducer ipfs ref-block)
> +         ipfs))))

This (ipfs-daemon-alive?) seems racy, although it's probably not.
Can we do

(define guix-eris-block-reducer
(case-lambda
(() (guard (c (oops-it-fails-because-the-daemon-cannot-be-
contacted? c)
#false)
(eris-block-ipfs-reducer))
[...]))

instead? (I don't think this will work as-is, because from the name and
thunkiness, it would appear that eris-block-ipfs-reducer returns a
procedure ...

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWwRBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7nxXAP9CAcbVz3Isk6TX52V5SGhjdExt
PLcXaJq7dAgpB44VfwEAqjgtVq1V7j2KDmmrZ9Om8e3E6+ys8okKmYBIMJia/QM=
=8qD5
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:24 +0100
(address . ~pukkamustard/eris@lists.sr.ht)
96a3137b8b746fcad9ef1eb75e1abfda5d01bfdb.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (3 lines)
> +  #:use-module (eris)
> +  #:use-module (eris blocks ipfs)

guile-eris is an optional dependency, so this needs to be autoloaded.
Or guix/eris.scm must only be compiled when guile-eris is available.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWwhRccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7tniAP9kASEY2aOmsyLv2tvwFXDDOimF
ogDnNWhDkbmvosQthAD/TR5q03yjYDu3Nv/gitR+b5d0ZeZS9eGYT+tQ4PI66A8=
=RBt/
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:28 +0100
Re: [bug#52555] [RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS.
(address . ~pukkamustard/eris@lists.sr.ht)
c6841903c177f934c7fc9d42a270907dbba59554.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (8 lines)
> +                    (let ((eris-urn _
> +                                    (eris-encode port
> +                                                 #:block-size
> +                                                 %eris-block-size-large
> +                                                 #:block-reducer
> +                                                 guix-eris-block-reducer)))
> + (uri->string eris-urn))))))))

IIUC (and quite probably I don't, because I've only being reading
things cursorly), eris-encode returns #false when the IPFS daemon is
down (because then guix-eris-block-reducer returns #false).

In that case, (uri->string eris-urn) = (uri->string #false) would
throw an exception:

scheme@(guile-user)> ((@ (web uri) uri->string) #false)
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
In procedure struct-vtable: Wrong type argument in position 1
(expecting struct): #f

Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue.
scheme@(guile-user) [1]> ,bt
In web/uri.scm:
336:17 1 (uri->string #f #:include-fragment? _)
In ice-9/boot-9.scm:
1669:16 0 (raise-exception _ #:continuable? _)

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWxaxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hHyAP9u5jfIW9GOQmUDoxjLJrBrB3cA
/zhz/MksltEGl16dtwD+M9uQeCUfT4Wqig+4eAxGYD952sqhmdmp8/9iI8W8ngM=
=b/X2
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:29 +0100
Re: [bug#52555] [RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(address . ~pukkamustard/eris@lists.sr.ht)
00bb85fa4fb62498b354ecb78d248df5209b6f29.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (4 lines)
> +(define (guix-eris-block-ref ref)
> +  "Dereference a block for decoding ERIS content"
> +  (eris-blocks-ipfs-ref ref))

'guix-eris-block-ref' just calls 'eris-blocks-ipfs-ref',
so I'm not seeing the point of this procedure.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWxzhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7uMzAP4u7P60iQCm0ohzXL8c0ZG6Zl0a
66kylFJ4J3x2BE+6UgEAgsWqyNKl0p3k/gg68VcHfsVnJMd0a+zy7V4houhF7As=
=iiau
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:33 +0100
(address . ~pukkamustard/eris@lists.sr.ht)
fbe6d8d556a796b580d0263cac87369bdc62b0d7.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (1 lines)
> +
Superfluous new empty line?

Toggle quote (5 lines)
>        (else
> -       (leave (G_ "unsupported substitute URI scheme: ~a~%")
> -              (uri->string uri)))))
> +       (if (and (eris-read-capability? uri))

guile-eris (which has the eris-read-capability? procedure) is an
optional dependency, so you have to check here is Guix was compiled
with guile-eris.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWyxxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7v7+AQCV4ylW2AvzZfYFWUufqDp6Fm2+
neoJm3Ip51Rj9gX/EgD9GWPVNtgpSMGfs3nbd9Vg2785Z3K0SsseF3FGI8o/sgA=
=1GlX
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:38 +0100
(address . ~pukkamustard/eris@lists.sr.ht)
0b0401a711d2196400b05a7f98ef5b757c99bf4b.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (6 lines)
> +  (define* (best-uri narinfo #:key (eris? #f))
> +    (if (and eris? (narinfo-eris-urn narinfo))
> +        (values (narinfo-eris-urn narinfo) "zstd" #f)
> +        (narinfo-best-uri narinfo #:fast-decompression?
> +                          %prefer-fast-decompression?)))

When Guix is compiled without guile-eris support,
'(and eris? (narinfo-eris-urn narinfo))' is the worst, not the best.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWz0xccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hfIAP4gQtcdPHk1ATJQ6Y2i1FdWxKRD
Hzwh+VxRPOOqAoleBQD+N+YpX65UT4K6zjdj7rmVzn/44mwqm+RRlGa2Kb0NeQg=
=9kGm
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:40 +0100
(address . ~pukkamustard/eris@lists.sr.ht)
d6bb147c902c0067d923321daf6d8db4a06fa537.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (6 lines)
> +  (define* (best-uri narinfo #:key (eris? #f))
> +    (if (and eris? (narinfo-eris-urn narinfo))
> +        (values (narinfo-eris-urn narinfo) "zstd" #f)
> +        (narinfo-best-uri narinfo #:fast-decompression?
> +                          %prefer-fast-decompression?)))

Why is ERIS the best here? Fast download speeds, decentralisation,
less network I/O, less heat production, more pronouncable than HTTPS?
I would add a comment here.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfW0ThccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rl8AP9JdJvZVUngfsqUuONq1psXpouT
FB5+4GT1KNZ8p15qwgEA0BH/56Hi+ezFF4RuMfpqYZkY/yF7KtM6j0a0va/qNgo=
=48HT
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 22:40 +0100
(address . ~pukkamustard/eris@lists.sr.ht)
e0156ce82b5e9a990d05c0b7bfb6626af5ce171a.camel@telenet.be
Maxime Devos schreef op za 29-01-2022 om 22:38 [+0100]:
Toggle quote (10 lines)
> pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
> > +  (define* (best-uri narinfo #:key (eris? #f))
> > +    (if (and eris? (narinfo-eris-urn narinfo))
> > +        (values (narinfo-eris-urn narinfo) "zstd" #f)
> > +        (narinfo-best-uri narinfo #:fast-decompression?
> > +                          %prefer-fast-decompression?)))
>
> When Guix is compiled without guile-eris support,
> '(and eris? (narinfo-eris-urn narinfo))' is the worst, not the best.

Nevermind, that's what (eris? #f) is for, I presume?

Greetings,
Maxime.
M
M
Maxime Devos wrote on 29 Jan 22:52 +0100
Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(address . ~pukkamustard/eris@lists.sr.ht)
73b50ffdca94407ef9fd7ef4875985a3b1c3c568.camel@telenet.be
Hi,

Is it possible for the following situation to happen?
If so, why not?

1. server A is authentic
2. server M is malicious, it tries to trick the client into
installing an incorrect substitute
3. (key of) server A is authorised
4. (key of) server M is _not_ authorised
5. server A and M are both in substitute-urls
6. server A only serves ‘classical’ substitutes, server B also serves
via ERIS+ipfs
7. Both A and M set the same FileHash, References, etc. in the
narinfo
8. However, M set an ERIS URN pointing to a backdoored substitute.
9. The client trusts A, and A and B have the same FileHash etc.,
so the client considers the narinfo of B to be authentic
because it has the same FileHash.
10. The client prefers ERIS above HTTP(S), so it downloads via M.
11. The client now installed a backdoored substitute!

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfW3KhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hJxAQDlxH9zz6F8LpkiMv07Hcyt35CG
YFb2CeCw2dbFO7qXNwD9HeDCIHVHzYQD/EvcGcYhQIocR7cUvrSnvPyFVHFErQ8=
=4RmQ
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 30 Jan 12:46 +0100
(address . ~pukkamustard/eris@lists.sr.ht)
ba490f009be9908063609f71800a2f00b4b1c1ae.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (6 lines)
> I have only tested this for fairly small packages (up to a few MB).
>
> One issue with IPFS might be that we have to create a new HTTP connection to
> the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem
> to support HTTP connection re-use

daemon supports connection reuse according to some people and doesn't
according to other people.

Toggle quote (2 lines)
> and neither does the Guile (web client).

Guix supports connection reuse, see 'call-with-cached-connection'
in (guix scripts substitute).

Toggle quote (2 lines)
> I fear this might become a performance issue.

IIUC, the performance problem primarily lies in the round-tripping
between the client and the server. If the client and the server are on
the same machine, then this round trip time is presumably small
compared to, say, localhost contacting ci.guix.gnu.org.

Still, connection reuse would be nice.

Toggle quote (6 lines)
> It seems possible to use IPFS more directly by exposing the Go code as a
> C library and then using that with the Guile FFI [1]. This is however a bit
> complicated and adds a lot of dependencies. In particular, this should not become
> a dependency of Guix itself. The performance of IPFS itself also needs to be
> evaluated, maybe the IPFS HTTP API will not be the bottle-neck.

Security-wise, libipfs doesn't seem great: libipfs starts the IPFS
daemon inside the process and guix/scripts/substitute.scm is run
as root.

Toggle quote (4 lines)
> As mentioned in previous mail a simple HTTP transport for blocks would be a
> good fallback. This would allow users to get missing blocks (things that
> somehow got dropped from IPFS) directly from a substitute server. [...]

Seems a good idea to me -- DHTs can be unreliable. I presume this will
be implemented with some kind of timeout: if no block is received
within N seconds, fallback to HTTP?

Also, don't forget to insert this missing block back into
IPFS/GNUnet/BitTorrent/..., otherwise less and less blocks will be
available until nothing is available anymore.

Toggle quote (5 lines)
> In any case, it would be necessary for the substitute server to store encoded
> blocks of the NAR. For this I think it makes sense to use a small database. We
> have bindings to use ERIS with GDBM [2]. It might also make sense to use
> SQLite, especially if there are other use-cases for such a database.

Wouldn't this be a huge database? IIRC, according to logs.guix.gnu.org
the size of the nars of the substitute servers are somewhere in the
200G-2T range or something like that.

To reduce the size of the database, perhaps you could let the database
be a mapping from block ids to the name of the nar + the position in
the nar, and encode the block on-demand?

The database doesn't seem necessary, the substitute server could have
some end-point

/publish-this-nar-again-into-IPFS/name-of-the-nar

which, when contacted, inserts the nar again into IPFS. Then when a
block was unavailable, the client contacts this end-point and retries.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfZ6jxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rANAP9328I61zxzX3OGMvtg/yulC2fN
hA2FYGhhmzeZUgJfWAD+Nz3A9TmanG1T0/QvW6bgstsy7kILYqCvXo7xF8onkAM=
=psyK
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 2 Feb 10:50 +0100
(name . Maxime Devos)(address . maximedevos@telenet.be)
86a6f9iofg.fsf@posteo.net
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (3 lines)
> There's an ipfs-service-type nowadays, so starting the daemon manually
> isn't required (if using Guix System).

Good point. Starting the daemon manually is only necessary if you don't
use the service. I don't use the IPFS service.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 10:56 +0100
(name . Maxime Devos)(address . maximedevos@telenet.be)
865ypxinr8.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (13 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
>> as transports.
>
> I have been writing a (Guile) Scheme port of GNUnet's client libraries
> (https://git.gnunet.org/gnunet-scheme.git/). Currently only NSE is
> supported, but I'm working on DHT. DHT search/put already works to a
> degree (see examples/web.scm), but there are plenty of sharp edges
> (see TODOs about disconnecting, reconnecting and stopping fibers,
> and see guix.scm for Guile bugs that are patched out and extra guile-
> fibers features).

Very interesting! I have been following your work on that a bit.

From what I understand gnunet-scheme interacts with the GNUNet services
and sends messages to the various GNUNet services. Is that correct?

Have you considered implementing the GNUNet protocols themeselves in
Guile? I.e. instead of connecting with the GNUNet services and sending
messages, implement R5N completely in Guile. IMHO this would be very
nice as one could use GNUNet protocols completely in Guile and not rely
on the GNUNet C code.

I believe this is somewhat the direction being taken with the GNUNet Go
implementation (https://github.com/bfix/gnunet-go)and also in line with
recent efforts to specify the individual GNUNet components and protocols
more independantly of one another (e.g. R5N is specified to work over IP

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 11:11 +0100
Re: [bug#52555] [RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(name . Maxime Devos)(address . maximedevos@telenet.be)
861r0linh1.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (9 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
>> +(define (guix-eris-block-ref ref)
>> +  "Dereference a block for decoding ERIS content"
>> +  (eris-blocks-ipfs-ref ref))
>
> 'guix-eris-block-ref' just calls 'eris-blocks-ipfs-ref',
> so I'm not seeing the point of this procedure.

Yes, currently it is an unnecessary level of abstraction.

The idea is that when there are multiple backends/transports they are
multiplexed here. E.g. guix-eris-block-ref would attempt to use
HTTP/IPFS or whatever to get the block. Whatever calls
guix-eris-block-ref does not need to know from where the blocks come.

I hope to make this more clear in a V3 that will add HTTP transport.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 11:16 +0100
Re: [bug#52555] [RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo
(name . Maxime Devos)(address . maximedevos@telenet.be)
86wnidh8ht.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (26 lines)
> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> +  (define (eris-encode-nar compressions)
>> +    (and (member %eris-zstd-compression compressions)
>> +         (let* ((nar (nar-cache-file cache item
>> +                                     #:compression %eris-zstd-compression))
>> +                (stat (stat nar #f)))
>> +           (and stat
>> +                (call-with-input-file nar
>> +                  (cut eris-encode->string <>
>> +                       #:block-size %eris-block-size-large))))))
>
> Why are exceptions turned into #f (in (stat nar #f))?
> Should this be done for all I/O errors, including, say, EOVERFLOW,
> ENOMEM or ENAMETOOLONG, or only for ENOENT?
>
> Is a race condition possible here? If so, maybe consider doing
> something like
>
> (catch 'system-error
> (lambda () (call-with-input-file ...))
> (lambda exception
> (and it-is-a-ENOENT
> (apply throw exception))))
>
> to avoid it?

A valid question. But (stat nar #f) is not something I introduced. It is
already in guix/scripts/publish.scm like that.

To me turning all exceptions to #f makes sense. Here we only want to
know if the file is readable. As the NAR is baked in the background by
another thread the case where the compressed NAR does exist yet will
happen. In that case we don't worry, we just don't publish the
`FileSize` and `ERIS` fields in the narinfo.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 11:28 +0100
Re: [bug#52555] [RFC PATCH v2 3/5] Add (guix eris).
(name . Maxime Devos)(address . maximedevos@telenet.be)
86sft1h7wh.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (17 lines)
> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> +(define (ipfs-daemon-alive?)
>> +  "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
>> +and #f else."
>> +  (with-exception-handler
>> +      (const #f)
>> +    (lambda _
>> +      (let ((response _
>> +                      (http-post (string-append (%ipfs-base-url)
>> +                                                "/api/v0/version"))))
>> +        (equal? 200 (response-code response))))
>> +    #:unwind? #t))
>
> This should preferably only be catching exceptions indicating that
> the daemon is down (exceptions indicating 404s, or system-errors
> indicating network errors, ...).

Yes, I guess it could be checked a bit finer. But at the end if an
exception happens then the IPFS daemon is probably not reachable, right?
If we don't care about the reason why it is not reachable then why
bother with catching finer grained exceptions?

Toggle quote (33 lines)
>> +
>> +(define guix-eris-block-reducer
>> +  (case-lambda
>> +
>> +    ;; Check if IPFS Daemon is running.
>> +    (() (if (ipfs-daemon-alive?)
>> +            (eris-blocks-ipfs-reducer)
>> +            #f))
>> +
>> +    ;; Completion. Nothing to do.
>> +    ((_) #t)
>> +
>> +    ((ipfs ref-block)
>> +     ;; If IPFS has been initialized store block there
>> +     (if ipfs
>> +         (eris-blocks-ipfs-reducer ipfs ref-block)
>> +         ipfs))))
>
> This (ipfs-daemon-alive?) seems racy, although it's probably not.
> Can we do
>
> (define guix-eris-block-reducer
> (case-lambda
> (() (guard (c (oops-it-fails-because-the-daemon-cannot-be-
> contacted? c)
> #false)
> (eris-block-ipfs-reducer))
> [...]))
>
> instead? (I don't think this will work as-is, because from the name and
> thunkiness, it would appear that eris-block-ipfs-reducer returns a
> procedure ...

Yes, eris-block-ipfs-reducer returns and SRFI-171 reducer. This is a
3-arity procedure that is either initialized, called with a block to
reduce and finalized.

The #f that the initialization case returns (0-arity call) is the state
of the reducer. In the block reducing case (2-ary call) the state (the
ipfs variable) is checked if ipfs is alive. If not the blocks are just
forgotten.

So guix-eris-block-reducer always returns a SRF-171 reducer, regardless
of if IPFS is alive or not. This is important as the ERIS URN can still
be computed without the IPFS daemon running.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 11:24 +0100
Re: [bug#52555] [RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS.
(name . Maxime Devos)(address . maximedevos@telenet.be)
86o83ph7uz.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:
Toggle quote (13 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
>> +                    (let ((eris-urn _
>> +                                    (eris-encode port
>> +                                                 #:block-size
>> +                                                 %eris-block-size-large
>> +                                                 #:block-reducer
>> +                                                 guix-eris-block-reducer)))
>> + (uri->string eris-urn))))))))
>
> IIUC (and quite probably I don't, because I've only being reading
> things cursorly), eris-encode returns #false when the IPFS daemon is
> down (because then guix-eris-block-reducer returns #false).
No, eris-encode will still return the ERIS URN. The blocks will just not
be stored in IPFS if the IPFS daemon is not running. See also my
response to the questions on guix-eris-block-reducer in (guix eris).
-pukkamustard
P
P
pukkamustard wrote on 2 Feb 11:38 +0100
Re: [bug#52555] [RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(name . Maxime Devos)(address . maximedevos@telenet.be)
86k0edh79j.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (12 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
>> +  (define* (best-uri narinfo #:key (eris? #f))
>> +    (if (and eris? (narinfo-eris-urn narinfo))
>> +        (values (narinfo-eris-urn narinfo) "zstd" #f)
>> +        (narinfo-best-uri narinfo #:fast-decompression?
>> +                          %prefer-fast-decompression?)))
>
> Why is ERIS the best here? Fast download speeds, decentralisation,
> less network I/O, less heat production, more pronouncable than HTTPS?
> I would add a comment here.

Those are all possible reasons. I think we first need to do some
experiments to see if any of those claims can be justified.

In general, the logic for when to use ERIS transports needs more
thought.

For one, I think it should be user configurable. I can imagine that
certain users do not want to use decentralized substitutes at all. Users
should be able to deactivate the entire ERIS thing.

For the default, I personally think it would be ok to try and use
ERIS. The only thing we absolutely need is a clean fallback logic that
transparently falls back to getting the entire NAR by HTTP. Currently
such a fallback is not yet implemented.

-pukkamustard
M
M
Maxime Devos wrote on 2 Feb 12:09 +0100
Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(name . pukkamustard)(address . pukkamustard@posteo.net)
3d9821d7eac6c81a5b1c8a5304a78ec733b2dba4.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 09:56 [+0000]:
Toggle quote (20 lines)
> Maxime Devos <maximedevos@telenet.be> writes:
>
> > [[PGP Signed Part:Undecided]]
> > pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
> > > I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
> > > as transports.
> >
> > I have been writing a (Guile) Scheme port of GNUnet's client libraries
> > (https://git.gnunet.org/gnunet-scheme.git/). Currently only NSE is
> > supported, but I'm working on DHT. DHT search/put already works to a
> > degree (see examples/web.scm), but there are plenty of sharp edges
> > (see TODOs about disconnecting, reconnecting and stopping fibers,
> > and see guix.scm for Guile bugs that are patched out and extra guile-
> > fibers features).
>
> Very interesting! I have been following your work on that a bit.
>
> From what I understand gnunet-scheme interacts with the GNUNet services
> and sends messages to the various GNUNet services. Is that correct?

Yes, it works like the C GNUnet client libraries, except it's in Guile
Scheme and a few different design decisions were made, e.g. w.r.t.
concurrency.

Toggle quote (4 lines)
> Have you considered implementing the GNUNet protocols themeselves in
> Guile? I.e. instead of connecting with the GNUNet services and sending
> messages, implement R5N completely in Guile.

I didn't, at least not _yet_. As-is, things are already complicated
enough and the client code seems a lot simpler than the service code.
Though perhaps in the future ...

E.g., for testing the DHT service, the test code effectively creates a
tiny, limited, in-memory DHT service (not communicating to any peers)
that's buggy in some respects (not yet committed, but will be in
tests/distributed-hash-table.scm).

Toggle quote (3 lines)
> IMHO this would be very nice as one could use GNUNet protocols completely
> in Guile and not rely on the GNUNet C code.

While it's not a priority, I'm not opposed to someday implementing the
services in Guile and testing whether they can communicate with C
peers.

However, keep in mind that GNUnet is supposed to be able to eventually
replace the TCP/IP stack, and running the DHT, NSE, NAT, FS, CADET,
TRANSPORT ... services in every web browser, in every mail client, in
all "guix substitute" and "guix perform-download" processes, etc. is
rather wasteful (memory-wise and CPU-wise), so I'd prefer this not to
be the _default_ option.

(I'm not sure if you were referring to that.)

Greetings,
Maxme.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfpmXRccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7mEeAPwOAn/O9DonokYiuo2Yq5+oLnHK
fIr8apEPKdTzy/O5sAD/TeShsCEUX0+DdtJF1xDXsYY4uRDd1ubLC/dowbvehA8=
=I6pm
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 2 Feb 11:51 +0100
(name . Maxime Devos)(address . maximedevos@telenet.be)
86fsp1h6ce.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (12 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> I have only tested this for fairly small packages (up to a few MB).
>>
>> One issue with IPFS might be that we have to create a new HTTP connection to
>> the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem
>> to support HTTP connection re-use
>
> According to <https://github.com/ipfs/go-ipfs/issues/3767>, the IPFS
> daemon supports connection reuse according to some people and doesn't
> according to other people.

Hm, from what I understand connection re-use is something introduced in
HTTP/2 and go-ipfs does not do HTTP/2

Toggle quote (5 lines)
>> and neither does the Guile (web client).
>
> Guix supports connection reuse, see 'call-with-cached-connection'
> in (guix scripts substitute).

Ah ok. Cool!

Toggle quote (9 lines)
>> I fear this might become a performance issue.
>
> IIUC, the performance problem primarily lies in the round-tripping
> between the client and the server. If the client and the server are on
> the same machine, then this round trip time is presumably small
> compared to, say, localhost contacting ci.guix.gnu.org.
>
> Still, connection reuse would be nice.

Remains to be seen if this is a problem.

It is considerably more pronounced than with regular usage of IPFS as we
make a HTTP request to IPFS for every 32KiB block instead of for an
entire file (what most people do when using the IPFS daemon).

Toggle quote (10 lines)
>> It seems possible to use IPFS more directly by exposing the Go code as a
>> C library and then using that with the Guile FFI [1]. This is however a bit
>> complicated and adds a lot of dependencies. In particular, this should not become
>> a dependency of Guix itself. The performance of IPFS itself also needs to be
>> evaluated, maybe the IPFS HTTP API will not be the bottle-neck.
>
> Security-wise, libipfs doesn't seem great: libipfs starts the IPFS
> daemon inside the process and guix/scripts/substitute.scm is run
> as root.

I agree.

Toggle quote (8 lines)
>> As mentioned in previous mail a simple HTTP transport for blocks would be a
>> good fallback. This would allow users to get missing blocks (things that
>> somehow got dropped from IPFS) directly from a substitute server. [...]
>
> Seems a good idea to me -- DHTs can be unreliable. I presume this will
> be implemented with some kind of timeout: if no block is received
> within N seconds, fallback to HTTP?

Yes, exactly.

Toggle quote (4 lines)
> Also, don't forget to insert this missing block back into
> IPFS/GNUnet/BitTorrent/..., otherwise less and less blocks will be
> available until nothing is available anymore.

This might be a bit of a burden for users. As you mention the size of
such a database might become considerable.

Toggle quote (13 lines)
>> In any case, it would be necessary for the substitute server to store encoded
>> blocks of the NAR. For this I think it makes sense to use a small database. We
>> have bindings to use ERIS with GDBM [2]. It might also make sense to use
>> SQLite, especially if there are other use-cases for such a database.
>
> Wouldn't this be a huge database? IIRC, according to logs.guix.gnu.org
> the size of the nars of the substitute servers are somewhere in the
> 200G-2T range or something like that.
>
> To reduce the size of the database, perhaps you could let the database
> be a mapping from block ids to the name of the nar + the position in
> the nar, and encode the block on-demand?

Yes! I've also been thinking of this - a "in-file" block store. I think
this makes a lot of sense for Guix but also other things (e.g. sharing
your music collection).

Another problem with IPFS/GNUNet is that they have their own storage. So
even if are clever about storing blocks in Guix, IPFS and GNUNet will
have their own copy of the blocks on disk. I think it would be much
nicer if DHTs/transport layers don't do block storage but are provided
with a callback from where they can get stored blocks. I believe this is
what OpenDHT does

I think we should propose such a change to the GNUNet R5N specification

Toggle quote (8 lines)
> The database doesn't seem necessary, the substitute server could have
> some end-point
>
> /publish-this-nar-again-into-IPFS/name-of-the-nar
>
> which, when contacted, inserts the nar again into IPFS. Then when a
> block was unavailable, the client contacts this end-point and retries.

But for a HTTP block endpoint we would still need such a database/block
storage.

I think it is important that we do not rely on IPFS for block
storage. The decentralized block distribution should work even if the
IPFS daemon is not available.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 12:10 +0100
(name . Maxime Devos)(address . maximedevos@telenet.be)
86bkzph5ux.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (26 lines)
> [[PGP Signed Part:Undecided]]
> Hi,
>
> Is it possible for the following situation to happen?
> If so, why not?
>
> 1. server A is authentic
> 2. server M is malicious, it tries to trick the client into
> installing an incorrect substitute
> 3. (key of) server A is authorised
> 4. (key of) server M is _not_ authorised
> 5. server A and M are both in substitute-urls
> 6. server A only serves ‘classical’ substitutes, server B also serves
> via ERIS+ipfs
> 7. Both A and M set the same FileHash, References, etc. in the
> narinfo
> 8. However, M set an ERIS URN pointing to a backdoored substitute.
> 9. The client trusts A, and A and B have the same FileHash etc.,
> so the client considers the narinfo of B to be authentic
> because it has the same FileHash.
> 10. The client prefers ERIS above HTTP(S), so it downloads via M.
> 11. The client now installed a backdoored substitute!
>
> Greetings,
> Maxime.

No this should not work.

The ERIS URN is only used if the entire narinfo is signed with a
authorized signature. The FileHash is not used when getting substitutes
via ERIS (being able to decode ERIS content implies integrity).

The interesting case that would be allowed with ERIS is following:

1. Server A is authentic and its key is authorized.
2. Servers M1 to MN are potentially malicious and their keys are not
authorized.
3. Server A and servers M1 to MN are in the substitute-urls.
4. Client gets Narinfo from server A and uses the ERIS URN from there.
5. Client can get blocks simultaneously from Server A and servers M1 to
MN.
6. Client decodes content with the ERIS URN and can be sure that they
have the valid substitute.

So client only needs to trust A but can use M1-MN (simultaneously) for
fetching the content.

-pukkamustard
M
M
Maxime Devos wrote on 2 Feb 12:27 +0100
(name . pukkamustard)(address . pukkamustard@posteo.net)
846716544b4424f02e383114ebcb52957b43dd4d.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 10:51 [+0000]:
Toggle quote (19 lines)
> > The database doesn't seem necessary, the substitute server could
> > have
> > some end-point
> >
> >    /publish-this-nar-again-into-IPFS/name-of-the-nar
> >
> > which, when contacted, inserts the nar again into IPFS.  Then when
> > a
> > block was unavailable, the client contacts this end-point and
> > retries.
>
> But for a HTTP block endpoint we would still need such a
> database/block
> storage.
>
> I think it is important that we do not rely on IPFS for block
> storage. The decentralized block distribution should work even if the
> IPFS daemon is not available.

Do we need a database at all?

E.g., if the client cannot download the data in the range [start, end]
because the corresponding block has disappeared, can it not simply
download that range from https://ci.guix.gnu.org/nar/[...]
(not sure about the URI) using a HTTP range request?

(Afterwards, the client should insert the block(s) back into
IPFS/GNUnet/whatever, maybe using this proposed ‘in-file block store’
such that other clients (using the same DHT mechanism) can benefit.)

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfpqjxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7q/tAQCx5U1HlZF9H7rfUztb00awEgdY
KRhb7frTXDXhQfzGwwD/Ro+r9CPRGyK2EQ1AqDDFgZUohQQrKHbAzL8yY+DG/Qk=
=93De
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 2 Feb 13:42 +0100
(name . Maxime Devos)(address . maximedevos@telenet.be)
8635l1h1mx.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (11 lines)
>> I think it is important that we do not rely on IPFS for block
>> storage. The decentralized block distribution should work even if the
>> IPFS daemon is not available.
>
> Do we need a database at all?
>
> E.g., if the client cannot download the data in the range [start, end]
> because the corresponding block has disappeared, can it not simply
> download that range from https://ci.guix.gnu.org/nar/[...]
> (not sure about the URI) using a HTTP range request?

This does not work as the mapping from block reference to location in
NAR can not be known by the client who only holds the ERIS
URN. Furthermore, some blocks will be intermediary nodes - they hold
references to content blocks (or other intermediary nodes) but not
content itself.

Toggle quote (4 lines)
> (Afterwards, the client should insert the block(s) back into
> IPFS/GNUnet/whatever, maybe using this proposed ‘in-file block store’
> such that other clients (using the same DHT mechanism) can benefit.)

It might make sense for some clients to make content available to other
clients and to go trough the extra effort of putting blocks back into
IPFS/GNUNet/whatever. But this should be optional. Maybe we can call
such clients "caching peers"?

IMO A client should by default only deal with things that are strictly
necessary for getting substitutes. The substistute servers (and caching
peers) should make sure substitutes are available to clients, whether
over IPFS/GNUNet/whatever or plain old HTTP.

-pukkamustard
M
M
Maxime Devos wrote on 2 Feb 16:07 +0100
(name . pukkamustard)(address . pukkamustard@posteo.net)
45e1dd2f05c6733eb74792af07194347fdd55ebd.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 12:42 [+0000]:
Toggle quote (19 lines)
> > (Afterwards, the client should insert the block(s) back into
> > IPFS/GNUnet/whatever, maybe using this proposed ‘in-file block
> > store’
> > such that other clients (using the same DHT mechanism) can
> > benefit.)
>
> It might make sense for some clients to make content available to
> other
> clients and to go trough the extra effort of putting blocks back into
> IPFS/GNUNet/whatever. But this should be optional. Maybe we can call
> such clients "caching peers"?
>
> IMO A client should by default only deal with things that are
> strictly
> necessary for getting substitutes. The substistute servers (and
> caching
> peers) should make sure substitutes are available to clients, whether
> over IPFS/GNUNet/whatever or plain old HTTP.

If re-inserting missing blocks back into the IPFS/GNUnet/whatever is
made optional and is off by default, then almost nobody will enable the
‘caching peer’ option and we will have freeloaders, somewhat defeating
the point of GNUnet/whatever.

In a classic setting (‘plain old HTTP’), serving and downloading is a
separate thing. But in a P2P setting, downloading cannot be separated
from uploading -- strictly speaking, a peer might be able to download
without uploading (depending on the P2P system), but that's anti-
social, not something that should be done by default.

However, if re-inserting missing blocks is _on_ by default, then there
doesn't seem to be any trouble.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfqeNBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hI+AQCyLDdDhhLnScMBo6MNl55F+DKo
Yaje3fToELjEOBmOUAD/YOvJCAsYd7BILqDAImAt4T8s+UnANcKzVtbWKXbklwc=
=ttyT
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 2 Feb 16:27 +0100
(name . pukkamustard)(address . pukkamustard@posteo.net)
84d930adac8c54120b176362e66a63ebf49179c6.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 12:42 [+0000]:
Toggle quote (10 lines)
> > E.g., if the client cannot download the data in the range [start,
> > end]
> > because the corresponding block has disappeared, can it not simply
> > download that range from https://ci.guix.gnu.org/nar/[...]
> > (not sure about the URI) using a HTTP range request?
>
> This does not work as the mapping from block reference to location in
> NAR can not be known by the client who only holds the ERIS
> URN.

The client not only knows the ERIS URN, it also knows the location of
the nar (over classical HTTP) because it's in the narinfo.

Toggle quote (4 lines)
> Furthermore, some blocks will be intermediary nodes - they hold
> references to content blocks (or other intermediary nodes) but not
> content itself.

If an intermediary node (responsible for, say, bytes 900--10000)
is missing, then the bytes 900--10000 could be downloaded via HTTP.
Whether the node is close to the top, or close to the bottom, in ERIS'
variant of Merkle trees, doesn't matter much.

Granted, if the nar is, say, 1 GiB, and the top-level block is missing,
then we'll have to download 1 GiB over HTTP, even if most lower blocks
exist on IPFS/GNUnet/whatever, which isn't really great.

We could also do some combination of the GDBM database and HTTP
Content-Range requests: most nodes are leaf nodes (*). Instead of
representing all nodes in the database, we could include only
(intermediate) nodes responsible for data of size, say, 4MiB.

(*) At least, that's the case for binary trees, presumably something
similar holds for ERIS.

I don't know the specifics for ERIS, but for (balanced) binary trees,
not storing the leaf nodes would save about 50% (**), which is a rather
nice space saving.

(**) This assumes the ‘block size’ is the size for storing two pointers
to the children, but in practice the block size would be quite a bit
larger, so there would be more space savings?

Perhaps we are overthinking things and the GDBM (***) database isn't
overly large, or perhaps missing blocks are sufficiently rare such that
we could simply download the _entire_ nar from classical HTTP in case
of missing blocks ...

(***) Guix uses SQlite databases, so I would use SQLite instead of GDBM
unless there's a compelling reason to use GDBM instead.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfqi+BccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7luwAP4x9bFYhc+pM+oN9Jed5nNRDUSl
xTW+PIkxKhsPSSkOtAD8DtBwnt0Ju57kJ/OTzouZ0ZwGJRzBUCZ1bkRntYZZPg4=
=r/kQ
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 2 Feb 16:36 +0100
Re: [bug#52555] [RFC PATCH v2 3/5] Add (guix eris).
(name . pukkamustard)(address . pukkamustard@posteo.net)
26bc58741a60c972226cd47832b27de955e4a57a.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 10:28 [+0000]:
Toggle quote (25 lines)
> > pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
> > > +(define (ipfs-daemon-alive?)
> > > +  "Attempt to connect to the IPFS daemon. Returns #t if the
> > > daemon is alive
> > > +and #f else."
> > > +  (with-exception-handler
> > > +      (const #f)
> > > +    (lambda _
> > > +      (let ((response _
> > > +                      (http-post (string-append (%ipfs-base-url)
> > > +                                               
> > > "/api/v0/version"))))
> > > +        (equal? 200 (response-code response))))
> > > +    #:unwind? #t))
> >
> > This should preferably only be catching exceptions indicating that
> > the daemon is down (exceptions indicating 404s, or system-errors
> > indicating network errors, ...).
>
> Yes, I guess it could be checked a bit finer. But at the end if an
> exception happens then the IPFS daemon is probably not reachable,
> right?
> If we don't care about the reason why it is not reachable then why
> bother with catching finer grained exceptions?

The exception could be caused by, say:

* an unbound variable
* wrong arity
* type error
* stack overflow
* prompt tag does not exist in current environment
* out of memory

Except for the last one, these causes are all bugs and hence shouldn't
be surpressed. Granted, this is a bit unlikely since this use of
'http-post' is very simple, but it's far from impossible for
(web client) to have a bug.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfqlGBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7lMuAQCTlCl2Q5LRrRQiIKgiwuimw1Sm
NXVCP9Pf/eOoQZMWAAD+NpY1trIo+TZLI+tmUcKBARROcYliOgUMS0RgLj1l3Qk=
=sgcN
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 3 Feb 21:36 +0100
Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(name . pukkamustard)(address . pukkamustard@posteo.net)
52ee517f75c66a8fd9e9823da016b5720b4d5d34.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 11:10 [+0000]:
Toggle quote (3 lines)
> The ERIS URN is only used if the entire narinfo is signed with a
> authorized signature.

Perhaps I'm missing something here, but in that case, shouldn't "ERIS"
be added to %mandatory-fields in (guix narinfo)?

Anyway, I don't see what prevents an unauthorised narinfo with a ERIS
URN to be used: the narinfo is chosen with

(define narinfo
(lookup-narinfo cache-urls store-item
(if (%allow-unauthenticated-substitutes?)
(const #t)
(cut valid-narinfo? <> acl))))

where lookup-narinfo is a tiny wrapper around lookup-narinfos/diverse.
lookup-narinfos/diverse considers both unauthorised and authorised
narinfos, and can choose an unauthorised narinfo if it's ‘equivalent’
to an authorised narinfo (using equivalent-narinfo?)

equivalent-narinfo? only looks at the hash, path, references and size,
and ignores the ERIS. As such, an unauthorised narinfo with a
malicious ERIS URN could be selected.

However, it turns out that all this doesn't really matter: whether the
port returned by 'fetch' in (guix scripts substitute) came from
file://, http://, https://or ERIS, the file hash is verified later
anyway:

;; Compute the actual nar hash as we read it.
((algorithm expected)
(narinfo-hash-algorithm+value narinfo))
((hashed get-hash)
(open-hash-input-port algorithm input)))

[...]

;; Check whether we got the data announced in NARINFO.
(let ((actual (get-hash)))
(if (bytevector=? actual expected)
[...]

False alarm I guess!

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfw80hccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7s/dAQDJWPpPuJCxdqcJv0tvWer5fKY/
Hr1YGeNT61UHexvazwD/X4/mwomnod+8Urvl9nhTeLUOa8Hs4iZ2tRYZR9nJ7wA=
=w/KP
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 4 Feb 11:20 +0100
(name . Maxime Devos)(address . maximedevos@telenet.be)
86tudeyknu.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (25 lines)
> pukkamustard schreef op wo 02-02-2022 om 11:10 [+0000]:
>> The ERIS URN is only used if the entire narinfo is signed with a
>> authorized signature.
>
> Perhaps I'm missing something here, but in that case, shouldn't "ERIS"
> be added to %mandatory-fields in (guix narinfo)?
>
> Anyway, I don't see what prevents an unauthorised narinfo with a ERIS
> URN to be used: the narinfo is chosen with
>
> (define narinfo
> (lookup-narinfo cache-urls store-item
> (if (%allow-unauthenticated-substitutes?)
> (const #t)
> (cut valid-narinfo? <> acl))))
>
> where lookup-narinfo is a tiny wrapper around lookup-narinfos/diverse.
> lookup-narinfos/diverse considers both unauthorised and authorised
> narinfos, and can choose an unauthorised narinfo if it's ‘equivalent’
> to an authorised narinfo (using equivalent-narinfo?)
>
> equivalent-narinfo? only looks at the hash, path, references and size,
> and ignores the ERIS. As such, an unauthorised narinfo with a
> malicious ERIS URN could be selected.

You're right. I was not aware that parts of unauthorized narinfos are
used when they are deemed equavelent to authorized narinfos with
equivalent-narinfo?.

Toggle quote (21 lines)
>
> However, it turns out that all this doesn't really matter: whether the
> port returned by 'fetch' in (guix scripts substitute) came from
> file://, http://, https:// or ERIS, the file hash is verified later
> anyway:
>
> ;; Compute the actual nar hash as we read it.
> ((algorithm expected)
> (narinfo-hash-algorithm+value narinfo))
> ((hashed get-hash)
> (open-hash-input-port algorithm input)))
>
> [...]
>
> ;; Check whether we got the data announced in NARINFO.
> (let ((actual (get-hash)))
> (if (bytevector=? actual expected)
> [...]
>
> False alarm I guess!

Yeah, good that the hash is checked. Still, I think we should not even
try downloading a ERIS URN that is not authorized.

I think adding a check to equivalent-narinfo? that makes sure that the
ERIS URNs are equivalent if present would fix this. wdyt?

-pukkamustard
M
M
Maxime Devos wrote on 4 Feb 17:16 +0100
(name . pukkamustard)(address . pukkamustard@posteo.net)
f05a19ad22a063662852f3799e465a6bb4d28ee8.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 10:51 [+0000]:
Toggle quote (7 lines)
> > Also, don't forget to insert this missing block back into
> > IPFS/GNUnet/BitTorrent/..., otherwise less and less blocks will be
> > available until nothing is available anymore.
>
> This might be a bit of a burden for users. As you mention the size of
> such a database might become considerable.

At least in GNUnet, there are quota on the size of the datastore
(and presumably, whatever the DHT service uses as database). When it's
exceeded, old blocks are removed. So I don't see a burden here,
assuming that the quota aren't overly large by default.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf1RXxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7tbgAQCIiwpFI5Vkp58Yf2ShVd29ShAW
qWSqyii0NGzBDXhwvQD+LNNDNEyqN59Ln2/gcNgL0dQw+u01A9WtwZ/IhWXJhAw=
=ky4X
-----END PGP SIGNATURE-----


?