[PATCH 0/1] An origin method for GNUnet FS URI's

Maxime Devos wrote on 24 Oct 2020 21:47

Recipients:(address . guix-patches@gnu.org)

Message-ID:5c72bcb9c86934deda97d952eb5cd459e615b313.camel@student.kuleuven.be

This patch defines a `gnunet-fetch' method, allowing for downloading
files from GNUnet by their GNUnet chk-URI.

This patch does not provide:
- a service configuration
- downloading substitutes from GNUnet
- fall-back to non-P2P (e.g. http://) or other P2P (e.g. ipfs://)
  systems
- downloading directories over GNUnet
- actual packages definitions using this method

Some issues and questions:
- (guix build gnunet) would call call-with-temporary-output-file
  from (guix utils), which isn't available when building derivations,
  so it has been copied to (guix build gnunet).  Is there any
  particular reason for call-with-temporary-output-file to be in
  (guix utils) and not (guix build utils)?
- Would it be possible somehow for url-fetch to support gnunet://fs/chk
  URIs? That way we could fall-back unto non-P2P URLs, which would be
  useful to bootstrap a P2P distribution from a non-P2P system.
- No timeouts have been implemented, so gnunet-download may spin
  forever if a source isn't available on GNUnet FS.

Some problematic points:
- (guix gnunet-download) calls gnunet-config from $PATH,
  to figure out connection details for (guix build gnunet)
- (guix build gnunet) requires the GNUnet FS daemon to bind
  to loopback, whereas a standard GNUnet setup would have
  the daemon bound to a Unix socket.

Example usage:

First make the GNUnet FS daemon accessible to Guix:
$ guix install gnunet
$ gnunet-config -s fs -o port -V 2094
$ gnunet-arm -s

Then publish the source tarball of the package to the GNUnet FS system:
$ guix environment --ad-hoc wget -- wget 
https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz
$ gnunet-publish hello-2.10.tar.gz

The output should look like this:

Toggle quote (6 lines)

> Publishing `$PWD/hello-2.10.tar.gz' done.

> URI is

> `gnunet://fs/chk/TY48PGS5RVX643NT2B7GDNFCBT4DWG692PF4YNHERR96K6MSFRZ4

> ZWRPQ4KVKZV29MGRZTWAMY9ETTST4B6VFM47JR2JS5PWBTPVXB0.8A9HRYABJ7HDA7B0P

> 37VG6D593>

The following test package can now be compiled:

$ cat > example.scm <<EOF
(define-module (example)
  #:use-module ((guix licenses)
                #:select (gpl3+))
  #:use-module (gnu packages)
  #:use-module (guix packages)
  #:use-module (guix utils)
  #:use-module (guix gnunet-download)
  #:use-module (guix build-system gnu)
  #:export (hello/gnunet))

(define-public hello/gnunet
  (package
    (name "hello-gnunet")
    (version "2.10")
    (source (origin
              (method gnunet-fetch)
              (uri
"gnunet://fs/chk/TY48PGS5RVX643NT2B7GDNFCBT4DWG692PF4YNHERR96K6MSFRZ4ZW
RPQ4KVKZV29MGRZTWAMY9ETTST4B6VFM47JR2JS5PWBTPVXB0.8A9HRYABJ7HDA7B0>
              (file-name "gnunet-hello-2.10.tar.gz")
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i")
)))
    (build-system gnu-build-system)
    (synopsis "Hello, GNUnet world! An example of a package with a
GNUnet chk-URI origin")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then
exits.  It
serves as an example of standard GNU coding practices.  As such, it
supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
hello/gnunet
EOF

$ ./pre-inst-env guix build -f example.scm

Maxime Devos (1):
  guix: Add (guix gnunet-download).

 Makefile.am              |   2 +
 doc/guix.texi            |   7 +++
 guix/build/gnunet.scm    | 113 +++++++++++++++++++++++++++++++++++++++
 guix/gnunet-download.scm |  89 ++++++++++++++++++++++++++++++
 4 files changed, 211 insertions(+)
 create mode 100644 guix/build/gnunet.scm
 create mode 100644 guix/gnunet-download.scm

-----BEGIN PGP SIGNATURE-----

iJcEABYIAD8WIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCX5SExSEcbWF4aW1lLmRl

dm9zQHN0dWRlbnQua3VsZXV2ZW4uYmUACgkQSePuIhkXJe6KWgD/XLInJPzNEYjD

OiuygHn8NgX38z9JpXfNIEqViy6tcUUA/i+j27oA9KBQW0vS2kqbqj2Gwrsx4gM3

bfWHKzh9oGwA

=L9RV

-----END PGP SIGNATURE-----

Maxime Devos wrote on 24 Oct 2020 21:54

[PATCH 1/1] guix: Add (guix gnunet-download).

Recipients:(address . 44199@debbugs.gnu.org)

Message-ID:a650e3207651d2a8c5357be3868f8581ee1692e6.camel@student.kuleuven.be

This method allows fetching sources over GNUnet's file-sharing
system, presuming GNUnet has been configured on the local system.

Missing:
- time-outs
- fetching substitutes over GNUnet
- fallback to legacy non-P2P servers
- GNUnet system service

* guix/gnunet-download.scm, guix/build/gnunet.scm: New files.
* Makefile.am (MODULES): Add them.
* doc/guix.texi (Defining Packages): Document gnunet-fetch.
---
 Makefile.am              |   2 +
 doc/guix.texi            |   7 +++
 guix/build/gnunet.scm    | 113 +++++++++++++++++++++++++++++++++++++++
 guix/gnunet-download.scm |  89 ++++++++++++++++++++++++++++++
 4 files changed, 211 insertions(+)
 create mode 100644 guix/build/gnunet.scm
 create mode 100644 guix/gnunet-download.scm

Toggle diff (283 lines)diff --git a/Makefile.am b/Makefile.am
index a75d9c1ffc..f046020017 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -94,6 +94,7 @@ MODULES =					\
   guix/android-repo-download.scm		\
   guix/bzr-download.scm            		\
   guix/git-download.scm				\
+  guix/gnunet-download.scm 			\
   guix/hg-download.scm				\
   guix/swh.scm					\
   guix/monads.scm				\
@@ -187,6 +188,7 @@ MODULES =					\
   guix/build/bzr.scm				\
   guix/build/copy-build-system.scm		\
   guix/build/git.scm				\
+  guix/build/gnunet.scm				\
   guix/build/hg.scm				\
   guix/build/glib-or-gtk-build-system.scm	\
   guix/build/gnu-bootstrap.scm			\
diff --git a/doc/guix.texi b/doc/guix.texi
index 8514dfe86f..8a5f0559f3 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -81,6 +81,7 @@ Copyright @copyright{} 2020 R Veera Kumar@*
 Copyright @copyright{} 2020 Pierre Langlois@*
 Copyright @copyright{} 2020 pinoaffe@*
 Copyright @copyright{} 2020 André Batista@*
+Copyright @copyright{} 2020 Maxime Devos@*
 
 Permission is granted to copy, distribute and/or modify this document
 under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -6595,6 +6596,12 @@ specified in the @code{uri} field as a
@code{git-reference} object; a
   (url "https://git.savannah.gnu.org/git/hello.git")
   (commit "v2.10"))
 @end lisp
+
+@vindex gnunet-fetch
+@item @var{gnunet-feth} from @code{(guix gnunet-download)}
+download a file specified by its GNUnet chk-URI.  To use
+this method, the GNUnet file-sharing daemon has to be configured
+to accept connections from the loopback networking interface.
 @end table
 
 @item @code{sha256}
diff --git a/guix/build/gnunet.scm b/guix/build/gnunet.scm
new file mode 100644
index 0000000000..3cee161cc2
--- /dev/null
+++ b/guix/build/gnunet.scm
@@ -0,0 +1,113 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2018 Mark H Weaver <mhw@netris.org>
+;;; Copyright © 2018 Björn Höfling <bjoern.hoefling@bjoernhoefling.de>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2020 Maxime Devos <maxime.devos@student.kuleuven.be>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify
it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or
(at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>;.
+
+(define-module (guix build gnunet)
+  #:use-module (guix build utils)
+  #:use-module (srfi srfi-34)
+  #:use-module (ice-9 format)
+  #:use-module (rnrs io ports)
+  #:export (gnunet-fetch))
+
+;;; Commentary:
+;;;
+;;; This is the build-side support code of (guix gnunet-download).  It 
allows
+;;; files of which the GNUnet chk-URI is known to be downloaded from
the GNUnet
+;;; file-sharing system.  The code has been derived from (guix build
hg).
+;;;
+;;; Code:
+
+;; Copied from (guix utils)
+(define (call-with-temporary-output-file proc)
+  "Call PROC with a name of a temporary file and open output port to
that
+file; close the file and delete it when leaving the dynamic extent of
this
+call."
+  (let* ((directory (or (getenv "TMPDIR") "/tmp"))
+         (template  (string-append directory "/guix-file.XXXXXX"))
+         (out       (mkstemp! template)))
+    (dynamic-wind
+      (lambda ()
+        #t)
+      (lambda ()
+        (proc template out))
+      (lambda ()
+        (false-if-exception (close out))
+        (false-if-exception (delete-file template))))))
+
+(define (gnunet-fs-up? port)
+  "#t if the GNUnet FS daemon seems to be up at @var{port}, #f
otherwise"
+  (let ((s (socket PF_INET SOCK_STREAM 0)))
+    (catch 'system-error
+      (lambda ()
+        (connect s AF_INET INADDR_LOOPBACK port)
+        (close-port s)
+        #t)
+      (lambda (tag function msg msg+ errno)
+        (close-port s)
+        (if (and (equal? function "connect")
+                 (equal? errno (list ECONNREFUSED)))
+            #f
+            (throw tag function msg msg+ errno))))))
+
+;; TODO: gnunet directories, time-outs, perhaps use guile-gnunet
+(define* (gnunet-fetch uri file
+                       #:key (gnunet-download-command "gnunet-
download"))
+  "Fetch a file identified by a GNUnet chk-URI @var{URI} into
@var{file}.
+@var{uri} must not be a directory.  Return #t on success, #f
otherwise."
+  (guard (c ((invoke-error? c)
+             (format (current-error-port)
+                     "gnunet-fetch: '~a~{ ~a~}' failed with exit code
~a~%"
+                     (invoke-error-program c)
+                     (invoke-error-arguments c)
+                     (or (invoke-error-exit-status c)
+                         (invoke-error-stop-signal c)
+                         (invoke-error-term-signal c)))
+             (false-if-exception (delete-file-recursively file))
+             #f))
+    (define port
+      (let ((p (getenv "gnunet port")))
+        (and p (< 0 (string-length p))
+             (string->number p))))
+    (define anonymity
+      (let ((a (getenv "GNUNET_ANONYMITY")))
+        (cond ((equal? a "") "1")
+              ((not a) "1")
+              (else a))))
+    ;; Check if the GNUnet daemon is up,
+    ;; otherwise gnunet-download might wait forever.
+    (if (or (not port) (gnunet-fs-up? port))
+        (call-with-temporary-output-file
+         (lambda (config-file-name config-output-port)
+           ;; Tell gnunet-download how to contact the FS daemon
+           (display (getenv "gnunet configuration") config-output-
port)
+           (flush-output-port config-output-port)
+           (invoke gnunet-download-command uri
+                   "-c" config-file-name
+                   "-V" ;; print progress information
+                   "-a" anonymity
+                   "-o" file)
+           #t))
+        (begin
+          (format (current-error-port)
+                  "gnunet-fetch: file-sharing daemon is down.~%")
+          #f))))
+
+;;; gnunet.scm ends here
diff --git a/guix/gnunet-download.scm b/guix/gnunet-download.scm
new file mode 100644
index 0000000000..8a825b90ae
--- /dev/null
+++ b/guix/gnunet-download.scm
@@ -0,0 +1,89 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>
+;;; Copyright © 2020 Maxime Devos <maxime.devos@student.kuleuven.be>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify
it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or
(at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>;.
+(define-module (guix gnunet-download)
+  #:use-module (guix packages)
+  #:use-module (guix store)
+  #:use-module (guix monads)
+  #:use-module (guix gexp)
+  #:use-module (ice-9 popen)
+  #:use-module (ice-9 rdelim)
+  #:export (gnunet-fetch))
+
+;;; An <origin> method that uses gnunet-download to fetch a specific
hash
+;;; the GNUnet file-sharing system.  The hash is specified as a GNUnet
chk-URI
+;;; string.  The code has been derived from (guix gx-download).
+;;;
+;;; Code:
+
+(define (gnunet-package)
+  "Return the default GNUnet package."
+  (let ((distro (resolve-interface '(gnu packages gnunet))))
+    (module-ref distro 'gnunet)))
+
+(define* (gnunet-configuration #:key (gnunet (gnunet-package)))
+  "Make a configuration file allowing the build process to talk
+with the GNUnet FS daemon."
+  ;; TODO: is it acceptable to assume
+  ;; the existence of gnunet-config in PATH?
+  ;; If not, can @var{gnunet} be compiled?
+  ;; Alternatively, parse .config/gnunet.conf manually.
+  ;;
+  ;; TODO: by default, GNUnet uses Unix sockets
+  ;; instead of IP for IPC. Can we poke a hole
+  ;; in the build process isolation allowing this
+  ;; setup?
+  (let* ((p (open-pipe* OPEN_READ "gnunet-config" "--section" "fs" "-
o" "PORT"))
+         (port (read-line p)))
+    (close-pipe p)
+    (values (format #f "[fs]~%PORT = ~a\n" port)
+            port)))
+
+(define* (gnunet-fetch uri hash-algo hash
+                       #:optional name
+                       #:key (system (%current-system)) (guile
(default-guile))
+                       (gnunet (gnunet-package)))
+  "Return a fixed-output derivation that fetches @var{uri}, a GNUnet
chk-URI
+string.  The output is expected to have hash @var{hash} of type
+@var{hash-algo}.  Use @var{name} as the file name, or a generic name
if #f."
+  (define build
+    (with-imported-modules '((guix build gnunet)
+                             (guix build utils))
+      #~(begin
+          (use-modules (guix build gnunet))
+          (or (gnunet-fetch '#$uri
+                            #$output
+                            #:gnunet-download-command
+                            (string-append #+gnunet "/bin/gnunet-
download"))))))
+  (define env-vars
+    (call-with-values (lambda () (gnunet-configuration #:gnunet
gnunet))
+      (lambda (configuration port)
+        `(("gnunet configuration" . ,configuration)
+          ("gnunet port" . ,port)))))
+  (mlet %store-monad ((guile (package->derivation guile system)))
+    (gexp->derivation (or name "gnunet-chk") build
+                      #:system system
+                      #:local-build? #t ;; don't offload downloads
+                      #:hash-algo hash-algo
+                      #:hash hash
+                      #:recursive? #f
+                      #:leaked-env-vars '("GNUNET_ANONYMITY")
+                      #:env-vars env-vars
+                      #:guile-for-build guile)))
+
+;;; gnunet-download.scm ends here

-----BEGIN PGP SIGNATURE-----

iJcEABYIAD8WIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCX5SGZCEcbWF4aW1lLmRl

dm9zQHN0dWRlbnQua3VsZXV2ZW4uYmUACgkQSePuIhkXJe5uAAEArZnvBoDnRG5J

FetXk6KkL/uGKiQmc6IBmjP6kqdwD/UA/2dGBa2mzP6UM1mNZcEOnZDHBjFuf4yI

PyQFqza+a3YO

=O+eh

-----END PGP SIGNATURE-----

zimoun wrote on 27 Oct 2020 14:39

Re: [bug#44199] [PATCH 0/1] An origin method for GNUnet FS URI's

Recipients:

Message-ID:86blgn4wk5.fsf@gmail.com

Dear,

Thank you for the patch. My questions are totally naive since I do not

know much about GNUnet.

On Sat, 24 Oct 2020 at 21:47, Maxime Devos <maxime.devos@student.kuleuven.be> wrote:

Toggle quote (10 lines)> This patch defines a `gnunet-fetch' method, allowing for downloading
> files from GNUnet by their GNUnet chk-URI.
>
> This patch does not provide:
> - a service configuration
> - downloading substitutes from GNUnet
> - fall-back to non-P2P (e.g. http://) or other P2P (e.g. ipfs://)
>   systems
> - downloading directories over GNUnet

This means it only works for archives as tarball, right?

Toggle quote (4 lines)

> - actual packages definitions using this method

> Some issues and questions:

[...]

Toggle quote (4 lines)

> - Would it be possible somehow for url-fetch to support gnunet://fs/chk

> URIs? That way we could fall-back unto non-P2P URLs, which would be

> useful to bootstrap a P2P distribution from a non-P2P system.

Who is the “we”? What do you mean by “url-fetch supports gnunet:// and

fall-back unto non-P2P”?

Some recent discussions are about content-address and fallback. For

example, roughly speaking ’git-fetch’ tries upstream, then the Guix

build farms, then Software Heritage (swh). For Git repo, it works

because the address from Guix side to SWH is straightforward. The 2

other VCS –hg and svn– supported by SWH should be implemented soon… who

knows! ;-)

The story about archives as tarball is a bit more complicated. The main

issue –as I understand it– can be summarized as: Guix knows the URL, the

integrity checksum and only at package time the content of the tarball.

Later in time, it is difficult to lookup because of this very address;

and some are around: nar, swh-id, ipfs, gnunet, etc.

Bridges to reassemble the content are currently discussed, e.g.,

https://git.ngyro.com/disarchive-db/

https://git.ngyro.com/disarchive

Well, today the fallback of tarball archive to SWH is not reliable.

What is your question? ;-)

Toggle quote (5 lines)

> Then publish the source tarball of the package to the GNUnet FS system:

> $ guix environment --ad-hoc wget -- wget

> https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz

> $ gnunet-publish hello-2.10.tar.gz

Naive question: are packages only available on GNUnet?

All the best,

simon

Maxime Devos wrote on 27 Oct 2020 19:50

Recipients:

Message-ID:e553302871cd653024e18d9b0a8f7cfce192e9fe.camel@student.kuleuven.be

zimoun schreef op di 27-10-2020 om 14:39 [+0100]:

Toggle quote (5 lines)> Dear,
> 
> Thank you for the patch.  My questions are totally naive since I do
> not
> know much about GNUnet.

Conceptually, the GNUnet file-sharing system is not unlike IPFS, dat,

or even torrents, although the details can vary wildly. I don't know

much more about GNUnet either except for how to install it, and how to

publish and search for files.

Toggle quote (15 lines)> On Sat, 24 Oct 2020 at 21:47, Maxime Devos <
> maxime.devos@student.kuleuven.be> wrote:
> > This patch defines a `gnunet-fetch' method, allowing for
> > downloading
> > files from GNUnet by their GNUnet chk-URI.
> > 
> > This patch does not provide:
> > - a service configuration
> > - downloading substitutes from GNUnet
> > - fall-back to non-P2P (e.g. http://) or other P2P (e.g. ipfs://)
> >   systems
> > - downloading directories over GNUnet
> 
> This means it only works for archives as tarball, right?

More-or-less, yes. I would assume a zip archive would work just fine as

well, and in the rare case where the source code consists of a single

file (e.g. search for the %boot-logo-patch in (gnu packages linux)),

you don't need an archive at all.

GNUnet has a special format for directories, presumably supporting

deduplication (I haven't checked), conceptually not unlike ipfs and

dat. It is probably not too much work to support this format, but I

would like to keep the first patch simple.

Toggle quote (10 lines)> > - Would it be possible somehow for url-fetch to support
> > gnunet://fs/chk
> >   URIs? That way we could fall-back unto non-P2P URLs, which would
> > be
> >   useful to bootstrap a P2P distribution from a non-P2P system.
> 
> Who is the “we”?  What do you mean by “url-fetch supports gnunet://
> and
> fall-back unto non-P2P”?

Presuming this patch is accepted in some form, ‘we’ refers to code in

Guix responsible for downloading archives. I should have formulated

that better. About the ‘fall-back unto non-P2P’:

The ultimate goal is to not rely on any centralised servers at all (not

even mirrors) (these are the non-P2P systems), and download all

archives and substitutes from GNUnet FS (or IPFS or dat, but I prefer

GNUnet). Of course, expecting everyone to stop using HTTP / ftp / etc.

for source distribution and instead use GNUnet immediately is foolish,

so ideally there would be a smooth transition path:

1. At first, maintainers would typically still publish the tarballs(*)

on a centralised (possibly mirrored) servers, say

https://ftp.gnu.org. The package definition would use the

url-fetch method.

2. If $MAINTAINER doesn't have access to or doesn't want to use any

publicly-available distribution site (e.g. due to censorship, or for

pseudonymity or simplicity), $MAINTAINER may instead use the

gnunet-fetch method in the package definition.

Alternatively, if the distribution site disappeared, Guix

maintainers have the option to point to GNUnet. (Or refer to swh,

I presume, but my goal is to decentralise.)

4. $MAINTAINER finds GNUnet convenient. However, GNUnet hasn't achieved

world-domination yet, so $MAINTAINER still publishes tarballs on

https://ftp.gnu.org or similar. Actually, even if GNUnet *was*

universally supported, a centralised server can still be useful:

P2P file-sharing systems by design may have to throw away old data

the local peer isn't interested in (due to limited disk space), and

$MAINTAINER might stop publishing the source code over GNUnet.

In such cases, a centralised server (and some mirrors) may be a

useful back-up, although it would still be preferred to use a

distributed P2P system (such as GNUnet or ipfs) when available

to limit the cost of running the centralised server (and mirrors).

(To limit the utilisation of the centralised server (and mirrors),

sources downloaded from the server (and mirrors) should be published

back into the GNUnet FS, but this can be as simple and crude as a

cron job `gnunet-publish /gnu/store/*.tar.gz`.)

My idea in this case is to specify the origin as:

(origin

(method url-fetch)

(uri (list "gnunet://fs/chk/etcetera"

(string-append "mirror://gnu/hello-" version

".tar.gz")))

etcetera)

However, this patch defines gnunet-fetch as a separate method

from url-fetch, url-fetch is left untouched, so url-fetch will

just ignore the gnunet://fs/chk URI's.

If I understand correctly, url-fetch is a special method, in that

the downloading code of the Guix daemon is used, and not the code

defined by the user. Nothing a `guix system reconfigure` cannot fix,

but nevertheless rather inconvenient for testing.

Toggle quote (8 lines)

> Some recent discussions are about content-address and fallback. For

> example, roughly speaking ’git-fetch’ tries upstream, then the Guix

> build farms, then Software Heritage (swh). For Git repo, it works

> because the address from Guix side to SWH is straightforward. The 2

> other VCS –hg and svn– supported by SWH should be implemented soon…

> who

> knows! ;-)

\me missed these discussions

Toggle quote (20 lines)> The story about archives as tarball is a bit more complicated.  The
> main
> issue –as I understand it– can be summarized as: Guix knows the URL,
> the
> integrity checksum and only at package time the content of the
> tarball.
> Later in time, it is difficult to lookup because of this very
> address;
> and some are around: nar, swh-id, ipfs, gnunet, etc.
> 
> Bridges to reassemble the content are currently discussed, e.g.,
> 
>    <https://git.ngyro.com/disarchive-db/>
>    <https://git.ngyro.com/disarchive>
> 
> Well, today the fallback of tarball archive to SWH is not reliable.
> 
> 
> What is your question? ;-)

The question to which answer? I'm not quite which answer you're

referring to, so some guesses:

Q: How does Guix figure out the GNUnet URI from the Guix (or nar, I

guess) hash?

A: Not automatically. The gnunet-fetch method as defined in this

patch needs to be passed the URI manually. However, an additional

service for GNUnet can be written that uses the DHT to map Guix

(or nar, or something else) hashes to corresponding GNUnet URI's.

(I'm not volunteering (yet))

Q: What about automatically generated tarballs (e.g. from git

repositories)?

A: Not addressed by this patch. The intention is to be able to replace

a http://*/*.tar.gz URL with a gnunet://fs/chk URI in package

definitions; source code repositories aren't supported by this

patch. (But perhaps a future patch could support this!)

Q: Integration with bridges?

A: Sounds nice, but I'm trying to keep things simple for the first

patch! (And I haven't heard of disarchive before now.)

Toggle quote (8 lines)

> > Then publish the source tarball of the package to the GNUnet FS

> > system:

> > $ guix environment --ad-hoc wget -- wget

> > https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz

> > $ gnunet-publish hello-2.10.tar.gz

> Naive question: are packages only available on GNUnet?

Naive interpretations of this question, and answers:

* Is the only thing available on GNUnet, *packages*?

While experimenting with GNUnet, I have seen some images published on

GNUnet. GNUnet also has other systems that the FS system, but they

aren't relevant to this path.

* Are packages *definitions* only available, on *GNUnet*?

I'm getting my Guix package definitions from the git repository

on Savannah, and I have never encountered any on the GNUnet

FS system.

* Is package *source code* only available, on *GNUnet*?

If someone published the source code (e.g. as a tarball) on GNUnet

with `gnunet-publish hello-2.10.tar.gz`, it is only published(*) on

GNUnet, and not somewhere else as well.

(*) I don't know exactly when one can be reasonably sure the file

will *remain* available for some time when published, except for

keeping the GNUnet daemon running continuously.

However, in practice, $MAINTAINER will publish the source code

somewhere else as well (e.g. https://ftp.gnu.org or perhaps ipfs).

This patch doesn't automatically publish source code of built or

downloaded packages on GNUnet, although that seems a useful service

to run as a daemon.

* If the gnunet-fetch method is used, will Guix try to get the source

code from GNUnet, and nowhere else?

Ignoring substitutes, yes. This is a limitation of defining

gnunet-fetch separately from url-fetch. I believe this has been

addressed earlier in my response e-mail.

Toggle quote (3 lines)

> All the best,

> simon

Likewise, maxime

Attachment: Maxime Devos.pgp

-----BEGIN PGP SIGNATURE-----

iJcEABYIAD8WIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCX5hr6iEcbWF4aW1lLmRl

dm9zQHN0dWRlbnQua3VsZXV2ZW4uYmUACgkQSePuIhkXJe6rXwD/ZvuV2F0+CXOR

aeeehho3jMT8oxE5ZmiF1oFILu6Qv7kA/1GCpvV0HKO7t3z++u7liNcCm/Ui7PBY

0DCP6YjQufYP

=R3pG

-----END PGP SIGNATURE-----

Maxime Devos wrote on 1 Nov 2020 01:05

Recipients:

Message-ID:05677842bc60336461f8fe77ebd3526b2b23efb9.camel@student.kuleuven.be

[CC'd to Timothy Sample because of discussion of defining a new format

for disarchive, and to gnunet-developers because of obvious reasons]

A small status update!

zimoun schreef op di 27-10-2020 om 14:39 [+0100]:

Toggle quote (22 lines)> [...]
> 
> The story about archives as tarball is a bit more complicated.  The
> main
> issue –as I understand it– can be summarized as: Guix knows the URL,
> the
> integrity checksum and only at package time the content of the
> tarball.
> Later in time, it is difficult to lookup because of this very
> address;
> and some are around: nar, swh-id, ipfs, gnunet, etc.
> 
> Bridges to reassemble the content are currently discussed, e.g.,
> 
>    <https://git.ngyro.com/disarchive-db/>
>    <https://git.ngyro.com/disarchive>
> 
> Well, today the fallback of tarball archive to SWH is not reliable.
> 
> 
> What is your question? ;-)

I looked a bit into the GNUnet FS code and disarchive discussions. The

part about tarballs seemed particularily relevant, as well as some

older discussion on preserving the executable bit when using IPFS.

Some issues with using GNUnet's directory format in GNUnet for Guix

substitutes to address:

* directory entries are not placed in any particular order.

Solution: sort by file-name

* there is no executable bit.

Solution: define a new metadata property (*).

This should only take a small patch to libextractor.

(*) Not sure about the correct terminology

* GNUnet sometimes inlines small files in directories,

but strictly speaking when to do so is left up to the implementation.

Solution: pick a fixed reference implementation.

* By default, when publishing, gnunet-publish uses libextractor

to figure out some meta-data (e.g. title, mime-type, album name),

which may return different meta-data depending on the implementation.

Solution: disable the use of libextractor, at least when GNUnet is

used by Guix.

I'm currently porting the directory creation code of GNUnet to Scheme

(but not any other GNUnet code), to be used by Guix (for publishing

substitutes) and disarchive (for reconstructing GNUnet directories).

After addressing these issues, I believe I will end up with a fairly

well-defined archive format.

<friendly-footer/>

-----BEGIN PGP SIGNATURE-----

iJcEABYIAD8WIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCX5374iEcbWF4aW1lLmRl

dm9zQHN0dWRlbnQua3VsZXV2ZW4uYmUACgkQSePuIhkXJe6L+wEAvGIuqyOhZ8aM

oiGQhaGINoV46XAmJ4JDx4vVpxON4zgBAK1wrsyGrBu5TmQbXdSPW9IdrmvSOR6P

S2XBUHs04mcL

=LVUJ

-----END PGP SIGNATURE-----

Ludovic Courtès wrote on 15 Nov 2020 22:13

Recipients:(name . Maxime Devos)(address . maxime.devos@student.kuleuven.be)(address . 44199@debbugs.gnu.org)

Message-ID:87mtzi5nni.fsf@gnu.org

Hi Maxime,

Maxime Devos <maxime.devos@student.kuleuven.be> skribis:

Toggle quote (3 lines)

> This patch defines a `gnunet-fetch' method, allowing for downloading

> files from GNUnet by their GNUnet chk-URI.

While I think this is a laudable goal, I’m reluctant to including GNUnet

support just yet because, as stated in recent release announcements,

GNUnet is still in flux and not considered “production ready”.

So I think we should keep it around and revisit this issue when GNUnet

is considered “stable”. WDYT?

Thanks,

Ludo’.

zimoun wrote on 16 Nov 2020 01:35

Recipients:

Message-ID:86361arvdh.fsf@gmail.com

Hi Maxim,

Thanks for your detailed answer.

You might be interested by the coming oneline Guix Days conference on

Sun. 22nd 2020. A session is specifically dedicated to a related topic:

How to distribute P2P? Please browse this week the blog guix.gnu.org

for the details.

On Tue, 27 Oct 2020 at 19:50, Maxime Devos <maxime.devos@student.kuleuven.be> wrote:

Toggle quote (7 lines)

> Q: How does Guix figure out the GNUnet URI from the Guix (or nar, I

> guess) hash?

> A: Not automatically. The gnunet-fetch method as defined in this

> patch needs to be passed the URI manually. However, an additional

> service for GNUnet can be written that uses the DHT to map Guix

> (or nar, or something else) hashes to corresponding GNUnet URI's.

From my understanding, this is a show stopper. It has to be solved

first going further, IMHO. It is not possible to write manually the URI

for all the packages. And as perhaps you read with the project

’disassemble’, it is not straightforward.

Toggle quote (7 lines)

> Q: What about automatically generated tarballs (e.g. from git

> repositories)?

> A: Not addressed by this patch. The intention is to be able to replace

> a http://*/*.tar.gz URL with a gnunet://fs/chk URI in package

> definitions; source code repositories aren't supported by this

> patch. (But perhaps a future patch could support this!)

I think this is the main issue: it is not affordable to replace for some

packages the current http://by gnunet://. Especially when GNUnet is

not “stable“.

Toggle quote (16 lines)> * Is package *source code* only available, on *GNUnet*?
>
>   If someone published the source code (e.g. as a tarball) on GNUnet
>   with `gnunet-publish hello-2.10.tar.gz`, it is only published(*) on
>   GNUnet, and not somewhere else as well.
>
>   (*) I don't know exactly when one can be reasonably sure the file
>   will *remain* available for some time when published, except for
>   keeping the GNUnet daemon running continuously.
>
>   However, in practice, $MAINTAINER will publish the source code
>   somewhere else as well (e.g. <https://ftp.gnu.org> or perhaps ipfs).
>   This patch doesn't automatically publish source code of built or
>   downloaded packages on GNUnet, although that seems a useful service 
>   to run as a daemon.

Therefore the corollary question is: how many tarballs currently used as

source by Guix are also available on GNUnet?

Thank you for your interest. And again I invite you to join the

discussion about P2P and Guix this Sunday 22nd. Read on the Guix

blog. :-)

All the best,

simon

Maxime Devos wrote on 18 Nov 2020 20:14

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 44199@debbugs.gnu.org)

Message-ID:efade17550e921b976a39312995503acb74ae017.camel@student.kuleuven.be

Hi Ludovic,

Ludovic Courtès schreef op zo 15-11-2020 om 22:13 [+0100]:

Toggle quote (10 lines)> [snip]
> While I think this is a laudable goal, I’m reluctant to including
> GNUnet
> support just yet because, as stated in recent release announcements,
> GNUnet is still in flux and not considered “production ready”.
> 
> So I think we should keep it around and revisit this issue when
> GNUnet
> is considered “stable”.  WDYT?

Sounds reasonable to me. There are also a lot of missing parts: a
service definition for Guix System, findings substitutes, finding
sources by hash (the one Guix uses, not the GNUnet hash) ..., so it
isn't like my rudimentary patch was usable on large scale anyway.

Greetings,
Maxime

-----BEGIN PGP SIGNATURE-----

iJcEABYIAD8WIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCX7VyjyEcbWF4aW1lLmRl

dm9zQHN0dWRlbnQua3VsZXV2ZW4uYmUACgkQSePuIhkXJe4VtgEAi0OogAM9vn2B

wchWM3/HcMj+9mpmBapUxSRHi61oYCsA/RzTNRmGwLxZ8ouB561mU8YWka9lFVuy

tQg5JEtxMfEO

=dUfT

-----END PGP SIGNATURE-----

Maxime Devos wrote on 18 Nov 2020 21:28

Recipients:

Message-ID:e6e8c509fff0b5ae2c32678865fba0bece3193d6.camel@student.kuleuven.be

Hi,

(btw it's Maxim*e*, not Maxim. The ‘e’ isn't pronounced but it's still

written.)

I'll try to address the various issues in separate e-mails.

zimoun schreef op ma 16-11-2020 om 01:35 [+0100

Toggle quote (2 lines)

> [snip]

> From my understanding, this is a show stopper. It has to be solved

first going further, IMHO. It is not possible to write manually the

URI for all the packages. And as perhaps you read with the project

’disassemble’, it is not straightforward.

I agree! I see three straightforward answers to this.

a) Fancy

Write a GNUnet service using the DHT to map the hashes used in origin

specifications (*) to URI's for the FS system. To let the local

contribution to the DHT survive peer restarts, maintain a database

(e.g. SQlite) of (Guix hash -> GNUnet hash) (^), that is expanded with

each successful source (or binary) substitution or build.

(Alternatively, as the DHT isn't anonymous,

place hash -> GNUnet hash references into some well-known name space.

Then hash lookup + FS should automatically be anonymous when desired.)

Possible issues: time out behaviour, the DHT is not anonymous.

Annoyance: probably requires extending the build daemon.

Perhaps try regular downloads (e.g. via HTTP/S, ftp, ...) in parallel

with the GNUnet download after a configurable delay?

Perhaps use a well-known GNUnet FS namespace instead of the DHT

for anonymous downloads?

(*) Also usable for package outputs, if the hash of the output is used

and not the hash of the outputs

(^) In case the database is full, delete some old entries

b) Simple, slow introduction (no additional GNUnet services required)

Extend (origin ...) with an optional gnunet-hash field.

Adjust ‘guix download’, ‘guix refresh’ and ‘guix import’

to emit the gnunet-hash (%) field. Plumb this field to the guix daemon

somehow. Same approach is possible for IPFS.

As packages are updated and new packages are defined, given sufficient

time, there will be more packages with a gnunet-hash field than not.

(%) Computing the gnunet-hash of a directory doesn't require

a full-fledged GNUnet installation. My scheme-gnunet repository

is not very far from the point where it can convert file trees +

libextractor metadata into bytevectors, without depending on C gnunet.

A TODO: different zlib's

would produce different bytevectors --> different GNUnet hash

--> perhaps always use a single version.

A TODO (for nix archives on GNUnet): define

EXTRACTOR_METATYPE_EXECUTABLE (or mimetype: application/x-executable).

Perhaps use mimetype: x-inode/symlink (or something like that) as well?

Repository URL: https://notabug.org/mdevos/scheme-gnunet

c) Not scalable, but may reduce network traffic to ci.guix.gnu.org & co

Like in a) keep a database of known (Guix hash -> GNUnet FS URI).

Perhaps make this available through a web interface or git repository

... wait, this sounds familiar ... this seems to fit well into the

‘disarchive’ project!

Greetings,

Maxime

-----BEGIN PGP SIGNATURE-----

iJcEABYIAD8WIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCX7WD1SEcbWF4aW1lLmRl

dm9zQHN0dWRlbnQua3VsZXV2ZW4uYmUACgkQSePuIhkXJe7bqAEAoDmT3o4g7bA2

KlNfK/0CvAylKeBL6s6A4G5/RSSuWHMBAJmOGs1KyOeYo3vigBmp8w4bwy5TnFj0

7/Q7srOIKCkF

=qgHe

-----END PGP SIGNATURE-----

Ludovic Courtès wrote on 18 Nov 2020 23:12

control message for bug #44199

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87wnyiiaan.fsf@gnu.org

tags 44199 wontfix
close 44199
quit

zimoun wrote on 18 Nov 2020 23:40

Re: [bug#44199] [PATCH 0/1] An origin method for GNUnet FS URI's

Recipients:

Message-ID:86o8ju2ss2.fsf@gmail.com

Dear,

On Wed, 18 Nov 2020 at 21:28, Maxime Devos <maxime.devos@student.kuleuven.be> wrote:

Toggle quote (3 lines)

> (btw it's Maxim*e*, not Maxim. The ‘e’ isn't pronounced but it's still

> written.)

Sorry.

Toggle quote (8 lines)

>> From my understanding, this is a show stopper. It has to be solved

> first going further, IMHO. It is not possible to write manually the

> URI for all the packages. And as perhaps you read with the project

> ’disassemble’, it is not straightforward.

> I agree! I see three straightforward answers to this.

Thanks for sharing your views.

Toggle quote (12 lines)> a) Fancy
>
> Write a GNUnet service using the DHT to map the hashes used in origin
> specifications (*) to URI's for the FS system. To let the local
> contribution to the DHT survive peer restarts, maintain a database
> (e.g. SQlite) of (Guix hash -> GNUnet hash) (^), that is expanded with
> each successful source (or binary) substitution or build.
>
> (Alternatively, as the DHT isn't anonymous,
> place hash -> GNUnet hash references into some well-known name space.
> Then hash lookup + FS should automatically be anonymous when desired.)

Yeah, for sure but both are a piece of tough work though.

Toggle quote (7 lines)

> b) Simple, slow introduction (no additional GNUnet services required)

> Extend (origin ...) with an optional gnunet-hash field.

> Adjust ‘guix download’, ‘guix refresh’ and ‘guix import’

> to emit the gnunet-hash (%) field. Plumb this field to the guix daemon

> somehow. Same approach is possible for IPFS.

Guix is not doing that for Software Heritage id for example (that’s why

dissamble exists :-)). So I am really doubtful that Guix will do it for

the more-experimental GNUnet.

However, if you want to implement proof-of-concept, you can use the

field ’properties’ which is more or less free. I mean, you can add

whatever you want in.

Toggle quote (7 lines)

> c) Not scalable, but may reduce network traffic to ci.guix.gnu.org & co

> Like in a) keep a database of known (Guix hash -> GNUnet FS URI).

> Perhaps make this available through a web interface or git repository

> ... wait, this sounds familiar ... this seems to fit well into the

> ‘disarchive’ project!

The most traffic issue is about substitutes, not really about sources.

And about substitutes, this patch #33899 [1] implements the support of

IPFS. Once it is ready, the pieces will be there somehow, so adding

GNUnet to distribute susbtitutes maybe would be an option.

Please join the conference [2] on this Sunday, a session to discuss all

this will be dedicated.

1: http://issues.guix.gnu.org/issue/33899

2: https://guix.gnu.org/en/blog/2020/online-guix-day-announce-2/

All the best,

simon

zimoun wrote on 18 Nov 2020 23:42

Recipients:(address . 44199-done@debbugs.gnu.org)

Message-ID:86lfey2so4.fsf@gmail.com

Hi Maxime,

On Wed, 18 Nov 2020 at 20:14, Maxime Devos <maxime.devos@student.kuleuven.be> wrote:

Toggle quote (17 lines)> Ludovic Courtès schreef op zo 15-11-2020 om 22:13 [+0100]:

>> [snip]
>> While I think this is a laudable goal, I’m reluctant to including
>> GNUnet
>> support just yet because, as stated in recent release announcements,
>> GNUnet is still in flux and not considered “production ready”.
>> 
>> So I think we should keep it around and revisit this issue when
>> GNUnet
>> is considered “stable”.  WDYT?
>
> Sounds reasonable to me. There are also a lot of missing parts: a
> service definition for Guix System, findings substitutes, finding
> sources by hash (the one Guix uses, not the GNUnet hash) ..., so it
> isn't like my rudimentary patch was usable on large scale anyway.

Therefore, I am closing. Feel free to reopen once GNUnet is considered

as (more) “stable”.

Thank you for your contribution.

All the best,

simon

Closed

Maxime Devos wrote on 27 Jan 2021 14:04

Subject

Recipients:(address . control@debbugs.gnu.org)

Message-ID:627784a420c349acd9e6b7e279bcd9ac77112489.camel@telenet.be

unarchive 44199

Maxime Devos wrote on 27 Jan 2021 14:07

Re: Info: Rehash Project

Recipients:(address . 44199@debbugs.gnu.org)

Message-ID:328647d66cf760388a26f2bbfaf3080801ca9ad0.camel@telenet.be

Hi Matias (and Guix, which I've CC'ed),

To Matias: a follow up message will follow.

Unfortunately, I've just taken a pause from Guix+GNUnet hacking

(though probably I'll occasionally resume hacking once in a while).

Some things that work now:

* The rehash service itself seems to work

(https://notabug.org/mdevos/rehash).

This is the service where peers add SHA512<->GNUnet FS URI mappings they

discover (replace SHA512 by whatever Guix uses).

* Unless I broke anything, the ‘remirror’ service

(actually just a daemon implementing a web

server to run locally) can proxy http: downloads.

Proxying https: is a little difficult, as ‘remirror’

would need to play man-in-the-middle, but

may be implemented eventually. Or maybe guix can

be patched to (optionally) not use the CONNECT method

for proxying https: downloads.

There is no ‘offloading’ to GNUnet yet, though.

* Perhaps a better approach for substitutes:

In the ‘scheme-gnunet’ repository (https://notabug.org/mdevos/scheme-gnunet/src/master/ROADMAP.org),

I've written a publish-store.scm and download-store.scm script,

that respectively upload and download an item

from the store/GNUnet FS (using the gnunet-publish and

gnunet-download binaries).

It's not plugged into the guix substituter

and guix publish yet, though. I'm a bit at a loss how to do this properly,

so I'm more-or-less waiting until (a future revision of) the IPFS patch

is merged, and then I'll try to add GNUnet as ‘just another p2p system’.

Greetings, Maxime

Maxime Devos <maximedevos@telenet.be>

PGP Key: C1F3 3EE2 0C52 8FDB 7DD7 011F 49E3 EE22 1917 25EE

Freenode handle: mdevos

-----BEGIN PGP SIGNATURE-----

iI0EABYIADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYBFiCRccbWF4aW1lZGV2

b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7pPTAQDTaskqInlgIBD7g93IkaptLS2v

ddCGN9fojF0WKPGnYAEAivUCGWCEvCknev8qngB+KIRBDH2CG0ikXaMDXUlOIgM=

=+8KI

-----END PGP SIGNATURE-----

Your comment

This issue is archived.

To comment on this conversation send email to 44199@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date