875xltklod.fsf@cock.li
Hello everyone.
I think Distributed substitute distribution has been on many people's
radar for a time. However, from what I understood of the discussion at
https://issues.guix.gnu.org/33899(which has been stagnant for many
years), actually achieving this is complicated because of the lack of a
good way to encode into IPFS store items in a way that correctly
deduplicates identical files (giving them the same hash) and works well
with the rest of the IPFS ecosystem. Solving this problem is hard and
until a spec for UnixFSv2 is created and implemented in the main IPFS
implementations, making perfect use of all the IPFS features will be
harder and require a lot of effort.
However, I would really like to have distributed substitutes, even if it
is done through a less-than optimal mechanism, and I feel like (I might be
wrong) that is a common sentiment.
The attached patch series adds to the `guix publish` daemon an option
`--ipfs-api` which, when present, makes it publish through ipfs each
compressed nar file it creates, and adds a corresponding entry in the
narinfo through the URL: ipfs://Qm... property.
on the `guix substitute` side, an `ipfs-fetch` function is also added to
download nars through a local gateway. This means that the new URL:
ipfs:Qm... fields can now be exploited as easily as the regular
nar/lzip/... urls.
These changes don't exploit every advantage of IPFS delivery
(deduplication, "true decentralization", etc), but enable a non-trivial
use-case: many machines on a local network can now distribute amongst
themselves substitutes (thus reducing the total needed bandwidth)
transparently.
WDYT?
From 80934014c6f603347996f90f2b915ebdb6b224e7 Mon Sep 17 00:00:00 2001
Message-ID: <80934014c6f603347996f90f2b915ebdb6b224e7.1738470881.git.terramorpha@cock.li>
In-Reply-To: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li>
References: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li>
From: terramorpha <terramorpha@cock.li>
Date: Sat, 1 Feb 2025 22:16:29 -0500
Subject: [PATCH 3/4] doc: publish: Add documentation for the --ipfs-api
argument.
---
doc/guix.texi | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
Toggle diff (26 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index bb5f29277f..fe1b3c08d9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -104,7 +104,7 @@
Copyright @copyright{} 2022 Aleksandr Vityazev@*
Copyright @copyright{} 2022 Philip M@sup{c}Grath@*
Copyright @copyright{} 2022 Karl Hallsby@*
-Copyright @copyright{} 2022 Justin Veilleux@*
+Copyright @copyright{} 2022, 2025 Justin Veilleux@*
Copyright @copyright{} 2022 Reily Siegel@*
Copyright @copyright{} 2022 Simon Streit@*
Copyright @copyright{} 2022 (@*
@@ -16198,6 +16198,10 @@ Invoking guix publish
service discovery (DNS-SD), currently @i{via} Guile-Avahi (@pxref{Top,,,
guile-avahi, Using Avahi in Guile Scheme Programs}).
+When the @option{--ipfs-api=URL} argument is passed and caching is
+enabled, the server will also make its substitutes available through the
+@uref{https://ipfs.io,IPFS} daemon available at the given url.
+
The general syntax is:
@example
--
2.47.1
From f71fb156b6b217cea0a181db13d88ccce69f0b5d Mon Sep 17 00:00:00 2001
Message-ID: <f71fb156b6b217cea0a181db13d88ccce69f0b5d.1738470881.git.terramorpha@cock.li>
In-Reply-To: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li>
References: <07bf18b20faa86375d2eb80a49a6a6b36ad294f9.1738470881.git.terramorpha@cock.li>
From: terramorpha <terramorpha@cock.li>
Date: Sat, 1 Feb 2025 22:41:28 -0500
Subject: [PATCH 4/4] services: guix-publish: Expose and document the
--ipfs-api argument.
Change-Id: Iecabcc79b11a263fd77b25e51ce9d02fcb3aeddc
---
doc/guix.texi | 4 ++++
gnu/services/base.scm | 7 ++++++-
2 files changed, 10 insertions(+), 1 deletion(-)
Toggle diff (49 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index fe1b3c08d9..ccd7b7d73f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20352,6 +20352,10 @@ Base Services
When true, advertise the service on the local network @i{via} the DNS-SD
protocol, using Avahi.
+@item @code{ipfs-api} (default: @code{#f})
+The url to an IPFS node api, when present (and @code{cache} has a
+value), publish the compressed NARs on the ipfs network.
+
This allows neighboring Guix devices with discovery on (see
@code{guix-configuration} above) to discover this @command{guix publish}
instance and to automatically download substitutes from it.
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 7331c030d7..f8c586d189 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -2252,6 +2252,8 @@ (define-record-type* <guix-publish-configuration>
(default "localhost"))
(advertise? guix-publish-advertise? ;boolean
(default #f))
+ (ipfs-api guix-publish-ipfs-api ;string to a url
+ (default #f))
(compression guix-publish-configuration-compression
(thunked)
(default (default-compression this-record)))
@@ -2291,7 +2293,7 @@ (define (guix-publish-shepherd-service config)
(match-record config <guix-publish-configuration>
(guix port host nar-path cache workers ttl negative-ttl
- cache-bypass-threshold advertise?)
+ cache-bypass-threshold advertise? ipfs-api)
(let ((command #~(list #$(file-append guix "/bin/guix")
"publish" "-u" "guix-publish"
"-p" #$(number->string port)
@@ -2301,6 +2303,9 @@ (define (guix-publish-shepherd-service config)
#$@(if advertise?
#~("--advertise")
#~())
+ #$@(if ipfs-api
+ #~((string-append "--ipfs-api=" ipfs-api))
+ #~())
#$@(if workers
#~((string-append "--workers="
#$(number->string
--
2.47.1