[PATCH] IPFS service definition

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxime Devos
Owner
unassigned
Submitted by
Maxime Devos
Severity
normal
M
M
Maxime Devos wrote on 15 Jan 2021 22:22
(address . guix-patches@gnu.org)
af02fc3aaa36504cd63cdc5d48bb1f8f31a46d83.camel@telenet.be
Hi Guix!

A patch defining simple ‘ipfs-service-type’ is attached. I've tested
this in a VM, and will test it on a ‘real’ system later. The gateway
is currently broken, it tries to redirect to non-existent subdomains
of localhost. Correcting this might require fiddling with the DNS
configuration.

Maxime
--
Maxime Devos <maximedevos@telenet.be>
PGP Key: C1F3 3EE2 0C52 8FDB 7DD7 011F 49E3 EE22 1917 25EE
Freenode handle: mdevos
-----BEGIN PGP SIGNATURE-----

iI0EABYIADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYAIHrxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7g/qAQCPTRlr/clMlltv4me82S2moBhv
xC7RohwRHRNasEmS+wD/S7pRLgw4RRd9szucH/Vrf3oaligMOd5niX83n3Alzgg=
=EEA3
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 22 Mar 2021 18:17
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 45905@debbugs.gnu.org)
874kh3w1j6.fsf@gnu.org
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (6 lines)
> A patch defining simple ‘ipfs-service-type’ is attached. I've tested
> this in a VM, and will test it on a ‘real’ system later. The gateway
> is currently broken, it tries to redirect to non-existent subdomains
> of localhost. Correcting this might require fiddling with the DNS
> configuration.

OK. That doesn’t prevent one from using it, right?

Toggle quote (14 lines)
> From c441bca727df67837652eb2f0b5ad23528fd11a3 Mon Sep 17 00:00:00 2001
> From: Maxime Devos <maximedevos@telenet.be>
> Date: Fri, 15 Jan 2021 21:46:42 +0100
> Subject: [PATCH] services: Add ipfs-service-type
>
> * gnu/services/networking.scm (ipfs-service-type)
> (%ipfs-home-mapping, %ipfs-environment)
> (%ipfs-accounts, %ipfs-home): New variables.
> (ipfs-configuration, ipfs-configuration?)
> (ipfs-configuration-package, ipfs-configuration-gateway)
> (ipfs-configuration-api, ipfs-shepherd-service)
> (ipfs-binary, %ipfs-activation): New procedures.
> * doc/guix.texi (Networking Services): Document it.

[...]

Toggle quote (3 lines)
> +@lisp
> +;; part of the operating-system declaration

I think you can omit this line.

Toggle quote (5 lines)
> +(service ipfs-service-type
> + (ipfs-configuration
> + (gateway "/ip4/127.0.0.1/tcp/8880")
> + (api "/ip4/127.0.0.1/tcp/8881")))

Indentation is left (should be aligned with ‘ipfs-service-type’.)

Toggle quote (9 lines)
> + (start #~(make-forkexec-constructor/container
> + #$ipfs-daemon-command
> + #:namespaces '#$(fold delq %namespaces '(user net))
> + #:mappings (list #$%ipfs-home-mapping)
> + #:log-file "/var/log/ipfs.log"
> + #:user "ipfs"
> + #:group "ipfs"
> + #:environment-variables #$%ipfs-environment))

Nice!

Toggle quote (29 lines)
> + ;; Run ipfs init and ipfs config from a container,
> + ;; in case the IPFS daemon was compromised at some point
> + ;; and ~/.ipfs is now a symlink to somewhere outside
> + ;; %ipfs-home.
> + (define container-gexp
> + (with-extensions (list shepherd)
> + (with-imported-modules (source-module-closure
> + '((gnu build shepherd)
> + (gnu system file-systems)))
> + #~(begin
> + (use-modules (gnu build shepherd)
> + (gnu system file-systems))
> + (let* ((constructor
> + (make-forkexec-constructor/container
> + (list #$inner-script)
> + #:namespaces '#$(fold delq %namespaces '(user))
> + #:mappings (list #$%ipfs-home-mapping)
> + #:user "ipfs"
> + #:group "ipfs"
> + #:environment-variables #$%ipfs-environment))
> + (pid (constructor)))
> + (waitpid pid))))))
> + ;; The activation may happen from the initrd, which uses
> + ;; a statically-linked guile, while the guix container
> + ;; procedures require a working dynamic-link.
> + (define container-script
> + (program-file "ipfs-activation-container" container-gexp))
> + #~(system* #$container-script))

That’s a bit involved, but it makes sense to me.

The patch LGTM. However, we usually commit services along with a system
test under (gnu tests …). The manual has info on how to run individual
system tests:


Could you write a test that ensures that basic functionality works? It
could be as simple as waiting for the service to be up, then invoking
‘ipfs add’ and ‘ipfs get’. WDYT?

Thank you!

Ludo’.
M
M
Maxime Devos wrote on 22 Mar 2021 19:40
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 45905@debbugs.gnu.org)
36af87b3ec48ed159cc237dcac93320817c74f58.camel@telenet.be
On Mon, 2021-03-22 at 18:17 +0100, Ludovic Courtès wrote:
Toggle quote (1 lines)
> Hi Maxime,
Hi

Toggle quote (10 lines)
> Maxime Devos <maximedevos@telenet.be> skribis:
>
> > A patch defining simple ‘ipfs-service-type’ is attached. I've tested
> > this in a VM, and will test it on a ‘real’ system later. The gateway
> > is currently broken, it tries to redirect to non-existent subdomains
> > of localhost. Correcting this might require fiddling with the DNS
> > configuration.
>
> OK. That doesn’t prevent one from using it, right?

Nah, the REST API presumably works just fine and there is plenty to see on
the webui:


Not perfect, but it might suffice for your purposes.
That reminds me the configuration can be modified from there.
I didn't figure how to disable that. Not ideal from a security
perspective, but at least its only loopback & ipfs is in a container.

Toggle quote (5 lines)
> > +@lisp
> > +;; part of the operating-system declaration

> I think you can omit this line.

I think I found that line somewhere & copied it for consistency,
but it has been some time ago.

Toggle quote (7 lines)
> > +(service ipfs-service-type
> > + (ipfs-configuration
> > + (gateway "/ip4/127.0.0.1/tcp/8880")
> > + (api "/ip4/127.0.0.1/tcp/8881")))
>
> Indentation is left (should be aligned with ‘ipfs-service-type’.)

Ok, not sure how this happened.

Toggle quote (5 lines)
> > + (start #~(make-forkexec-constructor/container
> > + [container stuff]
> > + #:environment-variables #$%ipfs-environment))
>
> Nice!
Yep! Also, this reminds me I'm not sure what the distinction between
#+ and #~ is in activation gexps, in shepherd services definitions,
etc.

Toggle quote (8 lines)
> > + ;; Run ipfs init and ipfs config from a container,
> > + ;; in case the IPFS daemon was compromised at some point
> > + ;; and ~/.ipfs is now a symlink to somewhere outside
> > + ;; %ipfs-home.
> > + (define container-gexp [complicated container stuff])
> >
> That’s a bit involved, but it makes sense to me.

Unfortunately, there are (non-container related) some more issues.
Last few weeks I've been seeing this error (/var/log/ipfs.log):

(start snip)
Error: fs-repo requires migration
Initializing daemon...
go-ipfs version: 0.8.0
Repo version: 11
System version: amd64/linux
Golang version: go1.14.15
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] Not running migrations of fs-repo now.
Please get fs-repo-migrations from https://dist.ipfs.io

Error: fs-repo requires migration
(end snip)

(Super hacky work-around:
rm -r /var/lib/ipfs
mkdir /var/lib/ipfs
chmod a-rwx /var/lib/ipfs
chmod u+rwx /var/lib/ipfs
chown ipfs:ipfs /var/lib/ipfs
sudo -u ipfs -g ipfs "`guix build go-ipfs`/bin/ipfs" init
# ^ this can take some seconds to complete
sudo -u ipfs -g ipfs "`guix build go-ipfs`/bin/ipfs" config API /ip4/127.0.0.1/tcp/5001
sudo -u ipfs -g ipfs "`guix build go-ipfs`/bin/ipfs" config Addresses.Gateway /ip4/127.0.0.1/tcp/8082
herd enable ipfs
herd start ipfs)

Unfortunately "fs-repo-migrations" does not seem to be packaged in Guix.
Apparently there has been a change in repo format in the go-ipfs v0.7.0
--> v0.8.0 upgrade. I believe for most users simply automatically running
the upgrades would be sufficient.

Now, how could we do this safely from shepherd? Maybe before starting open
a pipe, write "y\n" to it an pass it as file descriptor 0 (stdin) would
be sufficient? But shepherd always closes /dev/stdin before exec IIRC ..

Seems like shepherd needs support for file descriptor! I've a patch for
that, but it needs to be verified (& corrected likely) on GNU/Hurd. Feel
free to ask for the incomplete patch if you're impatient and want to finish
it yourself! (Seems to work on GNU/Linux in any case.)

Toggle quote (10 lines)
> The patch LGTM. However, we usually commit services along with a system
> test under (gnu tests …). The manual has info on how to run individual
> system tests:
>
> https://guix.gnu.org/manual/en/html_node/Running-the-Test-Suite.html
>
> Could you write a test that ensures that basic functionality works? It
> could be as simple as waiting for the service to be up, then invoking
> ‘ipfs add’ and ‘ipfs get’. WDYT?

Will look into it eventually, but I am currently occupied with other things
that have deadlines )-:. (Not feeling very inspired for a
writing/presentation assignment ...) (And I would rather hack on GNUnet
frankly; IPFS is more of a stop-gap to me for having some distributed
something for substitutes.) So feel free to beat me to it.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYIADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYFjkpRccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7q74AQCWznKm3gQg4zKipnOle2hpQbEo
x1mPjbKLJIXMYSk9DgEAwU30UR0baPI32ym4YvZORhhywcWH+B9J70naA0VerA4=
=IBUS
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 23 Mar 2021 14:08
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 45905@debbugs.gnu.org)
871rc6koew.fsf@gnu.org
Hi Maxime!

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (12 lines)
>> OK. That doesn’t prevent one from using it, right?
>
> Nah, the REST API presumably works just fine and there is plenty to see on
> the webui:
>
> http://localhost:5001/ipfs/bafybeif4zkmu7qdhkpf3pnhwxipylqleof7rl6ojbe7mq3fzogz6m4xk3i/#/
>
> Not perfect, but it might suffice for your purposes.
> That reminds me the configuration can be modified from there.
> I didn't figure how to disable that. Not ideal from a security
> perspective, but at least its only loopback & ipfs is in a container.

Good.

[...]

Toggle quote (4 lines)
> Yep! Also, this reminds me I'm not sure what the distinction between
> #+ and #~ is in activation gexps, in shepherd services definitions,
> etc.

#+ is ‘ungexp-native’. It makes sense if you consider a cross-compiled
system. Code in an activation gexp is meant to run on the target
system, so you want to use #$ (‘ungexp’) there.

You might want to use #+ when building things that can just as well be
built natively. For instance, the background image for GRUB must be
built by running Inkscape natively on the host system, so we use
#+inkscape (or similar) to do that.

I hope that makes sense.

Toggle quote (17 lines)
> Unfortunately, there are (non-container related) some more issues.
> Last few weeks I've been seeing this error (/var/log/ipfs.log):
>
> (start snip)
> Error: fs-repo requires migration
> Initializing daemon...
> go-ipfs version: 0.8.0
> Repo version: 11
> System version: amd64/linux
> Golang version: go1.14.15
> Found outdated fs-repo, migrations need to be run.
> Run migrations now? [y/N] Not running migrations of fs-repo now.
> Please get fs-repo-migrations from https://dist.ipfs.io
>
> Error: fs-repo requires migration
> (end snip)

Bah, I remember seeing that.

Toggle quote (5 lines)
> Unfortunately "fs-repo-migrations" does not seem to be packaged in Guix.
> Apparently there has been a change in repo format in the go-ipfs v0.7.0
> --> v0.8.0 upgrade. I believe for most users simply automatically running
> the upgrades would be sufficient.

Yes, I think so. We “just” need to package ‘fs-repo-migrations’ first.

Perhaps it’d be okay, as a first step, to provide an IPFS service that
doesn’t handle migrations automatically.

Toggle quote (4 lines)
> Now, how could we do this safely from shepherd? Maybe before starting open
> a pipe, write "y\n" to it an pass it as file descriptor 0 (stdin) would
> be sufficient? But shepherd always closes /dev/stdin before exec IIRC ..

You could have the ‘ipfs’ Shepherd service depend on, say, a one-shot
‘ipfs-migration’ service. The ‘ipfs-migration’ service would run
‘fs-repo-migrations’ if it’s necessary.

Toggle quote (16 lines)
>> The patch LGTM. However, we usually commit services along with a system
>> test under (gnu tests …). The manual has info on how to run individual
>> system tests:
>>
>> https://guix.gnu.org/manual/en/html_node/Running-the-Test-Suite.html
>>
>> Could you write a test that ensures that basic functionality works? It
>> could be as simple as waiting for the service to be up, then invoking
>> ‘ipfs add’ and ‘ipfs get’. WDYT?
>
> Will look into it eventually, but I am currently occupied with other things
> that have deadlines )-:. (Not feeling very inspired for a
> writing/presentation assignment ...) (And I would rather hack on GNUnet
> frankly; IPFS is more of a stop-gap to me for having some distributed
> something for substitutes.) So feel free to beat me to it.

I’m not offering to work on it :-), but hopefully you or maybe some
fellow contributor can finish it up in the coming weeks!

Thanks,
Ludo’.
M
M
Maxime Devos wrote on 28 Mar 2021 18:36
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 45905@debbugs.gnu.org)
4e1df2f011f5766d48c44d7231e562a889239e7a.camel@telenet.be
On Tue, 2021-03-23 at 14:08 +0100, Ludovic Courtès wrote:
Toggle quote (8 lines)
> [...]
> Yes, I think so. We “just” need to package ‘fs-repo-migrations’ first.
>
> Perhaps it’d be okay, as a first step, to provide an IPFS service that
> doesn’t handle migrations automatically.
>
> [...]

Punt for later.

Toggle quote (12 lines)
> > > The patch LGTM. However, we usually commit services along with a system
> > > test under (gnu tests …). The manual has info on how to run individual
> > > system tests:
> > >
> > > https://guix.gnu.org/manual/en/html_node/Running-the-Test-Suite.html
> > >
> > > Could you write a test that ensures that basic functionality works? It
> > > could be as simple as waiting for the service to be up, then invoking
> > > ‘ipfs add’ and ‘ipfs get’. WDYT?
> >
> > [...]

I have attached a revised patch series testing such basic functionality.
However, I tested the functionality with the HTTP interface instead of
with the command line, as the CLI tools assume the IPFS daemon is run
as the same user as the CLI tools.

(IIRC there is an implementation of the CLI tools somewhere that uses
the API endpoint instead of reading/writing to ~/.ipfs, but I don't
recall where.)

I have a problem: I can't run the test I wrote.

$ make && make check-system TESTS="ipfs"
Toggle quote (5 lines)
> [compilation bla bla]
> Compiling Scheme modules...
> Selected 1 system tests...
> (end of output)

For some reason, the test I wrote seems to be ignored.

(Also available from my public git repo:

Greetings,
Maxime.
From 732c018b9d24f0f36700c6f8715e989ee3d94663 Mon Sep 17 00:00:00 2001
From: Maxime Devos <maximedevos@telenet.be>
Date: Sun, 28 Mar 2021 17:01:49 +0200
Subject: [PATCH 3/3] gnu: tests: Test basic funtionality of the IPFS service.

It is tested whether the IPFS service listens
at the gateway and API ports and whether it
is possible to upload and download a bytevector.

TODO: this test isn't run for some reason:

$ make && make check-system TESTS="ipfs"
Toggle quote (5 lines)
> [compilation bla bla]
> Compiling Scheme modules...
> Selected 1 system tests...
> (end of output)

??? why isn't the IPFS test executed?

* gnu/tests/networking.scm
(%ipfs-os): New variable.
(run-ipfs-test): New procedure.
(%test-ipfs): New system test.
---
gnu/tests/networking.scm | 81 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 80 insertions(+), 1 deletion(-)

Toggle diff (105 lines)
diff --git a/gnu/tests/networking.scm b/gnu/tests/networking.scm
index 022663aa67..f886eac881 100644
--- a/gnu/tests/networking.scm
+++ b/gnu/tests/networking.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2017, 2020 Marius Bakke <marius@gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net>
+;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -34,7 +35,8 @@
#:use-module (gnu packages networking)
#:use-module (gnu services shepherd)
#:use-module (ice-9 match)
- #:export (%test-inetd %test-openvswitch %test-dhcpd %test-tor %test-iptables))
+ #:export (%test-inetd %test-openvswitch %test-dhcpd %test-tor %test-iptables
+ %test-ipfs))
(define %inetd-os
;; Operating system with 2 inetd services.
@@ -563,3 +565,80 @@ COMMIT
(name "iptables")
(description "Test a running iptables daemon.")
(value (run-iptables-test))))
+
+
+;;;
+;;; IPFS service
+;;;
+
+(define %ipfs-os
+ (simple-operating-system
+ (service ipfs-service-type)))
+
+(define (run-ipfs-test)
+ (define os
+ (marionette-operating-system %ipfs-os
+ #:imported-modules '((gnu services herd)
+ (guix ipfs))
+ #:requirements '(ipfs)))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (gnu build marionette)
+ (srfi srfi-64))
+
+ (define marionette
+ (make-marionette (list #$(virtual-machine os))))
+
+ (define (ipfs-is-alive?)
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd)
+ (srfi srfi-1))
+ (live-service-running
+ (find (lambda (live)
+ (memq 'ipfs
+ (live-service-provision live)))
+ (current-services))))
+ marionette))
+
+ ;; The default API endpoint port 5001 is used,
+ ;; so there is no need to parameterize %ipfs-base-url.
+ ;; By running this within the VM instead of outside the VM
+ ;; this system test does not have to forward any ports.
+ (define (add-data data)
+ (marionette-eval
+ `((@ (guix ipfs) add-contents) ,data)))
+ (define (read-contents object)
+ (marionette-eval
+ `((@ (guix ipfs) read-contents) ,object)))
+
+ (test-begin "ipfs")
+
+ ;; Test the IPFS service.
+
+ (test-assert "ipfs is alive" (ipfs-is-alive?))
+
+ (test-assert "ipfs is listening on the gateway"
+ (let ((default-port 8082))
+ (wait-for-tcp-port default-port marionette)))
+
+ (test-assert "ipfs is listening on the API endpoint"
+ (let ((default-port 5001))
+ (wait-for-tcp-port default-port marionette)))
+
+ (define test-bv (string->utf8 "hello ipfs!"))
+ (test-equal "can upload and download a file to/from ipfs"
+ test-bv
+ (read-contents (add-data test-bv)))
+
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+ (gexp->derivation "ipfs-test" test))
+
+(define %test-ipfs
+ (system-test
+ (name "ipfs")
+ (description "Test a running IPFS daemon configuration.")
+ (value (run-ipfs-test))))
--
2.31.0
-----BEGIN PGP SIGNATURE-----

iI0EABYIADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYGCwfhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7uXNAP918P4ZU2oJNrfzEN6M7uMhHzCn
8ehL24WfkJX/lXOKwAD6A+rtWQ1HNlYKp0Eld0MZwM45FPBnjtoV0k+6T100BQs=
=ac5P
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 29 Mar 2021 16:07
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 45905@debbugs.gnu.org)
877dlqoxy9.fsf@gnu.org
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (10 lines)
> TODO: this test isn't run for some reason:
>
> $ make && make check-system TESTS="ipfs"
>> [compilation bla bla]
>> Compiling Scheme modules...
>> Selected 1 system tests...
>> (end of output)
>
> ??? why isn't the IPFS test executed?

[...]

Toggle quote (41 lines)
> +(define (run-ipfs-test)
> + (define os
> + (marionette-operating-system %ipfs-os
> + #:imported-modules '((gnu services herd)
> + (guix ipfs))
> + #:requirements '(ipfs)))
> +
> + (define test
> + (with-imported-modules '((gnu build marionette))
> + #~(begin
> + (use-modules (gnu build marionette)
> + (srfi srfi-64))
> +
> + (define marionette
> + (make-marionette (list #$(virtual-machine os))))
> +
> + (define (ipfs-is-alive?)
> + (marionette-eval
> + '(begin
> + (use-modules (gnu services herd)
> + (srfi srfi-1))
> + (live-service-running
> + (find (lambda (live)
> + (memq 'ipfs
> + (live-service-provision live)))
> + (current-services))))
> + marionette))
> +
> + ;; The default API endpoint port 5001 is used,
> + ;; so there is no need to parameterize %ipfs-base-url.
> + ;; By running this within the VM instead of outside the VM
> + ;; this system test does not have to forward any ports.
> + (define (add-data data)
> + (marionette-eval
> + `((@ (guix ipfs) add-contents) ,data)))
> + (define (read-contents object)
> + (marionette-eval
> + `((@ (guix ipfs) read-contents) ,object)))
> +
> + (test-begin "ipfs")

[...]

Toggle quote (2 lines)
> + (gexp->derivation "ipfs-test" test))

You need to add:

(mkdir #$output)
(chdir #$output)

right before (test-begin "ipfs").

Failing to do that, you create “ipfs-test.drv” as a zero-output
derivation—i.e., a derivation that doesn’t produce any output. Since it
produces nothing, the daemon doesn’t bother running its code.

Nitpick: please avoid ‘@’. Instead, explicitly do:

(marionette-eval '(use-modules (ipfs)) marionette)

Alternatively, you can arrange to set up port forwarding for the VM and
use the (ipfs) module from the host rather than from the guest. This is
what (gnu tests ssh) does, for example.

As it stands, the test fails because you need to:

(define test
(with-extensions (list guile-json)
…))

so that Guile-JSON is available, and probably also:

(with-imported-modules '((ipfs))
…)

The rest LGTM.

HTH!

Ludo’.
L
L
Ludovic Courtès wrote on 29 Mar 2021 16:06
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 45905@debbugs.gnu.org)
878s66oxzj.fsf@gnu.org
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (10 lines)
> TODO: this test isn't run for some reason:
>
> $ make && make check-system TESTS="ipfs"
>> [compilation bla bla]
>> Compiling Scheme modules...
>> Selected 1 system tests...
>> (end of output)
>
> ??? why isn't the IPFS test executed?

[...]

Toggle quote (41 lines)
> +(define (run-ipfs-test)
> + (define os
> + (marionette-operating-system %ipfs-os
> + #:imported-modules '((gnu services herd)
> + (guix ipfs))
> + #:requirements '(ipfs)))
> +
> + (define test
> + (with-imported-modules '((gnu build marionette))
> + #~(begin
> + (use-modules (gnu build marionette)
> + (srfi srfi-64))
> +
> + (define marionette
> + (make-marionette (list #$(virtual-machine os))))
> +
> + (define (ipfs-is-alive?)
> + (marionette-eval
> + '(begin
> + (use-modules (gnu services herd)
> + (srfi srfi-1))
> + (live-service-running
> + (find (lambda (live)
> + (memq 'ipfs
> + (live-service-provision live)))
> + (current-services))))
> + marionette))
> +
> + ;; The default API endpoint port 5001 is used,
> + ;; so there is no need to parameterize %ipfs-base-url.
> + ;; By running this within the VM instead of outside the VM
> + ;; this system test does not have to forward any ports.
> + (define (add-data data)
> + (marionette-eval
> + `((@ (guix ipfs) add-contents) ,data)))
> + (define (read-contents object)
> + (marionette-eval
> + `((@ (guix ipfs) read-contents) ,object)))
> +
> + (test-begin "ipfs")

[...]

Toggle quote (2 lines)
> + (gexp->derivation "ipfs-test" test))

You need to add:

(mkdir #$output)
(chdir #$output)

right before (test-begin "ipfs").

Failing to do that, you create “ipfs-test.drv” as a zero-output
derivation—i.e., a derivation that doesn’t produce any output. Since it
produces nothing, the daemon doesn’t bother running its code.

Nitpick: please avoid ‘@’. Instead, explicitly do:

(marionette-eval '(use-modules (ipfs)) marionette)

Alternatively, you can arrange to set up port forwarding for the VM and
use the (ipfs) module from the host rather than from the guest. This is
what (gnu tests ssh) does, for example.

As it stands, the test fails because you need to:

(define test
(with-extensions (list guile-json)
…))

so that Guile-JSON is available, and probably also:

(with-imported-modules '((ipfs))
…)

The rest LGTM.

HTH!

Ludo’.
M
M
Maxime Devos wrote on 30 Mar 2021 15:37
[PATCH v3] IPFS service definition
(address . 45905@debbugs.gnu.org)
42e840c5a55968ecf8173e4eb84af7dc415f3a51.camel@telenet.be
Hi Guix,

Revised patch series is attached.

Changes in v2:

* let the shepherd service depend on (loopback) instead of
(networking)
* added a (broken) system test for the IPFS service

Changes in v3:

* added 'extensions' argument to 'marionette-operating-system'.
* fixed the system test
* tweaked the documentation formatting and removed a misleading comment
on forwarding (port forwarding would be fine here, as the test is run
in a container so there is no risk of port conflicts IIUC)

Ludovic Courtès wrote:
Toggle quote (3 lines)
> Nitpick: please avoid ‘@’. Instead, explicitly do:
> [...]

I actually prefer '(@ (...) ...)' here, but whatever. It's changed
in v3.

Toggle quote (5 lines)
> As it stands, the test fails because you need to:
> (define test
> (with-extensions (list guile-json)
> …))

As the uploading and downloading is done in the guest, not the host,
this needs to be done somewhat differently. That's what the patch
‘tests: Support package extensions in the backdoor REPL’ is for.

Greetings,
Maxime
From bbf35272775de63ad64aed98a2fa081374f28505 Mon Sep 17 00:00:00 2001
From: Maxime Devos <maximedevos@telenet.be>
Date: Tue, 30 Mar 2021 12:40:14 +0200
Subject: [PATCH 3/4] tests: Support package extensions in the backdoor REPL.

* gnu/tests.scm
(<marionette-configuration>): Add 'extensions' field.
(marionette-shepherd-service): Honour the field.
(with-import-modules-and-extensions): Define a combination
of 'with-import-modules' and 'with-extensions'.
---
gnu/tests.scm | 26 +++++++++++++++++++++++---
1 file changed, 23 insertions(+), 3 deletions(-)

Toggle diff (82 lines)
diff --git a/gnu/tests.scm b/gnu/tests.scm
index 3b10a6d5ac..eb636873a2 100644
--- a/gnu/tests.scm
+++ b/gnu/tests.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
;;; Copyright © 2017 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -74,13 +75,24 @@
(default "/dev/virtio-ports/org.gnu.guix.port.0"))
(imported-modules marionette-configuration-imported-modules
(default '()))
+ (extensions marionette-configuration-extensions
+ (default '())) ; list of packages
(requirements marionette-configuration-requirements ;list of symbols
(default '())))
+;; Hack: avoid indenting code beyond column 80 in marionette-shepherd-service.
+(define-syntax-rule (with-imported-modules-and-extensions imported-modules
+ extensions
+ gexp)
+ (with-imported-modules imported-modules
+ (with-extensions extensions
+ gexp)))
+
(define (marionette-shepherd-service config)
"Return the Shepherd service for the marionette REPL"
(match config
- (($ <marionette-configuration> device imported-modules requirement)
+ (($ <marionette-configuration> device imported-modules extensions
+ requirement)
(list (shepherd-service
(provision '(marionette))
@@ -90,7 +102,7 @@
(modules '((ice-9 match)
(srfi srfi-9 gnu)))
(start
- (with-imported-modules imported-modules
+ (with-imported-modules-and-extensions imported-modules extensions
#~(lambda ()
(define (self-quoting? x)
(letrec-syntax ((one-of (syntax-rules ()
@@ -154,11 +166,13 @@
(define* (marionette-operating-system os
#:key
(imported-modules '())
+ (extensions '())
(requirements '()))
"Return a marionetteed variant of OS such that OS can be used as a
marionette in a virtual machine--i.e., controlled from the host system. The
marionette service in the guest is started after the Shepherd services listed
-in REQUIREMENTS."
+in REQUIREMENTS. The packages in the list EXTENSIONS are made available from
+the backdoor REPL."
(operating-system
(inherit os)
;; Make sure the guest dies on error.
@@ -172,6 +186,7 @@ in REQUIREMENTS."
(services (cons (service marionette-service-type
(marionette-configuration
(requirements requirements)
+ (extensions extensions)
(imported-modules imported-modules)))
(operating-system-user-services os)))))
@@ -281,4 +296,9 @@ result."
"Return the list of system tests."
(reverse (fold-system-tests cons '())))
+
+;; Local Variables:
+;; eval: (put 'with-imported-modules-and-extensions 'scheme-indent-function 2)
+;; End:
+
;;; tests.scm ends here
--
2.31.1
From b9134c60d9e662dd497caf0c1819e3e04a5e8b4e Mon Sep 17 00:00:00 2001
From: Maxime Devos <maximedevos@telenet.be>
Date: Sun, 28 Mar 2021 17:01:49 +0200
Subject: [PATCH 4/4] gnu: tests: Test basic funtionality of the IPFS service.

It is tested whether the IPFS service listens
at the gateway and API ports and whether it
is possible to upload and download a bytevector.

* gnu/tests/networking.scm
(%ipfs-os): New variable.
(run-ipfs-test): New procedure.
(%test-ipfs): New system test.
---
gnu/tests/networking.scm | 92 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 91 insertions(+), 1 deletion(-)

Toggle diff (121 lines)
diff --git a/gnu/tests/networking.scm b/gnu/tests/networking.scm
index 022663aa67..453e63f52d 100644
--- a/gnu/tests/networking.scm
+++ b/gnu/tests/networking.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2017, 2020 Marius Bakke <marius@gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net>
+;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -29,12 +30,15 @@
#:use-module (guix gexp)
#:use-module (guix store)
#:use-module (guix monads)
+ #:use-module (guix modules)
#:use-module (gnu packages bash)
#:use-module (gnu packages linux)
#:use-module (gnu packages networking)
+ #:use-module (gnu packages guile)
#:use-module (gnu services shepherd)
#:use-module (ice-9 match)
- #:export (%test-inetd %test-openvswitch %test-dhcpd %test-tor %test-iptables))
+ #:export (%test-inetd %test-openvswitch %test-dhcpd %test-tor %test-iptables
+ %test-ipfs))
(define %inetd-os
;; Operating system with 2 inetd services.
@@ -563,3 +567,89 @@ COMMIT
(name "iptables")
(description "Test a running iptables daemon.")
(value (run-iptables-test))))
+
+
+;;;
+;;; IPFS service
+;;;
+
+(define %ipfs-os
+ (simple-operating-system
+ (service ipfs-service-type)))
+
+(define (run-ipfs-test)
+ (define os
+ (marionette-operating-system %ipfs-os
+ #:imported-modules (source-module-closure
+ '((gnu services herd)
+ (guix ipfs)))
+ #:extensions (list guile-json-4)
+ #:requirements '(ipfs)))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (gnu build marionette)
+ (rnrs bytevectors)
+ (srfi srfi-64)
+ (ice-9 binary-ports))
+
+ (define marionette
+ (make-marionette (list #$(virtual-machine os))))
+
+ (define (ipfs-is-alive?)
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd)
+ (srfi srfi-1))
+ (live-service-running
+ (find (lambda (live)
+ (memq 'ipfs
+ (live-service-provision live)))
+ (current-services))))
+ marionette))
+
+ ;; The default API endpoint port 5001 is used,
+ ;; so there is no need to parameterize %ipfs-base-url.
+ (define (add-data data)
+ (marionette-eval `(content-name (add-data ,data)) marionette))
+ (define (read-contents object)
+ (marionette-eval
+ `(let* ((input (read-contents ,object))
+ (all-input (get-bytevector-all input)))
+ (close-port input)
+ all-input)
+ marionette))
+
+ (marionette-eval '(use-modules (guix ipfs)) marionette)
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin "ipfs")
+
+ ;; Test the IPFS service.
+
+ (test-assert "ipfs is alive" (ipfs-is-alive?))
+
+ (test-assert "ipfs is listening on the gateway"
+ (let ((default-port 8082))
+ (wait-for-tcp-port default-port marionette)))
+
+ (test-assert "ipfs is listening on the API endpoint"
+ (let ((default-port 5001))
+ (wait-for-tcp-port default-port marionette)))
+
+ (define test-bv (string->utf8 "hello ipfs!"))
+ (test-equal "can upload and download a file to/from ipfs"
+ test-bv
+ (read-contents (add-data test-bv)))
+
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+ (gexp->derivation "ipfs-test" test))
+
+(define %test-ipfs
+ (system-test
+ (name "ipfs")
+ (description "Test a running IPFS daemon configuration.")
+ (value (run-ipfs-test))))
--
2.31.1
-----BEGIN PGP SIGNATURE-----

iI0EABYIADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYGMpqhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7it8AP9QGignIWYu/Ld+eendn57JkHrY
2vqwGGGXSyU2EeT4RAEA11x3F5zE5cLTG9bqejM/7mV7aNYLugGNqvn+lyEa/As=
=oRrp
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 12 Apr 2021 18:48
Re: bug#45905: [PATCH] IPFS service definition
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 45905-done@debbugs.gnu.org)
87im4ro3e8.fsf_-_@gnu.org
Hi Maxime,

Apologies for the delay — I had not seen this new version.

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (14 lines)
> From 74149efb0dbd1b412fdd14aa87bee80640ea5463 Mon Sep 17 00:00:00 2001
> From: Maxime Devos <maximedevos@telenet.be>
> Date: Fri, 15 Jan 2021 21:46:42 +0100
> Subject: [PATCH 1/4] services: Add ipfs-service-type
>
> * gnu/services/networking.scm (ipfs-service-type)
> (%ipfs-home-mapping, %ipfs-environment)
> (%ipfs-accounts, %ipfs-home): New variables.
> (ipfs-configuration, ipfs-configuration?)
> (ipfs-configuration-package, ipfs-configuration-gateway)
> (ipfs-configuration-api, ipfs-shepherd-service)
> (ipfs-binary, %ipfs-activation): New procedures.
> * doc/guix.texi (Networking Services): Document it.

[...]

Toggle quote (17 lines)
> From c1ca4e25ff35fabe89fc7a8b2b4d3521840236c9 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Fri, 28 Dec 2018 01:07:58 +0100
> Subject: [PATCH 2/4] Add (guix ipfs).
>
> This module allows for communicating with the IPFS
> gateway over the HTTP interface. The commit has been
> cherry-picked from <https://issues.guix.gnu.org/33899>.
>
> The procedures for adding and restoring file trees have
> been removed as according to a reply issue 33899, a different
> format will be used. The procedure 'add-data' has been
> exported as it will be used in the system test for IPFS.
>
> * guix/ipfs.scm: New file.
> * Makefile.am (MODULES): Add it.

[...]

Toggle quote (11 lines)
> From bbf35272775de63ad64aed98a2fa081374f28505 Mon Sep 17 00:00:00 2001
> From: Maxime Devos <maximedevos@telenet.be>
> Date: Tue, 30 Mar 2021 12:40:14 +0200
> Subject: [PATCH 3/4] tests: Support package extensions in the backdoor REPL.
>
> * gnu/tests.scm
> (<marionette-configuration>): Add 'extensions' field.
> (marionette-shepherd-service): Honour the field.
> (with-import-modules-and-extensions): Define a combination
> of 'with-import-modules' and 'with-extensions'.

[...]

Toggle quote (14 lines)
> From b9134c60d9e662dd497caf0c1819e3e04a5e8b4e Mon Sep 17 00:00:00 2001
> From: Maxime Devos <maximedevos@telenet.be>
> Date: Sun, 28 Mar 2021 17:01:49 +0200
> Subject: [PATCH 4/4] gnu: tests: Test basic funtionality of the IPFS service.
>
> It is tested whether the IPFS service listens
> at the gateway and API ports and whether it
> is possible to upload and download a bytevector.
>
> * gnu/tests/networking.scm
> (%ipfs-os): New variable.
> (run-ipfs-test): New procedure.
> (%test-ipfs): New system test.

Pushed all four patches as 68c9e0a56e008f19427bd213cf5b24bdd8fe5922.

Thanks!

Ludo’.
Closed
M
M
Maxime Devos wrote on 12 Apr 2021 20:35
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 45905-done@debbugs.gnu.org)
76c6d139042ddb60de88253af28393a258c4a599.camel@telenet.be
On Mon, 2021-04-12 at 18:48 +0200, Ludovic Courtès wrote:
Toggle quote (6 lines)
> [...]
>
> Pushed all four patches as 68c9e0a56e008f19427bd213cf5b24bdd8fe5922.
>
> Thanks!

Thanks!

Maxime.
Closed
?