[PATCH] Add package go-ethereum

DoneSubmitted by Helio Machado.
Details
4 participants
  • Helio Machado
  • Attila Lendvai
  • Marius Bakke
  • zimoun
Owner
unassigned
Severity
normal
H
H
Helio Machado wrote on 8 Oct 2020 23:59
(address . guix-patches@gnu.org)
CANe01w45Erh3=GbYek8X0PrisfZ0zGCG6tMRH-hcz6LaCUVjzA@mail.gmail.com
# Add package `go-ethereum`

These patches add the [go-ethereum][5] package and a small workaround that
allows experimental module support.

## [Checklist][1]

1. Code authentication: all the code is being pulled from Git repositories
and Go packages, so there isn't any specific code signature we can verify.

2. Synopsis and description: built from the official documentation and
around the style guidelines.

3. Results for `guix lint`: everything seems to be fine, and it seems to
have scheduled a [Software Heritage][2] archival (?).

4. Native build check: the package builds correctly on my platform
(x86-64), using `guix build` package.

5. Cross-build check: package was not tested on any other supported
platform because `go-build-system` [doesn't seem to support
cross-compilation][3].

6. Package deduplication: dependencies are being pulled from their
respective module repositories at build time; there isn't practically any
overlap between the package code and the Guix repositories.

7. Package size: as per `guix size`, this package requires a total of 418.7
MiB, with a package (self) size of 282.4 MiB.

8. Dependent packages: this is a freshly added package with no dependent
items.

9. Deterministic build: this package is fully deterministic and has fixed
hashes for both the initial download and the build derivation.

10. Inclusive language: documentation was written with neutrality in mind
and there isn't any obvious bias, though it might need proofreading.

11. Patch atomicity: this patch contains only one set of related changes,
though modifications affect two files.

12. Code formatting: added lines were formatted by using `indent-code.el`.

13. Source consistency: all the involved downloads are being pulled along
with a Git tag or commit, so results should be consistent enough.

14. Guix build: these changes don't affect Guix builds nor introduce any
warning.

15. Guix consistency: these changes do not break Guix in any obvious way.

## Desiderata

It would be great to have easier ways to replace Go module management
system with Guix in a seamless and painless way. Rust build system for Guix
has a cleaner implementation, majorly due to the existence of a
[centralized package registry][4].

[3]:
Attachment: file
H
H
Helio Machado wrote on 9 Oct 2020 14:11
Closing due to consistency issues
(address . 43872-done@debbugs.gnu.org)
CANe01w7ypkUjtc2p2TdZewDrC6xD-AExZno5a7uOhZrEZn=VQg@mail.gmail.com
This patch was a big hack and breaks on every architecture but x86-64
because the derivation hash changes when producing different binaries. I'm
leaving this for reference as it might be useful for a casual install, but
the proper solution probably involves a complete refactoring of the Guix Go
Build System.
Attachment: file
Closed
M
M
Marius Bakke wrote on 9 Oct 2020 20:34
87eem7fdt8.fsf@gnu.org
Hi!

Helio Machado <0x2b3bfa0@gmail.com> writes:

Toggle quote (6 lines)
> This patch was a big hack and breaks on every architecture but x86-64
> because the derivation hash changes when producing different binaries. I'm
> leaving this for reference as it might be useful for a casual install, but
> the proper solution probably involves a complete refactoring of the Guix Go
> Build System.

That is a very clever hack. :-)

The reason it works is because network access is allowed for
fixed-output derivations. So the Go build system is able to go online
and download everything it needs, instead of relying on declared
inputs. That is why different architectures produce different
derivation hashes: those are the build outputs.

The problem with this approach is that it is not deterministic. The
next time someone runs the same build, one of the Go dependencies may
have changed (i.e. a git tag has been overwritten, or a package declares
a dependency on a branch instead of a tag, etc).

Guix will detect if the output hash changes, but is unable reproduce the
earlier state because it does not know what has changed.

To properly package go-ethereum, you will need to declare each
dependency as separate packages. See e.g. 'syncthing' for an example.
Unfortunately there is no importer for Go yet, making it a rather
tedious task.

Since you don't seem to be afraid of hacking on Guix :-) consider
writing an "importer" for Go so you could simply:

guix import go -r github.com/ethereum/go-ethereum

...to create the necessary boilerplate.

Thanks!
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAl+ArVQACgkQoqBt8qM6
VPqrEAf/WeFN250sGAcjy9QGh8s1nHIfgAeqLPhz3lgdsscBfzcuzRp5nycqphXa
7vkPNWUfE2FrfPEgZDweD+9oJMpKDkGewbtskkLiBrXZj5sQcwROZUXKhM00y7Ci
WEGAeEv4tZFzC2+g4QFqmEYd+SOM7EhUUy09RioA2aF7PTGPA+yRHfkbHMNJRwK+
cGxju8lIhEzX/NZ3Fy+qDcaU77oVn5EIDOcmIDn6aB7h+aeXb4ZufVuQ+TdJcgGE
/6WJCqwsVF0e7yLrjixWnr2PqkFbV5xCW7nlEhTJxiheQijJZE7Zwond0Qgj4a1h
GA5BVqfkCotaAwDFUglAvjPk9LOoPQ==
=h9Md
-----END PGP SIGNATURE-----

Closed
H
H
Helio Machado wrote on 10 Oct 2020 20:07
CANe01w4SNUM0j7bdduM0WJvsDFkExBaw0O92hw9i8mGBCihitA@mail.gmail.com
Hi!
I started yesterday to improve the Go build system and create a solid
recursive importer. It implies some structural change for every package,
but it's probably worth it and I'll submit a new patch once it's ready.
I'm attaching some internal documentation for my hack for historical
purposes.
Thank you very much for your advice!
Attachment: file
# Go modules within Guix

Using the new [Go modules][4] with the Guix package manager poses some interesting issues, as the package importer is not implemented yet and Go's own package manager clashes heavily with the Guix package manager.

I wanted to port a package without needing to follow the cumbersome process of manually importing dozens of dependencies, so the less-friction hack turned to be the repurposing of the build derivation as a deterministic downloader module, able to pull all the dependencies with the native Go package manager as if it were one of the Guix official download methods.

The suggested approach is rather unorthodox and may not be a good long-term solution: after all, it resorts to all the possible trickery to reduce changes outside of the package to a minimum.

## Network access inside the build container
[As per the `libstore` source code][0], fixed-output derivations have unrestricted network access as long as their contents are kept deterministic and you provide a hash of the final state. _Id est_, you can gather files from a remote repository as long as you're able to get the same commit or version every time.

Toggle quote (2 lines)
> _When `hash` and `hash-algo` are given, a fixed-output derivation is created —i.e., one whose result is known in advance, such as a file download. If, in addition, `recursive?`` is true, then that fixed output may be an executable file or a directory and hash must be the hash of an archive containing this output. [(Source)][1]_

By extending [`guix/build-system/go.scm`][2], we can pass through three additional keyword arguments to `build-expression->derivation` specifying the derivation hash:

```patch
Toggle diff (351 lines)
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index f8ebaefb27..f6b6a7809d 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -88,6 +88,9 @@
                    (allow-go-reference? #f)
                    (system (%current-system))
                    (guile #f)
+                   (derivation-hash #f)
+                   (derivation-hash-algorithm #f)
+                   (derivation-hash-recursive? #f)
                    (imported-modules %go-build-system-modules)
                    (modules '((guix build go-build-system)
                               (guix build union)
@@ -114,6 +117,9 @@
                 #:build-flags ,build-flags
                 #:tests? ,tests?
                 #:allow-go-reference? ,allow-go-reference?
+                #:derivation-hash ,derivation-hash
+                ; #:derivation-hash-algorithm ,derivation-hash-algorithm ; FIXME
+                #:derivation-hash-recursive? ,derivation-hash-recursive?
                 #:inputs %build-inputs)))

   (define guile-for-build
@@ -131,6 +137,9 @@
                                 #:system system
                                 #:modules imported-modules
                                 #:outputs outputs
+                                #:hash derivation-hash
+                                #:hash-algo derivation-hash-algorithm
+                                #:recursive? derivation-hash-recursive?
                                 #:guile-for-build guile-for-build))

 (define go-build-system
```

This way, any package can create a fixed-output build derivation with network access in order to pull additional files during the build process without breaking the build determinism:

```scheme
(package
  (build-system go-build-system)
  (arguments
    (#:derivation-hash (base32 "···")
     #:derivation-hash-algorithm 'sha256
     #:derivation-hash-recursive? #t)))
```

## X509 certificates for network requests

Once we introduce networking capabilities inside the container, we need to provision it with certificates so it's able to authenticate secure connections to the outside world. In order to achieve this, we would need to include the `certs` package first:

```patch
diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index fe34d34491..03f8959b7f 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -46,6 +46,7 @@
   #:use-module (guix build-system go)
   #:use-module (gnu packages)
   #:use-module (gnu packages admin)
+  #:use-module (gnu packages certs)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages glib)
   #:use-module (gnu packages base)
```

Then, we can include the certificate bundle at the package level through the `inputs` keyword argument:

```scheme
(package
  (build-system go-build-system)
  (inputs `(("nss-certs" ,nss-certs))))
```

Additionally, we're going to need a few environment variables pointing to the actual certificate directory, because it won't be at the standard location:

```scheme
(package
  (build-system go-build-system)
  (arguments
   `(#:phases
      (modify-phases %standard-phases
        (add-after 'setup-go-environment 'add-certificates
          (lambda _
            (setenv "SSL_CERT_DIR" (string-append
              (assoc-ref %build-inputs "nss-certs")
              "/etc/ssl/certs"))
            (setenv "SSL_CERT_FILE" (string-append
              (getenv "SSL_CERT_DIR")
              "/ca-certificates.crt"))
            #t))))))
```

## Support for the new module system

Now, we need to enable the new module system through the `GO111MODULE` environment variable, as it's being disabled by default from [`guix/build/go-build-system.scm`][3] due to the previous lack of support. As this proposal is highly experimental and every package expects it set to `off`, it's probably better to toggle it on an the package level:

```scheme
(package
  (build-system go-build-system)
  (arguments
   `(#:phases
      (modify-phases %standard-phases
        (add-after 'setup-go-environment 'enable-modules
          (lambda _
            (setenv "GO111MODULE" "on")
            #t))))))
```

## Example package: `go-ethereum`

```scheme
(define-public go-github-com-ethereum-go-ethereum
  (package
    (name "go-github-com-ethereum-go-ethereum")
    (version "1.9.22")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/ethereum/go-ethereum")
                    (commit (string-append "v" version))))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "08i31xil2lygfcn2igsvn4hpg8xnf8l6g914f78hgl4wj6v1dja9"))))
    (build-system go-build-system)
    (arguments
     `(#:import-path "github.com/ethereum/go-ethereum"
       #:tests? #f ; tests are broken
       #:derivation-hash ,(base32 "1ab7gd90g85ciwsrw6zjaxs76j7y0a11kavp50xkwmm26x34sjss")
       #:derivation-hash-algorithm sha256
       #:derivation-hash-recursive? ,#t
       #:phases
       (modify-phases %standard-phases
         (add-after 'setup-go-environment 'modify-go-environment
           (lambda _
             (setenv "GO111MODULE" "on")
             (setenv "SSL_CERT_DIR" (string-append
               (assoc-ref %build-inputs "nss-certs")
               "/etc/ssl/certs"))
             (setenv "SSL_CERT_FILE" (string-append
               (getenv "SSL_CERT_DIR")
               "/ca-certificates.crt"))
             #t))
         (replace 'build
           (lambda* (#:key import-path build-flags #:allow-other-keys)
             (with-directory-excursion (string-append "src/" import-path)
               (invoke "go" "run" "build/ci.go" "install"))))
         (replace 'install
           (lambda* (#:key outputs import-path #:allow-other-keys)
             (let* ((out (assoc-ref outputs "out"))
               (source (string-append (getenv "GOPATH") "/src/" import-path "/build"))
               (dest out))
             (mkdir-p dest)
             (copy-recursively source dest #:keep-mtime? #t))))
         (replace 'check
           (lambda* (#:key tests? import-path #:allow-other-keys)
             (if tests?
               (with-directory-excursion (string-append "src/" import-path)
                 (invoke "go" "run" "build/ci.go" "test"))
               #t))))))
    (inputs `(("nss-certs" ,nss-certs)))
    (home-page "https://github.com/ethereum/go-ethereum")
    (synopsis "Official Go implementation of the Ethereum protocol")
    (description "This repository contains the official
Go implementation of the Ethereum protocol and command
line tools for interacting with the blockchain.")
    (license license:lgpl3)))
```

## Full patches, ready to apply

```patch
diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index fe34d34491..85c368942b 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -46,6 +46,7 @@
   #:use-module (guix build-system go)
   #:use-module (gnu packages)
   #:use-module (gnu packages admin)
+  #:use-module (gnu packages certs)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages glib)
   #:use-module (gnu packages base)
@@ -933,6 +934,62 @@ time.")
 is similar to Go's standard library @code{json} and @code{xml} package.")
     (license license:expat)))

+(define-public go-github-com-ethereum-go-ethereum
+  (package
+    (name "go-github-com-ethereum-go-ethereum")
+    (version "1.9.22")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/ethereum/go-ethereum")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "08i31xil2lygfcn2igsvn4hpg8xnf8l6g914f78hgl4wj6v1dja9"))))
+    (build-system go-build-system)
+    (arguments
+     `(#:import-path "github.com/ethereum/go-ethereum"
+       #:tests? #f ; tests are broken
+       #:derivation-hash ,(base32 "1ab7gd90g85ciwsrw6zjaxs76j7y0a11kavp50xkwmm26x34sjss")
+       #:derivation-hash-algorithm sha256
+       #:derivation-hash-recursive? ,#t
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'setup-go-environment 'modify-go-environment
+           (lambda _
+             (setenv "GO111MODULE" "on")
+             (setenv "SSL_CERT_DIR" (string-append
+               (assoc-ref %build-inputs "nss-certs")
+               "/etc/ssl/certs"))
+             (setenv "SSL_CERT_FILE" (string-append
+               (getenv "SSL_CERT_DIR")
+               "/ca-certificates.crt"))
+             #t))
+         (replace 'build
+           (lambda* (#:key import-path build-flags #:allow-other-keys)
+             (with-directory-excursion (string-append "src/" import-path)
+               (invoke "go" "run" "build/ci.go" "install"))))
+         (replace 'install
+           (lambda* (#:key outputs import-path #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+               (source (string-append (getenv "GOPATH") "/src/" import-path "/build"))
+               (dest out))
+             (mkdir-p dest)
+             (copy-recursively source dest #:keep-mtime? #t))))
+         (replace 'check
+           (lambda* (#:key tests? import-path #:allow-other-keys)
+             (if tests?
+               (with-directory-excursion (string-append "src/" import-path)
+                 (invoke "go" "run" "build/ci.go" "test"))
+               #t))))))
+    (inputs `(("nss-certs" ,nss-certs)))
+    (home-page "https://github.com/ethereum/go-ethereum")
+    (synopsis "Official Go implementation of the Ethereum protocol")
+    (description "This repository contains the official Go implementation of
+the Ethereum protocol and command line tools for interacting with the blockchain.")
+    (license license:lgpl3)))
+
 (define-public go-github-com-getsentry-raven-go
   (let ((commit "5c24d5110e0e198d9ae16f1f3465366085001d92")
         (revision "0"))
```

```patch
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index f8ebaefb27..e8743e3929 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -88,6 +88,9 @@
                    (allow-go-reference? #f)
                    (system (%current-system))
                    (guile #f)
+                   (derivation-hash #f)
+                   (derivation-hash-algorithm #f)
+                   (derivation-hash-recursive? #f)
                    (imported-modules %go-build-system-modules)
                    (modules '((guix build go-build-system)
                               (guix build union)
@@ -114,6 +117,9 @@
                 #:build-flags ,build-flags
                 #:tests? ,tests?
                 #:allow-go-reference? ,allow-go-reference?
+                #:derivation-hash ,derivation-hash
+                ; FIXME #:derivation-hash-algorithm ,derivation-hash-algorithm
+                #:derivation-hash-recursive? ,derivation-hash-recursive?
                 #:inputs %build-inputs)))

   (define guile-for-build
@@ -131,6 +137,9 @@
                                 #:system system
                                 #:modules imported-modules
                                 #:outputs outputs
+                                #:hash derivation-hash
+                                #:hash-algo derivation-hash-algorithm
+                                #:recursive? derivation-hash-recursive?
                                 #:guile-for-build guile-for-build))

 (define go-build-system
```

## Pending loose ends

* I've been unable to uncomment the `FIXME` line at `guix/build-system/go.scm` without getting an undecipherable [for me] stack trace. This should be fixed to keep compatibility with other hashing algorithms.

* We're only installing the built binary packages to the root of the derivation output, but there might be more interesting build artifacts to be added to use this package as a module.


# Guix testing environment

I'm documenting my recipe for creating a testing environment from a clean install.

## Upgrading the system
```bash
export PATH="/home/$USER/.config/guix/current/bin:$PATH"
```
```bash
guix pull
hash guix
guix pull
```
```bash
GUIX_PROFILE="/home/$USER/.guix-profile"
source "$GUIX_PROFILE/etc/profile"
```

## Installing the build dependencies
```bash
guix install git autoconf automake gettext texinfo graphviz help2man
```

## Getting the source code
```bash
git clone https://git.savannah.gnu.org/git/guix.git
cd guix
```

## Entering inside a container
```bash
guix environment guix --pure
```

## Setting up the project
```bash
./bootstrap
./configure --localstatedir=/var
make
```

## Sourcing `guix` profile inside the container
```bash
GUIX_PROFILE="/home/$USER/.guix-profile"
source "$GUIX_PROFILE/etc/profile"
```

## Running commands
```bash
./pre-inst-env guix --version
```

[0]: https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc#n1915
[1]: https://guix.gnu.org/manual/en/html_node/Derivations.html
[2]: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/go.scm#n129
[3]: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/go-build-system.scm#n149
[4]: https://blog.golang.org/using-go-modules
Closed
M
M
Marius Bakke wrote on 10 Oct 2020 22:35
87tuv1es4o.fsf@gnu.org
Helio Machado <0x2b3bfa0@gmail.com> writes:

Toggle quote (5 lines)
> Hi!
> I started yesterday to improve the Go build system and create a solid
> recursive importer. It implies some structural change for every package,
> but it's probably worth it and I'll submit a new patch once it's ready.

Exciting, good luck on the journey. Importers are decoupled from
build systems, so I'm curious what structural changes are imposed.

Feel free to drop by the #guix IRC channel on Freenode for
up-to-the-minute feedback. :-)

Thanks a lot for diving into this!
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAl+CGxcACgkQoqBt8qM6
VPpZ1wgApoxOUdFQF7RE0RyaWL7n2O7vm6nr3eyb7vMf/w7wcxBEzJ3ptFByJPph
tBo7SjLpuuJvgIby7wLr459qXIB2M3e2gH/TjD9EzExCCUwF5ur3CuFShYu2mgCW
SK1eJCKFp80nWft7omCetu9UH9Am8Y3xqg72UJa0TtC8p6Pz7CpndLUH5wuS5dHa
cBRtp2Nz1LkL7GSfnevYCOouZw/cj6mDdB0nEbccs/zHddeXhC0KCszAwyxGnuE1
m+FXl/eIXPczMY9Gk6McYg/L8pvCgow8QirLcHiH9VU95SOcS/4uoX3TSGCL0Sgo
FUYbOAAy2EJmTIoCDsfE6tLPnRk6BA==
=PrDw
-----END PGP SIGNATURE-----

Closed
A
A
Attila Lendvai wrote on 29 Aug 2021 17:53
(No Subject)
(name . control@debbugs.gnu.org)(address . control@debbugs.gnu.org)
utY8ui1mFUF7WYiYvC_w3sjVbNbumqDw541pRh-D0yK9MmvFuDSAeaj-N5QeF0PAQT2LLnn8t6eeDAzofalMtUwXzi-6YbnBhDPiK5T6Fg0=@lendvai.name
unarchive 43872
Attachment: file
A
A
Attila Lendvai wrote on 29 Aug 2021 17:56
(name . 43872@debbugs.gnu.org)(address . 43872@debbugs.gnu.org)
13xu_g-MkeM7DV1W1XsOjKTx_I1BGHYZNepHPM1gK6tBUMQNGNbnE3OaKQA0cLmTXLGGH0VgZa3HVE1_P4SSMVYWQdGvJB5xfRWkOqUBCmw=@lendvai.name
dear fellow hackers,

i'm wondering what happened to the go-ethereum package.

i'm new to guix. i came over from nixos to package the bee client of swarm (https://www.ethswarm.org/).i made a package/service for it on nixos, but the module/service handling there is not flexible enough, and i got discouraged by the seemingly pointless struggle with it.

go-ethereum is a dependency of bee, or more specifically, the clef binary of the go-ethereum project.

this issue is marked CLOSED, but i don't see go-ethereum anywhere in the repo, even though the go importer seems to have been merged.

is there something i can do to help? shall i test this package on guix HEAD? or shall i just wait patiently?

any guidance is appreciated,

- attila
PGP: 5D5F 45C7 DFCD 0A39
Attachment: file
Z
Z
zimoun wrote on 30 Aug 2021 10:01
(name . Attila Lendvai)(address . attila@lendvai.name)(name . 43872@debbugs.gnu.org)(address . 43872@debbugs.gnu.org)
CAJ3okZ2sO0hw4ef9OdxxuuShLRXpPzx5RxPFd9AQXjyZn3a6tQ@mail.gmail.com
Hi,

Thanks for your interest.

On Sun, 29 Aug 2021 at 18:43, Attila Lendvai <attila@lendvai.name> wrote:

Toggle quote (2 lines)
> i'm wondering what happened to the go-ethereum package.

As Marius wrote when closing the patch submission:

Toggle snippet (6 lines)
To properly package go-ethereum, you will need to declare each
dependency as separate packages. See e.g. 'syncthing' for an example.
Unfortunately there is no importer for Go yet, making it a rather
tedious task.

Toggle quote (2 lines)
> this issue is marked CLOSED, but i don't see go-ethereum anywhere in the repo, even though the go importer seems to have been merged.

The issue is marked closed because it was an incorrect way to package
it. The Go importer is indeed merged. Maybe, you could give a try
via this way.

Toggle quote (2 lines)
> is there something i can do to help? shall i test this package on guix HEAD? or shall i just wait patiently?

From my opinion, you would get more answers asking on the mailing
guix-devel or on the IRC channel #guix than by unarchiving this "old"
submission. :-)

All the best,
simon
?
Your comment

This issue is archived.

To comment on this conversation send email to 43872@debbugs.gnu.org