[PATCH] Add package go-ethereum

DoneSubmitted by Helio Machado.
Details
4 participants
  • Helio Machado
  • Attila Lendvai
  • Marius Bakke
  • zimoun
Owner
unassigned
Severity
normal
H
H
Helio Machado wrote on 8 Oct 2020 23:59
(address . guix-patches@gnu.org)
CANe01w45Erh3=GbYek8X0PrisfZ0zGCG6tMRH-hcz6LaCUVjzA@mail.gmail.com
# Add package `go-ethereum`
These patches add the [go-ethereum][5] package and a small workaround thatallows experimental module support.
## [Checklist][1]
1. Code authentication: all the code is being pulled from Git repositoriesand Go packages, so there isn't any specific code signature we can verify.
2. Synopsis and description: built from the official documentation andaround the style guidelines.
3. Results for `guix lint`: everything seems to be fine, and it seems tohave scheduled a [Software Heritage][2] archival (?).
4. Native build check: the package builds correctly on my platform(x86-64), using `guix build` package.
5. Cross-build check: package was not tested on any other supportedplatform because `go-build-system` [doesn't seem to supportcross-compilation][3].
6. Package deduplication: dependencies are being pulled from theirrespective module repositories at build time; there isn't practically anyoverlap between the package code and the Guix repositories.
7. Package size: as per `guix size`, this package requires a total of 418.7MiB, with a package (self) size of 282.4 MiB.
8. Dependent packages: this is a freshly added package with no dependentitems.
9. Deterministic build: this package is fully deterministic and has fixedhashes for both the initial download and the build derivation.
10. Inclusive language: documentation was written with neutrality in mindand there isn't any obvious bias, though it might need proofreading.
11. Patch atomicity: this patch contains only one set of related changes,though modifications affect two files.
12. Code formatting: added lines were formatted by using `indent-code.el`.
13. Source consistency: all the involved downloads are being pulled alongwith a Git tag or commit, so results should be consistent enough.
14. Guix build: these changes don't affect Guix builds nor introduce anywarning.
15. Guix consistency: these changes do not break Guix in any obvious way.
## Desiderata
It would be great to have easier ways to replace Go module managementsystem with Guix in a seamless and painless way. Rust build system for Guixhas a cleaner implementation, majorly due to the existence of a[centralized package registry][4].
[1]: https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html[2]: https://www.softwareheritage.org/[3]:https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/go.scm#n60[4]: https://crates.io[5]: https://github.com/ethereum/go-ethereum[6]: https://gist.github.com/0x2b3bfa0/1805037bf5efa2655fa71d32e69fdd14
Attachment: file
H
H
Helio Machado wrote on 9 Oct 2020 14:11
Closing due to consistency issues
(address . 43872-done@debbugs.gnu.org)
CANe01w7ypkUjtc2p2TdZewDrC6xD-AExZno5a7uOhZrEZn=VQg@mail.gmail.com
This patch was a big hack and breaks on every architecture but x86-64because the derivation hash changes when producing different binaries. I'mleaving this for reference as it might be useful for a casual install, butthe proper solution probably involves a complete refactoring of the Guix GoBuild System.
Attachment: file
Closed
M
M
Marius Bakke wrote on 9 Oct 2020 20:34
87eem7fdt8.fsf@gnu.org
Hi!
Helio Machado <0x2b3bfa0@gmail.com> writes:
Toggle quote (6 lines)> This patch was a big hack and breaks on every architecture but x86-64> because the derivation hash changes when producing different binaries. I'm> leaving this for reference as it might be useful for a casual install, but> the proper solution probably involves a complete refactoring of the Guix Go> Build System.
That is a very clever hack. :-)
The reason it works is because network access is allowed forfixed-output derivations. So the Go build system is able to go onlineand download everything it needs, instead of relying on declaredinputs. That is why different architectures produce differentderivation hashes: those are the build outputs.
The problem with this approach is that it is not deterministic. Thenext time someone runs the same build, one of the Go dependencies mayhave changed (i.e. a git tag has been overwritten, or a package declaresa dependency on a branch instead of a tag, etc).
Guix will detect if the output hash changes, but is unable reproduce theearlier state because it does not know what has changed.
To properly package go-ethereum, you will need to declare eachdependency as separate packages. See e.g. 'syncthing' for an example.Unfortunately there is no importer for Go yet, making it a rathertedious task.
Since you don't seem to be afraid of hacking on Guix :-) considerwriting an "importer" for Go so you could simply:
guix import go -r github.com/ethereum/go-ethereum
...to create the necessary boilerplate.
Thanks!
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAl+ArVQACgkQoqBt8qM6VPqrEAf/WeFN250sGAcjy9QGh8s1nHIfgAeqLPhz3lgdsscBfzcuzRp5nycqphXa7vkPNWUfE2FrfPEgZDweD+9oJMpKDkGewbtskkLiBrXZj5sQcwROZUXKhM00y7CiWEGAeEv4tZFzC2+g4QFqmEYd+SOM7EhUUy09RioA2aF7PTGPA+yRHfkbHMNJRwK+cGxju8lIhEzX/NZ3Fy+qDcaU77oVn5EIDOcmIDn6aB7h+aeXb4ZufVuQ+TdJcgGE/6WJCqwsVF0e7yLrjixWnr2PqkFbV5xCW7nlEhTJxiheQijJZE7Zwond0Qgj4a1hGA5BVqfkCotaAwDFUglAvjPk9LOoPQ===h9Md-----END PGP SIGNATURE-----
Closed
H
H
Helio Machado wrote on 10 Oct 2020 20:07
CANe01w4SNUM0j7bdduM0WJvsDFkExBaw0O92hw9i8mGBCihitA@mail.gmail.com
Hi!I started yesterday to improve the Go build system and create a solidrecursive importer. It implies some structural change for every package,but it's probably worth it and I'll submit a new patch once it's ready.I'm attaching some internal documentation for my hack for historicalpurposes.Thank you very much for your advice!
Attachment: file
# Go modules within Guix
Using the new [Go modules][4] with the Guix package manager poses some interesting issues, as the package importer is not implemented yet and Go's own package manager clashes heavily with the Guix package manager.
I wanted to port a package without needing to follow the cumbersome process of manually importing dozens of dependencies, so the less-friction hack turned to be the repurposing of the build derivation as a deterministic downloader module, able to pull all the dependencies with the native Go package manager as if it were one of the Guix official download methods.
The suggested approach is rather unorthodox and may not be a good long-term solution: after all, it resorts to all the possible trickery to reduce changes outside of the package to a minimum.
## Network access inside the build container[As per the `libstore` source code][0], fixed-output derivations have unrestricted network access as long as their contents are kept deterministic and you provide a hash of the final state. _Id est_, you can gather files from a remote repository as long as you're able to get the same commit or version every time.
Toggle quote (2 lines)> _When `hash` and `hash-algo` are given, a fixed-output derivation is created —i.e., one whose result is known in advance, such as a file download. If, in addition, `recursive?`` is true, then that fixed output may be an executable file or a directory and hash must be the hash of an archive containing this output. [(Source)][1]_
By extending [`guix/build-system/go.scm`][2], we can pass through three additional keyword arguments to `build-expression->derivation` specifying the derivation hash:
```patch
Toggle diff (351 lines)diff --git a/guix/build-system/go.scm b/guix/build-system/go.scmindex f8ebaefb27..f6b6a7809d 100644--- a/guix/build-system/go.scm+++ b/guix/build-system/go.scm@@ -88,6 +88,9 @@ (allow-go-reference? #f) (system (%current-system)) (guile #f)+ (derivation-hash #f)+ (derivation-hash-algorithm #f)+ (derivation-hash-recursive? #f) (imported-modules %go-build-system-modules) (modules '((guix build go-build-system) (guix build union)@@ -114,6 +117,9 @@ #:build-flags ,build-flags #:tests? ,tests? #:allow-go-reference? ,allow-go-reference?+ #:derivation-hash ,derivation-hash+ ; #:derivation-hash-algorithm ,derivation-hash-algorithm ; FIXME+ #:derivation-hash-recursive? ,derivation-hash-recursive? #:inputs %build-inputs)))
(define guile-for-build@@ -131,6 +137,9 @@ #:system system #:modules imported-modules #:outputs outputs+ #:hash derivation-hash+ #:hash-algo derivation-hash-algorithm+ #:recursive? derivation-hash-recursive? #:guile-for-build guile-for-build))
(define go-build-system```
This way, any package can create a fixed-output build derivation with network access in order to pull additional files during the build process without breaking the build determinism:
```scheme(package (build-system go-build-system) (arguments (#:derivation-hash (base32 "···") #:derivation-hash-algorithm 'sha256 #:derivation-hash-recursive? #t)))```
## X509 certificates for network requests
Once we introduce networking capabilities inside the container, we need to provision it with certificates so it's able to authenticate secure connections to the outside world. In order to achieve this, we would need to include the `certs` package first:
```patchdiff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scmindex fe34d34491..03f8959b7f 100644--- a/gnu/packages/golang.scm+++ b/gnu/packages/golang.scm@@ -46,6 +46,7 @@ #:use-module (guix build-system go) #:use-module (gnu packages) #:use-module (gnu packages admin)+ #:use-module (gnu packages certs) #:use-module (gnu packages gcc) #:use-module (gnu packages glib) #:use-module (gnu packages base)```
Then, we can include the certificate bundle at the package level through the `inputs` keyword argument:
```scheme(package (build-system go-build-system) (inputs `(("nss-certs" ,nss-certs))))```
Additionally, we're going to need a few environment variables pointing to the actual certificate directory, because it won't be at the standard location:
```scheme(package (build-system go-build-system) (arguments `(#:phases (modify-phases %standard-phases (add-after 'setup-go-environment 'add-certificates (lambda _ (setenv "SSL_CERT_DIR" (string-append (assoc-ref %build-inputs "nss-certs") "/etc/ssl/certs")) (setenv "SSL_CERT_FILE" (string-append (getenv "SSL_CERT_DIR") "/ca-certificates.crt")) #t))))))```
## Support for the new module system
Now, we need to enable the new module system through the `GO111MODULE` environment variable, as it's being disabled by default from [`guix/build/go-build-system.scm`][3] due to the previous lack of support. As this proposal is highly experimental and every package expects it set to `off`, it's probably better to toggle it on an the package level:
```scheme(package (build-system go-build-system) (arguments `(#:phases (modify-phases %standard-phases (add-after 'setup-go-environment 'enable-modules (lambda _ (setenv "GO111MODULE" "on") #t))))))```
## Example package: `go-ethereum`
```scheme(define-public go-github-com-ethereum-go-ethereum (package (name "go-github-com-ethereum-go-ethereum") (version "1.9.22") (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/ethereum/go-ethereum") (commit (string-append "v" version)))) (file-name (git-file-name name version)) (sha256 (base32 "08i31xil2lygfcn2igsvn4hpg8xnf8l6g914f78hgl4wj6v1dja9")))) (build-system go-build-system) (arguments `(#:import-path "github.com/ethereum/go-ethereum" #:tests? #f ; tests are broken #:derivation-hash ,(base32 "1ab7gd90g85ciwsrw6zjaxs76j7y0a11kavp50xkwmm26x34sjss") #:derivation-hash-algorithm sha256 #:derivation-hash-recursive? ,#t #:phases (modify-phases %standard-phases (add-after 'setup-go-environment 'modify-go-environment (lambda _ (setenv "GO111MODULE" "on") (setenv "SSL_CERT_DIR" (string-append (assoc-ref %build-inputs "nss-certs") "/etc/ssl/certs")) (setenv "SSL_CERT_FILE" (string-append (getenv "SSL_CERT_DIR") "/ca-certificates.crt")) #t)) (replace 'build (lambda* (#:key import-path build-flags #:allow-other-keys) (with-directory-excursion (string-append "src/" import-path) (invoke "go" "run" "build/ci.go" "install")))) (replace 'install (lambda* (#:key outputs import-path #:allow-other-keys) (let* ((out (assoc-ref outputs "out")) (source (string-append (getenv "GOPATH") "/src/" import-path "/build")) (dest out)) (mkdir-p dest) (copy-recursively source dest #:keep-mtime? #t)))) (replace 'check (lambda* (#:key tests? import-path #:allow-other-keys) (if tests? (with-directory-excursion (string-append "src/" import-path) (invoke "go" "run" "build/ci.go" "test")) #t)))))) (inputs `(("nss-certs" ,nss-certs))) (home-page "https://github.com/ethereum/go-ethereum") (synopsis "Official Go implementation of the Ethereum protocol") (description "This repository contains the officialGo implementation of the Ethereum protocol and commandline tools for interacting with the blockchain.") (license license:lgpl3)))```
## Full patches, ready to apply
```patchdiff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scmindex fe34d34491..85c368942b 100644--- a/gnu/packages/golang.scm+++ b/gnu/packages/golang.scm@@ -46,6 +46,7 @@ #:use-module (guix build-system go) #:use-module (gnu packages) #:use-module (gnu packages admin)+ #:use-module (gnu packages certs) #:use-module (gnu packages gcc) #:use-module (gnu packages glib) #:use-module (gnu packages base)@@ -933,6 +934,62 @@ time.") is similar to Go's standard library @code{json} and @code{xml} package.") (license license:expat)))
+(define-public go-github-com-ethereum-go-ethereum+ (package+ (name "go-github-com-ethereum-go-ethereum")+ (version "1.9.22")+ (source (origin+ (method git-fetch)+ (uri (git-reference+ (url "https://github.com/ethereum/go-ethereum")+ (commit (string-append "v" version))))+ (file-name (git-file-name name version))+ (sha256+ (base32+ "08i31xil2lygfcn2igsvn4hpg8xnf8l6g914f78hgl4wj6v1dja9"))))+ (build-system go-build-system)+ (arguments+ `(#:import-path "github.com/ethereum/go-ethereum"+ #:tests? #f ; tests are broken+ #:derivation-hash ,(base32 "1ab7gd90g85ciwsrw6zjaxs76j7y0a11kavp50xkwmm26x34sjss")+ #:derivation-hash-algorithm sha256+ #:derivation-hash-recursive? ,#t+ #:phases+ (modify-phases %standard-phases+ (add-after 'setup-go-environment 'modify-go-environment+ (lambda _+ (setenv "GO111MODULE" "on")+ (setenv "SSL_CERT_DIR" (string-append+ (assoc-ref %build-inputs "nss-certs")+ "/etc/ssl/certs"))+ (setenv "SSL_CERT_FILE" (string-append+ (getenv "SSL_CERT_DIR")+ "/ca-certificates.crt"))+ #t))+ (replace 'build+ (lambda* (#:key import-path build-flags #:allow-other-keys)+ (with-directory-excursion (string-append "src/" import-path)+ (invoke "go" "run" "build/ci.go" "install"))))+ (replace 'install+ (lambda* (#:key outputs import-path #:allow-other-keys)+ (let* ((out (assoc-ref outputs "out"))+ (source (string-append (getenv "GOPATH") "/src/" import-path "/build"))+ (dest out))+ (mkdir-p dest)+ (copy-recursively source dest #:keep-mtime? #t))))+ (replace 'check+ (lambda* (#:key tests? import-path #:allow-other-keys)+ (if tests?+ (with-directory-excursion (string-append "src/" import-path)+ (invoke "go" "run" "build/ci.go" "test"))+ #t))))))+ (inputs `(("nss-certs" ,nss-certs)))+ (home-page "https://github.com/ethereum/go-ethereum")+ (synopsis "Official Go implementation of the Ethereum protocol")+ (description "This repository contains the official Go implementation of+the Ethereum protocol and command line tools for interacting with the blockchain.")+ (license license:lgpl3)))+ (define-public go-github-com-getsentry-raven-go (let ((commit "5c24d5110e0e198d9ae16f1f3465366085001d92") (revision "0"))```
```patchdiff --git a/guix/build-system/go.scm b/guix/build-system/go.scmindex f8ebaefb27..e8743e3929 100644--- a/guix/build-system/go.scm+++ b/guix/build-system/go.scm@@ -88,6 +88,9 @@ (allow-go-reference? #f) (system (%current-system)) (guile #f)+ (derivation-hash #f)+ (derivation-hash-algorithm #f)+ (derivation-hash-recursive? #f) (imported-modules %go-build-system-modules) (modules '((guix build go-build-system) (guix build union)@@ -114,6 +117,9 @@ #:build-flags ,build-flags #:tests? ,tests? #:allow-go-reference? ,allow-go-reference?+ #:derivation-hash ,derivation-hash+ ; FIXME #:derivation-hash-algorithm ,derivation-hash-algorithm+ #:derivation-hash-recursive? ,derivation-hash-recursive? #:inputs %build-inputs)))
(define guile-for-build@@ -131,6 +137,9 @@ #:system system #:modules imported-modules #:outputs outputs+ #:hash derivation-hash+ #:hash-algo derivation-hash-algorithm+ #:recursive? derivation-hash-recursive? #:guile-for-build guile-for-build))
(define go-build-system```
## Pending loose ends
* I've been unable to uncomment the `FIXME` line at `guix/build-system/go.scm` without getting an undecipherable [for me] stack trace. This should be fixed to keep compatibility with other hashing algorithms.
* We're only installing the built binary packages to the root of the derivation output, but there might be more interesting build artifacts to be added to use this package as a module.

# Guix testing environment
I'm documenting my recipe for creating a testing environment from a clean install.
## Upgrading the system```bashexport PATH="/home/$USER/.config/guix/current/bin:$PATH"``````bashguix pullhash guixguix pull``````bashGUIX_PROFILE="/home/$USER/.guix-profile"source "$GUIX_PROFILE/etc/profile"```
## Installing the build dependencies```bashguix install git autoconf automake gettext texinfo graphviz help2man```
## Getting the source code```bashgit clone https://git.savannah.gnu.org/git/guix.gitcd guix```
## Entering inside a container```bashguix environment guix --pure```
## Setting up the project```bash./bootstrap./configure --localstatedir=/varmake```
## Sourcing `guix` profile inside the container```bashGUIX_PROFILE="/home/$USER/.guix-profile"source "$GUIX_PROFILE/etc/profile"```
## Running commands```bash./pre-inst-env guix --version```
[0]: https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc#n1915[1]: https://guix.gnu.org/manual/en/html_node/Derivations.html[2]: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/go.scm#n129[3]: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/go-build-system.scm#n149[4]: https://blog.golang.org/using-go-modules
Closed
M
M
Marius Bakke wrote on 10 Oct 2020 22:35
87tuv1es4o.fsf@gnu.org
Helio Machado <0x2b3bfa0@gmail.com> writes:
Toggle quote (5 lines)> Hi!> I started yesterday to improve the Go build system and create a solid> recursive importer. It implies some structural change for every package,> but it's probably worth it and I'll submit a new patch once it's ready.
Exciting, good luck on the journey. Importers are decoupled frombuild systems, so I'm curious what structural changes are imposed.
Feel free to drop by the #guix IRC channel on Freenode forup-to-the-minute feedback. :-)
Thanks a lot for diving into this!
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAl+CGxcACgkQoqBt8qM6VPpZ1wgApoxOUdFQF7RE0RyaWL7n2O7vm6nr3eyb7vMf/w7wcxBEzJ3ptFByJPphtBo7SjLpuuJvgIby7wLr459qXIB2M3e2gH/TjD9EzExCCUwF5ur3CuFShYu2mgCWSK1eJCKFp80nWft7omCetu9UH9Am8Y3xqg72UJa0TtC8p6Pz7CpndLUH5wuS5dHacBRtp2Nz1LkL7GSfnevYCOouZw/cj6mDdB0nEbccs/zHddeXhC0KCszAwyxGnuE1m+FXl/eIXPczMY9Gk6McYg/L8pvCgow8QirLcHiH9VU95SOcS/4uoX3TSGCL0SgoFUYbOAAy2EJmTIoCDsfE6tLPnRk6BA===PrDw-----END PGP SIGNATURE-----
Closed
A
A
Attila Lendvai wrote on 29 Aug 17:53 +0200
(No Subject)
(name . control@debbugs.gnu.org)(address . control@debbugs.gnu.org)
utY8ui1mFUF7WYiYvC_w3sjVbNbumqDw541pRh-D0yK9MmvFuDSAeaj-N5QeF0PAQT2LLnn8t6eeDAzofalMtUwXzi-6YbnBhDPiK5T6Fg0=@lendvai.name
unarchive 43872
Attachment: file
A
A
Attila Lendvai wrote on 29 Aug 17:56 +0200
(name . 43872@debbugs.gnu.org)(address . 43872@debbugs.gnu.org)
13xu_g-MkeM7DV1W1XsOjKTx_I1BGHYZNepHPM1gK6tBUMQNGNbnE3OaKQA0cLmTXLGGH0VgZa3HVE1_P4SSMVYWQdGvJB5xfRWkOqUBCmw=@lendvai.name
dear fellow hackers,
i'm wondering what happened to the go-ethereum package.
i'm new to guix. i came over from nixos to package the bee client of swarm (https://www.ethswarm.org/).i made a package/service for it on nixos, but the module/service handling there is not flexible enough, and i got discouraged by the seemingly pointless struggle with it.
go-ethereum is a dependency of bee, or more specifically, the clef binary of the go-ethereum project.
this issue is marked CLOSED, but i don't see go-ethereum anywhere in the repo, even though the go importer seems to have been merged.
is there something i can do to help? shall i test this package on guix HEAD? or shall i just wait patiently?
any guidance is appreciated,
- attilaPGP: 5D5F 45C7 DFCD 0A39
Attachment: file
Z
Z
zimoun wrote on 30 Aug 10:01 +0200
(name . Attila Lendvai)(address . attila@lendvai.name)(name . 43872@debbugs.gnu.org)(address . 43872@debbugs.gnu.org)
CAJ3okZ2sO0hw4ef9OdxxuuShLRXpPzx5RxPFd9AQXjyZn3a6tQ@mail.gmail.com
Hi,
Thanks for your interest.
On Sun, 29 Aug 2021 at 18:43, Attila Lendvai <attila@lendvai.name> wrote:
Toggle quote (2 lines)> i'm wondering what happened to the go-ethereum package.
As Marius wrote when closing the patch submission:
Toggle snippet (6 lines)To properly package go-ethereum, you will need to declare eachdependency as separate packages. See e.g. 'syncthing' for an example.Unfortunately there is no importer for Go yet, making it a rathertedious task.
Toggle quote (2 lines)> this issue is marked CLOSED, but i don't see go-ethereum anywhere in the repo, even though the go importer seems to have been merged.
The issue is marked closed because it was an incorrect way to packageit. The Go importer is indeed merged. Maybe, you could give a tryvia this way.
Toggle quote (2 lines)> is there something i can do to help? shall i test this package on guix HEAD? or shall i just wait patiently?
From my opinion, you would get more answers asking on the mailingguix-devel or on the IRC channel #guix than by unarchiving this "old"submission. :-)
All the best,simon
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 43872@debbugs.gnu.org