[PATCH 0/3] go-build-system and GOPATH improvements

OpenSubmitted by Marius Bakke.
Details
2 participants
  • Sarah Morgensen
  • Marius Bakke
Owner
unassigned
Severity
normal
M
M
Marius Bakke wrote on 27 Aug 17:10 +0200
(address . guix-patches@gnu.org)
20210827151052.12611-1-marius@gnu.org
These patches adjust the Go build system to use Guix's regularnative-search-paths mechanism instead of ad-hoc GOPATH trickery.
The context is that I needed to hack on a Go package, and was somewhatsurprised that my usual workflow of "guix environment PKG" did not work.
It still does not work "out of the box", but these patches bring it astep further. Now "all" that is needed is to:
$ cd ~/src/go-foo $ guix environment go-example-com-foo $ MYGOPATH="$HOME/tmp/go" $ NAMESPACE="$MYGOPATH/src/example.com/foo" $ mkdir -p $(dirname $NAMESPACE) $ ln -s $PWD $NAMESPACE # or git worktree add $NAMESPACE $ export GOPATH="$MYGOPATH:$GOPATH" $ go build # no 'go get' necessary!
I don't know how feasible it is to avoid making a local directory andsymlinking the project to the expected namespace. Still a complete Gonewbie, but this approach feels more natural and idiomatic Guix-wise.
Marius Bakke (3): build-system/go: Use a native-search-path for GOPATH. gnu: hyperledger-fabric: Do not assume GOPATH contains a single entry. gnu: go-gotest-tools-assert: Provide internal inputs with the source.
gnu/packages/golang.scm | 49 +++++++++++++++++++++++----------- gnu/packages/hyperledger.scm | 6 ++++- guix/build/go-build-system.scm | 37 +++---------------------- 3 files changed, 43 insertions(+), 49 deletions(-)
-- 2.31.1
M
M
Marius Bakke wrote on 27 Aug 17:13 +0200
[PATCH 1/3] build-system/go: Use a native-search-path for GOPATH.
(address . 50227@debbugs.gnu.org)
20210827151330.13112-1-marius@gnu.org
* gnu/packages/golang.scm (go-1.14)[native-search-paths]: New field.* guix/build/go-build-system.scm (setup-go-environment): Make the workingdirectory the first entry on GOPATH instead of creating a union.(unpack, install): Adjust accordingly.(go-package?, go-inputs): Remove variables.--- gnu/packages/golang.scm | 4 ++++ guix/build/go-build-system.scm | 37 ++++------------------------------ 2 files changed, 8 insertions(+), 33 deletions(-)
Toggle diff (104 lines)diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scmindex 72579d6bd2..3a5c6ddc3f 100644--- a/gnu/packages/golang.scm+++ b/gnu/packages/golang.scm@@ -257,6 +257,10 @@ in the style of communicating sequential processes (@dfn{CSP}).") (sha256 (base32 "1crh90qkvhlx23hwsi4wxy3l3h8973lr18135y6h1nnzzwr3n3ps"))))+ (native-search-paths+ (list (search-path-specification+ (variable "GOPATH")+ (files '("src/.." "pkg/.."))))) (arguments (substitute-keyword-arguments (package-arguments go-1.4) ((#:system system)diff --git a/guix/build/go-build-system.scm b/guix/build/go-build-system.scmindex 227df820db..01dd8cd249 100644--- a/guix/build/go-build-system.scm+++ b/guix/build/go-build-system.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2020 Jack Hill <jackhill@jackhill.us> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net> ;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>+;;; Copyright © 2021 Marius Bakke <marius@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -88,7 +89,6 @@ ;; a tmpdir when creating the inputs union. ;; * Use Go modules [4] ;; * Re-use compiled packages [5]-;; * Avoid the go-inputs hack ;; * Stop needing remove-go-references (-trimpath ? ) ;; * Remove module packages, only offering the full Git repos? This is ;; more idiomatic, I think, because Go downloads Git repos, not modules.@@ -143,23 +143,12 @@ dependencies, so it should be self-contained." (setenv "GOCACHE" "/tmp/go-cache") ;; Using the current working directory as GOPATH makes it easier for packagers ;; who need to manipulate the unpacked source code.- (setenv "GOPATH" (getcwd))+ (setenv "GOPATH" (string-append (getcwd) ":" (getenv "GOPATH"))) ;; Go 1.13 uses go modules by default. The go build system does not ;; currently support modules, so turn modules off to continue using the old ;; GOPATH behavior. (setenv "GO111MODULE" "off") (setenv "GOBIN" (string-append (assoc-ref outputs "out") "/bin"))- (let ((tmpdir (tmpnam)))- (match (go-inputs inputs)- (((names . directories) ...)- (union-build tmpdir (filter directory-exists? directories)- #:create-all-directories? #t- #:log-port (%make-void-port "w"))))- ;; XXX A little dance because (guix build union) doesn't use mkdir-p.- (copy-recursively tmpdir- (string-append (getenv "GOPATH"))- #:keep-mtime? #t)- (delete-file-recursively tmpdir)) #t) (define* (unpack #:key source import-path unpack-path #:allow-other-keys)@@ -191,31 +180,13 @@ unpacking." (display "WARNING: The Go import path is unset.\n")) (when (string-null? unpack-path) (set! unpack-path import-path))- (let ((dest (string-append (getenv "GOPATH") "/src/" unpack-path)))+ (let ((dest (string-append (getcwd) "/src/" unpack-path))) (mkdir-p dest) (if (file-is-directory? source) (copy-recursively source dest #:keep-mtime? #t) (unpack-maybe-strip source dest))) #t) -(define (go-package? name)- (string-prefix? "go-" name))--(define (go-inputs inputs)- "Return the alist of INPUTS that are Go software."- ;; XXX This should not check the file name of the store item. Instead we- ;; should pass, from the host side, the list of inputs that are packages using- ;; the go-build-system.- (alist-delete "go" ; Exclude the Go compiler- (alist-delete "source" ; Exclude the source code of the package being built- (filter (match-lambda- ((label . directory)- (go-package? ((compose package-name->name+version- strip-store-file-name)- directory)))- (_ #f))- inputs))))- (define* (build #:key import-path build-flags #:allow-other-keys) "Build the package named by IMPORT-PATH." (with-throw-handler@@ -249,7 +220,7 @@ XXX We can't make use of compiled libraries (Go \"packages\")." (if (string-null? import-path) ((display "WARNING: The Go import path is unset.\n"))) (let* ((out (assoc-ref outputs "out"))- (source (string-append (getenv "GOPATH") "/src/" import-path))+ (source (string-append "src/" import-path)) (dest (string-append out "/src/" import-path))) (mkdir-p dest) (copy-recursively source dest #:keep-mtime? #t)))-- 2.31.1
M
M
Marius Bakke wrote on 27 Aug 17:13 +0200
[PATCH 2/3] gnu: hyperledger-fabric: Do not assume GOPATH contains a single entry.
(address . 50227@debbugs.gnu.org)
20210827151330.13112-2-marius@gnu.org
* gnu/packages/hyperledger.scm (hyperledger-fabric)[arguments]: Adjust formultiple GOPATH entries.--- gnu/packages/hyperledger.scm | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
Toggle diff (23 lines)diff --git a/gnu/packages/hyperledger.scm b/gnu/packages/hyperledger.scmindex 82680cd5e4..5c36a81f44 100644--- a/gnu/packages/hyperledger.scm+++ b/gnu/packages/hyperledger.scm@@ -77,11 +77,15 @@ #:phases (modify-phases %standard-phases (replace 'build- (lambda _+ (lambda* (#:key outputs #:allow-other-keys) ;; Only linux-amd64 and linux-ppc64le seem to be supported at the moment. (invoke "make" "-j" (number->string (parallel-job-count)) "-C" "src/github.com/hyperledger/fabric"+ ;; The build system expects GOPATH to contain a single entry+ ;; and uses it to determine the installation directory.+ ;; Work around that.+ (string-append "GOTOOLS_BINDIR=" (assoc-ref outputs "out") "/bin") "release/linux-amd64"))) (add-after 'install 'install-commands (lambda* (#:key outputs #:allow-other-keys)-- 2.31.1
M
M
Marius Bakke wrote on 27 Aug 17:13 +0200
[PATCH 3/3] gnu: go-gotest-tools-assert: Provide internal inputs with the source.
(address . 50227@debbugs.gnu.org)
20210827151330.13112-3-marius@gnu.org
* gnu/packages/golang.scm (go-gotest-tools-assert)[inputs]: AddGO-GOTEST-TOOLS-INTERNAL-FORMAT, GO-GOTEST-TOOLS-INTERNAL-DIFFLIB, andGO-GOTEST-TOOLS-INTERNAL-SOURCE.[arguments]: Add phase to install a union of the above inputs.* gnu/packages/golang.scm (gotestsum)[native-inputs]: Don't add the abovementioned inputs.--- gnu/packages/golang.scm | 45 +++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 15 deletions(-)
Toggle diff (72 lines)diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scmindex 3a5c6ddc3f..295b442a2a 100644--- a/gnu/packages/golang.scm+++ b/gnu/packages/golang.scm@@ -20,7 +20,7 @@ ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net> ;;; Copyright © 2020 Nicolas Goaziou <mail@nicolasgoaziou.com> ;;; Copyright © 2020 Ryan Prior <rprior@protonmail.com>-;;; Copyright © 2020 Marius Bakke <marius@gnu.org>+;;; Copyright © 2020, 2021 Marius Bakke <marius@gnu.org> ;;; Copyright © 2020 raingloom <raingloom@riseup.net> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; Copyright © 2021 Ricardo Wurmus <rekado@elephly.net>@@ -5945,9 +5945,35 @@ gotest-tools."))) (arguments `(#:tests? #f ; Test failure concerning message formatting (FIXME) #:import-path "gotest.tools/assert"- #:unpack-path "gotest.tools"))- ;(propagated-inputs- ; `(("go-gotest-tools-internal-format" ,go-gotest-tools-internal-format)))+ #:unpack-path "gotest.tools"+ #:modules ((ice-9 match)+ (srfi srfi-26)+ ,@%go-build-system-modules)+ #:phases+ (modify-phases (@ (guix build go-build-system) %standard-phases)+ (add-before 'install 'install-internal-inputs+ (lambda* (#:key inputs outputs #:allow-other-keys)+ (let ((out (assoc-ref outputs "out")))+ ;; The Go compiler does not permit importing libraries with+ ;; "internal" in the path from anywhere except below the+ ;; package that uses them. Thus, install these inputs+ ;; alongside this package.+ (union-build+ out+ (match (filter (lambda (input)+ (string-prefix? "go-gotest-tools-internal"+ (car input)))+ inputs)+ (((names . directories) ...) directories))+ #:create-all-directories? #t+ #:log-port (%make-void-port "w"))))))))+ (inputs+ `(("go-gotest-tools-internal-format"+ ,go-gotest-tools-internal-format)+ ("go-gotest-tools-internal-difflib"+ ,go-gotest-tools-internal-difflib)+ ("go-gotest-tools-internal-source"+ ,go-gotest-tools-internal-source))) (native-inputs `(("go-github-com-pkg-errors" ,go-github-com-pkg-errors) ("go-github-com-google-go-cmp-cmp"@@ -5985,17 +6011,6 @@ test when a comparison fails.") ,go-github-com-jonboulle-clockwork) ("go-golang-org-x-crypto" ,go-golang-org-x-crypto) ("go-gotest-tools-assert" ,go-gotest-tools-assert)- ("go-github-com-google-go-cmp-cmp"- ,go-github-com-google-go-cmp-cmp)- ;; TODO: This would be better as a propagated-input of- ;; go-gotest-tools-assert, but that does not work for- ;; some reason.- ("go-gotest-tools-internal-format"- ,go-gotest-tools-internal-format)- ("go-gotest-tools-internal-difflib"- ,go-gotest-tools-internal-difflib)- ("go-gotest-tools-internal-source"- ,go-gotest-tools-internal-source) ("go-github-com-google-go-cmp-cmp" ,go-github-com-google-go-cmp-cmp))) (synopsis "Go test runner with output optimized for humans")-- 2.31.1
M
M
Marius Bakke wrote on 27 Aug 18:44 +0200
[PATCH] build-system/go: Trim store references using the native compiler option.
(address . 50227@debbugs.gnu.org)
20210827164423.17109-1-marius@gnu.org
* guix/build/go-build-system.scm (build): Add '-trimpath' to the 'go install'invocation.(remove-store-references, remove-go-references): Remove procedures.(%standard-phases): Don't include remove-go-references.* gnu/packages/docker.scm (docker)[arguments]: Add the '-trimpath' option tothe build flags. Remove phase remove-go-references.* gnu/packages/uucp.scm (nncp)[arguments]: Likewise.--- gnu/packages/docker.scm | 7 ++-- gnu/packages/uucp.scm | 8 ++-- guix/build/go-build-system.scm | 70 +++------------------------------- 3 files changed, 14 insertions(+), 71 deletions(-)
Toggle diff (169 lines)diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scmindex 8bac1b89ce..108f355aa7 100644--- a/gnu/packages/docker.scm+++ b/gnu/packages/docker.scm@@ -535,6 +535,8 @@ built-in registry server of Docker.") ;; Respectively, strip the symbol table and debug ;; information, and the DWARF symbol table. (setenv "LDFLAGS" "-s -w")+ ;; Trim store references from the compiled binary.+ (setenv "BUILDFLAGS" "-trimpath") ;; Make build faster (setenv "GOCACHE" "/tmp") #t))@@ -568,10 +570,7 @@ built-in registry server of Docker.") (install-file "bundles/dynbinary-daemon/dockerd" out-bin) (install-file (string-append "bundles/dynbinary-daemon/dockerd-" (getenv "VERSION"))- out-bin)- #t)))- (add-after 'install 'remove-go-references- (assoc-ref go:%standard-phases 'remove-go-references)))))+ out-bin))))))) (inputs `(("btrfs-progs" ,btrfs-progs) ("containerd" ,containerd) ; for containerd-shimdiff --git a/gnu/packages/uucp.scm b/gnu/packages/uucp.scmindex 120417dea1..9d39c88fe5 100644--- a/gnu/packages/uucp.scm+++ b/gnu/packages/uucp.scm@@ -127,6 +127,10 @@ between computers.") (substitute* (list "bin/default.do" "bin/hjson-cli.do" "test.do") ((" -mod=vendor") "") ((" -m") ""))+ (substitute* (list "bin/default.do" "bin/hjson-cli.do")+ ;; Prevent reference to the Go inputs in the compiled binaries.+ (("\\$GO build")+ "$GO build -trimpath")) ;; Use the correct module path. `go list` does not report the ;; correct module path since we have moved the source files. (substitute* "bin/default.do"@@ -138,9 +142,7 @@ between computers.") (replace 'check (lambda* (#:key tests? #:allow-other-keys) (when tests?- (invoke "contrib/do" "-c" "test"))))- (add-after 'install 'remove-go-references- (assoc-ref go:%standard-phases 'remove-go-references)))))+ (invoke "contrib/do" "-c" "test"))))))) (inputs `(("go-github-com-davecgh-go-xdr" ,go-github-com-davecgh-go-xdr) ("go-github-com-dustin-go-humanize" ,go-github-com-dustin-go-humanize)diff --git a/guix/build/go-build-system.scm b/guix/build/go-build-system.scmindex 37936fe5ca..fc5ee39c8d 100644--- a/guix/build/go-build-system.scm+++ b/guix/build/go-build-system.scm@@ -28,8 +28,6 @@ #:use-module (ice-9 match) #:use-module (ice-9 ftw) #:use-module (srfi srfi-1)- #:use-module (rnrs io ports)- #:use-module (rnrs bytevectors) #:export (%standard-phases go-build)) @@ -47,12 +45,9 @@ ;; structure called a 'workspace' [1]. This workspace can be found by Go via ;; the GOPATH environment variable. Typically, all Go source code and compiled ;; objects are kept in a single workspace, but GOPATH may be a list of-;; directories [2]. In this go-build-system we create a file system union of-;; the Go-language dependencies. Previously, we made GOPATH a list of store-;; directories, but stopped because Go programs started keeping references to-;; these directories in Go 1.11:-;; <https://bugs.gnu.org/33620>.-;;+;; directories [2], which we rely on here, with the caveat that the current+;; package must appear first on GOPATH.+; ;; Go software, whether a package or a command, is uniquely named using an ;; 'import path'. The import path is based on the URL of the software's source. ;; Because most source code is provided over the internet, the import path is@@ -88,7 +83,6 @@ ;; a tmpdir when creating the inputs union. ;; * Use Go modules [4] ;; * Re-use compiled packages [5]-;; * Stop needing remove-go-references (-trimpath ? ) ;; * Remove module packages, only offering the full Git repos? This is ;; more idiomatic, I think, because Go downloads Git repos, not modules. ;; What are the trade-offs?@@ -194,6 +188,8 @@ unpacking." (apply invoke "go" "install" "-v" ; print the name of packages as they are compiled "-x" ; print each command as it is invoked+ ;; Trim store references from the compiled binaries.+ "-trimpath" ;; Respectively, strip the symbol table and debug ;; information, and the DWARF symbol table. "-ldflags=-s -w"@@ -236,59 +232,6 @@ the standard install-license-files phase to first enter the correct directory." unpack-path)) (apply (assoc-ref gnu:%standard-phases 'install-license-files) args))) -(define* (remove-store-reference file file-name- #:optional (store (%store-directory)))- "Remove from FILE occurrences of FILE-NAME in STORE; return #t when FILE-NAME-is encountered in FILE, #f otherwise. This implementation reads FILE one byte at-a time, which is slow. Instead, we should use the Boyer-Moore string search-algorithm; there is an example in (guix build grafts)."- (define pattern- (string-take file-name- (+ 34 (string-length (%store-directory)))))-- (with-fluids ((%default-port-encoding #f))- (with-atomic-file-replacement file- (lambda (in out)- ;; We cannot use `regexp-exec' here because it cannot deal with- ;; strings containing NUL characters.- (format #t "removing references to `~a' from `~a'...~%" file-name file)- (setvbuf in 'block 65536)- (setvbuf out 'block 65536)- (fold-port-matches (lambda (match result)- (put-bytevector out (string->utf8 store))- (put-u8 out (char->integer #\/))- (put-bytevector out- (string->utf8- "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-"))- #t)- #f- pattern- in- (lambda (char result)- (put-u8 out (char->integer char))- result))))))--(define* (remove-go-references #:key allow-go-reference?- inputs outputs #:allow-other-keys)- "Remove any references to the Go compiler from the compiled Go executable-files in OUTPUTS."-;; We remove this spurious reference to save bandwidth when installing Go-;; executables. It would be better to not embed the reference in the first-;; place, but I'm not sure how to do that. The subject was discussed at:-;; <https://lists.gnu.org/archive/html/guix-devel/2017-10/msg00207.html>- (if allow-go-reference?- #t- (let ((go (assoc-ref inputs "go"))- (bin "/bin"))- (for-each (lambda (output)- (when (file-exists? (string-append (cdr output)- bin))- (for-each (lambda (file)- (remove-store-reference file go))- (find-files (string-append (cdr output) bin)))))- outputs)- #t)))- (define %standard-phases (modify-phases gnu:%standard-phases (delete 'bootstrap)@@ -299,8 +242,7 @@ files in OUTPUTS." (replace 'build build) (replace 'check check) (replace 'install install)- (replace 'install-license-files install-license-files)- (add-after 'install 'remove-go-references remove-go-references)))+ (replace 'install-license-files install-license-files))) (define* (go-build #:key inputs (phases %standard-phases) #:allow-other-keys #:rest args)-- 2.31.1
M
M
Marius Bakke wrote on 27 Aug 19:47 +0200
(address . 50227@debbugs.gnu.org)
87wno6n5ju.fsf@gnu.org
Marius Bakke <marius@gnu.org> skriver:
Toggle quote (8 lines)> * guix/build/go-build-system.scm (build): Add '-trimpath' to the 'go install'> invocation.> (remove-store-references, remove-go-references): Remove procedures.> (%standard-phases): Don't include remove-go-references.> * gnu/packages/docker.scm (docker)[arguments]: Add the '-trimpath' option to> the build flags. Remove phase remove-go-references.> * gnu/packages/uucp.scm (nncp)[arguments]: Likewise.
Some context for this patch, which I forgot to save before sending:
I happened to rediscover https://issues.guix.gnu.org/33620 and pulledout my yak shaving device again.
This patch removes the custom 'remove-store-references' procedure infavor of the native '-trimpath' option to 'go install', as alluded to inthe bug report as well as the build system commentary. I don't spot anydifference in sizes from 'master' apart from one package: Docker.
Docker explodes from 764.4 MiB to 1215.5 MiB with this patch even thoughit does use the '-trimpath' option. Perhaps -trimpath does not work aswell with dynamically linked executables as it does for static?
I'm willing to overlook this regression for now in favor of the reducedcomplexity, but comments welcome.
-----BEGIN PGP SIGNATURE-----
iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCYSklJQ8cbWFyaXVzQGdudS5vcmcACgkQ6HGLpZEUEHeJ9gD/Uq7jBi7fNKwJnDUx1sBTVWG8abmZstu3sJU2wFnmd+UA/0x+T5WzVa95c546RK7bleaOFOC0yKrN4rEiVd2YPbYG=ZQ24-----END PGP SIGNATURE-----
M
M
Marius Bakke wrote on 27 Aug 21:38 +0200
(address . 50227@debbugs.gnu.org)
87tujan0e7.fsf@gnu.org
Marius Bakke <marius@gnu.org> skriver:
Toggle quote (10 lines)> Marius Bakke <marius@gnu.org> skriver:>>> * guix/build/go-build-system.scm (build): Add '-trimpath' to the 'go install'>> invocation.>> (remove-store-references, remove-go-references): Remove procedures.>> (%standard-phases): Don't include remove-go-references.>> * gnu/packages/docker.scm (docker)[arguments]: Add the '-trimpath' option to>> the build flags. Remove phase remove-go-references.>> * gnu/packages/uucp.scm (nncp)[arguments]: Likewise.
[...]
Toggle quote (4 lines)> Docker explodes from 764.4 MiB to 1215.5 MiB with this patch even though> it does use the '-trimpath' option. Perhaps -trimpath does not work as> well with dynamically linked executables as it does for static?
The size difference comes from containerd, which has a custom buildsystem that does not add -trimpath. After adding the following hunk:
Toggle diff (18 lines)diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scmindex 88dccc2ae2..e1ddfc6c38 100644--- a/gnu/packages/docker.scm+++ b/gnu/packages/docker.scm@@ -221,6 +221,13 @@ Python without keeping their credentials in a Docker configuration file.") (("exec\\.LookPath\\(\"unpigz\"\\)") (string-append "\"" (assoc-ref inputs "pigz") "/bin/unpigz\", error(nil)"))))))+ (add-before 'build 'trim-store-references+ (lambda* (#:key import-path #:allow-other-keys)+ (substitute* (string-append "src/" import-path "/Makefile")+ ;; Pass the '-trimpath' option down to 'go build' in order+ ;; to avoid spurious store references.+ (("^GO_BUILD_FLAGS=")+ "GO_BUILD_FLAGS=-trimpath")))) (replace 'build (lambda* (#:key import-path #:allow-other-keys) (with-directory-excursion (string-append "src/" import-path)
...the size of Docker becomes 763.7 MiB, or 0.7 less than before.
I realize we can set the flag globally in go-build-system, instead ofjust for the build phase. Then we don't need to patch Docker,containerd, or anything else that does not use the stock build phase.
Toggle diff (33 lines)diff --git a/guix/build/go-build-system.scm b/guix/build/go-build-system.scmindex fc5ee39c8d..a6b9397d35 100644--- a/guix/build/go-build-system.scm+++ b/guix/build/go-build-system.scm@@ -137,6 +137,9 @@ dependencies, so it should be self-contained." ;; Using the current working directory as GOPATH makes it easier for packagers ;; who need to manipulate the unpacked source code. (setenv "GOPATH" (string-append (getcwd) ":" (getenv "GOPATH")))+ ;; Unconditionally set the -trimpath option to avoid spurious store references+ ;; from having multiple GOPATH entries. See <https://bugs.gnu.org/33620>.+ (setenv "GOFLAGS" "-trimpath") ;; Go 1.13 uses go modules by default. The go build system does not ;; currently support modules, so turn modules off to continue using the old ;; GOPATH behavior.@@ -188,8 +191,6 @@ unpacking." (apply invoke "go" "install" "-v" ; print the name of packages as they are compiled "-x" ; print each command as it is invoked- ;; Trim store references from the compiled binaries.- "-trimpath" ;; Respectively, strip the symbol table and debug ;; information, and the DWARF symbol table. "-ldflags=-s -w"@@ -202,6 +203,9 @@ unpacking." ;; Can this also install commands??? (define* (check #:key tests? import-path #:allow-other-keys) "Run the tests for the package named by IMPORT-PATH."+ ;; Remove the global -trimpath option because it can break some test+ ;; suites.+ (unsetenv "GOFLAGS") (when tests? (invoke "go" "test" import-path)) #t)
This may be a cleaner solution. Thoughts?
-----BEGIN PGP SIGNATURE-----
iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCYSk/QA8cbWFyaXVzQGdudS5vcmcACgkQ6HGLpZEUEHe7ggD/TEW7hZyJ2lvB7YjJo2j9VBEgrS1orktACioCKYPcSvoA/j1OmpU9+lCNzF+A5GSROTTacDXpoY8J7zsP2hlMd6AM=KQpw-----END PGP SIGNATURE-----
S
S
Sarah Morgensen wrote on 28 Aug 04:16 +0200
Re: [bug#50227] [PATCH 0/3] go-build-system and GOPATH improvements
(name . Marius Bakke)(address . marius@gnu.org)(address . 50227@debbugs.gnu.org)
86mtp248ku.fsf@mgsn.dev
Hello Marius,
(Apologies in advance for the length of this treatise! I did not havethe time to be concise.)
Marius Bakke <marius@gnu.org> writes:
Toggle quote (3 lines)> These patches adjust the Go build system to use Guix's regular> native-search-paths mechanism instead of ad-hoc GOPATH trickery.
I have been working on overhauling the Go build system behind thescenes; I expected to have a patch ready last week, but I fell down the"modules" rabbit hole after learning GOPATH is expected to be deprecatedas soon as 1.18. Sorry for the duplicated work! (I also have a Go 1.17ready to launch, but I've been attempting to nail down Go build systemchanges first so I didn't introduce anything incompatible.)
In any case, I hadn't thought of using search paths, that's quiteclever! I like it.
Before falling down the "modules" rabbit-hole, here is what was workingfor my GOPATH-based Go build system:
1. Install source in "out/share/go/src" rather than "out/src", and thensimply create a directory union of "/share/go/src" from inputs. Thisavoids accidentally including non-go packages with a "/src" directory.If you did this, then you could make the GOPATH search path be"/share/go/..".
2. Split the GOPATH as you have done (except with only two components;one for the package we're building and one for the union in 1).
3. Reuse build artifacts by copying the $GOPATH[0]/pkg directory to"out/share/go/pkg" in the install phase. They will be transparentlyused since they will be in GOPATH. (You can use "out/lib/go/pkg", butyou must set 'strip-directories' to avoid stripping Go archives, andthen include "/lib/go/.." in the GOPATH search path.) However, "goinstall" will eventually deprecate installing archives [0], perhaps evenbefore GOPATH is deprecated.
4. Use -trimpath instead of remove-go-references, as you did. Also, toavoid rebuilding the standard library with '-trimpath' for every package(since the Go build cache does not persist between build environments):
a) modify the Go package to build standard libraries with -trimpath, which would unfortunately mean most users of the Go package would find that ~180MB of space wasted; or b) build a '-trimpath' version of the standard library separately and use it with '-pkgdir' (which would prevent #3 from working) or by building a directory union of Go and Go-std-library-with-trimpath and setting GOROOT=/path/to/union.
Personally, I'm partial to a), along with removing the pre-compiledstandard library from the Go package since it ends up recompiled moreoften than not, is very fast to recompile, and it will eventually nolonger be distributed or used by Go [0].
Toggle quote (16 lines)>> The context is that I needed to hack on a Go package, and was somewhat> surprised that my usual workflow of "guix environment PKG" did not work.>> It still does not work "out of the box", but these patches bring it a> step further. Now "all" that is needed is to:>> $ cd ~/src/go-foo> $ guix environment go-example-com-foo> $ MYGOPATH="$HOME/tmp/go"> $ NAMESPACE="$MYGOPATH/src/example.com/foo"> $ mkdir -p $(dirname $NAMESPACE)> $ ln -s $PWD $NAMESPACE # or git worktree add $NAMESPACE> $ export GOPATH="$MYGOPATH:$GOPATH"> $ go build # no 'go get' necessary!
Interesting. I hadn't thought of the use-case for actually hacking on gopackages like this! I'll have to think of how modules-mode can be madeto work with this.
(A digression: the current issue with fully implementing module-awaremode is that Go really wants a specific version for each dependency. Ifwe just populate the module cache with the versions we have, it willinevitable complain when a package we try to build wants a version we donot have. I see a few solutions:
1. Put all dependencies in the module cache, and rewrite the mainmodule's go.mod (that is, add replace directives) to replace alldependencies with the versions we have.
2. Rewrite the go.mod to replace all dependencies with the localfilesystem path of the versions we have.
3. Put all dependencies in the vendor/ directory, and use -mod=vendor.Any pre-existing vendor directory must be handled properly.
These three solutions fail to allow re-using the build cache (andtherefore build artifacts), because Go computes the build cache keysdifferently for main and non-main modules. Building in Go is generallyfast, so we probably shouldn't compromise much to enable reusing thebuild cache, but a few ideas for doing so:
4. Set up a dummy go.mod out of the source tree, which 'replace's alldependencies AND the module we're building like in 1) or 2). This mayhave to account for replace directives in the go.mod of the module we'rebuilding, though.
5. Put the module we're building in the module cache, and build it with"go install module@version". The same caveat as in 4) applies, as wellas that "go install module@version" only works for main packages (thatis, packages which produce an executable).)
Toggle quote (5 lines)>> I don't know how feasible it is to avoid making a local directory and> symlinking the project to the expected namespace. Still a complete Go> newbie, but this approach feels more natural and idiomatic Guix-wise.
My intuition is that if you're working in GOPATH-mode, you already havea ~/go/src directory or similar, and your project is probably under~/go/src/my-project. Then, in order to hack on it Guix-like, you would
$ cd ~/go/src/my-project $ guix environment go-github-com-me-my-project $ export GOPATH=~/go:$GOPATH $ go build
I'm not sure what a similar idiom for Guix-like hacking in module-awaremode would be; we'd have to set GOMODCACHE or something, but it would bevery easy for Go to overwrite (or fail to overwrite) things withoutGOPROXY=off. Alternatively, if we make a "full" go proxy directorylayout, we can do
GOPROXY=file://path/to/gomodcache
or even a search path like
GOPROXY=file:///gnu/store/p1/gomodcache|file://gnu/store/p2/gomodcache
though I'm not sure how well that would scale w.r.t. number of packages.
Both of these GOPROXY methods have the advantage over setting GOMODCACHEthat the user could modify GOPROXY to include the default proxy, andwould still be able to get packages and versions not packaged by Guix.
I suppose there's no reason we couldn't set both GOPATH ande.g. GOPROXY.
[0] https://github.com/golang/go/issues/47257
--Sarah
M
M
Marius Bakke wrote on 28 Aug 16:52 +0200
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)(address . 50227@debbugs.gnu.org)
87r1edmxjk.fsf@gnu.org
Hi Sarah,
Sarah Morgensen <iskarian@mgsn.dev> skriver:
Toggle quote (5 lines)> Hello Marius,>> (Apologies in advance for the length of this treatise! I did not have> the time to be concise.)
No hurries, the insightful feedback is much appreciated.
Toggle quote (12 lines)> Marius Bakke <marius@gnu.org> writes:>>> These patches adjust the Go build system to use Guix's regular>> native-search-paths mechanism instead of ad-hoc GOPATH trickery.>> I have been working on overhauling the Go build system behind the> scenes; I expected to have a patch ready last week, but I fell down the> "modules" rabbit hole after learning GOPATH is expected to be deprecated> as soon as 1.18. Sorry for the duplicated work! (I also have a Go 1.17> ready to launch, but I've been attempting to nail down Go build system> changes first so I didn't introduce anything incompatible.)
That is excellent news, thank you! I briefly looked into "modules"while researching this, but realized that the rabbit hole would be toodeep for me. I did not notice that GOPATH was being deprecated however.
Toggle quote (15 lines)> In any case, I hadn't thought of using search paths, that's quite> clever! I like it.>> Before falling down the "modules" rabbit-hole, here is what was working> for my GOPATH-based Go build system:>> 1. Install source in "out/share/go/src" rather than "out/src", and then> simply create a directory union of "/share/go/src" from inputs. This> avoids accidentally including non-go packages with a "/src" directory.> If you did this, then you could make the GOPATH search path be> "/share/go/..".>> 2. Split the GOPATH as you have done (except with only two components;> one for the package we're building and one for the union in 1).
Installing to $out/share/go is much nicer indeed.
Toggle quote (8 lines)> 3. Reuse build artifacts by copying the $GOPATH[0]/pkg directory to> "out/share/go/pkg" in the install phase. They will be transparently> used since they will be in GOPATH. (You can use "out/lib/go/pkg", but> you must set 'strip-directories' to avoid stripping Go archives, and> then include "/lib/go/.." in the GOPATH search path.) However, "go> install" will eventually deprecate installing archives [0], perhaps even> before GOPATH is deprecated.
Right. The churn in Go's build tooling is surprising, I would expect itto slow down at this point! I suppose we can ignore the "pkg" artifactsfor now, or keep them in "share/go" as a stop-gap, since they will bedeprecated soon.
Toggle quote (18 lines)> 4. Use -trimpath instead of remove-go-references, as you did. Also, to> avoid rebuilding the standard library with '-trimpath' for every package> (since the Go build cache does not persist between build environments):>> a) modify the Go package to build standard libraries with -trimpath,> which would unfortunately mean most users of the Go package would> find that ~180MB of space wasted; or> > b) build a '-trimpath' version of the standard library separately and> use it with '-pkgdir' (which would prevent #3 from working) or by> building a directory union of Go and Go-std-library-with-trimpath> and setting GOROOT=/path/to/union.>> Personally, I'm partial to a), along with removing the pre-compiled> standard library from the Go package since it ends up recompiled more> often than not, is very fast to recompile, and it will eventually no> longer be distributed or used by Go [0].
Removing the compiled libraries sounds fine to me. I suppose we'llstill need -trimpath for executables ("main.go")?
Toggle quote (19 lines)>> The context is that I needed to hack on a Go package, and was somewhat>> surprised that my usual workflow of "guix environment PKG" did not work.>>>> It still does not work "out of the box", but these patches bring it a>> step further. Now "all" that is needed is to:>>>> $ cd ~/src/go-foo>> $ guix environment go-example-com-foo>> $ MYGOPATH="$HOME/tmp/go">> $ NAMESPACE="$MYGOPATH/src/example.com/foo">> $ mkdir -p $(dirname $NAMESPACE)>> $ ln -s $PWD $NAMESPACE # or git worktree add $NAMESPACE>> $ export GOPATH="$MYGOPATH:$GOPATH">> $ go build # no 'go get' necessary!>> Interesting. I hadn't thought of the use-case for actually hacking on go> packages like this! I'll have to think of how modules-mode can be made> to work with this.
It's not a very important feature, and I'm happy to scratch that itchagain once gomodules are first-class. :-)
Toggle quote (32 lines)> (A digression: the current issue with fully implementing module-aware> mode is that Go really wants a specific version for each dependency. If> we just populate the module cache with the versions we have, it will> inevitable complain when a package we try to build wants a version we do> not have. I see a few solutions:>> 1. Put all dependencies in the module cache, and rewrite the main> module's go.mod (that is, add replace directives) to replace all> dependencies with the versions we have.>> 2. Rewrite the go.mod to replace all dependencies with the local> filesystem path of the versions we have.>> 3. Put all dependencies in the vendor/ directory, and use -mod=vendor.> Any pre-existing vendor directory must be handled properly.>> These three solutions fail to allow re-using the build cache (and> therefore build artifacts), because Go computes the build cache keys> differently for main and non-main modules. Building in Go is generally> fast, so we probably shouldn't compromise much to enable reusing the> build cache, but a few ideas for doing so:>> 4. Set up a dummy go.mod out of the source tree, which 'replace's all> dependencies AND the module we're building like in 1) or 2). This may> have to account for replace directives in the go.mod of the module we're> building, though.>> 5. Put the module we're building in the module cache, and build it with> "go install module@version". The same caveat as in 4) applies, as well> as that "go install module@version" only works for main packages (that> is, packages which produce an executable).)
Thank you for this analysis. The vendoring option is compelling, if itdoes not require patching the go.mod files, and can work also forpackages where unbundling is not feasible (or downstream channels withless strict packaging policies).
For reusing build artifacts, perhaps we can piggy-back on whatever isimplemented for Bazel as mentioned in [0].
Toggle quote (13 lines)>> I don't know how feasible it is to avoid making a local directory and>> symlinking the project to the expected namespace. Still a complete Go>> newbie, but this approach feels more natural and idiomatic Guix-wise.>> My intuition is that if you're working in GOPATH-mode, you already have> a ~/go/src directory or similar, and your project is probably under> ~/go/src/my-project. Then, in order to hack on it Guix-like, you would>> $ cd ~/go/src/my-project> $ guix environment go-github-com-me-my-project> $ export GOPATH=~/go:$GOPATH> $ go build
That is much easier. You can tell I never hacked on Go before. :-)
Toggle quote (21 lines)> I'm not sure what a similar idiom for Guix-like hacking in module-aware> mode would be; we'd have to set GOMODCACHE or something, but it would be> very easy for Go to overwrite (or fail to overwrite) things without> GOPROXY=off. Alternatively, if we make a "full" go proxy directory> layout, we can do>> GOPROXY=file://path/to/gomodcache>> or even a search path like>> GOPROXY=file:///gnu/store/p1/gomodcache|file://gnu/store/p2/gomodcache>> though I'm not sure how well that would scale w.r.t. number of packages.>> Both of these GOPROXY methods have the advantage over setting GOMODCACHE> that the user could modify GOPROXY to include the default proxy, and> would still be able to get packages and versions not packaged by Guix.>> I suppose there's no reason we couldn't set both GOPATH and> e.g. GOPROXY.
GOPROXY seems like a great middle ground for local development.
Given the conflicting work here, what do you think we should do? I'mhappy to scrap this PR as it was largely an exercise to learngo-build-system, in addition to scratching a very minor itch.
Is the reduced complexity worth it while waiting for the gomodulesrewrite, and if so, are there parts that can be merged with your worksuch as using $out/share/go?
Let me know if I can be of assistance. :-)
Toggle quote (1 lines)> [0] https://github.com/golang/go/issues/47257
-----BEGIN PGP SIGNATURE-----
iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCYSpNrw8cbWFyaXVzQGdudS5vcmcACgkQ6HGLpZEUEHd1NwEAips91ZItQ8lyCOvqy99JEgtStC5K0TaQVnMQ6EaFHBgA/Rk5pg/pxBbyPEE+rZLfdYq7U+1/N3BwtmwiTiN6SSMC=/UXy-----END PGP SIGNATURE-----
S
S
Sarah Morgensen wrote on 29 Aug 08:17 +0200
Re: [bug#50227] [PATCH 3/3] gnu: go-gotest-tools-assert: Provide internal inputs with the source.
(name . Marius Bakke)(address . marius@gnu.org)(address . 50227@debbugs.gnu.org)
8635qs3hce.fsf@mgsn.dev
Hello,
Marius Bakke <marius@gnu.org> writes:
Toggle quote (81 lines)> * gnu/packages/golang.scm (go-gotest-tools-assert)[inputs]: Add> GO-GOTEST-TOOLS-INTERNAL-FORMAT, GO-GOTEST-TOOLS-INTERNAL-DIFFLIB, and> GO-GOTEST-TOOLS-INTERNAL-SOURCE.> [arguments]: Add phase to install a union of the above inputs.> * gnu/packages/golang.scm (gotestsum)[native-inputs]: Don't add the above> mentioned inputs.> ---> gnu/packages/golang.scm | 45 +++++++++++++++++++++++++++--------------> 1 file changed, 30 insertions(+), 15 deletions(-)>> diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm> index 3a5c6ddc3f..295b442a2a 100644> --- a/gnu/packages/golang.scm> +++ b/gnu/packages/golang.scm> @@ -20,7 +20,7 @@> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>> ;;; Copyright © 2020 Nicolas Goaziou <mail@nicolasgoaziou.com>> ;;; Copyright © 2020 Ryan Prior <rprior@protonmail.com>> -;;; Copyright © 2020 Marius Bakke <marius@gnu.org>> +;;; Copyright © 2020, 2021 Marius Bakke <marius@gnu.org>> ;;; Copyright © 2020 raingloom <raingloom@riseup.net>> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>> ;;; Copyright © 2021 Ricardo Wurmus <rekado@elephly.net>> @@ -5945,9 +5945,35 @@ gotest-tools.")))> (arguments> `(#:tests? #f ; Test failure concerning message formatting (FIXME)> #:import-path "gotest.tools/assert"> - #:unpack-path "gotest.tools"))> - ;(propagated-inputs> - ; `(("go-gotest-tools-internal-format" ,go-gotest-tools-internal-format)))> + #:unpack-path "gotest.tools"> + #:modules ((ice-9 match)> + (srfi srfi-26)> + ,@%go-build-system-modules)> + #:phases> + (modify-phases (@ (guix build go-build-system) %standard-phases)> + (add-before 'install 'install-internal-inputs> + (lambda* (#:key inputs outputs #:allow-other-keys)> + (let ((out (assoc-ref outputs "out")))> + ;; The Go compiler does not permit importing libraries with> + ;; "internal" in the path from anywhere except below the> + ;; package that uses them. Thus, install these inputs> + ;; alongside this package.> + (union-build> + out> + (match (filter (lambda (input)> + (string-prefix? "go-gotest-tools-internal"> + (car input)))> + inputs)> + (((names . directories) ...) directories))> + #:create-all-directories? #t> + #:log-port (%make-void-port "w"))))))))> + (inputs> + `(("go-gotest-tools-internal-format"> + ,go-gotest-tools-internal-format)> + ("go-gotest-tools-internal-difflib"> + ,go-gotest-tools-internal-difflib)> + ("go-gotest-tools-internal-source"> + ,go-gotest-tools-internal-source)))> (native-inputs> `(("go-github-com-pkg-errors" ,go-github-com-pkg-errors)> ("go-github-com-google-go-cmp-cmp"> @@ -5985,17 +6011,6 @@ test when a comparison fails.")> ,go-github-com-jonboulle-clockwork)> ("go-golang-org-x-crypto" ,go-golang-org-x-crypto)> ("go-gotest-tools-assert" ,go-gotest-tools-assert)> - ("go-github-com-google-go-cmp-cmp"> - ,go-github-com-google-go-cmp-cmp)> - ;; TODO: This would be better as a propagated-input of> - ;; go-gotest-tools-assert, but that does not work for> - ;; some reason.> - ("go-gotest-tools-internal-format"> - ,go-gotest-tools-internal-format)> - ("go-gotest-tools-internal-difflib"> - ,go-gotest-tools-internal-difflib)> - ("go-gotest-tools-internal-source"> - ,go-gotest-tools-internal-source)> ("go-github-com-google-go-cmp-cmp"> ,go-github-com-google-go-cmp-cmp)))> (synopsis "Go test runner with output optimized for humans")
Just piggybacking off this to add that I believe the "correct" wayforward to handle packages like these is to put all of them in a singlego-gotest-tools package, and modify the build system to build them all.
I've tested a proof-of-concept of this, based off of what Debian does[0]. Essentially:
1. Add two arguments to the build system, GO-PACKAGES-INCLUDE andGO-PACKAGES-EXCLUDE. GO-PACKAGES-INCLUDE defaults to something like'("IMPORT-PATH/...") and GO-PACKAGES-EXCLUDE defaults to the empty list.
2. Run "go list GO-PACKAGES-INCLUDE", which lists all packages matchingGO-PACKAGES-INCLUDE.
3. Remove any packages matching GO-PACKAGES-EXCLUDE (should this be aregex? I'm not sure), leaving us with GO-PACKAGES.
4. Run "go install ... GO-PACKAGES"
From my testing, this causes a LOT of packages to need edits, because itsurfaces a lot of hidden bugs. For example, suppose we have a Guixpackage "go-B-tool" with import path "B/tool" and another Guix package"go-use-B" which imports "B/tool/extra". If "B/tool/extra" is notimported by "B/tool", we will not have actually built or tested"B/tool/extra" so we will only encounter issues when building"go-use-B", even those the actual issue should be addressed in"go-B-tool".
In the case of the above package, we would merge all go-gotest-toolspackages into a go-gotest-tools-v3 package, with the import path"gotest.tools/v3", which is what its go.mod states. (Note that none ofthe sub-packages have their own go.mod, so it would cause issues downthe road with the module system to have each of those sub-packages be aGuix package.) With the above build-system changes, all of thepreviously-separate packages would be built and tested together. (If wewanted to exclude a problematic package which we didn't need, we couldremove the directory in a snippet.)
Depending on how many packages are affected, perhaps this part willwarrant a wip-go-build-system branch?
[0] https://manpages.debian.org/testing/dh-golang/Debian::Debhelper::Buildsystem::golang.3pm.en.html
--Sarah
S
S
Sarah Morgensen wrote on 4 Sep 00:55 +0200
Re: [bug#50227] [PATCH 0/3] go-build-system and GOPATH improvements
(name . Marius Bakke)(address . marius@gnu.org)(address . 50227@debbugs.gnu.org)
86ilzhxo97.fsf@mgsn.dev
Hello,
Marius Bakke <marius@gnu.org> writes:
Toggle quote (4 lines)> That is excellent news, thank you! I briefly looked into "modules"> while researching this, but realized that the rabbit hole would be too> deep for me. I did not notice that GOPATH was being deprecated however.
The Go 1.16 release notes actually stated that a deprecation noticewould be printed in Go 1.17 but then they didn't do that! I thinkthey're waiting for the "workspaces" feature [0] to be implementedfirst.
[0] https://github.com/golang/go/issues/45713
Toggle quote (21 lines)>> 4. Use -trimpath instead of remove-go-references, as you did. Also, to>> avoid rebuilding the standard library with '-trimpath' for every package>> (since the Go build cache does not persist between build environments):>>>> a) modify the Go package to build standard libraries with -trimpath,>> which would unfortunately mean most users of the Go package would>> find that ~180MB of space wasted; or>> >> b) build a '-trimpath' version of the standard library separately and>> use it with '-pkgdir' (which would prevent #3 from working) or by>> building a directory union of Go and Go-std-library-with-trimpath>> and setting GOROOT=/path/to/union.>>>> Personally, I'm partial to a), along with removing the pre-compiled>> standard library from the Go package since it ends up recompiled more>> often than not, is very fast to recompile, and it will eventually no>> longer be distributed or used by Go [0].>> Removing the compiled libraries sounds fine to me. I suppose we'll> still need -trimpath for executables ("main.go")?
Yes, we'll need '-trimpath' in all invocations of build tools, including'test' as well, or it will actually recompile everything without'-trimpath' and then test that!
Toggle quote (37 lines)>> (A digression: the current issue with fully implementing module-aware>> mode is that Go really wants a specific version for each dependency. If>> we just populate the module cache with the versions we have, it will>> inevitable complain when a package we try to build wants a version we do>> not have. I see a few solutions:>>>> 1. Put all dependencies in the module cache, and rewrite the main>> module's go.mod (that is, add replace directives) to replace all>> dependencies with the versions we have.>>>> 2. Rewrite the go.mod to replace all dependencies with the local>> filesystem path of the versions we have.>>>> 3. Put all dependencies in the vendor/ directory, and use -mod=vendor.>> Any pre-existing vendor directory must be handled properly.>>>> These three solutions fail to allow re-using the build cache (and>> therefore build artifacts), because Go computes the build cache keys>> differently for main and non-main modules. Building in Go is generally>> fast, so we probably shouldn't compromise much to enable reusing the>> build cache, but a few ideas for doing so:>>>> 4. Set up a dummy go.mod out of the source tree, which 'replace's all>> dependencies AND the module we're building like in 1) or 2). This may>> have to account for replace directives in the go.mod of the module we're>> building, though.>>>> 5. Put the module we're building in the module cache, and build it with>> "go install module@version". The same caveat as in 4) applies, as well>> as that "go install module@version" only works for main packages (that>> is, packages which produce an executable).)>> Thank you for this analysis. The vendoring option is compelling, if it> does not require patching the go.mod files, and can work also for> packages where unbundling is not feasible (or downstream channels with> less strict packaging policies).
Yes, and it has the upside that '-mod=vendor' automatically disablesnetwork access for most tools.
On the other hand, I think the only solution that properly moves in thedirection of allowing hacking on Go packages is the GOPROXY search-pathapproach.
This requires some more thought, for sure.
Toggle quote (3 lines)> For reusing build artifacts, perhaps we can piggy-back on whatever is> implemented for Bazel as mentioned in [0].
Bazel is an entire build system replacing cmd/go, which (presumably)constructs its own build graph, etc, so probably a little heavy-handedfor our use-case. We probably want to stay as close as possible tovanilla tooling.
Probably the best we can do is saving and re-using the build cache,since they seem insistent on moving everything into the cache.
Toggle quote (12 lines)> Given the conflicting work here, what do you think we should do? I'm> happy to scrap this PR as it was largely an exercise to learn> go-build-system, in addition to scratching a very minor itch.>> Is the reduced complexity worth it while waiting for the gomodules> rewrite, and if so, are there parts that can be merged with your work> such as using $out/share/go?>> Let me know if I can be of assistance. :-)>>> [0] https://github.com/golang/go/issues/47257
I think I would suggest breaking up the Go build system changes into:
1. Make the changes roughly included in your patch, along with making a"go-std-cache-for-build" package (hidden?) which will be an implicitinput in the Go build system (perhaps non-substitutable? it will befaster to build than download on nearly all machines), and seeding thebuild cache with it in 'setup-go-environment'. We skip re-using buildartifacts.
2. Update Go build system to use Go 1.16, leaving only docker with Go1.14 (via "#:go ,go-1.14"). We should be ready to do this as soon as[0] is fixed [1].
3. Enable building multiple Go packages in one Guix package, and mergeall Guix packages such that one Guix package == one module path.
4. Make gomodules changes.
5. Release Go 1.18, which will bootstrap from Go 1.16 or Gccgo 11 [2].
6. Update Go build system to use Go 1.18.
[0] https://issues.guix.gnu.org/49921[1] https://logs.guix.gnu.org/guix/2021-08-11.log#213401[2] https://github.com/golang/go/issues/44505 build: Adopt Go 11.6 asbootstrap toolchain for Go 1.18
--Sarah
S
S
Sarah Morgensen wrote on 10 Sep 02:51 +0200
control message for bug #50348
(address . control@debbugs.gnu.org)
8635qdmevt.fsf@mgsn.dev
block 50348 by 50227 49921 50495thanks
S
S
Sarah Morgensen wrote on 10 Sep 02:55 +0200
(address . control@debbugs.gnu.org)
861r5xmeq2.fsf@mgsn.dev
unblock 50348 by 50227 49921 50495thanks
Oops, wrong bug #.
S
S
Sarah Morgensen wrote on 10 Sep 02:56 +0200
control message for bug #50493
(address . control@debbugs.gnu.org)
86zgsll03k.fsf@mgsn.dev
block 50493 by 50495 49921 50227thanks
S
S
Sarah Morgensen wrote on 20 Sep 02:23 +0200
(address . control@debbugs.gnu.org)
E1mS75y-0006Lt-Ve@debbugs.gnu.org
unblock 50493 by 50227quit
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 50227@debbugs.gnu.org