[PATCH 0/2] Fix annoyance with "guix import go"

  • Open
  • quality assurance status badge
Details
4 participants
  • Timo Wilken
  • Ludovic Courtès
  • Christopher Baines
  • Simon Tournier
Owner
unassigned
Submitted by
Simon Tournier
Severity
normal
S
S
Simon Tournier wrote on 22 May 2023 18:04
(address . guix-patches@gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
cover.1684771100.git.zimon.toutoune@gmail.com
Hi,

As reported in [1], the Go importer returns backtrace. This patch fixes
that. The first patch is about sorting the versions. The second is about
warn but not exit. It allows to have a first draft when applying recursive
importer.

Compare (first patch),

Toggle snippet (15 lines)
$ guix import go go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@0.41
guix import: error: version v0.41 of go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc is not available

hint: Pick one of the following available versions: 0.26.0 0.36.0 0.15.0 0.41.0-rc.1 0.40.0 0.37.0 0.33.0 0.20.0 0.24.0 0.23.0 0.41.0 0.26.1 0.41.1 0.22.0
0.18.0 0.21.0 0.12.0 0.36.4 0.19.0 0.25.0 0.27.0 0.36.2 0.34.0 0.14.0 0.36.3 0.31.0 0.15.1 0.36.1 0.41.0-rc.2 0.32.0 0.28.0 0.35.0 0.42.0-rc.1 0.13.0
0.17.0 0.29.0 0.30.0 0.39.0 0.38.0 0.16.0.

$ ./pre-inst-env guix import go go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@0.41
guix import: error: version v0.41 of go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc is not available

hint: Pick one of the following available versions: 0.42.0-rc.1 0.41.1 0.41.0-rc.2 0.41.0-rc.1 0.41.0 0.40.0 0.39.0 0.38.0 0.37.0 0.36.4 0.36.3 0.36.2 0.36.1
0.36.0 0.35.0 0.34.0 0.33.0 0.32.0 0.31.0 0.30.0 0.29.0 0.28.0 0.27.0 0.26.1 0.26.0 0.25.0 0.24.0 0.23.0 0.22.0 0.21.0 0.20.0 0.19.0 0.18.0 0.17.0
0.16.0 0.15.1 0.15.0 0.14.0 0.13.0 0.12.0.

Then instead of,

Toggle snippet (33 lines)
$ guix import go go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
Backtrace:
14 (primitive-load "/home/simon/.config/guix/current/bin/guix")
In guix/ui.scm:
2300:7 13 (run-guix . _)
2263:10 12 (run-guix-command _ . _)
In guix/scripts/import.scm:
89:11 11 (guix-import . _)
In ice-9/boot-9.scm:
1752:10 10 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In guix/scripts/import/go.scm:
116:29 9 (_)
In ice-9/exceptions.scm:
406:15 8 (go-module->guix-package* . _)
In ice-9/boot-9.scm:
1752:10 7 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In guix/import/go.scm:
511:19 6 (go-module->guix-package "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" #:goproxy _ #:version _ # _)
In guix/git.scm:
291:4 5 (update-cached-checkout _ #:ref _ #:recursive? _ #:check-out? _ #:starting-commit _ #:log-port _ #:cache-directory _)
277:19 4 (resolve _)
In git/reference.scm:
60:8 3 (_ _ _)
In git/bindings.scm:
77:2 2 (raise-git-error _)
In ice-9/boot-9.scm:
1685:16 1 (raise-exception _ #:continuable? _)
1683:16 0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1683:16: In procedure raise-exception:
Git error: reference 'refs/tags/v0.41.1' not found

Now it returns (second patch),

Toggle snippet (49 lines)
$ ./pre-inst-env guix import go go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
guix import: warning: Git error: reference 'refs/tags/v0.41.1' not found in https://github.com/open-telemetry/opentelemetry-go-contrib
(define-public go-go-opentelemetry-io-contrib-instrumentation-google-golang-org-grpc-otelgrpc
(package
(name
"go-go-opentelemetry-io-contrib-instrumentation-google-golang-org-grpc-otelgrpc")
(version "0.41.1")
(source (origin
(method git-fetch)
(uri (git-reference
(url
"https://github.com/open-telemetry/opentelemetry-go-contrib")
(commit (string-append "v" version))))
(file-name (git-file-name name version))
(sha256
(base32
"0000000000000000000000000000000000000000000000000000"))))
(build-system go-build-system)
(arguments
'(#:import-path
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"))
(propagated-inputs `(("go-gopkg-in-yaml-v3" ,go-gopkg-in-yaml-v3)
("go-google-golang-org-genproto" ,go-google-golang-org-genproto)
("go-google-golang-org-appengine" ,go-google-golang-org-appengine)
("go-golang-org-x-text" ,go-golang-org-x-text)
("go-golang-org-x-sys" ,go-golang-org-x-sys)
("go-golang-org-x-oauth2" ,go-golang-org-x-oauth2)
("go-golang-org-x-net" ,go-golang-org-x-net)
("go-github-com-pmezard-go-difflib" ,go-github-com-pmezard-go-difflib)
("go-github-com-golang-protobuf" ,go-github-com-golang-protobuf)
("go-github-com-go-logr-stdr" ,go-github-com-go-logr-stdr)
("go-github-com-go-logr-logr" ,go-github-com-go-logr-logr)
("go-github-com-davecgh-go-spew" ,go-github-com-davecgh-go-spew)
("go-cloud-google-com-go-compute-metadata" ,go-cloud-google-com-go-compute-metadata)
("go-cloud-google-com-go-compute" ,go-cloud-google-com-go-compute)
("go-google-golang-org-protobuf" ,go-google-golang-org-protobuf)
("go-google-golang-org-grpc" ,go-google-golang-org-grpc)
("go-go-opentelemetry-io-otel-trace" ,go-go-opentelemetry-io-otel-trace)
("go-go-opentelemetry-io-otel-metric" ,go-go-opentelemetry-io-otel-metric)
("go-go-opentelemetry-io-otel" ,go-go-opentelemetry-io-otel)
("go-github-com-stretchr-testify" ,go-github-com-stretchr-testify)))
(home-page
"https://go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc")
(synopsis #f)
(description #f)
(license license:asl2.0)))[env]


Cheers,
simon


Simon Tournier (2):
guix: import: go: Sort hint about available versions.
guix: import: go: Catch Git error.

guix/import/go.scm | 30 ++++++++++++++++++++----------
1 file changed, 20 insertions(+), 10 deletions(-)


base-commit: 849286ba66c96534bddc04df1a47d5692cbc977e
--
2.38.1
S
S
Simon Tournier wrote on 22 May 2023 18:08
[PATCH 1/2] guix: import: go: Sort hint about available versions.
(address . 63647@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
5cb8f8989fccc4580d10b7e39633f8880c838236.1684771100.git.zimon.toutoune@gmail.com
* guix/import/go.scm (validate-version): Sort available versions.
---
guix/import/go.scm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

Toggle diff (26 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1eb..1943869162 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -34,6 +34,7 @@ (define-module (guix import go)
#:use-module (guix packages)
#:use-module (guix http-client)
#:use-module (guix memoization)
+ #:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
#:autoload (guix base32) (bytevector->nix-base32-string)
#:autoload (guix build utils) (mkdir-p)
@@ -588,8 +589,9 @@ (define (validate-version version available-versions module-path)
(condition (&fix-hint
(hint (format #f (G_ "Pick one of the following \
available versions:~{ ~a~}.")
- (map strip-v-prefix
- available-versions)))))))))
+ (sort (map strip-v-prefix
+ available-versions)
+ version>?)))))))))
(define* (go-module->guix-package module-path #:key
(goproxy "https://proxy.golang.org")
--
2.38.1
S
S
Simon Tournier wrote on 22 May 2023 18:08
[PATCH 2/2] guix: import: go: Catch Git error.
(address . 63647@debbugs.gnu.org)
307ae354ce980b521ae5878dd0710bd89f1ad0c5.1684771100.git.zimon.toutoune@gmail.com
* guix/import/go.scm (git-checkout-hash): Catch Git error.

Reported-by: Felix Lechner <felix.lechner@lease-up.com>.
---
guix/import/go.scm | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 1943869162..c6258296f6 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,7 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
-;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -36,9 +36,11 @@ (define-module (guix import go)
#:use-module (guix memoization)
#:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
- #:autoload (guix base32) (bytevector->nix-base32-string)
+ #:autoload (guix base32) (bytevector->nix-base32-string nix-base32-string->bytevector)
#:autoload (guix build utils) (mkdir-p)
+ #:autoload (guix ui) (warning)
#:autoload (gcrypt hash) (hash-algorithm sha256)
+ #:autoload (git structs) (git-error-message)
#:use-module (ice-9 format)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
@@ -507,12 +509,18 @@ (define* (git-checkout-hash url reference algorithm)
;; subsequent "guix import" invocations.
(mkdir-p cache)
(chmod cache #o700)
- (let-values (((checkout commit _)
- (parameterize ((%repository-cache-directory cache))
- (update-cached-checkout url
- #:ref
- `(tag-or-commit . ,reference)))))
- (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (catch 'git-error
+ (lambda _
+ (let-values (((checkout commit _)
+ (parameterize ((%repository-cache-directory cache))
+ (update-cached-checkout url
+ #:ref
+ `(tag-or-commit . ,reference)))))
+ (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (lambda (key error . rest)
+ (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
+ (nix-base32-string->bytevector
+ "0000000000000000000000000000000000000000000000000000"))))
(define (vcs->origin vcs-type vcs-repo-url version)
"Generate the `origin' block of a package depending on what type of source
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 17:41
[PATCH v2 2/3] guix: import: go: Catch Git error.
(address . 63647@debbugs.gnu.org)
f699a25c43805136ada95ef7c7213a78abf5909e.1685029183.git.zimon.toutoune@gmail.com
* guix/import/go.scm (git-checkout-hash): Catch Git error.

Reported-by: Felix Lechner <felix.lechner@lease-up.com>.
---
guix/import/go.scm | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 1943869162..c6258296f6 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,7 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
-;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -36,9 +36,11 @@ (define-module (guix import go)
#:use-module (guix memoization)
#:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
- #:autoload (guix base32) (bytevector->nix-base32-string)
+ #:autoload (guix base32) (bytevector->nix-base32-string nix-base32-string->bytevector)
#:autoload (guix build utils) (mkdir-p)
+ #:autoload (guix ui) (warning)
#:autoload (gcrypt hash) (hash-algorithm sha256)
+ #:autoload (git structs) (git-error-message)
#:use-module (ice-9 format)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
@@ -507,12 +509,18 @@ (define* (git-checkout-hash url reference algorithm)
;; subsequent "guix import" invocations.
(mkdir-p cache)
(chmod cache #o700)
- (let-values (((checkout commit _)
- (parameterize ((%repository-cache-directory cache))
- (update-cached-checkout url
- #:ref
- `(tag-or-commit . ,reference)))))
- (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (catch 'git-error
+ (lambda _
+ (let-values (((checkout commit _)
+ (parameterize ((%repository-cache-directory cache))
+ (update-cached-checkout url
+ #:ref
+ `(tag-or-commit . ,reference)))))
+ (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (lambda (key error . rest)
+ (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
+ (nix-base32-string->bytevector
+ "0000000000000000000000000000000000000000000000000000"))))
(define (vcs->origin vcs-type vcs-repo-url version)
"Generate the `origin' block of a package depending on what type of source
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 17:41
[PATCH v2 1/3] guix: import: go: Sort hint about available versions.
(address . 63647@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
a520128590d4b67af4c2b22ae167590b99e449dd.1685029183.git.zimon.toutoune@gmail.com
* guix/import/go.scm (validate-version): Sort available versions.
---
guix/import/go.scm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

Toggle diff (28 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1eb..1943869162 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -34,6 +34,7 @@ (define-module (guix import go)
#:use-module (guix packages)
#:use-module (guix http-client)
#:use-module (guix memoization)
+ #:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
#:autoload (guix base32) (bytevector->nix-base32-string)
#:autoload (guix build utils) (mkdir-p)
@@ -588,8 +589,9 @@ (define (validate-version version available-versions module-path)
(condition (&fix-hint
(hint (format #f (G_ "Pick one of the following \
available versions:~{ ~a~}.")
- (map strip-v-prefix
- available-versions)))))))))
+ (sort (map strip-v-prefix
+ available-versions)
+ version>?)))))))))
(define* (go-module->guix-package module-path #:key
(goproxy "https://proxy.golang.org")

base-commit: 99fc7e5dd3c9c92f9b6c2e51db9dc5b0481a56b2
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 17:41
[PATCH v2 3/3] guix: import: go: Use correct tag for go module in subdirectory.
(address . 63647@debbugs.gnu.org)
27b0652df204a77a5b38554a95128fb54ca64dd9.1685029183.git.zimon.toutoune@gmail.com
From: Stephen Paul Weber <singpolyma@singpolyma.net>

* guix/import/go.scm (vcs->origin): New argument module-path-subdirectory.
Use it to get Git tag as described by https://go.dev/ref/mod.

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/go.scm | 34 +++++++++++++++++++++++-----------
1 file changed, 23 insertions(+), 11 deletions(-)

Toggle diff (86 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index c6258296f6..f02c098f1c 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,6 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2022 Stephen Paul Weber <singpolyma@singpolyma.net>
;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
@@ -522,28 +523,34 @@ (define* (git-checkout-hash url reference algorithm)
(nix-base32-string->bytevector
"0000000000000000000000000000000000000000000000000000"))))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define (vcs->origin vcs-type vcs-repo-url module-path-subdirectory version)
"Generate the `origin' block of a package depending on what type of source
control system is being used."
(case vcs-type
((git)
- (let ((plain-version? (string=? version (go-version->git-ref version)))
- (v-prefixed? (string-prefix? "v" version)))
+ (let* ((plain-version? (string=? version (go-version->git-ref version)))
+ (v-prefixed? (string-prefix? "v" version))
+ ;; A module in a subdirectory has a tag prefixed by this
+ ;; subdirectory. See <https://go.dev/ref/mod>.
+ (tag-prefix (if (string=? module-path-subdirectory "") ;
+ ""
+ (string-append
+ (substring module-path-subdirectory 1) "/")))
+ (git-commit (if plain-version?
+ (string-append tag-prefix version)
+ (go-version->git-ref version))))
`(origin
(method git-fetch)
(uri (git-reference
(url ,vcs-repo-url)
- ;; This is done because the version field of the package,
- ;; which the generated quoted expression refers to, has been
- ;; stripped of any 'v' prefixed.
(commit ,(if (and plain-version? v-prefixed?)
- '(string-append "v" version)
+ `(string-append ,tag-prefix "v" version)
'(go-version->git-ref version)))))
(file-name (git-file-name name version))
(sha256
(base32
,(bytevector->nix-base32-string
- (git-checkout-hash vcs-repo-url (go-version->git-ref version)
+ (git-checkout-hash vcs-repo-url git-commit
(hash-algorithm sha256))))))))
((hg)
`(origin
@@ -621,12 +628,17 @@ (define* (go-module->guix-package module-path #:key
dependencies+versions
(map car dependencies+versions)))
(module-path-sans-suffix
- (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path)))
+ (if (string-prefix? "gopkg.in" module-path)
+ module-path
+ (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path))))
(guix-name (go-module->guix-package-name module-path))
- (root-module-path (module-path->repository-root module-path))
+ (root-module-path (module-path->repository-root module-path-sans-suffix))
;; The VCS type and URL are not included in goproxy information. For
;; this we need to fetch it from the official module page.
(meta-data (fetch-module-meta-data root-module-path))
+ (module-path-subdirectory
+ (substring module-path-sans-suffix
+ (string-length (module-meta-import-prefix meta-data))))
(vcs-type (module-meta-vcs meta-data))
(vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
(synopsis (go-package-synopsis module-path))
@@ -637,7 +649,7 @@ (define* (go-module->guix-package module-path #:key
(name ,guix-name)
(version ,(strip-v-prefix version*))
(source
- ,(vcs->origin vcs-type vcs-repo-url version*))
+ ,(vcs->origin vcs-type vcs-repo-url module-path-subdirectory version*))
(build-system go-build-system)
(arguments
'(#:import-path ,module-path
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 18:07
[PATCH v3 3/3] guix: import: go: Use correct tag for go module in subdirectory.
(address . 63647@debbugs.gnu.org)
ef9e802b1c3465665b96ec171cf2f7e9bf8f51e6.1685030838.git.zimon.toutoune@gmail.com
From: Stephen Paul Weber <singpolyma@singpolyma.net>

* guix/import/go.scm (vcs->origin): New argument module-path-subdirectory.
Use it to get Git tag as described by https://go.dev/ref/mod.

Co-authored-by: Simon Tournier <zimon.toutoune@gmail.com>
Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/go.scm | 35 ++++++++++++++++++++++++-----------
1 file changed, 24 insertions(+), 11 deletions(-)

Toggle diff (87 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index c6258296f6..c2f9065d94 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,6 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2022 Stephen Paul Weber <singpolyma@singpolyma.net>
;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
@@ -522,28 +523,35 @@ (define* (git-checkout-hash url reference algorithm)
(nix-base32-string->bytevector
"0000000000000000000000000000000000000000000000000000"))))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define (vcs->origin vcs-type vcs-repo-url module-path-subdirectory version)
"Generate the `origin' block of a package depending on what type of source
control system is being used."
(case vcs-type
((git)
- (let ((plain-version? (string=? version (go-version->git-ref version)))
- (v-prefixed? (string-prefix? "v" version)))
+ (let* ((plain-version? (string=? version (go-version->git-ref version)))
+ (v-prefixed? (string-prefix? "v" version))
+ ;; A module in a subdirectory has a tag prefixed by this
+ ;; subdirectory. See <https://go.dev/ref/mod>.
+ (tag-prefix (if (string=? module-path-subdirectory "") ;
+ ""
+ (string-append
+ (substring module-path-subdirectory 1) "/")))
+ (prefix (string-append tag-prefix "v"))
+ (git-commit (if plain-version?
+ (string-append tag-prefix version)
+ (go-version->git-ref version))))
`(origin
(method git-fetch)
(uri (git-reference
(url ,vcs-repo-url)
- ;; This is done because the version field of the package,
- ;; which the generated quoted expression refers to, has been
- ;; stripped of any 'v' prefixed.
(commit ,(if (and plain-version? v-prefixed?)
- '(string-append "v" version)
+ `(string-append ,prefix version)
'(go-version->git-ref version)))))
(file-name (git-file-name name version))
(sha256
(base32
,(bytevector->nix-base32-string
- (git-checkout-hash vcs-repo-url (go-version->git-ref version)
+ (git-checkout-hash vcs-repo-url git-commit
(hash-algorithm sha256))))))))
((hg)
`(origin
@@ -621,12 +629,17 @@ (define* (go-module->guix-package module-path #:key
dependencies+versions
(map car dependencies+versions)))
(module-path-sans-suffix
- (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path)))
+ (if (string-prefix? "gopkg.in" module-path)
+ module-path
+ (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path))))
(guix-name (go-module->guix-package-name module-path))
- (root-module-path (module-path->repository-root module-path))
+ (root-module-path (module-path->repository-root module-path-sans-suffix))
;; The VCS type and URL are not included in goproxy information. For
;; this we need to fetch it from the official module page.
(meta-data (fetch-module-meta-data root-module-path))
+ (module-path-subdirectory
+ (substring module-path-sans-suffix
+ (string-length (module-meta-import-prefix meta-data))))
(vcs-type (module-meta-vcs meta-data))
(vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
(synopsis (go-package-synopsis module-path))
@@ -637,7 +650,7 @@ (define* (go-module->guix-package module-path #:key
(name ,guix-name)
(version ,(strip-v-prefix version*))
(source
- ,(vcs->origin vcs-type vcs-repo-url version*))
+ ,(vcs->origin vcs-type vcs-repo-url module-path-subdirectory version*))
(build-system go-build-system)
(arguments
'(#:import-path ,module-path
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 18:07
[PATCH v3 2/3] guix: import: go: Catch Git error.
(address . 63647@debbugs.gnu.org)
f699a25c43805136ada95ef7c7213a78abf5909e.1685030838.git.zimon.toutoune@gmail.com
* guix/import/go.scm (git-checkout-hash): Catch Git error.

Reported-by: Felix Lechner <felix.lechner@lease-up.com>.
---
guix/import/go.scm | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 1943869162..c6258296f6 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,7 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
-;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -36,9 +36,11 @@ (define-module (guix import go)
#:use-module (guix memoization)
#:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
- #:autoload (guix base32) (bytevector->nix-base32-string)
+ #:autoload (guix base32) (bytevector->nix-base32-string nix-base32-string->bytevector)
#:autoload (guix build utils) (mkdir-p)
+ #:autoload (guix ui) (warning)
#:autoload (gcrypt hash) (hash-algorithm sha256)
+ #:autoload (git structs) (git-error-message)
#:use-module (ice-9 format)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
@@ -507,12 +509,18 @@ (define* (git-checkout-hash url reference algorithm)
;; subsequent "guix import" invocations.
(mkdir-p cache)
(chmod cache #o700)
- (let-values (((checkout commit _)
- (parameterize ((%repository-cache-directory cache))
- (update-cached-checkout url
- #:ref
- `(tag-or-commit . ,reference)))))
- (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (catch 'git-error
+ (lambda _
+ (let-values (((checkout commit _)
+ (parameterize ((%repository-cache-directory cache))
+ (update-cached-checkout url
+ #:ref
+ `(tag-or-commit . ,reference)))))
+ (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (lambda (key error . rest)
+ (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
+ (nix-base32-string->bytevector
+ "0000000000000000000000000000000000000000000000000000"))))
(define (vcs->origin vcs-type vcs-repo-url version)
"Generate the `origin' block of a package depending on what type of source
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 18:07
[PATCH v3 1/3] guix: import: go: Sort hint about available versions.
(address . 63647@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
a520128590d4b67af4c2b22ae167590b99e449dd.1685030838.git.zimon.toutoune@gmail.com
* guix/import/go.scm (validate-version): Sort available versions.
---
guix/import/go.scm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

Toggle diff (28 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1eb..1943869162 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -34,6 +34,7 @@ (define-module (guix import go)
#:use-module (guix packages)
#:use-module (guix http-client)
#:use-module (guix memoization)
+ #:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
#:autoload (guix base32) (bytevector->nix-base32-string)
#:autoload (guix build utils) (mkdir-p)
@@ -588,8 +589,9 @@ (define (validate-version version available-versions module-path)
(condition (&fix-hint
(hint (format #f (G_ "Pick one of the following \
available versions:~{ ~a~}.")
- (map strip-v-prefix
- available-versions)))))))))
+ (sort (map strip-v-prefix
+ available-versions)
+ version>?)))))))))
(define* (go-module->guix-package module-path #:key
(goproxy "https://proxy.golang.org")

base-commit: 99fc7e5dd3c9c92f9b6c2e51db9dc5b0481a56b2
--
2.38.1
L
L
Ludovic Courtès wrote on 5 Jun 2023 14:43
Re: bug#63647: [PATCH 0/2] Fix annoyance with "guix import go"
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 63647@debbugs.gnu.org)
87sfb6hyey.fsf_-_@gnu.org
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (2 lines)
> * guix/import/go.scm (validate-version): Sort available versions.

LGTM!

Toggle quote (8 lines)
> From: Stephen Paul Weber <singpolyma@singpolyma.net>
>
> * guix/import/go.scm (vcs->origin): New argument module-path-subdirectory.
> Use it to get Git tag as described by <https://go.dev/ref/mod>.
>
> Co-authored-by: Simon Tournier <zimon.toutoune@gmail.com>
> Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>

[...]

Toggle quote (5 lines)
> -(define (vcs->origin vcs-type vcs-repo-url version)
> +(define (vcs->origin vcs-type vcs-repo-url module-path-subdirectory version)
> "Generate the `origin' block of a package depending on what type of source
> control system is being used."

Maybe just ‘subdirectory’? (In general ‘path’ is used for search paths,
not for file names, but I see this module already used the term with
that meaning.)

Also please mention it in the docstring.

Should there be a test, or is it too tricky?

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 5 Jun 2023 14:45
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87o7luhybt.fsf_-_@gnu.org
Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (2 lines)
> * guix/import/go.scm (git-checkout-hash): Catch Git error.

[...]

Toggle quote (13 lines)
> + (catch 'git-error
> + (lambda _
> + (let-values (((checkout commit _)
> + (parameterize ((%repository-cache-directory cache))
> + (update-cached-checkout url
> + #:ref
> + `(tag-or-commit . ,reference)))))
> + (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
> + (lambda (key error . rest)
> + (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
> + (nix-base32-string->bytevector
> + "0000000000000000000000000000000000000000000000000000"))))

I’d rather let the exception through. How about adding
‘with-git-error-handling’ at the UI level, in (guix scripts import go)?

Thanks,
Ludo’.
S
S
Simon Tournier wrote on 6 Jun 2023 08:21
Re: [bug#63647] [PATCH 0/2] Fix annoyance with "guix import go"
(name . Ludovic Courtès)(address . ludo@gnu.org)
86cz29nm9n.fsf@gmail.com
Hi Ludo,

On Mon, 05 Jun 2023 at 14:45, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (22 lines)
> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>
>> * guix/import/go.scm (git-checkout-hash): Catch Git error.
>
> [...]
>
>> + (catch 'git-error
>> + (lambda _
>> + (let-values (((checkout commit _)
>> + (parameterize ((%repository-cache-directory cache))
>> + (update-cached-checkout url
>> + #:ref
>> + `(tag-or-commit . ,reference)))))
>> + (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
>> + (lambda (key error . rest)
>> + (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
>> + (nix-base32-string->bytevector
>> + "0000000000000000000000000000000000000000000000000000"))))
>
> I’d rather let the exception through. How about adding
> ‘with-git-error-handling’ at the UI level, in (guix scripts import go)?

What do you mean by “let the exception through”? It seems better to be
non-blocking and thus catch the exception then raise a meaningful
warning; it’s required when running with the option recursive.

Well, maybe an improvement could be in the addition of some
’report-git-warning’ and/or ’with-git-error-handling*’, in (guix git);
hum, I do not know.

Last, considering that the module (guix import go) already contains 4 UI
messages (G_), trying to move this warning about Git to (guix scripts
import go) will add some complexity – re-raise the exception 2 or 3
times, IIUC – and thus it will not change much about the UI, IMHO. I
mean, such move should be for all the messages or nothing.

Cheers,
simon
L
L
Ludovic Courtès wrote on 6 Jun 2023 17:42
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87zg5c61iv.fsf@gnu.org
Hello!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (27 lines)
> On Mon, 05 Jun 2023 at 14:45, Ludovic Courtès <ludo@gnu.org> wrote:
>> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>>
>>> * guix/import/go.scm (git-checkout-hash): Catch Git error.
>>
>> [...]
>>
>>> + (catch 'git-error
>>> + (lambda _
>>> + (let-values (((checkout commit _)
>>> + (parameterize ((%repository-cache-directory cache))
>>> + (update-cached-checkout url
>>> + #:ref
>>> + `(tag-or-commit . ,reference)))))
>>> + (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
>>> + (lambda (key error . rest)
>>> + (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
>>> + (nix-base32-string->bytevector
>>> + "0000000000000000000000000000000000000000000000000000"))))
>>
>> I’d rather let the exception through. How about adding
>> ‘with-git-error-handling’ at the UI level, in (guix scripts import go)?
>
> What do you mean by “let the exception through”? It seems better to be
> non-blocking and thus catch the exception then raise a meaningful
> warning; it’s required when running with the option recursive.

I thought it would be more appropriate to error out as soon as we have a
Git problem, rather than print a warning and emit an incorrect hash.

I understand that, when using ‘--recursive’, this means aborting the
whole process without producing anything. But maybe that’s better that
producing an incorrect (origin …) form?

Now, I don’t use ‘guix import go -r’ so it’s possible that I don’t
understand the scenario being considered here!

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 14 Jun 2023 23:09
Re: bug#63631: [PATCH] import: go: Handle subpackage versioning correctly.
(name . Timo Wilken)(address . guix@twilken.net)
87pm5xrbsg.fsf@gnu.org
Hi Timo,

Timo Wilken <guix@twilken.net> skribis:

Toggle quote (18 lines)
> Some Go source repositories (notably the Google Cloud SDK) contain multiple
> submodules and use a `refs/tags/<submodule>/<version>' tagging scheme.
>
> Fixes <https://bugs.gnu.org/54097>.
>
> * guix/import/go.scm (vcs->origin): Accept a module-path-suffix.
> (go-module->guix-package): Use the new parameter.
> ---
> Here's a patch that fixes the reported issue (bug#54097) for me. I've only
> tested this on the github.com/googleapis/google-cloud-go/compute package so
> far, though it seems to work there. Perhaps others have more testcases?
>
> I don't know enough about Go tooling to use it, so I've just patched the Guile
> logic of the importer. (I don't write Go, I just want to package stuff written
> in it.) In terms of performance, at least the repo contents are apparently
> cached by the first `git-checkout-hash' call, even if it fails, so the second
> call doesn't have to redownload them.

What you propose looks similar to part of the work Simon Tournier

What would you suggest? Simon?

Thanks for the patch, Timo!

Ludo’.
T
T
Timo Wilken wrote on 17 Jun 2023 17:12
(name . Ludovic Courtès)(address . ludo@gnu.org)
CTF06XBYWPT0.1MV6QA1B2OB98@lap.twilken.net
Hi Ludo', (hi everyone,)

On Wed Jun 14, 2023 at 11:09 PM CEST, Ludovic Courtès wrote:
Toggle quote (11 lines)
> Timo Wilken <guix@twilken.net> skribis:
> > Here's a patch that fixes the reported issue (bug#54097) for me. I've only
> > tested this on the github.com/googleapis/google-cloud-go/compute package so
> > far, though it seems to work there. Perhaps others have more testcases?
> >
> > I don't know enough about Go tooling to use it, so I've just patched the Guile
> > logic of the importer. (I don't write Go, I just want to package stuff written
> > in it.) In terms of performance, at least the repo contents are apparently
> > cached by the first `git-checkout-hash' call, even if it fails, so the second
> > call doesn't have to redownload them.

I've been testing my patch further this weekend, and I have a couple more
patches in the pipeline; I suppose I ought to clean those up and submit them.

In particular, I've got fixes for the following queued up locally:

1. Finding the `module-path-subdir' needs another case for e.g.
cloud.google.com/go/*.

2. My patch sometimes generates an unnecessary `go-version->git-ref' call.

3. Go versions need to be parsed from go.mod, since some packages require a
newer Go compiler than our default. This I've got a patch for, but this Go
version also ought to propagate up the dependency tree. I haven't found an
easy way to do that, since the importer seems to generate top-level
packages first, before descending the dep tree...

4. `fetch-module-meta-data' ought to ignore 4xx HTTP errors to follow the
spec; gonum.org/v1/gonum specifically depends on this behaviour.

I've been trying to recursively import github.com/matrix-org/dendrite, which
has a particularly large and hairy dependency tree. While I can now import it
without crashes, I can't build it from the imported package definitions yet --
mainly because of lots of dependency cycles in the generated packages, but
there may be more issues hidden beneath that.

Still, I can recommend it as a test of everyone's importer patches, since
it'll find a lot of edge cases in importing alone!

Toggle quote (3 lines)
> What you propose looks similar to part of the work Simon Tournier
> submitted at <https://issues.guix.gnu.org/63647>.

It seems lots of people have been working on the same problem -- in addition
to Simon's patches, I found a patch submitted by Elbek (issues 64035 & 64036;
Cc'd). I also forgot about the issue I submitted months ago (63001)...

Toggle quote (2 lines)
> What would you suggest? Simon?

Here's a brief comparison between Simon's patches and mine -- Simon's seem to
contain fixes for a couple more things than mine currently does:

1. Simon sorts available versions in an error message; this can presumably be
merged independently since it doesn't conflict with other patches.

2. Simon always prepends a "SUBDIR/" prefix to the tag if found, whereas I try
to find the plain "vX" tag first, then fall back to "SUBDIR/vX". Judging by
https://go.dev/ref/mod#vcs-version,Simon's approach seems more correct.
I'll change my implementation to match and try it out.

3. For detecting the `module-path-subdirectory' in Simon's patches: that's the
same approach I used initially, but I found I have to try `(substring
module-path (string-length import-prefix))' first (to handle e.g.
cloud.google.com/go/*). This is one of the things I haven't submitted
yet...

Toggle quote (2 lines)
> Thanks for the patch, Timo!

Thanks for your work in sorting through all of this, Ludo'!

Cheers,
Timo
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEU+w8BoVog92SNVvCL8eFBGgfabAFAmSNzaUACgkQL8eFBGgf
abDEaA//Y6k/Uckhvq+wllES0TjACxVG+2bY2E/wLZNoQShXF5Tn3ZQ9o2HOU7Gr
p8er/t/JkPLYm9w2MidUtvTms6qxp8H/YIZ12gzZ+GP/eoKa/iiqt42+KZvLS/Is
B2JWEcaJLFYy0w9nuubeuQpgcWEmxyk7/zb++GovD0tfYgczII3N1Z5fhRmeAj0N
FbKqfkBocxruYvBfcI5KhMV6Q1TjCwuAH+j2iwNmsiSnm7NTKbJxfCyO9+p5UF9w
O/em01p9J+Cg6EEFunwoi1pjoEAkVZCRtgaiP1SMgapq74UKKagvsWqMoW+JHpDf
uuiaVn1sLuTt+u9KGamYqaCj1NRcCfN7SjPxS7oTM4hsytVDmUB/eIHD6t3HntCP
Ev43fJKJwLoybQV0ptajnJggEEaEY1cegQG683jsD0ycrToh4YEhLe5fQsGhaLMA
7MSLjjf0K73BoFyZi+wYyfDPYqXRQmpb3Z6cKJPlF3v4IF13UtSxP0yBRdz930aI
1b1Q5khLQzQBI0fvxMNcZGPeA2RjPyNsXE8pKn9zEkIunzX9VYN8az++EjX928Ox
mmgjEUsLWMPCSMwGcEoZye6R5wJUG34LEkoQTbySf+314hlELPuGcLCatFgZZbXq
1Hn1upt13XYRYVCX/SXIpBJL9gdNbi2sSg8FVm4Aq+sQQqZzAKs=
=fjJn
-----END PGP SIGNATURE-----


S
S
Simon Tournier wrote on 16 Aug 2023 17:59
Re: bug#63001: bug#63631: [PATCH] import: go: Handle subpackage versioning correctly.
875y5ff05i.fsf@gmail.com
Hi Timo,

On Sat, 17 Jun 2023 at 17:12, "Timo Wilken" <guix@twilken.net> wrote:

Toggle quote (19 lines)
>> What would you suggest? Simon?
>
> Here's a brief comparison between Simon's patches and mine -- Simon's seem to
> contain fixes for a couple more things than mine currently does:
>
> 1. Simon sorts available versions in an error message; this can presumably be
> merged independently since it doesn't conflict with other patches.
>
> 2. Simon always prepends a "SUBDIR/" prefix to the tag if found, whereas I try
> to find the plain "vX" tag first, then fall back to "SUBDIR/vX". Judging by
> https://go.dev/ref/mod#vcs-version, Simon's approach seems more correct.
> I'll change my implementation to match and try it out.
>
> 3. For detecting the `module-path-subdirectory' in Simon's patches: that's the
> same approach I used initially, but I found I have to try `(substring
> module-path (string-length import-prefix))' first (to handle e.g.
> cloud.google.com/go/*). This is one of the things I haven't submitted
> yet...

Sorry if I have missed some patches or overlooked something. Do you
plan to send another patch series handling all?


Cheers,
simon
C
C
Christopher Baines wrote on 17 Oct 2023 10:31
tag 63647 moreinfo
(address . control@debbugs.gnu.org)
87h6mpy7jp.fsf@cbaines.net
tags 63647 + moreinfo
quit
?