[PATCH 0/2] Fix annoyance with "guix import go"

  • Open
  • quality assurance status badge
Details
4 participants
  • Timo Wilken
  • Ludovic Courtès
  • Christopher Baines
  • Simon Tournier
Owner
unassigned
Submitted by
Simon Tournier
Severity
normal
S
S
Simon Tournier wrote on 22 May 2023 18:04
(address . guix-patches@gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
cover.1684771100.git.zimon.toutoune@gmail.com
Hi,

As reported in [1], the Go importer returns backtrace. This patch fixes
that. The first patch is about sorting the versions. The second is about
warn but not exit. It allows to have a first draft when applying recursive
importer.

Compare (first patch),

Toggle snippet (15 lines)
$ guix import go go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@0.41
guix import: error: version v0.41 of go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc is not available

hint: Pick one of the following available versions: 0.26.0 0.36.0 0.15.0 0.41.0-rc.1 0.40.0 0.37.0 0.33.0 0.20.0 0.24.0 0.23.0 0.41.0 0.26.1 0.41.1 0.22.0
0.18.0 0.21.0 0.12.0 0.36.4 0.19.0 0.25.0 0.27.0 0.36.2 0.34.0 0.14.0 0.36.3 0.31.0 0.15.1 0.36.1 0.41.0-rc.2 0.32.0 0.28.0 0.35.0 0.42.0-rc.1 0.13.0
0.17.0 0.29.0 0.30.0 0.39.0 0.38.0 0.16.0.

$ ./pre-inst-env guix import go go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@0.41
guix import: error: version v0.41 of go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc is not available

hint: Pick one of the following available versions: 0.42.0-rc.1 0.41.1 0.41.0-rc.2 0.41.0-rc.1 0.41.0 0.40.0 0.39.0 0.38.0 0.37.0 0.36.4 0.36.3 0.36.2 0.36.1
0.36.0 0.35.0 0.34.0 0.33.0 0.32.0 0.31.0 0.30.0 0.29.0 0.28.0 0.27.0 0.26.1 0.26.0 0.25.0 0.24.0 0.23.0 0.22.0 0.21.0 0.20.0 0.19.0 0.18.0 0.17.0
0.16.0 0.15.1 0.15.0 0.14.0 0.13.0 0.12.0.

Then instead of,

Toggle snippet (33 lines)
$ guix import go go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
Backtrace:
14 (primitive-load "/home/simon/.config/guix/current/bin/guix")
In guix/ui.scm:
2300:7 13 (run-guix . _)
2263:10 12 (run-guix-command _ . _)
In guix/scripts/import.scm:
89:11 11 (guix-import . _)
In ice-9/boot-9.scm:
1752:10 10 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In guix/scripts/import/go.scm:
116:29 9 (_)
In ice-9/exceptions.scm:
406:15 8 (go-module->guix-package* . _)
In ice-9/boot-9.scm:
1752:10 7 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In guix/import/go.scm:
511:19 6 (go-module->guix-package "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" #:goproxy _ #:version _ # _)
In guix/git.scm:
291:4 5 (update-cached-checkout _ #:ref _ #:recursive? _ #:check-out? _ #:starting-commit _ #:log-port _ #:cache-directory _)
277:19 4 (resolve _)
In git/reference.scm:
60:8 3 (_ _ _)
In git/bindings.scm:
77:2 2 (raise-git-error _)
In ice-9/boot-9.scm:
1685:16 1 (raise-exception _ #:continuable? _)
1683:16 0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1683:16: In procedure raise-exception:
Git error: reference 'refs/tags/v0.41.1' not found

Now it returns (second patch),

Toggle snippet (49 lines)
$ ./pre-inst-env guix import go go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
guix import: warning: Git error: reference 'refs/tags/v0.41.1' not found in https://github.com/open-telemetry/opentelemetry-go-contrib
(define-public go-go-opentelemetry-io-contrib-instrumentation-google-golang-org-grpc-otelgrpc
(package
(name
"go-go-opentelemetry-io-contrib-instrumentation-google-golang-org-grpc-otelgrpc")
(version "0.41.1")
(source (origin
(method git-fetch)
(uri (git-reference
(url
"https://github.com/open-telemetry/opentelemetry-go-contrib")
(commit (string-append "v" version))))
(file-name (git-file-name name version))
(sha256
(base32
"0000000000000000000000000000000000000000000000000000"))))
(build-system go-build-system)
(arguments
'(#:import-path
"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"))
(propagated-inputs `(("go-gopkg-in-yaml-v3" ,go-gopkg-in-yaml-v3)
("go-google-golang-org-genproto" ,go-google-golang-org-genproto)
("go-google-golang-org-appengine" ,go-google-golang-org-appengine)
("go-golang-org-x-text" ,go-golang-org-x-text)
("go-golang-org-x-sys" ,go-golang-org-x-sys)
("go-golang-org-x-oauth2" ,go-golang-org-x-oauth2)
("go-golang-org-x-net" ,go-golang-org-x-net)
("go-github-com-pmezard-go-difflib" ,go-github-com-pmezard-go-difflib)
("go-github-com-golang-protobuf" ,go-github-com-golang-protobuf)
("go-github-com-go-logr-stdr" ,go-github-com-go-logr-stdr)
("go-github-com-go-logr-logr" ,go-github-com-go-logr-logr)
("go-github-com-davecgh-go-spew" ,go-github-com-davecgh-go-spew)
("go-cloud-google-com-go-compute-metadata" ,go-cloud-google-com-go-compute-metadata)
("go-cloud-google-com-go-compute" ,go-cloud-google-com-go-compute)
("go-google-golang-org-protobuf" ,go-google-golang-org-protobuf)
("go-google-golang-org-grpc" ,go-google-golang-org-grpc)
("go-go-opentelemetry-io-otel-trace" ,go-go-opentelemetry-io-otel-trace)
("go-go-opentelemetry-io-otel-metric" ,go-go-opentelemetry-io-otel-metric)
("go-go-opentelemetry-io-otel" ,go-go-opentelemetry-io-otel)
("go-github-com-stretchr-testify" ,go-github-com-stretchr-testify)))
(home-page
"https://go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc")
(synopsis #f)
(description #f)
(license license:asl2.0)))[env]


Cheers,
simon


Simon Tournier (2):
guix: import: go: Sort hint about available versions.
guix: import: go: Catch Git error.

guix/import/go.scm | 30 ++++++++++++++++++++----------
1 file changed, 20 insertions(+), 10 deletions(-)


base-commit: 849286ba66c96534bddc04df1a47d5692cbc977e
--
2.38.1
S
S
Simon Tournier wrote on 22 May 2023 18:08
[PATCH 1/2] guix: import: go: Sort hint about available versions.
(address . 63647@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
5cb8f8989fccc4580d10b7e39633f8880c838236.1684771100.git.zimon.toutoune@gmail.com
* guix/import/go.scm (validate-version): Sort available versions.
---
guix/import/go.scm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

Toggle diff (26 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1eb..1943869162 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -34,6 +34,7 @@ (define-module (guix import go)
#:use-module (guix packages)
#:use-module (guix http-client)
#:use-module (guix memoization)
+ #:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
#:autoload (guix base32) (bytevector->nix-base32-string)
#:autoload (guix build utils) (mkdir-p)
@@ -588,8 +589,9 @@ (define (validate-version version available-versions module-path)
(condition (&fix-hint
(hint (format #f (G_ "Pick one of the following \
available versions:~{ ~a~}.")
- (map strip-v-prefix
- available-versions)))))))))
+ (sort (map strip-v-prefix
+ available-versions)
+ version>?)))))))))
(define* (go-module->guix-package module-path #:key
(goproxy "https://proxy.golang.org")
--
2.38.1
S
S
Simon Tournier wrote on 22 May 2023 18:08
[PATCH 2/2] guix: import: go: Catch Git error.
(address . 63647@debbugs.gnu.org)
307ae354ce980b521ae5878dd0710bd89f1ad0c5.1684771100.git.zimon.toutoune@gmail.com
* guix/import/go.scm (git-checkout-hash): Catch Git error.

Reported-by: Felix Lechner <felix.lechner@lease-up.com>.
---
guix/import/go.scm | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 1943869162..c6258296f6 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,7 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
-;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -36,9 +36,11 @@ (define-module (guix import go)
#:use-module (guix memoization)
#:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
- #:autoload (guix base32) (bytevector->nix-base32-string)
+ #:autoload (guix base32) (bytevector->nix-base32-string nix-base32-string->bytevector)
#:autoload (guix build utils) (mkdir-p)
+ #:autoload (guix ui) (warning)
#:autoload (gcrypt hash) (hash-algorithm sha256)
+ #:autoload (git structs) (git-error-message)
#:use-module (ice-9 format)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
@@ -507,12 +509,18 @@ (define* (git-checkout-hash url reference algorithm)
;; subsequent "guix import" invocations.
(mkdir-p cache)
(chmod cache #o700)
- (let-values (((checkout commit _)
- (parameterize ((%repository-cache-directory cache))
- (update-cached-checkout url
- #:ref
- `(tag-or-commit . ,reference)))))
- (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (catch 'git-error
+ (lambda _
+ (let-values (((checkout commit _)
+ (parameterize ((%repository-cache-directory cache))
+ (update-cached-checkout url
+ #:ref
+ `(tag-or-commit . ,reference)))))
+ (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (lambda (key error . rest)
+ (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
+ (nix-base32-string->bytevector
+ "0000000000000000000000000000000000000000000000000000"))))
(define (vcs->origin vcs-type vcs-repo-url version)
"Generate the `origin' block of a package depending on what type of source
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 17:41
[PATCH v2 2/3] guix: import: go: Catch Git error.
(address . 63647@debbugs.gnu.org)
f699a25c43805136ada95ef7c7213a78abf5909e.1685029183.git.zimon.toutoune@gmail.com
* guix/import/go.scm (git-checkout-hash): Catch Git error.

Reported-by: Felix Lechner <felix.lechner@lease-up.com>.
---
guix/import/go.scm | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 1943869162..c6258296f6 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,7 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
-;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -36,9 +36,11 @@ (define-module (guix import go)
#:use-module (guix memoization)
#:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
- #:autoload (guix base32) (bytevector->nix-base32-string)
+ #:autoload (guix base32) (bytevector->nix-base32-string nix-base32-string->bytevector)
#:autoload (guix build utils) (mkdir-p)
+ #:autoload (guix ui) (warning)
#:autoload (gcrypt hash) (hash-algorithm sha256)
+ #:autoload (git structs) (git-error-message)
#:use-module (ice-9 format)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
@@ -507,12 +509,18 @@ (define* (git-checkout-hash url reference algorithm)
;; subsequent "guix import" invocations.
(mkdir-p cache)
(chmod cache #o700)
- (let-values (((checkout commit _)
- (parameterize ((%repository-cache-directory cache))
- (update-cached-checkout url
- #:ref
- `(tag-or-commit . ,reference)))))
- (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (catch 'git-error
+ (lambda _
+ (let-values (((checkout commit _)
+ (parameterize ((%repository-cache-directory cache))
+ (update-cached-checkout url
+ #:ref
+ `(tag-or-commit . ,reference)))))
+ (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (lambda (key error . rest)
+ (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
+ (nix-base32-string->bytevector
+ "0000000000000000000000000000000000000000000000000000"))))
(define (vcs->origin vcs-type vcs-repo-url version)
"Generate the `origin' block of a package depending on what type of source
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 17:41
[PATCH v2 1/3] guix: import: go: Sort hint about available versions.
(address . 63647@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
a520128590d4b67af4c2b22ae167590b99e449dd.1685029183.git.zimon.toutoune@gmail.com
* guix/import/go.scm (validate-version): Sort available versions.
---
guix/import/go.scm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

Toggle diff (28 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1eb..1943869162 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -34,6 +34,7 @@ (define-module (guix import go)
#:use-module (guix packages)
#:use-module (guix http-client)
#:use-module (guix memoization)
+ #:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
#:autoload (guix base32) (bytevector->nix-base32-string)
#:autoload (guix build utils) (mkdir-p)
@@ -588,8 +589,9 @@ (define (validate-version version available-versions module-path)
(condition (&fix-hint
(hint (format #f (G_ "Pick one of the following \
available versions:~{ ~a~}.")
- (map strip-v-prefix
- available-versions)))))))))
+ (sort (map strip-v-prefix
+ available-versions)
+ version>?)))))))))
(define* (go-module->guix-package module-path #:key
(goproxy "https://proxy.golang.org")

base-commit: 99fc7e5dd3c9c92f9b6c2e51db9dc5b0481a56b2
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 17:41
[PATCH v2 3/3] guix: import: go: Use correct tag for go module in subdirectory.
(address . 63647@debbugs.gnu.org)
27b0652df204a77a5b38554a95128fb54ca64dd9.1685029183.git.zimon.toutoune@gmail.com
From: Stephen Paul Weber <singpolyma@singpolyma.net>

* guix/import/go.scm (vcs->origin): New argument module-path-subdirectory.
Use it to get Git tag as described by https://go.dev/ref/mod.

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/go.scm | 34 +++++++++++++++++++++++-----------
1 file changed, 23 insertions(+), 11 deletions(-)

Toggle diff (86 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index c6258296f6..f02c098f1c 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,6 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2022 Stephen Paul Weber <singpolyma@singpolyma.net>
;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
@@ -522,28 +523,34 @@ (define* (git-checkout-hash url reference algorithm)
(nix-base32-string->bytevector
"0000000000000000000000000000000000000000000000000000"))))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define (vcs->origin vcs-type vcs-repo-url module-path-subdirectory version)
"Generate the `origin' block of a package depending on what type of source
control system is being used."
(case vcs-type
((git)
- (let ((plain-version? (string=? version (go-version->git-ref version)))
- (v-prefixed? (string-prefix? "v" version)))
+ (let* ((plain-version? (string=? version (go-version->git-ref version)))
+ (v-prefixed? (string-prefix? "v" version))
+ ;; A module in a subdirectory has a tag prefixed by this
+ ;; subdirectory. See <https://go.dev/ref/mod>.
+ (tag-prefix (if (string=? module-path-subdirectory "") ;
+ ""
+ (string-append
+ (substring module-path-subdirectory 1) "/")))
+ (git-commit (if plain-version?
+ (string-append tag-prefix version)
+ (go-version->git-ref version))))
`(origin
(method git-fetch)
(uri (git-reference
(url ,vcs-repo-url)
- ;; This is done because the version field of the package,
- ;; which the generated quoted expression refers to, has been
- ;; stripped of any 'v' prefixed.
(commit ,(if (and plain-version? v-prefixed?)
- '(string-append "v" version)
+ `(string-append ,tag-prefix "v" version)
'(go-version->git-ref version)))))
(file-name (git-file-name name version))
(sha256
(base32
,(bytevector->nix-base32-string
- (git-checkout-hash vcs-repo-url (go-version->git-ref version)
+ (git-checkout-hash vcs-repo-url git-commit
(hash-algorithm sha256))))))))
((hg)
`(origin
@@ -621,12 +628,17 @@ (define* (go-module->guix-package module-path #:key
dependencies+versions
(map car dependencies+versions)))
(module-path-sans-suffix
- (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path)))
+ (if (string-prefix? "gopkg.in" module-path)
+ module-path
+ (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path))))
(guix-name (go-module->guix-package-name module-path))
- (root-module-path (module-path->repository-root module-path))
+ (root-module-path (module-path->repository-root module-path-sans-suffix))
;; The VCS type and URL are not included in goproxy information. For
;; this we need to fetch it from the official module page.
(meta-data (fetch-module-meta-data root-module-path))
+ (module-path-subdirectory
+ (substring module-path-sans-suffix
+ (string-length (module-meta-import-prefix meta-data))))
(vcs-type (module-meta-vcs meta-data))
(vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
(synopsis (go-package-synopsis module-path))
@@ -637,7 +649,7 @@ (define* (go-module->guix-package module-path #:key
(name ,guix-name)
(version ,(strip-v-prefix version*))
(source
- ,(vcs->origin vcs-type vcs-repo-url version*))
+ ,(vcs->origin vcs-type vcs-repo-url module-path-subdirectory version*))
(build-system go-build-system)
(arguments
'(#:import-path ,module-path
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 18:07
[PATCH v3 3/3] guix: import: go: Use correct tag for go module in subdirectory.
(address . 63647@debbugs.gnu.org)
ef9e802b1c3465665b96ec171cf2f7e9bf8f51e6.1685030838.git.zimon.toutoune@gmail.com
From: Stephen Paul Weber <singpolyma@singpolyma.net>

* guix/import/go.scm (vcs->origin): New argument module-path-subdirectory.
Use it to get Git tag as described by https://go.dev/ref/mod.

Co-authored-by: Simon Tournier <zimon.toutoune@gmail.com>
Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/go.scm | 35 ++++++++++++++++++++++++-----------
1 file changed, 24 insertions(+), 11 deletions(-)

Toggle diff (87 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index c6258296f6..c2f9065d94 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,6 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2022 Stephen Paul Weber <singpolyma@singpolyma.net>
;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
@@ -522,28 +523,35 @@ (define* (git-checkout-hash url reference algorithm)
(nix-base32-string->bytevector
"0000000000000000000000000000000000000000000000000000"))))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define (vcs->origin vcs-type vcs-repo-url module-path-subdirectory version)
"Generate the `origin' block of a package depending on what type of source
control system is being used."
(case vcs-type
((git)
- (let ((plain-version? (string=? version (go-version->git-ref version)))
- (v-prefixed? (string-prefix? "v" version)))
+ (let* ((plain-version? (string=? version (go-version->git-ref version)))
+ (v-prefixed? (string-prefix? "v" version))
+ ;; A module in a subdirectory has a tag prefixed by this
+ ;; subdirectory. See <https://go.dev/ref/mod>.
+ (tag-prefix (if (string=? module-path-subdirectory "") ;
+ ""
+ (string-append
+ (substring module-path-subdirectory 1) "/")))
+ (prefix (string-append tag-prefix "v"))
+ (git-commit (if plain-version?
+ (string-append tag-prefix version)
+ (go-version->git-ref version))))
`(origin
(method git-fetch)
(uri (git-reference
(url ,vcs-repo-url)
- ;; This is done because the version field of the package,
- ;; which the generated quoted expression refers to, has been
- ;; stripped of any 'v' prefixed.
(commit ,(if (and plain-version? v-prefixed?)
- '(string-append "v" version)
+ `(string-append ,prefix version)
'(go-version->git-ref version)))))
(file-name (git-file-name name version))
(sha256
(base32
,(bytevector->nix-base32-string
- (git-checkout-hash vcs-repo-url (go-version->git-ref version)
+ (git-checkout-hash vcs-repo-url git-commit
(hash-algorithm sha256))))))))
((hg)
`(origin
@@ -621,12 +629,17 @@ (define* (go-module->guix-package module-path #:key
dependencies+versions
(map car dependencies+versions)))
(module-path-sans-suffix
- (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path)))
+ (if (string-prefix? "gopkg.in" module-path)
+ module-path
+ (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path))))
(guix-name (go-module->guix-package-name module-path))
- (root-module-path (module-path->repository-root module-path))
+ (root-module-path (module-path->repository-root module-path-sans-suffix))
;; The VCS type and URL are not included in goproxy information. For
;; this we need to fetch it from the official module page.
(meta-data (fetch-module-meta-data root-module-path))
+ (module-path-subdirectory
+ (substring module-path-sans-suffix
+ (string-length (module-meta-import-prefix meta-data))))
(vcs-type (module-meta-vcs meta-data))
(vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
(synopsis (go-package-synopsis module-path))
@@ -637,7 +650,7 @@ (define* (go-module->guix-package module-path #:key
(name ,guix-name)
(version ,(strip-v-prefix version*))
(source
- ,(vcs->origin vcs-type vcs-repo-url version*))
+ ,(vcs->origin vcs-type vcs-repo-url module-path-subdirectory version*))
(build-system go-build-system)
(arguments
'(#:import-path ,module-path
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 18:07
[PATCH v3 2/3] guix: import: go: Catch Git error.
(address . 63647@debbugs.gnu.org)
f699a25c43805136ada95ef7c7213a78abf5909e.1685030838.git.zimon.toutoune@gmail.com
* guix/import/go.scm (git-checkout-hash): Catch Git error.

Reported-by: Felix Lechner <felix.lechner@lease-up.com>.
---
guix/import/go.scm | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 1943869162..c6258296f6 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -6,7 +6,7 @@
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
-;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2021, 2023 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -36,9 +36,11 @@ (define-module (guix import go)
#:use-module (guix memoization)
#:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
- #:autoload (guix base32) (bytevector->nix-base32-string)
+ #:autoload (guix base32) (bytevector->nix-base32-string nix-base32-string->bytevector)
#:autoload (guix build utils) (mkdir-p)
+ #:autoload (guix ui) (warning)
#:autoload (gcrypt hash) (hash-algorithm sha256)
+ #:autoload (git structs) (git-error-message)
#:use-module (ice-9 format)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
@@ -507,12 +509,18 @@ (define* (git-checkout-hash url reference algorithm)
;; subsequent "guix import" invocations.
(mkdir-p cache)
(chmod cache #o700)
- (let-values (((checkout commit _)
- (parameterize ((%repository-cache-directory cache))
- (update-cached-checkout url
- #:ref
- `(tag-or-commit . ,reference)))))
- (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (catch 'git-error
+ (lambda _
+ (let-values (((checkout commit _)
+ (parameterize ((%repository-cache-directory cache))
+ (update-cached-checkout url
+ #:ref
+ `(tag-or-commit . ,reference)))))
+ (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
+ (lambda (key error . rest)
+ (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
+ (nix-base32-string->bytevector
+ "0000000000000000000000000000000000000000000000000000"))))
(define (vcs->origin vcs-type vcs-repo-url version)
"Generate the `origin' block of a package depending on what type of source
--
2.38.1
S
S
Simon Tournier wrote on 25 May 2023 18:07
[PATCH v3 1/3] guix: import: go: Sort hint about available versions.
(address . 63647@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
a520128590d4b67af4c2b22ae167590b99e449dd.1685030838.git.zimon.toutoune@gmail.com
* guix/import/go.scm (validate-version): Sort available versions.
---
guix/import/go.scm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

Toggle diff (28 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1eb..1943869162 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -34,6 +34,7 @@ (define-module (guix import go)
#:use-module (guix packages)
#:use-module (guix http-client)
#:use-module (guix memoization)
+ #:use-module (guix utils)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
#:autoload (guix base32) (bytevector->nix-base32-string)
#:autoload (guix build utils) (mkdir-p)
@@ -588,8 +589,9 @@ (define (validate-version version available-versions module-path)
(condition (&fix-hint
(hint (format #f (G_ "Pick one of the following \
available versions:~{ ~a~}.")
- (map strip-v-prefix
- available-versions)))))))))
+ (sort (map strip-v-prefix
+ available-versions)
+ version>?)))))))))
(define* (go-module->guix-package module-path #:key
(goproxy "https://proxy.golang.org")

base-commit: 99fc7e5dd3c9c92f9b6c2e51db9dc5b0481a56b2
--
2.38.1
L
L
Ludovic Courtès wrote on 5 Jun 2023 14:43
Re: bug#63647: [PATCH 0/2] Fix annoyance with "guix import go"
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 63647@debbugs.gnu.org)
87sfb6hyey.fsf_-_@gnu.org
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (2 lines)
> * guix/import/go.scm (validate-version): Sort available versions.

LGTM!

Toggle quote (8 lines)
> From: Stephen Paul Weber <singpolyma@singpolyma.net>
>
> * guix/import/go.scm (vcs->origin): New argument module-path-subdirectory.
> Use it to get Git tag as described by <https://go.dev/ref/mod>.
>
> Co-authored-by: Simon Tournier <zimon.toutoune@gmail.com>
> Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>

[...]

Toggle quote (5 lines)
> -(define (vcs->origin vcs-type vcs-repo-url version)
> +(define (vcs->origin vcs-type vcs-repo-url module-path-subdirectory version)
> "Generate the `origin' block of a package depending on what type of source
> control system is being used."

Maybe just ‘subdirectory’? (In general ‘path’ is used for search paths,
not for file names, but I see this module already used the term with
that meaning.)

Also please mention it in the docstring.

Should there be a test, or is it too tricky?

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 5 Jun 2023 14:45
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87o7luhybt.fsf_-_@gnu.org
Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (2 lines)
> * guix/import/go.scm (git-checkout-hash): Catch Git error.

[...]

Toggle quote (13 lines)
> + (catch 'git-error
> + (lambda _
> + (let-values (((checkout commit _)
> + (parameterize ((%repository-cache-directory cache))
> + (update-cached-checkout url
> + #:ref
> + `(tag-or-commit . ,reference)))))
> + (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
> + (lambda (key error . rest)
> + (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
> + (nix-base32-string->bytevector
> + "0000000000000000000000000000000000000000000000000000"))))

I’d rather let the exception through. How about adding
‘with-git-error-handling’ at the UI level, in (guix scripts import go)?

Thanks,
Ludo’.
S
S
Simon Tournier wrote on 6 Jun 2023 08:21
Re: [bug#63647] [PATCH 0/2] Fix annoyance with "guix import go"
(name . Ludovic Courtès)(address . ludo@gnu.org)
86cz29nm9n.fsf@gmail.com
Hi Ludo,

On Mon, 05 Jun 2023 at 14:45, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (22 lines)
> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>
>> * guix/import/go.scm (git-checkout-hash): Catch Git error.
>
> [...]
>
>> + (catch 'git-error
>> + (lambda _
>> + (let-values (((checkout commit _)
>> + (parameterize ((%repository-cache-directory cache))
>> + (update-cached-checkout url
>> + #:ref
>> + `(tag-or-commit . ,reference)))))
>> + (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
>> + (lambda (key error . rest)
>> + (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
>> + (nix-base32-string->bytevector
>> + "0000000000000000000000000000000000000000000000000000"))))
>
> I’d rather let the exception through. How about adding
> ‘with-git-error-handling’ at the UI level, in (guix scripts import go)?

What do you mean by “let the exception through”? It seems better to be
non-blocking and thus catch the exception then raise a meaningful
warning; it’s required when running with the option recursive.

Well, maybe an improvement could be in the addition of some
’report-git-warning’ and/or ’with-git-error-handling*’, in (guix git);
hum, I do not know.

Last, considering that the module (guix import go) already contains 4 UI
messages (G_), trying to move this warning about Git to (guix scripts
import go) will add some complexity – re-raise the exception 2 or 3
times, IIUC – and thus it will not change much about the UI, IMHO. I
mean, such move should be for all the messages or nothing.

Cheers,
simon
L
L
Ludovic Courtès wrote on 6 Jun 2023 17:42
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87zg5c61iv.fsf@gnu.org
Hello!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (27 lines)
> On Mon, 05 Jun 2023 at 14:45, Ludovic Courtès <ludo@gnu.org> wrote:
>> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>>
>>> * guix/import/go.scm (git-checkout-hash): Catch Git error.
>>
>> [...]
>>
>>> + (catch 'git-error
>>> + (lambda _
>>> + (let-values (((checkout commit _)
>>> + (parameterize ((%repository-cache-directory cache))
>>> + (update-cached-checkout url
>>> + #:ref
>>> + `(tag-or-commit . ,reference)))))
>>> + (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
>>> + (lambda (key error . rest)
>>> + (warning (G_ "Git error: ~a in ~a~%") (git-error-message error) url)
>>> + (nix-base32-string->bytevector
>>> + "0000000000000000000000000000000000000000000000000000"))))
>>
>> I’d rather let the exception through. How about adding
>> ‘with-git-error-handling’ at the UI level, in (guix scripts import go)?
>
> What do you mean by “let the exception through”? It seems better to be
> non-blocking and thus catch the exception then raise a meaningful
> warning; it’s required when running with the option recursive.

I thought it would be more appropriate to error out as soon as we have a
Git problem, rather than print a warning and emit an incorrect hash.

I understand that, when using ‘--recursive’, this means aborting the
whole process without producing anything. But maybe that’s better that
producing an incorrect (origin …) form?

Now, I don’t use ‘guix import go -r’ so it’s possible that I don’t
understand the scenario being considered here!

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 14 Jun 2023 23:09
Re: bug#63631: [PATCH] import: go: Handle subpackage versioning correctly.
(name . Timo Wilken)(address . guix@twilken.net)
87pm5xrbsg.fsf@gnu.org
Hi Timo,

Timo Wilken <guix@twilken.net> skribis:

Toggle quote (18 lines)
> Some Go source repositories (notably the Google Cloud SDK) contain multiple
> submodules and use a `refs/tags/<submodule>/<version>' tagging scheme.
>
> Fixes <https://bugs.gnu.org/54097>.
>
> * guix/import/go.scm (vcs->origin): Accept a module-path-suffix.
> (go-module->guix-package): Use the new parameter.
> ---
> Here's a patch that fixes the reported issue (bug#54097) for me. I've only
> tested this on the github.com/googleapis/google-cloud-go/compute package so
> far, though it seems to work there. Perhaps others have more testcases?
>
> I don't know enough about Go tooling to use it, so I've just patched the Guile
> logic of the importer. (I don't write Go, I just want to package stuff written
> in it.) In terms of performance, at least the repo contents are apparently
> cached by the first `git-checkout-hash' call, even if it fails, so the second
> call doesn't have to redownload them.

What you propose looks similar to part of the work Simon Tournier

What would you suggest? Simon?

Thanks for the patch, Timo!

Ludo’.
T
T
Timo Wilken wrote on 17 Jun 2023 17:12
(name . Ludovic Courtès)(address . ludo@gnu.org)
CTF06XBYWPT0.1MV6QA1B2OB98@lap.twilken.net
Hi Ludo', (hi everyone,)

On Wed Jun 14, 2023 at 11:09 PM CEST, Ludovic Courtès wrote:
Toggle quote (11 lines)
> Timo Wilken <guix@twilken.net> skribis:
> > Here's a patch that fixes the reported issue (bug#54097) for me. I've only
> > tested this on the github.com/googleapis/google-cloud-go/compute package so
> > far, though it seems to work there. Perhaps others have more testcases?
> >
> > I don't know enough about Go tooling to use it, so I've just patched the Guile
> > logic of the importer. (I don't write Go, I just want to package stuff written
> > in it.) In terms of performance, at least the repo contents are apparently
> > cached by the first `git-checkout-hash' call, even if it fails, so the second
> > call doesn't have to redownload them.

I've been testing my patch further this weekend, and I have a couple more
patches in the pipeline; I suppose I ought to clean those up and submit them.

In particular, I've got fixes for the following queued up locally:

1. Finding the `module-path-subdir' needs another case for e.g.
cloud.google.com/go/*.

2. My patch sometimes generates an unnecessary `go-version->git-ref' call.

3. Go versions need to be parsed from go.mod, since some packages require a
newer Go compiler than our default. This I've got a patch for, but this Go
version also ought to propagate up the dependency tree. I haven't found an
easy way to do that, since the importer seems to generate top-level
packages first, before descending the dep tree...

4. `fetch-module-meta-data' ought to ignore 4xx HTTP errors to follow the
spec; gonum.org/v1/gonum specifically depends on this behaviour.

I've been trying to recursively import github.com/matrix-org/dendrite, which
has a particularly large and hairy dependency tree. While I can now import it
without crashes, I can't build it from the imported package definitions yet --
mainly because of lots of dependency cycles in the generated packages, but
there may be more issues hidden beneath that.

Still, I can recommend it as a test of everyone's importer patches, since
it'll find a lot of edge cases in importing alone!

Toggle quote (3 lines)
> What you propose looks similar to part of the work Simon Tournier
> submitted at <https://issues.guix.gnu.org/63647>.

It seems lots of people have been working on the same problem -- in addition
to Simon's patches, I found a patch submitted by Elbek (issues 64035 & 64036;
Cc'd). I also forgot about the issue I submitted months ago (63001)...

Toggle quote (2 lines)
> What would you suggest? Simon?

Here's a brief comparison between Simon's patches and mine -- Simon's seem to
contain fixes for a couple more things than mine currently does:

1. Simon sorts available versions in an error message; this can presumably be
merged independently since it doesn't conflict with other patches.

2. Simon always prepends a "SUBDIR/" prefix to the tag if found, whereas I try
to find the plain "vX" tag first, then fall back to "SUBDIR/vX". Judging by
https://go.dev/ref/mod#vcs-version,Simon's approach seems more correct.
I'll change my implementation to match and try it out.

3. For detecting the `module-path-subdirectory' in Simon's patches: that's the
same approach I used initially, but I found I have to try `(substring
module-path (string-length import-prefix))' first (to handle e.g.
cloud.google.com/go/*). This is one of the things I haven't submitted
yet...

Toggle quote (2 lines)
> Thanks for the patch, Timo!

Thanks for your work in sorting through all of this, Ludo'!

Cheers,
Timo
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEU+w8BoVog92SNVvCL8eFBGgfabAFAmSNzaUACgkQL8eFBGgf
abDEaA//Y6k/Uckhvq+wllES0TjACxVG+2bY2E/wLZNoQShXF5Tn3ZQ9o2HOU7Gr
p8er/t/JkPLYm9w2MidUtvTms6qxp8H/YIZ12gzZ+GP/eoKa/iiqt42+KZvLS/Is
B2JWEcaJLFYy0w9nuubeuQpgcWEmxyk7/zb++GovD0tfYgczII3N1Z5fhRmeAj0N
FbKqfkBocxruYvBfcI5KhMV6Q1TjCwuAH+j2iwNmsiSnm7NTKbJxfCyO9+p5UF9w
O/em01p9J+Cg6EEFunwoi1pjoEAkVZCRtgaiP1SMgapq74UKKagvsWqMoW+JHpDf
uuiaVn1sLuTt+u9KGamYqaCj1NRcCfN7SjPxS7oTM4hsytVDmUB/eIHD6t3HntCP
Ev43fJKJwLoybQV0ptajnJggEEaEY1cegQG683jsD0ycrToh4YEhLe5fQsGhaLMA
7MSLjjf0K73BoFyZi+wYyfDPYqXRQmpb3Z6cKJPlF3v4IF13UtSxP0yBRdz930aI
1b1Q5khLQzQBI0fvxMNcZGPeA2RjPyNsXE8pKn9zEkIunzX9VYN8az++EjX928Ox
mmgjEUsLWMPCSMwGcEoZye6R5wJUG34LEkoQTbySf+314hlELPuGcLCatFgZZbXq
1Hn1upt13XYRYVCX/SXIpBJL9gdNbi2sSg8FVm4Aq+sQQqZzAKs=
=fjJn
-----END PGP SIGNATURE-----


S
S
Simon Tournier wrote on 16 Aug 2023 17:59
Re: bug#63001: bug#63631: [PATCH] import: go: Handle subpackage versioning correctly.
875y5ff05i.fsf@gmail.com
Hi Timo,

On Sat, 17 Jun 2023 at 17:12, "Timo Wilken" <guix@twilken.net> wrote:

Toggle quote (19 lines)
>> What would you suggest? Simon?
>
> Here's a brief comparison between Simon's patches and mine -- Simon's seem to
> contain fixes for a couple more things than mine currently does:
>
> 1. Simon sorts available versions in an error message; this can presumably be
> merged independently since it doesn't conflict with other patches.
>
> 2. Simon always prepends a "SUBDIR/" prefix to the tag if found, whereas I try
> to find the plain "vX" tag first, then fall back to "SUBDIR/vX". Judging by
> https://go.dev/ref/mod#vcs-version, Simon's approach seems more correct.
> I'll change my implementation to match and try it out.
>
> 3. For detecting the `module-path-subdirectory' in Simon's patches: that's the
> same approach I used initially, but I found I have to try `(substring
> module-path (string-length import-prefix))' first (to handle e.g.
> cloud.google.com/go/*). This is one of the things I haven't submitted
> yet...

Sorry if I have missed some patches or overlooked something. Do you
plan to send another patch series handling all?


Cheers,
simon
C
C
Christopher Baines wrote on 17 Oct 2023 10:31
tag 63647 moreinfo
(address . control@debbugs.gnu.org)
87h6mpy7jp.fsf@cbaines.net
tags 63647 + moreinfo
quit
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 63647@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 63647
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch