[PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

  • Open
  • quality assurance status badge
Details
2 participants
  • Christina O'Donnell
  • Sharlatan Hellseher
Owner
unassigned
Submitted by
Christina O'Donnell
Severity
normal
C
C
Christina O'Donnell wrote on 16 Mar 11:26 +0100
(address . guix-patches@gnu.org)(name . Christina O'Donnell)(address . cdo@mutix.org)
bfb6cb47bc6e2a05846653ca0b07470d4ae8ae05.1710584715.git.cdo@mutix.org
* guix/build-system/go.scm (go-version->git-ref): Add subdir keyword
parameter. This is needed because Go can have mutliple modules at different
versions in a single repo. It distinguishes their releases by using tags
with their subdirectory. See https://go.dev/ref/mod#vcs-version.

Change-Id: I68bc9e785e49877bb0b756de8458308549f4c957
---
guix/build-system/go.scm | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)

Toggle diff (36 lines)
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 0934fded07..94c5439dd1 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -56,11 +56,12 @@ (define %go-pseudo-version-rx
"([0-9A-Fa-f]{12})" ;commit hash
"(\\+incompatible)?$"))) ;optional +incompatible tag
-(define (go-version->git-ref version)
+(define* (go-version->git-ref version #:key subdir)
"Parse VERSION, a \"pseudo-version\" as defined at
-<https://golang.org/ref/mod#pseudo-versions>, and extract the commit hash from
-it, defaulting to full VERSION (stripped from the \"+incompatible\" suffix if
-present) if a pseudo-version pattern is not recognized."
+<https://golang.org/ref/mod#pseudo-versions>, and extract the commit hash from it,
+defaulting to full VERSION (stripped from the \"+incompatible\" suffix if present) if
+a pseudo-version pattern is not recognized. If SUBDIR is specified and this is not a
+pseudo-version, then this will prefix SUBDIR/ to the returned tag."
;; A module version like v1.2.3 is introduced by tagging a revision in the
;; underlying source repository. Untagged revisions can be referred to
;; using a "pseudo-version" like v0.0.0-yyyymmddhhmmss-abcdefabcdef, where
@@ -78,7 +79,9 @@ (define (go-version->git-ref version)
(match (regexp-exec %go-pseudo-version-rx version)))
(if match
(match:substring match 2)
- version)))
+ (if subdir
+ (string-append subdir "/" version)
+ version))))
(define (go-pseudo-version? version)
"True if VERSION is a Go pseudo-version, i.e., a version string made of a

base-commit: efc0ee1d7f2b704d3fc0c8aea0ef0ad1ac2972e1
--
2.41.0
C
C
Christina O'Donnell wrote on 16 Mar 11:30 +0100
[PATCH 2/3] import/go: Account for monorepo modules in the Go importer.
(address . 69827@debbugs.gnu.org)(name . Christina O'Donnell)(address . cdo@mutix.org)
e6fa7dee732e4fb0b186296781a321c1de8b143f.1710584715.git.cdo@mutix.org
This change allows for importing of modules situated in repos where they're
not at the root directory.

* guix/import/go.scm (go-module-verstion-string): Delete
(go-module-version-info): New procedure
(module-path->repository-root): Add parameter
(vcs->origin): Add parameter
(path-diff): New procedure

Change-Id: If50f7a951ce8e527e5ea44ed24db10d6a3676ff0
---
guix/import/go.scm | 71 +++++++++++++++++++++++++++++++++-------------
1 file changed, 51 insertions(+), 20 deletions(-)

Toggle diff (170 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..8276797d9a 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -122,15 +122,14 @@ (define (go-path-escape path)
(define (go.pkg.dev-info name)
(http-fetch* (string-append "https://pkg.go.dev/" name)))
-(define* (go-module-version-string goproxy name #:key version)
- "Fetch the version string of the latest version for NAME from the given
+(define* (go-module-version-info goproxy name #:key version)
+ "Fetch a JSON object encoding about the lastest version for NAME from the given
GOPROXY server, or for VERSION when specified."
(let ((file (if version
(string-append "@v/" version ".info")
"@latest")))
- (assoc-ref (json-fetch* (format #f "~a/~a/~a"
- goproxy (go-path-escape name) file))
- "Version")))
+ (json-fetch* (format #f "~a/~a/~a"
+ goproxy (go-path-escape name) file))))
(define* (go-module-available-versions goproxy name)
"Retrieve the available versions for a given module from the module proxy.
@@ -140,8 +139,12 @@ (define* (go-module-available-versions goproxy name)
(body (http-fetch* url))
(versions (remove string-null? (string-split body #\newline))))
(if (null? versions)
- (list (go-module-version-string goproxy name)) ;latest version
- versions)))
+ (begin
+ ;; If we haven't recieved any versions, look in the version-info json
+ ;; object and return a one-element list if found.
+ (or (and=> (assoc-ref (go-module-version-info goproxy name) "Version")
+ list))))
+ versions))
(define (go-package-licenses name)
"Retrieve the list of licenses that apply to NAME, a Go package or module
@@ -431,7 +434,7 @@ (define known-vcs
(/[A-Za-z0-9_.\\-]+)*$"
'git)))
-(define (module-path->repository-root module-path)
+(define (module-path->repository-root module-path version-info)
"Infer the repository root from a module path. Go modules can be
defined at any level of a repository tree, but querying for the meta tag
usually can only be done from the web page at the root of the repository,
@@ -452,6 +455,17 @@ (define (module-path->repository-root module-path)
(lambda (vcs)
(match:substring (regexp-exec (vcs-root-regex vcs)
module-path) 1)))
+ (and=> (assoc-ref version-info "Origin")
+ (lambda (origin)
+ (and=> (assoc-ref origin "Subdir")
+ (lambda (subdir)
+ ;; If version-info contains a 'subdir' and that is a suffix,
+ ;; then the repo-root can be found by stripping off the
+ ;; suffix.
+ (if (string-suffix? (string-append "/" subdir) module-path)
+ (string-drop-right module-path
+ (+ 1 (string-length subdir)))
+ #f)))))
(vcs-qualified-module-path->root-repo-url module-path)
module-path))
@@ -534,13 +548,21 @@ (define* (git-checkout-hash url reference algorithm)
`(tag-or-commit . ,reference)))))
(file-hash* checkout #:algorithm algorithm #:recursive? #true)))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define (vcs->origin vcs-type vcs-repo-url version subdir)
"Generate the `origin' block of a package depending on what type of source
control system is being used."
(case vcs-type
((git)
- (let ((plain-version? (string=? version (go-version->git-ref version)))
- (v-prefixed? (string-prefix? "v" version)))
+ (let* ((plain-version? (string=? version (go-version->git-ref version
+ #:subdir subdir)))
+ (v-prefixed? (string-prefix? "v" version))
+ ;; This is done because the version field of the package,
+ ;; which the generated quoted expression refers to, has been
+ ;; stripped of any 'v' prefixed.
+ (version-expr (if (and plain-version? v-prefixed?)
+ '(string-append "v" version)
+ `(go-version->git-ref version
+ ,@(if subdir `(#:subdir ,subdir) '())))))
`(origin
(method git-fetch)
(uri (git-reference
@@ -548,14 +570,13 @@ (define (vcs->origin vcs-type vcs-repo-url version)
;; This is done because the version field of the package,
;; which the generated quoted expression refers to, has been
;; stripped of any 'v' prefixed.
- (commit ,(if (and plain-version? v-prefixed?)
- '(string-append "v" version)
- '(go-version->git-ref version)))))
+ (commit ,version-expr)))
(file-name (git-file-name name version))
(sha256
(base32
,(bytevector->nix-base32-string
- (git-checkout-hash vcs-repo-url (go-version->git-ref version)
+ (git-checkout-hash vcs-repo-url (go-version->git-ref version
+ #:subdir subdir)
(hash-algorithm sha256))))))))
((hg)
`(origin
@@ -612,6 +633,12 @@ (define (validate-version version available-versions module-path)
(map strip-v-prefix
available-versions)))))))))
+(define (path-diff parent child)
+ (if (and (string-prefix? parent child) (not (string=? parent child)))
+ (let ((parent-len (string-length parent)))
+ (string-trim (substring child parent-len) (char-set #\/)))
+ #f))
+
(define* (go-module->guix-package module-path #:key
(goproxy "https://proxy.golang.org")
version
@@ -623,9 +650,11 @@ (define* (go-module->guix-package module-path #:key
(let* ((available-versions (go-module-available-versions goproxy module-path))
(version* (validate-version
(or (and version (ensure-v-prefix version))
- (go-module-version-string goproxy module-path)) ;latest
+ (assoc-ref (go-module-version-info goproxy module-path)
+ "Version")) ;latest
available-versions
module-path))
+ (version-info (go-module-version-info goproxy module-path #:version version*))
(content (fetch-go.mod goproxy module-path version*))
(min-go-version (second (go.mod-go-version (parse-go.mod content))))
(dependencies+versions (go.mod-requirements (parse-go.mod content)))
@@ -634,11 +663,13 @@ (define* (go-module->guix-package module-path #:key
(map car dependencies+versions)))
(module-path-sans-suffix
(match:prefix (string-match "([\\./]v[0-9]+)?$" module-path)))
- (guix-name (go-module->guix-package-name module-path))
- (root-module-path (module-path->repository-root module-path))
+ (guix-name (go-module->guix-package-name module-path-sans-suffix ))
+ (root-module-path (module-path->repository-root module-path-sans-suffix
+ version-info))
;; The VCS type and URL are not included in goproxy information. For
;; this we need to fetch it from the official module page.
(meta-data (fetch-module-meta-data root-module-path))
+ (subdir (path-diff root-module-path module-path-sans-suffix))
(vcs-type (module-meta-vcs meta-data))
(vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
(synopsis (go-package-synopsis module-path))
@@ -649,14 +680,14 @@ (define* (go-module->guix-package module-path #:key
(name ,guix-name)
(version ,(strip-v-prefix version*))
(source
- ,(vcs->origin vcs-type vcs-repo-url version*))
+ ,(vcs->origin vcs-type vcs-repo-url version* subdir))
(build-system go-build-system)
(arguments
(list ,@(if (version>? min-go-version (package-version (go-package)))
`(#:go ,(string->number min-go-version))
'())
#:import-path ,module-path
- ,@(if (string=? module-path-sans-suffix root-module-path)
+ ,@(if (string=? module-path root-module-path)
'()
`(#:unpack-path ,root-module-path))))
,@(maybe-propagated-inputs
--
2.41.0
C
C
Christina O'Donnell wrote on 16 Mar 11:30 +0100
[PATCH 3/3] import/go: Add diagnostics.
(address . 69827@debbugs.gnu.org)(name . Christina O'Donnell)(address . cdo@mutix.org)
e17bb3c54fea676d4c19d9a0ed577fbd26d8bc1e.1710584715.git.cdo@mutix.org
guix/import/go.scm (go-module-available-versions): Add warning when fetching
a list of versions from the proxy fails and raise exception when no version
can be found at all.
(module-path->repository-root): Warn when all attempts to find the repository
root have failed.
(fetch-module-meta-data): Raise exception when no meta element could be found.
(go-module->guix-package): Catch general exceptions and warn that the package
could not be imported.

Change-Id: I6dcdccc71f54bfec7110f6bfc5aeb8855502d1e3
---
guix/import/go.scm | 56 ++++++++++++++++++++++++++++++++++++----------
1 file changed, 44 insertions(+), 12 deletions(-)

Toggle diff (101 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 8276797d9a..5dd5b3d221 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -140,10 +140,15 @@ (define* (go-module-available-versions goproxy name)
(versions (remove string-null? (string-split body #\newline))))
(if (null? versions)
(begin
+ (warning (G_ "Empty list of versions on proxy ~a for package '~a'. Using latest.~%")
+ goproxy name)
;; If we haven't recieved any versions, look in the version-info json
;; object and return a one-element list if found.
(or (and=> (assoc-ref (go-module-version-info goproxy name) "Version")
- list))))
+ list)
+ (raise (make-compound-condition
+ (formatted-message (G_ "No versions available for '~a' on proxy ~a.")
+ name goproxy))))))
versions))
(define (go-package-licenses name)
@@ -467,7 +472,10 @@ (define (module-path->repository-root module-path version-info)
(+ 1 (string-length subdir)))
#f)))))
(vcs-qualified-module-path->root-repo-url module-path)
- module-path))
+ (begin
+ (warning (G_ "Unable to determine repository root of '~a'. Guessing '~a'.~%")
+ module-path module-path)
+ module-path)))
(define* (go-module->guix-package-name module-path #:optional version)
"Converts a module's path to the canonical Guix format for Go packages.
@@ -512,14 +520,19 @@ (define (fetch-module-meta-data module-path)
(select (sxpath `(// (meta (@ (equal? (name "go-import"))))
// content))))
(match (select (html->sxml meta-data #:strict? #t))
- (() #f) ;nothing selected
+ (() (raise (make-compound-condition
+ (formatted-message (G_ "no <meta/> element in result when accessing module path '~a' using go-get")
+ module-path))))
((('content content-text) ..1)
(or
(find (lambda (meta)
(string-prefix? (module-meta-import-prefix meta) module-path))
(map go-import->module-meta content-text))
;; Fallback to the first meta if no import prefixes match.
- (go-import->module-meta (first content-text)))))))
+ (go-import->module-meta (first content-text))
+ (raise (make-compound-condition
+ (formatted-message (G_ "unable to parse <meta/> when accessing module path '~a' using go-get")
+ module-path))))))))
(define (module-meta-data-repo-url meta-data goproxy)
"Return the URL where the fetcher which will be used can download the
@@ -716,16 +729,35 @@ (define go-module->guix-package*
;; consistently.
(setvbuf (current-error-port) 'none)
(let ((package-name (match args ((name _ ...) name))))
- (guard (c ((http-get-error? c)
- (warning (G_ "Failed to import package ~s.
+ (begin
+ (info (G_ "Importing package ~s...~%") package-name)
+ (guard (c ((http-get-error? c)
+ (warning (G_ "Failed to import package ~s.
reason: ~s could not be fetched: HTTP error ~a (~s).
This package and its dependencies won't be imported.~%")
- package-name
- (uri->string (http-get-error-uri c))
- (http-get-error-code c)
- (http-get-error-reason c))
- (values #f '())))
- (apply go-module->guix-package args)))))
+ package-name
+ (uri->string (http-get-error-uri c))
+ (http-get-error-code c)
+ (http-get-error-reason c))
+
+ (values #f '()))
+ ((formatted-message? c)
+ (warning (G_ "Failed to import package ~s.
+reason: ~a
+This package and its dependencies won't be imported.~%")
+ package-name
+ (apply format #f
+ (formatted-message-string c)
+ (formatted-message-arguments c)))
+ (values #f '()))
+ ((git-error? c)
+ (warning (G_ "Failed to import package ~s.
+reason: ~a
+This package and its dependencies won't be imported.~%")
+ package-name
+ (git-error-message c))
+ (values #f '())))
+ (apply go-module->guix-package args))))))
(define* (go-module-recursive-import package-name
#:key (goproxy "https://proxy.golang.org")
--
2.41.0
C
C
Christina O'Donnell wrote on 16 Mar 11:45 +0100
Fixing the go importer so that it can import modules located in a monorepo
(address . 69827@debbugs.gnu.org)
77a7ac18-b32a-8ba4-b400-5d168114c5db@mutix.org
Hi,

These three patches above allow the go importer to automatically import
modules that are located in a monorepo where the submodules are
versioned independently. See https://go.dev/ref/mod#vcs-version:

> If a module is defined in a subdirectory within the repository, that
is, the module subdirectory portion of the module path is not empty,
then each tag name must be prefixed with the module subdirectory,
followed by a slash. For example, the module golang.org/x/tools/gopls is
defined in the gopls subdirectory of the repository with root path
golang.org/x/tools. The version v0.4.0 of that module must have the tag
named gopls/v0.4.0 in that repository.

After this change, I am able to import github.com/gohugio/hugo and its
400 odd dependencies with much fewer failures.

I'll submit those patches separately once I've tested them.

Kind regards,

Christina

PS. This is my first time sending a patch series to debbugs, so I almost
certainly did something wrong. Feedback on that would be appreciated.
S
S
Sharlatan Hellseher wrote on 16 Apr 21:05 +0200
[PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.
(address . 69827@debbugs.gnu.org)
CAO+9K5rZTV+JCfnv+8906gMk2P8L9J4JH_ACt_kjwXj4dfvctw@mail.gmail.com
Hi,

Just a clarifying question. Do we definitely need to know
anything about the folder structure where go.mod governs the
module definition?

There is an API https://pkg.go.dev/ where the most of go projects
are registered, would it be easier just query it directly?

Thanks,
Oleg
Attachment: file
?