[PATCH 0/2] Improvements to the go importer

  • Open
  • quality assurance status badge
Details
3 participants
  • Efraim Flashner
  • Sharlatan Hellseher
  • Steve George
Owner
unassigned
Submitted by
Efraim Flashner
Severity
normal
E
E
Efraim Flashner wrote on 28 Nov 2023 11:18
(address . guix-patches@gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
cover.1701166089.git.efraim@flashner.co.il
I had intended to do some go packaging but found myself working on the
go importer instead.

According to the upstream go documentation, go.mod IS supposed to list
the indirect dependencies so that when someone runs 'go build' or 'go
install' all the dependencies are already listed. For us, the indirect
dependencies are already listed as propagated-inputs for the packages
which actually use them.

For the second patch I'm not super happy about using 'second' (yes,
(flatten (go.mod-directives go.mod 'go)) does return (version "1.18"))
but I couldn't find a nice way to return just the version string, and it
seemed better than cadr. Using version>? was the simplest way to do the
version compare, but I can copy the logic over if we'd rather not import
(guix utils).

Efraim Flashner (2):
guix: import: Don't include indirect dependencies in go.
guix: import: Report go version for go importer.

guix/import/go.scm | 37 +++++++++++++++++++++++++++++++------
1 file changed, 31 insertions(+), 6 deletions(-)


base-commit: 62376e3eb67644454bc655bed56be4be965bd13e
--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
E
E
Efraim Flashner wrote on 28 Nov 2023 11:21
[PATCH 1/2] guix: import: Don't include indirect dependencies in go.
(address . 67503@debbugs.gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
6314f6d57fcfd7f449af3b4b6ad7c425fd14936a.1701166089.git.efraim@flashner.co.il
* guix/import/go.scm (parse-go.mod)[define-peg-patern require]: Adjust
the peg pattern to reject lines with the 'indirect' comment in them.

Change-Id: I9618bbaa1cb8c6549ced875e3c8d32afc72c3b9b
---
guix/import/go.scm | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

Toggle diff (28 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1eb..940cdac4b0 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -7,6 +7,7 @@
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2023 Efraim Flashner <efraim@flashner.co.il>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -293,7 +294,10 @@ (define (parse-go.mod content)
;; The following directives may all be used solo or in a block
;; RequireSpec = ModulePath Version newline .
- (define-peg-pattern require all (and module-path version EOL))
+ (define-peg-pattern require all
+ (and module-path version
+ ;; We don't want the transitive dependencies.
+ (not-followed-by (and (* WS) "//" (* WS) "indirect")) EOL))
(define-peg-pattern require-top body
(and (ignore "require")
(or (and block-start (* (or require block-line)) block-end) require)))
--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
E
E
Efraim Flashner wrote on 28 Nov 2023 11:21
[PATCH 2/2] guix: import: Report go version for go importer.
(address . 67503@debbugs.gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
7a558e7d4aef26af09a8050b29c591f03a514494.1701166089.git.efraim@flashner.co.il
* guix/import/go.scm (go-package, go.mod-go-version): New procedures.
(go-module->guix-package): Add the #:go keyword in the generated package
definition if the required go is newer than the default go.

Change-Id: I8d005740a442330ac307a40a53764c803ceffc4f
---
guix/import/go.scm | 31 ++++++++++++++++++++++++++-----
1 file changed, 26 insertions(+), 5 deletions(-)

Toggle diff (81 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 940cdac4b0..dd9298808d 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -29,6 +29,7 @@ (define-module (guix import go)
#:use-module (guix git)
#:use-module (guix hash)
#:use-module (guix i18n)
+ #:use-module ((guix utils) #:select (version>?))
#:use-module (guix diagnostics)
#:use-module (guix import utils)
#:use-module (guix import json)
@@ -93,6 +94,11 @@ (define-module (guix import go)
;;; Code:
+(define (go-package)
+ "Return the 'go' package. This is a lazy reference so that we don't
+depend on (gnu packages golang)."
+ (module-ref (resolve-interface '(gnu packages golang)) 'go))
+
(define http-fetch*
;; Like http-fetch, but memoized and returning the body as a string.
(memoize (lambda args
@@ -314,7 +320,7 @@ (define (parse-go.mod content)
(define-peg-pattern with all (or (and module-path version) file-path))
(define-peg-pattern replace all (and original => with EOL))
(define-peg-pattern replace-top body
- (and (ignore "replace")
+ (and (ignore "replace")
(or (and block-start (* (or replace block-line)) block-end) replace)))
;; RetractSpec = ( Version | "[" Version "," Version "]" ) newline .
@@ -378,6 +384,17 @@ (define (go.mod-requirements go.mod)
;; Prevent inlining of this procedure, which is accessed by unit tests.
(set! go.mod-requirements go.mod-requirements)
+(define (go.mod-go-version go.mod)
+ "Return the minimum version of go required to specified by GO.MOD."
+ (let ((go-version (go.mod-directives go.mod 'go)))
+ (if (null? go-version)
+ ;; If the go directive is missing, go 1.16 is assumed.
+ '(version "1.16")
+ (flatten go-version))))
+
+;; Prevent inlining of this procedure, which is accessed by unit tests.
+(set! go.mod-go-version go.mod-go-version)
+
(define-record-type <vcs>
(%make-vcs url-prefix root-regex type)
vcs?
@@ -610,6 +627,7 @@ (define* (go-module->guix-package module-path #:key
available-versions
module-path))
(content (fetch-go.mod goproxy module-path version*))
+ (min-go-version (second (go.mod-go-version (parse-go.mod content))))
(dependencies+versions (go.mod-requirements (parse-go.mod content)))
(dependencies (if pin-versions?
dependencies+versions
@@ -634,10 +652,13 @@ (define* (go-module->guix-package module-path #:key
,(vcs->origin vcs-type vcs-repo-url version*))
(build-system go-build-system)
(arguments
- '(#:import-path ,module-path
- ,@(if (string=? module-path-sans-suffix root-module-path)
- '()
- `(#:unpack-path ,root-module-path))))
+ (list ,@(if (version>? min-go-version (package-version (go-package)))
+ `(#:go ,(string->number min-go-version))
+ '())
+ #:import-path ,module-path
+ ,@(if (string=? module-path-sans-suffix root-module-path)
+ '()
+ `(#:unpack-path ,root-module-path))))
,@(maybe-propagated-inputs
(map (match-lambda
((name version)
--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
E
E
Efraim Flashner wrote on 7 Dec 2023 12:05
[PATCH v2 2/2] guix: import: Report go version for go importer.
(address . 67503@debbugs.gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
2c3f8637eea1edd83116fc055a873b827266f977.1701412758.git.efraim@flashner.co.il
* guix/import/go.scm (go-package, go.mod-go-version): New procedures.
(go-module->guix-package): Add the #:go keyword in the generated package
definition if the required go is newer than the default go.
* tests/go.scm (mock-http-get): Use gexps for package arguments.

Change-Id: I8d005740a442330ac307a40a53764c803ceffc4f
---
guix/import/go.scm | 31 ++++++++++++++++++++++++++-----
tests/go.scm | 6 +++---
2 files changed, 29 insertions(+), 8 deletions(-)

Toggle diff (103 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 940cdac4b0..dd9298808d 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -29,6 +29,7 @@ (define-module (guix import go)
#:use-module (guix git)
#:use-module (guix hash)
#:use-module (guix i18n)
+ #:use-module ((guix utils) #:select (version>?))
#:use-module (guix diagnostics)
#:use-module (guix import utils)
#:use-module (guix import json)
@@ -93,6 +94,11 @@ (define-module (guix import go)
;;; Code:
+(define (go-package)
+ "Return the 'go' package. This is a lazy reference so that we don't
+depend on (gnu packages golang)."
+ (module-ref (resolve-interface '(gnu packages golang)) 'go))
+
(define http-fetch*
;; Like http-fetch, but memoized and returning the body as a string.
(memoize (lambda args
@@ -314,7 +320,7 @@ (define (parse-go.mod content)
(define-peg-pattern with all (or (and module-path version) file-path))
(define-peg-pattern replace all (and original => with EOL))
(define-peg-pattern replace-top body
- (and (ignore "replace")
+ (and (ignore "replace")
(or (and block-start (* (or replace block-line)) block-end) replace)))
;; RetractSpec = ( Version | "[" Version "," Version "]" ) newline .
@@ -378,6 +384,17 @@ (define (go.mod-requirements go.mod)
;; Prevent inlining of this procedure, which is accessed by unit tests.
(set! go.mod-requirements go.mod-requirements)
+(define (go.mod-go-version go.mod)
+ "Return the minimum version of go required to specified by GO.MOD."
+ (let ((go-version (go.mod-directives go.mod 'go)))
+ (if (null? go-version)
+ ;; If the go directive is missing, go 1.16 is assumed.
+ '(version "1.16")
+ (flatten go-version))))
+
+;; Prevent inlining of this procedure, which is accessed by unit tests.
+(set! go.mod-go-version go.mod-go-version)
+
(define-record-type <vcs>
(%make-vcs url-prefix root-regex type)
vcs?
@@ -610,6 +627,7 @@ (define* (go-module->guix-package module-path #:key
available-versions
module-path))
(content (fetch-go.mod goproxy module-path version*))
+ (min-go-version (second (go.mod-go-version (parse-go.mod content))))
(dependencies+versions (go.mod-requirements (parse-go.mod content)))
(dependencies (if pin-versions?
dependencies+versions
@@ -634,10 +652,13 @@ (define* (go-module->guix-package module-path #:key
,(vcs->origin vcs-type vcs-repo-url version*))
(build-system go-build-system)
(arguments
- '(#:import-path ,module-path
- ,@(if (string=? module-path-sans-suffix root-module-path)
- '()
- `(#:unpack-path ,root-module-path))))
+ (list ,@(if (version>? min-go-version (package-version (go-package)))
+ `(#:go ,(string->number min-go-version))
+ '())
+ #:import-path ,module-path
+ ,@(if (string=? module-path-sans-suffix root-module-path)
+ '()
+ `(#:unpack-path ,root-module-path))))
,@(maybe-propagated-inputs
(map (match-lambda
((name version)
diff --git a/tests/go.scm b/tests/go.scm
index a70a0ddbf5..d2e8846b30 100644
--- a/tests/go.scm
+++ b/tests/go.scm
@@ -1,6 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
-;;; Copyright ? 2021 Fran?ois Joulaud <francois.joulaud@radiofrance.com>
-;;; Copyright ? 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com>
+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -387,7 +387,7 @@ (define (mock-http-get testcase)
"0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"))))
(build-system go-build-system)
(arguments
- '(#:import-path "github.com/go-check/check"))
+ (list #:import-path "github.com/go-check/check"))
(propagated-inputs
`(("go-github-com-kr-pretty" ,go-github-com-kr-pretty)))
(home-page "https://github.com/go-check/check")
--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
E
E
Efraim Flashner wrote on 7 Dec 2023 12:05
[PATCH v2 1/2] guix: import: Don't include indirect dependencies in go.
(address . 67503@debbugs.gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
4231413995e058728e28814773ffc7e3c63d5104.1701412758.git.efraim@flashner.co.il
* guix/import/go.scm (parse-go.mod)[define-peg-patern require]: Adjust
the peg pattern to reject lines with the 'indirect' comment in them.

Change-Id: I9618bbaa1cb8c6549ced875e3c8d32afc72c3b9b
---
guix/import/go.scm | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

Toggle diff (28 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1eb..940cdac4b0 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -7,6 +7,7 @@
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2023 Efraim Flashner <efraim@flashner.co.il>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -293,7 +294,10 @@ (define (parse-go.mod content)
;; The following directives may all be used solo or in a block
;; RequireSpec = ModulePath Version newline .
- (define-peg-pattern require all (and module-path version EOL))
+ (define-peg-pattern require all
+ (and module-path version
+ ;; We don't want the transitive dependencies.
+ (not-followed-by (and (* WS) "//" (* WS) "indirect")) EOL))
(define-peg-pattern require-top body
(and (ignore "require")
(or (and block-start (* (or require block-line)) block-end) require)))
--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
S
S
Steve George wrote on 31 Oct 14:39 +0100
RE: Improvements to the go importer (bumping)
(address . 67503@debbugs.gnu.org)
ZyOIhG2SlSFMtIZg@dragon2
Hi,

Bumping this as a reminder ...
Unsure if go-team was on the original submission.

Futurile / Steve
S
S
Sharlatan Hellseher wrote on 8 Dec 19:00 +0100
[PATCH 0/2] Improvements to the go importer
(address . 67503@debbugs.gnu.org)
CAO+9K5rDQTY2J+4n6XnmO6LLxE8STri75rfCALEG1waL4qjQPQ@mail.gmail.com
Hi,

It looks like current go importer has that implementation.
What's missing is updater which is awere about pseudo
golang version and workspace specific tag (subdirs/v1.2.3).

--
Thanks,
Oleg
Attachment: file
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 67503@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 67503
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch