[PATCH 0/3] (WIP) Semantic version aware recusive importer for crates

DoneSubmitted by Martin Becze.
Details
10 participants
  • Efraim Flashner
  • Hartmut Goebel
  • Ivan Petkov
  • Jakub Kądziołka
  • Leo Famulari
  • Ludovic Courtès
  • Nicolas Goaziou
  • Marius Bakke
  • Martin Becze
  • Timothy Sample
Owner
unassigned
Severity
normal
Merged with
M
M
Martin Becze wrote on 28 Nov 2019 01:13
(address . guix-patches@gnu.org)
cover.1574897905.git.mjbecze@riseup.net
This patch add a new recusive importer (recusive-import-semver) which uses semantic versioning to find the correct version of dependencies. This procedure is then used by the crate importer. Since quite a few langague pms use semantic versioning I hope that other importers can also use it.
This patch has one problem that I'm aware of. recusive-import-semver relies on guile-semver. But how do I make it so that it is installed by default? Or altenativly how can I check a guile module exists or not, so that I can promt the user to install it?
Thanks,Martin Becze
Martin Becze (3): gnu: added new function, find-packages-by-name*/direct gnu: added new procedure, recusive-import-semver Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test.
gnu/packages.scm | 41 ++++++++ guix/import/crate.scm | 165 +++++++++++++++++-------------- guix/import/utils.scm | 181 ++++++++++++++++++++++++++++++++-- guix/scripts/import/crate.scm | 9 +- tests/crate.scm | 2 +- tests/import-utils.scm | 162 ++++++++++++++++++++++++++++++ tests/packages.scm | 13 +++ 7 files changed, 481 insertions(+), 92 deletions(-)
-- 2.24.0
M
M
Martin Becze wrote on 28 Nov 2019 01:16
[PATCH 1/3] gnu: added new function, find-packages-by-name*/direct
(address . 38408@debbugs.gnu.org)
d412799dfeddd8a990f933bb522c59cf81845cd9.1574897905.git.mjbecze@riseup.net
* gnu/packages.scm (find-packages-by-naem*/direct)--- gnu/packages.scm | 41 +++++++++++++++++++++++++++++++++++++++++ tests/packages.scm | 13 +++++++++++++ 2 files changed, 54 insertions(+)
Toggle diff (112 lines)diff --git a/gnu/packages.scm b/gnu/packages.scmindex 959777ff8f..cca2a393e5 100644--- a/gnu/packages.scm+++ b/gnu/packages.scm@@ -4,6 +4,7 @@ ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -52,7 +53,9 @@ %default-package-module-path fold-packages+ fold-packages* fold-available-packages+ find-packages-by-name*/direct find-newest-available-packages find-packages-by-name@@ -250,6 +253,23 @@ is guaranteed to never traverse the same package twice." init modules)) +(define* (fold-packages* proc init+ #:optional+ (modules (all-modules (%package-module-path)+ #:warn+ warn-about-load-error))+ #:key (select? (negate hidden-package?)))+ "Call (PROC PACKAGE RESULT) for each available package defined in one of+MODULES that matches SELECT?, using INIT as the initial value of RESULT. It+is guaranteed to never traverse the same package twice."+ (fold-module-public-variables* (lambda (module symbol var result)+ (let ((object (variable-ref var)))+ (if (and (package? object) (select? object))+ (proc module symbol object result)+ result)))+ init+ modules))+ (define %package-cache-file ;; Location of the package cache. "/lib/guix/package.cache")@@ -297,6 +317,27 @@ decreasing version order." matching) matching))))) +(define find-packages-by-name*/direct ;bypass the cache+ (let ((packages (delay+ (fold-packages* (lambda (mod sym p r)+ (vhash-cons (package-name p) (list mod sym p) r))+ vlist-null)))+ (version>? (match-lambda*+ (((_ _ versions) ..1)+ (apply version>? (map package-version versions))))))+ (lambda* (name #:optional version)+ "Return the list of (<module> <symbol> <package>) with the given NAME. If+ VERSION is not #f, then only return packages whose version is prefixed by+ VERSION, sorted in decreasing version order."+ (let ((matching (sort (vhash-fold* cons '() name (force packages))+ version>?)))+ (if version+ (filter (match-lambda+ ((_ _ package)+ (version-prefix? version (package-version package))))+ matching)+ matching)))))+ (define (cache-lookup cache name) "Lookup package NAME in CACHE. Return a list sorted in increasing version order."diff --git a/tests/packages.scm b/tests/packages.scmindex 423c5061aa..9f02b0d5d2 100644--- a/tests/packages.scm+++ b/tests/packages.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © Jan (janneke) Nieuwenhuizen <janneke@gnu.org>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -1135,11 +1136,23 @@ (((? (cut eq? hello <>))) #t) (wrong (pk 'find-packages-by-name wrong #f)))) +(test-assert "find-packages-by-name*/direct"+ (match (find-packages-by-name*/direct "hello")+ ((((? (cut eq? (resolve-interface '(gnu packages base)) <>))+ (? (cut eq? 'hello <>))+ (? (cut eq? hello <>)))) #t)))+ (test-assert "find-packages-by-name with version" (match (find-packages-by-name "hello" (package-version hello)) (((? (cut eq? hello <>))) #t) (wrong (pk 'find-packages-by-name wrong #f)))) +(test-assert "find-packages-by-name*/direct with version"+ (match (find-packages-by-name*/direct "hello" (package-version hello))+ ((((? (cut eq? (resolve-interface '(gnu packages base)) <>))+ (? (cut eq? 'hello <>))+ (? (cut eq? hello <>)))) #t)))+ (test-equal "find-packages-by-name with cache" (find-packages-by-name "guile") (call-with-temporary-directory-- 2.24.0
M
M
Martin Becze wrote on 28 Nov 2019 01:16
[PATCH 2/3] gnu: added new procedure, recusive-import-semver
(address . 38408@debbugs.gnu.org)
0b124a8a41c3488d3cda265d363eab49de7aca62.1574897905.git.mjbecze@riseup.net
* gnu/packages.scm (recusive-import-semver): New Procedure* gnu/packages.scm (package->definition)[arguments]: New argument, "latest"* tests/import-utils.scm: tests for recusive-import-semver--- guix/import/utils.scm | 181 +++++++++++++++++++++++++++++++++++++++-- tests/import-utils.scm | 162 ++++++++++++++++++++++++++++++++++++ 2 files changed, 336 insertions(+), 7 deletions(-)
Toggle diff (412 lines)diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 4694b6e7ef..6932614f8e 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2019 Robert Vollmert <rob@vllmrt.net>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -32,6 +33,7 @@ #:use-module (guix discovery) #:use-module (guix build-system) #:use-module (guix gexp)+ #:use-module (guix memoization) #:use-module (guix store) #:use-module (guix download) #:use-module (gnu packages)@@ -43,6 +45,8 @@ #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-41)+ #:use-module (semver)+ #:use-module (semver ranges) #:export (factorize-uri flatten@@ -69,7 +73,8 @@ guix-name - recursive-import))+ recursive-import+ recursive-import-semver)) (define (factorize-uri uri version) "Factorize URI, a package tarball URI as a string, such that any occurrences@@ -257,13 +262,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package)+(define* (package->definition guix-package #:optional (latest #t)) (match guix-package- (('package ('name (? string? name)) _ ...)- `(define-public ,(string->symbol name)- ,guix-package))- (('let anything ('package ('name (? string? name)) _ ...))- `(define-public ,(string->symbol name)+ ((or+ ('package ('name name) ('version version) . rest)+ ('let _ ('package ('name name) ('version version) . rest)))++ `(define-public ,(string->symbol (if latest+ name+ (string-append name "-" version))) ,guix-package)))) (define (build-system-modules)@@ -414,3 +421,163 @@ dependencies." step ;; initial state (step initial-state)))++(define* (recursive-import-semver #:key name+ (version #f)+ name->metadata+ metadata->package+ metadata-versions+ package-dependencies+ dependency-name+ dependency-range+ guix-name+ make-sexp)+ "Generates a stream of package expressions for the dependencies of the given +NAME and VERSION. The dependencies will be resolved using semantic versioning.+This procedure makes the assumption that most package repositories will, for a+given package provide some <metadata> on that package that includes what+versions of the package that are available and a list of dependencies for each+version. Dependencies are assumed to be composed of a NAME, a semantic RANGE and+other data.++This procedure takes the following keys:+ NAME - The name of the package to import+ VERSION - The version of the package to import+ NAME->METADATA - A procedure that takes a NAME of a package and returns that+package's <metadata>+ METADATA->PACKAGE A procedure that takes a package's <metadata> and VERSION +and returns the <package> for the given VERSION+ METADATA-VERSIONS A procedure that that takes a packages <metadata> and+returns a list of version as strings that are available for the given package+ PACKAGE-DEPENDENCIES a procedure that returns a list of <dependency> given a +<package>+ DEPENDENCY-NAME A procedure that takes a <dependency> and returns the its name+ DEPENDENCY-RANGE A procedure that takes a <dependency> and returns that+decency's range as a string+ GUIX-NAME A procedure that take a NAME and returns the Guix version of it+ MAKE-SEXP A procedure that takes <metadata>, <package> and a list of pairs+containing (EXPORT-NAME <dependency>), returning the package expression as an +s-expression"+ (define mem-name->metadata (memoize name->metadata))++ (define (latest? versions version)+ (equal? (car versions) version))++ (define (sorted-versions metadata)+ (sort (metadata-versions metadata) version>?))++ (define (name->versions name)+ (sorted-versions (mem-name->metadata name)))++ (define (semver-range-contains-string? range version)+ (semver-range-contains? range+ (string->semver version)))++ (define (guix-export-name name version)+ (let ((versions (name->versions name))+ (name (guix-name name)))+ (if (latest? versions version)+ name+ (string-append name "-" version))))++ ;; checks to see if we already defined or want to define a dep+ (define (find-known name range known)+ (match+ (find+ (match-lambda ((dep-name version)+ (and+ (string=? dep-name name)+ (semver-range-contains-string? range version))))+ known)++ (#f #f)+ ((name version) (list (guix-export-name name version) version #f)))+ )++ ;; searches searches for a package in guix+ (define (find-locally name range)+ (match+ (find+ (match-lambda ((_ _ package)+ (semver-range-contains-string?+ range+ (package-version package))))+ (find-packages-by-name*/direct (guix-name name)))+ (#f #f)+ ((_ export-symbol package) (list+ (symbol->string export-symbol)+ (package-version package) #f))))++ ;; searches for a package in some external repo+ (define (find-remote name range)+ (let* ((versions (name->versions name))+ (version (find+ (lambda (ver)+ (semver-range-contains-string? range ver))+ versions))+ (export-name (guix-export-name name version)))+ `(,export-name ,version #t)))+++ (define (find-dep-version dep known-deps)+ (let* ((name (dependency-name dep))+ (range (string->semver-range (dependency-range dep)))+ (export-name-version-needed+ (or (find-known name range known-deps)+ (find-locally name range)+ (find-remote name range))))+ `(,name ,@export-name-version-needed ,dep)+ ))++ (define (make-package-definition name version known-deps)+ (let* ((metadata (mem-name->metadata name))+ (versions (sorted-versions metadata))+ (package (metadata->package metadata version))+ (deps (map (lambda (dep)+ (find-dep-version dep known-deps))+ (package-dependencies package)))+ (sexp+ (make-sexp metadata package+ (map+ (match-lambda ((_ export-symbol _ _ dep)+ (list export-symbol dep)))+ deps))))+ (values+ (package->definition sexp (latest? versions version))+ (filter-map+ (match-lambda ((name _ version need? dep)+ (if need?+ (list name version)+ #f)))+ deps))))++ (define initial-state+ (list #f+ (list+ ;; packages to find+ (list name (if version+ version+ (car (name->versions name)))))+ ;; packages that have been found+ (list)))++ (define (step state)+ (match state+ ((prev ((next-name next-version) . rest) done)+ (receive (package dependencies)+ (make-package-definition next-name next-version+ (append done rest `((,next-name ,next-version))))+ (list+ package+ (append rest dependencies)+ (cons (list next-name next-version) done))))+ ((prev '() done)+ (list #f '() done))))+ + (stream-unfold+ ;; map: produce a stream element+ (match-lambda ((latest queue done) latest))+ ;; predicate+ (match-lambda ((latest queue done) latest))+ step+ (step initial-state)))diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex c3ab25d788..4ed3a5e1da 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>+;;; Copyright © 2016 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -24,6 +25,10 @@ #:use-module (guix packages) #:use-module (guix build-system) #:use-module (gnu packages)+ #:use-module (ice-9 match)+ #:use-module (srfi srfi-1)+ #:use-module (srfi srfi-9)+ #:use-module (srfi srfi-41) #:use-module (srfi srfi-64)) (test-begin "import-utils")@@ -120,4 +125,161 @@ ("license" . #f)))) (package-native-inputs (alist->package meta)))) +(define-record-type <metadata>+ (make-metadata name versions)+ metadata?+ (name metadata-name)+ (versions metadata-versions))++(define-record-type <package>+ (make-package version dependencies)+ package?+ (version package-version)+ (dependencies package-dependencies))++(define-record-type <dependency>+ (make-dependency name range)+ dependency?+ (name dependency-name)+ (range dependency-range))++(define (metadata-semver-versions metadata)+ (map (lambda (p)+ (package-version p))+ (metadata-versions metadata)))++(define (metadata->package metadata version)+ (find+ (lambda (package)+ (equal? (package-version package) version))+ (metadata-versions metadata)))++(define (make-sexp metadata package dependencies)+ `(package+ (name ,(guix-name (metadata-name metadata)))+ (version ,(package-version package))+ (dependcies ,(map+ (match-lambda ((public-name dep)+ (list (guix-name (dependency-name dep)) public-name)))+ dependencies))))++(define (guix-name name)+ (string-append "test-" name))++(define packages+ `(("no-deps" . (("1.0.0" . ()) ("0.1.0" . ())))+ ("one-dep" . (("1.0.0" . (("no-deps" "^1.0")))+ ("0.1.0" . (("no-deps" "^0.1.0")))))+ ("shared-dep" . (("1.0.0" . (("one-dep" "^0.1.0")+ ("no-deps" "*")))))+ ("recursive" . (("1.0.0" . (("recursive" "=1.0.0")))))+ ("already-packaged" . (("1.0.0" . (("rust" "~1.28")))))))++(define (name->metadata name)+ (let ((versions (assoc-ref packages name)))+ (make-metadata name+ (map+ (match-lambda+ ((version . deps)+ (make-package version+ (map+ (lambda (name-range)+ (apply make-dependency name-range))+ deps))))+ versions))))++(define* (test-recursive-importer name version #:optional (guix-name guix-name))+ (recursive-import-semver #:name name+ #:version version+ #:name->metadata name->metadata+ #:metadata->package metadata->package+ #:metadata-versions metadata-semver-versions+ #:package-dependencies package-dependencies+ #:dependency-name dependency-name+ #:dependency-range dependency-range+ #:guix-name guix-name+ #:make-sexp make-sexp))++(test-equal "recursive import test with no dependencies"+ `((define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "no-deps" "1.0.0")))++(test-equal "recursive import test with one dependencies"+ `((define-public test-one-dep+ (package+ (name "test-one-dep")+ (version "1.0.0")+ (dependcies (("test-no-deps" "test-no-deps")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "1.0.0")))++(test-equal "recursive import test with recursuve dependencies"+ `((define-public test-recursive+ (package+ (name "test-recursive")+ (version "1.0.0")+ (dependcies (("test-recursive" "test-recursive"))))))+ (stream->list (test-recursive-importer "recursive" "1.0.0")))++(test-equal "recursive import test with no dependencies using an old version"+ `((define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "no-deps" "0.1.0")))++(test-equal "recursive import test with one dependencies unsing an old version"+ `((define-public test-one-dep-0.1.0+ (package+ (name "test-one-dep")+ (version "0.1.0")+ (dependcies (("test-no-deps" "test-no-deps-0.1.0")))))+ (define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "0.1.0")))++(test-equal "recursive import test with with dependency that is already in the repo"+ `((define-public test-already-packaged+ (package (name "test-already-packaged")+ (version "1.0.0")+ (dependcies+ (("test-rust" "rust-1.28"))))))+ (stream->list (test-recursive-importer "already-packaged" "1.0.0" identity)))++(test-equal "shared dependencies"+ `((define-public test-shared-dep+ (package+ (name "test-shared-dep")+ (version "1.0.0")+ (dependcies (("test-one-dep" "test-one-dep-0.1.0")+ ("test-no-deps" "test-no-deps")))))+ (define-public test-one-dep-0.1.0+ (package+ (name "test-one-dep")+ (version "0.1.0")+ (dependcies (("test-no-deps" "test-no-deps-0.1.0")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ())))+ (define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies()))))+ (stream->list (test-recursive-importer "shared-dep" "1.0.0")))+ (test-end "import-utils")-- 2.24.0
M
M
Martin Becze wrote on 28 Nov 2019 01:16
[PATCH 3/3] Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test.
(address . 38408@debbugs.gnu.org)
052524339786cd4c0db5fda81547239c8bee6003.1574897905.git.mjbecze@riseup.net
* guix/import/crate.scm (make-crate-sexp): Use <crate> <crate-version> as args* guix/import/crate.scm (crate->crate-version): New Procedure* guix/import/crate.scm (crate->versions): New Procedure* guix/import/crate.scm (crate-recursive-import): Updated to user recursive-import-semver* guix/scripts/import/crate.scm (guix-import-crate): Remove `define-public` generation from UI* guix/tests/crate.scm: Updated tests--- guix/import/crate.scm | 165 ++++++++++++++++++---------------- guix/scripts/import/crate.scm | 9 +- tests/crate.scm | 2 +- 3 files changed, 91 insertions(+), 85 deletions(-)
Toggle diff (258 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 8dc014d232..da92c43b8c 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -38,6 +38,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name crate-recursive-import@@ -85,7 +86,7 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string- (kind crate-dependency-kind "kind" ;'normal | 'dev+ (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string @@ -111,7 +112,9 @@ record or #f if it was not found." (url (string-append (%crate-base-url) path))) (match (assoc-ref (or (json-fetch url) '()) "dependencies") ((? vector? vector)- (map json->crate-dependency (vector->list vector)))+ (filter (lambda (dep)+ (not (eq? (crate-dependency-kind dep) 'dev)))+ (map json->crate-dependency (vector->list vector)))) (_ '())))) @@ -141,62 +144,84 @@ record or #f if it was not found." ((args ...) `((arguments (,'quasiquote ,args)))))) -(define* (make-crate-sexp #:key name version cargo-inputs cargo-development-inputs- home-page synopsis description license- #:allow-other-keys)- "Return the `package' s-expression for a rust package with the given NAME,-VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION,-and LICENSE."- (let* ((port (http-fetch (crate-uri name version)))+(define (make-crate-sexp crate version* dependencies)+ "Return the `package' s-expression for a rust package given <crate>,+ <crate-version> and a list of <crate-dependency>"+ (define normal-dependency?+ (match-lambda ((_ dep) (not (eq? (crate-dependency-kind dep) 'dev)))))++ (define (string->license string)+ (match (regexp-exec %dual-license-rx string)+ (#f (list (spdx-string->license string)))+ (m (list (spdx-string->license (match:substring m 1))+ (spdx-string->license (match:substring m 2))))))++ (let* ((dep-crates dev-dep-crates (partition normal-dependency? dependencies))+ (cargo-inputs (sort (unzip1 dep-crates)+ string-ci<?))+ (cargo-development-inputs+ (sort (unzip1 dev-dep-crates)+ string-ci<?))+ (name (crate-name crate))+ (version (crate-version-number version*))+ (home-page (or (crate-home-page crate)+ (crate-repository crate)))+ (synopsis (crate-description crate))+ (description (crate-description crate))+ (license (and=> (crate-version-license version*)+ string->license))+ (port (http-fetch (crate-uri name version)) ) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name- cargo-development-inputs)) (pkg `(package- (name ,guix-name)- (version ,version)- (source (origin- (method url-fetch)- (uri (crate-uri ,name version))- (file-name (string-append name "-" version ".tar.gz"))- (sha256- (base32- ,(bytevector->nix-base32-string (port-sha256 port))))))- (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)- (maybe-cargo-development-inputs- cargo-development-inputs)))- (home-page ,(match home-page- (() "")- (_ home-page)))- (synopsis ,synopsis)- (description ,(beautify-description description))- (license ,(match license- (() #f)- ((license) license)- (_ `(list ,@license)))))))- (close-port port)- pkg))+ (name ,guix-name)+ (version ,version)+ (source (origin+ (method url-fetch)+ (uri (crate-uri ,name version))+ (file-name (string-append name "-" version ".crate"))+ (sha256+ (base32+ ,(bytevector->nix-base32-string (port-sha256 port))))))+ (build-system cargo-build-system)+ ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ (maybe-cargo-development-inputs+ cargo-development-inputs)))+ (home-page ,(match home-page+ (() "")+ (_ home-page)))+ (synopsis ,synopsis)+ (description ,(beautify-description description))+ (license ,(match license+ (() #f)+ ((license) license)+ (_ `(list ,@license)))))))++ (close-port port)+ pkg)) (define %dual-license-rx ;; Dual licensing is represented by a string such as "MIT OR Apache-2.0". ;; This regexp matches that. (make-regexp "^(.*) OR (.*)$")) +(define (crate->crate-version crate version-number)+ "returns the <crate-version> for a given CRATE and VERSION-NUMBER"+ (find (lambda (version)+ (string=? (crate-version-number version)+ version-number))+ (crate-versions crate)))++(define (crate->versions crate)+ "Returns a list of versions for a given CRATE"+ (map (lambda (version)+ (crate-version-number version))+ (crate-versions crate)))+ (define* (crate->guix-package crate-name #:optional version) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME."- (define (string->license string)- (match (regexp-exec %dual-license-rx string)- (#f (list (spdx-string->license string)))- (m (list (spdx-string->license (match:substring m 1))- (spdx-string->license (match:substring m 2))))))-- (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))- (define crate (lookup-crate crate-name)) @@ -205,38 +230,27 @@ latest version of CRATE-NAME." (crate-latest-version crate))) (define version*- (find (lambda (version)- (string=? (crate-version-number version)- version-number))- (crate-versions crate)))+ (crate->crate-version crate version-number)) - (and crate version*- (let* ((dependencies (crate-version-dependencies version*))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))- (values- (make-crate-sexp #:name crate-name- #:version (crate-version-number version*)- #:cargo-inputs cargo-inputs- #:cargo-development-inputs cargo-development-inputs- #:home-page (or (crate-home-page crate)- (crate-repository crate))- #:synopsis (crate-description crate)- #:description (crate-description crate)- #:license (and=> (crate-version-license version*)- string->license))- (append cargo-inputs cargo-development-inputs)))))+ (define dependencies (map+ (lambda (dep)+ (list (crate-name->package-name+ (crate-dependency-id dep)) dep))+ (crate-version-dependencies version*)))+ (make-crate-sexp crate version* dependencies)) -(define (crate-recursive-import crate-name)- (recursive-import crate-name #f- #:repo->guix-package (lambda (name repo)- (crate->guix-package name))- #:guix-name crate-name->package-name))+(define* (crate-recursive-import name #:optional version)+ (recursive-import-semver+ #:name name+ #:version version+ #:name->metadata lookup-crate+ #:metadata->package crate->crate-version+ #:metadata-versions crate->versions+ #:package-dependencies crate-version-dependencies+ #:dependency-name crate-dependency-id+ #:dependency-range crate-dependency-requirement+ #:guix-name crate-name->package-name+ #:make-sexp make-crate-sexp)) (define (guix-package->crate-name package) "Return the crate name of PACKAGE."@@ -285,4 +299,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release)))-diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex 4690cceb4d..85ae6fbe59 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -96,14 +96,7 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (reverse- (stream->list- (crate-recursive-import name))))+ (stream->list (crate-recursive-import name version)) (let ((sexp (crate->guix-package name version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%")diff --git a/tests/crate.scm b/tests/crate.scmindex c14862ad9f..b77cbb08c6 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -95,7 +95,7 @@ ('source ('origin ('method 'url-fetch) ('uri ('crate-uri "foo" 'version))- ('file-name ('string-append 'name "-" 'version ".tar.gz"))+ ('file-name ('string-append 'name "-" 'version ".crate")) ('sha256 ('base32 (? string? hash)))))-- 2.24.0
M
M
Martin Becze wrote on 29 Nov 2019 16:59
Re: [PATCH] WIP patches for the rust importer
(name . Efraim Flashner)(address . efraim@flashner.co.il)
8981451ac5d914dd5f53fa928741b846@riseup.net
On 2019-11-28 12:22, Efraim Flashner wrote:
Toggle quote (32 lines)> On Wed, Nov 27, 2019 at 04:36:20PM -0800, mjbecze@riseup.net wrote:>>>> > I'd love to see what you have so far if you want to share>>>> Okie Dokie, I posted it and cc'd ya.>>>> Also I took a look at your patches.>> 0001-import-crate-Don-t-include-optional-dependencies.patch should work>> just fine with my patch. But>> 0003-import-crate-Honor-versioned-dependencies-when-impor.patch will not>> work.>>>> I took a different route here with the naming. If you are interested take>> a look take a look at my second patch. (recusive-import-semver) only will>> add the version number to the name if the crate being imported is not the>> latest version. I thought this was more inline with the canonical names,>> but if always adding version number the export symbol is desirable it will>> simplify things.>>> > I'll take a look at it in a minute. I figured with the versioned> requirements we would always want to be specific in version numbers for> crate dependents so I figured it made sense. Also, if we did want to> provide an unversioned '-latest' version we could declare an extra> variable '(define-public rust-libc rust-libc-0.2)' and now rust-libc> points to rust-libc-0.2.> >> Also I added a function (find-packages-by-name*/direct) to packages.scm>> which will return the export symbol of a package that already exists. I>> use this in case there are some non-canocal export name already added.>>
I added the no-optional-dep logic to the recusive-semver patch(https://issues.guix.gnu.org/issue/38408),but it seems to breakpackages. I'm testing on the recursive importer on "hello-cli". Attachis the patch to add the logic and the before and after output for "guiximport crate -r hello-cli". Removing all the optional deps breaks claphere for some reason which I haven't figured out.
(define-module (gnu packages crates-io) #:use-module (guix build-system cargo) #:use-module (guix download) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix packages)) (define-public rust-hello-cli (package (name "rust-hello-cli") (version "0.2.2") (source (origin (method url-fetch) (uri (crate-uri "hello-cli" version)) (file-name (string-append name "-" version ".crate")) (sha256 (base32 "0qilj9b94ig7z160kazk41k2iy1hprdk5qwaym4fnf5x9fiksild")))) (build-system cargo-build-system) (arguments `(#:cargo-inputs (("rust-clap" ,rust-clap)))) (home-page "https://github.com/fpoli/rust-hello-cli") (synopsis "Prints a nice 'Hello World!' message") (description "Prints a nice 'Hello World!' message") (license license:expat))) (define-public rust-clap (package (name "rust-clap") (version "2.33.0") (source (origin (method url-fetch) (uri (crate-uri "clap" version)) (file-name (string-append name "-" version ".crate")) (sha256 (base32 "1nf6ld3bims1n5vfzhkvcb55pdzh04bbhzf8nil5vvw05nxzarsh")))) (build-system cargo-build-system) (arguments `(#:cargo-inputs (("rust-bitflags" ,rust-bitflags) ("rust-textwrap" ,rust-textwrap) ("rust-unicode-width" ,rust-unicode-width)))) (home-page "https://clap.rs/") (synopsis "A simple to use, efficient, and full-featured Command Line Argument Parser ") (description "This package provides a simple to use, efficient, and full-featured Command Line Argument Parser ") (license license:expat))) (define-public rust-bitflags (package (name "rust-bitflags") (version "1.2.1") (source (origin (method url-fetch) (uri (crate-uri "bitflags" version)) (file-name (string-append name "-" version ".crate")) (sha256 (base32 "14qnd5nq8p2almk79m4m8ydqhd413yaxsyjp5xd19g3mikzf47fg")))) (build-system cargo-build-system) (home-page "https://github.com/bitflags/bitflags") (synopsis "A macro to generate structures which behave like bitflags. ") (description "This package provides a macro to generate structures which behave like bitflags. ") (license #f))) (define-public rust-textwrap (package (name "rust-textwrap") (version "0.11.0") (source (origin (method url-fetch) (uri (crate-uri "textwrap" version)) (file-name (string-append name "-" version ".crate")) (sha256 (base32 "0q5hky03ik3y50s9sz25r438bc4nwhqc6dqwynv4wylc807n29nk")))) (build-system cargo-build-system) (arguments `(#:cargo-inputs (("rust-unicode-width" ,rust-unicode-width)))) (home-page "https://github.com/mgeisler/textwrap") (synopsis "Textwrap is a small library for word wrapping, indenting, and dedenting strings. You can use it to format strings (such as help and error messages) for display in commandline applications. It is designed to be efficient and handle Unicode characters correctly. ") (description "Textwrap is a small library for word wrapping, indenting, and dedenting strings. You can use it to format strings (such as help and error messages) for display in commandline applications. It is designed to be efficient and handle Unicode characters correctly. ") (license license:expat))) (define-public rust-unicode-width (package (name "rust-unicode-width") (version "0.1.6") (source (origin (method url-fetch) (uri (crate-uri "unicode-width" version)) (file-name (string-append name "-" version ".crate")) (sha256 (base32 "082f9hv1r3gcd1xl33whjhrm18p0w9i77zhhhkiccb5r47adn1vh")))) (build-system cargo-build-system) (home-page "https://github.com/unicode-rs/unicode-width") (synopsis "Determine displayed width of `char` and `str` types according to Unicode Standard Annex #11 rules. ") (description "Determine displayed width of `char` and `str` types according to Unicode Standard Annex #11 rules. ") (license #f))) rust-hello-cli
Attachment: before.scm (.16 MiB)
Toggle diff (38 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex da92c43b8c..355aaa140a 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019 Efraim Flashner <efraim@flashner.co.il> ;;; ;;; This file is part of GNU Guix. ;;;@@ -88,6 +89,7 @@ (id crate-dependency-id "crate_id") ;string (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol)+ (optional crate-dependency-optional "optional") ; 'true | 'false (requirement crate-dependency-requirement "req")) ;string (define (lookup-crate name)@@ -108,12 +110,16 @@ record or #f if it was not found." (define (crate-version-dependencies version) "Return the list of <crate-dependency> records of VERSION, a <crate-version>."+ (define (optional-dependency? dependency)+ (eq? (crate-dependency-optional dependency) #t))+ (let* ((path (assoc-ref (crate-version-links version) "dependencies")) (url (string-append (%crate-base-url) path))) (match (assoc-ref (or (json-fetch url) '()) "dependencies") ((? vector? vector) (filter (lambda (dep)- (not (eq? (crate-dependency-kind dep) 'dev)))+ (not (or (eq? (crate-dependency-kind dep) 'dev)+ (optional-dependency? dep)))) (map json->crate-dependency (vector->list vector)))) (_ '()))))
M
M
Martin Becze wrote on 30 Nov 2019 17:36
Re: [PATCH 3/3] Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test.
(address . 38408@debbugs.gnu.org)(address . efraim@flashner.co.il)
42cb010759c8355943b9e2cb71a66b93@riseup.net
On 2019-11-28 00:16, Martin Becze wrote:
Toggle quote (279 lines)> * guix/import/crate.scm (make-crate-sexp): Use <crate> <crate-version> as args> * guix/import/crate.scm (crate->crate-version): New Procedure> * guix/import/crate.scm (crate->versions): New Procedure> * guix/import/crate.scm (crate-recursive-import): Updated to user> recursive-import-semver> * guix/scripts/import/crate.scm (guix-import-crate): Remove> `define-public` generation from UI> * guix/tests/crate.scm: Updated tests> ---> guix/import/crate.scm | 165 ++++++++++++++++++----------------> guix/scripts/import/crate.scm | 9 +-> tests/crate.scm | 2 +-> 3 files changed, 91 insertions(+), 85 deletions(-)> > diff --git a/guix/import/crate.scm b/guix/import/crate.scm> index 8dc014d232..da92c43b8c 100644> --- a/guix/import/crate.scm> +++ b/guix/import/crate.scm> @@ -38,6 +38,7 @@> #:use-module (srfi srfi-1)> #:use-module (srfi srfi-2)> #:use-module (srfi srfi-26)> + #:use-module (srfi srfi-71)> #:export (crate->guix-package> guix-package->crate-name> crate-recursive-import> @@ -85,7 +86,7 @@> crate-dependency?> json->crate-dependency> (id crate-dependency-id "crate_id") ;string> - (kind crate-dependency-kind "kind" ;'normal | 'dev> + (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build> string->symbol)> (requirement crate-dependency-requirement "req")) ;string> > @@ -111,7 +112,9 @@ record or #f if it was not found."> (url (string-append (%crate-base-url) path)))> (match (assoc-ref (or (json-fetch url) '()) "dependencies")> ((? vector? vector)> - (map json->crate-dependency (vector->list vector)))> + (filter (lambda (dep)> + (not (eq? (crate-dependency-kind dep) 'dev)))> + (map json->crate-dependency (vector->list vector))))> (_> '()))))> > @@ -141,62 +144,84 @@ record or #f if it was not found."> ((args ...)> `((arguments (,'quasiquote ,args))))))> > -(define* (make-crate-sexp #:key name version cargo-inputs> cargo-development-inputs> - home-page synopsis description license> - #:allow-other-keys)> - "Return the `package' s-expression for a rust package with the given NAME,> -VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS,> DESCRIPTION,> -and LICENSE."> - (let* ((port (http-fetch (crate-uri name version)))> +(define (make-crate-sexp crate version* dependencies)> + "Return the `package' s-expression for a rust package given <crate>,> + <crate-version> and a list of <crate-dependency>"> + (define normal-dependency?> + (match-lambda ((_ dep) (not (eq? (crate-dependency-kind dep) 'dev)))))> +> + (define (string->license string)> + (match (regexp-exec %dual-license-rx string)> + (#f (list (spdx-string->license string)))> + (m (list (spdx-string->license (match:substring m 1))> + (spdx-string->license (match:substring m 2))))))> +> + (let* ((dep-crates dev-dep-crates (partition normal-dependency?> dependencies))> + (cargo-inputs (sort (unzip1 dep-crates)> + string-ci<?))> + (cargo-development-inputs> + (sort (unzip1 dev-dep-crates)> + string-ci<?))> + (name (crate-name crate))> + (version (crate-version-number version*))> + (home-page (or (crate-home-page crate)> + (crate-repository crate)))> + (synopsis (crate-description crate))> + (description (crate-description crate))> + (license (and=> (crate-version-license version*)> + string->license))> + (port (http-fetch (crate-uri name version)) )> (guix-name (crate-name->package-name name))> - (cargo-inputs (map crate-name->package-name cargo-inputs))> - (cargo-development-inputs (map crate-name->package-name> - cargo-development-inputs))> (pkg `(package> - (name ,guix-name)> - (version ,version)> - (source (origin> - (method url-fetch)> - (uri (crate-uri ,name version))> - (file-name (string-append name "-"> version ".tar.gz"))> - (sha256> - (base32> - ,(bytevector->nix-base32-string> (port-sha256 port))))))> - (build-system cargo-build-system)> - ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)> - (maybe-cargo-development-inputs> - cargo-development-inputs)))> - (home-page ,(match home-page> - (() "")> - (_ home-page)))> - (synopsis ,synopsis)> - (description ,(beautify-description description))> - (license ,(match license> - (() #f)> - ((license) license)> - (_ `(list ,@license)))))))> - (close-port port)> - pkg))> + (name ,guix-name)> + (version ,version)> + (source (origin> + (method url-fetch)> + (uri (crate-uri ,name version))> + (file-name (string-append name "-" version> ".crate"))> + (sha256> + (base32> + ,(bytevector->nix-base32-string> (port-sha256 port))))))> + (build-system cargo-build-system)> + ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)> + (maybe-cargo-development-inputs> + cargo-development-inputs)))> + (home-page ,(match home-page> + (() "")> + (_ home-page)))> + (synopsis ,synopsis)> + (description ,(beautify-description description))> + (license ,(match license> + (() #f)> + ((license) license)> + (_ `(list ,@license)))))))> +> + (close-port port)> + pkg))> > (define %dual-license-rx> ;; Dual licensing is represented by a string such as "MIT OR Apache-2.0".> ;; This regexp matches that.> (make-regexp "^(.*) OR (.*)$"))> > +(define (crate->crate-version crate version-number)> + "returns the <crate-version> for a given CRATE and VERSION-NUMBER"> + (find (lambda (version)> + (string=? (crate-version-number version)> + version-number))> + (crate-versions crate)))> +> +(define (crate->versions crate)> + "Returns a list of versions for a given CRATE"> + (map (lambda (version)> + (crate-version-number version))> + (crate-versions crate)))> +> (define* (crate->guix-package crate-name #:optional version)> "Fetch the metadata for CRATE-NAME from crates.io, and return the> `package' s-expression corresponding to that package, or #f on failure.> When VERSION is specified, attempt to fetch that version; otherwise fetch the> latest version of CRATE-NAME."> - (define (string->license string)> - (match (regexp-exec %dual-license-rx string)> - (#f (list (spdx-string->license string)))> - (m (list (spdx-string->license (match:substring m 1))> - (spdx-string->license (match:substring m 2))))))> -> - (define (normal-dependency? dependency)> - (eq? (crate-dependency-kind dependency) 'normal))> -> (define crate> (lookup-crate crate-name))> > @@ -205,38 +230,27 @@ latest version of CRATE-NAME."> (crate-latest-version crate)))> > (define version*> - (find (lambda (version)> - (string=? (crate-version-number version)> - version-number))> - (crate-versions crate)))> + (crate->crate-version crate version-number))> > - (and crate version*> - (let* ((dependencies (crate-version-dependencies version*))> - (dep-crates (filter normal-dependency? dependencies))> - (dev-dep-crates (remove normal-dependency? dependencies))> - (cargo-inputs (sort (map crate-dependency-id dep-crates)> - string-ci<?))> - (cargo-development-inputs> - (sort (map crate-dependency-id dev-dep-crates)> - string-ci<?)))> - (values> - (make-crate-sexp #:name crate-name> - #:version (crate-version-number version*)> - #:cargo-inputs cargo-inputs> - #:cargo-development-inputs cargo-development-inputs> - #:home-page (or (crate-home-page crate)> - (crate-repository crate))> - #:synopsis (crate-description crate)> - #:description (crate-description crate)> - #:license (and=> (crate-version-license version*)> - string->license))> - (append cargo-inputs cargo-development-inputs)))))> + (define dependencies (map> + (lambda (dep)> + (list (crate-name->package-name> + (crate-dependency-id dep)) dep))> + (crate-version-dependencies version*)))> + (make-crate-sexp crate version* dependencies))> > -(define (crate-recursive-import crate-name)> - (recursive-import crate-name #f> - #:repo->guix-package (lambda (name repo)> - (crate->guix-package name))> - #:guix-name crate-name->package-name))> +(define* (crate-recursive-import name #:optional version)> + (recursive-import-semver> + #:name name> + #:version version> + #:name->metadata lookup-crate> + #:metadata->package crate->crate-version> + #:metadata-versions crate->versions> + #:package-dependencies crate-version-dependencies> + #:dependency-name crate-dependency-id> + #:dependency-range crate-dependency-requirement> + #:guix-name crate-name->package-name> + #:make-sexp make-crate-sexp))> > (define (guix-package->crate-name package)> "Return the crate name of PACKAGE."> @@ -285,4 +299,3 @@ latest version of CRATE-NAME."> (description "Updater for crates.io packages")> (pred crate-package?)> (latest latest-release)))> -> diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scm> index 4690cceb4d..85ae6fbe59 100644> --- a/guix/scripts/import/crate.scm> +++ b/guix/scripts/import/crate.scm> @@ -96,14 +96,7 @@ Import and convert the crate.io package for> PACKAGE-NAME.\n"))> (package-name->name+version spec))> > (if (assoc-ref opts 'recursive)> - (map (match-lambda> - ((and ('package ('name name) . rest) pkg)> - `(define-public ,(string->symbol name)> - ,pkg))> - (_ #f))> - (reverse> - (stream->list> - (crate-recursive-import name))))> + (stream->list (crate-recursive-import name version))> (let ((sexp (crate->guix-package name version)))> (unless sexp> (leave (G_ "failed to download meta-data for package '~a'~%")> diff --git a/tests/crate.scm b/tests/crate.scm> index c14862ad9f..b77cbb08c6 100644> --- a/tests/crate.scm> +++ b/tests/crate.scm> @@ -95,7 +95,7 @@> ('source ('origin> ('method 'url-fetch)> ('uri ('crate-uri "foo" 'version))> - ('file-name ('string-append 'name "-" 'version ".tar.gz"))> + ('file-name ('string-append 'name "-" 'version ".crate"))> ('sha256> ('base32> (? string? hash)))))
I'm added a patch that will skips the building of libraries which Iwould assume most of the packages being imported are. This could beparametrized in the future.
From 3f2ff3b4dc4cdf8b0282316b9c2426291da8a6c7 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Sat, 30 Nov 2019 11:27:05 -0500Subject: [PATCH] added "#:skip-build? #t" to the output of (make-crate-sexp). Most the the packages imported will be libaries and won't need to build. The top level package will build them though.
* guix/import/crate.scm (make-crate-sexp): added "#:skip-build? #t" to the output--- guix/import/crate.scm | 3 ++- tests/crate.scm | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)
Toggle diff (30 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex da92c43b8c..5683369b7a 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -183,7 +183,8 @@ record or #f if it was not found." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ ,@(maybe-arguments (append `(#:skip-build? #t)+ (maybe-cargo-inputs cargo-inputs) (maybe-cargo-development-inputs cargo-development-inputs))) (home-page ,(match home-pagediff --git a/tests/crate.scm b/tests/crate.scmindex b77cbb08c6..64e5b6932e 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -102,7 +102,8 @@ ('build-system 'cargo-build-system) ('arguments ('quasiquote- ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))+ ('#:skip-build? #t+ #:cargo-inputs (("rust-bar" ('unquote rust-bar)))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary")-- 2.24.0
E
E
Efraim Flashner wrote on 1 Dec 2019 09:59
Re: [PATCH] WIP patches for the rust importer
(name . Martin Becze)(address . mjbecze@riseup.net)
20191201085941.GB14869@E5400
On Fri, Nov 29, 2019 at 07:59:35AM -0800, Martin Becze wrote:
Toggle quote (40 lines)> On 2019-11-28 12:22, Efraim Flashner wrote:> > On Wed, Nov 27, 2019 at 04:36:20PM -0800, mjbecze@riseup.net wrote:> >>> >> > I'd love to see what you have so far if you want to share> >>> >> Okie Dokie, I posted it and cc'd ya.> >>> >> Also I took a look at your patches.> >> 0001-import-crate-Don-t-include-optional-dependencies.patch should work> >> just fine with my patch. But> >> 0003-import-crate-Honor-versioned-dependencies-when-impor.patch will not> >> work.> >>> >> I took a different route here with the naming. If you are interested take> >> a look take a look at my second patch. (recusive-import-semver) only will> >> add the version number to the name if the crate being imported is not the> >> latest version. I thought this was more inline with the canonical names,> >> but if always adding version number the export symbol is desirable it will> >> simplify things.> >>> > > > I'll take a look at it in a minute. I figured with the versioned> > requirements we would always want to be specific in version numbers for> > crate dependents so I figured it made sense. Also, if we did want to> > provide an unversioned '-latest' version we could declare an extra> > variable '(define-public rust-libc rust-libc-0.2)' and now rust-libc> > points to rust-libc-0.2.> > > >> Also I added a function (find-packages-by-name*/direct) to packages.scm> >> which will return the export symbol of a package that already exists. I> >> use this in case there are some non-canocal export name already added.> >>> > I added the no-optional-dep logic to the recusive-semver patch> (https://issues.guix.gnu.org/issue/38408), but it seems to break> packages. I'm testing on the recursive importer on "hello-cli". Attach> is the patch to add the logic and the before and after output for "guix> import crate -r hello-cli". Removing all the optional deps breaks clap> here for some reason which I haven't figured out.
Looking at the two attached files I want to bring attention to the sizeof them:after.scm [text/plain, base64, utf-8, 5.7K]before.scm [text/plain, base64, utf-8, 226K]
One way to fix this (in addition to your "default to the don't build"argument) is to keep rust-clap in it's broken state and define adifferent rust-clap for when we actually want it and to have that useenough inputs to actually build. I do now actually realize that onlyreally works if we keep all the crates hidden, which I don't think wewant. So I guess that has us sending bug reports upstream that some ofthe optional dependencies aren't actually optional.
In any case, I think it'd be better to skip the optional dependenciesand then add them back in as needed to actually build the crates wewant.

-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl3jgP0ACgkQQarn3Mo9g1GTaBAAnMDGKZ8byZEL9rDyLAcVf8I+wFrZPw7D2KolaZ15EZppFeif9Vf4mxU6bFZzfb3EurqpTND3JpLFxLXoOEw/JWe5JqSzZBR4ejRWBbzEr33R2OC+qxfm6o93GIgf8BPFx7ZvBbJeN/wRbnZunlqcPaMfgrq6hHx5iNRrGEoMaBCSROi1t/t5ZTuFgm6+aAasFlZTwCANjV2zxpmpyOpkhskrN55i3JfTGkRMVkJ5r1BsUeLVlWTsw2n5BrM8x68ETSnSt84Exn5V1a2ffzlAhktS2AXp0X67bvzwgOZzmfaGzMFEM83Rx1ymo5JQJAP/2wLUTIvfPVOb3awOqm6J0Z8ulZCY9F2+9GhIChzldHr4vzCD+tjkK6du5ez/yFiXhN8TEPA+eNHDG1q5KxEqDRJm9JNQ7jWwrOhNkQC6VTKNtmrAlAQKwMkMf6Hf8y5BbiHXWNRaHzvQB0e7+oOW8tARrU4Av9M+vY0D4WjTHlJYLfhK9gkZ5rVmncQMWqE0SGkXk6ixyFh09UHSTnyy4UcZgCXlV5XIv60OpIVYsATPwMaWn92Q1/TRCQKkWm+BxG6eOjoHOkFAS7/gJ3NFLcsNGuvYdbdiuOrP8vR5+Qvot0Q7BQcL3wLA4hPsdTE2TtdtQH48vyyNQGz/ztQ3wbJJrZm9zZyZafklZj8c4Do==+UEn-----END PGP SIGNATURE-----

E
E
Efraim Flashner wrote on 1 Dec 2019 10:00
Re: [PATCH 3/3] Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test.
(name . Martin Becze)(address . mjbecze@riseup.net)(address . 38408@debbugs.gnu.org)
20191201090058.GC14869@E5400
On Sat, Nov 30, 2019 at 08:36:20AM -0800, Martin Becze wrote:
Toggle quote (284 lines)> On 2019-11-28 00:16, Martin Becze wrote:> > * guix/import/crate.scm (make-crate-sexp): Use <crate> <crate-version> as args> > * guix/import/crate.scm (crate->crate-version): New Procedure> > * guix/import/crate.scm (crate->versions): New Procedure> > * guix/import/crate.scm (crate-recursive-import): Updated to user> > recursive-import-semver> > * guix/scripts/import/crate.scm (guix-import-crate): Remove> > `define-public` generation from UI> > * guix/tests/crate.scm: Updated tests> > ---> > guix/import/crate.scm | 165 ++++++++++++++++++----------------> > guix/scripts/import/crate.scm | 9 +-> > tests/crate.scm | 2 +-> > 3 files changed, 91 insertions(+), 85 deletions(-)> > > > diff --git a/guix/import/crate.scm b/guix/import/crate.scm> > index 8dc014d232..da92c43b8c 100644> > --- a/guix/import/crate.scm> > +++ b/guix/import/crate.scm> > @@ -38,6 +38,7 @@> > #:use-module (srfi srfi-1)> > #:use-module (srfi srfi-2)> > #:use-module (srfi srfi-26)> > + #:use-module (srfi srfi-71)> > #:export (crate->guix-package> > guix-package->crate-name> > crate-recursive-import> > @@ -85,7 +86,7 @@> > crate-dependency?> > json->crate-dependency> > (id crate-dependency-id "crate_id") ;string> > - (kind crate-dependency-kind "kind" ;'normal | 'dev> > + (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build> > string->symbol)> > (requirement crate-dependency-requirement "req")) ;string> > > > @@ -111,7 +112,9 @@ record or #f if it was not found."> > (url (string-append (%crate-base-url) path)))> > (match (assoc-ref (or (json-fetch url) '()) "dependencies")> > ((? vector? vector)> > - (map json->crate-dependency (vector->list vector)))> > + (filter (lambda (dep)> > + (not (eq? (crate-dependency-kind dep) 'dev)))> > + (map json->crate-dependency (vector->list vector))))> > (_> > '()))))> > > > @@ -141,62 +144,84 @@ record or #f if it was not found."> > ((args ...)> > `((arguments (,'quasiquote ,args))))))> > > > -(define* (make-crate-sexp #:key name version cargo-inputs> > cargo-development-inputs> > - home-page synopsis description license> > - #:allow-other-keys)> > - "Return the `package' s-expression for a rust package with the given NAME,> > -VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS,> > DESCRIPTION,> > -and LICENSE."> > - (let* ((port (http-fetch (crate-uri name version)))> > +(define (make-crate-sexp crate version* dependencies)> > + "Return the `package' s-expression for a rust package given <crate>,> > + <crate-version> and a list of <crate-dependency>"> > + (define normal-dependency?> > + (match-lambda ((_ dep) (not (eq? (crate-dependency-kind dep) 'dev)))))> > +> > + (define (string->license string)> > + (match (regexp-exec %dual-license-rx string)> > + (#f (list (spdx-string->license string)))> > + (m (list (spdx-string->license (match:substring m 1))> > + (spdx-string->license (match:substring m 2))))))> > +> > + (let* ((dep-crates dev-dep-crates (partition normal-dependency?> > dependencies))> > + (cargo-inputs (sort (unzip1 dep-crates)> > + string-ci<?))> > + (cargo-development-inputs> > + (sort (unzip1 dev-dep-crates)> > + string-ci<?))> > + (name (crate-name crate))> > + (version (crate-version-number version*))> > + (home-page (or (crate-home-page crate)> > + (crate-repository crate)))> > + (synopsis (crate-description crate))> > + (description (crate-description crate))> > + (license (and=> (crate-version-license version*)> > + string->license))> > + (port (http-fetch (crate-uri name version)) )> > (guix-name (crate-name->package-name name))> > - (cargo-inputs (map crate-name->package-name cargo-inputs))> > - (cargo-development-inputs (map crate-name->package-name> > - cargo-development-inputs))> > (pkg `(package> > - (name ,guix-name)> > - (version ,version)> > - (source (origin> > - (method url-fetch)> > - (uri (crate-uri ,name version))> > - (file-name (string-append name "-"> > version ".tar.gz"))> > - (sha256> > - (base32> > - ,(bytevector->nix-base32-string> > (port-sha256 port))))))> > - (build-system cargo-build-system)> > - ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)> > - (maybe-cargo-development-inputs> > - cargo-development-inputs)))> > - (home-page ,(match home-page> > - (() "")> > - (_ home-page)))> > - (synopsis ,synopsis)> > - (description ,(beautify-description description))> > - (license ,(match license> > - (() #f)> > - ((license) license)> > - (_ `(list ,@license)))))))> > - (close-port port)> > - pkg))> > + (name ,guix-name)> > + (version ,version)> > + (source (origin> > + (method url-fetch)> > + (uri (crate-uri ,name version))> > + (file-name (string-append name "-" version> > ".crate"))> > + (sha256> > + (base32> > + ,(bytevector->nix-base32-string> > (port-sha256 port))))))> > + (build-system cargo-build-system)> > + ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)> > + (maybe-cargo-development-inputs> > + cargo-development-inputs)))> > + (home-page ,(match home-page> > + (() "")> > + (_ home-page)))> > + (synopsis ,synopsis)> > + (description ,(beautify-description description))> > + (license ,(match license> > + (() #f)> > + ((license) license)> > + (_ `(list ,@license)))))))> > +> > + (close-port port)> > + pkg))> > > > (define %dual-license-rx> > ;; Dual licensing is represented by a string such as "MIT OR Apache-2.0".> > ;; This regexp matches that.> > (make-regexp "^(.*) OR (.*)$"))> > > > +(define (crate->crate-version crate version-number)> > + "returns the <crate-version> for a given CRATE and VERSION-NUMBER"> > + (find (lambda (version)> > + (string=? (crate-version-number version)> > + version-number))> > + (crate-versions crate)))> > +> > +(define (crate->versions crate)> > + "Returns a list of versions for a given CRATE"> > + (map (lambda (version)> > + (crate-version-number version))> > + (crate-versions crate)))> > +> > (define* (crate->guix-package crate-name #:optional version)> > "Fetch the metadata for CRATE-NAME from crates.io, and return the> > `package' s-expression corresponding to that package, or #f on failure.> > When VERSION is specified, attempt to fetch that version; otherwise fetch the> > latest version of CRATE-NAME."> > - (define (string->license string)> > - (match (regexp-exec %dual-license-rx string)> > - (#f (list (spdx-string->license string)))> > - (m (list (spdx-string->license (match:substring m 1))> > - (spdx-string->license (match:substring m 2))))))> > -> > - (define (normal-dependency? dependency)> > - (eq? (crate-dependency-kind dependency) 'normal))> > -> > (define crate> > (lookup-crate crate-name))> > > > @@ -205,38 +230,27 @@ latest version of CRATE-NAME."> > (crate-latest-version crate)))> > > > (define version*> > - (find (lambda (version)> > - (string=? (crate-version-number version)> > - version-number))> > - (crate-versions crate)))> > + (crate->crate-version crate version-number))> > > > - (and crate version*> > - (let* ((dependencies (crate-version-dependencies version*))> > - (dep-crates (filter normal-dependency? dependencies))> > - (dev-dep-crates (remove normal-dependency? dependencies))> > - (cargo-inputs (sort (map crate-dependency-id dep-crates)> > - string-ci<?))> > - (cargo-development-inputs> > - (sort (map crate-dependency-id dev-dep-crates)> > - string-ci<?)))> > - (values> > - (make-crate-sexp #:name crate-name> > - #:version (crate-version-number version*)> > - #:cargo-inputs cargo-inputs> > - #:cargo-development-inputs cargo-development-inputs> > - #:home-page (or (crate-home-page crate)> > - (crate-repository crate))> > - #:synopsis (crate-description crate)> > - #:description (crate-description crate)> > - #:license (and=> (crate-version-license version*)> > - string->license))> > - (append cargo-inputs cargo-development-inputs)))))> > + (define dependencies (map> > + (lambda (dep)> > + (list (crate-name->package-name> > + (crate-dependency-id dep)) dep))> > + (crate-version-dependencies version*)))> > + (make-crate-sexp crate version* dependencies))> > > > -(define (crate-recursive-import crate-name)> > - (recursive-import crate-name #f> > - #:repo->guix-package (lambda (name repo)> > - (crate->guix-package name))> > - #:guix-name crate-name->package-name))> > +(define* (crate-recursive-import name #:optional version)> > + (recursive-import-semver> > + #:name name> > + #:version version> > + #:name->metadata lookup-crate> > + #:metadata->package crate->crate-version> > + #:metadata-versions crate->versions> > + #:package-dependencies crate-version-dependencies> > + #:dependency-name crate-dependency-id> > + #:dependency-range crate-dependency-requirement> > + #:guix-name crate-name->package-name> > + #:make-sexp make-crate-sexp))> > > > (define (guix-package->crate-name package)> > "Return the crate name of PACKAGE."> > @@ -285,4 +299,3 @@ latest version of CRATE-NAME."> > (description "Updater for crates.io packages")> > (pred crate-package?)> > (latest latest-release)))> > -> > diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scm> > index 4690cceb4d..85ae6fbe59 100644> > --- a/guix/scripts/import/crate.scm> > +++ b/guix/scripts/import/crate.scm> > @@ -96,14 +96,7 @@ Import and convert the crate.io package for> > PACKAGE-NAME.\n"))> > (package-name->name+version spec))> > > > (if (assoc-ref opts 'recursive)> > - (map (match-lambda> > - ((and ('package ('name name) . rest) pkg)> > - `(define-public ,(string->symbol name)> > - ,pkg))> > - (_ #f))> > - (reverse> > - (stream->list> > - (crate-recursive-import name))))> > + (stream->list (crate-recursive-import name version))> > (let ((sexp (crate->guix-package name version)))> > (unless sexp> > (leave (G_ "failed to download meta-data for package '~a'~%")> > diff --git a/tests/crate.scm b/tests/crate.scm> > index c14862ad9f..b77cbb08c6 100644> > --- a/tests/crate.scm> > +++ b/tests/crate.scm> > @@ -95,7 +95,7 @@> > ('source ('origin> > ('method 'url-fetch)> > ('uri ('crate-uri "foo" 'version))> > - ('file-name ('string-append 'name "-" 'version ".tar.gz"))> > + ('file-name ('string-append 'name "-" 'version ".crate"))> > ('sha256> > ('base32> > (? string? hash)))))> > I'm added a patch that will skips the building of libraries which I> would assume most of the packages being imported are. This could be> parametrized in the future.
I like this idea. Then we can also unhide all the rust crates. It helpswhen people search for them and when guix searches for them to see ifthey can be updated or need to be imported.

Toggle quote (46 lines)> From 3f2ff3b4dc4cdf8b0282316b9c2426291da8a6c7 Mon Sep 17 00:00:00 2001> From: Martin Becze <mjbecze@riseup.net>> Date: Sat, 30 Nov 2019 11:27:05 -0500> Subject: [PATCH] added "#:skip-build? #t" to the output of (make-crate-sexp).> Most the the packages imported will be libaries and won't need to build. The> top level package will build them though.> > * guix/import/crate.scm (make-crate-sexp): added "#:skip-build? #t" to the output> ---> guix/import/crate.scm | 3 ++-> tests/crate.scm | 3 ++-> 2 files changed, 4 insertions(+), 2 deletions(-)> > diff --git a/guix/import/crate.scm b/guix/import/crate.scm> index da92c43b8c..5683369b7a 100644> --- a/guix/import/crate.scm> +++ b/guix/import/crate.scm> @@ -183,7 +183,8 @@ record or #f if it was not found."> (base32> ,(bytevector->nix-base32-string (port-sha256 port))))))> (build-system cargo-build-system)> - ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)> + ,@(maybe-arguments (append `(#:skip-build? #t)> + (maybe-cargo-inputs cargo-inputs)> (maybe-cargo-development-inputs> cargo-development-inputs)))> (home-page ,(match home-page> diff --git a/tests/crate.scm b/tests/crate.scm> index b77cbb08c6..64e5b6932e 100644> --- a/tests/crate.scm> +++ b/tests/crate.scm> @@ -102,7 +102,8 @@> ('build-system 'cargo-build-system)> ('arguments> ('quasiquote> - ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))> + ('#:skip-build? #t> + #:cargo-inputs (("rust-bar" ('unquote rust-bar))))))> ('home-page "http://example.com")> ('synopsis "summary")> ('description "summary")> -- > 2.24.0>

-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl3jgUoACgkQQarn3Mo9g1GGEg/+I9QUTNDqunu8gNW9k3dKZJewqCvfye0cy4OZRjFAO6U+ucaZ1XfsyLfPr0Mqr00GsWR60dua2VRIrvRQDJUDvH8glhUilcvbogdE4OhKkUFjg3uQISkuMybdKU81NYY7RljJ7iq3+soT40z6H3RYIYGtU9hVVU6nPdC5iJExPhclk8QAnrtHX8Wh7lHpWYHyxQPRgsj5rTbbby+b2qDsk6t/6SWsaguhmkQvEf1ijom/0MVdP+pwqTSIPBtQUsvfJv+5l3tXsV5Di31ee217su8DMmGwNelG7Fpq5z45FLEenHmk18rtochpGk0Iupjon1gKSxT59+zliv4S/saD7oYayeTdcDfBgmx+goCxpJfmmnQd9tySVf6T3RkpZWYw9T8F6PAx4Tu2qBl2Bv/OznOd4w7itJkBqIRgX3Qe2rJ1RdqWYtFuqb8DHZ80l9A+6Q9GpHnbSXpBOQKD3sW0415jipERl0TF5mkTCLzikkGZFcgtm+gr8Ud0fds2KVMd/iLiX95MbgAYJoE0DgIZdpYq3WSznEBpa9pUo8UcMw5yttTa/qSU/5IyL3PR4AF9uPzlnI9mSHAKC1I0lTAxAdHfsql4vjqP/YYfz/Jh8v604BD/f8t0kjAghKuu7zFudz04oltJL2qiDzLjw2XQixiSKS8i0VceGLaV6CFa/Us==ss+7-----END PGP SIGNATURE-----

M
M
Martin Becze wrote on 2 Dec 2019 04:17
Re: [PATCH] WIP patches for the rust importer
(name . Efraim Flashner)(address . efraim@flashner.co.il)
d2f9b201395c65462b624055b5c65100@riseup.net
On 2019-12-01 08:59, Efraim Flashner wrote:
Toggle quote (58 lines)> On Fri, Nov 29, 2019 at 07:59:35AM -0800, Martin Becze wrote:>> On 2019-11-28 12:22, Efraim Flashner wrote:>> > On Wed, Nov 27, 2019 at 04:36:20PM -0800, mjbecze@riseup.net wrote:>> >>>> >> > I'd love to see what you have so far if you want to share>> >>>> >> Okie Dokie, I posted it and cc'd ya.>> >>>> >> Also I took a look at your patches.>> >> 0001-import-crate-Don-t-include-optional-dependencies.patch should work>> >> just fine with my patch. But>> >> 0003-import-crate-Honor-versioned-dependencies-when-impor.patch will not>> >> work.>> >>>> >> I took a different route here with the naming. If you are interested take>> >> a look take a look at my second patch. (recusive-import-semver) only will>> >> add the version number to the name if the crate being imported is not the>> >> latest version. I thought this was more inline with the canonical names,>> >> but if always adding version number the export symbol is desirable it will>> >> simplify things.>> >>>> >>> > I'll take a look at it in a minute. I figured with the versioned>> > requirements we would always want to be specific in version numbers for>> > crate dependents so I figured it made sense. Also, if we did want to>> > provide an unversioned '-latest' version we could declare an extra>> > variable '(define-public rust-libc rust-libc-0.2)' and now rust-libc>> > points to rust-libc-0.2.>> >>> >> Also I added a function (find-packages-by-name*/direct) to packages.scm>> >> which will return the export symbol of a package that already exists. I>> >> use this in case there are some non-canocal export name already added.>> >>>>>> I added the no-optional-dep logic to the recusive-semver patch>> (https://issues.guix.gnu.org/issue/38408), but it seems to break>> packages. I'm testing on the recursive importer on "hello-cli". Attach>> is the patch to add the logic and the before and after output for "guix>> import crate -r hello-cli". Removing all the optional deps breaks clap>> here for some reason which I haven't figured out.> > Looking at the two attached files I want to bring attention to the size> of them:> after.scm [text/plain, base64, utf-8, 5.7K]> before.scm [text/plain, base64, utf-8, 226K]> > One way to fix this (in addition to your "default to the don't build"> argument) is to keep rust-clap in it's broken state and define a> different rust-clap for when we actually want it and to have that use> enough inputs to actually build. I do now actually realize that only> really works if we keep all the crates hidden, which I don't think we> want. So I guess that has us sending bug reports upstream that some of> the optional dependencies aren't actually optional.> > In any case, I think it'd be better to skip the optional dependencies> and then add them back in as needed to actually build the crates we> want.
oh to be more clear. Even with "#:skip-build? #t" on all the rust libs,things fail if I omit one optional dependency from somewhere. I'm justtesting on hello-cli but hello-cli inculdes clap which has alot ofoptional deps. In ./guix/build-systems/cargo.scm on line 186 it says
"Cargo requires all transitive crate dependencies' sources to beavailablein its index, even if they are optional (this is so it can generatedeterministic Cargo.lock files regardless of the target platform orenabledfeatures). Thus we need all transitive crate dependencies for any cargodev-dependencies, but this is only needed when building/testing a cratedirectly(i.e. we will never need transitive dev-dependencies for any dependencycrates)."
I haven't really dug into the cargo build-system yet but my guess isthat the problem lies there. Allllsooo i totally could have missedsomething simple.. idk
I
I
Ivan Petkov wrote on 2 Dec 2019 05:01
(name . Martin Becze)(address . mjbecze@riseup.net)
322B85FF-6EA4-492E-B96A-1F326ADD136E@gmail.com
Hi Martin!
Toggle quote (22 lines)> On Dec 1, 2019, at 7:17 PM, Martin Becze <mjbecze@riseup.net> wrote:> > oh to be more clear. Even with "#:skip-build? #t" on all the rust libs,> things fail if I omit one optional dependency from somewhere. I'm just> testing on hello-cli but hello-cli inculdes clap which has alot of> optional deps. In ./guix/build-systems/cargo.scm on line 186 it says > > "Cargo requires all transitive crate dependencies' sources to be> available> in its index, even if they are optional (this is so it can generate> deterministic Cargo.lock files regardless of the target platform or> enabled> features). Thus we need all transitive crate dependencies for any cargo> dev-dependencies, but this is only needed when building/testing a crate> directly> (i.e. we will never need transitive dev-dependencies for any dependency> crates)."> > I haven't really dug into the cargo build-system yet but my guess is> that the problem lies there. Allllsooo i totally could have missed> something simple.. idk
Yes, this is the primary limitation of packaging rust crates into guixwithout manually needing to edit the Cargo.toml definition.
My opinion is that the easiest (to maintain) method for incorporatingrust packages into guix is to bite the bullet and perform a (source)import of all transitive dependencies that a package might need.Manually keeping up with tweaking the Cargo.toml file for every singlepotential package, across every single published version will makeyou go mad :)
Then the challenge would be how to automate the incremental importingfrom crates.io http://crates.io/ (for example, some crates require a specific x.y.z version ofa dependency, some are willing to work with x.y.*, some have ranges, etc.)and making sure that all packaged crates point to the appropriate dependenciesas they get updated in guix.
—Ivan
Attachment: file
M
M
Martin Becze wrote on 3 Dec 2019 00:10
(name . Ivan Petkov)(address . ivanppetkov@gmail.com)
785501a89e81d30db021047529e7262e@riseup.net
On 2019-12-02 04:01, Ivan Petkov wrote:
Toggle quote (55 lines)> Hi Martin!> >> On Dec 1, 2019, at 7:17 PM, Martin Becze <mjbecze@riseup.net> wrote:>> >> oh to be more clear. Even with "#:skip-build? #t" on all the rust>> libs,>> things fail if I omit one optional dependency from somewhere. I'm>> just>> testing on hello-cli but hello-cli inculdes clap which has alot of>> optional deps. In ./guix/build-systems/cargo.scm on line 186 it says>> >> >> "Cargo requires all transitive crate dependencies' sources to be>> available>> in its index, even if they are optional (this is so it can generate>> deterministic Cargo.lock files regardless of the target platform or>> enabled>> features). Thus we need all transitive crate dependencies for any>> cargo>> dev-dependencies, but this is only needed when building/testing a>> crate>> directly>> (i.e. we will never need transitive dev-dependencies for any>> dependency>> crates).">> >> I haven't really dug into the cargo build-system yet but my guess is>> that the problem lies there. Allllsooo i totally could have missed>> something simple.. idk> > Yes, this is the primary limitation of packaging rust crates into guix> without manually needing to edit the Cargo.toml definition.> > My opinion is that the easiest (to maintain) method for incorporating> rust packages into guix is to bite the bullet and perform a (source)> import of all transitive dependencies that a package might need.> Manually keeping up with tweaking the Cargo.toml file for every single> potential package, across every single published version will make> you go mad :)> > Then the challenge would be how to automate the incremental importing> from crates.io [1] (for example, some crates require a specific x.y.z> version of> a dependency, some are willing to work with x.y.*, some have ranges,> etc.)> and making sure that all packaged crates point to the appropriate> dependencies> as they get updated in guix.> > —Ivan> > Links:> ------> [1] http://crates.io
Hi Ivan,
Toggle quote (4 lines)> My opinion is that the easiest (to maintain) method for incorporating> rust packages into guix is to bite the bullet and perform a (source)> import of all transitive dependencies that a package might need.
When you say source import of the transitive dependencies, do you meanthat all the rust libs should just be source only or do you mean the toplevel package should have to declare all the transitive dependencies? Ifformer I agree but if it is the latter then I would consider rustpackaging to be broken.
Toggle quote (6 lines)> Then the challenge would be how to automate the incremental importing> from crates.io [1] (for example, some crates require a specific x.y.z> version of> a dependency, some are willing to work with x.y.*, some have ranges,> etc.)
Yes that is what this patch (https://issues.guix.gnu.org/issue/38408)does. It uses semantic versioning to select the correct package formcrate.io or if available from the alread packaged rust packages.
-Martin
I
I
Ivan Petkov wrote on 4 Dec 2019 03:40
(name . Martin Becze)(address . mjbecze@riseup.net)
A9ACD9FE-C0ED-45E8-9EBB-1F6C27C1392F@gmail.com
Hi Martin,
Toggle quote (6 lines)> On Dec 2, 2019, at 3:10 PM, Martin Becze <mjbecze@riseup.net> wrote:> > When you say source import of the transitive dependencies, do you mean> that all the rust libs should just be source only or do you mean the top> level package should have to declare all the transitive dependencies?
All rust libs should be source only imports, but each package definitionshould only declare dependencies on the crates it consumes directly andguix should figure out the rest (in other words, I’d expect there to be a one-to-one mapping between a Cargo.toml and a package definition).
For example, if crate foo depends on crate bar which depends on cratebaz, I’d expect the definitions to look like:
(define-public rust-foo (package (name “rust-foo") (source-input `((“bar” ,bar)))))
(define-public rust-bar (package (name “rust-bar") (source-input `((“baz” ,baz)))))
(define-public rust-baz (package (name “rust-baz")))
But while building foo (assuming it is some kind of application), guixwould ensure that bar and baz are available in the build environment.
IMO this direction would be the most maintainable in the long term.
—Ivan
Attachment: file
M
M
Martin Becze wrote on 4 Dec 2019 23:08
(name . Ivan Petkov)(address . ivanppetkov@gmail.com)
ac10db4dbb26caedb67b8150ec10f68e@riseup.net
On 2019-12-04 02:40, Ivan Petkov wrote:
Toggle quote (43 lines)> Hi Martin,> >> On Dec 2, 2019, at 3:10 PM, Martin Becze <mjbecze@riseup.net> wrote:>>>> When you say source import of the transitive dependencies, do you>> mean>> that all the rust libs should just be source only or do you mean the>> top>> level package should have to declare all the transitive>> dependencies?> > All rust libs should be source only imports, but each package> definition> should only declare dependencies on the crates it consumes directly> and> guix should figure out the rest (in other words, I’d expect there to> be a > one-to-one mapping between a Cargo.toml and a package definition).> > For example, if crate foo depends on crate bar which depends on crate> baz, I’d expect the definitions to look like:> > (define-public rust-foo> (package> (name “rust-foo")> (source-input `((“bar” ,bar)))))> > (define-public rust-bar> (package> (name “rust-bar")> (source-input `((“baz” ,baz)))))> > (define-public rust-baz> (package> (name “rust-baz")))> > But while building foo (assuming it is some kind of application), guix> would ensure that bar and baz are available in the build environment.> > IMO this direction would be the most maintainable in the long term.> > —Ivan
Yes agree and that is what (recusive-import-semver) for produces rust.
-Martin
M
M
Martin Becze wrote on 5 Dec 2019 21:05
[PATCH v2 0/5] Semantic version aware recusive importer for crates
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
cover.1575575779.git.mjbecze@riseup.net
This version just builds a bit on the prevouse version (https://issues.guix.gnu.org/issue/38408#0)I found while testing some crates have build-dependencies and nor dependencies that are the same. While in guix build and normal dependiences are treated the same way (ie source only). So the recusive importer was importing the duplicates so here we dedup the deps. Please let me know if there are any problems!
-Martin
Martin Becze (5): gnu: added new function, find-packages-by-name*/direct gnu: added new procedure, recusive-import-semver Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test. added "#:skip-build? #t" to the output of (make-crate-sexp). Most the the packages imported will be libaries and won't need to build. The top level package will build them though. guix: crate: Depublicated build and normal dependencies
gnu/packages.scm | 41 ++++++++ guix/import/crate.scm | 188 +++++++++++++++++++--------------- guix/import/utils.scm | 181 ++++++++++++++++++++++++++++++-- guix/scripts/import/crate.scm | 9 +- tests/crate.scm | 5 +- tests/import-utils.scm | 162 +++++++++++++++++++++++++++++ tests/packages.scm | 13 +++ 7 files changed, 501 insertions(+), 98 deletions(-)
-- 2.24.0
M
M
Martin Becze wrote on 5 Dec 2019 21:05
[PATCH v2 4/5] added "#:skip-build? #t" to the output of (make-crate-sexp). Most the the packages imported will be libaries and won't need to build. The top level package will build them though.
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
bf327fb427eba14f6d4d8d324c5c75135f39e79b.1575575779.git.mjbecze@riseup.net
* guix/import/crate.scm (make-crate-sexp): added "#:skip-build? #t" to the output--- guix/import/crate.scm | 3 ++- tests/crate.scm | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)
Toggle diff (30 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex da92c43b8c..5683369b7a 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -183,7 +183,8 @@ record or #f if it was not found." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ ,@(maybe-arguments (append `(#:skip-build? #t)+ (maybe-cargo-inputs cargo-inputs) (maybe-cargo-development-inputs cargo-development-inputs))) (home-page ,(match home-pagediff --git a/tests/crate.scm b/tests/crate.scmindex b77cbb08c6..64e5b6932e 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -102,7 +102,8 @@ ('build-system 'cargo-build-system) ('arguments ('quasiquote- ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))+ ('#:skip-build? #t+ #:cargo-inputs (("rust-bar" ('unquote rust-bar)))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary")-- 2.24.0
M
M
Martin Becze wrote on 5 Dec 2019 21:05
[PATCH v2 1/5] gnu: added new function, find-packages-by-name*/direct
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
dbe5240d37d3b42a6c1db1dfc07436bff5ed2297.1575575779.git.mjbecze@riseup.net
* gnu/packages.scm (find-packages-by-naem*/direct)--- gnu/packages.scm | 41 +++++++++++++++++++++++++++++++++++++++++ tests/packages.scm | 13 +++++++++++++ 2 files changed, 54 insertions(+)
Toggle diff (112 lines)diff --git a/gnu/packages.scm b/gnu/packages.scmindex 959777ff8f..cca2a393e5 100644--- a/gnu/packages.scm+++ b/gnu/packages.scm@@ -4,6 +4,7 @@ ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -52,7 +53,9 @@ %default-package-module-path fold-packages+ fold-packages* fold-available-packages+ find-packages-by-name*/direct find-newest-available-packages find-packages-by-name@@ -250,6 +253,23 @@ is guaranteed to never traverse the same package twice." init modules)) +(define* (fold-packages* proc init+ #:optional+ (modules (all-modules (%package-module-path)+ #:warn+ warn-about-load-error))+ #:key (select? (negate hidden-package?)))+ "Call (PROC PACKAGE RESULT) for each available package defined in one of+MODULES that matches SELECT?, using INIT as the initial value of RESULT. It+is guaranteed to never traverse the same package twice."+ (fold-module-public-variables* (lambda (module symbol var result)+ (let ((object (variable-ref var)))+ (if (and (package? object) (select? object))+ (proc module symbol object result)+ result)))+ init+ modules))+ (define %package-cache-file ;; Location of the package cache. "/lib/guix/package.cache")@@ -297,6 +317,27 @@ decreasing version order." matching) matching))))) +(define find-packages-by-name*/direct ;bypass the cache+ (let ((packages (delay+ (fold-packages* (lambda (mod sym p r)+ (vhash-cons (package-name p) (list mod sym p) r))+ vlist-null)))+ (version>? (match-lambda*+ (((_ _ versions) ..1)+ (apply version>? (map package-version versions))))))+ (lambda* (name #:optional version)+ "Return the list of (<module> <symbol> <package>) with the given NAME. If+ VERSION is not #f, then only return packages whose version is prefixed by+ VERSION, sorted in decreasing version order."+ (let ((matching (sort (vhash-fold* cons '() name (force packages))+ version>?)))+ (if version+ (filter (match-lambda+ ((_ _ package)+ (version-prefix? version (package-version package))))+ matching)+ matching)))))+ (define (cache-lookup cache name) "Lookup package NAME in CACHE. Return a list sorted in increasing version order."diff --git a/tests/packages.scm b/tests/packages.scmindex 423c5061aa..9f02b0d5d2 100644--- a/tests/packages.scm+++ b/tests/packages.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © Jan (janneke) Nieuwenhuizen <janneke@gnu.org>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -1135,11 +1136,23 @@ (((? (cut eq? hello <>))) #t) (wrong (pk 'find-packages-by-name wrong #f)))) +(test-assert "find-packages-by-name*/direct"+ (match (find-packages-by-name*/direct "hello")+ ((((? (cut eq? (resolve-interface '(gnu packages base)) <>))+ (? (cut eq? 'hello <>))+ (? (cut eq? hello <>)))) #t)))+ (test-assert "find-packages-by-name with version" (match (find-packages-by-name "hello" (package-version hello)) (((? (cut eq? hello <>))) #t) (wrong (pk 'find-packages-by-name wrong #f)))) +(test-assert "find-packages-by-name*/direct with version"+ (match (find-packages-by-name*/direct "hello" (package-version hello))+ ((((? (cut eq? (resolve-interface '(gnu packages base)) <>))+ (? (cut eq? 'hello <>))+ (? (cut eq? hello <>)))) #t)))+ (test-equal "find-packages-by-name with cache" (find-packages-by-name "guile") (call-with-temporary-directory-- 2.24.0
M
M
Martin Becze wrote on 5 Dec 2019 21:05
[PATCH v2 2/5] gnu: added new procedure, recusive-import-semver
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
18d8555e374e1d7682c5f7231be25dd572dacbf0.1575575779.git.mjbecze@riseup.net
* gnu/packages.scm (recusive-import-semver): New Procedure* gnu/packages.scm (package->definition)[arguments]: New argument, "latest"* tests/import-utils.scm: tests for recusive-import-semver--- guix/import/utils.scm | 181 +++++++++++++++++++++++++++++++++++++++-- tests/import-utils.scm | 162 ++++++++++++++++++++++++++++++++++++ 2 files changed, 336 insertions(+), 7 deletions(-)
Toggle diff (412 lines)diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 4694b6e7ef..6932614f8e 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2019 Robert Vollmert <rob@vllmrt.net>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -32,6 +33,7 @@ #:use-module (guix discovery) #:use-module (guix build-system) #:use-module (guix gexp)+ #:use-module (guix memoization) #:use-module (guix store) #:use-module (guix download) #:use-module (gnu packages)@@ -43,6 +45,8 @@ #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-41)+ #:use-module (semver)+ #:use-module (semver ranges) #:export (factorize-uri flatten@@ -69,7 +73,8 @@ guix-name - recursive-import))+ recursive-import+ recursive-import-semver)) (define (factorize-uri uri version) "Factorize URI, a package tarball URI as a string, such that any occurrences@@ -257,13 +262,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package)+(define* (package->definition guix-package #:optional (latest #t)) (match guix-package- (('package ('name (? string? name)) _ ...)- `(define-public ,(string->symbol name)- ,guix-package))- (('let anything ('package ('name (? string? name)) _ ...))- `(define-public ,(string->symbol name)+ ((or+ ('package ('name name) ('version version) . rest)+ ('let _ ('package ('name name) ('version version) . rest)))++ `(define-public ,(string->symbol (if latest+ name+ (string-append name "-" version))) ,guix-package)))) (define (build-system-modules)@@ -414,3 +421,163 @@ dependencies." step ;; initial state (step initial-state)))++(define* (recursive-import-semver #:key name+ (version #f)+ name->metadata+ metadata->package+ metadata-versions+ package-dependencies+ dependency-name+ dependency-range+ guix-name+ make-sexp)+ "Generates a stream of package expressions for the dependencies of the given +NAME and VERSION. The dependencies will be resolved using semantic versioning.+This procedure makes the assumption that most package repositories will, for a+given package provide some <metadata> on that package that includes what+versions of the package that are available and a list of dependencies for each+version. Dependencies are assumed to be composed of a NAME, a semantic RANGE and+other data.++This procedure takes the following keys:+ NAME - The name of the package to import+ VERSION - The version of the package to import+ NAME->METADATA - A procedure that takes a NAME of a package and returns that+package's <metadata>+ METADATA->PACKAGE A procedure that takes a package's <metadata> and VERSION +and returns the <package> for the given VERSION+ METADATA-VERSIONS A procedure that that takes a packages <metadata> and+returns a list of version as strings that are available for the given package+ PACKAGE-DEPENDENCIES a procedure that returns a list of <dependency> given a +<package>+ DEPENDENCY-NAME A procedure that takes a <dependency> and returns the its name+ DEPENDENCY-RANGE A procedure that takes a <dependency> and returns that+decency's range as a string+ GUIX-NAME A procedure that take a NAME and returns the Guix version of it+ MAKE-SEXP A procedure that takes <metadata>, <package> and a list of pairs+containing (EXPORT-NAME <dependency>), returning the package expression as an +s-expression"+ (define mem-name->metadata (memoize name->metadata))++ (define (latest? versions version)+ (equal? (car versions) version))++ (define (sorted-versions metadata)+ (sort (metadata-versions metadata) version>?))++ (define (name->versions name)+ (sorted-versions (mem-name->metadata name)))++ (define (semver-range-contains-string? range version)+ (semver-range-contains? range+ (string->semver version)))++ (define (guix-export-name name version)+ (let ((versions (name->versions name))+ (name (guix-name name)))+ (if (latest? versions version)+ name+ (string-append name "-" version))))++ ;; checks to see if we already defined or want to define a dep+ (define (find-known name range known)+ (match+ (find+ (match-lambda ((dep-name version)+ (and+ (string=? dep-name name)+ (semver-range-contains-string? range version))))+ known)++ (#f #f)+ ((name version) (list (guix-export-name name version) version #f)))+ )++ ;; searches searches for a package in guix+ (define (find-locally name range)+ (match+ (find+ (match-lambda ((_ _ package)+ (semver-range-contains-string?+ range+ (package-version package))))+ (find-packages-by-name*/direct (guix-name name)))+ (#f #f)+ ((_ export-symbol package) (list+ (symbol->string export-symbol)+ (package-version package) #f))))++ ;; searches for a package in some external repo+ (define (find-remote name range)+ (let* ((versions (name->versions name))+ (version (find+ (lambda (ver)+ (semver-range-contains-string? range ver))+ versions))+ (export-name (guix-export-name name version)))+ `(,export-name ,version #t)))+++ (define (find-dep-version dep known-deps)+ (let* ((name (dependency-name dep))+ (range (string->semver-range (dependency-range dep)))+ (export-name-version-needed+ (or (find-known name range known-deps)+ (find-locally name range)+ (find-remote name range))))+ `(,name ,@export-name-version-needed ,dep)+ ))++ (define (make-package-definition name version known-deps)+ (let* ((metadata (mem-name->metadata name))+ (versions (sorted-versions metadata))+ (package (metadata->package metadata version))+ (deps (map (lambda (dep)+ (find-dep-version dep known-deps))+ (package-dependencies package)))+ (sexp+ (make-sexp metadata package+ (map+ (match-lambda ((_ export-symbol _ _ dep)+ (list export-symbol dep)))+ deps))))+ (values+ (package->definition sexp (latest? versions version))+ (filter-map+ (match-lambda ((name _ version need? dep)+ (if need?+ (list name version)+ #f)))+ deps))))++ (define initial-state+ (list #f+ (list+ ;; packages to find+ (list name (if version+ version+ (car (name->versions name)))))+ ;; packages that have been found+ (list)))++ (define (step state)+ (match state+ ((prev ((next-name next-version) . rest) done)+ (receive (package dependencies)+ (make-package-definition next-name next-version+ (append done rest `((,next-name ,next-version))))+ (list+ package+ (append rest dependencies)+ (cons (list next-name next-version) done))))+ ((prev '() done)+ (list #f '() done))))+ + (stream-unfold+ ;; map: produce a stream element+ (match-lambda ((latest queue done) latest))+ ;; predicate+ (match-lambda ((latest queue done) latest))+ step+ (step initial-state)))diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex c3ab25d788..4ed3a5e1da 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>+;;; Copyright © 2016 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -24,6 +25,10 @@ #:use-module (guix packages) #:use-module (guix build-system) #:use-module (gnu packages)+ #:use-module (ice-9 match)+ #:use-module (srfi srfi-1)+ #:use-module (srfi srfi-9)+ #:use-module (srfi srfi-41) #:use-module (srfi srfi-64)) (test-begin "import-utils")@@ -120,4 +125,161 @@ ("license" . #f)))) (package-native-inputs (alist->package meta)))) +(define-record-type <metadata>+ (make-metadata name versions)+ metadata?+ (name metadata-name)+ (versions metadata-versions))++(define-record-type <package>+ (make-package version dependencies)+ package?+ (version package-version)+ (dependencies package-dependencies))++(define-record-type <dependency>+ (make-dependency name range)+ dependency?+ (name dependency-name)+ (range dependency-range))++(define (metadata-semver-versions metadata)+ (map (lambda (p)+ (package-version p))+ (metadata-versions metadata)))++(define (metadata->package metadata version)+ (find+ (lambda (package)+ (equal? (package-version package) version))+ (metadata-versions metadata)))++(define (make-sexp metadata package dependencies)+ `(package+ (name ,(guix-name (metadata-name metadata)))+ (version ,(package-version package))+ (dependcies ,(map+ (match-lambda ((public-name dep)+ (list (guix-name (dependency-name dep)) public-name)))+ dependencies))))++(define (guix-name name)+ (string-append "test-" name))++(define packages+ `(("no-deps" . (("1.0.0" . ()) ("0.1.0" . ())))+ ("one-dep" . (("1.0.0" . (("no-deps" "^1.0")))+ ("0.1.0" . (("no-deps" "^0.1.0")))))+ ("shared-dep" . (("1.0.0" . (("one-dep" "^0.1.0")+ ("no-deps" "*")))))+ ("recursive" . (("1.0.0" . (("recursive" "=1.0.0")))))+ ("already-packaged" . (("1.0.0" . (("rust" "~1.28")))))))++(define (name->metadata name)+ (let ((versions (assoc-ref packages name)))+ (make-metadata name+ (map+ (match-lambda+ ((version . deps)+ (make-package version+ (map+ (lambda (name-range)+ (apply make-dependency name-range))+ deps))))+ versions))))++(define* (test-recursive-importer name version #:optional (guix-name guix-name))+ (recursive-import-semver #:name name+ #:version version+ #:name->metadata name->metadata+ #:metadata->package metadata->package+ #:metadata-versions metadata-semver-versions+ #:package-dependencies package-dependencies+ #:dependency-name dependency-name+ #:dependency-range dependency-range+ #:guix-name guix-name+ #:make-sexp make-sexp))++(test-equal "recursive import test with no dependencies"+ `((define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "no-deps" "1.0.0")))++(test-equal "recursive import test with one dependencies"+ `((define-public test-one-dep+ (package+ (name "test-one-dep")+ (version "1.0.0")+ (dependcies (("test-no-deps" "test-no-deps")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "1.0.0")))++(test-equal "recursive import test with recursuve dependencies"+ `((define-public test-recursive+ (package+ (name "test-recursive")+ (version "1.0.0")+ (dependcies (("test-recursive" "test-recursive"))))))+ (stream->list (test-recursive-importer "recursive" "1.0.0")))++(test-equal "recursive import test with no dependencies using an old version"+ `((define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "no-deps" "0.1.0")))++(test-equal "recursive import test with one dependencies unsing an old version"+ `((define-public test-one-dep-0.1.0+ (package+ (name "test-one-dep")+ (version "0.1.0")+ (dependcies (("test-no-deps" "test-no-deps-0.1.0")))))+ (define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "0.1.0")))++(test-equal "recursive import test with with dependency that is already in the repo"+ `((define-public test-already-packaged+ (package (name "test-already-packaged")+ (version "1.0.0")+ (dependcies+ (("test-rust" "rust-1.28"))))))+ (stream->list (test-recursive-importer "already-packaged" "1.0.0" identity)))++(test-equal "shared dependencies"+ `((define-public test-shared-dep+ (package+ (name "test-shared-dep")+ (version "1.0.0")+ (dependcies (("test-one-dep" "test-one-dep-0.1.0")+ ("test-no-deps" "test-no-deps")))))+ (define-public test-one-dep-0.1.0+ (package+ (name "test-one-dep")+ (version "0.1.0")+ (dependcies (("test-no-deps" "test-no-deps-0.1.0")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ())))+ (define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies()))))+ (stream->list (test-recursive-importer "shared-dep" "1.0.0")))+ (test-end "import-utils")-- 2.24.0
M
M
Martin Becze wrote on 5 Dec 2019 21:05
[PATCH v2 3/5] Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test.
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
7cccd4231b9c1327d59ff39daf2aa7dd47a2e765.1575575779.git.mjbecze@riseup.net
* guix/import/crate.scm (make-crate-sexp): Use <crate> <crate-version> as args* guix/import/crate.scm (crate->crate-version): New Procedure* guix/import/crate.scm (crate->versions): New Procedure* guix/import/crate.scm (crate-recursive-import): Updated to user recursive-import-semver* guix/scripts/import/crate.scm (guix-import-crate): Remove `define-public` generation from UI* guix/tests/crate.scm: Updated tests--- guix/import/crate.scm | 165 ++++++++++++++++++---------------- guix/scripts/import/crate.scm | 9 +- tests/crate.scm | 2 +- 3 files changed, 91 insertions(+), 85 deletions(-)
Toggle diff (258 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 8dc014d232..da92c43b8c 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -38,6 +38,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name crate-recursive-import@@ -85,7 +86,7 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string- (kind crate-dependency-kind "kind" ;'normal | 'dev+ (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string @@ -111,7 +112,9 @@ record or #f if it was not found." (url (string-append (%crate-base-url) path))) (match (assoc-ref (or (json-fetch url) '()) "dependencies") ((? vector? vector)- (map json->crate-dependency (vector->list vector)))+ (filter (lambda (dep)+ (not (eq? (crate-dependency-kind dep) 'dev)))+ (map json->crate-dependency (vector->list vector)))) (_ '())))) @@ -141,62 +144,84 @@ record or #f if it was not found." ((args ...) `((arguments (,'quasiquote ,args)))))) -(define* (make-crate-sexp #:key name version cargo-inputs cargo-development-inputs- home-page synopsis description license- #:allow-other-keys)- "Return the `package' s-expression for a rust package with the given NAME,-VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION,-and LICENSE."- (let* ((port (http-fetch (crate-uri name version)))+(define (make-crate-sexp crate version* dependencies)+ "Return the `package' s-expression for a rust package given <crate>,+ <crate-version> and a list of <crate-dependency>"+ (define normal-dependency?+ (match-lambda ((_ dep) (not (eq? (crate-dependency-kind dep) 'dev)))))++ (define (string->license string)+ (match (regexp-exec %dual-license-rx string)+ (#f (list (spdx-string->license string)))+ (m (list (spdx-string->license (match:substring m 1))+ (spdx-string->license (match:substring m 2))))))++ (let* ((dep-crates dev-dep-crates (partition normal-dependency? dependencies))+ (cargo-inputs (sort (unzip1 dep-crates)+ string-ci<?))+ (cargo-development-inputs+ (sort (unzip1 dev-dep-crates)+ string-ci<?))+ (name (crate-name crate))+ (version (crate-version-number version*))+ (home-page (or (crate-home-page crate)+ (crate-repository crate)))+ (synopsis (crate-description crate))+ (description (crate-description crate))+ (license (and=> (crate-version-license version*)+ string->license))+ (port (http-fetch (crate-uri name version)) ) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name- cargo-development-inputs)) (pkg `(package- (name ,guix-name)- (version ,version)- (source (origin- (method url-fetch)- (uri (crate-uri ,name version))- (file-name (string-append name "-" version ".tar.gz"))- (sha256- (base32- ,(bytevector->nix-base32-string (port-sha256 port))))))- (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)- (maybe-cargo-development-inputs- cargo-development-inputs)))- (home-page ,(match home-page- (() "")- (_ home-page)))- (synopsis ,synopsis)- (description ,(beautify-description description))- (license ,(match license- (() #f)- ((license) license)- (_ `(list ,@license)))))))- (close-port port)- pkg))+ (name ,guix-name)+ (version ,version)+ (source (origin+ (method url-fetch)+ (uri (crate-uri ,name version))+ (file-name (string-append name "-" version ".crate"))+ (sha256+ (base32+ ,(bytevector->nix-base32-string (port-sha256 port))))))+ (build-system cargo-build-system)+ ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ (maybe-cargo-development-inputs+ cargo-development-inputs)))+ (home-page ,(match home-page+ (() "")+ (_ home-page)))+ (synopsis ,synopsis)+ (description ,(beautify-description description))+ (license ,(match license+ (() #f)+ ((license) license)+ (_ `(list ,@license)))))))++ (close-port port)+ pkg)) (define %dual-license-rx ;; Dual licensing is represented by a string such as "MIT OR Apache-2.0". ;; This regexp matches that. (make-regexp "^(.*) OR (.*)$")) +(define (crate->crate-version crate version-number)+ "returns the <crate-version> for a given CRATE and VERSION-NUMBER"+ (find (lambda (version)+ (string=? (crate-version-number version)+ version-number))+ (crate-versions crate)))++(define (crate->versions crate)+ "Returns a list of versions for a given CRATE"+ (map (lambda (version)+ (crate-version-number version))+ (crate-versions crate)))+ (define* (crate->guix-package crate-name #:optional version) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME."- (define (string->license string)- (match (regexp-exec %dual-license-rx string)- (#f (list (spdx-string->license string)))- (m (list (spdx-string->license (match:substring m 1))- (spdx-string->license (match:substring m 2))))))-- (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))- (define crate (lookup-crate crate-name)) @@ -205,38 +230,27 @@ latest version of CRATE-NAME." (crate-latest-version crate))) (define version*- (find (lambda (version)- (string=? (crate-version-number version)- version-number))- (crate-versions crate)))+ (crate->crate-version crate version-number)) - (and crate version*- (let* ((dependencies (crate-version-dependencies version*))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))- (values- (make-crate-sexp #:name crate-name- #:version (crate-version-number version*)- #:cargo-inputs cargo-inputs- #:cargo-development-inputs cargo-development-inputs- #:home-page (or (crate-home-page crate)- (crate-repository crate))- #:synopsis (crate-description crate)- #:description (crate-description crate)- #:license (and=> (crate-version-license version*)- string->license))- (append cargo-inputs cargo-development-inputs)))))+ (define dependencies (map+ (lambda (dep)+ (list (crate-name->package-name+ (crate-dependency-id dep)) dep))+ (crate-version-dependencies version*)))+ (make-crate-sexp crate version* dependencies)) -(define (crate-recursive-import crate-name)- (recursive-import crate-name #f- #:repo->guix-package (lambda (name repo)- (crate->guix-package name))- #:guix-name crate-name->package-name))+(define* (crate-recursive-import name #:optional version)+ (recursive-import-semver+ #:name name+ #:version version+ #:name->metadata lookup-crate+ #:metadata->package crate->crate-version+ #:metadata-versions crate->versions+ #:package-dependencies crate-version-dependencies+ #:dependency-name crate-dependency-id+ #:dependency-range crate-dependency-requirement+ #:guix-name crate-name->package-name+ #:make-sexp make-crate-sexp)) (define (guix-package->crate-name package) "Return the crate name of PACKAGE."@@ -285,4 +299,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release)))-diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex 4690cceb4d..85ae6fbe59 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -96,14 +96,7 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (reverse- (stream->list- (crate-recursive-import name))))+ (stream->list (crate-recursive-import name version)) (let ((sexp (crate->guix-package name version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%")diff --git a/tests/crate.scm b/tests/crate.scmindex c14862ad9f..b77cbb08c6 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -95,7 +95,7 @@ ('source ('origin ('method 'url-fetch) ('uri ('crate-uri "foo" 'version))- ('file-name ('string-append 'name "-" 'version ".tar.gz"))+ ('file-name ('string-append 'name "-" 'version ".crate")) ('sha256 ('base32 (? string? hash)))))-- 2.24.0
M
M
Martin Becze wrote on 5 Dec 2019 21:05
[PATCH v2 5/5] guix: crate: Depublicated build and normal dependencies
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
8ffe3fa2e5d612802069f2b214b03a66df05780b.1575575779.git.mjbecze@riseup.net
* guix/import/crate.scm: (crate-version-dependencies): dedup deps--- guix/import/crate.scm | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)
Toggle diff (41 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 5683369b7a..f3c36ba516 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -109,14 +109,26 @@ record or #f if it was not found." "Return the list of <crate-dependency> records of VERSION, a <crate-version>." (let* ((path (assoc-ref (crate-version-links version) "dependencies"))- (url (string-append (%crate-base-url) path)))- (match (assoc-ref (or (json-fetch url) '()) "dependencies")- ((? vector? vector)- (filter (lambda (dep)- (not (eq? (crate-dependency-kind dep) 'dev)))- (map json->crate-dependency (vector->list vector))))- (_- '()))))+ (url (string-append (%crate-base-url) path))+ (deps-list (match (assoc-ref (or (json-fetch url) '()) "dependencies")+ ((? vector? vector) (vector->list vector))+ (_+ '())))+ ;; turn the raw list into <dependency>'s and remove dev depenedencies+ (deps (filter-map (lambda (json)+ (let ((dep (json->crate-dependency json)))+ (if (eq? (crate-dependency-kind dep) 'dev)+ #f+ dep)))+ deps-list))+ ;; split normal and build dependencies+ (deps-normal deps-build (partition (lambda (dep)+ (eq? (crate-dependency-kind dep) 'normal))+ deps)))+ ;;remove duplicate normal and build dependencies+ (lset-union (lambda (a b)+ (string= (crate-dependency-id a) (crate-dependency-id a)))+ deps-normal deps-build))) ;;;-- 2.24.0
M
M
Martin Becze wrote on 6 Dec 2019 19:21
[PATCH v3 0/5] Semantic version aware recusive importer for crates
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
cover.1575656092.git.mjbecze@riseup.net
This version makes one little change from the prevous version (https://issues.guix.gnu.org/issue/38408#13).I found out that crates could have duplicate dependencies in for every possible target in their Cargo.toml. So just depuplicating the build dependencies agains the normal depedencies was not enough. We need to make sure all the dependencies are unqie!
-Martin
Martin Becze (5): gnu: added new function, find-packages-by-name*/direct gnu: added new procedure, recusive-import-semver Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test. added "#:skip-build? #t" to the output of (make-crate-sexp). Most the the packages imported will be libaries and won't need to build. The top level package will build them though. guix: crate: Depublicated dependencies
gnu/packages.scm | 41 ++++++++ guix/import/crate.scm | 188 +++++++++++++++++++--------------- guix/import/utils.scm | 181 ++++++++++++++++++++++++++++++-- guix/scripts/import/crate.scm | 9 +- tests/crate.scm | 5 +- tests/import-utils.scm | 162 +++++++++++++++++++++++++++++ tests/packages.scm | 13 +++ 7 files changed, 501 insertions(+), 98 deletions(-)
-- 2.24.0
M
M
Martin Becze wrote on 6 Dec 2019 19:21
[PATCH v3 1/5] gnu: added new function, find-packages-by-name*/direct
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
dbe5240d37d3b42a6c1db1dfc07436bff5ed2297.1575656092.git.mjbecze@riseup.net
* gnu/packages.scm (find-packages-by-naem*/direct)--- gnu/packages.scm | 41 +++++++++++++++++++++++++++++++++++++++++ tests/packages.scm | 13 +++++++++++++ 2 files changed, 54 insertions(+)
Toggle diff (112 lines)diff --git a/gnu/packages.scm b/gnu/packages.scmindex 959777ff8f..cca2a393e5 100644--- a/gnu/packages.scm+++ b/gnu/packages.scm@@ -4,6 +4,7 @@ ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -52,7 +53,9 @@ %default-package-module-path fold-packages+ fold-packages* fold-available-packages+ find-packages-by-name*/direct find-newest-available-packages find-packages-by-name@@ -250,6 +253,23 @@ is guaranteed to never traverse the same package twice." init modules)) +(define* (fold-packages* proc init+ #:optional+ (modules (all-modules (%package-module-path)+ #:warn+ warn-about-load-error))+ #:key (select? (negate hidden-package?)))+ "Call (PROC PACKAGE RESULT) for each available package defined in one of+MODULES that matches SELECT?, using INIT as the initial value of RESULT. It+is guaranteed to never traverse the same package twice."+ (fold-module-public-variables* (lambda (module symbol var result)+ (let ((object (variable-ref var)))+ (if (and (package? object) (select? object))+ (proc module symbol object result)+ result)))+ init+ modules))+ (define %package-cache-file ;; Location of the package cache. "/lib/guix/package.cache")@@ -297,6 +317,27 @@ decreasing version order." matching) matching))))) +(define find-packages-by-name*/direct ;bypass the cache+ (let ((packages (delay+ (fold-packages* (lambda (mod sym p r)+ (vhash-cons (package-name p) (list mod sym p) r))+ vlist-null)))+ (version>? (match-lambda*+ (((_ _ versions) ..1)+ (apply version>? (map package-version versions))))))+ (lambda* (name #:optional version)+ "Return the list of (<module> <symbol> <package>) with the given NAME. If+ VERSION is not #f, then only return packages whose version is prefixed by+ VERSION, sorted in decreasing version order."+ (let ((matching (sort (vhash-fold* cons '() name (force packages))+ version>?)))+ (if version+ (filter (match-lambda+ ((_ _ package)+ (version-prefix? version (package-version package))))+ matching)+ matching)))))+ (define (cache-lookup cache name) "Lookup package NAME in CACHE. Return a list sorted in increasing version order."diff --git a/tests/packages.scm b/tests/packages.scmindex 423c5061aa..9f02b0d5d2 100644--- a/tests/packages.scm+++ b/tests/packages.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © Jan (janneke) Nieuwenhuizen <janneke@gnu.org>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -1135,11 +1136,23 @@ (((? (cut eq? hello <>))) #t) (wrong (pk 'find-packages-by-name wrong #f)))) +(test-assert "find-packages-by-name*/direct"+ (match (find-packages-by-name*/direct "hello")+ ((((? (cut eq? (resolve-interface '(gnu packages base)) <>))+ (? (cut eq? 'hello <>))+ (? (cut eq? hello <>)))) #t)))+ (test-assert "find-packages-by-name with version" (match (find-packages-by-name "hello" (package-version hello)) (((? (cut eq? hello <>))) #t) (wrong (pk 'find-packages-by-name wrong #f)))) +(test-assert "find-packages-by-name*/direct with version"+ (match (find-packages-by-name*/direct "hello" (package-version hello))+ ((((? (cut eq? (resolve-interface '(gnu packages base)) <>))+ (? (cut eq? 'hello <>))+ (? (cut eq? hello <>)))) #t)))+ (test-equal "find-packages-by-name with cache" (find-packages-by-name "guile") (call-with-temporary-directory-- 2.24.0
M
M
Martin Becze wrote on 6 Dec 2019 19:21
[PATCH v3 2/5] gnu: added new procedure, recusive-import-semver
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
18d8555e374e1d7682c5f7231be25dd572dacbf0.1575656092.git.mjbecze@riseup.net
* gnu/packages.scm (recusive-import-semver): New Procedure* gnu/packages.scm (package->definition)[arguments]: New argument, "latest"* tests/import-utils.scm: tests for recusive-import-semver--- guix/import/utils.scm | 181 +++++++++++++++++++++++++++++++++++++++-- tests/import-utils.scm | 162 ++++++++++++++++++++++++++++++++++++ 2 files changed, 336 insertions(+), 7 deletions(-)
Toggle diff (412 lines)diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 4694b6e7ef..6932614f8e 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2019 Robert Vollmert <rob@vllmrt.net>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -32,6 +33,7 @@ #:use-module (guix discovery) #:use-module (guix build-system) #:use-module (guix gexp)+ #:use-module (guix memoization) #:use-module (guix store) #:use-module (guix download) #:use-module (gnu packages)@@ -43,6 +45,8 @@ #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-41)+ #:use-module (semver)+ #:use-module (semver ranges) #:export (factorize-uri flatten@@ -69,7 +73,8 @@ guix-name - recursive-import))+ recursive-import+ recursive-import-semver)) (define (factorize-uri uri version) "Factorize URI, a package tarball URI as a string, such that any occurrences@@ -257,13 +262,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package)+(define* (package->definition guix-package #:optional (latest #t)) (match guix-package- (('package ('name (? string? name)) _ ...)- `(define-public ,(string->symbol name)- ,guix-package))- (('let anything ('package ('name (? string? name)) _ ...))- `(define-public ,(string->symbol name)+ ((or+ ('package ('name name) ('version version) . rest)+ ('let _ ('package ('name name) ('version version) . rest)))++ `(define-public ,(string->symbol (if latest+ name+ (string-append name "-" version))) ,guix-package)))) (define (build-system-modules)@@ -414,3 +421,163 @@ dependencies." step ;; initial state (step initial-state)))++(define* (recursive-import-semver #:key name+ (version #f)+ name->metadata+ metadata->package+ metadata-versions+ package-dependencies+ dependency-name+ dependency-range+ guix-name+ make-sexp)+ "Generates a stream of package expressions for the dependencies of the given +NAME and VERSION. The dependencies will be resolved using semantic versioning.+This procedure makes the assumption that most package repositories will, for a+given package provide some <metadata> on that package that includes what+versions of the package that are available and a list of dependencies for each+version. Dependencies are assumed to be composed of a NAME, a semantic RANGE and+other data.++This procedure takes the following keys:+ NAME - The name of the package to import+ VERSION - The version of the package to import+ NAME->METADATA - A procedure that takes a NAME of a package and returns that+package's <metadata>+ METADATA->PACKAGE A procedure that takes a package's <metadata> and VERSION +and returns the <package> for the given VERSION+ METADATA-VERSIONS A procedure that that takes a packages <metadata> and+returns a list of version as strings that are available for the given package+ PACKAGE-DEPENDENCIES a procedure that returns a list of <dependency> given a +<package>+ DEPENDENCY-NAME A procedure that takes a <dependency> and returns the its name+ DEPENDENCY-RANGE A procedure that takes a <dependency> and returns that+decency's range as a string+ GUIX-NAME A procedure that take a NAME and returns the Guix version of it+ MAKE-SEXP A procedure that takes <metadata>, <package> and a list of pairs+containing (EXPORT-NAME <dependency>), returning the package expression as an +s-expression"+ (define mem-name->metadata (memoize name->metadata))++ (define (latest? versions version)+ (equal? (car versions) version))++ (define (sorted-versions metadata)+ (sort (metadata-versions metadata) version>?))++ (define (name->versions name)+ (sorted-versions (mem-name->metadata name)))++ (define (semver-range-contains-string? range version)+ (semver-range-contains? range+ (string->semver version)))++ (define (guix-export-name name version)+ (let ((versions (name->versions name))+ (name (guix-name name)))+ (if (latest? versions version)+ name+ (string-append name "-" version))))++ ;; checks to see if we already defined or want to define a dep+ (define (find-known name range known)+ (match+ (find+ (match-lambda ((dep-name version)+ (and+ (string=? dep-name name)+ (semver-range-contains-string? range version))))+ known)++ (#f #f)+ ((name version) (list (guix-export-name name version) version #f)))+ )++ ;; searches searches for a package in guix+ (define (find-locally name range)+ (match+ (find+ (match-lambda ((_ _ package)+ (semver-range-contains-string?+ range+ (package-version package))))+ (find-packages-by-name*/direct (guix-name name)))+ (#f #f)+ ((_ export-symbol package) (list+ (symbol->string export-symbol)+ (package-version package) #f))))++ ;; searches for a package in some external repo+ (define (find-remote name range)+ (let* ((versions (name->versions name))+ (version (find+ (lambda (ver)+ (semver-range-contains-string? range ver))+ versions))+ (export-name (guix-export-name name version)))+ `(,export-name ,version #t)))+++ (define (find-dep-version dep known-deps)+ (let* ((name (dependency-name dep))+ (range (string->semver-range (dependency-range dep)))+ (export-name-version-needed+ (or (find-known name range known-deps)+ (find-locally name range)+ (find-remote name range))))+ `(,name ,@export-name-version-needed ,dep)+ ))++ (define (make-package-definition name version known-deps)+ (let* ((metadata (mem-name->metadata name))+ (versions (sorted-versions metadata))+ (package (metadata->package metadata version))+ (deps (map (lambda (dep)+ (find-dep-version dep known-deps))+ (package-dependencies package)))+ (sexp+ (make-sexp metadata package+ (map+ (match-lambda ((_ export-symbol _ _ dep)+ (list export-symbol dep)))+ deps))))+ (values+ (package->definition sexp (latest? versions version))+ (filter-map+ (match-lambda ((name _ version need? dep)+ (if need?+ (list name version)+ #f)))+ deps))))++ (define initial-state+ (list #f+ (list+ ;; packages to find+ (list name (if version+ version+ (car (name->versions name)))))+ ;; packages that have been found+ (list)))++ (define (step state)+ (match state+ ((prev ((next-name next-version) . rest) done)+ (receive (package dependencies)+ (make-package-definition next-name next-version+ (append done rest `((,next-name ,next-version))))+ (list+ package+ (append rest dependencies)+ (cons (list next-name next-version) done))))+ ((prev '() done)+ (list #f '() done))))+ + (stream-unfold+ ;; map: produce a stream element+ (match-lambda ((latest queue done) latest))+ ;; predicate+ (match-lambda ((latest queue done) latest))+ step+ (step initial-state)))diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex c3ab25d788..4ed3a5e1da 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>+;;; Copyright © 2016 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -24,6 +25,10 @@ #:use-module (guix packages) #:use-module (guix build-system) #:use-module (gnu packages)+ #:use-module (ice-9 match)+ #:use-module (srfi srfi-1)+ #:use-module (srfi srfi-9)+ #:use-module (srfi srfi-41) #:use-module (srfi srfi-64)) (test-begin "import-utils")@@ -120,4 +125,161 @@ ("license" . #f)))) (package-native-inputs (alist->package meta)))) +(define-record-type <metadata>+ (make-metadata name versions)+ metadata?+ (name metadata-name)+ (versions metadata-versions))++(define-record-type <package>+ (make-package version dependencies)+ package?+ (version package-version)+ (dependencies package-dependencies))++(define-record-type <dependency>+ (make-dependency name range)+ dependency?+ (name dependency-name)+ (range dependency-range))++(define (metadata-semver-versions metadata)+ (map (lambda (p)+ (package-version p))+ (metadata-versions metadata)))++(define (metadata->package metadata version)+ (find+ (lambda (package)+ (equal? (package-version package) version))+ (metadata-versions metadata)))++(define (make-sexp metadata package dependencies)+ `(package+ (name ,(guix-name (metadata-name metadata)))+ (version ,(package-version package))+ (dependcies ,(map+ (match-lambda ((public-name dep)+ (list (guix-name (dependency-name dep)) public-name)))+ dependencies))))++(define (guix-name name)+ (string-append "test-" name))++(define packages+ `(("no-deps" . (("1.0.0" . ()) ("0.1.0" . ())))+ ("one-dep" . (("1.0.0" . (("no-deps" "^1.0")))+ ("0.1.0" . (("no-deps" "^0.1.0")))))+ ("shared-dep" . (("1.0.0" . (("one-dep" "^0.1.0")+ ("no-deps" "*")))))+ ("recursive" . (("1.0.0" . (("recursive" "=1.0.0")))))+ ("already-packaged" . (("1.0.0" . (("rust" "~1.28")))))))++(define (name->metadata name)+ (let ((versions (assoc-ref packages name)))+ (make-metadata name+ (map+ (match-lambda+ ((version . deps)+ (make-package version+ (map+ (lambda (name-range)+ (apply make-dependency name-range))+ deps))))+ versions))))++(define* (test-recursive-importer name version #:optional (guix-name guix-name))+ (recursive-import-semver #:name name+ #:version version+ #:name->metadata name->metadata+ #:metadata->package metadata->package+ #:metadata-versions metadata-semver-versions+ #:package-dependencies package-dependencies+ #:dependency-name dependency-name+ #:dependency-range dependency-range+ #:guix-name guix-name+ #:make-sexp make-sexp))++(test-equal "recursive import test with no dependencies"+ `((define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "no-deps" "1.0.0")))++(test-equal "recursive import test with one dependencies"+ `((define-public test-one-dep+ (package+ (name "test-one-dep")+ (version "1.0.0")+ (dependcies (("test-no-deps" "test-no-deps")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "1.0.0")))++(test-equal "recursive import test with recursuve dependencies"+ `((define-public test-recursive+ (package+ (name "test-recursive")+ (version "1.0.0")+ (dependcies (("test-recursive" "test-recursive"))))))+ (stream->list (test-recursive-importer "recursive" "1.0.0")))++(test-equal "recursive import test with no dependencies using an old version"+ `((define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "no-deps" "0.1.0")))++(test-equal "recursive import test with one dependencies unsing an old version"+ `((define-public test-one-dep-0.1.0+ (package+ (name "test-one-dep")+ (version "0.1.0")+ (dependcies (("test-no-deps" "test-no-deps-0.1.0")))))+ (define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "0.1.0")))++(test-equal "recursive import test with with dependency that is already in the repo"+ `((define-public test-already-packaged+ (package (name "test-already-packaged")+ (version "1.0.0")+ (dependcies+ (("test-rust" "rust-1.28"))))))+ (stream->list (test-recursive-importer "already-packaged" "1.0.0" identity)))++(test-equal "shared dependencies"+ `((define-public test-shared-dep+ (package+ (name "test-shared-dep")+ (version "1.0.0")+ (dependcies (("test-one-dep" "test-one-dep-0.1.0")+ ("test-no-deps" "test-no-deps")))))+ (define-public test-one-dep-0.1.0+ (package+ (name "test-one-dep")+ (version "0.1.0")+ (dependcies (("test-no-deps" "test-no-deps-0.1.0")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ())))+ (define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies()))))+ (stream->list (test-recursive-importer "shared-dep" "1.0.0")))+ (test-end "import-utils")-- 2.24.0
M
M
Martin Becze wrote on 6 Dec 2019 19:21
[PATCH v3 3/5] Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test.
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
7cccd4231b9c1327d59ff39daf2aa7dd47a2e765.1575656092.git.mjbecze@riseup.net
* guix/import/crate.scm (make-crate-sexp): Use <crate> <crate-version> as args* guix/import/crate.scm (crate->crate-version): New Procedure* guix/import/crate.scm (crate->versions): New Procedure* guix/import/crate.scm (crate-recursive-import): Updated to user recursive-import-semver* guix/scripts/import/crate.scm (guix-import-crate): Remove `define-public` generation from UI* guix/tests/crate.scm: Updated tests--- guix/import/crate.scm | 165 ++++++++++++++++++---------------- guix/scripts/import/crate.scm | 9 +- tests/crate.scm | 2 +- 3 files changed, 91 insertions(+), 85 deletions(-)
Toggle diff (258 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 8dc014d232..da92c43b8c 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -38,6 +38,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name crate-recursive-import@@ -85,7 +86,7 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string- (kind crate-dependency-kind "kind" ;'normal | 'dev+ (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string @@ -111,7 +112,9 @@ record or #f if it was not found." (url (string-append (%crate-base-url) path))) (match (assoc-ref (or (json-fetch url) '()) "dependencies") ((? vector? vector)- (map json->crate-dependency (vector->list vector)))+ (filter (lambda (dep)+ (not (eq? (crate-dependency-kind dep) 'dev)))+ (map json->crate-dependency (vector->list vector)))) (_ '())))) @@ -141,62 +144,84 @@ record or #f if it was not found." ((args ...) `((arguments (,'quasiquote ,args)))))) -(define* (make-crate-sexp #:key name version cargo-inputs cargo-development-inputs- home-page synopsis description license- #:allow-other-keys)- "Return the `package' s-expression for a rust package with the given NAME,-VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION,-and LICENSE."- (let* ((port (http-fetch (crate-uri name version)))+(define (make-crate-sexp crate version* dependencies)+ "Return the `package' s-expression for a rust package given <crate>,+ <crate-version> and a list of <crate-dependency>"+ (define normal-dependency?+ (match-lambda ((_ dep) (not (eq? (crate-dependency-kind dep) 'dev)))))++ (define (string->license string)+ (match (regexp-exec %dual-license-rx string)+ (#f (list (spdx-string->license string)))+ (m (list (spdx-string->license (match:substring m 1))+ (spdx-string->license (match:substring m 2))))))++ (let* ((dep-crates dev-dep-crates (partition normal-dependency? dependencies))+ (cargo-inputs (sort (unzip1 dep-crates)+ string-ci<?))+ (cargo-development-inputs+ (sort (unzip1 dev-dep-crates)+ string-ci<?))+ (name (crate-name crate))+ (version (crate-version-number version*))+ (home-page (or (crate-home-page crate)+ (crate-repository crate)))+ (synopsis (crate-description crate))+ (description (crate-description crate))+ (license (and=> (crate-version-license version*)+ string->license))+ (port (http-fetch (crate-uri name version)) ) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name- cargo-development-inputs)) (pkg `(package- (name ,guix-name)- (version ,version)- (source (origin- (method url-fetch)- (uri (crate-uri ,name version))- (file-name (string-append name "-" version ".tar.gz"))- (sha256- (base32- ,(bytevector->nix-base32-string (port-sha256 port))))))- (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)- (maybe-cargo-development-inputs- cargo-development-inputs)))- (home-page ,(match home-page- (() "")- (_ home-page)))- (synopsis ,synopsis)- (description ,(beautify-description description))- (license ,(match license- (() #f)- ((license) license)- (_ `(list ,@license)))))))- (close-port port)- pkg))+ (name ,guix-name)+ (version ,version)+ (source (origin+ (method url-fetch)+ (uri (crate-uri ,name version))+ (file-name (string-append name "-" version ".crate"))+ (sha256+ (base32+ ,(bytevector->nix-base32-string (port-sha256 port))))))+ (build-system cargo-build-system)+ ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ (maybe-cargo-development-inputs+ cargo-development-inputs)))+ (home-page ,(match home-page+ (() "")+ (_ home-page)))+ (synopsis ,synopsis)+ (description ,(beautify-description description))+ (license ,(match license+ (() #f)+ ((license) license)+ (_ `(list ,@license)))))))++ (close-port port)+ pkg)) (define %dual-license-rx ;; Dual licensing is represented by a string such as "MIT OR Apache-2.0". ;; This regexp matches that. (make-regexp "^(.*) OR (.*)$")) +(define (crate->crate-version crate version-number)+ "returns the <crate-version> for a given CRATE and VERSION-NUMBER"+ (find (lambda (version)+ (string=? (crate-version-number version)+ version-number))+ (crate-versions crate)))++(define (crate->versions crate)+ "Returns a list of versions for a given CRATE"+ (map (lambda (version)+ (crate-version-number version))+ (crate-versions crate)))+ (define* (crate->guix-package crate-name #:optional version) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME."- (define (string->license string)- (match (regexp-exec %dual-license-rx string)- (#f (list (spdx-string->license string)))- (m (list (spdx-string->license (match:substring m 1))- (spdx-string->license (match:substring m 2))))))-- (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))- (define crate (lookup-crate crate-name)) @@ -205,38 +230,27 @@ latest version of CRATE-NAME." (crate-latest-version crate))) (define version*- (find (lambda (version)- (string=? (crate-version-number version)- version-number))- (crate-versions crate)))+ (crate->crate-version crate version-number)) - (and crate version*- (let* ((dependencies (crate-version-dependencies version*))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))- (values- (make-crate-sexp #:name crate-name- #:version (crate-version-number version*)- #:cargo-inputs cargo-inputs- #:cargo-development-inputs cargo-development-inputs- #:home-page (or (crate-home-page crate)- (crate-repository crate))- #:synopsis (crate-description crate)- #:description (crate-description crate)- #:license (and=> (crate-version-license version*)- string->license))- (append cargo-inputs cargo-development-inputs)))))+ (define dependencies (map+ (lambda (dep)+ (list (crate-name->package-name+ (crate-dependency-id dep)) dep))+ (crate-version-dependencies version*)))+ (make-crate-sexp crate version* dependencies)) -(define (crate-recursive-import crate-name)- (recursive-import crate-name #f- #:repo->guix-package (lambda (name repo)- (crate->guix-package name))- #:guix-name crate-name->package-name))+(define* (crate-recursive-import name #:optional version)+ (recursive-import-semver+ #:name name+ #:version version+ #:name->metadata lookup-crate+ #:metadata->package crate->crate-version+ #:metadata-versions crate->versions+ #:package-dependencies crate-version-dependencies+ #:dependency-name crate-dependency-id+ #:dependency-range crate-dependency-requirement+ #:guix-name crate-name->package-name+ #:make-sexp make-crate-sexp)) (define (guix-package->crate-name package) "Return the crate name of PACKAGE."@@ -285,4 +299,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release)))-diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex 4690cceb4d..85ae6fbe59 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -96,14 +96,7 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (reverse- (stream->list- (crate-recursive-import name))))+ (stream->list (crate-recursive-import name version)) (let ((sexp (crate->guix-package name version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%")diff --git a/tests/crate.scm b/tests/crate.scmindex c14862ad9f..b77cbb08c6 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -95,7 +95,7 @@ ('source ('origin ('method 'url-fetch) ('uri ('crate-uri "foo" 'version))- ('file-name ('string-append 'name "-" 'version ".tar.gz"))+ ('file-name ('string-append 'name "-" 'version ".crate")) ('sha256 ('base32 (? string? hash)))))-- 2.24.0
M
M
Martin Becze wrote on 6 Dec 2019 19:21
[PATCH v3 4/5] added "#:skip-build? #t" to the output of (make-crate-sexp). Most the the packages imported will be libaries and won't need to build. The top level package will build them though.
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
bf327fb427eba14f6d4d8d324c5c75135f39e79b.1575656092.git.mjbecze@riseup.net
* guix/import/crate.scm (make-crate-sexp): added "#:skip-build? #t" to the output--- guix/import/crate.scm | 3 ++- tests/crate.scm | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)
Toggle diff (30 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex da92c43b8c..5683369b7a 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -183,7 +183,8 @@ record or #f if it was not found." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ ,@(maybe-arguments (append `(#:skip-build? #t)+ (maybe-cargo-inputs cargo-inputs) (maybe-cargo-development-inputs cargo-development-inputs))) (home-page ,(match home-pagediff --git a/tests/crate.scm b/tests/crate.scmindex b77cbb08c6..64e5b6932e 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -102,7 +102,8 @@ ('build-system 'cargo-build-system) ('arguments ('quasiquote- ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))+ ('#:skip-build? #t+ #:cargo-inputs (("rust-bar" ('unquote rust-bar)))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary")-- 2.24.0
M
M
Martin Becze wrote on 6 Dec 2019 19:21
[PATCH v3 5/5] guix: crate: Depublicated dependencies
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
b756799cd226d1f76577e683d1d0821acff7de99.1575656092.git.mjbecze@riseup.net
* guix/import/crate.scm: (crate-version-dependencies): dedup deps--- guix/import/crate.scm | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)
Toggle diff (41 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 5683369b7a..f3c36ba516 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -109,14 +109,26 @@ record or #f if it was not found." "Return the list of <crate-dependency> records of VERSION, a <crate-version>." (let* ((path (assoc-ref (crate-version-links version) "dependencies"))- (url (string-append (%crate-base-url) path)))- (match (assoc-ref (or (json-fetch url) '()) "dependencies")- ((? vector? vector)- (filter (lambda (dep)- (not (eq? (crate-dependency-kind dep) 'dev)))- (map json->crate-dependency (vector->list vector))))- (_- '()))))+ (url (string-append (%crate-base-url) path))+ (deps-list (match (assoc-ref (or (json-fetch url) '()) "dependencies")+ ((? vector? vector) (vector->list vector))+ (_+ '())))+ ;; turn the raw list into <dependency>'s and remove dev depenedencies+ (deps (filter-map (lambda (json)+ (let ((dep (json->crate-dependency json)))+ (if (eq? (crate-dependency-kind dep) 'dev)+ #f+ dep)))+ deps-list))+ ;; split normal and build dependencies+ (deps-normal deps-build (partition (lambda (dep)+ (eq? (crate-dependency-kind dep) 'normal))+ deps)))+ ;;remove duplicate normal and build dependencies+ (lset-union (lambda (a b)+ (string= (crate-dependency-id a) (crate-dependency-id a)))+ deps-normal deps-build))) ;;;-- 2.24.0
M
M
Martin Becze wrote on 10 Dec 2019 20:23
[PATCH v4 0/6] Semantic version aware recusive importer for crates
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
cover.1576005195.git.mjbecze@riseup.net
Hi Guix,This version adds a feature. You can an import a crate using a semantive version range. ie. "guix import crate -r ripgrep@^1". Let me know if you have any suggetions or stuff to fix on this patch! Thanks.
-Martin
Martin Becze (6): gnu: added new function, find-packages-by-name*/direct gnu: added new procedure, recusive-import-semver Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test. added "#:skip-build? #t" to the output of (make-crate-sexp). Most the the packages imported will be libaries and won't need to build. The top level package will build them though. guix: crate: Depublicated dependencies guix: import: recursive-import-semver: allow the range of a package to be specified when begining import.
gnu/packages.scm | 41 ++++++++ guix/import/crate.scm | 186 +++++++++++++++++--------------- guix/import/utils.scm | 192 ++++++++++++++++++++++++++++++++-- guix/scripts/import/crate.scm | 9 +- tests/crate.scm | 5 +- tests/import-utils.scm | 175 +++++++++++++++++++++++++++++++ tests/packages.scm | 13 +++ 7 files changed, 522 insertions(+), 99 deletions(-)
-- 2.24.0
M
M
Martin Becze wrote on 10 Dec 2019 20:23
[PATCH v4 1/6] gnu: added new function, find-packages-by-name*/direct
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
d83466c676763462ce625fe04df176c7681fc836.1576005195.git.mjbecze@riseup.net
* gnu/packages.scm (find-packages-by-naem*/direct)--- gnu/packages.scm | 41 +++++++++++++++++++++++++++++++++++++++++ tests/packages.scm | 13 +++++++++++++ 2 files changed, 54 insertions(+)
Toggle diff (112 lines)diff --git a/gnu/packages.scm b/gnu/packages.scmindex 959777ff8f..cca2a393e5 100644--- a/gnu/packages.scm+++ b/gnu/packages.scm@@ -4,6 +4,7 @@ ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -52,7 +53,9 @@ %default-package-module-path fold-packages+ fold-packages* fold-available-packages+ find-packages-by-name*/direct find-newest-available-packages find-packages-by-name@@ -250,6 +253,23 @@ is guaranteed to never traverse the same package twice." init modules)) +(define* (fold-packages* proc init+ #:optional+ (modules (all-modules (%package-module-path)+ #:warn+ warn-about-load-error))+ #:key (select? (negate hidden-package?)))+ "Call (PROC PACKAGE RESULT) for each available package defined in one of+MODULES that matches SELECT?, using INIT as the initial value of RESULT. It+is guaranteed to never traverse the same package twice."+ (fold-module-public-variables* (lambda (module symbol var result)+ (let ((object (variable-ref var)))+ (if (and (package? object) (select? object))+ (proc module symbol object result)+ result)))+ init+ modules))+ (define %package-cache-file ;; Location of the package cache. "/lib/guix/package.cache")@@ -297,6 +317,27 @@ decreasing version order." matching) matching))))) +(define find-packages-by-name*/direct ;bypass the cache+ (let ((packages (delay+ (fold-packages* (lambda (mod sym p r)+ (vhash-cons (package-name p) (list mod sym p) r))+ vlist-null)))+ (version>? (match-lambda*+ (((_ _ versions) ..1)+ (apply version>? (map package-version versions))))))+ (lambda* (name #:optional version)+ "Return the list of (<module> <symbol> <package>) with the given NAME. If+ VERSION is not #f, then only return packages whose version is prefixed by+ VERSION, sorted in decreasing version order."+ (let ((matching (sort (vhash-fold* cons '() name (force packages))+ version>?)))+ (if version+ (filter (match-lambda+ ((_ _ package)+ (version-prefix? version (package-version package))))+ matching)+ matching)))))+ (define (cache-lookup cache name) "Lookup package NAME in CACHE. Return a list sorted in increasing version order."diff --git a/tests/packages.scm b/tests/packages.scmindex 423c5061aa..9f02b0d5d2 100644--- a/tests/packages.scm+++ b/tests/packages.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © Jan (janneke) Nieuwenhuizen <janneke@gnu.org>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -1135,11 +1136,23 @@ (((? (cut eq? hello <>))) #t) (wrong (pk 'find-packages-by-name wrong #f)))) +(test-assert "find-packages-by-name*/direct"+ (match (find-packages-by-name*/direct "hello")+ ((((? (cut eq? (resolve-interface '(gnu packages base)) <>))+ (? (cut eq? 'hello <>))+ (? (cut eq? hello <>)))) #t)))+ (test-assert "find-packages-by-name with version" (match (find-packages-by-name "hello" (package-version hello)) (((? (cut eq? hello <>))) #t) (wrong (pk 'find-packages-by-name wrong #f)))) +(test-assert "find-packages-by-name*/direct with version"+ (match (find-packages-by-name*/direct "hello" (package-version hello))+ ((((? (cut eq? (resolve-interface '(gnu packages base)) <>))+ (? (cut eq? 'hello <>))+ (? (cut eq? hello <>)))) #t)))+ (test-equal "find-packages-by-name with cache" (find-packages-by-name "guile") (call-with-temporary-directory-- 2.24.0
M
M
Martin Becze wrote on 10 Dec 2019 20:23
[PATCH v4 2/6] gnu: added new procedure, recusive-import-semver
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
5b82d5ab2c9d579d279db055b8f12509587234fa.1576005195.git.mjbecze@riseup.net
* gnu/packages.scm (recusive-import-semver): New Procedure* gnu/packages.scm (package->definition)[arguments]: New argument, "latest"* tests/import-utils.scm: tests for recusive-import-semver--- guix/import/utils.scm | 181 +++++++++++++++++++++++++++++++++++++++-- tests/import-utils.scm | 162 ++++++++++++++++++++++++++++++++++++ 2 files changed, 336 insertions(+), 7 deletions(-)
Toggle diff (412 lines)diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 4694b6e7ef..6932614f8e 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2019 Robert Vollmert <rob@vllmrt.net>+;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -32,6 +33,7 @@ #:use-module (guix discovery) #:use-module (guix build-system) #:use-module (guix gexp)+ #:use-module (guix memoization) #:use-module (guix store) #:use-module (guix download) #:use-module (gnu packages)@@ -43,6 +45,8 @@ #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-41)+ #:use-module (semver)+ #:use-module (semver ranges) #:export (factorize-uri flatten@@ -69,7 +73,8 @@ guix-name - recursive-import))+ recursive-import+ recursive-import-semver)) (define (factorize-uri uri version) "Factorize URI, a package tarball URI as a string, such that any occurrences@@ -257,13 +262,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package)+(define* (package->definition guix-package #:optional (latest #t)) (match guix-package- (('package ('name (? string? name)) _ ...)- `(define-public ,(string->symbol name)- ,guix-package))- (('let anything ('package ('name (? string? name)) _ ...))- `(define-public ,(string->symbol name)+ ((or+ ('package ('name name) ('version version) . rest)+ ('let _ ('package ('name name) ('version version) . rest)))++ `(define-public ,(string->symbol (if latest+ name+ (string-append name "-" version))) ,guix-package)))) (define (build-system-modules)@@ -414,3 +421,163 @@ dependencies." step ;; initial state (step initial-state)))++(define* (recursive-import-semver #:key name+ (version #f)+ name->metadata+ metadata->package+ metadata-versions+ package-dependencies+ dependency-name+ dependency-range+ guix-name+ make-sexp)+ "Generates a stream of package expressions for the dependencies of the given +NAME and VERSION. The dependencies will be resolved using semantic versioning.+This procedure makes the assumption that most package repositories will, for a+given package provide some <metadata> on that package that includes what+versions of the package that are available and a list of dependencies for each+version. Dependencies are assumed to be composed of a NAME, a semantic RANGE and+other data.++This procedure takes the following keys:+ NAME - The name of the package to import+ VERSION - The version of the package to import+ NAME->METADATA - A procedure that takes a NAME of a package and returns that+package's <metadata>+ METADATA->PACKAGE A procedure that takes a package's <metadata> and VERSION +and returns the <package> for the given VERSION+ METADATA-VERSIONS A procedure that that takes a packages <metadata> and+returns a list of version as strings that are available for the given package+ PACKAGE-DEPENDENCIES a procedure that returns a list of <dependency> given a +<package>+ DEPENDENCY-NAME A procedure that takes a <dependency> and returns the its name+ DEPENDENCY-RANGE A procedure that takes a <dependency> and returns that+decency's range as a string+ GUIX-NAME A procedure that take a NAME and returns the Guix version of it+ MAKE-SEXP A procedure that takes <metadata>, <package> and a list of pairs+containing (EXPORT-NAME <dependency>), returning the package expression as an +s-expression"+ (define mem-name->metadata (memoize name->metadata))++ (define (latest? versions version)+ (equal? (car versions) version))++ (define (sorted-versions metadata)+ (sort (metadata-versions metadata) version>?))++ (define (name->versions name)+ (sorted-versions (mem-name->metadata name)))++ (define (semver-range-contains-string? range version)+ (semver-range-contains? range+ (string->semver version)))++ (define (guix-export-name name version)+ (let ((versions (name->versions name))+ (name (guix-name name)))+ (if (latest? versions version)+ name+ (string-append name "-" version))))++ ;; checks to see if we already defined or want to define a dep+ (define (find-known name range known)+ (match+ (find+ (match-lambda ((dep-name version)+ (and+ (string=? dep-name name)+ (semver-range-contains-string? range version))))+ known)++ (#f #f)+ ((name version) (list (guix-export-name name version) version #f)))+ )++ ;; searches searches for a package in guix+ (define (find-locally name range)+ (match+ (find+ (match-lambda ((_ _ package)+ (semver-range-contains-string?+ range+ (package-version package))))+ (find-packages-by-name*/direct (guix-name name)))+ (#f #f)+ ((_ export-symbol package) (list+ (symbol->string export-symbol)+ (package-version package) #f))))++ ;; searches for a package in some external repo+ (define (find-remote name range)+ (let* ((versions (name->versions name))+ (version (find+ (lambda (ver)+ (semver-range-contains-string? range ver))+ versions))+ (export-name (guix-export-name name version)))+ `(,export-name ,version #t)))+++ (define (find-dep-version dep known-deps)+ (let* ((name (dependency-name dep))+ (range (string->semver-range (dependency-range dep)))+ (export-name-version-needed+ (or (find-known name range known-deps)+ (find-locally name range)+ (find-remote name range))))+ `(,name ,@export-name-version-needed ,dep)+ ))++ (define (make-package-definition name version known-deps)+ (let* ((metadata (mem-name->metadata name))+ (versions (sorted-versions metadata))+ (package (metadata->package metadata version))+ (deps (map (lambda (dep)+ (find-dep-version dep known-deps))+ (package-dependencies package)))+ (sexp+ (make-sexp metadata package+ (map+ (match-lambda ((_ export-symbol _ _ dep)+ (list export-symbol dep)))+ deps))))+ (values+ (package->definition sexp (latest? versions version))+ (filter-map+ (match-lambda ((name _ version need? dep)+ (if need?+ (list name version)+ #f)))+ deps))))++ (define initial-state+ (list #f+ (list+ ;; packages to find+ (list name (if version+ version+ (car (name->versions name)))))+ ;; packages that have been found+ (list)))++ (define (step state)+ (match state+ ((prev ((next-name next-version) . rest) done)+ (receive (package dependencies)+ (make-package-definition next-name next-version+ (append done rest `((,next-name ,next-version))))+ (list+ package+ (append rest dependencies)+ (cons (list next-name next-version) done))))+ ((prev '() done)+ (list #f '() done))))+ + (stream-unfold+ ;; map: produce a stream element+ (match-lambda ((latest queue done) latest))+ ;; predicate+ (match-lambda ((latest queue done) latest))+ step+ (step initial-state)))diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex c3ab25d788..4ed3a5e1da 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>+;;; Copyright © 2016 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -24,6 +25,10 @@ #:use-module (guix packages) #:use-module (guix build-system) #:use-module (gnu packages)+ #:use-module (ice-9 match)+ #:use-module (srfi srfi-1)+ #:use-module (srfi srfi-9)+ #:use-module (srfi srfi-41) #:use-module (srfi srfi-64)) (test-begin "import-utils")@@ -120,4 +125,161 @@ ("license" . #f)))) (package-native-inputs (alist->package meta)))) +(define-record-type <metadata>+ (make-metadata name versions)+ metadata?+ (name metadata-name)+ (versions metadata-versions))++(define-record-type <package>+ (make-package version dependencies)+ package?+ (version package-version)+ (dependencies package-dependencies))++(define-record-type <dependency>+ (make-dependency name range)+ dependency?+ (name dependency-name)+ (range dependency-range))++(define (metadata-semver-versions metadata)+ (map (lambda (p)+ (package-version p))+ (metadata-versions metadata)))++(define (metadata->package metadata version)+ (find+ (lambda (package)+ (equal? (package-version package) version))+ (metadata-versions metadata)))++(define (make-sexp metadata package dependencies)+ `(package+ (name ,(guix-name (metadata-name metadata)))+ (version ,(package-version package))+ (dependcies ,(map+ (match-lambda ((public-name dep)+ (list (guix-name (dependency-name dep)) public-name)))+ dependencies))))++(define (guix-name name)+ (string-append "test-" name))++(define packages+ `(("no-deps" . (("1.0.0" . ()) ("0.1.0" . ())))+ ("one-dep" . (("1.0.0" . (("no-deps" "^1.0")))+ ("0.1.0" . (("no-deps" "^0.1.0")))))+ ("shared-dep" . (("1.0.0" . (("one-dep" "^0.1.0")+ ("no-deps" "*")))))+ ("recursive" . (("1.0.0" . (("recursive" "=1.0.0")))))+ ("already-packaged" . (("1.0.0" . (("rust" "~1.28")))))))++(define (name->metadata name)+ (let ((versions (assoc-ref packages name)))+ (make-metadata name+ (map+ (match-lambda+ ((version . deps)+ (make-package version+ (map+ (lambda (name-range)+ (apply make-dependency name-range))+ deps))))+ versions))))++(define* (test-recursive-importer name version #:optional (guix-name guix-name))+ (recursive-import-semver #:name name+ #:version version+ #:name->metadata name->metadata+ #:metadata->package metadata->package+ #:metadata-versions metadata-semver-versions+ #:package-dependencies package-dependencies+ #:dependency-name dependency-name+ #:dependency-range dependency-range+ #:guix-name guix-name+ #:make-sexp make-sexp))++(test-equal "recursive import test with no dependencies"+ `((define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "no-deps" "1.0.0")))++(test-equal "recursive import test with one dependencies"+ `((define-public test-one-dep+ (package+ (name "test-one-dep")+ (version "1.0.0")+ (dependcies (("test-no-deps" "test-no-deps")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "1.0.0")))++(test-equal "recursive import test with recursuve dependencies"+ `((define-public test-recursive+ (package+ (name "test-recursive")+ (version "1.0.0")+ (dependcies (("test-recursive" "test-recursive"))))))+ (stream->list (test-recursive-importer "recursive" "1.0.0")))++(test-equal "recursive import test with no dependencies using an old version"+ `((define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "no-deps" "0.1.0")))++(test-equal "recursive import test with one dependencies unsing an old version"+ `((define-public test-one-dep-0.1.0+ (package+ (name "test-one-dep")+ (version "0.1.0")+ (dependcies (("test-no-deps" "test-no-deps-0.1.0")))))+ (define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "0.1.0")))++(test-equal "recursive import test with with dependency that is already in the repo"+ `((define-public test-already-packaged+ (package (name "test-already-packaged")+ (version "1.0.0")+ (dependcies+ (("test-rust" "rust-1.28"))))))+ (stream->list (test-recursive-importer "already-packaged" "1.0.0" identity)))++(test-equal "shared dependencies"+ `((define-public test-shared-dep+ (package+ (name "test-shared-dep")+ (version "1.0.0")+ (dependcies (("test-one-dep" "test-one-dep-0.1.0")+ ("test-no-deps" "test-no-deps")))))+ (define-public test-one-dep-0.1.0+ (package+ (name "test-one-dep")+ (version "0.1.0")+ (dependcies (("test-no-deps" "test-no-deps-0.1.0")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ())))+ (define-public test-no-deps-0.1.0+ (package+ (name "test-no-deps")+ (version "0.1.0")+ (dependcies()))))+ (stream->list (test-recursive-importer "shared-dep" "1.0.0")))+ (test-end "import-utils")-- 2.24.0
M
M
Martin Becze wrote on 10 Dec 2019 20:23
[PATCH v4 4/6] added "#:skip-build? #t" to the output of (make-crate-sexp). Most the the packages imported will be libaries and won't need to build. The top level package will build them though.
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
a734de722430cdd288cb0ed21dc8515fc89fdd3a.1576005195.git.mjbecze@riseup.net
* guix/import/crate.scm (make-crate-sexp): added "#:skip-build? #t" to the output--- guix/import/crate.scm | 3 ++- tests/crate.scm | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-)
Toggle diff (30 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex da92c43b8c..5683369b7a 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -183,7 +183,8 @@ record or #f if it was not found." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ ,@(maybe-arguments (append `(#:skip-build? #t)+ (maybe-cargo-inputs cargo-inputs) (maybe-cargo-development-inputs cargo-development-inputs))) (home-page ,(match home-pagediff --git a/tests/crate.scm b/tests/crate.scmindex b77cbb08c6..64e5b6932e 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -102,7 +102,8 @@ ('build-system 'cargo-build-system) ('arguments ('quasiquote- ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))+ ('#:skip-build? #t+ #:cargo-inputs (("rust-bar" ('unquote rust-bar)))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary")-- 2.24.0
M
M
Martin Becze wrote on 10 Dec 2019 20:23
[PATCH v4 3/6] Rewrote some of guix/import/crate.scm to use recursive-import-semver and updated script and test.
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
583af30cbbd946af96c7f9cb856df9d1170287c7.1576005195.git.mjbecze@riseup.net
* guix/import/crate.scm (make-crate-sexp): Use <crate> <crate-version> as args* guix/import/crate.scm (crate->crate-version): New Procedure* guix/import/crate.scm (crate->versions): New Procedure* guix/import/crate.scm (crate-recursive-import): Updated to user recursive-import-semver* guix/scripts/import/crate.scm (guix-import-crate): Remove `define-public` generation from UI* guix/tests/crate.scm: Updated tests--- guix/import/crate.scm | 165 ++++++++++++++++++---------------- guix/scripts/import/crate.scm | 9 +- tests/crate.scm | 2 +- 3 files changed, 91 insertions(+), 85 deletions(-)
Toggle diff (258 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 8dc014d232..da92c43b8c 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -38,6 +38,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name crate-recursive-import@@ -85,7 +86,7 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string- (kind crate-dependency-kind "kind" ;'normal | 'dev+ (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string @@ -111,7 +112,9 @@ record or #f if it was not found." (url (string-append (%crate-base-url) path))) (match (assoc-ref (or (json-fetch url) '()) "dependencies") ((? vector? vector)- (map json->crate-dependency (vector->list vector)))+ (filter (lambda (dep)+ (not (eq? (crate-dependency-kind dep) 'dev)))+ (map json->crate-dependency (vector->list vector)))) (_ '())))) @@ -141,62 +144,84 @@ record or #f if it was not found." ((args ...) `((arguments (,'quasiquote ,args)))))) -(define* (make-crate-sexp #:key name version cargo-inputs cargo-development-inputs- home-page synopsis description license- #:allow-other-keys)- "Return the `package' s-expression for a rust package with the given NAME,-VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION,-and LICENSE."- (let* ((port (http-fetch (crate-uri name version)))+(define (make-crate-sexp crate version* dependencies)+ "Return the `package' s-expression for a rust package given <crate>,+ <crate-version> and a list of <crate-dependency>"+ (define normal-dependency?+ (match-lambda ((_ dep) (not (eq? (crate-dependency-kind dep) 'dev)))))++ (define (string->license string)+ (match (regexp-exec %dual-license-rx string)+ (#f (list (spdx-string->license string)))+ (m (list (spdx-string->license (match:substring m 1))+ (spdx-string->license (match:substring m 2))))))++ (let* ((dep-crates dev-dep-crates (partition normal-dependency? dependencies))+ (cargo-inputs (sort (unzip1 dep-crates)+ string-ci<?))+ (cargo-development-inputs+ (sort (unzip1 dev-dep-crates)+ string-ci<?))+ (name (crate-name crate))+ (version (crate-version-number version*))+ (home-page (or (crate-home-page crate)+ (crate-repository crate)))+ (synopsis (crate-description crate))+ (description (crate-description crate))+ (license (and=> (crate-version-license version*)+ string->license))+ (port (http-fetch (crate-uri name version)) ) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name- cargo-development-inputs)) (pkg `(package- (name ,guix-name)- (version ,version)- (source (origin- (method url-fetch)- (uri (crate-uri ,name version))- (file-name (string-append name "-" version ".tar.gz"))- (sha256- (base32- ,(bytevector->nix-base32-string (port-sha256 port))))))- (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)- (maybe-cargo-development-inputs- cargo-development-inputs)))- (home-page ,(match home-page- (() "")- (_ home-page)))- (synopsis ,synopsis)- (description ,(beautify-description description))- (license ,(match license- (() #f)- ((license) license)- (_ `(list ,@license)))))))- (close-port port)- pkg))+ (name ,guix-name)+ (version ,version)+ (source (origin+ (method url-fetch)+ (uri (crate-uri ,name version))+ (file-name (string-append name "-" version ".crate"))+ (sha256+ (base32+ ,(bytevector->nix-base32-string (port-sha256 port))))))+ (build-system cargo-build-system)+ ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ (maybe-cargo-development-inputs+ cargo-development-inputs)))+ (home-page ,(match home-page+ (() "")+ (_ home-page)))+ (synopsis ,synopsis)+ (description ,(beautify-description description))+ (license ,(match license+ (() #f)+ ((license) license)+ (_ `(list ,@license)))))))++ (close-port port)+ pkg)) (define %dual-license-rx ;; Dual licensing is represented by a string such as "MIT OR Apache-2.0". ;; This regexp matches that. (make-regexp "^(.*) OR (.*)$")) +(define (crate->crate-version crate version-number)+ "returns the <crate-version> for a given CRATE and VERSION-NUMBER"+ (find (lambda (version)+ (string=? (crate-version-number version)+ version-number))+ (crate-versions crate)))++(define (crate->versions crate)+ "Returns a list of versions for a given CRATE"+ (map (lambda (version)+ (crate-version-number version))+ (crate-versions crate)))+ (define* (crate->guix-package crate-name #:optional version) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME."- (define (string->license string)- (match (regexp-exec %dual-license-rx string)- (#f (list (spdx-string->license string)))- (m (list (spdx-string->license (match:substring m 1))- (spdx-string->license (match:substring m 2))))))-- (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))- (define crate (lookup-crate crate-name)) @@ -205,38 +230,27 @@ latest version of CRATE-NAME." (crate-latest-version crate))) (define version*- (find (lambda (version)- (string=? (crate-version-number version)- version-number))- (crate-versions crate)))+ (crate->crate-version crate version-number)) - (and crate version*- (let* ((dependencies (crate-version-dependencies version*))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))- (values- (make-crate-sexp #:name crate-name- #:version (crate-version-number version*)- #:cargo-inputs cargo-inputs- #:cargo-development-inputs cargo-development-inputs- #:home-page (or (crate-home-page crate)- (crate-repository crate))- #:synopsis (crate-description crate)- #:description (crate-description crate)- #:license (and=> (crate-version-license version*)- string->license))- (append cargo-inputs cargo-development-inputs)))))+ (define dependencies (map+ (lambda (dep)+ (list (crate-name->package-name+ (crate-dependency-id dep)) dep))+ (crate-version-dependencies version*)))+ (make-crate-sexp crate version* dependencies)) -(define (crate-recursive-import crate-name)- (recursive-import crate-name #f- #:repo->guix-package (lambda (name repo)- (crate->guix-package name))- #:guix-name crate-name->package-name))+(define* (crate-recursive-import name #:optional version)+ (recursive-import-semver+ #:name name+ #:version version+ #:name->metadata lookup-crate+ #:metadata->package crate->crate-version+ #:metadata-versions crate->versions+ #:package-dependencies crate-version-dependencies+ #:dependency-name crate-dependency-id+ #:dependency-range crate-dependency-requirement+ #:guix-name crate-name->package-name+ #:make-sexp make-crate-sexp)) (define (guix-package->crate-name package) "Return the crate name of PACKAGE."@@ -285,4 +299,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release)))-diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex 4690cceb4d..85ae6fbe59 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -96,14 +96,7 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (reverse- (stream->list- (crate-recursive-import name))))+ (stream->list (crate-recursive-import name version)) (let ((sexp (crate->guix-package name version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%")diff --git a/tests/crate.scm b/tests/crate.scmindex c14862ad9f..b77cbb08c6 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -95,7 +95,7 @@ ('source ('origin ('method 'url-fetch) ('uri ('crate-uri "foo" 'version))- ('file-name ('string-append 'name "-" 'version ".tar.gz"))+ ('file-name ('string-append 'name "-" 'version ".crate")) ('sha256 ('base32 (? string? hash)))))-- 2.24.0
M
M
Martin Becze wrote on 10 Dec 2019 20:23
[PATCH v4 5/6] guix: crate: Depublicated dependencies
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
46a5a5ed450c386ee1a17c02f73d613369c61657.1576005195.git.mjbecze@riseup.net
* guix/import/crate.scm: (crate-version-dependencies): dedup deps--- guix/import/crate.scm | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
Toggle diff (38 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 5683369b7a..535ac2d8e5 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -109,15 +109,22 @@ record or #f if it was not found." "Return the list of <crate-dependency> records of VERSION, a <crate-version>." (let* ((path (assoc-ref (crate-version-links version) "dependencies"))- (url (string-append (%crate-base-url) path)))- (match (assoc-ref (or (json-fetch url) '()) "dependencies")- ((? vector? vector)- (filter (lambda (dep)- (not (eq? (crate-dependency-kind dep) 'dev)))- (map json->crate-dependency (vector->list vector))))- (_- '()))))-+ (url (string-append (%crate-base-url) path))+ (deps-list (match (assoc-ref (or (json-fetch url) '()) "dependencies")+ ((? vector? vector) (vector->list vector))+ (_+ '())))+ ;; turn the raw list into <dependency>'s and remove dev depenedencies+ (deps (filter-map (lambda (json)+ (let ((dep (json->crate-dependency json)))+ (if (eq? (crate-dependency-kind dep) 'dev)+ #f+ dep)))+ deps-list)))+ ;;remove duplicate dependencies+ (apply lset-adjoin `(,(lambda (a b)+ (string-ci=? (crate-dependency-id a) (crate-dependency-id b)))+ () ,@deps)))) ;;; ;;; Converting crates to Guix packages.-- 2.24.0
M
M
Martin Becze wrote on 10 Dec 2019 20:23
[PATCH v4 6/6] guix: import: recursive-import-semver: allow the range of a package to be specified when begining import.
(address . 38408@debbugs.gnu.org)(name . Martin Becze)(address . mjbecze@riseup.net)
dc13c06ce905b66f4c4fe3c55ed1f79dabd98229.1576005195.git.mjbecze@riseup.net
* guix/import/crate.scm (crate-recursive-import) changed param version to range* guix/import/util.scm (recursive-import-semver) changed param version to range* guix/tests/import-utils.scm added range test for (recursive-import-semver)--- guix/import/crate.scm | 5 +-- guix/import/utils.scm | 69 ++++++++++++++++++++++++------------------ tests/import-utils.scm | 15 ++++++++- 3 files changed, 57 insertions(+), 32 deletions(-)
Toggle diff (185 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 535ac2d8e5..cd9ab61cca 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -247,10 +247,11 @@ latest version of CRATE-NAME." (crate-version-dependencies version*))) (make-crate-sexp crate version* dependencies)) -(define* (crate-recursive-import name #:optional version)++(define* (crate-recursive-import name #:optional range) (recursive-import-semver #:name name- #:version version+ #:range (if range range "*") #:name->metadata lookup-crate #:metadata->package crate->crate-version #:metadata-versions crate->versionsdiff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 6932614f8e..35d5c79286 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -422,8 +422,9 @@ dependencies." ;; initial state (step initial-state))) -(define* (recursive-import-semver #:key name- (version #f)+(define* (recursive-import-semver #:key+ name+ (range "*") name->metadata metadata->package metadata-versions@@ -433,7 +434,7 @@ dependencies." guix-name make-sexp) "Generates a stream of package expressions for the dependencies of the given -NAME and VERSION. The dependencies will be resolved using semantic versioning.+NAME and version RANGE. The dependencies will be resolved using semantic versioning. This procedure makes the assumption that most package repositories will, for a given package provide some <metadata> on that package that includes what versions of the package that are available and a list of dependencies for each@@ -442,7 +443,7 @@ other data. This procedure takes the following keys: NAME - The name of the package to import- VERSION - The version of the package to import+ RANGE - The version range of the package to import NAME->METADATA - A procedure that takes a NAME of a package and returns that package's <metadata> METADATA->PACKAGE A procedure that takes a package's <metadata> and VERSION @@ -473,6 +474,8 @@ s-expression" (semver-range-contains? range (string->semver version))) + ;; given a name of a package and a version number this returns the export+ ;; symbol that will be used (define (guix-export-name name version) (let ((versions (name->versions name)) (name (guix-name name)))@@ -518,14 +521,17 @@ s-expression" (export-name (guix-export-name name version))) `(,export-name ,version #t))) + (define (find-dep-version-by-name-range name range-string known-deps)+ (let ((range (string->semver-range range-string)))+ (or (find-known name range known-deps)+ (find-locally name range)+ (find-remote name range)))) (define (find-dep-version dep known-deps) (let* ((name (dependency-name dep))- (range (string->semver-range (dependency-range dep)))+ (range (dependency-range dep)) (export-name-version-needed- (or (find-known name range known-deps)- (find-locally name range)- (find-remote name range))))+ (find-dep-version-by-name-range name range known-deps))) `(,name ,@export-name-version-needed ,dep) )) @@ -536,12 +542,12 @@ s-expression" (deps (map (lambda (dep) (find-dep-version dep known-deps)) (package-dependencies package)))+ (deps-with-export-symbol (map+ (match-lambda ((_ export-symbol _ _ dep)+ (list export-symbol dep)))+ deps)) (sexp- (make-sexp metadata package- (map- (match-lambda ((_ export-symbol _ _ dep)- (list export-symbol dep)))- deps))))+ (make-sexp metadata package deps-with-export-symbol))) (values (package->definition sexp (latest? versions version)) (filter-map@@ -551,15 +557,12 @@ s-expression" #f))) deps)))) - (define initial-state- (list #f- (list- ;; packages to find- (list name (if version- version- (car (name->versions name)))))- ;; packages that have been found- (list)))+ (define (initial-state name version)+ `(#f+ ;; packages to find+ ,(list (list name version))+ ;; packages that have been found+ ())) (define (step state) (match state@@ -573,11 +576,19 @@ s-expression" (cons (list next-name next-version) done)))) ((prev '() done) (list #f '() done))))++ (define (create-stream initial-state)+ (stream-unfold+ ;; map: produce a stream element+ (match-lambda ((latest queue done) latest))+ ;; predicate+ (match-lambda ((latest queue done) latest))+ step+ (step initial-state))) - (stream-unfold- ;; map: produce a stream element- (match-lambda ((latest queue done) latest))- ;; predicate- (match-lambda ((latest queue done) latest))- step- (step initial-state)))+ (match (find-dep-version-by-name-range name range '())+ ((_ version #t)+ (create-stream (initial-state name version)))+ ;; if the initial package alread exsits then just return its export symbol+ ((export-name _ #f)+ (list->stream (list export-name)))))diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex 4ed3a5e1da..022b8f2b32 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -190,7 +190,7 @@ (define* (test-recursive-importer name version #:optional (guix-name guix-name)) (recursive-import-semver #:name name- #:version version+ #:range version #:name->metadata name->metadata #:metadata->package metadata->package #:metadata-versions metadata-semver-versions@@ -250,6 +250,19 @@ (dependcies ())))) (stream->list (test-recursive-importer "one-dep" "0.1.0"))) +(test-equal "recursive import test with a version range"+ `((define-public test-one-dep+ (package+ (name "test-one-dep")+ (version "1.0.0")+ (dependcies (("test-no-deps" "test-no-deps")))))+ (define-public test-no-deps+ (package+ (name "test-no-deps")+ (version "1.0.0")+ (dependcies ()))))+ (stream->list (test-recursive-importer "one-dep" "*")))+ (test-equal "recursive import test with with dependency that is already in the repo" `((define-public test-already-packaged (package (name "test-already-packaged")-- 2.24.0
M
M
Martin Becze wrote on 17 Dec 2019 00:30
Rewrote recursive-import-semver based on topological-sort
(address . 38408@debbugs.gnu.org)
b506552fb2bfdfbd6a9ba6240c6838a6@riseup.net
Hello Guix,
I rewrote the recursive-import-semver procedure to use the newtopological-sort procedure. I also return a list instead of a streamlike (recursive-import). So to recap (recursive-import-semver) importsdependency using semantic version to find the correct version of adependency. The crate importer has also be converted to userecursive-import-semver. It will now use recursive-import-semver when"guix import crate -r" is used. You can also specify the range that youwould like to import such as guix import crate -r rand@^0.6". Here is anexample of the format that it will produce.
(define-public rust-bincode-1.2.1 (package (name "rust-bincode") (version "1.2.1") (source (origin (method url-fetch) (uri (crate-uri "bincode" version)) (file-name (string-append name "-" version ".crate")) (sha256 (base32 "1gvxm3n67xv1874fwxmnlircdlphlk1hcw75ykrrnw9l2nky4lsp")))) (build-system cargo-build-system) (arguments `(#:skip-build? #t #:cargo-inputs (("rust-byteorder-1.3.2" ,rust-byteorder-1.3.2) ("rust-serde-1.0.103" ,rust-serde-1.0.103)))) (home-page "https://github.com/servo/bincode") (synopsis "A binary serialization / deserialization strategy that uses Serdefor transforming structs into bytes and vice versa!") (description "This package provides a binary serialization / deserializationstrategy that uses Serde for transforming structs into bytes and viceversa!") (license license:expat)))
-Martin
From aa6aaeacb5f91508f4158999ba6e7f95e8309bed Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Mon, 16 Dec 2019 18:11:38 -0500Subject: [PATCH 4/4] gnu: scripts: import: crate: Remove `define-public` generation from UI
* guix/scripts/import/crate.scm (guix-import-crate): Remove `define-public` generation from UI--- guix/scripts/import/crate.scm | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
Toggle diff (20 lines)diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex 92034dab3c..9a08c9b8b4 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -95,12 +95,7 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (crate-recursive-import name))+ (crate-recursive-import name version) (let ((sexp (crate->guix-package name version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%")-- 2.24.0
L
L
Ludovic Courtès wrote on 19 Dec 2019 23:00
Re: [bug#38408] [PATCH v4 1/6] gnu: added new function, find-packages-by-name*/direct
(name . Martin Becze)(address . mjbecze@riseup.net)(address . 38408@debbugs.gnu.org)
877e2sm0vv.fsf@gnu.org
Hello!
I’m not a Crate expert so I’m only commenting on non-Crate-specificbits.
Martin Becze <mjbecze@riseup.net> skribis:
Toggle quote (2 lines)> * gnu/packages.scm (find-packages-by-naem*/direct)
[...]
Toggle quote (14 lines)> +(define* (fold-packages* proc init> + #:optional> + (modules (all-modules (%package-module-path)> + #:warn> + warn-about-load-error))> + #:key (select? (negate hidden-package?)))> + "Call (PROC PACKAGE RESULT) for each available package defined in one of> +MODULES that matches SELECT?, using INIT as the initial value of RESULT. It> +is guaranteed to never traverse the same package twice."> + (fold-module-public-variables* (lambda (module symbol var result)> + (let ((object (variable-ref var)))> + (if (and (package? object) (select? object))> + (proc module symbol object result)
I’m wary of exposing variable names, especially in such a central API.
Toggle quote (2 lines)> +(define find-packages-by-name*/direct ;bypass the cache
Providing an explicit cache bypassing method also sounds worrying to me:the cache is supposed to be transparent and semantics-preserving.
More generally, I think adding new features to an importer shouldn’trequire modifications in this area, as a matter of separating concerns.
WDYT?
Thanks,Ludo’.
L
L
Ludovic Courtès wrote on 19 Dec 2019 23:07
Re: [bug#38408] [PATCH v4 2/6] gnu: added new procedure, recusive-import-semver
(name . Martin Becze)(address . mjbecze@riseup.net)(address . 38408@debbugs.gnu.org)
87woaskm08.fsf@gnu.org
Martin Becze <mjbecze@riseup.net> skribis:
Toggle quote (2 lines)> * gnu/packages.scm (recusive-import-semver): New Procedure> * gnu/packages.scm (package->definition)[arguments]: New argument, "latest"
^This is actually guix/import/utils.scm. :-)
Toggle quote (13 lines)> * tests/import-utils.scm: tests for recusive-import-semver
> +(define* (recursive-import-semver #:key name> + (version #f)> + name->metadata> + metadata->package> + metadata-versions> + package-dependencies> + dependency-name> + dependency-range> + guix-name> + make-sexp)
That’s intimidating. :-)
Since there’s currently a single user, the Crate importer, I wouldrather have semver-handling directly in (guix import crate). Sure wecan try to write it in a way that clearly separates semver handling fromCrate-specific bits, but we shouldn’t try to make it too generic at thispoint, IMO.
WDYT?
The <crate> and <crate-version> records provide the information neededto implement what you have in mind, I think.
Thanks,Ludo’.
M
M
Martin Becze wrote on 20 Dec 2019 19:37
Re: [bug#38408] [PATCH v4 1/6] gnu: added new function, find-packages-by-name*/direct
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 38408@debbugs.gnu.org)
6e56589639ea75bfec2c97f7e9e31ad9@riseup.net
Toggle quote (12 lines)> > Providing an explicit cache bypassing method also sounds worrying to me:> the cache is supposed to be transparent and semantics-preserving.> > More generally, I think adding new features to an importer shouldn’t> require modifications in this area, as a matter of separating concerns.> > WDYT?> > Thanks,> Ludo’.
yes I agree, I removed that in the last version! Which also rebases offthe new topological sort procedure. I'll attach the latest patches here,in case you missed it.
From aa6aaeacb5f91508f4158999ba6e7f95e8309bed Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Mon, 16 Dec 2019 18:11:38 -0500Subject: [PATCH v5 4/4] gnu: scripts: import: crate: Remove `define-public` generation from UI
* guix/scripts/import/crate.scm (guix-import-crate): Remove `define-public` generation from UI--- guix/scripts/import/crate.scm | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
Toggle diff (20 lines)diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex 92034dab3c..9a08c9b8b4 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -95,12 +95,7 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (crate-recursive-import name))+ (crate-recursive-import name version) (let ((sexp (crate->guix-package name version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%")-- 2.24.1
M
M
Martin Becze wrote on 20 Dec 2019 19:46
Re: [bug#38408] [PATCH v4 2/6] gnu: added new procedure, recusive-import-semver
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 38408@debbugs.gnu.org)
74e229c9340764337bbea3752a4f8c50@riseup.net
Toggle quote (8 lines)> Since there’s currently a single user, the Crate importer, I would> rather have semver-handling directly in (guix import crate). Sure we> can try to write it in a way that clearly separates semver handling from> Crate-specific bits, but we shouldn’t try to make it too generic at this> point, IMO.> > WDYT?
I don't have strong opinion but I'm happy to write it this way!
L
L
Ludovic Courtès wrote on 27 Dec 2019 19:38
Re: [bug#38408] [PATCH v4 1/6] gnu: added new function, find-packages-by-name*/direct
(name . Martin Becze)(address . mjbecze@riseup.net)(address . 38408@debbugs.gnu.org)
87blrtfwbx.fsf@gnu.org
Hi Martin,
Sorry for the late reply.
Martin Becze <mjbecze@riseup.net> skribis:
Toggle quote (15 lines)>> Providing an explicit cache bypassing method also sounds worrying to me:>> the cache is supposed to be transparent and semantics-preserving.>> >> More generally, I think adding new features to an importer shouldn’t>> require modifications in this area, as a matter of separating concerns.>> >> WDYT?>> >> Thanks,>> Ludo’.>> yes I agree, I removed that in the last version! Which also rebases off> the new topological sort procedure. I'll attach the latest patches here,> in case you missed it.
Thanks!
Toggle quote (8 lines)> From eeffdf569c4d7fbfd843e0b48404b6a2f3d46343 Mon Sep 17 00:00:00 2001> From: Martin Becze <mjbecze@riseup.net>> Date: Mon, 16 Dec 2019 17:08:16 -0500> Subject: [PATCH v5 1/4] guix: import: added recusive-import-semver>> * guix/import/utils.scm (recusive-import-semver): New Varible> * guix/import/utils.scm (package->definition)[arguments]: Add append-verions option
[...]
Toggle quote (38 lines)> +(define* (recursive-import-semver #:key> + name> + (range "*")> + name->metadata> + metadata->package> + metadata-versions> + package-dependencies> + dependency-name> + dependency-range> + guix-name> + make-sexp)> + "Generates a list of package expressions for the dependencies of the given > +NAME and version RANGE. The dependencies will be resolved using semantic versioning.> +This procedure makes the assumption that most package repositories will, for a> +given package provide some <metadata> on that package that includes what> +versions of the package that are available and a list of dependencies for each> +version. Dependencies are assumed to be composed of a NAME, a semantic RANGE and> +other data.> +> +This procedure takes the following keys:> + NAME - The name of the package to import> + RANGE - The version range of the package to import> + NAME->METADATA - A procedure that takes a NAME of a package and returns that> +package's <metadata>> + METADATA->PACKAGE A procedure that takes a package's <metadata> and VERSION > +and returns the <package> for the given VERSION> + METADATA-VERSIONS A procedure that that takes a packages <metadata> and> +returns a list of version as strings that are available for the given package> + PACKAGE-DEPENDENCIES a procedure that returns a list of <dependency> given a > +<package>> + DEPENDENCY-NAME A procedure that takes a <dependency> and returns the its name> + DEPENDENCY-RANGE A procedure that takes a <dependency> and returns that> +decency's range as a string> + GUIX-NAME A procedure that take a NAME and returns the Guix version of it> + MAKE-SEXP A procedure that takes <metadata>, <package> and a list of pairs> +containing (EXPORT-NAME <dependency>), returning the package expression as an > +s-expression"
As noted in my previous message, I think this interface is too complex,and since it’s used in a single importer, it would be best to have itdirectly in (guix import crate).
If/when there’s a need to share that logic among several importers, thenwe can look for ways to factorize whatever needs to be factorized.
In the meantime, there are probably semver-related things that couldnaturally to a helper (guix import semver) module, although perhaps mostof that is already provided by guile-semver?
Toggle quote (2 lines)> + (define-record-type <node-dependency>
Also, as a rule of thumb, we wouldn’t want to duplicate these data typesand related code.
WDYT?
Thanks,Ludo’.
M
M
Martin Becze wrote on 18 Jan 2020 17:40
Re: [bug#38408] [PATCH v6] Semantic version aware recusive importer for crates
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 38408@debbugs.gnu.org)
4c06cf99-30bc-2440-e833-52781376e302@riseup.net
okkkie! finally got this rewote! Patches attached.
Toggle quote (1 lines)> As noted in my previous message, I think this interface is too > complex, and since it’s used in a single importer, it would be best >
to have it directly in (guix import crate).This is now done!
Toggle quote (1 lines)> In the meantime, there are probably semver-related things that could > naturally to a helper (guix import semver) module, although perhaps >
most of that is already provided by guile-semver?guile-semver is pretty easy to work with so I didn't need to use many helpers. The one I did use I think I will try to submit to guile-semver itself.
On 12/27/19 1:38 PM, Ludovic Courtès wrote:
Toggle quote (1 lines)> Hi Martin, > > Sorry for the late reply. > > Martin Becze <mjbecze@riseup.net>
skribis: > >>> Providing an explicit cache bypassing method also sounds >>> worrying to me: the cache is supposed to be transparent and >>> semantics-preserving. >>> >>> More generally, I think adding new features to an importer >>> shouldn’t require modifications in this area, as a matter of >>> separating concerns. >>> >>> WDYT? >>> >>> Thanks, Ludo’. >> >> yes I agree, I removed that in the last version! Which also >> rebases off the new topological sort procedure. I'll attach the >> latest patches here, in case you missed it. > > Thanks! > >> From eeffdf569c4d7fbfd843e0b48404b6a2f3d46343 Mon Sep 17 00:00:00 >> 2001 From: Martin Becze <mjbecze@riseup.net> Date: Mon, 16 Dec >> 2019 17:08:16 -0500 Subject: [PATCH v5 1/4] guix: import: added >> recusive-import-semver >> >> * guix/import/utils.scm (recusive-import-semver): New Varible * >> guix/import/utils.scm (package->definition)[arguments]: Add >> append-verions option > > [...] > >> +(define* (recursive-import-semver #:key + name + (range "*") + >> name->metadata + metadata->package + metadata-versions + >> package-dependencies + dependency-name + dependency-range + >> guix-name + make-sexp) + "Generates a list of package expressions >> for the dependencies of the given +NAME and version RANGE. The >> dependencies will be resolved using semantic versioning. +This >> procedure makes the assumption that most package repositories will, >> for a +given package provide some <metadata> on that package that >> includes what +versions of the package that are available and a >> list of dependencies for each +version. Dependencies are assumed to >> be composed of a NAME, a semantic RANGE and +other data. + +This >> procedure takes the following keys: + NAME - The name of the >> package to import + RANGE - The version range of the package to >> import + NAME->METADATA - A procedure that takes a NAME of a >> package and returns that +package's <metadata> + METADATA->PACKAGE >> A procedure that takes a package's <metadata> and VERSION +and >> returns the <package> for the given VERSION + METADATA-VERSIONS A >> procedure that that takes a packages <metadata> and +returns a >> list of version as strings that are available for the given package >> + PACKAGE-DEPENDENCIES a procedure that returns a list of >> <dependency> given a +<package> + DEPENDENCY-NAME A procedure >> that takes a <dependency> and returns the its name + >> DEPENDENCY-RANGE A procedure that takes a <dependency> and returns >> that +decency's range as a string + GUIX-NAME A procedure that >> take a NAME and returns the Guix version of it + MAKE-SEXP A >> procedure that takes <metadata>, <package> and a list of pairs >> +containing (EXPORT-NAME <dependency>), returning the package >> expression as an +s-expression" > > As noted in my previous message, I think this interface is too > complex, and since it’s used in a single importer, it would be best > to have it directly in (guix import crate). > > If/when there’s a need to share that logic among several importers, > then we can look for ways to factorize whatever needs to be > factorized. > > In the meantime, there are probably semver-related things that could > naturally to a helper (guix import semver) module, although perhaps > most of that is already provided by guile-semver? > >> + (define-record-type <node-dependency> > > Also, as a rule of thumb, we wouldn’t want to duplicate these data > types and related code. > > WDYT? > > Thanks, Ludo’.
Attachment: file
From 7747d27ed32069acb193b3811b30a5a3cc8cfd5b Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Sat, 18 Jan 2020 05:05:03 -0500Subject: [PATCH v6 1/2] guix: import: (recursive-import) Allow for version numbers
* guix/import/utils.scm (package->definition) [arguments] added optional `append-version?`* guix/import/utils.scm (recursive-import) [arguments] added key `version` and moved `repo` to be a key* tests/import-utils.scm
* guix/import/cran.scm (cran->guix-package) [argument]: change `repo` to a key* guix/import/cran.scm (cran-recursive-import) [argument]: change `repo` to a key* guix/scripts/import/cran.scm: change `repo` to a key* guix/import/elpa.scm (elpa->guix-pakcage) [argumnets]: change `repo` to a key* guix/import/elpa.scm (elpa-recursive-import) [argumnets]: change `repo` to a key* guix/scripts/import/elpa.scm: change `repo` to a key* guix/import/gem.scm (gem->guix-package) [arguments]: change `repo` to a key* guix/import/gem.scm (recursive-import) [arguments]: change `repo` to a key* guix/import/opam.scm (opam-recurive-import) [arguments]: change `repo` to a key* guix/import/pypi.scm (pypi-recursive-import) [arguments]: change `repo` to a key* guix/import/stackage.scm (stackage-recursive-import) [arguments]: change `repo` to a key--- guix/import/cran.scm | 8 +++-- guix/import/elpa.scm | 6 ++-- guix/import/gem.scm | 6 ++-- guix/import/opam.scm | 5 ++-- guix/import/pypi.scm | 5 ++-- guix/import/stackage.scm | 5 ++-- guix/import/utils.scm | 57 +++++++++++++++++++++++------------- guix/scripts/import/cran.scm | 5 ++-- guix/scripts/import/elpa.scm | 4 ++- tests/import-utils.scm | 8 +++-- 10 files changed, 69 insertions(+), 40 deletions(-)
Toggle diff (344 lines)diff --git a/guix/import/cran.scm b/guix/import/cran.scmindex bcb37ed250..9e05dfcba8 100644--- a/guix/import/cran.scm+++ b/guix/import/cran.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015, 2016, 2017, 2018, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2015, 2016, 2017, 2019, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -506,7 +507,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file." (define cran->guix-package (memoize- (lambda* (package-name #:optional (repo 'cran))+ (lambda* (package-name #:key (repo 'cran) #:allow-other-keys) "Fetch the metadata for PACKAGE-NAME from REPO and return the `package' s-expression corresponding to that package, or #f on failure." (let ((description (fetch-description repo package-name)))@@ -521,8 +522,9 @@ s-expression corresponding to that package, or #f on failure." (cran->guix-package package-name 'cran)) (else (values #f '())))))))) -(define* (cran-recursive-import package-name #:optional (repo 'cran))- (recursive-import package-name repo+(define* (cran-recursive-import package-name #:key (repo 'cran))+ (recursive-import package-name+ #:repo repo #:repo->guix-package cran->guix-package #:guix-name cran-guix-name)) diff --git a/guix/import/elpa.scm b/guix/import/elpa.scmindex 2d4487dba0..9140bcdc34 100644--- a/guix/import/elpa.scm+++ b/guix/import/elpa.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2015, 2016, 2017, 2018, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -245,7 +246,7 @@ type '<elpa-package>'." (license ,license)) dependencies-names))) -(define* (elpa->guix-package name #:optional (repo 'gnu))+(define* (elpa->guix-package name #:key (repo 'gnu) #:allow-other-keys) "Fetch the package NAME from REPO and produce a Guix package S-expression." (match (fetch-elpa-package name repo) (#f #f)@@ -301,7 +302,8 @@ type '<elpa-package>'." (define elpa-guix-name (cut guix-name "emacs-" <>)) (define* (elpa-recursive-import package-name #:optional (repo 'gnu))- (recursive-import package-name repo+ (recursive-import package-name+ #:repo repo #:repo->guix-package elpa->guix-package #:guix-name elpa-guix-name)) diff --git a/guix/import/gem.scm b/guix/import/gem.scmindex 0bf9ff2552..e744d9e69d 100644--- a/guix/import/gem.scm+++ b/guix/import/gem.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015 David Thompson <davet@gnu.org> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -117,7 +118,7 @@ VERSION, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES." ((license) (license->symbol license)) (_ `(list ,@(map license->symbol licenses))))))) -(define* (gem->guix-package package-name #:optional (repo 'rubygems) version)+(define* (gem->guix-package package-name #:key (repo 'rubygems) version) "Fetch the metadata for PACKAGE-NAME from rubygems.org, and return the `package' s-expression corresponding to that package, or #f on failure." (let ((package (rubygems-fetch package-name)))@@ -201,6 +202,7 @@ package on RubyGems." (latest latest-release))) (define* (gem-recursive-import package-name #:optional version)- (recursive-import package-name '()+ (recursive-import package-name+ #:repo '() #:repo->guix-package gem->guix-package #:guix-name ruby-package-name))diff --git a/guix/import/opam.scm b/guix/import/opam.scmindex 394415fdd4..87c823a98c 100644--- a/guix/import/opam.scm+++ b/guix/import/opam.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2018 Julien Lepiller <julien@lepiller.eu>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -311,8 +312,8 @@ or #f on failure." dependencies)))))))) (define (opam-recursive-import package-name)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (opam->guix-package name)) #:guix-name ocaml-name->guix-name)) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scmindex 354cae9c4c..f0702d6403 100644--- a/guix/import/pypi.scm+++ b/guix/import/pypi.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com> ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -415,8 +416,8 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." description license)))))))) (define (pypi-recursive-import package-name)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (pypi->guix-package name)) #:guix-name python->package-name)) diff --git a/guix/import/stackage.scm b/guix/import/stackage.scmindex 14150201b5..6091cf2c64 100644--- a/guix/import/stackage.scm+++ b/guix/import/stackage.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -108,8 +109,8 @@ included in the Stackage LTS release." (leave-with-message "~a: Stackage package not found" package-name)))))) (define (stackage-recursive-import package-name . args)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (apply stackage->guix-package (cons name args))) #:guix-name hackage-name->package-name)) diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex d17d400ddf..59430d3e66 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2019 Robert Vollmert <rob@vllmrt.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -44,6 +45,7 @@ #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (factorize-uri flatten@@ -258,13 +260,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package)+(define* (package->definition guix-package #:optional append-version?) (match guix-package- (('package ('name (? string? name)) _ ...)- `(define-public ,(string->symbol name)- ,guix-package))- (('let anything ('package ('name (? string? name)) _ ...))- `(define-public ,(string->symbol name)+ ((or+ ('package ('name name) ('version version) . rest)+ ('let _ ('package ('name name) ('version version) . rest)))++ `(define-public ,(string->symbol (if append-version?+ (string-append name "-" version)+ version)) ,guix-package)))) (define (build-system-modules)@@ -399,32 +403,43 @@ obtain a node's uniquely identifying \"key\"." (cons head result) (set-insert (node-name head) visited)))))))) -(define* (recursive-import package-name repo- #:key repo->guix-package guix-name+(define* (recursive-import package-name+ #:key repo->guix-package guix-name version repo #:allow-other-keys) "Return a list of package expressions for PACKAGE-NAME and all its dependencies, sorted in topological order. For each package,-call (REPO->GUIX-PACKAGE NAME REPO), which should return a package expression-and a list of dependencies; call (GUIX-NAME NAME) to obtain the Guix package-name corresponding to the upstream name."+call (REPO->GUIX-PACKAGE NAME :KEYS version repo), which should return a+package expression and a list of dependencies; call (GUIX-NAME NAME) to+obtain the Guix package name corresponding to the upstream name." (define-record-type <node>- (make-node name package dependencies)+ (make-node name version package dependencies) node? (name node-name)+ (version node-version) (package node-package) (dependencies node-dependencies)) - (define (exists? name)- (not (null? (find-packages-by-name (guix-name name)))))+ (define (exists? name version)+ (not (null? (find-packages-by-name (guix-name name) version)))) - (define (lookup-node name)- (receive (package dependencies) (repo->guix-package name repo)- (make-node name package dependencies)))+ (define (lookup-node name version)+ (let* ((package dependencies (repo->guix-package name+ #:version version+ #:repo repo))+ (normilizied-deps (map (match-lambda+ ((name version) (list name version))+ (name (list name #f))) dependencies)))+ (make-node name version package normilizied-deps))) (map node-package- (topological-sort (list (lookup-node package-name))+ (topological-sort (list (lookup-node package-name version)) (lambda (node)- (map lookup-node- (remove exists?+ (map (lambda (name-version)+ (apply lookup-node name-version))+ (remove (lambda (name-version)+ (apply exists? name-version)) (node-dependencies node))))- node-name)))+ (lambda (node)+ (string-append+ (node-name node)+ (or (node-version node) ""))))))diff --git a/guix/scripts/import/cran.scm b/guix/scripts/import/cran.scmindex d6f371ef3a..bc266ad9da 100644--- a/guix/scripts/import/cran.scm+++ b/guix/scripts/import/cran.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2015, 2017, 2019 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -98,10 +99,10 @@ Import and convert the CRAN package for PACKAGE-NAME.\n")) ;; Recursive import (map package->definition (cran-recursive-import package-name- (or (assoc-ref opts 'repo) 'cran)))+ #:repo (or (assoc-ref opts 'repo) 'cran))) ;; Single import (let ((sexp (cran->guix-package package-name- (or (assoc-ref opts 'repo) 'cran))))+ #:repo (or (assoc-ref opts 'repo) 'cran)))) (unless sexp (leave (G_ "failed to download description for package '~a'~%") package-name))diff --git a/guix/scripts/import/elpa.scm b/guix/scripts/import/elpa.scmindex d270d2b4bc..07ac07a3d5 100644--- a/guix/scripts/import/elpa.scm+++ b/guix/scripts/import/elpa.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -102,7 +103,8 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n")) (_ #f)) (elpa-recursive-import package-name (or (assoc-ref opts 'repo) 'gnu)))- (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo))))+ (let ((sexp (elpa->guix-package package-name+ #:repo (assoc-ref opts 'repo)))) (unless sexp (leave (G_ "failed to download package '~a'~%") package-name)) sexp)))diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex 87dda3238f..2357ea5c40 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -48,15 +49,16 @@ (package (name "foo") (inputs `(("bar" ,bar)))))- (recursive-import "foo" 'repo+ (recursive-import "foo"+ #:repo 'repo #:repo->guix-package (match-lambda*- (("foo" 'repo)+ (("foo" #:version #f #:repo 'repo) (values '(package (name "foo") (inputs `(("bar" ,bar)))) '("bar")))- (("bar" 'repo)+ (("bar" #:version #f #:repo 'repo) (values '(package (name "bar")) '())))-- 2.25.0
From a405e3ffdf2716b9920f6b74e4690c9b7731f67a Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Sat, 18 Jan 2020 10:44:18 -0500Subject: [PATCH v6 2/2] guix: import: crate: Use semver to resovle module versions
* guix/import/crate.scm (make-crate-sexp): formatting, added '#:skip-build?' to build system args; added package definition geneation* guix/import/crate.scm (crate->guix-package): [arguments] moved `verions` to a key. Use semver to resolve the correct module versions* guix/import/crate.scm (crate-name->package0name): [arguments] add #:optional `version` arguement* guix/scripts/import/crate.scm remove package definition generation; changed `version` to a key* tests/crate.scm: added version data to (recursuve-import) test--- guix/import/crate.scm | 140 +++++++++------- guix/scripts/import/crate.scm | 11 +- tests/crate.scm | 290 +++++++++++++++++++--------------- tests/elpa.scm | 3 +- 4 files changed, 258 insertions(+), 186 deletions(-)
Toggle diff (654 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 57823c3639..6847a7046b 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>-;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019, 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -28,6 +28,7 @@ #:use-module (guix import json) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:)+ #:use-module (guix memoization) #:use-module (guix monads) #:use-module (guix packages) #:use-module (guix upstream)@@ -35,9 +36,12 @@ #:use-module (ice-9 match) #:use-module (ice-9 regex) #:use-module (json)+ #:use-module (semver)+ #:use-module (semver ranges) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name string->license@@ -86,7 +90,7 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string- (kind crate-dependency-kind "kind" ;'normal | 'dev+ (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string @@ -105,6 +109,8 @@ record or #f if it was not found." (json->crate `(,@alist ("actual_versions" . ,versions)))))))) +(define mem-lookup-crate (memoize lookup-crate))+ (define (crate-version-dependencies version) "Return the list of <crate-dependency> records of VERSION, a <crate-version>."@@ -150,34 +156,40 @@ VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTIO and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name- cargo-development-inputs))+ (cargo-inputs+ (map+ (lambda (name-version)+ (apply crate-name->package-name name-version)) cargo-inputs))+ (cargo-development-inputs+ (map+ (lambda (name-version)+ (apply crate-name->package-name name-version)) cargo-development-inputs)) (pkg `(package- (name ,guix-name)- (version ,version)- (source (origin- (method url-fetch)- (uri (crate-uri ,name version))- (file-name (string-append name "-" version ".tar.gz"))- (sha256- (base32- ,(bytevector->nix-base32-string (port-sha256 port))))))- (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)- (maybe-cargo-development-inputs+ (name ,guix-name)+ (version ,version)+ (source (origin+ (method url-fetch)+ (uri (crate-uri ,name version))+ (file-name (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ ,(bytevector->nix-base32-string (port-sha256 port))))))+ (build-system cargo-build-system)+ ,@(maybe-arguments (append `(#:skip-build? #t)+ (maybe-cargo-inputs cargo-inputs)+ (maybe-cargo-development-inputs cargo-development-inputs)))- (home-page ,(match home-page- (() "")- (_ home-page)))- (synopsis ,synopsis)- (description ,(beautify-description description))- (license ,(match license- (() #f)- ((license) license)- (_ `(list ,@license)))))))- (close-port port)- pkg))+ (home-page ,(match home-page+ (() "")+ (_ home-page)))+ (synopsis ,synopsis)+ (description ,(beautify-description description))+ (license ,(match license+ (() #f)+ ((license) license)+ (_ `(list ,@license)))))))+ (close-port port)+ (package->definition pkg #t))) (define (string->license string) (filter-map (lambda (license)@@ -188,37 +200,60 @@ and LICENSE." 'unknown-license!))) (string-split string (string->char-set " /")))) -(define* (crate->guix-package crate-name #:optional version)+(define* (crate->guix-package crate-name #:key version #:allow-other-keys) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME." + (define (semver-range-contains-string? range version)+ (semver-range-contains? (string->semver-range range)+ (string->semver version)))+ (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))+ (or (eq? (crate-dependency-kind dependency) 'build)+ (eq? (crate-dependency-kind dependency) 'normal))) (define crate- (lookup-crate crate-name))+ (mem-lookup-crate crate-name)) (define version-number (or version (crate-latest-version crate))) - (define version*+ (define (find-version crate range)+ "finds the a vesion of a crate that fulfils the semver <range>" (find (lambda (version)- (string=? (crate-version-number version)- version-number))+ (semver-range-contains-string?+ range+ (crate-version-number version))) (crate-versions crate))) + (define version*+ (find-version crate version-number))++ (define (sort-map-deps deps)+ "sorts the dependencies and maps the dependencies to a list+ containing pairs of (name version)"+ (sort (map (lambda (dep)+ (let* ((name (crate-dependency-id dep))+ (crate (mem-lookup-crate name))+ (req (crate-dependency-requirement dep))+ (ver (find-version crate req)))+ (list name+ (crate-version-number ver))))+ deps)+ (match-lambda* (((_ name) ...)+ (apply string-ci<? name)))))+ (and crate version*- (let* ((dependencies (crate-version-dependencies version*))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))+ (let* ((dependencies (crate-version-dependencies version*))+ (dep-crates dev-dep-crates (partition normal-dependency? dependencies))+ (cargo-inputs (sort-map-deps dep-crates))+ ;; for now we are skipping the resolution of the development inputs+ ;; since most crates are libaries and we only want to test at the+ ;; app level. This probably should be parameterized though.+ (cargo-development-inputs '())) (values (make-crate-sexp #:name crate-name #:version (crate-version-number version*)@@ -230,15 +265,12 @@ latest version of CRATE-NAME." #:description (crate-description crate) #:license (and=> (crate-version-license version*) string->license))- (append cargo-inputs cargo-development-inputs)))))+ cargo-inputs)))) -(define* (crate-recursive-import crate-name #:optional version)- (recursive-import crate-name #f- #:repo->guix-package- (lambda (name repo)- (let ((version (and (string=? name crate-name)- version)))- (crate->guix-package name version)))+(define* (crate-recursive-import crate-name #:key version)+ (recursive-import crate-name+ #:repo->guix-package crate->guix-package+ #:version version #:guix-name crate-name->package-name)) (define (guix-package->crate-name package)@@ -252,8 +284,11 @@ latest version of CRATE-NAME." (match parts ((name _ ...) name)))) -(define (crate-name->package-name name)- (string-append "rust-" (string-join (string-split name #\_) "-")))+(define* (crate-name->package-name name #:optional version)+ (let ((name (guix-name "rust-" name)))+ (if version+ (string-append name "-" version)+ name))) ;;;@@ -288,4 +323,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release)))-diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex d834518c18..552628cfc7 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -2,7 +2,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 David Thompson <davet@gnu.org> ;;; Copyright © 2016 David Craven <david@craven.ch>-;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019, 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -95,13 +95,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (crate-recursive-import name version))- (let ((sexp (crate->guix-package name version)))+ (crate-recursive-import name #:version version)+ (let ((sexp (crate->guix-package name #:version version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%") (if versiondiff --git a/tests/crate.scm b/tests/crate.scmindex aa51faebf9..39561d5745 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2014 David Thompson <davet@gnu.org> ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -54,8 +55,9 @@ "{ \"dependencies\": [ {- \"crate_id\": \"bar\",+ \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } ] }")@@ -88,18 +90,22 @@ { \"crate_id\": \"intermediate-1\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, ] }")@@ -132,14 +138,17 @@ { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } ] }")@@ -172,6 +181,7 @@ { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, ] }")@@ -252,34 +262,48 @@ (open-input-string test-foo-crate)) ("https://crates.io/api/v1/crates/foo/1.0.0/download" (set! test-source-hash- (bytevector->nix-base32-string- (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies" (open-input-string test-foo-dependencies))+ ("https://crates.io/api/v1/crates/leaf-alice"+ (open-input-string test-leaf-alice-crate))+ ("https://crates.io/api/v1/crates/leaf-alice/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/leaf-alice/1.0.0/dependencies"+ (open-input-string test-leaf-alice-dependencies)) (_ (error "Unexpected URL: " url)))))- (match (crate->guix-package "foo")- (('package- ('name "rust-foo")- ('version "1.0.0")- ('source ('origin- ('method 'url-fetch)- ('uri ('crate-uri "foo" 'version))- ('file-name ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- (string=? test-source-hash hash))- (x- (pk 'fail x #f)))))++ (match (crate->guix-package "foo")+ ((define-public rust-foo-1.0.0+ (package (name "rust-foo")+ (version "1.0.0")+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "foo" 'version))+ (file-name (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system 'cargo-build-system)+ (arguments+ ('quasiquote+ (#:skip-build? #t+ #:cargo-inputs+ (("rust-leaf-alice-1.0.0" ('unquote rust-leaf-alice-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))++ (string=? test-source-hash hash))+ (x+ (pk 'fail x #f))))) (test-assert "cargo-recursive-import" ;; Replace network resources with sample data.@@ -334,105 +358,123 @@ (_ (error "Unexpected URL: " url))))) (match (crate-recursive-import "root") ;; rust-intermediate-2 has no dependency on the rust-leaf-alice package, so this is a valid ordering- ((('package- ('name "rust-leaf-alice")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "leaf-alice" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-leaf-bob")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "leaf-bob" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-intermediate-2")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "intermediate-2" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-intermediate-1")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "intermediate-1" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-intermediate-2" ('unquote rust-intermediate-2))- ("rust-leaf-alice" ('unquote rust-leaf-alice))- ("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-root")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "root" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-intermediate-1" ('unquote rust-intermediate-1))- ("rust-intermediate-2" ('unquote rust-intermediate-2))- ("rust-leaf-alice" ('unquote rust-leaf-alice))- ("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0))))+ (((define-public rust-leaf-alice-1.0.0+ (package+ (name "rust-leaf-alice")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "leaf-alice" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments ('quasiquote (#:skip-build? #t)))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-leaf-bob-1.0.0+ (package+ (name "rust-leaf-bob")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "leaf-bob" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments ('quasiquote (#:skip-build? #t)))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-intermediate-2-1.0.0+ (package+ (name "rust-intermediate-2")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "intermediate-2" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build? #t+ #:cargo-inputs+ (("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-intermediate-1-1.0.0+ (package+ (name "rust-intermediate-1")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "intermediate-1" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build? #t+ #:cargo-inputs+ (("rust-intermediate-2-1.0.0"+ ,rust-intermediate-2-1.0.0)+ ("rust-leaf-alice-1.0.0"+ ('unquote rust-leaf-alice-1.0.0))+ ("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-root-1.0.0+ (package+ (name "rust-root")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "root" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build?+ #t #:cargo-inputs+ (("rust-intermediate-1-1.0.0"+ ('unquote rust-intermediate-1-1.0.0))+ ("rust-intermediate-2-1.0.0"+ ('unquote rust-intermediate-2-1.0.0))+ ("rust-leaf-alice-1.0.0"+ ('unquote rust-leaf-alice-1.0.0))+ ("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))) #t) (x (pk 'fail x #f)))))diff --git a/tests/elpa.scm b/tests/elpa.scmindex b70539bda6..a008cf993c 100644--- a/tests/elpa.scm+++ b/tests/elpa.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -51,7 +52,7 @@ (200 "This is the description.") (200 "fake tarball contents")) (parameterize ((current-http-proxy (%local-url)))- (match (elpa->guix-package pkg 'gnu/http)+ (match (elpa->guix-package pkg #:repo 'gnu/http) (('package ('name "emacs-auctex") ('version "11.88.6")-- 2.25.0
M
M
Martin Becze wrote on 25 Jan 2020 11:17
Re: Importers as independent packages?
(name . Ludovic Courtès)(address . ludo@gnu.org)
5ff41b52-03b6-c223-862f-2b480c1eed48@riseup.net
Thank you Ludo,I added guile-semver to guix in the attached patch set, and I tested it by running ./pre-inst-env guix environment guix, which installed the guile-semver.
Toggle quote (2 lines)> and Makefile.am may have to check whether guile-semver is available.)
I didn't see anything in the Makefile.am that looks to check for guile modules. Let me know if anything needs fixing!
=Martin
From e4b022ce72582691dadae1b9f31ad6243914a5db Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Sat, 18 Jan 2020 05:05:03 -0500Subject: [PATCH v7 1/3] guix: import: (recursive-import) Allow for version numbers
* guix/import/utils.scm (package->definition) [arguments] added optional `append-version?`* guix/import/utils.scm (recursive-import) [arguments] added key `version` and moved `repo` to be a key* tests/import-utils.scm
* guix/import/cran.scm (cran->guix-package) [argument]: change `repo` to a key* guix/import/cran.scm (cran-recursive-import) [argument]: change `repo` to a key* guix/scripts/import/cran.scm: change `repo` to a key* guix/import/elpa.scm (elpa->guix-pakcage) [argumnets]: change `repo` to a key* guix/import/elpa.scm (elpa-recursive-import) [argumnets]: change `repo` to a key* guix/scripts/import/elpa.scm: change `repo` to a key* guix/import/gem.scm (gem->guix-package) [arguments]: change `repo` to a key* guix/import/gem.scm (recursive-import) [arguments]: change `repo` to a key* guix/import/opam.scm (opam-recurive-import) [arguments]: change `repo` to a key* guix/import/pypi.scm (pypi-recursive-import) [arguments]: change `repo` to a key* guix/import/stackage.scm (stackage-recursive-import) [arguments]: change `repo` to a key--- guix/import/cran.scm | 8 +++-- guix/import/elpa.scm | 6 ++-- guix/import/gem.scm | 6 ++-- guix/import/opam.scm | 5 ++-- guix/import/pypi.scm | 5 ++-- guix/import/stackage.scm | 5 ++-- guix/import/utils.scm | 57 +++++++++++++++++++++++------------- guix/scripts/import/cran.scm | 5 ++-- guix/scripts/import/elpa.scm | 4 ++- tests/import-utils.scm | 8 +++-- 10 files changed, 69 insertions(+), 40 deletions(-)
Toggle diff (344 lines)diff --git a/guix/import/cran.scm b/guix/import/cran.scmindex bcb37ed250..9e05dfcba8 100644--- a/guix/import/cran.scm+++ b/guix/import/cran.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015, 2016, 2017, 2018, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2015, 2016, 2017, 2019, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -506,7 +507,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file." (define cran->guix-package (memoize- (lambda* (package-name #:optional (repo 'cran))+ (lambda* (package-name #:key (repo 'cran) #:allow-other-keys) "Fetch the metadata for PACKAGE-NAME from REPO and return the `package' s-expression corresponding to that package, or #f on failure." (let ((description (fetch-description repo package-name)))@@ -521,8 +522,9 @@ s-expression corresponding to that package, or #f on failure." (cran->guix-package package-name 'cran)) (else (values #f '())))))))) -(define* (cran-recursive-import package-name #:optional (repo 'cran))- (recursive-import package-name repo+(define* (cran-recursive-import package-name #:key (repo 'cran))+ (recursive-import package-name+ #:repo repo #:repo->guix-package cran->guix-package #:guix-name cran-guix-name)) diff --git a/guix/import/elpa.scm b/guix/import/elpa.scmindex 2d4487dba0..9140bcdc34 100644--- a/guix/import/elpa.scm+++ b/guix/import/elpa.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2015, 2016, 2017, 2018, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -245,7 +246,7 @@ type '<elpa-package>'." (license ,license)) dependencies-names))) -(define* (elpa->guix-package name #:optional (repo 'gnu))+(define* (elpa->guix-package name #:key (repo 'gnu) #:allow-other-keys) "Fetch the package NAME from REPO and produce a Guix package S-expression." (match (fetch-elpa-package name repo) (#f #f)@@ -301,7 +302,8 @@ type '<elpa-package>'." (define elpa-guix-name (cut guix-name "emacs-" <>)) (define* (elpa-recursive-import package-name #:optional (repo 'gnu))- (recursive-import package-name repo+ (recursive-import package-name+ #:repo repo #:repo->guix-package elpa->guix-package #:guix-name elpa-guix-name)) diff --git a/guix/import/gem.scm b/guix/import/gem.scmindex 0bf9ff2552..e744d9e69d 100644--- a/guix/import/gem.scm+++ b/guix/import/gem.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015 David Thompson <davet@gnu.org> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -117,7 +118,7 @@ VERSION, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES." ((license) (license->symbol license)) (_ `(list ,@(map license->symbol licenses))))))) -(define* (gem->guix-package package-name #:optional (repo 'rubygems) version)+(define* (gem->guix-package package-name #:key (repo 'rubygems) version) "Fetch the metadata for PACKAGE-NAME from rubygems.org, and return the `package' s-expression corresponding to that package, or #f on failure." (let ((package (rubygems-fetch package-name)))@@ -201,6 +202,7 @@ package on RubyGems." (latest latest-release))) (define* (gem-recursive-import package-name #:optional version)- (recursive-import package-name '()+ (recursive-import package-name+ #:repo '() #:repo->guix-package gem->guix-package #:guix-name ruby-package-name))diff --git a/guix/import/opam.scm b/guix/import/opam.scmindex 394415fdd4..87c823a98c 100644--- a/guix/import/opam.scm+++ b/guix/import/opam.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2018 Julien Lepiller <julien@lepiller.eu>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -311,8 +312,8 @@ or #f on failure." dependencies)))))))) (define (opam-recursive-import package-name)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (opam->guix-package name)) #:guix-name ocaml-name->guix-name)) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scmindex 354cae9c4c..f0702d6403 100644--- a/guix/import/pypi.scm+++ b/guix/import/pypi.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com> ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -415,8 +416,8 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." description license)))))))) (define (pypi-recursive-import package-name)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (pypi->guix-package name)) #:guix-name python->package-name)) diff --git a/guix/import/stackage.scm b/guix/import/stackage.scmindex 14150201b5..6091cf2c64 100644--- a/guix/import/stackage.scm+++ b/guix/import/stackage.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -108,8 +109,8 @@ included in the Stackage LTS release." (leave-with-message "~a: Stackage package not found" package-name)))))) (define (stackage-recursive-import package-name . args)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (apply stackage->guix-package (cons name args))) #:guix-name hackage-name->package-name)) diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex d17d400ddf..59430d3e66 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2019 Robert Vollmert <rob@vllmrt.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -44,6 +45,7 @@ #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (factorize-uri flatten@@ -258,13 +260,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package)+(define* (package->definition guix-package #:optional append-version?) (match guix-package- (('package ('name (? string? name)) _ ...)- `(define-public ,(string->symbol name)- ,guix-package))- (('let anything ('package ('name (? string? name)) _ ...))- `(define-public ,(string->symbol name)+ ((or+ ('package ('name name) ('version version) . rest)+ ('let _ ('package ('name name) ('version version) . rest)))++ `(define-public ,(string->symbol (if append-version?+ (string-append name "-" version)+ version)) ,guix-package)))) (define (build-system-modules)@@ -399,32 +403,43 @@ obtain a node's uniquely identifying \"key\"." (cons head result) (set-insert (node-name head) visited)))))))) -(define* (recursive-import package-name repo- #:key repo->guix-package guix-name+(define* (recursive-import package-name+ #:key repo->guix-package guix-name version repo #:allow-other-keys) "Return a list of package expressions for PACKAGE-NAME and all its dependencies, sorted in topological order. For each package,-call (REPO->GUIX-PACKAGE NAME REPO), which should return a package expression-and a list of dependencies; call (GUIX-NAME NAME) to obtain the Guix package-name corresponding to the upstream name."+call (REPO->GUIX-PACKAGE NAME :KEYS version repo), which should return a+package expression and a list of dependencies; call (GUIX-NAME NAME) to+obtain the Guix package name corresponding to the upstream name." (define-record-type <node>- (make-node name package dependencies)+ (make-node name version package dependencies) node? (name node-name)+ (version node-version) (package node-package) (dependencies node-dependencies)) - (define (exists? name)- (not (null? (find-packages-by-name (guix-name name)))))+ (define (exists? name version)+ (not (null? (find-packages-by-name (guix-name name) version)))) - (define (lookup-node name)- (receive (package dependencies) (repo->guix-package name repo)- (make-node name package dependencies)))+ (define (lookup-node name version)+ (let* ((package dependencies (repo->guix-package name+ #:version version+ #:repo repo))+ (normilizied-deps (map (match-lambda+ ((name version) (list name version))+ (name (list name #f))) dependencies)))+ (make-node name version package normilizied-deps))) (map node-package- (topological-sort (list (lookup-node package-name))+ (topological-sort (list (lookup-node package-name version)) (lambda (node)- (map lookup-node- (remove exists?+ (map (lambda (name-version)+ (apply lookup-node name-version))+ (remove (lambda (name-version)+ (apply exists? name-version)) (node-dependencies node))))- node-name)))+ (lambda (node)+ (string-append+ (node-name node)+ (or (node-version node) ""))))))diff --git a/guix/scripts/import/cran.scm b/guix/scripts/import/cran.scmindex d6f371ef3a..bc266ad9da 100644--- a/guix/scripts/import/cran.scm+++ b/guix/scripts/import/cran.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2015, 2017, 2019 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -98,10 +99,10 @@ Import and convert the CRAN package for PACKAGE-NAME.\n")) ;; Recursive import (map package->definition (cran-recursive-import package-name- (or (assoc-ref opts 'repo) 'cran)))+ #:repo (or (assoc-ref opts 'repo) 'cran))) ;; Single import (let ((sexp (cran->guix-package package-name- (or (assoc-ref opts 'repo) 'cran))))+ #:repo (or (assoc-ref opts 'repo) 'cran)))) (unless sexp (leave (G_ "failed to download description for package '~a'~%") package-name))diff --git a/guix/scripts/import/elpa.scm b/guix/scripts/import/elpa.scmindex d270d2b4bc..07ac07a3d5 100644--- a/guix/scripts/import/elpa.scm+++ b/guix/scripts/import/elpa.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -102,7 +103,8 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n")) (_ #f)) (elpa-recursive-import package-name (or (assoc-ref opts 'repo) 'gnu)))- (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo))))+ (let ((sexp (elpa->guix-package package-name+ #:repo (assoc-ref opts 'repo)))) (unless sexp (leave (G_ "failed to download package '~a'~%") package-name)) sexp)))diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex 87dda3238f..2357ea5c40 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -48,15 +49,16 @@ (package (name "foo") (inputs `(("bar" ,bar)))))- (recursive-import "foo" 'repo+ (recursive-import "foo"+ #:repo 'repo #:repo->guix-package (match-lambda*- (("foo" 'repo)+ (("foo" #:version #f #:repo 'repo) (values '(package (name "foo") (inputs `(("bar" ,bar)))) '("bar")))- (("bar" 'repo)+ (("bar" #:version #f #:repo 'repo) (values '(package (name "bar")) '())))-- 2.25.0
From 3ecb0a58a6d2fbb30e36d9fa3648afe932f6e465 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Sat, 18 Jan 2020 10:44:18 -0500Subject: [PATCH v7 2/3] guix: import: crate: Use semver to resovle module versions
* guix/import/crate.scm (make-crate-sexp): formatting, added '#:skip-build?' to build system args; added package definition geneation* guix/import/crate.scm (crate->guix-package): [arguments] moved `verions` to a key. Use semver to resolve the correct module versions* guix/import/crate.scm (crate-name->package0name): [arguments] add #:optional `version` arguement* guix/scripts/import/crate.scm remove package definition generation; changed `version` to a key* tests/crate.scm: added version data to (recursuve-import) test--- guix/import/crate.scm | 140 +++++++++------- guix/scripts/import/crate.scm | 11 +- tests/crate.scm | 290 +++++++++++++++++++--------------- tests/elpa.scm | 3 +- 4 files changed, 258 insertions(+), 186 deletions(-)
Toggle diff (654 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 57823c3639..6847a7046b 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>-;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019, 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -28,6 +28,7 @@ #:use-module (guix import json) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:)+ #:use-module (guix memoization) #:use-module (guix monads) #:use-module (guix packages) #:use-module (guix upstream)@@ -35,9 +36,12 @@ #:use-module (ice-9 match) #:use-module (ice-9 regex) #:use-module (json)+ #:use-module (semver)+ #:use-module (semver ranges) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name string->license@@ -86,7 +90,7 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string- (kind crate-dependency-kind "kind" ;'normal | 'dev+ (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string @@ -105,6 +109,8 @@ record or #f if it was not found." (json->crate `(,@alist ("actual_versions" . ,versions)))))))) +(define mem-lookup-crate (memoize lookup-crate))+ (define (crate-version-dependencies version) "Return the list of <crate-dependency> records of VERSION, a <crate-version>."@@ -150,34 +156,40 @@ VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTIO and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name- cargo-development-inputs))+ (cargo-inputs+ (map+ (lambda (name-version)+ (apply crate-name->package-name name-version)) cargo-inputs))+ (cargo-development-inputs+ (map+ (lambda (name-version)+ (apply crate-name->package-name name-version)) cargo-development-inputs)) (pkg `(package- (name ,guix-name)- (version ,version)- (source (origin- (method url-fetch)- (uri (crate-uri ,name version))- (file-name (string-append name "-" version ".tar.gz"))- (sha256- (base32- ,(bytevector->nix-base32-string (port-sha256 port))))))- (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)- (maybe-cargo-development-inputs+ (name ,guix-name)+ (version ,version)+ (source (origin+ (method url-fetch)+ (uri (crate-uri ,name version))+ (file-name (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ ,(bytevector->nix-base32-string (port-sha256 port))))))+ (build-system cargo-build-system)+ ,@(maybe-arguments (append `(#:skip-build? #t)+ (maybe-cargo-inputs cargo-inputs)+ (maybe-cargo-development-inputs cargo-development-inputs)))- (home-page ,(match home-page- (() "")- (_ home-page)))- (synopsis ,synopsis)- (description ,(beautify-description description))- (license ,(match license- (() #f)- ((license) license)- (_ `(list ,@license)))))))- (close-port port)- pkg))+ (home-page ,(match home-page+ (() "")+ (_ home-page)))+ (synopsis ,synopsis)+ (description ,(beautify-description description))+ (license ,(match license+ (() #f)+ ((license) license)+ (_ `(list ,@license)))))))+ (close-port port)+ (package->definition pkg #t))) (define (string->license string) (filter-map (lambda (license)@@ -188,37 +200,60 @@ and LICENSE." 'unknown-license!))) (string-split string (string->char-set " /")))) -(define* (crate->guix-package crate-name #:optional version)+(define* (crate->guix-package crate-name #:key version #:allow-other-keys) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME." + (define (semver-range-contains-string? range version)+ (semver-range-contains? (string->semver-range range)+ (string->semver version)))+ (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))+ (or (eq? (crate-dependency-kind dependency) 'build)+ (eq? (crate-dependency-kind dependency) 'normal))) (define crate- (lookup-crate crate-name))+ (mem-lookup-crate crate-name)) (define version-number (or version (crate-latest-version crate))) - (define version*+ (define (find-version crate range)+ "finds the a vesion of a crate that fulfils the semver <range>" (find (lambda (version)- (string=? (crate-version-number version)- version-number))+ (semver-range-contains-string?+ range+ (crate-version-number version))) (crate-versions crate))) + (define version*+ (find-version crate version-number))++ (define (sort-map-deps deps)+ "sorts the dependencies and maps the dependencies to a list+ containing pairs of (name version)"+ (sort (map (lambda (dep)+ (let* ((name (crate-dependency-id dep))+ (crate (mem-lookup-crate name))+ (req (crate-dependency-requirement dep))+ (ver (find-version crate req)))+ (list name+ (crate-version-number ver))))+ deps)+ (match-lambda* (((_ name) ...)+ (apply string-ci<? name)))))+ (and crate version*- (let* ((dependencies (crate-version-dependencies version*))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))+ (let* ((dependencies (crate-version-dependencies version*))+ (dep-crates dev-dep-crates (partition normal-dependency? dependencies))+ (cargo-inputs (sort-map-deps dep-crates))+ ;; for now we are skipping the resolution of the development inputs+ ;; since most crates are libaries and we only want to test at the+ ;; app level. This probably should be parameterized though.+ (cargo-development-inputs '())) (values (make-crate-sexp #:name crate-name #:version (crate-version-number version*)@@ -230,15 +265,12 @@ latest version of CRATE-NAME." #:description (crate-description crate) #:license (and=> (crate-version-license version*) string->license))- (append cargo-inputs cargo-development-inputs)))))+ cargo-inputs)))) -(define* (crate-recursive-import crate-name #:optional version)- (recursive-import crate-name #f- #:repo->guix-package- (lambda (name repo)- (let ((version (and (string=? name crate-name)- version)))- (crate->guix-package name version)))+(define* (crate-recursive-import crate-name #:key version)+ (recursive-import crate-name+ #:repo->guix-package crate->guix-package+ #:version version #:guix-name crate-name->package-name)) (define (guix-package->crate-name package)@@ -252,8 +284,11 @@ latest version of CRATE-NAME." (match parts ((name _ ...) name)))) -(define (crate-name->package-name name)- (string-append "rust-" (string-join (string-split name #\_) "-")))+(define* (crate-name->package-name name #:optional version)+ (let ((name (guix-name "rust-" name)))+ (if version+ (string-append name "-" version)+ name))) ;;;@@ -288,4 +323,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release)))-diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex d834518c18..552628cfc7 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -2,7 +2,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 David Thompson <davet@gnu.org> ;;; Copyright © 2016 David Craven <david@craven.ch>-;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019, 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -95,13 +95,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (crate-recursive-import name version))- (let ((sexp (crate->guix-package name version)))+ (crate-recursive-import name #:version version)+ (let ((sexp (crate->guix-package name #:version version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%") (if versiondiff --git a/tests/crate.scm b/tests/crate.scmindex aa51faebf9..39561d5745 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2014 David Thompson <davet@gnu.org> ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -54,8 +55,9 @@ "{ \"dependencies\": [ {- \"crate_id\": \"bar\",+ \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } ] }")@@ -88,18 +90,22 @@ { \"crate_id\": \"intermediate-1\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, ] }")@@ -132,14 +138,17 @@ { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } ] }")@@ -172,6 +181,7 @@ { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, ] }")@@ -252,34 +262,48 @@ (open-input-string test-foo-crate)) ("https://crates.io/api/v1/crates/foo/1.0.0/download" (set! test-source-hash- (bytevector->nix-base32-string- (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies" (open-input-string test-foo-dependencies))+ ("https://crates.io/api/v1/crates/leaf-alice"+ (open-input-string test-leaf-alice-crate))+ ("https://crates.io/api/v1/crates/leaf-alice/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/leaf-alice/1.0.0/dependencies"+ (open-input-string test-leaf-alice-dependencies)) (_ (error "Unexpected URL: " url)))))- (match (crate->guix-package "foo")- (('package- ('name "rust-foo")- ('version "1.0.0")- ('source ('origin- ('method 'url-fetch)- ('uri ('crate-uri "foo" 'version))- ('file-name ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- (string=? test-source-hash hash))- (x- (pk 'fail x #f)))))++ (match (crate->guix-package "foo")+ ((define-public rust-foo-1.0.0+ (package (name "rust-foo")+ (version "1.0.0")+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "foo" 'version))+ (file-name (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system 'cargo-build-system)+ (arguments+ ('quasiquote+ (#:skip-build? #t+ #:cargo-inputs+ (("rust-leaf-alice-1.0.0" ('unquote rust-leaf-alice-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))++ (string=? test-source-hash hash))+ (x+ (pk 'fail x #f))))) (test-assert "cargo-recursive-import" ;; Replace network resources with sample data.@@ -334,105 +358,123 @@ (_ (error "Unexpected URL: " url))))) (match (crate-recursive-import "root") ;; rust-intermediate-2 has no dependency on the rust-leaf-alice package, so this is a valid ordering- ((('package- ('name "rust-leaf-alice")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "leaf-alice" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-leaf-bob")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "leaf-bob" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-intermediate-2")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "intermediate-2" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-intermediate-1")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "intermediate-1" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-intermediate-2" ('unquote rust-intermediate-2))- ("rust-leaf-alice" ('unquote rust-leaf-alice))- ("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-root")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "root" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-intermediate-1" ('unquote rust-intermediate-1))- ("rust-intermediate-2" ('unquote rust-intermediate-2))- ("rust-leaf-alice" ('unquote rust-leaf-alice))- ("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0))))+ (((define-public rust-leaf-alice-1.0.0+ (package+ (name "rust-leaf-alice")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "leaf-alice" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments ('quasiquote (#:skip-build? #t)))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-leaf-bob-1.0.0+ (package+ (name "rust-leaf-bob")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "leaf-bob" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments ('quasiquote (#:skip-build? #t)))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-intermediate-2-1.0.0+ (package+ (name "rust-intermediate-2")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "intermediate-2" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build? #t+ #:cargo-inputs+ (("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-intermediate-1-1.0.0+ (package+ (name "rust-intermediate-1")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "intermediate-1" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build? #t+ #:cargo-inputs+ (("rust-intermediate-2-1.0.0"+ ,rust-intermediate-2-1.0.0)+ ("rust-leaf-alice-1.0.0"+ ('unquote rust-leaf-alice-1.0.0))+ ("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-root-1.0.0+ (package+ (name "rust-root")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "root" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build?+ #t #:cargo-inputs+ (("rust-intermediate-1-1.0.0"+ ('unquote rust-intermediate-1-1.0.0))+ ("rust-intermediate-2-1.0.0"+ ('unquote rust-intermediate-2-1.0.0))+ ("rust-leaf-alice-1.0.0"+ ('unquote rust-leaf-alice-1.0.0))+ ("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))) #t) (x (pk 'fail x #f)))))diff --git a/tests/elpa.scm b/tests/elpa.scmindex b70539bda6..a008cf993c 100644--- a/tests/elpa.scm+++ b/tests/elpa.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -51,7 +52,7 @@ (200 "This is the description.") (200 "fake tarball contents")) (parameterize ((current-http-proxy (%local-url)))- (match (elpa->guix-package pkg 'gnu/http)+ (match (elpa->guix-package pkg #:repo 'gnu/http) (('package ('name "emacs-auctex") ('version "11.88.6")-- 2.25.0
From c68ac6dc2ae5c3e2ffbbccfd05ad14e9eb3fda60 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Sat, 25 Jan 2020 04:57:13 -0500Subject: [PATCH v7 3/3] Added Guile-Semver as a dependency to guix
* configure.ac: added check for guile-semver* gnu/packages/package-management.scm (guix): added guile-semver as dep--- configure.ac | 7 +++++++ doc/guix.texi | 2 ++ gnu/packages/package-management.scm | 7 +++++-- 3 files changed, 14 insertions(+), 2 deletions(-)
Toggle diff (79 lines)diff --git a/configure.ac b/configure.acindex 06e86c209f..461ccaa8e7 100644--- a/configure.ac+++ b/configure.ac@@ -118,12 +118,19 @@ if test "x$have_guile_git" != "xyes"; then AC_MSG_ERROR([Guile-Git is missing; please install it.]) fi +dnl Check for Guile-Semver+GUILE_MODULE_AVAILABLE([have_guile_semver], [(semver)])+if test "x$have_guile_semver" != "xyes"; then+ AC_MSG_ERROR([Guile-Semver is missing; please install it.])+fi+ dnl Check for Guile-JSON. GUIX_CHECK_GUILE_JSON if test "x$guix_cv_have_recent_guile_json" != "xyes"; then AC_MSG_ERROR([Guile-JSON is missing; please install it.]) fi + dnl Guile-Sqlite3 is used by the (guix store ...) modules. GUIX_CHECK_GUILE_SQLITE3 if test "x$guix_cv_have_recent_guile_sqlite3" != "xyes"; thendiff --git a/doc/guix.texi b/doc/guix.texiindex d674b9484f..eb6d980bca 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -72,6 +72,7 @@ Copyright @copyright{} 2019 Guillaume Le Vaillant@* Copyright @copyright{} 2020 Leo Prikler@* Copyright @copyright{} 2019, 2020 Simon Tournier@* Copyright @copyright{} 2020 Wiktor Żelazny@*+Copyright @copyright{} 2020 Martin Becze@* Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or@@ -761,6 +762,7 @@ or later; @uref{https://gitlab.com/guile-git/guile-git, Guile-Git}, from August 2017 or later; @item @uref{https://savannah.nongnu.org/projects/guile-json/, Guile-JSON} 3.x;+@item @uref{https://ngyro.com/software/guile-semver.html, Guile-Semver} 0.1.x; @item @url{https://zlib.net, zlib}; @item @url{https://www.gnu.org/software/make/, GNU Make}. @end itemizediff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scmindex 422d4f1959..c456071a87 100644--- a/gnu/packages/package-management.scm+++ b/gnu/packages/package-management.scm@@ -11,6 +11,7 @@ ;;; Copyright © 2018, 2019 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2019, 2020 Efraim Flashner <efraim@flashner.co.il> ;;; Copyright © 2019 Jonathan Brielmaier <jonathan.brielmaier@web.de>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -248,8 +249,9 @@ (ssh (assoc-ref inputs "guile-ssh")) (gnutls (assoc-ref inputs "gnutls")) (locales (assoc-ref inputs "glibc-utf8-locales"))+ (semver (assoc-ref inputs "guile-semver")) (deps (list gcrypt json sqlite gnutls- git bs ssh))+ git bs ssh semver)) (effective (read-line (open-pipe* OPEN_READ@@ -322,7 +324,8 @@ ("guile-json" ,guile-json-3) ("guile-sqlite3" ,guile-sqlite3) ("guile-ssh" ,guile-ssh)- ("guile-git" ,guile-git)))+ ("guile-git" ,guile-git)+ ("guile-semver",guile-semver))) (home-page "https://www.gnu.org/software/guix/") (synopsis "Functional package manager for installed software packages and versions")-- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 1/8] guix: import: (recursive-import) Allow for version numbers
(address . 38408@debbugs.gnu.org)
4b0a9afd6559ee87734ef06d1d8adc0e60e63ee2.1580817140.git.mjbecze@riseup.net
This adds a key VERSION to (recursive-import) and move the paramter REPO to akey. This also changes all the things that rely on (recursive-import)
* guix/import/utils.scm (package->definition): added optional `append-version?`* guix/import/utils.scm (recursive-import): added key `version` and moved `repo` to be a key
* guix/import/cran.scm (cran->guix-package): change `repo` to a key* guix/import/cran.scm (cran-recursive-import): change `repo` to a key* guix/scripts/import/cran.scm: change `repo` to a key* guix/import/elpa.scm (elpa->guix-pakcage): change `repo` to a key* guix/import/elpa.scm (elpa-recursive-import): change `repo` to a key* guix/scripts/import/elpa.scm: change `repo` to a key* guix/import/gem.scm (gem->guix-package): change `repo` to a key* guix/import/gem.scm (recursive-import): change `repo` to a key* guix/import/opam.scm (opam-recurive-import): change `repo` to a key* guix/import/pypi.scm (pypi-recursive-import): change `repo` to a key* guix/import/stackage.scm (stackage-recursive-import): change `repo` to a key--- guix/import/cran.scm | 8 +++-- guix/import/elpa.scm | 6 ++-- guix/import/gem.scm | 6 ++-- guix/import/opam.scm | 5 ++-- guix/import/pypi.scm | 5 ++-- guix/import/stackage.scm | 5 ++-- guix/import/utils.scm | 57 +++++++++++++++++++++++------------- guix/scripts/import/cran.scm | 5 ++-- guix/scripts/import/elpa.scm | 4 ++- tests/elpa.scm | 3 +- tests/import-utils.scm | 8 +++-- 11 files changed, 71 insertions(+), 41 deletions(-)
Toggle diff (365 lines)diff --git a/guix/import/cran.scm b/guix/import/cran.scmindex bcb37ed250..9e05dfcba8 100644--- a/guix/import/cran.scm+++ b/guix/import/cran.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015, 2016, 2017, 2018, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2015, 2016, 2017, 2019, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -506,7 +507,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file." (define cran->guix-package (memoize- (lambda* (package-name #:optional (repo 'cran))+ (lambda* (package-name #:key (repo 'cran) #:allow-other-keys) "Fetch the metadata for PACKAGE-NAME from REPO and return the `package' s-expression corresponding to that package, or #f on failure." (let ((description (fetch-description repo package-name)))@@ -521,8 +522,9 @@ s-expression corresponding to that package, or #f on failure." (cran->guix-package package-name 'cran)) (else (values #f '())))))))) -(define* (cran-recursive-import package-name #:optional (repo 'cran))- (recursive-import package-name repo+(define* (cran-recursive-import package-name #:key (repo 'cran))+ (recursive-import package-name+ #:repo repo #:repo->guix-package cran->guix-package #:guix-name cran-guix-name)) diff --git a/guix/import/elpa.scm b/guix/import/elpa.scmindex 2d4487dba0..9140bcdc34 100644--- a/guix/import/elpa.scm+++ b/guix/import/elpa.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2015, 2016, 2017, 2018, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -245,7 +246,7 @@ type '<elpa-package>'." (license ,license)) dependencies-names))) -(define* (elpa->guix-package name #:optional (repo 'gnu))+(define* (elpa->guix-package name #:key (repo 'gnu) #:allow-other-keys) "Fetch the package NAME from REPO and produce a Guix package S-expression." (match (fetch-elpa-package name repo) (#f #f)@@ -301,7 +302,8 @@ type '<elpa-package>'." (define elpa-guix-name (cut guix-name "emacs-" <>)) (define* (elpa-recursive-import package-name #:optional (repo 'gnu))- (recursive-import package-name repo+ (recursive-import package-name+ #:repo repo #:repo->guix-package elpa->guix-package #:guix-name elpa-guix-name)) diff --git a/guix/import/gem.scm b/guix/import/gem.scmindex 0bf9ff2552..e744d9e69d 100644--- a/guix/import/gem.scm+++ b/guix/import/gem.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015 David Thompson <davet@gnu.org> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -117,7 +118,7 @@ VERSION, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES." ((license) (license->symbol license)) (_ `(list ,@(map license->symbol licenses))))))) -(define* (gem->guix-package package-name #:optional (repo 'rubygems) version)+(define* (gem->guix-package package-name #:key (repo 'rubygems) version) "Fetch the metadata for PACKAGE-NAME from rubygems.org, and return the `package' s-expression corresponding to that package, or #f on failure." (let ((package (rubygems-fetch package-name)))@@ -201,6 +202,7 @@ package on RubyGems." (latest latest-release))) (define* (gem-recursive-import package-name #:optional version)- (recursive-import package-name '()+ (recursive-import package-name+ #:repo '() #:repo->guix-package gem->guix-package #:guix-name ruby-package-name))diff --git a/guix/import/opam.scm b/guix/import/opam.scmindex 394415fdd4..87c823a98c 100644--- a/guix/import/opam.scm+++ b/guix/import/opam.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2018 Julien Lepiller <julien@lepiller.eu>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -311,8 +312,8 @@ or #f on failure." dependencies)))))))) (define (opam-recursive-import package-name)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (opam->guix-package name)) #:guix-name ocaml-name->guix-name)) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scmindex 354cae9c4c..f0702d6403 100644--- a/guix/import/pypi.scm+++ b/guix/import/pypi.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com> ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -415,8 +416,8 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." description license)))))))) (define (pypi-recursive-import package-name)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (pypi->guix-package name)) #:guix-name python->package-name)) diff --git a/guix/import/stackage.scm b/guix/import/stackage.scmindex 14150201b5..6091cf2c64 100644--- a/guix/import/stackage.scm+++ b/guix/import/stackage.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -108,8 +109,8 @@ included in the Stackage LTS release." (leave-with-message "~a: Stackage package not found" package-name)))))) (define (stackage-recursive-import package-name . args)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (apply stackage->guix-package (cons name args))) #:guix-name hackage-name->package-name)) diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex d17d400ddf..59430d3e66 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2019 Robert Vollmert <rob@vllmrt.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -44,6 +45,7 @@ #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (factorize-uri flatten@@ -258,13 +260,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package)+(define* (package->definition guix-package #:optional append-version?) (match guix-package- (('package ('name (? string? name)) _ ...)- `(define-public ,(string->symbol name)- ,guix-package))- (('let anything ('package ('name (? string? name)) _ ...))- `(define-public ,(string->symbol name)+ ((or+ ('package ('name name) ('version version) . rest)+ ('let _ ('package ('name name) ('version version) . rest)))++ `(define-public ,(string->symbol (if append-version?+ (string-append name "-" version)+ version)) ,guix-package)))) (define (build-system-modules)@@ -399,32 +403,43 @@ obtain a node's uniquely identifying \"key\"." (cons head result) (set-insert (node-name head) visited)))))))) -(define* (recursive-import package-name repo- #:key repo->guix-package guix-name+(define* (recursive-import package-name+ #:key repo->guix-package guix-name version repo #:allow-other-keys) "Return a list of package expressions for PACKAGE-NAME and all its dependencies, sorted in topological order. For each package,-call (REPO->GUIX-PACKAGE NAME REPO), which should return a package expression-and a list of dependencies; call (GUIX-NAME NAME) to obtain the Guix package-name corresponding to the upstream name."+call (REPO->GUIX-PACKAGE NAME :KEYS version repo), which should return a+package expression and a list of dependencies; call (GUIX-NAME NAME) to+obtain the Guix package name corresponding to the upstream name." (define-record-type <node>- (make-node name package dependencies)+ (make-node name version package dependencies) node? (name node-name)+ (version node-version) (package node-package) (dependencies node-dependencies)) - (define (exists? name)- (not (null? (find-packages-by-name (guix-name name)))))+ (define (exists? name version)+ (not (null? (find-packages-by-name (guix-name name) version)))) - (define (lookup-node name)- (receive (package dependencies) (repo->guix-package name repo)- (make-node name package dependencies)))+ (define (lookup-node name version)+ (let* ((package dependencies (repo->guix-package name+ #:version version+ #:repo repo))+ (normilizied-deps (map (match-lambda+ ((name version) (list name version))+ (name (list name #f))) dependencies)))+ (make-node name version package normilizied-deps))) (map node-package- (topological-sort (list (lookup-node package-name))+ (topological-sort (list (lookup-node package-name version)) (lambda (node)- (map lookup-node- (remove exists?+ (map (lambda (name-version)+ (apply lookup-node name-version))+ (remove (lambda (name-version)+ (apply exists? name-version)) (node-dependencies node))))- node-name)))+ (lambda (node)+ (string-append+ (node-name node)+ (or (node-version node) ""))))))diff --git a/guix/scripts/import/cran.scm b/guix/scripts/import/cran.scmindex d6f371ef3a..bc266ad9da 100644--- a/guix/scripts/import/cran.scm+++ b/guix/scripts/import/cran.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2015, 2017, 2019 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -98,10 +99,10 @@ Import and convert the CRAN package for PACKAGE-NAME.\n")) ;; Recursive import (map package->definition (cran-recursive-import package-name- (or (assoc-ref opts 'repo) 'cran)))+ #:repo (or (assoc-ref opts 'repo) 'cran))) ;; Single import (let ((sexp (cran->guix-package package-name- (or (assoc-ref opts 'repo) 'cran))))+ #:repo (or (assoc-ref opts 'repo) 'cran)))) (unless sexp (leave (G_ "failed to download description for package '~a'~%") package-name))diff --git a/guix/scripts/import/elpa.scm b/guix/scripts/import/elpa.scmindex d270d2b4bc..07ac07a3d5 100644--- a/guix/scripts/import/elpa.scm+++ b/guix/scripts/import/elpa.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -102,7 +103,8 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n")) (_ #f)) (elpa-recursive-import package-name (or (assoc-ref opts 'repo) 'gnu)))- (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo))))+ (let ((sexp (elpa->guix-package package-name+ #:repo (assoc-ref opts 'repo)))) (unless sexp (leave (G_ "failed to download package '~a'~%") package-name)) sexp)))diff --git a/tests/elpa.scm b/tests/elpa.scmindex b70539bda6..a008cf993c 100644--- a/tests/elpa.scm+++ b/tests/elpa.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -51,7 +52,7 @@ (200 "This is the description.") (200 "fake tarball contents")) (parameterize ((current-http-proxy (%local-url)))- (match (elpa->guix-package pkg 'gnu/http)+ (match (elpa->guix-package pkg #:repo 'gnu/http) (('package ('name "emacs-auctex") ('version "11.88.6")diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex 87dda3238f..2357ea5c40 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -48,15 +49,16 @@ (package (name "foo") (inputs `(("bar" ,bar)))))- (recursive-import "foo" 'repo+ (recursive-import "foo"+ #:repo 'repo #:repo->guix-package (match-lambda*- (("foo" 'repo)+ (("foo" #:version #f #:repo 'repo) (values '(package (name "foo") (inputs `(("bar" ,bar)))) '("bar")))- (("bar" 'repo)+ (("bar" #:version #f #:repo 'repo) (values '(package (name "bar")) '())))-- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 0/8] recursive semver crate importer!
(address . 38408@debbugs.gnu.org)
cover.1580817140.git.mjbecze@riseup.net
Here is the another version of the recursive semver crate importer! And hopefully the best one so far. The first 3 commits actully implement the and add semver support. The rest are mainly ergonomics such as* triming version numbers from package name* better deduplication of dependencies* top level importing of development dependenies
I think it has incorpated the feedback i got from everyone so far, but if i forgot something or if there is more to add let me know!
Cheers~Martin
Martin Becze (8): guix: import: (recursive-import) Allow for version numbers guix: import: crate: Use semver to resovle module versions Added Guile-Semver as a dependency to guix guix: import: utils: allow generation of inputs to be version aware guix: import: crate: deduplicate dependencies guix: import: crate: memorize crate->guix-package guix: import: utils: trim patch version from names guix: import: parametrized importing of dev dependencies
configure.ac | 7 + doc/guix.texi | 2 + gnu/packages/package-management.scm | 7 +- guix/import/cran.scm | 8 +- guix/import/crate.scm | 111 +++++++---- guix/import/elpa.scm | 6 +- guix/import/gem.scm | 6 +- guix/import/opam.scm | 5 +- guix/import/pypi.scm | 5 +- guix/import/stackage.scm | 5 +- guix/import/utils.scm | 79 +++++--- guix/scripts/import/cran.scm | 5 +- guix/scripts/import/crate.scm | 13 +- guix/scripts/import/elpa.scm | 4 +- tests/crate.scm | 289 ++++++++++++++++------------ tests/elpa.scm | 3 +- tests/import-utils.scm | 8 +- 17 files changed, 346 insertions(+), 217 deletions(-)
-- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 2/8] guix: import: crate: Use semver to resovle module versions
(address . 38408@debbugs.gnu.org)
7dbe9a73fe56e1116d3207ef3cb9547a32b9a773.1580817140.git.mjbecze@riseup.net
* guix/import/crate.scm (make-crate-sexp): formatting, added '#:skip-build?' to build system args; added package definition geneation* guix/import/crate.scm (crate->guix-package): Use semver to resolve the correct module versions* tests/crate.scm: added version data to (recursuve-import) test--- guix/import/crate.scm | 87 ++++++---- guix/scripts/import/crate.scm | 11 +- tests/crate.scm | 290 +++++++++++++++++++--------------- 3 files changed, 225 insertions(+), 163 deletions(-)
Toggle diff (579 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 57823c3639..84b152620c 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>-;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019, 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -35,9 +35,12 @@ #:use-module (ice-9 match) #:use-module (ice-9 regex) #:use-module (json)+ #:use-module (semver)+ #:use-module (semver ranges) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name string->license@@ -86,7 +89,7 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string- (kind crate-dependency-kind "kind" ;'normal | 'dev+ (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string @@ -150,9 +153,14 @@ VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTIO and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name- cargo-development-inputs))+ (cargo-inputs+ (map+ (lambda (name-version)+ (apply crate-name->package-name name-version)) cargo-inputs))+ (cargo-development-inputs+ (map+ (lambda (name-version)+ (apply crate-name->package-name name-version)) cargo-development-inputs)) (pkg `(package (name ,guix-name) (version ,version)@@ -164,9 +172,10 @@ and LICENSE." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ ,@(maybe-arguments (append '(#:skip-build? #t)+ (maybe-cargo-inputs cargo-inputs) (maybe-cargo-development-inputs- cargo-development-inputs)))+ cargo-development-inputs))) (home-page ,(match home-page (() "") (_ home-page)))@@ -177,7 +186,7 @@ and LICENSE." ((license) license) (_ `(list ,@license))))))) (close-port port)- pkg))+ (package->definition pkg #t))) (define (string->license string) (filter-map (lambda (license)@@ -188,14 +197,19 @@ and LICENSE." 'unknown-license!))) (string-split string (string->char-set " /")))) -(define* (crate->guix-package crate-name #:optional version)+(define* (crate->guix-package crate-name #:key version #:allow-other-keys) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME." + (define (semver-range-contains-string? range version)+ (semver-range-contains? (string->semver-range range)+ (string->semver version)))+ (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))+ (or (eq? (crate-dependency-kind dependency) 'build)+ (eq? (crate-dependency-kind dependency) 'normal))) (define crate (lookup-crate crate-name))@@ -204,21 +218,36 @@ latest version of CRATE-NAME." (or version (crate-latest-version crate))) - (define version*+ (define (find-version crate range)+ "finds the a vesion of a crate that fulfils the semver <range>" (find (lambda (version)- (string=? (crate-version-number version)- version-number))+ (semver-range-contains-string?+ range+ (crate-version-number version))) (crate-versions crate))) + (define version*+ (find-version crate version-number))++ (define (sort-map-deps deps)+ "sorts the dependencies and maps the dependencies to a list+ containing pairs of (name version)"+ (sort (map (lambda (dep)+ (let* ((name (crate-dependency-id dep))+ (crate (lookup-crate name))+ (req (crate-dependency-requirement dep))+ (ver (find-version crate req)))+ (list name+ (crate-version-number ver))))+ deps)+ (match-lambda* (((_ name) ...)+ (apply string-ci<? name)))))+ (and crate version*- (let* ((dependencies (crate-version-dependencies version*))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))+ (let* ((dependencies (crate-version-dependencies version*))+ (dep-crates dev-dep-crates (partition normal-dependency? dependencies))+ (cargo-inputs (sort-map-deps dep-crates))+ (cargo-development-inputs '())) (values (make-crate-sexp #:name crate-name #:version (crate-version-number version*)@@ -230,15 +259,12 @@ latest version of CRATE-NAME." #:description (crate-description crate) #:license (and=> (crate-version-license version*) string->license))- (append cargo-inputs cargo-development-inputs)))))+ cargo-inputs)))) -(define* (crate-recursive-import crate-name #:optional version)- (recursive-import crate-name #f- #:repo->guix-package- (lambda (name repo)- (let ((version (and (string=? name crate-name)- version)))- (crate->guix-package name version)))+(define* (crate-recursive-import crate-name #:key version)+ (recursive-import crate-name+ #:repo->guix-package crate->guix-package+ #:version version #:guix-name crate-name->package-name)) (define (guix-package->crate-name package)@@ -253,7 +279,7 @@ latest version of CRATE-NAME." ((name _ ...) name)))) (define (crate-name->package-name name)- (string-append "rust-" (string-join (string-split name #\_) "-")))+ (guix-name "rust-" name)) ;;;@@ -288,4 +314,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release)))-diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex d834518c18..552628cfc7 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -2,7 +2,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 David Thompson <davet@gnu.org> ;;; Copyright © 2016 David Craven <david@craven.ch>-;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019, 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -95,13 +95,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (crate-recursive-import name version))- (let ((sexp (crate->guix-package name version)))+ (crate-recursive-import name #:version version)+ (let ((sexp (crate->guix-package name #:version version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%") (if versiondiff --git a/tests/crate.scm b/tests/crate.scmindex aa51faebf9..39561d5745 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2014 David Thompson <davet@gnu.org> ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -54,8 +55,9 @@ "{ \"dependencies\": [ {- \"crate_id\": \"bar\",+ \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } ] }")@@ -88,18 +90,22 @@ { \"crate_id\": \"intermediate-1\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, ] }")@@ -132,14 +138,17 @@ { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } ] }")@@ -172,6 +181,7 @@ { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, ] }")@@ -252,34 +262,48 @@ (open-input-string test-foo-crate)) ("https://crates.io/api/v1/crates/foo/1.0.0/download" (set! test-source-hash- (bytevector->nix-base32-string- (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies" (open-input-string test-foo-dependencies))+ ("https://crates.io/api/v1/crates/leaf-alice"+ (open-input-string test-leaf-alice-crate))+ ("https://crates.io/api/v1/crates/leaf-alice/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/leaf-alice/1.0.0/dependencies"+ (open-input-string test-leaf-alice-dependencies)) (_ (error "Unexpected URL: " url)))))- (match (crate->guix-package "foo")- (('package- ('name "rust-foo")- ('version "1.0.0")- ('source ('origin- ('method 'url-fetch)- ('uri ('crate-uri "foo" 'version))- ('file-name ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- (string=? test-source-hash hash))- (x- (pk 'fail x #f)))))++ (match (crate->guix-package "foo")+ ((define-public rust-foo-1.0.0+ (package (name "rust-foo")+ (version "1.0.0")+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "foo" 'version))+ (file-name (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system 'cargo-build-system)+ (arguments+ ('quasiquote+ (#:skip-build? #t+ #:cargo-inputs+ (("rust-leaf-alice-1.0.0" ('unquote rust-leaf-alice-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))++ (string=? test-source-hash hash))+ (x+ (pk 'fail x #f))))) (test-assert "cargo-recursive-import" ;; Replace network resources with sample data.@@ -334,105 +358,123 @@ (_ (error "Unexpected URL: " url))))) (match (crate-recursive-import "root") ;; rust-intermediate-2 has no dependency on the rust-leaf-alice package, so this is a valid ordering- ((('package- ('name "rust-leaf-alice")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "leaf-alice" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-leaf-bob")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "leaf-bob" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-intermediate-2")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "intermediate-2" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-intermediate-1")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "intermediate-1" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-intermediate-2" ('unquote rust-intermediate-2))- ("rust-leaf-alice" ('unquote rust-leaf-alice))- ("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0)))- ('package- ('name "rust-root")- ('version (? string? ver))- ('source- ('origin- ('method 'url-fetch)- ('uri ('crate-uri "root" 'version))- ('file-name- ('string-append 'name "-" 'version ".tar.gz"))- ('sha256- ('base32- (? string? hash)))))- ('build-system 'cargo-build-system)- ('arguments- ('quasiquote- ('#:cargo-inputs (("rust-intermediate-1" ('unquote rust-intermediate-1))- ("rust-intermediate-2" ('unquote rust-intermediate-2))- ("rust-leaf-alice" ('unquote rust-leaf-alice))- ("rust-leaf-bob" ('unquote rust-leaf-bob))))))- ('home-page "http://example.com")- ('synopsis "summary")- ('description "summary")- ('license ('list 'license:expat 'license:asl2.0))))+ (((define-public rust-leaf-alice-1.0.0+ (package+ (name "rust-leaf-alice")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "leaf-alice" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments ('quasiquote (#:skip-build? #t)))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-leaf-bob-1.0.0+ (package+ (name "rust-leaf-bob")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "leaf-bob" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments ('quasiquote (#:skip-build? #t)))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-intermediate-2-1.0.0+ (package+ (name "rust-intermediate-2")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "intermediate-2" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build? #t+ #:cargo-inputs+ (("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-intermediate-1-1.0.0+ (package+ (name "rust-intermediate-1")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "intermediate-1" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build? #t+ #:cargo-inputs+ (("rust-intermediate-2-1.0.0"+ ,rust-intermediate-2-1.0.0)+ ("rust-leaf-alice-1.0.0"+ ('unquote rust-leaf-alice-1.0.0))+ ("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))+ (define-public rust-root-1.0.0+ (package+ (name "rust-root")+ (version (? string? ver))+ (source+ (origin+ (method url-fetch)+ (uri (crate-uri "root" version))+ (file-name+ (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ (? string? hash)))))+ (build-system cargo-build-system)+ (arguments+ ('quasiquote (#:skip-build?+ #t #:cargo-inputs+ (("rust-intermediate-1-1.0.0"+ ('unquote rust-intermediate-1-1.0.0))+ ("rust-intermediate-2-1.0.0"+ ('unquote rust-intermediate-2-1.0.0))+ ("rust-leaf-alice-1.0.0"+ ('unquote rust-leaf-alice-1.0.0))+ ("rust-leaf-bob-1.0.0"+ ('unquote rust-leaf-bob-1.0.0))))))+ (home-page "http://example.com")+ (synopsis "summary")+ (description "summary")+ (license (list license:expat license:asl2.0))))) #t) (x (pk 'fail x #f)))))-- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 4/8] guix: import: utils: allow generation of inputs to be version aware
(address . 38408@debbugs.gnu.org)
c504652952de9de7902ab938efd7182e91456dad.1580817140.git.mjbecze@riseup.net
* guix/import/utils.scm (package-names->package-inputs): Added the ability to handle (name version) pairs* guix/import/crate.scm (make-crate-sexp): cleaned up input field generation--- guix/import/crate.scm | 17 +++++++++-------- guix/import/utils.scm | 21 ++++++++++++++------- 2 files changed, 23 insertions(+), 15 deletions(-)
Toggle diff (64 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 84b152620c..9128314370 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -151,16 +151,17 @@ record or #f if it was not found." "Return the `package' s-expression for a rust package with the given NAME, VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."+ (define (format-inputs inputs)+ (map+ (match-lambda+ ((name version) (list (crate-name->package-name name)+ (version-major+minor version))))+ inputs))+ (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs- (map- (lambda (name-version)- (apply crate-name->package-name name-version)) cargo-inputs))- (cargo-development-inputs- (map- (lambda (name-version)- (apply crate-name->package-name name-version)) cargo-development-inputs))+ (cargo-inputs (format-inputs cargo-inputs))+ (cargo-development-inputs (format-inputs cargo-development-inputs)) (pkg `(package (name ,guix-name) (version ,version)diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 59430d3e66..518877d476 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -233,13 +233,20 @@ into a proper sentence and by using two spaces between sentences." cleaned 'pre ". " 'post))) (define* (package-names->package-inputs names #:optional (output #f))- "Given a list of PACKAGE-NAMES, and an optional OUTPUT, tries to generate a-quoted list of inputs, as suitable to use in an 'inputs' field of a package-definition."- (map (lambda (input)- (cons* input (list 'unquote (string->symbol input))- (or (and output (list output))- '())))+ "Given a list of PACKAGE-NAMES or (PACKAGE-NAME VERSION) pairs, and an+optional OUTPUT, tries to generate a quoted list of inputs, as suitable to+use in an 'inputs' field of a package definition."+ (define (make-input input version)+ (cons* input (list 'unquote (string->symbol+ (if version+ (string-append input "-" version)+ input)))+ (or (and output (list output))+ '())))++ (map (match-lambda+ ((input version) (make-input input version))+ (input (make-input input #f))) names)) (define* (maybe-inputs package-names #:optional (output #f))-- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 5/8] guix: import: crate: deduplicate dependencies
(address . 38408@debbugs.gnu.org)
24fe155b78d5df8046e5fa1953fb5e07b7532f75.1580817140.git.mjbecze@riseup.net
* guix/import/crate.scm (crate-version-dependencies): deduplicate dependencies--- guix/import/crate.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Toggle diff (15 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 9128314370..a82e5e877a 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -115,7 +115,7 @@ record or #f if it was not found." (url (string-append (%crate-base-url) path))) (match (assoc-ref (or (json-fetch url) '()) "dependencies") ((? vector? vector)- (map json->crate-dependency (vector->list vector)))+ (delete-duplicates (map json->crate-dependency (vector->list vector)))) (_ '())))) -- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 3/8] Added Guile-Semver as a dependency to guix
(address . 38408@debbugs.gnu.org)
8a86631d201313b1da427a5ceb2ca5f201e6546c.1580817140.git.mjbecze@riseup.net
* configure.ac: added check for guile-semver* gnu/packages/package-management.scm (guix): added guile-semver as dep--- configure.ac | 7 +++++++ doc/guix.texi | 2 ++ gnu/packages/package-management.scm | 7 +++++-- 3 files changed, 14 insertions(+), 2 deletions(-)
Toggle diff (79 lines)diff --git a/configure.ac b/configure.acindex 06e86c209f..461ccaa8e7 100644--- a/configure.ac+++ b/configure.ac@@ -118,12 +118,19 @@ if test "x$have_guile_git" != "xyes"; then AC_MSG_ERROR([Guile-Git is missing; please install it.]) fi +dnl Check for Guile-Semver+GUILE_MODULE_AVAILABLE([have_guile_semver], [(semver)])+if test "x$have_guile_semver" != "xyes"; then+ AC_MSG_ERROR([Guile-Semver is missing; please install it.])+fi+ dnl Check for Guile-JSON. GUIX_CHECK_GUILE_JSON if test "x$guix_cv_have_recent_guile_json" != "xyes"; then AC_MSG_ERROR([Guile-JSON is missing; please install it.]) fi + dnl Guile-Sqlite3 is used by the (guix store ...) modules. GUIX_CHECK_GUILE_SQLITE3 if test "x$guix_cv_have_recent_guile_sqlite3" != "xyes"; thendiff --git a/doc/guix.texi b/doc/guix.texiindex 956c25ba9e..8d3d5935cd 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -72,6 +72,7 @@ Copyright @copyright{} 2019 Guillaume Le Vaillant@* Copyright @copyright{} 2020 Leo Prikler@* Copyright @copyright{} 2019, 2020 Simon Tournier@* Copyright @copyright{} 2020 Wiktor Żelazny@*+Copyright @copyright{} 2020 Martin Becze@* Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or@@ -762,6 +763,7 @@ or later; @uref{https://gitlab.com/guile-git/guile-git, Guile-Git}, from August 2017 or later; @item @uref{https://savannah.nongnu.org/projects/guile-json/, Guile-JSON} 3.x;+@item @uref{https://ngyro.com/software/guile-semver.html, Guile-Semver} 0.1.x; @item @url{https://zlib.net, zlib}; @item @url{https://www.gnu.org/software/make/, GNU Make}. @end itemizediff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scmindex 422d4f1959..c456071a87 100644--- a/gnu/packages/package-management.scm+++ b/gnu/packages/package-management.scm@@ -11,6 +11,7 @@ ;;; Copyright © 2018, 2019 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2019, 2020 Efraim Flashner <efraim@flashner.co.il> ;;; Copyright © 2019 Jonathan Brielmaier <jonathan.brielmaier@web.de>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -248,8 +249,9 @@ (ssh (assoc-ref inputs "guile-ssh")) (gnutls (assoc-ref inputs "gnutls")) (locales (assoc-ref inputs "glibc-utf8-locales"))+ (semver (assoc-ref inputs "guile-semver")) (deps (list gcrypt json sqlite gnutls- git bs ssh))+ git bs ssh semver)) (effective (read-line (open-pipe* OPEN_READ@@ -322,7 +324,8 @@ ("guile-json" ,guile-json-3) ("guile-sqlite3" ,guile-sqlite3) ("guile-ssh" ,guile-ssh)- ("guile-git" ,guile-git)))+ ("guile-git" ,guile-git)+ ("guile-semver",guile-semver))) (home-page "https://www.gnu.org/software/guix/") (synopsis "Functional package manager for installed software packages and versions")-- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 6/8] guix: import: crate: memorize crate->guix-package
(address . 38408@debbugs.gnu.org)
e0d885f9b4f5c84428e908486d606a56a5591fdc.1580817140.git.mjbecze@riseup.net
This adds memorization to procedures that involve network lookups.(mem-lookup-crate) is used on every dependency of a package to findit's versions. (mem-crate->guix-package) is needed becuase(topological-sort) depduplicates after dependencies have been turnedinto dependencies.
* guix/import/crate.scm (mem-crate->guix-package, mem-lookup-crate)--- guix/import/crate.scm | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
Toggle diff (54 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex a82e5e877a..630f4d3749 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -28,6 +28,7 @@ #:use-module (guix import json) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:)+ #:use-module (guix memoization) #:use-module (guix monads) #:use-module (guix packages) #:use-module (guix upstream)@@ -108,6 +109,8 @@ record or #f if it was not found." (json->crate `(,@alist ("actual_versions" . ,versions)))))))) +(define mem-lookup-crate (memoize lookup-crate))+ (define (crate-version-dependencies version) "Return the list of <crate-dependency> records of VERSION, a <crate-version>."@@ -213,7 +216,7 @@ latest version of CRATE-NAME." (eq? (crate-dependency-kind dependency) 'normal))) (define crate- (lookup-crate crate-name))+ (mem-lookup-crate crate-name)) (define version-number (or version@@ -235,7 +238,7 @@ latest version of CRATE-NAME." containing pairs of (name version)" (sort (map (lambda (dep) (let* ((name (crate-dependency-id dep))- (crate (lookup-crate name))+ (crate (mem-lookup-crate name)) (req (crate-dependency-requirement dep)) (ver (find-version crate req))) (list name@@ -262,9 +265,11 @@ latest version of CRATE-NAME." string->license)) cargo-inputs)))) +(define mem-crate->guix-package (memoize crate->guix-package))+ (define* (crate-recursive-import crate-name #:key version) (recursive-import crate-name- #:repo->guix-package crate->guix-package+ #:repo->guix-package mem-crate->guix-package #:version version #:guix-name crate-name->package-name)) -- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 7/8] guix: import: utils: trim patch version from names
(address . 38408@debbugs.gnu.org)
ea2b16bf2e4b4739bb2a8f406d5172fee5a109ab.1580817140.git.mjbecze@riseup.net
* guix/import/utils.scm (package->definition): trim patch version from names* tests/crate.scm: updated the tests--- guix/import/utils.scm | 7 ++++--- tests/crate.scm | 44 +++++++++++++++++++++---------------------- 2 files changed, 26 insertions(+), 25 deletions(-)
Toggle diff (133 lines)diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 518877d476..b902f008fd 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -273,9 +273,10 @@ package definition." ('package ('name name) ('version version) . rest) ('let _ ('package ('name name) ('version version) . rest))) - `(define-public ,(string->symbol (if append-version?- (string-append name "-" version)- version))+ `(define-public ,(string->symbol+ (if append-version?+ (string-append name "-" (version-major+minor version))+ version)) ,guix-package)))) (define (build-system-modules)diff --git a/tests/crate.scm b/tests/crate.scmindex 39561d5745..893dd70fc9 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -279,7 +279,7 @@ (_ (error "Unexpected URL: " url))))) (match (crate->guix-package "foo")- ((define-public rust-foo-1.0.0+ ((define-public rust-foo-1.0 (package (name "rust-foo") (version "1.0.0") (source@@ -295,7 +295,7 @@ ('quasiquote (#:skip-build? #t #:cargo-inputs- (("rust-leaf-alice-1.0.0" ('unquote rust-leaf-alice-1.0.0))))))+ (("rust-leaf-alice" ('unquote rust-leaf-alice-1.0)))))) (home-page "http://example.com") (synopsis "summary") (description "summary")@@ -358,7 +358,7 @@ (_ (error "Unexpected URL: " url))))) (match (crate-recursive-import "root") ;; rust-intermediate-2 has no dependency on the rust-leaf-alice package, so this is a valid ordering- (((define-public rust-leaf-alice-1.0.0+ (((define-public rust-leaf-alice-1.0 (package (name "rust-leaf-alice") (version (? string? ver))@@ -377,7 +377,7 @@ (synopsis "summary") (description "summary") (license (list license:expat license:asl2.0))))- (define-public rust-leaf-bob-1.0.0+ (define-public rust-leaf-bob-1.0 (package (name "rust-leaf-bob") (version (? string? ver))@@ -396,7 +396,7 @@ (synopsis "summary") (description "summary") (license (list license:expat license:asl2.0))))- (define-public rust-intermediate-2-1.0.0+ (define-public rust-intermediate-2-1.0 (package (name "rust-intermediate-2") (version (? string? ver))@@ -413,13 +413,13 @@ (arguments ('quasiquote (#:skip-build? #t #:cargo-inputs- (("rust-leaf-bob-1.0.0"+ (("rust-leaf-bob" ('unquote rust-leaf-bob-1.0.0)))))) (home-page "http://example.com") (synopsis "summary") (description "summary") (license (list license:expat license:asl2.0))))- (define-public rust-intermediate-1-1.0.0+ (define-public rust-intermediate-1-1.0 (package (name "rust-intermediate-1") (version (? string? ver))@@ -436,17 +436,17 @@ (arguments ('quasiquote (#:skip-build? #t #:cargo-inputs- (("rust-intermediate-2-1.0.0"- ,rust-intermediate-2-1.0.0)- ("rust-leaf-alice-1.0.0"- ('unquote rust-leaf-alice-1.0.0))- ("rust-leaf-bob-1.0.0"- ('unquote rust-leaf-bob-1.0.0))))))+ (("rust-intermediate-2"+ ,rust-intermediate-2-1.0)+ ("rust-leaf-alice"+ ('unquote rust-leaf-alice-1.0))+ ("rust-leaf-bob"+ ('unquote rust-leaf-bob-1.0)))))) (home-page "http://example.com") (synopsis "summary") (description "summary") (license (list license:expat license:asl2.0))))- (define-public rust-root-1.0.0+ (define-public rust-root-1.0 (package (name "rust-root") (version (? string? ver))@@ -463,14 +463,14 @@ (arguments ('quasiquote (#:skip-build? #t #:cargo-inputs- (("rust-intermediate-1-1.0.0"- ('unquote rust-intermediate-1-1.0.0))- ("rust-intermediate-2-1.0.0"- ('unquote rust-intermediate-2-1.0.0))- ("rust-leaf-alice-1.0.0"- ('unquote rust-leaf-alice-1.0.0))- ("rust-leaf-bob-1.0.0"- ('unquote rust-leaf-bob-1.0.0))))))+ (("rust-intermediate-1"+ ('unquote rust-intermediate-1-1.0))+ ("rust-intermediate-2"+ ('unquote rust-intermediate-2-1.0))+ ("rust-leaf-alice"+ ('unquote rust-leaf-alice-1.0))+ ("rust-leaf-bob"+ ('unquote rust-leaf-bob-1.0)))))) (home-page "http://example.com") (synopsis "summary") (description "summary")-- 2.25.0
M
M
Martin Becze wrote on 4 Feb 2020 13:18
[PATCH v9 8/8] guix: import: parametrized importing of dev dependencies
(address . 38408@debbugs.gnu.org)
7e2ac30581e11f1efc9b24258f51fc6d04c60272.1580817140.git.mjbecze@riseup.net
This changes the behavoir of the recusive crate importer so that it willinclude the importing of development dependencies for the top level packagebut will not inculded the development dependencies for any other imported package.
* guix/import/crate.scm (crate->guix-package, make-crate-sexp)<guix import crate>: added new parameter--- guix/import/crate.scm | 28 ++++++++++++++++++++-------- guix/scripts/import/crate.scm | 4 ++-- tests/crate.scm | 3 +-- 3 files changed, 23 insertions(+), 12 deletions(-)
Toggle diff (112 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 630f4d3749..a370fddffe 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -149,7 +149,7 @@ record or #f if it was not found." `((arguments (,'quasiquote ,args)))))) (define* (make-crate-sexp #:key name version cargo-inputs cargo-development-inputs- home-page synopsis description license+ home-page synopsis description license build? #:allow-other-keys) "Return the `package' s-expression for a rust package with the given NAME, VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION,@@ -176,7 +176,9 @@ and LICENSE." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append '(#:skip-build? #t)+ ,@(maybe-arguments (append (if build?+ '()+ '(#:skip-build? #t)) (maybe-cargo-inputs cargo-inputs) (maybe-cargo-development-inputs cargo-development-inputs)))@@ -201,11 +203,13 @@ and LICENSE." 'unknown-license!))) (string-split string (string->char-set " /")))) -(define* (crate->guix-package crate-name #:key version #:allow-other-keys)+(define* (crate->guix-package crate-name #:key version include-dev-deps?+ #:allow-other-keys) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the-latest version of CRATE-NAME."+latest version of CRATE-NAME. If INCLUDE-DEV-DEPS is true then this+will also lookup the development dependencs for the given crate." (define (semver-range-contains-string? range version) (semver-range-contains? (string->semver-range range)@@ -251,9 +255,12 @@ latest version of CRATE-NAME." (let* ((dependencies (crate-version-dependencies version*)) (dep-crates dev-dep-crates (partition normal-dependency? dependencies)) (cargo-inputs (sort-map-deps dep-crates))- (cargo-development-inputs '()))+ (cargo-development-inputs (if include-dev-deps?+ (sort-map-deps dev-dep-crates)+ '()))) (values- (make-crate-sexp #:name crate-name+ (make-crate-sexp #:build? include-dev-deps?+ #:name crate-name #:version (crate-version-number version*) #:cargo-inputs cargo-inputs #:cargo-development-inputs cargo-development-inputs@@ -263,13 +270,18 @@ latest version of CRATE-NAME." #:description (crate-description crate) #:license (and=> (crate-version-license version*) string->license))- cargo-inputs))))+ (append cargo-inputs cargo-development-inputs))))) (define mem-crate->guix-package (memoize crate->guix-package)) (define* (crate-recursive-import crate-name #:key version) (recursive-import crate-name- #:repo->guix-package mem-crate->guix-package+ #:repo->guix-package+ (lambda* params+ ;; only download the development dependencies for the top level package+ (let ((include-dev-deps? (equal? (car params) crate-name)))+ (apply mem-crate->guix-package+ (append params `(#:include-dev-deps? ,include-dev-deps?))))) #:version version #:guix-name crate-name->package-name)) diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex 552628cfc7..9252c52dfa 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -96,13 +96,13 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (if (assoc-ref opts 'recursive) (crate-recursive-import name #:version version)- (let ((sexp (crate->guix-package name #:version version)))+ (let ((sexp (crate->guix-package name #:version version #:include-dev-deps? #t))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%") (if version (string-append name "@" version) name)))- sexp)))+ (list sexp)))) (() (leave (G_ "too few arguments~%"))) ((many ...)diff --git a/tests/crate.scm b/tests/crate.scmindex 893dd70fc9..6fb9b772d8 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -461,8 +461,7 @@ (? string? hash))))) (build-system cargo-build-system) (arguments- ('quasiquote (#:skip-build?- #t #:cargo-inputs+ ('quasiquote (#:cargo-inputs (("rust-intermediate-1" ('unquote rust-intermediate-1-1.0)) ("rust-intermediate-2"-- 2.25.0
E
E
Efraim Flashner wrote on 17 Feb 2020 11:03
Re: [PATCH v9 1/8] guix: import: (recursive-import) Allow for version numbers
(name . Martin Becze)(address . mjbecze@riseup.net)
20200217100309.GH1968@E5400
The tree has moved a bit since you sent the patch, here's a copy I madethat applies.
-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
From 72be956188ead1aa11a1e9df972ad803323041cc Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Tue, 4 Feb 2020 07:18:18 -0500Subject: [PATCH 1/8] guix: import: (recursive-import) Allow for version numbers
This adds a key VERSION to (recursive-import) and move the paramter REPO to akey. This also changes all the things that rely on (recursive-import)
* guix/import/utils.scm (package->definition): added optional `append-version?`* guix/import/utils.scm (recursive-import): added key `version` and moved `repo` to be a key
* guix/import/cran.scm (cran->guix-package): change `repo` to a key* guix/import/cran.scm (cran-recursive-import): change `repo` to a key* guix/scripts/import/cran.scm: change `repo` to a key* guix/import/elpa.scm (elpa->guix-pakcage): change `repo` to a key* guix/import/elpa.scm (elpa-recursive-import): change `repo` to a key* guix/scripts/import/elpa.scm: change `repo` to a key* guix/import/gem.scm (gem->guix-package): change `repo` to a key* guix/import/gem.scm (recursive-import): change `repo` to a key* guix/import/opam.scm (opam-recurive-import): change `repo` to a key* guix/import/pypi.scm (pypi-recursive-import): change `repo` to a key* guix/import/stackage.scm (stackage-recursive-import): change `repo` to a key--- guix/import/cran.scm | 8 +++-- guix/import/elpa.scm | 6 ++-- guix/import/gem.scm | 4 ++- guix/import/opam.scm | 5 +-- guix/import/pypi.scm | 5 +-- guix/import/stackage.scm | 5 +-- guix/import/utils.scm | 59 ++++++++++++++++++++++-------------- guix/scripts/import/cran.scm | 5 +-- guix/scripts/import/elpa.scm | 4 ++- tests/elpa.scm | 3 +- tests/import-utils.scm | 8 +++-- 11 files changed, 71 insertions(+), 41 deletions(-)
Toggle diff (357 lines)diff --git a/guix/import/cran.scm b/guix/import/cran.scmindex bcb37ed250..9e05dfcba8 100644--- a/guix/import/cran.scm+++ b/guix/import/cran.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015, 2016, 2017, 2018, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2015, 2016, 2017, 2019, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -506,7 +507,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file." (define cran->guix-package (memoize- (lambda* (package-name #:optional (repo 'cran))+ (lambda* (package-name #:key (repo 'cran) #:allow-other-keys) "Fetch the metadata for PACKAGE-NAME from REPO and return the `package' s-expression corresponding to that package, or #f on failure." (let ((description (fetch-description repo package-name)))@@ -521,8 +522,9 @@ s-expression corresponding to that package, or #f on failure." (cran->guix-package package-name 'cran)) (else (values #f '())))))))) -(define* (cran-recursive-import package-name #:optional (repo 'cran))- (recursive-import package-name repo+(define* (cran-recursive-import package-name #:key (repo 'cran))+ (recursive-import package-name+ #:repo repo #:repo->guix-package cran->guix-package #:guix-name cran-guix-name)) diff --git a/guix/import/elpa.scm b/guix/import/elpa.scmindex 2d4487dba0..9140bcdc34 100644--- a/guix/import/elpa.scm+++ b/guix/import/elpa.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2015, 2016, 2017, 2018, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -245,7 +246,7 @@ type '<elpa-package>'." (license ,license)) dependencies-names))) -(define* (elpa->guix-package name #:optional (repo 'gnu))+(define* (elpa->guix-package name #:key (repo 'gnu) #:allow-other-keys) "Fetch the package NAME from REPO and produce a Guix package S-expression." (match (fetch-elpa-package name repo) (#f #f)@@ -301,7 +302,8 @@ type '<elpa-package>'." (define elpa-guix-name (cut guix-name "emacs-" <>)) (define* (elpa-recursive-import package-name #:optional (repo 'gnu))- (recursive-import package-name repo+ (recursive-import package-name+ #:repo repo #:repo->guix-package elpa->guix-package #:guix-name elpa-guix-name)) diff --git a/guix/import/gem.scm b/guix/import/gem.scmindex bd5d5b3569..54f158fa65 100644--- a/guix/import/gem.scm+++ b/guix/import/gem.scm@@ -3,6 +3,7 @@ ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -200,6 +201,7 @@ package on RubyGems." (latest latest-release))) (define* (gem-recursive-import package-name #:optional version)- (recursive-import package-name '()+ (recursive-import package-name+ #:repo '() #:repo->guix-package gem->guix-package #:guix-name ruby-package-name))diff --git a/guix/import/opam.scm b/guix/import/opam.scmindex 394415fdd4..87c823a98c 100644--- a/guix/import/opam.scm+++ b/guix/import/opam.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2018 Julien Lepiller <julien@lepiller.eu>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -311,8 +312,8 @@ or #f on failure." dependencies)))))))) (define (opam-recursive-import package-name)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (opam->guix-package name)) #:guix-name ocaml-name->guix-name)) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scmindex 6897f42be3..abd933e2e1 100644--- a/guix/import/pypi.scm+++ b/guix/import/pypi.scm@@ -6,6 +6,7 @@ ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2019 Maxim Cournoyer <maxim.cournoyer@gmail.com> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -429,8 +430,8 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." description license)))))))) (define (pypi-recursive-import package-name)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (pypi->guix-package name)) #:guix-name python->package-name)) diff --git a/guix/import/stackage.scm b/guix/import/stackage.scmindex 14150201b5..6091cf2c64 100644--- a/guix/import/stackage.scm+++ b/guix/import/stackage.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -108,8 +109,8 @@ included in the Stackage LTS release." (leave-with-message "~a: Stackage package not found" package-name)))))) (define (stackage-recursive-import package-name . args)- (recursive-import package-name #f- #:repo->guix-package (lambda (name repo)+ (recursive-import package-name+ #:repo->guix-package (lambda (name . _) (apply stackage->guix-package (cons name args))) #:guix-name hackage-name->package-name)) diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex d17d400ddf..8c434a3eea 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2019 Robert Vollmert <rob@vllmrt.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -44,6 +45,7 @@ #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (factorize-uri flatten@@ -258,13 +260,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package)+(define* (package->definition guix-package #:optional append-version?) (match guix-package- (('package ('name (? string? name)) _ ...)- `(define-public ,(string->symbol name)- ,guix-package))- (('let anything ('package ('name (? string? name)) _ ...))- `(define-public ,(string->symbol name)+ ((or+ ('package ('name name) ('version version) . rest)+ ('let _ ('package ('name name) ('version version) . rest)))++ `(define-public ,(string->symbol (if append-version?+ (string-append name "-" version)+ version)) ,guix-package)))) (define (build-system-modules)@@ -399,32 +403,43 @@ obtain a node's uniquely identifying \"key\"." (cons head result) (set-insert (node-name head) visited)))))))) -(define* (recursive-import package-name repo- #:key repo->guix-package guix-name+(define* (recursive-import package-name+ #:key repo->guix-package guix-name version repo #:allow-other-keys) "Return a list of package expressions for PACKAGE-NAME and all its dependencies, sorted in topological order. For each package,-call (REPO->GUIX-PACKAGE NAME REPO), which should return a package expression-and a list of dependencies; call (GUIX-NAME NAME) to obtain the Guix package-name corresponding to the upstream name."+call (REPO->GUIX-PACKAGE NAME :KEYS version repo), which should return a+package expression and a list of dependencies; call (GUIX-NAME NAME) to+obtain the Guix package name corresponding to the upstream name." (define-record-type <node>- (make-node name package dependencies)+ (make-node name version package dependencies) node? (name node-name)+ (version node-version) (package node-package) (dependencies node-dependencies)) - (define (exists? name)- (not (null? (find-packages-by-name (guix-name name)))))+ (define (exists? name version)+ (not (null? (find-packages-by-name (guix-name name) version)))) - (define (lookup-node name)- (receive (package dependencies) (repo->guix-package name repo)- (make-node name package dependencies)))+ (define (lookup-node name version)+ (let* ((package dependencies (repo->guix-package name+ #:version version+ #:repo repo))+ (normilizied-deps (map (match-lambda+ ((name version) (list name version))+ (name (list name #f))) dependencies)))+ (make-node name version package normilizied-deps))) (map node-package- (topological-sort (list (lookup-node package-name))+ (topological-sort (list (lookup-node package-name version)) (lambda (node)- (map lookup-node- (remove exists?- (node-dependencies node))))- node-name)))+ (map (lambda (name-version)+ (apply lookup-node name-version))+ (remove (lambda (name-version)+ (apply exists? name-version))+ (node-dependencies node))))+ (lambda (node)+ (string-append+ (node-name node)+ (or (node-version node) ""))))))diff --git a/guix/scripts/import/cran.scm b/guix/scripts/import/cran.scmindex d6f371ef3a..bc266ad9da 100644--- a/guix/scripts/import/cran.scm+++ b/guix/scripts/import/cran.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2015, 2017, 2019 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -98,10 +99,10 @@ Import and convert the CRAN package for PACKAGE-NAME.\n")) ;; Recursive import (map package->definition (cran-recursive-import package-name- (or (assoc-ref opts 'repo) 'cran)))+ #:repo (or (assoc-ref opts 'repo) 'cran))) ;; Single import (let ((sexp (cran->guix-package package-name- (or (assoc-ref opts 'repo) 'cran))))+ #:repo (or (assoc-ref opts 'repo) 'cran)))) (unless sexp (leave (G_ "failed to download description for package '~a'~%") package-name))diff --git a/guix/scripts/import/elpa.scm b/guix/scripts/import/elpa.scmindex d270d2b4bc..07ac07a3d5 100644--- a/guix/scripts/import/elpa.scm+++ b/guix/scripts/import/elpa.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -102,7 +103,8 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n")) (_ #f)) (elpa-recursive-import package-name (or (assoc-ref opts 'repo) 'gnu)))- (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo))))+ (let ((sexp (elpa->guix-package package-name+ #:repo (assoc-ref opts 'repo)))) (unless sexp (leave (G_ "failed to download package '~a'~%") package-name)) sexp)))diff --git a/tests/elpa.scm b/tests/elpa.scmindex b70539bda6..a008cf993c 100644--- a/tests/elpa.scm+++ b/tests/elpa.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch> ;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -51,7 +52,7 @@ (200 "This is the description.") (200 "fake tarball contents")) (parameterize ((current-http-proxy (%local-url)))- (match (elpa->guix-package pkg 'gnu/http)+ (match (elpa->guix-package pkg #:repo 'gnu/http) (('package ('name "emacs-auctex") ('version "11.88.6")diff --git a/tests/import-utils.scm b/tests/import-utils.scmindex 87dda3238f..2357ea5c40 100644--- a/tests/import-utils.scm+++ b/tests/import-utils.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -48,15 +49,16 @@ (package (name "foo") (inputs `(("bar" ,bar)))))- (recursive-import "foo" 'repo+ (recursive-import "foo"+ #:repo 'repo #:repo->guix-package (match-lambda*- (("foo" 'repo)+ (("foo" #:version #f #:repo 'repo) (values '(package (name "foo") (inputs `(("bar" ,bar)))) '("bar")))- (("bar" 'repo)+ (("bar" #:version #f #:repo 'repo) (values '(package (name "bar")) '())))-- 2.25.0
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl5KZN0ACgkQQarn3Mo9g1HpEA/9FCvCEmDejdFsSyQg+OfqUc5ODs9IOq1AFAsNlLMKcEH6a1ZsBTPPgqWzboXzb2BxLcSuiTMbMqpvJSW+ZmNBGRVRFkrvV/kf8hc7CXizRSmqEL9OVSQ3gda7Yh1L/V4hB3dZ0BgnhT4YI318NZMx8MixcNaQJ6uOUr9dJHJpOYPUGwHBkJpKwQVwZ9cI41pos12kJ9BceEv8TA3OV1qtY6gNw3Co8TLKsGY+IG/jhyGQufLPFyPJFebfxFabXbsk5nua2HvH4N4Yck3t3BGmh7T28WXImAOAKonsRgXtpO9PYNlw91II3AqQsuESG2Aar971p96N0UfchgHraxBeTZuONBGnfQoZ1nj/pLMIuI9cu0hdDOl9CfaBY/jNDc39FkZDlP3FdKILgMoTrKhqNZSW3VavtBoH/14qqT6K0f9Fxwt+ymH4klqnwAF5ZoulAvJdLAxesJJ8HHrVE9EBKK9zKDaEeH0+kprwAUMpWpN4PQolyHovNqUy47iglyRa4DIOFmENO3TPpAL12PwZ/yA1bRu8FpoFBsPUd7m/Eko2lK45I7XGuKXW0GRrYQ3a7T0cXrOPSpNUuLhclaroCKWdblcH+idj4jvELLmJq1RXZ5SROd8pu+KBucuvmJvRuWuhAE9CqiUi7Yq1LI4wa7AnKX/jLJNtq6HI5tgt5+M==Enae-----END PGP SIGNATURE-----

E
E
Efraim Flashner wrote on 17 Feb 2020 11:03
Re: [PATCH v9 3/8] Added Guile-Semver as a dependency to guix
(name . Martin Becze)(address . mjbecze@riseup.net)
20200217100345.GI1968@E5400
The tree has moved a bit since you sent the patch, here's a copy I madethat applies

-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
From 578d6f023c706df999c1b1b1bb23c9771b279857 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Tue, 4 Feb 2020 07:18:20 -0500Subject: [PATCH 3/8] Added Guile-Semver as a dependency to guix
* configure.ac: added check for guile-semver* gnu/packages/package-management.scm (guix): added guile-semver as dep
Signed-off-by: Efraim Flashner <efraim@flashner.co.il>--- configure.ac | 6 ++++++ doc/guix.texi | 2 ++ gnu/packages/package-management.scm | 7 +++++-- 3 files changed, 13 insertions(+), 2 deletions(-)
Toggle diff (72 lines)diff --git a/configure.ac b/configure.acindex 06e86c209f..0896c23955 100644--- a/configure.ac+++ b/configure.ac@@ -118,6 +118,12 @@ if test "x$have_guile_git" != "xyes"; then AC_MSG_ERROR([Guile-Git is missing; please install it.]) fi +dnl Check for Guile-Semver+GUILE_MODULE_AVAILABLE([have_guile_semver], [(semver)])+if test "x$have_guile_semver" != "xyes"; then+ AC_MSG_ERROR([Guile-Semver is missing; please install it.])+fi+ dnl Check for Guile-JSON. GUIX_CHECK_GUILE_JSON if test "x$guix_cv_have_recent_guile_json" != "xyes"; thendiff --git a/doc/guix.texi b/doc/guix.texiindex aa50340fe2..bc6a431b4c 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -74,6 +74,7 @@ Copyright @copyright{} 2019, 2020 Simon Tournier@* Copyright @copyright{} 2020 Wiktor Żelazny@* Copyright @copyright{} 2020 Damien Cassou@* Copyright @copyright{} 2020 Jakub Kądziołka@*+Copyright @copyright{} 2020 Martin Becze@* Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or@@ -764,6 +765,7 @@ or later; @uref{https://gitlab.com/guile-git/guile-git, Guile-Git}, from August 2017 or later; @item @uref{https://savannah.nongnu.org/projects/guile-json/, Guile-JSON} 3.x;+@item @uref{https://ngyro.com/software/guile-semver.html, Guile-Semver} 0.1.x; @item @url{https://zlib.net, zlib}; @item @url{https://www.gnu.org/software/make/, GNU Make}. @end itemizediff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scmindex 09888ca9e2..2419f49184 100644--- a/gnu/packages/package-management.scm+++ b/gnu/packages/package-management.scm@@ -11,6 +11,7 @@ ;;; Copyright © 2018, 2019 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2019, 2020 Efraim Flashner <efraim@flashner.co.il> ;;; Copyright © 2019 Jonathan Brielmaier <jonathan.brielmaier@web.de>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -248,8 +249,9 @@ (ssh (assoc-ref inputs "guile-ssh")) (gnutls (assoc-ref inputs "gnutls")) (locales (assoc-ref inputs "glibc-utf8-locales"))+ (semver (assoc-ref inputs "guile-semver")) (deps (list gcrypt json sqlite gnutls- git bs ssh))+ git bs ssh semver)) (effective (read-line (open-pipe* OPEN_READ@@ -322,7 +324,8 @@ ("guile-json" ,guile-json-3) ("guile-sqlite3" ,guile-sqlite3) ("guile-ssh" ,guile-ssh)- ("guile-git" ,guile-git)))+ ("guile-git" ,guile-git)+ ("guile-semver",guile-semver))) (home-page "https://www.gnu.org/software/guix/") (synopsis "Functional package manager for installed software packages and versions")-- 2.25.0
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl5KZQEACgkQQarn3Mo9g1GYqxAAq5/Hi0t3NY4UC9tDFcZVz45JQ/ChWtuWdrd5WZUeNpwil2wPJ9LrBa3yPiigraurhVw8OBk/LBtK/ydyDU7seLQ2WVCZWgCLAFunVdJoqnNNklZWJ4HGFMZ869MmNYmrocV8yeJcHNGMp+adSsQMutpTlxqTHMcHOILHDYASMb/tQ0n+4AB1LEdiGo870H7fFezz6OFPaA1b7DaYx+yZ1+SEKJbAjyxw7Qp3LLn76yS8Cjth5eC/2BGrl1JKDTnxtwLJjIi6zy65JqPGJ94JFhc+mASPi9UJywTVv43Xur6H/JDA/eXjG1oitVgpJX4nXJBXZx7V6Aan1AkACsZb8gT+o+psuhPe2D7aKBJEeYz5eX51r9F1UeiFRRjWMjm1Ha9wgtbd0N+zKRLqopKabJWHU4Lo+ceejLV7me+JZllyAqJCANSBlbYFWvqPn9p73YtJhUhIk8Egi9QDsMVKb4jyYqm+WNDhQHDTDEwzs+skoD6k0ATpMhxlk3kRIw3PJ+L40MRYRwiA8zQUjy9VOETIxhOUX/tdqZBzeJoqJZSNqr6Q85BV1tS50n+TFeGNZb7IIu+TwhjfXXkW4DHkyVIGQprrTJlSoP5egyKZBocGXfWpDyGSz76sYbdPiZZqevmJt75NQ8GwaXzmGqrwQjOaYJilnfWdEErJHR0KlwE==7c6s-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 17 Feb 2020 15:35
Re: [bug#38408] [PATCH v9 2/8] guix: import: crate: Use semver to resovle module versions
(name . Martin Becze)(address . mjbecze@riseup.net)
87y2t17047.fsf@gnu.org
Hi Martin & Efraim,
Thinking more about it, I’m not sure I fully understand why we need topay attention to semver here:
Martin Becze <mjbecze@riseup.net> skribis:
Toggle quote (32 lines)> +(define* (crate->guix-package crate-name #:key version #:allow-other-keys)> "Fetch the metadata for CRATE-NAME from crates.io, and return the> `package' s-expression corresponding to that package, or #f on failure.> When VERSION is specified, attempt to fetch that version; otherwise fetch the> latest version of CRATE-NAME."> > + (define (semver-range-contains-string? range version)> + (semver-range-contains? (string->semver-range range)> + (string->semver version)))> +> (define (normal-dependency? dependency)> - (eq? (crate-dependency-kind dependency) 'normal))> + (or (eq? (crate-dependency-kind dependency) 'build)> + (eq? (crate-dependency-kind dependency) 'normal)))> > (define crate> (lookup-crate crate-name))> @@ -204,21 +218,36 @@ latest version of CRATE-NAME."> (or version> (crate-latest-version crate)))> > - (define version*> + (define (find-version crate range)> + "finds the a vesion of a crate that fulfils the semver <range>"> (find (lambda (version)> - (string=? (crate-version-number version)> - version-number))> + (semver-range-contains-string?> + range> + (crate-version-number version)))> (crate-versions crate)))
The reason I wonder is that the HTTP API gives us rather precisedependency requirements:
Toggle snippet (4 lines)scheme@(guix import crate)> (crate-version-dependencies (car (crate-versions (lookup-crate "blake2-rfc"))))$8 = (#<<crate-dependency> id: "arrayvec" kind: normal requirement: "^0.4.6"> #<<crate-dependency> id: "constant_time_eq" kind: normal requirement: "^0.1.0"> #<<crate-dependency> id: "data-encoding" kind: dev requirement: "^2.0.0"> #<<crate-dependency> id: "clippy" kind: normal requirement: "^0.0.41">)
In the example above, the importer could “just” fetch version 0.4.6 ofarrayvec, version 0.1.0 of constant_time_eq, etc., no?
It’s an approximation because the caret (^) means more than just this,but hopefully it’s a good approximation.
Am I missing something?
Ludo’.
L
L
Ludovic Courtès wrote on 17 Feb 2020 15:36
Re: [PATCH v9 3/8] Added Guile-Semver as a dependency to guix
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87wo8l702y.fsf@gnu.org
Hi,
Efraim Flashner <efraim@flashner.co.il> skribis:
Toggle quote (8 lines)> From 578d6f023c706df999c1b1b1bb23c9771b279857 Mon Sep 17 00:00:00 2001> From: Martin Becze <mjbecze@riseup.net>> Date: Tue, 4 Feb 2020 07:18:20 -0500> Subject: [PATCH 3/8] Added Guile-Semver as a dependency to guix>> * configure.ac: added check for guile-semver> * gnu/packages/package-management.scm (guix): added guile-semver as dep
[...]
Toggle quote (6 lines)> +dnl Check for Guile-Semver> +GUILE_MODULE_AVAILABLE([have_guile_semver], [(semver)])> +if test "x$have_guile_semver" != "xyes"; then> + AC_MSG_ERROR([Guile-Semver is missing; please install it.])> +fi
I think a hard dependency like this is too much.
I would very much prefer to deal with it similar to how we deal withGuile-Newt or Guile-Charting: a soft dependency that’s entirelyoptional.
But I guess that also depends on what Guile-Semver is used for.I just posted a question on this topic in that thread. :-)
Thanks for reviving this patch series!
Ludo’.
E
E
Efraim Flashner wrote on 17 Feb 2020 15:57
Re: [bug#38408] [PATCH v9 2/8] guix: import: crate: Use semver to resovle module versions
(name . Ludovic Courtès)(address . ludo@gnu.org)
20200217145759.GM1968@E5400
On Mon, Feb 17, 2020 at 03:35:20PM +0100, Ludovic Courtès wrote:
Toggle quote (57 lines)> Hi Martin & Efraim,> > Thinking more about it, I’m not sure I fully understand why we need to> pay attention to semver here:> > Martin Becze <mjbecze@riseup.net> skribis:> > > +(define* (crate->guix-package crate-name #:key version #:allow-other-keys)> > "Fetch the metadata for CRATE-NAME from crates.io, and return the> > `package' s-expression corresponding to that package, or #f on failure.> > When VERSION is specified, attempt to fetch that version; otherwise fetch the> > latest version of CRATE-NAME."> > > > + (define (semver-range-contains-string? range version)> > + (semver-range-contains? (string->semver-range range)> > + (string->semver version)))> > +> > (define (normal-dependency? dependency)> > - (eq? (crate-dependency-kind dependency) 'normal))> > + (or (eq? (crate-dependency-kind dependency) 'build)> > + (eq? (crate-dependency-kind dependency) 'normal)))> > > > (define crate> > (lookup-crate crate-name))> > @@ -204,21 +218,36 @@ latest version of CRATE-NAME."> > (or version> > (crate-latest-version crate)))> > > > - (define version*> > + (define (find-version crate range)> > + "finds the a vesion of a crate that fulfils the semver <range>"> > (find (lambda (version)> > - (string=? (crate-version-number version)> > - version-number))> > + (semver-range-contains-string?> > + range> > + (crate-version-number version)))> > (crate-versions crate)))> > The reason I wonder is that the HTTP API gives us rather precise> dependency requirements:> > --8<---------------cut here---------------start------------->8---> scheme@(guix import crate)> (crate-version-dependencies (car (crate-versions (lookup-crate "blake2-rfc"))))> $8 = (#<<crate-dependency> id: "arrayvec" kind: normal requirement: "^0.4.6"> #<<crate-dependency> id: "constant_time_eq" kind: normal requirement: "^0.1.0"> #<<crate-dependency> id: "data-encoding" kind: dev requirement: "^2.0.0"> #<<crate-dependency> id: "clippy" kind: normal requirement: "^0.0.41">)> --8<---------------cut here---------------end--------------->8---> > In the example above, the importer could “just” fetch version 0.4.6 of> arrayvec, version 0.1.0 of constant_time_eq, etc., no?> > It’s an approximation because the caret (^) means more than just this,> but hopefully it’s a good approximation.> > Am I missing something?> > Ludo’.
Here we're looking at a minimum of 0.4.6 for arrayvec. According tohere¹ we'd really want to import 0.4.12, which is the latest 0.4.xrelease.
¹ https://crates.io/crates/arrayvec/versions

-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl5KqfcACgkQQarn3Mo9g1HUPw//V4YL5yf/EOoR7XB/s3SMPKSVQKIz/VbcqqvxZCPgVI0GQ0RShuVyTGmh9gkG2FTNem9NAMte14rr1bYuVMdba25XKZWSiuvNK4p5CeWrjXVuJTSd8059UUWzmVRoxqOU1hAxqPvowfoO4SFjIBQQE1NTAstWd7uXrWk+SZgJJ4+RpBaaCKtXUz8C8eqfCkt8XUJeoUwfafOi2bdGF9MHAkeZy3o8bSfgZ4QsTiIsmvUca07ERtxAszLXE4+gBb/sW4etW8imoEVRcahfa3HkSgPO2oFl4SL/DknoutbCwkVV3zOj9gZlo/21YI8uXDkEzhqTpdpKb7MJEXQHmnyvAQxZt8wmBkNoOhQKrXsaaLNRQFd7zazJA2p0F/GZdBTwmr4rTBQfDwyxns6zQmv65F1FbaatHVE8V/+ZnJXsevYMK0PJZXn/XrTm6yyuHmc+9H+Rgi4XNAtSoVpjcCcgS9yv8yfEBErIwlPchm8fkVMokywpnYLjR21RcBR7AyqjUZdckHCDQl9TK7pBZ4TqmAf3HIT1mpfHoY01Dp1owncwZYeY+vjOHgqyjMKlEkRTJpY5WA+FQbxfSeJMlUrWWWFJaKONh5MhrdUqAPphNn3pOStKMob1kKlE5ZSZFe6Qz43mjifSEVQ1IOfu1SOHoVI1geQi3KnFvvke+rLueEo==zRpi-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 17 Feb 2020 16:37
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87v9o55ioa.fsf@gnu.org
Efraim Flashner <efraim@flashner.co.il> skribis:
Toggle quote (2 lines)> On Mon, Feb 17, 2020 at 03:35:20PM +0100, Ludovic Courtès wrote:
[...]
Toggle quote (19 lines)>> --8<---------------cut here---------------start------------->8--->> scheme@(guix import crate)> (crate-version-dependencies (car (crate-versions (lookup-crate "blake2-rfc"))))>> $8 = (#<<crate-dependency> id: "arrayvec" kind: normal requirement: "^0.4.6"> #<<crate-dependency> id: "constant_time_eq" kind: normal requirement: "^0.1.0"> #<<crate-dependency> id: "data-encoding" kind: dev requirement: "^2.0.0"> #<<crate-dependency> id: "clippy" kind: normal requirement: "^0.0.41">)>> --8<---------------cut here---------------end--------------->8--->> >> In the example above, the importer could “just” fetch version 0.4.6 of>> arrayvec, version 0.1.0 of constant_time_eq, etc., no?>> >> It’s an approximation because the caret (^) means more than just this,>> but hopefully it’s a good approximation.>> >> Am I missing something?>> >> Ludo’.>> Here we're looking at a minimum of 0.4.6 for arrayvec. According to> here¹ we'd really want to import 0.4.12, which is the latest 0.4.x> release.
That’s why I wrote that 0.4.6 is an approximation (probably a good onebecause it’s apparently known to work.)
We can do something smarter, but then it’s only useful if the updater isequally smart—that is, it can update 0.4.6 to 0.4.13 whenever thatversion is out, knowing that blake2-rfc will still work fine.
Tricky! WDYT?
Ludo’.
M
M
Martin Becze wrote on 18 Feb 2020 09:56
a6e2462b-6843-319c-99b1-e28106b21f04@riseup.net
On 2/17/20 10:37 AM, Ludovic Courtès wrote:
Toggle quote (3 lines)> That’s why I wrote that 0.4.6 is an approximation (probably a good one> because it’s apparently known to work.)
Just grabbing the version from the semver range would work for some ranage would break on Hyphenated ranges (1.2.3 - 2), Combining ranges (>=0.14 <16) and on the asterisk range operator (1.*.* or 2.*)
Currently we are just trying to pick the most recent version that fits in the semver range.
Toggle quote (4 lines)> We can do something smarter, but then it’s only useful if the updater is> equally smart—that is, it can update 0.4.6 to 0.4.13 whenever that> version is out, knowing that blake2-rfc will still work fine.
Yep argeed! I would like to fix the updater as well, but i thought i should wait to send that in after this one gets in. Also it can quickly turns in to a SAT problem. I think we have two basic options though.
1) update everything to the newest possible version (easiest and this is what the importer does currently)
2) make the smallest possible dependency graph for all packages (harder, involves a SAT solver)
M
M
Martin Becze wrote on 18 Feb 2020 10:30
Re: [PATCH v9 3/8] Added Guile-Semver as a dependency to guix
bfd45370-f479-7ee6-030e-4a83ac2777fc@riseup.net
I'm looking at guile-charting now, and i don't understand how it is being used as a soft dependency. in guix/scripts/size.scm line 195 it is getting used with "(module-autoload!"... is that it?
On 2/17/20 9:36 AM, Ludovic Courtès wrote:
Toggle quote (33 lines)> Hi,> > Efraim Flashner <efraim@flashner.co.il> skribis:> >> From 578d6f023c706df999c1b1b1bb23c9771b279857 Mon Sep 17 00:00:00 2001>> From: Martin Becze <mjbecze@riseup.net>>> Date: Tue, 4 Feb 2020 07:18:20 -0500>> Subject: [PATCH 3/8] Added Guile-Semver as a dependency to guix>>>> * configure.ac: added check for guile-semver>> * gnu/packages/package-management.scm (guix): added guile-semver as dep> > [...]> >> +dnl Check for Guile-Semver>> +GUILE_MODULE_AVAILABLE([have_guile_semver], [(semver)])>> +if test "x$have_guile_semver" != "xyes"; then>> + AC_MSG_ERROR([Guile-Semver is missing; please install it.])>> +fi> > I think a hard dependency like this is too much.> > I would very much prefer to deal with it similar to how we deal with> Guile-Newt or Guile-Charting: a soft dependency that’s entirely> optional.> > But I guess that also depends on what Guile-Semver is used for.> I just posted a question on this topic in that thread. :-)> > Thanks for reviving this patch series!> > Ludo’.>
L
L
Ludovic Courtès wrote on 20 Feb 2020 10:40
(name . Martin Becze)(address . mjbecze@riseup.net)
874kvld2bl.fsf@gnu.org
Hi,
Martin Becze <mjbecze@riseup.net> skribis:
Toggle quote (4 lines)> I'm looking at guile-charting now, and i don't understand how it is> being used as a soft dependency. in guix/scripts/size.scm line 195 it> is getting used with "(module-autoload!"... is that it?
Yes, exactly. The thing is, Guile-Charting is not used at all unlessone passes the ‘--map-file’ option to ‘guix size’.
Ludo’.
M
M
Martin Becze wrote on 20 Feb 2020 17:54
(name . Ludovic Courtès)(address . ludo@gnu.org)
f7d9f9d1-e3aa-4a3a-432f-1b37d37d4207@riseup.net
Ok cool! I have tested it now. Attached is a patch that adds that behavior. Please drop the first patch and apply this one at the end.

On 2/20/20 4:40 AM, Ludovic Courtès wrote:
Toggle quote (2 lines)> exactly. The thing is, Guile-Charting is not used at all unless> one passes
From ef8f1a6365c321662248885d4c97c949c5c0a167 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Thu, 20 Feb 2020 11:49:11 -0500Subject: [PATCH v9] guix: import: crate: Added guile-semver as a soft dep
* guix/import/crate.scm Added guile-semver as a soft dep--- guix/import/crate.scm | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
Toggle diff (27 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex a370fddffe..91e38839bd 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -36,8 +36,6 @@ #:use-module (ice-9 match) #:use-module (ice-9 regex) #:use-module (json)- #:use-module (semver)- #:use-module (semver ranges) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)@@ -94,6 +92,11 @@ string->symbol) (requirement crate-dependency-requirement "req")) ;string +(module-autoload! (current-module)+ '(semver) '(string->semver))+(module-autoload! (current-module)+ '(semver ranges) '(string->semver-range semver-range-contains?))+ (define (lookup-crate name) "Look up NAME on https://crates.io and return the corresopnding <crate> record or #f if it was not found."-- 2.25.1
L
L
Leo Famulari wrote on 20 Feb 2020 19:53
Re: [bug#38408] [PATCH v9 0/8] recursive semver crate importer!
(name . Martin Becze)(address . mjbecze@riseup.net)
20200220185310.GA12545@jasmine.lan
On Tue, Feb 04, 2020 at 07:18:17AM -0500, Martin Becze wrote:
Toggle quote (5 lines)> Here is the another version of the recursive semver crate importer! And hopefully the best one so far. The first 3 commits actully implement the and add semver support. The rest are mainly ergonomics such as> * triming version numbers from package name> * better deduplication of dependencies> * top level importing of development dependenies
Wow, this is great! It makes it possible to get a (very close to)working rav1e package. The only things really missing are non-Rustdependencies of rav1e but I think that is not in scope for yourimporter.
Toggle quote (2 lines)> I think it has incorpated the feedback i got from everyone so far, but if i forgot something or if there is more to add let me know!
A minor bug: Sometimes if a dependency is already existing, the importerwill construct its variable name incorrectly.
For example, some of the packages I imported require version 2.33 ofrust-clap. We already have this package, and the variable is named'rust-clap-2', but the importer makes the packages depend on'rust-clap-2.33'. It has to be adjusted by hand.
You should be able to reproduce the bug with the following command. Itdoesn't matter if guile-semver is a hard dependency or autoloaded.
`guix environment guix --ad-hoc guile-json guile-semver -- ./pre-inst-env guix import crate --recursive rav1e`
Also, it prints the skip-build? argument on multiple lines, like this...
#:skip-build?#t
M
M
Martin Becze wrote on 21 Feb 2020 09:35
(name . Leo Famulari)(address . leo@famulari.name)
02e2004c-0271-b0fa-3919-04428d6d0570@riseup.net
On 2/20/20 1:53 PM, Leo Famulari wrote:
Toggle quote (3 lines)> 'rust-clap-2', but the importer makes the packages depend on> 'rust-clap-2.33'. It has to be adjusted by hand.
Thats actually not a bug, I think we probably should change rust-clap-2 -> rust-clap-2.33.
Toggle quote (6 lines)> > Also, it prints the skip-build? argument on multiple lines, like this...> > #:skip-build?> #t
Yeah that is annoying, its a problem with (ice-9 pretty-print). Would be nice to fix.
L
L
Ludovic Courtès wrote on 21 Feb 2020 10:01
Re: [PATCH v9 3/8] Added Guile-Semver as a dependency to guix
(name . Martin Becze)(address . mjbecze@riseup.net)
87sgj4nwjw.fsf@gnu.org
Hi Martin,
Martin Becze <mjbecze@riseup.net> skribis:
Toggle quote (5 lines)> +(module-autoload! (current-module)> + '(semver) '(string->semver))> +(module-autoload! (current-module)> + '(semver ranges) '(string->semver-range semver-range-contains?))
Sounds good. Could you please squash it with the commit that addssupport for semver?
Also, we may want to add guile-semver to ‘dependencies’ in‘compiled-guix’ in (guix self). That way, a pulled guix will haveguile-semver available, and thus ‘guix import crate’ will work out ofthe box.
Thanks,Ludo’.
E
E
Efraim Flashner wrote on 21 Feb 2020 13:15
Re: [bug#38408] [PATCH v9 0/8] recursive semver crate importer!
B96501C1-83D8-4C8E-9A43-24365D4D693B@flashner.co.il
On February 21, 2020 8:35:31 AM UTC, Martin Becze <mjbecze@riseup.net> wrote:
Toggle quote (10 lines)>>>On 2/20/20 1:53 PM, Leo Famulari wrote:>> 'rust-clap-2', but the importer makes the packages depend on>> 'rust-clap-2.33'. It has to be adjusted by hand.>>Thats actually not a bug, I think we probably should change rust-clap-2 >-> rust-clap-2.33.>
Talking to others at FOSDEM the other distro maintainers are pretty sure that all of the rust-clap-2 versions should be compatible, and similarly for other packages with a major version other than 0. So I'd personally prefer to change them to just the major version (and keep it as rust-clap-2).
Toggle quote (9 lines)>> >> Also, it prints the skip-build? argument on multiple lines, like this...>> >> #:skip-build?>> #t>>Yeah that is annoying, its a problem with (ice-9 pretty-print). Would be >nice to fix.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
M
M
Martin Becze wrote on 21 Feb 2020 17:25
Re: [PATCH v9 3/8] Added Guile-Semver as a dependency to guix
(name . Ludovic Courtès)(address . ludo@gnu.org)
8453e3ce-bc5a-3bb3-1bfb-deaca7ca11d3@riseup.net
On 2/21/20 4:01 AM, Ludovic Courtès wrote:
Toggle quote (5 lines)> Hi Martin,
> Sounds good. Could you please squash it with the commit that adds> support for semver?
Squashed and attached as v10-0002-guix-import-crate-Use-semver-to-resovle-module-v.patch
Toggle quote (5 lines)> Also, we may want to add guile-semver to ‘dependencies’ in> ‘compiled-guix’ in (guix self). That way, a pulled guix will have> guile-semver available, and thus ‘guix import crate’ will work out of> the box.
I added that it is attached as v10-0008-guix-self-added-guile-semver-as-a-depenedency.patchBut I'm not sure how to test guix pull to see if it correctly brought in guile-semver!
From ee19656f5e63955e43922301f1abf32cfc629779 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Fri, 21 Feb 2020 10:41:44 -0500Subject: [PATCH v10 8/8] guix: self: added guile-semver as a depenedency
* guix/self.scm (compliled-guix) added guile-semver as a depenedency--- guix/self.scm | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
Toggle diff (40 lines)diff --git a/guix/self.scm b/guix/self.scmindex 6b633f9bc0..c0506a0a60 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -54,6 +55,7 @@ ("guile-git" (ref '(gnu packages guile) 'guile3.0-git)) ("guile-sqlite3" (ref '(gnu packages guile) 'guile3.0-sqlite3)) ("guile-gcrypt" (ref '(gnu packages gnupg) 'guile3.0-gcrypt))+ ("guile-semver" (ref '(gnu packages guile-xyz) 'guile-semver)) ("gnutls" (ref '(gnu packages tls) 'guile3.0-gnutls)) ("zlib" (ref '(gnu packages compression) 'zlib)) ("lzlib" (ref '(gnu packages compression) 'lzlib))@@ -682,6 +684,9 @@ Info manual." (define guile-gcrypt (specification->package "guile-gcrypt")) + (define guile-semver+ (specification->package "guile-semver"))+ (define gnutls (specification->package "gnutls")) @@ -690,7 +695,7 @@ Info manual." (cons (list "x" package) (package-transitive-propagated-inputs package))) (list guile-gcrypt gnutls guile-git guile-json- guile-ssh guile-sqlite3))+ guile-ssh guile-sqlite3 guile-semver)) (((labels packages _ ...) ...) packages))) -- 2.25.1
From 3f1d5662f75abda64fb042f2d12dd8afc5523e17 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Mon, 3 Feb 2020 16:19:49 -0500Subject: [PATCH v10 7/8] guix: import: parametrized importing of dev dependencies
This changes the behavoir of the recusive crate importer so that it willinclude the importing of development dependencies for the top level packagebut will not inculded the development dependencies for any other imported package.
* guix/import/crate.scm (crate->guix-package, make-crate-sexp)<guix import crate>: added new parameter--- guix/import/crate.scm | 28 ++++++++++++++++++++-------- guix/scripts/import/crate.scm | 4 ++-- tests/crate.scm | 3 +-- 3 files changed, 23 insertions(+), 12 deletions(-)
Toggle diff (112 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 00ac6ee318..91e38839bd 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -152,7 +152,7 @@ record or #f if it was not found." `((arguments (,'quasiquote ,args)))))) (define* (make-crate-sexp #:key name version cargo-inputs cargo-development-inputs- home-page synopsis description license+ home-page synopsis description license build? #:allow-other-keys) "Return the `package' s-expression for a rust package with the given NAME, VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION,@@ -179,7 +179,9 @@ and LICENSE." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append '(#:skip-build? #t)+ ,@(maybe-arguments (append (if build?+ '()+ '(#:skip-build? #t)) (maybe-cargo-inputs cargo-inputs) (maybe-cargo-development-inputs cargo-development-inputs)))@@ -204,11 +206,13 @@ and LICENSE." 'unknown-license!))) (string-split string (string->char-set " /")))) -(define* (crate->guix-package crate-name #:key version #:allow-other-keys)+(define* (crate->guix-package crate-name #:key version include-dev-deps?+ #:allow-other-keys) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the-latest version of CRATE-NAME."+latest version of CRATE-NAME. If INCLUDE-DEV-DEPS is true then this+will also lookup the development dependencs for the given crate." (define (semver-range-contains-string? range version) (semver-range-contains? (string->semver-range range)@@ -254,9 +258,12 @@ latest version of CRATE-NAME." (let* ((dependencies (crate-version-dependencies version*)) (dep-crates dev-dep-crates (partition normal-dependency? dependencies)) (cargo-inputs (sort-map-deps dep-crates))- (cargo-development-inputs '()))+ (cargo-development-inputs (if include-dev-deps?+ (sort-map-deps dev-dep-crates)+ '()))) (values- (make-crate-sexp #:name crate-name+ (make-crate-sexp #:build? include-dev-deps?+ #:name crate-name #:version (crate-version-number version*) #:cargo-inputs cargo-inputs #:cargo-development-inputs cargo-development-inputs@@ -266,13 +273,18 @@ latest version of CRATE-NAME." #:description (crate-description crate) #:license (and=> (crate-version-license version*) string->license))- cargo-inputs))))+ (append cargo-inputs cargo-development-inputs))))) (define mem-crate->guix-package (memoize crate->guix-package)) (define* (crate-recursive-import crate-name #:key version) (recursive-import crate-name- #:repo->guix-package mem-crate->guix-package+ #:repo->guix-package+ (lambda* params+ ;; only download the development dependencies for the top level package+ (let ((include-dev-deps? (equal? (car params) crate-name)))+ (apply mem-crate->guix-package+ (append params `(#:include-dev-deps? ,include-dev-deps?))))) #:version version #:guix-name crate-name->package-name)) diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex 552628cfc7..9252c52dfa 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -96,13 +96,13 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (if (assoc-ref opts 'recursive) (crate-recursive-import name #:version version)- (let ((sexp (crate->guix-package name #:version version)))+ (let ((sexp (crate->guix-package name #:version version #:include-dev-deps? #t))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%") (if version (string-append name "@" version) name)))- sexp)))+ (list sexp)))) (() (leave (G_ "too few arguments~%"))) ((many ...)diff --git a/tests/crate.scm b/tests/crate.scmindex 893dd70fc9..6fb9b772d8 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -461,8 +461,7 @@ (? string? hash))))) (build-system cargo-build-system) (arguments- ('quasiquote (#:skip-build?- #t #:cargo-inputs+ ('quasiquote (#:cargo-inputs (("rust-intermediate-1" ('unquote rust-intermediate-1-1.0)) ("rust-intermediate-2"-- 2.25.1
From 58f6c611aa4d49e9fef0ea6a0ed7125cd3942bef Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Thu, 30 Jan 2020 11:19:13 -0500Subject: [PATCH v10 6/8] guix: import: utils: trim patch version from names
* guix/import/utils.scm (package->definition): trim patch version from names* tests/crate.scm: updated the tests--- guix/import/utils.scm | 7 ++++--- tests/crate.scm | 44 +++++++++++++++++++++---------------------- 2 files changed, 26 insertions(+), 25 deletions(-)
Toggle diff (133 lines)diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex c60a164271..649b9f9b85 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -273,9 +273,10 @@ package definition." ('package ('name name) ('version version) . rest) ('let _ ('package ('name name) ('version version) . rest))) - `(define-public ,(string->symbol (if append-version?- (string-append name "-" version)- version))+ `(define-public ,(string->symbol+ (if append-version?+ (string-append name "-" (version-major+minor version))+ version)) ,guix-package)))) (define (build-system-modules)diff --git a/tests/crate.scm b/tests/crate.scmindex 39561d5745..893dd70fc9 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -279,7 +279,7 @@ (_ (error "Unexpected URL: " url))))) (match (crate->guix-package "foo")- ((define-public rust-foo-1.0.0+ ((define-public rust-foo-1.0 (package (name "rust-foo") (version "1.0.0") (source@@ -295,7 +295,7 @@ ('quasiquote (#:skip-build? #t #:cargo-inputs- (("rust-leaf-alice-1.0.0" ('unquote rust-leaf-alice-1.0.0))))))+ (("rust-leaf-alice" ('unquote rust-leaf-alice-1.0)))))) (home-page "http://example.com") (synopsis "summary") (description "summary")@@ -358,7 +358,7 @@ (_ (error "Unexpected URL: " url))))) (match (crate-recursive-import "root") ;; rust-intermediate-2 has no dependency on the rust-leaf-alice package, so this is a valid ordering- (((define-public rust-leaf-alice-1.0.0+ (((define-public rust-leaf-alice-1.0 (package (name "rust-leaf-alice") (version (? string? ver))@@ -377,7 +377,7 @@ (synopsis "summary") (description "summary") (license (list license:expat license:asl2.0))))- (define-public rust-leaf-bob-1.0.0+ (define-public rust-leaf-bob-1.0 (package (name "rust-leaf-bob") (version (? string? ver))@@ -396,7 +396,7 @@ (synopsis "summary") (description "summary") (license (list license:expat license:asl2.0))))- (define-public rust-intermediate-2-1.0.0+ (define-public rust-intermediate-2-1.0 (package (name "rust-intermediate-2") (version (? string? ver))@@ -413,13 +413,13 @@ (arguments ('quasiquote (#:skip-build? #t #:cargo-inputs- (("rust-leaf-bob-1.0.0"+ (("rust-leaf-bob" ('unquote rust-leaf-bob-1.0.0)))))) (home-page "http://example.com") (synopsis "summary") (description "summary") (license (list license:expat license:asl2.0))))- (define-public rust-intermediate-1-1.0.0+ (define-public rust-intermediate-1-1.0 (package (name "rust-intermediate-1") (version (? string? ver))@@ -436,17 +436,17 @@ (arguments ('quasiquote (#:skip-build? #t #:cargo-inputs- (("rust-intermediate-2-1.0.0"- ,rust-intermediate-2-1.0.0)- ("rust-leaf-alice-1.0.0"- ('unquote rust-leaf-alice-1.0.0))- ("rust-leaf-bob-1.0.0"- ('unquote rust-leaf-bob-1.0.0))))))+ (("rust-intermediate-2"+ ,rust-intermediate-2-1.0)+ ("rust-leaf-alice"+ ('unquote rust-leaf-alice-1.0))+ ("rust-leaf-bob"+ ('unquote rust-leaf-bob-1.0)))))) (home-page "http://example.com") (synopsis "summary") (description "summary") (license (list license:expat license:asl2.0))))- (define-public rust-root-1.0.0+ (define-public rust-root-1.0 (package (name "rust-root") (version (? string? ver))@@ -463,14 +463,14 @@ (arguments ('quasiquote (#:skip-build? #t #:cargo-inputs- (("rust-intermediate-1-1.0.0"- ('unquote rust-intermediate-1-1.0.0))- ("rust-intermediate-2-1.0.0"- ('unquote rust-intermediate-2-1.0.0))- ("rust-leaf-alice-1.0.0"- ('unquote rust-leaf-alice-1.0.0))- ("rust-leaf-bob-1.0.0"- ('unquote rust-leaf-bob-1.0.0))))))+ (("rust-intermediate-1"+ ('unquote rust-intermediate-1-1.0))+ ("rust-intermediate-2"+ ('unquote rust-intermediate-2-1.0))+ ("rust-leaf-alice"+ ('unquote rust-leaf-alice-1.0))+ ("rust-leaf-bob"+ ('unquote rust-leaf-bob-1.0)))))) (home-page "http://example.com") (synopsis "summary") (description "summary")-- 2.25.1
From 6a4e13a7b16c7647fd47f3412bb27252dbd054a1 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Thu, 30 Jan 2020 11:17:00 -0500Subject: [PATCH v10 5/8] guix: import: crate: memorize crate->guix-package
This adds memorization to procedures that involve network lookups.(mem-lookup-crate) is used on every dependency of a package to findit's versions. (mem-crate->guix-package) is needed becuase(topological-sort) depduplicates after dependencies have been turnedinto dependencies.
* guix/import/crate.scm (mem-crate->guix-package, mem-lookup-crate)--- guix/import/crate.scm | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
Toggle diff (54 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex b2a3dd7e70..00ac6ee318 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -28,6 +28,7 @@ #:use-module (guix import json) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:)+ #:use-module (guix memoization) #:use-module (guix monads) #:use-module (guix packages) #:use-module (guix upstream)@@ -111,6 +112,8 @@ record or #f if it was not found." (json->crate `(,@alist ("actual_versions" . ,versions)))))))) +(define mem-lookup-crate (memoize lookup-crate))+ (define (crate-version-dependencies version) "Return the list of <crate-dependency> records of VERSION, a <crate-version>."@@ -216,7 +219,7 @@ latest version of CRATE-NAME." (eq? (crate-dependency-kind dependency) 'normal))) (define crate- (lookup-crate crate-name))+ (mem-lookup-crate crate-name)) (define version-number (or version@@ -238,7 +241,7 @@ latest version of CRATE-NAME." containing pairs of (name version)" (sort (map (lambda (dep) (let* ((name (crate-dependency-id dep))- (crate (lookup-crate name))+ (crate (mem-lookup-crate name)) (req (crate-dependency-requirement dep)) (ver (find-version crate req))) (list name@@ -265,9 +268,11 @@ latest version of CRATE-NAME." string->license)) cargo-inputs)))) +(define mem-crate->guix-package (memoize crate->guix-package))+ (define* (crate-recursive-import crate-name #:key version) (recursive-import crate-name- #:repo->guix-package crate->guix-package+ #:repo->guix-package mem-crate->guix-package #:version version #:guix-name crate-name->package-name)) -- 2.25.1
From f93800713be11754aba4572009c859d30256adac Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Thu, 30 Jan 2020 10:52:28 -0500Subject: [PATCH v10 4/8] guix: import: crate: deduplicate dependencies
* guix/import/crate.scm (crate-version-dependencies): deduplicate dependencies--- guix/import/crate.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Toggle diff (15 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 7e61bc21b6..b2a3dd7e70 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -118,7 +118,7 @@ record or #f if it was not found." (url (string-append (%crate-base-url) path))) (match (assoc-ref (or (json-fetch url) '()) "dependencies") ((? vector? vector)- (map json->crate-dependency (vector->list vector)))+ (delete-duplicates (map json->crate-dependency (vector->list vector)))) (_ '())))) -- 2.25.1
From ee5d7412e0cad9bad9be3b26195cd298ed37ca30 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Mon, 27 Jan 2020 06:45:10 -0500Subject: [PATCH v10 3/8] guix: import: utils: allow generation of inputs to be version aware
* guix/import/utils.scm (package-names->package-inputs): Added the ability to handle (name version) pairs* guix/import/crate.scm (make-crate-sexp): cleaned up input field generation--- guix/import/crate.scm | 17 +++++++++-------- guix/import/utils.scm | 21 ++++++++++++++------- 2 files changed, 23 insertions(+), 15 deletions(-)
Toggle diff (64 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex d711820b81..7e61bc21b6 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -154,16 +154,17 @@ record or #f if it was not found." "Return the `package' s-expression for a rust package with the given NAME, VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."+ (define (format-inputs inputs)+ (map+ (match-lambda+ ((name version) (list (crate-name->package-name name)+ (version-major+minor version))))+ inputs))+ (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs- (map- (lambda (name-version)- (apply crate-name->package-name name-version)) cargo-inputs))- (cargo-development-inputs- (map- (lambda (name-version)- (apply crate-name->package-name name-version)) cargo-development-inputs))+ (cargo-inputs (format-inputs cargo-inputs))+ (cargo-development-inputs (format-inputs cargo-development-inputs)) (pkg `(package (name ,guix-name) (version ,version)diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 8c434a3eea..c60a164271 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -233,13 +233,20 @@ into a proper sentence and by using two spaces between sentences." cleaned 'pre ". " 'post))) (define* (package-names->package-inputs names #:optional (output #f))- "Given a list of PACKAGE-NAMES, and an optional OUTPUT, tries to generate a-quoted list of inputs, as suitable to use in an 'inputs' field of a package-definition."- (map (lambda (input)- (cons* input (list 'unquote (string->symbol input))- (or (and output (list output))- '())))+ "Given a list of PACKAGE-NAMES or (PACKAGE-NAME VERSION) pairs, and an+optional OUTPUT, tries to generate a quoted list of inputs, as suitable to+use in an 'inputs' field of a package definition."+ (define (make-input input version)+ (cons* input (list 'unquote (string->symbol+ (if version+ (string-append input "-" version)+ input)))+ (or (and output (list output))+ '())))++ (map (match-lambda+ ((input version) (make-input input version))+ (input (make-input input #f))) names)) (define* (maybe-inputs package-names #:optional (output #f))-- 2.25.1
From ed3365f11107556d45b189da6588e353b2ef5e46 Mon Sep 17 00:00:00 2001From: Martin Becze <mjbecze@riseup.net>Date: Tue, 4 Feb 2020 03:50:48 -0500Subject: [PATCH v10 2/8] guix: import: crate: Use semver to resovle module versions
* guix/import/crate.scm (make-crate-sexp): formatting, added '#:skip-build?' to build system args; added package definition geneation* guix/import/crate.scm (crate->guix-package): Use semver to resolve the correct module versions* guix/import/crate.scm Added guile-semver as a soft dep* tests/crate.scm: added version data to (recursuve-import) test--- guix/import/crate.scm | 90 +++++++---- guix/scripts/import/crate.scm | 11 +- tests/crate.scm | 290 +++++++++++++++++++--------------- 3 files changed, 228 insertions(+), 163 deletions(-)
Toggle diff (582 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 57823c3639..d711820b81 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>-;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019, 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -38,6 +38,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26)+ #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name string->license@@ -86,10 +87,15 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string- (kind crate-dependency-kind "kind" ;'normal | 'dev+ (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string +(module-autoload! (current-module)+ '(semver) '(string->semver))+(module-autoload! (current-module)+ '(semver ranges) '(string->semver-range semver-range-contains?))+ (define (lookup-crate name) "Look up NAME on https://crates.io and return the corresopnding <crate> record or #f if it was not found."@@ -150,9 +156,14 @@ VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTIO and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name- cargo-development-inputs))+ (cargo-inputs+ (map+ (lambda (name-version)+ (apply crate-name->package-name name-version)) cargo-inputs))+ (cargo-development-inputs+ (map+ (lambda (name-version)+ (apply crate-name->package-name name-version)) cargo-development-inputs)) (pkg `(package (name ,guix-name) (version ,version)@@ -164,9 +175,10 @@ and LICENSE." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ ,@(maybe-arguments (append '(#:skip-build? #t)+ (maybe-cargo-inputs cargo-inputs) (maybe-cargo-development-inputs- cargo-development-inputs)))+ cargo-development-inputs))) (home-page ,(match home-page (() "") (_ home-page)))@@ -177,7 +189,7 @@ and LICENSE." ((license) license) (_ `(list ,@license))))))) (close-port port)- pkg))+ (package->definition pkg #t))) (define (string->license string) (filter-map (lambda (license)@@ -188,14 +200,19 @@ and LICENSE." 'unknown-license!))) (string-split string (string->char-set " /")))) -(define* (crate->guix-package crate-name #:optional version)+(define* (crate->guix-package crate-name #:key version #:allow-other-keys) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME." + (define (semver-range-contains-string? range version)+ (semver-range-contains? (string->semver-range range)+ (string->semver version)))+ (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))+ (or (eq? (crate-dependency-kind dependency) 'build)+ (eq? (crate-dependency-kind dependency) 'normal))) (define crate (lookup-crate crate-name))@@ -204,21 +221,36 @@ latest version of CRATE-NAME." (or version (crate-latest-version crate))) - (define version*+ (define (find-version crate range)+ "finds the a vesion of a crate that fulfils the semver <range>" (find (lambda (version)- (string=? (crate-version-number version)- version-number))+ (semver-range-contains-string?+ range+ (crate-version-number version))) (crate-versions crate))) + (define version*+ (find-version crate version-number))++ (define (sort-map-deps deps)+ "sorts the dependencies and maps the dependencies to a list+ containing pairs of (name version)"+ (sort (map (lambda (dep)+ (let* ((name (crate-dependency-id dep))+ (crate (lookup-crate name))+ (req (crate-dependency-requirement dep))+ (ver (find-version crate req)))+ (list name+ (crate-version-number ver))))+ deps)+ (match-lambda* (((_ name) ...)+ (apply string-ci<? name)))))+ (and crate version*- (let* ((dependencies (crate-version-dependencies version*))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))+ (let* ((dependencies (crate-version-dependencies version*))+ (dep-crates dev-dep-crates (partition normal-dependency? dependencies))+ (cargo-inputs (sort-map-deps dep-crates))+ (cargo-development-inputs '())) (values (make-crate-sexp #:name crate-name #:version (crate-version-number version*)@@ -230,15 +262,12 @@ latest version of CRATE-NAME." #:description (crate-description crate) #:license (and=> (crate-version-license version*) string->license))- (append cargo-inputs cargo-development-inputs)))))+ cargo-inputs)))) -(define* (crate-recursive-import crate-name #:optional version)- (recursive-import crate-name #f- #:repo->guix-package- (lambda (name repo)- (let ((version (and (string=? name crate-name)- version)))- (crate->guix-package name version)))+(define* (crate-recursive-import crate-name #:key version)+ (recursive-import crate-name+ #:repo->guix-package crate->guix-package+ #:version version #:guix-name crate-name->package-name)) (define (guix-package->crate-name package)@@ -253,7 +282,7 @@ latest version of CRATE-NAME." ((name _ ...) name)))) (define (crate-name->package-name name)- (string-append "rust-" (string-join (string-split name #\_) "-")))+ (guix-name "rust-" name)) ;;;@@ -288,4 +317,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release)))-diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex d834518c18..552628cfc7 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -2,7 +2,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 David Thompson <davet@gnu.org> ;;; Copyright © 2016 David Craven <david@craven.ch>-;;; Copyright © 2019 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2019, 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -95,13 +95,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive)- (map (match-lambda- ((and ('package ('name name) . rest) pkg)- `(define-public ,(string->symbol name)- ,pkg))- (_ #f))- (crate-recursive-import name version))- (let ((sexp (crate->guix-package name version)))+ (crate-recursive-import name #:version version)+ (let ((sexp (crate->guix-package name #:version version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%") (if versiondiff --git a/tests/crate.scm b/tests/crate.scmindex aa51faebf9..39561d5745 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2014 David Thompson <davet@gnu.org> ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; ;;; This file is part of GNU Guix. ;;;@@ -54,8 +55,9 @@ "{ \"dependencies\": [ {- \"crate_id\": \"bar\",+ \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } ] }")@@ -88,18 +90,22 @@ { \"crate_id\": \"intermediate-1\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, ] }")@@ -132,14 +138,17 @@ { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", } ] }")@@ -172,6 +181,7 @@ { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\",+ \"req\": \"1.0.0\", }, ] }")@@ -252,34 +262,48 @@ (open-input-string test-foo-crate)) ("https://crates.io/api/v1/crates/foo/1.0.0/download" (set! test-source-hash- (bytevector->nix-base32-string- (sha256 (string->bytevector "empty file\n" "utf-8"))))