[PATCH] import: crate: add recursive option

DoneSubmitted by Karl Meakin.
Details
5 participants
  • Brian Leung
  • Efraim Flashner
  • Ivan Petkov
  • Karl Meakin
  • Ludovic Courtès
Owner
unassigned
Severity
normal
K
K
Karl Meakin wrote on 20 May 2019 20:23
(address . guix-patches@gnu.org)(name . Karl Meakin)(address . Karlwfmeakin@gmail.com)
20190520182306.11899-1-Karlwfmeakin@gmail.com
* guix/script/import/crate.scm: Add recursive option.* guix/import/crate.scm (crate-recursive-import): New variable.--- doc/guix.texi | 8 ++++++++ guix/import/crate.scm | 24 ++++++++++++++++-------- guix/scripts/import/crate.scm | 27 ++++++++++++++++++++++----- 3 files changed, 46 insertions(+), 13 deletions(-)
Toggle diff (145 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex ae9ad0739e..636bb7521d 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -8787,6 +8787,14 @@ in Guix. Import metadata from the crates.io Rust package repository @uref{https://crates.io, crates.io}. +@table @code+@item --recursive+@itemx -r+Traverse the dependency graph of the given upstream package recursively+and generate package expressions for all those packages that are not yet+in Guix.+@end table+ @item opam @cindex OPAM @cindex OCamldiff --git a/guix/import/crate.scm b/guix/import/crate.scmindex e0b400d054..d8554b0e7a 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -37,6 +37,7 @@ #:use-module (srfi srfi-26) #:export (crate->guix-package guix-package->crate-name+ crate-recursive-import %crate-updater)) (define (crate-fetch crate-name callback)@@ -86,8 +87,8 @@ VERSION, INPUTS, NATIVE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (inputs (map crate-name->package-name inputs))- (native-inputs (map crate-name->package-name native-inputs))+ (input-packages (map crate-name->package-name inputs))+ (native-input-packages (map crate-name->package-name native-inputs)) (pkg `(package (name ,guix-name) (version ,version)@@ -99,8 +100,8 @@ VERSION, INPUTS, NATIVE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-native-inputs native-inputs "src")- ,@(maybe-inputs inputs "src")+ ,@(maybe-native-inputs native-input-packages "src")+ ,@(maybe-inputs input-packages "src") (home-page ,(match home-page (() "") (_ home-page)))@@ -111,12 +112,14 @@ VERSION, INPUTS, NATIVE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." ((license) license) (_ `(list ,@license))))))) (close-port port)- pkg))+ (values pkg inputs))) -(define (crate->guix-package crate-name)- "Fetch the metadata for CRATE-NAME from crates.io, and return the+(define crate->guix-package+ (memoize+ (lambda* (crate-name _)+ "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure."- (crate-fetch crate-name make-crate-sexp))+ (crate-fetch crate-name make-crate-sexp)))) (define (guix-package->crate-name package) "Return the crate name of PACKAGE."@@ -158,6 +161,11 @@ VERSION, INPUTS, NATIVE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." (version version) (urls (list url))))) +(define* (crate-recursive-import package-name)+ (recursive-import package-name #f+ #:repo->guix-package crate->guix-package+ #:guix-name crate-name->package-name))+ (define %crate-updater (upstream-updater (name 'crates)diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex cab9a4397b..8fadcdd57c 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -27,6 +27,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-37)+ #:use-module (srfi srfi-41) #:use-module (ice-9 match) #:use-module (ice-9 format) #:export (guix-import-crate))@@ -45,6 +46,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (display (G_ " -h, --help display this help and exit")) (display (G_ "+ -r, --recursive import packages recursively"))+ (display (G_ " -V, --version display version information and exit")) (newline) (show-bug-report-information))@@ -58,6 +61,9 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (option '(#\V "version") #f #f (lambda args (show-version-and-exit "guix import crate")))+ (option '(#\r "recursive") #f #f+ (lambda (opt name arg result)+ (alist-cons 'recursive #t result))) %standard-import-options)) @@ -83,11 +89,22 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (reverse opts)))) (match args ((package-name)- (let ((sexp (crate->guix-package package-name)))- (unless sexp- (leave (G_ "failed to download meta-data for package '~a'~%")- package-name))- sexp))+ (if (assoc-ref opts 'recursive)+ ;; Recursive import+ (map (match-lambda+ ((and ('package ('name name) . rest) pkg)+ `(define-public ,(string->symbol name)+ ,pkg))+ (_ #f))+ (reverse+ (stream->list+ (crate-recursive-import package-name))))+ ;; Single import+ (let ((sexp (crate->guix-package package-name #f)))+ (unless sexp+ (leave (G_ "failed to download meta-data for package '~a'~%")+ package-name))+ sexp))) (() (leave (G_ "too few arguments~%"))) ((many ...)-- 2.21.0
L
L
Ludovic Courtès wrote on 24 May 2019 17:42
(name . Karl Meakin)(address . karl.w.f.meakin@gmail.com)
87woifuadk.fsf@gnu.org
Hi Karl,
Karl Meakin <karl.w.f.meakin@gmail.com> skribis:
Toggle quote (3 lines)> * guix/script/import/crate.scm: Add recursive option.> * guix/import/crate.scm (crate-recursive-import): New variable.
Thanks for working on this! Please also mention the guix.texi changehere.
I prefer to let Ivan and Danny comment on this. My only question is:would it be possible to augment ‘tests/crate.scm’ with tests forrecursive imports? That would be great.
Thanks,Ludo’.
I
I
Ivan Petkov wrote on 25 May 2019 21:38
(address . 35813@debbugs.gnu.org)
F514F3C2-FA56-4958-916B-488EBF898E15@gmail.com
Hi Karl,
Thanks for the patch, overall the implementation looks good to me!
Toggle quote (5 lines)> On May 24, 2019, at 8:42 AM, Ludovic Courtès <ludo@gnu.org> wrote:> > would it be possible to augment ‘tests/crate.scm’ with tests for> recursive imports? That would be great.
I second Ludo’s comment here, having some tests around this would be great!
Also, have you had a chance to test this by importing a Rust application?I tried to import hexyl (which has a small dependency closure compared to otherapplications), but the final result had some missing crate input definitions(e.g. Maybe a crate depends on rust-foo, but rust-foo wasn't defined in theoutput either). Maybe I did something wrong, but it would be good to confirm!
Also, don't worry about actually trying to build any imported packages as ofyet, you might hit a cycle issue between proc-macro2 and quote. I've got apatch open[0] for fixing the cargo-build-system to handle this!
--Ivan
[0]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35318 https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35318
Attachment: file
B
B
Brian Leung wrote on 20 Jul 2019 20:30
(address . 35813@debbugs.gnu.org)
CAAc=MEztxqF7z4mi50_CDv64PNwSt8=snm-XmEDZ3GuVJO31vw@mail.gmail.com
What's the status on this? I was about to start working on a recursiveimporter for Rust crates until I noticed this patch.
Attachment: file
I
I
Ivan Petkov wrote on 20 Jul 2019 23:43
(name . Brian Leung)(address . bkleung89@gmail.com)
FB09817B-0F88-4040-887B-87DF52F9522D@gmail.com
Hi Brian,
Toggle quote (4 lines)> On Jul 20, 2019, at 11:30 AM, Brian Leung <bkleung89@gmail.com> wrote:> > What's the status on this? I was about to start working on a recursive importer for Rust crates until I noticed this patch.
On the patch side a couple of updates are still needed:* a rebase on the latest master* an update to also include the native-inputs/dev-dependency inputs in the loop for importing additional crates* some tests to verify the behavior
Not sure if Karl has a had a chance to update this yet
—Ivan
B
B
Brian Leung wrote on 5 Aug 2019 19:50
[PATCH] Add crate-recursive-import.
(address . 35813@debbugs.gnu.org)
CAAc=MEyL1Fp_qHrEUVFn207D_3NABCAOxPtq8ezc9mDaS3wo9A@mail.gmail.com
I took Karl's changes and updated them accordingly. I've also added a smalltest. The patch containing his importer, my changes, and my test isattached (the commit was made using my name--not sure if I should insteadapply Karl's patch).
Attachment: file
From 407959e97656803981ebf69f83b96a0be2ff6cb7 Mon Sep 17 00:00:00 2001From: Brian Leung <bkleung89@gmail.com>Date: Sat, 20 Jul 2019 21:35:14 +0200Subject: [PATCH] gnu: Add crate-recursive-import.
* guix/import/crate.scm (crate-recursive-import): New variable.* guix/script/import/crate.scm: Add recursive option.* guix/tests/crate.scm (crate-recursive-import): New test.------ guix/import/crate.scm | 31 ++++++++---- guix/scripts/import/crate.scm | 28 +++++++++-- tests/crate.scm | 92 +++++++++++++++++++++++++++++++++-- 3 files changed, 132 insertions(+), 19 deletions(-)
Toggle diff (263 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 52c5cb1c30..51f55ea708 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -36,6 +36,7 @@ #:use-module (srfi srfi-2) #:use-module (srfi srfi-26) #:export (crate->guix-package+ crate-recursive-import guix-package->crate-name %crate-updater)) @@ -109,8 +110,8 @@ VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTIO and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name+ (inputs (map crate-name->package-name cargo-inputs))+ (development-inputs (map crate-name->package-name cargo-development-inputs)) (pkg `(package (name ,guix-name)@@ -123,9 +124,9 @@ and LICENSE." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ ,@(maybe-arguments (append (maybe-cargo-inputs inputs) (maybe-cargo-development-inputs- cargo-development-inputs)))+ development-inputs))) (home-page ,(match home-page (() "") (_ home-page)))@@ -136,12 +137,22 @@ and LICENSE." ((license) license) (_ `(list ,@license))))))) (close-port port)- pkg))--(define (crate->guix-package crate-name)- "Fetch the metadata for CRATE-NAME from crates.io, and return the-`package' s-expression corresponding to that package, or #f on failure."- (crate-fetch crate-name make-crate-sexp))+ ;; (pretty-print guix-name)+ (values pkg (append cargo-development-inputs cargo-inputs))+ ;; pkg+ ))++(define crate->guix-package+ (memoize+ (lambda* (crate-name)+ "Fetch the metadata for CRATE-NAME from crates.io, and return the+ `package' s-expression corresponding to that package, or #f on failure."+ (crate-fetch crate-name make-crate-sexp))))++(define* (crate-recursive-import package-name)+ (recursive-import package-name #f+ #:repo->guix-package (lambda (name _) (crate->guix-package name))+ #:guix-name crate-name->package-name)) (define (guix-package->crate-name package) "Return the crate name of PACKAGE."diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex cab9a4397b..b18cab8286 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -27,6 +27,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-37)+ #:use-module (srfi srfi-41) #:use-module (ice-9 match) #:use-module (ice-9 format) #:export (guix-import-crate))@@ -45,6 +46,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (display (G_ " -h, --help display this help and exit")) (display (G_ "+ -r, --recursive import packages recursively"))+ (display (G_ " -V, --version display version information and exit")) (newline) (show-bug-report-information))@@ -58,6 +61,9 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (option '(#\V "version") #f #f (lambda args (show-version-and-exit "guix import crate")))+ (option '(#\r "recursive") #f #f+ (lambda (opt name arg result)+ (alist-cons 'recursive #t result))) %standard-import-options)) @@ -83,11 +89,23 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (reverse opts)))) (match args ((package-name)- (let ((sexp (crate->guix-package package-name)))- (unless sexp- (leave (G_ "failed to download meta-data for package '~a'~%")- package-name))- sexp))+ (if (assoc-ref opts 'recursive)+ ;; Recursive import+ (map (match-lambda+ ((and ('package ('name name) . rest) pkg)+ `(define-public ,(string->symbol name)+ ,pkg))+ (_ #f))+ (reverse+ (stream->list+ (crate-recursive-import package-name))))+ ;; Single import+ (let ((sexp (crate->guix-package package-name ;; #f+ )))+ (unless sexp+ (leave (G_ "failed to download meta-data for package '~a'~%")+ package-name))+ sexp))) (() (leave (G_ "too few arguments~%"))) ((many ...)diff --git a/tests/crate.scm b/tests/crate.scmindex 72c3a13350..1787d4f2f6 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -25,9 +25,10 @@ #:use-module (guix tests) #:use-module (ice-9 iconv) #:use-module (ice-9 match)+ #:use-module (srfi srfi-41) #:use-module (srfi srfi-64)) -(define test-crate+(define test-foo-crate "{ \"crate\": { \"max_version\": \"1.0.0\",@@ -39,7 +40,7 @@ } }") -(define test-dependencies+(define test-foo-dependencies "{ \"dependencies\": [ {@@ -49,6 +50,23 @@ ] }") +(define test-bar-crate+ "{+ \"crate\": {+ \"max_version\": \"1.0.0\",+ \"name\": \"bar\",+ \"license\": \"MIT/Apache-2.0\",+ \"description\": \"summary\",+ \"homepage\": \"http://example.com\",+ \"repository\": \"http://example.com\",+ }+}")++(define test-bar-dependencies+ "{+ \"dependencies\": []+}")+ (define test-source-hash "") @@ -68,14 +86,14 @@ (lambda (url . rest) (match url ("https://crates.io/api/v1/crates/foo"- (open-input-string test-crate))+ (open-input-string test-foo-crate)) ("https://crates.io/api/v1/crates/foo/1.0.0/download" (set! test-source-hash (bytevector->nix-base32-string (sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies"- (open-input-string test-dependencies))+ (open-input-string test-foo-dependencies)) (_ (error "Unexpected URL: " url))))) (match (crate->guix-package "foo") (('package@@ -100,4 +118,70 @@ (x (pk 'fail x #f))))) +(test-assert "cargo-recursive-import"+ ;; Replace network resources with sample data.+ (mock ((guix http-client) http-fetch+ (lambda (url . rest)+ (match url+ ("https://crates.io/api/v1/crates/foo"+ (open-input-string test-foo-crate))+ ("https://crates.io/api/v1/crates/foo/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies"+ (open-input-string test-foo-dependencies))+ ("https://crates.io/api/v1/crates/bar"+ (open-input-string test-bar-crate))+ ("https://crates.io/api/v1/crates/bar/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/bar/1.0.0/dependencies"+ (open-input-string test-bar-dependencies))+ (_ (error "Unexpected URL: " url)))))+ (match (stream->list (crate-recursive-import "foo"))+ ((('package+ ('name "rust-foo")+ ('version (? string? ver))+ ('source+ ('origin+ ('method 'url-fetch)+ ('uri ('crate-uri "foo" 'version))+ ('file-name+ ('string-append 'name "-" 'version ".tar.gz"))+ ('sha256+ ('base32+ (? string? hash)))))+ ('build-system 'cargo-build-system)+ ('arguments+ ('quasiquote+ ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))+ ('home-page "http://example.com")+ ('synopsis "summary")+ ('description "summary")+ ('license ('list 'license:expat 'license:asl2.0)))+ ('package+ ('name "rust-bar")+ ('version (? string? ver))+ ('source+ ('origin+ ('method 'url-fetch)+ ('uri ('crate-uri "bar" 'version))+ ('file-name+ ('string-append 'name "-" 'version ".tar.gz"))+ ('sha256+ ('base32+ (? string? hash)))))+ ('build-system 'cargo-build-system)+ ('home-page "http://example.com")+ ('synopsis "summary")+ ('description "summary")+ ('license ('list 'license:expat 'license:asl2.0))))+ #t)+ (x+ (pk 'fail x #f)))))+ (test-end "crate")-- 2.22.0
B
B
Brian Leung wrote on 6 Aug 2019 05:42
(address . 35813@debbugs.gnu.org)
CAAc=MEyHs+TsKEC1icvMp=ORwqAmxJvsVmVOiBcMRq3qW6VdxA@mail.gmail.com
OK, I updated to remove print statements I missed.
On Mon, Aug 5, 2019 at 7:50 PM Brian Leung <bkleung89@gmail.com> wrote:
Toggle quote (5 lines)> I took Karl's changes and updated them accordingly. I've also added a> small test. The patch containing his importer, my changes, and my test is> attached (the commit was made using my name--not sure if I should instead> apply Karl's patch).>
Attachment: file
From a3528e0b3333864528805150a16caad88c07fd7a Mon Sep 17 00:00:00 2001From: Brian Leung <bkleung89@gmail.com>Date: Sat, 20 Jul 2019 21:35:14 +0200Subject: [PATCH] gnu: Add crate-recursive-import.
* guix/import/crate.scm (crate-recursive-import): New variable.* guix/script/import/crate.scm: Add recursive option.* guix/tests/crate.scm (crate-recursive-import): New test.------ guix/import/crate.scm | 28 +++++++---- guix/scripts/import/crate.scm | 28 +++++++++-- tests/crate.scm | 92 +++++++++++++++++++++++++++++++++-- 3 files changed, 129 insertions(+), 19 deletions(-)
Toggle diff (260 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex 52c5cb1c30..355f0264bc 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -36,6 +36,7 @@ #:use-module (srfi srfi-2) #:use-module (srfi srfi-26) #:export (crate->guix-package+ crate-recursive-import guix-package->crate-name %crate-updater)) @@ -109,8 +110,8 @@ VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTIO and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name+ (inputs (map crate-name->package-name cargo-inputs))+ (development-inputs (map crate-name->package-name cargo-development-inputs)) (pkg `(package (name ,guix-name)@@ -123,9 +124,9 @@ and LICENSE." (base32 ,(bytevector->nix-base32-string (port-sha256 port)))))) (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)+ ,@(maybe-arguments (append (maybe-cargo-inputs inputs) (maybe-cargo-development-inputs- cargo-development-inputs)))+ development-inputs))) (home-page ,(match home-page (() "") (_ home-page)))@@ -136,12 +137,19 @@ and LICENSE." ((license) license) (_ `(list ,@license))))))) (close-port port)- pkg))--(define (crate->guix-package crate-name)- "Fetch the metadata for CRATE-NAME from crates.io, and return the-`package' s-expression corresponding to that package, or #f on failure."- (crate-fetch crate-name make-crate-sexp))+ (values pkg (append cargo-development-inputs cargo-inputs))))++(define crate->guix-package+ (memoize+ (lambda* (crate-name)+ "Fetch the metadata for CRATE-NAME from crates.io, and return the+ `package' s-expression corresponding to that package, or #f on failure."+ (crate-fetch crate-name make-crate-sexp))))++(define* (crate-recursive-import package-name)+ (recursive-import package-name #f+ #:repo->guix-package (lambda (name _) (crate->guix-package name))+ #:guix-name crate-name->package-name)) (define (guix-package->crate-name package) "Return the crate name of PACKAGE."diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex cab9a4397b..b18cab8286 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -27,6 +27,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-37)+ #:use-module (srfi srfi-41) #:use-module (ice-9 match) #:use-module (ice-9 format) #:export (guix-import-crate))@@ -45,6 +46,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (display (G_ " -h, --help display this help and exit")) (display (G_ "+ -r, --recursive import packages recursively"))+ (display (G_ " -V, --version display version information and exit")) (newline) (show-bug-report-information))@@ -58,6 +61,9 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (option '(#\V "version") #f #f (lambda args (show-version-and-exit "guix import crate")))+ (option '(#\r "recursive") #f #f+ (lambda (opt name arg result)+ (alist-cons 'recursive #t result))) %standard-import-options)) @@ -83,11 +89,23 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (reverse opts)))) (match args ((package-name)- (let ((sexp (crate->guix-package package-name)))- (unless sexp- (leave (G_ "failed to download meta-data for package '~a'~%")- package-name))- sexp))+ (if (assoc-ref opts 'recursive)+ ;; Recursive import+ (map (match-lambda+ ((and ('package ('name name) . rest) pkg)+ `(define-public ,(string->symbol name)+ ,pkg))+ (_ #f))+ (reverse+ (stream->list+ (crate-recursive-import package-name))))+ ;; Single import+ (let ((sexp (crate->guix-package package-name ;; #f+ )))+ (unless sexp+ (leave (G_ "failed to download meta-data for package '~a'~%")+ package-name))+ sexp))) (() (leave (G_ "too few arguments~%"))) ((many ...)diff --git a/tests/crate.scm b/tests/crate.scmindex 72c3a13350..1787d4f2f6 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -25,9 +25,10 @@ #:use-module (guix tests) #:use-module (ice-9 iconv) #:use-module (ice-9 match)+ #:use-module (srfi srfi-41) #:use-module (srfi srfi-64)) -(define test-crate+(define test-foo-crate "{ \"crate\": { \"max_version\": \"1.0.0\",@@ -39,7 +40,7 @@ } }") -(define test-dependencies+(define test-foo-dependencies "{ \"dependencies\": [ {@@ -49,6 +50,23 @@ ] }") +(define test-bar-crate+ "{+ \"crate\": {+ \"max_version\": \"1.0.0\",+ \"name\": \"bar\",+ \"license\": \"MIT/Apache-2.0\",+ \"description\": \"summary\",+ \"homepage\": \"http://example.com\",+ \"repository\": \"http://example.com\",+ }+}")++(define test-bar-dependencies+ "{+ \"dependencies\": []+}")+ (define test-source-hash "") @@ -68,14 +86,14 @@ (lambda (url . rest) (match url ("https://crates.io/api/v1/crates/foo"- (open-input-string test-crate))+ (open-input-string test-foo-crate)) ("https://crates.io/api/v1/crates/foo/1.0.0/download" (set! test-source-hash (bytevector->nix-base32-string (sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies"- (open-input-string test-dependencies))+ (open-input-string test-foo-dependencies)) (_ (error "Unexpected URL: " url))))) (match (crate->guix-package "foo") (('package@@ -100,4 +118,70 @@ (x (pk 'fail x #f))))) +(test-assert "cargo-recursive-import"+ ;; Replace network resources with sample data.+ (mock ((guix http-client) http-fetch+ (lambda (url . rest)+ (match url+ ("https://crates.io/api/v1/crates/foo"+ (open-input-string test-foo-crate))+ ("https://crates.io/api/v1/crates/foo/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies"+ (open-input-string test-foo-dependencies))+ ("https://crates.io/api/v1/crates/bar"+ (open-input-string test-bar-crate))+ ("https://crates.io/api/v1/crates/bar/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/bar/1.0.0/dependencies"+ (open-input-string test-bar-dependencies))+ (_ (error "Unexpected URL: " url)))))+ (match (stream->list (crate-recursive-import "foo"))+ ((('package+ ('name "rust-foo")+ ('version (? string? ver))+ ('source+ ('origin+ ('method 'url-fetch)+ ('uri ('crate-uri "foo" 'version))+ ('file-name+ ('string-append 'name "-" 'version ".tar.gz"))+ ('sha256+ ('base32+ (? string? hash)))))+ ('build-system 'cargo-build-system)+ ('arguments+ ('quasiquote+ ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))+ ('home-page "http://example.com")+ ('synopsis "summary")+ ('description "summary")+ ('license ('list 'license:expat 'license:asl2.0)))+ ('package+ ('name "rust-bar")+ ('version (? string? ver))+ ('source+ ('origin+ ('method 'url-fetch)+ ('uri ('crate-uri "bar" 'version))+ ('file-name+ ('string-append 'name "-" 'version ".tar.gz"))+ ('sha256+ ('base32+ (? string? hash)))))+ ('build-system 'cargo-build-system)+ ('home-page "http://example.com")+ ('synopsis "summary")+ ('description "summary")+ ('license ('list 'license:expat 'license:asl2.0))))+ #t)+ (x+ (pk 'fail x #f)))))+ (test-end "crate")-- 2.22.0
B
B
Brian Leung wrote on 6 Aug 2019 18:03
CAAc=MExmu0mmcB6BVatRYyTpsVQHiYiSDhC3nzJCX7fXMHzZ9g@mail.gmail.com
Should have sent this to you too, Ivan.
On Tue, Aug 6, 2019 at 5:42 AM Brian Leung <bkleung89@gmail.com> wrote:
Toggle quote (10 lines)> OK, I updated to remove print statements I missed.>> On Mon, Aug 5, 2019 at 7:50 PM Brian Leung <bkleung89@gmail.com> wrote:>>> I took Karl's changes and updated them accordingly. I've also added a>> small test. The patch containing his importer, my changes, and my test is>> attached (the commit was made using my name--not sure if I should instead>> apply Karl's patch).>>>
Attachment: file
E
E
Efraim Flashner wrote on 8 Aug 2019 12:39
Re: [bug#35813] [PATCH] Add crate-recursive-import.
(name . Brian Leung)(address . bkleung89@gmail.com)
20190808103956.GD5507@E2140
On Tue, Aug 06, 2019 at 06:03:23PM +0200, Brian Leung wrote:
Toggle quote (15 lines)> Should have sent this to you too, Ivan.> > On Tue, Aug 6, 2019 at 5:42 AM Brian Leung <bkleung89@gmail.com> wrote:> > > OK, I updated to remove print statements I missed.> >> > On Mon, Aug 5, 2019 at 7:50 PM Brian Leung <bkleung89@gmail.com> wrote:> >> >> I took Karl's changes and updated them accordingly. I've also added a> >> small test. The patch containing his importer, my changes, and my test is> >> attached (the commit was made using my name--not sure if I should instead> >> apply Karl's patch).> >>> >
I ran 'guix import crate -r afl' on a machine where I had a bunch ofcrates pre-packaged the (very shorted output looked like this:
(define-public rust-xdg...
(define-public rust-xdg...
(define-public rust-afl (package (name "rust-afl") (version "0.4.4") (source (origin (method url-fetch) (uri (crate-uri "afl" version)) (file-name (string-append name "-" version ".tar.gz")) (sha256 (base32 "14k6hnwzqn7rrs0hs87vcfqj4334k9wff38d15378frlxpviaard")))) (build-system cargo-build-system) (arguments `(#:cargo-inputs (("rust-cc" ,rust-cc) ("rust-clap" ,rust-clap) ("rust-rustc-version" ,rust-rustc-version) ("rust-xdg" ,rust-xdg)) #:cargo-development-inputs (("rust-rustc-version" ,rust-rustc-version) ("rust-xdg" ,rust-xdg)))) (home-page "https://github.com/rust-fuzz/afl.rs") (synopsis "Fuzzing Rust code with american-fuzzy-lop") (description "Fuzzing Rust code with american-fuzzy-lop") (license #f)))
I know rust-xdg is there twice, but IMO it should only be printed once.
also 'guix import crate -r rusty-fork' gives me #f'guix import crate rusty-fork' gives me:guix import: error: failed to download meta-data for package 'rusty-fork'

-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl1L++sACgkQQarn3Mo9g1GtBA//QcgZ57jbzLlVV/yvMwgp7MuCEx6ZeKKNkWeXg4sT6KzcBoYcPUNKDXhfcO8LrZptUHgZPOR+JLsnSIApHlUEjY2BNIBT4l2ZJKUFaCd0J5SZiuvU6vYWBsSjQQMoW065hZOtAhoYga9m1rsIfOWkMO5jzyyu/+9RNgcgnjRu5H+O9wRttuKXCmihbaNQOC0z91SLbwNf5uCkpUhSgluTid/3GON6v52yKm4qL6sCMtYDUe59p02N5Rth6ZYjnNwOSHBz+9zSYDAY9mRaza17OWdh3ecVXyDK8+MI6AMc7Lt1peiPcUSmQJs8Yen1rNbZ+0eh9f4pWR242kWvpQrTpErc8eFrjVX27XNyTPClPeVOofjdDoAO6ntTjQF4MGrvjvafwUvTXZmfGPihnVVL6RKKJQf6KR3n3RDNhkEfVyFZ2tK+OOczEHcyMZUWp30S/8zzIKFpSBjDL0RleL75RbPCLsFx5GmB1Dk8SXB4uFgO3+KRziA4ZKfZrwtCuV586+LV/FYDNTy/3p/+oaaTCzFPcR0XECaI9x6/OzF28vkUaUY75D85ZYxvxcPTYBGSFAQUqbbZArcOj/0+Uj4U0JIHXbfdo/LsyOsuzWbi6oOv++lFzP8QYPpsj3flw+DcKb/V3KsL/CwOWztwhsSRN+cys3UXyz+calKnQD1Lzls==9Fqc-----END PGP SIGNATURE-----

B
B
Brian Leung wrote on 7 Sep 2019 23:49
(name . Efraim Flashner)(address . efraim@flashner.co.il)
CAAc=MEyUOF8_fqbDu8b+q30uhxUhHKyf5MvW2eNNPwcL8_+FtA@mail.gmail.com
Hi Efraim,
An updated patch is attached.
I was and still am confused why my previous patch was yielding duplicates.I'm not getting duplicates right now, even with packages that previouslyyielded duplicates (I tried this on ripgrep); please test and let me knowif there are any issues. Maybe I fixed the issue in the course of rebasing?
And "guix import crate -r asfeusnoetuhesont" should now print "failed todownload meta-data for...", though I don't know how to do this moreidiomatically without using error or leave (which would end the recursiveimport earlier than desired). And the double quotes actually appear, whichis not ideal. I'd appreciate any advice on how to clean this up.
Best,Brian
On Thu, Aug 8, 2019 at 12:39 PM Efraim Flashner <efraim@flashner.co.il>wrote:
Toggle quote (70 lines)> On Tue, Aug 06, 2019 at 06:03:23PM +0200, Brian Leung wrote:> > Should have sent this to you too, Ivan.> >> > On Tue, Aug 6, 2019 at 5:42 AM Brian Leung <bkleung89@gmail.com> wrote:> >> > > OK, I updated to remove print statements I missed.> > >> > > On Mon, Aug 5, 2019 at 7:50 PM Brian Leung <bkleung89@gmail.com>> wrote:> > >> > >> I took Karl's changes and updated them accordingly. I've also added a> > >> small test. The patch containing his importer, my changes, and my> test is> > >> attached (the commit was made using my name--not sure if I should> instead> > >> apply Karl's patch).> > >>> > >>> I ran 'guix import crate -r afl' on a machine where I had a bunch of> crates pre-packaged the (very shorted output looked like this:>> (define-public rust-xdg> ...>> (define-public rust-xdg> ...>> (define-public rust-afl> (package> (name "rust-afl")> (version "0.4.4")> (source> (origin> (method url-fetch)> (uri (crate-uri "afl" version))> (file-name> (string-append name "-" version ".tar.gz"))> (sha256> (base32> "14k6hnwzqn7rrs0hs87vcfqj4334k9wff38d15378frlxpviaard"))))> (build-system cargo-build-system)> (arguments> `(#:cargo-inputs> (("rust-cc" ,rust-cc)> ("rust-clap" ,rust-clap)> ("rust-rustc-version" ,rust-rustc-version)> ("rust-xdg" ,rust-xdg))> #:cargo-development-inputs> (("rust-rustc-version" ,rust-rustc-version)> ("rust-xdg" ,rust-xdg))))> (home-page "https://github.com/rust-fuzz/afl.rs")> (synopsis> "Fuzzing Rust code with american-fuzzy-lop")> (description> "Fuzzing Rust code with american-fuzzy-lop")> (license #f)))>> I know rust-xdg is there twice, but IMO it should only be printed once.>> also 'guix import crate -r rusty-fork' gives me #f> 'guix import crate rusty-fork' gives me:> guix import: error: failed to download meta-data for package 'rusty-fork'>>> --> Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר> GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351> Confidentiality cannot be guaranteed on emails sent or received unencrypted>
Attachment: file
From 751bf2367edf54015792f339dcaca797cd7da937 Mon Sep 17 00:00:00 2001From: Brian Leung <bkleung89@gmail.com>Date: Sat, 20 Jul 2019 21:35:14 +0200Subject: [PATCH] gnu: Add crate-recursive-import.
* guix/import/crate.scm (crate-recursive-import): New variable.* guix/script/import/crate.scm: Add recursive option.* guix/tests/crate.scm (crate-recursive-import): New test.------ guix/import/crate.scm | 131 +++++++++++++------------ guix/import/utils.scm | 16 ++-- guix/scripts/import/crate.scm | 32 ++++++- tests/crate.scm | 173 +++++++++++++++++++++++++++++++++- 4 files changed, 273 insertions(+), 79 deletions(-)
Toggle diff (482 lines)diff --git a/guix/import/crate.scm b/guix/import/crate.scmindex f6057dbf8b..5e81c015d8 100644--- a/guix/import/crate.scm+++ b/guix/import/crate.scm@@ -38,6 +38,7 @@ #:use-module (srfi srfi-2) #:use-module (srfi srfi-26) #:export (crate->guix-package+ crate-recursive-import guix-package->crate-name %crate-updater)) @@ -147,78 +148,86 @@ VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTIO and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name))- (cargo-inputs (map crate-name->package-name cargo-inputs))- (cargo-development-inputs (map crate-name->package-name+ (inputs (map crate-name->package-name cargo-inputs))+ (development-inputs (map crate-name->package-name cargo-development-inputs)) (pkg `(package- (name ,guix-name)- (version ,version)- (source (origin- (method url-fetch)- (uri (crate-uri ,name version))- (file-name (string-append name "-" version ".tar.gz"))- (sha256- (base32- ,(bytevector->nix-base32-string (port-sha256 port))))))- (build-system cargo-build-system)- ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs)- (maybe-cargo-development-inputs- cargo-development-inputs)))- (home-page ,(match home-page- (() "")- (_ home-page)))- (synopsis ,synopsis)- (description ,(beautify-description description))- (license ,(match license- (() #f)- ((license) license)- (_ `(list ,@license)))))))- (close-port port)- pkg))+ (name ,guix-name)+ (version ,version)+ (source (origin+ (method url-fetch)+ (uri (crate-uri ,name version))+ (file-name (string-append name "-" version ".tar.gz"))+ (sha256+ (base32+ ,(bytevector->nix-base32-string (port-sha256 port))))))+ (build-system cargo-build-system)+ ,@(maybe-arguments (append (maybe-cargo-inputs inputs)+ (maybe-cargo-development-inputs+ development-inputs)))+ (home-page ,(match home-page+ (() "")+ (_ home-page)))+ (synopsis ,synopsis)+ (description ,(beautify-description description))+ (license ,(match license+ (() #f)+ ((license) license)+ (_ `(list ,@license)))))))+ (close-port port)+ (values pkg+ (lset-union equal? cargo-inputs cargo-development-inputs)))) (define %dual-license-rx ;; Dual licensing is represented by a string such as "MIT OR Apache-2.0". ;; This regexp matches that. (make-regexp "^(.*) OR (.*)$")) -(define (crate->guix-package crate-name)- "Fetch the metadata for CRATE-NAME from crates.io, and return the-`package' s-expression corresponding to that package, or #f on failure."- (define (string->license string)- (match (regexp-exec %dual-license-rx string)- (#f (list (spdx-string->license string)))- (m (list (spdx-string->license (match:substring m 1))- (spdx-string->license (match:substring m 2))))))+(define (string->license string)+ (match (regexp-exec %dual-license-rx string)+ (#f (list (spdx-string->license string)))+ (m (list (spdx-string->license (match:substring m 1))+ (spdx-string->license (match:substring m 2))))))++(define (normal-dependency? dependency)+ (eq? (crate-dependency-kind dependency) 'normal)) - (define (normal-dependency? dependency)- (eq? (crate-dependency-kind dependency) 'normal))+(define crate->guix-package+ (memoize+ (lambda (crate-name)+ "Fetch the metadata for CRATE-NAME from crates.io, and return the+`package' s-expression corresponding to that package, or #f on failure."+ (define crate+ (lookup-crate crate-name)) - (define crate- (lookup-crate crate-name))+ (and crate+ (let* ((version (find (lambda (version)+ (string=? (crate-version-number version)+ (crate-latest-version crate)))+ (crate-versions crate)))+ (dependencies (crate-version-dependencies version))+ (dep-crates (filter normal-dependency? dependencies))+ (dev-dep-crates (remove normal-dependency? dependencies))+ (cargo-inputs (sort (map crate-dependency-id dep-crates)+ string-ci<?))+ (cargo-development-inputs+ (sort (map crate-dependency-id dev-dep-crates)+ string-ci<?)))+ (make-crate-sexp #:name crate-name+ #:version (crate-version-number version)+ #:cargo-inputs cargo-inputs+ #:cargo-development-inputs cargo-development-inputs+ #:home-page (or (crate-home-page crate)+ (crate-repository crate))+ #:synopsis (crate-description crate)+ #:description (crate-description crate)+ #:license (and=> (crate-version-license version)+ string->license))))))) - (and crate- (let* ((version (find (lambda (version)- (string=? (crate-version-number version)- (crate-latest-version crate)))- (crate-versions crate)))- (dependencies (crate-version-dependencies version))- (dep-crates (filter normal-dependency? dependencies))- (dev-dep-crates (remove normal-dependency? dependencies))- (cargo-inputs (sort (map crate-dependency-id dep-crates)- string-ci<?))- (cargo-development-inputs- (sort (map crate-dependency-id dev-dep-crates)- string-ci<?)))- (make-crate-sexp #:name crate-name- #:version (crate-version-number version)- #:cargo-inputs cargo-inputs- #:cargo-development-inputs cargo-development-inputs- #:home-page (or (crate-home-page crate)- (crate-repository crate))- #:synopsis (crate-description crate)- #:description (crate-description crate)- #:license (and=> (crate-version-license version)- string->license)))))+(define* (crate-recursive-import package-name)+ (recursive-import package-name #f+ #:repo->guix-package (lambda (name _) (crate->guix-package name))+ #:guix-name crate-name->package-name)) (define (guix-package->crate-name package) "Return the crate name of PACKAGE."diff --git a/guix/import/utils.scm b/guix/import/utils.scmindex 252875eeab..e58f5cba94 100644--- a/guix/import/utils.scm+++ b/guix/import/utils.scm@@ -381,16 +381,16 @@ dependencies." ((prev (next . rest) done) (define (handle? dep) (and- (not (equal? dep next))- (not (member dep done))- (not (exists? dep))))+ (not (equal? dep next))+ (not (member dep done))+ (not (exists? dep)))) (receive (package . dependencies) (repo->guix-package next repo) (list- (if package package '()) ;; default #f on failure would interrupt- (if package- (lset-union equal? rest (filter handle? (car dependencies)))- rest)- (cons next done))))+ (or package next)+ (if package+ (lset-union equal? rest (filter handle? (car dependencies)))+ rest)+ (cons next done)))) ((prev '() done) (list #f '() done)))) diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scmindex cab9a4397b..9970b1a231 100644--- a/guix/scripts/import/crate.scm+++ b/guix/scripts/import/crate.scm@@ -27,6 +27,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-37)+ #:use-module (srfi srfi-41) #:use-module (ice-9 match) #:use-module (ice-9 format) #:export (guix-import-crate))@@ -45,6 +46,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (display (G_ " -h, --help display this help and exit")) (display (G_ "+ -r, --recursive import packages recursively"))+ (display (G_ " -V, --version display version information and exit")) (newline) (show-bug-report-information))@@ -58,6 +61,9 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (option '(#\V "version") #f #f (lambda args (show-version-and-exit "guix import crate")))+ (option '(#\r "recursive") #f #f+ (lambda (opt name arg result)+ (alist-cons 'recursive #t result))) %standard-import-options)) @@ -83,11 +89,27 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (reverse opts)))) (match args ((package-name)- (let ((sexp (crate->guix-package package-name)))- (unless sexp- (leave (G_ "failed to download meta-data for package '~a'~%")- package-name))- sexp))+ (if (assoc-ref opts 'recursive)+ ;; Recursive import+ (map (match-lambda+ ((and ('package ('name name) . rest) pkg)+ `(define-public ,(string->symbol name)+ ,pkg))+ ((and string? pkg-name)+ ;; (format #f (G_ "failed to download meta-data for package '~a'") dep-name)+ (string-append "failed to download meta-data for package '"+ pkg-name+ "'"))+ (_ #f))+ (reverse+ (stream->list+ (crate-recursive-import package-name))))+ ;; Single import+ (let ((sexp (crate->guix-package package-name)))+ (unless sexp+ (leave (G_ "failed to download meta-data for package '~a'~%")+ package-name))+ sexp))) (() (leave (G_ "too few arguments~%"))) ((many ...)diff --git a/tests/crate.scm b/tests/crate.scmindex c14862ad9f..8e7b0bda9b 100644--- a/tests/crate.scm+++ b/tests/crate.scm@@ -26,9 +26,10 @@ #:use-module (guix tests) #:use-module (ice-9 iconv) #:use-module (ice-9 match)+ #:use-module (srfi srfi-41) #:use-module (srfi srfi-64)) -(define test-crate+(define test-foo-crate "{ \"crate\": { \"max_version\": \"1.0.0\",@@ -50,16 +51,81 @@ } }") -(define test-dependencies+(define test-foo-dependencies "{ \"dependencies\": [ { \"crate_id\": \"bar\", \"kind\": \"normal\",+ },+ {+ \"crate_id\": \"baz\",+ \"kind\": \"normal\",+ }+ ]+}")++(define test-bar-crate+ "{+ \"crate\": {+ \"max_version\": \"1.0.0\",+ \"name\": \"bar\",+ \"description\": \"summary\",+ \"homepage\": \"http://example.com\",+ \"repository\": \"http://example.com\",+ \"keywords\": [\"dummy\" \"test\"],+ \"categories\": [\"test\"]+ \"actual_versions\": [+ { \"id\": \"bar\",+ \"num\": \"1.0.0\",+ \"license\": \"MIT OR Apache-2.0\",+ \"links\": {+ \"dependencies\": \"/api/v1/crates/bar/1.0.0/dependencies\"+ }+ }+ ]+ \"license\": \"MIT OR Apache-2.0\",+ }+}")++(define test-bar-dependencies+ "{+ \"dependencies\": [+ {+ \"crate_id\": \"baz\",+ \"kind\": \"normal\", } ] }") +(define test-baz-crate+ "{+ \"crate\": {+ \"max_version\": \"1.0.0\",+ \"name\": \"baz\",+ \"description\": \"summary\",+ \"homepage\": \"http://example.com\",+ \"repository\": \"http://example.com\",+ \"keywords\": [\"dummy\" \"test\"],+ \"categories\": [\"test\"]+ \"actual_versions\": [+ { \"id\": \"baz\",+ \"num\": \"1.0.0\",+ \"license\": \"MIT OR Apache-2.0\",+ \"links\": {+ \"dependencies\": \"/api/v1/crates/baz/1.0.0/dependencies\"+ }+ }+ ]+ \"license\": \"MIT OR Apache-2.0\",+ }+}")++(define test-baz-dependencies+ "{+\"dependencies\": []+}")+ (define test-source-hash "") @@ -79,14 +145,14 @@ (lambda (url . rest) (match url ("https://crates.io/api/v1/crates/foo"- (open-input-string test-crate))+ (open-input-string test-foo-crate)) ("https://crates.io/api/v1/crates/foo/1.0.0/download" (set! test-source-hash (bytevector->nix-base32-string (sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies"- (open-input-string test-dependencies))+ (open-input-string test-foo-dependencies)) (_ (error "Unexpected URL: " url))))) (match (crate->guix-package "foo") (('package@@ -102,7 +168,8 @@ ('build-system 'cargo-build-system) ('arguments ('quasiquote- ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))))))+ ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))+ ("rust-baz" ('unquote rust-baz)))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary")@@ -111,4 +178,100 @@ (x (pk 'fail x #f))))) +(test-assert "cargo-recursive-import"+ ;; Replace network resources with sample data.+ (mock ((guix http-client) http-fetch+ (lambda (url . rest)+ (match url+ ("https://crates.io/api/v1/crates/foo"+ (open-input-string test-foo-crate))+ ("https://crates.io/api/v1/crates/foo/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies"+ (open-input-string test-foo-dependencies))+ ("https://crates.io/api/v1/crates/bar"+ (open-input-string test-bar-crate))+ ("https://crates.io/api/v1/crates/bar/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/bar/1.0.0/dependencies"+ (open-input-string test-bar-dependencies))+ ("https://crates.io/api/v1/crates/baz"+ (open-input-string test-baz-crate))+ ("https://crates.io/api/v1/crates/baz/1.0.0/download"+ (set! test-source-hash+ (bytevector->nix-base32-string+ (sha256 (string->bytevector "empty file\n" "utf-8"))))+ (open-input-string "empty file\n"))+ ("https://crates.io/api/v1/crates/baz/1.0.0/dependencies"+ (open-input-string test-baz-dependencies))+ (_ (error "Unexpected URL: " url)))))+ (match (stream->list (crate-recursive-import "foo"))+ ((('package+ ('name "rust-foo")+ ('version (? string? ver))+ ('source+ ('origin+ ('method 'url-fetch)+ ('uri ('crate-uri "foo" 'version))+ ('file-name+ ('string-append 'name "-" 'version ".tar.gz"))+ ('sha256+ ('base32+ (? string? hash)))))+ ('build-system 'cargo-build-system)+ ('arguments+ ('quasiquote+ ('#:cargo-inputs (("rust-bar" ('unquote rust-bar))+ ("rust-baz" ('unquote rust-baz))))))+ ('home-page "http://example.com")+ ('synopsis "summary")+ ('description "summary")+ ('license ('list 'license:expat 'license:asl2.0)))+ ('package+ ('name "rust-bar")+ ('version (? string? ver))+ ('source+ ('origin+ ('method 'url-fetch)+ ('uri ('crate-uri "bar" 'version))+ ('file-name+ ('string-append 'name "-" 'version ".tar.gz"))+ ('sha256+ ('base32+ (? string? hash)))))+ ('build-system 'cargo-build-system)+ ('arguments+ ('quasiquote+ ('#:cargo-inputs (("rust-baz" ('unquote rust-baz))))))+ ('home-page "http://example.com")+ ('synopsis "summary")+ ('description "summary")+ ('license ('list 'license:expat 'license:asl2.0)))+ ('package+ ('name "rust-baz")+ ('version (? string? ver))+ ('source+ ('origin+ ('method 'url-fetch)+ ('uri ('crate-uri "baz" 'version))+ ('file-name+ ('string-append 'name "-" 'version ".tar.gz"))+ ('sha256+ ('base32+ (? string? hash)))))+ ('build-system 'cargo-build-system)+ ('home-page "http://example.com")+ ('synopsis "summary")+ ('description "summary")+ ('license ('list 'license:expat 'license:asl2.0))))+ #t)+ (x+ (pk 'fail x #f)))))+ (test-end "crate")-- 2.23.0
E
E
Efraim Flashner wrote on 8 Sep 2019 09:57
(name . Brian Leung)(address . bkleung89@gmail.com)
20190908075730.GA977@E5400
As a simple test I ran 'guix import crate encoding -r' and it gave methe 6 packages I expected. 'guix import crate winapi -r' only gave merust-winapi, as the dependent crates are already packaged. When I tried'guix import crate rand -r' it found the updated version and startedimporting all the new dependencies also.
It looks good. I'm tempted to leave it running with 'guix import crateserde -r' just to see if we're ever going to make it there.
I see that it imports A then B then C then D, and prints out D then Cthen B then A. For the one I tested with is still rust-encoding. I'lltry my hand at ascii art:
encoding | ------------------------------------------------------- | | | | |japanese korean simpchinese singlebyte tradchinese | | | | | ------------------------------------------------------- | encoding-tests
import went encoding, japanese, tests, korean, simpchinese, singlebyte,tradchinese
I think the only thing I would wish for would be to do tests, then thelanguages and then encoding (best for upstreaming one at a time), or todo them alphabetically (plop them in alphabetically all at once). ThisI'm happy to live without I think.
The other thing was I ran 'guix import crate security-framework -r' andafter ~40 crates it crashed on me with: web/http.scm:1186:15: In procedure read-response-line: Bad Read-Header-Line header: #<eof>and I would prefer to have the ~40 crates it did grab first to beprinted out and not lost. Between these two I would like most to notlose the imported crates than worrying over the printed order.
Great job! From what I've tested I think it's ready as-is and anychanges would just be gravy.

-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl10tGYACgkQQarn3Mo9g1EKBxAAhZlAiOJPpc3Yu4Qg0hUegfq0tpMRIefiTZ+YNMmQeQ6A50UVOctwraJo088Co86+EYIfWfDN43Dm78SrYEviE5rhVqpPbEaMQYKMpRyI2tGPHdLmOM0ijiZ0UakCGakWcQhfKBwlqRxdKhimk0HU/7k0ruixbtAYHhsvflQ7guf5nziSxLccZgLqWLI34r7ZWB+VYeqqTZuaGOqnsEk1Y0+KESySkm2pi6sc2qImGuVqqtO+KXbCiAQqtOPorw+G/x3Wk71ebM1ZAKK+nVmZPya2eeFZvVvvAELM4eBOljfSpRTDQjGB/kFEKgtuefh8WQdyPCuQGkeI+SRl/npQbrPelPbjF5tzdbGv0X766OPEFlY0VH3Q2bKgHVePtizoK0XYHrdVB+mBw5bA/qc8xsdrsSXpV3yy9x12OzPD5/RTJOdbgw56xHtXVAC+cQQCxOUXsBadge9iI9UrVP4GzPLcIXRADDxBzxnRUiHP4hbThQ2181dpol3w9LXc84iE5tvZbGJJnl5a9q70ACtsvZ326ajSmzNXpMD6t5WCyk2qTwmeDfQ3CVbIPJ7xdx+mp1WRTiFSwtgqe8DqQ4dJ54hVqcOWZOwB6hT3IDuzh48ezNWrWMqc0yeE1V1HJz5AsLBW+Ftcgk3NIc1IsdZ1q5+lDFfmhZkTDJLe7pLPgqc==7hxM-----END PGP SIGNATURE-----

B
B
Brian Leung wrote on 13 Oct 2019 09:42
(name . Efraim Flashner)(address . efraim@flashner.co.il)
CAAc=MEx=2ja4qba-dms=7L3uXX2X-EcMyECf6_Cs0Zq9jHfrWA@mail.gmail.com
Closing this since someone else already merged their own crate recursiveimporter.
On Sun, Sep 8, 2019 at 12:57 AM Efraim Flashner <efraim@flashner.co.il>wrote:
Toggle quote (48 lines)> As a simple test I ran 'guix import crate encoding -r' and it gave me> the 6 packages I expected. 'guix import crate winapi -r' only gave me> rust-winapi, as the dependent crates are already packaged. When I tried> 'guix import crate rand -r' it found the updated version and started> importing all the new dependencies also.>> It looks good. I'm tempted to leave it running with 'guix import crate> serde -r' just to see if we're ever going to make it there.>> I see that it imports A then B then C then D, and prints out D then C> then B then A. For the one I tested with is still rust-encoding. I'll> try my hand at ascii art:>> encoding> |> -------------------------------------------------------> | | | | |> japanese korean simpchinese singlebyte tradchinese> | | | | |> -------------------------------------------------------> |> encoding-tests>> import went encoding, japanese, tests, korean, simpchinese, singlebyte,> tradchinese>> I think the only thing I would wish for would be to do tests, then the> languages and then encoding (best for upstreaming one at a time), or to> do them alphabetically (plop them in alphabetically all at once). This> I'm happy to live without I think.>> The other thing was I ran 'guix import crate security-framework -r' and> after ~40 crates it crashed on me with:> web/http.scm:1186:15: In procedure read-response-line:> Bad Read-Header-Line header: #<eof>> and I would prefer to have the ~40 crates it did grab first to be> printed out and not lost. Between these two I would like most to not> lose the imported crates than worrying over the printed order.>> Great job! From what I've tested I think it's ready as-is and any> changes would just be gravy.>>> --> Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר> GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351> Confidentiality cannot be guaranteed on emails sent or received unencrypted>
Attachment: file
Closed
?
Your comment

This issue is archived.

To comment on this conversation send email to 35813@debbugs.gnu.org