[PATCH] build-system/cargo: Propagate crates across builds.

DoneSubmitted by Efraim Flashner.
Details
3 participants
  • Efraim Flashner
  • Leo Famulari
  • pelzflorian (Florian Pelz)
Owner
unassigned
Severity
normal
E
E
Efraim Flashner wrote on 9 Feb 11:58 +0100
(address . guix-patches@gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
20210209105843.13230-1-efraim@flashner.co.il
* guix/build-system/cargo.scm (cargo-build): Add cargo-package-flags,install-source flags.* guix/build/cargo-build-system.scm (unpack-rust-crates, package): Newprocedures.(install): Also install crate sources.(%standard-phases): Add new phases.* doc/guix.texi (Packaging-guidelines)[Rust Crates]: Adjust to changesin the cargo-build-system.---
With this patch, each current crate will also install its sources, bothin a .crate (tar.gz) and unpacked.
(ins)efraim@3900XT ~$ tree -d /gnu/store/anklck1x25qk43dk5p442iarfcpknwwp-rust-bencher-0.1.5/gnu/store/anklck1x25qk43dk5p442iarfcpknwwp-rust-bencher-0.1.5└── share ├── cargo │   ├── registry │   └── src │   └── bencher-0.1.5 │   └── benches └── doc └── rust-bencher-0.1.5
8 directories
A new phase after 'unpack will find the .crate files inshare/cargo/registry, copy them to target/registry and unpack them intovendor-dir for use with the next build. At the end of that build all the.crate tarballs are copied to that package's %out/share/cargo/registry,and also unpacked into %out/share/cargo/src.
This means that with this patch and some modifications to existingpackages to use the new features it should be possible to run 'guixenvironment rust-foo' and then in the .cargo/config set$GUIX_ENVIRONMENT/share/cargo/src as a 'directory source' for cargo.
I have to look more into source-replacement¹ to see if I can organizethe .crate files in %out/share/cargo/registry into a local-registryformat for use as a 'cache' for crates.io.
¹ https://doc.rust-lang.org/cargo/reference/source-replacement.html

doc/guix.texi | 15 ++++--- guix/build-system/cargo.scm | 4 ++ guix/build/cargo-build-system.scm | 66 +++++++++++++++++++++++++++++-- 3 files changed, 77 insertions(+), 8 deletions(-)
Toggle diff (179 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex 7d18703283..0d153a1470 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -32,7 +32,7 @@ Copyright @copyright{} 2015, 2016, 2017, 2019, 2020 Leo Famulari@* Copyright @copyright{} 2015, 2016, 2017, 2018, 2019, 2020 Ricardo Wurmus@* Copyright @copyright{} 2016 Ben Woodcroft@* Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@*-Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Efraim Flashner@*+Copyright @copyright{} 2016, 2017, 2018, 2019, 2020, 2021 Efraim Flashner@* Copyright @copyright{} 2016 John Darrington@* Copyright @copyright{} 2016, 2017 Nikita Gillmann@* Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen@*@@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the It adds @code{rustc} and @code{cargo} to the set of inputs. A different Rust package can be specified with the @code{#:rust} parameter. -Regular cargo dependencies should be added to the package definition via the-@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the+Regular cargo dependencies should be added to the package definition similarly+to other packages; those needed only at build time to native-inputs, others to+inputs. If you need to add source-only crates then you should add them to via+the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the spec can be a package or a source definition. Note that the spec must evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml} file at its root, or it will be ignored. Similarly, cargo dev-dependencies@@ -7449,8 +7451,11 @@ In its @code{configure} phase, this build system will make any source inputs specified in the @code{#:cargo-inputs} and @code{#:cargo-development-inputs} parameters available to cargo. It will also remove an included @code{Cargo.lock} file to be recreated by @code{cargo} during the-@code{build} phase. The @code{install} phase installs the binaries-defined by the crate.+@code{build} phase. The @code{package} phase will run @code{cargo package}+to create a source crate for future use. The @code{install} phase installs+the binaries defined by the crate. Unless @code{install-source? #f} is+defined it will also install a source crate repository of itself and unpacked+sources, to ease in future hacking on rust packages. @end defvr @defvr {Scheme Variable} chicken-build-systemdiff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scmindex 6c8edf6bac..01667ef045 100644--- a/guix/build-system/cargo.scm+++ b/guix/build-system/cargo.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>+;;; Copyright © 2021 Efraim Flashner <efraim@flashner.co.il> ;;; ;;; This file is part of GNU Guix. ;;;@@ -77,8 +78,10 @@ to NAME and VERSION." (vendor-dir "guix-vendor") (cargo-build-flags ''("--release")) (cargo-test-flags ''("--release"))+ (cargo-package-flags ''("--no-metadata" "--no-verify")) (features ''()) (skip-build? #f)+ (install-source? #t) (phases '(@ (guix build cargo-build-system) %standard-phases)) (outputs '("out"))@@ -106,6 +109,7 @@ to NAME and VERSION." #:vendor-dir ,vendor-dir #:cargo-build-flags ,cargo-build-flags #:cargo-test-flags ,cargo-test-flags+ #:cargo-package-flags ,cargo-package-flags #:features ,features #:skip-build? ,skip-build? #:tests? ,(and tests? (not skip-build?))diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scmindex 1d21b33895..0328ede4cb 100644--- a/guix/build/cargo-build-system.scm+++ b/guix/build/cargo-build-system.scm@@ -2,7 +2,7 @@ ;;; Copyright © 2016 David Craven <david@craven.ch> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com> ;;; Copyright © 2019 Ivan Petkov <ivanppetkov@gmail.com>-;;; Copyright © 2019, 2020 Efraim Flashner <efraim@flashner.co.il>+;;; Copyright © 2019, 2020, 2021 Efraim Flashner <efraim@flashner.co.il> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net> ;;; Copyright © 2020 Marius Bakke <marius@gnu.org> ;;;@@ -73,6 +73,38 @@ Cargo.toml file present at its root." " | cut -d/ -f2" " | grep -q '^Cargo.toml$'"))))) +(define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys)+ (define (inputs->rust-inputs inputs)+ "Filter using the label part from INPUTS."+ (filter (lambda (input)+ (match input+ ((name . _) (rust-package? name))))+ inputs))+ (define (inputs->directories inputs)+ "Extract the directory part from INPUTS."+ (match inputs+ (((names . directories) ...)+ directories)))++ (let ((rust-inputs (inputs->directories (inputs->rust-inputs inputs))))+ (unless (null? rust-inputs)+ (mkdir-p "target/package")+ (mkdir-p vendor-dir)+ ;; TODO: copy only regular inputs to target/package, not native-inputs.+ (for-each (lambda (input-crate)+ (copy-recursively (string-append input-crate+ "/share/cargo/registry")+ "target/package"))+ (delete-duplicates rust-inputs))++ (for-each (lambda (crate)+ (invoke "tar" "xzf" crate "-C" vendor-dir))+ (find-files "target/package" "\\.crate$"))))+ #t)++(define (rust-package? name)+ (string-prefix? "rust-" name))+ (define* (configure #:key inputs (vendor-dir "guix-vendor") #:allow-other-keys)@@ -170,9 +202,23 @@ directory = '" port) (apply invoke "cargo" "test" cargo-test-flags) #t)) -(define* (install #:key inputs outputs skip-build? features #:allow-other-keys)+(define* (package #:key+ (cargo-package-flags '("--no-metadata" "--no-verify"))+ #:allow-other-keys)+ "Run 'cargo-package' for a given Cargo package."+ (apply invoke `("cargo" "package" ,@cargo-package-flags)))++(define* (install #:key+ inputs+ outputs+ skip-build?+ (install-source? #t)+ features+ #:allow-other-keys) "Install a given Cargo package."- (let* ((out (assoc-ref outputs "out")))+ (let* ((out (assoc-ref outputs "out"))+ (registry (string-append out "/share/cargo/registry"))+ (sources (string-append out "/share/cargo/src"))) (mkdir-p out) ;; Make cargo reuse all the artifacts we just built instead@@ -186,6 +232,18 @@ directory = '" port) (invoke "cargo" "install" "--no-track" "--path" "." "--root" out "--features" (string-join features))) + (when install-source?+ ;; Install crate tarballs and unpacked sources for later use.+ ;; TODO: Is there a better format/directory for these files?+ (mkdir-p sources)+ (for-each (lambda (crate)+ (install-file crate registry))+ (find-files "target/package" "\\.crate$"))++ (for-each (lambda (crate)+ (invoke "tar" "xzf" crate "-C" sources))+ (find-files registry "\\.crate$")))+ #t)) (define %standard-phases@@ -195,6 +253,8 @@ directory = '" port) (replace 'build build) (replace 'check check) (replace 'install install)+ (add-after 'build 'package package)+ (add-after 'unpack 'unpack-rust-crates unpack-rust-crates) (add-after 'patch-generated-file-shebangs 'patch-cargo-checksums patch-cargo-checksums))) (define* (cargo-build #:key inputs (phases %standard-phases)-- 2.30.0
E
E
Efraim Flashner wrote on 14 Mar 18:59 +0100
Re: bug#46399: Acknowledgement ([PATCH] build-system/cargo: Propagate crates across builds.)
(address . 46399-done@debbugs.gnu.org)
YE5PC0wlQHNh3/3i@3900XT
Pushed with minor revisions. Also patched a few packages which needed#:install-source #? in order to continue to build.
-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmBOTwsACgkQQarn3Mo9g1HtmxAAoAJ0S0YwbQZvuNAgV3ZWPeb2l/mQjYFHnGKAPq25kzwNSXWGetGqtobVeq7LB7z/uzeK04ZrzltHfWhnDTmeo7exTa0aTrs+LtnyXuPKR6SVGI9boepnZUP5bEgCBTQWGOW0OuMBUGPgoz1QD0ZkGAm45x5KXpiM4k7TN6cwxj0wugB5fiLt6r0a3BR/0Y3OshV/sjhMqEn3mb7lNhvbM8y96NJGC5+m0kJsVxW0ICKpoN4pEqZB3B5Fs3lDcCJuvziwDhX8NUs9UIi3kcdsGAZfVbNfd5scei3pHHSWohUBuamBCTzBgBx0KtmnaEcWrXE5c7RDTYCUZOkbBPAX9UslnMFlg5wbCZOhT0/03a+nAj32bgko9sdsAXr9LSBUejNL9QtxCc9k5A1utF7BXqzZSfjgRWJVopKLY6MTXpYpp9MQiq9R61FzjllqxS7ZlBeYitub1kDE0Ub5E3uU42vKJIvQXxo0Ybu2GMt1mZX50GkmpZCWuSZLSxoFLzhB06XGNZ1ed62fPmyJQNMpKh20UIqbrbi0It5mLp/V+wSdLvsA6p2ZAWiqQ1xwGxpvgKwQU3QWCJy9wDztolusRXWX7Bwtyla4TqpexytgKEG9XaEwU+2G3mXdviHydNCS98leDGgcxV6XfpcrRnmuS8jmOrMI3R4pTaG2RvDtTgQ==0Pfy-----END PGP SIGNATURE-----

Closed
P
P
pelzflorian (Florian Pelz) wrote on 11 Apr 17:11 +0200
Re: [bug#46399] [PATCH] build-system/cargo: Propagate crates across builds.
(name . Efraim Flashner)(address . efraim@flashner.co.il)(address . 46399@debbugs.gnu.org)
20210411151052.ckjjpbrpqgfp43tl@pelzflorian.localdomain
Hello Efraim!
Thank you for improving Rust development with Guix. I have a questionabout this change to doc/guix.texi when translating. Sorry fornoticing so late just before the string freeze.
On Tue, Feb 09, 2021 at 12:58:43PM +0200, Efraim Flashner wrote:
Toggle quote (19 lines)> diff --git a/doc/guix.texi b/doc/guix.texi> index 7d18703283..0d153a1470 100644> --- a/doc/guix.texi> +++ b/doc/guix.texi> […]> @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the> It adds @code{rustc} and @code{cargo} to the set of inputs.> A different Rust package can be specified with the @code{#:rust} parameter.> > -Regular cargo dependencies should be added to the package definition via the> -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the> +Regular cargo dependencies should be added to the package definition similarly> +to other packages; those needed only at build time to native-inputs, others to> +inputs. If you need to add source-only crates then you should add them to via> +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the> spec can be a package or a source definition. Note that the spec must> evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml}> file at its root, or it will be ignored. Similarly, cargo dev-dependencies
I have no experience in Rust. Do I understand correctly thatsource-only crates among the inputs should be added both to inputs andto #:cargo-inputs? Is the
Toggle quote (2 lines)> +inputs. If you need to add source-only crates then you should add them to via
a typo which should read “add them too” instead of “add them to”?
I look at the subsequent changef35ab79e033aa8e4955e9c9100c74ecbb3ce4d0b to rust-wayland-commons
Toggle quote (21 lines)> diff --git a/gnu/packages/crates-graphics.scm b/gnu/packages/crates-graphics.scm> index d67e9dc02f..7dde00ffdb 100644> --- a/gnu/packages/crates-graphics.scm> +++ b/gnu/packages/crates-graphics.scm> @@ -2415,8 +2415,11 @@ the wayland protocol, client side.")> (arguments> - `(#:skip-build? #t> - #:cargo-inputs> + `(#:cargo-inputs> (("rust-nix" ,rust-nix-0.18)> ("rust-once-cell" ,rust-once-cell-1)> - ("rust-smallvec" ,rust-smallvec-1)> - ("rust-wayland-sys" ,rust-wayland-sys-0.28))))> + ("rust-smallvec" ,rust-smallvec-1))))> + (inputs> + `(("rust-nix" ,rust-nix-0.18)> + ("rust-once-cell" ,rust-once-cell-1)> + ("rust-smallvec" ,rust-smallvec-1)> + ("rust-wayland-sys" ,rust-wayland-sys-0.28)))> (home-page "https://github.com/smithay/wayland-rs")
and do not understand why rust-wayland-sys is only part of inputs butrust-once-cell is also among the cargo-inputs. Also the inputs areempty for older versions of rust-wayland-commons.
Could you clarify what is written in the doc/guix.texi?
Regards,Florian
E
E
Efraim Flashner wrote on 12 Apr 08:40 +0200
(name . pelzflorian (Florian Pelz))(address . pelzflorian@pelzflorian.de)(address . 46399@debbugs.gnu.org)
YHPrUS/hT3Kp2hcQ@3900XT
On Sun, Apr 11, 2021 at 05:11:21PM +0200, pelzflorian (Florian Pelz) wrote:
Toggle quote (34 lines)> Hello Efraim!> > Thank you for improving Rust development with Guix. I have a question> about this change to doc/guix.texi when translating. Sorry for> noticing so late just before the string freeze.> > On Tue, Feb 09, 2021 at 12:58:43PM +0200, Efraim Flashner wrote:> > diff --git a/doc/guix.texi b/doc/guix.texi> > index 7d18703283..0d153a1470 100644> > --- a/doc/guix.texi> > +++ b/doc/guix.texi> > […]> > @@ -7437,8 +7437,10 @@ supports builds of packages using Cargo, the build tool of the> > It adds @code{rustc} and @code{cargo} to the set of inputs.> > A different Rust package can be specified with the @code{#:rust} parameter.> > > > -Regular cargo dependencies should be added to the package definition via the> > -@code{#:cargo-inputs} parameter as a list of name and spec pairs, where the> > +Regular cargo dependencies should be added to the package definition similarly> > +to other packages; those needed only at build time to native-inputs, others to> > +inputs. If you need to add source-only crates then you should add them to via> > +the @code{#:cargo-inputs} parameter as a list of name and spec pairs, where the> > spec can be a package or a source definition. Note that the spec must> > evaluate to a path to a gzipped tarball which includes a @code{Cargo.toml}> > file at its root, or it will be ignored. Similarly, cargo dev-dependencies> > I have no experience in Rust. Do I understand correctly that> source-only crates among the inputs should be added both to inputs and> to #:cargo-inputs? Is the> > > +inputs. If you need to add source-only crates then you should add them to via> > a typo which should read “add them too” instead of “add them to”?
I went back and refreshed myself with what I wrote there. source-onlycrates should be added to both regular inputs and to #:cargo-inputs.Actually converting the ~1000 rust packages to actually make good use ofthe changes will allow us to drop nearly all the #:cargo-inputs and justuse regular inputs, but until then we still need the propagated sourcemagic that comes from #:cargo-inputs.
Yes, definitely a typo, it should be "add them too".
Toggle quote (28 lines)> I look at the subsequent change> f35ab79e033aa8e4955e9c9100c74ecbb3ce4d0b to rust-wayland-commons> > > diff --git a/gnu/packages/crates-graphics.scm b/gnu/packages/crates-graphics.scm> > index d67e9dc02f..7dde00ffdb 100644> > --- a/gnu/packages/crates-graphics.scm> > +++ b/gnu/packages/crates-graphics.scm> > @@ -2415,8 +2415,11 @@ the wayland protocol, client side.")> > (arguments> > - `(#:skip-build? #t> > - #:cargo-inputs> > + `(#:cargo-inputs> > (("rust-nix" ,rust-nix-0.18)> > ("rust-once-cell" ,rust-once-cell-1)> > - ("rust-smallvec" ,rust-smallvec-1)> > - ("rust-wayland-sys" ,rust-wayland-sys-0.28))))> > + ("rust-smallvec" ,rust-smallvec-1))))> > + (inputs> > + `(("rust-nix" ,rust-nix-0.18)> > + ("rust-once-cell" ,rust-once-cell-1)> > + ("rust-smallvec" ,rust-smallvec-1)> > + ("rust-wayland-sys" ,rust-wayland-sys-0.28)))> > (home-page "https://github.com/smithay/wayland-rs")> > and do not understand why rust-wayland-sys is only part of inputs but> rust-once-cell is also among the cargo-inputs. Also the inputs are> empty for older versions of rust-wayland-commons.
#:cargo-inputs are "magic" in that they propagate their own #:cargo-inputsto future builds. The changes to the cargo-build-system mimic this so wecan use regular inputs and not #:cargo-inputs. rust-wayland-sys-0.28 hasenough of its dependencies converted that it can be used as just aninput but rust-once-cell-1 hasn't yet so it still needs to be in#:cargo-inputs. As far as the other versions of rust-wayland-commons, Istarted by focusing on alacritty so we could remove some of the buildphases and move them to the crates which actually needed patching.
Toggle quote (5 lines)> Could you clarify what is written in the doc/guix.texi?> > Regards,> Florian
I think for now the best option is to add the rust inputs of all typesto both inputs and #:cargo-inputs (or native-inputs and#:cargo-development-inputs) and then remove them from #:cargo-inputswhen it's possible to do so. Ultimately Someone™ should revive thewip-rust branch and convert all the crates over in one go and adjust theimporter and manual.
-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmBz604ACgkQQarn3Mo9g1EE2BAAo853e56xlnVuZG/r1qvzIBBZqZS/9EYrD17pRH+cUgQPYsLaYrNzy04r3j8XPn0t1bOWSiVh08s4OY6kcvvFgNPy8QnAQz319ocvgthh0bXyEKsvvPCFCrQgNp+NSr48r8ZVu6blZhWL+fZ3SAWLB69Yyb6gQZy5P/0EsnulJdNfSgOEB4rfmzh8OMr1a5B6cv3GnJLS7x7K3QSGWxSxMDoHTX8oj0Ze8PQLrH8J8n9joujte+xOhQGxWgDVm09Xrk/1iXXjvCIzbPQ3JIvDZ+a92sU91IYkwV6S9G2D7O/vJGIN9IKExfJPBfZgoWH2mk2U0HKunhLY9z9Ot6JgUi6WuX7tXQXE5cNN4kktLqTA3f7fEUSwyK2LQ8O8WsnSqmkUuECyVY/tQTDGHDdl92dgqldol3cCKJ4KJPFYR21p9H/SZsYgk76CuEbGZW+Q1rYD+Fc9hVTlzX6Y766suuGr4CuhTW4rHYUOo/Pa1nPfHFjsz2ZIx0tDG7jRSXiJtiTPsYCTbTS/DnHaWKLxAXlO+ZukUXPeqmWWAs7bVBbhFf8YFHK6FHHdunWeinAJiUCSlYFTIOqx/yHFry5ozab0GaF/1nGDBCbskyJGhSIoHUuoVgKg5bieO2keEzAgI+KjUlSlaRBF112GEmYcJeLGJUGxoDpzxW7B4p26JHo==zcEO-----END PGP SIGNATURE-----

P
P
pelzflorian (Florian Pelz) wrote on 13 Apr 14:31 +0200
(name . Efraim Flashner)(address . efraim@flashner.co.il)(address . 46399@debbugs.gnu.org)
20210413123139.t2s5jhskvep4syhk@pelzflorian.localdomain
On Mon, Apr 12, 2021 at 09:40:17AM +0300, Efraim Flashner wrote:
Toggle quote (3 lines)> I went back and refreshed myself with what I wrote there. source-only> crates should be added to both regular inputs and to #:cargo-inputs.
Thank you for clarifying. I will translate the manual accordingly.
Toggle quote (3 lines)> #:cargo-inputs are "magic" in that they propagate their own #:cargo-inputs> to future builds.
I had not known about this difference.
Toggle quote (7 lines)> I think for now the best option is to add the rust inputs of all types> to both inputs and #:cargo-inputs (or native-inputs and> #:cargo-development-inputs) and then remove them from #:cargo-inputs> when it's possible to do so. Ultimately Someone™ should revive the> wip-rust branch and convert all the crates over in one go and adjust the> importer and manual.
I suppose this paragraph of your reply should be added to the manual.
Regards,Florian
L
L
Leo Famulari wrote on 13 Apr 19:49 +0200
(name . pelzflorian (Florian Pelz))(address . pelzflorian@pelzflorian.de)
YHXZpaThKVksayl3@jasmine.lan
On Tue, Apr 13, 2021 at 02:31:39PM +0200, pelzflorian (Florian Pelz) wrote:
Toggle quote (6 lines)> On Mon, Apr 12, 2021 at 09:40:17AM +0300, Efraim Flashner wrote:> > I went back and refreshed myself with what I wrote there. source-only> > crates should be added to both regular inputs and to #:cargo-inputs.> > Thank you for clarifying. I will translate the manual accordingly.
We are in "string freeze" for the upcoming release, so please delay yourchanges to the manual until the release is completed.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 46399@debbugs.gnu.org