[PATCH] guix: Compress and decompress xz archives in parallel.

DoneSubmitted by Efraim Flashner.
Details
6 participants
  • Efraim Flashner
  • Kei Kebreau
  • Leo Famulari
  • Ludovic Courtès
  • Marius Bakke
  • Glenn Morris
Owner
unassigned
Severity
normal
E
E
Efraim Flashner wrote on 23 Mar 2017 10:47
E1cr0d6-0004an-8g@eggs.gnu.org
* guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".* guix/utils.scm (decompressed-port, compressed-port,compressed-output-port): Same.--- guix/scripts/pack.scm | 3 ++- guix/utils.scm | 7 ++++--- 2 files changed, 6 insertions(+), 4 deletions(-)
Toggle diff (61 lines)diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scmindex 626c592e1..9e91bc22a 100644--- a/guix/scripts/pack.scm+++ b/guix/scripts/pack.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il> ;;; ;;; This file is part of GNU Guix. ;;;@@ -59,7 +60,7 @@ (compressor "lzip" "lz" #~(#+(file-append lzip "/bin/lzip") "-9")) (compressor "xz" "xz"- #~(#+(file-append xz "/bin/xz") "-e"))+ #~(#+(file-append xz "/bin/xz") "-e -T0")) (compressor "bzip2" "bz2" #~(#+(file-append bzip2 "/bin/bzip2") "-9")))) diff --git a/guix/utils.scm b/guix/utils.scmindex bc90686de..fb962df8b 100644--- a/guix/utils.scm+++ b/guix/utils.scm@@ -5,6 +5,7 @@ ;;; Copyright © 2014 Ian Denhardt <ian@zenhack.net> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org> ;;; Copyright © 2015 David Thompson <davet@gnu.org>+;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il> ;;; ;;; This file is part of GNU Guix. ;;;@@ -154,7 +155,7 @@ a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-dc") input))- ('xz (filtered-port `(,%xz "-dc") input))+ ('xz (filtered-port `(,%xz "-dc -T0") input)) ('gzip (filtered-port `(,%gzip "-dc") input)) (else (error "unsupported compression scheme" compression)))) @@ -164,7 +165,7 @@ a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-c") input))- ('xz (filtered-port `(,%xz "-c") input))+ ('xz (filtered-port `(,%xz "-c -T0") input)) ('gzip (filtered-port `(,%gzip "-c") input)) (else (error "unsupported compression scheme" compression)))) @@ -221,7 +222,7 @@ program--e.g., '(\"--fast\")." (match compression ((or #f 'none) (values output '())) ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output))- ('xz (filtered-output-port `(,%xz "-c" ,@options) output))+ ('xz (filtered-output-port `(,%xz "-c -T0" ,@options) output)) ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output)) (else (error "unsupported compression scheme" compression)))) -- 2.12.1
G
G
Glenn Morris wrote on 24 Mar 2017 20:33
control message for bug 26225
(address . control@debbugs.gnu.org)
E1crUxR-00085h-5y@fencepost.gnu.org
reassign 26225 guix-patches
G
G
Glenn Morris wrote on 24 Mar 2017 20:36
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
(address . 26225@debbugs.gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
5vvaqyii1c.fsf@fencepost.gnu.org
Reassigned to guix-patches.(You must include a Package: line if reporting by bcc(?) ).
Efraim Flashner wrote:
Toggle quote (66 lines)> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".> * guix/utils.scm (decompressed-port, compressed-port,> compressed-output-port): Same.> ---> guix/scripts/pack.scm | 3 ++-> guix/utils.scm | 7 ++++---> 2 files changed, 6 insertions(+), 4 deletions(-)>> diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm> index 626c592e1..9e91bc22a 100644> --- a/guix/scripts/pack.scm> +++ b/guix/scripts/pack.scm> @@ -1,5 +1,6 @@> ;;; GNU Guix --- Functional package management for GNU> ;;; Copyright © 2015, 2017 Ludovic Courtès <ludo@gnu.org>> +;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>> ;;;> ;;; This file is part of GNU Guix.> ;;;> @@ -59,7 +60,7 @@> (compressor "lzip" "lz"> #~(#+(file-append lzip "/bin/lzip") "-9"))> (compressor "xz" "xz"> - #~(#+(file-append xz "/bin/xz") "-e"))> + #~(#+(file-append xz "/bin/xz") "-e -T0"))> (compressor "bzip2" "bz2"> #~(#+(file-append bzip2 "/bin/bzip2") "-9"))))> > diff --git a/guix/utils.scm b/guix/utils.scm> index bc90686de..fb962df8b 100644> --- a/guix/utils.scm> +++ b/guix/utils.scm> @@ -5,6 +5,7 @@> ;;; Copyright © 2014 Ian Denhardt <ian@zenhack.net>> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>> ;;; Copyright © 2015 David Thompson <davet@gnu.org>> +;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>> ;;;> ;;; This file is part of GNU Guix.> ;;;> @@ -154,7 +155,7 @@ a symbol such as 'xz."> (match compression> ((or #f 'none) (values input '()))> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))> - ('xz (filtered-port `(,%xz "-dc") input))> + ('xz (filtered-port `(,%xz "-dc -T0") input))> ('gzip (filtered-port `(,%gzip "-dc") input))> (else (error "unsupported compression scheme" compression))))> > @@ -164,7 +165,7 @@ a symbol such as 'xz."> (match compression> ((or #f 'none) (values input '()))> ('bzip2 (filtered-port `(,%bzip2 "-c") input))> - ('xz (filtered-port `(,%xz "-c") input))> + ('xz (filtered-port `(,%xz "-c -T0") input))> ('gzip (filtered-port `(,%gzip "-c") input))> (else (error "unsupported compression scheme" compression))))> > @@ -221,7 +222,7 @@ program--e.g., '(\"--fast\")."> (match compression> ((or #f 'none) (values output '()))> ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output))> - ('xz (filtered-output-port `(,%xz "-c" ,@options) output))> + ('xz (filtered-output-port `(,%xz "-c -T0" ,@options) output))> ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output))> (else (error "unsupported compression scheme" compression))))
L
L
Leo Famulari wrote on 31 Mar 2017 00:22
20170330222235.GB10357@jasmine
Efraim Flashner wrote:
Toggle quote (15 lines)> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".> * guix/utils.scm (decompressed-port, compressed-port,> compressed-output-port): Same.
> --- a/guix/utils.scm> +++ b/guix/utils.scm> @@ -154,7 +155,7 @@ a symbol such as 'xz."> (match compression> ((or #f 'none) (values input '()))> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))> - ('xz (filtered-port `(,%xz "-dc") input))> + ('xz (filtered-port `(,%xz "-dc -T0") input))> ('gzip (filtered-port `(,%gzip "-dc") input))> (else (error "unsupported compression scheme" compression))))
Parallel decompression isn't implemented yet, but the extra '-T0' optiondoesn't seem to break anything.
The changes LGTM.
I sent a companion patch to https://bugs.gnu.org/26316 that enablesthreaded compression when building source tarballs.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAljdhSsACgkQJkb6MLrKfwhkRBAAz6Rd76JFEETSgiWnlatIAoOraOBSk8jXNNgMixc4jCtYaQEIS3M7cOA/QRz/VTlDZgONfhSxEPOp0vHAMrxhwdGSMIW9vqstb+mnAxzS4zIHW/nxrUQ5mkLiIgzxQqQcT3+9tPxjqcEYC74qRo2rHFGbY3KM9Ksb+G5C7LsbC2wGAG+pik3a7OtHmvOa2GVAE09vNldeW3P8spt595oDqye+Er3jjY0BZEHkpxK6ZG1qVHL0SP9QTkeTpkEFIJSjX9ofeLt1atrd+b+oQ5/37K7+WxYzyJEskHLjl5ZziknyQk8K9nwzDz+AbWWn1qW6BhddnXMY/7eRFACNJCU99w6ng03K6YlLumZxizpkLN+ISGy/7rVjcWC+Yr9lUguLRoO0yXo0SXNBjFvedxHUKMarqcbwKLMCf9jvHOwio5E/jQifvLJvJXeibHyhmd3bkJm7hY0YWYfOGkLh/VjbDmfHShzLaDcb6dIsTsaFTfpTa/ll+5rqREpXBpkhdc+LXz/J6cUAU1+Ry+hR7Lu2XlnSKcYWnjPIagyttg1BqxHPXyGji7Z3Fmrnpwp2wxSagRdVGOOCbA0SxdPZ/TbG92yAMjhU8y1TyFzPBgMJQqUrnvOynpgZlHFp820uI1rWMFcqoZrkvPCEyWPdX4S+YGDslqv61Xwdw45DTf7ihGQ==ViJO-----END PGP SIGNATURE-----

K
K
Kei Kebreau wrote on 31 Mar 2017 19:56
(name . Leo Famulari)(address . leo@famulari.name)
87h9295nze.fsf@openmailbox.org
Leo Famulari <leo@famulari.name> writes:
Toggle quote (24 lines)> Efraim Flashner wrote:>> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".>> * guix/utils.scm (decompressed-port, compressed-port,>> compressed-output-port): Same.>>> --- a/guix/utils.scm>> +++ b/guix/utils.scm>> @@ -154,7 +155,7 @@ a symbol such as 'xz.">> (match compression>> ((or #f 'none) (values input '()))>> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))>> - ('xz (filtered-port `(,%xz "-dc") input))>> + ('xz (filtered-port `(,%xz "-dc -T0") input))>> ('gzip (filtered-port `(,%gzip "-dc") input))>> (else (error "unsupported compression scheme" compression))))>> Parallel decompression isn't implemented yet, but the extra '-T0' option> doesn't seem to break anything.>> The changes LGTM.>> I sent a companion patch to <https://bugs.gnu.org/26316> that enables> threaded compression when building source tarballs.
LGTM as well.
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEg7ZwOtzKO2lLzi2m5qXuPBlGeg0FAljemFUACgkQ5qXuPBlGeg15DQ//WCd4qGdVJRgi+mo8Z2Oq9XF1xR7P4ud4J5kRGFqZbWzOysFldBC6a4ANI9f2uLX0lh1+uj+WMUSyZxHO4pwiB1PC2K9EK4Clx5WwZLC9gqxueWhIQ41XI6RqqgHHlNxfZhaTeI6qglpLtxdZUZqF1RDrjvBZXSVd4oYty/EhRvaJbKYhkUN+CaeVPVwcg2Wh5K7SeYC1IuMBTp6fppue/QmfLUhgr9F6IzYxXN9UoSImxXz1whQlj1f4/dV4TzbGGPIbczWpVuTE1jxa20yFoOm3YnkkAU9F8111fC3G8UPXjWWT9mQPYf2/Xrup/5dyX5FtORU1492Y0imZWTgvAoIoTDkjpe68r62ztzuuR+pv6pZwKdnXZ7DojHow9nF6bINquK7x/FYcRO2buIAQrj18KgohrsMa4/WbCKNqr7oUz8MoLwpFOH/SSVTfDZlP52ay8koSnhe2e+TXH6UyIkJ9MPejkPiBxS1s4yP90BnF9w9eexXLYDtyeimR9uq/E+e11My58eFCT8mIj+hNs4r6DQEwpBGXSPtyBSWwFSuCXdxIBY7OxYJ8iQHcALnelvdCv99Hk+k0mEVCMyT4ChHrWtkajIZpsUxkEUOPApUSzdEp7jmAU/BYOqPOtehOZsR2tVY629VNsBFWD5YCedzh7CbEfx4B0uSg2dyrimQ==iwIp-----END PGP SIGNATURE-----
L
L
Leo Famulari wrote on 1 Apr 2017 01:01
(name . Kei Kebreau)(address . kei@openmailbox.org)
20170331230158.GA9642@jasmine
On Fri, Mar 31, 2017 at 01:56:37PM -0400, Kei Kebreau wrote:
Toggle quote (28 lines)> Leo Famulari <leo@famulari.name> writes:> > > Efraim Flashner wrote:> >> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".> >> * guix/utils.scm (decompressed-port, compressed-port,> >> compressed-output-port): Same.> >> >> --- a/guix/utils.scm> >> +++ b/guix/utils.scm> >> @@ -154,7 +155,7 @@ a symbol such as 'xz."> >> (match compression> >> ((or #f 'none) (values input '()))> >> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))> >> - ('xz (filtered-port `(,%xz "-dc") input))> >> + ('xz (filtered-port `(,%xz "-dc -T0") input))> >> ('gzip (filtered-port `(,%gzip "-dc") input))> >> (else (error "unsupported compression scheme" compression))))> >> > Parallel decompression isn't implemented yet, but the extra '-T0' option> > doesn't seem to break anything.> >> > The changes LGTM.> >> > I sent a companion patch to <https://bugs.gnu.org/26316> that enables> > threaded compression when building source tarballs.> > LGTM as well.
AFAICT, this doesn't cause mass rebuilds.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlje3+AACgkQJkb6MLrKfwif8g/+Nxyq4Jp56kYniociZcomjXKfPz2tCvd06pNtQObyuZlIQqNCU66bQIoDA09A3aQPbnc88eTqv3bE1a6KJH7h9suPzvls2/j5ddah5MHDZABh+YT3yCNQW7noxseINtlkbYxGXo5aCcJ6cEjLmzF60FmgN93/wpNJOXXMXAtyF4IWccgB/lovRQNeZvm6RLxY95Xfj5kiw0SOsNTRnxOtwY2oRvNrcKYKSeaBoaSPOaD8RQNS0Y1tBH8EdSvb52NenO8LqqQScY3QDvj2LpoLs6KsGUE7YKGJNQj4lU/HAJWT9yxyCjwHO0krZEAmYfX7CeO4y54lo9h3uuqArJIhq0/glotqxqikWmWKpRvK9clpthPRD07bWz5u0cNmbJw7+a/ykqDo6PmPtWItN+wn0DvFufIUBJCJhd0jSat9VfUVeJoTCqfP1par5k5s4P46BFZS9npPDQPBcVqVaEI8HX+VNAGpfYByE9Q4/mieKGVFFMsgD9E7EiMTojtjYPcSBnQgogY6qjZKSQhBj4Q3GdQVUUKjPFTZhyncS/mDaPbTnoXfUTx3KzmA+kSEjKEXZkTwr7pvTkBWydvSToVOq36ovX997rlbFL6lDh2ONBY7BLrqtNicQ0Jc2qTXtftNLOlUqwCmbiFyFXiIZu4zUSFn71dp0EQbi3/anyDSruU==oLYK-----END PGP SIGNATURE-----

M
M
Marius Bakke wrote on 1 Apr 2017 08:52
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
878tnkfwlv.fsf@kirby.i-did-not-set--mail-host-address--so-tickle-me
Leo Famulari <leo@famulari.name> writes:
Toggle quote (24 lines)> Efraim Flashner wrote:>> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".>> * guix/utils.scm (decompressed-port, compressed-port,>> compressed-output-port): Same.>>> --- a/guix/utils.scm>> +++ b/guix/utils.scm>> @@ -154,7 +155,7 @@ a symbol such as 'xz.">> (match compression>> ((or #f 'none) (values input '()))>> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))>> - ('xz (filtered-port `(,%xz "-dc") input))>> + ('xz (filtered-port `(,%xz "-dc -T0") input))>> ('gzip (filtered-port `(,%gzip "-dc") input))>> (else (error "unsupported compression scheme" compression))))>> Parallel decompression isn't implemented yet, but the extra '-T0' option> doesn't seem to break anything.>> The changes LGTM.>> I sent a companion patch to <https://bugs.gnu.org/26316> that enables> threaded compression when building source tarballs.
Is (parallel-job-count) accessible here? It would be nice to respect it.
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAljfTiwACgkQoqBt8qM6VPrBWQgApc4bdfvSCVtABSMaB+kRw9WSK5S/jcuYEIocUNxnqJJeQo+MF8nYNBPii0g1NlWrXr8nonRO0iuMIZLbi74iy4sv4ghGjRXIO4SZfWzsc8ESh+rZH9j2Wnq0AEDn7oo5klBgWlHnHzsV5DabwQm/6fYFC0ozoBuZw6Xxhnxdg+FMUT8+7DhwexZGtzfjOD6eoWGVA11N8693EW8KWEO+ViDZNteN5dmCtb7TN7h/1hw4jY3vOn78VLVnp6ZuH5qxJYDc0e/jcR+PACdX4DZE3ucmt1YS6A0oj38vHPrmjbuRRY4g3+FVmVRkwtgfbuGFw4Gv4kS/lmG3Uj3HJBnFeg===nXr1-----END PGP SIGNATURE-----
L
L
Leo Famulari wrote on 2 Apr 2017 23:21
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
(name . Marius Bakke)(address . mbakke@fastmail.com)
20170402212118.GA28696@jasmine
On Sat, Apr 01, 2017 at 08:52:28AM +0200, Marius Bakke wrote:
Toggle quote (18 lines)> > Efraim Flashner wrote:> >> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".> >> * guix/utils.scm (decompressed-port, compressed-port,> >> compressed-output-port): Same.> >> >> --- a/guix/utils.scm> >> +++ b/guix/utils.scm> >> @@ -154,7 +155,7 @@ a symbol such as 'xz."> >> (match compression> >> ((or #f 'none) (values input '()))> >> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))> >> - ('xz (filtered-port `(,%xz "-dc") input))> >> + ('xz (filtered-port `(,%xz "-dc -T0") input))> >> ('gzip (filtered-port `(,%gzip "-dc") input))> >> (else (error "unsupported compression scheme" compression))))> > Is (parallel-job-count) accessible here? It would be nice to respect it.
I agree, it would be good if we respected it here, if possible.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAljha04ACgkQJkb6MLrKfwjmyhAA4Kh8awEp7TBs8a83qSg1wzjOhMtmpZ3ocYpi70KWtQ9ZiyOTrWZMiy36EvfbJweY+wUUqcSxbytP3nGRjRyxXSFyMw7KmTMvmGTlp2vLrw8Jn8s1mrrDvUoUcWZC5Y8lT5LUTo/vYikk0qjppBjiLtwIe+voHGqKD+2rs9pySP5iHiT5rduRpWXOR0UkLfsQkoDjkmzRbxNbwIi1TtNcUxjwhRcUEdnX4xQE1G9M4zdFIWarfos6nffDfTfOf/7ClPmGB9eZi2vfPZgOJv8GM5t/u4yJVY/goKSqim6hnzmRpMzBKpqHbYu1YVacG2ulnCpVbDTlN+yMeasVmBGW2c2Pzpz+Ecr1GKjGWq9D8SIigBsnK6GI9N9O+f2DOmA3aWoKGnSui2InYVVAcGPlK435+Caycn4OSYZMkcOzn7E0rTowCBoYsNy7n+8piwJ1UWb1qhImuxKNnJqFYTmef8DgMBbhS4PJXw/ZKYRm4qmk+u6uq9UPYOBgFkoSuISww/9qDBQ8Y/Gqz1SHu6CEIPYJzA8pDJyPWrbtPqv46fKoIS4rlnZREGFbYGJ2XSiirr5tMi7ukQCoJogL8jG2mpbGrxzw148QtSVAleJPte4BDb2f50XVja251pjvLXSt7nuHJ9yJRnwwREIBssUXu3/9ZWvb9/YG5KTzo7tbEB0==JfVK-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 7 Apr 2017 22:59
control message for bug #26225
(address . control@debbugs.gnu.org)
87o9w8szlf.fsf@gnu.org
tags 26225 fixedclose 26225
?
Your comment

This issue is archived.

To comment on this conversation send email to 26225@debbugs.gnu.org