[PATCH] guix: Compress and decompress xz archives in parallel.

  • Done
  • quality assurance status badge
Details
6 participants
  • Efraim Flashner
  • Kei Kebreau
  • Leo Famulari
  • Ludovic Courtès
  • Marius Bakke
  • Glenn Morris
Owner
unassigned
Submitted by
Efraim Flashner
Severity
normal
E
E
Efraim Flashner wrote on 23 Mar 2017 10:47
E1cr0d6-0004an-8g@eggs.gnu.org
* guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
* guix/utils.scm (decompressed-port, compressed-port,
compressed-output-port): Same.
---
guix/scripts/pack.scm | 3 ++-
guix/utils.scm | 7 ++++---
2 files changed, 6 insertions(+), 4 deletions(-)

Toggle diff (61 lines)
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 626c592e1..9e91bc22a 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2015, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -59,7 +60,7 @@
(compressor "lzip" "lz"
#~(#+(file-append lzip "/bin/lzip") "-9"))
(compressor "xz" "xz"
- #~(#+(file-append xz "/bin/xz") "-e"))
+ #~(#+(file-append xz "/bin/xz") "-e -T0"))
(compressor "bzip2" "bz2"
#~(#+(file-append bzip2 "/bin/bzip2") "-9"))))
diff --git a/guix/utils.scm b/guix/utils.scm
index bc90686de..fb962df8b 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2014 Ian Denhardt <ian@zenhack.net>
;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
;;; Copyright © 2015 David Thompson <davet@gnu.org>
+;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -154,7 +155,7 @@ a symbol such as 'xz."
(match compression
((or #f 'none) (values input '()))
('bzip2 (filtered-port `(,%bzip2 "-dc") input))
- ('xz (filtered-port `(,%xz "-dc") input))
+ ('xz (filtered-port `(,%xz "-dc -T0") input))
('gzip (filtered-port `(,%gzip "-dc") input))
(else (error "unsupported compression scheme" compression))))
@@ -164,7 +165,7 @@ a symbol such as 'xz."
(match compression
((or #f 'none) (values input '()))
('bzip2 (filtered-port `(,%bzip2 "-c") input))
- ('xz (filtered-port `(,%xz "-c") input))
+ ('xz (filtered-port `(,%xz "-c -T0") input))
('gzip (filtered-port `(,%gzip "-c") input))
(else (error "unsupported compression scheme" compression))))
@@ -221,7 +222,7 @@ program--e.g., '(\"--fast\")."
(match compression
((or #f 'none) (values output '()))
('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output))
- ('xz (filtered-output-port `(,%xz "-c" ,@options) output))
+ ('xz (filtered-output-port `(,%xz "-c -T0" ,@options) output))
('gzip (filtered-output-port `(,%gzip "-c" ,@options) output))
(else (error "unsupported compression scheme" compression))))
--
2.12.1
G
G
Glenn Morris wrote on 24 Mar 2017 20:33
control message for bug 26225
(address . control@debbugs.gnu.org)
E1crUxR-00085h-5y@fencepost.gnu.org
reassign 26225 guix-patches
G
G
Glenn Morris wrote on 24 Mar 2017 20:36
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
(address . 26225@debbugs.gnu.org)(name . Efraim Flashner)(address . efraim@flashner.co.il)
5vvaqyii1c.fsf@fencepost.gnu.org
Reassigned to guix-patches.
(You must include a Package: line if reporting by bcc(?) ).

Efraim Flashner wrote:

Toggle quote (66 lines)
> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> * guix/utils.scm (decompressed-port, compressed-port,
> compressed-output-port): Same.
> ---
> guix/scripts/pack.scm | 3 ++-
> guix/utils.scm | 7 ++++---
> 2 files changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
> index 626c592e1..9e91bc22a 100644
> --- a/guix/scripts/pack.scm
> +++ b/guix/scripts/pack.scm
> @@ -1,5 +1,6 @@
> ;;; GNU Guix --- Functional package management for GNU
> ;;; Copyright © 2015, 2017 Ludovic Courtès <ludo@gnu.org>
> +;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -59,7 +60,7 @@
> (compressor "lzip" "lz"
> #~(#+(file-append lzip "/bin/lzip") "-9"))
> (compressor "xz" "xz"
> - #~(#+(file-append xz "/bin/xz") "-e"))
> + #~(#+(file-append xz "/bin/xz") "-e -T0"))
> (compressor "bzip2" "bz2"
> #~(#+(file-append bzip2 "/bin/bzip2") "-9"))))
>
> diff --git a/guix/utils.scm b/guix/utils.scm
> index bc90686de..fb962df8b 100644
> --- a/guix/utils.scm
> +++ b/guix/utils.scm
> @@ -5,6 +5,7 @@
> ;;; Copyright © 2014 Ian Denhardt <ian@zenhack.net>
> ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
> ;;; Copyright © 2015 David Thompson <davet@gnu.org>
> +;;; Copyright © 2017 Efraim Flashner <efraim@flashner.co.il>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> (match compression
> ((or #f 'none) (values input '()))
> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
> - ('xz (filtered-port `(,%xz "-dc") input))
> + ('xz (filtered-port `(,%xz "-dc -T0") input))
> ('gzip (filtered-port `(,%gzip "-dc") input))
> (else (error "unsupported compression scheme" compression))))
>
> @@ -164,7 +165,7 @@ a symbol such as 'xz."
> (match compression
> ((or #f 'none) (values input '()))
> ('bzip2 (filtered-port `(,%bzip2 "-c") input))
> - ('xz (filtered-port `(,%xz "-c") input))
> + ('xz (filtered-port `(,%xz "-c -T0") input))
> ('gzip (filtered-port `(,%gzip "-c") input))
> (else (error "unsupported compression scheme" compression))))
>
> @@ -221,7 +222,7 @@ program--e.g., '(\"--fast\")."
> (match compression
> ((or #f 'none) (values output '()))
> ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output))
> - ('xz (filtered-output-port `(,%xz "-c" ,@options) output))
> + ('xz (filtered-output-port `(,%xz "-c -T0" ,@options) output))
> ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output))
> (else (error "unsupported compression scheme" compression))))
L
L
Leo Famulari wrote on 31 Mar 2017 00:22
20170330222235.GB10357@jasmine
Efraim Flashner wrote:
Toggle quote (15 lines)
> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> * guix/utils.scm (decompressed-port, compressed-port,
> compressed-output-port): Same.

> --- a/guix/utils.scm
> +++ b/guix/utils.scm
> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> (match compression
> ((or #f 'none) (values input '()))
> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
> - ('xz (filtered-port `(,%xz "-dc") input))
> + ('xz (filtered-port `(,%xz "-dc -T0") input))
> ('gzip (filtered-port `(,%gzip "-dc") input))
> (else (error "unsupported compression scheme" compression))))

Parallel decompression isn't implemented yet, but the extra '-T0' option
doesn't seem to break anything.

The changes LGTM.

I sent a companion patch to https://bugs.gnu.org/26316 that enables
threaded compression when building source tarballs.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAljdhSsACgkQJkb6MLrK
fwhkRBAAz6Rd76JFEETSgiWnlatIAoOraOBSk8jXNNgMixc4jCtYaQEIS3M7cOA/
QRz/VTlDZgONfhSxEPOp0vHAMrxhwdGSMIW9vqstb+mnAxzS4zIHW/nxrUQ5mkLi
IgzxQqQcT3+9tPxjqcEYC74qRo2rHFGbY3KM9Ksb+G5C7LsbC2wGAG+pik3a7OtH
mvOa2GVAE09vNldeW3P8spt595oDqye+Er3jjY0BZEHkpxK6ZG1qVHL0SP9QTkeT
pkEFIJSjX9ofeLt1atrd+b+oQ5/37K7+WxYzyJEskHLjl5ZziknyQk8K9nwzDz+A
bWWn1qW6BhddnXMY/7eRFACNJCU99w6ng03K6YlLumZxizpkLN+ISGy/7rVjcWC+
Yr9lUguLRoO0yXo0SXNBjFvedxHUKMarqcbwKLMCf9jvHOwio5E/jQifvLJvJXei
bHyhmd3bkJm7hY0YWYfOGkLh/VjbDmfHShzLaDcb6dIsTsaFTfpTa/ll+5rqREpX
Bpkhdc+LXz/J6cUAU1+Ry+hR7Lu2XlnSKcYWnjPIagyttg1BqxHPXyGji7Z3Fmrn
pwp2wxSagRdVGOOCbA0SxdPZ/TbG92yAMjhU8y1TyFzPBgMJQqUrnvOynpgZlHFp
820uI1rWMFcqoZrkvPCEyWPdX4S+YGDslqv61Xwdw45DTf7ihGQ=
=ViJO
-----END PGP SIGNATURE-----


K
K
Kei Kebreau wrote on 31 Mar 2017 19:56
(name . Leo Famulari)(address . leo@famulari.name)
87h9295nze.fsf@openmailbox.org
Leo Famulari <leo@famulari.name> writes:

Toggle quote (24 lines)
> Efraim Flashner wrote:
>> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
>> * guix/utils.scm (decompressed-port, compressed-port,
>> compressed-output-port): Same.
>
>> --- a/guix/utils.scm
>> +++ b/guix/utils.scm
>> @@ -154,7 +155,7 @@ a symbol such as 'xz."
>> (match compression
>> ((or #f 'none) (values input '()))
>> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
>> - ('xz (filtered-port `(,%xz "-dc") input))
>> + ('xz (filtered-port `(,%xz "-dc -T0") input))
>> ('gzip (filtered-port `(,%gzip "-dc") input))
>> (else (error "unsupported compression scheme" compression))))
>
> Parallel decompression isn't implemented yet, but the extra '-T0' option
> doesn't seem to break anything.
>
> The changes LGTM.
>
> I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> threaded compression when building source tarballs.

LGTM as well.
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEg7ZwOtzKO2lLzi2m5qXuPBlGeg0FAljemFUACgkQ5qXuPBlG
eg15DQ//WCd4qGdVJRgi+mo8Z2Oq9XF1xR7P4ud4J5kRGFqZbWzOysFldBC6a4AN
I9f2uLX0lh1+uj+WMUSyZxHO4pwiB1PC2K9EK4Clx5WwZLC9gqxueWhIQ41XI6Rq
qgHHlNxfZhaTeI6qglpLtxdZUZqF1RDrjvBZXSVd4oYty/EhRvaJbKYhkUN+CaeV
PVwcg2Wh5K7SeYC1IuMBTp6fppue/QmfLUhgr9F6IzYxXN9UoSImxXz1whQlj1f4
/dV4TzbGGPIbczWpVuTE1jxa20yFoOm3YnkkAU9F8111fC3G8UPXjWWT9mQPYf2/
Xrup/5dyX5FtORU1492Y0imZWTgvAoIoTDkjpe68r62ztzuuR+pv6pZwKdnXZ7Do
jHow9nF6bINquK7x/FYcRO2buIAQrj18KgohrsMa4/WbCKNqr7oUz8MoLwpFOH/S
SVTfDZlP52ay8koSnhe2e+TXH6UyIkJ9MPejkPiBxS1s4yP90BnF9w9eexXLYDty
eimR9uq/E+e11My58eFCT8mIj+hNs4r6DQEwpBGXSPtyBSWwFSuCXdxIBY7OxYJ8
iQHcALnelvdCv99Hk+k0mEVCMyT4ChHrWtkajIZpsUxkEUOPApUSzdEp7jmAU/BY
OqPOtehOZsR2tVY629VNsBFWD5YCedzh7CbEfx4B0uSg2dyrimQ=
=iwIp
-----END PGP SIGNATURE-----

L
L
Leo Famulari wrote on 1 Apr 2017 01:01
(name . Kei Kebreau)(address . kei@openmailbox.org)
20170331230158.GA9642@jasmine
On Fri, Mar 31, 2017 at 01:56:37PM -0400, Kei Kebreau wrote:
Toggle quote (28 lines)
> Leo Famulari <leo@famulari.name> writes:
>
> > Efraim Flashner wrote:
> >> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> >> * guix/utils.scm (decompressed-port, compressed-port,
> >> compressed-output-port): Same.
> >
> >> --- a/guix/utils.scm
> >> +++ b/guix/utils.scm
> >> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> >> (match compression
> >> ((or #f 'none) (values input '()))
> >> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
> >> - ('xz (filtered-port `(,%xz "-dc") input))
> >> + ('xz (filtered-port `(,%xz "-dc -T0") input))
> >> ('gzip (filtered-port `(,%gzip "-dc") input))
> >> (else (error "unsupported compression scheme" compression))))
> >
> > Parallel decompression isn't implemented yet, but the extra '-T0' option
> > doesn't seem to break anything.
> >
> > The changes LGTM.
> >
> > I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> > threaded compression when building source tarballs.
>
> LGTM as well.

AFAICT, this doesn't cause mass rebuilds.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlje3+AACgkQJkb6MLrK
fwif8g/+Nxyq4Jp56kYniociZcomjXKfPz2tCvd06pNtQObyuZlIQqNCU66bQIoD
A09A3aQPbnc88eTqv3bE1a6KJH7h9suPzvls2/j5ddah5MHDZABh+YT3yCNQW7no
xseINtlkbYxGXo5aCcJ6cEjLmzF60FmgN93/wpNJOXXMXAtyF4IWccgB/lovRQNe
Zvm6RLxY95Xfj5kiw0SOsNTRnxOtwY2oRvNrcKYKSeaBoaSPOaD8RQNS0Y1tBH8E
dSvb52NenO8LqqQScY3QDvj2LpoLs6KsGUE7YKGJNQj4lU/HAJWT9yxyCjwHO0kr
ZEAmYfX7CeO4y54lo9h3uuqArJIhq0/glotqxqikWmWKpRvK9clpthPRD07bWz5u
0cNmbJw7+a/ykqDo6PmPtWItN+wn0DvFufIUBJCJhd0jSat9VfUVeJoTCqfP1par
5k5s4P46BFZS9npPDQPBcVqVaEI8HX+VNAGpfYByE9Q4/mieKGVFFMsgD9E7EiMT
ojtjYPcSBnQgogY6qjZKSQhBj4Q3GdQVUUKjPFTZhyncS/mDaPbTnoXfUTx3KzmA
+kSEjKEXZkTwr7pvTkBWydvSToVOq36ovX997rlbFL6lDh2ONBY7BLrqtNicQ0Jc
2qTXtftNLOlUqwCmbiFyFXiIZu4zUSFn71dp0EQbi3/anyDSruU=
=oLYK
-----END PGP SIGNATURE-----


M
M
Marius Bakke wrote on 1 Apr 2017 08:52
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
878tnkfwlv.fsf@kirby.i-did-not-set--mail-host-address--so-tickle-me
Leo Famulari <leo@famulari.name> writes:

Toggle quote (24 lines)
> Efraim Flashner wrote:
>> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
>> * guix/utils.scm (decompressed-port, compressed-port,
>> compressed-output-port): Same.
>
>> --- a/guix/utils.scm
>> +++ b/guix/utils.scm
>> @@ -154,7 +155,7 @@ a symbol such as 'xz."
>> (match compression
>> ((or #f 'none) (values input '()))
>> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
>> - ('xz (filtered-port `(,%xz "-dc") input))
>> + ('xz (filtered-port `(,%xz "-dc -T0") input))
>> ('gzip (filtered-port `(,%gzip "-dc") input))
>> (else (error "unsupported compression scheme" compression))))
>
> Parallel decompression isn't implemented yet, but the extra '-T0' option
> doesn't seem to break anything.
>
> The changes LGTM.
>
> I sent a companion patch to <https://bugs.gnu.org/26316> that enables
> threaded compression when building source tarballs.

Is (parallel-job-count) accessible here? It would be nice to respect it.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAljfTiwACgkQoqBt8qM6
VPrBWQgApc4bdfvSCVtABSMaB+kRw9WSK5S/jcuYEIocUNxnqJJeQo+MF8nYNBPi
i0g1NlWrXr8nonRO0iuMIZLbi74iy4sv4ghGjRXIO4SZfWzsc8ESh+rZH9j2Wnq0
AEDn7oo5klBgWlHnHzsV5DabwQm/6fYFC0ozoBuZw6Xxhnxdg+FMUT8+7DhwexZG
tzfjOD6eoWGVA11N8693EW8KWEO+ViDZNteN5dmCtb7TN7h/1hw4jY3vOn78VLVn
p6ZuH5qxJYDc0e/jcR+PACdX4DZE3ucmt1YS6A0oj38vHPrmjbuRRY4g3+FVmVRk
wtgfbuGFw4Gv4kS/lmG3Uj3HJBnFeg==
=nXr1
-----END PGP SIGNATURE-----

L
L
Leo Famulari wrote on 2 Apr 2017 23:21
Re: bug#26225: [PATCH] guix: Compress and decompress xz archives in parallel.
(name . Marius Bakke)(address . mbakke@fastmail.com)
20170402212118.GA28696@jasmine
On Sat, Apr 01, 2017 at 08:52:28AM +0200, Marius Bakke wrote:
Toggle quote (18 lines)
> > Efraim Flashner wrote:
> >> * guix/scripts/pack.scm (%compressors): Add flag '-T0' when calling "xz".
> >> * guix/utils.scm (decompressed-port, compressed-port,
> >> compressed-output-port): Same.
> >
> >> --- a/guix/utils.scm
> >> +++ b/guix/utils.scm
> >> @@ -154,7 +155,7 @@ a symbol such as 'xz."
> >> (match compression
> >> ((or #f 'none) (values input '()))
> >> ('bzip2 (filtered-port `(,%bzip2 "-dc") input))
> >> - ('xz (filtered-port `(,%xz "-dc") input))
> >> + ('xz (filtered-port `(,%xz "-dc -T0") input))
> >> ('gzip (filtered-port `(,%gzip "-dc") input))
> >> (else (error "unsupported compression scheme" compression))))
>
> Is (parallel-job-count) accessible here? It would be nice to respect it.

I agree, it would be good if we respected it here, if possible.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAljha04ACgkQJkb6MLrK
fwjmyhAA4Kh8awEp7TBs8a83qSg1wzjOhMtmpZ3ocYpi70KWtQ9ZiyOTrWZMiy36
EvfbJweY+wUUqcSxbytP3nGRjRyxXSFyMw7KmTMvmGTlp2vLrw8Jn8s1mrrDvUoU
cWZC5Y8lT5LUTo/vYikk0qjppBjiLtwIe+voHGqKD+2rs9pySP5iHiT5rduRpWXO
R0UkLfsQkoDjkmzRbxNbwIi1TtNcUxjwhRcUEdnX4xQE1G9M4zdFIWarfos6nffD
fTfOf/7ClPmGB9eZi2vfPZgOJv8GM5t/u4yJVY/goKSqim6hnzmRpMzBKpqHbYu1
YVacG2ulnCpVbDTlN+yMeasVmBGW2c2Pzpz+Ecr1GKjGWq9D8SIigBsnK6GI9N9O
+f2DOmA3aWoKGnSui2InYVVAcGPlK435+Caycn4OSYZMkcOzn7E0rTowCBoYsNy7
n+8piwJ1UWb1qhImuxKNnJqFYTmef8DgMBbhS4PJXw/ZKYRm4qmk+u6uq9UPYOBg
FkoSuISww/9qDBQ8Y/Gqz1SHu6CEIPYJzA8pDJyPWrbtPqv46fKoIS4rlnZREGFb
YGJ2XSiirr5tMi7ukQCoJogL8jG2mpbGrxzw148QtSVAleJPte4BDb2f50XVja25
1pjvLXSt7nuHJ9yJRnwwREIBssUXu3/9ZWvb9/YG5KTzo7tbEB0=
=JfVK
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 7 Apr 2017 22:59
control message for bug #26225
(address . control@debbugs.gnu.org)
87o9w8szlf.fsf@gnu.org
tags 26225 fixed
close 26225
?