[PATCH] packages, scripts, utils: Respect (parallel-job-count) when xz-compressing.

DoneSubmitted by Marius Bakke.
Details
2 participants
  • Ludovic Courtès
  • Marius Bakke
Owner
unassigned
Severity
normal
M
M
Marius Bakke wrote on 1 Apr 2018 14:54
(address . guix-patches@gnu.org)(name . Marius Bakke)(address . mbakke@fastmail.com)
20180401125431.11996-1-mbakke@fastmail.com
* guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use theamount of cores specified by e.g. 'guix build -c'.* guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise. While at it,use the long form '--threads=' for clarity.* guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):Likewise.--- guix/packages.scm | 3 ++- guix/scripts/pack.scm | 7 +++++-- guix/utils.scm | 18 +++++++++++++++--- 3 files changed, 22 insertions(+), 6 deletions(-)
Toggle diff (84 lines)diff --git a/guix/packages.scm b/guix/packages.scmindex ab4b6278d..cca2c6357 100644--- a/guix/packages.scm+++ b/guix/packages.scm@@ -604,7 +604,8 @@ specifies modules in scope when evaluating SNIPPET." ;; threaded compression (introduced in ;; 5.2.0), but it ignores the extra flag. (string-append "--use-compress-program="- #+xz "/bin/xz --threads=0")+ #+xz "/bin/xz --threads="+ (number->string (parallel-job-count))) ;; avoid non-determinism in the archive "--mtime=@0" "--owner=root:0"diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scmindex 488638adc..818de0f32 100644--- a/guix/scripts/pack.scm+++ b/guix/scripts/pack.scm@@ -65,7 +65,9 @@ (compressor "lzip" ".lz" #~(#+(file-append lzip "/bin/lzip") "-9")) (compressor "xz" ".xz"- #~(#+(file-append xz "/bin/xz") "-e -T0"))+ #~(#+(file-append xz "/bin/xz")+ (string-append "-e --threads="+ (number->string (parallel-job-count))))) (compressor "bzip2" ".bz2" #~(#+(file-append bzip2 "/bin/bzip2") "-9")) (compressor "none" "" #f)))@@ -73,7 +75,8 @@ ;; This one is only for use in this module, so don't put it in %compressors. (define bootstrap-xz (compressor "bootstrap-xz" ".xz"- #~(#+(file-append %bootstrap-coreutils&co "/bin/xz") "-e -T0")))+ #~(#+(file-append %bootstrap-coreutils&co "/bin/xz")+ (string-append "-e --threads=" (number->string (parallel-job-count)))))) (define (lookup-compressor name) "Return the compressor object called NAME. Error out if it could not bediff --git a/guix/utils.scm b/guix/utils.scmindex 92e45de61..043034e99 100644--- a/guix/utils.scm+++ b/guix/utils.scm@@ -172,7 +172,11 @@ a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-dc") input))- ('xz (filtered-port `(,%xz "-dc" "-T0") input))+ ('xz (filtered-port+ `(,%xz "-dc" (string-append+ "--threads="+ (number->string (parallel-job-count))))+ input)) ('gzip (filtered-port `(,%gzip "-dc") input)) (else (error "unsupported compression scheme" compression)))) @@ -182,7 +186,11 @@ a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-c") input))- ('xz (filtered-port `(,%xz "-c" "-T0") input))+ ('xz (filtered-port+ `(,%xz "-c" (string-append+ "--threads="+ (number->string (parallel-job-count))))+ input)) ('gzip (filtered-port `(,%gzip "-c") input)) (else (error "unsupported compression scheme" compression)))) @@ -239,7 +247,11 @@ program--e.g., '(\"--fast\")." (match compression ((or #f 'none) (values output '())) ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output))- ('xz (filtered-output-port `(,%xz "-c" "-T0" ,@options) output))+ ('xz (filtered-output-port+ `(,%xz "-c" (string-append+ "--threads="+ (number->string (parallel-job-count)))+ ,@options) output)) ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output)) (else (error "unsupported compression scheme" compression)))) -- 2.16.3
L
L
Ludovic Courtès wrote on 4 Apr 2018 13:42
(name . Marius Bakke)(address . mbakke@fastmail.com)(address . 31015@debbugs.gnu.org)
877epnfbau.fsf@gnu.org
Hello!
Marius Bakke <mbakke@fastmail.com> skribis:
Toggle quote (7 lines)> * guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the> amount of cores specified by e.g. 'guix build -c'.> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise. While at it,> use the long form '--threads=' for clarity.> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):> Likewise.
Wasn’t there an issue that parallel xz is non-deterministic? I vaguelyremember something like this when we were producing with glibc tarballsout of the Git repo.
If so, we should instead use -T1, even if that means getting worseperformance (I hear that lzip “works better” than xz, so we should giveit a try if we want both performance and good compression.)
Besides, the guix/packages.scm part would lead to a full rebuild, so Ithink it’s a bit late for ‘core-updates’.
Thanks,Ludo’.
L
L
Ludovic Courtès wrote on 13 May 2018 22:37
(name . Marius Bakke)(address . mbakke@fastmail.com)(address . 31015@debbugs.gnu.org)
87a7t3l2ah.fsf@gnu.org
Hello Marius,
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (13 lines)> Marius Bakke <mbakke@fastmail.com> skribis:>>> * guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the>> amount of cores specified by e.g. 'guix build -c'.>> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise. While at it,>> use the long form '--threads=' for clarity.>> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):>> Likewise.>> Wasn’t there an issue that parallel xz is non-deterministic? I vaguely> remember something like this when we were producing with glibc tarballs> out of the Git repo.
That was indeed the case, so I’m closing this issue. Let’s reopen it ifwe have more info or new ideas! :-)
Ludo’.
L
L
Ludovic Courtès wrote on 13 May 2018 22:38
control message for bug #31015
(address . control@debbugs.gnu.org)
878t8nl2ab.fsf@gnu.org
tags 31015 wontfixclose 31015
?
Your comment

This issue is archived.

To comment on this conversation send email to 31015@debbugs.gnu.org