Hi! The patch below is a followup to the thread started in December: https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html It provides a naïve but apparently good enough way for ‘guix substitute’to choose the compression method that yields the best speed given theCPU and current networking conditions. On a recent x86_64 laptop with fast networking, using ci.guix.gnu.org,the effect so far is to choose gzip substitutes, which indeed providesslightly faster substitute installation. When ci.guix provides zstdsubstitutes, the speedup will be higher. I have yet to check that it sticks to lzip when bandwidth is low. Thoughts? Thanks,Ludo’.
From 3f95a1ac04c5e178a7fedfc2d03c07bcb1075ead Mon Sep 17 00:00:00 2001From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <email@example.com>Date: Sun, 14 Mar 2021 15:05:30 +0100Subject: [PATCH] substitute: Choose compression method based on past CPU usage. This stems from the observation that substitute download can beCPU-bound when high-speed networks are in use: https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html * guix/narinfo.scm (decompresses-faster?): New procedure.(narinfo-best-uri): Add #:fast-decompression?.* guix/scripts/substitute.scm (%prefer-fast-decompression?): Newvariable.(call-with-cpu-usage-monitoring): New procedure.(with-cpu-usage-monitoring): New macro.(display-narinfo-data, process-substitution): Pass #:fast-decompression?to 'narinfo-best-uri'.(process-substitution): Wrap 'restore-file' call in'with-cpu-usage-monitoring'. Set '%prefer-fast-decompression?'.--- guix/narinfo.scm | 27 ++++++++++++++++--- guix/scripts/substitute.scm | 53 ++++++++++++++++++++++++++++++++----- 2 files changed, 69 insertions(+), 11 deletions(-)
Toggle quote (15 lines)> The patch below is a followup to the thread started in December:>> https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html>> It provides a naïve but apparently good enough way for ‘guix substitute’> to choose the compression method that yields the best speed given the> CPU and current networking conditions.>> On a recent x86_64 laptop with fast networking, using ci.guix.gnu.org,> the effect so far is to choose gzip substitutes, which indeed provides> slightly faster substitute installation. When ci.guix provides zstd> substitutes, the speedup will be higher.>> I have yet to check that it sticks to lzip when bandwidth is low.
I did that, using ‘tc’, and it works as expected, staying on lzip. Pushed as 9da5ec7099b992a8969a17627548cd341c01bd90 with two minortweaks: lowered the low hysteresis threshold, and added a comment on howto use ‘tc’ to test the behavior on “slow” networks. Rather than running ‘guix build’ followed by ‘guix gc’, I found thatmanually invoking ‘guix substitute’ was nicer (long line ahead!): ( echo substitute /gnu/store/svv4826f8zfj8grl2qa17xnxk3acsppc-elixir-1.11.4 /tmp/t1; echo substitute /gnu/store/d9dk53m7pwx1dc1p97zm0q323gpk70f9-poezio-0.13.1 /tmp/t4; echo substitute /gnu/store/mra8i18y9gjavhmdlkbb10m4miinirgz-ocaml-4.11.1 /tmp/t2; echo substitute /gnu/store/ay2j5mp20j9vbhibcwp5lmmcmhqkdnga-vim-full-8.2.2632 /tmp/t3; echo substitute /gnu/store/svv4826f8zfj8grl2qa17xnxk3acsppc-elixir-1.11.4 /tmp/t5; echo substitute /gnu/store/ay2j5mp20j9vbhibcwp5lmmcmhqkdnga-vim-full-8.2.2632 /tmp/t6) | GUIX_ALLOW_UNAUTHENTICATED_SUBSTITUTES=yes ./pre-inst-env guix substitute --substitute 4>&2 Note that this change won’t take effect until we update the ‘guix’package. Ludo’.