duplicate downloads when running “guix pull”
(address . bug-guix@gnu.org)
87ed6g5ryr.fsf@gmail.com
Hi,
Well, I notice several similar downloads when running “guix pull” or
“guix time-machine”; see complete log below.
This reports and investigates 2 different origins for these duplicates:
1. graft with multiple outputs;
2. remaining derivation about boot or mesboot.
For instance, when pulling it reads:
bash-minimal-5.1.16 568KiB 165KiB/s 00:03 ???????????????????? 100.0%
bash-minimal-5.1.16 569KiB 163KiB/s 00:03 ???????????????????? 100.0%
bzip2-1.0.8 55KiB 224KiB/s 00:00 ???????????????????? 100.0%
bzip2-1.0.8 55KiB 246KiB/s 00:00 ???????????????????? 100.0%
bzip2-1.0.8 55KiB 308KiB/s 00:00 ???????????????????? 100.0%
gcc-11.3.0-lib 4.8MiB 128KiB/s 00:38 ???????????????????? 100.0%
gcc-11.3.0-lib 4.8MiB 140KiB/s 00:35 ???????????????????? 100.0%
guile-3.0.9 7.0MiB 181KiB/s 00:40 ???????????????????? 100.0%
guile-3.0.9 7.0MiB 193KiB/s 00:37 ???????????????????? 100.0%
gzip-1.12 81KiB 43KiB/s 00:02 ???????????????????? 100.0%
gzip-1.12 81KiB 343KiB/s 00:00 ???????????????????? 100.0%
xz-5.2.8 366KiB 85KiB/s 00:04 ???????????????????? 100.0%
xz-5.2.8 366KiB 193KiB/s 00:02 ???????????????????? 100.0%
xz-5.2.8 366KiB 203KiB/s 00:02 ???????????????????? 100.0%
Based on the information below, it’s straightforward to extract the
profile and then trace it. Else, I guess this should reproduce:
guix pull --commit=4c194859f7c5956b03a4ccaa1c9ad22576c80c59 -p /tmp/this
Indeed, the profile refers to 2 bash-minimal, 3 bzip2, etc.
Toggle snippet (22 lines)
$ guix gc -R $(readlink -f /tmp/this) | grep bash-minimal
/gnu/store/9lhcyibqxfxni0n9z3x33bs66691bxrj-bash-minimal-5.1.16
/gnu/store/fl3l5wx8qynjrvx5lilz6c38hb77cf36-bash-minimal-5.1.16
$ guix gc -R $(readlink -f /tmp/this) | grep bzip2-1
/gnu/store/al9hskxmgzq0b1ff463djdjrdk7ypffj-bzip2-1.0.8
/gnu/store/bzcfrvhd8n8fnyi5gpfmsjmqwrgnpbp9-bzip2-1.0.8
/gnu/store/m0c98mlg3fv3096p6xc7arcw9v6zcxi9-bzip2-1.0.8
$ guix gc -R $(readlink -f /tmp/this) | grep gcc
/gnu/store/3sw5s4qlp8lka77y2jps8pvs4ldawdrs-gcc-11.3.0-lib
/gnu/store/ixwl0vnvm9zrrg89nxh62llh3li701dw-gcc-11.3.0-lib
$ guix gc -R $(readlink -f /tmp/this) | grep guile-3
/gnu/store/sg5kqvv9is29xh60innp13rlvc03yvr0-guile-3.0.9
/gnu/store/93jzwshnwlsijjf97ssdpw3w516cd6ja-guile-3.0.9
$ guix gc -R $(readlink -f /tmp/this) | grep xz-5
/gnu/store/cpmq6yl1ls1qnv2knzfh56bpbs2wss1q-xz-5.2.8
/gnu/store/8gxdqg8azmlwbdg5l9pnwab4pib9mwr4-xz-5.2.8
/gnu/store/d8y2fpk9d3ajmg6rpmn0rnb0dcmbk82w-xz-5.2.8
That’s consistent with the download. :-) The duplicates can be verified
with:
guix gc -R $(readlink -f /tmp/this) | cut -f2- -d'-' | sort | uniq -c | sort -n
Let give a look to the derivations producing these bzip2 outputs. The 3
derivations are derivations for grafting.
The difference for one compared to the two others is the ’static’
output.
/gnu/store/zmgwf3wa5zbal6zckrl187rlyh64yl8k-bzip2-1.0.8.drv
->
/gnu/store/m0c98mlg3fv3096p6xc7arcw9v6zcxi9-bzip2-1.0.8
/gnu/store/ha8hvl9215d6s5gb1rxiiai0rv6zi23w-bzip2-1.0.8-static
Obviously, the builder contains a reference to an old-output ’static’,
thus its content-address is different. Well, that’s annoying… at least
that appears to me as a bug. It’s similar for the case of ’xz’.
For information, the derivation that is grafted here reads:
/gnu/store/slv5z1nqd2i6kd90c78z6kfjgb3llnpi-bzip2-1.0.8.drv
Now, let give a look to:
/gnu/store/bzcfrvhd8n8fnyi5gpfmsjmqwrgnpbp9-bzip2-1.0.8
produced by:
/gnu/store/1fc3cl93512p61khr7grnd9zpvzywv83-bzip2-1.0.8.drv
Well, this derivation also grafts this other derivation:
/gnu/store/slv5z1nqd2i6kd90c78z6kfjgb3llnpi-bzip2-1.0.8.drv
Other said, the same as above. In summary, the same identical binaries
but differently content-addressed are downloaded twice.
Toggle snippet (8 lines)
$ diff -r --no-dereference \
> /gnu/store/m0c98mlg3fv3096p6xc7arcw9v6zcxi9-bzip2-1.0.8 \
> /gnu/store/bzcfrvhd8n8fnyi5gpfmsjmqwrgnpbp9-bzip2-1.0.8
Symbolic links /gnu/store/m0c98mlg3fv3096p6xc7arcw9v6zcxi9-bzip2-1.0.8/bin/bzcmp and /gnu/store/bzcfrvhd8n8fnyi5gpfmsjmqwrgnpbp9-bzip2-1.0.8/bin/bzcmp differ
Symbolic links /gnu/store/m0c98mlg3fv3096p6xc7arcw9v6zcxi9-bzip2-1.0.8/bin/bzegrep and /gnu/store/bzcfrvhd8n8fnyi5gpfmsjmqwrgnpbp9-bzip2-1.0.8/bin/bzegrep differ
Symbolic links /gnu/store/m0c98mlg3fv3096p6xc7arcw9v6zcxi9-bzip2-1.0.8/bin/bzfgrep and /gnu/store/bzcfrvhd8n8fnyi5gpfmsjmqwrgnpbp9-bzip2-1.0.8/bin/bzfgrep differ
Symbolic links /gnu/store/m0c98mlg3fv3096p6xc7arcw9v6zcxi9-bzip2-1.0.8/bin/bzless and /gnu/store/bzcfrvhd8n8fnyi5gpfmsjmqwrgnpbp9-bzip2-1.0.8/bin/bzless differ
Ok, done with the first part of the bug report. Now let show the second
part.
For the other duplicate, let compare the remaining derivation:
/gnu/store/0ir8nkhbzdd8dpjr7z0myy38d5gggr24-bzip2-1.0.8.drv
->
/gnu/store/al9hskxmgzq0b1ff463djdjrdk7ypffj-bzip2-1.0.8
This grafts the derivation:
/gnu/store/fc83kzqb5ry2f6bdlch1xhp3fkpiwc9p-bzip2-1.0.8.drv
which comes from some boot (boot0 and mesboot) as it reads in module
(gnu packages commencement). Well the builder contains:
("C_INCLUDE_PATH"
("include" "lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/include")
":" directory #f)
and many inputs contains the suffix -boot0 or -mesboot.
Last, since we are at it. :-) The derivation
/gnu/store/zmgwf3wa5zbal6zckrl187rlyh64yl8k-bzip2-1.0.8.drv
contains two mentions of bash-minimal:
/gnu/store/qvbq1b2pfjp0ka0kb0l3g264vimny30n-bash-minimal-5.1.16.drv
/gnu/store/r1sjrj2z2hn1s1ab7iqydk4vhc655gv8-bash-minimal-5.1.16.drv
Well, first this one:
/gnu/store/qvbq1b2pfjp0ka0kb0l3g264vimny30n-bash-minimal-5.1.16.drv
produces:
/gnu/store/fl3l5wx8qynjrvx5lilz6c38hb77cf36-bash-minimal-5.1.16
and as seen previously, it is part of the download.
Then, this qvbq1b2pfjp0ka0… derivation has r1sjrj2z2hn1s1a… as inputs too.
All in all, I do not expect to download the same more than once.
Cheers,
simon
PS: I am also surprised by the 78 messages:
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
but that’s another story. ;-)
Toggle snippet (178 lines)
$ guix describe
Generation 6 Jul 19 2024 12:18:52 (current)
guix 1aa8dfa
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 1aa8dfaeec3c6e4e587aadf7440246f7c5c04b9f
$ guix time-machine -q -- describe
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 4c19485 (1,127 new commits)...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
module-import 2KiB 240KiB/s 00:00 ???????????????????? 100.0%
module-import-compiled 1.3MiB 86KiB/s 00:15 ???????????????????? 100.0%
compute-guix-derivation 1KiB 1.5MiB/s 00:00 ???????????????????? 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
guile-zlib-0.2.1 43KiB 27KiB/s 00:02 ???????????????????? 100.0%
guix-daemon-1.4.0-24.9a2ddcc 262KiB 326KiB/s 00:01 ???????????????????? 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/yhakjkmb61dx7qn478l5j3l40vz4hyi4-profile.drv
/gnu/store/6ihpafcybca4va5xkdh35r4jgkq4ncvv-guix-cli-core-modules.drv
/gnu/store/f40iamijq94yj2bwjiwszsb6j1y7wgak-guix-system-tests-modules.drv
/gnu/store/nyjmpcn5bskdbzbz1rm8b30xyzinwn1x-guix-cli-modules.drv
/gnu/store/s3rjx81l3za7qnkihv9im8m9ldhljmhp-guix-packages-base-modules.drv
/gnu/store/vzzknpwcv2q1vsgimnlklk6glswj4hk1-guix-extra-modules.drv
/gnu/store/zv8q7zskkq1jv37z3x7p3iy7a43k36j9-guix-system-modules.drv
/gnu/store/f6lha1zn60km2w8gs05hj7y2xllv213i-guix-command.drv
/gnu/store/ggbf3w5pn8qzsv3amnqll5ynnhf9dz9w-guix-daemon.drv
16 items will be downloaded
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
guix-package-cache 396KiB/s 00:03 | 1.1MiB transferred
bash-minimal-5.1.16 568KiB 165KiB/s 00:03 ???????????????????? 100.0%
gcc-11.3.0-lib 4.8MiB 140KiB/s 00:35 ???????????????????? 100.0%
gcc-11.3.0-lib 4.8MiB 128KiB/s 00:38 ???????????????????? 100.0%
guix-cli 124KiB/s 00:03 | 398KiB transferred
bash-minimal-5.1.16 569KiB 163KiB/s 00:03 ???????????????????? 100.0%
bzip2-1.0.8 55KiB 308KiB/s 00:00 ???????????????????? 100.0%
bzip2-1.0.8 55KiB 246KiB/s 00:00 ???????????????????? 100.0%
bzip2-1.0.8 55KiB 224KiB/s 00:00 ???????????????????? 100.0%
expat-2.5.0 102KiB 273KiB/s 00:00 ???????????????????? 100.0%
gdbm-1.23 213KiB 34KiB/s 00:06 ???????????????????? 100.0%
gmp-6.2.1 571KiB 42KiB/s 00:14 ???????????????????? 100.0%
grep-3.8 220KiB 106KiB/s 00:02 ???????????????????? 100.0%
guile-bzip2-0.1.0 55KiB 246KiB/s 00:00 ???????????????????? 100.0%
guix-cli-core 104KiB/s 00:11 | 1.1MiB transferred
guix-cli-modules 258B 245KiB/s 00:00 ???????????????????? 100.0%
guix-cli-core-modules 263B 302KiB/s 00:00 ???????????????????? 100.0%
guix-extra 2.9MiB 169KiB/s 00:18 ???????????????????? 100.0%
guix-locale-guix 412KiB 120KiB/s 00:03 ???????????????????? 100.0%
guix-extra-modules 260B 244KiB/s 00:00 ???????????????????? 100.0%
guix-locale-guix-packages 191KiB 296KiB/s 00:01 ???????????????????? 100.0%
guix-manual 5.3MiB 183KiB/s 00:30 ???????????????????? 100.0%
guix-packages-base 180KiB/s 02:09 | 22.7MiB transferred
guix-system 173KiB/s 01:01 | 10.2MiB transferred
guix-packages-base-modules 267B 215KiB/s 00:00 ???????????????????? 100.0%
guix-system-modules 258B 322KiB/s 00:00 ???????????????????? 100.0%
guix-system-tests 840KiB 153KiB/s 00:05 ???????????????????? 100.0%
gzip-1.12 81KiB 343KiB/s 00:00 ???????????????????? 100.0%
guix-system-tests-modules 264B 288KiB/s 00:00 ???????????????????? 100.0%
gzip-1.12 81KiB 43KiB/s 00:02 ???????????????????? 100.0%
http-parser-2.9.4-1.ec8b5ee 18KiB 162KiB/s 00:00 ???????????????????? 100.0%
libcap-2.64 82KiB 86KiB/s 00:01 ???????????????????? 100.0%
libdaemon-0.14 29KiB 90KiB/s 00:00 ???????????????????? 100.0%
libevent-2.1.12 304KiB 117KiB/s 00:03 ???????????????????? 100.0%
libffi-3.4.4 56KiB 497KiB/s 00:00 ???????????????????? 100.0%
libgc-8.2.2 218KiB 219KiB/s 00:01 ???????????????????? 100.0%
libgpg-error-1.45 222KiB 187KiB/s 00:01 ???????????????????? 100.0%
libtasn1-4.19.0 106KiB 280KiB/s 00:00 ???????????????????? 100.0%
libgcrypt-1.10.1 552KiB 153KiB/s 00:04 ???????????????????? 100.0%
libunistring-1.0 529KiB 102KiB/s 00:05 ???????????????????? 100.0%
guile-gcrypt-0.4.0 165KiB 35KiB/s 00:05 ???????????????????? 100.0%
libidn2-2.3.4 170KiB 41KiB/s 00:04 ???????????????????? 100.0%
libxau-1.0.10 8KiB 1.0MiB/s 00:00 ???????????????????? 100.0%
libxdmcp-1.1.3 24KiB 58KiB/s 00:00 ???????????????????? 100.0%
lzlib-1.13 51KiB 69KiB/s 00:01 ???????????????????? 100.0%
libxcb-1.15 279KiB 43KiB/s 00:06 ???????????????????? 100.0%
guile-lzlib-0.3.0 36KiB 85KiB/s 00:00 ???????????????????? 100.0%
libx11-1.8.7 669KiB 67KiB/s 00:10 ???????????????????? 100.0%
ncurses-6.2.20210619 633KiB 203KiB/s 00:03 ???????????????????? 100.0%
dbus-1.15.8 281KiB 219KiB/s 00:01 ???????????????????? 100.0%
nettle-3.8.1 362KiB 230KiB/s 00:02 ???????????????????? 100.0%
nghttp2-1.58.0-lib 93KiB 60KiB/s 00:02 ???????????????????? 100.0%
openssl-3.0.8 2.2MiB 130KiB/s 00:17 ???????????????????? 100.0%
p11-kit-0.24.1 433KiB 146KiB/s 00:03 ???????????????????? 100.0%
mit-krb5-1.20 1.0MiB 144KiB/s 00:07 ???????????????????? 100.0%
pkg-config-0.29.2 187KiB 131KiB/s 00:01 ???????????????????? 100.0%
readline-8.1.2 409KiB 174KiB/s 00:02 ???????????????????? 100.0%
guile-3.0.9 7.0MiB 181KiB/s 00:40 ???????????????????? 100.0%
guile-3.0.9 7.0MiB 193KiB/s 00:37 ???????????????????? 100.0%
guile-wrapper 3KiB 712KiB/s 00:00 ???????????????????? 100.0%
sqlite-3.39.3 1.1MiB 209KiB/s 00:05 ???????????????????? 100.0%
tar-1.34 745KiB 211KiB/s 00:04 ???????????????????? 100.0%
guile-sqlite3-0.1.3 57KiB 264KiB/s 00:00 ???????????????????? 100.0%
util-linux-2.37.4-lib 1.5MiB 195KiB/s 00:08 ???????????????????? 100.0%
xz-5.2.8 366KiB 203KiB/s 00:02 ????????????????
This message was truncated. Download the full message here.