Grafts leads to inefficient substitute info retrieval

DoneSubmitted by Ludovic Courtès.
Details
4 participants
  • Alex Kost
  • David Craven
  • Ludovic Courtès
  • Mark H Weaver
Owner
unassigned
Severity
important
Merged with
L
L
Ludovic Courtès wrote on 11 Mar 2016 17:52
(address . bug-guix@gnu.org)
8737rxx8gk.fsf@gnu.org
As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
1. Fetching substitute info about the things being built so that it can determine its references, which in turns allows it to determine whether they need to be grafted.
2. Building stuff, as a last resort, so that it can determine its references.
Case #1 is hopefully going to be the most common.
The problem with #1 is that when building a profile, we do one‘package-derivation’ call for each package in the profile, whichtranslates in one ‘graft-derivation’ call for each relevant package¹,which translates into one ‘references/substitutes’ call for each.
Concretely, what this means is this:
$ guix package -u substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0% substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0% […] The following files would be downloaded:
Each of the initial “updating list” message corresponds to an HTTPrequest for a single narinfo file, which can take around 1 second.
Instead, the ideal thing would be to fetch the narinfo files for all therelevant packages at once; that way, we’d spawn ‘guix substitute’ onlyonce, and it would benefit from HTTP pipelining (one round-trip insteadof N.)
To achieve this, I’m thinking of extending gexp code such that gexpcompilers can return a list of applicable grafts. The ‘package’compiler would do #:graft? #f and instead let ‘gexp->derivation’ call‘graft-derivation’.
I’ll give it a try and report back.
Ludo’.
¹ A package is “relevant” if ‘package-grafts’ returns a non-empty list.
A
A
Alex Kost wrote on 12 Mar 2016 10:23
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22990@debbugs.gnu.org)
87a8m4123s.fsf@gmail.com
Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
Toggle quote (9 lines)> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:>> 1. Fetching substitute info about the things being built so that it> can determine its references, which in turns allows it to determine> whether they need to be grafted.>> 2. Building stuff, as a last resort, so that it can determine its> references.
I noticed that #1 is happening even with --no-substitutes option. Is itintended?
I've tried this:
$ guix build --dry-run --no-substitutes muttsubstitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%The following derivations would be built: /gnu/store/a2w22xlmfwkgwx4vw11dxc6zrdmww435-mutt-1.5.24.drv ...
-- Alex
L
L
Ludovic Courtès wrote on 13 Mar 2016 13:11
(name . Alex Kost)(address . alezost@gmail.com)(address . 22990@debbugs.gnu.org)
874mcazifb.fsf@gnu.org
Alex Kost <alezost@gmail.com> skribis:
Toggle quote (14 lines)> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:>>>> 1. Fetching substitute info about the things being built so that it>> can determine its references, which in turns allows it to determine>> whether they need to be grafted.>>>> 2. Building stuff, as a last resort, so that it can determine its>> references.>> I noticed that #1 is happening even with --no-substitutes option. Is it> intended?
Not really, but I see this is because ‘substitutable-path-info’ (calledfrom ‘references/substitutes’, called from ‘graft-derivation’) worksregardless of whether substitutes are enabled:
Toggle snippet (10 lines)scheme@(guile-user)> ,use(guix)scheme@(guile-user)> (define s (open-connection))scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)$2 = #tscheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")$3 = #fscheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))$4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)
However, substitutes are not downloaded, so in this regard--no-substitutes is honored.
Ludo’.
L
L
Ludovic Courtès wrote on 15 Mar 2016 09:24
(address . 22990@debbugs.gnu.org)
87mvq0tagl.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (13 lines)> Concretely, what this means is this:>> $ guix package -u> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> […]> The following files would be downloaded:
This is somewhat mitigated by commitsf09aea1b58b3ef961d3cc712f116fe4617bc8f90 and264fdedb408ba3620d1e361de6c77e7925025301.
In addition, 026ca50fa4c46a8e280cd51621bbec76b12c0757 slightly improves‘guix substitute’ performance.
Ludo’.
M
M
Mark H Weaver wrote on 15 Mar 2016 19:49
(name . Ludovic Courtès)(address . ludo@gnu.org)
8737rrmv8s.fsf@netris.org
ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (29 lines)> Alex Kost <alezost@gmail.com> skribis:>>> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:>>>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:>>>>>> 1. Fetching substitute info about the things being built so that it>>> can determine its references, which in turns allows it to determine>>> whether they need to be grafted.>>>>>> 2. Building stuff, as a last resort, so that it can determine its>>> references.>>>> I noticed that #1 is happening even with --no-substitutes option. Is it>> intended?>> Not really, but I see this is because ‘substitutable-path-info’ (called> from ‘references/substitutes’, called from ‘graft-derivation’) works> regardless of whether substitutes are enabled:>> scheme@(guile-user)> ,use(guix)> scheme@(guile-user)> (define s (open-connection))> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)> $2 = #t> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")> $3 = #f> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))> $4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)
Is the information from the substitute server authenticated by checkinghydra's signature against the list of keys in /etc/guix/acls?
The reason I ask is that if the set of runtime dependencies received isincomplete, it could lead to incorrect grafting, namely that referencesto compromised libraries could be retained.
Toggle quote (3 lines)> However, substitutes are not downloaded, so in this regard> --no-substitutes is honored.
It depends on the intent of --no-substitutes. If the intent is to avoidtrusting the substitute server, then by relying on the accuracy of theruntime dependency data from Hydra, we are failing to honor that intent.
That said, I think it's okay to document that --no-substitutes alone isnot sufficient to avoid trusting a substitute server, and that theproper way to accomplish that is to make sure its key is not in/etc/guix/acls.
What do you think?
Thanks, Mark
L
L
Ludovic Courtès wrote on 15 Mar 2016 22:50
(name . Mark H Weaver)(address . mhw@netris.org)
87twk7qulu.fsf@gnu.org
Mark H Weaver <mhw@netris.org> skribis:
Toggle quote (34 lines)> ludo@gnu.org (Ludovic Courtès) writes:>>> Alex Kost <alezost@gmail.com> skribis:>>>>> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:>>>>>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:>>>>>>>> 1. Fetching substitute info about the things being built so that it>>>> can determine its references, which in turns allows it to determine>>>> whether they need to be grafted.>>>>>>>> 2. Building stuff, as a last resort, so that it can determine its>>>> references.>>>>>> I noticed that #1 is happening even with --no-substitutes option. Is it>>> intended?>>>> Not really, but I see this is because ‘substitutable-path-info’ (called>> from ‘references/substitutes’, called from ‘graft-derivation’) works>> regardless of whether substitutes are enabled:>>>> scheme@(guile-user)> ,use(guix)>> scheme@(guile-user)> (define s (open-connection))>> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)>> $2 = #t>> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")>> $3 = #f>> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))>> $4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)>> Is the information from the substitute server authenticated by checking> hydra's signature against the list of keys in /etc/guix/acls?
Yes.
Toggle quote (12 lines)>> However, substitutes are not downloaded, so in this regard>> --no-substitutes is honored.>> It depends on the intent of --no-substitutes. If the intent is to avoid> trusting the substitute server, then by relying on the accuracy of the> runtime dependency data from Hydra, we are failing to honor that intent.>> That said, I think it's okay to document that --no-substitutes alone is> not sufficient to avoid trusting a substitute server, and that the> proper way to accomplish that is to make sure its key is not in> /etc/guix/acls.
The sysadmin gets to choose which principals are trusted; unprivilegedusers can only shrink this set.
The weird thing is that in this case, passing --substitute-urls='' onthe client side would effectively disable substitutes entirely.
Toggle quote (2 lines)> What do you think?
We could augment the doc for --no-substitutes, I guess we should firstdocument that grafting relies on server-provided info.
Ludo’.
L
L
Ludovic Courtès wrote on 9 Dec 2016 21:48
control message for bug #25137
(address . control@debbugs.gnu.org)
87shpw7rsr.fsf@gnu.org
merge 25137 22990
L
L
Ludovic Courtès wrote on 8 Jan 2017 22:44
Re: bug#22990: Grafts leads to inefficient substitute info retrieval
(address . 22990@debbugs.gnu.org)
87k2a5rzue.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (26 lines)> Concretely, what this means is this:>> $ guix package -u> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> […]> The following files would be downloaded:>> Each of the initial “updating list” message corresponds to an HTTP> request for a single narinfo file, which can take around 1 second.>> Instead, the ideal thing would be to fetch the narinfo files for all the> relevant packages at once; that way, we’d spawn ‘guix substitute’ only> once, and it would benefit from HTTP pipelining (one round-trip instead> of N.)>> To achieve this, I’m thinking of extending gexp code such that gexp> compilers can return a list of applicable grafts. The ‘package’> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call> ‘graft-derivation’.
The ‘wip-gexp-grafts’ branch does that. Namely, it’s possible to knowwhat grafts would apply to a gexp derivation build, so that one canfirst build ungrafted, and then apply the grafts to the results.So for a profile, we’d first build the profile as is, and only thenwould we graft it.
Right now the tip of this branch is a hack such that ‘guix package’:
1. Builds the original (ungrafted) derivation of the profile;
2. Manually calls ‘graft-derivation’ on that, passing it the list of applicable grafts.
Conceptually it’s what we want to do, but the drawback is that thecaller (here ‘guix package’) goes through a lot of hops to get the listof grafts and to apply it.
I think we should instead have a way to annotate a derivation with alist of grafts, as well as a procedure to build that derivations in twophases (first the original derivation, then the grafts).
To be continued…
Ludo’.
L
L
Ludovic Courtès wrote on 9 Jan 2017 23:55
(address . 22990@debbugs.gnu.org)
877f633kt1.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (2 lines)> ludo@gnu.org (Ludovic Courtès) skribis:
[...]
Toggle quote (26 lines)>> To achieve this, I’m thinking of extending gexp code such that gexp>> compilers can return a list of applicable grafts. The ‘package’>> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call>> ‘graft-derivation’.>> The ‘wip-gexp-grafts’ branch does that. Namely, it’s possible to know> what grafts would apply to a gexp derivation build, so that one can> first build ungrafted, and then apply the grafts to the results.> So for a profile, we’d first build the profile as is, and only then> would we graft it.>> Right now the tip of this branch is a hack such that ‘guix package’:>> 1. Builds the original (ungrafted) derivation of the profile;>> 2. Manually calls ‘graft-derivation’ on that, passing it the list of> applicable grafts.>> Conceptually it’s what we want to do, but the drawback is that the> caller (here ‘guix package’) goes through a lot of hops to get the list> of grafts and to apply it.>> I think we should instead have a way to annotate a derivation with a> list of grafts, as well as a procedure to build that derivations in two> phases (first the original derivation, then the grafts).
The current iteration introduces “build continuation”: a derivation canbe annotated with a continuation, and the ‘build-things’ procedure willloop over continuations and return the final results (that only worksfor derivations directly passed as an argument to ‘build-things’.)
Example (where Guile 2.0.12 replaced by 2.0.13):
Toggle snippet (16 lines)$ ./pre-inst-env guix package -p foo -i guile-json guile-ssh gdbThe following packages will be installed: guile-json 0.5.0 /gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0 guile-ssh 0.10.2 /gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2 gdb 7.12 /gnu/store/d2a4lmycc13ssdf47a9h410knlfqqq41-gdb-7.12
applying 6 grafts to /gnu/store/s7w6r9ih52kdzcl7kprf2rq48s69zh98-profile3 packages in profileThe following environment variable definitions may be needed: export PATH="foo/bin${PATH:+:}$PATH"$ guix gc -R foo | grep guile/gnu/store/7fisf4frrgsjzmknjbab1dal23wxrp8d-guile-2.0.13/gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2/gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0
That seems to be a good model.
Now we must make sure that it works also for ‘guix build foo’ and ‘guixsystem build’.
Ludo’.
L
L
Ludovic Courtès wrote on 11 Jan 2017 10:54
(address . 22990@debbugs.gnu.org)
874m15yl99.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (37 lines)> ludo@gnu.org (Ludovic Courtès) skribis:>>> ludo@gnu.org (Ludovic Courtès) skribis:>> [...]>>>> To achieve this, I’m thinking of extending gexp code such that gexp>>> compilers can return a list of applicable grafts. The ‘package’>>> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call>>> ‘graft-derivation’.>>>> The ‘wip-gexp-grafts’ branch does that. Namely, it’s possible to know>> what grafts would apply to a gexp derivation build, so that one can>> first build ungrafted, and then apply the grafts to the results.>> So for a profile, we’d first build the profile as is, and only then>> would we graft it.>>>> Right now the tip of this branch is a hack such that ‘guix package’:>>>> 1. Builds the original (ungrafted) derivation of the profile;>>>> 2. Manually calls ‘graft-derivation’ on that, passing it the list of>> applicable grafts.>>>> Conceptually it’s what we want to do, but the drawback is that the>> caller (here ‘guix package’) goes through a lot of hops to get the list>> of grafts and to apply it.>>>> I think we should instead have a way to annotate a derivation with a>> list of grafts, as well as a procedure to build that derivations in two>> phases (first the original derivation, then the grafts).>> The current iteration introduces “build continuation”: a derivation can> be annotated with a continuation, and the ‘build-things’ procedure will> loop over continuations and return the final results (that only works> for derivations directly passed as an argument to ‘build-things’.)
On second thought, the whole idea of applying grafts on the final result(instead of applying grafts at each step like we do now) doesn’t fly.It works well for things like a profile or the system derivation, butbreaks for less trivial things.
For example, if you’re building a VM image or a binary tarball, youreally need to graft packages early on; trying to graft the VM image orbinary tarball wouldn’t have the desired effect.
I’ve pushed ‘wip-gexp-grafts’ again but it seems to be a dead end.
Ludo’.
D
D
David Craven wrote on 11 Jan 2017 11:51
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22990@debbugs.gnu.org)
CAL1_imnWoGFQ=NBjghV1wracigF676j4zomgCsfCTPO+JiPbKQ@mail.gmail.com
Toggle quote (9 lines)> On second thought, the whole idea of applying grafts on the final result> (instead of applying grafts at each step like we do now) doesn’t fly.> It works well for things like a profile or the system derivation, but> breaks for less trivial things.
> For example, if you’re building a VM image or a binary tarball, you> really need to graft packages early on; trying to graft the VM image or> binary tarball wouldn’t have the desired effect.
Isn't a system derivation or a profile derivation an intermediate stepto these derivations? Can't there be a flag or something called#:already-grafted? #t?
L
L
Ludovic Courtès wrote on 11 Jan 2017 22:19
(name . David Craven)(address . david@craven.ch)(address . 22990@debbugs.gnu.org)
87k2a1wazs.fsf@gnu.org
David Craven <david@craven.ch> skribis:
Toggle quote (12 lines)>> On second thought, the whole idea of applying grafts on the final result>> (instead of applying grafts at each step like we do now) doesn’t fly.>> It works well for things like a profile or the system derivation, but>> breaks for less trivial things.>>> For example, if you’re building a VM image or a binary tarball, you>> really need to graft packages early on; trying to graft the VM image or>> binary tarball wouldn’t have the desired effect.>> Isn't a system derivation or a profile derivation an intermediate step> to these derivations?
Yes, you’re right. However the implementation I had come up with reliedon “build continuations”, which only worked for the “top-level”derivations (those you pass to ‘build-derivations’.)
Toggle quote (2 lines)> Can't there be a flag or something called #:already-grafted? #t?
Yeah, we need something like that. I need to chew a bit more on this tofind a nice way to achieve that.
Thanks for your feedback, I feel less lonely now. :-)
Ludo’.
L
L
Ludovic Courtès wrote on 27 May 2017 12:03
control message for bug #22990
(address . control@debbugs.gnu.org)
87tw46vcor.fsf@gnu.org
severity 22990 important
L
L
Ludovic Courtès wrote on 29 Mar 15:41 +0200
Re: bug#22990: Grafts leads to inefficient substitute info retrieval
(address . 22990-done@debbugs.gnu.org)
87tv271dob.fsf@gnu.org
Hi younger self!
ludo@gnu.org (Ludovic Courtès) skribis:
Toggle quote (32 lines)> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:>> 1. Fetching substitute info about the things being built so that it> can determine its references, which in turns allows it to determine> whether they need to be grafted.>> 2. Building stuff, as a last resort, so that it can determine its> references.>> Case #1 is hopefully going to be the most common.>> The problem with #1 is that when building a profile, we do one> ‘package-derivation’ call for each package in the profile, which> translates in one ‘graft-derivation’ call for each relevant package¹,> which translates into one ‘references/substitutes’ call for each.>> Concretely, what this means is this:>> $ guix package -u> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%> […]> The following files would be downloaded:>> Each of the initial “updating list” message corresponds to an HTTP> request for a single narinfo file, which can take around 1 second.
This is finally fixed with commit710854304b1ab29332edcb76f3de532e0724c197, as discussed earlier thisweek¹. \o/
The solution is simpler and more orthogonal than what was envisionedearlier in this thread, which is nice.
I also potentially addresses other uses of “dynamic dependencies”elsewhere in the code base, such as ‘remote-eval’ calls in ‘guixdeploy’.
Feedback welcome!
Ludo’.
¹ https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00337.html
Closed
?