Grafts leads to inefficient substitute info retrieval

  • Done
  • quality assurance status badge
Details
4 participants
  • Alex Kost
  • David Craven
  • Ludovic Courtès
  • Mark H Weaver
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
important
Merged with
L
L
Ludovic Courtès wrote on 11 Mar 2016 17:52
(address . bug-guix@gnu.org)
8737rxx8gk.fsf@gnu.org
As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:

1. Fetching substitute info about the things being built so that it
can determine its references, which in turns allows it to determine
whether they need to be grafted.

2. Building stuff, as a last resort, so that it can determine its
references.

Case #1 is hopefully going to be the most common.

The problem with #1 is that when building a profile, we do one
‘package-derivation’ call for each package in the profile, which
translates in one ‘graft-derivation’ call for each relevant package¹,
which translates into one ‘references/substitutes’ call for each.

Concretely, what this means is this:

$ guix package -u
substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
[…]
The following files would be downloaded:

Each of the initial “updating list” message corresponds to an HTTP
request for a single narinfo file, which can take around 1 second.

Instead, the ideal thing would be to fetch the narinfo files for all the
relevant packages at once; that way, we’d spawn ‘guix substitute’ only
once, and it would benefit from HTTP pipelining (one round-trip instead
of N.)

To achieve this, I’m thinking of extending gexp code such that gexp
compilers can return a list of applicable grafts. The ‘package’
compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
‘graft-derivation’.

I’ll give it a try and report back.

Ludo’.

¹ A package is “relevant” if ‘package-grafts’ returns a non-empty list.
A
A
Alex Kost wrote on 12 Mar 2016 10:23
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22990@debbugs.gnu.org)
87a8m4123s.fsf@gmail.com
Ludovic Courtès (2016-03-11 19:52 +0300) wrote:

Toggle quote (9 lines)
> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>
> 1. Fetching substitute info about the things being built so that it
> can determine its references, which in turns allows it to determine
> whether they need to be grafted.
>
> 2. Building stuff, as a last resort, so that it can determine its
> references.

I noticed that #1 is happening even with --no-substitutes option. Is it
intended?

I've tried this:

$ guix build --dry-run --no-substitutes mutt
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
The following derivations would be built:
/gnu/store/a2w22xlmfwkgwx4vw11dxc6zrdmww435-mutt-1.5.24.drv
...

--
Alex
L
L
Ludovic Courtès wrote on 13 Mar 2016 13:11
(name . Alex Kost)(address . alezost@gmail.com)(address . 22990@debbugs.gnu.org)
874mcazifb.fsf@gnu.org
Alex Kost <alezost@gmail.com> skribis:

Toggle quote (14 lines)
> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>
>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>
>> 1. Fetching substitute info about the things being built so that it
>> can determine its references, which in turns allows it to determine
>> whether they need to be grafted.
>>
>> 2. Building stuff, as a last resort, so that it can determine its
>> references.
>
> I noticed that #1 is happening even with --no-substitutes option. Is it
> intended?

Not really, but I see this is because ‘substitutable-path-info’ (called
from ‘references/substitutes’, called from ‘graft-derivation’) works
regardless of whether substitutes are enabled:

Toggle snippet (10 lines)
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> (define s (open-connection))
scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
$2 = #t
scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
$3 = #f
scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
$4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)

However, substitutes are not downloaded, so in this regard
--no-substitutes is honored.

Ludo’.
L
L
Ludovic Courtès wrote on 15 Mar 2016 09:24
(address . 22990@debbugs.gnu.org)
87mvq0tagl.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (13 lines)
> Concretely, what this means is this:
>
> $ guix package -u
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> […]
> The following files would be downloaded:

This is somewhat mitigated by commits
f09aea1b58b3ef961d3cc712f116fe4617bc8f90 and
264fdedb408ba3620d1e361de6c77e7925025301.

In addition, 026ca50fa4c46a8e280cd51621bbec76b12c0757 slightly improves
‘guix substitute’ performance.

Ludo’.
M
M
Mark H Weaver wrote on 15 Mar 2016 19:49
(name . Ludovic Courtès)(address . ludo@gnu.org)
8737rrmv8s.fsf@netris.org
ludo@gnu.org (Ludovic Courtès) writes:

Toggle quote (29 lines)
> Alex Kost <alezost@gmail.com> skribis:
>
>> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>>
>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>>
>>> 1. Fetching substitute info about the things being built so that it
>>> can determine its references, which in turns allows it to determine
>>> whether they need to be grafted.
>>>
>>> 2. Building stuff, as a last resort, so that it can determine its
>>> references.
>>
>> I noticed that #1 is happening even with --no-substitutes option. Is it
>> intended?
>
> Not really, but I see this is because ‘substitutable-path-info’ (called
> from ‘references/substitutes’, called from ‘graft-derivation’) works
> regardless of whether substitutes are enabled:
>
> scheme@(guile-user)> ,use(guix)
> scheme@(guile-user)> (define s (open-connection))
> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
> $2 = #t
> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
> $3 = #f
> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
> $4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)

Is the information from the substitute server authenticated by checking
hydra's signature against the list of keys in /etc/guix/acls?

The reason I ask is that if the set of runtime dependencies received is
incomplete, it could lead to incorrect grafting, namely that references
to compromised libraries could be retained.

Toggle quote (3 lines)
> However, substitutes are not downloaded, so in this regard
> --no-substitutes is honored.

It depends on the intent of --no-substitutes. If the intent is to avoid
trusting the substitute server, then by relying on the accuracy of the
runtime dependency data from Hydra, we are failing to honor that intent.

That said, I think it's okay to document that --no-substitutes alone is
not sufficient to avoid trusting a substitute server, and that the
proper way to accomplish that is to make sure its key is not in
/etc/guix/acls.

What do you think?

Thanks,
Mark
L
L
Ludovic Courtès wrote on 15 Mar 2016 22:50
(name . Mark H Weaver)(address . mhw@netris.org)
87twk7qulu.fsf@gnu.org
Mark H Weaver <mhw@netris.org> skribis:

Toggle quote (34 lines)
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Alex Kost <alezost@gmail.com> skribis:
>>
>>> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>>>
>>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>>>
>>>> 1. Fetching substitute info about the things being built so that it
>>>> can determine its references, which in turns allows it to determine
>>>> whether they need to be grafted.
>>>>
>>>> 2. Building stuff, as a last resort, so that it can determine its
>>>> references.
>>>
>>> I noticed that #1 is happening even with --no-substitutes option. Is it
>>> intended?
>>
>> Not really, but I see this is because ‘substitutable-path-info’ (called
>> from ‘references/substitutes’, called from ‘graft-derivation’) works
>> regardless of whether substitutes are enabled:
>>
>> scheme@(guile-user)> ,use(guix)
>> scheme@(guile-user)> (define s (open-connection))
>> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
>> $2 = #t
>> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
>> $3 = #f
>> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
>> $4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)
>
> Is the information from the substitute server authenticated by checking
> hydra's signature against the list of keys in /etc/guix/acls?

Yes.

Toggle quote (12 lines)
>> However, substitutes are not downloaded, so in this regard
>> --no-substitutes is honored.
>
> It depends on the intent of --no-substitutes. If the intent is to avoid
> trusting the substitute server, then by relying on the accuracy of the
> runtime dependency data from Hydra, we are failing to honor that intent.
>
> That said, I think it's okay to document that --no-substitutes alone is
> not sufficient to avoid trusting a substitute server, and that the
> proper way to accomplish that is to make sure its key is not in
> /etc/guix/acls.

The sysadmin gets to choose which principals are trusted; unprivileged
users can only shrink this set.

The weird thing is that in this case, passing --substitute-urls='' on
the client side would effectively disable substitutes entirely.

Toggle quote (2 lines)
> What do you think?

We could augment the doc for --no-substitutes, I guess we should first
document that grafting relies on server-provided info.

Ludo’.
L
L
Ludovic Courtès wrote on 9 Dec 2016 21:48
control message for bug #25137
(address . control@debbugs.gnu.org)
87shpw7rsr.fsf@gnu.org
merge 25137 22990
L
L
Ludovic Courtès wrote on 8 Jan 2017 22:44
Re: bug#22990: Grafts leads to inefficient substitute info retrieval
(address . 22990@debbugs.gnu.org)
87k2a5rzue.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (26 lines)
> Concretely, what this means is this:
>
> $ guix package -u
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> […]
> The following files would be downloaded:
>
> Each of the initial “updating list” message corresponds to an HTTP
> request for a single narinfo file, which can take around 1 second.
>
> Instead, the ideal thing would be to fetch the narinfo files for all the
> relevant packages at once; that way, we’d spawn ‘guix substitute’ only
> once, and it would benefit from HTTP pipelining (one round-trip instead
> of N.)
>
> To achieve this, I’m thinking of extending gexp code such that gexp
> compilers can return a list of applicable grafts. The ‘package’
> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
> ‘graft-derivation’.

The ‘wip-gexp-grafts’ branch does that. Namely, it’s possible to know
what grafts would apply to a gexp derivation build, so that one can
first build ungrafted, and then apply the grafts to the results.
So for a profile, we’d first build the profile as is, and only then
would we graft it.

Right now the tip of this branch is a hack such that ‘guix package’:

1. Builds the original (ungrafted) derivation of the profile;

2. Manually calls ‘graft-derivation’ on that, passing it the list of
applicable grafts.

Conceptually it’s what we want to do, but the drawback is that the
caller (here ‘guix package’) goes through a lot of hops to get the list
of grafts and to apply it.

I think we should instead have a way to annotate a derivation with a
list of grafts, as well as a procedure to build that derivations in two
phases (first the original derivation, then the grafts).

To be continued…

Ludo’.
L
L
Ludovic Courtès wrote on 9 Jan 2017 23:55
(address . 22990@debbugs.gnu.org)
877f633kt1.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (2 lines)
> ludo@gnu.org (Ludovic Courtès) skribis:

[...]

Toggle quote (26 lines)
>> To achieve this, I’m thinking of extending gexp code such that gexp
>> compilers can return a list of applicable grafts. The ‘package’
>> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
>> ‘graft-derivation’.
>
> The ‘wip-gexp-grafts’ branch does that. Namely, it’s possible to know
> what grafts would apply to a gexp derivation build, so that one can
> first build ungrafted, and then apply the grafts to the results.
> So for a profile, we’d first build the profile as is, and only then
> would we graft it.
>
> Right now the tip of this branch is a hack such that ‘guix package’:
>
> 1. Builds the original (ungrafted) derivation of the profile;
>
> 2. Manually calls ‘graft-derivation’ on that, passing it the list of
> applicable grafts.
>
> Conceptually it’s what we want to do, but the drawback is that the
> caller (here ‘guix package’) goes through a lot of hops to get the list
> of grafts and to apply it.
>
> I think we should instead have a way to annotate a derivation with a
> list of grafts, as well as a procedure to build that derivations in two
> phases (first the original derivation, then the grafts).

The current iteration introduces “build continuation”: a derivation can
be annotated with a continuation, and the ‘build-things’ procedure will
loop over continuations and return the final results (that only works
for derivations directly passed as an argument to ‘build-things’.)

Example (where Guile 2.0.12 replaced by 2.0.13):

Toggle snippet (16 lines)
$ ./pre-inst-env guix package -p foo -i guile-json guile-ssh gdb
The following packages will be installed:
guile-json 0.5.0 /gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0
guile-ssh 0.10.2 /gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2
gdb 7.12 /gnu/store/d2a4lmycc13ssdf47a9h410knlfqqq41-gdb-7.12

applying 6 grafts to /gnu/store/s7w6r9ih52kdzcl7kprf2rq48s69zh98-profile
3 packages in profile
The following environment variable definitions may be needed:
export PATH="foo/bin${PATH:+:}$PATH"
$ guix gc -R foo | grep guile
/gnu/store/7fisf4frrgsjzmknjbab1dal23wxrp8d-guile-2.0.13
/gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2
/gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0

That seems to be a good model.

Now we must make sure that it works also for ‘guix build foo’ and ‘guix
system build’.

Ludo’.
L
L
Ludovic Courtès wrote on 11 Jan 2017 10:54
(address . 22990@debbugs.gnu.org)
874m15yl99.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (37 lines)
> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> ludo@gnu.org (Ludovic Courtès) skribis:
>
> [...]
>
>>> To achieve this, I’m thinking of extending gexp code such that gexp
>>> compilers can return a list of applicable grafts. The ‘package’
>>> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
>>> ‘graft-derivation’.
>>
>> The ‘wip-gexp-grafts’ branch does that. Namely, it’s possible to know
>> what grafts would apply to a gexp derivation build, so that one can
>> first build ungrafted, and then apply the grafts to the results.
>> So for a profile, we’d first build the profile as is, and only then
>> would we graft it.
>>
>> Right now the tip of this branch is a hack such that ‘guix package’:
>>
>> 1. Builds the original (ungrafted) derivation of the profile;
>>
>> 2. Manually calls ‘graft-derivation’ on that, passing it the list of
>> applicable grafts.
>>
>> Conceptually it’s what we want to do, but the drawback is that the
>> caller (here ‘guix package’) goes through a lot of hops to get the list
>> of grafts and to apply it.
>>
>> I think we should instead have a way to annotate a derivation with a
>> list of grafts, as well as a procedure to build that derivations in two
>> phases (first the original derivation, then the grafts).
>
> The current iteration introduces “build continuation”: a derivation can
> be annotated with a continuation, and the ‘build-things’ procedure will
> loop over continuations and return the final results (that only works
> for derivations directly passed as an argument to ‘build-things’.)

On second thought, the whole idea of applying grafts on the final result
(instead of applying grafts at each step like we do now) doesn’t fly.
It works well for things like a profile or the system derivation, but
breaks for less trivial things.

For example, if you’re building a VM image or a binary tarball, you
really need to graft packages early on; trying to graft the VM image or
binary tarball wouldn’t have the desired effect.

I’ve pushed ‘wip-gexp-grafts’ again but it seems to be a dead end.

Ludo’.
D
D
David Craven wrote on 11 Jan 2017 11:51
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 22990@debbugs.gnu.org)
CAL1_imnWoGFQ=NBjghV1wracigF676j4zomgCsfCTPO+JiPbKQ@mail.gmail.com
Toggle quote (9 lines)
> On second thought, the whole idea of applying grafts on the final result
> (instead of applying grafts at each step like we do now) doesn’t fly.
> It works well for things like a profile or the system derivation, but
> breaks for less trivial things.

> For example, if you’re building a VM image or a binary tarball, you
> really need to graft packages early on; trying to graft the VM image or
> binary tarball wouldn’t have the desired effect.

Isn't a system derivation or a profile derivation an intermediate step
to these derivations? Can't there be a flag or something called
#:already-grafted? #t?
L
L
Ludovic Courtès wrote on 11 Jan 2017 22:19
(name . David Craven)(address . david@craven.ch)(address . 22990@debbugs.gnu.org)
87k2a1wazs.fsf@gnu.org
David Craven <david@craven.ch> skribis:

Toggle quote (12 lines)
>> On second thought, the whole idea of applying grafts on the final result
>> (instead of applying grafts at each step like we do now) doesn’t fly.
>> It works well for things like a profile or the system derivation, but
>> breaks for less trivial things.
>
>> For example, if you’re building a VM image or a binary tarball, you
>> really need to graft packages early on; trying to graft the VM image or
>> binary tarball wouldn’t have the desired effect.
>
> Isn't a system derivation or a profile derivation an intermediate step
> to these derivations?

Yes, you’re right. However the implementation I had come up with relied
on “build continuations”, which only worked for the “top-level”
derivations (those you pass to ‘build-derivations’.)

Toggle quote (2 lines)
> Can't there be a flag or something called #:already-grafted? #t?

Yeah, we need something like that. I need to chew a bit more on this to
find a nice way to achieve that.

Thanks for your feedback, I feel less lonely now. :-)

Ludo’.
L
L
Ludovic Courtès wrote on 27 May 2017 12:03
control message for bug #22990
(address . control@debbugs.gnu.org)
87tw46vcor.fsf@gnu.org
severity 22990 important
L
L
Ludovic Courtès wrote on 29 Mar 2020 15:41
Re: bug#22990: Grafts leads to inefficient substitute info retrieval
(address . 22990-done@debbugs.gnu.org)
87tv271dob.fsf@gnu.org
Hi younger self!

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (32 lines)
> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>
> 1. Fetching substitute info about the things being built so that it
> can determine its references, which in turns allows it to determine
> whether they need to be grafted.
>
> 2. Building stuff, as a last resort, so that it can determine its
> references.
>
> Case #1 is hopefully going to be the most common.
>
> The problem with #1 is that when building a profile, we do one
> ‘package-derivation’ call for each package in the profile, which
> translates in one ‘graft-derivation’ call for each relevant package¹,
> which translates into one ‘references/substitutes’ call for each.
>
> Concretely, what this means is this:
>
> $ guix package -u
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
> […]
> The following files would be downloaded:
>
> Each of the initial “updating list” message corresponds to an HTTP
> request for a single narinfo file, which can take around 1 second.

This is finally fixed with commit
710854304b1ab29332edcb76f3de532e0724c197, as discussed earlier this
week¹. \o/

The solution is simpler and more orthogonal than what was envisioned
earlier in this thread, which is nice.

I also potentially addresses other uses of “dynamic dependencies”
elsewhere in the code base, such as ‘remote-eval’ calls in ‘guix
deploy’.

Feedback welcome!

Ludo’.

Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 22990@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 22990
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch