substitute servers should be preferred according to their coverage rate

  • Open
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
normal
Merged with
M
M
Maxim Cournoyer wrote on 29 Oct 2021 04:07
(name . bug-guix)(address . bug-guix@gnu.org)
87k0hwa8ic.fsf@gmail.com
Hello,

When using substitute servers discovery, I've noticed that if one of the
substitute servers doesn't have any substitutes available, it'll keep
getting tried instead of others, leading to a slide-show of substitutes
updates such as:

Toggle snippet (39 lines)
normalized load on machine '127.0.0.1' is 0.04
building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
normalized load on machine '127.0.0.1' is 0.04
building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://127.0.0.1:8080'... 100.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://127.0.0.1:8080'... 100.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%

We should implement some scheme to prefer querying high-substitute
servers first, instead of wasting time querying servers always failed
queries; this would greatly improve performance when using substitute
discovery for example combined with low coverage.

Thanks!

Maxim
L
L
Ludovic Courtès wrote on 7 Nov 2021 16:11
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 51472@debbugs.gnu.org)
874k8oknlj.fsf@gnu.org
Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (16 lines)
> When using substitute servers discovery, I've noticed that if one of the
> substitute servers doesn't have any substitutes available, it'll keep
> getting tried instead of others, leading to a slide-show of substitutes
> updates such as:
>
> normalized load on machine '127.0.0.1' is 0.04
> building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
> process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
> normalized load on machine '127.0.0.1' is 0.04
> building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%

We’d need to check why this particular server is checked repeatedly.
The fact that it displays “0.0%” doesn’t mean that the server lacks
substitutes, but that it does not reply to ‘GET /xyz.narinfo’ requests,
for example because it’s off-line (see

Toggle quote (5 lines)
> We should implement some scheme to prefer querying high-substitute
> servers first, instead of wasting time querying servers always failed
> queries; this would greatly improve performance when using substitute
> discovery for example combined with low coverage.

There are several problems with that. First one is that you can’t tell
what substitute coverage is until you’ve actually made those GET
requests. Second one is that substitute coverage varies and it’s not an
absolute measure; for example, if a server provides substitutes for only
0.1% of all the packages, but that’s precisely the 0.1% you care about,
it’s more valuable than the one that has 99% of the packages but lacks
those you want.

There are other issues such as the fact that current semantics is to
respect the order of substitute URLs, which is presumably chosen by the
user according to their own criteria: download speed, bandwidth usage,
etc.

I hope this makes sense!

Ludo’.
M
M
Maxim Cournoyer wrote on 11 Jan 2022 04:43
(name . Ludovic Courtès)(address . ludo@gnu.org)
87pmozndi2.fsf@gmail.com
merge 48808 51472
thanks

Hello Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (46 lines)
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> When using substitute servers discovery, I've noticed that if one of the
>> substitute servers doesn't have any substitutes available, it'll keep
>> getting tried instead of others, leading to a slide-show of substitutes
>> updates such as:
>>
>> normalized load on machine '127.0.0.1' is 0.04
>> building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
>> process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
>> normalized load on machine '127.0.0.1' is 0.04
>> building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>
> We’d need to check why this particular server is checked repeatedly.
> The fact that it displays “0.0%” doesn’t mean that the server lacks
> substitutes, but that it does not reply to ‘GET /xyz.narinfo’ requests,
> for example because it’s off-line (see
> <https://issues.guix.gnu.org/48808>.)
>
>> We should implement some scheme to prefer querying high-substitute
>> servers first, instead of wasting time querying servers always failed
>> queries; this would greatly improve performance when using substitute
>> discovery for example combined with low coverage.
>
> There are several problems with that. First one is that you can’t tell
> what substitute coverage is until you’ve actually made those GET
> requests. Second one is that substitute coverage varies and it’s not an
> absolute measure; for example, if a server provides substitutes for only
> 0.1% of all the packages, but that’s precisely the 0.1% you care about,
> it’s more valuable than the one that has 99% of the packages but lacks
> those you want.
>
> There are other issues such as the fact that current semantics is to
> respect the order of substitute URLs, which is presumably chosen by the
> user according to their own criteria: download speed, bandwidth usage,
> etc.
>
> I hope this makes sense!

It does! I agree that it'd be tricky to get this right; makes me
realize that my problem is probably due to #48808, and fixing that one
would probably have avoided that bug report :-).

I'm merging this one with 48808.

Thank you!

Maxim
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 51472@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 51472
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch