guix weather not checking all substitute urls

Done

Details

3 participants

Efraim Flashner
Ludovic Courtès
zimoun

Owner: unassigned

Submitted by: Efraim Flashner

Severity: normal

Efraim Flashner wrote on 11 Nov 2020 12:30

Recipients:(address . bug-guix@gnu.org)

Message-ID:20201111113005.GB12414@E5400

Attachment: file

zimoun wrote on 16 Nov 2020 18:12

Recipients:

Message-ID:86ft59ql7t.fsf@gmail.com

Hi Efraim,

On Wed, 11 Nov 2020 at 13:30, Efraim Flashner <efraim@flashner.co.il> wrote:

Toggle quote (5 lines)

> I would have assumed that it would check for all of the substitute urls

> I listed and not just the first one. Most of the packages listed below

> I've test built on my pine64 so they shouldn't appear in the list

> anymore.

I think #44688 [1] fixes it. Please let me know.

1: http://issues.guix.gnu.org/issue/44688

Examples:

Toggle snippet (37 lines)$ ./pre-inst-env guix weather --substitute-urls="https://bayfront.guix.gnu.org https://ci.guix.gnu.org"
computing 15,391 package derivations for x86_64-linux...
looking for 16,707 store items on https://bayfront.guix.gnu.org...
updating substitutes from 'https://bayfront.guix.gnu.org'... 100.0%
https://bayfront.guix.gnu.org
  55.6% substitutes available (9,289 out of 16,707)
  at least 51,869.2 MiB of nars (compressed)
  82,986.2 MiB on disk (uncompressed)
  0.001 seconds per request (24.6 seconds in total)
  679.0 requests per second

  0.3% (20 out of 7,418) of the missing items are queued
  32 queued builds
      x86_64-linux: 32 (100.0%)
  build rate: 13.83 builds per hour
      x86_64-linux: 13.83 builds per hour
looking for 16,707 store items on https://ci.guix.gnu.org...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
https://ci.guix.gnu.org
  88.9% substitutes available (14,860 out of 16,707)
  at least 78,929.4 MiB of nars (compressed)
  130,797.2 MiB on disk (uncompressed)
  0.002 seconds per request (26.9 seconds in total)
  620.2 requests per second

  0.5% (10 out of 1,847) of the missing items are queued
  at least 1,000 queued builds
      x86_64-linux: 493 (49.3%)
      aarch64-linux: 140 (14.0%)
      i686-linux: 346 (34.6%)
      armhf-linux: 21 (2.1%)
  build rate: 38.98 builds per hour
      x86_64-linux: 13.29 builds per hour
      i686-linux: 12.32 builds per hour
      aarch64-linux: 6.78 builds per hour
      armhf-linux: 6.59 builds per hour

Toggle snippet (55 lines)$ ./pre-inst-env guix weather --substitute-urls="https://guix.tobias.gr https://ci.guix.gnu.org" -c 100
computing 15,391 package derivations for x86_64-linux...
looking for 16,707 store items on https://guix.tobias.gr...
updating substitutes from 'https://guix.tobias.gr'... 100.0%
https://guix.tobias.gr
  35.0% substitutes available (5,855 out of 16,707)
  unknown substitute sizes
  52,230.7 MiB on disk (uncompressed)
  0.017 seconds per request (291.8 seconds in total)
  57.3 requests per second
  (continuous integration information unavailable)
updating substitutes from 'https://guix.tobias.gr'... 100.0%
6777 packages are missing from 'https://guix.tobias.gr' for 'x86_64-linux', among which:
  16564	bootar@1a	/gnu/store/x2iy3ka299sk1x97an783jl4pfbn60pj-bootar-1a 
  16561	bootstrap-mes-rewired@0.19	/gnu/store/a4m8c80mrys8w3vlm6vdnq05c7a1warl-bootstrap-mes-rewired-0.19 
  16495	ld-wrapper-boot3@0	/gnu/store/v3wwjj9i9zqglq4g2yczf6cxsh028qfb-ld-wrapper-boot3-0 
  9743	bison@3.5.3	/gnu/store/rxli27pbpr9x9c8mk9cbp6masnwaszrs-bison-3.5.3 
  1831	xkbcomp-intermediate@1.4.3	/gnu/store/fjwjb14b5irngvv9r06dmkhhds566vw2-xkbcomp-intermediate-1.4.3 
   788	bootar@1a	/gnu/store/x2iy3ka299sk1x97an783jl4pfbn60pj-bootar-1a 
   785	bootstrap-mes-rewired@0.19	/gnu/store/a4m8c80mrys8w3vlm6vdnq05c7a1warl-bootstrap-mes-rewired-0.19 
   719	ld-wrapper-boot3@0	/gnu/store/v3wwjj9i9zqglq4g2yczf6cxsh028qfb-ld-wrapper-boot3-0 
   595	bootar@1a	/gnu/store/x2iy3ka299sk1x97an783jl4pfbn60pj-bootar-1a 
   592	bootstrap-mes-rewired@0.19	/gnu/store/a4m8c80mrys8w3vlm6vdnq05c7a1warl-bootstrap-mes-rewired-0.19 
   526	ld-wrapper-boot3@0	/gnu/store/v3wwjj9i9zqglq4g2yczf6cxsh028qfb-ld-wrapper-boot3-0 
   339	bison@3.5.3	/gnu/store/rxli27pbpr9x9c8mk9cbp6masnwaszrs-bison-3.5.3 
   305	go@1.4-bootstrap-20171003	/gnu/store/asw6676iy7i0a8azpra2870w5hkq7bvv-go-1.4-bootstrap-20171003-doc /gnu/store/j20pik1y8psi78v7zb5ksqybn198inx7-go-1.4-bootstrap-20171003 /gnu/store/csfqsyyysbg9afnck405wmkhrlrblmza-go-1.4-bootstrap-20171003-tests 
   211	xkbcomp-intermediate@1.4.3	/gnu/store/fjwjb14b5irngvv9r06dmkhhds566vw2-xkbcomp-intermediate-1.4.3 
   121	bison@3.5.3	/gnu/store/rxli27pbpr9x9c8mk9cbp6masnwaszrs-bison-3.5.3 
   113	r-codetools@0.2-16	/gnu/store/k217j4c4ra2hfdldva075dmvq89ww1dn-r-codetools-0.2-16 
looking for 16,707 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  88.9% substitutes available (14,860 out of 16,707)
  at least 78,929.4 MiB of nars (compressed)
  130,797.2 MiB on disk (uncompressed)
  0.001 seconds per request (10.1 seconds in total)
  1,650.0 requests per second

  0.5% (10 out of 1,847) of the missing items are queued
  at least 1,000 queued builds
      x86_64-linux: 493 (49.3%)
      aarch64-linux: 140 (14.0%)
      i686-linux: 346 (34.6%)
      armhf-linux: 21 (2.1%)
  build rate: 38.70 builds per hour
      x86_64-linux: 13.20 builds per hour
      i686-linux: 12.23 builds per hour
      aarch64-linux: 6.73 builds per hour
      armhf-linux: 6.54 builds per hour
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
1251 packages are missing from 'https://ci.guix.gnu.org' for 'x86_64-linux', among which:
   112	java-snappy@1.1.7.5	/gnu/store/mq1xagq5sn9mfhpfg7j2cn1dn4vsds72-java-snappy-1.1.7.5 

compared to the current behaviour:

Toggle snippet (18 lines)$ guix weather --substitute-urls="https://bayfront.guix.gnu.org https://ci.guix.gnu.org"
computing 15,287 package derivations for x86_64-linux...
looking for 16,584 store items on https://bayfront.guix.gnu.org...
updating substitutes from 'https://bayfront.guix.gnu.org'... 100.0%
https://bayfront.guix.gnu.org
  54.4% substitutes available (9,026 out of 16,584)
  at least 49,365.8 MiB of nars (compressed)
  81,792.9 MiB on disk (uncompressed)
  0.003 seconds per request (41.6 seconds in total)
  398.0 requests per second

  0.1% (10 out of 7,558) of the missing items are queued
  32 queued builds
      x86_64-linux: 32 (100.0%)
  build rate: 13.84 builds per hour
      x86_64-linux: 13.84 builds per hour

(Note that Bayfront is working hard. ;-))

All the best,

simon

Efraim Flashner wrote on 16 Nov 2020 20:30

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 44574@debbugs.gnu.org)

Message-ID:20201116193011.GP987@E5400

Attachment: file

zimoun wrote on 16 Nov 2020 21:41

Recipients:(name . Efraim Flashner)(address . efraim@flashner.co.il)(address . 44574@debbugs.gnu.org)

Message-ID:864klpqbji.fsf@gmail.com

Hi Efraim,

On Mon, 16 Nov 2020 at 21:30, Efraim Flashner <efraim@flashner.co.il> wrote:

Toggle quote (14 lines)>> > I would have assumed that it would check for all of the substitute urls
>> > I listed and not just the first one. Most of the packages listed below
>> > I've test built on my pine64 so they shouldn't appear in the list
>> > anymore.
>> 
>> I think #44688 [1] fixes it.  Please let me know.
>> 
>> 1: <http://issues.guix.gnu.org/issue/44688>
>
> I had hoped for a unified list of the missing packages between the two
> or more build machines but the separate output is also useful. With a
> unified list it's possible to see which packages are missing overall and
> need fixing/building.

Ah, this a feature request. ;-) I agree it could be really helpful.

Cheers,

simon

Efraim Flashner wrote on 16 Nov 2020 21:49

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 44574@debbugs.gnu.org)

Message-ID:20201116204954.GR987@E5400

On Mon, Nov 16, 2020 at 09:41:53PM +0100, zimoun wrote:

Toggle quote (21 lines)> Hi Efraim,
> 
> On Mon, 16 Nov 2020 at 21:30, Efraim Flashner <efraim@flashner.co.il> wrote:
> 
> >> > I would have assumed that it would check for all of the substitute urls
> >> > I listed and not just the first one. Most of the packages listed below
> >> > I've test built on my pine64 so they shouldn't appear in the list
> >> > anymore.
> >> 
> >> I think #44688 [1] fixes it.  Please let me know.
> >> 
> >> 1: <http://issues.guix.gnu.org/issue/44688>
> >
> > I had hoped for a unified list of the missing packages between the two
> > or more build machines but the separate output is also useful. With a
> > unified list it's possible to see which packages are missing overall and
> > need fixing/building.
> 
> Ah, this a feature request. ;-)  I agree it could be really helpful.
> 

Lets merge 44688 anyway, it does address something that's lacking.

-- 
Efraim Flashner   <efraim@flashner.co.il>   ????? ?????
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

zimoun wrote on 17 Nov 2020 17:01

Recipients:(address . 44574@debbugs.gnu.org)

Message-ID:86k0uknfau.fsf@gmail.com

Hi,

On Mon, 16 Nov 2020 at 22:49, Efraim Flashner <efraim@flashner.co.il> wrote:

Toggle quote (11 lines)

>> >> > I would have assumed that it would check for all of the substitute urls

>> >> > I listed and not just the first one. Most of the packages listed below

>> >> > I've test built on my pine64 so they shouldn't appear in the list

>> >> > anymore.

>> >>

>> >> I think #44688 [1] fixes it. Please let me know.

>> >>

>> >> 1: <http://issues.guix.gnu.org/issue/44688>

> Lets merge 44688 anyway, it does address something that's lacking.

Ludo, does #44688 go to version-1.2.0 or master? It is simple patch

that fixes:

guix weather \

--substitute-urls=“https://bayfront.guix.gnu.orghttps://ci.guix.gnu.org”

where the current behaviour process only one server.

Cheers,

simon

Ludovic Courtès wrote on 17 Nov 2020 21:45

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87h7pnpv95.fsf@gnu.org

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (21 lines)> On Mon, 16 Nov 2020 at 22:49, Efraim Flashner <efraim@flashner.co.il> wrote:
>
>>> >> > I would have assumed that it would check for all of the substitute urls
>>> >> > I listed and not just the first one. Most of the packages listed below
>>> >> > I've test built on my pine64 so they shouldn't appear in the list
>>> >> > anymore.
>>> >> 
>>> >> I think #44688 [1] fixes it.  Please let me know.
>>> >> 
>>> >> 1: <http://issues.guix.gnu.org/issue/44688>
>
>> Lets merge 44688 anyway, it does address something that's lacking.
>
> Ludo, does #44688 go to version-1.2.0 or master?  It is simple patch
> that fixes:
>
>   guix weather \
>        --substitute-urls=“https://bayfront.guix.gnu.org https://ci.guix.gnu.org”
>
> where the current behaviour process only one server.

It’s OK for ‘version-1.2.0’. Thank you!

Ludo’.

Ludovic Courtès wrote on 3 Dec 2020 18:07

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87y2iebyxs.fsf@gnu.org

This was fixed by c2cc5f0a006fd7797039dae24274833118ea7b47, closing!

Closed

zimoun wrote on 4 Dec 2020 10:35

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:86wnxxyku9.fsf@gmail.com

Hi Ludo,

On Thu, 03 Dec 2020 at 18:07, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)

> This was fixed by c2cc5f0a006fd7797039dae24274833118ea7b47, closing!

Thanks to close this forgotten one. But it does not count for the

“December old bug squashing“ effort. ;-)

Cheers,

simon

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 44574@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date