nar 404 leads to hard ‘guix substitute’ crash

Done

Details

3 participants

Ludovic Courtès
Christopher Baines
Simon Tournier

Owner: unassigned

Submitted by: Ludovic Courtès

Severity: important

Ludovic Courtès wrote on 22 May 2023 00:12

nar 404 leads to hard ‘guix substitute’ crash

Recipients:(address . bug-guix@gnu.org)

Message-ID:87cz2ttju2.fsf@inria.fr

Like this:

Toggle snippet (63 lines) gst-plugins-bad-minimal-1.22.2  650KiB                 6.8MiB/s 00:00 ???????????????????? 100.0%
 guile-bytestructures-1.0.10  222KiB                    4.1MiB/s 00:00 ???????????????????? 100.0%
Backtrace:
In ice-9/boot-9.scm:
  1752:10 19 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          18 (apply-smob/0 #<thunk 7f6df0f172a0>)
In ice-9/boot-9.scm:
    724:2 17 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8 16 (_ #(#(#<directory (guile-user) 7f6df0f1cc80>)))
In guix/ui.scm:
   2300:7 15 (run-guix . _)
  2263:10 14 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1752:10 12 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
   854:15 11 (_)
    646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    463:7  8 (download-nar #<<narinfo> path: "/gnu/store/dg0rm1mp9q…> …)
In ice-9/boot-9.scm:
  1747:15  7 (with-exception-handler #<procedure 7f6df0fa71e0 at ic…> …)
  1685:16  6 (raise-exception _ #:continuable? _)
  1683:16  5 (raise-exception _ #:continuable? _)
  1685:16  4 (raise-exception _ #:continuable? _)
  1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1685:16  2 (raise-exception _ #:continuable? _)
  1683:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &http-get-error:
      uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1" query: #f fragment: #f>
      code: 404
      reason: "Not Found"
      headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 39 minute: 59 hour: 21 day: 21 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
  2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1: HTTP download failed: 404 (\"Not Found\")"
substitution of /gnu/store/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1 failed
guix system: error: corrupt input while restoring archive from #<closed: file 7f0e7113b3f0>
$ guix describe
Generation 263  May 21 2023 23:56:04    (current)
  guix c5bc698
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: c5bc698e8922d78ed85989985cc2ceb034de2f23
  guile 36fd2b4
    repository URL: https://git.savannah.gnu.org/git/guile.git
    branch: main
    commit: 36fd2b4920ae926c79b936c29e739e71a6dff2bc
  shepherd 954742c
    repository URL: https://git.savannah.gnu.org/git/shepherd.git
    branch: master
    commit: 954742cd0d066ac4be7de4a031d7e229fd445c5d
$ guix package -p /run/current-system/profile -I guix
guix      	1.4.0-6.dc5430c	out	/gnu/store/cgjddvw9zay626z8hyxl0zmn1354c24k-guix-1.4.0-6.dc5430c
guix-icons	0.1            	out	/gnu/store/arcpwjvhp3ws1i6g1sdnfxxrxc3zq4m9-guix-icons-0.1

This might be related to 8bd4126917f59f4af9a4323c3d5699201862dca2. The

two ‘download-nar’ calls are guarded against ‘http-get-error?’ though.

Ludo’.

Ludovic Courtès wrote on 22 May 2023 09:59

control message for bug #63634

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87o7mczthk.fsf@gnu.org

severity 63634 important

quit

Ludovic Courtès wrote on 22 May 2023 10:00

Re: bug#63634: nar 404 leads to hard ‘guix subs titute’ crash

Recipients:(address . 63634@debbugs.gnu.org)

Message-ID:87jzx0ztgi.fsf@gnu.org

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (25 lines)> In guix/scripts/substitute.scm:
>    854:15 11 (_)
>     646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
> In ice-9/boot-9.scm:
>   1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
> In guix/scripts/substitute.scm:
>     463:7  8 (download-nar #<<narinfo> path: "/gnu/store/dg0rm1mp9q…> …)
> In ice-9/boot-9.scm:
>   1747:15  7 (with-exception-handler #<procedure 7f6df0fa71e0 at ic…> …)
>   1685:16  6 (raise-exception _ #:continuable? _)
>   1683:16  5 (raise-exception _ #:continuable? _)
>   1685:16  4 (raise-exception _ #:continuable? _)
>   1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
>   1685:16  2 (raise-exception _ #:continuable? _)
>   1683:16  1 (raise-exception _ #:continuable? _)
>   1685:16  0 (raise-exception _ #:continuable? _)
>
> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> ERROR:
>   1. &http-get-error:
>       uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1" query: #f fragment: #f>
>       code: 404
>       reason: "Not Found"
>       headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 39 minute: 59 hour: 21 day: 21 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))

Possibly duplicate of https://issues.guix.gnu.org/55820.

Simon Tournier wrote on 22 May 2023 10:19

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 63634@debbugs.gnu.org)

Message-ID:87ttw4ye13.fsf@gmail.com

Hi,

On lun., 22 mai 2023 at 00:12, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (3 lines)

> This might be related to 8bd4126917f59f4af9a4323c3d5699201862dca2. The

> two ‘download-nar’ calls are guarded against ‘http-get-error?’ though.

Well, I am currently hitting something very similar.

Toggle snippet (42 lines)$ guix build rust-base64@0.13 -S
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0,0 MB will be downloaded:
  /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
substituting /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz...
Backtrace:
          16 (primitive-load "/gnu/store/jpcpva9gg7l337d0icn88pf76l4…")
In guix/ui.scm:
   2263:7 15 (run-guix . _)
  2226:10 14 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1752:10 12 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
   856:15 11 (_)
    650:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    467:7  8 (download-nar #<<narinfo> path: "/gnu/store/sx6sr6cs1x…> …)
In ice-9/boot-9.scm:
  1747:15  7 (with-exception-handler #<procedure 7f6a0ca624e0 at ic…> …)
  1685:16  6 (raise-exception _ #:continuable? _)
  1683:16  5 (raise-exception _ #:continuable? _)
  1685:16  4 (raise-exception _ #:continuable? _)
  1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1685:16  2 (raise-exception _ #:continuable? _)
  1683:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &http-get-error:
      uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz" query: #f fragment: #f>
      code: 404
      reason: "Not Found"
      headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 28 minute: 9 hour: 8 day: 22 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
  2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz: HTTP download failed: 404 (\"Not Found\")"
substitution of /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz failed
guix build: error: some substitutes for the outputs of derivation `/gnu/store/qcpzwblzpa2wprc4vwlm128bwckfj8rj-rust-base64-0.13.1.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source

Then the option “--fallback” works.

Toggle quote (3 lines)

> The

> two ‘download-nar’ calls are guarded against ‘http-get-error?’ though.

Well, the fallback should happen transparently.

It rings this bell

https://issues.guix.gnu.org/msgid/f7db538ea324545f2e297b6a6cdcd937f889ef63.1683293896.git.dev@jpoiret.xyz.

Cheers,

simon

Ludovic Courtès wrote on 22 May 2023 11:32

Recipients:(address . 63634@debbugs.gnu.org)(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87a5xwzp6w.fsf@gnu.org

I experienced it a second time and have more info:

Toggle snippet (45 lines)substituting /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0...
Backtrace:
In ice-9/boot-9.scm:
  1752:10 19 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          18 (apply-smob/0 #<thunk 7f4f9bf172a0>)
In ice-9/boot-9.scm:
    724:2 17 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8 16 (_ #(#(#<directory (guile-user) 7f4f9bf1cc80>)))
In guix/ui.scm:
   2300:7 15 (run-guix . _)
  2263:10 14 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1752:10 12 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
   854:15 11 (_)
    646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    463:7  8 (download-nar #<<narinfo> path: "/gnu/store/apw1y9nf8r…> …)
In ice-9/boot-9.scm:
  1747:15  7 (with-exception-handler #<procedure 7f4f9846af00 at ic…> …)
  1685:16  6 (raise-exception _ #:continuable? _)
  1683:16  5 (raise-exception _ #:continuable? _)
  1685:16  4 (raise-exception _ #:continuable? _)
  1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1685:16  2 (raise-exception _ #:continuable? _)
  1683:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &http-get-error:
      uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0" query: #f fragment: #f>
      code: 404
      reason: "Not Found"
      headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 31 minute: 59 hour: 8 day: 22 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
  2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0: HTTP download failed: 404 (\"Not Found\")"
substitution of /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 failed
guix build: error: corrupt input while restoring archive from #<closed: file 7f1693f9c9a0>

I have ‘--substitute-urls=https://ci.guix.gnu.orghttps://bordeaux.guix.gnu.org’.

Let’s see what’s available on each server:

Toggle snippet (48 lines)$ wget -O- https://ci.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
--2023-05-22 11:00:08--  https://ci.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-05-22 11:00:08 ERROR 404: Not Found.
$ wget -O- https://bordeaux.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
--2023-05-22 11:03:36--  https://bordeaux.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1715 (1.7K) [text/plain]
Saving to: ‘STDOUT’

-                                     0%[                                                                   ]       0  --.-KB/s               StorePath: /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
NarHash: sha256:1am3v254jr9dpgd9r4dkjqxfgbp1xm0vl1hxl49cmmh97dg117z8
NarSize: 144790464
References: 8rz7yh6zdzp4b78f4n9wqj3hav2md4d4-isl-0.24 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 brzrf6qx5d83j1zvirc1xk14wrhyx4hf-binutils-cross-aarch64-linux-gnu-2.38 cinj0g8krh5s428jawsazawlfarp912k-gcc-cross-aarch64-linux-gnu-11.3.0-lib cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0 gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1 rbjq50q2ik8c1glkj5f0ksnwfz64g16g-mpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 rjgpaj54q1q9n3hcvpzyp29fgi2nx6ls-ld-wrapper-aarch64-linux-gnu-0 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 y4ndp9n5krkaiwgvw58c61n7h8ls9b0f-glibc-cross-aarch64-linux-gnu-2.35
System: x86_64-linux
Deriver: fqb09jygm70cmq03xmjm7al6gzajp1p5-gcc-cross-aarch64-linux-gnu-11.3.0.drv
Signature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNzA2MTI1QUFGMjcxQzM2MDM3RkY2MTIwMzI0NzI2NTVDMkU3OEY2NTA3OUY0MzgxMzY4NDhDQzA0OTVFRTIxRiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDg1RTA2ODE3RTFENTdCQzE1NTAxOUNGNjA0QTQxMTlFRjNBREVGQ0VFOTE4NjQzMDIzQTExQzk4RDM3REVBOCMpCiAgIChzICMwMUIwMkQyNjI0MDhCMDIzMUMxMUVDODI1Q0Q2MTkzNTU5NDRCQUNBMzg4NjEwRDIxOEIzNTVCNjM1NjFCQjk3IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==
URL: nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Compression: lzip
FileSize: 24767829
-                                   100%[==================================================================>]   1.67K  --.-KB/s    in 0s      

2023-05-22 11:03:36 (20.6 MB/s) - written to stdout [1715/1715]

$ wget -O/dev/null https://bordeaux.guix.gnu.org/nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
--2023-05-22 11:04:02--  https://bordeaux.guix.gnu.org/nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24767829 (24M) [text/plain]
Saving to: ‘/dev/null’

/dev/null                           100%[==================================================================>]  23.62M  44.8MB/s    in 0.5s    

2023-05-22 11:04:03 (44.8 MB/s) - ‘/dev/null’ saved [24767829/24767829]

$ wget -O/dev/null https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
--2023-05-22 11:04:28--  https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-05-22 11:04:28 ERROR 404: Not Found.

So it seems ‘guix substitute’ picked /nar/zstd, even though

bordeaux.guix is not advertising that.

Or is it?

Toggle snippet (4 lines)

$ sudo cat /var/guix/substitute/cache/kzwjeblndsbkjzmjailrt4bnhguil7tqjmewzcyw22hgajbhfy3q/apw1y9nf8rqgxvjnlr1isbhpd502bcs5

(narinfo (version 2) (cache-uri "https://bordeaux.guix.gnu.org") (date 1683633757) (ttl 15552000) (value "StorePath: /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nNarHash: sha256:1am3v254jr9dpgd9r4dkjqxfgbp1xm0vl1hxl49cmmh97dg117z8\nNarSize: 144790464\nReferences: 8rz7yh6zdzp4b78f4n9wqj3hav2md4d4-isl-0.24 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 brzrf6qx5d83j1zvirc1xk14wrhyx4hf-binutils-cross-aarch64-linux-gnu-2.38 cinj0g8krh5s428jawsazawlfarp912k-gcc-cross-aarch64-linux-gnu-11.3.0-lib cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0 gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1 rbjq50q2ik8c1glkj5f0ksnwfz64g16g-mpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 rjgpaj54q1q9n3hcvpzyp29fgi2nx6ls-ld-wrapper-aarch64-linux-gnu-0 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 y4ndp9n5krkaiwgvw58c61n7h8ls9b0f-glibc-cross-aarch64-linux-gnu-2.35\nSystem: x86_64-linux\nDeriver: fqb09jygm70cmq03xmjm7al6gzajp1p5-gcc-cross-aarch64-linux-gnu-11.3.0.drv\nSignature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNzA2MTI1QUFGMjcxQzM2MDM3RkY2MTIwMzI0NzI2NTVDMkU3OEY2NTA3OUY0MzgxMzY4NDhDQzA0OTVFRTIxRiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDg1RTA2ODE3RTFENTdCQzE1NTAxOUNGNjA0QTQxMTlFRjNBREVGQ0VFOTE4NjQzMDIzQTExQzk4RDM3REVBOCMpCiAgIChzICMwMUIwMkQyNjI0MDhCMDIzMUMxMUVDODI1Q0Q2MTkzNTU5NDRCQUNBMzg4NjEwRDIxOEIzNTVCNjM1NjFCQjk3IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==\nURL: nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: lzip\nFileSize: 24767829\nURL: nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: zstd\nFileSize: 39822104"))ludo@ribbon ~/src/guix [env]$

So it seems that bordeaux.guix did advertise zstd at some point (and

that narinfo is still in cache) but no longer does.

Chris, can you confirm?

I’ll see how to get ‘guix substitute’ to handle that gracefully.

Ludo’.

Ludovic Courtès wrote on 22 May 2023 11:36

Recipients:(name . Simon Tournier)(address . zimon.toutoune@gmail.com)

Message-ID:875y8kzp17.fsf@gnu.org

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (2 lines)

> 2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz:HTTP download failed: 404 (\"Not Found\")"

Look:

Toggle snippet (13 lines)$ wget -qO- https://bordeaux.guix.gnu.org/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x.narinfo
StorePath: /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
NarHash: sha256:0pynnlsiiq03h7ysfkzw53wkf6hfmplb63haz6bjqdc686rnf4q1
NarSize: 49128
References: 
System: x86_64-linux
Deriver: qcpzwblzpa2wprc4vwlm128bwckfj8rj-rust-base64-0.13.1.tar.xz.drv
Signature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjRUE2MTcwNUZENjNCOTU1NzcwNjIxN0EyQjczN0U5NzI2RjM4Rjk1NkQwRTRCQjI4MjVBRTA2MTdDQjcxMDQ1MyMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDNGRjU3ODU1RTNERDMxQkJFMUY2NDEyMkU0MzNBNzcwMEZGNkI1ODExRUQ0NTBCQzVEQTcwMkQyNDk3ODZEMCMpCiAgIChzICMwN0ZCQjREN0YwNTExQTlBNzZENzBFNzUxQTc2NTM1MzE0MzU1Mzk1RUUwNzgzODU3REM2NEYxRDIxMTEzQjMxIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==
URL: nar/lzip/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
Compression: lzip
FileSize: 49781

See? zstd’s gone! This confirms my hypothesis.

BTW, ‘guix publish’ has provisions to not compress already-compressed

files, as in the case above. We should really share code, Chris.

Ludo’.

Christopher Baines wrote on 22 May 2023 12:59

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 63634@debbugs.gnu.org)

Message-ID:87fs7ofx3v.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (13 lines)> So it seems ‘guix substitute’ picked /nar/zstd, even though
> bordeaux.guix is not advertising that.
>
> Or is it?
>
> $ sudo cat /var/guix/substitute/cache/kzwjeblndsbkjzmjailrt4bnhguil7tqjmewzcyw22hgajbhfy3q/apw1y9nf8rqgxvjnlr1isbhpd502bcs5
> (narinfo (version 2) (cache-uri "https://bordeaux.guix.gnu.org") (date 1683633757) (ttl 15552000) (value "StorePath: /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nNarHash: sha256:1am3v254jr9dpgd9r4dkjqxfgbp1xm0vl1hxl49cmmh97dg117z8\nNarSize: 144790464\nReferences: 8rz7yh6zdzp4b78f4n9wqj3hav2md4d4-isl-0.24 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 brzrf6qx5d83j1zvirc1xk14wrhyx4hf-binutils-cross-aarch64-linux-gnu-2.38 cinj0g8krh5s428jawsazawlfarp912k-gcc-cross-aarch64-linux-gnu-11.3.0-lib cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0 gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1 rbjq50q2ik8c1glkj5f0ksnwfz64g16g-mpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 rjgpaj54q1q9n3hcvpzyp29fgi2nx6ls-ld-wrapper-aarch64-linux-gnu-0 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 y4ndp9n5krkaiwgvw58c61n7h8ls9b0f-glibc-cross-aarch64-linux-gnu-2.35\nSystem: x86_64-linux\nDeriver: fqb09jygm70cmq03xmjm7al6gzajp1p5-gcc-cross-aarch64-linux-gnu-11.3.0.drv\nSignature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNzA2MTI1QUFGMjcxQzM2MDM3RkY2MTIwMzI0NzI2NTVDMkU3OEY2NTA3OUY0MzgxMzY4NDhDQzA0OTVFRTIxRiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDg1RTA2ODE3RTFENTdCQzE1NTAxOUNGNjA0QTQxMTlFRjNBREVGQ0VFOTE4NjQzMDIzQTExQzk4RDM3REVBOCMpCiAgIChzICMwMUIwMkQyNjI0MDhCMDIzMUMxMUVDODI1Q0Q2MTkzNTU5NDRCQUNBMzg4NjEwRDIxOEIzNTVCNjM1NjFCQjk3IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==\nURL: nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: lzip\nFileSize: 24767829\nURL: nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: zstd\nFileSize: 39822104"))ludo@ribbon ~/src/guix [env]$
>
> So it seems that bordeaux.guix did advertise zstd at some point (and
> that narinfo is still in cache) but no longer does.
>
> Chris, can you confirm?

Yeah, this would have been a consequence of reducing the cache size [1].

1: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=ce8d3000fda6b80c0cf3e6f8204e0ee293024e6b

Christopher Baines wrote on 22 May 2023 13:02

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 63634@debbugs.gnu.org)

Message-ID:87bkicfw6e.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (25 lines)> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>
>> 2. &message:
> "https://bordeaux.guix.gnu.org/nar/zstd/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz:
> HTTP download failed: 404 (\"Not Found\")"
>
> Look:
>
> $ wget -qO- https://bordeaux.guix.gnu.org/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x.narinfo
> StorePath: /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
> NarHash: sha256:0pynnlsiiq03h7ysfkzw53wkf6hfmplb63haz6bjqdc686rnf4q1
> NarSize: 49128
> References: 
> System: x86_64-linux
> Deriver: qcpzwblzpa2wprc4vwlm128bwckfj8rj-rust-base64-0.13.1.tar.xz.drv
> Signature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjRUE2MTcwNUZENjNCOTU1NzcwNjIxN0EyQjczN0U5NzI2RjM4Rjk1NkQwRTRCQjI4MjVBRTA2MTdDQjcxMDQ1MyMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDNGRjU3ODU1RTNERDMxQkJFMUY2NDEyMkU0MzNBNzcwMEZGNkI1ODExRUQ0NTBCQzVEQTcwMkQyNDk3ODZEMCMpCiAgIChzICMwN0ZCQjREN0YwNTExQTlBNzZENzBFNzUxQTc2NTM1MzE0MzU1Mzk1RUUwNzgzODU3REM2NEYxRDIxMTEzQjMxIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==
> URL: nar/lzip/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
> Compression: lzip
> FileSize: 49781
>
> See? zstd’s gone!  This confirms my hypothesis.
>
> BTW, ‘guix publish’ has provisions to not compress already-compressed
> files, as in the case above.  We should really share code, Chris.

I'm not sure if this is what you have in mind regarding code sharing,
but I'll use compressed-file? from (guix utils) in the nar-herder to
avoid generating cached compressions for these kind of files in the
future.

Ludovic Courtès wrote on 22 May 2023 13:51

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 63634@debbugs.gnu.org)

Message-ID:87y1lgy46x.fsf@gnu.org

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (2 lines)

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

Toggle quote (8 lines)

>> BTW, ‘guix publish’ has provisions to not compress already-compressed

>> files, as in the case above. We should really share code, Chris.

> I'm not sure if this is what you have in mind regarding code sharing,

> but I'll use compressed-file? from (guix utils) in the nar-herder to

> avoid generating cached compressions for these kind of files in the

> future.

Great, thank you.

Ideally we’d have a single ‘guix publish’ implementation, or at least

most of it would be a module shared by both ‘guix publish’ and

‘nar-herder’ because it’s tricky enough that we’d rather not have two of

them. The (guix narinfo) module you introduced is a step in that

direction and as discussed earlier, we need to go further.

Ludo’.

Ludovic Courtès wrote on 22 May 2023 13:54

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 63634@debbugs.gnu.org)

Message-ID:87mt1wy41m.fsf@gnu.org

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (2 lines)

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

Toggle quote (9 lines)

>> So it seems that bordeaux.guix did advertise zstd at some point (and

>> that narinfo is still in cache) but no longer does.

>> Chris, can you confirm?

> Yeah, this would have been a consequence of reducing the cache size [1].

> 1: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=ce8d3000fda6b80c0cf3e6f8204e0ee293024e6b

As discussed on IRC, I think nar-herder should honor the TTL it

advertises for all the URLs of the narinfo (lzip, zstd, etc.) because

that’s how ‘guix substitute’ currently interprets the TTL.

Eventually it would be nice for ‘guix substitute’ to interpret the TTL

as applying to one of the URLs (not necessarily all of them), but this

kind of change takes months if not years to be widely deployed because

users rarely (if ever) update guix-daemon.

Ludo’.

Ludovic Courtès wrote on 22 May 2023 17:16

Recipients:

Message-ID:878rdgwg4a.fsf@gnu.org

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (25 lines)>     646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
> In ice-9/boot-9.scm:
>   1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
> In guix/scripts/substitute.scm:
>     463:7  8 (download-nar #<<narinfo> path: "/gnu/store/dg0rm1mp9q…> …)
> In ice-9/boot-9.scm:
>   1747:15  7 (with-exception-handler #<procedure 7f6df0fa71e0 at ic…> …)
>   1685:16  6 (raise-exception _ #:continuable? _)
>   1683:16  5 (raise-exception _ #:continuable? _)
>   1685:16  4 (raise-exception _ #:continuable? _)
>   1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
>   1685:16  2 (raise-exception _ #:continuable? _)
>   1683:16  1 (raise-exception _ #:continuable? _)
>   1685:16  0 (raise-exception _ #:continuable? _)
>
> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> ERROR:
>   1. &http-get-error:
>       uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1" query: #f fragment: #f>
>       code: 404
>       reason: "Not Found"
>       headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 39 minute: 59 hour: 21 day: 21 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
>   2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1: HTTP download failed: 404 (\"Not Found\")"
> substitution of /gnu/store/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1 failed

[...]

Toggle quote (3 lines)

> This might be related to 8bd4126917f59f4af9a4323c3d5699201862dca2. The

> two ‘download-nar’ calls are guarded against ‘http-get-error?’ though.

That the exception went through despite the ‘network-error?’ guard is

fixed by 3f59fd6d114548480c719d4b8f8509bdf3e8dcca.

The reason was that the ‘&http-get-error’ condition was getting wrapped

in a ‘kind-and-args’ exception upon rethrow, as illustrated here:

Toggle snippet (12 lines)scheme@(guile-user)> (catch #t
		       (lambda () (raise-exception (condition (&message (message "boo")))))
		       (lambda args args))
$5 = (%exception #<&message message: "boo">)
scheme@(guile-user)> (with-exception-handler (lambda (e) e)
		       (lambda () (apply throw $5))
		       #:unwind? #t)
$6 = #<&compound-exception components: (#<&error> #<&irritants irritants: (#<&message message: "boo">)> #<&exception-with-kind-and-args kind: %exception args: (#<&message message: "boo">)>)>
scheme@(guile-user)> (message-condition? $6)
$7 = #f

Ludo’.

Ludovic Courtès wrote on 22 May 2023 17:35

Recipients:(address . 63634@debbugs.gnu.org)(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87zg5wv0pi.fsf@gnu.org

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (3 lines)

> So it seems that bordeaux.guix did advertise zstd at some point (and

> that narinfo is still in cache) but no longer does.

Here’s a fix:

https://issues.guix.gnu.org/63646

Ludo’.

Ludovic Courtès wrote on 30 May 2023 00:19

Re: bug#63646: [PATCH] substitute: If a server's nar URL is 404, try the next one(s).

Recipients:

Message-ID:87cz2ieq78.fsf_-_@gnu.org

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (14 lines)> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> +  (define (try-fetch choices)
>> +    (match choices
>> +      (((uri compression file-size) rest ...)
>> +       (guard (c ((and (pair? rest) (network-error? c))
>> +                  (warning (G_ "download from '~a' failed, trying next URL~%")
>> +                           (uri->string uri))
>
> I realized we can change ‘network-error?’ to ‘http-get-error?’ above.
> Otherwise, we could find ourselves trying several nar URLs on the same
> server when the error is ETIMEDOUT or ECONNREFUSED, which would be a
> waste of time.

Pushed as 8af9a2aa5fa2fa5b00234c1cbe12e9aff60888a0.

Ludo’.

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 63634@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date