guix publish daemon on Hydra became dysfunctional; needed restart
(address . bug-guix@gnu.org)
While trying to update my GuixSD system in the last hour, I found that
every attempt by the substituter to download NARs resulted in a 500
"Internal Server Error":
Toggle snippet (11 lines)
mhw@jojen ~$ guix package -u
substitute: ^Msubstitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 0.0%^Msubstitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
Downloading https://mirror.hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz (2.3MiB installed)...
guix substitute: error: download from 'https://mirror.hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz' failed: 500, "Internal Server Error"
guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/l58cq0l2j2ikyimjvxyv1dakha2ai6rx-gnome-calendar-3.22.2.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
mhw@jojen ~$ guix package -u
Downloading https://mirror.hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz (2.3MiB installed)...
guix substitute: error: download from 'https://mirror.hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz' failed: 500, "Internal Server Error"
guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/l58cq0l2j2ikyimjvxyv1dakha2ai6rx-gnome-calendar-3.22.2.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
Bypassing the mirror didn't help:
Toggle snippet (8 lines)
mhw@jojen ~$ guix package -u . --substitute-urls=https://hydra.gnu.org
substitute: ^Msubstitute: updating list of substitutes from 'https://hydra.gnu.org'... 0.0%^Msubstitute: updating list of substitutes from 'https://hydra.gnu.org'... 100.0%
substitute: ^Msubstitute: updating list of substitutes from 'https://hydra.gnu.org'... 0.0%^Msubstitute: updating list of substitutes from 'https://hydra.gnu.org'... 100.0%
Downloading https://hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz (2.3MiB installed)...
guix substitute: error: download from 'https://hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz' failed: 500, "Internal Server Error"
guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/l58cq0l2j2ikyimjvxyv1dakha2ai6rx-gnome-calendar-3.22.2.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
Here's what I found in the recent output of the 'guix publish' server on
Hydra (running in root's screen session):
Toggle snippet (42 lines)
GET /74ch6nvjfkj3i56nygwijnaghlpi01d4.narinfo
In guix/scripts/publish.scm:
393:2 2 (render-narinfo/cached #<build-daemon 256.96 211f7c0> ...)
In guix/store.scm:
663:9 1 (query-path-from-hash-part #<build-daemon 256.96 2...> #)
In unknown file:
0 (put-bytevector #<input-output: socket 12> #vu8(# ...) ...)
ERROR: In procedure fport_write: Broken pipe
GET /guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz
In guix/scripts/publish.scm:
491:8 2 (render-nar #<build-daemon 256.96 211f7c0> #<<requ...> ...)
In guix/store.scm:
648:0 1 (valid-path? #<build-daemon 256.96 211f7c0> "/gnu/sto...")
In unknown file:
0 (put-bytevector #<input-output: socket 12> #vu8(1 ...) ...)
ERROR: In procedure fport_write: Broken pipe
GET /zhxab8rkbbalgrlhg86q8pzpqi3s6q36.narinfo
In guix/scripts/publish.scm:
393:2 2 (render-narinfo/cached #<build-daemon 256.96 211f7c0> ...)
In guix/store.scm:
663:9 1 (query-path-from-hash-part #<build-daemon 256.96 2...> #)
In unknown file:
0 (put-bytevector #<input-output: socket 12> #vu8(# ...) ...)
ERROR: In procedure fport_write: Broken pipe
GET /z2rygpahxbpkwx8k164bi5lr5lann357.narinfo
In guix/scripts/publish.scm:
393:2 2 (render-narinfo/cached #<build-daemon 256.96 211f7c0> ...)
In guix/store.scm:
663:9 1 (query-path-from-hash-part #<build-daemon 256.96 2...> #)
In unknown file:
0 (put-bytevector #<input-output: socket 12> #vu8(# ...) ...)
ERROR: In procedure fport_write: Broken pipe
GET /yw635wxpm4ac40khzd9w6j9v3n72a708.narinfo
In guix/scripts/publish.scm:
393:2 2 (render-narinfo/cached #<build-daemon 256.96 211f7c0> ...)
In guix/store.scm:
663:9 1 (query-path-from-hash-part #<build-daemon 256.96 2...> #)
In unknown file:
0 (put-bytevector #<input-output: socket 12> #vu8(# ...) ...)
ERROR: In procedure fport_write: Broken pipe
I killed the process, restarted it, and now things seem to be working
again, but it would be good to know what went wrong here.
Mark