From debbugs-submit-bounces@debbugs.gnu.org Fri Apr 28 18:34:08 2017 Received: (at submit) by debbugs.gnu.org; 28 Apr 2017 22:34:08 +0000 Received: from localhost ([127.0.0.1]:45212 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d4ESm-00084b-8N for submit@debbugs.gnu.org; Fri, 28 Apr 2017 18:34:08 -0400 Received: from eggs.gnu.org ([208.118.235.92]:36147) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d4ESj-000847-M8 for submit@debbugs.gnu.org; Fri, 28 Apr 2017 18:34:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d4ESd-0006kl-6f for submit@debbugs.gnu.org; Fri, 28 Apr 2017 18:34:00 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:44213) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1d4ESd-0006ka-3V for submit@debbugs.gnu.org; Fri, 28 Apr 2017 18:33:59 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55334) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d4ESb-0004X5-FC for bug-guix@gnu.org; Fri, 28 Apr 2017 18:33:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d4ESW-0006a3-DC for bug-guix@gnu.org; Fri, 28 Apr 2017 18:33:57 -0400 Received: from world.peace.net ([50.252.239.5]:38201) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d4ESW-0006NL-9Q for bug-guix@gnu.org; Fri, 28 Apr 2017 18:33:52 -0400 Received: from turntable.mit.edu ([18.160.0.29] helo=jojen) by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1d4ESL-0001oF-0R; Fri, 28 Apr 2017 18:33:41 -0400 From: Mark H Weaver To: bug-guix@gnu.org Subject: guix publish daemon on Hydra became dysfunctional; needed restart Date: Fri, 28 Apr 2017 18:33:30 -0400 Message-ID: <877f24xiv9.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) While trying to update my GuixSD system in the last hour, I found that every attempt by the substituter to download NARs resulted in a 500 "Internal Server Error": --8<---------------cut here---------------start------------->8--- mhw@jojen ~$ guix package -u substitute: ^Msubstitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 0.0%^Msubstitute: updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0% Downloading https://mirror.hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz (2.3MiB installed)... guix substitute: error: download from 'https://mirror.hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz' failed: 500, "Internal Server Error" guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/l58cq0l2j2ikyimjvxyv1dakha2ai6rx-gnome-calendar-3.22.2.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source mhw@jojen ~$ guix package -u Downloading https://mirror.hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz (2.3MiB installed)... guix substitute: error: download from 'https://mirror.hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz' failed: 500, "Internal Server Error" guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/l58cq0l2j2ikyimjvxyv1dakha2ai6rx-gnome-calendar-3.22.2.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source --8<---------------cut here---------------end--------------->8--- Bypassing the mirror didn't help: --8<---------------cut here---------------start------------->8--- mhw@jojen ~$ guix package -u . --substitute-urls=https://hydra.gnu.org substitute: ^Msubstitute: updating list of substitutes from 'https://hydra.gnu.org'... 0.0%^Msubstitute: updating list of substitutes from 'https://hydra.gnu.org'... 100.0% substitute: ^Msubstitute: updating list of substitutes from 'https://hydra.gnu.org'... 0.0%^Msubstitute: updating list of substitutes from 'https://hydra.gnu.org'... 100.0% Downloading https://hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz (2.3MiB installed)... guix substitute: error: download from 'https://hydra.gnu.org/guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz' failed: 500, "Internal Server Error" guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/l58cq0l2j2ikyimjvxyv1dakha2ai6rx-gnome-calendar-3.22.2.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source --8<---------------cut here---------------end--------------->8--- Here's what I found in the recent output of the 'guix publish' server on Hydra (running in root's screen session): --8<---------------cut here---------------start------------->8--- GET /74ch6nvjfkj3i56nygwijnaghlpi01d4.narinfo In guix/scripts/publish.scm: 393:2 2 (render-narinfo/cached # ...) In guix/store.scm: 663:9 1 (query-path-from-hash-part # #) In unknown file: 0 (put-bytevector # #vu8(# ...) ...) ERROR: In procedure fport_write: Broken pipe GET /guix/nar/qz4mg7sid6avdav158lhr6wziqswpjmx-gnome-calendar-3.22.2.tar.xz In guix/scripts/publish.scm: 491:8 2 (render-nar # #< ...) In guix/store.scm: 648:0 1 (valid-path? # "/gnu/sto...") In unknown file: 0 (put-bytevector # #vu8(1 ...) ...) ERROR: In procedure fport_write: Broken pipe GET /zhxab8rkbbalgrlhg86q8pzpqi3s6q36.narinfo In guix/scripts/publish.scm: 393:2 2 (render-narinfo/cached # ...) In guix/store.scm: 663:9 1 (query-path-from-hash-part # #) In unknown file: 0 (put-bytevector # #vu8(# ...) ...) ERROR: In procedure fport_write: Broken pipe GET /z2rygpahxbpkwx8k164bi5lr5lann357.narinfo In guix/scripts/publish.scm: 393:2 2 (render-narinfo/cached # ...) In guix/store.scm: 663:9 1 (query-path-from-hash-part # #) In unknown file: 0 (put-bytevector # #vu8(# ...) ...) ERROR: In procedure fport_write: Broken pipe GET /yw635wxpm4ac40khzd9w6j9v3n72a708.narinfo In guix/scripts/publish.scm: 393:2 2 (render-narinfo/cached # ...) In guix/store.scm: 663:9 1 (query-path-from-hash-part # #) In unknown file: 0 (put-bytevector # #vu8(# ...) ...) ERROR: In procedure fport_write: Broken pipe --8<---------------cut here---------------end--------------->8--- I killed the process, restarted it, and now things seem to be working again, but it would be good to know what went wrong here. Mark