From debbugs-submit-bounces@debbugs.gnu.org Sun Apr 07 12:43:30 2019 Received: (at submit) by debbugs.gnu.org; 7 Apr 2019 16:43:30 +0000 Received: from localhost ([127.0.0.1]:48647 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDAtG-00080l-GW for submit@debbugs.gnu.org; Sun, 07 Apr 2019 12:43:30 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42480) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDAtE-00080X-7T for submit@debbugs.gnu.org; Sun, 07 Apr 2019 12:43:29 -0400 Received: from lists.gnu.org ([209.51.188.17]:39203) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hDAt8-00061R-TL for submit@debbugs.gnu.org; Sun, 07 Apr 2019 12:43:22 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56861) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hDAt7-00013r-Ou for bug-guix@gnu.org; Sun, 07 Apr 2019 12:43:22 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_40,URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hDAt6-00060g-Pj for bug-guix@gnu.org; Sun, 07 Apr 2019 12:43:21 -0400 Received: from world.peace.net ([64.112.178.59]:56576) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hDAt6-0005x5-JP for bug-guix@gnu.org; Sun, 07 Apr 2019 12:43:20 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDAsu-0007vo-Lh; Sun, 07 Apr 2019 12:43:08 -0400 From: Mark H Weaver To: bug-guix@gnu.org Subject: Hydra offloads often get stuck while exporting build requisites Date: Sun, 07 Apr 2019 12:41:38 -0400 Message-ID: <87mul17oo2.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 64.112.178.59 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) It has become extremely frequent for builds offloaded by hydra.gnu.org to its x86 build slave hydra.gnunet.org to get stuck indefinitely while exporting prerequisites for the build to the build slave. As I write this, both of hydra.gnunet.org's build slots (one for x86_64-linux, and one for i686-linux) are stuck in this way. Here are the stuck builds: https://hydra.gnu.org/build/3432052 https://hydra.gnu.org/build/3432472 and here are the tails of their nix build logs: --8<---------------cut here---------------start------------->8--- performing build 3432052 these derivations will be built: /gnu/store/k27i3lkb38gr3mw0mridymhik3qsg6w7-font-fira-sans-4.202.drv process 14769 acquired build slot '/var/guix/offload/hydra.gnunet.org/1' load on machine 'hydra.gnunet.org' is 1.54 (normalized: 0.385) sending 1 store item to 'hydra.gnunet.org'... exporting path `/gnu/store/gzd2cisahj50nff16p8ji813p683p5r4-font-fira-sans-4.202-checkout' --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- performing build 3432472 these derivations will be built: /gnu/store/5ivay4l7bn0sqsi7k53j4qv3kndrby17-font-google-material-design-icons-3.0.1.drv process 8985 acquired build slot '/var/guix/offload/hydra.gnunet.org/0' load on machine 'hydra.gnunet.org' is 1.98 (normalized: 0.99) sending 1 store item to 'hydra.gnunet.org'... exporting path `/gnu/store/kaj5xgnz04l4mzgj05sc5v6j7cvpbrrd-font-google-material-design-icons-3.0.1-checkout' --8<---------------cut here---------------end--------------->8--- The first of these builds has been stuck for about 10.25 hours, and the second has been stuck for 11 hours. In the recent past, I've discovered them stuck for over 24 hours. Mark