From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 08 04:21:49 2018 Received: (at 21097) by debbugs.gnu.org; 8 Jun 2018 08:21:49 +0000 Received: from localhost ([127.0.0.1]:38660 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fRCeb-00076F-AD for submit@debbugs.gnu.org; Fri, 08 Jun 2018 04:21:49 -0400 Received: from mail-pg0-f48.google.com ([74.125.83.48]:41936) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fRCeZ-000762-RH for 21097@debbugs.gnu.org; Fri, 08 Jun 2018 04:21:48 -0400 Received: by mail-pg0-f48.google.com with SMTP id l65-v6so6035500pgl.8 for <21097@debbugs.gnu.org>; Fri, 08 Jun 2018 01:21:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:user-agent:date :message-id:mime-version; bh=p0U1IXctEs0z3PmbaQPDFHrv4WG40yOZnsJptz5Xwfk=; b=odRkUwWWOqvXYT+Ovt0KLvehZYXDNsrvAaMJdWsQT84WO0GM3TNWyIBpomoWE5fNiU AafUmthuIm2p/9gkbToirieP1NtM96BYD6V3joEuZDWVz7a3u0oVlR3DzdqjtxWkWM6w coIE6CxovG1FKCDhVFLxw96b++B/EWsRigkpsJ/HPi2pEwBU3PCv5aQY12weKYjDp3hb gkyM6SO0WMVpbo9WPhZkbyeWLyG7E9lTaNsS1zmzAsqCr4TV9a+CQbqo+aBwD79Orrlq o9UsFbuTtAaD4iOCza4bdHZ77w1qVyO0YW43uCm5v5Gj9auMFSnGfsbZJvRLYvf7vmNj hylg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :user-agent:date:message-id:mime-version; bh=p0U1IXctEs0z3PmbaQPDFHrv4WG40yOZnsJptz5Xwfk=; b=AZAiYmxBTHUAuFmrBMepzWXBfbLa+OdVD5UWstQnAqNc+wa3fZJvudZ8HWE2/LrlXo bPWXxa6wW26i2nJ+BHYcKPVleHMSautdldKQyMlPeevt5JDpD1Mfxmy1D7VR1KnaQcP0 NfSfZzXfbBXQb1bgY4DpaoM3N9iBYG6jul5Aq5iUOkjctTIIHJV0PDVZg1jNonE00Ct9 B1T0Dbz7uaP0jNB2c7/771Y1KdV+d+dxW8NJZJ2S2ai6z+O46JkzVifjOQnHHwLVRe6O amCGzezIu2fziEyvIlzF2nQOM42alP3rcs8ipUfjyMJy7PqyNwLXKVvbV4VXu9OWh5+Z m7vg== X-Gm-Message-State: APt69E2zY3vmXr/kpRAOWqZah/8ENZari+C79bf+UztY+8rikV3Kco1F fe4/x30CiEmNYfB2rh1+TLf6HA== X-Google-Smtp-Source: ADUXVKKGT/4Odqjcb+rGh3Qp/zRPucsB5BYv94/z+8w4FBCKZ3VmPfYIUlt+qALh60EFS2igC42k3g== X-Received: by 2002:a62:211a:: with SMTP id h26-v6mr5016800pfh.133.1528446101403; Fri, 08 Jun 2018 01:21:41 -0700 (PDT) Received: from garuda.local (c-24-18-253-84.hsd1.wa.comcast.net. [24.18.253.84]) by smtp.gmail.com with ESMTPSA id t5-v6sm1426088pfh.32.2018.06.08.01.21.39 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 08 Jun 2018 01:21:39 -0700 (PDT) From: Chris Marusich X-Google-Original-From: Chris Marusich To: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: bug#21097: verify-store test failure on armhf-linux In-Reply-To: <877flybt9v.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 03 Nov 2015 23:41:16 +0100") References: <87k2tu3b10.fsf@netris.org> <877flybt9v.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) Date: Fri, 08 Jun 2018 01:21:33 -0700 Message-ID: <87fu1xy9gy.fsf@garuda.local.i-did-not-set--mail-host-address--so-tickle-me> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 21097 Cc: Mark H Weaver , 21097@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable ludo@gnu.org (Ludovic Court=C3=A8s) writes: > I=E2=80=99ve become convinced that this is due to parallelism: several > guix-daemon processes run at the same time. In this case, I bet this > process tries to remove an item from the ValidPaths table while another > is trying to add it in the Refs table or something. > > In dc57d527 I added #:parallel-tests? #f for =E2=80=98guix-devel=E2=80=99= . Eventually > we should fix the makefile to run this test alone, as is done for > =E2=80=98guix-gc.sh=E2=80=99. In the 2 years and 7 months since we disabled parallel tests in commit dc57d527aee4eb18ec5fb345f90d6637bbd1a4d2 to work around this bug, we may have allowed other parallelism bugs to quietly creep in. Today, I observed a parallel test failure that seems unrelated to the original bug reported here. And anecdotally, I feel that the tests frequently fail spuriously when I run them in parallel. Until we get to the bottom of this, I agree that the best thing to do is to always run the tests in serial. For completeness, below I'll report the failure I observed today. On my x86_64-linux GuixSD machine, using Guix version 0ec430f79530ee343c175347952f91a78adca5ec (this is what my ~/.config/guix/latest points to), I entered a Guix development environment via "guix environment guix". In Guix's Git repository, I checked out commit 4dd91dff477b9717b3fa494b23976e4d69ab7dfc (the current tip of core-updates) and ran the following commands: ./bootstrap && ./configure --localstatedir=3D/var && make -j \ && make -j check The following tests failed: FAIL: tests/guix-hash.sh FAIL: tests/guix-download.sh FAIL: tests/guix-build.sh FAIL: tests/guix-package.sh FAIL: tests/guix-system.sh When I immediately ran "make recheck" without making any changes, the same 5 tests passed. Note that this ran the tests in serial because I omitted -j. When I ran the same 5 tests again in parallel using the following command, they all passed: make -j check TESTS=3D"tests/guix-hash.sh tests/guix-download.sh \ tests/guix-build.sh tests/guix-package.sh tests/guix-system.sh" I also tried running just tests/guix-hash.sh and tests/guix-download.sh together 10 times in serial and then 10 times in parallel. Unfortunately, this didn't reproduce the failure, either (i.e., all 20 test runs passed). All in all, this seems to suggest that the failures I observed might be caused by a parallelism bug when running the entire test suite. Regarding the cause of failure, the 5 tests all failed with a message like the following: =2D-8<---------------cut here---------------start------------->8--- ERROR: In procedure canonicalize-path: In procedure canonicalize-path: No such file or directory + guix download --version Backtrace: In ice-9/boot-9.scm: 2875:24 19 (_) 222:17 18 (map1 (((guix utils)) ((guix config)) ((guix #)) ((=E2=80=A6))= =E2=80=A6)) 2788:17 17 (resolve-interface (guix utils) #:select _ #:hide _ # _ =E2=80= =A6) 2714:10 16 (_ (guix utils) _ _ #:ensure _) 2982:16 15 (try-module-autoload _ _) 2312:4 14 (save-module-excursion #) 3002:22 13 (_) In unknown file: 12 (primitive-load-path "guix/utils" #) In guix/utils.scm: 26:0 11 (_) In ice-9/boot-9.scm: 2862:4 10 (define-module* _ #:filename _ #:pure _ #:version _ # _ =E2=80= =A6) 2875:24 9 (_) 222:17 8 (map1 (((guix config)) ((srfi srfi-1)) ((srfi #)) (#) =E2=80= =A6)) 2788:17 7 (resolve-interface (guix config) #:select _ #:hide _ # _ =E2= =80=A6) 2714:10 6 (_ (guix config) _ _ #:ensure _) 2982:16 5 (try-module-autoload _ _) 2312:4 4 (save-module-excursion #) 3002:22 3 (_) In unknown file: 2 (primitive-load-path "guix/config" #) In guix/config.scm: 86:6 1 (_) In unknown file: 0 (canonicalize-path "/home/marusich/guix/test-tmp/db") =2D-8<---------------cut here---------------end--------------->8--- All the test failures looked the same, except that instead of "guix download --version", the equivalent command (e.g., "guix system =2D-version") was invoked. I realize this information doesn't help solve the original bug reported here. However, it's a real failure, so I hope it'll be useful. In any case, it shows that there are probably multiple parallelism bugs lurking in our code now. We're going to have to solve all those parallelism bugs before we can reliably run the tests in parallel again. =2D-=20 Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAlsaPI0ACgkQ3UCaFdgi Rp3+/g//VWvLRC78ldJ/Xxf1ErllS8hiYsHl5jtTxaU3hm/EnPqOe34P6nKWPh67 DMicUYcNgRWSkx5qEP19WuSjq9c21HQo6NTgZFhrkZR5SLzw44HqO1OZOACuGtRu exbkD774mbgEJyDbEZwhN/6G/V9myyjF07/h1ZkSqV58Rh2e9DVA2Wdjojr+OoIK RcM0WtcJwHa5XNdsJF6Mlgwsjjmv6vtE9gkS27iL59f2U0TlSJUQMkylLxa6uq2r bMUfxspURkk0nyC62kFGeP2g7m7rIT3bNj2C751sDRLh9VJeu+5D6g1klRb1YIlA 2WFgqU39B1TGIMO9a/JG/PaRAxpHsdQt+hS9Urf7i/E88eEQ1xbxF3CuUzqZ2kBm 07nHpx1yI0ek+MdD/gnL8wbGL/1ClZ20IWLuSgMStNa1V2hE/QFzoeKDt+KUZecG +tAaxBj4TKlCQCvf24OZiyskH9gjwobFOD86JWb4MZ1rmeMdInUTN2HK+2DOCUvG 3qfaFa64LpqqeRqyXQtirN8q33vlonzBUesxfu4IIKxhvpVbAbYPOkzLfoK9khac m9I5JdAbfXs/Vr7ZJ+bTJEcz3OgX4g0+pl6ZT4+qDLf/4LiTKKVxEjfghE7xiHrQ Z7KH2D5mBnjhdI1dJYHxkNj4jv4+d6LgGQKkYkAQXbPqQjh/ijo= =vzSU -----END PGP SIGNATURE----- --=-=-=--