From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 23 12:37:03 2020 Received: (at submit) by debbugs.gnu.org; 23 Jun 2020 16:37:03 +0000 Received: from localhost ([127.0.0.1]:37026 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnlux-00021U-9H for submit@debbugs.gnu.org; Tue, 23 Jun 2020 12:37:03 -0400 Received: from lists.gnu.org ([209.51.188.17]:45702) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnluv-000215-Ef for submit@debbugs.gnu.org; Tue, 23 Jun 2020 12:37:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43310) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jnluv-00029C-6Y for guix-patches@gnu.org; Tue, 23 Jun 2020 12:37:01 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:44423) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jnlup-0008GS-0E for guix-patches@gnu.org; Tue, 23 Jun 2020 12:37:00 -0400 Received: from localhost (unknown [46.237.174.39]) by mira.cbaines.net (Postfix) with ESMTPSA id D82EA27BBE1 for ; Tue, 23 Jun 2020 17:36:51 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 5d911ac2 for ; Tue, 23 Jun 2020 16:36:49 +0000 (UTC) From: Christopher Baines To: guix-patches@gnu.org Subject: [PATCH] database: register-items: reduce transaction scope. Date: Tue, 23 Jun 2020 17:36:49 +0100 Message-Id: <20200623163649.32444-1-mail@cbaines.net> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/23 12:36:52 X-ACL-Warn: Detected OS = ??? X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) It was made transactional in a4678c6ba18d8dbd79d931f80426eebf61be7ebe, with the reasoning to prevent broken intermediate states from being visible. I think this means something like an entry being in ValidPaths, but the Refs not being inserted. Using a transaction for this makes sense, but I think using one single transaction for the whole register-items call is unnecessary to avoid broken states from being visible, and could block other writes to the store database while register-items is running. Because the deduplication and resetting timestamps happens within the transaction as well, even though these things don't involve the database, writes to the database will still be blocked while this is happening. To reduce the potential for register-items to block other writers to the database for extended periods, this commit moves the transaction to just wrap the call to sqlite-register. This is the one place where writes occur, so that should prevent the broken intermediate states issue above. The one difference this will make is some of the registered items will be visible to other connections while others may be still being added. I think this is OK, as it's equivalent to just registering different items. * guix/store/database.scm (register-items): Reduce transaction scope. --- guix/store/database.scm | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/guix/store/database.scm b/guix/store/database.scm index a38e4d7e52..767335bc0f 100644 --- a/guix/store/database.scm +++ b/guix/store/database.scm @@ -441,24 +441,25 @@ in the database; #f means \"now\". Write a progress report to LOG-PORT." (when reset-timestamps? (reset-timestamps real-file-name)) (let-values (((hash nar-size) (nar-sha256 real-file-name))) - (sqlite-register db #:path to-register - #:references (store-info-references item) - #:deriver (store-info-deriver item) - #:hash (string-append "sha256:" - (bytevector->base16-string hash)) - #:nar-size nar-size - #:time registration-time) + (call-with-retrying-transaction db + (lambda () + (sqlite-register db #:path to-register + #:references (store-info-references item) + #:deriver (store-info-deriver item) + #:hash (string-append + "sha256:" + (bytevector->base16-string hash)) + #:nar-size nar-size + #:time registration-time))) (when deduplicate? (deduplicate real-file-name hash #:store store-dir))))) - (call-with-retrying-transaction db - (lambda () - (let* ((prefix (format #f "registering ~a items" (length items))) - (progress (progress-reporter/bar (length items) - prefix log-port))) - (call-with-progress-reporter progress - (lambda (report) - (for-each (lambda (item) - (register db item) - (report)) - items))))))) + (let* ((prefix (format #f "registering ~a items" (length items))) + (progress (progress-reporter/bar (length items) + prefix log-port))) + (call-with-progress-reporter progress + (lambda (report) + (for-each (lambda (item) + (register db item) + (report)) + items))))) -- 2.26.2