[PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

DoneSubmitted by Jan (janneke) Nieuwenhuizen.
Details
2 participants
  • Jan (janneke) Nieuwenhuizen
  • Ludovic Courtès
Owner
unassigned
Severity
normal
J
J
Jan (janneke) Nieuwenhuizen wrote on 1 Jul 16:29 +0200
(address . bug-guix@gnu.org)
20200701142929.26919-1-janneke@gnu.org
Hello Guix!
I configured my system with a childhurd service and added
Toggle snippet (8 lines) (build-machine (name "childhurd") (system "i586-gnu") (host-key "ssh-ed25519 ... root@childhurd") (user "root") (private-key "/home/janneke/.ssh/id_rsa_childhurd"))
to /etc/guix/machines. Sadly, running guix offload test gives
Toggle snippet (7 lines)$ guix offload testguix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")guix offload: 'childhurd' is running GNU Guile 3.0.4guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
So, I looked into the Debian/Hurd sqlite3 package for inspiration and founddebian/patches/20-hurd-locking-style.patch...well, that should fix it, right?
Sadly, this patch seems to break sqlite3 altogether.
Toggle snippet (34 lines)$ guix offload testguix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...Backtrace:In ice-9/boot-9.scm: 1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)In unknown file: 10 (apply-smob/0 #<thunk 7f3c99b1f480>)In ice-9/boot-9.scm: 718:2 9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)In ice-9/eval.scm: 619:8 8 (_ #(#(#<directory (guile-user) 7f3c99743f00>)))In guix/ui.scm: 1949:12 7 (run-guix-command _ . _) 663:2 6 (call-with-error-handling _)In ice-9/boot-9.scm: 1736:10 5 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _) 1731:15 4 (with-exception-handler #<procedure 7f3c9546d3f0 at ice-9/boot-9.scm:1815:7 (exn)> _ #:unwind? _ # …)In guix/scripts/offload.scm: 647:6 3 (check-machine-availability _ _)In srfi/srfi-1.scm: 650:11 2 (for-each #<procedure assert-node-has-guix (node name)> (#<<inferior> pid: pipe socket: #<input…>) …)In guix/scripts/offload.scm: 582:2 1 (assert-node-has-guix #<<inferior> pid: pipe socket: #<input-output: channel (open) 7f3c954a2500…> …)In ice-9/boot-9.scm: 1669:16 0 (raise-exception _ #:continuable? _)
ice-9/boot-9.scm:1669:16: In procedure raise-exception:ERROR: 1. &inferior-exception: arguments: (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: unable to open database file" status: 1>>) inferior: #<<inferior> pid: pipe socket: #<input-output: channel (open) 7f3c954a2500> close: #<procedure close-port (_)> version: (0 1 1) packages: #<promise #<procedure 7f3c954a8000 at guix/inferior.scm:161:32 ()>> table: #<promise #<procedure 7f3c954a56f0 at guix/inferior.scm:162:32 ()>>> stack: ((#f ("ice-9/boot-9.scm" 1763 13)) (raise-exception ("ice-9/boot-9.scm" 1666 16)) (raise-exception ("ice-9/boot-9.scm" 1668 16)) (#f ("guix/store.scm" 582 42)) (with-exception-handler ("ice-9/boot-9.scm" 1735 10)) (open-connection ("guix/store.scm" 545 2)) (call-with-store ("guix/store.scm" 626 15)) (#f (#f #f #f)) (#f ("guix/repl.scm" 92 21)) (with-exception-handler ("ice-9/boot-9.scm" 1735 10)) (with-exception-handler ("ice-9/boot-9.scm" 1730 15)) (#f ("guix/repl.scm" 119 7)))
Maybe we're missing some file_lock patch that Debian has? Where to look,glibc, hurd, ...? Ideas?
Greetings,Janneke
Jan (janneke) Nieuwenhuizen (3): gnu: Add sqlite/hurd with locking fix. gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd. gnu: guix: Use sqlite/hurd for locking on the Hurd.
gnu/local.mk | 1 + gnu/packages/guile.scm | 2 +- gnu/packages/package-management.scm | 2 +- gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++ gnu/packages/sqlite.scm | 9 +++++ 5 files changed, 63 insertions(+), 2 deletions(-) create mode 100644 gnu/packages/patches/sqlite3-hurd.patch
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 1 Jul 16:31 +0200
[PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
(address . 42151@debbugs.gnu.org)
20200701143118.26985-2-janneke@gnu.org
This fixes
guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
* gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead ofsqlite.--- gnu/packages/guile.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Toggle diff (16 lines)diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scmindex a32bd33293..c651e2427a 100644--- a/gnu/packages/guile.scm+++ b/gnu/packages/guile.scm@@ -665,7 +665,7 @@ Guile's foreign function interface.") ("pkg-config" ,pkg-config))) (inputs `(("guile" ,guile-3.0)- ("sqlite" ,sqlite)))+ ("sqlite" ,sqlite/hurd))) (synopsis "Access SQLite databases from Guile") (description "This package provides Guile bindings to the SQLite database system.")-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 1 Jul 16:31 +0200
[PATCH 1/3] gnu: Add sqlite/hurd with locking fix.
(address . 42151@debbugs.gnu.org)
20200701143118.26985-1-janneke@gnu.org
* gnu/packages/patches/sqlite3-hurd.patch: New file.* gnu/local.mk (dist_patch_DATA): Add it.* gnu/packages/sqlite.scm (sqlite/hurd): New variable.--- gnu/local.mk | 1 + gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++ gnu/packages/sqlite.scm | 9 +++++ 3 files changed, 61 insertions(+) create mode 100644 gnu/packages/patches/sqlite3-hurd.patch
Toggle diff (96 lines)diff --git a/gnu/local.mk b/gnu/local.mkindex 5e9dba5ab7..d5451bf2d5 100644--- a/gnu/local.mk+++ b/gnu/local.mk@@ -1358,6 +1358,7 @@ dist_patch_DATA = \ %D%/packages/patches/sdl-pango-header-guard.patch \ %D%/packages/patches/sdl-pango-matrix_declarations.patch \ %D%/packages/patches/sdl-pango-sans-serif.patch \+ %D%/packages/patches/sqlite3-hurd.patch \ %D%/packages/patches/patchutils-test-perms.patch \ %D%/packages/patches/patch-hurd-path-max.patch \ %D%/packages/patches/perl-autosplit-default-time.patch \diff --git a/gnu/packages/patches/sqlite3-hurd.patch b/gnu/packages/patches/sqlite3-hurd.patchnew file mode 100644index 0000000000..de87a30cb1--- /dev/null+++ b/gnu/packages/patches/sqlite3-hurd.patch@@ -0,0 +1,51 @@+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch+Upstream status: Not offered upstream.++diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c+--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode+ # include <sys/mman.h>+ #endif+ +-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ /* # include <sys/ioctl.h> */+ # include <sys/file.h>+ # include <sys/param.h>+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id+ **+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off+ */+-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ + /*+ ** Retry flock() calls that fail with EINTR+@@ -38586,7 +38586,7 @@ IOMETHODS(+ 0 /* xShmMap method */+ )+ +-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ IOMETHODS(+ flockIoFinder, /* Finder function name */+ flockIoMethods, /* sqlite3_io_methods object name */+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){+ UNIXVFS("unix", autolockIoFinder ),+ #elif OS_VXWORKS+ UNIXVFS("unix", vxworksIoFinder ),++#elif defined(__GNU__)++ UNIXVFS("unix", flockIoFinder ),+ #else+ UNIXVFS("unix", posixIoFinder ),+ #endif+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){+ #if OS_VXWORKS+ UNIXVFS("unix-namedsem", semIoFinder ),+ #endif+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)+ UNIXVFS("unix-posix", posixIoFinder ),+ #endif+ #if SQLITE_ENABLE_LOCKING_STYLEdiff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scmindex 8468131ddf..d5c44b0d0f 100644--- a/gnu/packages/sqlite.scm+++ b/gnu/packages/sqlite.scm@@ -11,6 +11,7 @@ ;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org> ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr> ;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -107,3 +108,11 @@ is in the public domain.") ;; commit fad5b1a6d8d9c36bea5785ae4fbc1beb37e644d7. (define-public sqlite-with-column-metadata (deprecated-package "sqlite-with-column-metadata" sqlite))++(define-public sqlite/hurd+ ;; TODO move into sqlite on the next rebuild cycle.+ (package+ (inherit sqlite)+ (name "sqlite-for-hurd")+ (source (origin (inherit (package-source sqlite))+ (patches (search-patches "sqlite3-hurd.patch"))))))-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 1 Jul 16:31 +0200
[PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.
(address . 42151@debbugs.gnu.org)
20200701143118.26985-3-janneke@gnu.org
* gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.--- gnu/packages/package-management.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Toggle diff (16 lines)diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scmindex 068d46566c..52c5a1e60d 100644--- a/gnu/packages/package-management.scm+++ b/gnu/packages/package-management.scm@@ -343,7 +343,7 @@ $(prefix)/etc/init.d\n"))) ("zlib" ,zlib) ;for 'guix publish' ("lzlib" ,lzlib) ;for 'guix publish' and 'guix substitute' - ("sqlite" ,sqlite)+ ("sqlite" ,sqlite/hurd) ("libgcrypt" ,libgcrypt) ("guile" ,guile-3.0-latest)-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 1 Jul 19:01 +0200
Re: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol
(address . bug-guix@gnu.org)
87o8ozcgq3.fsf@gnu.org
Jan (janneke) Nieuwenhuizen writes:
Toggle quote (3 lines)> Maybe we're missing some file_lock patch that Debian has? Where to look,> glibc, hurd, ...? Ideas?
So...I found a way to reproduce the feature/bug: run "pragma synchronous = normal;"in two instances.
I created a db.sqlite using
sh -x tests/guix-pack-localstatedir.sh
(after removing the cleanup trap).
Then, doing
Toggle snippet (20 lines)$ sqlite3 db.sqlite SQLite version 3.31.1 2020-01-27 19:55:54Enter ".help" for usage hints.sqlite> SELECT * FROM ValidPaths;1|/gnu/store/pqciscj9304544h3yaqb5m70fggsr293-manual-database|sha256:ccd93bed3a0018f892abc5db92bf182c3e2e986f41a8ded168915b4d9e85fa18|1||26642|/gnu/store/mzfkrxd4w8vqrmyrx169wj8wyw7r8i37-bash|sha256:6f887d45fa0f7e59e55c6d7ba86a3d8c35369c7afbb3a5829b8ed226bfef4a66|1||13518803|/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0|sha256:dc3e6b577e995c093849454fe4b9c30e87ece6462c54369ff249121f178c5476|1||127188084|/gnu/store/6z4h8g6s73zmcn0h6sm63p9d47ih4b63-info-dir|sha256:fd154332e6d35c95f12e41d6168a93945b4737fcbaf734bcaf04ca771a106544|1||4325|/gnu/store/mj7szx64bdh080rlhzhrvcg6viyba73s-profile|sha256:e32ae4d07fc2298a72e6942f21ba85c0300a84bbaf5794a8c5f0393a8dc9f253|1||2960sqlite> pragma synchronous = normal;sqlite> C-c C-z^Z[2]+ Stopped sqlite3 db.sqlite[148]18:55:06 janneke@dundal:~/src/guix/master [env]$ sqlite3 db.sqlite SQLite version 3.31.1 2020-01-27 19:55:54Enter ".help" for usage hints.sqlite> pragma synchronous = normal;sqlite>
works nice on GNU/Linux and on current master Childhurd gives
Toggle snippet (26 lines)$ ssh childhurdLast login: Wed Jul 1 12:08:17 2020 from 10.0.2.2 This is the GNU Hurd. Welcome.
root@childhurd ~# sqlite3 db.sqlite SQLite version 3.31.1 2020-01-27 19:55:54Enter ".help" for usage hints.sqlite> SELECT * FROM ValidPaths;1|/gnu/store/pqciscj9304544h3yaqb5m70fggsr293-manual-database|sha256:ccd93bed3a0018f892abc5db92bf182c3e2e986f41a8ded168915b4d9e85fa18|1||26642|/gnu/store/mzfkrxd4w8vqrmyrx169wj8wyw7r8i37-bash|sha256:6f887d45fa0f7e59e55c6d7ba86a3d8c35369c7afbb3a5829b8ed226bfef4a66|1||13518803|/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0|sha256:dc3e6b577e995c093849454fe4b9c30e87ece6462c54369ff249121f178c5476|1||127188084|/gnu/store/6z4h8g6s73zmcn0h6sm63p9d47ih4b63-info-dir|sha256:fd154332e6d35c95f12e41d6168a93945b4737fcbaf734bcaf04ca771a106544|1||4325|/gnu/store/mj7szx64bdh080rlhzhrvcg6viyba73s-profile|sha256:e32ae4d07fc2298a72e6942f21ba85c0300a84bbaf5794a8c5f0393a8dc9f253|1||2960sqlite> pragma synchronous = normal;sqlite> C-c C-z^Z[1]+ Stopped sqlite3 db.sqliteroot@childhurd ~# sqlite3 db.sqlite SQLite version 3.31.1 2020-01-27 19:55:54Enter ".help" for usage hints.sqlite> pragma synchronous = normal;Error: locking protocolsqlite>
On Debian/Hurd (which I hoped had a patched, fixed sqlite3), it doesn'twork at all
Toggle snippet (25 lines)$ ssh hurdGNU debian 0.9 GNU-Mach 1.8+git20191117-486/Hurd-0.9 i686-AT386 This is the GNU Hurd. Welcome.
The Hurd is not Linux. Make sure to readhttp://www.debian.org/ports/hurd/hurd-installto check out the few things you _need_ to know.Also check out the FAQhttp://www.gnu.org/software/hurd/faq.htmlor its latest version onhttp://darnassus.sceen.net/~hurd-web/faq/
To read a short intro on some nice features of the Hurd, just have a look atthe translator_primer file, for example via 'nano translator_primer'Last login: Wed Jul 1 12:58:44 2020 from 10.0.2.2janneke@debian:~$ sqlite3 db.sqliteSQLite version 3.32.3 2020-06-18 14:00:33Enter ".help" for usage hints.sqlite> SELECT * FROM ValidPaths;Error: unable to open database filesqlite>
So, still a similar question...is the patch simply broken/bitrotted, ordid some file locking change on the Hurd?
Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 2 Jul 14:49 +0200
Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87imf6rsir.fsf@gnu.org
Hi!
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
Toggle quote (6 lines)> $ guix offload test> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...> guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")> guix offload: 'childhurd' is running GNU Guile 3.0.4> guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
Does sqlite pass its tests on GNU/Hurd?
Does it help if you set ‘settings.fsyncMetadata = false’ in the daemon?As a stop-gap, we could add a command-line option if that helps.
The “synchronous = normal” mode translates to ‘fsync’ calls, right? Ifyou rpctrace sqlite3, do you see ‘file_sync’ calls failing?
My 2¢!
Ludo’.
J
J
Jan Nieuwenhuizen wrote on 3 Jul 12:03 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
873668zzip.fsf@gnu.org
Ludovic Courtès writes:
Hi!
Toggle quote (13 lines)> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:>>> $ guix offload test>> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...>> guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")>> guix offload: 'childhurd' is running GNU Guile 3.0.4>> guix offload: error: exception occurred on remote host 'localhost':>> (%exception #<inferior-object #<&store-protocol-error message:>> "setting synchronous mode: locking protocol" status: 1>>)
> Does it help if you set ‘settings.fsyncMetadata = false’ in the daemon?> As a stop-gap, we could add a command-line option if that helps.
Tried that, thanks. No, it does not help. (But that's good news, see below!)
Toggle quote (3 lines)> The “synchronous = normal” mode translates to ‘fsync’ calls, right? If> you rpctrace sqlite3, do you see ‘file_sync’ calls failing?
Tried that before, rpctrace hangs before I see something useful.
Toggle quote (2 lines)> Does sqlite pass its tests on GNU/Hurd?
That's the (or at least a) right question: YES!
Toggle quote (2 lines)> My 2¢!
Thanks -- it seems that buys us a pretty cheap fix after all, $-wise ;)
It turns out that Debian's patch (and thus this patch series) isprobably OK: It fixes the locking problem on the Hurd, while exposinganother bug, apparently: "unable to open database file".
It seems there is a compatibility bug/problem/thing with the db.sqlitethat we produce on GNU/Linux. While an unpatched sqlite3 on the Hurdcan read it, and work with it, the unpatched sqlite has lockingproblems. I found a workaround, though: dumping and loading thedatabase file.
Look...
Toggle snippet (26 lines)$ scp childhurd2:/var/guix/db/db.sqlite db.sqlite-origdb.sqlite 100% 144KB 5.8MB/s 00:00 11:30:37 janneke@dundal:~/tmp [env]$ sqlite3 db.sqlite-orig .dump > db.dump11:30:45 janneke@dundal:~/tmp [env]$ sqlite3 -init db.dump db.sqlite-init .quit-- Loading resources from db.dump11:30:49 janneke@dundal:~/tmp [env]$ cmp db.sqlite-orig db.sqlite-init db.sqlite-orig db.sqlite-init differ: byte 19, line 1[1]11:31:11 janneke@dundal:~/tmp [env]$ scp db.sqlite-init childhurd2:/var/guix/db/db.sqlitedb.sqlite-init 100% 144KB 7.3MB/s 00:00 11:31:21 janneke@dundal:~/tmp [env]$ guix offload testguix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...guix offload: Guix is usable on 'localhost' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")guix offload: 'localhost' is running GNU Guile 3.0.4sending 1 store item (0 MiB) to 'localhost'...exporting path `/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'guix offload: 'localhost' successfully imported '/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'retrieving 1 store item from 'localhost'...guix offload: successfully imported '/gnu/store/gxz6hzyc1cy3m1w9l7f2dk6rcspvymxf-import-test' from 'localhost'11:31:29 janneke@dundal:~/tmp [env]
So...about the compatibility problem. I tried to diff the db.sqlite-origdb.sqlite-init binary files: they look completely different. Not surehow to handle this workaround, maybe we can insert a two system* callssomewhere when building the disk image?
Greetings,Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 3 Jul 15:51 +0200
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87fta8ogf2.fsf@gnu.org
Hi!
Jan Nieuwenhuizen <janneke@gnu.org> skribis:
Toggle quote (2 lines)> Thanks -- it seems that buys us a pretty cheap fix after all, $-wise ;)
Heheh. :-)
Toggle quote (42 lines)> It turns out that Debian's patch (and thus this patch series) is> probably OK: It fixes the locking problem on the Hurd, while exposing> another bug, apparently: "unable to open database file".>> It seems there is a compatibility bug/problem/thing with the db.sqlite> that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd> can read it, and work with it, the unpatched sqlite has locking> problems. I found a workaround, though: dumping and loading the> database file.>> Look...>> $ scp childhurd2:/var/guix/db/db.sqlite db.sqlite-orig> db.sqlite 100% 144KB 5.8MB/s 00:00 > 11:30:37 janneke@dundal:~/tmp [env]> $ sqlite3 db.sqlite-orig .dump > db.dump> 11:30:45 janneke@dundal:~/tmp [env]> $ sqlite3 -init db.dump db.sqlite-init .quit> -- Loading resources from db.dump> 11:30:49 janneke@dundal:~/tmp [env]> $ cmp db.sqlite-orig db.sqlite-init > db.sqlite-orig db.sqlite-init differ: byte 19, line 1> [1]11:31:11 janneke@dundal:~/tmp [env]> $ scp db.sqlite-init childhurd2:/var/guix/db/db.sqlite> db.sqlite-init 100% 144KB 7.3MB/s 00:00 > 11:31:21 janneke@dundal:~/tmp [env]> $ guix offload test> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...> guix offload: Guix is usable on 'localhost' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")> guix offload: 'localhost' is running GNU Guile 3.0.4> sending 1 store item (0 MiB) to 'localhost'...> exporting path `/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'> guix offload: 'localhost' successfully imported '/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'> retrieving 1 store item from 'localhost'...> guix offload: successfully imported '/gnu/store/gxz6hzyc1cy3m1w9l7f2dk6rcspvymxf-import-test' from 'localhost'> 11:31:29 janneke@dundal:~/tmp [env]>> So...about the compatibility problem. I tried to diff the db.sqlite-orig> db.sqlite-init binary files: they look completely different. Not sure> how to handle this workaround, maybe we can insert a two system* calls> somewhere when building the disk image?
Weird, weird!
Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ createdfrom the same ‘sqlite3 -init’ command on GNU/Linux? (Perhaps loadingthe dump reorders entries or something.)
I think the binary format of the database is supposed to bearchitecture-independent and filling it is supposed to be deterministic.
Congrats for getting this far anyway!
Ludo’.
J
J
Jan Nieuwenhuizen wrote on 3 Jul 17:27 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87sge8wrdf.fsf@gnu.org
Ludovic Courtès writes:
Hi!
Toggle quote (6 lines)>> It seems there is a compatibility bug/problem/thing with the db.sqlite>> that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd>> can read it, and work with it, the unpatched sqlite has locking>> problems. I found a workaround, though: dumping and loading the>> database file.
[..]
Toggle quote (10 lines)>> So...about the compatibility problem. I tried to diff the db.sqlite-orig>> db.sqlite-init binary files: they look completely different. Not sure>> how to handle this workaround, maybe we can insert a two system* calls>> somewhere when building the disk image?>> Weird, weird!>> Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created> from the same ‘sqlite3 -init’ command on GNU/Linux?
Yeah, they are identical. The initial dump can only be created atm onGNU/Linux; the dump can be loaded (obviously) anywhere like so
Toggle quote (2 lines)>> $ sqlite3 -init db.dump db.sqlite-init .quit
and the resulting initial db.sqlite is the same. Guess we can do thatby hand for now...
Toggle quote (2 lines)> (Perhaps loading the dump reorders entries or something.)
Yes, "or something" certainly! :) I have no clue...
If/when we decide to pinpoint this bug, what could be a first step? Whois creating the database right now, is that the C++ daemon orguile-sqlite3. IWBN to have that code create/save a smaller version andsee when reading fails.
Toggle quote (3 lines)> I think the binary format of the database is supposed to be> architecture-independent and filling it is supposed to be deterministic.
Yes, that works.
Toggle quote (2 lines)> Congrats for getting this far anyway!
Yeah... \o/
Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 4 Jul 10:12 +0200
[PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87y2nzr94w.fsf_-_@gnu.org
Jan Nieuwenhuizen writes:
Adding one additional patch to this series, hence 4/3 (see attached)fixes the sqlite problem.
With this over-complete series we can now do offload builds to achildhurd:
Toggle snippet (17 lines)$ guix build -e '(@@ (gnu packages commencement) rsync-boot0)'sending 74 store items (365 MiB) to 'localhost'...exporting path `/gnu/store/1kf05mhh31wl89r3mlx52x5sxrjivi7i-module-import-compiled-guile-builder'exporting path `/gnu/store/7jxamp3g8wdr6vwrisqfmrncxh8nbfb4-mirrors'[...]exporting path `/gnu/store/x8wnf0h0ipibgzadlbmll8bcnhvv1yaq-diffutils-boot0-3.7'offloading '/gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv' to 'localhost'...offloading build of /gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv to 'localhost'[...]@ build-succeeded /gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv -retrieving 1 store item from 'localhost'...importing file or directory '/gnu/store/haf6mlm8xa6s2q918s05pijl6ql17mnq-rsync-3.1.3'...guix offload: error: corrupt input while restoring archive from #<input: string 7ff4794cee00>guix build: error: build of `/gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv' failed[1]10:04:55 janneke@dundal:~/src/guix/master [env]
After the build succeeds, the download fails but lets first get thispatch series done...
Janneke
From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>Date: Fri, 3 Jul 2020 23:45:20 +0200Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.Content-Transfer-Encoding: 8bitContent-Type: text/plain; charset=UTF-8
This fixes https://bugs.gnu.org/42151.
* guix/store/database.scm (call-with-database): When building for the Hurd,do not set journal_model=WAL.--- guix/store/database.scm | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
Toggle diff (47 lines)diff --git a/guix/store/database.scm b/guix/store/database.scmindex a38e4d7e52..da46b0abce 100644--- a/guix/store/database.scm+++ b/guix/store/database.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org> ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -20,6 +21,7 @@ (define-module (guix store database) #:use-module (sqlite3) #:use-module (guix config)+ #:use-module (guix gexp) #:use-module (guix serialization) #:use-module (guix store deduplication) #:use-module (guix base16)@@ -27,6 +29,7 @@ #:use-module (guix build syscalls) #:use-module ((guix build utils) #:select (mkdir-p executable-file?))+ #:use-module (guix utils) #:use-module (guix build store-copy) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11)@@ -105,9 +108,12 @@ create it and initialize it as a new database." (mkdir-p (dirname file)) #t))) (db (sqlite-open file)))- ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED- ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.- (sqlite-exec db "PRAGMA journal_mode=WAL;")+ ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.+ (unless (let-system (system target)+ (equal? target "i586-pc-gnu"))+ ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED+ ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.+ (sqlite-exec db "PRAGMA journal_mode=WAL;")) ;; Install a busy handler such that, when the database is locked, sqlite ;; retries until 30 seconds have passed, at which point it gives up and-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 7 Jul 07:04 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87d057apb4.fsf@gnu.org
Jan Nieuwenhuizen writes:
Toggle quote (5 lines)> Jan Nieuwenhuizen writes:>> Adding one additional patch to this series, hence 4/3 (see attached)> fixes the sqlite problem.
For clarity -- patches 1-4 really do fix offloading to the Hurd and this...
Toggle quote (3 lines)> After the build succeeds, the download fails but lets first get this> patch series done...
...is not true; you gat that when you (I) forgot to add keys in/etc/guix/
As seen on IRC, possibly Ricardo will an update for the QEMUcommand-line (at least for berlin), adding "--cpu base".
Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 10 Jul 14:03 +0200
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87v9iv1ssj.fsf@gnu.org
Hi!
Jan Nieuwenhuizen <janneke@gnu.org> skribis:
Toggle quote (57 lines)>>From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001> From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>> Date: Fri, 3 Jul 2020 23:45:20 +0200> Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.> Content-Transfer-Encoding: 8bit> Content-Type: text/plain; charset=UTF-8>> This fixes <https://bugs.gnu.org/42151>.>> * guix/store/database.scm (call-with-database): When building for the Hurd,> do not set journal_model=WAL.> ---> guix/store/database.scm | 12 +++++++++---> 1 file changed, 9 insertions(+), 3 deletions(-)>> diff --git a/guix/store/database.scm b/guix/store/database.scm> index a38e4d7e52..da46b0abce 100644> --- a/guix/store/database.scm> +++ b/guix/store/database.scm> @@ -1,6 +1,7 @@> ;;; GNU Guix --- Functional package management for GNU> ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>> ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>> ;;;> ;;; This file is part of GNU Guix.> ;;;> @@ -20,6 +21,7 @@> (define-module (guix store database)> #:use-module (sqlite3)> #:use-module (guix config)> + #:use-module (guix gexp)> #:use-module (guix serialization)> #:use-module (guix store deduplication)> #:use-module (guix base16)> @@ -27,6 +29,7 @@> #:use-module (guix build syscalls)> #:use-module ((guix build utils)> #:select (mkdir-p executable-file?))> + #:use-module (guix utils)> #:use-module (guix build store-copy)> #:use-module (srfi srfi-1)> #:use-module (srfi srfi-11)> @@ -105,9 +108,12 @@ create it and initialize it as a new database."> (mkdir-p (dirname file))> #t)))> (db (sqlite-open file)))> - ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED> - ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.> - (sqlite-exec db "PRAGMA journal_mode=WAL;")> + ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.> + (unless (let-system (system target)> + (equal? target "i586-pc-gnu"))> + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED> + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.> + (sqlite-exec db "PRAGMA journal_mode=WAL;"))
I think this has the same effect as:
(unless #t (sqlite-exec db "PRAGMA journal_mode=WAL;"))
because this code is not in a gexp, so ‘let-system’ evaluates to anobject (not #f), and that’s it, it’s never lowered and the ‘equal?’ callis never made.
You want to disable WAL mode not just when running this code natively onGNU/Hurd, but also when building a database that will eventually be usedon GNU/Hurd, right?
In that case, I think you’ll have to add, say, a #:wal-mode? parameterto ‘call-with-database’ (defaulting to #true), and change the caller toset it appropriately. The caller, directly or indirectly, is in a gexp,where you can use the ‘let-system’ expression above.
Does that make sense?
Thanks, and apologies for the delay!
Ludo’.
L
L
Ludovic Courtès wrote on 10 Jul 14:05 +0200
Re: bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87r1tj1so4.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
Toggle quote (4 lines)> * gnu/packages/patches/sqlite3-hurd.patch: New file.> * gnu/local.mk (dist_patch_DATA): Add it.> * gnu/packages/sqlite.scm (sqlite/hurd): New variable.
[…]
Toggle quote (5 lines)> +++ b/gnu/packages/patches/sqlite3-hurd.patch> @@ -0,0 +1,51 @@> +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch> +Upstream status: Not offered upstream.
The URL is 404, could you fix it? It would also be great if you couldadd a sentence or two explaining what this works around, so that ourfuture selves know how to deal with this patch. :-)
Otherwise LGTM, you can push with changes along these lines!
Ludo’.
L
L
Ludovic Courtès wrote on 10 Jul 14:06 +0200
Re: bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87mu471smz.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
Toggle quote (21 lines)> This fixes>> guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)>> * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of> sqlite.> ---> gnu/packages/guile.scm | 2 +-> 1 file changed, 1 insertion(+), 1 deletion(-)>> diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm> index a32bd33293..c651e2427a 100644> --- a/gnu/packages/guile.scm> +++ b/gnu/packages/guile.scm> @@ -665,7 +665,7 @@ Guile's foreign function interface.")> ("pkg-config" ,pkg-config)))> (inputs> `(("guile" ,guile-3.0)> - ("sqlite" ,sqlite)))> + ("sqlite" ,sqlite/hurd)))
This should be guarded by ‘if (hurd-target?)’, right?
Ludo’.
L
L
Ludovic Courtès wrote on 10 Jul 14:07 +0200
Re: bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87imev1sl6.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
Toggle quote (2 lines)> * gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.
Same here.
Which makes me thing: how about applying the sqlite/hurd patch in abuild phase of sqlite that would be Hurd-specific? That way, we’d stillhave a single sqlite package.
Thanks,Ludo’.
L
L
Ludovic Courtès wrote on 10 Jul 14:34 +0200
Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87sgdzzgyz.fsf@gnu.org
Hi,
Jan Nieuwenhuizen <janneke@gnu.org> skribis:
Toggle quote (34 lines)> Ludovic Courtès writes:>> Hi!>>>> It seems there is a compatibility bug/problem/thing with the db.sqlite>>> that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd>>> can read it, and work with it, the unpatched sqlite has locking>>> problems. I found a workaround, though: dumping and loading the>>> database file.>> [..]>>>> So...about the compatibility problem. I tried to diff the db.sqlite-orig>>> db.sqlite-init binary files: they look completely different. Not sure>>> how to handle this workaround, maybe we can insert a two system* calls>>> somewhere when building the disk image?>>>> Weird, weird!>>>> Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created>> from the same ‘sqlite3 -init’ command on GNU/Linux?>> Yeah, they are identical. The initial dump can only be created atm on> GNU/Linux; the dump can be loaded (obviously) anywhere like so>>>> $ sqlite3 -init db.dump db.sqlite-init .quit>> and the resulting initial db.sqlite is the same. Guess we can do that> by hand for now...>>> (Perhaps loading the dump reorders entries or something.)>> Yes, "or something" certainly! :) I have no clue...
Sorry for catching up days later but… did disabling WAL mode fix thisdiscrepancy? IOW, without WAL mode, is the file produced on GNU/Linuxbit-identical to that produced on GNU/Hurd?
Toggle quote (5 lines)> If/when we decide to pinpoint this bug, what could be a first step? Who> is creating the database right now, is that the C++ daemon or> guile-sqlite3. IWBN to have that code create/save a smaller version and> see when reading fails.
As you found out it’s created by (guix store database) in this case.
Thanks,Ludo’.
J
J
Jan Nieuwenhuizen wrote on 10 Jul 17:40 +0200
Re: bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87a707v0ns.fsf@gnu.org
Ludovic Courtès writes:
Hi!
Toggle quote (10 lines)> Jan Nieuwenhuizen <janneke@gnu.org> skribis:>>>>From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001>> From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>>> Date: Fri, 3 Jul 2020 23:45:20 +0200>> Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.>> Content-Transfer-Encoding: 8bit>> Content-Type: text/plain; charset=UTF-8>>>> This fixes <https://bugs.gnu.org/42151>.
[...]
Toggle quote (4 lines)>> diff --git a/guix/store/database.scm b/guix/store/database.scm>> index a38e4d7e52..da46b0abce 100644>> --- a/guix/store/database.scm>> +++ b/guix/store/database.scm
[...]
Toggle quote (16 lines)>> + ;; Using WAL breaks for the Hurd https://bugs.gnu.org/42151.>> + (unless (let-system (system target)>> + (equal? target "i586-pc-gnu"))>> + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED>> + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.>> + (sqlite-exec db "PRAGMA journal_mode=WAL;"))>> I think this has the same effect as:>> (unless #t> (sqlite-exec db "PRAGMA journal_mode=WAL;"))>> because this code is not in a gexp, so ‘let-system’ evaluates to an> object (not #f), and that’s it, it’s never lowered and the ‘equal?’ call> is never made.
Oops. Great catch, thanks!
Toggle quote (4 lines)> You want to disable WAL mode not just when running this code natively on> GNU/Hurd, but also when building a database that will eventually be used> on GNU/Hurd, right?
Exactly.
Toggle quote (7 lines)> In that case, I think you’ll have to add, say, a #:wal-mode? parameter> to ‘call-with-database’ (defaulting to #true), and change the caller to> set it appropriately. The caller, directly or indirectly, is in a gexp,> where you can use the ‘let-system’ expression above.>> Does that make sense?
Yes, very much so. [...] After some typing, it turns out that the wholelet-system is gone, we can set use a hard-coded #:wal-mode? #t inhurd-initialize-root-partition.
Toggle quote (2 lines)> Thanks, and apologies for the delay!
Sure, I'm happy I didn't "just push" ;-)
I'm sending a v2 patch set in a minute.
Greetings,Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 17:42 +0200
[PATCH v2 2/4] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710154205.31342-2-janneke@gnu.org
This fixes
guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
* gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead ofsqlite.--- gnu/packages/guile.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Toggle diff (16 lines)diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scmindex a32bd33293..c651e2427a 100644--- a/gnu/packages/guile.scm+++ b/gnu/packages/guile.scm@@ -665,7 +665,7 @@ Guile's foreign function interface.") ("pkg-config" ,pkg-config))) (inputs `(("guile" ,guile-3.0)- ("sqlite" ,sqlite)))+ ("sqlite" ,sqlite/hurd))) (synopsis "Access SQLite databases from Guile") (description "This package provides Guile bindings to the SQLite database system.")-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 17:42 +0200
[PATCH v2 3/4] gnu: guix: Use sqlite/hurd for locking on the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710154205.31342-3-janneke@gnu.org
* gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.--- gnu/packages/package-management.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Toggle diff (16 lines)diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scmindex 9986976cc6..64c6931011 100644--- a/gnu/packages/package-management.scm+++ b/gnu/packages/package-management.scm@@ -343,7 +343,7 @@ $(prefix)/etc/init.d\n"))) ("zlib" ,zlib) ;for 'guix publish' ("lzlib" ,lzlib) ;for 'guix publish' and 'guix substitute' - ("sqlite" ,sqlite)+ ("sqlite" ,sqlite/hurd) ("libgcrypt" ,libgcrypt) ("guile" ,guile-3.0-latest)-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 17:42 +0200
[PATCH v2 1/4] gnu: Add sqlite/hurd with locking fix.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710154205.31342-1-janneke@gnu.org
* gnu/packages/patches/sqlite3-hurd.patch: New file.* gnu/local.mk (dist_patch_DATA): Add it.* gnu/packages/sqlite.scm (sqlite/hurd): Use it in new variable.--- gnu/local.mk | 1 + gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++ gnu/packages/sqlite.scm | 9 +++++ 3 files changed, 61 insertions(+) create mode 100644 gnu/packages/patches/sqlite3-hurd.patch
Toggle diff (96 lines)diff --git a/gnu/local.mk b/gnu/local.mkindex 5c3b391960..e452004945 100644--- a/gnu/local.mk+++ b/gnu/local.mk@@ -1370,6 +1370,7 @@ dist_patch_DATA = \ %D%/packages/patches/sdl-pango-header-guard.patch \ %D%/packages/patches/sdl-pango-matrix_declarations.patch \ %D%/packages/patches/sdl-pango-sans-serif.patch \+ %D%/packages/patches/sqlite3-hurd.patch \ %D%/packages/patches/patchutils-test-perms.patch \ %D%/packages/patches/patch-hurd-path-max.patch \ %D%/packages/patches/perl-autosplit-default-time.patch \diff --git a/gnu/packages/patches/sqlite3-hurd.patch b/gnu/packages/patches/sqlite3-hurd.patchnew file mode 100644index 0000000000..de87a30cb1--- /dev/null+++ b/gnu/packages/patches/sqlite3-hurd.patch@@ -0,0 +1,51 @@+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch+Upstream status: Not offered upstream.++diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c+--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode+ # include <sys/mman.h>+ #endif+ +-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ /* # include <sys/ioctl.h> */+ # include <sys/file.h>+ # include <sys/param.h>+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id+ **+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off+ */+-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ + /*+ ** Retry flock() calls that fail with EINTR+@@ -38586,7 +38586,7 @@ IOMETHODS(+ 0 /* xShmMap method */+ )+ +-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ IOMETHODS(+ flockIoFinder, /* Finder function name */+ flockIoMethods, /* sqlite3_io_methods object name */+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){+ UNIXVFS("unix", autolockIoFinder ),+ #elif OS_VXWORKS+ UNIXVFS("unix", vxworksIoFinder ),++#elif defined(__GNU__)++ UNIXVFS("unix", flockIoFinder ),+ #else+ UNIXVFS("unix", posixIoFinder ),+ #endif+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){+ #if OS_VXWORKS+ UNIXVFS("unix-namedsem", semIoFinder ),+ #endif+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)+ UNIXVFS("unix-posix", posixIoFinder ),+ #endif+ #if SQLITE_ENABLE_LOCKING_STYLEdiff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scmindex 8468131ddf..d5c44b0d0f 100644--- a/gnu/packages/sqlite.scm+++ b/gnu/packages/sqlite.scm@@ -11,6 +11,7 @@ ;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org> ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr> ;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -107,3 +108,11 @@ is in the public domain.") ;; commit fad5b1a6d8d9c36bea5785ae4fbc1beb37e644d7. (define-public sqlite-with-column-metadata (deprecated-package "sqlite-with-column-metadata" sqlite))++(define-public sqlite/hurd+ ;; TODO move into sqlite on the next rebuild cycle.+ (package+ (inherit sqlite)+ (name "sqlite-for-hurd")+ (source (origin (inherit (package-source sqlite))+ (patches (search-patches "sqlite3-hurd.patch"))))))-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 17:42 +0200
[PATCH v2 4/4] image: Do not set journal_model=WAL for the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710154205.31342-4-janneke@gnu.org
This fixes https://bugs.gnu.org/42151.
* gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #fin call to ...* gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?parameter, pass it to ...(register-closure): ... this, add #:wal-mode? parameter, pass it to ...* guix/store/database.scm (with-database): ... this, add #:wal-mode?parameter, pass it to ...(call-with-database): ... this, add #:wal-mode? parameter; whenset to #f, do not set journal_model=WAL.--- gnu/build/image.scm | 26 ++++++++++++++++---------- gnu/system/images/hurd.scm | 4 ++-- guix/store/database.scm | 29 ++++++++++++++++++++--------- 3 files changed, 38 insertions(+), 21 deletions(-)
Toggle diff (156 lines)diff --git a/gnu/build/image.scm b/gnu/build/image.scmindex e7b0418182..d8efa73f16 100644--- a/gnu/build/image.scm+++ b/gnu/build/image.scm@@ -131,20 +131,23 @@ given CONFIG file." (define* (register-closure prefix closure #:key (deduplicate? #t) (reset-timestamps? #t)- (schema (sql-schema)))+ (schema (sql-schema))+ (wal-mode? #t)) "Register CLOSURE in PREFIX, where PREFIX is the directory name of the target store and CLOSURE is the name of a file containing a reference graph as produced by #:references-graphs.. As a side effect, if RESET-TIMESTAMPS? is true, reset timestamps on store files and, if DEDUPLICATE? is true,-deduplicates files common to CLOSURE and the rest of PREFIX."+deduplicates files common to CLOSURE and the rest of PREFIX. Pass WAL-MODE?+to call-with-database." (let ((items (call-with-input-file closure read-reference-graph))) (parameterize ((sql-schema schema)) (with-database (store-database-file #:prefix prefix) db- (register-items db items- #:prefix prefix- #:deduplicate? deduplicate?- #:reset-timestamps? reset-timestamps?- #:registration-time %epoch)))))+ #:wal-mode? wal-mode?+ (register-items db items+ #:prefix prefix+ #:deduplicate? deduplicate?+ #:reset-timestamps? reset-timestamps?+ #:registration-time %epoch))))) (define* (initialize-efi-partition root #:key@@ -164,14 +167,16 @@ deduplicates files common to CLOSURE and the rest of PREFIX." (register-closures? #t) system-directory make-device-nodes+ (wal-mode? #t) #:allow-other-keys) "Initialize the given ROOT directory. Use BOOTCFG and BOOTCFG-LOCATION to install the bootloader configuration. If REGISTER-CLOSURES? is true, register REFERENCES-GRAPHS in the store. If DEDUPLICATE? is true, then also deduplicate files common to CLOSURES and the-rest of the store when registering the closures. SYSTEM-DIRECTORY is the name-of the directory of the 'system' derivation."+rest of the store when registering the closures. SYSTEM-DIRECTORY is the name+of the directory of the 'system' derivation. Pass WAL-MODE? to+register-closure." (populate-root-file-system system-directory root) (populate-store references-graphs root) @@ -184,7 +189,8 @@ of the directory of the 'system' derivation." (register-closure root closure #:reset-timestamps? #t- #:deduplicate? deduplicate?))+ #:deduplicate? deduplicate?+ #:wal-mode? wal-mode?)) references-graphs)) (when bootloader-installerdiff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scmindex 31942e7386..4e1db37104 100644--- a/gnu/system/images/hurd.scm+++ b/gnu/system/images/hurd.scm@@ -61,8 +61,8 @@ #~(lambda* (#:rest args) (apply initialize-root-partition (append args- (list #:make-device-nodes- make-hurd-device-nodes)))))+ (list #:make-device-nodes make-hurd-device-nodes+ #:wal-mode? #f))))) (define hurd-disk-image (imagediff --git a/guix/store/database.scm b/guix/store/database.scmindex a38e4d7e52..50b66ce282 100644--- a/guix/store/database.scm+++ b/guix/store/database.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org> ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -20,6 +21,7 @@ (define-module (guix store database) #:use-module (sqlite3) #:use-module (guix config)+ #:use-module (guix gexp) #:use-module (guix serialization) #:use-module (guix store deduplication) #:use-module (guix base16)@@ -27,6 +29,7 @@ #:use-module (guix build syscalls) #:use-module ((guix build utils) #:select (mkdir-p executable-file?))+ #:use-module (guix utils) #:use-module (guix build store-copy) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11)@@ -97,17 +100,20 @@ as specified by SQL-SCHEMA." (sqlite-exec db (call-with-input-file schema get-string-all))) -(define (call-with-database file proc)+(define* (call-with-database file proc #:key (wal-mode? #t)) "Pass PROC a database record corresponding to FILE. If FILE doesn't exist,-create it and initialize it as a new database."+create it and initialize it as a new database. Unless WAL-MODE? is set to #f,+set journal_mode=WAL." (let ((new? (and (not (file-exists? file)) (begin (mkdir-p (dirname file)) #t))) (db (sqlite-open file)))- ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED- ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.- (sqlite-exec db "PRAGMA journal_mode=WAL;")+ ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.+ (when wal-mode?+ ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED+ ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.+ (sqlite-exec db "PRAGMA journal_mode=WAL;")) ;; Install a busy handler such that, when the database is locked, sqlite ;; retries until 30 seconds have passed, at which point it gives up and@@ -200,10 +206,15 @@ prior to returning." ;; Default location of the store database. (string-append %store-database-directory "/db.sqlite")) -(define-syntax-rule (with-database file db exp ...)- "Open DB from FILE and close it when the dynamic extent of EXP... is left.-If FILE doesn't exist, create it and initialize it as a new database."- (call-with-database file (lambda (db) exp ...)))+(define-syntax with-database+ (syntax-rules ()+ "Open DB from FILE and close it when the dynamic extent of EXP... is left.+If FILE doesn't exist, create it and initialize it as a new database. Pass+#:wal-mode? to call-with-database."+ ((_ file db #:wal-mode? wal-mode? exp ...)+ (call-with-database file (lambda (db) exp ...) #:wal-mode? wal-mode?))+ ((_ file db exp ...)+ (call-with-database file (lambda (db) exp ...))))) (define (sqlite-finalize stmt) ;; As of guile-sqlite3 0.1.0, cached statements aren't reset when-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 10 Jul 18:25 +0200
Re: bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87v9ivtk07.fsf@gnu.org
Ludovic Courtès writes:
Toggle quote (10 lines)> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>> * gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.>> Same here.>> Which makes me thing: how about applying the sqlite/hurd patch in a> build phase of sqlite that would be Hurd-specific? That way, we’d still> have a single sqlite package.
Ah, yes that sounds better. We don't need sqlite/hurd "sqlite-for-hurd"and all that. I'll give that a go!
Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 10 Jul 20:34 +0200
Re: bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87sgdzte1s.fsf@gnu.org
Ludovic Courtès writes:
Toggle quote (15 lines)> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:>>> * gnu/packages/patches/sqlite3-hurd.patch: New file.>> * gnu/local.mk (dist_patch_DATA): Add it.>> * gnu/packages/sqlite.scm (sqlite/hurd): New variable.>> […]>>> +++ b/gnu/packages/patches/sqlite3-hurd.patch>> @@ -0,0 +1,51 @@>> +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch>> +Upstream status: Not offered upstream.>> The URL is 404, could you fix it?
That's weird...Changing it (in completely rewritten patch) to
https://sources.debian.org/patches/sqlite3/3.32.3-1/20-hurd-locking-style.patch/
Toggle quote (4 lines)> It would also be great if you could> add a sentence or two explaining what this works around, so that our> future selves know how to deal with this patch. :-)
Ah, sure!
Toggle quote (2 lines)> Otherwise LGTM, you can push with changes along these lines!
As it's so much changed, I'll send it by one more time :-)
Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 10 Jul 20:34 +0200
Re: bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87r1tjte0t.fsf@gnu.org
Ludovic Courtès writes:
Toggle quote (25 lines)> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:>>> This fixes>>>> guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)>>>> * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of>> sqlite.>> --->> gnu/packages/guile.scm | 2 +->> 1 file changed, 1 insertion(+), 1 deletion(-)>>>> diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm>> index a32bd33293..c651e2427a 100644>> --- a/gnu/packages/guile.scm>> +++ b/gnu/packages/guile.scm>> @@ -665,7 +665,7 @@ Guile's foreign function interface.")>> ("pkg-config" ,pkg-config)))>> (inputs>> `(("guile" ,guile-3.0)>> - ("sqlite" ,sqlite)))>> + ("sqlite" ,sqlite/hurd)))>> This should be guarded by ‘if (hurd-target?)’, right?
Yeah...but all changed now...look out for v3. Thanks!
Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 20:35 +0200
[PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710183551.12807-2-janneke@gnu.org
This fixes https://bugs.gnu.org/42151.
* gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #fin call to ...* gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?parameter, pass it to ...(register-closure): ... this, add #:wal-mode? parameter, pass it to ...* guix/store/database.scm (with-database): ... this, add #:wal-mode?parameter, pass it to ...(call-with-database): ... this, add #:wal-mode? parameter; whenset to #f, do not set journal_model=WAL.--- gnu/build/image.scm | 26 ++++++++++++++++---------- gnu/system/images/hurd.scm | 4 ++-- guix/store/database.scm | 29 ++++++++++++++++++++--------- 3 files changed, 38 insertions(+), 21 deletions(-)
Toggle diff (156 lines)diff --git a/gnu/build/image.scm b/gnu/build/image.scmindex e7b0418182..d8efa73f16 100644--- a/gnu/build/image.scm+++ b/gnu/build/image.scm@@ -131,20 +131,23 @@ given CONFIG file." (define* (register-closure prefix closure #:key (deduplicate? #t) (reset-timestamps? #t)- (schema (sql-schema)))+ (schema (sql-schema))+ (wal-mode? #t)) "Register CLOSURE in PREFIX, where PREFIX is the directory name of the target store and CLOSURE is the name of a file containing a reference graph as produced by #:references-graphs.. As a side effect, if RESET-TIMESTAMPS? is true, reset timestamps on store files and, if DEDUPLICATE? is true,-deduplicates files common to CLOSURE and the rest of PREFIX."+deduplicates files common to CLOSURE and the rest of PREFIX. Pass WAL-MODE?+to call-with-database." (let ((items (call-with-input-file closure read-reference-graph))) (parameterize ((sql-schema schema)) (with-database (store-database-file #:prefix prefix) db- (register-items db items- #:prefix prefix- #:deduplicate? deduplicate?- #:reset-timestamps? reset-timestamps?- #:registration-time %epoch)))))+ #:wal-mode? wal-mode?+ (register-items db items+ #:prefix prefix+ #:deduplicate? deduplicate?+ #:reset-timestamps? reset-timestamps?+ #:registration-time %epoch))))) (define* (initialize-efi-partition root #:key@@ -164,14 +167,16 @@ deduplicates files common to CLOSURE and the rest of PREFIX." (register-closures? #t) system-directory make-device-nodes+ (wal-mode? #t) #:allow-other-keys) "Initialize the given ROOT directory. Use BOOTCFG and BOOTCFG-LOCATION to install the bootloader configuration. If REGISTER-CLOSURES? is true, register REFERENCES-GRAPHS in the store. If DEDUPLICATE? is true, then also deduplicate files common to CLOSURES and the-rest of the store when registering the closures. SYSTEM-DIRECTORY is the name-of the directory of the 'system' derivation."+rest of the store when registering the closures. SYSTEM-DIRECTORY is the name+of the directory of the 'system' derivation. Pass WAL-MODE? to+register-closure." (populate-root-file-system system-directory root) (populate-store references-graphs root) @@ -184,7 +189,8 @@ of the directory of the 'system' derivation." (register-closure root closure #:reset-timestamps? #t- #:deduplicate? deduplicate?))+ #:deduplicate? deduplicate?+ #:wal-mode? wal-mode?)) references-graphs)) (when bootloader-installerdiff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scmindex 31942e7386..4e1db37104 100644--- a/gnu/system/images/hurd.scm+++ b/gnu/system/images/hurd.scm@@ -61,8 +61,8 @@ #~(lambda* (#:rest args) (apply initialize-root-partition (append args- (list #:make-device-nodes- make-hurd-device-nodes)))))+ (list #:make-device-nodes make-hurd-device-nodes+ #:wal-mode? #f))))) (define hurd-disk-image (imagediff --git a/guix/store/database.scm b/guix/store/database.scmindex a38e4d7e52..50b66ce282 100644--- a/guix/store/database.scm+++ b/guix/store/database.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org> ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -20,6 +21,7 @@ (define-module (guix store database) #:use-module (sqlite3) #:use-module (guix config)+ #:use-module (guix gexp) #:use-module (guix serialization) #:use-module (guix store deduplication) #:use-module (guix base16)@@ -27,6 +29,7 @@ #:use-module (guix build syscalls) #:use-module ((guix build utils) #:select (mkdir-p executable-file?))+ #:use-module (guix utils) #:use-module (guix build store-copy) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11)@@ -97,17 +100,20 @@ as specified by SQL-SCHEMA." (sqlite-exec db (call-with-input-file schema get-string-all))) -(define (call-with-database file proc)+(define* (call-with-database file proc #:key (wal-mode? #t)) "Pass PROC a database record corresponding to FILE. If FILE doesn't exist,-create it and initialize it as a new database."+create it and initialize it as a new database. Unless WAL-MODE? is set to #f,+set journal_mode=WAL." (let ((new? (and (not (file-exists? file)) (begin (mkdir-p (dirname file)) #t))) (db (sqlite-open file)))- ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED- ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.- (sqlite-exec db "PRAGMA journal_mode=WAL;")+ ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.+ (when wal-mode?+ ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED+ ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.+ (sqlite-exec db "PRAGMA journal_mode=WAL;")) ;; Install a busy handler such that, when the database is locked, sqlite ;; retries until 30 seconds have passed, at which point it gives up and@@ -200,10 +206,15 @@ prior to returning." ;; Default location of the store database. (string-append %store-database-directory "/db.sqlite")) -(define-syntax-rule (with-database file db exp ...)- "Open DB from FILE and close it when the dynamic extent of EXP... is left.-If FILE doesn't exist, create it and initialize it as a new database."- (call-with-database file (lambda (db) exp ...)))+(define-syntax with-database+ (syntax-rules ()+ "Open DB from FILE and close it when the dynamic extent of EXP... is left.+If FILE doesn't exist, create it and initialize it as a new database. Pass+#:wal-mode? to call-with-database."+ ((_ file db #:wal-mode? wal-mode? exp ...)+ (call-with-database file (lambda (db) exp ...) #:wal-mode? wal-mode?))+ ((_ file db exp ...)+ (call-with-database file (lambda (db) exp ...))))) (define (sqlite-finalize stmt) ;; As of guile-sqlite3 0.1.0, cached statements aren't reset when-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 20:35 +0200
[PATCH v3 1/2] gnu: sqlite: Add locking-mode fix for the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710183551.12807-1-janneke@gnu.org
* gnu/packages/patches/sqlite-hurd.patch: New file.* gnu/local.mk (dist_patch_DATA): Add it.* gnu/packages/sqlite.scm (sqlite)[native-inputs]: Add it when buildingfor the Hurd.[arguments]: Apply it when building for the Hurd.--- gnu/local.mk | 1 + gnu/packages/patches/sqlite-hurd.patch | 58 ++++++++++++++++++++++++++ gnu/packages/sqlite.scm | 19 +++++++++ 3 files changed, 78 insertions(+) create mode 100644 gnu/packages/patches/sqlite-hurd.patch
Toggle diff (130 lines)diff --git a/gnu/local.mk b/gnu/local.mkindex 5c3b391960..42c9d0f379 100644--- a/gnu/local.mk+++ b/gnu/local.mk@@ -1370,6 +1370,7 @@ dist_patch_DATA = \ %D%/packages/patches/sdl-pango-header-guard.patch \ %D%/packages/patches/sdl-pango-matrix_declarations.patch \ %D%/packages/patches/sdl-pango-sans-serif.patch \+ %D%/packages/patches/sqlite-hurd.patch \ %D%/packages/patches/patchutils-test-perms.patch \ %D%/packages/patches/patch-hurd-path-max.patch \ %D%/packages/patches/perl-autosplit-default-time.patch \diff --git a/gnu/packages/patches/sqlite-hurd.patch b/gnu/packages/patches/sqlite-hurd.patchnew file mode 100644index 0000000000..d80a2c5be8--- /dev/null+++ b/gnu/packages/patches/sqlite-hurd.patch@@ -0,0 +1,58 @@+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.32.3-1/20-hurd-locking-style.patch+Upstream status: Not upstreamed.++This patch is needed to get offloading to work.++Sqlite can use simple file locking mode, but that does not work for the Hurd;+a second sqlite process fails with a "locking protocol" error.++See also: https://bugs.debian.org/529734.++diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c+--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode+ # include <sys/mman.h>+ #endif+ +-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ /* # include <sys/ioctl.h> */+ # include <sys/file.h>+ # include <sys/param.h>+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id+ **+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off+ */+-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ + /*+ ** Retry flock() calls that fail with EINTR+@@ -38586,7 +38586,7 @@ IOMETHODS(+ 0 /* xShmMap method */+ )+ +-#if SQLITE_ENABLE_LOCKING_STYLE++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)+ IOMETHODS(+ flockIoFinder, /* Finder function name */+ flockIoMethods, /* sqlite3_io_methods object name */+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){+ UNIXVFS("unix", autolockIoFinder ),+ #elif OS_VXWORKS+ UNIXVFS("unix", vxworksIoFinder ),++#elif defined(__GNU__)++ UNIXVFS("unix", flockIoFinder ),+ #else+ UNIXVFS("unix", posixIoFinder ),+ #endif+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){+ #if OS_VXWORKS+ UNIXVFS("unix-namedsem", semIoFinder ),+ #endif+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)+ UNIXVFS("unix-posix", posixIoFinder ),+ #endif+ #if SQLITE_ENABLE_LOCKING_STYLEdiff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scmindex 8468131ddf..eeb77749d8 100644--- a/gnu/packages/sqlite.scm+++ b/gnu/packages/sqlite.scm@@ -11,6 +11,7 @@ ;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org> ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr> ;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org> ;;; ;;; This file is part of GNU Guix. ;;;@@ -29,6 +30,7 @@ (define-module (gnu packages sqlite) #:use-module (gnu packages)+ #:use-module (gnu packages hurd) #:use-module (gnu packages readline) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix packages)@@ -65,6 +67,11 @@ "1bj936svd8i5g25xd1bj52hj4zca01fgl3sqkj86z9q5pkz4wa32")))) (build-system gnu-build-system) (inputs `(("readline" ,readline)))+ (native-inputs (if (hurd-target?)+ ;; TODO move into origin on the next rebuild cycle.+ `(("hurd-locking-mode.patch"+ ,@(search-patches "sqlite-hurd.patch")))+ '())) (outputs '("out" "static")) (arguments `(#:configure-flags@@ -79,6 +86,18 @@ ;; Column metadata is required by GNU Jami and Qt, et.al. "-DSQLITE_ENABLE_COLUMN_METADATA")) #:phases (modify-phases %standard-phases+ ;; TODO: remove in the next rebuild cycle+ ,@(if (hurd-target?)+ `((add-after 'unpack 'patch-sqlite/hurd+ (lambda* (#:key inputs native-inputs+ #:allow-other-keys)+ (let ((patch (assoc-ref+ (if ,(%current-target-system)+ native-inputs+ inputs)+ "hurd-locking-mode.patch")))+ (invoke "patch" "-p1" "--force" "-i" patch)))))+ '()) (add-after 'install 'move-static-library (lambda* (#:key outputs #:allow-other-keys) (let* ((out (assoc-ref outputs "out"))-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 11 Jul 16:18 +0200
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87v9iuw2x5.fsf@gnu.org
Hi!
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
Toggle quote (6 lines)> * gnu/packages/patches/sqlite-hurd.patch: New file.> * gnu/local.mk (dist_patch_DATA): Add it.> * gnu/packages/sqlite.scm (sqlite)[native-inputs]: Add it when building> for the Hurd.> [arguments]: Apply it when building for the Hurd.
LGTM, thank you!
L
L
Ludovic Courtès wrote on 11 Jul 16:22 +0200
Re: bug#42151: [PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87r1tiw2qt.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
Toggle quote (12 lines)> This fixes https://bugs.gnu.org/42151.>> * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f> in call to ...> * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?> parameter, pass it to ...> (register-closure): ... this, add #:wal-mode? parameter, pass it to ...> * guix/store/database.scm (with-database): ... this, add #:wal-mode?> parameter, pass it to ...> (call-with-database): ... this, add #:wal-mode? parameter; when> set to #f, do not set journal_model=WAL.
[...]
Toggle quote (10 lines)> +++ b/gnu/system/images/hurd.scm> @@ -61,8 +61,8 @@> #~(lambda* (#:rest args)> (apply initialize-root-partition> (append args> - (list #:make-device-nodes> - make-hurd-device-nodes)))))> + (list #:make-device-nodes make-hurd-device-nodes> + #:wal-mode? #f)))))
IWBN to have an XXX comment here stating why we disable WAL mode onGNU/Hurd, possibly referencing this issue. That way, we can eventuallyrevisit the issue and hopefully get rid of this workaround.
Apart from that the patch LGTM, thanks!
Ludo’.
J
J
Jan Nieuwenhuizen wrote on 11 Jul 16:36 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151-done@debbugs.gnu.org)
87365yf79x.fsf@gnu.org
Ludovic Courtès writes:
Toggle quote (6 lines)> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:>>> This fixes <https://bugs.gnu.org/42151>.>>>> * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f>> in call to ...
[...]
Toggle quote (7 lines)>> + (list #:make-device-nodes make-hurd-device-nodes>> + #:wal-mode? #f)))))>> IWBN to have an XXX comment here stating why we disable WAL mode on> GNU/Hurd, possibly referencing this issue. That way, we can eventually> revisit the issue and hopefully get rid of this workaround.
Sure, done!
Toggle quote (2 lines)> Apart from that the patch LGTM, thanks!
Yay! Pushed series to master as 4b9eecd322e566783369795ebea63a479b51f486.
Let the offloadings commence!
Janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
Closed
?