[PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol

  • Done
  • quality assurance status badge
Details
2 participants
  • Jan (janneke) Nieuwenhuizen
  • Ludovic Courtès
Owner
unassigned
Submitted by
Jan (janneke) Nieuwenhuizen
Severity
normal
J
J
Jan (janneke) Nieuwenhuizen wrote on 1 Jul 2020 16:29
(address . bug-guix@gnu.org)
20200701142929.26919-1-janneke@gnu.org
Hello Guix!

I configured my system with a childhurd service and added

Toggle snippet (8 lines)
(build-machine
(name "childhurd")
(system "i586-gnu")
(host-key "ssh-ed25519 ... root@childhurd")
(user "root")
(private-key "/home/janneke/.ssh/id_rsa_childhurd"))

to /etc/guix/machines. Sadly, running guix offload test gives

Toggle snippet (7 lines)
$ guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: 'childhurd' is running GNU Guile 3.0.4
guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)

So, I looked into the Debian/Hurd sqlite3 package for inspiration and found
debian/patches/20-hurd-locking-style.patch...well, that should fix it, right?

Sadly, this patch seems to break sqlite3 altogether.

Toggle snippet (34 lines)
$ guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
Backtrace:
In ice-9/boot-9.scm:
1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In unknown file:
10 (apply-smob/0 #<thunk 7f3c99b1f480>)
In ice-9/boot-9.scm:
718:2 9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
In ice-9/eval.scm:
619:8 8 (_ #(#(#<directory (guile-user) 7f3c99743f00>)))
In guix/ui.scm:
1949:12 7 (run-guix-command _ . _)
663:2 6 (call-with-error-handling _)
In ice-9/boot-9.scm:
1736:10 5 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
1731:15 4 (with-exception-handler #<procedure 7f3c9546d3f0 at ice-9/boot-9.scm:1815:7 (exn)> _ #:unwind? _ # …)
In guix/scripts/offload.scm:
647:6 3 (check-machine-availability _ _)
In srfi/srfi-1.scm:
650:11 2 (for-each #<procedure assert-node-has-guix (node name)> (#<<inferior> pid: pipe socket: #<input…>) …)
In guix/scripts/offload.scm:
582:2 1 (assert-node-has-guix #<<inferior> pid: pipe socket: #<input-output: channel (open) 7f3c954a2500…> …)
In ice-9/boot-9.scm:
1669:16 0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
ERROR:
1. &inferior-exception:
arguments: (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: unable to open database file" status: 1>>)
inferior: #<<inferior> pid: pipe socket: #<input-output: channel (open) 7f3c954a2500> close: #<procedure close-port (_)> version: (0 1 1) packages: #<promise #<procedure 7f3c954a8000 at guix/inferior.scm:161:32 ()>> table: #<promise #<procedure 7f3c954a56f0 at guix/inferior.scm:162:32 ()>>>
stack: ((#f ("ice-9/boot-9.scm" 1763 13)) (raise-exception ("ice-9/boot-9.scm" 1666 16)) (raise-exception ("ice-9/boot-9.scm" 1668 16)) (#f ("guix/store.scm" 582 42)) (with-exception-handler ("ice-9/boot-9.scm" 1735 10)) (open-connection ("guix/store.scm" 545 2)) (call-with-store ("guix/store.scm" 626 15)) (#f (#f #f #f)) (#f ("guix/repl.scm" 92 21)) (with-exception-handler ("ice-9/boot-9.scm" 1735 10)) (with-exception-handler ("ice-9/boot-9.scm" 1730 15)) (#f ("guix/repl.scm" 119 7)))

Maybe we're missing some file_lock patch that Debian has? Where to look,
glibc, hurd, ...? Ideas?

Greetings,
Janneke

Jan (janneke) Nieuwenhuizen (3):
gnu: Add sqlite/hurd with locking fix.
gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
gnu: guix: Use sqlite/hurd for locking on the Hurd.

gnu/local.mk | 1 +
gnu/packages/guile.scm | 2 +-
gnu/packages/package-management.scm | 2 +-
gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++
gnu/packages/sqlite.scm | 9 +++++
5 files changed, 63 insertions(+), 2 deletions(-)
create mode 100644 gnu/packages/patches/sqlite3-hurd.patch

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 1 Jul 2020 16:31
[PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
(address . 42151@debbugs.gnu.org)
20200701143118.26985-2-janneke@gnu.org
This fixes

guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)

* gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of
sqlite.
---
gnu/packages/guile.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index a32bd33293..c651e2427a 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -665,7 +665,7 @@ Guile's foreign function interface.")
("pkg-config" ,pkg-config)))
(inputs
`(("guile" ,guile-3.0)
- ("sqlite" ,sqlite)))
+ ("sqlite" ,sqlite/hurd)))
(synopsis "Access SQLite databases from Guile")
(description
"This package provides Guile bindings to the SQLite database system.")
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 1 Jul 2020 16:31
[PATCH 1/3] gnu: Add sqlite/hurd with locking fix.
(address . 42151@debbugs.gnu.org)
20200701143118.26985-1-janneke@gnu.org
* gnu/packages/patches/sqlite3-hurd.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/sqlite.scm (sqlite/hurd): New variable.
---
gnu/local.mk | 1 +
gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++
gnu/packages/sqlite.scm | 9 +++++
3 files changed, 61 insertions(+)
create mode 100644 gnu/packages/patches/sqlite3-hurd.patch

Toggle diff (96 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index 5e9dba5ab7..d5451bf2d5 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1358,6 +1358,7 @@ dist_patch_DATA = \
%D%/packages/patches/sdl-pango-header-guard.patch \
%D%/packages/patches/sdl-pango-matrix_declarations.patch \
%D%/packages/patches/sdl-pango-sans-serif.patch \
+ %D%/packages/patches/sqlite3-hurd.patch \
%D%/packages/patches/patchutils-test-perms.patch \
%D%/packages/patches/patch-hurd-path-max.patch \
%D%/packages/patches/perl-autosplit-default-time.patch \
diff --git a/gnu/packages/patches/sqlite3-hurd.patch b/gnu/packages/patches/sqlite3-hurd.patch
new file mode 100644
index 0000000000..de87a30cb1
--- /dev/null
+++ b/gnu/packages/patches/sqlite3-hurd.patch
@@ -0,0 +1,51 @@
+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch
+Upstream status: Not offered upstream.
+
+diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c
+--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100
++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200
+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode
+ # include <sys/mman.h>
+ #endif
+
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ /* # include <sys/ioctl.h> */
+ # include <sys/file.h>
+ # include <sys/param.h>
+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id
+ **
+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off
+ */
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+
+ /*
+ ** Retry flock() calls that fail with EINTR
+@@ -38586,7 +38586,7 @@ IOMETHODS(
+ 0 /* xShmMap method */
+ )
+
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ IOMETHODS(
+ flockIoFinder, /* Finder function name */
+ flockIoMethods, /* sqlite3_io_methods object name */
+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){
+ UNIXVFS("unix", autolockIoFinder ),
+ #elif OS_VXWORKS
+ UNIXVFS("unix", vxworksIoFinder ),
++#elif defined(__GNU__)
++ UNIXVFS("unix", flockIoFinder ),
+ #else
+ UNIXVFS("unix", posixIoFinder ),
+ #endif
+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){
+ #if OS_VXWORKS
+ UNIXVFS("unix-namedsem", semIoFinder ),
+ #endif
+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS
++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)
+ UNIXVFS("unix-posix", posixIoFinder ),
+ #endif
+ #if SQLITE_ENABLE_LOCKING_STYLE
diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm
index 8468131ddf..d5c44b0d0f 100644
--- a/gnu/packages/sqlite.scm
+++ b/gnu/packages/sqlite.scm
@@ -11,6 +11,7 @@
;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org>
;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -107,3 +108,11 @@ is in the public domain.")
;; commit fad5b1a6d8d9c36bea5785ae4fbc1beb37e644d7.
(define-public sqlite-with-column-metadata
(deprecated-package "sqlite-with-column-metadata" sqlite))
+
+(define-public sqlite/hurd
+ ;; TODO move into sqlite on the next rebuild cycle.
+ (package
+ (inherit sqlite)
+ (name "sqlite-for-hurd")
+ (source (origin (inherit (package-source sqlite))
+ (patches (search-patches "sqlite3-hurd.patch"))))))
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 1 Jul 2020 16:31
[PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.
(address . 42151@debbugs.gnu.org)
20200701143118.26985-3-janneke@gnu.org
* gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.
---
gnu/packages/package-management.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 068d46566c..52c5a1e60d 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -343,7 +343,7 @@ $(prefix)/etc/init.d\n")))
("zlib" ,zlib) ;for 'guix publish'
("lzlib" ,lzlib) ;for 'guix publish' and 'guix substitute'
- ("sqlite" ,sqlite)
+ ("sqlite" ,sqlite/hurd)
("libgcrypt" ,libgcrypt)
("guile" ,guile-3.0-latest)
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 1 Jul 2020 19:01
Re: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol
(address . bug-guix@gnu.org)
87o8ozcgq3.fsf@gnu.org
Jan (janneke) Nieuwenhuizen writes:

Toggle quote (3 lines)
> Maybe we're missing some file_lock patch that Debian has? Where to look,
> glibc, hurd, ...? Ideas?

So...I found a way to reproduce the feature/bug: run "pragma synchronous = normal;"
in two instances.

I created a db.sqlite using

sh -x tests/guix-pack-localstatedir.sh

(after removing the cleanup trap).

Then, doing

Toggle snippet (20 lines)
$ sqlite3 db.sqlite
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> SELECT * FROM ValidPaths;
1|/gnu/store/pqciscj9304544h3yaqb5m70fggsr293-manual-database|sha256:ccd93bed3a0018f892abc5db92bf182c3e2e986f41a8ded168915b4d9e85fa18|1||2664
2|/gnu/store/mzfkrxd4w8vqrmyrx169wj8wyw7r8i37-bash|sha256:6f887d45fa0f7e59e55c6d7ba86a3d8c35369c7afbb3a5829b8ed226bfef4a66|1||1351880
3|/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0|sha256:dc3e6b577e995c093849454fe4b9c30e87ece6462c54369ff249121f178c5476|1||12718808
4|/gnu/store/6z4h8g6s73zmcn0h6sm63p9d47ih4b63-info-dir|sha256:fd154332e6d35c95f12e41d6168a93945b4737fcbaf734bcaf04ca771a106544|1||432
5|/gnu/store/mj7szx64bdh080rlhzhrvcg6viyba73s-profile|sha256:e32ae4d07fc2298a72e6942f21ba85c0300a84bbaf5794a8c5f0393a8dc9f253|1||2960
sqlite> pragma synchronous = normal;
sqlite> C-c C-z^Z
[2]+ Stopped sqlite3 db.sqlite
[148]18:55:06 janneke@dundal:~/src/guix/master [env]
$ sqlite3 db.sqlite
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> pragma synchronous = normal;
sqlite>

works nice on GNU/Linux and on current master Childhurd gives

Toggle snippet (26 lines)
$ ssh childhurd
Last login: Wed Jul 1 12:08:17 2020 from 10.0.2.2
??????
??????
?????? This is the GNU Hurd. Welcome.

root@childhurd ~# sqlite3 db.sqlite
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> SELECT * FROM ValidPaths;
1|/gnu/store/pqciscj9304544h3yaqb5m70fggsr293-manual-database|sha256:ccd93bed3a0018f892abc5db92bf182c3e2e986f41a8ded168915b4d9e85fa18|1||2664
2|/gnu/store/mzfkrxd4w8vqrmyrx169wj8wyw7r8i37-bash|sha256:6f887d45fa0f7e59e55c6d7ba86a3d8c35369c7afbb3a5829b8ed226bfef4a66|1||1351880
3|/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0|sha256:dc3e6b577e995c093849454fe4b9c30e87ece6462c54369ff249121f178c5476|1||12718808
4|/gnu/store/6z4h8g6s73zmcn0h6sm63p9d47ih4b63-info-dir|sha256:fd154332e6d35c95f12e41d6168a93945b4737fcbaf734bcaf04ca771a106544|1||432
5|/gnu/store/mj7szx64bdh080rlhzhrvcg6viyba73s-profile|sha256:e32ae4d07fc2298a72e6942f21ba85c0300a84bbaf5794a8c5f0393a8dc9f253|1||2960
sqlite> pragma synchronous = normal;
sqlite> C-c C-z^Z
[1]+ Stopped sqlite3 db.sqlite
root@childhurd ~# sqlite3 db.sqlite
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> pragma synchronous = normal;
Error: locking protocol
sqlite>

On Debian/Hurd (which I hoped had a patched, fixed sqlite3), it doesn't
work at all

Toggle snippet (25 lines)
$ ssh hurd
GNU debian 0.9 GNU-Mach 1.8+git20191117-486/Hurd-0.9 i686-AT386
??????
??????
?????? This is the GNU Hurd. Welcome.

The Hurd is not Linux. Make sure to read
http://www.debian.org/ports/hurd/hurd-install
to check out the few things you _need_ to know.
Also check out the FAQ
http://www.gnu.org/software/hurd/faq.html
or its latest version on
http://darnassus.sceen.net/~hurd-web/faq/

To read a short intro on some nice features of the Hurd, just have a look at
the translator_primer file, for example via 'nano translator_primer'
Last login: Wed Jul 1 12:58:44 2020 from 10.0.2.2
janneke@debian:~$ sqlite3 db.sqlite
SQLite version 3.32.3 2020-06-18 14:00:33
Enter ".help" for usage hints.
sqlite> SELECT * FROM ValidPaths;
Error: unable to open database file
sqlite>

So, still a similar question...is the patch simply broken/bitrotted, or
did some file locking change on the Hurd?

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 2 Jul 2020 14:49
Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87imf6rsir.fsf@gnu.org
Hi!

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (6 lines)
> $ guix offload test
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> guix offload: 'childhurd' is running GNU Guile 3.0.4
> guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)

Does sqlite pass its tests on GNU/Hurd?

Does it help if you set ‘settings.fsyncMetadata = false’ in the daemon?
As a stop-gap, we could add a command-line option if that helps.

The “synchronous = normal” mode translates to ‘fsync’ calls, right? If
you rpctrace sqlite3, do you see ‘file_sync’ calls failing?

My 2¢!

Ludo’.
J
J
Jan Nieuwenhuizen wrote on 3 Jul 2020 12:03
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
873668zzip.fsf@gnu.org
Ludovic Courtès writes:

Hi!

Toggle quote (13 lines)
> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> $ guix offload test
>> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
>> guix offload: Guix is usable on 'childhurd' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
>> guix offload: 'childhurd' is running GNU Guile 3.0.4
>> guix offload: error: exception occurred on remote host 'localhost':
>> (%exception #<inferior-object #<&store-protocol-error message:
>> "setting synchronous mode: locking protocol" status: 1>>)

> Does it help if you set ‘settings.fsyncMetadata = false’ in the daemon?
> As a stop-gap, we could add a command-line option if that helps.

Tried that, thanks. No, it does not help. (But that's good news, see below!)

Toggle quote (3 lines)
> The “synchronous = normal” mode translates to ‘fsync’ calls, right? If
> you rpctrace sqlite3, do you see ‘file_sync’ calls failing?

Tried that before, rpctrace hangs before I see something useful.

Toggle quote (2 lines)
> Does sqlite pass its tests on GNU/Hurd?

That's the (or at least a) right question: YES!

Toggle quote (2 lines)
> My 2¢!

Thanks -- it seems that buys us a pretty cheap fix after all, $-wise ;)

It turns out that Debian's patch (and thus this patch series) is
probably OK: It fixes the locking problem on the Hurd, while exposing
another bug, apparently: "unable to open database file".

It seems there is a compatibility bug/problem/thing with the db.sqlite
that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd
can read it, and work with it, the unpatched sqlite has locking
problems. I found a workaround, though: dumping and loading the
database file.

Look...

Toggle snippet (26 lines)
$ scp childhurd2:/var/guix/db/db.sqlite db.sqlite-orig
db.sqlite 100% 144KB 5.8MB/s 00:00
11:30:37 janneke@dundal:~/tmp [env]
$ sqlite3 db.sqlite-orig .dump > db.dump
11:30:45 janneke@dundal:~/tmp [env]
$ sqlite3 -init db.dump db.sqlite-init .quit
-- Loading resources from db.dump
11:30:49 janneke@dundal:~/tmp [env]
$ cmp db.sqlite-orig db.sqlite-init
db.sqlite-orig db.sqlite-init differ: byte 19, line 1
[1]11:31:11 janneke@dundal:~/tmp [env]
$ scp db.sqlite-init childhurd2:/var/guix/db/db.sqlite
db.sqlite-init 100% 144KB 7.3MB/s 00:00
11:31:21 janneke@dundal:~/tmp [env]
$ guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
guix offload: Guix is usable on 'localhost' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: 'localhost' is running GNU Guile 3.0.4
sending 1 store item (0 MiB) to 'localhost'...
exporting path `/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'
guix offload: 'localhost' successfully imported '/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'
retrieving 1 store item from 'localhost'...
guix offload: successfully imported '/gnu/store/gxz6hzyc1cy3m1w9l7f2dk6rcspvymxf-import-test' from 'localhost'
11:31:29 janneke@dundal:~/tmp [env]

So...about the compatibility problem. I tried to diff the db.sqlite-orig
db.sqlite-init binary files: they look completely different. Not sure
how to handle this workaround, maybe we can insert a two system* calls
somewhere when building the disk image?

Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 3 Jul 2020 15:51
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87fta8ogf2.fsf@gnu.org
Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (2 lines)
> Thanks -- it seems that buys us a pretty cheap fix after all, $-wise ;)

Heheh. :-)

Toggle quote (42 lines)
> It turns out that Debian's patch (and thus this patch series) is
> probably OK: It fixes the locking problem on the Hurd, while exposing
> another bug, apparently: "unable to open database file".
>
> It seems there is a compatibility bug/problem/thing with the db.sqlite
> that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd
> can read it, and work with it, the unpatched sqlite has locking
> problems. I found a workaround, though: dumping and loading the
> database file.
>
> Look...
>
> $ scp childhurd2:/var/guix/db/db.sqlite db.sqlite-orig
> db.sqlite 100% 144KB 5.8MB/s 00:00
> 11:30:37 janneke@dundal:~/tmp [env]
> $ sqlite3 db.sqlite-orig .dump > db.dump
> 11:30:45 janneke@dundal:~/tmp [env]
> $ sqlite3 -init db.dump db.sqlite-init .quit
> -- Loading resources from db.dump
> 11:30:49 janneke@dundal:~/tmp [env]
> $ cmp db.sqlite-orig db.sqlite-init
> db.sqlite-orig db.sqlite-init differ: byte 19, line 1
> [1]11:31:11 janneke@dundal:~/tmp [env]
> $ scp db.sqlite-init childhurd2:/var/guix/db/db.sqlite
> db.sqlite-init 100% 144KB 7.3MB/s 00:00
> 11:31:21 janneke@dundal:~/tmp [env]
> $ guix offload test
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> guix offload: Guix is usable on 'localhost' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> guix offload: 'localhost' is running GNU Guile 3.0.4
> sending 1 store item (0 MiB) to 'localhost'...
> exporting path `/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'
> guix offload: 'localhost' successfully imported '/gnu/store/y6b7bjqsazmm6jsyj5y80dqqajysw64p-export-test'
> retrieving 1 store item from 'localhost'...
> guix offload: successfully imported '/gnu/store/gxz6hzyc1cy3m1w9l7f2dk6rcspvymxf-import-test' from 'localhost'
> 11:31:29 janneke@dundal:~/tmp [env]
>
> So...about the compatibility problem. I tried to diff the db.sqlite-orig
> db.sqlite-init binary files: they look completely different. Not sure
> how to handle this workaround, maybe we can insert a two system* calls
> somewhere when building the disk image?

Weird, weird!

Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created
from the same ‘sqlite3 -init’ command on GNU/Linux? (Perhaps loading
the dump reorders entries or something.)

I think the binary format of the database is supposed to be
architecture-independent and filling it is supposed to be deterministic.

Congrats for getting this far anyway!

Ludo’.
J
J
Jan Nieuwenhuizen wrote on 3 Jul 2020 17:27
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87sge8wrdf.fsf@gnu.org
Ludovic Courtès writes:

Hi!

Toggle quote (6 lines)
>> It seems there is a compatibility bug/problem/thing with the db.sqlite
>> that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd
>> can read it, and work with it, the unpatched sqlite has locking
>> problems. I found a workaround, though: dumping and loading the
>> database file.

[..]

Toggle quote (10 lines)
>> So...about the compatibility problem. I tried to diff the db.sqlite-orig
>> db.sqlite-init binary files: they look completely different. Not sure
>> how to handle this workaround, maybe we can insert a two system* calls
>> somewhere when building the disk image?
>
> Weird, weird!
>
> Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created
> from the same ‘sqlite3 -init’ command on GNU/Linux?

Yeah, they are identical. The initial dump can only be created atm on
GNU/Linux; the dump can be loaded (obviously) anywhere like so

Toggle quote (2 lines)
>> $ sqlite3 -init db.dump db.sqlite-init .quit

and the resulting initial db.sqlite is the same. Guess we can do that
by hand for now...

Toggle quote (2 lines)
> (Perhaps loading the dump reorders entries or something.)

Yes, "or something" certainly! :) I have no clue...

If/when we decide to pinpoint this bug, what could be a first step? Who
is creating the database right now, is that the C++ daemon or
guile-sqlite3. IWBN to have that code create/save a smaller version and
see when reading fails.

Toggle quote (3 lines)
> I think the binary format of the database is supposed to be
> architecture-independent and filling it is supposed to be deterministic.

Yes, that works.

Toggle quote (2 lines)
> Congrats for getting this far anyway!

Yeah... \o/

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 4 Jul 2020 10:12
[PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87y2nzr94w.fsf_-_@gnu.org
Jan Nieuwenhuizen writes:

Adding one additional patch to this series, hence 4/3 (see attached)
fixes the sqlite problem.

With this over-complete series we can now do offload builds to a
childhurd:

Toggle snippet (17 lines)
$ guix build -e '(@@ (gnu packages commencement) rsync-boot0)'
sending 74 store items (365 MiB) to 'localhost'...
exporting path `/gnu/store/1kf05mhh31wl89r3mlx52x5sxrjivi7i-module-import-compiled-guile-builder'
exporting path `/gnu/store/7jxamp3g8wdr6vwrisqfmrncxh8nbfb4-mirrors'
[...]
exporting path `/gnu/store/x8wnf0h0ipibgzadlbmll8bcnhvv1yaq-diffutils-boot0-3.7'
offloading '/gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv' to 'localhost'...
offloading build of /gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv to 'localhost'
[...]
@ build-succeeded /gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv -
retrieving 1 store item from 'localhost'...
importing file or directory '/gnu/store/haf6mlm8xa6s2q918s05pijl6ql17mnq-rsync-3.1.3'...
guix offload: error: corrupt input while restoring archive from #<input: string 7ff4794cee00>
guix build: error: build of `/gnu/store/3xmmbjfqrl4p4sn8vljfdikypb0vi5am-rsync-3.1.3.drv' failed
[1]10:04:55 janneke@dundal:~/src/guix/master [env]

After the build succeeds, the download fails but lets first get this
patch series done...

Janneke
From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001
From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>
Date: Fri, 3 Jul 2020 23:45:20 +0200
Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8


* guix/store/database.scm (call-with-database): When building for the Hurd,
do not set journal_model=WAL.
---
guix/store/database.scm | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)

Toggle diff (47 lines)
diff --git a/guix/store/database.scm b/guix/store/database.scm
index a38e4d7e52..da46b0abce 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>
;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -20,6 +21,7 @@
(define-module (guix store database)
#:use-module (sqlite3)
#:use-module (guix config)
+ #:use-module (guix gexp)
#:use-module (guix serialization)
#:use-module (guix store deduplication)
#:use-module (guix base16)
@@ -27,6 +29,7 @@
#:use-module (guix build syscalls)
#:use-module ((guix build utils)
#:select (mkdir-p executable-file?))
+ #:use-module (guix utils)
#:use-module (guix build store-copy)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
@@ -105,9 +108,12 @@ create it and initialize it as a new database."
(mkdir-p (dirname file))
#t)))
(db (sqlite-open file)))
- ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
- ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
- (sqlite-exec db "PRAGMA journal_mode=WAL;")
+ ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
+ (unless (let-system (system target)
+ (equal? target "i586-pc-gnu"))
+ ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
+ ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
+ (sqlite-exec db "PRAGMA journal_mode=WAL;"))
;; Install a busy handler such that, when the database is locked, sqlite
;; retries until 30 seconds have passed, at which point it gives up and
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 7 Jul 2020 07:04
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87d057apb4.fsf@gnu.org
Jan Nieuwenhuizen writes:

Toggle quote (5 lines)
> Jan Nieuwenhuizen writes:
>
> Adding one additional patch to this series, hence 4/3 (see attached)
> fixes the sqlite problem.

For clarity -- patches 1-4 really do fix offloading to the Hurd and this...

Toggle quote (3 lines)
> After the build succeeds, the download fails but lets first get this
> patch series done...

...is not true; you gat that when you (I) forgot to add keys in
/etc/guix/

As seen on IRC, possibly Ricardo will an update for the QEMU
command-line (at least for berlin), adding "--cpu base".

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 10 Jul 2020 14:03
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87v9iv1ssj.fsf@gnu.org
Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (57 lines)
>>From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001
> From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>
> Date: Fri, 3 Jul 2020 23:45:20 +0200
> Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
> Content-Transfer-Encoding: 8bit
> Content-Type: text/plain; charset=UTF-8
>
> This fixes <https://bugs.gnu.org/42151>.
>
> * guix/store/database.scm (call-with-database): When building for the Hurd,
> do not set journal_model=WAL.
> ---
> guix/store/database.scm | 12 +++++++++---
> 1 file changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/guix/store/database.scm b/guix/store/database.scm
> index a38e4d7e52..da46b0abce 100644
> --- a/guix/store/database.scm
> +++ b/guix/store/database.scm
> @@ -1,6 +1,7 @@
> ;;; GNU Guix --- Functional package management for GNU
> ;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>
> ;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>
> +;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
> ;;;
> ;;; This file is part of GNU Guix.
> ;;;
> @@ -20,6 +21,7 @@
> (define-module (guix store database)
> #:use-module (sqlite3)
> #:use-module (guix config)
> + #:use-module (guix gexp)
> #:use-module (guix serialization)
> #:use-module (guix store deduplication)
> #:use-module (guix base16)
> @@ -27,6 +29,7 @@
> #:use-module (guix build syscalls)
> #:use-module ((guix build utils)
> #:select (mkdir-p executable-file?))
> + #:use-module (guix utils)
> #:use-module (guix build store-copy)
> #:use-module (srfi srfi-1)
> #:use-module (srfi srfi-11)
> @@ -105,9 +108,12 @@ create it and initialize it as a new database."
> (mkdir-p (dirname file))
> #t)))
> (db (sqlite-open file)))
> - ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
> - ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
> - (sqlite-exec db "PRAGMA journal_mode=WAL;")
> + ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
> + (unless (let-system (system target)
> + (equal? target "i586-pc-gnu"))
> + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
> + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
> + (sqlite-exec db "PRAGMA journal_mode=WAL;"))

I think this has the same effect as:

(unless #t
(sqlite-exec db "PRAGMA journal_mode=WAL;"))

because this code is not in a gexp, so ‘let-system’ evaluates to an
object (not #f), and that’s it, it’s never lowered and the ‘equal?’ call
is never made.

You want to disable WAL mode not just when running this code natively on
GNU/Hurd, but also when building a database that will eventually be used
on GNU/Hurd, right?

In that case, I think you’ll have to add, say, a #:wal-mode? parameter
to ‘call-with-database’ (defaulting to #true), and change the caller to
set it appropriately. The caller, directly or indirectly, is in a gexp,
where you can use the ‘let-system’ expression above.

Does that make sense?

Thanks, and apologies for the delay!

Ludo’.
L
L
Ludovic Courtès wrote on 10 Jul 2020 14:05
Re: bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87r1tj1so4.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (4 lines)
> * gnu/packages/patches/sqlite3-hurd.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add it.
> * gnu/packages/sqlite.scm (sqlite/hurd): New variable.

[…]

Toggle quote (5 lines)
> +++ b/gnu/packages/patches/sqlite3-hurd.patch
> @@ -0,0 +1,51 @@
> +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch
> +Upstream status: Not offered upstream.

The URL is 404, could you fix it? It would also be great if you could
add a sentence or two explaining what this works around, so that our
future selves know how to deal with this patch. :-)

Otherwise LGTM, you can push with changes along these lines!

Ludo’.
L
L
Ludovic Courtès wrote on 10 Jul 2020 14:06
Re: bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87mu471smz.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (21 lines)
> This fixes
>
> guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
>
> * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of
> sqlite.
> ---
> gnu/packages/guile.scm | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
> index a32bd33293..c651e2427a 100644
> --- a/gnu/packages/guile.scm
> +++ b/gnu/packages/guile.scm
> @@ -665,7 +665,7 @@ Guile's foreign function interface.")
> ("pkg-config" ,pkg-config)))
> (inputs
> `(("guile" ,guile-3.0)
> - ("sqlite" ,sqlite)))
> + ("sqlite" ,sqlite/hurd)))

This should be guarded by ‘if (hurd-target?)’, right?

Ludo’.
L
L
Ludovic Courtès wrote on 10 Jul 2020 14:07
Re: bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87imev1sl6.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (2 lines)
> * gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.

Same here.

Which makes me thing: how about applying the sqlite/hurd patch in a
build phase of sqlite that would be Hurd-specific? That way, we’d still
have a single sqlite package.

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 10 Jul 2020 14:34
Re: bug#42151: [PATCH 0/3] offload to Childhurd fails: setting synchronous mode: locking protocol
(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87sgdzzgyz.fsf@gnu.org
Hi,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (34 lines)
> Ludovic Courtès writes:
>
> Hi!
>
>>> It seems there is a compatibility bug/problem/thing with the db.sqlite
>>> that we produce on GNU/Linux. While an unpatched sqlite3 on the Hurd
>>> can read it, and work with it, the unpatched sqlite has locking
>>> problems. I found a workaround, though: dumping and loading the
>>> database file.
>
> [..]
>
>>> So...about the compatibility problem. I tried to diff the db.sqlite-orig
>>> db.sqlite-init binary files: they look completely different. Not sure
>>> how to handle this workaround, maybe we can insert a two system* calls
>>> somewhere when building the disk image?
>>
>> Weird, weird!
>>
>> Could you compare ‘db.dump’ created on GNU/Hurd with ‘db.dump’ created
>> from the same ‘sqlite3 -init’ command on GNU/Linux?
>
> Yeah, they are identical. The initial dump can only be created atm on
> GNU/Linux; the dump can be loaded (obviously) anywhere like so
>
>>> $ sqlite3 -init db.dump db.sqlite-init .quit
>
> and the resulting initial db.sqlite is the same. Guess we can do that
> by hand for now...
>
>> (Perhaps loading the dump reorders entries or something.)
>
> Yes, "or something" certainly! :) I have no clue...

Sorry for catching up days later but… did disabling WAL mode fix this
discrepancy? IOW, without WAL mode, is the file produced on GNU/Linux
bit-identical to that produced on GNU/Hurd?

Toggle quote (5 lines)
> If/when we decide to pinpoint this bug, what could be a first step? Who
> is creating the database right now, is that the C++ daemon or
> guile-sqlite3. IWBN to have that code create/save a smaller version and
> see when reading fails.

As you found out it’s created by (guix store database) in this case.

Thanks,
Ludo’.
J
J
Jan Nieuwenhuizen wrote on 10 Jul 2020 17:40
Re: bug#42151: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87a707v0ns.fsf@gnu.org
Ludovic Courtès writes:

Hi!

Toggle quote (10 lines)
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>>>From dc6f96fc7de50602fb28d7ad7b8cbff09e55f538 Mon Sep 17 00:00:00 2001
>> From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>
>> Date: Fri, 3 Jul 2020 23:45:20 +0200
>> Subject: [PATCH 4/3] database: Do not use journal_model=WAL for the Hurd.
>> Content-Transfer-Encoding: 8bit
>> Content-Type: text/plain; charset=UTF-8
>>
>> This fixes <https://bugs.gnu.org/42151>.
[...]
Toggle quote (4 lines)
>> diff --git a/guix/store/database.scm b/guix/store/database.scm
>> index a38e4d7e52..da46b0abce 100644
>> --- a/guix/store/database.scm
>> +++ b/guix/store/database.scm
[...]
Toggle quote (16 lines)
>> + ;; Using WAL breaks for the Hurd https://bugs.gnu.org/42151.
>> + (unless (let-system (system target)
>> + (equal? target "i586-pc-gnu"))
>> + ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
>> + ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
>> + (sqlite-exec db "PRAGMA journal_mode=WAL;"))
>
> I think this has the same effect as:
>
> (unless #t
> (sqlite-exec db "PRAGMA journal_mode=WAL;"))
>
> because this code is not in a gexp, so ‘let-system’ evaluates to an
> object (not #f), and that’s it, it’s never lowered and the ‘equal?’ call
> is never made.

Oops. Great catch, thanks!

Toggle quote (4 lines)
> You want to disable WAL mode not just when running this code natively on
> GNU/Hurd, but also when building a database that will eventually be used
> on GNU/Hurd, right?

Exactly.

Toggle quote (7 lines)
> In that case, I think you’ll have to add, say, a #:wal-mode? parameter
> to ‘call-with-database’ (defaulting to #true), and change the caller to
> set it appropriately. The caller, directly or indirectly, is in a gexp,
> where you can use the ‘let-system’ expression above.
>
> Does that make sense?

Yes, very much so. [...] After some typing, it turns out that the whole
let-system is gone, we can set use a hard-coded #:wal-mode? #t in
hurd-initialize-root-partition.

Toggle quote (2 lines)
> Thanks, and apologies for the delay!

Sure, I'm happy I didn't "just push" ;-)

I'm sending a v2 patch set in a minute.

Greetings,
Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 2020 17:42
[PATCH v2 2/4] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710154205.31342-2-janneke@gnu.org
This fixes

guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)

* gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of
sqlite.
---
gnu/packages/guile.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index a32bd33293..c651e2427a 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -665,7 +665,7 @@ Guile's foreign function interface.")
("pkg-config" ,pkg-config)))
(inputs
`(("guile" ,guile-3.0)
- ("sqlite" ,sqlite)))
+ ("sqlite" ,sqlite/hurd)))
(synopsis "Access SQLite databases from Guile")
(description
"This package provides Guile bindings to the SQLite database system.")
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 2020 17:42
[PATCH v2 3/4] gnu: guix: Use sqlite/hurd for locking on the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710154205.31342-3-janneke@gnu.org
* gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.
---
gnu/packages/package-management.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 9986976cc6..64c6931011 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -343,7 +343,7 @@ $(prefix)/etc/init.d\n")))
("zlib" ,zlib) ;for 'guix publish'
("lzlib" ,lzlib) ;for 'guix publish' and 'guix substitute'
- ("sqlite" ,sqlite)
+ ("sqlite" ,sqlite/hurd)
("libgcrypt" ,libgcrypt)
("guile" ,guile-3.0-latest)
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 2020 17:42
[PATCH v2 1/4] gnu: Add sqlite/hurd with locking fix.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710154205.31342-1-janneke@gnu.org
* gnu/packages/patches/sqlite3-hurd.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/sqlite.scm (sqlite/hurd): Use it in new variable.
---
gnu/local.mk | 1 +
gnu/packages/patches/sqlite3-hurd.patch | 51 +++++++++++++++++++++++++
gnu/packages/sqlite.scm | 9 +++++
3 files changed, 61 insertions(+)
create mode 100644 gnu/packages/patches/sqlite3-hurd.patch

Toggle diff (96 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index 5c3b391960..e452004945 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1370,6 +1370,7 @@ dist_patch_DATA = \
%D%/packages/patches/sdl-pango-header-guard.patch \
%D%/packages/patches/sdl-pango-matrix_declarations.patch \
%D%/packages/patches/sdl-pango-sans-serif.patch \
+ %D%/packages/patches/sqlite3-hurd.patch \
%D%/packages/patches/patchutils-test-perms.patch \
%D%/packages/patches/patch-hurd-path-max.patch \
%D%/packages/patches/perl-autosplit-default-time.patch \
diff --git a/gnu/packages/patches/sqlite3-hurd.patch b/gnu/packages/patches/sqlite3-hurd.patch
new file mode 100644
index 0000000000..de87a30cb1
--- /dev/null
+++ b/gnu/packages/patches/sqlite3-hurd.patch
@@ -0,0 +1,51 @@
+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch
+Upstream status: Not offered upstream.
+
+diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c
+--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100
++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200
+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode
+ # include <sys/mman.h>
+ #endif
+
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ /* # include <sys/ioctl.h> */
+ # include <sys/file.h>
+ # include <sys/param.h>
+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id
+ **
+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off
+ */
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+
+ /*
+ ** Retry flock() calls that fail with EINTR
+@@ -38586,7 +38586,7 @@ IOMETHODS(
+ 0 /* xShmMap method */
+ )
+
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ IOMETHODS(
+ flockIoFinder, /* Finder function name */
+ flockIoMethods, /* sqlite3_io_methods object name */
+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){
+ UNIXVFS("unix", autolockIoFinder ),
+ #elif OS_VXWORKS
+ UNIXVFS("unix", vxworksIoFinder ),
++#elif defined(__GNU__)
++ UNIXVFS("unix", flockIoFinder ),
+ #else
+ UNIXVFS("unix", posixIoFinder ),
+ #endif
+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){
+ #if OS_VXWORKS
+ UNIXVFS("unix-namedsem", semIoFinder ),
+ #endif
+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS
++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)
+ UNIXVFS("unix-posix", posixIoFinder ),
+ #endif
+ #if SQLITE_ENABLE_LOCKING_STYLE
diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm
index 8468131ddf..d5c44b0d0f 100644
--- a/gnu/packages/sqlite.scm
+++ b/gnu/packages/sqlite.scm
@@ -11,6 +11,7 @@
;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org>
;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -107,3 +108,11 @@ is in the public domain.")
;; commit fad5b1a6d8d9c36bea5785ae4fbc1beb37e644d7.
(define-public sqlite-with-column-metadata
(deprecated-package "sqlite-with-column-metadata" sqlite))
+
+(define-public sqlite/hurd
+ ;; TODO move into sqlite on the next rebuild cycle.
+ (package
+ (inherit sqlite)
+ (name "sqlite-for-hurd")
+ (source (origin (inherit (package-source sqlite))
+ (patches (search-patches "sqlite3-hurd.patch"))))))
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 2020 17:42
[PATCH v2 4/4] image: Do not set journal_model=WAL for the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710154205.31342-4-janneke@gnu.org

* gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
in call to ...
* gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?
parameter, pass it to ...
(register-closure): ... this, add #:wal-mode? parameter, pass it to ...
* guix/store/database.scm (with-database): ... this, add #:wal-mode?
parameter, pass it to ...
(call-with-database): ... this, add #:wal-mode? parameter; when
set to #f, do not set journal_model=WAL.
---
gnu/build/image.scm | 26 ++++++++++++++++----------
gnu/system/images/hurd.scm | 4 ++--
guix/store/database.scm | 29 ++++++++++++++++++++---------
3 files changed, 38 insertions(+), 21 deletions(-)

Toggle diff (156 lines)
diff --git a/gnu/build/image.scm b/gnu/build/image.scm
index e7b0418182..d8efa73f16 100644
--- a/gnu/build/image.scm
+++ b/gnu/build/image.scm
@@ -131,20 +131,23 @@ given CONFIG file."
(define* (register-closure prefix closure
#:key
(deduplicate? #t) (reset-timestamps? #t)
- (schema (sql-schema)))
+ (schema (sql-schema))
+ (wal-mode? #t))
"Register CLOSURE in PREFIX, where PREFIX is the directory name of the
target store and CLOSURE is the name of a file containing a reference graph as
produced by #:references-graphs.. As a side effect, if RESET-TIMESTAMPS? is
true, reset timestamps on store files and, if DEDUPLICATE? is true,
-deduplicates files common to CLOSURE and the rest of PREFIX."
+deduplicates files common to CLOSURE and the rest of PREFIX. Pass WAL-MODE?
+to call-with-database."
(let ((items (call-with-input-file closure read-reference-graph)))
(parameterize ((sql-schema schema))
(with-database (store-database-file #:prefix prefix) db
- (register-items db items
- #:prefix prefix
- #:deduplicate? deduplicate?
- #:reset-timestamps? reset-timestamps?
- #:registration-time %epoch)))))
+ #:wal-mode? wal-mode?
+ (register-items db items
+ #:prefix prefix
+ #:deduplicate? deduplicate?
+ #:reset-timestamps? reset-timestamps?
+ #:registration-time %epoch)))))
(define* (initialize-efi-partition root
#:key
@@ -164,14 +167,16 @@ deduplicates files common to CLOSURE and the rest of PREFIX."
(register-closures? #t)
system-directory
make-device-nodes
+ (wal-mode? #t)
#:allow-other-keys)
"Initialize the given ROOT directory. Use BOOTCFG and BOOTCFG-LOCATION to
install the bootloader configuration.
If REGISTER-CLOSURES? is true, register REFERENCES-GRAPHS in the store. If
DEDUPLICATE? is true, then also deduplicate files common to CLOSURES and the
-rest of the store when registering the closures. SYSTEM-DIRECTORY is the name
-of the directory of the 'system' derivation."
+rest of the store when registering the closures. SYSTEM-DIRECTORY is the name
+of the directory of the 'system' derivation. Pass WAL-MODE? to
+register-closure."
(populate-root-file-system system-directory root)
(populate-store references-graphs root)
@@ -184,7 +189,8 @@ of the directory of the 'system' derivation."
(register-closure root
closure
#:reset-timestamps? #t
- #:deduplicate? deduplicate?))
+ #:deduplicate? deduplicate?
+ #:wal-mode? wal-mode?))
references-graphs))
(when bootloader-installer
diff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scm
index 31942e7386..4e1db37104 100644
--- a/gnu/system/images/hurd.scm
+++ b/gnu/system/images/hurd.scm
@@ -61,8 +61,8 @@
#~(lambda* (#:rest args)
(apply initialize-root-partition
(append args
- (list #:make-device-nodes
- make-hurd-device-nodes)))))
+ (list #:make-device-nodes make-hurd-device-nodes
+ #:wal-mode? #f)))))
(define hurd-disk-image
(image
diff --git a/guix/store/database.scm b/guix/store/database.scm
index a38e4d7e52..50b66ce282 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>
;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -20,6 +21,7 @@
(define-module (guix store database)
#:use-module (sqlite3)
#:use-module (guix config)
+ #:use-module (guix gexp)
#:use-module (guix serialization)
#:use-module (guix store deduplication)
#:use-module (guix base16)
@@ -27,6 +29,7 @@
#:use-module (guix build syscalls)
#:use-module ((guix build utils)
#:select (mkdir-p executable-file?))
+ #:use-module (guix utils)
#:use-module (guix build store-copy)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
@@ -97,17 +100,20 @@ as specified by SQL-SCHEMA."
(sqlite-exec db (call-with-input-file schema get-string-all)))
-(define (call-with-database file proc)
+(define* (call-with-database file proc #:key (wal-mode? #t))
"Pass PROC a database record corresponding to FILE. If FILE doesn't exist,
-create it and initialize it as a new database."
+create it and initialize it as a new database. Unless WAL-MODE? is set to #f,
+set journal_mode=WAL."
(let ((new? (and (not (file-exists? file))
(begin
(mkdir-p (dirname file))
#t)))
(db (sqlite-open file)))
- ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
- ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
- (sqlite-exec db "PRAGMA journal_mode=WAL;")
+ ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
+ (when wal-mode?
+ ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
+ ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
+ (sqlite-exec db "PRAGMA journal_mode=WAL;"))
;; Install a busy handler such that, when the database is locked, sqlite
;; retries until 30 seconds have passed, at which point it gives up and
@@ -200,10 +206,15 @@ prior to returning."
;; Default location of the store database.
(string-append %store-database-directory "/db.sqlite"))
-(define-syntax-rule (with-database file db exp ...)
- "Open DB from FILE and close it when the dynamic extent of EXP... is left.
-If FILE doesn't exist, create it and initialize it as a new database."
- (call-with-database file (lambda (db) exp ...)))
+(define-syntax with-database
+ (syntax-rules ()
+ "Open DB from FILE and close it when the dynamic extent of EXP... is left.
+If FILE doesn't exist, create it and initialize it as a new database. Pass
+#:wal-mode? to call-with-database."
+ ((_ file db #:wal-mode? wal-mode? exp ...)
+ (call-with-database file (lambda (db) exp ...) #:wal-mode? wal-mode?))
+ ((_ file db exp ...)
+ (call-with-database file (lambda (db) exp ...)))))
(define (sqlite-finalize stmt)
;; As of guile-sqlite3 0.1.0, cached statements aren't reset when
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 10 Jul 2020 18:25
Re: bug#42151: [PATCH 3/3] gnu: guix: Use sqlite/hurd for locking on the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87v9ivtk07.fsf@gnu.org
Ludovic Courtès writes:

Toggle quote (10 lines)
> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

>> * gnu/packages/package-management.scm (guix)[inputs]: Use sqlite/hurd.
>
> Same here.
>
> Which makes me thing: how about applying the sqlite/hurd patch in a
> build phase of sqlite that would be Hurd-specific? That way, we’d still
> have a single sqlite package.

Ah, yes that sounds better. We don't need sqlite/hurd "sqlite-for-hurd"
and all that. I'll give that a go!

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 10 Jul 2020 20:34
Re: bug#42151: [PATCH 1/3] gnu: Add sqlite/hurd with locking fix.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87sgdzte1s.fsf@gnu.org
Ludovic Courtès writes:

Toggle quote (15 lines)
> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> * gnu/packages/patches/sqlite3-hurd.patch: New file.
>> * gnu/local.mk (dist_patch_DATA): Add it.
>> * gnu/packages/sqlite.scm (sqlite/hurd): New variable.
>
> […]
>
>> +++ b/gnu/packages/patches/sqlite3-hurd.patch
>> @@ -0,0 +1,51 @@
>> +Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.31.1-5/20-hurd-locking-style.patch
>> +Upstream status: Not offered upstream.
>
> The URL is 404, could you fix it?

That's weird...Changing it (in completely rewritten patch) to


Toggle quote (4 lines)
> It would also be great if you could
> add a sentence or two explaining what this works around, so that our
> future selves know how to deal with this patch. :-)

Ah, sure!

Toggle quote (2 lines)
> Otherwise LGTM, you can push with changes along these lines!

As it's so much changed, I'll send it by one more time :-)

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan Nieuwenhuizen wrote on 10 Jul 2020 20:34
Re: bug#42151: [PATCH 2/3] gnu: guile-sqlite3: Use sqlite/hurd for locking on the Hurd.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151@debbugs.gnu.org)
87r1tjte0t.fsf@gnu.org
Ludovic Courtès writes:

Toggle quote (25 lines)
> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> This fixes
>>
>> guix offload: error: exception occurred on remote host 'localhost': (%exception #<inferior-object #<&store-protocol-error message: "setting synchronous mode: locking protocol" status: 1>>)
>>
>> * gnu/packages/guile.scm (guile-sqlite3)[inputs]: Use sqlite/hurd instead of
>> sqlite.
>> ---
>> gnu/packages/guile.scm | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
>> index a32bd33293..c651e2427a 100644
>> --- a/gnu/packages/guile.scm
>> +++ b/gnu/packages/guile.scm
>> @@ -665,7 +665,7 @@ Guile's foreign function interface.")
>> ("pkg-config" ,pkg-config)))
>> (inputs
>> `(("guile" ,guile-3.0)
>> - ("sqlite" ,sqlite)))
>> + ("sqlite" ,sqlite/hurd)))
>
> This should be guarded by ‘if (hurd-target?)’, right?

Yeah...but all changed now...look out for v3. Thanks!

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 2020 20:35
[PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710183551.12807-2-janneke@gnu.org

* gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
in call to ...
* gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?
parameter, pass it to ...
(register-closure): ... this, add #:wal-mode? parameter, pass it to ...
* guix/store/database.scm (with-database): ... this, add #:wal-mode?
parameter, pass it to ...
(call-with-database): ... this, add #:wal-mode? parameter; when
set to #f, do not set journal_model=WAL.
---
gnu/build/image.scm | 26 ++++++++++++++++----------
gnu/system/images/hurd.scm | 4 ++--
guix/store/database.scm | 29 ++++++++++++++++++++---------
3 files changed, 38 insertions(+), 21 deletions(-)

Toggle diff (156 lines)
diff --git a/gnu/build/image.scm b/gnu/build/image.scm
index e7b0418182..d8efa73f16 100644
--- a/gnu/build/image.scm
+++ b/gnu/build/image.scm
@@ -131,20 +131,23 @@ given CONFIG file."
(define* (register-closure prefix closure
#:key
(deduplicate? #t) (reset-timestamps? #t)
- (schema (sql-schema)))
+ (schema (sql-schema))
+ (wal-mode? #t))
"Register CLOSURE in PREFIX, where PREFIX is the directory name of the
target store and CLOSURE is the name of a file containing a reference graph as
produced by #:references-graphs.. As a side effect, if RESET-TIMESTAMPS? is
true, reset timestamps on store files and, if DEDUPLICATE? is true,
-deduplicates files common to CLOSURE and the rest of PREFIX."
+deduplicates files common to CLOSURE and the rest of PREFIX. Pass WAL-MODE?
+to call-with-database."
(let ((items (call-with-input-file closure read-reference-graph)))
(parameterize ((sql-schema schema))
(with-database (store-database-file #:prefix prefix) db
- (register-items db items
- #:prefix prefix
- #:deduplicate? deduplicate?
- #:reset-timestamps? reset-timestamps?
- #:registration-time %epoch)))))
+ #:wal-mode? wal-mode?
+ (register-items db items
+ #:prefix prefix
+ #:deduplicate? deduplicate?
+ #:reset-timestamps? reset-timestamps?
+ #:registration-time %epoch)))))
(define* (initialize-efi-partition root
#:key
@@ -164,14 +167,16 @@ deduplicates files common to CLOSURE and the rest of PREFIX."
(register-closures? #t)
system-directory
make-device-nodes
+ (wal-mode? #t)
#:allow-other-keys)
"Initialize the given ROOT directory. Use BOOTCFG and BOOTCFG-LOCATION to
install the bootloader configuration.
If REGISTER-CLOSURES? is true, register REFERENCES-GRAPHS in the store. If
DEDUPLICATE? is true, then also deduplicate files common to CLOSURES and the
-rest of the store when registering the closures. SYSTEM-DIRECTORY is the name
-of the directory of the 'system' derivation."
+rest of the store when registering the closures. SYSTEM-DIRECTORY is the name
+of the directory of the 'system' derivation. Pass WAL-MODE? to
+register-closure."
(populate-root-file-system system-directory root)
(populate-store references-graphs root)
@@ -184,7 +189,8 @@ of the directory of the 'system' derivation."
(register-closure root
closure
#:reset-timestamps? #t
- #:deduplicate? deduplicate?))
+ #:deduplicate? deduplicate?
+ #:wal-mode? wal-mode?))
references-graphs))
(when bootloader-installer
diff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scm
index 31942e7386..4e1db37104 100644
--- a/gnu/system/images/hurd.scm
+++ b/gnu/system/images/hurd.scm
@@ -61,8 +61,8 @@
#~(lambda* (#:rest args)
(apply initialize-root-partition
(append args
- (list #:make-device-nodes
- make-hurd-device-nodes)))))
+ (list #:make-device-nodes make-hurd-device-nodes
+ #:wal-mode? #f)))))
(define hurd-disk-image
(image
diff --git a/guix/store/database.scm b/guix/store/database.scm
index a38e4d7e52..50b66ce282 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017, 2019 Caleb Ristvedt <caleb.ristvedt@cune.org>
;;; Copyright © 2018, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -20,6 +21,7 @@
(define-module (guix store database)
#:use-module (sqlite3)
#:use-module (guix config)
+ #:use-module (guix gexp)
#:use-module (guix serialization)
#:use-module (guix store deduplication)
#:use-module (guix base16)
@@ -27,6 +29,7 @@
#:use-module (guix build syscalls)
#:use-module ((guix build utils)
#:select (mkdir-p executable-file?))
+ #:use-module (guix utils)
#:use-module (guix build store-copy)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
@@ -97,17 +100,20 @@ as specified by SQL-SCHEMA."
(sqlite-exec db (call-with-input-file schema get-string-all)))
-(define (call-with-database file proc)
+(define* (call-with-database file proc #:key (wal-mode? #t))
"Pass PROC a database record corresponding to FILE. If FILE doesn't exist,
-create it and initialize it as a new database."
+create it and initialize it as a new database. Unless WAL-MODE? is set to #f,
+set journal_mode=WAL."
(let ((new? (and (not (file-exists? file))
(begin
(mkdir-p (dirname file))
#t)))
(db (sqlite-open file)))
- ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
- ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
- (sqlite-exec db "PRAGMA journal_mode=WAL;")
+ ;; Using WAL breaks for the Hurd <https://bugs.gnu.org/42151>.
+ (when wal-mode?
+ ;; Turn DB in "write-ahead log" mode, which should avoid SQLITE_LOCKED
+ ;; errors when we have several readers: <https://www.sqlite.org/wal.html>.
+ (sqlite-exec db "PRAGMA journal_mode=WAL;"))
;; Install a busy handler such that, when the database is locked, sqlite
;; retries until 30 seconds have passed, at which point it gives up and
@@ -200,10 +206,15 @@ prior to returning."
;; Default location of the store database.
(string-append %store-database-directory "/db.sqlite"))
-(define-syntax-rule (with-database file db exp ...)
- "Open DB from FILE and close it when the dynamic extent of EXP... is left.
-If FILE doesn't exist, create it and initialize it as a new database."
- (call-with-database file (lambda (db) exp ...)))
+(define-syntax with-database
+ (syntax-rules ()
+ "Open DB from FILE and close it when the dynamic extent of EXP... is left.
+If FILE doesn't exist, create it and initialize it as a new database. Pass
+#:wal-mode? to call-with-database."
+ ((_ file db #:wal-mode? wal-mode? exp ...)
+ (call-with-database file (lambda (db) exp ...) #:wal-mode? wal-mode?))
+ ((_ file db exp ...)
+ (call-with-database file (lambda (db) exp ...)))))
(define (sqlite-finalize stmt)
;; As of guile-sqlite3 0.1.0, cached statements aren't reset when
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
J
J
Jan (janneke) Nieuwenhuizen wrote on 10 Jul 2020 20:35
[PATCH v3 1/2] gnu: sqlite: Add locking-mode fix for the Hurd.
(address . 42151@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20200710183551.12807-1-janneke@gnu.org
* gnu/packages/patches/sqlite-hurd.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/sqlite.scm (sqlite)[native-inputs]: Add it when building
for the Hurd.
[arguments]: Apply it when building for the Hurd.
---
gnu/local.mk | 1 +
gnu/packages/patches/sqlite-hurd.patch | 58 ++++++++++++++++++++++++++
gnu/packages/sqlite.scm | 19 +++++++++
3 files changed, 78 insertions(+)
create mode 100644 gnu/packages/patches/sqlite-hurd.patch

Toggle diff (130 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index 5c3b391960..42c9d0f379 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1370,6 +1370,7 @@ dist_patch_DATA = \
%D%/packages/patches/sdl-pango-header-guard.patch \
%D%/packages/patches/sdl-pango-matrix_declarations.patch \
%D%/packages/patches/sdl-pango-sans-serif.patch \
+ %D%/packages/patches/sqlite-hurd.patch \
%D%/packages/patches/patchutils-test-perms.patch \
%D%/packages/patches/patch-hurd-path-max.patch \
%D%/packages/patches/perl-autosplit-default-time.patch \
diff --git a/gnu/packages/patches/sqlite-hurd.patch b/gnu/packages/patches/sqlite-hurd.patch
new file mode 100644
index 0000000000..d80a2c5be8
--- /dev/null
+++ b/gnu/packages/patches/sqlite-hurd.patch
@@ -0,0 +1,58 @@
+Adapted from Debian: https://sources.debian.org/patches/sqlite3/3.32.3-1/20-hurd-locking-style.patch
+Upstream status: Not upstreamed.
+
+This patch is needed to get offloading to work.
+
+Sqlite can use simple file locking mode, but that does not work for the Hurd;
+a second sqlite process fails with a "locking protocol" error.
+
+See also: https://bugs.debian.org/529734.
+
+diff -purN sqlite-autoconf-3310100/sqlite3.c sqlite-autoconf-3310100-/sqlite3.c
+--- sqlite-autoconf-3310100/sqlite3.c 2020-01-27 21:25:19.000000000 +0100
++++ sqlite-autoconf-3310100-/sqlite3.c 2020-07-01 11:50:13.768333806 +0200
+@@ -33189,7 +33189,7 @@ SQLITE_PRIVATE const char *sqlite3Opcode
+ # include <sys/mman.h>
+ #endif
+
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ /* # include <sys/ioctl.h> */
+ # include <sys/file.h>
+ # include <sys/param.h>
+@@ -35676,7 +35676,7 @@ static int dotlockClose(sqlite3_file *id
+ **
+ ** Omit this section if SQLITE_ENABLE_LOCKING_STYLE is turned off
+ */
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+
+ /*
+ ** Retry flock() calls that fail with EINTR
+@@ -38586,7 +38586,7 @@ IOMETHODS(
+ 0 /* xShmMap method */
+ )
+
+-#if SQLITE_ENABLE_LOCKING_STYLE
++#if SQLITE_ENABLE_LOCKING_STYLE || defined(__GNU__)
+ IOMETHODS(
+ flockIoFinder, /* Finder function name */
+ flockIoMethods, /* sqlite3_io_methods object name */
+@@ -41142,6 +41142,8 @@ SQLITE_API int sqlite3_os_init(void){
+ UNIXVFS("unix", autolockIoFinder ),
+ #elif OS_VXWORKS
+ UNIXVFS("unix", vxworksIoFinder ),
++#elif defined(__GNU__)
++ UNIXVFS("unix", flockIoFinder ),
+ #else
+ UNIXVFS("unix", posixIoFinder ),
+ #endif
+@@ -41151,7 +41153,7 @@ SQLITE_API int sqlite3_os_init(void){
+ #if OS_VXWORKS
+ UNIXVFS("unix-namedsem", semIoFinder ),
+ #endif
+-#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS
++#if SQLITE_ENABLE_LOCKING_STYLE || OS_VXWORKS || defined(__GNU__)
+ UNIXVFS("unix-posix", posixIoFinder ),
+ #endif
+ #if SQLITE_ENABLE_LOCKING_STYLE
diff --git a/gnu/packages/sqlite.scm b/gnu/packages/sqlite.scm
index 8468131ddf..eeb77749d8 100644
--- a/gnu/packages/sqlite.scm
+++ b/gnu/packages/sqlite.scm
@@ -11,6 +11,7 @@
;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org>
;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
+;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -29,6 +30,7 @@
(define-module (gnu packages sqlite)
#:use-module (gnu packages)
+ #:use-module (gnu packages hurd)
#:use-module (gnu packages readline)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
@@ -65,6 +67,11 @@
"1bj936svd8i5g25xd1bj52hj4zca01fgl3sqkj86z9q5pkz4wa32"))))
(build-system gnu-build-system)
(inputs `(("readline" ,readline)))
+ (native-inputs (if (hurd-target?)
+ ;; TODO move into origin on the next rebuild cycle.
+ `(("hurd-locking-mode.patch"
+ ,@(search-patches "sqlite-hurd.patch")))
+ '()))
(outputs '("out" "static"))
(arguments
`(#:configure-flags
@@ -79,6 +86,18 @@
;; Column metadata is required by GNU Jami and Qt, et.al.
"-DSQLITE_ENABLE_COLUMN_METADATA"))
#:phases (modify-phases %standard-phases
+ ;; TODO: remove in the next rebuild cycle
+ ,@(if (hurd-target?)
+ `((add-after 'unpack 'patch-sqlite/hurd
+ (lambda* (#:key inputs native-inputs
+ #:allow-other-keys)
+ (let ((patch (assoc-ref
+ (if ,(%current-target-system)
+ native-inputs
+ inputs)
+ "hurd-locking-mode.patch")))
+ (invoke "patch" "-p1" "--force" "-i" patch)))))
+ '())
(add-after 'install 'move-static-library
(lambda* (#:key outputs #:allow-other-keys)
(let* ((out (assoc-ref outputs "out"))
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
L
L
Ludovic Courtès wrote on 11 Jul 2020 16:18
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87v9iuw2x5.fsf@gnu.org
Hi!

"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (6 lines)
> * gnu/packages/patches/sqlite-hurd.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add it.
> * gnu/packages/sqlite.scm (sqlite)[native-inputs]: Add it when building
> for the Hurd.
> [arguments]: Apply it when building for the Hurd.

LGTM, thank you!
L
L
Ludovic Courtès wrote on 11 Jul 2020 16:22
Re: bug#42151: [PATCH v3 2/2] image: Do not set journal_model=WAL for the Hurd.
(name . Jan (janneke) Nieuwenhuizen)(address . janneke@gnu.org)(address . 42151@debbugs.gnu.org)
87r1tiw2qt.fsf@gnu.org
"Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:

Toggle quote (12 lines)
>
> * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
> in call to ...
> * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode?
> parameter, pass it to ...
> (register-closure): ... this, add #:wal-mode? parameter, pass it to ...
> * guix/store/database.scm (with-database): ... this, add #:wal-mode?
> parameter, pass it to ...
> (call-with-database): ... this, add #:wal-mode? parameter; when
> set to #f, do not set journal_model=WAL.

[...]

Toggle quote (10 lines)
> +++ b/gnu/system/images/hurd.scm
> @@ -61,8 +61,8 @@
> #~(lambda* (#:rest args)
> (apply initialize-root-partition
> (append args
> - (list #:make-device-nodes
> - make-hurd-device-nodes)))))
> + (list #:make-device-nodes make-hurd-device-nodes
> + #:wal-mode? #f)))))

IWBN to have an XXX comment here stating why we disable WAL mode on
GNU/Hurd, possibly referencing this issue. That way, we can eventually
revisit the issue and hopefully get rid of this workaround.

Apart from that the patch LGTM, thanks!

Ludo’.
J
J
Jan Nieuwenhuizen wrote on 11 Jul 2020 16:36
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42151-done@debbugs.gnu.org)
87365yf79x.fsf@gnu.org
Ludovic Courtès writes:

Toggle quote (6 lines)
> "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org> skribis:
>
>> This fixes <https://bugs.gnu.org/42151>.
>>
>> * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f
>> in call to ...
[...]
Toggle quote (7 lines)
>> + (list #:make-device-nodes make-hurd-device-nodes
>> + #:wal-mode? #f)))))
>
> IWBN to have an XXX comment here stating why we disable WAL mode on
> GNU/Hurd, possibly referencing this issue. That way, we can eventually
> revisit the issue and hopefully get rid of this workaround.

Sure, done!

Toggle quote (2 lines)
> Apart from that the patch LGTM, thanks!

Yay! Pushed series to master as 4b9eecd322e566783369795ebea63a479b51f486.

Let the offloadings commence!

Janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
Closed
?