Manual database index.db embeds timestamps

  • Done
  • quality assurance status badge
Details
4 participants
  • Ruud van Asseldonk
  • Sergey Poznyakoff
  • Ludovic Courtès
  • Ricardo Wurmus
Owner
unassigned
Submitted by
Ruud van Asseldonk
Severity
normal
R
R
Ruud van Asseldonk wrote on 10 Dec 2017 23:47
(address . bug-guix@gnu.org)
c5553a2e-4a7d-f712-f35a-2279b137fe87@veniogames.com
The manual database file index.db embeds mtimes of the files it refers
to, making it not reproducible. This is an issue for building
reproducible profiles (as produced by `guix pack` for example).

The number that are not reproducible can be seen with `accessdb
index.db`, which will output something like

$version$ -> "2.5.0"
acme-client -> "- 1 1 1512941854 498178709 B - - gz secure ACME client"

And when built again on a clean installation:

$version$ -> "2.5.0"
acme-client -> "- 1 1 1512942998 296814690 B - - gz secure ACME client"

I asked the man-db maintainer about this, who replied:

Toggle quote (8 lines)
> Those are timestamps, seconds and nanoseconds respectively. You should
> get reproducible output if you make sure the mtimes of manual pages are
> reproducible. (man-db needs to keep track of mtimes so that it knows
> when a database is out of date.) If the manual pages come straight from
> the source package, this should be straightforward; if they're
> generated, you'll need some strategy to ensure that they're generated
> with a reproducible mtime.

Now the strange thing is, when I check the profile directory, both the
man file symlinks, and the files they refer to, have the mtime set to
epoch. I suspected that the mtime would be reset after the manual
database had already been built, but inserting calls to utime after
`(symlink manpage dest-file)` in the `manual-database` procedure, did
not resolve the issue.

The mtime that ends up in index.db does correspond to a recent time, and
for larger databases the timestamps can differ among entries, but so far
I have only seen them differ in the nanoseconds. The seconds timestamp
looks like the timestamp at which the database was generated. So either
the mtime of the input files is not epoch when man-db runs, or man-db is
somehow setting the mtime of every entry to the current time as it adds
them.

I looked at the man-db source and I did not find anything that looked
like it was storing current times as mtimes in the database, although I
did not look very carefully. I suspect that the mtimes of the files to
scan are actually wrong. It should be possible to confirm this by
introducing a delay when creating the man files, and when making the
symlinks, and seeing if the delay shows up in the database.

Help with diagnosing this further, or ideas about what could be going on
and how to resolve it, would be appreciated. I am willing to work on a
fix, but I am not very familiar with Guix yet.

Kind regards,

Ruud van Asseldonk
L
L
Ludovic Courtès wrote on 15 Dec 2017 22:30
(name . Ruud van Asseldonk)(address . dev+guix@veniogames.com)(address . 29654@debbugs.gnu.org)
87mv2jfzmi.fsf@gnu.org
Hi Ruud,

Ruud van Asseldonk <dev+guix@veniogames.com> skribis:

Toggle quote (15 lines)
> The manual database file index.db embeds mtimes of the files it refers
> to, making it not reproducible. This is an issue for building
> reproducible profiles (as produced by `guix pack` for example).
>
> The number that are not reproducible can be seen with `accessdb
> index.db`, which will output something like
>
> $version$ -> "2.5.0"
> acme-client -> "- 1 1 1512941854 498178709 B - - gz secure ACME client"
>
> And when built again on a clean installation:
>
> $version$ -> "2.5.0"
> acme-client -> "- 1 1 1512942998 296814690 B - - gz secure ACME client"

Good catch.

I was already motivated to create the database directly from Scheme
(‘man-db’ is quite slow), so that gave me an additional excuse. ;-)

The attached patch does that. The timestamps are always set to zero.

You can check for reproducibility by doing:

guix package -p foo -i bar baz
guix build --check /gnu/store/…-manual-database.drv

where the .drv is the one shown in the ‘guix package’ output.

Unfortunately, this is not fully deterministic: when running --check
several times in a row, I occasionally get different results. I suspect
GDBM’s output is not fully deterministic.

Thoughts?

Ludo’.
From 9da89cbdda484c106ff2706f09089d87bdf88a45 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Fri, 15 Dec 2017 22:08:34 +0100
Subject: [PATCH 1/2] gnu: guile-gdbm-ffi: Default to Guile 2.2.

* gnu/packages/guile.scm (guile-gdbm-ffi)[inputs]: Switch to GUILE-2.2.
(guile2.0-gdbm-ffi, guile2.2-gdbm-ffi): New variables.
---
gnu/packages/guile.scm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)

Toggle diff (28 lines)
diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index c0fda71ea..887e360a3 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -1097,7 +1097,7 @@ inspired by the SCSH regular expression system.")
;; compile to the destination
(compile-file gdbm.scm-dest gdbm.go-dest)))))
(inputs
- `(("guile" ,guile-2.0)))
+ `(("guile" ,guile-2.2)))
(propagated-inputs
`(("gdbm" ,gdbm)))
(home-page "https://github.com/ijp/guile-gdbm")
@@ -1107,8 +1107,11 @@ inspired by the SCSH regular expression system.")
Guile's foreign function interface.")
(license license:gpl3+)))
+(define-public guile2.0-gdbm-ffi
+ (package-for-guile-2.0 guile-gdbm-ffi))
+
(define-public guile2.2-gdbm-ffi
- (package-for-guile-2.2 guile-gdbm-ffi))
+ (deprecated-package "guile2.2-gdbm-ffi" guile-gdbm-ffi))
(define-public guile-sqlite3
(let ((commit "607721fe1174a299e45d457acacf94eefb964071"))
--
2.15.1
Attachment: 0002-DRAFT-profiles-Use-guix-man-db-to-create-the-manual-.patch
L
L
Ludovic Courtès wrote on 15 Dec 2017 22:54
(name . Ruud van Asseldonk)(address . dev+guix@veniogames.com)(address . 29654@debbugs.gnu.org)
87ind7fyip.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (3 lines)
> I was already motivated to create the database directly from Scheme
> (‘man-db’ is quite slow), so that gave me an additional excuse. ;-)

FWIW, on my profile with 237 packages corresponding to 19,122 man pages,
on a warm cache, (guix man-db) takes ~8 seconds whereas the previous
method (invoking ‘man-db’) takes ~26 seconds.

My guess is that the main difference is due to not forking a gzip
process for each man page.

Ludo’.
R
R
Ricardo Wurmus wrote on 16 Dec 2017 01:35
(name . Ludovic Courtès)(address . ludo@gnu.org)
87po7fo6gv.fsf@elephly.net
Hi Ludo,

Toggle quote (5 lines)
> I was already motivated to create the database directly from Scheme
> (‘man-db’ is quite slow), so that gave me an additional excuse. ;-)
>
> The attached patch does that. The timestamps are always set to zero.

That’s great!

Toggle quote (4 lines)
> Unfortunately, this is not fully deterministic: when running --check
> several times in a row, I occasionally get different results. I suspect
> GDBM’s output is not fully deterministic.

Hmm, I dumped the contents of the generated databases with gdbm_dump and
couldn’t find any difference aside from the header (which is produced by
gdbm_dump itself). Diffoscope shows a lot of differences, though.

I thought that maybe the difference comes from the fact that upon adding
new entries gdbm grows the hash table. After setting the initial size
to a multiple of the number of entries I haven’t been able to generate a
non-reproducible database.

My only change is in “write-mandb-database”:

(gdbm-open file GDBM_WRCREAT #:block-size (* 512 (length entries)))

I tried this:

./pre-inst-env guix package -p foo -i coreutils guile
for i in `seq 30`; do ./pre-inst-env guix build --check -K /gnu/store/pg3684khpj69py40v7p76b90r9q4j2lv-manual-database.drv; done

Seems fine. Coincidence or did I get lucky?

Toggle quote (14 lines)
> +(define (entry->string entry)
> + "Return the wire format for ENTRY as a string."
> + (match entry
> + (($ <mandb-entry> file name section synopsis)
> + (string-append (abbreviate-file-name file) "\t"
> + (number->string section) "\t"
> + (number->string section)
> +
> + ;; Timestamps, that we always set to the epoch.
> + "\t0\t0"
> +
> + ;; XXX: Weird things.
> + "\tB\t-\t-\tgz\t"

What’s that?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
L
L
Ludovic Courtès wrote on 17 Dec 2017 00:51
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87wp1mcjty.fsf@gnu.org
Howdy Ricardo,

Ricardo Wurmus <rekado@elephly.net> skribis:

Toggle quote (24 lines)
>> Unfortunately, this is not fully deterministic: when running --check
>> several times in a row, I occasionally get different results. I suspect
>> GDBM’s output is not fully deterministic.
>
> Hmm, I dumped the contents of the generated databases with gdbm_dump and
> couldn’t find any difference aside from the header (which is produced by
> gdbm_dump itself). Diffoscope shows a lot of differences, though.
>
> I thought that maybe the difference comes from the fact that upon adding
> new entries gdbm grows the hash table. After setting the initial size
> to a multiple of the number of entries I haven’t been able to generate a
> non-reproducible database.
>
> My only change is in “write-mandb-database”:
>
> (gdbm-open file GDBM_WRCREAT #:block-size (* 512 (length entries)))
>
> I tried this:
>
> ./pre-inst-env guix package -p foo -i coreutils guile
> for i in `seq 30`; do ./pre-inst-env guix build --check -K /gnu/store/pg3684khpj69py40v7p76b90r9q4j2lv-manual-database.drv; done
>
> Seems fine. Coincidence or did I get lucky?

I checked with the program below. It helps, but does not entirely fix
it:
(use-modules (guix man-db) (guix hash) (guix base32)) (define %database "/tmp/index.db") (let loop () (false-if-exception (delete-file %database)) (write-mandb-database %database (mandb-entries "/home/ludo/.guix-profile/share/man")) (pk (stat:size (stat %database)) (bytevector->nix-base32-string (file-sha256 %database))) (loop))
Valgrind reports this:

Toggle snippet (22 lines)
==8395== Syscall param write(buf) points to uninitialised byte(s)
==8395== at 0x53E4A8D: ??? (in /gnu/store/3h31zsqxjjg52da5gp3qmhkh4x8klhah-glibc-2.25/lib/libpthread-2.25.so)
==8395== by 0xACAF44D: _gdbm_full_write (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==8395== by 0xACAC6AD: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==8395== by 0x55FA0BF: ffi_call_unix64 (in /gnu/store/kvi64k387hqdrn59gsgd09brxh65jxjj-libffi-3.2.1/lib/libffi.so.6.0.4)
==8395== by 0x55F8EE0: ffi_call (in /gnu/store/kvi64k387hqdrn59gsgd09brxh65jxjj-libffi-3.2.1/lib/libffi.so.6.0.4)
==8395== by 0x4E8C23C: scm_i_foreign_call (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4EF9243: vm_regular_engine (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4EFC7B9: scm_call_n (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4E80A06: scm_primitive_eval (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4E80A62: scm_eval (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4ECBA6F: scm_shell (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4E974AC: invoke_main_func (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== Address 0xced0044 is 4 bytes inside a block of size 8,388,608 alloc'd
==8395== at 0x4C2AAD6: malloc (in /gnu/store/p2b1rzqlpdqbhn42g76xzgykbivwc063-valgrind-3.12.0/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8395== by 0xACAC5E6: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==8395== by 0x55FA0BF: ffi_call_unix64 (in /gnu/store/kvi64k387hqdrn59gsgd09brxh65jxjj-libffi-3.2.1/lib/libffi.so.6.0.4)
==8395== by 0x55F8EE0: ffi_call (in /gnu/store/kvi64k387hqdrn59gsgd09brxh65jxjj-libffi-3.2.1/lib/libffi.so.6.0.4)
==8395== by 0x4E8C23C: scm_i_foreign_call (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)


Toggle quote (16 lines)
>> +(define (entry->string entry)
>> + "Return the wire format for ENTRY as a string."
>> + (match entry
>> + (($ <mandb-entry> file name section synopsis)
>> + (string-append (abbreviate-file-name file) "\t"
>> + (number->string section) "\t"
>> + (number->string section)
>> +
>> + ;; Timestamps, that we always set to the epoch.
>> + "\t0\t0"
>> +
>> + ;; XXX: Weird things.
>> + "\tB\t-\t-\tgz\t"
>
> What’s that?

In db_store.c it’s done like this:

Toggle snippet (14 lines)
MYDBM_SET (cont, xasprintf (
"%s\t%s\t%s\t%ld\t%ld\t%c\t%s\t%s\t%s\t%s",
dash_if_unset (in->name),
in->ext,
in->sec,
(long) in->mtime.tv_sec,
in->mtime.tv_nsec,
in->id,
in->pointer,
in->filter,
in->comp,
in->whatis));

and db_storage.h says:

Toggle snippet (21 lines)
struct mandata {
struct mandata *next; /* ptr to next structure, if any */
char *addr; /* ptr to memory containing the fields */

char *name; /* Name of page, if != key */

/* The following are all const because they should be pointers to
* parts of strings allocated elsewhere (often the addr field above)
* and should not be written through or freed themselves.
*/
const char *ext; /* Filename ext w/o comp ext */
const char *sec; /* Section name/number */
char id; /* id for this entry */
const char *pointer; /* id related file pointer */
const char *comp; /* Compression extension */
const char *filter; /* filters needed for the page */
const char *whatis; /* whatis description for page */
struct timespec mtime; /* mod time for file */
};

The ‘B’ part gives the kind of manual page:

Toggle snippet (10 lines)
/* These definitions give an inherent precedence to each particular type
of manual page:
ULT_MAN: ultimate manual page, the full source nroff file.
SO_MAN: source nroff file containing .so request to an ULT_MAN.
WHATIS_MAN: virtual `whatis referenced' page pointing to an ULT_MAN.
STRAY_CAT: pre-formatted manual page with no source.
WHATIS_CAT: virtual `whatis referenced' page pointing to a STRAY_CAT. */

I’ve updated man-db.scm to handle that better.

Thanks,
Ludo’.
Toggle diff (115 lines)
diff --git a/guix/man-db.scm b/guix/man-db.scm
index b42558b06..3ce268547 100644
--- a/guix/man-db.scm
+++ b/guix/man-db.scm
@@ -29,6 +29,7 @@
mandb-entry-name
mandb-entry-section
mandb-entry-synopsis
+ mandb-entry-kind
mandb-entries
write-mandb-database))
@@ -47,12 +48,13 @@
(module-use! (current-module) (resolve-interface '(gdbm)))
(define-record-type <mandb-entry>
- (mandb-entry file-name name section synopsis)
+ (mandb-entry file-name name section synopsis kind)
mandb-entry?
(file-name mandb-entry-file-name) ;e.g., "../abiword.1.gz"
(name mandb-entry-name) ;e.g., "ABIWORD"
(section mandb-entry-section) ;number
- (synopsis mandb-entry-synopsis)) ;string
+ (synopsis mandb-entry-synopsis) ;string
+ (kind mandb-entry-kind)) ;'ultimate | 'link
(define (mandb-entry<? entry1 entry2)
(match entry1
@@ -74,16 +76,26 @@
(define (entry->string entry)
"Return the wire format for ENTRY as a string."
(match entry
- (($ <mandb-entry> file name section synopsis)
+ (($ <mandb-entry> file name section synopsis kind)
+ ;; See db_store.c:make_content in man-db for the format.
(string-append (abbreviate-file-name file) "\t"
(number->string section) "\t"
(number->string section)
- ;; Timestamps, that we always set to the epoch.
+ ;; Timestamp that we always set to the epoch.
"\t0\t0"
- ;; XXX: Weird things.
- "\tB\t-\t-\tgz\t"
+ ;; See "db_storage.h" in man-db for the different kinds.
+ "\t"
+ (case kind
+ ((ultimate) "A") ;ultimate man page
+ ((link) "B") ;".so" link to other man page
+ (else "A")) ;something that doesn't matter much
+
+ "\t-\t-\t"
+
+ (if (string-suffix? ".gz" file) "gz" "")
+ "\t"
synopsis "\x00"))))
@@ -94,7 +106,8 @@
(define (write-mandb-database file entries)
"Write ENTRIES to FILE as a man-db database. FILE is usually
\".../index.db\", and is a GDBM database."
- (let ((db (gdbm-open file GDBM_WRCREAT)))
+ (let ((db (gdbm-open file GDBM_WRCREAT
+ #:block-size (* 512 (length entries)))))
(gdbm-set! db %version-key %version-value)
;; Write ENTRIES in sorted order so we get deterministic output.
@@ -141,33 +154,37 @@
(string->number (string-drop (string-drop-right str 1) 1))
(string->number str)))
+ ;; Note: This works for both gzipped and uncompressed files.
(call-with-gzip-input-port (open-file file "r0")
(lambda (port)
(let loop ((name #f)
(section #f)
- (synopsis #f))
+ (synopsis #f)
+ (kind 'ultimate))
(if (and name section synopsis)
- (mandb-entry file name section synopsis)
+ (mandb-entry file name section synopsis kind)
(let ((line (read-line port)))
(if (eof-object? line)
- (mandb-entry file name (or section 0) (or synopsis ""))
+ (mandb-entry file name (or section 0) (or synopsis "")
+ kind)
(match (string-tokenize line)
((".TH" name (= string->number* section) _ ...)
- (loop name section synopsis))
+ (loop name section synopsis kind))
((".SH" (or "NAME" "\"NAME\""))
- (loop name section (read-synopsis port)))
+ (loop name section (read-synopsis port) kind))
((".so" link)
(match (and=> (resolve link)
(cut man-page->entry <> resolve))
(#f
- (loop name section synopsis))
+ (loop name section synopsis 'link))
(alias
(mandb-entry file
(mandb-entry-name alias)
(mandb-entry-section alias)
- (mandb-entry-synopsis alias)))))
+ (mandb-entry-synopsis alias)
+ 'link))))
(_
- (loop name section synopsis))))))))))
+ (loop name section synopsis kind))))))))))
(define (man-files directory)
"Return the list of man pages found under DIRECTORY, recursively."
L
L
Ludovic Courtès wrote on 17 Dec 2017 01:04
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87shcacj8p.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (5 lines)
> ==8395== Syscall param write(buf) points to uninitialised byte(s)
> ==8395== at 0x53E4A8D: ??? (in /gnu/store/3h31zsqxjjg52da5gp3qmhkh4x8klhah-glibc-2.25/lib/libpthread-2.25.so)
> ==8395== by 0xACAF44D: _gdbm_full_write (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
> ==8395== by 0xACAC6AD: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)

This is almost certainly an uninitialized malloc’d area that goes
straight to disk as can be seen by running the script I gave before with
‘MALLOC_PERTURB_’ set:

Toggle snippet (31 lines)
ludo@ribbon ~/src/guix$ MALLOC_PERTURB_=22 ./pre-inst-env guile t.scm

;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na")

;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na")

;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na")

;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na")
C-c C-c
ludo@ribbon ~/src/guix$ MALLOC_PERTURB_=44 ./pre-inst-env guile t.scm

;;; (33554432 "03kl992ypwxxp4cplbhz05b04ihjh96d3pldwgz7qaj4ls0qssr3")

;;; (33554432 "03kl992ypwxxp4cplbhz05b04ihjh96d3pldwgz7qaj4ls0qssr3")

;;; (33554432 "03kl992ypwxxp4cplbhz05b04ihjh96d3pldwgz7qaj4ls0qssr3")

;;; (33554432 "03kl992ypwxxp4cplbhz05b04ihjh96d3pldwgz7qaj4ls0qssr3")
C-c C-c
ludo@ribbon ~/src/guix$ MALLOC_PERTURB_=22 ./pre-inst-env guile t.scm

;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na")

;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na")

;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na")

;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na")

So for now, until gdbm is fixed, we can always work around the issue by
setting MALLOC_PERTURB_.

How does that sound?

Ludo’.
L
L
Ludovic Courtès wrote on 17 Dec 2017 01:26
GDBM output is not deterministic
(address . bug-gdbm@gnu.org)
87d13eci8m.fsf_-_@gnu.org
Hello Sergey & co.,

While investigating https://bugs.gnu.org/29654, we realized that
databases produced by GDBM are not bit-for-bit reproducible. This can
be seen with this program:

Toggle snippet (11 lines)
#include <unistd.h>
#include <gdbm.h>

int
main ()
{
unlink ("/tmp/t.db");
gdbm_open ("/tmp/t.db", 0, GDBM_WRCREAT, 0644, NULL);
}

Under Valgrind, we get:

Toggle snippet (11 lines)
==15981== Syscall param write(buf) points to uninitialised byte(s)
==15981== at 0x5335080: __write_nocancel (in /gnu/store/3h31zsqxjjg52da5gp3qmhkh4x8klhah-glibc-2.25/lib/libc-2.25.so)
==15981== by 0x4E3E44D: _gdbm_full_write (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==15981== by 0x4E3B6AD: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==15981== by 0x400743: main (in /home/ludo/src/guix/a.out)
==15981== Address 0x55fb204 is 4 bytes inside a block of size 4,096 alloc'd
==15981== at 0x4C2AAD6: malloc (in /gnu/store/p2b1rzqlpdqbhn42g76xzgykbivwc063-valgrind-3.12.0/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15981== by 0x4E3B5E6: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==15981== by 0x400743: main (in /home/ludo/src/guix/a.out)

… which suggests that an uninitialized block is being written straightly
to disk.

If we tell libc to initialize ‘malloc’ areas that were otherwise left
uninitialized, we can confirm the problem (again with the program
above):

Toggle snippet (46 lines)
$ for i in `seq 0 20` ; do MALLOC_PERTURB_=$i ./a.out ; sha256sum /tmp/t.db ; done
0483a04a43d0a08a5a0aa71d09b262b62eb2504e85c5fe7104b783a5f0f7ae15 /tmp/t.db
29122b2994face5b5a716c8cd0e6beda7f05c3441441447872c2f15e7e2db308 /tmp/t.db
34a130820ab6d5e638adae55e83cb784fec036c2da4727a113d3d1191331bd7c /tmp/t.db
9604ce76ea1f1b37968bd037798c0eb212a1e654f060b7c3d66011620b249212 /tmp/t.db
a1c275c1bf1b5980166431bd7245d2d649499a8cad233059abf2a6cbf2dec19b /tmp/t.db
1cbe43f146a158ac4c21064feac1371e7a6bb4ac9d0013b15acfe7fec754486f /tmp/t.db
83778759475790f1377adadd77b9d2eee7db7e5e43e006ef04e805f4b4309590 /tmp/t.db
6b80d00119fd806d05f8863056919ea0a6b66a0771d16439b072e2cc2eac6b58 /tmp/t.db
f1f65b95e39f74cff12c3e830eea77a9841c8793f4441c6a8bfa9ad92d765d8d /tmp/t.db
3dd2479f44c0be15e02f0040fcef3d765a306dbd2a47b3557cb19fb1282b0b05 /tmp/t.db
fed4cc15d84f676737954e80ab594034e47d2f2b7cbcd12dad722dddff6c5afb /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
3d9bccf127fd76b56617435c412aeec0c8fc95ab6b9d5f9e3f24060398899c26 /tmp/t.db
743808bd91a42e2c149a30bbdfacadbaa4124f82e01ea43323d5c1fd10de5499 /tmp/t.db
86a92606713ca03cab09f1cc97463184ed0c4d2051d7512dd98b136bc7ca1465 /tmp/t.db
93f5428a4fde073cab1737c1efd64e2554731db1ab51d77aec95c81592b014a0 /tmp/t.db
b93fd6692eeebfc5b49b64857de7647a88e541be7d4fd4e62fb6771cb4b00618 /tmp/t.db
ba08b6acee5812c3b3770f842aca7435729acc1082cddf8bfe26024f9ea8f54e /tmp/t.db
30dc2e3d1ea0150e4a200435b1f609de726680ebfc8661a9d121afbd4cf0b0a8 /tmp/t.db
0de2cc18e445ff0a2697cd2783f379fa202d57020db9a73c55c4f7b8fd5cac74 /tmp/t.db
dd938784d3799c5a575abc3fe24191f3e1ac593643719a9694f9fed32ed272da /tmp/t.db
$ for i in `seq 0 20` ; do MALLOC_PERTURB_=11 ./a.out ; sha256sum /tmp/t.db ; done
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db

I guess the fix is to use ‘calloc’ instead of ‘malloc’ for this specific
allocation.

WDYT?

Ludo’.
L
S
S
Sergey Poznyakoff wrote on 17 Dec 2017 10:19
Re: GDBM output is not deterministic
(name . Ludovic Courtès)(address . ludo@gnu.org)
20171217111925.30092@ulysses.gnu.org.ua
Hi Ludovic,

Toggle quote (2 lines)
> I guess the fix is to use calloc

Yes, that's quite reasonable. I'll fix that. Thanks for reporting!

Regards,
Sergey
R
R
Ruud van Asseldonk wrote on 17 Dec 2017 20:51
Re: bug#29654: Manual database index.db embeds timestamps
(address . 29654@debbugs.gnu.org)
b82a388c-e087-cd6f-ceeb-66a31324f601@veniogames.com
Thanks so much for looking into this!

Unfortunately, I am unable to pull that commit. Guix pull output:

Updating from Git repository at
Building from Git commit b8396f96bfeadfa63e7ad2afc2ab5a37f37f5f81...
The following derivation will be built:
/gnu/store/m8wcw35f4d33rzf54zzcrg0g80m0zi4r-guix-latest.drv
copying and compiling to
'/gnu/store/2g14x91l9wh4wv8zsvnkfblrrdfcysg4-guix-latest' with Guile
2.2.2...
loading... 17.1% of 659 filesBacktrace:
12 (primitive-load "/gnu/store/9pgqmhjclkayr5bdhqfk69sbn84?")
In ./guix/build/pull.scm:
127:8 11 (build-guix _ _ #:system _ #:storedir _ #:localstatedir ?)
In ./guix/build/compile.scm:
158:6 10 (compile-files _ _ ("/gnu/store/2g14x91l9wh4wv8zsvn?" ?) ?)
107:11 9 (load-files "/gnu/store/2g14x91l9wh4wv8zsvnkfblrrdfcys?" ?)
In ice-9/boot-9.scm:
2792:17 8 (resolve-interface (guix man-db) #:select _ #:hide _ # _ ?)
2718:10 7 (_ (guix man-db) _ _ #:ensure _)
2986:16 6 (try-module-autoload _ _)
2316:4 5 (save-module-excursion #<procedure 44847e0 at ice-9/boo?>)
3006:22 4 (_)
In unknown file:
3 (primitive-load-path "guix/man-db" #<procedure 40e90a0 ?>)
In ice-9/eval.scm:
196:43 2 (_ #f)
In ice-9/boot-9.scm:
2795:6 1 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ ?)
In unknown file:
0 (scm-error misc-error #f "~A ~S" ("no code for modu?" ?) ?)

ERROR: In procedure scm-error:
ERROR: no code for module (gdbm)

This happens even in an environment with guile-gdbm-ffi and gdbm
packages. Am I missing something?

Best,

Ruud

Ludovic Courtès wrote:
Toggle quote (12 lines)
> Hello,
>
> I went ahead and pushed (guix man-db) with MALLOC_PERTURB_ set as a
> workaround:
>
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b8396f96bfeadfa63e7ad2afc2ab5a37f37f5f81
>
> AFAICS this fixes the problem.
>
> Thanks,
> Ludo’.
>
L
L
Ludovic Courtès wrote on 17 Dec 2017 22:32
(name . Ruud van Asseldonk)(address . dev+guix@veniogames.com)
87lgi182hg.fsf@gnu.org
Ruud van Asseldonk <dev+guix@veniogames.com> skribis:

Toggle quote (35 lines)
> Unfortunately, I am unable to pull that commit. Guix pull output:
>
> Updating from Git repository at
> 'https://git.savannah.gnu.org/git/guix.git'...
> Building from Git commit b8396f96bfeadfa63e7ad2afc2ab5a37f37f5f81...
> The following derivation will be built:
> /gnu/store/m8wcw35f4d33rzf54zzcrg0g80m0zi4r-guix-latest.drv
> copying and compiling to
> '/gnu/store/2g14x91l9wh4wv8zsvnkfblrrdfcysg4-guix-latest' with Guile
> 2.2.2...
> loading... 17.1% of 659 filesBacktrace:
> 12 (primitive-load "/gnu/store/9pgqmhjclkayr5bdhqfk69sbn84?")
> In ./guix/build/pull.scm:
> 127:8 11 (build-guix _ _ #:system _ #:storedir _ #:localstatedir ?)
> In ./guix/build/compile.scm:
> 158:6 10 (compile-files _ _ ("/gnu/store/2g14x91l9wh4wv8zsvn?" ?) ?)
> 107:11 9 (load-files "/gnu/store/2g14x91l9wh4wv8zsvnkfblrrdfcys?" ?)
> In ice-9/boot-9.scm:
> 2792:17 8 (resolve-interface (guix man-db) #:select _ #:hide _ # _ ?)
> 2718:10 7 (_ (guix man-db) _ _ #:ensure _)
> 2986:16 6 (try-module-autoload _ _)
> 2316:4 5 (save-module-excursion #<procedure 44847e0 at ice-9/boo?>)
> 3006:22 4 (_)
> In unknown file:
> 3 (primitive-load-path "guix/man-db" #<procedure 40e90a0 ?>)
> In ice-9/eval.scm:
> 196:43 2 (_ #f)
> In ice-9/boot-9.scm:
> 2795:6 1 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ ?)
> In unknown file:
> 0 (scm-error misc-error #f "~A ~S" ("no code for modu?" ?) ?)
>
> ERROR: In procedure scm-error:
> ERROR: no code for module (gdbm)

My bad, fixed in 16613d230b3d9a9cf307c5c5d3899eb0a0c93b0e, which makes
GDBM is “soft dependency” (we only need it when building the manual
database, not when building Guix itself, after all.)

Thanks!

Ludo’.
Closed
?