guix gc: error: executing SQLite statement: database disk image is malformed

Open

Details

4 participants

Chris Marusich
Mike Gerwitz
pkill9
Ricardo Wurmus

Owner: unassigned

Submitted by: Chris Marusich

Severity: normal

Chris Marusich wrote on 16 Jul 2019 09:41

Recipients:(address . bug-guix@gnu.org)

Message-ID:87v9w2ctpm.fsf@gmail.com

Hi,

My disk was filling up, so I tried to run "guix gc", but I got an error

instead:

Toggle snippet (8 lines)

$ guix gc -C 25GiB

...

deleting `/gnu/store/n0gyzfw77ik35ld9d0d4737w88f11m4b-profile.drv'

deleting `/gnu/store/fl7w0dlki7c906isiiflf9ka4c49zcmi-ca-certificate-bundle.drv'

deleting `/gnu/store/ipn4xvvb3wrbx4lhzwdyyylvj42vyg6f-xdg-desktop-database.drv'

guix gc: error: executing SQLite statement: database disk image is malformed

What does this mean?  How concerned should I be?  How can I fix it?  I
looked for existing bug reports, but I didn't find anything.  I haven't
tried running the command a second time.

FYI, I was building one package using my Guix installation via "guix
build --cores=1", and another package using a Guix checkout via
"./pre-inst-env guix build --cores", when I decided to run this "guix
gc" command in another shell.  I don't know if that matters.  My
expectation is that I can run GC even when building software, thanks to
the guix-daemon's safe concurrent garbage collection algorithm.

-- 
Chris

Ricardo Wurmus wrote on 16 Jul 2019 14:44

Recipients:(address . cmmarusich@gmail.com)(address . 36687@debbugs.gnu.org)

Message-ID:87tvbmgndy.fsf@elephly.net

Chris Marusich <cmmarusich@gmail.com> writes:

Toggle quote (14 lines)> My disk was filling up, so I tried to run "guix gc", but I got an error
> instead:
>
> --8<---------------cut here---------------start------------->8---
> $ guix gc -C 25GiB
> ...
> deleting `/gnu/store/n0gyzfw77ik35ld9d0d4737w88f11m4b-profile.drv'
> deleting `/gnu/store/fl7w0dlki7c906isiiflf9ka4c49zcmi-ca-certificate-bundle.drv'
> deleting `/gnu/store/ipn4xvvb3wrbx4lhzwdyyylvj42vyg6f-xdg-desktop-database.drv'
> guix gc: error: executing SQLite statement: database disk image is malformed
> --8<---------------cut here---------------end--------------->8---
>
> What does this mean?

I think this means that the database has been corrupted.

Toggle quote (5 lines)

> FYI, I was building one package using my Guix installation via "guix

> build --cores=1", and another package using a Guix checkout via

> "./pre-inst-env guix build --cores", when I decided to run this "guix

> gc" command in another shell. I don't know if that matters.

I don’t think this is related.

Toggle quote (4 lines)

> My expectation is that I can run GC even when building software,

> thanks to the guix-daemon's safe concurrent garbage collection

> algorithm.

That’s correct.

Ricardo

Chris Marusich wrote on 19 Jul 2019 07:30

Recipients:(name . Ricardo Wurmus)(address . rekado@elephly.net)(address . 36687@debbugs.gnu.org)

Message-ID:87pnm6egm4.fsf@gmail.com

Ricardo Wurmus <rekado@elephly.net> writes:

Toggle quote (18 lines)> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> My disk was filling up, so I tried to run "guix gc", but I got an error
>> instead:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ guix gc -C 25GiB
>> ...
>> deleting `/gnu/store/n0gyzfw77ik35ld9d0d4737w88f11m4b-profile.drv'
>> deleting `/gnu/store/fl7w0dlki7c906isiiflf9ka4c49zcmi-ca-certificate-bundle.drv'
>> deleting `/gnu/store/ipn4xvvb3wrbx4lhzwdyyylvj42vyg6f-xdg-desktop-database.drv'
>> guix gc: error: executing SQLite statement: database disk image is malformed
>> --8<---------------cut here---------------end--------------->8---
>>
>> What does this mean?
>
> I think this means that the database has been corrupted.

I ran it a second time. It printed more messages, but curiously it

errored out after saying that it deleted another

xdg-desktop-database.drv:

Toggle snippet (9 lines)

$ guix gc -C 25GiB

...

deleting `/gnu/store/d5z2lazx11qz571mwy59gwlqxidj0h4r-gtk-im-modules.drv'

deleting `/gnu/store/gc36aax38fs194ll5s17j8a8l8arm09a-gtk-icon-themes.drv'

deleting `/gnu/store/hbxvgm2pc6f6jn2fai30izc7f5kd25qd-gtk-im-modules.drv'

deleting `/gnu/store/hrc7yacy4ckmp3zi5pgmzd5ciqzv40ih-xdg-desktop-database.drv'

guix gc: error: executing SQLite statement: database disk image is malformed

I ran it a third time, and it printed:

Toggle snippet (7 lines)

$ guix gc -C 25GiB

finding garbage collector roots...

deleting garbage...

deleting `/gnu/store/1dqflkm3029ncacx5vl9r95ldfcfs3m0-pugixml-1.9.drv'

guix gc: error: executing SQLite statement: database disk image is malformed

I ran it a fourth time, and it printed:

Toggle snippet (9 lines)

$ guix gc -C 25GiB

...

deleting `/gnu/store/w0nbsjjb1pj9pi1aim1qcbjq93zss9pn-libwpd-0.10.2.drv'

deleting `/gnu/store/ywdgn73h9kdfcvs6cr1nfjqqj245p788-librevenge-0.0.4.drv'

deleting `/gnu/store/ryawkapgxirj6yp54zb5vak027xxk2i1-cppunit-1.14.0.drv'

deleting `/gnu/store/g2p20iss1r58r4ch2qggzyqn9d5q4kaf-cppunit-1.14.0.tar.gz.drv'

guix gc: error: executing SQLite statement: database disk image is malformed

I wonder what has brought my installation into this state.  I can't
think of a way to fix it since I don't even know what caused it, so I
will probably re-install Guix to work around the issue, but before I do
that, is there anything we can check to understand why the database
corruption has occurred?

For the record, I do sometimes abruptly power off my machine, and it
does rarely abruptly crash (I think I have reported this when it occurs,
but I can't be sure).  In such cases, when booting, I see messages
telling me that the system is recovering the journal from my ext4 file
system, and I do not recall ever seeing an error from that process, but
who knows what mischief I've been causing to my system over the years.

-- 
Chris

Ricardo Wurmus wrote on 19 Jul 2019 10:04

Recipients:(name . Chris Marusich)(address . cmmarusich@gmail.com)(address . 36687@debbugs.gnu.org)

Message-ID:87blxqfo25.fsf@elephly.net

Chris Marusich <cmmarusich@gmail.com> writes:

Toggle quote (2 lines)

> is there anything we can check to understand why the database

> corruption has occurred?

[…]

Toggle quote (4 lines)

> For the record, I do sometimes abruptly power off my machine, and it

> does rarely abruptly crash (I think I have reported this when it occurs,

> but I can't be sure).

This can be a cause of corruption.

Have you manually run fsck on the unmounted disk yet?

Ricardo

Chris Marusich wrote on 3 Aug 2019 11:11

Recipients:(name . Ricardo Wurmus)(address . rekado@elephly.net)

Message-ID:87wofuipfp.fsf@gmail.com

Ricardo Wurmus <rekado@elephly.net> writes:

Toggle quote (13 lines)> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> is there anything we can check to understand why the database
>> corruption has occurred?
> […]
>> For the record, I do sometimes abruptly power off my machine, and it
>> does rarely abruptly crash (I think I have reported this when it occurs,
>> but I can't be sure).
>
> This can be a cause of corruption.
>
> Have you manually run fsck on the unmounted disk yet?

Yes, I have.  The file system containing the database is my root file
system, and it is ext4.  On my system, Guix runs e2fsck on the root file
system automatically on each boot [1].  However, just to be sure, I ran
it manually on the unmounted device with the -f option to force a check.
It reported the following (I've copied this output by hand):

Toggle snippet (22 lines)root@spaceship ~# e2fsck -f /dev/mapper/home
e2fsck 1.45.2 (27-May-2019)
Pass 1: Checking inodes, blocks, and sizes
Inode 290237 extent tree (at level 2) could be narrower.  Optimize<y>? yes
Inode 291644 extent tree (at level 2) could be narrower.  Optimize<y>? yes
Inode 446822 extent tree (at level 1) could be narrower.  Optimize<y>? yes
Inode 3016396 extent tree (at level 2) could be narrower.  Optimize<y>? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Problem in HTREE directory inode 12451844: block #21606 has bad min hash
Invalid HTREE directory inode 12451844 (/gnu/store/.links).  Clear HTree index<y>? yes
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information

root: ***** FILE SYSTEM WAS MODIFIED *****
root: 1761282/14000128 files (0.4% non-contiguous), 50020294/55985664 blocks
root@spaceship ~# echo $?
1

Unfortunately, when I rebooted and tried "guix gc", the same problem as

before occurred.

I saw that Mike Gerwitz encountered the same error, so I thought I would

try his work-around [2]. Before proceeding, I stopped guix-daemon

("sudo herd stop guix-daemon") and verified that no processes were

accessing the database file ("sudo fuser db.sqlite"). Then I made a

backup of my entire /var/guix directory.

I dumped the database to SQL and recreated it like so:

sudo sqlite3 db.sqlite .dump > /tmp/db.sqlite.dump

sudo rm db.sqlite

sudo sqlite3 db.sqlite < /tmp/db.sqlite.dump

This succeeded without errors or warnings, and it reduced the size of

the db.sqlite file from 145 MiB to 49 MiB. Although Mike's dump ended

with a "ROLLBACK" and needed to be manually modified, mine ended with a

"COMMIT" and did not require manual modification.

I then started guix-daemon ("sudo herd start guix-daemon") and attempted

to run "guix gc -C 1GiB", which succeeded. I followed that up with

another "guix gc -C 40GiB", which also succeeded.

The sqlite documentation says that this method of dumping and recreating

the database is supported [3], so I think it's a valid work-around.

However, you never know... Dumping and restoring the database file

obviously changed it, so even if the restored database is "equivalent"

to the original, it's still conceivable that the difference might cause

a problem somehow. After all, obviously it was different enough that

guix-daemon stopped failing. But maybe I'm being too pessimistic.

In any case, it's notable that guix-daemon claimed the database was

corrupt, even though sqlite3 didn't complain about it. Since the size

was reduced, I suppose sqlite3 cleaned it up or optimized it somehow.

Maybe in doing so, it fixed something that guix-daemon didn't know how

to handle? That's just a guess.

It's concerning that this "corruption" has occurred for two different

people recently. We might want to check the Nix bug reports to see if

they've encountered a bug like this. I did a (very) cursory search and

found only this possibly similar, but possibly unrelated issue:

https://github.com/NixOS/nixpkgs/issues/3958

It may be worth looking harder before diving too deeply into debugging.

If you can think of anything I can do to drill into the files or debug

this more, please let me know. If the database files do not contain

sensitive information, I would also be willing to share them with you

for further debugging.

Footnotes:

[1] https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/linux-boot.scm?id=afb986e77cd669c2f21953f501f7893237730ca7#n383

[2] https://lists.gnu.org/archive/html/help-guix/2019-08/msg00012.html

[3] https://sqlite.org/cli.html#dump

Chris

Mike Gerwitz wrote on 3 Aug 2019 15:06

Recipients:(name . Chris Marusich)(address . cmmarusich@gmail.com)

Message-ID:87pnlmtn2p.fsf@gnu.org

Sorry I hadn't seen this bug report; it was opened after I researched

this issue a couple months ago and I didn't bother checking since.

Here's the thread I posted to help-guix: <87lfwbuj27.fsf@gnu.org>.

Mike Gerwitz

Free Software Hacker+Activist | GNU Maintainer & Volunteer

GPG: D6E9 B930 028A 6C38 F43B 2388 FEF6 3574 5E6F 6D05

https://mikegerwitz.com

Chris Marusich wrote on 17 Nov 2019 00:30

Recipients:(name . Chris Marusich)(address . cmmarusich@gmail.com)

Message-ID:8736en762m.fsf@garuda.local.i-did-not-set--mail-host-address--so-tickle-me

Chris Marusich <cmmarusich@gmail.com> writes:

Toggle quote (4 lines)

> I wonder what has brought my installation into this state. I can't

> think of a way to fix it since I don't even know what caused it, so I

> will probably re-install Guix to work around the issue

I re-installed Guix to work around the issue.

Chris

pkill9 wrote on 30 Dec 2023 03:25

guix gc: error: executing SQLite statement: database disk image is malformed

Recipients:(address . 36687@debbugs.gnu.org)

Message-ID:13KG6S.9JVA2Q8FX3TN@runbox.com

Hi, I fixed this issue using sqlite3's built-in recover command:

sudo mv /var/guix/db/db.sqlite /var/guix/db/db.sqlite.bak

sqlite3 /var/guix/db/db.sqlite.bak ".recover" | sudo sqlite3

/var/guix/db/db.sqlite

Solution was found at

https://stackoverflow.com/questions/18259692/how-to-recover-a-corrupt-sqlite3-database/57872238#57872238

Attachment: file

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 36687@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 36687

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date