A "live" profile is missing

My root user's latest default package installation profile has gone

missing:

------

# ls -l ~root/.guix-profile

lrwxrwxrwx 1 root root 45 Jun 13 2017 /root/.guix-profile -> /var/guix/profiles/per-user/root/guix-profile

# realpath ~root/.guix-profile

/gnu/store/1ph7vxq4hhqr1hjkmgin16p1cqx3v904-profile

# realpath --canonicalize-existing ~root/.guix-profile

realpath: /root/.guix-profile: No such file or directory

# ls -l /var/guix/profiles/per-user/root/

total 0

lrwxrwxrwx 1 root root 21 Jan 18 13:24 current-guix -> current-guix-113-link

lrwxrwxrwx 1 root root 51 Dec 6 18:50 current-guix-109-link -> /gnu/store/1bng1iikqj2h9hfqylcd47ya1cmjf2ww-profile

lrwxrwxrwx 1 root root 51 Dec 8 20:05 current-guix-110-link -> /gnu/store/kpp7rq67df1751g6al5fxv7anm4h6fvv-profile

lrwxrwxrwx 1 root root 51 Dec 20 17:45 current-guix-111-link -> /gnu/store/yg9r3vys1cx3xwsqmbl2kq20ih4xbryi-profile

lrwxrwxrwx 1 root root 51 Jan 13 03:27 current-guix-112-link -> /gnu/store/4aj0ydxfrbfz2yshlv1qg6kr5qw17mjz-profile

lrwxrwxrwx 1 root root 51 Jan 18 13:24 current-guix-113-link -> /gnu/store/5kfv0zzf1mmkyl3c9q42piwcb0h3fyzj-profile

-rw-r--r-- 1 root root 0 Jan 19 13:44 current-guix.lock

lrwxrwxrwx 1 root root 21 Jan 18 13:25 guix-profile -> guix-profile-197-link

lrwxrwxrwx 1 root root 51 Dec 20 17:47 guix-profile-192-link -> /gnu/store/hnmdaga6m5d1jvq1yjs0w1w10r8ffsha-profile

lrwxrwxrwx 1 root root 51 Dec 22 21:32 guix-profile-193-link -> /gnu/store/wfz6qbzd6imqg7cg0w5f8arnpzc5a8ck-profile

lrwxrwxrwx 1 root root 51 Jan 1 01:44 guix-profile-194-link -> /gnu/store/daixf0dvmxlavplc1v987sfqmkjv258j-profile

lrwxrwxrwx 1 root root 51 Jan 9 13:32 guix-profile-195-link -> /gnu/store/w14sbx2ja9g178lxbl8173c3xbbcapf9-profile

lrwxrwxrwx 1 root root 51 Jan 13 03:29 guix-profile-196-link -> /gnu/store/99qhzhhrm73mawbzcgp9kfcxhzs51kk3-profile

lrwxrwxrwx 1 root root 51 Jan 18 13:25 guix-profile-197-link -> /gnu/store/1ph7vxq4hhqr1hjkmgin16p1cqx3v904-profile

-rw-r--r-- 1 root root 0 Jan 18 13:25 guix-profile.lock

# ls -l /gnu/store/1ph7vxq4hhqr1hjkmgin16p1cqx3v904-profile

ls: cannot access '/gnu/store/1ph7vxq4hhqr1hjkmgin16p1cqx3v904-profile': No such file or directory

------

That's weird, right? Any ideas?

I noticed a change in the garbage collector performance recently. It

seemed to slow down by a lot, with frequent pauses. I wonder if that

could be related?

Here are the generations of my current-guix profile (this is not the

profile that is broken):

------

# guix package -p ~/.config/guix/current -l

Generation 109 Dec 06 2020 18:50:21

guix 8f80565 out /gnu/store/ikwp5p0nfhlmp2xvxyh0l5x7349ws24k-guix-8f805657b

Generation 110 Dec 08 2020 20:05:00

+ guix 02b21f3 out /gnu/store/b30hyzr2ln51dc7sdrwgzqklmkl71sni-guix-02b21f35c

- guix 8f80565 out /gnu/store/ikwp5p0nfhlmp2xvxyh0l5x7349ws24k-guix-8f805657b

Generation 111 Dec 20 2020 17:45:49

+ guix cfee860 out /gnu/store/xvswkzhadlsdlkxj2yikfq60vakvmpi8-guix-cfee8608c

- guix 02b21f3 out /gnu/store/b30hyzr2ln51dc7sdrwgzqklmkl71sni-guix-02b21f35c

Generation 112 Jan 13 2021 03:27:33

+ guix 645a28e out /gnu/store/08rk3q8s9qqzph0hah66mx1ws2ssll15-guix-645a28ee9

- guix cfee860 out /gnu/store/xvswkzhadlsdlkxj2yikfq60vakvmpi8-guix-cfee8608c

Generation 113 Jan 18 2021 13:24:01 (current)

+ guix c20eee7 out /gnu/store/zpmabbkh27myw01gjfczbrlqjr0ih5pd-guix-c20eee77a

- guix 645a28e out /gnu/store/08rk3q8s9qqzph0hah66mx1ws2ssll15-guix-645a28ee9

------

On Tue, Jan 19, 2021 at 01:56:13PM -0500, Leo Famulari wrote:

I checked, and this store item is not in the Guix database.

I just talked with Leo on IRC and he thinks the following might be

related.

`guix` stopped working, now taking any arguments without effect. I

found out that the associated store item is empty!

$which guix

/home/thorwil/.config/guix/current/bin/guix

$ls -l /home/thorwil/.config/guix/current/bin/guix

... /gnu/store/xa1xx4gpnvvf4wpzx63v1swl7gvqyw5d-guix-command

$file /gnu/store/xa1xx4gpnvvf4wpzx63v1swl7gvqyw5d-guix-command

... empty

This is on just about 1 week old hardware, Ubuntu Unity 20.10, Kernel

5.8.0-38, no power failures, ext4 file system. No direct manipulation,

just using `guix pull`, updating, installing and removing packages.

Meanwhile, `sudo guix pull` still worked.

As suggested by Leo, I did:

$cd /var/guix/profiles/per-user/thorwil

$rm current-guix

$ln -s current-guix-4-link current-guix

(Trying to `ln -s current-guix-4-link current-guix` without moving

current-guix out of the way first, failed with:

"ln: failed to create symbolic link 'current-guix/current-guix-4-link':

Read-only file system".)

--

Thorsten Wilms <t_w_@freenet.de>

Hi Leo,

Leo Famulari <leo@famulari.name> skribis:

Could it be that you (1) configured a checkout with the wrong

--localstatedir, (2) started guix-daemon from that checkout, and (3) ran

‘guix gc’ or equivalent?

The ‘configure’ script normally prevents #1, but we never know.

Thanks,

Ludo’.

On Thu, Jan 21, 2021 at 01:57:46PM +0100, Ludovic Courtès wrote:

1) I could have misconfigured a checkout, although it is unlikely. I

don't see any incorrect './configure' invocations in my shell history.

2) I never use the guix-daemon from a Git checkout. Really! I don't even

know off-hand where the built guix-daemon is located. I would have to

find it.

3) I run `guix gc` several times per week.

Hi Leo,

Leo Famulari <leo@famulari.name> skribis:

What does ‘grep ^localstatedir config.log’ return?

OK.

Alright.

So this is really weird. AFAICS nothing has changed in this area. You

could try building a new profile generation and then run things like

‘guix gc --list-live’ to see if the profile (in /gnu/store) is there,

and also ‘guix gc --list-roots’ to see if the profile itself (normally

/var/guix/profiles/per-user/$USER/guix-profile) is listed.

HTH,

Ludo’.

On Mon, Jan 25, 2021 at 07:11:33PM +0100, Ludovic Courtès wrote:

For my "master" and "staging" worktrees, it returns:

localstatedir='/var'

However, I've since done a full garbage collection and reconfigured, so

it doesn't really prove anything.

--list-live actually starts by printing this:

------

$ guix gc --list-live

finding garbage collector roots...

skipping invalid root from `/var/guix/profiles/per-user/root/guix-profile-197-link' to `/gnu/store/1ph7vxq4hhqr1hjkmgin16p1cqx3v904-profile'

determining live/dead paths...

[... and then the paths are printed ...]

------

And --list-roots does include the bogus profile, exactly as described in

the warning from --list-live:

------

# guix gc --list-roots

/var/guix/profiles/per-user/root/current-guix-113-link

/var/guix/profiles/per-user/root/current-guix-109-link

/var/guix/profiles/per-user/root/current-guix-110-link

/var/guix/profiles/per-user/root/current-guix-111-link

/var/guix/profiles/per-user/root/current-guix-112-link

/var/guix/profiles/per-user/root/current-guix-113-link

/var/guix/profiles/per-user/root/guix-profile-196-link

/var/guix/profiles/per-user/root/guix-profile-192-link

/var/guix/profiles/per-user/root/guix-profile-193-link

/var/guix/profiles/per-user/root/guix-profile-194-link

/var/guix/profiles/per-user/root/guix-profile-195-link

/var/guix/profiles/per-user/root/guix-profile-196-link

/var/guix/profiles/per-user/root/guix-profile-197-link

/var/guix/profiles/per-user/leo/current-guix-152-link

/var/guix/profiles/per-user/leo/current-guix-147-link

/var/guix/profiles/per-user/leo/current-guix-148-link

/var/guix/profiles/per-user/leo/current-guix-149-link

/var/guix/profiles/per-user/leo/current-guix-150-link

/var/guix/profiles/per-user/leo/current-guix-151-link

/var/guix/profiles/per-user/leo/current-guix-152-link

/var/guix/profiles/per-user/leo/guix-profile-642-link

/var/guix/profiles/per-user/leo/guix-profile-631-link

/var/guix/profiles/per-user/leo/guix-profile-632-link

/var/guix/profiles/per-user/leo/guix-profile-633-link

/var/guix/profiles/per-user/leo/guix-profile-634-link

/var/guix/profiles/per-user/leo/guix-profile-635-link

/var/guix/profiles/per-user/leo/guix-profile-636-link

/var/guix/profiles/per-user/leo/guix-profile-637-link

/var/guix/profiles/per-user/leo/guix-profile-638-link

/var/guix/profiles/per-user/leo/guix-profile-639-link

/var/guix/profiles/per-user/leo/guix-profile-640-link

/var/guix/profiles/per-user/leo/guix-profile-641-link

/var/guix/profiles/per-user/leo/guix-profile-642-link

/var/guix/profiles/per-user/leo/current-guix-147-link

/home/leo/.cache/guix/inferiors/xme6psb4eddn2fvq262cxw6yxepo7reqzpkqf7r24hpemqkqddoq

/var/guix/profiles/per-user/leo/current-guix-152-link

/root/guix-profile-staging-1-link

/var/guix/profiles/per-user/root/current-guix-112-link

/var/guix/profiles/per-user/root/current-guix-109-link

/home/leo/.cache/guix/inferiors/erkfkimh2njpo5acqlz4ep5o6iihv3bgmfxpahltoityb3h5sguq

/var/guix/profiles/per-user/root/current-guix-111-link

/var/guix/profiles/per-user/leo/current-guix-150-link

/var/guix/profiles/per-user/root/current-guix-110-link

/var/guix/profiles/per-user/root/current-guix-113-link

/var/guix/profiles/per-user/leo/current-guix-148-link

/home/leo/.cache/guix/inferiors/jtzxrv5w2ppetxbu5sfxa45y36fjvgygksqu3uu7o36kizufmhta

/root/.config/guix/latest

/home/leo/.cache/guix/inferiors/xrlvjdtbkf6yqqcgwo34arbfpnb4lszopf7bcj4mq3etpl6dxbra

/var/guix/profiles/per-user/leo/current-guix-149-link

/var/guix/profiles/per-user/leo/current-guix-151-link

/var/guix/profiles

------

Regarding the suspicious "/root/guix-profile-staging-1-link", it's

leftover from testing the staging branch. I created it with `guix

pull --branch=staging --profile=/root/guix-profile-staging`.

And, it's time to delete /root/.config/guix/latest :)

Leo Famulari <leo@famulari.name> skribis:

OK.

That means the target of this symlink was already invalid/non-existent,

right?

OK.

If there’s a root pointing to the profile, and the profile no longer

exists, it could be that the root was created after the profile had been

deleted. AFAIK that cannot happen in normal operation, so you’d really

have to fiddle with /var/guix/profiles to achieve that.

So… I don’t see how this could happen!

Ludo’.

Leo Famulari <leo@famulari.name> skribis:

OK.

That means the target of this symlink was already invalid/non-existent,

right?

OK.

If there’s a root pointing to the profile, and the profile no longer

exists, it could be that the root was created after the profile had been

deleted. AFAIK that cannot happen in normal operation, so you’d really

have to fiddle with /var/guix/profiles to achieve that.

So… I don’t see how this could happen!

Ludo’.

On Mon, Jan 25, 2021 at 09:55:09PM +0100, Ludovic Courtès wrote:

Right, that store item did not exist.

Mysterious...

tags 45992 + moreinfo

quit

I don't think that this message is really relevant to the bug report. It

describes some corrupted store items on my computer and might be

interesting as examples of what goes wrong when /gnu/store is writable.

This is on Debian, where /gnu/store is just a "regular directory",

without the protection you get on Guix System.

I noticed that three store items have been altered somehow:

------

$ guix gc --verify=contents

reading the store...

checking path existence...

checking hashes...

path `/gnu/store/4aj0ydxfrbfz2yshlv1qg6kr5qw17mjz-profile' was modified! expected hash `2f0a69784b287ba297082209b928e46b2f8f4925cab00996cd23699a468cc8d1', got `bb3067ed7fa0e11b7f57ad9d4d8428ce9312a1b7d0bd638d7ce16e447e452dee'

path `/gnu/store/5kfv0zzf1mmkyl3c9q42piwcb0h3fyzj-profile' was modified! expected hash `9f8281ec9f8e5bc173739b36e8ffc5e088667f17e57e496f51a22f59340d4643', got `b4e187418d96c984f543f50a0fc3709d90056423c66dc09e7548aa06dcdb8e9a'

path `/gnu/store/7y3lvk3xf4im8n44337mc6y0ccysvfia-font-dejavu-2.37' was modified! expected hash `5f4f3e0d3eb79e67fc38654a1277709a95ec1ca85253d4c40cb2f9e5514cafa7', got `e726b2163b08b3bd7edb95f29d8e29c0c70ae356d7a4d898ade486b71b0f80ed'

------

None of these are the "missing" profile.

Both of the profiles have some suspicious links like these:

------

ls -l /gnu/store/4aj0ydxfrbfz2yshlv1qg6kr5qw17mjz-profile

total 20

lrwxrwxrwx 3 root root 62 Dec 31 1969 bin -> /gnu/store/08rk3q8s9qqzph0hah66mx1ws2ssll15-guix-645a28ee9/bin

lrwxrwxrwx 1 root root 12 Jan 20 13:27 current-guix -> current-guix

lrwxrwxrwx 1 root root 21 Jan 20 13:28 current-guix-112-link -> current-guix-112-link

lrwxrwxrwx 1 root root 21 Jan 20 13:28 current-guix-113-link -> current-guix-113-link

dr-xr-xr-x 2 root root 4096 Dec 31 1969 etc

dr-xr-xr-x 2 root root 4096 Dec 31 1969 lib

-r--r--r-- 2 root root 851 Dec 31 1969 manifest

dr-xr-xr-x 3 root root 4096 Dec 31 1969 share

------

The timestamps lead me to think I messed these up while trying to

workaround the missing profile that spurred this bug report.

In the font-dejavu store item, I noticed that not all the files have

their timestamps set correctly:

------

/gnu/store/7y3lvk3xf4im8n44337mc6y0ccysvfia-font-dejavu-2.37

└── [Dec 31 1969] share

├── [Dec 31 1969] doc

│   └── [Dec 31 1969] font-dejavu-2.37

│   └── [Dec 31 1969] LICENSE

├── [Dec 31 1969] fontconfig

│   └── [Dec 31 1969] conf.avail

│   ├── [Dec 31 1969] 20-unhint-small-dejavu-sans.conf

│   ├── [Dec 31 1969] 20-unhint-small-dejavu-sans-mono.conf

│   ├── [Dec 31 1969] 20-unhint-small-dejavu-serif.conf

│   ├── [Dec 31 1969] 57-dejavu-sans.conf

│   ├── [Dec 31 1969] 57-dejavu-sans-mono.conf

│   └── [Dec 31 1969] 57-dejavu-serif.conf

└── [Dec 31 1969] fonts

├── [Dec 31 1969] truetype

│   ├── [Dec 31 1969] DejaVuMathTeXGyre.ttf

│   ├── [Dec 31 1969] DejaVuSans-BoldOblique.ttf

│   ├── [Dec 31 1969] DejaVuSans-Bold.ttf

│   ├── [Dec 31 1969] DejaVuSansCondensed-BoldOblique.ttf

│   ├── [Dec 31 1969] DejaVuSansCondensed-Bold.ttf

│   ├── [Dec 31 1969] DejaVuSansCondensed-Oblique.ttf

│   ├── [Dec 31 1969] DejaVuSansCondensed.ttf

│   ├── [Dec 31 1969] DejaVuSans-ExtraLight.ttf

│   ├── [Dec 31 1969] DejaVuSansMono-BoldOblique.ttf

│   ├── [Dec 31 1969] DejaVuSansMono-Bold.ttf

│   ├── [Dec 31 1969] DejaVuSansMono-Oblique.ttf

│   ├── [Dec 31 1969] DejaVuSansMono.ttf

│   ├── [Dec 31 1969] DejaVuSans-Oblique.ttf

│   ├── [Dec 31 1969] DejaVuSans.ttf

│   ├── [Dec 31 1969] DejaVuSerif-BoldItalic.ttf

│   ├── [Dec 31 1969] DejaVuSerif-Bold.ttf

│   ├── [Dec 31 1969] DejaVuSerifCondensed-BoldItalic.ttf

│   ├── [Dec 31 1969] DejaVuSerifCondensed-Bold.ttf

│   ├── [Dec 31 1969] DejaVuSerifCondensed-Italic.ttf

│   ├── [Dec 31 1969] DejaVuSerifCondensed.ttf

│   ├── [Dec 31 1969] DejaVuSerif-Italic.ttf

│   ├── [Dec 31 1969] DejaVuSerif.ttf

│   └── [Jun 17 2020] .uuid

└── [Jun 17 2020] .uuid

------

What's up with those '.uuid' files?

Hi Leo,

Leo Famulari <leo@famulari.name> skribis:

We now provide a ‘gnu-store.mount’ systemd service that makes /gnu/store

read-only, as on Guix System. I suppose it wasn’t enabled on this

machine, right?

[...]

Not 100% sure, but my theory is that it’s a cache created by Fontconfig

and its friends. That can happen when running a Guix application as

root.

It’s in fact a relatively frequent problem and the reason why Guix and

before that NixOS made the store read-only.

HTH!

Ludo’.

On Sun, Jan 31, 2021 at 05:40:50PM +0100, Ludovic Courtès wrote:

Thanks for pointing that out. This installation of Guix is very old. I'm

going to make it use that gnu-store.mount service now!

Okay!