Guix time machine provenance/manifest reproducibility issue?

Open

Details

2 participants

Denis 'GNUtoo' Carikli
zimoun

Owner: unassigned

Submitted by: Denis 'GNUtoo' Carikli

Severity: normal

Denis 'GNUtoo' Carikli wrote on 1 Aug 2021 02:21

Recipients:(address . bug-guix@gnu.org)

Message-ID:20210801022142.2117e06e@primarylaptop.localdomain

Hi,

I've been trying to reproduce a tarball

(sz1lkq3ryr5iv6amy6f3d2pziks27g28-tarball-pack.tar.xz) that I generated

with guix pack on guix master the 28 January 2021.

To build it, in January, I used the following commands:

Toggle quote (9 lines)

> guix pull

> guix pack \

> --compression=xz \

> --save-provenance \

> -RR \

> --symlink=/usr/local/bin/repo=bin/repo \

> --symlink=/usr/local/bin/repo-env.sh=etc/profile \

> git-repo le-certs nss-certs git python-certifi

That tarball is publicly available in the Replicant ftp server[1].

The extracted provenance file (named manifest) has the following

content:

Toggle quote (124 lines)> ;; This file was automatically generated and is for internal use only.
> ;; It cannot be passed to the '--manifest' option.
> 
> (manifest
>   (version 3)
>   (packages
>     (("git-repo"
>       "2.4.1"
>       "out"
>       "/gnu/store/d4frkcdq15a7gyfjdggwg44ryi46fa2d-git-repo-2.4.1R"
>       (propagated-inputs ())
>       (search-paths ())
>       (properties
>         (provenance
>           (repository
>             (version 0)
>             (url "https://git.savannah.gnu.org/git/guix.git")
>             (branch "master")
>             (commit
>               "f9bd4621dd92a9415276706b476b9bd2973411fa")
>             (introduction
>               (channel-introduction
>                 (version 0)
>                 (commit
>                   "9edb3f66fd807b096b48283debdcddccfea34bad")
>                 (signer
>                   "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A
> 54FA"))))))) ("le-certs"
>       "0"
>       "out"
>       "/gnu/store/x004p4hnyy0ickg2f5msvrpszhy9hzpl-le-certs-0R"
>       (propagated-inputs ())
>       (search-paths ())
>       (properties
>         (provenance
>           (repository
>             (version 0)
>             (url "https://git.savannah.gnu.org/git/guix.git")
>             (branch "master")
>             (commit
>               "f9bd4621dd92a9415276706b476b9bd2973411fa")
>             (introduction
>               (channel-introduction
>                 (version 0)
>                 (commit
>                   "9edb3f66fd807b096b48283debdcddccfea34bad")
>                 (signer
>                   "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A
> 54FA"))))))) ("nss-certs"
>       "3.57"
>       "out"
>       "/gnu/store/shc8qpw1y2k7q668rx4gl6aff0wp1n6v-nss-certs-3.57R"
>       (propagated-inputs ())
>       (search-paths ())
>       (properties
>         (provenance
>           (repository
>             (version 0)
>             (url "https://git.savannah.gnu.org/git/guix.git")
>             (branch "master")
>             (commit
>               "f9bd4621dd92a9415276706b476b9bd2973411fa")
>             (introduction
>               (channel-introduction
>                 (version 0)
>                 (commit
>                   "9edb3f66fd807b096b48283debdcddccfea34bad")
>                 (signer
>                   "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A
> 54FA"))))))) ("git"
>       "2.30.0"
>       "out"
>       "/gnu/store/378nlw54nxy991jcilnnbrxasnfvv9wl-git-2.30.0R"
>       (propagated-inputs ())
>       (search-paths
>         (("GIT_SSL_CAINFO"
>           ("etc/ssl/certs/ca-certificates.crt")
>           #f
>           regular
>           #f)
>          ("GIT_EXEC_PATH"
>           ("libexec/git-core")
>           #f
>           directory
>           #f)))
>       (properties
>         (provenance
>           (repository
>             (version 0)
>             (url "https://git.savannah.gnu.org/git/guix.git")
>             (branch "master")
>             (commit
>               "f9bd4621dd92a9415276706b476b9bd2973411fa")
>             (introduction
>               (channel-introduction
>                 (version 0)
>                 (commit
>                   "9edb3f66fd807b096b48283debdcddccfea34bad")
>                 (signer
>                   "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A
> 54FA"))))))) ("python-certifi"
>       "2020.11.8"
>       "out"
>       "/gnu/store/hmp6ab9kw1z3hjns9h1fm3afsq4g6j7x-python-certifi-2020.11.8R"
>       (propagated-inputs ())
>       (search-paths ())
>       (properties
>         (provenance
>           (repository
>             (version 0)
>             (url "https://git.savannah.gnu.org/git/guix.git")
>             (branch "master")
>             (commit
>               "f9bd4621dd92a9415276706b476b9bd2973411fa")
>             (introduction
>               (channel-introduction
>                 (version 0)
>                 (commit
>                   "9edb3f66fd807b096b48283debdcddccfea34bad")
>                 (signer
>                   "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A
> 54FA"))))))))))

So I tried to reproduce it with the following command:

Toggle quote (10 lines)> guix time-machine \
> 	--commit=f9bd4621dd92a9415276706b476b9bd2973411fa -- \
> 	pack \
> 		--compression=xz \
> 		--save-provenance \
> 		-RR \
> 		--symlink=/usr/local/bin/repo=bin/repo \
> 		--symlink=/usr/local/bin/repo-env.sh=etc/profile \
>  git-repo le-certs nss-certs git python-certifi

But the new tarball filename was different.

vivien in #guix helped me a lot by trying to build that tarball too and

me and viven have the same filename with guix-time-machine:

bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack.tar.xz

We then managed to get to the root cause of the difference.

All the binaries were the sames. All the differences comes from the

fact that the provenance file (named 'manifest') is different.

That difference then produces a different profile name and also affects

/usr/bin as that references the profile.

Diffing the two provenance files gives that:

Toggle quote (61 lines)> +++
> bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack/gnu/store/216jiimdyw7zyx8s9b3fz67aw69ydkvw-profile/manifest
> 1970-01-01 01:00:01.000000000 +0100 @@ -15,9 +15,10 @@ (repository
>              (version 0)
>              (url "https://git.savannah.gnu.org/git/guix.git")
> -            (branch "master")
> +            (branch #f)
>              (commit
>                "f9bd4621dd92a9415276706b476b9bd2973411fa")
> +            (name guix)
>              (introduction
>                (channel-introduction
>                  (version 0)
> @@ -36,9 +37,10 @@
>            (repository
>              (version 0)
>              (url "https://git.savannah.gnu.org/git/guix.git")
> -            (branch "master")
> +            (branch #f)
>              (commit
>                "f9bd4621dd92a9415276706b476b9bd2973411fa")
> +            (name guix)
>              (introduction
>                (channel-introduction
>                  (version 0)
> @@ -57,9 +59,10 @@
>            (repository
>              (version 0)
>              (url "https://git.savannah.gnu.org/git/guix.git")
> -            (branch "master")
> +            (branch #f)
>              (commit
>                "f9bd4621dd92a9415276706b476b9bd2973411fa")
> +            (name guix)
>              (introduction
>                (channel-introduction
>                  (version 0)
> @@ -88,9 +91,10 @@
>            (repository
>              (version 0)
>              (url "https://git.savannah.gnu.org/git/guix.git")
> -            (branch "master")
> +            (branch #f)
>              (commit
>                "f9bd4621dd92a9415276706b476b9bd2973411fa")
> +            (name guix)
>              (introduction
>                (channel-introduction
>                  (version 0)
> @@ -109,9 +113,10 @@
>            (repository
>              (version 0)
>              (url "https://git.savannah.gnu.org/git/guix.git")
> -            (branch "master")
> +            (branch #f)
>              (commit
>                "f9bd4621dd92a9415276706b476b9bd2973411fa")
> +            (name guix)
>              (introduction
>                (channel-introduction

I've tried to add --branch=master to guix time-machine and used guix

gc -D to remove the older tarball as it didn't rebuild it even with

--rounds=2, and at the end I still got the exact same

bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack.tar.xz tarball (I've

compared both with cmp).

Am I doing something wrong, or is there an issue that needs to be fixed

somehow?

References:

-----------

[1]https://ftp.osuosl.org/pub/replicant/build-tools/repo/28-01-2021/

Denis.

zimoun wrote on 17 Aug 2021 14:11

Recipients:

Message-ID:86k0kkclu3.fsf@gmail.com

Hi,

Thanks for the report.

On Sun, 01 Aug 2021 at 02:21, Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> wrote:

Toggle quote (15 lines)> Diffing the two provenance files gives that:
>> +++
>> bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack/gnu/store/216jiimdyw7zyx8s9b3fz67aw69ydkvw-profile/manifest
>> 1970-01-01 01:00:01.000000000 +0100 @@ -15,9 +15,10 @@ (repository
>>              (version 0)
>>              (url "https://git.savannah.gnu.org/git/guix.git")
>> -            (branch "master")
>> +            (branch #f)
>>              (commit
>>                "f9bd4621dd92a9415276706b476b9bd2973411fa")
>> +            (name guix)
>>              (introduction
>>                (channel-introduction
>>                  (version 0)

Well, I think it comes from ’channel-list’ in the ’time-machine’.

Specifically, it reads in guix/scripts/pull.scm:

Toggle snippet (4 lines)

(channel (inherit guix)

(url url) (commit commit) (branch #f)))

other said, the name of the branch is “lost”. Hum, I do not know if

this is done on purpose or not. Maybe this change

Toggle snippet (12 lines)             (cons (match ref
                     (('commit . commit)
                      (channel (inherit guix)
                               (url url) (commit commit))
                     (('branch . branch)
                      (channel (inherit guix)
                               (url url) (commit #f) (branch branch)))
                     (#f
                      (channel (inherit guix) (url url))))
                   (remove guix-channel? channels))

is enough. But, I do not know what would happens for:

guix pull --commit=<hash>

where <hash> is not a commit from the branch master.

All the best,

simon

Denis 'GNUtoo' Carikli wrote on 2 Sep 2021 00:27

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 49801@debbugs.gnu.org)

Message-ID:20210902002742.3866243a@primarylaptop.localdomain

Hi again.

With and without this patch:

Toggle quote (16 lines)> diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
> index fb8ce50fa7..af1cf77f07 100644
> --- a/guix/scripts/pull.scm
> +++ b/guix/scripts/pull.scm
> @@ -739,7 +739,7 @@ Use '~/.config/guix/channels.scm' instead."))
>               (cons (match ref
>                       (('commit . commit)
>                        (channel (inherit guix)
> -                               (url url) (commit commit) (branch
> #f)))
> +                               (url url) (commit commit)))
>                       (('branch . branch)
>                        (channel (inherit guix)
>                                 (url url) (commit #f) (branch
> branch)))

on top of 95c29d2746943733cbe8df7013854d45bb0df413 ("gnu: electron-cash:

Update to 4.2.5." which is today's master HEAD), I get the same diff

with and without time-machine.

I made and used this Makefile to build two hello tarball in both cases:

Toggle quote (22 lines)> COMMIT ?= 95c29d2746943733cbe8df7013854d45bb0df413
> 
> all: \
> 	hello-guix-$(COMMIT).tar.xz \
> 	hello-time-machine-$(COMMIT).tar.xz \
> 
> hello-guix-$(COMMIT).tar.xz:
> 	install -m 644 \
> 		`../pre-inst-env \
> 		guix pack \
> 		--compression=xz --save-provenance hello` \
> 	$@
> 
> hello-time-machine-$(COMMIT).tar.xz:
> 	install -m 644 \
> 		`../pre-inst-env guix time-machine \
> 			--branch=master \
> 			--commit=$(COMMIT) \
> 			-- \
> 			pack --compression=xz --save-provenance hello` \
> 	$@

And once the file named manifest is extracted from both tarballs I get

this diff (with and without your slightly modified patch):

Toggle quote (24 lines)> --- ./hello-guix-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile/manifest	1970-01-01 01:00:01.000000000 +0100
> +++ ./hello-time-machine-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/30pf6ppiqpjsjaaiw35kc5lp6dcixpf1-profile/manifest	1970-01-01 01:00:01.000000000 +0100
> @@ -12,4 +12,19 @@
>        "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10"
>        (propagated-inputs ())
>        (search-paths ())
> -      (properties)))))
> +      (properties
> +        (provenance
> +          (repository
> +            (version 0)
> +            (url "https://git.savannah.gnu.org/git/guix.git")
> +            (branch #f)
> +            (commit
> +              "95c29d2746943733cbe8df7013854d45bb0df413")
> +            (name guix)
> +            (introduction
> +              (channel-introduction
> +                (version 0)
> +                (commit
> +                  "9edb3f66fd807b096b48283debdcddccfea34bad")
> +                (signer
> +                  "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))))))))

PS: In the diff at the top there is a slight difference with the patch

that you suggested: I only removed (branch #f) so I end up with one

more parenthesis at the end.

Denis.

zimoun wrote on 2 Sep 2021 10:10

Recipients:(name . Denis 'GNUtoo' Carikli)(address . GNUtoo@cyberdimension.org)(address . 49801@debbugs.gnu.org)

Message-ID:CAJ3okZ0cU72g479HwbCTcTb1hUURD_W+2aV3z+XTTEf=1KwztQ@mail.gmail.com

Hi Denis,

Thanks for the investigation and the attempt.

Well, I miss if it works or not...

On Thu, 2 Sept 2021 at 00:27, Denis 'GNUtoo' Carikli

<GNUtoo@cyberdimension.org> wrote:

Toggle quote (21 lines)> With and without this patch:
> > diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
> > index fb8ce50fa7..af1cf77f07 100644
> > --- a/guix/scripts/pull.scm
> > +++ b/guix/scripts/pull.scm
> > @@ -739,7 +739,7 @@ Use '~/.config/guix/channels.scm' instead."))
> >               (cons (match ref
> >                       (('commit . commit)
> >                        (channel (inherit guix)
> > -                               (url url) (commit commit) (branch
> > #f)))
> > +                               (url url) (commit commit)))
> >                       (('branch . branch)
> >                        (channel (inherit guix)
> >                                 (url url) (commit #f) (branch
> > branch)))
>
> on top of 95c29d2746943733cbe8df7013854d45bb0df413 ("gnu: electron-cash:
> Update to 4.2.5." which is today's master HEAD), I get the same diff
> with and without time-machine.

...here I understand the patch fixes the issue...

Toggle quote (49 lines)> I made and used this Makefile to build two hello tarball in both cases:
> > COMMIT ?= 95c29d2746943733cbe8df7013854d45bb0df413
> >
> > all: \
> >       hello-guix-$(COMMIT).tar.xz \
> >       hello-time-machine-$(COMMIT).tar.xz \
> >
> > hello-guix-$(COMMIT).tar.xz:
> >       install -m 644 \
> >               `../pre-inst-env \
> >               guix pack \
> >               --compression=xz --save-provenance hello` \
> >       $@
> >
> > hello-time-machine-$(COMMIT).tar.xz:
> >       install -m 644 \
> >               `../pre-inst-env guix time-machine \
> >                       --branch=master \
> >                       --commit=$(COMMIT) \
> >                       -- \
> >                       pack --compression=xz --save-provenance hello` \
> >       $@
>
> And once the file named manifest is extracted from both tarballs I get
> this diff (with and without your slightly modified patch):
> > --- ./hello-guix-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> > +++ ./hello-time-machine-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/30pf6ppiqpjsjaaiw35kc5lp6dcixpf1-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> > @@ -12,4 +12,19 @@
> >        "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10"
> >        (propagated-inputs ())
> >        (search-paths ())
> > -      (properties)))))
> > +      (properties
> > +        (provenance
> > +          (repository
> > +            (version 0)
> > +            (url "https://git.savannah.gnu.org/git/guix.git")
> > +            (branch #f)
> > +            (commit
> > +              "95c29d2746943733cbe8df7013854d45bb0df413")
> > +            (name guix)
> > +            (introduction
> > +              (channel-introduction
> > +                (version 0)
> > +                (commit
> > +                  "9edb3f66fd807b096b48283debdcddccfea34bad")
> > +                (signer
> > +                  "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))))))))

...but then here I see it does not fix it.

However, because you run "./pre-inst-env guix pack --save-provenance",

it seems expected that the 'properties' is empty. From my

understanding, '(find guix-channels? channels)' does not return the

'guix' channel because it is the current Git checkout. It is not the

case with "guix time-machine" because it creates an inferior using the

'guix' channel.

Moreover, if you want to try the patch, you need to run:

./pre-inst-env guix pull -p /tmp/new

./tmp/new/bin/guix describe # return commit 12345

./tmp/new/bin/guix pack --save-provenance

./tmp/new/bin/guix time-machine --commit=12345 -- pack --save-provenance

and be careful with the '--localstatedir' and '--sysconfdir' variables

at './configure' time.

Well, from my point of view, the Guix way would be:

guix describe -f channels > channels.scm

guix pack --save-provenance

then later or elsewehere

guix time-machine -C channels.scm -- pack --save-provenance

Although, it will not fix the bug you are exposing. :-)

WDYT?

Last, I have not carefully checked and maybe I am wrong, the both

options "--commit=1234 --branch=master" are exclusive I guess; i.e.,

the argument 'master' passed to '--branch' is not used in this case,

IIUC.

Cheers,

simon

Denis 'GNUtoo' Carikli wrote on 2 Sep 2021 16:12

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 49801@debbugs.gnu.org)

Message-ID:20210902161219.3dab2f9b@primarylaptop.localdomain

On Thu, 2 Sep 2021 10:10:22 +0200

zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (5 lines)> Hi Denis,
> 
> Thanks for the investigation and the attempt.
> 
> Well, I miss if it works or not...

It doesn't work.

The issue was that if you build a tarball with guix pack, without guix

time-machine, you can't reproduce it with guix-time-machine.

Between the two tarballs, everything is the same but the provenance

file.

So here the idea is to make sure that the provenance file is the same

between tarballs made with and without guix time-machine.

Here I get a diff between tarballs made with and without guix

time-machine, with or without your patch, so the patch doesn't fix it

yet for guix master of yesterday.

Between when I reported the bug and the test I did yesterday, the HEAD

of guix master changed though.

Denis.

zimoun wrote on 2 Sep 2021 21:30

Recipients:(name . Denis 'GNUtoo' Carikli)(address . GNUtoo@cyberdimension.org)(address . 49801@debbugs.gnu.org)

Message-ID:864kb23hcp.fsf@gmail.com

Hi Denis,

On Thu, 02 Sep 2021 at 16:12, Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> wrote:

Toggle quote (6 lines)

> Between the two tarballs, everything is the same but the provenance

> file.

> So here the idea is to make sure that the provenance file is the same

> between tarballs made with and without guix time-machine.

Yes, for sure. IMHO, the Guix way would be:

guix describe -f channels > channels.scm

guix pack --save-provenance

then later or elsewehere

guix time-machine -C channels.scm -- pack --save-provenance

It is a workaround of the bug your reported. ;-)

Does it work?

Toggle quote (4 lines)

> Here I get a diff between tarballs made with and without guix

> time-machine, with or without your patch, so the patch doesn't fix it

> yet for guix master of yesterday.

I think your tests about the patch are not correct. As I wrote, this is

what you should try, IIUC:

Toggle snippet (12 lines)

Moreover, if you want to try the patch, you need to run:

./pre-inst-env guix pull -p /tmp/new

./tmp/new/bin/guix describe # return commit 12345

./tmp/new/bin/guix pack --save-provenance

./tmp/new/bin/guix time-machine --commit=12345 -- pack --save-provenance

and be careful with the '--localstatedir' and '--sysconfdir' variables

at './configure' time.

All the best,

simon

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 49801@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date