[PATCH] Fix determinism issue in guix package

  • Done
  • quality assurance status badge
Details
5 participants
  • Hilton Chain
  • Jakob Kirsch
  • Ludovic Courtès
  • Maxim Cournoyer
  • Simon Tournier
Owner
unassigned
Submitted by
Jakob Kirsch
Severity
important
J
J
Jakob Kirsch wrote on 30 Oct 19:06 +0100
(address . guix-patches@gnu.org)
ZyJ1wSX6cPIdnLpN@kernelpanicroom
I've recently noticed that `guix challenge guix` returns different hashes for each substitute server and also every build with `guix build --no-grafts guix` fails.
Running `guix build --no-grafts guix --cores=1` seems to reliably produce the same hash though. As someone pointed out on XMPP, Guile seems to have issues with parallel builds so this patch disables them for the guix package.

I think this has high importance because the main guix package should be reproducible in order to trust the whole chain of packages.
From 0f3df56dd0c430c09ba2839c9e2d5b32948201ae Mon Sep 17 00:00:00 2001
Message-ID: <0f3df56dd0c430c09ba2839c9e2d5b32948201ae.1730311414.git.jakob.kirsch@web.de>
From: Jakob Kirsch <jakob.kirsch@web.de>
Date: Wed, 30 Oct 2024 19:02:15 +0100
Subject: [PATCH v1] gnu: guix: Fix determinism issue

* gnu/packages/package-management.scm (guix): Fix determinism issue by disabling parallel build.

Change-Id: Ie28e16ed1f15cbc0da0c0d70b2c461e2baa3ff0a
---
gnu/packages/package-management.scm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 1763d2d59f..478a74385b 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -199,7 +199,8 @@ (define-public guix
(file-name (string-append "guix-" version "-checkout"))))
(build-system gnu-build-system)
(arguments
- `(#:configure-flags (list
+ `(#:parallel-build? #false ; for reproducibility
+ #:configure-flags (list

;; Provide channel metadata for 'guix describe'.
;; Don't pass '--with-channel-url' and

base-commit: d6f775c30c6f47e174f6110d1089edc6315600e4
--
2.46.0
J
J
Jakob Kirsch wrote on 30 Oct 19:13 +0100
(no subject)
(address . control@debbugs.gnu.org)
ZyJ3RG3bpC0kpqlw@kernelpanicroom
severity 74112 important
quit
H
H
Hilton Chain wrote on 5 Nov 11:31 +0100
Re: bug#74204: Guix is not reproducible
(name . Jakob Kirsch)(address . jakob.kirsch@web.de)
87fro6yntj.wl-hako@ultrarare.space
Hi Jakob,

On Tue, 05 Nov 2024 02:38:34 +0800,
Jakob Kirsch via Bug reports for GNU Guix wrote:
Toggle quote (10 lines)
>
> I've recently noticed that `guix challenge guix` fails since guix isn't
> reproducible at the moment. As someone pointed out on XMPP, this is due to
> parallelism issues with Guile so running `guix build guix --no-grafts
> --cores=1 --check` reliably produces the same output.
>
> I've sent a simple patch in #74112 to address this and I think this is
> important since the main guix package should definitely be reproducible so it
> can be trusted.

I can confirm the reproducibility issue.

I have two x86_64-linux machines building guix to verify the fix, I'll apply
your patch once they produce matching outputs.

Thanks for reporting!
H
H
Hilton Chain wrote on 5 Nov 16:25 +0100
Re: [bug#74112] bug#74204: Guix is not reproducible
(name . Jakob Kirsch)(address . jakob.kirsch@web.de)
87ed3pzora.wl-hako@ultrarare.space
Hello again,

On Tue, 05 Nov 2024 18:31:04 +0800,
Hilton Chain via Guix-patches via wrote:
Toggle quote (20 lines)
>
> Hi Jakob,
>
> On Tue, 05 Nov 2024 02:38:34 +0800,
> Jakob Kirsch via Bug reports for GNU Guix wrote:
> >
> > I've recently noticed that `guix challenge guix` fails since guix isn't
> > reproducible at the moment. As someone pointed out on XMPP, this is due to
> > parallelism issues with Guile so running `guix build guix --no-grafts
> > --cores=1 --check` reliably produces the same output.
> >
> > I've sent a simple patch in #74112 to address this and I think this is
> > important since the main guix package should definitely be reproducible so it
> > can be trusted.
>
> I can confirm the reproducibility issue.
>
> I have two x86_64-linux machines building guix to verify the fix, I'll apply
> your patch once they produce matching outputs.

Took me quite a while to build 5 rounds. :)

Toggle snippet (4 lines)
$ guix hash --serializer=nar /gnu/store/fs7x07jfn7igpkwv3alrs9by21q70y13-guix-1.4.0-26.5ab3c4c
0kh87wb4qn97kwzrf4igal71cjvv143j6jr2y3dwfzcy1madj1ll

Applied #74112 as 4c56d0cccdc44e12484b26332715f54768738c5f, thanks!
Closed
S
S
Simon Tournier wrote on 7 Nov 18:54 +0100
Re: bug#74204: [bug#74112] Guix is not reproducible
87pln70w16.fsf@gmail.com
Hi,

On Tue, 05 Nov 2024 at 23:25, Hilton Chain via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:

Toggle quote (14 lines)
>> I can confirm the reproducibility issue.
>>
>> I have two x86_64-linux machines building guix to verify the fix, I'll apply
>> your patch once they produce matching outputs.
>
> Took me quite a while to build 5 rounds. :)
>
> --8<---------------cut here---------------start------------->8---
> $ guix hash --serializer=nar /gnu/store/fs7x07jfn7igpkwv3alrs9by21q70y13-guix-1.4.0-26.5ab3c4c
> 0kh87wb4qn97kwzrf4igal71cjvv143j6jr2y3dwfzcy1madj1ll
> --8<---------------cut here---------------end--------------->8---
>
> Applied #74112 as 4c56d0cccdc44e12484b26332715f54768738c5f, thanks!

Maybe I am missing something. To my knowledge, .go files produced by
Guile are not always reproducible, see bug#20272 [1]. And, from my
understanding, Guix cannot be reproducible until this bug had been
fixed. Therefore, I am not convinced that this patch is worth under
this frame considering the build-time penalty it brings.

That’s said, maybe it’s better than nothing and the package ’guix’ is
barely built after all. I do not know.

What people think?

Cheers,
simon

Closed
M
M
Maxim Cournoyer wrote on 10 Nov 10:11 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87a5e7xxkl.fsf@gmail.com
Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (29 lines)
> Hi,
>
> On Tue, 05 Nov 2024 at 23:25, Hilton Chain via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:
>
>>> I can confirm the reproducibility issue.
>>>
>>> I have two x86_64-linux machines building guix to verify the fix, I'll apply
>>> your patch once they produce matching outputs.
>>
>> Took me quite a while to build 5 rounds. :)
>>
>> --8<---------------cut here---------------start------------->8---
>> $ guix hash --serializer=nar /gnu/store/fs7x07jfn7igpkwv3alrs9by21q70y13-guix-1.4.0-26.5ab3c4c
>> 0kh87wb4qn97kwzrf4igal71cjvv143j6jr2y3dwfzcy1madj1ll
>> --8<---------------cut here---------------end--------------->8---
>>
>> Applied #74112 as 4c56d0cccdc44e12484b26332715f54768738c5f, thanks!
>
> Maybe I am missing something. To my knowledge, .go files produced by
> Guile are not always reproducible, see bug#20272 [1]. And, from my
> understanding, Guix cannot be reproducible until this bug had been
> fixed. Therefore, I am not convinced that this patch is worth under
> this frame considering the build-time penalty it brings.
>
> That’s said, maybe it’s better than nothing and the package ’guix’ is
> barely built after all. I do not know.
>
> What people think?

Perhaps we should set the default parallel-build? to #f in the
guile-build-system at least in the meantime, with a prominent comment as
to why and a reference to the upstream issue? Many Guile packages use
the gnu-build-system so that wouldn't cover all of them like
'guix'... I'm not sure.

It'd be nicer to fix the underlying guile issue (again?), but I doubt
many people are up to this.

--
Thanks,
Maxim
Closed
L
L
Ludovic Courtès wrote on 14 Nov 11:01 +0100
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87o72ip20w.fsf@gnu.org
Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (2 lines)
> Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

Toggle quote (17 lines)
>> Maybe I am missing something. To my knowledge, .go files produced by
>> Guile are not always reproducible, see bug#20272 [1]. And, from my
>> understanding, Guix cannot be reproducible until this bug had been
>> fixed. Therefore, I am not convinced that this patch is worth under
>> this frame considering the build-time penalty it brings.
>>
>> That’s said, maybe it’s better than nothing and the package ’guix’ is
>> barely built after all. I do not know.
>>
>> What people think?
>
> Perhaps we should set the default parallel-build? to #f in the
> guile-build-system at least in the meantime, with a prominent comment as
> to why and a reference to the upstream issue? Many Guile packages use
> the gnu-build-system so that wouldn't cover all of them like
> 'guix'... I'm not sure.

Sounds good to me: packages using ‘guile-build-system’ are usually
relatively small so the impact is negligible.

‘guix’ is a little different because it takes so much time to build
sequentially…

Ludo’.
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 74112@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74112
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch