groff build is not reproducible

  • Done
  • quality assurance status badge
Details
3 participants
  • Danny Milosavljevic
  • Ludovic Courtès
  • Marius Bakke
Owner
unassigned
Submitted by
Danny Milosavljevic
Severity
normal

Debbugs page

Danny Milosavljevic wrote 8 years ago
(address . bug-guix@gnu.org)
20170706025451.312ddebe@scratchpost.org
on core-updates
Danny Milosavljevic wrote 8 years ago
[PATCH] gnu: groff: Remove timestamps.
(address . 27593@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20170706010144.31568-1-dannym@scratchpost.org

* gnu/packages/groff.scm (groff)[source]: Remove timestamps.
---
gnu/packages/groff.scm | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)

Toggle diff (26 lines)
diff --git a/gnu/packages/groff.scm b/gnu/packages/groff.scm
index 46e1ccf23..e5dd732ab 100644
--- a/gnu/packages/groff.scm
+++ b/gnu/packages/groff.scm
@@ -40,7 +40,20 @@
(uri (string-append "mirror://gnu/groff/groff-" version
".tar.gz"))
(sha256 (base32
- "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))))
+ "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))
+ (modules '((guix build utils)))
+ (snippet
+ '(begin
+ (substitute* "mdate.sh"
+ (("^LANGUAGE=") "
+
+echo \"1 Jan 1970\"
+exit 0
+
+"))
+ (substitute* '("src/devices/grops/ps.cpp"
+ "src/devices/grohtml/post-html.cpp")
+ (("time\\(0\\)") "0"))))))
(build-system gnu-build-system)
(outputs '("out"
"doc")) ;12MiB of PS, PDF, HTML, and examples
Danny Milosavljevic wrote 8 years ago
[PATCH v2] gnu: groff: Remove timestamps.
(address . 27593@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20170706014708.16491-1-dannym@scratchpost.org

* gnu/packages/groff.scm (groff)[source]: Remove timestamps.
---
gnu/packages/groff.scm | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)

Toggle diff (29 lines)
diff --git a/gnu/packages/groff.scm b/gnu/packages/groff.scm
index 46e1ccf23..1a74194a6 100644
--- a/gnu/packages/groff.scm
+++ b/gnu/packages/groff.scm
@@ -40,7 +40,23 @@
(uri (string-append "mirror://gnu/groff/groff-" version
".tar.gz"))
(sha256 (base32
- "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))))
+ "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))
+ (modules '((guix build utils)))
+ (snippet
+ '(begin
+ (substitute* "mdate.sh"
+ (("^LANGUAGE=") "
+
+echo \"1 Jan 1970\"
+exit 0
+
+"))
+ (substitute* '("src/devices/grops/ps.cpp"
+ "src/devices/grohtml/post-html.cpp"
+ "src/roff/troff/input.cpp")
+ (("time\\(0\\)") "0"))
+ (substitute* "src/devices/gropdf/gropdf.pl"
+ (("\\(time\\)") "(0)"))))))
(build-system gnu-build-system)
(outputs '("out"
"doc")) ;12MiB of PS, PDF, HTML, and examples
Marius Bakke wrote 8 years ago
87wp7hahpm.fsf@fastmail.com
Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (28 lines)
>
> * gnu/packages/groff.scm (groff)[source]: Remove timestamps.
> ---
> gnu/packages/groff.scm | 18 +++++++++++++++++-
> 1 file changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/groff.scm b/gnu/packages/groff.scm
> index 46e1ccf23..1a74194a6 100644
> --- a/gnu/packages/groff.scm
> +++ b/gnu/packages/groff.scm
> @@ -40,7 +40,23 @@
> (uri (string-append "mirror://gnu/groff/groff-" version
> ".tar.gz"))
> (sha256 (base32
> - "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))))
> + "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))
> + (modules '((guix build utils)))
> + (snippet
> + '(begin
> + (substitute* "mdate.sh"
> + (("^LANGUAGE=") "
> +
> +echo \"1 Jan 1970\"
> +exit 0
> +
> +"))

It would be more obvious what's going on if we substitute the "mdate.sh"
invocation from the Makefile instead. OTOH, this is more
future-proof. No strong opinion, but feels a bit "hacky".

Note that you can use \n here.

Toggle quote (7 lines)
> + (substitute* '("src/devices/grops/ps.cpp"
> + "src/devices/grohtml/post-html.cpp"
> + "src/roff/troff/input.cpp")
> + (("time\\(0\\)") "0"))
> + (substitute* "src/devices/gropdf/gropdf.pl"
> + (("\\(time\\)") "(0)"))))))

Are all of these really a cause of indeterminism in the build process?
It looks like this could make "legitimate" uses of groff return epoch 0
at runtime, I'm not sure if that is okay.

Which output files are fixed by these substitutions? Are there other
ways to deal with the indeterminism, e.g. substitute them in place?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlliKNYACgkQoqBt8qM6
VPrgTgf/cceemkIgOwTf19dCw6YELn43WJs9aTOCU7J9sYbQjt3UIsUCktMIo64r
4azVtVZbROgXwJjxLYGqZLkH4vUJ5yzkPpUQUWvMPHf7SbcFC2yBGpu+cjGjFqdp
Q5/0qomrY1BKm8dckvAlFLYBFyAVRWTCw/9v+u3SSOHHrKcBPvjjClxvkFcx8opm
QSC1SyM3jOy+pa34mvWh6fDbzC0VVqPgdxcAKjO0iN+wmhEc2DQiFS+ptgUbzTNt
J7g+dyGdiDQayi8SyZc4qKFn5iyc6TV28RxOYe86wJQ4TZ9Jr87dmQVGpLiQ56EQ
WMhr/g8G3Vu8qxU5U/Gl2zm5vK1ttA==
=GtU3
-----END PGP SIGNATURE-----

Ludovic Courtès wrote 8 years ago
(name . Marius Bakke)(address . mbakke@fastmail.com)
87eftpv177.fsf@gnu.org
Marius Bakke <mbakke@fastmail.com> skribis:

Toggle quote (50 lines)
> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
>> Fixes <https://bugs.gnu.org/27593>.
>>
>> * gnu/packages/groff.scm (groff)[source]: Remove timestamps.
>> ---
>> gnu/packages/groff.scm | 18 +++++++++++++++++-
>> 1 file changed, 17 insertions(+), 1 deletion(-)
>>
>> diff --git a/gnu/packages/groff.scm b/gnu/packages/groff.scm
>> index 46e1ccf23..1a74194a6 100644
>> --- a/gnu/packages/groff.scm
>> +++ b/gnu/packages/groff.scm
>> @@ -40,7 +40,23 @@
>> (uri (string-append "mirror://gnu/groff/groff-" version
>> ".tar.gz"))
>> (sha256 (base32
>> - "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))))
>> + "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))
>> + (modules '((guix build utils)))
>> + (snippet
>> + '(begin
>> + (substitute* "mdate.sh"
>> + (("^LANGUAGE=") "
>> +
>> +echo \"1 Jan 1970\"
>> +exit 0
>> +
>> +"))
>
> It would be more obvious what's going on if we substitute the "mdate.sh"
> invocation from the Makefile instead. OTOH, this is more
> future-proof. No strong opinion, but feels a bit "hacky".
>
> Note that you can use \n here.
>
>> + (substitute* '("src/devices/grops/ps.cpp"
>> + "src/devices/grohtml/post-html.cpp"
>> + "src/roff/troff/input.cpp")
>> + (("time\\(0\\)") "0"))
>> + (substitute* "src/devices/gropdf/gropdf.pl"
>> + (("\\(time\\)") "(0)"))))))
>
> Are all of these really a cause of indeterminism in the build process?
> It looks like this could make "legitimate" uses of groff return epoch 0
> at runtime, I'm not sure if that is okay.
>
> Which output files are fixed by these substitutions? Are there other
> ways to deal with the indeterminism, e.g. substitute them in place?

Sometimes it’s better to make such changes available as a .patch file so
that we can more clearly see from the context that we’re patching the
right thing. Perhaps that’s one such case here.

BTW, looking at issues.yml in
git://git.debian.org/git/reproducible/notes.git leads me to:


Debian has a SOURCE_DATE_EPOCH patch:


Perhaps we could just use that? That way, the default behavior of groff
would remained unchanged.

Apparently upstream wasn’t convinced by the idea of SOURCE_DATE_EPOCH,
so we could also restart the discussion with upstream in parallel.

However, it would be nice to commit this groff change quickly so we can
go on with core-updates builds. :-)

Thoughts?

Ludo’.
Danny Milosavljevic wrote 8 years ago
[PATCH v3] gnu: groff: Remove timestamps.
(name . Ludovic Courtès)(address . ludo@gnu.org)
20170709232746.7a7af62b@scratchpost.org
Hi Ludo,

On Sun, 09 Jul 2017 21:51:08 +0200
ludo@gnu.org (Ludovic Courtès) wrote:
Toggle quote (7 lines)
> Debian has a SOURCE_DATE_EPOCH patch:
>
> http://sources.debian.net/patches/groff/1.22.3-9/source-date-epoch.patch/
>
> Perhaps we could just use that? That way, the default behavior of groff
> would remained unchanged.

Yeah, it seems to work fine.

Apparently git send-email doesn't work for me right now.

So I attached the patch manually here...
Attachment: 0001-gnu-groff-Remove-timestamps.patch
Marius Bakke wrote 8 years ago
(address . 27593@debbugs.gnu.org)
87tw2l9swp.fsf@fastmail.com
Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (17 lines)
> Hi Ludo,
>
> On Sun, 09 Jul 2017 21:51:08 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>> Debian has a SOURCE_DATE_EPOCH patch:
>>
>> http://sources.debian.net/patches/groff/1.22.3-9/source-date-epoch.patch/
>>
>> Perhaps we could just use that? That way, the default behavior of groff
>> would remained unchanged.
>
> Yeah, it seems to work fine.
>
> Apparently git send-email doesn't work for me right now.
>
> So I attached the patch manually here...

[...]

Toggle quote (5 lines)
> - "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))))
> + "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))
> + (patches (search-patches "groff-source-date-epoch.patch"))
> + (modules '((guix build utils)))))

(guix build utils) is no longer needed. Apart from that LGTM!
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAllipmYACgkQoqBt8qM6
VPoMTQf/ZqPlVatWSrohbV+cg+uFyeYI+O8BrgbECSlp71mLqvmqIik4YIK1wlAL
Bn4kj4k5WmwtQ+0GJcu2GNkRWhsTiQxeUO3jyzY1TYlmh5v8cIYXw1rC+3AkLRuY
KUF260fQy7bJLY9nuWBu1YkLKjRNjuTrSFLRgx0XhggLLjbn3OtRxWERAHTdA3Q8
TATLYnduzwKTQ8fKNM/VllfeImXs5Bni2Pg3cWDWALlZ8j0lj3AjnTEQ7qJ9IlUl
Y+CdpmtqPmBVVjb9Ugw8UoFTciPJG8R+iRH4duBUY3Sv/udqtviUSAGX78HvpTP8
idUgHHyXBOpVwJp9cW7mqpZM7pCYTw==
=EuK9
-----END PGP SIGNATURE-----

Ludovic Courtès wrote 8 years ago
(name . Marius Bakke)(address . mbakke@fastmail.com)
87r2xo8xvs.fsf@gnu.org
Marius Bakke <mbakke@fastmail.com> skribis:

Toggle quote (28 lines)
> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
>> Hi Ludo,
>>
>> On Sun, 09 Jul 2017 21:51:08 +0200
>> ludo@gnu.org (Ludovic Courtès) wrote:
>>> Debian has a SOURCE_DATE_EPOCH patch:
>>>
>>> http://sources.debian.net/patches/groff/1.22.3-9/source-date-epoch.patch/
>>>
>>> Perhaps we could just use that? That way, the default behavior of groff
>>> would remained unchanged.
>>
>> Yeah, it seems to work fine.
>>
>> Apparently git send-email doesn't work for me right now.
>>
>> So I attached the patch manually here...
>
> [...]
>
>> - "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))))
>> + "1998v2kcs288d3y7kfxpvl369nqi06zbbvjzafyvyl3pr7bajj1s"))
>> + (patches (search-patches "groff-source-date-epoch.patch"))
>> + (modules '((guix build utils)))))
>
> (guix build utils) is no longer needed. Apart from that LGTM!

+1!

Ludo'.
Danny Milosavljevic wrote 8 years ago
(name . Ludovic Courtès)(address . ludo@gnu.org)
20170710124045.11b7e190@scratchpost.org
Thanks!

Pushed to core-updates as 9146543f8dce997c194aea93bf4c7a7a4144c8c9. Note that for real reproducibility the ghostscript patches are required too, and a phase in groff that sets the environment variable.

I've sent a new series for ghostscript.
Danny Milosavljevic wrote 8 years ago
[PATCH] gnu: groff: Make build reproducible.
(address . 27593@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20170710163704.25613-1-dannym@scratchpost.org
* gnu/packages/groff.scm (groff)[arguments]: Add phase "setenv".
---
gnu/packages/groff.scm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

Toggle diff (20 lines)
diff --git a/gnu/packages/groff.scm b/gnu/packages/groff.scm
index 67dd1dbfa..6e1a60e9b 100644
--- a/gnu/packages/groff.scm
+++ b/gnu/packages/groff.scm
@@ -56,7 +56,14 @@
("perl" ,perl)
("psutils" ,psutils)
("texinfo" ,texinfo)))
- (arguments '(#:parallel-build? #f)) ; parallel build fails
+ (arguments
+ `(#:parallel-build? #f ; parallel build fails
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'setenv
+ (lambda _
+ (setenv "GS_GENERATE_UUIDS" "0")
+ #t)))))
(synopsis "Typesetting from plain text mixed with formatting commands")
(description
"Groff is a typesetting package that reads plain text and produces
Marius Bakke wrote 8 years ago
87r2xo9kmj.fsf@fastmail.com
Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (2 lines)
> * gnu/packages/groff.scm (groff)[arguments]: Add phase "setenv".

What does this add to the previous patch? Can you add a comment in the
code explaining what it's for?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAllj0FQACgkQoqBt8qM6
VPq0fgf/cLd08h+/+ipdrssO8qPdgZHcQ/F1jUPe/xSIYiWzdmhtMFA+MKLOJeu/
Ad28mUVBHf7OlTSz7zNEjpOEjOayZca2r3fnhqHsfGGTNRKgYmLgv+OwUfGNz39l
biasleUpso8vIB1gU16qwUA3uW6eLHrk57GpjgFfvGJdVI+or+brmamMfZpRpLF2
jA8/UajaHtNx8/P8/LZuSIHl7ruFuyNz9X3qZ9vWNeVYTFeii4oWu5PWNCyzI1ei
5UjhojTZ8nzSVk0CQyRqQ4iyTRdz54uYH9dCYlsAQHFgtYEjmvGiAcclO34Y++xY
2ie7CDcxpYT8IfTOo8+ioXSp4N1Pww==
=MLQo
-----END PGP SIGNATURE-----

Danny Milosavljevic wrote 8 years ago
(name . Marius Bakke)(address . mbakke@fastmail.com)(address . 27593@debbugs.gnu.org)
20170710212845.587fd165@scratchpost.org
Hi Marius,

On Mon, 10 Jul 2017 21:07:00 +0200
Marius Bakke <mbakke@fastmail.com> wrote:

Toggle quote (7 lines)
> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
> > * gnu/packages/groff.scm (groff)[arguments]: Add phase "setenv".
>
> What does this add to the previous patch? Can you add a comment in the
> code explaining what it's for?

It replaces the previous patch by using a new patched-ghostscript feature:
- ghostscript can now be instructed to not print the problematic parts (timestamps in headers, UUIDs in headers) that were the original cause of the "ever-changing PDFs" problem.

The previous approach was fighting the symptoms, where every time a problem appeared in one of the outputs, I'd patch it somewhere else so that, if it did make it to ghostscript, ghostscript would put fake timestamps and would derive UUIDs from them - which upstream recommended we should definitely not do.

That was really not scalable or sane.

Also, the original ghostscript states that they are printing the current system date into PDFs (rather than, say, the input file mtime), against better judgement, for compatibility. Just no.

So this patch prevents ghostscript from putting the timestamps or the document UUIDs into the PDFs in the first place (I don't mean it puts a fake one there now - I mean it leaves the fields off entirely). Much simpler.

I'll add a comment.
Ludovic Courtès wrote 8 years ago
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 27593@debbugs.gnu.org)
87fue3xqu7.fsf@gnu.org
Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (23 lines)
> * gnu/packages/groff.scm (groff)[arguments]: Add phase "setenv".
> ---
> gnu/packages/groff.scm | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/groff.scm b/gnu/packages/groff.scm
> index 67dd1dbfa..6e1a60e9b 100644
> --- a/gnu/packages/groff.scm
> +++ b/gnu/packages/groff.scm
> @@ -56,7 +56,14 @@
> ("perl" ,perl)
> ("psutils" ,psutils)
> ("texinfo" ,texinfo)))
> - (arguments '(#:parallel-build? #f)) ; parallel build fails
> + (arguments
> + `(#:parallel-build? #f ; parallel build fails
> + #:phases
> + (modify-phases %standard-phases
> + (add-after 'unpack 'setenv
> + (lambda _
> + (setenv "GS_GENERATE_UUIDS" "0")
> + #t)))))

With a comment like Marius wrote, it’d be perfect; otherwise LGTM.

However, since ‘core-updates’ is now building and we’d like to make
progress on it, could you push it to ‘core-updates-next’ or similar?

Thanks,
Ludo’.
Ludovic Courtès wrote 8 years ago
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 27593@debbugs.gnu.org)
87shfv8vh3.fsf@gnu.org
Hi Danny,

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (30 lines)
> Danny Milosavljevic <dannym@scratchpost.org> skribis:
>
>> * gnu/packages/groff.scm (groff)[arguments]: Add phase "setenv".
>> ---
>> gnu/packages/groff.scm | 9 ++++++++-
>> 1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/gnu/packages/groff.scm b/gnu/packages/groff.scm
>> index 67dd1dbfa..6e1a60e9b 100644
>> --- a/gnu/packages/groff.scm
>> +++ b/gnu/packages/groff.scm
>> @@ -56,7 +56,14 @@
>> ("perl" ,perl)
>> ("psutils" ,psutils)
>> ("texinfo" ,texinfo)))
>> - (arguments '(#:parallel-build? #f)) ; parallel build fails
>> + (arguments
>> + `(#:parallel-build? #f ; parallel build fails
>> + #:phases
>> + (modify-phases %standard-phases
>> + (add-after 'unpack 'setenv
>> + (lambda _
>> + (setenv "GS_GENERATE_UUIDS" "0")
>> + #t)))))
>
> With a comment like Marius wrote, it’d be perfect; otherwise LGTM.
>
> However, since ‘core-updates’ is now building and we’d like to make
> progress on it, could you push it to ‘core-updates-next’ or similar?

Maybe now is the time to push this to ‘core-updates’?

(I was just hit by this reproducibility issue for a PDF file made with
groff…)

Cheers,
Ludo’.
Danny Milosavljevic wrote 8 years ago
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 27593@debbugs.gnu.org)
20170909231742.5819bb31@scratchpost.org
Hi Ludo,

On Sat, 09 Sep 2017 22:31:36 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (5 lines)
> Maybe now is the time to push this to ‘core-updates’?
>
> (I was just hit by this reproducibility issue for a PDF file made with
> groff…)

Isn't it pushed already?

If so, bad sign. Maybe the new environment variable doesn't always work?

Do you have a diff of what is changed in the PDF? It should have left the creationdate and moddate off, so it shouldn't list those in the diff...
Ludovic Courtès wrote 8 years ago
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 27593-done@debbugs.gnu.org)
87tw0a7l3r.fsf@gnu.org
Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (12 lines)
> Hi Ludo,
>
> On Sat, 09 Sep 2017 22:31:36 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Maybe now is the time to push this to ‘core-updates’?
>>
>> (I was just hit by this reproducibility issue for a PDF file made with
>> groff…)
>
> Isn't it pushed already?

Oh it is, but in the new ‘core-updates’ (I was looking at ‘master’.).

Perfect, happily closing the bug now. :-)

Thanks,
Ludo’.
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 27593@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 27593
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch
You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags
mumi command -t +confirmed -t +easy
Or, remove the moreinfo tag and set the help tag
mumi command -t -moreinfo -t +help