ghostscript-with-cups is not reproducible

diff -ur --no-dereference guix-build-ghostscript-with-cups-9.14.0.drv-0/gnu-ghostscript-9.14.0/soobj/gsromfs1_.c guix-build-ghostscript-with-cups-9.14.0.drv-1/gnu-ghostscript-9.14.0/

soobj/gsromfs1_.c

--- guix-build-ghostscript-with-cups-9.14.0.drv-0/gnu-ghostscript-9.14.0/soobj/gsromfs1_.c 2017-07-03 19:45:46.632983314 +0200

+++ guix-build-ghostscript-with-cups-9.14.0.drv-1/gnu-ghostscript-9.14.0/soobj/gsromfs1_.c 2017-07-03 19:56:25.401286255 +0200

@@ -5,7 +5,7 @@

#include "time_.h"

- time_t gs_romfs_buildtime = 1499103945;

+ time_t gs_romfs_buildtime = 1499104584;

static uint32_t node_0[] = {

0x5cdc0280, /* compression_flag_bit + file length */

diff -ur --no-dereference guix-build-ghostscript-with-cups-9.14.0.drv-0/gnu-ghostscript-9.14.0/soobj/gsromfs1.c guix-build-ghostscript-with-cups-9.14.0.drv-1/gnu-ghostscript-9.14.0/soobj/gsromfs1.c

--- guix-build-ghostscript-with-cups-9.14.0.drv-0/gnu-ghostscript-9.14.0/soobj/gsromfs1.c 2017-07-03 19:45:46.660983573 +0200

+++ guix-build-ghostscript-with-cups-9.14.0.drv-1/gnu-ghostscript-9.14.0/soobj/gsromfs1.c 2017-07-03 19:56:25.437286605 +0200

@@ -5,7 +5,7 @@

#include "time_.h"

- time_t gs_romfs_buildtime = 1499103945;

+ time_t gs_romfs_buildtime = 1499104584;

static uint32_t node_0[] = {

0x5cdc0280, /* compression_flag_bit + file length */

---

gnu/packages/ghostscript.scm | 5 +++++

1 file changed, 5 insertions(+)

On Mon, Jul 03, 2017 at 08:16:46PM +0200, Danny Milosavljevic wrote:

is it possible to patch this in a snippet?

--

Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר

GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351

Confidentiality cannot be guaranteed on emails sent or received unencrypted

---

gnu/packages/ghostscript.scm | 10 +++++++---

1 file changed, 7 insertions(+), 3 deletions(-)

On Mon, Jul 03, 2017 at 08:16:46PM +0200, Danny Milosavljevic wrote:

Same feedback as for netpbm: If ghostscript works with this change and

we haven't built it yet for core-updates on Hydra, it's okay to make

this change.

But I'm not reviewing the details of the change itself; I'm technically

on vacation this week ;)

Efraim Flashner <efraim@flashner.co.il> skribis:

Agreed; your second version LGTM.

Like Leo, I’d invite you to check the state of ‘core-updates’. :-)

Depending on how far it went, perhaps we can push it there along with

the netpbm one, cancel pending builds, and start a new evaluation (let

me know if you want to do these.)

Thanks,

Ludo’.

close 27567

close 27563

---

gnu/local.mk | 1 +

gnu/packages/ghostscript.scm | 3 ++-

.../patches/ghostscript-no-header-uuid.patch | 28 ++++++++++++++++++++++

3 files changed, 31 insertions(+), 1 deletion(-)

create mode 100644 gnu/packages/patches/ghostscript-no-header-uuid.patch

---

gnu/local.mk | 1 +

gnu/packages/ghostscript.scm | 3 ++-

.../patches/ghostscript-no-header-uuid.patch | 28 ++++++++++++++++++++++

3 files changed, 31 insertions(+), 1 deletion(-)

create mode 100644 gnu/packages/patches/ghostscript-no-header-uuid.patch

So this is what's needed to finally make ghostscript, netpbm and groff

reproducible. Groff just finished its 38th build on my machine and it

finally compared the rounds as equal.

I'm posting those here in order to make sure we all agree that this is

the way to go.

The patchset patches PDF creation in ghostscript. It's for core-updates.

The PDF file has a trailer field "/ID" which is required only when

encrypting. But ghostscript derives it from the current time.

So I figured leaving it off if allowed would be the easiest fix.

If it's not there then it can't change :P

Also, newer PDF files have an RDF header specifying some extra information

in an XML-like format. For example there's an instance UUID (PDF/A specifies

that it's recommended to set this to an empty string), and a document UUID.

The latter again is time-based.

This patchset

Because of the printf-style functions, it has to split up the printfs a bit,

but really it just makes one of the parts printed optional - in multiple

places (because PDF trailers can be chained).

Danny Milosavljevic (2):

gnu: ghostscript: Don't write document UUID; use "" as instance UUID.

gnu: ghostscript: Write document ID only when encrypting.

gnu/local.mk | 2 +

gnu/packages/ghostscript.scm | 4 +-

.../patches/ghostscript-no-header-id.patch | 47 ++++++++++++++++++++++

.../patches/ghostscript-no-header-uuid.patch | 28 +++++++++++++

4 files changed, 80 insertions(+), 1 deletion(-)

create mode 100644 gnu/packages/patches/ghostscript-no-header-id.patch

create mode 100644 gnu/packages/patches/ghostscript-no-header-uuid.patch

---

gnu/local.mk | 1 +

gnu/packages/ghostscript.scm | 1 +

.../patches/ghostscript-no-header-id.patch | 47 ++++++++++++++++++++++

3 files changed, 49 insertions(+)

create mode 100644 gnu/packages/patches/ghostscript-no-header-id.patch

Upstream bug report: https://bugs.ghostscript.com/show_bug.cgi?id=698208.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

[...]

Please add a comment at the top of the patch to explain what it does and

what its upstream status is or if there were discussions around it.

What’s the impact of simply removing all of this from generated PDF

files? Should we instead make it optional? For instance, we could

introduce a new environment variable, say “GHOSTSCRIPT_PDF_UUID”, and

use that as the document UUID when it’s defined. In our build

environments, we’d always set GHOSTSCRIPT_PDF_UUID to a known value to

ensure determinism.

WDYT?

Also, what does Debian do? :-)

Thank you for working on it!

Ludo’.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

If it’s time-based, then the solution may be to honor SOURCE_DATE_EPOCH.

I asked on #reproducible-builds (OFTC). A patch had been proposed

upstream but rejected:

http://bugs.ghostscript.com/show_bug.cgi?id=696765

See also

https://wiki.debian.org/ReproducibleBuilds/PdfGeneratedByGhostscript.

Ludo’.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

[...]

Please add an explanation here.

Also, do you know whether the PDF specs are OK with that? Might be good

to discuss with upstream, we wouldn’t want to generate somewhat broken

PDFs. WDYT?

Thank you,

Ludo’.

Hi Ludo,

On Fri, 07 Jul 2017 14:02:04 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Yeah, at the upstream bug link https://bugs.ghostscript.com/show_bug.cgi?id=698208 we discussed that (somewhat). While they don't want to carry the patches (because they don't want to lose functionality) they explained that it might well be that *future* versions of the spec could make ID and UUID mandatory.

Right now there's a stringent spec, called PDF/A (for "archiving"; which is intended for governing bodies where you don't want existing documents that dynamically alter their contents after some time - like with Javascript or something) which already sets the instance UUID to "". So I just set it to "" always rather than just for PDF/A.

Also, as far as I understand the "/ID" is currently only mandatory when encrypting, although in the future it might change.

That leaves the document UUID - and upstream, in some of the other bugreports, explained that they want UNIQUE document UUIDs. So I figured that we should just leave it off - so it's not the same over multiple documents. They are definitely not fine with non-unique UUIDs.

This RDF metadata stuff (the instance UUID and document UUID) is quite new. In a former life I wrote PDF parsers and I didn't handle the RDF back then at all. So I guess it would even work to leave the entire RDF metadata off - after all, it worked back then.

If someone is well-versed in XMP RDF metadata for PDF, I wonder what is better: leaving the entire RDF off or just leaving the element containing the document id (as an attribute) off. Currently, the patch does the latter. The specification by adobe (XMP Specification Part 1, ISO 16684-1:2011(E) Annex A) says "The use of robust GUIDs is encouraged; having globally unique values is important" but as far as I can see doesn't say whether they are mandatory.

I also thought of patching groff instead. But it seems that groff is now searching for a maintainer - I'm not sure anyone would integrate it there. Also, I'm not well-versed in perl. Also, patching finished PDFs (using regexps or something) is kinda dangerous because nobody *forces* you to encode the streams (think: attachements) in PDFs. So it could be that some other non-PDF thing is integrated into the PDF as a stream and the regexp substituter would just substitute it in there as well.

There's a program "pdfmark" which is supposed to be for changing the metadata for PDFs but upstream said that it can't change those fields. It could change the CreationDate, ModDate etc.

In short, I think the lowest risk is patching ghostscript as we did here.

Hi Ludo,

On Fri, 07 Jul 2017 14:00:09 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Upstream says definitely not. The UUIDs are supposed to be unique and they don't want anyone writing fixed UUIDs into documents (except for "" for the instance ID which they themselves do).

I think there could be some enterprise search engine which associates a document with other resources using the document UUID - and if everyone went and reused UUIDs it would be very confused.

That's why I left it off.

I don't know. It's just one metadata element - and it's recent. I mean that there are lots of (old) PDF files that don't have it in the first place.

Upstream definitely doesn't want that - at least not if it's a constant value for all PDFs.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

OK, makes sense. Maybe we can still have it disabled (or enabled) by

environment variable instead of having it removed wholesale?

Ludo’.

Hmm... can you access the patch linked there (under "Solution") ?

On Fri, 07 Jul 2017 17:18:15 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Sure. Any suggestions for the name of the environment variable? Also, where would we set it so the build processes of all the other packages actually pick it up?

Would it disable and re-enable all these things at once? :

On Fri, Jul 07, 2017 at 03:21:49PM +0200, Danny Milosavljevic wrote:

I think the lowest risk is to do nothing to Ghostscript and move the PDF

documentation to a separate 'doc' output. Then, we could have

reproducible binaries and ignore the PDF issues for now. Does anyone

know how many packages include PDF documentation built with Ghostscript?

I think the next lowest risk is to do nothing.

I think it's risky to patch Ghostscript, for a few reasons:

1) The patches don't include provenance information, so it's difficult

to find any other discussion of them. I'd like for the Ghostscript

maintainers to have reviewed the proposed changes, both for code

correctness and for PDF-specific issues.

2) At least some of the patches in the related Ghostscript discussions

seem to be proof of concepts rather than finished code:

So, if these patches came from there, we'd want to be extra careful.

By the way, this is the patch used for Debian's latest Ghostscript

package:

That patch was not reviewed on a public forum, at least nothing I can

find with Google. Again, I'd want to get the Ghostscript team's advice.

Hi Leo,

On Fri, 7 Jul 2017 12:21:51 -0400

Leo Famulari <leo@famulari.name> wrote:

Aren't the derivations of the doc outputs still a problem? For example, Hydra will run out of space sooner or later because it keeps building them, right?

No, I wrote the ones here without external sources (except for the direct discussion on my newish upstream bug report, and the PDF and XMP specifications - whatever worth they have).

On such an approach they advised that we should only generate *unique* UUIDs. But the UUIDs are generated from these times. So that linked patch would generate multiple non-unique uuids on systems.

That's why I removed the entire UUID and Time sections and actually didn't fiddle with the ghostscript-internal times at all. Builds reproducibly.

I wonder how many packages actually use the ghostscript pdf writer too. How to find that out?

Note that groff itself also fails to build reproducibly without the patches.

In any case, the patch 2/2 is quite tame (it looks scary because of the printf splitting, but it's actually just either leaving "/ID[...]" off or not, globally).

But I understand that it would be even easier to do nothing. Wouldn't make the stuff reproducible, though.

I'd vote for an environment variable to disable UUID printing and also Time header printing. That way it would do everything normally in regular usage - but when used in packages, it would just not *print* the problematic stuff. No internal state is changed at all by the patches.

On Fri, Jul 07, 2017 at 06:42:25PM +0200, Danny Milosavljevic wrote:

Do these timestamps and UUID affect the derivations? I figured they only

affected the result of running the derivation — that is, the output of

the build process. Those outputs are what we'd like to create

reproducibly, but they don't cause rebuilds if they are not

reproducible.

If a package's dependency graph is identical to before, Guix (and I

assume Hydra) will not rebuild it, even if we humans know that the built

output is unreproducible, such as when timestamps are embedded.

My apologies if I misinterpreted your question.

We run out of space and have to garbage collect periodically anyways.

Regardless, once we own the Hydra machine, I'd like for us to buy a huge

amount of storage and keep built outputs for much longer than we do now.

In practice, it's not really possible to go back in time more than 6

months of Guix, due to missing upstream sources and test suites with

expiration dates.

Ah, thanks for the clarification.

Okay, thank you for explaining this (especially if you already explained

it! It's hard to join a conversation like this halfway through). I'll

read your patches carefully later today.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

That sounds like a reasonable approach to me.

I’d make it opt-out to minimize disruption—i.e., the env var would allow

users to disable UUID generation, which would still be enabled as before

otherwise.

Ludo’.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

It’s 404, but Leo sent a link to the patch on debian.org.

Ludo’.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

For CreationDate/ModDate, I think it should honor SOURCE_DATE_EPOCH as

in

https://anonscm.debian.org/git/printing/ghostscript.git/tree/debian/patches/2010_add_build_timestamp_setting.patch?id=e2bf3ad7026afe13636d4937430c3fdae7854078.

For the two UUIDs (and “ID” too?), maybe we can use, say,

GS_GENERATE_UUIDS; if set to 0 or “no” it’s disable, otherwise it’s

enabled.

Eventually we can add it to gnu-build-system.scm, but for now, given

that core-updates is well built, we should add it on a case-by-case

basis. I don’t think there are that many packages that produce PDFs,

but I could be wrong.

How does that sound?

Thank you,

Ludo’.

On Fri, 07 Jul 2017 19:51:10 +0200

ludo@gnu.org (Ludovic Courtès) wrote:

Really? I've been leaving them off, too. Especially because of this funny comment in the upstream ghostscript:

/* Initialize the IDs allocated at startup. */

void

pdf_initialize_ids(gx_device_pdf * pdev)

{

...

/*

* Acrobat Distiller sets CreationDate and ModDate to the current

* date and time, rather than (for example) %%CreationDate from the

* PostScript file. We think this is wrong, but we do the same.

*/

{

... proceed to set CreationDate and ModDate to the current time.

}

}

That would look like this:

if (!getenv("GS_GENERATE_UUIDS") || strcmp(getenv("GS_GENERATE_UUIDS"), "0") == 0 || strcmp(getenv("GS_GENERATE_UUIDS"), "no") == 0) ...

Okay :)

On Fri, Jul 07, 2017 at 01:24:07PM -0400, Leo Famulari wrote:

I didn't get around to it :/

I think that if you are Ludo and confident and everything seems to work,

the patches are fine, right? :)

Danny Milosavljevic <dannym@scratchpost.org> skribis:

I guess they hamper reproducibility if they’re always created? In that

case, they need to follow SOURCE_DATE_EPOCH; if OTOH they’re only

created in specific cases that don’t matter much, we can leave them.

Yes.

Thanks!

Ludo’.

Danny Milosavljevic (3):

gnu: ghostscript: Make "/ID" optional, depending on environment

variable.

gnu: ghostscript: Make XMP UUID headers optional, depending on

environment variable.

gnu: ghostscript: Make "/CreationDate", "/ModDate" optoinal, depending

on environment variable.

gnu/local.mk | 3 ++

gnu/packages/ghostscript.scm | 10 ++---

.../ghostscript-no-header-creationdate.patch | 16 +++++++

.../patches/ghostscript-no-header-id.patch | 49 ++++++++++++++++++++++

.../patches/ghostscript-no-header-uuid.patch | 43 +++++++++++++++++++

5 files changed, 116 insertions(+), 5 deletions(-)

create mode 100644 gnu/packages/patches/ghostscript-no-header-creationdate.patch

create mode 100644 gnu/packages/patches/ghostscript-no-header-id.patch

create mode 100644 gnu/packages/patches/ghostscript-no-header-uuid.patch

---

gnu/local.mk | 1 +

gnu/packages/ghostscript.scm | 3 +-

.../patches/ghostscript-no-header-uuid.patch | 43 ++++++++++++++++++++++

3 files changed, 46 insertions(+), 1 deletion(-)

create mode 100644 gnu/packages/patches/ghostscript-no-header-uuid.patch

---

gnu/local.mk | 1 +

gnu/packages/ghostscript.scm | 9 +++-

.../patches/ghostscript-no-header-id.patch | 49 ++++++++++++++++++++++

3 files changed, 57 insertions(+), 2 deletions(-)

create mode 100644 gnu/packages/patches/ghostscript-no-header-id.patch

---

gnu/local.mk | 1 +

gnu/packages/ghostscript.scm | 10 ++--------

.../patches/ghostscript-no-header-creationdate.patch | 16 ++++++++++++++++

3 files changed, 19 insertions(+), 8 deletions(-)

create mode 100644 gnu/packages/patches/ghostscript-no-header-creationdate.patch

Danny Milosavljevic <dannym@scratchpost.org> skribis:

[...]

Please include a description of what the patch does and what its

upstream status is (you can write that it was not submitted upstream but

that similar patches were discussed, linking to this bug and upstream

URLs.)

Extra line. :-)

I haven’t tested the patch, but if you can confirm that ps2pdf still

generates valid PDF files after this change, both with and without

UUIDs, then that’s fine with me.

Thank you!

Ludo’.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

[...]

Does this also apply to ‘core-updates’?

Rather ‘strcasecmp’ for the last one.

If the resulting ps2pdf works, OK for ‘core-updates’!

Ludo’.

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Hmm, what is this patch against? Was it generated against a previous

WIP tree?

Ludo’.

I tested it, with bug# 27593 it works and creates a reproducible groff.

So I pushed this set to core-updates; closing this bug report.

Phiew, finally almost done. There's one more patch for groff at bug# 27593 in order to set the environment variable GS_GENERATE_UUIDS to "0".