Gettext introduces timestamps in .mo files

OpenSubmitted by Ludovic Courtès.
Details
4 participants
  • Julien Lepiller
  • Ludovic Courtès
  • Miguel Ángel Arruga Vivas
  • zimoun
Owner
unassigned
Severity
normal
L
L
Ludovic Courtès wrote on 24 Mar 2017 23:54
(address . bug-guix@gnu.org)
8760iyxp3t.fsf@gnu.org
Gettext 0.19.8.1 (current core-updates,
77ab6983a19ef307558ab2607920158d6bb94ba8) introduces timestamps in .mo
file, without honoring SOURCE_DATE_EPOCH, which leads to
non-reproducible builds (for example ‘guix’).

Ludo’.
Z
Z
zimoun wrote on 1 Dec 2020 19:46
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 26247@debbugs.gnu.org)
865z5luzxx.fsf@gmail.com
Hi Ludo,

This old bug #26247 about timestamp in .mo files from Gettext is still
open:


On Fri, 24 Mar 2017 at 23:54, ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (5 lines)
> Gettext 0.19.8.1 (current core-updates,
> 77ab6983a19ef307558ab2607920158d6bb94ba8) introduces timestamps in .mo
> file, without honoring SOURCE_DATE_EPOCH, which leads to
> non-reproducible builds (for example ‘guix’).

Is it still relevant? Since Gettext is now at 0.20.1. How can I
reproduce the issue? Because the usual:

guix build gettext --no-grafts --check

works fine.


Cheers,
simon
L
L
Ludovic Courtès wrote on 3 Dec 2020 11:32
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 26247@debbugs.gnu.org)
87v9djgox5.fsf@gnu.org
Hi Simon,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (10 lines)
> On Fri, 24 Mar 2017 at 23:54, ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Gettext 0.19.8.1 (current core-updates,
>> 77ab6983a19ef307558ab2607920158d6bb94ba8) introduces timestamps in .mo
>> file, without honoring SOURCE_DATE_EPOCH, which leads to
>> non-reproducible builds (for example ‘guix’).
>
> Is it still relevant? Since Gettext is now at 0.20.1. How can I
> reproduce the issue?

I still see this:

Toggle snippet (19 lines)
$ guix challenge guix --substitute-urls="https://ci.guix.gnu.org https://bayfront.guix.gnu.org"
updating substitutes from 'https://bayfront.guix.gnu.org'... 100.0%
/gnu/store/babcmx68gkfxwzr3rmccan88dqjiqzb4-guix-1.2.0-3.35a32fe contents differ:
no local build for '/gnu/store/babcmx68gkfxwzr3rmccan88dqjiqzb4-guix-1.2.0-3.35a32fe'
https://ci.guix.gnu.org/nar/lzip/babcmx68gkfxwzr3rmccan88dqjiqzb4-guix-1.2.0-3.35a32fe: 13wvxga668grzs0p6sp0ghvdiy96nc9w71vs11djjkypsaf7wpw1
https://bayfront.guix.gnu.org/nar/lzip/babcmx68gkfxwzr3rmccan88dqjiqzb4-guix-1.2.0-3.35a32fe: 1rpwil9h2whjd9dbwpikxn8prkg924nhljglwj9yjijh578nlfr8
differing files:
/share/locale/en@quot/LC_MESSAGES/guix.mo
/share/locale/en@quot/LC_MESSAGES/guix-packages.mo
/share/locale/en@boldquot/LC_MESSAGES/guix.mo
/share/locale/en@boldquot/LC_MESSAGES/guix-packages.mo
/share/info/guix-cookbook.de.info.gz
/lib/guile/3.0/site-ccache/guix/workers.go
/lib/guile/3.0/site-ccache/guix/ui.go
/lib/guile/3.0/site-ccache/guix/swh.go
/lib/guile/3.0/site-ccache/guix/svn-download.go
[…]

‘--diff=diffoscope’ is not an option here because it takes too long
looking at all the .go files…

A focused diff shows this:

Toggle snippet (41 lines)
$ diffoscope --exclude-directory-metadata=yes /tmp/{t1,t2}/share/locale/en@quot/
--- /tmp/t1/share/locale/en@quot/
+++ /tmp/t2/share/locale/en@quot/
│ --- /tmp/t1/share/locale/en@quot/LC_MESSAGES
├── +++ /tmp/t2/share/locale/en@quot/LC_MESSAGES
│ │ --- /tmp/t1/share/locale/en@quot/LC_MESSAGES/guix-packages.mo
│ ├── +++ /tmp/t2/share/locale/en@quot/LC_MESSAGES/guix-packages.mo
│ │ ├── msgunfmt {}
│ │ │ @@ -1,12 +1,12 @@
│ │ │ msgid ""
│ │ │ msgstr ""
│ │ │ "Project-Id-Version: guix 1.2.0-3.35a32fe\n"
│ │ │ "Report-Msgid-Bugs-To: bug-guix@gnu.org\n"
│ │ │ -"PO-Revision-Date: 2020-11-29 18:33+0000\n"
│ │ │ +"PO-Revision-Date: 2020-12-02 10:10+0000\n"
│ │ │ "Last-Translator: Automatically generated\n"
│ │ │ "Language-Team: none\n"
│ │ │ "Language: en@quot\n"
│ │ │ "MIME-Version: 1.0\n"
│ │ │ "Content-Type: text/plain; charset=UTF-8\n"
│ │ │ "Content-Transfer-Encoding: 8bit\n"
│ │ │ "Plural-Forms: nplurals=2; plural=(n != 1);\n"
│ │ --- /tmp/t1/share/locale/en@quot/LC_MESSAGES/guix.mo
│ ├── +++ /tmp/t2/share/locale/en@quot/LC_MESSAGES/guix.mo
│ │ ├── msgunfmt {}
│ │ │ @@ -1,12 +1,12 @@
│ │ │ msgid ""
│ │ │ msgstr ""
│ │ │ "Project-Id-Version: guix 1.2.0-3.35a32fe\n"
│ │ │ "Report-Msgid-Bugs-To: bug-guix@gnu.org\n"
│ │ │ -"PO-Revision-Date: 2020-11-29 18:33+0000\n"
│ │ │ +"PO-Revision-Date: 2020-12-02 10:10+0000\n"
│ │ │ "Last-Translator: Automatically generated\n"
│ │ │ "Language-Team: none\n"
│ │ │ "Language: en@quot\n"
│ │ │ "MIME-Version: 1.0\n"
│ │ │ "Content-Type: text/plain; charset=UTF-8\n"
│ │ │ "Content-Transfer-Encoding: 8bit\n"
│ │ │ "Plural-Forms: nplurals=2; plural=(n != 1);\n"

Ludo’.
J
J
Julien Lepiller wrote on 3 Dec 2020 13:07
(address . 26247@debbugs.gnu.org)
EB5BE5DC-EF6C-4311-9A07-B2A06E71E65A@lepiller.eu
So it's not gettext itself, but our build system that generates the en@quote and en@boldquote files. Do we really need them? If so, we should find a way to generate them reproducibly.

Le 3 décembre 2020 05:32:54 GMT-05:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
Toggle quote (91 lines)
>Hi Simon,
>
>zimoun <zimon.toutoune@gmail.com> skribis:
>
>> On Fri, 24 Mar 2017 at 23:54, ludo@gnu.org (Ludovic Courtès) wrote:
>>
>>> Gettext 0.19.8.1 (current core-updates,
>>> 77ab6983a19ef307558ab2607920158d6bb94ba8) introduces timestamps in
>.mo
>>> file, without honoring SOURCE_DATE_EPOCH, which leads to
>>> non-reproducible builds (for example ‘guix’).
>>
>> Is it still relevant? Since Gettext is now at 0.20.1. How can I
>> reproduce the issue?
>
>I still see this:
>
>--8<---------------cut here---------------start------------->8---
>$ guix challenge guix --substitute-urls="https://ci.guix.gnu.org
>https://bayfront.guix.gnu.org"
>updating substitutes from 'https://bayfront.guix.gnu.org'... 100.0%
>/gnu/store/babcmx68gkfxwzr3rmccan88dqjiqzb4-guix-1.2.0-3.35a32fe
>contents differ:
>no local build for
>'/gnu/store/babcmx68gkfxwzr3rmccan88dqjiqzb4-guix-1.2.0-3.35a32fe'
>https://ci.guix.gnu.org/nar/lzip/babcmx68gkfxwzr3rmccan88dqjiqzb4-guix-1.2.0-3.35a32fe:
>13wvxga668grzs0p6sp0ghvdiy96nc9w71vs11djjkypsaf7wpw1
>https://bayfront.guix.gnu.org/nar/lzip/babcmx68gkfxwzr3rmccan88dqjiqzb4-guix-1.2.0-3.35a32fe:
>1rpwil9h2whjd9dbwpikxn8prkg924nhljglwj9yjijh578nlfr8
> differing files:
> /share/locale/en@quot/LC_MESSAGES/guix.mo
> /share/locale/en@quot/LC_MESSAGES/guix-packages.mo
> /share/locale/en@boldquot/LC_MESSAGES/guix.mo
> /share/locale/en@boldquot/LC_MESSAGES/guix-packages.mo
> /share/info/guix-cookbook.de.info.gz
> /lib/guile/3.0/site-ccache/guix/workers.go
> /lib/guile/3.0/site-ccache/guix/ui.go
> /lib/guile/3.0/site-ccache/guix/swh.go
> /lib/guile/3.0/site-ccache/guix/svn-download.go
>[…]
>--8<---------------cut here---------------end--------------->8---
>
>‘--diff=diffoscope’ is not an option here because it takes too long
>looking at all the .go files…
>
>A focused diff shows this:
>
>--8<---------------cut here---------------start------------->8---
>$ diffoscope --exclude-directory-metadata=yes
>/tmp/{t1,t2}/share/locale/en@quot/
>--- /tmp/t1/share/locale/en@quot/
>+++ /tmp/t2/share/locale/en@quot/
>│ --- /tmp/t1/share/locale/en@quot/LC_MESSAGES
>├── +++ /tmp/t2/share/locale/en@quot/LC_MESSAGES
>│ │ --- /tmp/t1/share/locale/en@quot/LC_MESSAGES/guix-packages.mo
>│ ├── +++ /tmp/t2/share/locale/en@quot/LC_MESSAGES/guix-packages.mo
>│ │ ├── msgunfmt {}
>│ │ │ @@ -1,12 +1,12 @@
>│ │ │ msgid ""
>│ │ │ msgstr ""
>│ │ │ "Project-Id-Version: guix 1.2.0-3.35a32fe\n"
>│ │ │ "Report-Msgid-Bugs-To: bug-guix@gnu.org\n"
>│ │ │ -"PO-Revision-Date: 2020-11-29 18:33+0000\n"
>│ │ │ +"PO-Revision-Date: 2020-12-02 10:10+0000\n"
>│ │ │ "Last-Translator: Automatically generated\n"
>│ │ │ "Language-Team: none\n"
>│ │ │ "Language: en@quot\n"
>│ │ │ "MIME-Version: 1.0\n"
>│ │ │ "Content-Type: text/plain; charset=UTF-8\n"
>│ │ │ "Content-Transfer-Encoding: 8bit\n"
>│ │ │ "Plural-Forms: nplurals=2; plural=(n != 1);\n"
>│ │ --- /tmp/t1/share/locale/en@quot/LC_MESSAGES/guix.mo
>│ ├── +++ /tmp/t2/share/locale/en@quot/LC_MESSAGES/guix.mo
>│ │ ├── msgunfmt {}
>│ │ │ @@ -1,12 +1,12 @@
>│ │ │ msgid ""
>│ │ │ msgstr ""
>│ │ │ "Project-Id-Version: guix 1.2.0-3.35a32fe\n"
>│ │ │ "Report-Msgid-Bugs-To: bug-guix@gnu.org\n"
>│ │ │ -"PO-Revision-Date: 2020-11-29 18:33+0000\n"
>│ │ │ +"PO-Revision-Date: 2020-12-02 10:10+0000\n"
>│ │ │ "Last-Translator: Automatically generated\n"
>│ │ │ "Language-Team: none\n"
>│ │ │ "Language: en@quot\n"
>│ │ │ "MIME-Version: 1.0\n"
>│ │ │ "Content-Type: text/plain; charset=UTF-8\n"
>│ │ │ "Content-Transfer-Encoding: 8bit\n"
>│ │ │ "Plural-Forms: nplurals=2; plural=(n != 1);\n"
>--8<---------------cut here---------------end--------------->8---
>
>Ludo’.
Attachment: file
L
L
Ludovic Courtès wrote on 3 Dec 2020 15:41
(name . Julien Lepiller)(address . julien@lepiller.eu)
87blfbdka0.fsf@gnu.org
Julien Lepiller <julien@lepiller.eu> skribis:

Toggle quote (2 lines)
> So it's not gettext itself, but our build system that generates the en@quote and en@boldquote files. Do we really need them? If so, we should find a way to generate them reproducibly.

They’re generated automatically by the gettext Makefilery that we use,
so I don’t think there’s much we can do?

We could remove them from ‘LINGUAS’, but we’d be losing something,
wouldn’t we?

Ludo’.
M
M
Miguel Ángel Arruga Vivas wrote on 11 Dec 2020 14:26
(name . Ludovic Courtès)(address . ludo@gnu.org)
875z58fp8e.fsf@gmail.com
Hi!

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (9 lines)
> Julien Lepiller <julien@lepiller.eu> skribis:
>
>> So it's not gettext itself, but our build system that generates the
>> en@quote and en@boldquote files. Do we really need them? If so, we
>> should find a way to generate them reproducibly.
>
> They’re generated automatically by the gettext Makefilery that we use,
> so I don’t think there’s much we can do?

The issue isn't on those files but on POT generation. Currently
xgettext doesn't honor SOURCE_DATE_EPOCH to fill POT-Creation-Date,
which is used to fill PO-Revision-Date for auto-generated po files.
These files are included into tarballs generated by make dist, therefore
its date is already fixed, but they aren't present on our git tree---for
obvious reasons.

This bug has already been reported upstream[1] and probably I'll push it
as soon as I have more test cases prepared and receive a brief review,
but I can prepare a patch for previous versions if it's needed too.

Best regards,
Miguel

L
L
Ludovic Courtès wrote on 14 Dec 2020 10:09
(name . Miguel Ángel Arruga Vivas)(address . rosen644835@gmail.com)
87sg88wy7p.fsf@gnu.org
Hi Miguel,

Miguel Ángel Arruga Vivas <rosen644835@gmail.com> skribis:

Toggle quote (18 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Julien Lepiller <julien@lepiller.eu> skribis:
>>
>>> So it's not gettext itself, but our build system that generates the
>>> en@quote and en@boldquote files. Do we really need them? If so, we
>>> should find a way to generate them reproducibly.
>>
>> They’re generated automatically by the gettext Makefilery that we use,
>> so I don’t think there’s much we can do?
>
> The issue isn't on those files but on POT generation. Currently
> xgettext doesn't honor SOURCE_DATE_EPOCH to fill POT-Creation-Date,
> which is used to fill PO-Revision-Date for auto-generated po files.
> These files are included into tarballs generated by make dist, therefore
> its date is already fixed, but they aren't present on our git tree---for
> obvious reasons.

OK.

Toggle quote (4 lines)
> This bug has already been reported upstream[1] and probably I'll push it
> as soon as I have more test cases prepared and receive a brief review,
> but I can prepare a patch for previous versions if it's needed too.

[...]

Toggle quote (2 lines)
Thanks for getting to the bottom of this and for preparing a patch
upstream!

In ‘core-updates’, we could either wait for the next Gettext release or
apply your patch in the meantime.

Ludo’.
?