openjdk@9/10 sources not reproducible

  • Done
  • quality assurance status badge
Details
5 participants
  • Björn Höfling
  • Jonathan Brielmaier
  • Lars-Dominik Braun
  • Ludovic Courtès
  • Simon Tournier
Owner
unassigned
Submitted by
Lars-Dominik Braun
Severity
normal
L
L
Lars-Dominik Braun wrote on 9 Mar 2023 10:48
(address . bug-guix@gnu.org)
ZAmrhccMfljep8+i@noor.fritz.box
Hi,

it looks like the (auto-generated) tarballs for openjdk@9 and openjdk@10
changed their hash, causing a hash mismatch via

guix build -S openjdk@9 --no-substitutes --no-grafts

I’m not sure why it uses these tarballs in the first place, since we
have a hg-download.

Lars
B
B
Björn Höfling wrote on 12 Mar 2023 00:06
(name . Lars-Dominik Braun)(address . lars@6xq.net)(address . 62071@debbugs.gnu.org)
20230312000655.3f66d7fd@tangletp
On Thu, 9 Mar 2023 10:48:53 +0100
Lars-Dominik Braun <lars@6xq.net> wrote:

Toggle quote (6 lines)
> Hi,
>
> it looks like the (auto-generated) tarballs for openjdk@9 and
> openjdk@10 changed their hash, causing a hash mismatch via
>
> guix build -S openjdk@9 --no-substitutes --no-grafts

I can confirm this.

I found the old versions of openjdk 9 and 10 on a server of mine. I
will compare old/new tomorrow (oh, better say: later today).

Toggle quote (3 lines)
> I’m not sure why it uses these tarballs in the first place, since we
> have a hg-download.

I don't know. Maybe there was no hg-download yet when we added OpenJDK9?

Björn
-----BEGIN PGP SIGNATURE-----

iF0EAREKAB0WIQQiGUP0np8nb5SZM4K/KGy2WT5f/QUCZA0JjwAKCRC/KGy2WT5f
/cfuAJ9F262CVMXHFElH3XmSQv4sb7PDWQCgqV1qTDZOgcicr7O/gzQIcw9oGkw=
=Dtbz
-----END PGP SIGNATURE-----


B
B
Björn Höfling wrote on 12 Mar 2023 22:00
(name . Lars-Dominik Braun)(address . lars@6xq.net)(address . 62071@debbugs.gnu.org)
20230312220021.22bfff4f@tangletp
On Thu, 9 Mar 2023 10:48:53 +0100
Lars-Dominik Braun <lars@6xq.net> wrote:

Toggle quote (10 lines)
> Hi,
>
> it looks like the (auto-generated) tarballs for openjdk@9 and
> openjdk@10 changed their hash, causing a hash mismatch via
>
> guix build -S openjdk@9 --no-substitutes --no-grafts
>
> I’m not sure why it uses these tarballs in the first place, since we
> have a hg-download.

I compared for JDK9 the two tarballs (old and new hash) and there is no
difference in the content (according to diffoscope). Also, if I
hg-clone the repository/tag (and add the .hg_archival.txt file), all
three directory trees have the same hash value according to guix hash
-rx

Thus, it seams like their artifacts are not stable, as we saw it
for autogenerated artifacts on github.

I will check the same for JDK10 and will prepare a patch within the
next two days.

Björn
-----BEGIN PGP SIGNATURE-----

iF0EAREKAB0WIQQiGUP0np8nb5SZM4K/KGy2WT5f/QUCZA49ZQAKCRC/KGy2WT5f
/ZPVAJwJBM3IhJAHQObQZUJlxg949Gd1TACeNDPnn+F00mB5ADw9uZ8bmWX9L9g=
=+1IK
-----END PGP SIGNATURE-----


S
S
Simon Tournier wrote on 13 Mar 2023 14:50
(address . 62071@debbugs.gnu.org)
87o7owlpz8.fsf@gmail.com
Hi,

On dim., 12 mars 2023 at 22:00, Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

Toggle quote (6 lines)
> I compared for JDK9 the two tarballs (old and new hash) and there is no
> difference in the content (according to diffoscope). Also, if I
> hg-clone the repository/tag (and add the .hg_archival.txt file), all
> three directory trees have the same hash value according to guix hash
> -rx

So maybe it comes from some parameters of the compressor; maybe they
changed their level. Well, I do not know how to check that.

Maybe the best is to switch from url-fetch to hg-fetch. WDYT?

Cheers,
simon
B
B
Björn Höfling wrote on 16 Mar 2023 10:12
(name . Lars-Dominik Braun)(address . lars@6xq.net)(address . 62071-done@debbugs.gnu.org)
20230316101212.45619eec@tangletp
On Sun, 12 Mar 2023 22:00:21 +0100
Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

Toggle quote (13 lines)
> On Thu, 9 Mar 2023 10:48:53 +0100
> Lars-Dominik Braun <lars@6xq.net> wrote:
>
> > Hi,
> >
> > it looks like the (auto-generated) tarballs for openjdk@9 and
> > openjdk@10 changed their hash, causing a hash mismatch via
> >
> > guix build -S openjdk@9 --no-substitutes --no-grafts
> >
> > I’m not sure why it uses these tarballs in the first place, since we
> > have a hg-download.

Changed to hg-download in commit(s):

7636c49b45adb9870cf416c64bde032ec858a820

Thanks for pointing this out.

Björn
-----BEGIN PGP SIGNATURE-----

iF0EAREKAB0WIQQiGUP0np8nb5SZM4K/KGy2WT5f/QUCZBLdbAAKCRC/KGy2WT5f
/cvrAKCX2tIMh/Lcnt+TkIGaWDmMfztqqQCeNppX9Ndwy3OaCU7p+3l49E+qY18=
=5nU7
-----END PGP SIGNATURE-----


Closed
L
L
Ludovic Courtès wrote on 16 Mar 2023 12:48
(name . Björn Höfling)(address . bjoern.hoefling@bjoernhoefling.de)
878rfwgbng.fsf@gnu.org
Hi Björn,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

Toggle quote (3 lines)
> I will check the same for JDK10 and will prepare a patch within the
> next two days.

Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
commit!

For the record, there are two remaining issues:

1. Reproducibility of past revisions. If we lose copies of the
auto-generated tarballs, then OpenJDK in past revisions of Guix is
irreparably lost. We should check whether/how to get them in
Disarchive + SWH.

2. Mercurial/SWH bridge. While SWH has a one-to-one mapping with Git
(you can ask it for a specific Git commit ID), that’s not true for
hg. This is a more general problem, but as things are today,
there’s no automatic SWH fallback if the upstream hg server
vanishes.

Thanks,
Ludo’.
J
J
Jonathan Brielmaier wrote on 16 Mar 2023 13:37
openjdk@9/10 sources not reproducible
(address . 62071@debbugs.gnu.org)
b2c12554-aca9-0d74-5a52-d447be8f86fe@web.de
I’m not sure why it uses these tarballs in the first place, since we
have a hg-download.

-> I guess a reason could be that downloading via hg is quite slow.
Thats at least my impression when fetching the "comm" repository for
Thunderbird with mecurial. Tarballs and git checkout tend to be way
faster...
B
B
Björn Höfling wrote on 17 Mar 2023 23:10
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
20230317231031.4a7eb099@tangletp
On Thu, 16 Mar 2023 12:48:19 +0100
Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (17 lines)
> Hi Björn,
>
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
>
> > I will check the same for JDK10 and will prepare a patch within the
> > next two days.
>
> Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
> commit!
>
> For the record, there are two remaining issues:
>
> 1. Reproducibility of past revisions. If we lose copies of the
> auto-generated tarballs, then OpenJDK in past revisions of Guix
> is irreparably lost. We should check whether/how to get them in
> Disarchive + SWH.

How do we do that? Adding git repos to SWH is something I can achieve,
but adding tarballs to SWH was always opaque to me.

I find no reference in the manual about Disarchive. Ideally, is there a
linter for just adding a package to the disarchive database?

Toggle quote (6 lines)
> 2. Mercurial/SWH bridge. While SWH has a one-to-one mapping with
> Git (you can ask it for a specific Git commit ID), that’s not true for
> hg. This is a more general problem, but as things are today,
> there’s no automatic SWH fallback if the upstream hg server
> vanishes.

For git, I know and appreciate that the linter can directly add a
missing repo to SWH. During linting, I recogniced this is missing for
hg.

I was thinking a second time about it and found that not only the newer
development of OpenJDK is on GitHub, but also the older versions are
available. So I could add another patch like this:

+ (method git-fetch)
+ (uri (git-reference

WDYT?

Björn
-----BEGIN PGP SIGNATURE-----

iF0EAREKAB0WIQQiGUP0np8nb5SZM4K/KGy2WT5f/QUCZBTlVwAKCRC/KGy2WT5f
/UTFAKC6Nbe4Uc4F71mhNsJySzonXCJzfgCeL003G2uF0s1Kw+ePFz+jFSIv7o0=
=bIQP
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 20 Mar 2023 10:08
(name . Björn Höfling)(address . bjoern.hoefling@bjoernhoefling.de)
871qljpz71.fsf@inria.fr
Hello,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

Toggle quote (26 lines)
> On Thu, 16 Mar 2023 12:48:19 +0100
> Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> Hi Björn,
>>
>> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
>>
>> > I will check the same for JDK10 and will prepare a patch within the
>> > next two days.
>>
>> Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
>> commit!
>>
>> For the record, there are two remaining issues:
>>
>> 1. Reproducibility of past revisions. If we lose copies of the
>> auto-generated tarballs, then OpenJDK in past revisions of Guix
>> is irreparably lost. We should check whether/how to get them in
>> Disarchive + SWH.
>
> How do we do that? Adding git repos to SWH is something I can achieve,
> but adding tarballs to SWH was always opaque to me.
>
> I find no reference in the manual about Disarchive. Ideally, is there a
> linter for just adding a package to the disarchive database?

SWH periodically ingests the contents of tarballs (not tarballs
themselves) that appear in https://guix.gnu.org/sources.json. We’d
need to check whether it has the contents of those tarballs.

Then https://disarchive.guix.gnu.org is populated by the CI job at

Are the openjdk 9 and 10 tarballs archived?

Let’s look at their origins as of commit
1e6ddceb8318d413745ca1c9d91fde01b1e0364b. We can construct their
Disarchive URL by first converting their SHA256 to hex:

Toggle snippet (6 lines)
scheme@(guile-user)> ,use(guix base32)
scheme@(guile-user)> ,use(guix base16)
scheme@(guile-user)> (bytevector->base16-string (nix-base32-string->bytevector "01ihmyf7k5z17wbr7xig7y40l9f01d5zjgkcmawn1102hw5kchpq"))
$5 = "f842360b87028460b9aa6c3ef94b0bc0250a883f2ff693173fe197799caf3006"

That gives us:


Both are 404. But like I wrote, this is expected: they are bzip2
tarballs and Disarchive doesn’t support bzip2 (yet).

Toggle quote (20 lines)
>> 2. Mercurial/SWH bridge. While SWH has a one-to-one mapping with
>> Git (you can ask it for a specific Git commit ID), that’s not true for
>> hg. This is a more general problem, but as things are today,
>> there’s no automatic SWH fallback if the upstream hg server
>> vanishes.
>
> For git, I know and appreciate that the linter can directly add a
> missing repo to SWH. During linting, I recogniced this is missing for
> hg.
>
> I was thinking a second time about it and found that not only the newer
> development of OpenJDK is on GitHub, but also the older versions are
> available. So I could add another patch like this:
>
> + (method git-fetch)
> + (uri (git-reference
> + (url "https://github.com/openjdk/jdk9")
>
> WDYT?

That’s a good idea. It shouldn’t change the SHA256 (if it does,
something’s wrong) so it looks like an no-brainer.

Thanks!

Ludo’.
S
S
Simon Tournier wrote on 3 Apr 2023 23:52
86h6twy6of.fsf@gmail.com
Hi,

On Mon, 20 Mar 2023 at 10:08, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (11 lines)
>> I was thinking a second time about it and found that not only the newer
>> development of OpenJDK is on GitHub, but also the older versions are
>> available. So I could add another patch like this:
>>
>> + (method git-fetch)
>> + (uri (git-reference
>> + (url "https://github.com/openjdk/jdk9")
>
> That’s a good idea. It shouldn’t change the SHA256 (if it does,
> something’s wrong) so it looks like an no-brainer.

Well, by experience, it is rare that the released tarball contain the
exact same content as the tagged commit. If it is the case (same SHA256
for both tarball and GitHub tagged release), indeed no-brainer. :-)

Else, some work still remain for the complete preservation of Guix. ;-)

Björn, what is the status of this SHA256?

Cheers,
simon
?