icedtea-3 binaries contain references to icedtea-2

  • Done
  • quality assurance status badge
Details
8 participants
  • Andreas Enge
  • Björn Höfling
  • Gábor Boskovits
  • Carlo Zancanaro
  • Leo Famulari
  • Ludovic Courtès
  • Ricardo Wurmus
  • Ricardo Wurmus
Owner
unassigned
Submitted by
Ricardo Wurmus
Severity
important
R
R
Ricardo Wurmus wrote on 5 Jun 2018 12:47
(address . bug-guix@gnu.org)(name . Gábor Boskovits)(address . boskovits@gmail.com)
877end1pda.fsf@mdc-berlin.de
Many binaries of the icedtea-3 package contain references to icedtea-2,
which was used to build it. Because of these references users of
icedtea-3 have to *also* download icedtea-2. And because icedtea-2
contains references to icedtea-1, they *also* need to download that
version.

It is possible that icedtea-1 contains references to the bootstrap JDK,
which will then also have to be downloaded.

We should figure out a way to remove all of these references.

--
Ricardo
G
G
Gábor Boskovits wrote on 5 Jun 2018 13:49
CAE4v=pgmd_B9UBEVT+DQ7Ypd7ZdeLcT7zZQJh-C1VuDc5cCBLg@mail.gmail.com
Do you think we can do something like rebuild icedtea3 with icedtea3, and
then rewrite the references? I guess this way the dependency loop could be
broken, as we could detach the last icedtea3 build from the packages used
for bootstrapping.
We could even make on more rebuilding, if necessary. WDYT?
Attachment: file
L
L
Leo Famulari wrote on 5 Jun 2018 14:45
(name . Gábor Boskovits)(address . boskovits@gmail.com)
20180605124503.GA25671@jasmine.lan
On Tue, Jun 05, 2018 at 01:49:12PM +0200, Gábor Boskovits wrote:
Toggle quote (5 lines)
> Do you think we can do something like rebuild icedtea3 with icedtea3, and
> then rewrite the references? I guess this way the dependency loop could be
> broken, as we could detach the last icedtea3 build from the packages used
> for bootstrapping.

If the references are not important (that is, if they are not used), we
could also rewrite them to something that doesn't exist. Currently the
go-build-system does this so that Go executables don't refer to the Go
compiler, which is very large.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlsWhcwACgkQJkb6MLrK
fwgy/RAAiRv340ZZLLnqLVIlBxo/ly8+AyI+YE2TOHJ/Fg1uIz1hVnfDuzKaWcoX
r9qWkGmKrAP3Sg7mz4b2+fwYTctJ5Fg/IQd0JxQFu/Wv9FqeJsgX2rb7PQNPB2+I
q4bt7H5wTiWitYfv1T+MOAhU8GbsSAba82LwW7m1OasvZFrrV/VDxo74+qVdYXSK
dIlhTC7xHkFNfcgIEMEfw8mzo6ycZhdMdOdDfoNs1VWI8gQJmo5ZrvtuDVBeoKX7
h6PWQefr+KnKyL6B+WNAw2S1jueU6jTuX4o2GhpBsrLxFWC/+yzF8WNaGcwiFXu5
YyzPb1dlCn53nZNMh0Dmm9Sk7vxecc1ranTrY7m4zLHsY1em7rI6rrYALWOSvqGM
+g0jvE01Bsrq6pKfrTm7U/vbeKC9S4p8FPFoDQyAGYCN609uI2pWD61pPX4xEU8R
0mxWng/SLlnmGzK1D0i7Ke5du9lp9uw7hjvbCpvfKSEuOAUp+z1TcyKtUVGrzrJl
ejXs6qK5z8E//o3US1TiW0ewtGaCISIy3QItrFw51kyiN74FXMxKQ0PDWGPAtVow
W7uCtxrJvxeF25OH7y9GlmprcAsa3ndGEZRNfJ4socYQOXypnFkp8wNJK2qyo+ax
z/raR82kqLBoPHchkdcJrp0zdy3h0Nbo2AfnYsgJYKHvl3c4GYc=
=Ii/X
-----END PGP SIGNATURE-----


G
G
Gábor Boskovits wrote on 5 Jun 2018 16:45
(name . Leo Famulari)(address . leo@famulari.name)
CAE4v=phb6TALdVasVFTYEN-rjMowzMK8_Lk1aq-+JtadHZOqbg@mail.gmail.com
2018-06-05 14:45 GMT+02:00 Leo Famulari <leo@famulari.name>:

Toggle quote (13 lines)
> On Tue, Jun 05, 2018 at 01:49:12PM +0200, Gábor Boskovits wrote:
> > Do you think we can do something like rebuild icedtea3 with icedtea3, and
> > then rewrite the references? I guess this way the dependency loop could
> be
> > broken, as we could detach the last icedtea3 build from the packages used
> > for bootstrapping.
>
> If the references are not important (that is, if they are not used), we
> could also rewrite them to something that doesn't exist. Currently the
> go-build-system does this so that Go executables don't refer to the Go
> compiler, which is very large.
>

Ok, I will have a look at his, and try to isolate these references. This
looks like a
good first step to evaluate what can be done next.
Attachment: file
R
R
Ricardo Wurmus wrote on 5 Jun 2018 14:36
Re: icedtea-3 binaries contain references to icedtea-2
(name . Gábor Boskovits)(address . boskovits@gmail.com)(address . 31719@debbugs.gnu.org)
871sdl1kbg.fsf@elephly.net
Gábor Boskovits <boskovits@gmail.com> writes:

Toggle quote (6 lines)
> Do you think we can do something like rebuild icedtea3 with icedtea3, and
> then rewrite the references? I guess this way the dependency loop could be
> broken, as we could detach the last icedtea3 build from the packages used
> for bootstrapping.
> We could even make on more rebuilding, if necessary. WDYT?

Yes, that’s an option, but I’m not sure it is the best option. Icedtea
builds already take a very long time. Rebuilding the JDK after the
“build” phase would almost double the build time.

I would be happier if we could prevent references to the bootstrap JDK
from ending up in the binaries. Before we can do that we need to
understand why they are there. Are they necessary or are they just
accidental?

--
Ricardo
G
G
Gábor Boskovits wrote on 10 Sep 2018 23:00
icedtea-3 binaries contain references to icedtea-2
CAE4v=phPpvTzwTKQAUzWnYZDyNLVvcYWkk1ueLAuqsk7fuvpmw@mail.gmail.com
It seems that this is caused by having the files in a directory that is not
stripped.
I will try to have another look at that.
Attachment: file
R
R
Ricardo Wurmus wrote on 14 May 2020 20:02
(address . 31719@debbugs.gnu.org)(name . Gábor Boskovits)(address . boskovits@gmail.com)
87eermied9.fsf@elephly.net
This seems to affect the openjdk packages as well, so a user of OpenJDK
12 will have to download *all* JDKs.

Gábor, have you been able to identify locations that retain references
to the build JDK?

--
Ricardo
A
A
Andreas Enge wrote on 27 Feb 2021 12:17
Chains of dependencies getting longer
(address . 31719@debbugs.gnu.org)
YDoqWNwgubzB+5Fb@jurong
Hello,

trying a "guix build openjdk", I see this:
2136.0 MB would be downloaded:
/gnu/store/243algr6h60j46spn5dqhjc4mhkd0a0p-libelf-0.8.13
/gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
/gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
/gnu/store/52bp22jrvwfqa5fmlv2y5vxcir74m8jb-openjdk-10.46-jdk
/gnu/store/bp86qd2vgaqahqam3n89xl1wz02g0srs-openjdk-10.46
/gnu/store/a8f5ksk3gb4i1wrn7m5fm5ijc0zwwg2w-openjdk-11.28-jdk
/gnu/store/9ni61z4nhvkrxbzj6gwip8zrb7zbkx6l-openjdk-11.28
/gnu/store/837pl2vmq9haiw3kmsnh72fkwvcljipk-openjdk-12.33-jdk
/gnu/store/ll5dgf42x9smclrc1mx4gyk3dl683m77-openjdk-13.0-jdk
/gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
/gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
/gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0

So the problem is getting worse and worse over time, which makes a solution
more important.

Andreas
L
L
Ludovic Courtès wrote on 1 Mar 2021 11:04
(name . Andreas Enge)(address . andreas@enge.fr)(address . 31719@debbugs.gnu.org)
87mtvntckt.fsf@gnu.org
Hi,

Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (15 lines)
> trying a "guix build openjdk", I see this:
> 2136.0 MB would be downloaded:
> /gnu/store/243algr6h60j46spn5dqhjc4mhkd0a0p-libelf-0.8.13
> /gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
> /gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
> /gnu/store/52bp22jrvwfqa5fmlv2y5vxcir74m8jb-openjdk-10.46-jdk
> /gnu/store/bp86qd2vgaqahqam3n89xl1wz02g0srs-openjdk-10.46
> /gnu/store/a8f5ksk3gb4i1wrn7m5fm5ijc0zwwg2w-openjdk-11.28-jdk
> /gnu/store/9ni61z4nhvkrxbzj6gwip8zrb7zbkx6l-openjdk-11.28
> /gnu/store/837pl2vmq9haiw3kmsnh72fkwvcljipk-openjdk-12.33-jdk
> /gnu/store/ll5dgf42x9smclrc1mx4gyk3dl683m77-openjdk-13.0-jdk
> /gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
> /gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
> /gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0

Weird. I see this:

Toggle snippet (20 lines)
$ guix build openjdk -n
369.8 MB would be downloaded:
/gnu/store/676kcvxnnjfcy960fy4xkr10fbn3ciay-openjdk-14.0-doc
/gnu/store/dk8rkrfkdgsmy1a33n0sc39fhqggy1nh-openjdk-14.0-jdk
/gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0
$ guix build openjdk@13 -n
358.0 MB would be downloaded:
/gnu/store/hipy0g8zpnicqcwhx5ygx9fmmkbgbw6w-openjdk-13.0-doc
/gnu/store/ff9nn7ipbpnrrvsg0djdrzm0a8v1jx5j-openjdk-13.0-jdk
/gnu/store/nb6p8aqjjw923vwd7safbh0w5q3mr9fq-openjdk-13.0
$ guix size openjdk@14 | grep openjdk
/gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0 389.3 295.8 76.0%
$ guix describe
Generacio 175 Feb 04 2021 22:52:40 (nuna)
guix 5ae09d7
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 5ae09d7979a0696d862b9555314eab199f7ce576

What this shows is that the compiled openjdk 14 does not depend on
previous versions; likewise for version 13.

So the long dependency chain is a built-time-only dependency chain.
It’s a problem when building from source but not when using substitutes.

Ludo’.
A
A
Andreas Enge wrote on 1 Mar 2021 13:00
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 31719@debbugs.gnu.org)
YDzXeFFnGenPdZfn@jurong
Hello,

Am Mon, Mar 01, 2021 at 11:04:02AM +0100 schrieb Ludovic Courtï¿œs:
Toggle quote (3 lines)
> Andreas Enge <andreas@enge.fr> skribis:
> > trying a "guix build openjdk", I see this:
> > 2136.0 MB would be downloaded
...
Toggle quote (2 lines)
> > /gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
> > /gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
...
Toggle quote (13 lines)
> > /gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
> > /gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
> > /gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0
>
> Weird. I see this:
>
> --8<---------------cut here---------------start------------->8---
> $ guix build openjdk -n
> 369.8 MB would be downloaded:
> /gnu/store/676kcvxnnjfcy960fy4xkr10fbn3ciay-openjdk-14.0-doc
> /gnu/store/dk8rkrfkdgsmy1a33n0sc39fhqggy1nh-openjdk-14.0-jdk
> /gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0

It is strange we do not see the same thing! I just tried it again, and am
getting the same result as above. Notice that openjdk@14 is available as a
substitute, but still all previous versions are downloaded, while nothing
is built. The same with the most recent master commit (7ca43b0a1e).

Andreas
L
L
Ludovic Courtès wrote on 1 Mar 2021 22:33
(name . Andreas Enge)(address . andreas@enge.fr)
87o8g2pni4.fsf@gnu.org
Hi!

Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (26 lines)
> Am Mon, Mar 01, 2021 at 11:04:02AM +0100 schrieb Ludovic Courtès:
>> Andreas Enge <andreas@enge.fr> skribis:
>> > trying a "guix build openjdk", I see this:
>> > 2136.0 MB would be downloaded
> ...
>> > /gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
>> > /gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
> ...
>> > /gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
>> > /gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
>> > /gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0
>>
>> Weird. I see this:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ guix build openjdk -n
>> 369.8 MB would be downloaded:
>> /gnu/store/676kcvxnnjfcy960fy4xkr10fbn3ciay-openjdk-14.0-doc
>> /gnu/store/dk8rkrfkdgsmy1a33n0sc39fhqggy1nh-openjdk-14.0-jdk
>> /gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0
>
> It is strange we do not see the same thing! I just tried it again, and am
> getting the same result as above. Notice that openjdk@14 is available as a
> substitute, but still all previous versions are downloaded, while nothing
> is built.

Wait, with a newer commit, I get the whole chain as well:

Toggle snippet (21 lines)
$ guix build openjdk -n
2,135.0 MB would be downloaded:
/gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
/gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
/gnu/store/52bp22jrvwfqa5fmlv2y5vxcir74m8jb-openjdk-10.46-jdk
/gnu/store/bp86qd2vgaqahqam3n89xl1wz02g0srs-openjdk-10.46
/gnu/store/a8f5ksk3gb4i1wrn7m5fm5ijc0zwwg2w-openjdk-11.28-jdk
/gnu/store/9ni61z4nhvkrxbzj6gwip8zrb7zbkx6l-openjdk-11.28
/gnu/store/837pl2vmq9haiw3kmsnh72fkwvcljipk-openjdk-12.33-jdk
/gnu/store/ll5dgf42x9smclrc1mx4gyk3dl683m77-openjdk-13.0-jdk
/gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
/gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
/gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0
$ guix describe
Generacio 176 Mar 01 2021 11:39:49 (nuna)
guix 7ca43b0
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 7ca43b0a1e2215abe0df0708f31decace8e68911

But again, from that Feb. 4th commit, everything looks fine:

Toggle snippet (7 lines)
$ guix time-machine --commit=5ae09d7979a0696d862b9555314eab199f7ce576 -- build openjdk -n
369.8 MB would be downloaded:
/gnu/store/676kcvxnnjfcy960fy4xkr10fbn3ciay-openjdk-14.0-doc
/gnu/store/dk8rkrfkdgsmy1a33n0sc39fhqggy1nh-openjdk-14.0-jdk
/gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0

It must be a recent regression. I think this is an unintended
side-effect of 44425e1f2a96aee39a21dff634abb9b77b44261e and
84805ef205b7fa12bfaa7b23da06993cab64c40b (see

Toggle snippet (15 lines)
$ guix time-machine --commit=44425e1f2a96aee39a21dff634abb9b77b44261e -- build openjdk -n
2,135.9 MB would be downloaded:
/gnu/store/wd9jyzy7i2dxbxahc0pa8mq22any08zx-openjdk-9.181-jdk
/gnu/store/7zxhm2xmyqds8hvlhifi9d16p9ln9k4b-openjdk-9.181
/gnu/store/j5sh6c7qhm6vis6fvjyxs2jch9dhz085-openjdk-10.46-jdk
/gnu/store/7w3vd4y0jac7d74gw62qmfv7br11k64r-openjdk-10.46
/gnu/store/0sb9z8ds8ly7pwmviaqixq9wq1rp13g5-openjdk-11.28-jdk
/gnu/store/4nf35d4v363bxrfmp85sk671swzmbj5k-openjdk-11.28
/gnu/store/iwi28gcfxrsq541367m4hnp44jnnrkdq-openjdk-12.33-jdk
/gnu/store/2w5gg7v9sy54s1jfnymgvg6qwqnkzr8h-openjdk-13.0-jdk
/gnu/store/sgqw343zlqqn4vpdq4v9m4jy6ar9i99f-openjdk-14.0-doc
/gnu/store/8l4pb8rh9zbqjn14ipbps7nn8wd0jg4w-openjdk-14.0-jdk
/gnu/store/1d5bzdzrir8k13bkclx3a9cq59gw5j4h-openjdk-14.0

Björn, WDYT? I suppose we need to filter out some library names in
‘patch-jni-libs’?

While at it, we should add a #:disallowed-references keyword to prevent
that from popping up in the future.

Thanks,
Ludo’.
B
B
Björn Höfling wrote on 1 Mar 2021 23:15
(name . Ludovic Courtès)(address . ludo@gnu.org)
20210301231534.1d4e440a@alma-ubu.fritz.box
On Mon, 01 Mar 2021 22:33:55 +0100
Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (24 lines)
> It must be a recent regression. I think this is an unintended
> side-effect of 44425e1f2a96aee39a21dff634abb9b77b44261e and
> 84805ef205b7fa12bfaa7b23da06993cab64c40b (see
> <https://issues.guix.gnu.org/41177>):
>
> --8<---------------cut here---------------start------------->8---
> $ guix time-machine --commit=44425e1f2a96aee39a21dff634abb9b77b44261e
> -- build openjdk -n 2,135.9 MB would be downloaded:
> /gnu/store/wd9jyzy7i2dxbxahc0pa8mq22any08zx-openjdk-9.181-jdk
> /gnu/store/7zxhm2xmyqds8hvlhifi9d16p9ln9k4b-openjdk-9.181
> /gnu/store/j5sh6c7qhm6vis6fvjyxs2jch9dhz085-openjdk-10.46-jdk
> /gnu/store/7w3vd4y0jac7d74gw62qmfv7br11k64r-openjdk-10.46
> /gnu/store/0sb9z8ds8ly7pwmviaqixq9wq1rp13g5-openjdk-11.28-jdk
> /gnu/store/4nf35d4v363bxrfmp85sk671swzmbj5k-openjdk-11.28
> /gnu/store/iwi28gcfxrsq541367m4hnp44jnnrkdq-openjdk-12.33-jdk
> /gnu/store/2w5gg7v9sy54s1jfnymgvg6qwqnkzr8h-openjdk-13.0-jdk
> /gnu/store/sgqw343zlqqn4vpdq4v9m4jy6ar9i99f-openjdk-14.0-doc
> /gnu/store/8l4pb8rh9zbqjn14ipbps7nn8wd0jg4w-openjdk-14.0-jdk
> /gnu/store/1d5bzdzrir8k13bkclx3a9cq59gw5j4h-openjdk-14.0
> --8<---------------cut here---------------end--------------->8---
>
> Björn, WDYT? I suppose we need to filter out some library names in
> ‘patch-jni-libs’?

Ouch. While installing a JDK last week, I had the same feeling that the
dependency chain down was a bit too long, but then forgot to look at it
again.

When I read today Andreas' email, it popped up again ... I will have a
look at it.

Toggle quote (3 lines)
> While at it, we should add a #:disallowed-references keyword to
> prevent that from popping up in the future.

I wasn't aware of this and will also have a look.

Björn
-----BEGIN PGP SIGNATURE-----

iF0EAREKAB0WIQQiGUP0np8nb5SZM4K/KGy2WT5f/QUCYD1nhgAKCRC/KGy2WT5f
/YRuAJ0d5nbr6U170CMdH+LcHsMeDUz7pQCeOteabX6suZttApKF2t3z4CW8DsM=
=JLny
-----END PGP SIGNATURE-----


R
R
Ricardo Wurmus wrote on 14 Apr 2021 09:36
(name . Björn Höfling)(address . bjoern.hoefling@bjoernhoefling.de)
878s5ltj0o.fsf@elephly.net
Hi Björn,

Toggle quote (46 lines)
> On Mon, 01 Mar 2021 22:33:55 +0100
> Ludovic Courtès <ludo@gnu.org> wrote:
>
>> It must be a recent regression. I think this is an unintended
>> side-effect of 44425e1f2a96aee39a21dff634abb9b77b44261e and
>> 84805ef205b7fa12bfaa7b23da06993cab64c40b (see
>> <https://issues.guix.gnu.org/41177>):
>>
>> --8<---------------cut
>> here---------------start------------->8---
>> $ guix time-machine
>> --commit=44425e1f2a96aee39a21dff634abb9b77b44261e
>> -- build openjdk -n 2,135.9 MB would be downloaded:
>> /gnu/store/wd9jyzy7i2dxbxahc0pa8mq22any08zx-openjdk-9.181-jdk
>> /gnu/store/7zxhm2xmyqds8hvlhifi9d16p9ln9k4b-openjdk-9.181
>> /gnu/store/j5sh6c7qhm6vis6fvjyxs2jch9dhz085-openjdk-10.46-jdk
>> /gnu/store/7w3vd4y0jac7d74gw62qmfv7br11k64r-openjdk-10.46
>> /gnu/store/0sb9z8ds8ly7pwmviaqixq9wq1rp13g5-openjdk-11.28-jdk
>> /gnu/store/4nf35d4v363bxrfmp85sk671swzmbj5k-openjdk-11.28
>> /gnu/store/iwi28gcfxrsq541367m4hnp44jnnrkdq-openjdk-12.33-jdk
>> /gnu/store/2w5gg7v9sy54s1jfnymgvg6qwqnkzr8h-openjdk-13.0-jdk
>> /gnu/store/sgqw343zlqqn4vpdq4v9m4jy6ar9i99f-openjdk-14.0-doc
>> /gnu/store/8l4pb8rh9zbqjn14ipbps7nn8wd0jg4w-openjdk-14.0-jdk
>> /gnu/store/1d5bzdzrir8k13bkclx3a9cq59gw5j4h-openjdk-14.0
>> --8<---------------cut
>> here---------------end--------------->8---
>>
>> Björn, WDYT? I suppose we need to filter out some library
>> names in
>> ‘patch-jni-libs’?
>
> Ouch. While installing a JDK last week, I had the same feeling
> that the
> dependency chain down was a bit too long, but then forgot to
> look at it
> again.
>
> When I read today Andreas' email, it popped up again ... I will
> have a
> look at it.
>
>> While at it, we should add a #:disallowed-references keyword to
>> prevent that from popping up in the future.
>
> I wasn't aware of this and will also have a look.

Have you been able to identify what’s wrong with the JDK chain?
I’d like us to drop these extra references soon so that they don’t
appear in the upcoming release.

--
Ricardo
B
B
Björn Höfling wrote on 16 Apr 2021 21:22
(name . Ricardo Wurmus)(address . rekado@elephly.net)
20210416212233.05ac87de@alma-ubu.fritz.box
Hi Ricardo,

On Wed, 14 Apr 2021 09:36:23 +0200
Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (4 lines)
> Have you been able to identify what’s wrong with the JDK chain?
> I’d like us to drop these extra references soon so that they don’t
> appear in the upcoming release.

Sorry, I currently don't find the time to look at the problem. Would
someone else give it a try?

Björn
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQQiGUP0np8nb5SZM4K/KGy2WT5f/QUCYHnj+QAKCRC/KGy2WT5f
/T1rAKConJ8oQrNtdYKrWVVdU/n7NEMKNgCePC4yNna3TXLgWdFKfcMCRfzX2tc=
=EFoc
-----END PGP SIGNATURE-----


C
C
Carlo Zancanaro wrote on 17 Apr 2021 00:26
(name . Björn Höfling)(address . bjoern.hoefling@bjoernhoefling.de)
87fszpc1du.fsf@zancanaro.id.au
On Sat, Apr 17 2021, Björn Höfling wrote:
Toggle quote (3 lines)
> Sorry, I currently don't find the time to look at the problem.
> Would someone else give it a try?

I just had a quick look, and I think we can resolve it by changing
the patch-jni-libs to not rewrite references to "mlib_image" and
"splashscreen". It looks like they're provided as part of the JVM
built itself, but our build phase hard codes a reference to the
previous JVM's build result instead of using the current JVM's
build result. They're the only dependencies I've found so far, but
I've only looked at openjdk10 and openjdk14.

I'll put together a patch later today, if I haven't been beaten to
it by then.

Carlo
C
C
Carlo Zancanaro wrote on 17 Apr 2021 09:11
87czutbd2f.fsf@zancanaro.id.au
Here's a patch that should clean up these runtime dependencies.
It's a bit specific to this particular case, but I think that
might be fine for now.

I think it would make more sense for native inputs to not have
their paths included in LIBRARY_PATH. Does it even make sense for
them to be there? I thought LIBRARY_PATH was for compilers to find
dependencies when compiling so they can link their output binaries
against them. Having native inputs show up there seems wrong.

I'm in the process of rebuilding Java from icedtea-8 upwards to
check, but I have already tested that modifying openjdk 9 and 10
leads to "guix gc --references" show that openjdk 10 does not
depend on openjdk 9. I have also tested that I can run some
complex Java programs on my machine using the openjdk 10 built
using this patch.

Carlo
From f98dc5ad5662cc62f198d8f50e7dd719cf941315 Mon Sep 17 00:00:00 2001
From: Carlo Zancanaro <carlo@zancanaro.id.au>
Date: Sat, 17 Apr 2021 16:33:06 +1000
Subject: [PATCH] gnu: Clean up runtime dependencies between Java versions.

* gnu/packages/java.scm (icedtea-8, openjdk9, openjdk11): Don't consider
icedtea/openjdk input paths when rewriting JNI libraries.
---
gnu/packages/java.scm | 36 ++++++++++++++++++++++++++----------
1 file changed, 26 insertions(+), 10 deletions(-)

Toggle diff (71 lines)
diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 207f136513..3c4013ab6f 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -2,7 +2,7 @@
;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2016 Leo Famulari <leo@famulari.name>
;;; Copyright © 2016, 2017 Roel Janssen <roel@gnu.org>
-;;; Copyright © 2017, 2019 Carlo Zancanaro <carlo@zancanaro.id.au>
+;;; Copyright © 2017, 2019, 2021 Carlo Zancanaro <carlo@zancanaro.id.au>
;;; Copyright © 2017-2020 Julien Lepiller <julien@lepiller.eu>
;;; Copyright © 2017 Thomas Danckaert <post@thomasdanckaert.be>
;;; Copyright © 2016, 2017, 2018 Alex Vong <alexvong1995@gmail.com>
@@ -1792,8 +1792,13 @@ new Date();"))
(add-after 'unpack 'patch-jni-libs
;; Hardcode dynamically loaded libraries.
(lambda _
- (let* ((library-path (search-path-as-string->list
- (getenv "LIBRARY_PATH")))
+ (use-modules (srfi srfi-1))
+ (define (icedtea-or-openjdk? path)
+ (or (string-contains path "openjdk")
+ (string-contains path "icedtea")))
+ (let* ((library-path (remove icedtea-or-openjdk?
+ (search-path-as-string->list
+ (getenv "LIBRARY_PATH"))))
(find-library (lambda (name)
(search-path
library-path
@@ -1931,12 +1936,18 @@ new Date();"))
(add-after 'unpack 'patch-jni-libs
;; Hardcode dynamically loaded libraries.
(lambda _
- (let* ((library-path (search-path-as-string->list
- (getenv "LIBRARY_PATH")))
+ (use-modules (srfi srfi-1))
+ (define (icedtea-or-openjdk? path)
+ (or (string-contains path "openjdk")
+ (string-contains path "icedtea")))
+ (let* ((library-path (remove icedtea-or-openjdk?
+ (search-path-as-string->list
+ (getenv "LIBRARY_PATH"))))
(find-library (lambda (name)
- (search-path
- library-path
- (string-append "lib" name ".so")))))
+ (or (search-path
+ library-path
+ (string-append "lib" name ".so"))
+ (string-append "lib" name ".so")))))
(for-each
(lambda (file)
(catch 'decoding-error
@@ -2139,8 +2150,13 @@ new Date();"))
(add-after 'unpack 'patch-jni-libs
;; Hardcode dynamically loaded libraries.
(lambda _
- (let* ((library-path (search-path-as-string->list
- (getenv "LIBRARY_PATH")))
+ (use-modules (srfi srfi-1))
+ (define (icedtea-or-openjdk? path)
+ (or (string-contains path "openjdk")
+ (string-contains path "icedtea")))
+ (let* ((library-path (remove icedtea-or-openjdk?
+ (search-path-as-string->list
+ (getenv "LIBRARY_PATH"))))
(find-library (lambda (name)
(search-path
library-path
--
2.31.1
C
C
Carlo Zancanaro wrote on 17 Apr 2021 11:38
87a6pxb69o.fsf@zancanaro.id.au
On Sat, Apr 17 2021, Carlo Zancanaro wrote:
Toggle quote (3 lines)
> I'm in the process of rebuilding Java from icedtea-8 upwards to
> check,

I've now built and checked the JRE/JDKs from 10 to 14, and none of
them retain a reference to any other JRE/JDK according to "guix gc
--references".

Carlo
A
A
Andreas Enge wrote on 19 Apr 2021 18:22
(name . Carlo Zancanaro)(address . carlo@zancanaro.id.au)
YH2uQKCqLAmBiPE3@jurong
Hello Carlo,

Am Sat, Apr 17, 2021 at 07:38:11PM +1000 schrieb Carlo Zancanaro:
Toggle quote (6 lines)
> On Sat, Apr 17 2021, Carlo Zancanaro wrote:
> > I'm in the process of rebuilding Java from icedtea-8 upwards to check,
>
> I've now built and checked the JRE/JDKs from 10 to 14, and none of them
> retain a reference to any other JRE/JDK according to "guix gc --references".

thanks a lot for your patch! I am not very knowledgeable on this matter, but
after updating my system am right now once again bitten by the problem:
The following derivations would be built:
/gnu/store/0bxpgqps79007jky37dwzgm08wlsgj02-openjdk-10.46.drv
/gnu/store/2xwsm2plrbk7l4510hrqp9y7b5vv9v5j-module-import-compiled.drv
1683.5 MB would be downloaded:
/gnu/store/0gzv5208m2prqbnsqdffcnz7mwfqa684-openjdk-10.46.tar.xz
/gnu/store/5g6ng9a2ifgrv57mzdaysf2lq9rkixxk-make-4.2.1
/gnu/store/243algr6h60j46spn5dqhjc4mhkd0a0p-libelf-0.8.13
/gnu/store/crc4cgsdls10isy8prjdddvsp21wafx6-openjdk-9.181-jdk
/gnu/store/6amc3php7qyqxagak5xnmv2k81wm7w26-openjdk-9.181
/gnu/store/bms6jg5qwn927qmbh9xi30ifhsxxdazq-openjdk-11.28-jdk
/gnu/store/lylv6ng5gwqpi930c6y7sglh2k8byjn6-openjdk-11.28
/gnu/store/z27mhqpylrbmwkcgrb100p6jbflxhq5h-openjdk-12.33-jdk
/gnu/store/xz500ry11dihwn9kij1kmzkci1lmnqjf-openjdk-13.0-jdk
/gnu/store/a1s66hlj10nkkcxlk6s2dzq4iip4mh4k-openjdk-14.0-doc
/gnu/store/cyssia0a5lh8mb5czd7155y7sl31aryl-openjdk-14.0-jdk
/gnu/store/630lvja8vak1bkf9vsbbh29cp79j4pwp-openjdk-14.0
guix build: warning: at least 5922.4 MB needed but only 3539.4 MB available in /gnu/store

So I would have to download lots of openjdk and even build one in the middle
of the chain myself.

So unless someone else objects, I intend to apply the patch, build the
latest openjdk, and push it if everything goes well.

Notice that the original problem reported by Ricardo seems to have been
fixed independently - "guix build icedtea" only downloads icedtea@3, and
not @2 on current master.

Thanks,

Andreas
L
L
Ludovic Courtès wrote on 19 Apr 2021 18:45
control message for bug #31719
(address . control@debbugs.gnu.org)
87v98idxz3.fsf@gnu.org
severity 31719 important
quit
L
L
Ludovic Courtès wrote on 19 Apr 2021 18:46
control message for bug #47297
(address . control@debbugs.gnu.org)
87tuo2dxyp.fsf@gnu.org
block 47297 by 31719
quit
R
R
Ricardo Wurmus wrote on 19 Apr 2021 20:18
Re: bug#31719: Chains of dependencies getting longer
(name . Andreas Enge)(address . andreas@enge.fr)
87wnsym944.fsf@elephly.net
Andreas Enge <andreas@enge.fr> writes:

Toggle quote (4 lines)
> So unless someone else objects, I intend to apply the patch,
> build the
> latest openjdk, and push it if everything goes well.

Sounds good.

I just looked over the patch, and while I’m not sure it’s the best
way to do things (matching “openjdk” or “icedtea” in the package
name seems a little error prone in the presence of packages whose
names might include these strings), but I think it’s a definite
improvement.

Thank you Carlo!

--
Ricardo
A
A
Andreas Enge wrote on 20 Apr 2021 10:34
(name . Ricardo Wurmus)(address . rekado@elephly.net)
YH6SCaSNrRiqocqT@jurong
Hello,

Am Mon, Apr 19, 2021 at 08:18:03PM +0200 schrieb Ricardo Wurmus:
Toggle quote (5 lines)
> I just looked over the patch, and while I’m not sure it’s the best way to do
> things (matching “openjdk” or “icedtea” in the package name seems a little
> error prone in the presence of packages whose names might include these
> strings), but I think it’s a definite improvement.

I just pushed the patch to master. Thanks a lot, Carlo! It has definitely
solved my problem: I can now compile an Android project after downloading
a single openjdk package.

It would be nice if someone else could close the bug if you feel the problem
is solved, or otherwise leave it open to discuss further possible improvements.

Andreas
C
C
Carlo Zancanaro wrote on 20 Apr 2021 11:01
(name . Ricardo Wurmus)(address . rekado@elephly.net)(address . 31719@debbugs.gnu.org)
AEF2E3EB-DEFC-4877-A270-8A46975FA192@zancanaro.id.au
On 20 April 2021 4:18:03 am AEST, Ricardo Wurmus <rekado@elephly.net> wrote:
Toggle quote (2 lines)
>I just looked over the patch, and while I’m not sure it’s the best way to do things (matching “openjdk” or “icedtea” in the package name seems a little error prone in the presence of packages whose names might include these strings), but I think it’s a definite improvement.

Yeah, I'm agreed on that. I tried for a while to find a way to exclude native inputs, but as far as I could tell they all ended up as inputs within the build phases. I'd be happy to change it if somebody can give me a hint about how to pick out the native inputs.

Carlo
L
L
Ludovic Courtès wrote on 20 Apr 2021 11:24
Re: bug#31719: icedtea-3 binaries contain references to icedtea-2
(name . Andreas Enge)(address . andreas@enge.fr)
878s5de2ac.fsf_-_@gnu.org
Hi,

Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (13 lines)
> Am Mon, Apr 19, 2021 at 08:18:03PM +0200 schrieb Ricardo Wurmus:
>> I just looked over the patch, and while I’m not sure it’s the best way to do
>> things (matching “openjdk” or “icedtea” in the package name seems a little
>> error prone in the presence of packages whose names might include these
>> strings), but I think it’s a definite improvement.
>
> I just pushed the patch to master. Thanks a lot, Carlo! It has definitely
> solved my problem: I can now compile an Android project after downloading
> a single openjdk package.
>
> It would be nice if someone else could close the bug if you feel the problem
> is solved, or otherwise leave it open to discuss further possible improvements.

I think we can close it. I have the attached improvements that I can
commit to ‘staging’ (or ‘core-updates’?) to avoid another rebuild.

Thanks!

Ludo’.
From 253d0485a9307c4e08afc058d7dafcd56025f9a6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Tue, 20 Apr 2021 00:26:01 +0200
Subject: [PATCH] java fixlet

---
gnu/packages/java.scm | 35 +++++++++++++++++++++++++++--------
1 file changed, 27 insertions(+), 8 deletions(-)

Toggle diff (103 lines)
diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 3c4013ab6f..b780f7a85f 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -1749,6 +1749,9 @@ IcedTea build harness.")
((guix build ant-build-system)
(guix build syscalls)
,@%gnu-build-system-modules)
+
+ #:disallowed-references ((,icedtea-7 "jdk"))
+
,@(substitute-keyword-arguments (package-arguments icedtea-7)
((#:modules modules)
`((guix build utils)
@@ -1792,10 +1795,13 @@ new Date();"))
(add-after 'unpack 'patch-jni-libs
;; Hardcode dynamically loaded libraries.
(lambda _
- (use-modules (srfi srfi-1))
+ (define remove
+ (@ (srfi srfi-1) remove))
+
(define (icedtea-or-openjdk? path)
(or (string-contains path "openjdk")
(string-contains path "icedtea")))
+
(let* ((library-path (remove icedtea-or-openjdk?
(search-path-as-string->list
(getenv "LIBRARY_PATH"))))
@@ -1898,6 +1904,9 @@ new Date();"))
#:imported-modules
((guix build syscalls)
,@%gnu-build-system-modules)
+
+ #:disallowed-references (,icedtea-8 (,icedtea-8 "jdk"))
+
#:phases
(modify-phases %standard-phases
(add-after 'patch-source-shebangs 'fix-java-shebangs
@@ -1936,18 +1945,20 @@ new Date();"))
(add-after 'unpack 'patch-jni-libs
;; Hardcode dynamically loaded libraries.
(lambda _
- (use-modules (srfi srfi-1))
+ (define remove
+ (@ (srfi srfi-1) remove))
+
(define (icedtea-or-openjdk? path)
(or (string-contains path "openjdk")
(string-contains path "icedtea")))
+
(let* ((library-path (remove icedtea-or-openjdk?
(search-path-as-string->list
(getenv "LIBRARY_PATH"))))
(find-library (lambda (name)
- (or (search-path
- library-path
- (string-append "lib" name ".so"))
- (string-append "lib" name ".so")))))
+ (search-path
+ library-path
+ (string-append "lib" name ".so")))))
(for-each
(lambda (file)
(catch 'decoding-error
@@ -2090,7 +2101,9 @@ new Date();"))
"--with-libjpeg=system"
"--with-native-debug-symbols=zipped"
(string-append "--prefix=" (assoc-ref outputs "out")))
- #t))))))
+ #t))))
+ ((#:disallowed-references _ '())
+ `(,openjdk9 (,openjdk9 "jdk")))))
(native-inputs
`(("openjdk9" ,openjdk9)
("openjdk9:jdk" ,openjdk9 "jdk")
@@ -2120,6 +2133,9 @@ new Date();"))
(arguments
`(#:imported-modules ((guix build syscalls)
,@%gnu-build-system-modules)
+
+ #:disallowed-references (,openjdk10 (,openjdk10 "jdk"))
+
#:tests? #f; requires jtreg
;; TODO package jtreg
#:configure-flags
@@ -2150,10 +2166,13 @@ new Date();"))
(add-after 'unpack 'patch-jni-libs
;; Hardcode dynamically loaded libraries.
(lambda _
- (use-modules (srfi srfi-1))
+ (define remove
+ (@ (srfi srfi-1) remove))
+
(define (icedtea-or-openjdk? path)
(or (string-contains path "openjdk")
(string-contains path "icedtea")))
+
(let* ((library-path (remove icedtea-or-openjdk?
(search-path-as-string->list
(getenv "LIBRARY_PATH"))))
--
2.31.1
A
A
Andreas Enge wrote on 20 Apr 2021 11:36
(name . Ludovic Courtès)(address . ludo@gnu.org)
YH6giG5JWZAqBYiq@jurong
Am Tue, Apr 20, 2021 at 11:24:59AM +0200 schrieb Ludovic Courtès:
Toggle quote (3 lines)
> I think we can close it. I have the attached improvements that I can
> commit to ‘staging’ (or ‘core-updates’?) to avoid another rebuild.

I think they can easily go to master; the time for rebuilds is not very
high, I just ran into trouble since I had little space left on the device
so some manual intervention was required to gc intermediate packages.

There also seem to be very few dependencies if "guix refresh -l openjdk"
is to be trusted.

Andreas
C
C
Carlo Zancanaro wrote on 20 Apr 2021 13:32
(name . Ludovic Courtès)(address . ludo@gnu.org)
87tuo18a4l.fsf@zancanaro.id.au
Hi Ludo!

On Tue, Apr 20 2021, Ludovic Courtès wrote:
Toggle quote (13 lines)
> (find-library (lambda (name)
> - (or (search-path
> - library-path
> - (string-append "lib"
> name ".so"))
> - (string-append "lib"
> name ".so")))))
> + (search-path
> + library-path
> + (string-append "lib" name
> ".so")))))
> (for-each

As discussed on IRC, the "or" is actually important here to avoid
substituting #f as the library name. I've attached a patch on top
of yours that adds the "or" back (including the other two that I
missed in my earlier patch), and also switches to "string-append"
which is less sensitive to this problem.

I have built up to openjdk11 with this patch, and I see less #f's
in the result. There are still some in the compiled libraries, but
I haven't investigated thoroughly as to whether they're correct or
not.

Carlo
L
L
Ludovic Courtès wrote on 20 Apr 2021 18:38
(name . Carlo Zancanaro)(address . carlo@zancanaro.id.au)
87bla8c3mn.fsf@gnu.org
Hi Carlo,

Carlo Zancanaro <carlo@zancanaro.id.au> skribis:

Toggle quote (9 lines)
> From 60101b27543b7cc41a052d5bec95234ea4977d35 Mon Sep 17 00:00:00 2001
> From: Carlo Zancanaro <carlo@zancanaro.id.au>
> Date: Tue, 20 Apr 2021 21:22:20 +1000
> Subject: [PATCH] gnu: Fix openjdk library substitution when libraries aren't
> found
>
> * gnu/packages/java.scm (icedtea-8, openjdk9, openjdk11): Fix JNI library
> substitution to not substitute #f if the library can't be found.

Thanks for the update patch! Please mention ‘find-library’ in the
commit log, for clarity.

I think you can go and push it to master given that OpenJDK is probably
broken due to those #f right now. However, please push an additional
commit at the same time that adds those #:disallowed-references I
proposed. (Though you’ll have to rebuild openjdk@11 to be sure.)

Sounds good?

Ludo’.
C
C
Carlo Zancanaro wrote on 20 Apr 2021 23:00
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 31719@debbugs.gnu.org)
11712F58-FFF2-4C28-83F8-79F5CDE574EE@zancanaro.id.au
On 21 April 2021 2:38:56 am AEST, "Ludovic Courtès" <ludo@gnu.org> wrote:
Toggle quote (2 lines)
>Sounds good?

Sounds good, except I don't have commit privileges. ? Can someone else push it for me?
L
L
Ludovic Courtès wrote on 21 Apr 2021 14:40
(name . Carlo Zancanaro)(address . carlo@zancanaro.id.au)(address . 31719-done@debbugs.gnu.org)
875z0f7quv.fsf@gnu.org
Hi,

Carlo Zancanaro <carlo@zancanaro.id.au> skribis:

Toggle quote (5 lines)
> On 21 April 2021 2:38:56 am AEST, "Ludovic Courtès" <ludo@gnu.org> wrote:
>>Sounds good?
>
> Sounds good, except I don't have commit privileges. ? Can someone else push it for me?

Oops! I adjusted the patch so it would apply to ‘master’, and followed
up with two commits:

f8acd1aeef gnu: openjdk: Disallow references to the JDK used for build.
e511a1d327 gnu: openjdk: Avoid non-top-level 'use-modules'.
698c4365ba gnu: openjdk: Fix library substitution when libraries aren't found.

I rebuilt everything up to openjdk@11.

Thanks!

Ludo’.
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 31719@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 31719
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch