Go retains a reference to GCC

  • Open
  • quality assurance status badge
Details
2 participants
  • Sarah Morgensen
  • Ludovic Courtès
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 2 Feb 2020 23:42
(address . bug-Guix@gnu.org)
87v9oo391q.fsf@gnu.org
Hello!

It seems that Go unduly retains a reference to GCC:

Toggle snippet (30 lines)
$ guix size go
store item total self
/gnu/store/g4rrz9rnr8ssbnj3gx3dmsxv4glll8nf-go-1.12.15 646.3 355.9 55.1%
/gnu/store/x3jx25cd3q363mr7nbgzrhrv1vza6cf7-gcc-7.4.0 177.4 107.2 16.6%
/gnu/store/ziinjmbnq004866mwjrczsk12wf35qb8-perl-5.30.0 148.2 57.1 8.8%
/gnu/store/ahqgl4h89xqj695lgqvsaf6zh2nhy4pj-glibc-2.29 37.4 35.8 5.5%
/gnu/store/64xzk1kwk30n7arqgmxs2qy7wzcnc8i3-gcc-7.4.0-lib 70.1 32.8 5.1%
/gnu/store/2plcy91lypnbbysb18ymnhaw3zwk8pg1-gcc-7.4.0-lib 70.0 32.6 5.0%
/gnu/store/9kzrrccpzl6i1sfwb0drb00gi2gwk0x0-coreutils-8.31 90.0 16.5 2.6%
/gnu/store/dvs3acxwfnwgc7yma6h3y937ri2li47y-gmp-6.1.2 72.6 2.6 0.4%
/gnu/store/9mmsilz9avdl49i6a6nj5mzfyim8ihv2-tzdata-2019c 2.0 2.0 0.3%
/gnu/store/cp72ncw4prnsga65n3pzll07hpsg524f-bash-static-5.0.7 1.6 1.6 0.2%
/gnu/store/29jhbbg1hf557x8j53f9sxd9imlmf02a-bash-minimal-5.0.7 38.4 1.0 0.2%
/gnu/store/nffbgghxyvrj29lcgxs5fpmi3sx9zzql-acl-2.2.53 70.7 0.5 0.1%
/gnu/store/in1738m2zvhgpz78n2yqa972sdzc42ss-attr-2.4.48 70.3 0.3 0.0%
/gnu/store/y127kw0vmw46zcrvfz0i5s2rynq7ddjv-zlib-1.2.11 37.6 0.2 0.0%
/gnu/store/waw5ci4lazbf2a1x9v6gw1v274nk0gny-libcap-2.27 70.2 0.2 0.0%
/gnu/store/hzk6d08yiih0alc2raciah570plkjxzh-net-base-5.3 0.0 0.0 0.0%
total: 646.3 MiB
$ guix describe
Generacio 126 Jan 27 2020 23:59:19 (nuna)
guix 93f4511
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 93f4511eb0c9b33f5083c2a04f4148e0a494059c
$ grep -r x3jx25cd3q363mr7nbgzrhrv1vza6cf7 /gnu/store/g4rrz9rnr8ssbnj3gx3dmsxv4glll8nf-go-1.12.15
Duuma dosiero /gnu/store/g4rrz9rnr8ssbnj3gx3dmsxv4glll8nf-go-1.12.15/bin/go kongruas
Duuma dosiero /gnu/store/g4rrz9rnr8ssbnj3gx3dmsxv4glll8nf-go-1.12.15/pkg/tool/linux_amd64/cgo kongruas

Ludo’.
S
S
Sarah Morgensen wrote on 4 Jul 2021 07:30
(name . Ludovic Courtès)(address . ludo@gnu.org)
86k0m6abn2.fsf@mgsn.dev
Hello all,

(I have CC'd Tobias since we briefly discussed this on #guix recently.)

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (10 lines)
> Hello!
>
> It seems that Go unduly retains a reference to GCC:
>
> $ guix size go
> store item total self
> /gnu/store/g4rrz9rnr8ssbnj3gx3dmsxv4glll8nf-go-1.12.15 646.3 355.9 55.1%
> /gnu/store/x3jx25cd3q363mr7nbgzrhrv1vza6cf7-gcc-7.4.0 177.4 107.2 16.6%
> [...]

This is still the case:

$ guix size go
store item total self
/gnu/store/vvly7zgn981brb37v8y8a7f9psx7c6ay-go-1.16.5 570.0 371.5 65.2%
/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0 178.5 107.3 18.8%
/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31 38.4 36.7 6.4%
/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib 71.0 32.6 5.7%
/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32 88.0 17.0 3.0%
/gnu/store/kl68v5mclwp511xgpsl2h1s9gmsdxpzh-tzdata-2021a 1.9 1.9 0.3%
/gnu/store/mmhimfwmmidf09jw1plw3aw1g1zn2nkh-bash-static-5.0.16 1.6 1.6 0.3%
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16 39.4 1.0 0.2%
/gnu/store/g2s5jfkfd4k973wb58476b1bbv9zpm6m-zlib-1.2.11 38.6 0.2 0.0%
/gnu/store/zfbbn61ij7w0bl4wbrwi87x5ghqx968c-net-base-5.3 0.0 0.0 0.0%
total: 570.0 MiB

But... is the "baking-in" of (a particular version of) GCC a bad thing?
I am not sure either way.

Go invokes gcc when compiling with cgo, and cgo (or at least the usage
of standard libraries which use cgo) is getting fairly common. If we do
not provide a default gcc with Go, a plain "go build" will produce an
error if it encounters something which uses cgo and it can't find gcc:

$ go build
# runtime/cgo
cgo: exec gcc: exec: "gcc": executable file not found in $PATH

The user would then have to either install a gcc-toolchain, or figure
out that they must set CGO_ENABLED=0. From this perspective, it is more
convenient to have GCC baked-in for the average user, who likely has no
reason to want a different CC anyway.

On the other hand, currently GCC is baked-in to Go in two ways:

* CC is set to /gnu/store/...-gcc-7.5.0/bin/gcc
* Go is patched so that it adds /gnu/store/...-gcc-7.5.0-lib/lib as a
runpath when linking with cgo files

This means that even if the user provides a different CC, the
gcc-7.5.0-lib dir will also be in the runpath. I do not know if, or how
much, this would conflict with other gcc-lib runpaths.

I have experimented with a couple ways of removing the gcc-7.5.0 reference:

1. Simply set CC=gcc. This works to remove gcc-7.5.0 from references,
but we still get a gcc-7.5.0-lib runpath. We can't remove this runpath
completely, as anything using cgo-enabled parts of the standard library
require it, and Go does not save the library location anywhere.

2. Make Go require external linking for anything using cgo, which would
remove the need to patch internal linking at all. Some platforms do not
support internally linking cgo at all, so Go should have no trouble
handling this. It does break some tests which expect to be able to
internally link, but I have not yet found any actual packages it breaks.

My instinct says that removing the reference, and doing so via #2, is
the way to go, but I am just a newcomer to Guix. WDYT?

--
Sarah
S
S
Sarah Morgensen wrote on 6 Jul 2021 22:17
(name . Ludovic Courtès)(address . ludo@gnu.org)
86k0m3jiyc.fsf@mgsn.dev
Hello,

A quick addendum...

Sarah Morgensen <iskarian@mgsn.dev> writes:

Toggle quote (5 lines)
> This means that even if the user provides a different CC, the
> gcc-7.5.0-lib dir will also be in the runpath. I do not know if, or how
> much, this would conflict with other gcc-lib runpaths.
>

I have just seen the consequences of this: the binary is unable to load
symbols from the newer library. More info at

Toggle quote (8 lines)
> I have experimented with a couple ways of removing the gcc-7.5.0 reference:
>
> 1. Simply set CC=gcc. This works to remove gcc-7.5.0 from references,
> but we still get a gcc-7.5.0-lib runpath. We can't remove this runpath
> completely, as anything using cgo-enabled parts of the standard library
> require it, and Go does not save the library location anywhere.
>

--
Sarah
L
L
Ludovic Courtès wrote on 7 Jul 2021 23:49
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)
87k0m1okv5.fsf@gnu.org
Hi,

Sarah Morgensen <iskarian@mgsn.dev> skribis:

Toggle quote (40 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello!
>>
>> It seems that Go unduly retains a reference to GCC:
>>
>> $ guix size go
>> store item total self
>> /gnu/store/g4rrz9rnr8ssbnj3gx3dmsxv4glll8nf-go-1.12.15 646.3 355.9 55.1%
>> /gnu/store/x3jx25cd3q363mr7nbgzrhrv1vza6cf7-gcc-7.4.0 177.4 107.2 16.6%
>> [...]
>
> This is still the case:
>
> $ guix size go
> store item total self
> /gnu/store/vvly7zgn981brb37v8y8a7f9psx7c6ay-go-1.16.5 570.0 371.5 65.2%
> /gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0 178.5 107.3 18.8%
> /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31 38.4 36.7 6.4%
> /gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib 71.0 32.6 5.7%
> /gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32 88.0 17.0 3.0%
> /gnu/store/kl68v5mclwp511xgpsl2h1s9gmsdxpzh-tzdata-2021a 1.9 1.9 0.3%
> /gnu/store/mmhimfwmmidf09jw1plw3aw1g1zn2nkh-bash-static-5.0.16 1.6 1.6 0.3%
> /gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16 39.4 1.0 0.2%
> /gnu/store/g2s5jfkfd4k973wb58476b1bbv9zpm6m-zlib-1.2.11 38.6 0.2 0.0%
> /gnu/store/zfbbn61ij7w0bl4wbrwi87x5ghqx968c-net-base-5.3 0.0 0.0 0.0%
> total: 570.0 MiB
>
> But... is the "baking-in" of (a particular version of) GCC a bad thing?
> I am not sure either way.
>
> Go invokes gcc when compiling with cgo, and cgo (or at least the usage
> of standard libraries which use cgo) is getting fairly common. If we do
> not provide a default gcc with Go, a plain "go build" will produce an
> error if it encounters something which uses cgo and it can't find gcc:
>
> $ go build
> # runtime/cgo
> cgo: exec gcc: exec: "gcc": executable file not found in $PATH

Ah, I didn’t know about cgo (a helper for C bindings, right?).

I think it’s a case where “dynamic composition” (i.e., looking for gcc
in $PATH at run time) is preferable, because there are lots of
situations where gcc is not needed at all.

[...]

Toggle quote (7 lines)
> I have experimented with a couple ways of removing the gcc-7.5.0 reference:
>
> 1. Simply set CC=gcc. This works to remove gcc-7.5.0 from references,
> but we still get a gcc-7.5.0-lib runpath. We can't remove this runpath
> completely, as anything using cgo-enabled parts of the standard library
> require it, and Go does not save the library location anywhere.

Sounds good to me. (gcc-7.5.0-lib is always in the RUNPATH of
executables, we don’t have to worry about this one.)

Toggle quote (6 lines)
> 2. Make Go require external linking for anything using cgo, which would
> remove the need to patch internal linking at all. Some platforms do not
> support internally linking cgo at all, so Go should have no trouble
> handling this. It does break some tests which expect to be able to
> internally link, but I have not yet found any actual packages it breaks.

What do you mean by “external linking” and “internal linking” in this
context? (I know very little about Go.)

Thanks!

Ludo’.
S
S
Sarah Morgensen wrote on 8 Jul 2021 03:54
(name . Ludovic Courtès)(address . ludo@gnu.org)
86bl7dk1t6.fsf@mgsn.dev
Hi,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (11 lines)
>> Go invokes gcc when compiling with cgo, and cgo (or at least the usage
>> of standard libraries which use cgo) is getting fairly common. If we do
>> not provide a default gcc with Go, a plain "go build" will produce an
>> error if it encounters something which uses cgo and it can't find gcc:
>>
>> $ go build
>> # runtime/cgo
>> cgo: exec gcc: exec: "gcc": executable file not found in $PATH
>
> Ah, I didn’t know about cgo (a helper for C bindings, right?).

Yes, cgo allows you to compile Go programs which interface with C, as
well as straight .c files.

Toggle quote (17 lines)
>
> I think it’s a case where “dynamic composition” (i.e., looking for gcc
> in $PATH at run time) is preferable, because there are lots of
> situations where gcc is not needed at all.
>
> [...]
>
>> I have experimented with a couple ways of removing the gcc-7.5.0 reference:
>>
>> 1. Simply set CC=gcc. This works to remove gcc-7.5.0 from references,
>> but we still get a gcc-7.5.0-lib runpath. We can't remove this runpath
>> completely, as anything using cgo-enabled parts of the standard library
>> require it, and Go does not save the library location anywhere.
>
> Sounds good to me. (gcc-7.5.0-lib is always in the RUNPATH of
> executables, we don’t have to worry about this one.)

I recently discovered that there is actually an issue with this
particular approach. If the user uses a newer gcc-toolchain, the
always-added gcc-7.5.0-lib shadows the newer libraries and newer symbols
are unavailable. See https://issues.guix.gnu.org/36823.

Toggle quote (10 lines)
>
>> 2. Make Go require external linking for anything using cgo, which would
>> remove the need to patch internal linking at all. Some platforms do not
>> support internally linking cgo at all, so Go should have no trouble
>> handling this. It does break some tests which expect to be able to
>> internally link, but I have not yet found any actual packages it breaks.
>
> What do you mean by “external linking” and “internal linking” in this
> context? (I know very little about Go.)

"external linking" => Go invokes gcc to link object files together
"internal linking" => Go does the linking itself

--
Sarah
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 39400@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 39400
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch