Golang programs keeping references [gnu: go: Update default to 1.11.]

  • Done
  • quality assurance status badge
Details
One participant
  • Leo Famulari
Owner
unassigned
Submitted by
Leo Famulari
Severity
normal
Merged with
L
L
Leo Famulari wrote on 5 Dec 2018 03:16
(address . bug-guix@gnu.org)
20181205021601.GA22628@jasmine.lan
----- Forwarded message from Leo Famulari <leo@famulari.name> -----

Date: Mon, 12 Nov 2018 12:29:25 -0500
From: Leo Famulari <leo@famulari.name>
To: guix-devel@gnu.org
Cc: Pierre Neidhardt <ambrevar@gmail.com>
Subject: Golang programs keeping references [gnu: go: Update default to 1.11.]
User-Agent: Mutt/1.10.1 (2018-07-13)

On Mon, Nov 12, 2018 at 04:32:46AM -0500, Pierre Neidhardt wrote:
Toggle quote (8 lines)
> commit 9a65a052016572b61e3c4247fcdf9e0478656f71
> Author: Pierre Neidhardt <mail@ambrevar.xyz>
> Date: Sun Nov 11 22:02:18 2018 +0100
>
> gnu: go: Update default to 1.11.
>
> * gnu/packages/golang.scm (go): Update default to 1.11.

I noticed that since this change, Go programs (that is, command-line
executables) keep references to their inputs:

$ guix gc --references $(./pre-inst-env guix build --no-grafts kurly)
/gnu/store/2b2md66fbzyspsmd5dj6zkj9hilac40r-tzdata-2018e
/gnu/store/4iwksvq53rlzphfp3xvp63ihlw226c0n-go-github-com-aki237-nscjar-0.0.0-0.e2df936
/gnu/store/5rxdjbk8h0bh1hbaan8y8ib13va2bcmw-net-base-5.3
/gnu/store/ahvdlp6y44qj6kx63rmx1sq8r61x3zc2-go-github-com-alsm-ioprogress-0.0.0-0.063c372
/gnu/store/f8yps0l8p371jgzh6cki0z5n2kgfjiwy-go-github-com-urfave-cli-1.19.1-0.934abfb
/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27
/gnu/store/pp0bakrbyv9xmp1kyv2114l19s11b74z-gcc-6.4.0-lib

Previously, they did not:

$ guix gc --references $(guix build --no-grafts kurly)
/gnu/store/2b2md66fbzyspsmd5dj6zkj9hilac40r-tzdata-2018e
/gnu/store/5rxdjbk8h0bh1hbaan8y8ib13va2bcmw-net-base-5.3
/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27
/gnu/store/pp0bakrbyv9xmp1kyv2114l19s11b74z-gcc-6.4.0-lib

Is this expected? I thought that Go programs were always statically
linked.



----- End forwarded message -----
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlwHNOEACgkQJkb6MLrK
fwimChAA1l4ncYXHAmRY+9JWzUt3aVHGnJKZB3Lyl076uwM/sjpGgglTp10JFX8H
WGi7H/eqxV5DlQkW47T4MZvjDkG6FqFhbV/mGThc2spfEMGIULNvTZY8D7DqyYbo
aKte8vh3PCBwQHmkHZgFLBPGkMs3Mn7CLBKGatQ3XnaWeYyfaApG/ZuDRYiTREqX
plojmwhk/bkTqGPzDi4Spy8M7HcyB6ZeuvALbA+hhJzLg3IRyHw49/hRQqQyAxmb
pGL8digJPz7eV0RNXIrrPOqS09iOzpjV6LDL4qzaGuEtbrkeH+8aue/lo8n/ky9N
4NzVwZQh9Pbf59jC7xMdZh6fPCra9v6X3lWdsJo7wV7rONBTioUmQ+Ccsbcc7rYC
06z5FeJ1gTI/SW7w9VNxoRP0WNYk4a+nO5k8qD6xmk1XFWfnFdm9R4yzFv/c5Hay
mzkdyRODrPQmM/swdAF1u7Fw3cxPMMWkRT4K0TgJJem1EJWqrkJEIQHP+xNjmps8
Ikp8GFzXZ7CZbwQ7nVQrmT4aTr7qe1OAENWS1gjILjiVpj3t4L2YPUpRHyDb4g3B
sHHEDcjjpPjyY0X/3XlxH8S3i0CL8zONcGiitJHERPLXiKoBMxxA4sMDlnRXiXjf
Tx/xgJwZZEbhWV43IILU2019HAhhWq8P5h3wkR1/jdqB0l3CQ4U=
=lXa+
-----END PGP SIGNATURE-----


L
L
Leo Famulari wrote on 15 Dec 2018 20:54
[mail@ambrevar.xyz: Re: Golang programs keeping references [gnu: go: Update default to 1.11.]]
(address . 33620@debbugs.gnu.org)
20181215195457.GD9685@jasmine.lan
----- Forwarded message from Pierre Neidhardt <mail@ambrevar.xyz> -----

Date: Sun, 09 Dec 2018 19:46:03 +0100
From: Pierre Neidhardt <mail@ambrevar.xyz>
To: Leo Famulari <leo@famulari.name>
Cc: guix-devel@gnu.org
Subject: Re: Golang programs keeping references [gnu: go: Update default to 1.11.]
User-agent: mu4e 1.0; emacs 26.1

I've investigated the possible solutions to avoid including the paths into the
binaries.

I've found this:


It's still unresolved and only planned for Go 1.13.

In the meantime, I've played with the -trimpath option to see if I could get
something out of it.

I've added this to Go's (build) function:

Toggle snippet (4 lines)
"-asmflags=all=-trimpath=/gnu/store"
"-gcflags=all=-trimpath=/gnu/store"

The resulting binary is indeed trimmed, but that's not enough: it seems that
Guix detects the remaining part of the path as a store item and includes it in
the list of requisites. Is this really how Guix works? It does not need the
full path?

Go supports only one call to -trimpath, so we can't even set this to the Go
inputs.

Regarding Boyer-Moore over the binary, we would have to apply the changes for
all recursive Go libraries. Now is there a reliable way to detect what's a Go
library and what is not? We don't want to patch non-Go libraries, right?
(Let's not forget that there is CGo...)

If we can't think of a way to detect a Go library, Boyer-Moore does not seem to
be a solution either. And we might have to wait for Go 1.13...

--
Pierre Neidhardt



----- End forwarded message -----
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlwVXBEACgkQJkb6MLrK
fwhBYw/9HRo55iLh+r5+AlGJBUksSqBofjodfx2z2miyfj0IVJZCoP0uREe8+Ki4
ocvS/E57ncMv7jhERJUrNnoxoBaEtYb2Cugpmk7Ncw/9jqJqZTlRPa3Z98mc+QGt
cw6vdvJ6BtpvCscK/MtLEATV0VMfLMJwqEjB6wMsvI1gdkJ3xJlkXlhqHzJgFBAt
mMJeKse1Mx8I5WC3bvCoOBMlxEQT5aFB2o0tQwDm05ao4jE0y/vmSwYtXKmBTQwH
7mXoyZ1Z/gdL5ENv5YKkmWCYLN3fjEomhijFRzFEMWINoknIvII8kpTEExe9P8NA
BDyN3VmCwTv7RvbDojpjXx/MW8kljp4cjk2rYHm8pjXYp+JhXzSnpaMMTrEmd27k
Ef/8PNpyJWvK+/fi+3z2eMox4rehsqCVtRslggA9rKAIoke2V/m82quWy5GMoIfi
E2E5WPdyDK5G9cGFOCr7u6ETSFhr6fqoe6+9T/cCR3o4seh6uF0slx2SyWwD4lFX
utb84IivKD+57nN76Hg7qS8I31HqT0JzJbumWGmZuREk9UiqLaEpuH7I0nJYiJ+t
xA6lwbeUzWt2WjHtC8mg4qYl5lPJs4rX4jzbUglWMV0k5QgZ2lq0LxrMOmT9n3yM
RuhdilEhnbRKmPQtLiHnNW74Dq+wagj8z/d9edE0Y9LDM6rAaW0=
=cXb9
-----END PGP SIGNATURE-----


L
L
Leo Famulari wrote on 16 Dec 2018 22:26
(no subject)
(address . control@debbugs.gnu.org)
20181216212623.GA32482@jasmine.lan
merge 33741 33620
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlwWwv8ACgkQJkb6MLrK
fwhztg//fsLVRKJJ1OHzOA9i0aMZJII8pm7Xy1ZWrzmhxUpDq083xjkHEltEIYj+
grdNKEwjfbyVGiH+4fomRMdqZSCXJJrmdVL1bvUEJYNmURD9vRi2g4ofn/gQ0svV
EBwpH96FZs2clVgZ9XRbNJcI9RuUmcMOolNIde9n+CmrdKlKD5BK9YoW0EocqXY2
DHOjfUlYtKXMFDXzzFH5tQ54A7n/ytK1zfRrNTccxdb4WjPLcJKB4jzvCf9Y85sp
T6/qBi9PcoG7PiLwqmEeqSnlxy8dCoeepU0gIACdCgLbtjOjMGn4SkcdCrMrxuek
ZTBrPAM88ymIoMfntOG7JXwqCQySaP/bm56DFOs8ThtIlGZkKFBOaGX2fgNQSoGO
2ZIM+lDd5J1OFHBDnKmOEAchdHGAIVdPgBlOWYgxlfb5CiZCQzpV7V059wnvI2Ah
vDI/UXt3hnAZAsd0MICtA4jNYt6avSAdxeM1ZUkcCp2QR+LcP1Fb55tF+GhN4tmf
bvyc9YxRjSnlSetGQjfTSJBSMMQKhnUKUvTTsR0ukPSZKaUKJSq4g90OxdyuCfuq
0iAZKjMj8liEmt4aScH6Di94nuHv1/UKgIWVQ5mD8ez4TrbboUbyukTFLI8kc7N1
a5HiAzeuCqhgJwxLuM3bNq2bW01SxCuDT2HzYsStP+yw9jy7Dlo=
=KMK0
-----END PGP SIGNATURE-----


L
L
Leo Famulari wrote on 14 Mar 2019 20:44
Re: [mail@ambrevar.xyz: Re: Golang programs keeping references [gnu: go: Update default to 1.11.]]
(address . 33620-done@debbugs.gnu.org)
20190314194415.GB17982@jasmine.lan
The issue of Go packages keeping references to all their Go-language is
inputs is resolved with commit e3900a4d64e4bf6f426809d6bff058e5a2ae9bc8.

This commit basically avoids bringing the store paths into the build
environment at all by symlinking them into a union directory in the
TMPDIR. This is a bit of a hack but it's actually more idiomatic in Go
to have all the inputs in a single directory ("workspace" in Go). The
previous technique of passing a list of directories in the GOPATH
variable is officially supported but is definitely a 2nd-class technique
in practice.

----- Forwarded message from Pierre Neidhardt <mail@ambrevar.xyz> -----
Toggle quote (12 lines)
> I've added this to Go's (build) function:
>
> --8<---------------cut here---------------start------------->8---
> "-asmflags=all=-trimpath=/gnu/store"
> "-gcflags=all=-trimpath=/gnu/store"
> --8<---------------cut here---------------end--------------->8---
>
> The resulting binary is indeed trimmed, but that's not enough: it seems that
> Guix detects the remaining part of the path as a store item and includes it in
> the list of requisites. Is this really how Guix works? It does not need the
> full path?

Not having read the reference scanner code carefully, I expect that it
ignores the '/gnu/store/' since this path is actually configurable.

Toggle quote (5 lines)
> Regarding Boyer-Moore over the binary, we would have to apply the changes for
> all recursive Go libraries. Now is there a reliable way to detect what's a Go
> library and what is not? We don't want to patch non-Go libraries, right?
> (Let's not forget that there is CGo...)

In (guix build-system go), I'd like to construct of list of inputs that
use the go-build-system and pass this list to the biuld side. This would
be useful for other things, but could also be used to detect which
inputs are Go libraries.

Toggle quote (3 lines)
> If we can't think of a way to detect a Go library, Boyer-Moore does not seem to
> be a solution either. And we might have to wait for Go 1.13...

Waiting for upstream is always the easiest way :)

But it would still be nice to have Boyer-Moore available in (guix build
utils) for whenever we want to use it. Or even in Guile itself...
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlyKrw8ACgkQJkb6MLrK
fwjqmRAAzOUIUa6m2/jL2hawh24O+ais4tPAj4WfpdilVBgkXMKbKs8nzkXUocEl
cahgMCZQuV8EQPTYYHrMPLToZ2UIbIMinu8LSrfBAI2IWmghH7BxlotufQcj6nF8
46UpM2gnf6TSMvuzSCZPqrV4JwSpUaO+hPyRDKg6mahMc01kYoIhrCRc9pJoALnc
9DfxsX9GAGtrDQvCz7Q7zGhahQsmJpX9+MTTCla6cVLGFEAsIZESTmnl7GTH8jdu
2R2Fnshu/auJM1qYBME9jWRRXwtCajW39YWbWnn/7+Lupo43qxvk/vz34fzjPPIP
8z1gxa+sp/GQNOveeM3jdilyGxMGGmUWbEAiNXgB7cYO+cSMyQcdU9lLYAwh7TeQ
gLozcujqTufR/cwsuErt48ZUVO67ybWOsYRKm37/SuFUz9PsBtptftDFhPEhFYeP
uMp1n7FkJNtmvi1eKYn2/rs/cjUplGHL5heyFFxB5nB5vXJ3bvnsj9GzXS+ARa0m
Ctns8MEhC5nZrJTGhoetzbBBjaQLOkINmb58MRmuRg3NQ84I67N6E85urPf8WSUQ
YjcvP7ScPnEmJ+6NRpxjQA8llN1Puo1brLMCqt349uoHIFOzncGwLivzZiqrEfpR
0j+u6k9e5Eh0lamuuYLBpJXHErSeesrhWt58yLhAYP4JskHNXWI=
=4/wz
-----END PGP SIGNATURE-----


Closed
?