go incremental builds broken

OpenSubmitted by Hank Donnay.
Details
5 participants
  • Efraim Flashner
  • Hank Donnay
  • Leo Famulari
  • Ludovic Courtès
  • zimoun
Owner
unassigned
Severity
normal
H
H
Hank Donnay wrote on 16 Feb 2017 16:05
(address . bug-guix@gnu.org)
CAD4UWkjnkSBYmAfk+E5QAvO9H29Y1xS2vu_h8eB3ZypJ=7NHwg@mail.gmail.com
Hello,
It seems the guix's go package is broken when the go tool is used forincremental builds. Any attempt to use 'install' or 'build -i' resultsin an attempt to write to the store. A one-liner:
guix environment --ad-hoc go -- bash -c 'export t=$(mktemp -d); cd$t && export GOPATH=$(pwd) GOBIN=$(pwd)/bin && go install cmd/go; cd&& rm -rf $t'
Another command reports that (seemingly) the entire stdlib is marked as stale:
guix environment --ad-hoc go -- bash -c 'export t=$(mktemp -d); cd$t && export GOPATH=$(pwd) GOBIN=$(pwd)/bin && go list -f '\''{{join.Deps "\n"}}'\'' cmd/go | xargs -n1 go list -f '\''{{if.Stale}}{{.ImportPath}}: {{.StaleReason}}{{end}}'\''; cd && rm -rf $t'
The function for determining staleness is here (after the giantcomment explaining the reasoning):https://golang.org/src/cmd/go/pkg.go#L1111
I don't see anything wrong with the package definition, but could bemissing something. My only hunch at this point is that something mightbe modifying src/runtime/internal/sys/zversion.go, as that entire fileis included in the build ID computation.
Thanks,
H
H
Hank Donnay wrote on 24 Feb 2017 20:50
Re: bug#25752: Acknowledgement (go incremental builds broken)
(address . 25752@debbugs.gnu.org)
CAD4UWkjG0POAOYM-E90k2UDFFhnro1s5wd3qT1=pZPMqhOLTXw@mail.gmail.com
After talking on IRC, it seems like grafts are the problem. Adding`--no-grafts` to the above commands has them succeed.
The GOROOT (store location, in this case) gets encoded in thegenerated source by the 'dist' tool, so changing that is going tocause issues.
Is there a way to disable grafts on a per-package basis, in the definition?
On Thu, Feb 16, 2017 at 10:06 AM, GNU bug Tracking System<help-debbugs@gnu.org> wrote:
Toggle quote (21 lines)> Thank you for filing a new bug report with debbugs.gnu.org.>> This is an automatically generated reply to let you know your message> has been received.>> Your message is being forwarded to the package maintainers and other> interested parties for their attention; they will reply in due course.>> Your message has been sent to the package maintainer(s):> bug-guix@gnu.org>> If you wish to submit further information on this problem, please> send it to 25752@debbugs.gnu.org.>> Please do not send mail to help-debbugs@gnu.org unless you wish> to report a problem with the Bug-tracking system.>> --> 25752: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=25752> GNU Bug Tracking System> Contact help-debbugs@gnu.org with problems
L
L
Leo Famulari wrote on 25 Feb 2017 16:58
(name . Hank Donnay)(address . hdonnay@gmail.com)(address . 25752@debbugs.gnu.org)
20170225155814.GB3227@jasmine
On Fri, Feb 24, 2017 at 02:50:10PM -0500, Hank Donnay wrote:
Toggle quote (9 lines)> After talking on IRC, it seems like grafts are the problem. Adding> `--no-grafts` to the above commands has them succeed.> > The GOROOT (store location, in this case) gets encoded in the> generated source by the 'dist' tool, so changing that is going to> cause issues.> > Is there a way to disable grafts on a per-package basis, in the definition?
You can disable them on the command-line for any tool that buildsthings, using the '--no-grafts' option, which is a "Common build option"[0]. For example, `guix environment --ad-hoc go --no-grafts`.
If you do this, you should understand that the ungrafted Go package willcontain some serious security vulnerabilities, since we only use graftsto fix vulnerabilities in core packages. [1]
We should figure out why this issue manifests when go is grafted, andfix the root cause.
[0]https://www.gnu.org/software/guix/manual/html_node/Common-Build-Options.html
[1]https://www.gnu.org/software/guix/manual/html_node/Security-Updates.html
L
L
Ludovic Courtès wrote on 7 Mar 2017 22:50
Re: bug#25752: go incremental builds broken
(name . Hank Donnay)(address . hdonnay@gmail.com)(address . 25752@debbugs.gnu.org)
87fuiorc3b.fsf@gnu.org
Hello,
Hank Donnay <hdonnay@gmail.com> skribis:
Toggle quote (4 lines)> The function for determining staleness is here (after the giant> comment explaining the reasoning):> https://golang.org/src/cmd/go/pkg.go#L1111
This method relies on the build ID to, which is defined like this (info"(ld) Options"):
`--build-id' `--build-id=STYLE' Request the creation of a `.note.gnu.build-id' ELF note section or a `.buildid' COFF section. The contents of the note are unique bits identifying this linked file. STYLE can be `uuid' to use 128 random bits, `sha1' to use a 160-bit SHA1 hash on the normative parts of the output contents, `md5' to use a 128-bit MD5 hash on the normative parts of the output contents, or `0xHEXSTRING' to use a chosen bit string specified as an even number of hexadecimal digits (`-' and `:' characters between digit pairs are ignored). If STYLE is omitted, `sha1' is used.
The `md5' and `sha1' styles produces an identifier that is always the same in an identical output file, but will be unique among all nonidentical output files. It is not intended to be compared as a checksum for the file's contents. A linked file may be changed later by other tools, but the build ID bit string identifying the original linked file does not change.
Passing `none' for STYLE disables the setting from any `--build-id' options earlier on the command line.
I suppose Go uses one of md5 or sha1, which is a good thing since itallows for reproducible builds.
However, grafting breaks this, similarly to https://bugs.gnu.org/19973since they change file contents without recomputing the build ID.
Having Go use --build-id=uuid would work around the problem, but itwould also prevent bit-reproducible builds.
Perhaps our grafting code will have to handle .note.gnu.build-idspecially.
Thoughts?
Thanks for reporting the issue,Ludo’.
Z
Z
zimoun wrote on 18 Dec 2020 21:00
Re: bug#25752: Acknowledgement (go incremental builds broken)
(name . Leo Famulari)(address . leo@famulari.name)
86mtyavq9d.fsf@gmail.com
Hi,
On Sat, 25 Feb 2017 at 10:58, Leo Famulari <leo@famulari.name> wrote:
Toggle quote (21 lines)> On Fri, Feb 24, 2017 at 02:50:10PM -0500, Hank Donnay wrote:>> After talking on IRC, it seems like grafts are the problem. Adding>> `--no-grafts` to the above commands has them succeed.>> >> The GOROOT (store location, in this case) gets encoded in the>> generated source by the 'dist' tool, so changing that is going to>> cause issues.>> >> Is there a way to disable grafts on a per-package basis, in the definition?>> You can disable them on the command-line for any tool that builds> things, using the '--no-grafts' option, which is a "Common build option"> [0]. For example, `guix environment --ad-hoc go --no-grafts`.>> If you do this, you should understand that the ungrafted Go package will> contain some serious security vulnerabilities, since we only use grafts> to fix vulnerabilities in core packages. [1]>> We should figure out why this issue manifests when go is grafted, and> fix the root cause.
I suspect closing this bug had been forgotten. If yes, feel free toclose it. Otherwise, I do not understand what is the next actionablestep.
http://issues.guix.gnu.org/issue/25752
All the best,simon
L
L
Leo Famulari wrote on 18 Dec 2020 22:31
(name . zimoun)(address . zimon.toutoune@gmail.com)
X90fuo+5BdnG7q8P@jasmine.lan
On Fri, Dec 18, 2020 at 09:00:14PM +0100, zimoun wrote:
Toggle quote (4 lines)> I suspect closing this bug had been forgotten. If yes, feel free to> close it. Otherwise, I do not understand what is the next actionable> step.
I don't think this bug has been fixed, although I have not checked.Currently, Go incremental builds and library re-use is not functional inGuix, unless I missed a new development.
Z
Z
zimoun wrote on 11 Jan 13:46 +0100
(name . Leo Famulari)(address . leo@famulari.name)
86im83tzd3.fsf@gmail.com
Hi Leo,
On Fri, 18 Dec 2020 at 16:31, Leo Famulari <leo@famulari.name> wrote:
Toggle quote (4 lines)> I don't think this bug has been fixed, although I have not checked.> Currently, Go incremental builds and library re-use is not functional in> Guix, unless I missed a new development.
Thanks. Could you provide a Go example for reproducing? Well, areproducer to track what could be wrong on the grafting mechanism forGo. If I understand correctly.

All the best,simon
E
E
Efraim Flashner wrote on 11 Jan 21:01 +0100
(name . zimoun)(address . zimon.toutoune@gmail.com)
X/yujodIHrnX0s3T@3900XT
On Mon, Jan 11, 2021 at 01:46:48PM +0100, zimoun wrote:
Toggle quote (13 lines)> Hi Leo,> > On Fri, 18 Dec 2020 at 16:31, Leo Famulari <leo@famulari.name> wrote:> > > I don't think this bug has been fixed, although I have not checked.> > Currently, Go incremental builds and library re-use is not functional in> > Guix, unless I missed a new development.> > Thanks. Could you provide a Go example for reproducing? Well, a> reproducer to track what could be wrong on the grafting mechanism for> Go. If I understand correctly.>
I actually spent a bit of time on this today/yesterday. I modified thego-build-system to copy the pkg and src folders from build to build(instead of just the src folders) and the pkg folders were ignored.Upstream documentation isn't clear about when the pkg folder is used andwhen it is ignored.
One thing I did try was to set the pkgdir to (string-append (getenv"GOROOT") "/pkg"), but then it didn't find the pkg folder from go:out. Ididn't try copying go:out/pkg to GOROOT/pkg to see if that would fix it;I assumed that even if it did work we wouldn't want to propagate all ofgo with the go packages.
-- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנרGPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl/8rosACgkQQarn3Mo9g1EbwxAAvkZuax5OBUTzbjtqMxsU7f8NDTekN5HccR6A4/W2OcGQst5wW2K22jEKjtDXThHCzPPmap44RXs4il5J+23Ypy0qf8YlaFi5ieVMVSS9jgM4HiWcBuBZBTM2N+4YL7Bu9jNpucYINIJtLY+4+cDIdsEMce0Re7MkD4+QAdlTchh4PxEmyf9iiCCaTKkBsWehlw/j22dTCjpPUrQp109fyE+No1qFUQ+ie6dkq96BsNJMWKF1QcSIdkxmhQ//eESzcJUWeJfr0uaaesEl1YYHumot0nor5RI1P5e+VCC2SkRpEhasubrSlpxqQnPvjjQactm6krM0QWqzCK7Gkj+U9ThuXILAFNSyvzIH4QueMWXtYLM/PdmzFItYzSZ/C8/vMqh0XvqLDDyjGejcA3o95MaASo+hGrK4NlXdiyeLOOqH3Fj9dG0ipV3PiE72yS3lxrWK5x6zre2FNv3Gv04tLsKRtdQ7bzcumQ1eiDOq/ReHQf1vT41XKA2qDNVR35Q85wqP8OHHFoCSJ4BIg1O5gn69iHresmo7+TK03ul2IIfHpxTQEpxIZVhMHqRcsKEPjNgx++7vUyAoSOOMJxJTL+Feto3tNMrwmukfiE58/rEn6sLsi4WFa2YWQpaj5Jql8G7lF7RiFJO58HOS1G3TQKuG+x/gWGvLBF6uBoDp9n4==Clx4-----END PGP SIGNATURE-----

Z
Z
zimoun wrote on 9 Jun 23:38 +0200
(name . Efraim Flashner)(address . efraim@flashner.co.il)
86v96mlnv9.fsf@gmail.com
Hi,
On Mon, 11 Jan 2021 at 22:01, Efraim Flashner <efraim@flashner.co.il> wrote:
Toggle quote (24 lines)> On Mon, Jan 11, 2021 at 01:46:48PM +0100, zimoun wrote:>> On Fri, 18 Dec 2020 at 16:31, Leo Famulari <leo@famulari.name> wrote:>> >> > I don't think this bug has been fixed, although I have not checked.>> > Currently, Go incremental builds and library re-use is not functional in>> > Guix, unless I missed a new development.>> >> Thanks. Could you provide a Go example for reproducing? Well, a>> reproducer to track what could be wrong on the grafting mechanism for>> Go. If I understand correctly.>> >> I actually spent a bit of time on this today/yesterday. I modified the> go-build-system to copy the pkg and src folders from build to build> (instead of just the src folders) and the pkg folders were ignored.> Upstream documentation isn't clear about when the pkg folder is used and> when it is ignored.>> One thing I did try was to set the pkgdir to (string-append (getenv> "GOROOT") "/pkg"), but then it didn't find the pkg folder from go:out. I> didn't try copying go:out/pkg to GOROOT/pkg to see if that would fix it;> I assumed that even if it did work we wouldn't want to propagate all of> go with the go packages.
What is the status of this bug? Since I am not using Go at all, it ishard for me to have a reproducer. Is it possible to provide areproducer? Thanks.
Cheers,simon
Z
Z
zimoun wrote on 14 Sep 12:43 +0200
bug#25752: go incremental builds broken
(name . Leo Famulari)(address . leo@famulari.name)
86zgsftp3c.fsf_-_@gmail.com
Hi,
On Fri, 18 Dec 2020 at 16:31, Leo Famulari <leo@famulari.name> wrote:
Toggle quote (9 lines)> On Fri, Dec 18, 2020 at 09:00:14PM +0100, zimoun wrote:>> I suspect closing this bug had been forgotten. If yes, feel free to>> close it. Otherwise, I do not understand what is the next actionable>> step.>> I don't think this bug has been fixed, although I have not checked.> Currently, Go incremental builds and library re-use is not functional in> Guix, unless I missed a new development.
What is the status of this bug? Since I am not using Go at all, it ishard for me to have a reproducer. Is it possible to provide areproducer? Thanks.
Cheers,simon
PS: I asked the same thing 13 weeks ago. Therefore, without any answerand giving this bug is really old not so much commented, I propose toclose it.
Z
Z
zimoun wrote on 14 Sep 12:39 +0200
control message for bug #25752
(address . control@debbugs.gnu.org)
8635q7v3tz.fsf@gmail.com
tags 25752 + moreinfoquit
Z
Z
zimoun wrote 44 hours ago
Re: bug#25752: Acknowledgement (go incremental builds broken)
(name . Efraim Flashner)(address . efraim@flashner.co.il)
86fsrjsosb.fsf@gmail.com
Hi,
On Mon, 11 Jan 2021 at 22:01, Efraim Flashner <efraim@flashner.co.il> wrote:
Toggle quote (24 lines)> On Mon, Jan 11, 2021 at 01:46:48PM +0100, zimoun wrote:>> On Fri, 18 Dec 2020 at 16:31, Leo Famulari <leo@famulari.name> wrote:>>>> > I don't think this bug has been fixed, although I have not checked.>> > Currently, Go incremental builds and library re-use is not functional in>> > Guix, unless I missed a new development.>>>> Thanks. Could you provide a Go example for reproducing? Well, a>> reproducer to track what could be wrong on the grafting mechanism for>> Go. If I understand correctly.>>>> I actually spent a bit of time on this today/yesterday. I modified the> go-build-system to copy the pkg and src folders from build to build> (instead of just the src folders) and the pkg folders were ignored.> Upstream documentation isn't clear about when the pkg folder is used and> when it is ignored.>> One thing I did try was to set the pkgdir to (string-append (getenv> "GOROOT") "/pkg"), but then it didn't find the pkg folder from go:out. I> didn't try copying go:out/pkg to GOROOT/pkg to see if that would fix it;> I assumed that even if it did work we wouldn't want to propagate all of> go with the go packages.
What is the status of this old bug [1]? It is marked ’moreinfo’ becauseit misses a reproducer to be able to act on. Therefore, I am going toclose it.

Cheers,simon
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 25752@debbugs.gnu.org