gnu/ci.go: Embeds build path, breaking reproducible builds

OpenSubmitted by Vagrant Cascadian.
Details
3 participants
  • Ludovic Courtès
  • Mathieu Othacehe
  • Vagrant Cascadian
Owner
unassigned
Severity
normal
V
V
Vagrant Cascadian wrote on 24 Nov 2020 05:19
(address . bug-guix@gnu.org)
87r1ojgzdh.fsf@yucca
Something in gnu/ci.scm is embedding the build path when compiled intognu/ci.go, as can be seen:
https://tests.reproducible-builds.org/debian/rb-pkg/experimental/amd64/diffoscope-results/guix.html
./usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache/gnu/ci.go strings --all --bytes=8 {} Offset 85, 15 lines modified Offset 85, 15 lines modified ... 92 /build/1st/guix-1.2.0~rc2/gnu/ci.scm 92 /build/2/guix-1.2.0~rc2/2nd/gnu/ci.scm
While guix builds of guix are typically built with a consistent buildpath, it would be nice to fix this issue for other environments wherethe build path may vary between builds.

My *wild* guess is it maybe has something to do with the use ofcanonicalize-path:
(define (find-current-checkout arguments) "Find the first checkout of ARGUMENTS that provided the current file.Return #f if no such checkout is found." (let ((current-root (canonicalize-path (string-append (dirname (current-filename)) "/..")))) (find (lambda (argument) (and=> (assq-ref argument 'file-name) (lambda (name) (string=? name current-root)))) arguments)))
Either directly or indirectly... does canonicalize-path resolve at buildtime? run time? somewhere in-between? Or is this a red herring, andsomething else entirely is responsible? I'll let a competent schemerponder this! :)

Thanks for all your help!

live well, vagrant
-----BEGIN PGP SIGNATURE-----
iHUEARYKAB0WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCX7yJ6gAKCRDcUY/If5cWqlmmAQD+b4gNNqA6atRtAHhIqm5o4r5EqMgH/9uiGy4s5VsBpAD8CtYB6YkOAAsAK5sxn7qvHLyvpGepXa2VrrmtWHNTUgg==zJG9-----END PGP SIGNATURE-----
L
L
Ludovic Courtès wrote on 26 Nov 2020 22:39
(name . Vagrant Cascadian)(address . vagrant@reproducible-builds.org)
87blfj23yb.fsf@gnu.org
Hi!
Vagrant Cascadian <vagrant@reproducible-builds.org> skribis:
Toggle quote (14 lines)> My *wild* guess is it maybe has something to do with the use of> canonicalize-path:>> (define (find-current-checkout arguments)> "Find the first checkout of ARGUMENTS that provided the current file.> Return #f if no such checkout is found."> (let ((current-root> (canonicalize-path> (string-append (dirname (current-filename)) "/.."))))> (find (lambda (argument)> (and=> (assq-ref argument 'file-name)> (lambda (name)> (string=? name current-root)))) arguments)))
‘canonicalize-path’ is called at run time, so that’s fine. However,‘current-filename’ is a macro that captures the source file name atbuild time, so it’s the likely culprit here.
I was going to go with something like:
Toggle diff (14 lines)diff --git a/gnu/ci.scm b/gnu/ci.scmindex 5548d9560e..0bacfbe025 100644--- a/gnu/ci.scm+++ b/gnu/ci.scm@@ -488,7 +488,8 @@ valid." Return #f if no such checkout is found." (let ((current-root (canonicalize-path- (string-append (dirname (current-filename)) "/.."))))+ (string-append (dirname (search-path %load-path "gnu/ci.scm"))+ "/..")))) (find (lambda (argument) (and=> (assq-ref argument 'file-name) (lambda (name)
… but I don’t think we can assume that the checkout is in the‘%load-path’ when this code is executed. WDYT, Mathieu?
Looking at f71b0a0012d46bd30ead1a14ed58fd59647415e2, which introducedthis, there might be other options too.
Thanks,Ludo’.
V
V
Vagrant Cascadian wrote on 27 Nov 2020 21:25
(name . Ludovic Courtès)(address . ludo@gnu.org)
878samh7j2.fsf@yucca
On 2020-11-26, Ludovic Courtès wrote:
Toggle quote (42 lines)> Vagrant Cascadian <vagrant@reproducible-builds.org> skribis:>> My *wild* guess is it maybe has something to do with the use of>> canonicalize-path:>>>> (define (find-current-checkout arguments)>> "Find the first checkout of ARGUMENTS that provided the current file.>> Return #f if no such checkout is found.">> (let ((current-root>> (canonicalize-path>> (string-append (dirname (current-filename)) "/.."))))>> (find (lambda (argument)>> (and=> (assq-ref argument 'file-name)>> (lambda (name)>> (string=? name current-root)))) arguments)))>> ‘canonicalize-path’ is called at run time, so that’s fine. However,> ‘current-filename’ is a macro that captures the source file name at> build time, so it’s the likely culprit here.>> I was going to go with something like:>> diff --git a/gnu/ci.scm b/gnu/ci.scm> index 5548d9560e..0bacfbe025 100644> --- a/gnu/ci.scm> +++ b/gnu/ci.scm> @@ -488,7 +488,8 @@ valid."> Return #f if no such checkout is found."> (let ((current-root> (canonicalize-path> - (string-append (dirname (current-filename)) "/.."))))> + (string-append (dirname (search-path %load-path "gnu/ci.scm"))> + "/.."))))> (find (lambda (argument)> (and=> (assq-ref argument 'file-name)> (lambda (name)>> … but I don’t think we can assume that the checkout is in the> ‘%load-path’ when this code is executed. WDYT, Mathieu?>> Looking at f71b0a0012d46bd30ead1a14ed58fd59647415e2, which introduced> this, there might be other options too.
This does resolve the reproducibility issues after some "quick" testing;no idea if it breaks anything, though I didn't see any new test suitefailures (though my builds are patched to skip many tests)...

live well, vagrant
-----BEGIN PGP SIGNATURE-----
iHUEARYKAB0WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCX8FgoQAKCRDcUY/If5cWqrW3AP4zrVVyDNDv2kxfNWWCrqfxOnmgYNKhXG8nlg0H9vb6pgEAttq02lrkEthURDqtXmcO+pjoMZmboFK6T3b4397S+go==G6Px-----END PGP SIGNATURE-----
M
M
Mathieu Othacehe wrote on 1 Dec 2020 10:41
(name . Ludovic Courtès)(address . ludo@gnu.org)
87wny13lub.fsf@gnu.org
Hey,
Toggle quote (3 lines)> … but I don’t think we can assume that the checkout is in the> ‘%load-path’ when this code is executed. WDYT, Mathieu?
Cuirass happens to add checkouts to the %load-path just before loadingthis file.
I tested your path, it works fine. I think it is acceptable as this (gnuci) interface needs a big rework anyway.
I can apply your patch if it's ok for you.
Thanks,
Mathieu
L
L
Ludovic Courtès wrote on 3 Dec 2020 14:23
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87o8jbdnvb.fsf@gnu.org
Hi,
Mathieu Othacehe <othacehe@gnu.org> skribis:
Toggle quote (6 lines)>> … but I don’t think we can assume that the checkout is in the>> ‘%load-path’ when this code is executed. WDYT, Mathieu?>> Cuirass happens to add checkouts to the %load-path just before loading> this file.
Is that systematic? Isn’t it only when ‘load_path_inputs’ is non-empty?
Toggle quote (5 lines)> I tested your path, it works fine. I think it is acceptable as this (gnu> ci) interface needs a big rework anyway.>> I can apply your patch if it's ok for you.
Sure if you’re confident you can go ahead. :-)
Thanks,Ludo’.
?