clojure-build-system fails to compile -- backtrace from language/tree-il/peval.scm

  • Open
  • quality assurance status badge
Details
2 participants
  • Maxim Cournoyer
  • Maxime Devos
Owner
unassigned
Submitted by
Maxime Devos
Severity
normal
M
M
Maxime Devos wrote on 10 Aug 2022 18:59
(address . bug-guix@gnu.org)
8cd4199a-7db2-ee39-1939-3783e020dc9c@telenet.be
I started with a guix checkout at commit
d519305d83d08058e4def2c4d72fe62102d9599d. Everything was compiled,
according to "make".
Then I switched to current master: b21d05d232ec0aba5abec20e83cc52c1d5163cc3.
Now I get the following error:
Toggle quote (79 lines)
> In language/tree-il/peval.scm:
>     799:6 19 (loop _ #<vhash 7f81d96446e0 10 pairs> #f effect)
>   1661:48 18 (loop _ _ _ effect)
> In srfi/srfi-1.scm:
>    586:17 17 (map1 (#<tree-il (lexical install #{ g377}#)> #<tree-il
> (const documentation)> #<tree-il (const "Standard 'install' phase for
> clojure-build-system.")>))
> In language/tree-il/peval.scm:
>    853:11 16 (loop _ _ #f value)
>    346:22 15 (visit-operand #<<operand> var: #<<var> name: install
> gensym: #{ g377}# refcount: 2 set?: #f> sym: #{install 412}# visit:
> #<procedure 7f81d96447a0 at language/tree-il/peval.scm:977:40 (exp
> counter ctx)> source: #<tree-il (primcall values (call (@ (guix build
> java-utils) install-jars) (const "./")))> visit-count: 1 use-count: 0
> copyable?: #t residual-value: #f constant-value: #f alias: #f> _ value
> _ _)
>   1286:21 14 (loop _ #<vhash 7f81d96242c0 8 pairs> #<<counter> effort:
> #<variable 7f81d9876ae0 value: 0> size: #<variable 7f81d9876ad0 value:
> 0> continuation: #<procedure abort ()> recursive?: #t data:
> #<<operand> var: #<<var> name: install gensym: #{ g377}# refcount: 2
> set?: #f> sym: #{install 412}# visit: #<procedure 7f81d96447a0 at
> language/tree-il/peval.scm:977:40 (exp counter ctx)> source: #<tree-il
> (primcall values (call (@ (guix build java-utils) install-…> …)
> In srfi/srfi-1.scm:
>    586:17 13 (map1 (#<tree-il (call (@ (guix build java-utils)
> install-jars) (const "./"))>))
> In language/tree-il/peval.scm:
>   1625:20 12 (loop _ #<vhash 7f81d96242c0 8 pairs> #<<counter> effort:
> #<variable 7f81d9876ae0 value: 0> size: #<variable 7f81d9876ad0 value:
> 0> continuation: #<procedure abort ()> recursive?: #t data:
> #<<operand> var: #<<var> name: install gensym: #{ g377}# refcount: 2
> set?: #f> sym: #{install 412}# visit: #<procedure 7f81d96447a0 at
> language/tree-il/peval.scm:977:40 (exp counter ctx)> source: #<tree-il
> (primcall values (call (@ (guix build java-utils) install-…> …)
>    981:20 11 (loop _ #<vhash 7f81d96242c0 8 pairs> #<<counter> effort:
> #<variable 7f81d9876710 value: 65> size: #<variable 7f81d9876700
> value: 20> continuation: #<procedure abort ()> recursive?: #f data:
> #<tree-il (lambda ((name . install-jars)) (lambda-case
> (((jar-directory) #f #f #f () (t413)) (lambda () (lambda-case ((() #f
> #f (#t (#:outputs outputs #f)) ((const #f)) (t414)) (let (share)
> (t415) ((call (@ (guile) string-append) (call (@ (guile) assoc-ref)
> (lex…> …)
>     799:6 10 (loop #<tree-il (lambda () (lambda-case ((() #f #f (#t
> (#:outputs outputs #f)) ((const #f)) (t414)) (let (share) (t415)
> ((call (@ (guile) string-append) (call (@ (guile) assoc-ref) (lexical
> outputs t414) (const "out")) (const "/share/java"))) (seq (call (@
> (guile) for-each) (lambda () (lambda-case (((f) #f #f #f () (t416))
> (call (@ (guix build utils) install-file) (lexical f t416) (lexical
> share t415))))) (call (@ (guix build utils) find-files) (lex…> …)
>   1699:60  9 (loop _ #<vhash 7f81d964b200 10 pairs> #<<counter>
> effort: #<variable 7f81d9876710 value: 65> size: #<variable
> 7f81d9876700 value: 20> continuation: #<procedure abort ()>
> recursive?: #f data: #<tree-il (lambda ((name . install-jars))
> (lambda-case (((jar-directory) #f #f #f () (t413)) (lambda ()
> (lambda-case ((() #f #f (#t (#:outputs outputs #f)) ((const #f))
> (t414)) (let (share) (t415) ((call (@ (guile) string-append) (call (@
> (guile) assoc-ref) (le…> …)
> In srfi/srfi-1.scm:
>    586:17  8 (map1 (#f))
> In language/tree-il/peval.scm:
>   1691:38  7 (new-sym _)
> In ice-9/boot-9.scm:
>   1685:16  6 (raise-exception _ #:continuable? _)
>   1685:16  5 (raise-exception _ #:continuable? _)
>   1780:13  4 (_ #<&compound-exception components:
> (#<&assertion-failure> #<&origin origin: "cdr"> #<&message message:
> "Wrong type argument in position 1 (expecting pair): ~S"> #<&irritants
> irritants: (#f)> #<&exception-with-kind-and-args kind: wrong-type-arg
> args: ("cdr" "Wrong type argument in position 1 (expecting pair): ~S"
> (#f) (#f))>)>)
> In guix/build/compile.scm:
>     191:6  3 (_ _ . _)
> In ice-9/boot-9.scm:
>   1747:15  2 (with-exception-handler #<procedure 7f81dac67270 at
> ice-9/boot-9.scm:1831:7 (exn)> _ #:unwind? _ #:unwind-for-type _)
> In guix/build/compile.scm:
>    194:22  1 (_)
> In unknown file:
>            0 (make-stack #t)
Looking at language/tree-il/peval.scm 1691:38, it appears that somehow
the 'old' symbol does not appear in the environment.
For a minimal test case, remove everything but the 'define-module' and
the '(define-with-docs install [...])'.
Looks like something is going wrong with 'define-with-docs' (without
-with-docs and without the docstring, it compiles fine).
Greetings,
Maxime.
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
M
M
M
Maxim Cournoyer wrote on 19 Aug 2022 22:37
Re: bug#57121: clojure-build-system fails to compile -- backtrace from language/tree-il/peval.scm
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 57121@debbugs.gnu.org)
87ilmohswc.fsf@gmail.com
Hi Maxime.

Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (2 lines)
> Can't reproduce after touching java-utils.scm.

I got this issue a couple times too. I attributed it (without analysis)
to Guile's failure to keep track of changes to macro expanded code.

More like something to track on the side of Guile, I would think.

I'd suggest to close it here, since it's not reproducible.

Thanks,

Maxim
M
M
Maxime Devos wrote on 19 Aug 2022 22:58
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 57121@debbugs.gnu.org)
e43d48d5-1223-8149-e354-c30107d02c49@telenet.be
On 19-08-2022 22:37, Maxim Cournoyer wrote:
Toggle quote (7 lines)
> Hi Maxime.
>
> Maxime Devos <maximedevos@telenet.be> writes:
>
>> Can't reproduce after touching java-utils.scm.
> I got this issue a couple times too. I attributed it (without analysis)
> to Guile's failure to keep track of changes to macro expanded code.
It might be inlining. I don't see how macro expansion matters here. I
think it's dependency tracking in general.
Toggle quote (1 lines)
> More like something to track on the side of Guile, I would think.
I had a patch for build-aux/compile-all.scm that adds a form of
dependency tracking: https://issues.guix.gnu.org/50384. If we teach it
about (define-module (foo) #:use-module (bar)) --> (bar) is a dependency
of (foo) (using parts of source-module-closure?) (and drop the
search-patch things), then it seems solved to me.
As there is a known path to a solution, I wouldn't close this.
These patches are for Guix' build system.  I don't see anything that
could be done on the Guile side, except for eventually migrating some
dependency tracking stuff over to Guile -- "gcc" has an -M option to use
in combination with "make", maybe Guile could have something similar. 
Actually acting on the dependency information (which is part of the
patches) isn't something Guile can do, that seems more something for
"make" (when using Autotools and not like how Guix uses Autotools),
guile-build-system or build-aux/compile-all.scm to me.
Toggle quote (1 lines)
> I'd suggest to close it here, since it's not reproducible.
Just wait a few months or something, it keeps popping up -- I have
encountered this one some time in the past.
Also, non-determinism failures (which is a cause of irreproducibility)
are bugs in Guix.
Greetings,
Maxime
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
M
M
Maxim Cournoyer wrote on 22 Aug 2022 17:32
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 57121@debbugs.gnu.org)
878rngi99p.fsf@gmail.com
Hi Maxim,

Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (13 lines)
> On 19-08-2022 22:37, Maxim Cournoyer wrote:
>
>> Hi Maxime.
>>
>> Maxime Devos <maximedevos@telenet.be> writes:
>>
>>> Can't reproduce after touching java-utils.scm.
>> I got this issue a couple times too. I attributed it (without analysis)
>> to Guile's failure to keep track of changes to macro expanded code.
>
> It might be inlining. I don't see how macro expansion matters here. I
> think it's dependency tracking in general.

Dependency tracking is probably what I meant.

Toggle quote (14 lines)
>> More like something to track on the side of Guile, I would think.
>
> I had a patch for build-aux/compile-all.scm that adds a form of
> dependency tracking: <https://issues.guix.gnu.org/50384>. If we teach
> it about (define-module (foo) #:use-module (bar)) --> (bar) is a
> dependency of (foo) (using parts of source-module-closure?) (and drop
> the search-patch things), then it seems solved to me.
>
> As there is a known path to a solution, I wouldn't close this.
>
> These patches are for Guix' build system.  I don't see anything that
> could be done on the Guile side, except for eventually migrating some
> dependency tracking stuff over to Guile

If a module imports a different module, and that module changes, even if
it's macro, Guile should not blindly reuse the stale .go like it
currently does. It should complain and evaluate from source instead.

That would cover the base and avoid breakage. After, if it known how to
do that, yes, it seems it'd be useful to have something similar to 'gcc
-M' to provide the needed intelligence to the build system.

Does that make sense?

Thanks,

Maxim
M
M
Maxime Devos wrote on 22 Aug 2022 20:10
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 57121@debbugs.gnu.org)
d35f4d4d-830d-991a-1d55-33689182ed67@telenet.be
On 22-08-2022 17:32, Maxim Cournoyer wrote:
Toggle quote (12 lines)
>> These patches are for Guix' build system.  I don't see anything that
>> could be done on the Guile side, except for eventually migrating some
>> dependency tracking stuff over to Guile
> If a module imports a different module, and that module changes, even if
> it's macro, Guile should not blindly reuse the stale .go like it
> currently does. It should complain and evaluate from source instead.
>
> That would cover the base and avoid breakage. After, if it known how to
> do that, yes, it seems it'd be useful to have something similar to 'gcc
> -M' to provide the needed intelligence to the build system.
>
> Does that make sense?
Sounds reasonable, though we could go for something less general in Guix
first.
Greetings,
Maxime.
Attachment: file
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
M
M
Maxim Cournoyer wrote on 23 Aug 2022 06:06
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 57121@debbugs.gnu.org)
87y1vffvsv.fsf@gmail.com
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (17 lines)
> On 22-08-2022 17:32, Maxim Cournoyer wrote:
>>> These patches are for Guix' build system.  I don't see anything that
>>> could be done on the Guile side, except for eventually migrating some
>>> dependency tracking stuff over to Guile
>> If a module imports a different module, and that module changes, even if
>> it's macro, Guile should not blindly reuse the stale .go like it
>> currently does. It should complain and evaluate from source instead.
>>
>> That would cover the base and avoid breakage. After, if it known how to
>> do that, yes, it seems it'd be useful to have something similar to 'gcc
>> -M' to provide the needed intelligence to the build system.
>>
>> Does that make sense?
>
> Sounds reasonable, though we could go for something less general in
> Guix first.

I'd rather avoiding adding more complexity in Guix if it can be fixed
upstream; where it'd benefit everyone most.

Thanks,

Maxim
M
M
Maxime Devos wrote on 23 Aug 2022 11:07
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 57121@debbugs.gnu.org)
ae2686e0-0ad3-1ae6-cfae-9340e5b6868b@telenet.be
On 23-08-2022 06:06, Maxim Cournoyer wrote:
Toggle quote (21 lines)
> Hi Maxime,
>
> Maxime Devos<maximedevos@telenet.be> writes:
>
>> On 22-08-2022 17:32, Maxim Cournoyer wrote:
>>>> These patches are for Guix' build system.  I don't see anything that
>>>> could be done on the Guile side, except for eventually migrating some
>>>> dependency tracking stuff over to Guile
>>> If a module imports a different module, and that module changes, even if
>>> it's macro, Guile should not blindly reuse the stale .go like it
>>> currently does. It should complain and evaluate from source instead.
>>>
>>> That would cover the base and avoid breakage. After, if it known how to
>>> do that, yes, it seems it'd be useful to have something similar to 'gcc
>>> -M' to provide the needed intelligence to the build system.
>>>
>>> Does that make sense?
>> Sounds reasonable, though we could go for something less general in
>> Guix first.
> I'd rather avoiding adding more complexity in Guix if it can be fixed
> upstream; where it'd benefit everyone most.
It cannot be solved upstream, this is not a pure Guile matter but also a
build system matter. Even if this the 'if that module changes, it ...
should evaluate from source' was implemented, the build system still
needs to know to compile the module.
More concretely, build-aux/compile-all.scm uses the following to
determine what needs to be recompiled:
Toggle quote (4 lines)
> (define (file-needs-compilation? file)
>   (let ((go (scm->go file)))
>     (or (not (file-exists? go))
>         (file-mtime<? go file))))
--- just interpreting imported modules that have changed doesn't cause
the importing module to be recompiled.
Also, at least one Guile maintainer wants dependency tracking to be
orthogonal:
Toggle quote (7 lines)
> I guess I could live with an 80% dependency tracking solution. However,
> my criteria for it would be that (1) it should be orthogonal (e.g., not
> requiring explicit calls to ‘notice-dependency’ in many places), and (2)
> it should be self-contained and reasonably small. Perhaps having
> ‘load*’ or similar instrument ‘open-file’ or similar could help achieve
> that?
Your proposal to do in Guile does not appear to match that criterium.
The notice-dependency could be eventually upstreamed, but the issue does
not appear to be fixable upstream.
Greetings,
Maxime.
Attachment: file
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
M
M
Maxim Cournoyer wrote on 27 Aug 2022 17:53
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 57121@debbugs.gnu.org)
87tu5xbs32.fsf@gmail.com
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (28 lines)
> On 23-08-2022 06:06, Maxim Cournoyer wrote:
>> Hi Maxime,
>>
>> Maxime Devos<maximedevos@telenet.be> writes:
>>
>>> On 22-08-2022 17:32, Maxim Cournoyer wrote:
>>>>> These patches are for Guix' build system.  I don't see anything that
>>>>> could be done on the Guile side, except for eventually migrating some
>>>>> dependency tracking stuff over to Guile
>>>> If a module imports a different module, and that module changes, even if
>>>> it's macro, Guile should not blindly reuse the stale .go like it
>>>> currently does. It should complain and evaluate from source instead.
>>>>
>>>> That would cover the base and avoid breakage. After, if it known how to
>>>> do that, yes, it seems it'd be useful to have something similar to 'gcc
>>>> -M' to provide the needed intelligence to the build system.
>>>>
>>>> Does that make sense?
>>> Sounds reasonable, though we could go for something less general in
>>> Guix first.
>> I'd rather avoiding adding more complexity in Guix if it can be fixed
>> upstream; where it'd benefit everyone most.
>
> It cannot be solved upstream, this is not a pure Guile matter but also
> a build system matter. Even if this the 'if that module changes, it
> ... should evaluate from source' was implemented, the build system
> still needs to know to compile the module.

OK. I'd welcome fixing as much of it as possible in Guile, then adding
the Guix-specific bits on top of it. We already carry a bit too many
things in Guix that could be in Guile proper, in my opinion.

Toggle quote (11 lines)
> More concretely, build-aux/compile-all.scm uses the following to
> determine what needs to be recompiled:
>
>> (define (file-needs-compilation? file)
>>   (let ((go (scm->go file)))
>>     (or (not (file-exists? go))
>>         (file-mtime<? go file))))
> --- just interpreting imported modules that have changed doesn't cause
> the importing module to be recompiled.
>

[...]

OK, I see that since we already implement such mechanism in Guix, it's
on our shoulders to fix it there. I'm changing my mind but with what I
said above (as much of the fix to go in Guile, the rest on top in Guix).

Thank you,

Maxim
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 57121@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 57121
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch