taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis: > Sadly that assumption isn't met when autoloads are involved. > Minimal-ish test-case: > > - Check out 0889321. > > - Build it. > > - Edit gnu/build/activation.scm and gnu/build/linux-boot.scm to contain > merely the following expressions, respectively: > > (define-module (gnu build activation) > #:use-module (gnu build linux-boot)) > > (define-module (gnu build linux-boot) > #:autoload (system base compile) (compile-file)) > > - Run make again. > > If you're on a multi-core system, you will probably get an error saying > something weird like "no such language scheme". Do you have a clear explanation of why this happens? I would expect (system base compile) to already be loaded for instance, so it’s not clear to me what’s going on. Or is it just the mutation of (gnu build linux-boot) that’s causing problems? > Solution proposals: > > 1. s/par-for-each/for-each/. Will make compilation slower on multi-core > machines. We would do the same for guix pull, which is a bit sad > because it's so fast right now. Very simple solution though. > > 2. We find out some partitioning of the Scheme modules such that there > is minimal overlap in total loaded modules when the modules in one > subset are each loaded by one Guile process. Then each Guile process > loads & compiles the modules in its given subset serially, but these > Guile processes run in parallel. This could speed things up even > more than now because the module-loading phases of the processes > would be parallel too. It also has the side-effect that less memory > is consumed the fewer cores you have (because less Scheme modules > loaded into memory at once). If someone (Ludo?) has a good general > overview of Guix's module graph then maybe they can come up with a > sensible partitioning of the modules, say into 4 subsets (maxing out > benefits at quad-core), such that loading all modules in one subset > loads a minimal amount of modules that are outside that subset. That > should be the only challenging part of this solution. > > 3. We do nothing for now since this bug triggers rarely, and can be > worked around by simply re-running make. (We just have to hope that > it doesn't trigger on guix pull or on clean builds after some commit; > there's no "just rerun make" in guix pull or an automated build of > Guix.) AFAIU Wingo expressed motivation to make Guile's module > system thread safe, so this problem would then truly disappear. Short-term, I’d do #1 or #3; probably #1 though, because random failures are no fun, and we know they can happen. Longer-term, I’m not convinced by #2. I think I would instead build packages in reverse topological order, probably serially at first, which would address (with the caveat that the (gnu packages …) modules cannot be topologically-sorted, but OTOH they typically don’t use macros, so we’re fine.) That would require a tool to extract and the ‘define-module’ forms and build a graph from there. But really, we must fix , an in particular, ‘compile-file’ should not mutate the global module name space. I think we could do something like: (define (compile-file* …) (let ((root the-root-module) (compile-root (copy-module the-root-module))) (dynamic-wind (lambda () (set! the-root-module compile-root) ;; ditto with the-scm-module ) (lambda () (compile-file …)) (lambda () (set! the-root-module root) ;; … )))) It’s unclear how costly ‘copy-module’ would be, and the whole strategy depends on it. Eventually it seems clear that Guile proper needs to address this use case, and needs to provide thread-safe modules. Ludo’.