[PATCH 0/6] Start breaking up import cycles

Import cycles make some packaging things harder and prevent some

proposed optimisations to "guix pull", let's start eliminating them.

TBC ...

This partially breaks a cycle between (gnu packages linux) and

(gnu packages gtk). It is not sufficient though, see next patch.

---

gnu/packages/linux.scm | 38 --------------------------------------

gnu/packages/xorg.scm | 41 +++++++++++++++++++++++++++++++++++++++++

2 files changed, 41 insertions(+), 38 deletions(-)

Currently, there's a cycle between ncurses, linux, audio and music.

By moving audacity to its own module, 'audio' does not have to import

'music' and 'linux' anymore. Looking at the output of

"guix graph --type=module", there still appears to be cycles, so

this commit will not sufficient by itself.

from the old module. Add a few comments to the old module to discourage

re-importing the 'linux' and 'music' module in the future.

---

gnu/local.mk | 1 +

gnu/packages/audio.scm | 140 ++---------------------------------------

2 files changed, 5 insertions(+), 136 deletions(-)

This (partially?) breaks a cycle between (gnu packages linux) and

(gnu packages networking).

---

gnu/packages/linux.scm | 30 ++----------------------------

gnu/packages/networking.scm | 28 ++++++++++++++++++++++++++++

2 files changed, 30 insertions(+), 28 deletions(-)

This (partially?) breaks a cycle between (gnu packages linux),

(gnu packages haskell-xyz) and (gnu packages golang).

It seems a bit cheaty to me but it works.

'go' and 'pandoc'. Remove now unneeded imports.

---

gnu/packages/linux.scm | 18 ++++++++++++------

1 file changed, 12 insertions(+), 6 deletions(-)

This (partially?) breaks a cycle between (gnu packages linux)

and (gnu packages check). There's more work to be done though,

see future patches -- stay tuned!

(python-pamela,earlyoom,multipath-tools,mtd-utils,procenv):

Don't import (gnu packages check), instead access that module

via 'module-ref' + 'resolve-interface'.

---

gnu/packages/linux.scm | 31 +++++++++++++++++++++++++------

1 file changed, 25 insertions(+), 6 deletions(-)

This partially breaks a cycle between (gnu packages linux) and

(gnu packages gtk). It is not sufficient though, see next patch.

---

gnu/local.mk | 1 +

gnu/packages/linux.scm | 192 +---------------------------------

gnu/packages/tlp.scm | 228 +++++++++++++++++++++++++++++++++++++++++

3 files changed, 231 insertions(+), 190 deletions(-)

create mode 100644 gnu/packages/tlp.scm

Maxime Devos schreef op wo 23-03-2022 om 19:46 [+0100]:

The copyright lines are based on a mix of "git blame" and "git log --

grep the-package"; I might have missed some people ...

Hi Maxime,

Am Mittwoch, dem 23.03.2022 um 19:49 +0100 schrieb Maxime Devos:

I agree that breaking up cycles is a good thing, but I disagree with

some of the decisions you've made here. For instance, I oppose the use

of single-package modules, because those more often than not simply

clutter the file system.

I'm not sure if Guile's #:autoload could do anything to fix these

issues (I suppose not), but long term I think guile modules should

support a style that is basically (resolve-interface) + (module-ref) in

the manner Guix needs, but declaratively. While we do not have that in

place yet, I suggest something like the following:

(define (check-package-ref pkg)

(module-ref (resolve-interface '(gnu packages check)) pkg))

...

or

(define check-package-ref

(let ((iface (resolve-interface '(gnu packages check))))

(lambda (pkg) (resolve-interface iface pkg))))

I'm not sure if the second will have the intended effect. It appears

to me as though the key to breaking these cycles is moving them into a

context that is not evaluated on the top of the file, e.g. a thunked

field or in the case of my first suggestion a procedure.

In either case, declaring all these lazy dependencies near the module

definition would have the added benefit, that people could see them

being lazily imported and thus no longer need the #:use-modules

comment.

WDYT?

Liliana Marie Prikler schreef op do 24-03-2022 om 08:22 [+0100]:

There are some other sound applications in (gnu packages audio)

and (gnu packages music), so maybe I can make a (gnu packages audio-

apps) module where 'audacity' and other applications like 'calf' can

reside?

For reducing the contribution of (gnu packages compression) on the

cycle issue, I've (in not yet submitted patched) separated many things

into (gnu packages compression-xyz), perhaps I can merge

(gnu packages patools) into (gnu packages compression-xyz)?

Greetings,

Maxime.

Am Donnerstag, dem 24.03.2022 um 16:05 +0100 schrieb Maxime Devos:

I'm not sure. IIUC, audio should be for audio systems, codecs, etc.

whereas music sounds like a particular niche containing music players

etc. Perhaps the cycle could more appropriately been broken by moving

stuff from music to audio? Alternatively, declaring music as lazy

import as shown in my previous mail might help making these

interdependencies both "cycle-free" and visible. (I'm pretty sure we

also have "sound" lying around somewhere to put more oil into the

fire.)

Here too, I think a classification into compression algorithms in

compression and backup/archival tools in another file (we do have

backup IIRC) would make the most sense. Though obviously, we'd have to

do compression algorithms implemented in Rust in a special rust-

compression file to avoid circles or use the cycle killer lambda trick.

I'm not sure if "compression-xyz" would be a helpful label, and it

might just become the next root of circular dependencies if abused.

Cheers

Liliana Marie Prikler schreef op do 24-03-2022 om 16:38 [+0100]:

Except for zpaq, there don't appear to be any backup tools in (gnu

packages compression). There are no rust compression things in

(gnu packages compression) (yet!). But yes, there's some opportunity

for abuse here. There _are_ archival tools in (gnu packages

compression), e.g. tar and unzip. Moving them to (gnu packages

backup) wouldn't bring any benefit though, since they are used by

practically everything.

Anyway, I'll continue trying to break cycles ...

Greetings,

Maxime.

Hi,

On Thu, 24 Mar 2022 at 08:23, Liliana Marie Prikler

<liliana.prikler@ist.tugraz.at> wrote:

Well, instead of opinions in the vacuum of matter, we need to profile

and decide on performance report. The number of files and the number

of package per file should consider the performance of:

- compilation by developer

- guix pull

- guix search (or any other)

- guix search --load-path

- etc.

I agree that lazyness is a good thing and a good direction. However,

let be pragmatic with what we have now. :-) What are the performance

comparison between breaking many cycles as Maxime is proposing vs

using many 'module-ref' + 'resolve-interface' instead of break?

Cheers,

simon

On Thu, Mar 24, 2022 at 08:22:09AM +0100, Liliana Marie Prikler wrote:

The file system can hold many files. The Guix codebase is nowhere near

the limit...

zimoun schreef op do 24-03-2022 om 17:58 [+0100]:

Currently this patch series does not improve anything much, according

to "guix graph --type=module hello | wc --lines". I'm now introducing

some module-ref+resolve-interface --- it's very convenient, but I'm not

yet at a significant result.

Anyway, restructuring modules and lazy loading can be complementary,

whatever's convenient.

Greetings,

Maxime.

Maxime Devos schreef op wo 23-03-2022 om 19:46 [+0100]:

Status update: previously the output of

"guix graph -t module hello --max-depth=9 |wc --lines" reports ~1040

lines. With WIP patches, it now reports 280 lines. I'm gradually

increasing the max-depth and each time investigating how to leave xorg,

gtk, ... out of the output.

Greetings,

Maxime.

Am Donnerstag, dem 24.03.2022 um 19:07 +0100 schrieb Maxime Devos:

For the record, my suggestion to declare lazily loaded modules near the

top is based on the fact that Maxime's current patch set uses them to

break up cycles in a manner that also requires a comment in the define-

module clause for the sake of clarity. As a nice side effect, it makes

it so that two-liners in the inputs field become one-liners.

The question is (on a per-module basis) whether we consider this cheat

fine or whether we want to move things into different files (and

which). I so far haven't heard a good argument for the case of

audacity I raised. "It breaks cycles" is not good enough when we

consider the potential existence of other cuts (e.g. "audio-apps",

although perhaps a more specific "audio-editors" similar to how we have

"image-viewers" might make more sense), as well as the cheat of lazy

imports.

simon, you raise some important performance metrics, but there is such

a thing as optimizing for the wrong metric. There are other variables

to consider, like time to grep, "does it make sense that X belongs to Y

and Z doesn't", etc., when it comes to ease of contributing. Declaring

some modules banned for a given other module has an adverse effect

here, in my opinion, and thus I claim that we need easily accessible

ways of using those supposedly banned modules.

Btw. regarding style, I think declaring a function @PACKAGE_MODULE,

i.e. a literal '@ followed by the last symbol in the module's name,

would be the easiest, as one could read (@PACKAGE_MODULE arg) as a

shorthand for (@ (gnu packages PACKAGE_MODULE) arg). Somewhat off-

topic, what's the rationale behind not using @ syntax? Does @ have

different semantics from resolve-interface + module-ref?

Cheers

Am Donnerstag, dem 24.03.2022 um 13:05 -0400 schrieb Leo Famulari:

That's not a good argument, though. Theoretically, you could have one

file per package, or even one folder per package as the Gentoo folks

do. Clearly, this is not an appeal to file system limits, but to

limitations of humans and their tools. Have you tried exploring

/gnu/store? Merely loading it takes a considerable amount of time.

Cheers

Liliana Marie Prikler schreef op do 24-03-2022 om 16:38 [+0100]:

I'm not sure where the ‘I'm not sure’ comes from --- audacity is not an

audio system like pulseaudio or alsa, not a codec implementation like

libvorbis, and it is a sound player (and editor), so IIUC, audacity

does not fit into (gnu packages audio).

While it can be used for modifying and playing music, it is more

general than that, hence the suggestion of (gnu packages audio-apps)

(where some other packages can be moved to as well, maybe 'gnaural'?).

Though granted, it's difficult to make a strict distinction between

audio and music.

(gnu packages music) is full of applications using gtk+ or gles or the

like. Moving them to (gnu packages audio) would make (gnu packages

audio) depend on (gnu packages gtk) and friends, which seems counter-

productive to me.

Greetings,

Maxime.

Am Freitag, dem 25.03.2022 um 11:26 +0100 schrieb Maxime Devos:

It's not particularly specific to audacity in this case, it's that I

think that "-app" does not make for a useful distinction. Consider

fluidsynth. Is it an app, a library, something else? If fluidsynth

was causing circular imports and moving it to audio-synthesizers fixed

things, that'd be fine by me. gnaural could also fit into

synthesizers. However, putting both in the same file might still be an

issue from a cycle perspective, because the latter needs gtk, whereas

the former is content with having just glib (although that one appears

benign on surface, as at the very least gtk does not audio directly).

That's the status quo (through gnaural and audacity for example). To

make a more educated guess, which cycle do we aim to address here? Is

there a meaningful cut that can be made (e.g. the offending packages

are all "audio editors", "audio synthesizers", etc.) or do we have to

separate good and bad based on their inputs?

Cheers

Liliana Marie Prikler schreef op vr 25-03-2022 om 12:47 [+0100]:

Some cycles:

(gnu packages pulseaudio) -> (gnu packages audio) -> (gnu packages gtk)

/(gnu packages qt)-> (gnu packages pulseaudio) + the world

(gnu packages pulseaudio) -> (gnu packages audio) -> (gnu packages

webkit) -> (gnu packages gstreamer) + the world -> (gnu packages

pulseaudio) + the world

Suggested cut: audio libraries like flac, libogg, libvorbis, opus,

wildmidi, vo-aacenc, tinyalsa ... can go in (gnu packages audio), other

things go somewhere else, especially if they need expensive imports.

Greetings,

Maxime.

Am Freitag, dem 25.03.2022 um 15:12 +0100 schrieb Maxime Devos:

Hmm, is it all codecs? In that case, I'd suggest making a smaller (gnu

packages audio-codecs), that can be used by (gnu packages audio-

systems) [including (tiny)alsa, pulseaudio, jack, ...] that can be used

by the rest of the world. Having that, we could move "the rest" into

audio-xyz (or let it simply remain "audio"). Would that be actionable?

(Note: There might still be debate w.r.t. the above split when

considering synthesizers, as they are technically not codecs, but we

still need to distinguish between low-level synths like fluidsynth and

wildmidi vs. full-on sound stations.)

(Note2: Of course, this assumes that neither audio-codecs nor audio-

systems will ever need to import any of the rust stuff. *sigh*)

Cheers

Maxime Devos schreef op wo 23-03-2022 om 19:46 [+0100]:

I (locally) reverted the six patches I sent previously and I tried out

the 'module-ref'+'resolve-interface' strategy on (gnu packages linux)

(see attached patch). WDYT of repeating this with a few other

‘strategic’ modules, e.g. (gnu packages compression), (gnu packages

perl), (gnu packages version-control)?

Greetings,

Maxime.

One of the things I've tried out, is moving 'bc' from (gnu packages

algebra) to (gnu packages base). It allowed removing (gnu packages

algebra) from the imports of many packages. Does this seem reasonable?

Hi Liliana and Maxime,

On Fri, 25 Mar 2022 at 09:44, Liliana Marie Prikler

<liliana.prikler@ist.tugraz.at> wrote:

To be honest, I am not sure to understand the aim of reorganizing the

modules... I mean, to me, the only important metrics is the

performance of the end-user. If there is no performance improvement

when cutting cycle, then it appears to me pointless to cut cycles. :-)

Moreover, set an arbitrary boundary between packages is... arbitrary.

You can spend close to eternity for discussing "does it make sense

that X belongs to Y and Z doesn't". To me, such activity is like

"tagging" (assign a specific word belonging to a finite set of words),

it is usually a lot of effort and energy for, at the end, few, if not

none, pragmatic outcomes.

Last, for classification (assign a package to one module depending on

the affinity with the other packages of that module), well, it could

almost arbitrary (manual depending of human choice) as it is now or it

could be self-organized depending on the data themself. From my point

of view, it could be interesting to apply some kind of self-organized

map (SOM) and other related things. It could be help for many other

issues as "search". Pointers for what they are worth:

Cheers,

simon

zimoun schreef op vr 25-03-2022 om 18:05 [+0100]:

FWIW, there are three goals here:

* Allowing writing stuff like

(use-modules (gnu packages ncurses))

(package

(name "some-terminal-app")

[...]

;; Work-around the ‘search path of dependencies not propagated’ bug.

(native-search-paths (package-native-search-paths ncurses)))

without getting 'unbound variable' errors.

Alternatively, the search-path-specification could be defined in,

say, (guix search-paths) next to $PATH but that proposal seems

to have been rejected.

* Making "compute-guix-derivation" faster by reducing the number of

(uncompiled!) package module files it needs to load.

* Eventually making the ‘incremental compilation’ by fine-grained derivations

proposal from the ‘Faster "guix pull" by incremental compilation and

non-circular modules?’ thread [0] feasible.

As a side benefit, it makes the output of "guix graph --type=module foo"

less cluttered and presumably reduces the module closure (a likely

side-effect of breaking cycles), making "guix show foo" (*) a bit faster.

(*) FWIW, on my machine, "guix show guix" takes 1.6s.

[0]: https://lists.nongnu.org/archive/html/guix-devel/2022-02/msg00193.html

Greetings,

Maxime.

Hi Maxime,

On Fri, 25 Mar 2022 at 18:46, Maxime Devos <maximedevos@telenet.be> wrote:

Thanks for the detailed and clear explanations. It was my initial

understanding that cutting cycles can improve the performances, and

IMHO, timings are required for comparing apple to apple; as I tried to

explain [1]. Then the thread have let me the impression that the

performance improvement was not the aim -- thanks for clarifying.

1: https://issues.guix.gnu.org/54539#12

To be precise, "guix show guix" could be drastically improved by

adapting the already existing package.cache, i.e., resume the lengthy

work of https://issues.guix.gnu.org/39258. :-)

However, such cache would be useless for "guix show -L path/to/others

foo" where performance can be really poor; especially on spinning hard

disk. Well, thanks for working on that by trying to tackle the cycle

of modules.

Cheers,

simon

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

Sorry for the late reply.

Some of the changes you propose may make sense (and should be applied),

but we shouldn’t overplay the role of such changes.

If you follow the logic, breaking up import cycles would mean, in the

end, having one file per package.

But would that be enough? Probably not, because low-level packages are

bound to depend on high-level packages—e.g., glibc depends on Python,

some other low-level tool might depend on Pandoc (GHC), librsvg depends

on Rust, and so on.

IOW, since the graph of build dependency really is a graph, and not a

tree, there’ll always be import cycles.

(guix self), the module that ‘guix pull’ uses, already automatically

splits package modules into two groups. It’s not as modular as we’d

like, but it’s a start. What would be useful is to come up with metrics

and tools to reduce the closure of the “guix-packages-base” group.

WDYT?

Thanks,

Ludo’.

Ludovic Courtès schreef op di 19-04-2022 om 11:17 [+0200]:

Not necessarily, (gnu packages minetest) has multiple packages

(minetest and some of its mods) but it doesn't cause any cycles (no

other module, except sort-of (guix build-system minetest), imports it.)

That one appears to be, at least currently, a bit of a special case

though.

The graph of build dependencies (in terms of derivations) is a tree,

the build daemon doesn't allow cyclic derivations. So I think that by

letting the module graph be a coarser version of the derivation graph

but still a tree (except for the bootstrap packages gcc, sed, ... whose

modules may import each other).

Maybe:

a tool that determines a minimal set of (importing module ->

imported module tuples) that needs to be lazified to reduce the

closure size (in number of modules) in guix-packages-base by N

and:

extend "guix style" to perform these changes

Maybe the ‘number of imports lazified -> number of modules in guix-

packages-base’ function has some sweet spot somewhere.

I think it would be easier though to work our way up before going to

"guix pull" -- first "hello", then "util-linux, then "guile-avahi",

then "guile-ssh", then "sqlite" ... and only eventually guix itself.

Also, even if the closure of "guix-packages-base" cannot be reduced,

making it (mostly) a tree would allow splitting the group into multiple

parts (see ‘Faster "guix pull" by incremental compilation and non-

circular modules?’).

Alternative:

* make _all_ package module imports lazy -- #:autoload everything!

guix-packages-base might then need to be set manually though ...

Greetings,

Maxime.

Ludovic Courtès schreef op di 19-04-2022 om 11:17 [+0200]:

Maybe as a first step, "guix style" could be taught to trim unused

imports? When writing the patches it turned out that some imports were

unnecessary and could therefore be removed ...

Greetings,

Maxime.

Hi!

Maxime Devos <maximedevos@telenet.be> skribis:

Yes, that would be nice.

Ludo’.

Maxime Devos <maximedevos@telenet.be> skribis:

I think so. All the historical package modules started that way.

It’s a directed acyclic graph (DAG), not a tree.

I thought so, but came to the conclusion that it’s hardly feasible in

practice.

Currently ‘source-module-closure’ considers #:autoloaded modules as part

of the closure; we could change that though and indeed, that might prove

helpful in this case.

Could be.

I don’t know, having spent some time on this, I feel like there’s no

easy solution. But it could be that using autoloads at least in the

right places would help shrink ‘guix-packages-base’. Worth a try!

Ludo’.

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding Java.

---

gnu/packages/fontutils.scm | 71 ++++++++++++++++++++------------------

1 file changed, 37 insertions(+), 34 deletions(-)

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding Java.

---

gnu/packages/ruby.scm | 64 +++++++++++++++++++++----------------------

1 file changed, 31 insertions(+), 33 deletions(-)

On 03-09-2022 18:43, Maxime Devos wrote:

(The linter is currently running in the background)

Greetings,

Maxime.

See first commit in the series for the procedure.

---

gnu/packages/documentation.scm | 43 +++++++++++++++++-----------------

1 file changed, 22 insertions(+), 21 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/python-xyz.scm | 282 +++++++++++++++++++++++-------------

1 file changed, 182 insertions(+), 100 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/tcl.scm | 13 ++++++-------

1 file changed, 6 insertions(+), 7 deletions(-)

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding Java.

---

gnu/packages/xml.scm | 47 +++++++++++++++++++++++++-------------------

1 file changed, 27 insertions(+), 20 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/perl.scm | 68 ++++++++++++++++++++++++++-----------------

1 file changed, 41 insertions(+), 27 deletions(-)

Goal: reduce the amount of modules that "compute-guix-derivation" needs to

load (reducing memory usage). As a side benefit, the start-up latency

of various "guix build ..." commands should decrease -- at worst, it should

remain as the same as before.

Delete #:use-module, check that "./pre-inst-env guix build -d guix" still

works. If so, turn the use-module into an #:autoload. If not, reinstate

the #:use-module. Some imported modules are loaded anyway by dependencies

and as such could theoretically be left intact for the purposes of

https://issues.guix.gnu.org/54539, but that's more developer work for no

apparent benefit.

Sometimes, the module wasn't used at all. In that case, it is removed.

On its own, this commit doesn't accomplish much, so a report will be given

in the latest commit of the series.

---

gnu/packages/package-management.scm | 155 +++++++++++++++-------------

1 file changed, 86 insertions(+), 69 deletions(-)

It's not used by Guix according to "./pre-inst-env guix build guix".

Theoretically (gnu packages pkg-config) could be autoloaded, but

it seems pointless.

See first commit in the series for the procedure.

---

gnu/packages/base.scm | 3 +--

1 file changed, 1 insertion(+), 2 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/crypto.scm | 98 +++++++++++++++++++++++------------------

1 file changed, 54 insertions(+), 44 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/check.scm | 63 ++++++++++++++++++++++++++++--------------

1 file changed, 43 insertions(+), 20 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/backup.scm | 99 ++++++++++++++++++++++-------------------

1 file changed, 53 insertions(+), 46 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/gnupg.scm | 74 ++++++++++++++++++++++--------------------

1 file changed, 39 insertions(+), 35 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/gettext.scm | 29 +++++++++++++++++------------

1 file changed, 17 insertions(+), 12 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/python.scm | 32 +++++++++++++++-----------------

1 file changed, 15 insertions(+), 17 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/guile-xyz.scm | 119 +++++++++++++++++++------------------

1 file changed, 61 insertions(+), 58 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/admin.scm | 185 +++++++++++++++++++++++------------------

1 file changed, 104 insertions(+), 81 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/databases.scm | 212 ++++++++++++++++++++++---------------

1 file changed, 128 insertions(+), 84 deletions(-)

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding loading the Java-related

modules when only the guix package is required.

---

gnu/packages/icu4c.scm | 7 ++++---

1 file changed, 4 insertions(+), 3 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/docbook.scm | 32 +++++++++++++++++++++-----------

1 file changed, 21 insertions(+), 11 deletions(-)

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding loading the Go-related

modules when only the guix package is required.

---

gnu/packages/curl.scm | 31 +++++++++++++++++--------------

1 file changed, 17 insertions(+), 14 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/elf.scm | 19 +++++++++----------

1 file changed, 9 insertions(+), 10 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/linux.scm | 167 +++++++++++++++++++++--------------------

1 file changed, 87 insertions(+), 80 deletions(-)

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding the GTK and Qt stacks.

---

gnu/packages/compression.scm | 53 ++++++++++++++++++------------------

1 file changed, 26 insertions(+), 27 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/hurd.scm | 30 +++++++++++++++---------------

1 file changed, 15 insertions(+), 15 deletions(-)

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding Java.

---

gnu/packages/algebra.scm | 66 +++++++++++++++++++++-------------------

1 file changed, 35 insertions(+), 31 deletions(-)

See first commit in the series for the procedure.

---

gnu/packages/cmake.scm | 26 +++++++++++++-------------

1 file changed, 13 insertions(+), 13 deletions(-)

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding loading the Go-related

modules when only the guix package is required.

---

gnu/packages/version-control.scm | 133 +++++++++++++++++--------------

1 file changed, 75 insertions(+), 58 deletions(-)

See first commit in the series for the procedure.

This helps with, but is not sufficient for, avoiding Java, Go and Qt.

---

gnu/packages/web.scm | 219 +++++++++++++++++++++++++------------------

1 file changed, 128 insertions(+), 91 deletions(-)

TODO: check that pulling still works and that

"./pre-inst-env guix lint --checker=derivation" doesn't report porblems.

(All the performance numbers below are for a SSD, on a HDD I would expect

the differences to be more pronounced.)

The number of modules indirectly used by the Guix package is reduced from

$ guix graph --type=module guix | grep -F 'label' | wc --lines

to

$ ./pre-inst-envguix graph --type=module guix | grep -F 'label' | wc --lines

(Note: you will have to patch source-module-closure first to ignore #:autoload.)

I.e., it is reduced by about half!

Memory usage and time spent in GC, before and after the patch (run thrice).

Before:

$ GUIX_PROFILING=gc guix build -d -e '(@ (gnu packages package-management) guix)' --no-grafts

Garbage collection statistics:

heap size: 72.46 MiB

allocated: 184.59 MiB

GC times: 15

time spent in GC: 1.04 seconds (56% of user time)

Garbage collection statistics:

heap size: 72.52 MiB

allocated: 184.60 MiB

GC times: 15

time spent in GC: 1.11 seconds (58% of user time)

Garbage collection statistics:

heap size: 72.46 MiB

allocated: 184.59 MiB

GC times: 15

time spent in GC: 1.05 seconds (57% of user time)

After:

$ GUIX_PROFILING=gc ./pre-inst-env guix build -d -e '(@ (gnu packages package-management) guix)' --no-grafts

Garbage collection statistics:

heap size: 46.95 MiB

allocated: 174.24 MiB

GC times: 17

time spent in GC: 0.88 seconds (55% of user time)

Garbage collection statistics:

heap size: 46.95 MiB

allocated: 174.24 MiB

GC times: 17

time spent in GC: 0.88 seconds (54% of user time)

Garbage collection statistics:

heap size: 46.95 MiB

allocated: 174.23 MiB

GC times: 17

time spent in GC: 0.88 seconds (55% of user time)

The heap size is lower: 72.50 MiB -> 45.95 MiB.

Don't know what's up with the GC here, but the percentage didn't increase,

so doesn't seem bad to me.

Now, start-up latency.

First, do

$ echo powersave | sudo tee /sys/devices/system/cpu/cpufreq/policy*/scaling_governor

to avoid the OS from interfering by making the CPU faster.

Before:

$ guix shell hyperfine -- hyperfine -w2 -- "guix build -d -e '(@ (gnu packages package-management) guix)' --no-grafts"

Time (mean ± σ): 4.137 s ± 0.017 s [User: 5.516 s, System: 0.270 s]

Range (min … max): 4.108 s … 4.160 s 10 runs

After:

$ guix shell hyperfine -- hyperfine -w2 -- "./pre-inst-env guix build -d -e '(@ (gnu packages package-management) guix)' --no-grafts"

Time (mean ± σ): 3.830 s ± 0.038 s [User: 4.976 s, System: 0.241 s]

Range (min … max): 3.794 s … 3.922 s 10 runs

-- a small but measurable improvement. This was with warm disk caches, what about cold

disk caches? As the number of loaded modules is reduced, the number of 'read from disk<->Guile determines

what to read next' roundtrips decreases, which should make things faster (especially on spinning disks).

Before:

$ guix shell hyperfine -- hyperfine --prepare='sync; echo 3 |sudo tee /proc/sys/vm/drop_caches' -w2 -- "guix build -d -e '(@ (gnu packages package-management) guix)' --no-grafts"

Benchmark #1: guix build -d -e '(@ (gnu packages package-management) guix)' --no-grafts

Time (mean ± σ): 5.146 s ± 0.072 s [User: 5.647 s, System: 0.442 s]

Range (min … max): 5.072 s … 5.274 s 10 runs

After:

guix shell hyperfine -- hyperfine --prepare='sync; echo 3 |sudo tee /proc/sys/vm/drop_caches' -w2 -- "./pre-inst-env guix build -d -e '(@ (gnu packages package-management) guix)' --no-grafts"

Benchmark #1: ./pre-inst-env guix build -d -e '(@ (gnu packages package-management) guix)' --no-grafts

Time (mean ± σ): 4.536 s ± 0.032 s [User: 4.924 s, System: 0.329 s]

Range (min … max): 4.480 s … 4.577 s 10 runs

Again, an improvement.

Let's look at the number of syscalls this time. I don't know what's up with the symlinks.

Before:

$ guix shell strace -- strace -c guix build -e '(@ (gnu packages package-management) guix)' -d --no-grafts

% time seconds usecs/call calls errors syscall

------ ----------- ----------- --------- --------- ------------------

26,39 0,015695 2 6350 5754 readlink

18,64 0,011083 1 6808 read

16,97 0,010092 26 377 87 futex

12,87 0,007654 2 2768 268 newfstatat

8,57 0,005099 1 2576 write

5,13 0,003053 2 1024 212 openat

3,74 0,002222 2 774 mmap

2,17 0,001288 2 568 mprotect

2,15 0,001280 1 815 close

1,64 0,000976 1 770 3 lseek

0,77 0,000456 2 197 getcwd

------ ----------- ----------- --------- --------- ------------------

100,00 0,059467 2 23217 6327 total

After:

./pre-inst-env guix shell strace -- strace -c guix build -e '(@ (gnu packages package-management) guix)' -d --no-grafts

% time seconds usecs/call calls errors syscall

------ ----------- ----------- --------- --------- ------------------

32,70 0,014266 1 7820 5684 newfstatat

19,25 0,008398 1 6808 read

11,25 0,004907 12 391 90 futex

10,25 0,004470 1 2580 write

7,67 0,003345 2 1622 1621 readlink

7,33 0,003198 3 847 212 openat

3,84 0,001676 2 561 mmap

2,27 0,000992 2 391 mprotect

2,27 0,000991 1 638 close

1,66 0,000726 1 592 3 lseek

0,83 0,000361 1 197 getcwd

0,28 0,000123 5 21 pread64

[ other syscalls removed for brevity ]

------ ----------- ----------- --------- --------- ------------------

100,00 0,043627 1 22643 7613 total

'mmap' is called less, I don't know how much of that can be attributed

to less .go files being mmapped and how much to the lower memory usage.

I'd assume that the reduction in 'openat' uses (1024->847) is due to

less modules (and maybe as a consequence, less patch files?).

There were a few things unclear with the performance numbers above, but I

think I can state that this patch series does, in fact, reduce memory usage

(CPU time and heap size for computing "guix build -d guix --no-grafts"),

albeit less than I hoped for.

The same methods can be applied for other packages as well -- maybe in the

future, we could simply do an #:autoload for _every_ imported package module.

The latest commit was a bit ad-hoc, to save time making this patch series.

I would expcet that, with some additional work, more modules can be removed

from the graph.

---

gnu/packages/audio.scm | 29 ++++---

gnu/packages/build-tools.scm | 14 ++--

gnu/packages/c.scm | 2 +-

gnu/packages/cdrom.scm | 2 +-

gnu/packages/code.scm | 6 +-

gnu/packages/cpp.scm | 10 +--

gnu/packages/cups.scm | 6 +-

gnu/packages/dns.scm | 2 +-

gnu/packages/freedesktop.scm | 22 +++---

gnu/packages/gdb.scm | 2 +-

gnu/packages/gl.scm | 6 +-

gnu/packages/glib.scm | 2 +-

gnu/packages/gnome.scm | 117 ++++++++++++++++------------

gnu/packages/graphics.scm | 44 +++++------

gnu/packages/gstreamer.scm | 25 +++---

gnu/packages/gtk.scm | 13 ++--

gnu/packages/image-processing.scm | 6 +-

gnu/packages/image.scm | 12 +--

gnu/packages/inkscape.scm | 3 +-

gnu/packages/libusb.scm | 5 +-

gnu/packages/llvm.scm | 7 +-

gnu/packages/logging.scm | 2 +-

gnu/packages/maths.scm | 31 ++++----

gnu/packages/mpi.scm | 6 +-

gnu/packages/music.scm | 39 ++++++----

gnu/packages/networking.scm | 24 ++++--

gnu/packages/openldap.scm | 2 +-

gnu/packages/password-utils.scm | 2 +-

gnu/packages/pdf.scm | 24 +++---

gnu/packages/photo.scm | 8 +-

gnu/packages/polkit.scm | 4 +-

gnu/packages/python-check.scm | 6 +-

gnu/packages/python-compression.scm | 2 +-

gnu/packages/python-web.scm | 19 +++--

gnu/packages/qt.scm | 15 ++--

gnu/packages/samba.scm | 4 +-

gnu/packages/sdl.scm | 6 +-

gnu/packages/search.scm | 4 +-

gnu/packages/serialization.scm | 4 +-

gnu/packages/shells.scm | 37 +++++++--

gnu/packages/sphinx.scm | 8 +-

gnu/packages/ssh.scm | 7 +-

gnu/packages/tex.scm | 8 +-

gnu/packages/textutils.scm | 12 ++-

gnu/packages/time.scm | 5 +-

gnu/packages/tls.scm | 9 ++-

gnu/packages/video.scm | 62 +++++++++------

gnu/packages/wget.scm | 2 +-

gnu/packages/xdisorg.scm | 5 +-

gnu/packages/xorg.scm | 19 +++--

50 files changed, 397 insertions(+), 314 deletions(-)