[PATCH 0/2] Split guix build into more steps for 32bit hosts.

  • Done
  • quality assurance status badge
Details
3 participants
  • Dr. Arne Babenhauserheide
  • Janneke Nieuwenhuizen
  • Ludovic Courtès
Owner
unassigned
Submitted by
Janneke Nieuwenhuizen
Severity
normal
J
J
Janneke Nieuwenhuizen wrote on 22 Aug 2023 19:17
(address . bug-guix@gnu.org)
cover.1692723764.git.janneke@gnu.org
Hi!

Initially writter for the Hurd (see hurd-team
This afternoon on IRC, Maxim confirmed my suspicion that this could be 32bit
issue, rather than a Hurd problem

I didn't submit these for the Hurd yet, as I didn't really feel comfortable
with the 26-way split. I tried a 5-way split and that still gave OOM errors
on the Hurd and I didn't feel like hand-coding a 10-way split, so yeah.

Also, the 5-way split in the Makefile.am will produce percentages over
100 which suggests a typo...but I couldn't spot it.

Lastly, these patches feel like a workaround for Guile / libgc memory
management? Otoh, being able to build Guix on 32bit hosts is kinda nice
too... So yeah.

Greetings,
Janneke

Janneke Nieuwenhuizen (2):
build: Build gnu/packages/*.go in five steps.
self: Build gnu/packages/*.go in 26 steps on 32bit.

Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++-------
guix/self.scm | 31 ++++++++++++++++++++++++--
2 files changed, 83 insertions(+), 10 deletions(-)


base-commit: c655231b72ac28b5a433069fcf86a835c9c83691
--
2.41.0
J
J
Janneke Nieuwenhuizen wrote on 22 Aug 2023 19:19
[PATCH 1/2] build: Build gnu/packages/*.go in five steps.
(address . 65456@debbugs.gnu.org)
606351ff2416c9086638a1c379b84ba811b384fa.1692723764.git.janneke@gnu.org
This breaks-up packages into five chunks of ~150,000 lines, allowing guix
build --target=i586-pc-gnu from an x86 host.

This is a followup to 1aa7ee52c6c520c2dbbdb06f1381466e9fd96294.

* Makefile.am (first_half): Rename to...
(first_quart): ...this, and also split into...
(second_quart): ...this.
(third_quart, MODULES_PACKAGES3, MODULES_PACKAGE4): New variables.
(make-packages3-go, make-packages4-go): New targets.
(make-packages-go): Add them.
---
Makefile.am | 62 ++++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 54 insertions(+), 8 deletions(-)

Toggle diff (88 lines)
diff --git a/Makefile.am b/Makefile.am
index 738532f839..7c8c215f86 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -776,19 +776,45 @@ endef
# in <https://issues.guix.gnu.org/48963>. Each 'eval' call below creates a
# 'make-*-go' phony target that builds the corresponding subset.
-first_half := \
- gnu/packages/a% gnu/packages/b% gnu/packages/c% gnu/packages/d% \
- gnu/packages/e% gnu/packages/f% gnu/packages/g% gnu/packages/h% \
- gnu/packages/i% gnu/packages/j% gnu/packages/k% gnu/packages/l%
+first_fifth := \
+ gnu/packages/a% gnu/packages/b% gnu/packages/c%
+
+second_fifth := \
+ gnu/packages/d% gnu/packages/e% gnu/packages/f%
+
+third_fifth := \
+ gnu/packages/g% gnu/packages/h% gnu/packages/i%
+
+fourth_fifth := \
+ gnu/packages/j% gnu/packages/k% gnu/packages/l% gnu/packages/m% \
+ gnu/packages/n% gnu/packages/o%
MODULES_CORE := guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULES)))
-MODULES_PACKAGES1 := $(filter $(first_half),$(MODULES))
-MODULES_PACKAGES2 := $(filter-out $(first_half),$(filter gnu/packages/%,$(MODULES)))
-MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)
+MODULES_PACKAGES1 := $(filter $(first_fifth),$(MODULES))
+MODULES_PACKAGES2 := $(filter $(second_fifth),$(MODULES))
+MODULES_PACKAGES3 := $(filter $(third_fifth),$(MODULES))
+MODULES_PACKAGES4 := $(filter $(fourth_fifth),$(MODULES))
+MODULES_PACKAGES5 := $(filter-out $(first_fifth) $(second_fifth) \
+ $(third_fifth) $(fourth_fifth), \
+ $(filter gnu/packages/%,$(MODULES)))
+MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \
+ $(MODULES_PACKAGES3) $(MODULES_PACKAGES4) \
+ $(MODULES_PACKAGES5)
MODULES_SYSTEM := gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES)))
MODULES_CLI := $(filter guix/scripts/%,$(MODULES))
MODULES_PO := guix/build/po.scm
+print-p1:
+ @echo $(MODULES_PACKAGES1)
+print-p2:
+ @echo $(MODULES_PACKAGES2)
+print-p3:
+ @echo $(MODULES_PACKAGES3)
+print-p4:
+ @echo $(MODULES_PACKAGES4)
+print-p5:
+ @echo $(MODULES_PACKAGES5)
+
$(eval $(call guile-compilation-rule,make-core-go, \
$(MODULES_CORE) guix/config.scm $(dist_noinst_DATA), \
0))
@@ -804,7 +830,27 @@ $(eval $(call guile-compilation-rule,make-packages2-go, \
$(words $(MODULES_CORE) $(MODULES_PACKAGES1))))
.PHONY: make-packages2-go
-make-packages-go: make-packages1-go make-packages2-go
+$(eval $(call guile-compilation-rule,make-packages3-go, \
+ $(MODULES_PACKAGES3) make-core-go make-packages1-go make-packages2-go, \
+ $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2))))
+.PHONY: make-packages3-go
+
+$(eval $(call guile-compilation-rule,make-packages4-go, \
+ $(MODULES_PACKAGES4) make-core-go make-packages1-go make-packages2-go \
+ make-packages3-go, \
+ $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \
+ $(MODULES_PACKAGES3))))
+.PHONY: make-packages4-go
+
+$(eval $(call guile-compilation-rule,make-packages5-go, \
+ $(MODULES_PACKAGES5) make-core-go make-packages1-go make-packages2-go \
+ make-packages3-go make-packages4-go, \
+ $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \
+ $(MODULES_PACKAGES3) $(MODULES_PACKAGES4))))
+.PHONY: make-packages5-go
+
+make-packages-go: make-packages1-go make-packages2-go \
+ make-packages3-go make-packages4-go make-packages5-go
.PHONY: make-packages-go
$(eval $(call guile-compilation-rule,make-system-go, \
--
2.41.0
J
J
Janneke Nieuwenhuizen wrote on 22 Aug 2023 19:19
[PATCH 2/2] self: Build gnu/packages/*.go in 26 steps on 32bit.
(address . 65456@debbugs.gnu.org)
887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org
Similar to the Makefile.am change, this breaks-up packages into 26 chunks
when building on 32bit. Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define.
Use it when building on a "i586" or "i686" cpu.
---
guix/self.scm | 31 +++++++++++++++++++++++++++++--
1 file changed, 29 insertions(+), 2 deletions(-)

Toggle diff (67 lines)
diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..474715fe7a 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files
'((guix build compile)
(guix build utils)))
#~(begin
- (use-modules (srfi srfi-26)
+ (use-modules (srfi srfi-1)
+ (srfi srfi-26)
+ (srfi srfi-71)
(ice-9 match)
(ice-9 format)
+ (ice-9 regex)
(ice-9 threads)
(guix build compile)
(guix build utils))
@@ -1251,6 +1255,26 @@ (define* (compiled-modules name module-tree module-files
#:report-load report-load
#:report-compilation report-compilation)))
+ (define (process-directory/32bit directory files output)
+ (let* ((chunks (map (compose
+ (cute partition <> files)
+ (lambda (regex)
+ (cute string-match regex <>))
+ (cute string-append "^gnu/packages/" <>)
+ (cute make-string 1 <>)
+ integer->char
+ (cute + (char->integer #\a) <>))
+ (iota 26)))
+ (chunks (filter pair? chunks)))
+ (for-each
+ (lambda (chunck)
+ (compile-files directory #$output chunck
+ #:workers (parallel-job-count)
+ #:report-load report-load
+ #:report-compilation report-compilation)
+ (gc))
+ chunks)))
+
(setvbuf (current-output-port) 'line)
(setvbuf (current-error-port) 'line)
@@ -1277,7 +1301,10 @@ (define* (compiled-modules name module-tree module-files
(mkdir #$output)
(chdir #+module-tree)
- (process-directory "." '#+module-files #$output)
+ (match (string-split %host-type #\-)
+ (((or "i586" "i686") rest ...)
+ (process-directory/32bit "." '#+module-files #$output))
+ (_ (process-directory "." '#+module-files #$output)))
(newline))))
(computed-file name build
--
2.41.0
J
J
Janneke Nieuwenhuizen wrote on 22 Aug 2023 19:33
Re: [PATCH 1/2] build: Build gnu/packages/*.go in five steps.
(address . 65456@debbugs.gnu.org)
87sf8b7zj1.fsf@gnu.org
Janneke Nieuwenhuizen writes:

Toggle quote (1 lines)
> * Makefile.am (first_half): Rename to...
[..]
Toggle quote (12 lines)
> +print-p1:
> + @echo $(MODULES_PACKAGES1)
> +print-p2:
> + @echo $(MODULES_PACKAGES2)
> +print-p3:
> + @echo $(MODULES_PACKAGES3)
> +print-p4:
> + @echo $(MODULES_PACKAGES4)
> +print-p5:
> + @echo $(MODULES_PACKAGES5)
> +

...I mean to remove this debugging code, but it might come in handy for
review.

--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
J
J
Janneke Nieuwenhuizen wrote on 22 Aug 2023 19:49
[PATCH v2 1/2] build: Build gnu/packages/*.go in five steps.
(address . 65456@debbugs.gnu.org)
99867a1bd9869a0fb0f3beacaaec9cec9090cb1e.1692726496.git.janneke@gnu.org
This breaks-up packages into five chunks of ~150,000 lines, allowing guix
build --target=i586-pc-gnu from an x86 host.

This is a followup to 1aa7ee52c6c520c2dbbdb06f1381466e9fd96294.

* Makefile.am (first_half): Rename to...
(first_quart): ...this, and also split into...
(second_quart): ...this.
(third_quart, MODULES_PACKAGES3, MODULES_PACKAGE4): New variables.
(make-packages3-go, make-packages4-go): New targets.
(make-packages-go): Add them.
---
Makefile.am | 51 +++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 43 insertions(+), 8 deletions(-)

Toggle diff (75 lines)
diff --git a/Makefile.am b/Makefile.am
index 738532f839..922913355c 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -776,15 +776,30 @@ endef
# in <https://issues.guix.gnu.org/48963>. Each 'eval' call below creates a
# 'make-*-go' phony target that builds the corresponding subset.
-first_half := \
- gnu/packages/a% gnu/packages/b% gnu/packages/c% gnu/packages/d% \
- gnu/packages/e% gnu/packages/f% gnu/packages/g% gnu/packages/h% \
- gnu/packages/i% gnu/packages/j% gnu/packages/k% gnu/packages/l%
+first_fifth := \
+ gnu/packages/a% gnu/packages/b% gnu/packages/c%
+
+second_fifth := \
+ gnu/packages/d% gnu/packages/e% gnu/packages/f%
+
+third_fifth := \
+ gnu/packages/g% gnu/packages/h% gnu/packages/i%
+
+fourth_fifth := \
+ gnu/packages/j% gnu/packages/k% gnu/packages/l% gnu/packages/m% \
+ gnu/packages/n% gnu/packages/o%
MODULES_CORE := guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULES)))
-MODULES_PACKAGES1 := $(filter $(first_half),$(MODULES))
-MODULES_PACKAGES2 := $(filter-out $(first_half),$(filter gnu/packages/%,$(MODULES)))
-MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2)
+MODULES_PACKAGES1 := $(filter $(first_fifth),$(MODULES))
+MODULES_PACKAGES2 := $(filter $(second_fifth),$(MODULES))
+MODULES_PACKAGES3 := $(filter $(third_fifth),$(MODULES))
+MODULES_PACKAGES4 := $(filter $(fourth_fifth),$(MODULES))
+MODULES_PACKAGES5 := $(filter-out $(first_fifth) $(second_fifth) \
+ $(third_fifth) $(fourth_fifth), \
+ $(filter gnu/packages/%,$(MODULES)))
+MODULES_PACKAGES := $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \
+ $(MODULES_PACKAGES3) $(MODULES_PACKAGES4) \
+ $(MODULES_PACKAGES5)
MODULES_SYSTEM := gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES)))
MODULES_CLI := $(filter guix/scripts/%,$(MODULES))
MODULES_PO := guix/build/po.scm
@@ -804,7 +819,27 @@ $(eval $(call guile-compilation-rule,make-packages2-go, \
$(words $(MODULES_CORE) $(MODULES_PACKAGES1))))
.PHONY: make-packages2-go
-make-packages-go: make-packages1-go make-packages2-go
+$(eval $(call guile-compilation-rule,make-packages3-go, \
+ $(MODULES_PACKAGES3) make-core-go make-packages1-go make-packages2-go, \
+ $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2))))
+.PHONY: make-packages3-go
+
+$(eval $(call guile-compilation-rule,make-packages4-go, \
+ $(MODULES_PACKAGES4) make-core-go make-packages1-go make-packages2-go \
+ make-packages3-go, \
+ $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \
+ $(MODULES_PACKAGES3))))
+.PHONY: make-packages4-go
+
+$(eval $(call guile-compilation-rule,make-packages5-go, \
+ $(MODULES_PACKAGES5) make-core-go make-packages1-go make-packages2-go \
+ make-packages3-go make-packages4-go, \
+ $(words $(MODULES_CORE) $(MODULES_PACKAGES1) $(MODULES_PACKAGES2) \
+ $(MODULES_PACKAGES3) $(MODULES_PACKAGES4))))
+.PHONY: make-packages5-go
+
+make-packages-go: make-packages1-go make-packages2-go \
+ make-packages3-go make-packages4-go make-packages5-go
.PHONY: make-packages-go
$(eval $(call guile-compilation-rule,make-system-go, \

base-commit: c655231b72ac28b5a433069fcf86a835c9c83691
--
2.41.0
J
J
Janneke Nieuwenhuizen wrote on 22 Aug 2023 19:49
[PATCH v2 2/2] self: Build gnu/packages/*.go in 26 steps.
(address . 65456@debbugs.gnu.org)
c886fd26f18bc0075bd46299d24c44742d241f35.1692726496.git.janneke@gnu.org
Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks
when building on 32bit. Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
"gnu/packages" into 26 chunks.
---
guix/self.scm | 33 ++++++++++++++++++++++++++-------
1 file changed, 26 insertions(+), 7 deletions(-)

Toggle diff (62 lines)
diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..151fa9f452 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files
'((guix build compile)
(guix build utils)))
#~(begin
- (use-modules (srfi srfi-26)
+ (use-modules (srfi srfi-1)
+ (srfi srfi-26)
+ (srfi srfi-71)
(ice-9 match)
(ice-9 format)
+ (ice-9 regex)
(ice-9 threads)
(guix build compile)
(guix build utils))
@@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files
(force-output))
(define (process-directory directory files output)
- ;; Hide compilation warnings.
- (parameterize ((current-warning-port (%make-void-port "w")))
- (compile-files directory #$output files
- #:workers (parallel-job-count)
- #:report-load report-load
- #:report-compilation report-compilation)))
+ ;; Split gnu/packages in 26 chunks to avoid OOM errors
+ (let* ((chunks (map (compose
+ (cute partition <> files)
+ (lambda (regex)
+ (cute string-match regex <>))
+ (cute string-append "^gnu/packages/" <>)
+ (cute make-string 1 <>)
+ integer->char
+ (cute + (char->integer #\a) <>))
+ (iota 26)))
+ (chunks (filter pair? chunks)))
+ (for-each
+ (lambda (chunck)
+ (parameterize ((current-warning-port (%make-void-port "w")))
+ ;; Hide compilation warnings.
+ (compile-files directory #$output chunck
+ #:workers (parallel-job-count)
+ #:report-load report-load
+ #:report-compilation report-compilation))
+ (gc))
+ chunks)))
(setvbuf (current-output-port) 'line)
(setvbuf (current-error-port) 'line)
--
2.41.0
J
J
Janneke Nieuwenhuizen wrote on 22 Aug 2023 19:55
(address . 65456@debbugs.gnu.org)
87o7iz7yip.fsf@gnu.org
Janneke Nieuwenhuizen writes:

Toggle quote (2 lines)
> diff --git a/guix/self.scm b/guix/self.scm
> index 81a36e007f..151fa9f452 100644
[..]

Toggle quote (2 lines)
> + (parameterize ((current-warning-port (%make-void-port "w")))

meant to move this comment

Toggle quote (2 lines)
> + ;; Hide compilation warnings.

above the parameterize; fixed locally


Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
J
J
Janneke Nieuwenhuizen wrote on 22 Aug 2023 21:28
(address . 65456-done@debbugs.gnu.org)
87jztm98qy.fsf@gnu.org
Janneke Nieuwenhuizen writes:

After a LGTM from Ludo and discussing with Maxim on IRC, pushed to
master as 5898b2e8a3dbf7797e83b39a2783c5b543015725

we could maybe use Guile in make to split building .go files in
Makefile.am algorithmically like we do in guix/self.scm.

Greetings,
Janneke

--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
Closed
L
L
Ludovic Courtès wrote on 22 Aug 2023 23:48
control message for bug #65456
(address . control@debbugs.gnu.org)
87ttsq3g0n.fsf@gnu.org
reopen 65456
tags 65456 - fixed patch
quit
L
L
Ludovic Courtès wrote on 22 Aug 2023 23:51
Re: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.
(name . Janneke Nieuwenhuizen)(address . janneke@gnu.org)
87pm3e3fuu.fsf_-_@gnu.org
Hi,

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (6 lines)
> Similar to the Makefile.am change, this breaks-up packages into 26 chunks
> when building on 32bit. Also force garbage collection.
>
> * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define.
> Use it when building on a "i586" or "i686" cpu.

.go files of a Guix built with this new (guix self) would not be found:

Toggle snippet (12 lines)
newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/share/guile/site/3.0/guix/ui.scm", {st_mode=S_IFREG|0444, st_size=95284, ...}, 0) = 0
newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/share/guile/site/3.0/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/ludo/.guix-home/profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/ludo/.guix-home/profile/lib/guile/3.0/site-ccache/guix/ui.go", 0x7ffc4bf07640, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/run/current-system/profile/lib/guile/3.0/site-ccache/guix/ui.go", {st_mode=S_IFREG|0444, st_size=530053, ...}, 0) = 0

In fact, guix/*.go is entirely missing, it seems:

$ --8<---------------cut here---------------start------------->8---
ls /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix
ls: cannot access '/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/guix': No such file or directory
$ ls /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/lib/guile/3.0/site-ccache/gnu
packages/
Toggle snippet (4 lines)
With the patch, without the patch:

$ find -L /gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union -name \*.go | wc -l
719
$ find -L /gnu/store/4dm864f7iif1vg9sz355gdqkgb7air98-guix-module-union -name \*.go | wc -l
1188
Toggle snippet (4 lines)
The problem would then manifest like this:

$ /gnu/store/57x6wrf78147lif1ss0ck2z3a0323b9l-guix-20230822.21/bin/guix build hello
error: license:arphic-1999: unbound variable
hint: Did you forget a `use-modules' form?

error: googletest: unbound variable
hint: Did you forget a `use-modules' form?

[…]

Throw to key `unbound-variable' with args `("resolve-interface" "no binding `~A' in module ~A" (shared-mime-info (gnu packages gnome)) #f)'.
Backtrace:
In guix/store.scm:
659:37 19 (thunk)
1298:8 18 (call-with-build-handler #<procedure 7f82c427ac60 at guix/ui.scm:1196:2 (continu…> …)
In guix/scripts/build.scm:
584:2 17 (_)
In srfi/srfi-1.scm:
673:15 16 (append-map _ _ . _)
586:17 15 (map1 ((argument . "hello") (build-mode . 0) (graft? . #t) (substitutes? . #t) # …))
In guix/scripts/build.scm:
604:31 14 (_ _)
In gnu/packages.scm:
485:2 13 (%find-package "hello" "hello" #f)
365:6 12 (find-best-packages-by-name _ _)
295:56 11 (_ "hello" _)
In unknown file:
10 (force #<promise #<procedure 7f82c6c5eee0 at gnu/packages.scm:285:18 ()>>)
In gnu/packages.scm:
242:33 9 (fold-packages #<procedure 7f82c21125f0 at gnu/packages.scm:286:35 (p r)> #<vlis…> …)
In guix/discovery.scm:
158:11 8 (all-modules _ #:warn _)
In srfi/srfi-1.scm:
460:18 7 (fold #<procedure 7f82c6cc7f00 at guix/discovery.scm:142:8 (spec result)> _ ((. #)))
In guix/discovery.scm:
148:19 6 (_ _ ())
115:5 5 (scheme-modules _ _ #:warn _)
In srfi/srfi-1.scm:
691:23 4 (filter-map #<procedure 7f82c6cc7de0 at guix/discovery.scm:115:16 (file)> _ . _)
In guix/discovery.scm:
123:24 3 (_ . _)
In guix/ui.scm:
343:2 2 (report-unbound-variable-error _ #:frame _)
In ice-9/boot-9.scm:
1685:16 1 (raise-exception _ #:continuable? _)
1685:16 0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Throw to key `match-error' with args `("match" "no matching pattern" (unbound-variable "resolve-interface" "no binding `~A' in module ~A" (shared-mime-info (gnu packages gnome)) #f))'.
Toggle snippet (11 lines)
This is because an old (guix licenses) module would be loaded.

I’m reverting for now.

Note that (guix self) is very sensitive, so we should test it
thoroughly. Here I used ‘make as-derivation’ and ran the resulting
‘guix’ command.

Ludo’.
J
J
Janneke Nieuwenhuizen wrote on 23 Aug 2023 08:16
Re: bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps.
(name . Ludovic Courtès)(address . ludo@gnu.org)
871qfu8erx.fsf_-_@gnu.org
Ludovic Courtès writes:

Hi!

Toggle quote (12 lines)
> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> Similar to the Makefile.am change, this breaks-up packages into 26 chunks
>> when building on 32bit. Also force garbage collection.
>>
>> * guix/self.scm (compiled-modules)[process-directory/32bit]: New inner define.
>> Use it when building on a "i586" or "i686" cpu.
>
> .go files of a Guix built with this new (guix self) would not be found:
>
> newfstatat(AT_FDCWD, "/gnu/store/pm43nabwng5rm8irmfhw2wk39hip8xr6-guix-module-union/share/guile/site/3.0/guix/ui.scm", {st_mode=S_IFREG|0444, st_size=95284, ...}, 0) = 0

Hmm, how did I miss this?

Toggle quote (2 lines)
> In fact, guix/*.go is entirely missing, it seems:

Right...that makes sense now that I look at the code again.

Wait...I probably even experienced this breakage after running `guix
pull' on the Hurd but failed to notice its cause, the missing guix/*.go,
and ascribed it to "something" being broken on Hurd. So the good news
is that we'll most probably have guix pull work on the Hurd after fixing
this!

Toggle snippet (6 lines)
(cute partition <> files)
(lambda (regex)
(cute string-match regex <>))
(cute string-append "^gnu/packages/" <>)

it still has the "gnu/packages" directory name hardcoded here. (I even
believe this was kind-of intentional, meaning not to split-up builds of
other directories...). The result is, of course, that partition always
produces an empty result for any '("guix/foo.scm") files!

It is fixed by doing

Toggle snippet (3 lines)
(cute string-append "^" directory "/" <>)

i.e., also splitting guix/ 26 ways.

Toggle quote (2 lines)
> I’m reverting for now.

Thank you for looking into and taking care of this mess I made, sorry!

Toggle quote (3 lines)
> Note that (guix self) is very sensitive, so we should test it
> thoroughly.

Using the attached patch, I ran

Toggle snippet (13 lines)
$ guix pull --branch=master --url=$PWD
[..]
08:11:02 janneke@drakenpad:~/src/guix/master
$ guix describe
Git checkout:
repository: /home/janneke/src/guix/master/
branch: master
commit: a639a64d590067d3fe928be3ecc9d2d4cc2e8df3
08:11:04 janneke@drakenpad:~/src/guix/master
$ guix build hello
/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1

Toggle quote (3 lines)
> Here I used ‘make as-derivation’ and ran the resulting
> ‘guix’ command.

Ah, nice that guix works too for me too now

Toggle snippet (8 lines)
$ make as-derivation
[..]
/gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06
08:13:15 janneke@drakenpad:~/src/guix/master [env]
$ /gnu/store/lqckq8f3c5s8jb5glwk8ancb781fc7km-guix-20230823.06/bin/guix build hello
/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1

Greetings,
Janneke
From a639a64d590067d3fe928be3ecc9d2d4cc2e8df3 Mon Sep 17 00:00:00 2001
Message-ID: <a639a64d590067d3fe928be3ecc9d2d4cc2e8df3.1692770000.git.janneke@gnu.org>
From: Janneke Nieuwenhuizen <janneke@gnu.org>
Date: Thu, 22 Jun 2023 08:30:25 +0200
Subject: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26
steps.

Similar to the Makefile.am change, this breaks-up gnu/packages into 26 chunks
when building on 32bit. Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
directories into 26 chunks.
---
guix/self.scm | 33 ++++++++++++++++++++++++++-------
1 file changed, 26 insertions(+), 7 deletions(-)

Toggle diff (64 lines)
diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..e5a85eaad6 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -1210,9 +1211,12 @@ (define* (compiled-modules name module-tree module-files
'((guix build compile)
(guix build utils)))
#~(begin
- (use-modules (srfi srfi-26)
+ (use-modules (srfi srfi-1)
+ (srfi srfi-26)
+ (srfi srfi-71)
(ice-9 match)
(ice-9 format)
+ (ice-9 regex)
(ice-9 threads)
(guix build compile)
(guix build utils))
@@ -1244,12 +1248,27 @@ (define* (compiled-modules name module-tree module-files
(force-output))
(define (process-directory directory files output)
- ;; Hide compilation warnings.
- (parameterize ((current-warning-port (%make-void-port "w")))
- (compile-files directory #$output files
- #:workers (parallel-job-count)
- #:report-load report-load
- #:report-compilation report-compilation)))
+ ;; Split gnu/packages in 26 chunks to avoid OOM errors
+ (let* ((chunks (map (compose
+ (cute partition <> files)
+ (lambda (regex)
+ (cute string-match regex <>))
+ (cute string-append "^" directory "/" <>)
+ (cute make-string 1 <>)
+ integer->char
+ (cute + (char->integer #\a) <>))
+ (iota 26)))
+ (chunks (filter pair? chunks)))
+ (for-each
+ (lambda (chunck)
+ ;; Hide compilation warnings.
+ (parameterize ((current-warning-port (%make-void-port "w")))
+ (compile-files directory #$output chunck
+ #:workers (parallel-job-count)
+ #:report-load report-load
+ #:report-compilation report-compilation))
+ (gc))
+ chunks)))
(setvbuf (current-output-port) 'line)
(setvbuf (current-error-port) 'line)

base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea
--
2.41.0
--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
J
J
Janneke Nieuwenhuizen wrote on 23 Aug 2023 11:41
Re: bug#65456: [PATCH v4] self: Build directories in chunks of max 25 files at a time.
(name . Ludovic Courtès)(address . ludo@gnu.org)
87h6oq6qpg.fsf_-_@gnu.org
Janneke Nieuwenhuizen writes:

Hi!

Toggle quote (3 lines)
> Ludovic Courtès writes:
>> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>>
[..]

Toggle quote (9 lines)
> it still has the "gnu/packages" directory name hardcoded here. (I even
> believe this was kind-of intentional, meaning not to split-up builds of
> other directories...). The result is, of course, that partition always
> produces an empty result for any '("guix/foo.scm") files!
>
> It is fixed by doing
>
> (cute string-append "^" directory "/" <>)

Well, almost...In my test script it worked, but directory is "." in the
actual self.scm.

So with v3 no .go files get built at all...and then a naive `guix build
hello' seems to work!; `guix shell' for example fails. Comparing the
list of files from `find /gnu/store/...-guix-20230823.08 -follow' is
also helpful.

I realised that there's no need to use the name-based splitting hack
that Makefile.am uses and changed the creation of chunks to simply use a
maximum chunk length of 25 files.

New version attached.

Greetings,
Janneke
From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org>
From: Janneke Nieuwenhuizen <janneke@gnu.org>
Date: Thu, 22 Jun 2023 08:30:25 +0200
Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
time.

Similar to split build of make-go in Makefile.am, this breaks-up building
directories into chunks of max 25 files. Also force garbage collection.

* guix/self.scm (compiled-modules)[process-directory]: Split building of
directories into chunks of max 25 files.
---
guix/self.scm | 27 ++++++++++++++++++++-------
1 file changed, 20 insertions(+), 7 deletions(-)

Toggle diff (56 lines)
diff --git a/guix/self.scm b/guix/self.scm
index 81a36e007f..fc2dfd8131 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017-2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>
+;;; Copyright © 2023 Janneke Nieuwenhuizen <janneke@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -1210,7 +1211,8 @@ (define* (compiled-modules name module-tree module-files
'((guix build compile)
(guix build utils)))
#~(begin
- (use-modules (srfi srfi-26)
+ (use-modules (srfi srfi-1)
+ (srfi srfi-26)
(ice-9 match)
(ice-9 format)
(ice-9 threads)
@@ -1244,12 +1246,23 @@ (define* (compiled-modules name module-tree module-files
(force-output))
(define (process-directory directory files output)
- ;; Hide compilation warnings.
- (parameterize ((current-warning-port (%make-void-port "w")))
- (compile-files directory #$output files
- #:workers (parallel-job-count)
- #:report-load report-load
- #:report-compilation report-compilation)))
+ (let* ((size 25) ;compile max 25 files a time
+ (chunks (unfold
+ (lambda (seed) (< (length seed) size)) ;p
+ (cute take <> size) ;f
+ (cute drop <> size) ;g
+ files ;seed
+ list))) ;tail
+ (for-each
+ (lambda (chunck)
+ ;; Hide compilation warnings.
+ (parameterize ((current-warning-port (%make-void-port "w")))
+ (compile-files directory output chunck
+ #:workers (parallel-job-count)
+ #:report-load report-load
+ #:report-compilation report-compilation))
+ (gc))
+ chunks)))
(setvbuf (current-output-port) 'line)
(setvbuf (current-error-port) 'line)

base-commit: f4d0d0bd5e7d0e67281d84d81068f7fd5eb480ea
--
2.41.0
--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
J
J
Janneke Nieuwenhuizen wrote on 24 Aug 2023 07:33
Re: bug#65456: [PATCH v3] self: Build guix/ and gnu/packages/ directories in 26 steps.
(name . Ludovic Courtès)(address . ludo@gnu.org)
87zg2h57il.fsf@gnu.org
Janneke Nieuwenhuizen writes:

Hi!

Toggle quote (6 lines)
> Ludovic Courtès writes:
>
> Hi!
>
>> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>>
[..]
Toggle quote (10 lines)
>> In fact, guix/*.go is entirely missing, it seems:
>
> Right...that makes sense now that I look at the code again.
>
> Wait...I probably even experienced this breakage after running `guix
> pull' on the Hurd but failed to notice its cause, the missing guix/*.go,
> and ascribed it to "something" being broken on Hurd. So the good news
> is that we'll most probably have guix pull work on the Hurd after fixing
> this!

So, some good news at last; I can confirm that using v4 of this patch we
now have "guix pull", pulling from a local git directory, now fully
working on the Hurd! (When pulling from a git url, I get "Illegal
instruction" while receiving objects.)

See log below.

Greetings,
Janneke

/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=https://gitlab.com/janneke/guix--branch=wip-hurd
Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'...
guix pull: error: Git error: the SSL certificate is invalid
/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=http://gitlab.com/janneke/guix--branch=wip-hurd
Updating channel 'guix' from Git repository at 'http://gitlab.com/janneke/guix'...
guix pull: error: Git error: the SSL certificate is invalid
/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix install nss-certs
guix install: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.

The following package will be installed:
nss-certs 3.88.1

substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/z1jyx8lyhgr1gykiky5wjh5mncwvp6ls-profile.drv
/gnu/store/p3kkmm8ap3z68irnzl6dsq9clkzrg9cj-nss-certs-3.88.1.drv
/gnu/store/cjkisarl0gcwrvgcc3hklz63x7dz6ji2-nss-3.88.1.tar.xz.drv
/gnu/store/iwsyc1bvgn9x7qg0x3an5g2q23a6l2xk-certdata2pem-0.0.0.drv

71.6 MB will be downloaded
certdata2pem.c 2KiB 23KiB/s 00:00 [##################] 100.0%
nss-3.88.1.tar.gz 68.3MiB 2.1MiB/s 00:32 [##################] 100.0%
building /gnu/store/iwsyc1bvgn9x7qg0x3an5g2q23a6l2xk-certdata2pem-0.0.0.drv...
building /gnu/store/cjkisarl0gcwrvgcc3hklz63x7dz6ji2-nss-3.88.1.tar.xz.drv...
building /gnu/store/p3kkmm8ap3z68irnzl6dsq9clkzrg9cj-nss-certs-3.88.1.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
killing process 174: Invalid argument
killing process 175: Invalid argument
/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ guix pull --url=https://gitlab.com/janneke/guix--branch=wip-hurd
Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'...
guix pull: error: Git error: the SSL certificate is invalid
/ssh:childhurd1:/root/src/guix/wip-hurd/ #$ bash -login
root@guixydevel ~/src/guix/wip-hurd#
root@guixydevel ~/src/guix/wip-hurd# guix pull --url=https://gitlab.com/janneke/guix--branch=wip-hurd
Updating channel 'guix' from Git repository at 'https://gitlab.com/janneke/guix'...
receiving objects 49% [################################## ]Illegal instruction (core dumped)
root@guixydevel ~/src/guix/wip-hurd# guix pull --url=$PWD --branch=wip-hurd
Updating channel 'guix' from Git repository at '/root/src/guix/wip-hurd'...
guix pull: error: Git error: cannot locate remote-tracking branch 'origin/keyring'
root@guixydevel ~/src/guix/wip-hurd# git branch keyring origin/keyring
branch 'keyring' set up to track 'origin/keyring'.
root@guixydevel ~/src/guix/wip-hurd# guix pull --url=$PWD --branch=wip-hurd
Updating channel 'guix' from Git repository at '/root/src/guix/wip-hurd'...
Authenticating channel 'guix', commits 9edb3f6 to beb2704 (26 new commits)...
guix pull: warning: pulled channel 'guix' from a mirror of https://git.savannah.gnu.org/git/guix.git,which might be stale
Building from this channel:
guix /root/src/guix/wip-hurd beb2704
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
building /gnu/store/67p25fiinjvm1h0ga8bays6d0kyy0dp4-config.scm.drv...
building /gnu/store/iy189p9rd62ymac23l83h6vz86sx775y-git.scm.drv...
building /gnu/store/w5ld3vgpsd7im25zqjrk7rxpw3j6km3j-hash.scm.drv...
building /gnu/store/151iwm03mirbpbgsrrmxnpr97bj5w1ps-module-import.drv...
building /gnu/store/sdnlhfdqrg0kkh33adg4dcb15qnx4205-module-import.drv...
building /gnu/store/s835yp2frsy9mf4ha7q0b3mrg4mc78sc-module-import-compiled.drv...
building /gnu/store/sc3ik9br6k1fzkczg51qr54d9ilkl268-module-import-compiled.drv...
building /gnu/store/yn9hi0kmyygk29s79fa44nqh0wqip82n-compute-guix-derivation.drv...
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
bash 389KiB/s 00:02 | 907KiB transferred
dbus-1.14.0 1.2MiB/s 00:00 | 355KiB transferred
guile-json-3.5.0 5.9MiB/s 00:00 | 61KiB transferre|
guix-1.4.0-12.329069a-checkout 2.2MiB/s 00:09 | 20.3MiB transferred
libdaemon-0.14 +inf.0TiB/s 00:00 | 39KiB transferred
libevent-2.1.12 167KiB/s 00:03 | 463KiB transferred
mkdir 155KiB/s 00:03 | 432KiB transferred
avahi-0.8 229KiB/s 00:02 | 466KiB transferred
tar 3.2MiB/s 00:00 | 840KiB transferred
guile-avahi-0.4.1 26KiB/s 00:03 | 73KiB transferred
xz 670KiB/s 00:01 | 529KiB transferred
building /gnu/store/nr3r518p14rj0672q37c320fkd00r01y-guix-daemon-1.4.0-12.329069a.drv...
\ 'build' phaseild-log 286 52
CXX nix/nix-daemon/guix_daemon-guix-daemon.o
| 'build' phase47
CXX nix/libstore/libstore_a-pathlocks.o
/ 'make-dynamic-linker-cache' phas/
killing process 283: Invalid argument
killing process 284: Invalid argument
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/k6z2f845lqzkz2h4q60w089rzl7p7ay7-profile.drv
/gnu/store/k3rsfdspx1bqicpsv6vn6crw9lxqpwba-guix-beb270468.drv
/gnu/store/8wyidapyb2qibc1qsizm6r64fz2xjcw9-guix-manual.drv
/gnu/store/q2a7nzpg6i5qrh7w1mzzbp12lgwd0f75-guix-translated-texinfo.drv
/gnu/store/9fcfgcbswidhr8snjf7zqfyn4znahnmf-guix-beb270468-modules.drv
/gnu/store/2kdrxg03299x7n0gb3zrlwvrdgna3gi7-guix-packages-base-modules.drv
/gnu/store/vz2mic633i2biqn8v4b4znkl8mxdyasy-guix-packages-base.drv
/gnu/store/0hxl8ck28afjvx5q6ga0sgpfywbchizw-guix-core.drv
/gnu/store/n8z3r1vp95lrzbbq17h04j087cm6hmv2-guix-core-source.drv
/gnu/store/va8ncxqnbmljnxf5vn0rv62mz6lkyk0m-config.scm.drv
/gnu/store/v7mn3gb4jbh0qq5yy161rckw4vww6gbv-guix-extra.drv
/gnu/store/5qhkmkxwap8fykf5y1fwxxq9kz2yw2x2-guix-core-modules.drv
/gnu/store/fwjkn1cf3yjxg8znddbri1f560g60rb2-guix-cli-core-modules.drv
/gnu/store/x2h3sb1vpf2nf58byyxfgl79b4pyckn9-guix-cli-core.drv
/gnu/store/l07fiikl45adg3iv6ff4p0f3qbf6splw-guix-system-modules.drv
/gnu/store/gzgizh4wmbzdhdc9gmx9q9dz2jkxrdcd-guix-system.drv
/gnu/store/i8qmsx9myyykv8s2n6f3mrg20nclb43k-guix-packages.drv
/gnu/store/ljphi1irng8cc3vnqpfcfxy0rq1i4nv8-guix-config-modules.drv
/gnu/store/f4n6bxkzb1r06j1r600ivdll3cbhcvxv-guix-config.drv
/gnu/store/hq8s053m2bi184xr5wnymwi56hx2820k-guix-config-source.drv
/gnu/store/k6mz0x5yv13dsmhg0247zj19yb5ckjp0-config.scm.drv
/gnu/store/lqnmqbi39g7rxmfzs31ricahanaqmbbl-guix-packages-modules.drv
/gnu/store/rcfi26j7jyfnmc9cwp2cznacmm40viii-guix-extra-modules.drv
/gnu/store/sx0lrfjii302l72y2rzry1g1pn9x7awh-guix-system-tests-modules.drv
/gnu/store/43qri996hp9lmjk3dbqp19w2j7s0ixf2-guix-system-tests.drv
/gnu/store/z2wrv851s184asfp90zccmbzi37xbfmz-guix-cli.drv
/gnu/store/650wrs5h0css6baqq2zyj4cyn46dlcyx-guix-home.drv
/gnu/store/vvby7fdf2500fy7v6vl7jp6xcvd0psik-guix-cli-modules.drv
/gnu/store/xj31wyyq7i234bgss9dbagcci8siny5s-guix-home-modules.drv
/gnu/store/bjb1s22wxm427aig9kwygc7298nkzfjj-guix-daemon.drv
/gnu/store/kfnmjag44n6d6giwkmlckf5gqnxx2wid-guix-command.drv
/gnu/store/abh52n6ddp3i9w2hj75vqlgx44ynfwv5-guix-locale-guix-packages.drv
/gnu/store/p9kdrf7vxihq5gl85wmlpqrz3pd16naq-guix-locale-guix.drv
/gnu/store/shgbmdpa189mk7kza5fi6x1dmh0d9d0z-guix-module-union.drv
/gnu/store/jl04xar2gc4c0aybslgcnzg2b8ir551p-guix-misc.drv
/gnu/store/66fa0pvf2c24kl3dvhh4n4f8j4s0kzhi-guix-daemon.cil.drv
/gnu/store/50fk0z57vprxh13pp000alz6200lm9r6-profile.drv
/gnu/store/ij3cnmqmcfg8jd81j1k87xski8cifp2h-inferior-script.scm.drv

5 items will be downloaded
module-import-compiled 800KiB/s 00:00 | 64KiB transferred
guile-wrapper 160KiB/s 00:00 | 3KiB transferred
module-import-compiled 541KiB/s 00:00 | 81KiB transferred
module-import-compiled 860KiB/s 00:00 | 138KiB transferred
module-import-compiled 328KiB/s 00:00 | 10KiB transferred
building /gnu/store/k6mz0x5yv13dsmhg0247zj19yb5ckjp0-config.scm.drv...
building /gnu/store/va8ncxqnbmljnxf5vn0rv62mz6lkyk0m-config.scm.drv...
building /gnu/store/hq8s053m2bi184xr5wnymwi56hx2820k-guix-config-source.drv...
building /gnu/store/n8z3r1vp95lrzbbq17h04j087cm6hmv2-guix-core-source.drv...
building /gnu/store/f4n6bxkzb1r06j1r600ivdll3cbhcvxv-guix-config.drv...
building /gnu/store/0hxl8ck28afjvx5q6ga0sgpfywbchizw-guix-core.drv...
building /gnu/store/ljphi1irng8cc3vnqpfcfxy0rq1i4nv8-guix-config-modules.drv...
building /gnu/store/5qhkmkxwap8fykf5y1fwxxq9kz2yw2x2-guix-core-modules.drv...
building /gnu/store/66fa0pvf2c24kl3dvhh4n4f8j4s0kzhi-guix-daemon.cil.drv...
building /gnu/store/v7mn3gb4jbh0qq5yy161rckw4vww6gbv-guix-extra.drv...
building /gnu/store/abh52n6ddp3i9w2hj75vqlgx44ynfwv5-guix-locale-guix-packages.drv...
building /gnu/store/rcfi26j7jyfnmc9cwp2cznacmm40viii-guix-extra-modules.drv...
building /gnu/store/p9kdrf7vxihq5gl85wmlpqrz3pd16naq-guix-locale-guix.drv...
building /gnu/store/jl04xar2gc4c0aybslgcnzg2b8ir551p-guix-misc.drv...
building /gnu/store/vz2mic633i2biqn8v4b4znkl8mxdyasy-guix-packages-base.drv...
building /gnu/store/q2a7nzpg6i5qrh7w1mzzbp12lgwd0f75-guix-translated-texinfo.drv...
building /gnu/store/x2h3sb1vpf2nf58byyxfgl79b4pyckn9-guix-cli-core.drv...
building /gnu/store/8wyidapyb2qibc1qsizm6r64fz2xjcw9-guix-manual.drv...
building /gnu/store/fwjkn1cf3yjxg8znddbri1f560g60rb2-guix-cli-core-modules.drv...
building /gnu/store/2kdrxg03299x7n0gb3zrlwvrdgna3gi7-guix-packages-base-modules.drv...
building /gnu/store/i8qmsx9myyykv8s2n6f3mrg20nclb43k-guix-packages.drv...
building /gnu/store/lqnmqbi39g7rxmfzs31ricahanaqmbbl-guix-packages-modules.drv...
building /gnu/store/gzgizh4wmbzdhdc9gmx9q9dz2jkxrdcd-guix-system.drv...
building /gnu/store/650wrs5h0css6baqq2zyj4cyn46dlcyx-guix-home.drv...
building /gnu/store/l07fiikl45adg3iv6ff4p0f3qbf6splw-guix-system-modules.drv...
building /gnu/store/z2wrv851s184asfp90zccmbzi37xbfmz-guix-cli.drv...
building /gnu/store/xj31wyyq7i234bgss9dbagcci8siny5s-guix-home-modules.drv...
building /gnu/store/vvby7fdf2500fy7v6vl7jp6xcvd0psik-guix-cli-modules.drv...
building /gnu/store/43qri996hp9lmjk3dbqp19w2j7s0ixf2-guix-system-tests.drv...
building /gnu/store/sx0lrfjii302l72y2rzry1g1pn9x7awh-guix-system-tests-modules.drv...
building /gnu/store/9fcfgcbswidhr8snjf7zqfyn4znahnmf-guix-beb270468-modules.drv...
building /gnu/store/shgbmdpa189mk7kza5fi6x1dmh0d9d0z-guix-module-union.drv...
building /gnu/store/kfnmjag44n6d6giwkmlckf5gqnxx2wid-guix-command.drv...
building /gnu/store/bjb1s22wxm427aig9kwygc7298nkzfjj-guix-daemon.drv...
building /gnu/store/k3rsfdspx1bqicpsv6vn6crw9lxqpwba-guix-beb270468.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
building /gnu/store/ij3cnmqmcfg8jd81j1k87xski8cifp2h-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
hint: Consider setting the necessary environment variables by running:

GUIX_PROFILE="/root/.config/guix/current"
. "$GUIX_PROFILE/etc/profile"

Alternately, see `guix package --search-paths -p "/root/.config/guix/current"'.


hint: After setting `PATH', run `hash guix' to make sure your shell refers to
`/root/.config/guix/current/bin/guix'.

root@guixydevel ~/src/guix/wip-hurd# GUIX_PROFILE="/root/.config/guix/current"
. "$GUIX_PROFILE/etc/profile"
~/src/guix/wip-hurd# type -p guix
/root/.config/guix/current/bin/guix
root@guixydevel ~/src/guix/wip-hurd# guix --version
guix (GNU Guix) beb27046836d60974390f6a940167e8b5def1066
Copyright (C) 2023 the Guix authors
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
root@guixydevel ~/src/guix/wip-hurd# guix describe
Generation 1 Aug 24 2023 01:34:21 (current)
guix beb2704
repository URL: /root/src/guix/wip-hurd
branch: wip-hurd
commit: beb27046836d60974390f6a940167e8b5def1066
root@guixydevel ~/src/guix/wip-hurd# guix shell -D guix
hint: Consider passing the `--check' option once to make sure your shell does not clobber
environment variables.

root@guixydevel ~/src/guix/wip-hurd [env]# guix build hello
substitute: updating substitutes from 'http://dezyne.org:8181'... 100.0%
The following file will be downloaded:
/gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1
substituting /gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1...
hello-2.12.1 297KiB/s 00:00 | 74KiB transferred

/gnu/store/cqnh7hih2n42mfx3x9yy11i3jzyn6gqj-hello-2.12.1
root@guixydevel ~/src/guix/wip-hurd [env]#

--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
D
D
Dr. Arne Babenhauserheide wrote on 24 Aug 2023 08:44
(name . Janneke Nieuwenhuizen)(address . janneke@gnu.org)
87zg2hvt05.fsf@web.de
Janneke Nieuwenhuizen <janneke@gnu.org> writes:

Toggle quote (4 lines)
> So, some good news at last; I can confirm that using v4 of this patch we
> now have "guix pull", pulling from a local git directory, now fully
> working on the Hurd!

That’s awesome! Thank you for your work!

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de
-----BEGIN PGP SIGNATURE-----

iQJEBAEBCAAuFiEE801qEjXQSQPNItXAE++NRSQDw+sFAmTm/GoQHGFybmVfYmFi
QHdlYi5kZQAKCRAT741FJAPD6ziZEACyFGL3aNHfDk39Q5gClsX0Obv0uhQqxELa
SLy1+cSYOp9edrK+n3KNFLO4FGhK7cvvv1AVV+ExefCW080zRL5RskoOK34nAbRb
n8DHj9htOmVqZylSFZXkBqUBNlxMBSnbMFIcx8HnAiqI7Bju7Bu9cKZP8rXmlNQ+
XryzaueTaJIppsGd6lYHBZUrQsP6JgcofnSa/1qpSMNDAX5WUxVbZV1Z73VZWvZJ
eJcYiI76JVf9wOLuHq7gZrvbehp/NKmqm66qMyLxZ8yRUehwwdnItShyKfKgYi4q
ssQv1MWYlm4Q5kv+sTY1EyHdhw703D4m518KIcBhG9zAefz8v62oaI0qC2Oa7mIs
Wm80Gsrc4OMkCGW1vHFlZv1hKqgYk1LfJ7mEJvV3nRgyRxQCLeguvfNT+J1EoLg6
S+7Mq3GkAvnSUs3MsfJwmyXD/IjC2WsNDmgppA84xZ2F+g3rGJzHZ8U2zJfMfeAK
XLy5TRm5A9FMHn+Dc379cfD/9ikdi0y1FkLcfCOiJ5weQsxCMcmYCOal9RkJm/+Q
9V2+w1KjOPANpwGIhenWzV0zxNhDBHlj96Lu5xEZJH3weJXM4qMg5GdOep7czMlq
U+FNrQJnbMFML7foA9LJFITWC23L2hYacOP4X7JUOSqCH4G4JXyURd2Ef3XIO5Yd
/ZLXcZ5tlIjEBAEBCAAuFiEE3Si95tmHXKvOSosd3M8NswvBBUgFAmTm/GoQHGFy
bmVfYmFiQHdlYi5kZQAKCRDczw2zC8EFSGFrA/9sN2IEACGKnYVJTn6YsCYyI5XQ
gOrB+f+Y6mUWJMiyGxbftAgXMyYunuchN5Y7JOz/C9Qq9zUn+gW7tA3Fc3c2DsfS
/EI19S0yfP59L633+7FX5WxR6wrBK6DFNvn/55YMhHzqGbwIkiBUc/4VTbp+iFO/
7PQMJ0bCdbnCZhBeAw==
=idoo
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 24 Aug 2023 16:42
Re: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.
(name . Janneke Nieuwenhuizen)(address . janneke@gnu.org)
87zg2gy013.fsf_-_@gnu.org
Hi!

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (10 lines)
>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
> Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org>
> From: Janneke Nieuwenhuizen <janneke@gnu.org>
> Date: Thu, 22 Jun 2023 08:30:25 +0200
> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
> time.
>
> Similar to split build of make-go in Makefile.am, this breaks-up building
> directories into chunks of max 25 files. Also force garbage collection.

The big difference with ‘make-go’ is that ‘make-go’ spawns a new process
for each chunk of files: each process starts with an empty heap, which
is not the case here as we reuse the same process.

However, (guix self) is already splitting gnu/packages/*.scm in two
pieces: ‘guix-packages-base’ and ‘guix-packages’. The former is the
closure of (gnu packages base), and the latter contains the remaining
files. Unfortunately this is uneven:

Toggle snippet (20 lines)
$ readlink -f $(type -P guix)
/gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command
$ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages-base
/gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base
/gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source
/gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules
$ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base -type f |wc -l
361
$ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages$
/gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages
ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages | wc -l
64
$ guix describe
Generation 271 Aug 20 2023 23:48:59 (current)
guix a0f5885
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05

Maxime Devos looked into this a while back:


Toggle quote (3 lines)
> * guix/self.scm (compiled-modules)[process-directory]: Split building of
> directories into chunks of max 25 files.

[...]

Toggle quote (17 lines)
> (define (process-directory directory files output)
> - ;; Hide compilation warnings.
> - (parameterize ((current-warning-port (%make-void-port "w")))
> - (compile-files directory #$output files
> - #:workers (parallel-job-count)
> - #:report-load report-load
> - #:report-compilation report-compilation)))
> + (let* ((size 25) ;compile max 25 files a time
> + (chunks (unfold
> + (lambda (seed) (< (length seed) size)) ;p
> + (cute take <> size) ;f
> + (cute drop <> size) ;g
> + files ;seed
> + list))) ;tail
> + (for-each
> + (lambda (chunck)

s/chunck/chunk/

Can you confirm that this reduces memory usage observably? One way to
check that would be to print (gc-stats) from ‘process-directory’, with
and without the change. Could you give it a try?

Intuitively, I don’t see why it would eat less memory; maybe peak memory
usage is lower because we do less at once?

Also, I think we should remove the explicit (gc) call: it should not be
necessary, and if we depend on that, something’s wrong.

Anyhow, thanks for tackling this issue!

Ludo’.
J
J
Janneke Nieuwenhuizen wrote on 1 Sep 2023 14:48
(name . Ludovic Courtès)(address . ludo@gnu.org)
87h6oeys7t.fsf@verum.com
Ludovic Courtès writes:

Hello!

Toggle quote (16 lines)
> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>
>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
>> Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org>
>> From: Janneke Nieuwenhuizen <janneke@gnu.org>
>> Date: Thu, 22 Jun 2023 08:30:25 +0200
>> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
>> time.
>>
>> Similar to split build of make-go in Makefile.am, this breaks-up building
>> directories into chunks of max 25 files. Also force garbage collection.
>
> The big difference with ‘make-go’ is that ‘make-go’ spawns a new process
> for each chunk of files: each process starts with an empty heap, which
> is not the case here as we reuse the same process.

Right.

Toggle quote (5 lines)
> However, (guix self) is already splitting gnu/packages/*.scm in two
> pieces: ‘guix-packages-base’ and ‘guix-packages’. The former is the
> closure of (gnu packages base), and the latter contains the remaining
> files. Unfortunately this is uneven:

Okay...

Toggle quote (23 lines)
> $ readlink -f $(type -P guix)
> /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command
> $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages-base
> /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base
> /gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source
> /gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules
> $ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base -type f |wc -l
> 361
> $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages$
> /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages
> ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages | wc -l
> 64
> $ guix describe
> Generation 271 Aug 20 2023 23:48:59 (current)
> guix a0f5885
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05
>
> Maxime Devos looked into this a while back:
>
> https://issues.guix.gnu.org/54539

Oh my....

Toggle quote (7 lines)
>> * guix/self.scm (compiled-modules)[process-directory]: Split building of
>> directories into chunks of max 25 files.
>> + (for-each
>> + (lambda (chunck)
>
> s/chunck/chunk/

Oops, fixed.

Toggle quote (4 lines)
> Can you confirm that this reduces memory usage observably? One way to
> check that would be to print (gc-stats) from ‘process-directory’, with
> and without the change. Could you give it a try?

What a good and seemingly simple question. After a week of
instrumentation and testing, my answer can only be: I tried, and maybe.
(see below).

Toggle quote (3 lines)
> Intuitively, I don’t see why it would eat less memory; maybe peak memory
> usage is lower because we do less at once?

Okay...

Toggle quote (5 lines)
> Also, I think we should remove the explicit (gc) call: it should not be
> necessary, and if we depend on that, something’s wrong.

> Anyhow, thanks for tackling this issue!

Hehe. You've probably seen Josselin's recent GraphML backend effort
that might really help to address this? I'm afraid this patch can maybe
only postpone what really needs to be done...

There is gc-stats output from a successful `guix pull' or `make
as-derivation' on Guix/Hurd, that I can show you, and I've tried more
than 20 times; it always fails (OOM, hang, spontaneous reset, ...).

Below is a typical output of gc-stats on the Hurd for building self.scm,
when heap-size peaks (using the the max 25 files patch):

Toggle snippet (9 lines)
((gc-time-taken . 1530)
(heap-size . 2,625,474,560)
(heap-free-size . 1127989248)
(heap-total-allocated . 1337029496)
(heap-allocated-since-gc . 28728)
(protected-objects . 28)
(gc-times . 324))

notice that it's *much* bigger (more than twice) than my findings on
linux-64 below. I have no idea why this is of what it might mean...

So I turned to Guix GNU/Linux to get some gc-stat measurements. What
you see below is the maximum head-size at any point (I also have
heap-total-allocated but I think that's irrelevant? and initially didn't
use a script that measured the time).

Toggle snippet (32 lines)
* guix/self.scm: Vanilla, not chunked; print gc-stats.
((gc-time-taken . 27319485051)
(heap-size . 1,360,330,752)
(heap-free-size . 285,696,000)
(heap-total-allocated . 74,067,590,944)
(heap-allocated-since-gc . 186,250,144)
(protected-objects . 28)
(gc-times . 464))
real 24m36.643s

* guix/self.scm: Split building of directories into 26 chunks; print gc-stats.
(heap-size . 1,131,298,816)

* guix/self.scm: Split building of directories into 26 chunks; no gc; print gc-stats.
(heap-size . 1,121,116,160)

* guix/self.scm: Chunks of 25 files; run gc; print gc-stats.
(heap-size . 1,066,725,376)

* guix/self.scm: Chunks of 50 files; no gc; print gc-stats.
(heap-size . 1,299,230,720)
real 26m40.708s

* guix/self.scm: Chunks of 25 files; no gc; print gc-stats.
(heap-size . 1,024,045,056) ; 1st run
real 28m4.451s

* guix/self.scm: Chunks of 10 files; no gc; print gc-stats.
(heap-size . 1,077,895,168)
real 30m14.049s

...strangely enough, if we assume that these statistics translate to the
Hurd, using chunks of max 25 files seems to be a sort of sweet spot?
25% less peak memory (~300MB), "only" 12% (3"45') slower... though not
great for GNU/Linux users...

I have produced a handful of successful `guix pull's (from a local
checked-out worktree) using the 26-way split and chunks of max-25 files
patches, but sadly also many more attempts failed. Initially, when
creating this patch series, I was convinced this fixed building on the
Hurd, but I'm much less enthusiastic now.

So I still have a slight preference for using the latest max-25-files
patch, but I'm sorry to say that I cannot back it up with tangible data.
All in all a rather discouraging week with much effort spent for little
gain. Hopefully Josselin can do some of his magic here :)

Greetings,
Janneke

--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
J
J
Janneke Nieuwenhuizen wrote on 16 Sep 2023 17:16
(name . Ludovic Courtès)(address . ludo@gnu.org)
87led6ywqb.fsf@gnu.org
Janneke Nieuwenhuizen writes:

Hi!

Toggle quote (7 lines)
> Ludovic Courtès writes:
>
> Hello!
>
>> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
[..]
Toggle quote (77 lines)
>> Anyhow, thanks for tackling this issue!
>
> Hehe. You've probably seen Josselin's recent GraphML backend effort
> that might really help to address this? I'm afraid this patch can maybe
> only postpone what really needs to be done...
>
> There is gc-stats output from a successful `guix pull' or `make
> as-derivation' on Guix/Hurd, that I can show you, and I've tried more
> than 20 times; it always fails (OOM, hang, spontaneous reset, ...).
>
> Below is a typical output of gc-stats on the Hurd for building self.scm,
> when heap-size peaks (using the the max 25 files patch):
>
> ((gc-time-taken . 1530)
> (heap-size . 2,625,474,560)
> (heap-free-size . 1127989248)
> (heap-total-allocated . 1337029496)
> (heap-allocated-since-gc . 28728)
> (protected-objects . 28)
> (gc-times . 324))
>
>
> notice that it's *much* bigger (more than twice) than my findings on
> linux-64 below. I have no idea why this is of what it might mean...
>
> So I turned to Guix GNU/Linux to get some gc-stat measurements. What
> you see below is the maximum head-size at any point (I also have
> heap-total-allocated but I think that's irrelevant? and initially didn't
> use a script that measured the time).
>
> * guix/self.scm: Vanilla, not chunked; print gc-stats.
> ((gc-time-taken . 27319485051)
> (heap-size . 1,360,330,752)
> (heap-free-size . 285,696,000)
> (heap-total-allocated . 74,067,590,944)
> (heap-allocated-since-gc . 186,250,144)
> (protected-objects . 28)
> (gc-times . 464))
> real 24m36.643s
>
> * guix/self.scm: Split building of directories into 26 chunks; print gc-stats.
> (heap-size . 1,131,298,816)
>
> * guix/self.scm: Split building of directories into 26 chunks; no gc; print gc-stats.
> (heap-size . 1,121,116,160)
>
> * guix/self.scm: Chunks of 25 files; run gc; print gc-stats.
> (heap-size . 1,066,725,376)
>
> * guix/self.scm: Chunks of 50 files; no gc; print gc-stats.
> (heap-size . 1,299,230,720)
> real 26m40.708s
>
> * guix/self.scm: Chunks of 25 files; no gc; print gc-stats.
> (heap-size . 1,024,045,056) ; 1st run
> real 28m4.451s
>
> * guix/self.scm: Chunks of 10 files; no gc; print gc-stats.
> (heap-size . 1,077,895,168)
> real 30m14.049s
>
> ...strangely enough, if we assume that these statistics translate to the
> Hurd, using chunks of max 25 files seems to be a sort of sweet spot?
> 25% less peak memory (~300MB), "only" 12% (3"45') slower... though not
> great for GNU/Linux users...
>
> I have produced a handful of successful `guix pull's (from a local
> checked-out worktree) using the 26-way split and chunks of max-25 files
> patches, but sadly also many more attempts failed. Initially, when
> creating this patch series, I was convinced this fixed building on the
> Hurd, but I'm much less enthusiastic now.
>
> So I still have a slight preference for using the latest max-25-files
> patch, but I'm sorry to say that I cannot back it up with tangible data.
> All in all a rather discouraging week with much effort spent for little
> gain. Hopefully Josselin can do some of his magic here :)

Anyway, I've finally

Toggle snippet (6 lines)
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b2cc649999ed97b62517dcfb7cf21ee731c97487>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f2cfb4a85c82882c571274573fd7ff646d380b63>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=41df5c5289c970b40a002a32dcfdcb61ddc5a2f5>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=bd303443be97c5aa5f8696f6f1ecc5e40b919874>

managed to get all remaining necessary dependencies built to run `guix
system reconfigure' on the Hurd*. The only patch that we need that's not
on master yet is this one.

I'm still rooting for Josselin's graphml cycle analysis and/or lechner
Felix's bespoke make rewrite will render this hack obsolete real soon
now...

If there are no objections I'll go forward and install this patch by the
end of the weekend.

Greetings,
Janneke

*) using the hurd-team branch, grub install still fails, apparently
because "someone" imagines /dev/wd0 to be /dev/hd0, but the new
system derivation is being enabled.

--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
J
J
Janneke Nieuwenhuizen wrote on 18 Sep 2023 06:52
(name . Ludovic Courtès)(address . ludo@gnu.org)
87led4m6ak.fsf@gnu.org
Janneke Nieuwenhuizen writes:

Hi!

Toggle quote (7 lines)
> Janneke Nieuwenhuizen writes:
>
>> Ludovic Courtès writes:

> If there are no objections I'll go forward and install this patch by the
> end of the weekend.

So, I've been running guix pull from hurd-team on the Hurd all day.
Starting with the max-25 files split without (gc), it failed 5
consecutive times building guix-packages-base. Because the initial
patch worked pretty well and had the (gc) call (and yes, also used the
26-way split), I re-added (gc), and now the fourth attempt succeeded.
After `guix-packages-base' built, `guix-packages' then built right away.

I seem to remember guix-packages was the main problem when I wrote the
first version of this patch some four months ago, but that one split the
list of packages 26-way of course. That suggests that the (combined)
compilation of certain files is especially problematic, or that it has
since become more problematic (https://issues.guix.gnu.org/66063).

On the other hand, we may be able to use more than 4GiB of memory on the
Hurd some time soon (and after we upgrade and compile using
--enable-pae)

Pushed to master as 658de25e990a56265e2398a3737e9cf1eb57af68

Greetings,
Janneke

--
Janneke Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com| Avatar® https://AvatarAcademy.com
Closed
?