High memory usage during guix pull (i686-linux, guile jit)

  • Open
  • quality assurance status badge
Details
One participant
  • Dariqq
Owner
unassigned
Submitted by
Dariqq
Severity
normal
D
D
Dariqq wrote on 16 Nov 13:58 +0100
(address . bug-guix@gnu.org)
3faad437-c87c-44d0-ad77-9e42418bf2dd@posteo.net
Hi,

Using 'guix pull' on i686-linux regularly fails when building
'guix-packages-base':

E.g this recent run on ci.guix.gnu.org:

Log from me trying to pull 3e8d3d80f41e016cdfe80e488a78c2351c94fef8:

Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 64 KiB
GC Warning: Out of Memory! Heap size: 3436 MiB. Returning NULL!
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
[ 19/ 20] compiling... 90.0% of 10 filesallocating JIT code buffer
failed: Cannot allocate memory
JIT failed due to resource exhaustion
disabling automatic JIT compilation
allocate_stack failed: Cannot allocate memory
Warning: Unwind-only stack overflow exception; skipping pre-unwind handler.
[...]

The error suggests this might be related to guile jit compiling things

Disabling it by adding GUILE_JIT_THRESHOLD=-1 to the guix-daemon
environment got things working for me again.
D
D
Dariqq wrote 6 days ago
(address . 74381@debbugs.gnu.org)
a9f06663-01e4-4b47-8a1e-7330135d5c4c@posteo.net
It turns out disabling guile jit does not work reliably and even without
it guix pull may oom building guix-packages-base using 3GB+ of RAM.

My workaround for now is to run 'guix pull -s i686-linux -p
/tmp/whatever' on an x86_64 machine (to have more RAM available) with
guile jit disabled multiple times until it succeeds (not exactly sure
why that works, is something getting cached somewhere?) and use that
machine to serve substitutes.

After mentioning this on irc today janneke mentioned this commit
which reduces the number of files per chunk from 25 to 10 to get guix
pull working on the 32 bit hurd.

I tried reducing that even further to 5 in my local checkout and the max
RAM usage went to (a little bit more reasonable) 1.5-2 GB (both on i686
natively and when emulating with -s i686 with and without guile jit).
This would be still not enough for my little machine but might be enough
for the substitute servers to build it reliably.


I also wanted to try this in a i586-gnu childhurd but guix pull
immediately segfaulted.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74381@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74381
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch