OOM error in graphical installer tests.

DoneSubmitted by Mathieu Othacehe.
Details
3 participants
  • Bengt Richter
  • Ludovic Courtès
  • Mathieu Othacehe
Owner
unassigned
Severity
important
M
M
Mathieu Othacehe wrote on 23 Apr 2020 11:49
(address . bug-guix@gnu.org)
877dy6o7kl.fsf@gmail.com
Hello,
The "gui-installed-os" test fails on master because "guix system init"command is OOM killed. Log of the failure is attached.
Thanks,
Mathieu
Attachment: oom.log
L
L
Ludovic Courtès wrote on 24 Apr 2020 16:25
control message for bug #40790
(address . control@debbugs.gnu.org)
87368thsfp.fsf@gnu.org
severity 40790 importantquit
L
L
Ludovic Courtès wrote on 24 Apr 2020 16:27
Re: bug#40790: OOM error in graphical installer tests.
(name . Mathieu Othacehe)(address . m.othacehe@gmail.com)(address . 40790@debbugs.gnu.org)
87wo65gdry.fsf@gnu.org
Hi Mathieu,
Mathieu Othacehe <m.othacehe@gmail.com> skribis:
Toggle quote (3 lines)> The "gui-installed-os" test fails on master because "guix system init"> command is OOM killed. Log of the failure is attached.
[...]
Toggle quote (2 lines)> Apr 23 11:43:27 localhost installer[230]: command ("guix" "system" "init" "--fallback" "--no-grafts" "--no-substitutes" "/mnt/etc/config.scm" "/mnt") failed with exit code #finstaller[230]: Umounted /remove suc
Can we throw a little bit more memory at it? :-)
Is there a commit that trigger this failure?
Thanks,Ludo’.
M
M
Mathieu Othacehe wrote on 29 Apr 2020 10:21
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 40790-done@debbugs.gnu.org)
87v9lisnwm.fsf@gmail.com
Hello,
Toggle quote (2 lines)> Can we throw a little bit more memory at it? :-)
That's what I did with ae1a0f758 :)
Toggle quote (2 lines)> Is there a commit that trigger this failure?
I didn't have the courage to bisect it, given how long this testis. Fixing it, I discovered two more issues:
* "sherpherd services" test was failing because a wrong swap device was used (fixed with be3abf28a).
* I have some hangs when doing the final testing in marionette. Still trying to understand why.
Thanks,
Mathieu
Closed
M
M
Mathieu Othacehe wrote on 29 Apr 2020 17:32
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 40790-done@debbugs.gnu.org)
874kt2xq74.fsf@gmail.com
Toggle quote (3 lines)> * I have some hangs when doing the final testing in marionette. Still> trying to understand why.
Fixed with c537d2bcb7c652b4776435cd1e2a5119af43dd78. Still an issueremaining. "gui-installed-desktop-os-encrypted" closure is about 5.5GiBwhich makes too much files for libisofs:
Toggle snippet (7 lines)xorriso : UPDATE : 427934 files added in 19 secondslibisofs: FAILURE : HFS+ map nodes aren't implementedlibisofs: FAILURE : Too much files to mangle, cannot guarantee unique file namesxorriso : FAILURE : Failed to prepare session write runxorriso : NOTE : -return_with SORRY 32 triggered by problem severity FAILURE
Not sure how to fix it. Someone?
Thanks,
Mathieu
Closed
L
L
Ludovic Courtès wrote on 30 Apr 2020 23:10
(name . Mathieu Othacehe)(address . m.othacehe@gmail.com)(address . 40790-done@debbugs.gnu.org)
87v9lgof2q.fsf@gnu.org
Hi,
Mathieu Othacehe <m.othacehe@gmail.com> skribis:
Toggle quote (4 lines)>> Can we throw a little bit more memory at it? :-)>> That's what I did with ae1a0f758 :)
Oh, good.
Next we should of course find out why ‘guix system init’ is consuming somuch memory. I haven’t tried GC-profiling that yet, but here’s a roughfigure:
Toggle snippet (15 lines)$ GUIX_PROFILING=gc GUIX_PROFILING_EVENTS=exit guix system build -d --no-grafts gnu/system/examples/desktop.tmpl/gnu/store/2by26x4pvvp9vi1f0lmrq39wqqkj0s0f-system.drvGarbage collection statistics: heap size: 98.80 MiB allocated: 548.35 MiB GC times: 26 time spent in GC: 1.84 seconds (36% of user time)$ guix describeGeneracio 140 Apr 29 2020 19:43:22 (nuna) guix 19987f5 repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: 19987f5efe0acb65918802681f3a18cbb6a07c2a
So at first sight, it’s surprising that it couldn’t fit in the VM with800┬áMiB of RAM. Oh well, we’ll see!
Nitpick: please include the canonical bug URL in commit logs ratherthan a link to the mailing list, like so:
Fixes https://bugs.gnu.org/40790.
That makes it easier to grep for fixed bugs.
Toggle quote (6 lines)> I didn't have the courage to bisect it, given how long this test> is. Fixing it, I discovered two more issues:>> * "sherpherd services" test was failing because a wrong swap device was> used (fixed with be3abf28a).
Good catch!
Toggle quote (3 lines)> * I have some hangs when doing the final testing in marionette. Still> trying to understand why.
Oh.
Thank you!
Ludo’.
Closed
B
B
Bengt Richter wrote on 1 May 2020 12:13
(name . Ludovic Courtès)(address . ludo@gnu.org)
20200501101341.GB8427@LionPure
Hi Ludo, et al,
On +2020-04-30 23:10:05 +0200, Ludovic Courtès wrote:[...]
rc1: best-practice: commit logs: message syntax
Toggle snippet (8 lines)> Nitpick: please include the canonical bug URL in commit logs rather> than a link to the mailing list, like so:> > Fixes <https://bugs.gnu.org/40790>.> > That makes it easier to grep for fixed bugs.
ISTM your "nit-pick" is a good example of an embedded mail-list nugget.
Here's my wish:If you and other developers notice (without false modesty ;-) that youare posting a good example of something, then if you would enclose itin guix snippet-delimiters as above, with a single prefixed linestarting with "primary-focus-subject:" and optionally followed by othergreppable (or list-matchable) sub-categories or ad-hoc words or names or tags,then these nuggets would be easy to extract automatically, tocreate a treasure-trove of searchable useful goodies.
I would hope to see useful one-liners and examples of package definition,system configs, debugging methods, small scripts, and good info-URLs etc.delimited and prefixed.
(The "primary-focus-subject:" of "rc1:" would permit us to invent improvedsyntaxes and evolve them as "rc2" etc until we settle on a canonical setfor primary foci, (and PEG grammar for the rest, if we want to get fancy).
Perhaps the servers could have a gem-and-nugget-bot authoring an htmlblog of these things, with automatic attibutions etc from email headers?
-- Regards,Bengt Richter
Closed
L
L
Ludovic Courtès wrote on 2 May 2020 15:54
(name . Mathieu Othacehe)(address . m.othacehe@gmail.com)(address . 40790@debbugs.gnu.org)
87imhe7881.fsf@gnu.org
Hi,
Mathieu Othacehe <m.othacehe@gmail.com> skribis:
Toggle quote (5 lines)>> * I have some hangs when doing the final testing in marionette. Still>> trying to understand why.>> Fixed with c537d2bcb7c652b4776435cd1e2a5119af43dd78.
Cool, sorry for breaking things. :-/
Toggle quote (11 lines)> Still an issue remaining. "gui-installed-desktop-os-encrypted" closure> is about 5.5GiB which makes too much files for libisofs:>> xorriso : UPDATE : 427934 files added in 19 seconds> libisofs: FAILURE : HFS+ map nodes aren't implemented> libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file names> xorriso : FAILURE : Failed to prepare session write run> xorriso : NOTE : -return_with SORRY 32 triggered by problem severity FAILURE>> Not sure how to fix it. Someone?
I tried the attached patch, which would select just one desktopenvironments, thinking it might appease Xorriso, but actually it stillOOMs at the very last step:
Toggle snippet (9 lines)xorriso : UPDATE : 141879 file filters processed in 848 secondsxorriso : UPDATE : 142026 file filters processed in 849 secondsxorriso : UPDATE : 142102 file filters processed in 850 secondsxorriso : NOTE : Copying to System Area: 512 bytes from file '/gnu/store/dvfl5ax7pj6c62f1ccc27qcrcb8nafr7-grub-hybrid-2.04/lib/grub/i386-pc/boot_hybrid.img'[13266.747348] xorriso invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0[13266.749415] CPU: 0 PID: 251 Comm: xorriso Not tainted 5.4.36-gnu #1[13266.750777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
Testing again with this extra patch:
Toggle diff (13 lines)diff --git a/gnu/system/vm.scm b/gnu/system/vm.scmindex 6f81ac16ff..9f240b50ee 100644--- a/gnu/system/vm.scm+++ b/gnu/system/vm.scm@@ -356,7 +356,7 @@ INPUTS is a list of inputs (as for packages)." #:substitutable? substitutable? ;; Xorriso seems to be quite memory-hungry, so increase the VM's RAM size.- #:memory-size 512))+ #:memory-size 1024)) (define* (qemu-image #:key (name "qemu-image")
I wonder if why it’s taking so much memory, and whether there’ssomething else at play, like temporary files written to the overlaidtmpfs and thus eating memory.
Thoughts?
Ludo’.
Toggle diff (62 lines)diff --git a/gnu/tests/install.scm b/gnu/tests/install.scmindex 019e21fd39..941958d698 100644--- a/gnu/tests/install.scm+++ b/gnu/tests/install.scm@@ -957,7 +957,7 @@ build (current-guix) and then store a couple of full system images.") (define* (gui-test-program marionette #:key- (desktop? #f)+ (desktops '()) (encrypted? #f)) #~(let () (define (screenshot file)@@ -971,7 +971,9 @@ build (current-guix) and then store a couple of full system images.") (setvbuf (current-output-port) 'none) (setvbuf (current-error-port) 'none) - (marionette-eval* '(use-modules (gnu installer tests))+ (marionette-eval* '(use-modules (gnu installer tests)+ (srfi srfi-1)+ (srfi srfi-26)) #$marionette) ;; Arrange so that 'converse' prints debugging output to the console.@@ -1016,7 +1018,9 @@ build (current-guix) and then store a couple of full system images.") (marionette-eval* '(choose-services installer-socket #:choose-desktop-environment?- (const #$desktop?)+ (lambda (desktop)+ (any (cut string-contains desktop <>)+ '#$desktops)) #:choose-network-service? (const #f)) #$marionette)@@ -1110,7 +1114,7 @@ build (current-guix) and then store a couple of full system images.") (define* (guided-installation-test name #:key- (desktop? #f)+ (desktops '()) (encrypted? #f) target-os (install-size 'guess)@@ -1132,7 +1136,7 @@ build (current-guix) and then store a couple of full system images.") (lambda (marionette) (gui-test-program marionette- #:desktop? desktop?+ #:desktops desktops #:encrypted? encrypted?)))) (command (qemu-command/writable-image image))) (run-basic-test target-os command name@@ -1155,7 +1159,7 @@ build (current-guix) and then store a couple of full system images.") ;; desktop environments in a single test to reduce the overhead. (define %test-gui-installed-desktop-os-encrypted (guided-installation-test "gui-installed-desktop-os-encrypted"- #:desktop? #t+ #:desktops '("GNOME") #:encrypted? #t #:target-os (installation-target-desktop-os-for-gui-tests
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 40790@debbugs.gnu.org