Guile-SSH channel finalization leads to segfaults on Guile 2.2

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Mark H Weaver
  • Artyom Poptsov
Owner
unassigned
Submitted by
Mark H Weaver
Severity
serious
M
M
Mark H Weaver wrote on 18 May 2017 03:55
On Hydra, offload crashes while trying to build linux-libre source
(address . bug-guix@gnu.org)
87h90japz0.fsf@netris.org
On Hydra, the builds of linux-libre-4.11.1's source code aborted on all
three hydra-supported-systems:


The failure output on x86_64 and i686 are roughly the same:

@ hook-failed /gnu/store/1nlgkdh3s91f7l3zzsxh6np0xbqzq574-linux-libre-4.11.1-gnu.tar.xz.drv - 11 builder for `/gnu/store/1nlgkdh3s91f7l3zzsxh6np0xbqzq574-linux-libre-4.11.1-gnu.tar.xz.drv' failed due to signal 11 (Segmentation fault)

On armhf, the error seems to happen in the same place, but with far more
detailed output, included below.

Mark


*** Error in `/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/bin/guile': realloc(): invalid next size: 0x00000000024617d0 ***
======= Backtrace: =========
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x70fd5)[0x7f77e8343fd5]
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x773a6)[0x7f77e834a3a6]
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x7a3a9)[0x7f77e834d3a9]
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(realloc+0x156)[0x7f77e834e6e6]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xdc6b)[0x7f77e2e24c6b]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xddce)[0x7f77e2e24dce]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xe50a)[0x7f77e2e2550a]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xe7b2)[0x7f77e2e257b2]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(ssh_channel_close+0x47)[0x7f77e2e27f87]
/gnu/store/avy681pwf979kbwiv9k75c5h7jdink2c-guile2.2-ssh-0.11.0/lib/libguile-ssh.so.11(+0xa597)[0x7f77e3290597]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0x83785)[0x7f77e9f00785]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_close_port+0x8b)[0x7f77e9f03a0b]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0xc2c4d)[0x7f77e9f3fc4d]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_call_n+0x16a)[0x7f77e9f432aa]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0xb54e6)[0x7f77e9f324e6]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_internal_catch+0xe)[0x7f77e9f329ee]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0x820a4)[0x7f77e9eff0a4]
/gnu/store/j4p7cnxb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1(GC_invoke_finalizers+0x6f)[0x7f77e9c1f70f]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_run_finalizers+0x9)[0x7f77e9ed0099]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0x530f5)[0x7f77e9ed00f5]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0x43cba)[0x7f77e9ec0cba]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0xc2c4d)[0x7f77e9f3fc4d]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_call_n+0x16a)[0x7f77e9f432aa]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0xb54e6)[0x7f77e9f324e6]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0x442a0)[0x7f77e9ec12a0]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_c_with_continuation_barrier+0x35)[0x7f77e9ec1335]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0xb410c)[0x7f77e9f3110c]
/gnu/store/j4p7cnxb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1(GC_call_with_stack_base+0x12)[0x7f77e9c26732]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_with_guile+0x28)[0x7f77e9f31498]
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0(+0x7454)[0x7f77e99f3454]
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(clone+0x3f)[0x7f77e83bb7bf]
======= Memory map: ========
00400000-00401000 r-xp 00000000 ca:01 87687205 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/bin/guile
00601000-00602000 r--p 00001000 ca:01 87687205 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/bin/guile
00602000-00603000 rw-p 00002000 ca:01 87687205 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/bin/guile
0124a000-02482000 rw-p 00000000 00:00 0 [heap]
7f77d4000000-7f77d4022000 rw-p 00000000 00:00 0
7f77d4022000-7f77d8000000 ---p 00000000 00:00 0
7f77dc000000-7f77dc021000 rw-p 00000000 00:00 0
7f77dc021000-7f77e0000000 ---p 00000000 00:00 0
7f77e2576000-7f77e2588000 r-xp 00000000 ca:01 86119335 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libresolv-2.25.so
7f77e2588000-7f77e2788000 ---p 00012000 ca:01 86119335 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libresolv-2.25.so
7f77e2788000-7f77e2789000 r--p 00012000 ca:01 86119335 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libresolv-2.25.so
7f77e2789000-7f77e278a000 rw-p 00013000 ca:01 86119335 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libresolv-2.25.so
7f77e278a000-7f77e278c000 rw-p 00000000 00:00 0
7f77e278c000-7f77e2791000 r-xp 00000000 ca:01 86119300 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libnss_dns-2.25.so
7f77e2791000-7f77e2990000 ---p 00005000 ca:01 86119300 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libnss_dns-2.25.so
7f77e2990000-7f77e2991000 r--p 00004000 ca:01 86119300 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libnss_dns-2.25.so
7f77e2991000-7f77e2992000 rw-p 00005000 ca:01 86119300 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libnss_dns-2.25.so
7f77e2992000-7f77e299d000 r-xp 00000000 ca:01 86119304 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libnss_files-2.25.so
7f77e299d000-7f77e2b9c000 ---p 0000b000 ca:01 86119304 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libnss_files-2.25.so
7f77e2b9c000-7f77e2b9d000 r--p 0000a000 ca:01 86119304 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libnss_files-2.25.so
7f77e2b9d000-7f77e2b9e000 rw-p 0000b000 ca:01 86119304 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libnss_files-2.25.so
7f77e2b9e000-7f77e2ba4000 rw-p 00000000 00:00 0
7f77e2ba4000-7f77e2bc4000 r--p 00000000 ca:01 54790438 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/pk-crypto.go
7f77e2bc4000-7f77e2bcd000 rw-p 00020000 ca:01 54790438 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/pk-crypto.go
7f77e2bcd000-7f77e2bd1000 r--p 00029000 ca:01 54790438 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/pk-crypto.go
7f77e2bd1000-7f77e2be1000 r--p 00000000 ca:01 54790440 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/pki.go
7f77e2be1000-7f77e2be4000 rw-p 00010000 ca:01 54790440 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/pki.go
7f77e2be4000-7f77e2be6000 r--p 00013000 ca:01 54790440 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/pki.go
7f77e2be6000-7f77e2bf6000 r--p 00000000 ca:01 54790430 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/nar.go
7f77e2bf6000-7f77e2bf9000 rw-p 00010000 ca:01 54790430 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/nar.go
7f77e2bf9000-7f77e2bfb000 r--p 00013000 ca:01 54790430 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/nar.go
7f77e2bfb000-7f77e2c0b000 r--p 00000000 ca:01 54790501 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/ssh.go
7f77e2c0b000-7f77e2c0e000 rw-p 00010000 ca:01 54790501 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/ssh.go
7f77e2c0e000-7f77e2c0f000 r--p 00013000 ca:01 54790501 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/ssh.go
7f77e2c0f000-7f77e2c16000 r-xp 00000000 ca:01 86119365 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/librt-2.25.so
7f77e2c16000-7f77e2e15000 ---p 00007000 ca:01 86119365 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/librt-2.25.so
7f77e2e15000-7f77e2e16000 r--p 00006000 ca:01 86119365 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/librt-2.25.so
7f77e2e16000-7f77e2e17000 rw-p 00007000 ca:01 86119365 /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/librt-2.25.so
7f77e2e17000-7f77e2e81000 r-xp 00000000 ca:01 93586244 /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4.4.1
7f77e2e81000-7f77e3081000 ---p 0006a000 ca:01 93586244 /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4.4.1
7f77e3081000-7f77e3082000 r--p 0006a000 ca:01 93586244 /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4.4.1
7f77e3082000-7f77e3084000 rw-p 0006b000 ca:01 93586244 /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4.4.1
7f77e3084000-7f77e3085000 r-xp 00000000 ca:01 93586248 /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh_threads.so.4.4.1
7f77e3085000-7f77e3284000 ---p 00001000 ca:01 93586248 /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh_threads.so.4.4.1
7f77e3284000-7f77e3285000 r--p 00000000 ca:01 93586248 /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh_threads.so.4.4.1
7f77e3285000-7f77e3286000 rw-p 00001000 ca:01 93586248 /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh_threads.so.4.4.1
7f77e3286000-7f77e329a000 r-xp 00000000 ca:01 54399701 /gnu/store/avy681pwf979kbwiv9k75c5h7jdink2c-guile2.2-ssh-0.11.0/lib/libguile-ssh.so.11.0.0
7f77e329a000-7f77e3499000 ---p 00014000 ca:01 54399701 /gnu/store/avy681pwf979kbwiv9k75c5h7jdink2c-guile2.2-ssh-0.11.0/lib/libguile-ssh.so.11.0.0
7f77e3499000-7f77e349a000 r--p 00013000 ca:01 54399701 /gnu/store/avy681pwf979kbwiv9k75c5h7jdink2c-guile2.2-ssh-0.11.0/lib/libguile-ssh.so.11.0.0
7f77e349a000-7f77e349b000 rw-p 00014000 ca:01 54399701 /gnu/store/avy681pwf979kbwiv9k75c5h7jdink2c-guile2.2-ssh-0.11.0/lib/libguile-ssh.so.11.0.0
7f77e349b000-7f77e34db000 r--p 00000000 ca:01 54790487 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/scripts/offload.go
7f77e34db000-7f77e34f4000 rw-p 00040000 ca:01 54790487 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/scripts/offload.go
7f77e34f4000-7f77e34ff000 r--p 00059000 ca:01 54790487 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/scripts/offload.go
7f77e34ff000-7f77e3500000 rw-p 00000000 00:00 0
7f77e3500000-7f77e3501000 ---p 00000000 00:00 0
7f77e3501000-7f77e3d01000 rw-p 00000000 00:00 0
7f77e3d01000-7f77e3d11000 r--p 00000000 ca:01 87688157 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/srfi/srfi-14.go
7f77e3d11000-7f77e3d12000 rw-p 00010000 ca:01 87688157 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/srfi/srfi-14.go
7f77e3d12000-7f77e3d13000 r--p 00011000 ca:01 87688157 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/srfi/srfi-14.go
7f77e3d13000-7f77e3d23000 r--p 00000000 ca:01 87688283 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo/string-utils.go
7f77e3d23000-7f77e3d24000 rw-p 00010000 ca:01 87688283 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo/string-utils.go
7f77e3d24000-7f77e3d28000 r--p 00011000 ca:01 87688283 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo/string-utils.go
7f77e3d28000-7f77e3d38000 r--p 00000000 ca:01 87687395 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo/plain-text.go
7f77e3d38000-7f77e3d3b000 rw-p 00010000 ca:01 87687395 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo/plain-text.go
7f77e3d3b000-7f77e3d3e000 r--p 00013000 ca:01 87687395 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo/plain-text.go
7f77e3d3e000-7f77e3d4e000 r--p 00000000 ca:01 87688209 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/transform.go
7f77e3d4e000-7f77e3d4f000 rw-p 00010000 ca:01 87688209 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/transform.go
7f77e3d4f000-7f77e3d50000 r--p 00011000 ca:01 87688209 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/transform.go
7f77e3d50000-7f77e3d70000 r--p 00000000 ca:01 87688208 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/ssax.go
7f77e3d70000-7f77e3d7f000 rw-p 00020000 ca:01 87688208 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/ssax.go
7f77e3d7f000-7f77e3d85000 r--p 0002f000 ca:01 87688208 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/ssax.go
7f77e3d85000-7f77e3d95000 r--p 00000000 ca:01 87688206 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/ssax/input-parse.go
7f77e3d95000-7f77e3d97000 rw-p 00010000 ca:01 87688206 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/ssax/input-parse.go
7f77e3d97000-7f77e3d99000 r--p 00012000 ca:01 87688206 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/ssax/input-parse.go
7f77e3d99000-7f77e3da9000 r--p 00000000 ca:01 87688203 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/simple.go
7f77e3da9000-7f77e3dab000 rw-p 00010000 ca:01 87688203 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/simple.go
7f77e3dab000-7f77e3dae000 r--p 00012000 ca:01 87688203 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/sxml/simple.go
7f77e3dae000-7f77e3dbe000 r--p 00000000 ca:01 87688284 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo.go
7f77e3dbe000-7f77e3dc5000 rw-p 00010000 ca:01 87688284 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo.go
7f77e3dc5000-7f77e3dc9000 r--p 00017000 ca:01 87688284 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/texinfo.go
7f77e3dc9000-7f77e3dd9000 r--p 00000000 ca:01 87688168 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/srfi/srfi-31.go
7f77e3dd9000-7f77e3dda000 rw-p 00010000 ca:01 87688168 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/srfi/srfi-31.go
7f77e3dda000-7f77e3ddb000 r--p 00011000 ca:01 87688168 /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/guile/2.2/ccache/srfi/srfi-31.go
7f77e3ddb000-7f77e3dfb000 r--p 00000000 ca:01 54790425 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/licenses.go
7f77e3dfb000-7f77e3e05000 rw-p 00020000 ca:01 54790425 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/licenses.go
7f77e3e05000-7f77e3e08000 r--p 0002a000 ca:01 54790425 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/licenses.go
7f77e3e08000-7f77e3e48000 r--p 00000000 ca:01 54790442 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/profiles.go
7f77e3e48000-7f77e3e63000 rw-p 00040000 ca:01 54790442 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/profiles.go
7f77e3e63000-7f77e3e73000 r--p 0005b000 ca:01 54790442 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/profiles.go
7f77e3e73000-7f77e3e93000 r--p 00000000 ca:01 54790498 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/search-paths.go
7f77e3e93000-7f77e3e9f000 rw-p 00020000 ca:01 54790498 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/search-paths.go
7f77e3e9f000-7f77e3ea3000 r--p 0002c000 ca:01 54790498 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/search-paths.go
7f77e3ea3000-7f77e3ed3000 r--p 00000000 ca:01 54790272 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/build-system.go
7f77e3ed3000-7f77e3ee6000 rw-p 00030000 ca:01 54790272 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/build-system.go
7f77e3ee6000-7f77e3eec000 r--p 00043000 ca:01 54790272 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/build-system.go
7f77e3eec000-7f77e3f4c000 r--p 00000000 ca:01 54790433 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/packages.go
7f77e3f4c000-7f77e3f79000 rw-p 00060000 ca:01 54790433 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/packages.go
7f77e3f79000-7f77e3f8d000 r--p 0008d000 ca:01 54790433 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/packages.go
7f77e3f8d000-7f77e3fad000 r--p 00000000 ca:01 54790383 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/grafts.go
7f77e3fad000-7f77e3fb9000 rw-p 00020000 ca:01 54790383 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/grafts.go
7f77e3fb9000-7f77e3fbe000 r--p 0002c000 ca:01 54790383 /gnu/store/d4wwx93gqizx132zjk7h1ir7rzph0pig-guix-0.12.0-10.ba2260d/lib/guile/2.2/site-ccache/guix/grafts.go
7f77e3fbe000-7f77e3fce000 r--p 00000000 ca:01 54790500
This message was truncated. Download the full message here.
M
M
Mark H Weaver wrote on 18 May 2017 04:02
(address . 26976@debbugs.gnu.org)
87d1b6c47j.fsf@netris.org
Mark H Weaver <mhw@netris.org> writes:

Toggle quote (14 lines)
> On Hydra, the builds of linux-libre-4.11.1's source code aborted on all
> three hydra-supported-systems:
>
> https://hydra.gnu.org/build/2071337 (x86_64)
> https://hydra.gnu.org/build/2071533 (i686)
> https://hydra.gnu.org/build/2071510 (armhf)
>
> The failure output on x86_64 and i686 are roughly the same:
>
> @ hook-failed /gnu/store/1nlgkdh3s91f7l3zzsxh6np0xbqzq574-linux-libre-4.11.1-gnu.tar.xz.drv - 11 builder for `/gnu/store/1nlgkdh3s91f7l3zzsxh6np0xbqzq574-linux-libre-4.11.1-gnu.tar.xz.drv' failed due to signal 11 (Segmentation fault)
>
> On armhf, the error seems to happen in the same place, but with far more
> detailed output, included below.

The problem is not limited to linux-libre. Many important builds are
failing in the same way, including the 'guix' package:


I would say that Hydra is mostly broken right now. This might be
related to the recent switch on Hydra to guix-based-on-guile-2.2.

Mark
L
L
Ludovic Courtès wrote on 18 May 2017 21:39
control message for bug #26976
(address . control@debbugs.gnu.org)
8737c2ar9z.fsf@gnu.org
severity 26976 serious
L
L
Ludovic Courtès wrote on 19 May 2017 00:00
Re: bug#26976: On Hydra, offload crashes while trying to build linux-libre source
(name . Mark H Weaver)(address . mhw@netris.org)
87h90h966f.fsf@gnu.org
Hi Mark,

(Cc: Artyom. Artyom, this is about what looks like a bug in Guile-SSH
when used with Guile 2.2; see https://bugs.gnu.org/26976.)

Mark H Weaver <mhw@netris.org> skribis:

Toggle quote (14 lines)
> *** Error in `/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/bin/guile': realloc(): invalid next size: 0x00000000024617d0 ***
> ======= Backtrace: =========
> /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x70fd5)[0x7f77e8343fd5]
> /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x773a6)[0x7f77e834a3a6]
> /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x7a3a9)[0x7f77e834d3a9]
> /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(realloc+0x156)[0x7f77e834e6e6]
> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xdc6b)[0x7f77e2e24c6b]
> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xddce)[0x7f77e2e24dce]
> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xe50a)[0x7f77e2e2550a]
> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xe7b2)[0x7f77e2e257b2]
> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(ssh_channel_close+0x47)[0x7f77e2e27f87]
> /gnu/store/avy681pwf979kbwiv9k75c5h7jdink2c-guile2.2-ssh-0.11.0/lib/libguile-ssh.so.11(+0xa597)[0x7f77e3290597]
> /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0x83785)[0x7f77e9f00785]

This looks like a double-free and ‘ssh_channel_close’ has only one call
site, which is ‘ptob_close’, the ‘close’ function for the channel port
type in Guile-SSH.

I’m quite confident that the attached patch fixes the problem. However,
I haven’t found a scenario in Guile 2.2 where the ‘close’ method could
be called more than once, and I cannot reproduce the bug on my machine.
Thoughts?

I suggest applying it to the ‘guile-ssh’ package in Guix.

Thanks,
Ludo’.
Toggle diff (34 lines)
diff --git a/libguile-ssh/channel-type.c b/libguile-ssh/channel-type.c
index 3dd641f..0839854 100644
--- a/libguile-ssh/channel-type.c
+++ b/libguile-ssh/channel-type.c
@@ -229,10 +229,11 @@ ptob_close (SCM channel)
ssh_channel_free (ch->ssh_channel);
}
+ SCM_SETSTREAM (channel, NULL);
+
#if USING_GUILE_BEFORE_2_2
scm_gc_free (pt->write_buf, pt->write_buf_size, "port write buffer");
scm_gc_free (pt->read_buf, pt->read_buf_size, "port read buffer");
- SCM_SETSTREAM (channel, NULL);
return 0;
#endif
diff --git a/libguile-ssh/sftp-file-type.c b/libguile-ssh/sftp-file-type.c
index 8879924..f87cf03 100644
--- a/libguile-ssh/sftp-file-type.c
+++ b/libguile-ssh/sftp-file-type.c
@@ -224,10 +224,11 @@ ptob_close (SCM sftp_file)
sftp_close (fd->file);
}
+ SCM_SETSTREAM (sftp_file, NULL);
+
#if USING_GUILE_BEFORE_2_2
scm_gc_free (pt->write_buf, pt->write_buf_size, "port write buffer");
scm_gc_free (pt->read_buf, pt->read_buf_size, "port read buffer");
- SCM_SETSTREAM (sftp_file, NULL);
return 1;
#endif
A
A
Artyom Poptsov wrote on 19 May 2017 05:20
(name . Ludovic Courtès)(address . ludo@gnu.org)
87r2zljzvx.fsf@gmail.com
Hello Ludovic and Mark,

thanks for notifying me about the bug and especially for the patch
attached. I'll take a look into it in the next few days.

- Artyom

--
Artyom V. Poptsov <poptsov.artyom@gmail.com>; GPG Key: 0898A02F
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEr9DIunjgTzVb3YfdTA30jTkDNZoFAlkeZJMACgkQTA30jTkD
NZr/EQ/9FMWQt1TyoSkStZftTy+U6eNSR+6hg/dSE1O7V6jmNG1Oxxe6ui2v8Jyt
4L0dpqBMGYV03Ui0zWGzXOKft8K814OWOipVAUEMd5CpG6l4ruhUzyf6TxSksz7+
RpfBYHCa2W8stadydtRq3b87PI4tTr7tHNEDc3G3woKZin70cjV3ipU31AK0dkvM
s+/srMKyvThcjLG/sI/gb1gTIs24eWMb8BE2DCgh+VJDxDGqp6F9j9mNV7SJ1kGC
KA5PCeFqiZF/ovOdqAURTc0E8KfLzacjIK7wfmxl/iGJXoJnVofxYJv144Jr+ZeM
H6NAe2CcR1VicxwByT0unvf0SaflRNYYvrQyerlB9FG/nitprDlO2gKaAHA6CN69
dqMP9LuKerOHx9sM30jW3RE3vzAhIIDPMEZux5UJaRWY9+paeivPn3t8ZJqplrVG
hfAqpGHo4r0Mzm9nFlTccH/83l5ux36TejD4sc4lynDdlSYeMin0ObiB/FEJbKzT
kscJNXrAwcP4+oBVW4feJpbC6IzZqRGK654/3YDUAZCRrUqdY33l4GfEf8/M8TEH
pLOBX2iQZNRv84Qx3rIFZWE634VD3MtQGd4Shv6CWsSuKjDAEwOtt/j23Si8xpYj
WfARx7qv8JjswTffiPOknaX0GJbCSpl2wg+QMm7a+FvUfOeDy3M=
=KsFq
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 19 May 2017 14:44
(name . Mark H Weaver)(address . mhw@netris.org)
87pof5dnja.fsf@gnu.org
Hi again,

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (30 lines)
> (Cc: Artyom. Artyom, this is about what looks like a bug in Guile-SSH
> when used with Guile 2.2; see <https://bugs.gnu.org/26976>.)
>
> Mark H Weaver <mhw@netris.org> skribis:
>
>> *** Error in `/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/bin/guile': realloc(): invalid next size: 0x00000000024617d0 ***
>> ======= Backtrace: =========
>> /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x70fd5)[0x7f77e8343fd5]
>> /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x773a6)[0x7f77e834a3a6]
>> /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x7a3a9)[0x7f77e834d3a9]
>> /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(realloc+0x156)[0x7f77e834e6e6]
>> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xdc6b)[0x7f77e2e24c6b]
>> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xddce)[0x7f77e2e24dce]
>> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xe50a)[0x7f77e2e2550a]
>> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xe7b2)[0x7f77e2e257b2]
>> /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(ssh_channel_close+0x47)[0x7f77e2e27f87]
>> /gnu/store/avy681pwf979kbwiv9k75c5h7jdink2c-guile2.2-ssh-0.11.0/lib/libguile-ssh.so.11(+0xa597)[0x7f77e3290597]
>> /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0x83785)[0x7f77e9f00785]
>
> This looks like a double-free and ‘ssh_channel_close’ has only one call
> site, which is ‘ptob_close’, the ‘close’ function for the channel port
> type in Guile-SSH.
>
> I’m quite confident that the attached patch fixes the problem. However,
> I haven’t found a scenario in Guile 2.2 where the ‘close’ method could
> be called more than once, and I cannot reproduce the bug on my machine.
> Thoughts?
>
> I suggest applying it to the ‘guile-ssh’ package in Guix.

I went ahead and did that, in an attempt to salvage our build farm:


The patch clearly fixes potential issues (at least use-after-free) so it
seemed appropriate to apply it anyway.

I’ve deployed Guix built against this patched Guile-SSH on
hydra.gnu.org. I tried offloading the linux-libre build that you
mentioned, Mark, and that no longer crashed right away. I’ve restarted
the queue-runner and I’m now monitoring the first few builds to see how
it goes:

https://hydra.gnu.org/build/2054610 - segfaulted as before

#2054610 segfaulted early on:

Toggle snippet (9 lines)
process 1808 acquired build slot '/var/guix/offload/hydra.gnunet.org/1'
load on machine 'hydra.gnunet.org' is 0.23 (normalized: 0.115)
process 1808 acquired build slot '/var/guix/offload/guix.sjd.se/0'
load on machine 'guix.sjd.se' is 0.01 (normalized: 0.005)
sending 4 store items to 'guix.sjd.se'...
exporting path `/gnu/store/gi7r1v65zqhh8riqprq8nchfc9v9k156-guix-current'
unknown Nix trace message: @ hook-failed /gnu/store/7d688059y8j4hif7hkjs1cifqcnklw1k-guix-0.12.0-11.ce92d26+.drv - 11 builder for `/gnu/store/7d688059y8j4hif7hkjs1cifqcnklw1k-guix-0.12.0-11.ce92d26+.drv' failed due to signal 11 (Segmentation fault)

I managed to reproduce it and to get a backtrace:

Toggle snippet (23 lines)
@ build-started /gnu/store/7d688059y8j4hif7hkjs1cifqcnklw1k-guix-0.12.0-11.ce92d26+.drv - i686-linux /var/log/guix/drvs/7d//688059y8j4hif7hkjs1cifqcnklw1k-guix-0.12.0-11.ce92d26+.drv
sending 4 store items to 'guix.sjd.se'...
exporting path `/gnu/store/gi7r1v65zqhh8riqprq8nchfc9v9k156-guix-current'
*** Error in `/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/bin/guile': realloc(): invalid next size: 0x0000000001c7c020 ***
======= Backtrace: =========
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x70fd5)[0x7f6f8336afd5]
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x773a6)[0x7f6f833713a6]
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(+0x7a3a9)[0x7f6f833743a9]
/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6(realloc+0x156)[0x7f6f833756e6]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xdc6b)[0x7f6f7de4bc6b]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xdd7d)[0x7f6f7de4bd7d]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0x39793)[0x7f6f7de77793]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0x1eea7)[0x7f6f7de5cea7]
/gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4(+0xf598)[0x7f6f7de4d598]
/gnu/store/ql5h9hxh5560d42xdirh0yxzrgii6i0m-guile-ssh-0.11.0/lib/libguile-ssh.so.11(+0xa5ee)[0x7f6f7e2b75ee]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0x8672c)[0x7f6f84f2a72c]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_put_bytevector+0x94)[0x7f6f84f31de4]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(+0xc2c4d)[0x7f6f84f66c4d]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_call_n+0x16a)[0x7f6f84f6a2aa]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_primitive_eval+0x27)[0x7f6f84eee8d7]
/gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1(scm_primitive_load+0xdb)[0x7f6f84f0a6eb]

Cleaner backtrace from the core dumped:

Toggle snippet (21 lines)
(gdb) bt
#0 0x00007f6f8332d2c4 in raise () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#1 0x00007f6f8332e72a in abort () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#2 0x00007f6f8336afda in __libc_message () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#3 0x00007f6f833713a6 in malloc_printerr () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#4 0x00007f6f833743a9 in _int_realloc () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#5 0x00007f6f833756e6 in realloc () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#6 0x00007f6f7de4bc6b in realloc_buffer () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#7 0x00007f6f7de4bd7d in ssh_buffer_reinit () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#8 0x00007f6f7de77793 in compress_buffer () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#9 0x00007f6f7de5cea7 in packet_send2 () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#10 0x00007f6f7de4d598 in channel_write_common () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#11 0x00007f6f7e2b75ee in write_to_channel_port () from /gnu/store/ql5h9hxh5560d42xdirh0yxzrgii6i0m-guile-ssh-0.11.0/lib/libguile-ssh.so.11
#12 0x00007f6f84f2a72c in scm_i_write_bytes () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1
#13 0x00007f6f84f31de4 in scm_put_bytevector () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1
#14 0x00007f6f84f66c4d in vm_regular_engine () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1
#15 0x00007f6f84f6a2aa in scm_call_n () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1
#16 0x00007f6f84eee8d7 in scm_primitive_eval () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1
#17 0x00007f6f84f0a6eb in scm_primitive_load () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1


Toggle snippet (30 lines)
sending 5 store items to 'hydra-slave2.netris.org'...
exporting path `/gnu/store/yfks7lndwf36arp3xwah5dc07qwk749c-kwidgetsaddons-5.34.0-guile-builder'
exporting path `/gnu/store/zhvvhgdyakxbav26l33zg00x3byns22l-kwidgetsaddons-5.34.0.tar.xz.drv'
exporting path `/gnu/store/kn0hzhnic5qd7aqipyn9firg3nhx2m1n-kwidgetsaddons-5.34.0.drv'
exporting path `/gnu/store/mkvvbawa78dkfdyajlipas41fr5nn0hd-kwidgetsaddons-5.34.0.tar.xz'
Backtrace:
11 (primitive-load "/gnu/store/ys7ghld9ql7knl11mpb3b072nvy?")
In guix/ui.scm:
1264:8 10 (run-guix-command _ . _)
In guix/scripts/offload.scm:
650:22 9 (guix-offload . _)
In ice-9/boot-9.scm:
837:9 8 (catch _ _ #<procedure 7f27d6ab5930 at guix/ui.scm:449?> ?)
837:9 7 (catch _ _ #<procedure 7f27d6ab5948 at guix/ui.scm:510?> ?)
In guix/scripts/offload.scm:
340:4 6 (transfer-and-offload #<derivation /gnu/store/kn0hzhni?> ?)
In guix/ssh.scm:
221:4 5 (send-files _ _ _ #:recursive? _ #:log-port _)
In guix/store.scm:
1193:12 4 (export-paths #<build-daemon 256.97 199e340> _ #<outpu?> ?)
1173:22 3 (export-path #<build-daemon 256.97 199e340> _ #<output?> ?)
580:13 2 (process-stderr _ _)
543:10 1 (dump-port #<input-output: socket 18> #<output: channe?> ?)
In unknown file:
0 (put-bytevector #<output: channel (closed) 15f9a20> # 0 #)

ERROR: In procedure put-bytevector:
ERROR: Throw to key `guile-ssh-error' with args `("write_to_channel_port" "Socket error: Invalid argument" #<output: channel (closed) 15f9a20> #f)'.

… which could be a related problem (it’s the same backtrace).

I’ve stopped the queue-runner while investigating. To be continued…

Ludo’.
L
L
Ludovic Courtès wrote on 20 May 2017 00:36
(name . Mark H Weaver)(address . mhw@netris.org)
87mva88oe9.fsf@gnu.org
Hi again,

I was able to reproduce it on hydra.gnu.org with:

guix copy --to=hydra.gnunet.org /gnu/store/gi7r1v65zqhh8riqprq8nchfc9v9k156-guix-current

which leads most of the time to SIGSEGV (I couldn’t get the SIGSEGV on
my laptop):

Toggle snippet (15 lines)
Thread 1 received signal SIGSEGV, Segmentation fault.
0x00007f121ff61ac7 in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
(gdb) bt
#0 0x00007f121ff61ac7 in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#1 0x00007f121ff6378d in deflate () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#2 0x00007f121e281451 in gzip_compress () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#3 0x00007f121e281783 in compress_buffer () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#4 0x00007f121e266ea7 in packet_send2 () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#5 0x00007f121e257598 in channel_write_common () from /gnu/store/vlc43y485v80sgq7iw60hzy4pw5r52d2-libssh-0.7.4/lib/libssh.so.4
#6 0x00007f121e6c15ee in write_to_channel_port () from /gnu/store/ql5h9hxh5560d42xdirh0yxzrgii6i0m-guile-ssh-0.11.0/lib/libguile-ssh.so.11
#7 0x00007f122532472c in scm_i_write_bytes () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1
#8 0x00007f122532bde4 in scm_put_bytevector () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1
#9 0x00007f1225360c4d in vm_regular_engine () from /gnu/store/5zx29y44nrqj0s8h3jlvlj82k8hj4dxs-guile-2.2.2/lib/libguile-2.2.so.1

Unfortunately, when running under “valgrind --trace-children=yes”, it
doesn’t segfault and Valgrind doesn’t report anything interesting
(neither on hydra.gnu.org nor on my laptop).

I built libssh and guile-ssh with -fsanitize=address on my laptop but
that doesn’t report anything useful either.

Suggestions welcome!

Ludo’.
M
M
Mark H Weaver wrote on 20 May 2017 18:59
(name . Ludovic Courtès)(address . ludo@gnu.org)
87h90ffoqp.fsf@netris.org
ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (2 lines)
> I’ve stopped the queue-runner while investigating. To be continued…

For now, I would suggest downgrading the 'guix' on Hydra to be based on
guile-2.0 again, so that it can be building things while we sort this
out. What do you think?

Mark
L
L
Ludovic Courtès wrote on 20 May 2017 20:02
(name . Mark H Weaver)(address . mhw@netris.org)
87k25b5rvo.fsf@gnu.org
Mark H Weaver <mhw@netris.org> skribis:

Toggle quote (7 lines)
> ludo@gnu.org (Ludovic Courtès) writes:
>> I’ve stopped the queue-runner while investigating. To be continued…
>
> For now, I would suggest downgrading the 'guix' on Hydra to be based on
> guile-2.0 again, so that it can be building things while we sort this
> out. What do you think?

Sounds reasonable, let’s give it a try.

I’ve added a ‘guile2.0-guix’ package in commit
82ce81ba06ea1babfdb0d1c447e972461c1d3018.

If it works, that would suggest that the culprit is the 2.2 code in
Guile-SSH.

Ludo’.
L
L
Ludovic Courtès wrote on 20 May 2017 23:59
(name . Mark H Weaver)(address . mhw@netris.org)
877f1b5gv9.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (7 lines)
> I was able to reproduce it on hydra.gnu.org with:
>
> guix copy --to=hydra.gnunet.org /gnu/store/gi7r1v65zqhh8riqprq8nchfc9v9k156-guix-current
>
> which leads most of the time to SIGSEGV (I couldn’t get the SIGSEGV on
> my laptop):

Similar backtrace with debugging symbols:

Toggle snippet (80 lines)
#0 0x00007fc24587dac7 in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
No symbol table info available.
#1 0x00007fc24587f78d in deflate () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
No symbol table info available.
#2 0x00007fc243b9d4e1 in gzip_compress (session=session@entry=0x10817d0, source=source@entry=0x1082170, level=<optimized out>) at /tmp/guix-build-libssh-0.7.4.drv-0/libssh-0.7.4/src/gzip.c:85
zout = 0x10845a0
in_ptr = 0x1946bc0
in_size = 24744
dest = 0x1085900
out_buf = "\000\000\000\000\000\000\000\000\000\000\235\000\000\000z\377\000\000\200\000\000\000\374\376\000\000\000\200\205\001\000\000\000\200\237\034\000\000@?9\000\000\000\000\000\000\260\032\000\000\000\200\177\064b\004\067\000\000\000\000\270\000\000\000\000@;\224\003\000\000\002\000\000\000\000\000\000\000\000\000\200\253\002\000\000\000\300G9\000\000\200\f7\246\000\000\000\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000\000\230\004\000\000\000f\332\222F\004\000\000\000\064\367\032\067\006\000\000\000\000h\000\000\000\000\000\000\000\000\000\260", '\000' <repeats 13 times>, "v\000\000\000\000 ", '\000' <repeats 46 times>...
len = <optimized out>
status = <optimized out>
__func__ = "gzip_compress"
#3 0x00007fc243b9d813 in compress_buffer (session=session@entry=0x10817d0, buf=0x1082170) at /tmp/guix-build-libssh-0.7.4.drv-0/libssh-0.7.4/src/gzip.c:106
dest = 0x0
#4 0x00007fc243b82f37 in packet_send2 (session=session@entry=0x10817d0) at /tmp/guix-build-libssh-0.7.4.drv-0/libssh-0.7.4/src/packet.c:535
blocksize = <optimized out>
hmac_type = SSH_HMAC_SHA256
currentlen = 24744
hmac = 0x0
padstring = '\000' <repeats 31 times>
rc = -1
finallen = <optimized out>
payloadsize = 24744
compsize = <optimized out>
padding = <optimized out>
header = "\000\000\000\000"
__func__ = "packet_send2"
#5 0x00007fc243b83885 in packet_send (session=session@entry=0x10817d0) at /tmp/guix-build-libssh-0.7.4.drv-0/libssh-0.7.4/src/packet.c:604
No locals.
#6 0x00007fc243b74f4a in ssh_channel_send_eof (channel=0x10874c0) at /tmp/guix-build-libssh-0.7.4.drv-0/libssh-0.7.4/src/channels.c:1085
session = 0x10817d0
rc = -1
err = <optimized out>
__func__ = "ssh_channel_send_eof"
#7 0x00007fc243b75085 in ssh_channel_close (channel=0x10874c0) at /tmp/guix-build-libssh-0.7.4.drv-0/libssh-0.7.4/src/channels.c:1128
session = 0x10817d0
rc = 0
__func__ = "ssh_channel_close"
#8 0x00007fc243fdd59f in ptob_close (channel=0x14983a0) at channel-type.c:228
ch = 0x14983c0
#9 0x00007fc24ac3d785 in release_port (port=0x14983a0) at ports.c:158
pt = 0x14983a0
port = 0x14983a0
pt = <optimized out>
cur = 1
next = <optimized out>
#10 0x00007fc24ac40a0b in scm_close_port (port=0x14983a0) at ports.c:887
No locals.
#11 0x00007fc24ac7cc4d in vm_regular_engine (thread=0xffff21fa, vp=0xd24f30, registers=0x11c4b70, resume=-56837) at vm-engine.c:784

[...]

(gdb) frame 2
#2 0x00007fc243b9d4e1 in gzip_compress (session=session@entry=0x10817d0, source=source@entry=0x1082170, level=<optimized out>) at /tmp/guix-build-libssh-0.7.4.drv-0/libssh-0.7.4/src/gzip.c:85
85 in /tmp/guix-build-libssh-0.7.4.drv-0/libssh-0.7.4/src/gzip.c
(gdb) p *zout
$4 = {next_in = 0x1949b1e ":key inputs outputs #:allow-other-keys)\n", ' ' <repeats 21 times>, ";; Util-linux comes with a bunch of completion files for\n", ' ' <repeats 21 times>, ";; its own commands which are more sophisticated and\n "..., avail_in = 18681, total_in = 3741615, next_out = 0x7fc2472681e6 "", avail_out = 790, total_out = 1120084, msg = 0x0, state = 0x11c3420, zalloc = 0x7fc24588ab50 <zcalloc>,
zfree = 0x7fc24588ab60 <zcfree>, opaque = 0x0, data_type = 0, adler = 3865149116, reserved = 0}
(gdb) p out_buf
$5 = "\000\000\000\000\000\000\000\000\000\000\235\000\000\000z\377\000\000\200\000\000\000\374\376\000\000\000\200\205\001\000\000\000\200\237\034\000\000@?9\000\000\000\000\000\000\260\032\000\000\000\200\177\064b\004\067\000\000\000\000\270\000\000\000\000@;\224\003\000\000\002\000\000\000\000\000\000\000\000\000\200\253\002\000\000\000\300G9\000\000\200\f7\246\000\000\000\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000\000\230\004\000\000\000f\332\222F\004\000\000\000\064\367\032\067\006\000\000\000\000h\000\000\000\000\000\000\000\000\000\260", '\000' <repeats 13 times>, "v\000\000\000\000 ", '\000' <repeats 46 times>...
(gdb) p &out_buf
$6 = (unsigned char (*)[4092]) 0x7fc247267500
(gdb) p (char*)zout->next_out - (char*)&out_buf
$7 = 3302
(gdb) p $7 + zout->avail_out
$8 = 4092

[...]

(gdb) p *source
$18 = {data = 0x1946bc0 "^", used = 24744, allocated = 32768, pos = 0, secure = 0}
(gdb) p in_ptr
$19 = (void *) 0x1946bc0
(gdb) p (char*)zout->next_in - (char*) in_ptr
$20 = 12126
(gdb) p $20 + zout->avail_in
$21 = 30807

$21 here is above source->used, which suggestes the callee, ‘deflate’,
could end up reading 6K beyond the end of ‘source->data’.

Thoughts?

Ludo’.
L
L
Ludovic Courtès wrote on 21 May 2017 00:21
(name . Mark H Weaver)(address . mhw@netris.org)
87inkv41a2.fsf@gnu.org
ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (14 lines)
> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>> I’ve stopped the queue-runner while investigating. To be continued…
>>
>> For now, I would suggest downgrading the 'guix' on Hydra to be based on
>> guile-2.0 again, so that it can be building things while we sort this
>> out. What do you think?
>
> Sounds reasonable, let’s give it a try.
>
> I’ve added a ‘guile2.0-guix’ package in commit
> 82ce81ba06ea1babfdb0d1c447e972461c1d3018.

I’ve installed ‘guile2.0-guix’ on hydra.gnu.org, and it’s now running it
for guix-daemon and ‘guix publish’. Let’s see…

Ludo’.
L
L
Ludovic Courtès wrote on 5 Jun 2017 23:33
(name . Mark H Weaver)(address . mhw@netris.org)
87k24q86hk.fsf@gnu.org
Hi,

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (7 lines)
> I was able to reproduce it on hydra.gnu.org with:
>
> guix copy --to=hydra.gnunet.org /gnu/store/gi7r1v65zqhh8riqprq8nchfc9v9k156-guix-current
>
> which leads most of the time to SIGSEGV (I couldn’t get the SIGSEGV on
> my laptop):

Data from a recent run (libssh 0.7.5):

Toggle snippet (88 lines)
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f53f1b27700 (LWP 20388) 0x00007f53f013bac7 in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
2 Thread 0x7f53f59bcf00 (LWP 20384) 0x00007f53f013b97e in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
3 Thread 0x7f53f28cb700 (LWP 20387) 0x00007f53f4ff458f in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
4 Thread 0x7f53f38cd700 (LWP 20385) 0x00007f53f4ff458f in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
5 Thread 0x7f53ef2fb700 (LWP 20391) 0x00007f53f4ff7aed in read () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
6 Thread 0x7f53f30cc700 (LWP 20386) 0x00007f53f4ff458f in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
(gdb) bt
#0 0x00007f53f013bac7 in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#1 0x00007f53f013d78d in deflate () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#2 0x00007f53ee459531 in gzip_compress (session=session@entry=0x1f3a7d0, source=source@entry=0x1f2b370, level=<optimized out>) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:85
#3 0x00007f53ee459863 in compress_buffer (session=session@entry=0x1f3a7d0, buf=0x1f2b370) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:106
#4 0x00007f53ee43ef87 in packet_send2 (session=session@entry=0x1f3a7d0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:535
#5 0x00007f53ee43f8d5 in packet_send (session=session@entry=0x1f3a7d0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:604
#6 0x00007f53ee430f4a in ssh_channel_send_eof (channel=0x1f44960) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1085
#7 0x00007f53ee431085 in ssh_channel_close (channel=0x1f44960) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1128
#8 0x00007f53ee89959f in ptob_close (channel=0x2451020) at channel-type.c:228
#9 0x00007f53f54fb785 in release_port (port=0x2451020) at ports.c:158
#10 0x00007f53f54fea0b in scm_close_port (port=0x2451020) at ports.c:887
#11 0x00007f53f553ac4d in vm_regular_engine (thread=0xfff51068, vp=0x1cacf30, registers=0x2080010, resume=-716695) at vm-engine.c:784
#12 0x00007f53f553e2aa in scm_call_n (proc=proc@entry=0x2408e40, argv=argv@entry=0x0, nargs=nargs@entry=0) at vm.c:1257
#13 0x00007f53f54c17d9 in scm_call_0 (proc=proc@entry=0x2408e40) at eval.c:481
#14 0x00007f53f552d4e6 in catch (tag=tag@entry=0x404, thunk=0x2408e40, handler=0x2408e20, pre_unwind_handler=0x4) at throw.c:137
#15 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>, pre_unwind_handler=<optimized out>) at throw.c:254
#16 0x00007f53f552d9df in scm_c_catch (tag=tag@entry=0x404, body=body@entry=0x7f53f54fea60 <do_close>, body_data=<optimized out>, handler=<optimized out>, handler_data=handler_data@entry=0x0, pre_unwind_handler=pre_unwind_handler@entry=0x0, pre_unwind_handler_data=0x0) at throw.c:377
#17 0x00007f53f552d9ee in scm_internal_catch (tag=tag@entry=0x404, body=body@entry=0x7f53f54fea60 <do_close>, body_data=<optimized out>, handler=<optimized out>, handler_data=handler_data@entry=0x0) at throw.c:386
#18 0x00007f53f54fa0a4 in finalize_port (ptr=<optimized out>, data=<optimized out>) at ports.c:701
#19 0x00007f53f521a70f in GC_invoke_finalizers () from /gnu/store/j4p7cnxb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1
#20 0x00007f53f54cb099 in scm_run_finalizers () at finalizers.c:387
#21 0x00007f53f54cb0f5 in finalization_thread_proc (unused=<optimized out>) at finalizers.c:223
#22 0x00007f53f54bbcba in c_body (d=0x7f53f1b26e60) at continuations.c:422
#23 0x00007f53f553ac4d in vm_regular_engine (thread=0xfff51068, vp=0x1cacf30, registers=0x2080010, resume=-716695) at vm-engine.c:784
#24 0x00007f53f553e2aa in scm_call_n (proc=proc@entry=0x1c04460, argv=argv@entry=0x0, nargs=nargs@entry=0) at vm.c:1257
#25 0x00007f53f54c17d9 in scm_call_0 (proc=proc@entry=0x1c04460) at eval.c:481
#26 0x00007f53f552d4e6 in catch (tag=tag@entry=0x404, thunk=0x1c04460, handler=0x1c043e0, pre_unwind_handler=0x1c04360) at throw.c:137
#27 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>, pre_unwind_handler=<optimized out>) at throw.c:254
#28 0x00007f53f552d9df in scm_c_catch (tag=tag@entry=0x404, body=body@entry=0x7f53f54bbcb0 <c_body>, body_data=body_data@entry=0x7f53f1b26e60, handler=handler@entry=0x7f53f54bbf40 <c_handler>, handler_data=handler_data@entry=0x7f53f1b26e60, pre_unwind_handler=pre_unwind_handler@entry=0x7f53f54bbda0 <pre_unwind_handler>, pre_unwind_handler_data=0x1bd6c80) at throw.c:377
#29 0x00007f53f54bc2a0 in scm_i_with_continuation_barrier (body=body@entry=0x7f53f54bbcb0 <c_body>, body_data=body_data@entry=0x7f53f1b26e60, handler=handler@entry=0x7f53f54bbf40 <c_handler>, handler_data=handler_data@entry=0x7f53f1b26e60, pre_unwind_handler=pre_unwind_handler@entry=0x7f53f54bbda0 <pre_unwind_handler>, pre_unwind_handler_data=0x1bd6c80) at continuations.c:360
#30 0x00007f53f54bc335 in scm_c_with_continuation_barrier (func=<optimized out>, data=<optimized out>) at continuations.c:456
#31 0x00007f53f552c10c in with_guile (base=0x7f53f1b26ec0, data=0x7f53f1b26ef0) at threads.c:660
#32 0x00007f53f5221732 in GC_call_with_stack_base () from /gnu/store/j4p7cnxb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1
#33 0x00007f53f552c498 in scm_i_with_guile (dynamic_state=<optimized out>, data=<optimized out>, func=<optimized out>) at threads.c:703
#34 scm_with_guile (func=<optimized out>, data=<optimized out>) at threads.c:709
#35 0x00007f53f4fee454 in start_thread () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#36 0x00007f53f39b67bf in clone () from /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f53f59bcf00 (LWP 20384))]
#0 0x00007f53f013b97e in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
(gdb) bt
#0 0x00007f53f013b97e in deflate_fast () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#1 0x00007f53f013d78d in deflate () from /gnu/store/jwkcd7siv6fcyl0qsg607bg9c8ap0gqr-zlib-1.2.11/lib/libz.so.1
#2 0x00007f53ee459531 in gzip_compress (session=session@entry=0x1f3a7d0, source=source@entry=0x1f2b370, level=<optimized out>) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:85
#3 0x00007f53ee459863 in compress_buffer (session=session@entry=0x1f3a7d0, buf=0x1f2b370) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/gzip.c:106
#4 0x00007f53ee43ef87 in packet_send2 (session=session@entry=0x1f3a7d0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:535
#5 0x00007f53ee43f8d5 in packet_send (session=session@entry=0x1f3a7d0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/packet.c:604
#6 0x00007f53ee42f628 in channel_write_common (channel=0x1f3cc20, data=data@entry=0x25af020, len=len@entry=65536, is_stderr=is_stderr@entry=0) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1321
#7 0x00007f53ee431127 in ssh_channel_write (channel=<optimized out>, data=data@entry=0x25af020, len=len@entry=65536) at /tmp/guix-build-libssh-0.7.5.drv-0/libssh-0.7.5/src/channels.c:1370
#8 0x00007f53ee8995ee in write_to_channel_port (channel=0x24ce5a0, src=<optimized out>, start=<optimized out>, count=65536) at channel-type.c:180
#9 0x00007f53f54fe72c in scm_i_write_bytes (port=0x24ce5a0, src=0x25af000, start=0, count=65536) at ports.c:2842
#10 0x00007f53f5505de4 in scm_put_bytevector (port=0x24ce5a0, bv=0x25af000, start=0x2, count=<optimized out>) at r6rs-ports.c:614
#11 0x00007f53f553ac4d in vm_regular_engine (thread=0x15c0, vp=0x1bdaf30, registers=0xc665f, resume=5) at vm-engine.c:784
#12 0x00007f53f553e2aa in scm_call_n (proc=0x7f53f59b4030, argv=argv@entry=0x7fffa1436c28, nargs=nargs@entry=1) at vm.c:1257
#13 0x00007f53f54c28d7 in scm_primitive_eval (exp=0x1cd7dd0) at eval.c:662
#14 0x00007f53f54de6eb in scm_primitive_load (filename=<optimized out>) at load.c:123
#15 0x00007f53f553ac4d in vm_regular_engine (thread=0x15c0, vp=0x1bdaf30, registers=0xc665f, resume=5) at vm-engine.c:784
#16 0x00007f53f553e2aa in scm_call_n (proc=0x7f53f59b4030, argv=argv@entry=0x7fffa1436e78, nargs=nargs@entry=1) at vm.c:1257
#17 0x00007f53f54c28d7 in scm_primitive_eval (exp=exp@entry=0x1cbc4e0) at eval.c:662
#18 0x00007f53f54c2933 in scm_eval (exp=0x1cbc4e0, module_or_state=module_or_state@entry=0x1c9f140) at eval.c:696
#19 0x00007f53f550d9d0 in scm_shell (argc=6, argv=0x7fffa14374d8) at script.c:454
#20 0x00007f53f54d94ed in invoke_main_func (body_data=0x7fffa1437380) at init.c:340
#21 0x00007f53f54bbcba in c_body (d=0x7fffa14372c0) at continuations.c:422
#22 0x00007f53f553ac4d in vm_regular_engine (thread=0x15c0, vp=0x1bdaf30, registers=0xc665f, resume=5) at vm-engine.c:784
#23 0x00007f53f553e2aa in scm_call_n (proc=proc@entry=0x1ba97a0, argv=argv@entry=0x0, nargs=nargs@entry=0) at vm.c:1257
#24 0x00007f53f54c17d9 in scm_call_0 (proc=proc@entry=0x1ba97a0) at eval.c:481
#25 0x00007f53f552d4e6 in catch (tag=tag@entry=0x404, thunk=0x1ba97a0, handler=0x1ba9780, pre_unwind_handler=0x1ba9720) at throw.c:137
#26 0x00007f53f552d825 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>, pre_unwind_handler=<optimized out>) at throw.c:254
#27 0x00007f53f552d9df in scm_c_catch (tag=tag@entry=0x404, body=body@entry=0x7f53f54bbcb0 <c_body>, body_data=body_data@entry=0x7fffa14372c0, handler=handler@entry=0x7f53f54bbf40 <c_handler>, handler_data=handler_data@entry=0x7fffa14372c0, pre_unwind_handler=pre_unwind_handler@entry=0x7f53f54bbda0 <pre_unwind_handler>, pre_unwind_handler_data=0x1bd6c80) at throw.c:377
#28 0x00007f53f54bc2a0 in scm_i_with_continuation_barrier (body=body@entry=0x7f53f54bbcb0 <c_body>, body_data=body_data@entry=0x7fffa14372c0, handler=handler@entry=0x7f53f54bbf40 <c_handler>, handler_data=handler_data@entry=0x7fffa14372c0, pre_unwind_handler=pre_unwind_handler@entry=0x7f53f54bbda0 <pre_unwind_handler>, pre_unwind_handler_data=0x1bd6c80) at continuations.c:360
#29 0x00007f53f54bc335 in scm_c_with_continuation_barrier (func=<optimized out>, data=<optimized out>) at continuations.c:456
#30 0x00007f53f552c10c in with_guile (base=0x7fffa1437320, data=0x7fffa1437350) at threads.c:660
#31 0x00007f53f5221732 in GC_call_with_stack_base () from /gnu/store/j4p7cnxb6byf1wra3yz6pk8av4g5mwkl-libgc-7.6.0/lib/libgc.so.1
#32 0x00007f53f552c498 in scm_i_with_guile (dynamic_state=<optimized out>, data=data@entry=0x7fffa1437350, func=func@entry=0x7f53f54d94d0 <invoke_main_func>) at threads.c:703
#33 scm_with_guile (func=func@entry=0x7f53f54d94d0 <invoke_main_func>, data=data@entry=0x7fffa1437380) at threads.c:709
#34 0x00007f53f54d9682 in scm_boot_guile (argc=argc@entry=6, argv=argv@entry=0x7fffa14374d8, main_func=main_func@entry=0x400ca0 <inner_main>, closure=closure@entry=0x0) at init.c:323
#35 0x0000000000400b40 in main (argc=6, argv=0x7fffa14374d8) at guile.c:101

So we have the finalization thread closing a channel of session
0x1f3a7d0 (which causes a write on the channel), and the main thread
writing to a channel of that same session.

If we look at src/gzip.c:95 in libssh, it starts like this:

Toggle snippet (5 lines)
static ssh_buffer gzip_compress(ssh_session session,ssh_buffer source,int level){
z_stream *zout = session->current_crypto->compress_out_ctx;
void *in_ptr = buffer_get_rest(source);

AIUI, that means there’s one output compression buffer per session, and
it’s not thread-safe (in Guile 2.2 finalizers are called from a separate
thread.)

I think the fix, in Guile-SSH, is to associate each libssh object
(session, channel, etc.) with a mutex, and to protect all uses of the
libssh object by that mutex.

(Alternately we could change libssh to move its compression buffers to
channels instead of sessions, but that may not be enough.)

WDYT, Artyom? Would you be able to work on this? :-)

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 13 Jun 2017 14:29
control message for bug #26976
(address . control@debbugs.gnu.org)
87efuo2hri.fsf@gnu.org
retitle 26976 Guile-SSH channel finalization leads to segfaults on Guile 2.2
L
L
Ludovic Courtès wrote on 13 Jun 2017 23:32
Re: bug#26976: On Hydra, offload crashes while trying to build linux-libre source
(name . Mark H Weaver)(address . mhw@netris.org)
87shj3tvzd.fsf@gnu.org
Hello,

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (8 lines)
> AIUI, that means there’s one output compression buffer per session, and
> it’s not thread-safe (in Guile 2.2 finalizers are called from a separate
> thread.)
>
> I think the fix, in Guile-SSH, is to associate each libssh object
> (session, channel, etc.) with a mutex, and to protect all uses of the
> libssh object by that mutex.

I’ve pushed a workaround that seems to work (‘guix copy’ commands that
previously segfaulted on hydra.gnu.org no longer do):


The idea is simply to explicitly close all the channel ports. That way,
when their finalizer gets called, it doesn’t invoke libssh code.

(guix build offload) and (guix ssh) already explicitly close all the
channel ports they open, so I found that the culprit is ‘node-eval’ in
Guile-SSH. The patch changes ‘node-eval’ to explicitly close the RREPL
channel upon completion. (BTW, performance-wise, it may be best to
avoid opening a new channel every time ‘node-eval’ is called.)

That may be good enough for Guix, but of course that’s a workaround and
not a proper fix.

I’ll do some more testing and then maybe try switching hydra.gnu.org to
Guile 2.2 again.

Thanks,
Ludo’.
M
M
Mark H Weaver wrote on 14 Jun 2017 08:58
(name . Ludovic Courtès)(address . ludo@gnu.org)
87k24fys10.fsf@netris.org
ludo@gnu.org (Ludovic Courtès) writes:

Toggle quote (19 lines)
> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> ludo@gnu.org (Ludovic Courtès) writes:
>>>> I’ve stopped the queue-runner while investigating. To be continued…
>>>
>>> For now, I would suggest downgrading the 'guix' on Hydra to be based on
>>> guile-2.0 again, so that it can be building things while we sort this
>>> out. What do you think?
>>
>> Sounds reasonable, let’s give it a try.
>>
>> I’ve added a ‘guile2.0-guix’ package in commit
>> 82ce81ba06ea1babfdb0d1c447e972461c1d3018.
>
> I’ve installed ‘guile2.0-guix’ on hydra.gnu.org, and it’s now running it
> for guix-daemon and ‘guix publish’. Let’s see…

We're now seeing many builds aborting with errors like this:

guix offload: error: failed to connect to `#<input-output: channel (open) 269a9f0>': Protocol error

For example:


Mark
M
M
Mark H Weaver wrote on 14 Jun 2017 09:15
(name . Ludovic Courtès)(address . ludo@gnu.org)
87fuf3yr9d.fsf@netris.org
Mark H Weaver <mhw@netris.org> writes:

Toggle quote (9 lines)
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> I’ve installed ‘guile2.0-guix’ on hydra.gnu.org, and it’s now running it
>> for guix-daemon and ‘guix publish’. Let’s see…
>
> We're now seeing many builds aborting with errors like this:
>
> guix offload: error: failed to connect to `#<input-output: channel (open) 269a9f0>': Protocol error

Nevermind. It turned out to be because two of the armhf build slaves
(both of the Novenas that I host) rebooted and didn't have guix-daemon
running.

Mark
L
L
Ludovic Courtès wrote on 14 Jun 2017 22:10
(name . Mark H Weaver)(address . mhw@netris.org)
87r2yms52k.fsf@gnu.org
(+ Cc: guix-sysadmin)

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (20 lines)
> I’ve pushed a workaround that seems to work (‘guix copy’ commands that
> previously segfaulted on hydra.gnu.org no longer do):
>
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8e469b67f95cfe5b95405b503b8ee315fdf8ce66
>
> The idea is simply to explicitly close all the channel ports. That way,
> when their finalizer gets called, it doesn’t invoke libssh code.
>
> (guix build offload) and (guix ssh) already explicitly close all the
> channel ports they open, so I found that the culprit is ‘node-eval’ in
> Guile-SSH. The patch changes ‘node-eval’ to explicitly close the RREPL
> channel upon completion. (BTW, performance-wise, it may be best to
> avoid opening a new channel every time ‘node-eval’ is called.)
>
> That may be good enough for Guix, but of course that’s a workaround and
> not a proper fix.
>
> I’ll do some more testing and then maybe try switching hydra.gnu.org to
> Guile 2.2 again.

After some more testing, I’ve upgraded hydra.gnu.org to
/gnu/store/4p56wc3lmr177ffw0zkjdwcmm3cb18wx-guix-0.13.0-2.de9d8f0 (from
/gnu/store/cxmj38x6rh0ykq3d5dlqbxr5h1zgiiaf-guile2.0-guix-0.13.0-1.a6d728b).
So far so good, but let’s keep an eye on the ongoing builds. We can
always roll back if it turns out my optimism was misplaced.

Ludo’.
L
L
Ludovic Courtès wrote on 27 Jul 2017 12:14
(name . Mark H Weaver)(address . mhw@netris.org)
87eft25gql.fsf@gnu.org
Hello,

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (5 lines)
> I’ve pushed a workaround that seems to work (‘guix copy’ commands that
> previously segfaulted on hydra.gnu.org no longer do):
>
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8e469b67f95cfe5b95405b503b8ee315fdf8ce66

The workaround has proved to work well for ‘guix offload’ and ‘guix
copy’ so I’m closing this bug on our side.

In Guile-SSH though, I think there’s still the potential for finalizers
getting called in a separate thread and triggering the issue we’ve seen
here. Am I right, Artyom?

If I’m not mistaken, that calls for a fix as described in
protecting libssh objects with mutexes.

Ludo’.
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 26976@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 26976
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch