Recurring timeout error while using offload

  • Open
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
important
M
M
Maxim Cournoyer wrote on 1 Sep 2021 07:35
Recurring timeout eror while using offload
(name . bug-guix)(address . bug-guix@gnu.org)
87k0k0eu40.fsf@gmail.com
Hello Guix,

I'm encountered timeout error that seem to be caused by a particular
payload rather than networking issues. With a specific payload (a
bunch of rust package to build on the core-updates-frozen branch), it
seems to be triggered every time.

Here's the derivations to be built:

Toggle snippet (22 lines)
$ ./pre-inst-env guix build rust -n
The following derivations would be built:
/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv
/gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv
/gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv
/gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv
/gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv
/gnu/store/0z13dkldzsxf87f4kgi3b2545bm39nnh-rustc-1.51.0-src.tar.xz.drv
/gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv
/gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv
/gnu/store/izgl0l21r5lwd3d3glxqwmigl0315b26-rustc-1.49.0-src.tar.xz.drv
/gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv
/gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv
/gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv
/gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv
/gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv
/gnu/store/m7z7jdd2my4p9p5mjwa0cna4fzhd1v8j-rust-1.43.0.drv
/gnu/store/pw4ynvgbnk1j6fxlh37a9wd8fyi7bjnc-rustc-1.50.0-src.tar.xz.drv
/gnu/store/g9izbhyhc6jcalxgbxc6aas3aa66h2aa-rustc-1.52.1-src.tar.xz.drv
/gnu/store/jbr3f7360x3ww5rn6nnb803yxq7ppck3-rustc-1.53.0-src.tar.xz.drv

And here is what the error looks like:

Toggle snippet (29 lines)
rustc-1.52.1-src/src/llvm-project/clang/test/Misc/message-length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Driver/fmessage-length.c
rustc-1.52.1-src/src/llvm-project/compiler-rt/test/hwasan/TestCases/stack-history-length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Sema/vector_swizzle_length.c
rustc-1.52.1-src/src/llvm-project/clang/test/Sema/array-bounds-ptr-arith.c
rustc-1.52.1-src/src/llvm-project/clang/test/Analysis/ptr-arith.c
rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/engine/tb_asnmth.c
rustc-1.52.1-src/src/llvm-project/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_cnth.c
rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/pem/pem_oth.c
rustc-1.52.1-src/vendor/libgit2-sys/libgit2/src/transports/auth.c
guix offload: error: failed to connect to '127.0.0.1': Timeout connecting to 127.0.0.1
cannot build derivation `/gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed

real 2m50.854s
user 0m11.302s
sys 0m0.661s

It occurs shortly after heavy scrolling starts.

I've verified that it's not my network link at least on my local
network; any ideas what I should look into for debugging? Perhaps the
logs/strace of the remote SSH server?

Thanks,

Maxim
M
M
Maxim Cournoyer wrote on 1 Sep 2021 18:57
(address . 50312@debbugs.gnu.org)
87pmtscjy5.fsf@gmail.com
Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

Toggle quote (40 lines)
> And here is what the error looks like:
>
> rustc-1.52.1-src/src/llvm-project/clang/test/Misc/message-length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Driver/fmessage-length.c
> rustc-1.52.1-src/src/llvm-project/compiler-rt/test/hwasan/TestCases/stack-history-length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Sema/vector_swizzle_length.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Sema/array-bounds-ptr-arith.c
> rustc-1.52.1-src/src/llvm-project/clang/test/Analysis/ptr-arith.c
> rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/engine/tb_asnmth.c
> rustc-1.52.1-src/src/llvm-project/clang/test/CodeGen/aarch64-sve-intrinsics/acle_sve_cnth.c
> rustc-1.52.1-src/vendor/openssl-src/openssl/crypto/pem/pem_oth.c
> rustc-1.52.1-src/vendor/libgit2-sys/libgit2/src/transports/auth.c
> guix offload: error: failed to connect to '127.0.0.1': Timeout connecting to 127.0.0.1
> cannot build derivation `/gnu/store/4i5xm0wcasilh55f9gfcrzp2sjdkd507-rust-1.44.1.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/h3mxv9zlx8q3x0m0s5gvin2n1ndliivw-rust-1.45.2.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/7ib14fx50651av5bq5g195cqf2azsvhc-rust-1.46.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/a9qwa9cl90xrm2dpa3hnd3ak54nqss8n-rust-1.47.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/n0dpka76b9q0w4r2k912qq5p2a57bz6c-rust-1.48.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/h5959ysczr862ldwh6f85h10895c4csp-rust-1.49.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/q9mbicnmrpd1c98vmi5y0gyjnpr3gqbn-rust-1.50.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/72572pjayh2j5frr187i7axjzfd01kp6-rust-1.51.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/9v5yl0fg4dj1n82pkznksasmdjgi541w-rust-1.52.1.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/x1v25nclqps677j1g5dvyck2njaxy0ya-rust-1.53.0.drv': 1 dependencies couldn't be built
> cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
> guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed
>
> real 2m50.854s
> user 0m11.302s
> sys 0m0.661s
>
> It occurs shortly after heavy scrolling starts.
>
> I've verified that it's not my network link at least on my local
> network; any ideas what I should look into for debugging? Perhaps the
> logs/strace of the remote SSH server?
>
> Thanks,
>
> Maxim

Now that I've manually built them by small batches of 4 (that would not
fail), I reached the last item rust-1.54, which failed like so:

Toggle snippet (35 lines)
rustc-1.54.0-src/vendor/lzma-sys/xz-5.2/tests/files/good-0cat-empty.xz
source is at 'rustc-1.54.0-src'
Backtrace:
12 (primitive-load "/gnu/store/j2hl7fmsbh7jmixpy8dy07finqp…")
In ice-9/eval.scm:
619:8 11 (_ #(#(#<directory (guile-user) 7ffff5fdbc80> "rus…") #))
In ice-9/boot-9.scm:
142:2 10 (dynamic-wind _ _ #<procedure 7ffff5fc8e00 at ice-9/eva…>)
In system/base/compile.scm:
352:28 9 (compile _ #:from _ #:to _ #:env _ #:optimization-level …)
265:44 8 (_ _ _)
265:44 7 (_ _ _)
265:44 6 (_ _ _)
261:27 5 (_ _ _)
In ice-9/boot-9.scm:
2835:4 4 (save-module-excursion _)
In language/bytecode/spec.scm:
35:19 3 (_)
In ice-9/boot-9.scm:
260:13 2 (for-each #<procedure delete-file-recursively (dir #:k…> …)
In guix/build/utils.scm:
446:23 1 (delete-file-recursively "vendor/jemalloc-sys/jemalloc" …)
In unknown file:
0 (lstat "vendor/jemalloc-sys/jemalloc")

ERROR: In procedure lstat:
In procedure lstat: No such file or directory: "vendor/jemalloc-sys/jemalloc"
note: keeping build directory `/tmp/guix-build-rustc-1.54.0-src.tar.xz.drv-0'
builder for `/gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv' failed with exit code 1
build of /gnu/store/bmc82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv failed
View build log at '/var/log/guix/drvs/bm/c82kpr11gpk1xy7d3l6hy37cs58i7d-rustc-1.54.0-src.tar.xz.drv'.
cannot build derivation `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/nv3l1lyx4x33gx80pxk8033g01b6bv0m-rust-1.54.0.drv' failed

Perhaps this failure was badly handled in the parallel build an reported
as a timeout? I'm just guessing, but reporting it as it may be at
cause.

Thanks,

Maxim
L
L
Ludovic Courtès wrote on 13 Sep 2021 14:05
control message for bug #50312
(address . control@debbugs.gnu.org)
87y2808yvf.fsf@gnu.org
severity 50312 important
quit
M
M
Maxim Cournoyer wrote on 14 Apr 2023 15:17
(address . control@debbugs.gnu.org)
87bkjqk3gm.fsf@gmail.com
retitle 50312 Recurring timeout error while using offload
quit
?