[Cuirass] ‘remote-server’ wrecks havoc when misbehaving clients connect

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 4 Dec 2023 22:46
(address . bug-guix@gnu.org)
871qc1myj6.fsf@inria.fr
The ‘cuirass remote-server’ process goes awry when misbehaved clients
connect; specifically, some of its fibers crash, leaving it running but
inactive:

Toggle snippet (48 lines)
2023-12-04 10:13:14 periodic update: 0 resumable, 0 failed builds
2023-12-04 10:13:28 error: Connection reset by peer when replying to xx.xx.xx.254.
2023-12-04 10:13:42 error: invalid log received from xx.xx.xx.254
2023-12-04 10:13:42 error: EOF while receiving log from xx.xx.xx.254
2023-12-04 10:13:48 Uncaught exception in task:
2023-12-04 10:13:48 In fibers.scm:
2023-12-04 10:13:48 186:20 8 (_)
2023-12-04 10:13:48 145:21 7 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48 1752:10 6 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
2023-12-04 10:13:48 695:11 5 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48 1747:15 4 (with-exception-handler #<procedure 7f982c2abd50 at ice-9/boot-9.scm:1831:7 (exn)> _ #:unwind? _ #:unwind-for-type _)
2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
2023-12-04 10:13:48 437:23 3 (serve-build-requests _ #<<channel> getq: #<atomic-box 7f982e764390 value: (() #(#<atomic-box 7f98283753a0 value: W> #<procedure resume (thunk)>))> getq-gc-co
unter: #<atomic-box 7f982e764320 value: 16> putq: #<atomic-box 7f982e764240 value: (())> putq-gc-counter: #<atomic-box 7f982e764230 value: 31>>)
2023-12-04 10:13:48 In cuirass/remote.scm:
2023-12-04 10:13:48 466:6 2 (receive-message _ #:router? _)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48 1685:16 1 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48 1685:16 0 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:13:48 Throw to key `match-error' with args `("match" "no matching pattern" (#<zmq-message 7f981881d9a0> #<zmq-message 7f981881d900> #<zmq-message 7f981881d860>))'.
2023-12-04 10:13:48 Uncaught exception in task:
2023-12-04 10:13:48 In fibers.scm:
2023-12-04 10:13:48 172:8 2 (_)
2023-12-04 10:13:48 In cuirass/notification.scm:
2023-12-04 10:13:48 162:15 1 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48 1685:16 0 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:13:48 Throw to key `psql-query-error' with args `(#f #f "no connection to the server\n")'.
2023-12-04 10:14:06 0 items queued for eventual download
2023-12-04 10:14:07 Uncaught exception in task:
2023-12-04 10:14:07 In fibers.scm:
2023-12-04 10:14:07 172:8 2 (_)
2023-12-04 10:14:07 In cuirass/scripts/remote-server.scm:
2023-12-04 10:14:07 430:10 1 (_)
2023-12-04 10:14:07 In ice-9/boot-9.scm:
2023-12-04 10:14:07 1685:16 0 (raise-exception _ #:continuable? _)
2023-12-04 10:14:07 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:14:07 Throw to key `psql-query-error' with args `(#f #f "no connection to the server\n")'.
2023-12-04 10:14:30 error: invalid log received from xx.xx.xx.254
2023-12-04 10:14:30 error: invalid log received from xx.xx.xx.254
2023-12-04 10:14:35 error: invalid log received from xx.xx.xx.254

Ludo’.
M
M
Maxim Cournoyer wrote on 12 Dec 2023 18:03
Re: bug#67629: [Cuirass] ‘remote-server ’ wrecks havoc when misbehaving clients connect
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 67629@debbugs.gnu.org)
875y13bbew.fsf@gmail.com
Hi,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

Toggle quote (4 lines)
> The ‘cuirass remote-server’ process goes awry when misbehaved clients
> connect; specifically, some of its fibers crash, leaving it running but
> inactive:

I guess what I reported in bug#67633 is a symptom of the above problem,
perhaps?

--
Thanks,
Maxim
L
L
Ludovic Courtès wrote on 3 Apr 11:09 +0200
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 67629@debbugs.gnu.org)
87v84yzuaw.fsf@gnu.org
Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (9 lines)
> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> The ‘cuirass remote-server’ process goes awry when misbehaved clients
>> connect; specifically, some of its fibers crash, leaving it running but
>> inactive:
>
> I guess what I reported in bug#67633 is a symptom of the above problem,
> perhaps?

It could be yes, if ‘remote-server’ crashed right at that moment.

Ludo’.
L
L
Ludovic Courtès wrote on 3 Apr 11:13 +0200
(address . 67629-done@debbugs.gnu.org)
87o7aqzu55.fsf@gnu.org
Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

Toggle quote (29 lines)
> The ‘cuirass remote-server’ process goes awry when misbehaved clients
> connect; specifically, some of its fibers crash, leaving it running but
> inactive:
>
> 2023-12-04 10:13:14 periodic update: 0 resumable, 0 failed builds
> 2023-12-04 10:13:28 error: Connection reset by peer when replying to xx.xx.xx.254.
> 2023-12-04 10:13:42 error: invalid log received from xx.xx.xx.254
> 2023-12-04 10:13:42 error: EOF while receiving log from xx.xx.xx.254
> 2023-12-04 10:13:48 Uncaught exception in task:
> 2023-12-04 10:13:48 In fibers.scm:
> 2023-12-04 10:13:48 186:20 8 (_)
> 2023-12-04 10:13:48 145:21 7 (_)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48 1752:10 6 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
> 2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
> 2023-12-04 10:13:48 695:11 5 (_)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48 1747:15 4 (with-exception-handler #<procedure 7f982c2abd50 at ice-9/boot-9.scm:1831:7 (exn)> _ #:unwind? _ #:unwind-for-type _)
> 2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
> 2023-12-04 10:13:48 437:23 3 (serve-build-requests _ #<<channel> getq: #<atomic-box 7f982e764390 value: (() #(#<atomic-box 7f98283753a0 value: W> #<procedure resume (thunk)>))> getq-gc-co
> unter: #<atomic-box 7f982e764320 value: 16> putq: #<atomic-box 7f982e764240 value: (())> putq-gc-counter: #<atomic-box 7f982e764230 value: 31>>)
> 2023-12-04 10:13:48 In cuirass/remote.scm:
> 2023-12-04 10:13:48 466:6 2 (receive-message _ #:router? _)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48 1685:16 1 (raise-exception _ #:continuable? _)
> 2023-12-04 10:13:48 1685:16 0 (raise-exception _ #:continuable? _)
> 2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> 2023-12-04 10:13:48 Throw to key `match-error' with args `("match" "no matching pattern" (#<zmq-message 7f981881d9a0> #<zmq-message 7f981881d900> #<zmq-message 7f981881d860>))'.

With commit 9a1452ee021c9f773424961cfeef47ca0b7c5c5a, ‘cuirass
remote-server’ handles this situation gracefully.

I’m still curious about who’s sending us garbage: it has to be valid
zmq, just with the wrong number of parts. From the logs, it seemed like
a port scanner or something on the MDC network was sending garbage on
all the open ports, but that would need to be a sophisticated one.

Ludo’.
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 67629@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 67629
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch