[Cuirass] ‘remote-server’ wrecks havoc when misbehaving clients connect

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 4 Dec 2023 22:46
(address . bug-guix@gnu.org)
871qc1myj6.fsf@inria.fr
The ‘cuirass remote-server’ process goes awry when misbehaved clients
connect; specifically, some of its fibers crash, leaving it running but
inactive:

Toggle snippet (48 lines)
2023-12-04 10:13:14 periodic update: 0 resumable, 0 failed builds
2023-12-04 10:13:28 error: Connection reset by peer when replying to xx.xx.xx.254.
2023-12-04 10:13:42 error: invalid log received from xx.xx.xx.254
2023-12-04 10:13:42 error: EOF while receiving log from xx.xx.xx.254
2023-12-04 10:13:48 Uncaught exception in task:
2023-12-04 10:13:48 In fibers.scm:
2023-12-04 10:13:48 186:20 8 (_)
2023-12-04 10:13:48 145:21 7 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48 1752:10 6 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
2023-12-04 10:13:48 695:11 5 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48 1747:15 4 (with-exception-handler #<procedure 7f982c2abd50 at ice-9/boot-9.scm:1831:7 (exn)> _ #:unwind? _ #:unwind-for-type _)
2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
2023-12-04 10:13:48 437:23 3 (serve-build-requests _ #<<channel> getq: #<atomic-box 7f982e764390 value: (() #(#<atomic-box 7f98283753a0 value: W> #<procedure resume (thunk)>))> getq-gc-co
unter: #<atomic-box 7f982e764320 value: 16> putq: #<atomic-box 7f982e764240 value: (())> putq-gc-counter: #<atomic-box 7f982e764230 value: 31>>)
2023-12-04 10:13:48 In cuirass/remote.scm:
2023-12-04 10:13:48 466:6 2 (receive-message _ #:router? _)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48 1685:16 1 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48 1685:16 0 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:13:48 Throw to key `match-error' with args `("match" "no matching pattern" (#<zmq-message 7f981881d9a0> #<zmq-message 7f981881d900> #<zmq-message 7f981881d860>))'.
2023-12-04 10:13:48 Uncaught exception in task:
2023-12-04 10:13:48 In fibers.scm:
2023-12-04 10:13:48 172:8 2 (_)
2023-12-04 10:13:48 In cuirass/notification.scm:
2023-12-04 10:13:48 162:15 1 (_)
2023-12-04 10:13:48 In ice-9/boot-9.scm:
2023-12-04 10:13:48 1685:16 0 (raise-exception _ #:continuable? _)
2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:13:48 Throw to key `psql-query-error' with args `(#f #f "no connection to the server\n")'.
2023-12-04 10:14:06 0 items queued for eventual download
2023-12-04 10:14:07 Uncaught exception in task:
2023-12-04 10:14:07 In fibers.scm:
2023-12-04 10:14:07 172:8 2 (_)
2023-12-04 10:14:07 In cuirass/scripts/remote-server.scm:
2023-12-04 10:14:07 430:10 1 (_)
2023-12-04 10:14:07 In ice-9/boot-9.scm:
2023-12-04 10:14:07 1685:16 0 (raise-exception _ #:continuable? _)
2023-12-04 10:14:07 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
2023-12-04 10:14:07 Throw to key `psql-query-error' with args `(#f #f "no connection to the server\n")'.
2023-12-04 10:14:30 error: invalid log received from xx.xx.xx.254
2023-12-04 10:14:30 error: invalid log received from xx.xx.xx.254
2023-12-04 10:14:35 error: invalid log received from xx.xx.xx.254

Ludo’.
M
M
Maxim Cournoyer wrote on 12 Dec 2023 18:03
Re: bug#67629: [Cuirass] ‘remote-server ’ wrecks havoc when misbehaving clients connect
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 67629@debbugs.gnu.org)
875y13bbew.fsf@gmail.com
Hi,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

Toggle quote (4 lines)
> The ‘cuirass remote-server’ process goes awry when misbehaved clients
> connect; specifically, some of its fibers crash, leaving it running but
> inactive:

I guess what I reported in bug#67633 is a symptom of the above problem,
perhaps?

--
Thanks,
Maxim
L
L
Ludovic Courtès wrote on 3 Apr 11:09 +0200
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 67629@debbugs.gnu.org)
87v84yzuaw.fsf@gnu.org
Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (9 lines)
> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> The ‘cuirass remote-server’ process goes awry when misbehaved clients
>> connect; specifically, some of its fibers crash, leaving it running but
>> inactive:
>
> I guess what I reported in bug#67633 is a symptom of the above problem,
> perhaps?

It could be yes, if ‘remote-server’ crashed right at that moment.

Ludo’.
L
L
Ludovic Courtès wrote on 3 Apr 11:13 +0200
(address . 67629-done@debbugs.gnu.org)
87o7aqzu55.fsf@gnu.org
Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

Toggle quote (29 lines)
> The ‘cuirass remote-server’ process goes awry when misbehaved clients
> connect; specifically, some of its fibers crash, leaving it running but
> inactive:
>
> 2023-12-04 10:13:14 periodic update: 0 resumable, 0 failed builds
> 2023-12-04 10:13:28 error: Connection reset by peer when replying to xx.xx.xx.254.
> 2023-12-04 10:13:42 error: invalid log received from xx.xx.xx.254
> 2023-12-04 10:13:42 error: EOF while receiving log from xx.xx.xx.254
> 2023-12-04 10:13:48 Uncaught exception in task:
> 2023-12-04 10:13:48 In fibers.scm:
> 2023-12-04 10:13:48 186:20 8 (_)
> 2023-12-04 10:13:48 145:21 7 (_)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48 1752:10 6 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
> 2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
> 2023-12-04 10:13:48 695:11 5 (_)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48 1747:15 4 (with-exception-handler #<procedure 7f982c2abd50 at ice-9/boot-9.scm:1831:7 (exn)> _ #:unwind? _ #:unwind-for-type _)
> 2023-12-04 10:13:48 In cuirass/scripts/remote-server.scm:
> 2023-12-04 10:13:48 437:23 3 (serve-build-requests _ #<<channel> getq: #<atomic-box 7f982e764390 value: (() #(#<atomic-box 7f98283753a0 value: W> #<procedure resume (thunk)>))> getq-gc-co
> unter: #<atomic-box 7f982e764320 value: 16> putq: #<atomic-box 7f982e764240 value: (())> putq-gc-counter: #<atomic-box 7f982e764230 value: 31>>)
> 2023-12-04 10:13:48 In cuirass/remote.scm:
> 2023-12-04 10:13:48 466:6 2 (receive-message _ #:router? _)
> 2023-12-04 10:13:48 In ice-9/boot-9.scm:
> 2023-12-04 10:13:48 1685:16 1 (raise-exception _ #:continuable? _)
> 2023-12-04 10:13:48 1685:16 0 (raise-exception _ #:continuable? _)
> 2023-12-04 10:13:48 ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> 2023-12-04 10:13:48 Throw to key `match-error' with args `("match" "no matching pattern" (#<zmq-message 7f981881d9a0> #<zmq-message 7f981881d900> #<zmq-message 7f981881d860>))'.

With commit 9a1452ee021c9f773424961cfeef47ca0b7c5c5a, ‘cuirass
remote-server’ handles this situation gracefully.

I’m still curious about who’s sending us garbage: it has to be valid
zmq, just with the wrong number of parts. From the logs, it seemed like
a port scanner or something on the MDC network was sending garbage on
all the open ports, but that would need to be a sophisticated one.

Ludo’.
Closed
?