Interactive prompt opened upon shepherd config file error

Done

Details

3 participants

Ludovic Courtès
Ludovic Courtès
Christopher Baines

Owner: unassigned

Submitted by: Ludovic Courtès

Severity: important

Ludovic Courtès wrote on 23 May 12:59 +0200

Recipients:(address . bug-guix@gnu.org)

Message-ID:87sey894kj.fsf@inria.fr

Hello,

One problem we noticed in the analysis of the boot problem of bayfront

after the recent downtime¹ is that an interactive REPL would be opened

after an unbound variable was found in the shepherd config file:

Toggle snippet (18 lines)[   13.098907] shepherd[1]: Service root started.
[   13.100711] shepherd[1]: Service root running with value #t.
[   13.103824] shepherd[1]: Service root has been started.
[   13.426102] shepherd[1]: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
[   13.428099] shepherd[1]: Unbound variable: make-forkexec-constructor/container
[   13.429912] shepherd[1]: 
[   13.431108] shepherd[1]: Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
[   13.441983] shepherd[1]: GNU Guile 3.0.9
[   13.442728] shepherd[1]: Copyright (C) 1995-2023 Free Software Foundation, Inc.
[   13.443947] shepherd[1]: 
[   13.444427] shepherd[1]: Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
[   13.445679] shepherd[1]: This program is free software, and you are welcome to redistribute it
[   13.446919] shepherd[1]: under certain conditions; type `,show c' for 
[   13.447072] shepherd[1]: details.
[   13.448737] shepherd[1]: 
[   13.449239] shepherd[1]: Enter `,help' for help.

This was unhelpful because we couldn’t interact with that REPL remotely

(no IPMI). Even when you can interact, it’s of limited use; in this

case, if you type “,q”, it tries to continue and fails:

Toggle snippet (43 lines)

Uncaught exception in task:

In fibers.scm:

172:8 7 (_)

In ice-9/exceptions.scm:

406:15 6 (_)

In ice-9/boot-9.scm:

1752:10 5 (with-exception-handler _ _ #:unwind? _ # _)

In shepherd/service.scm:

824:39 4 (_)

this is because we’re effectively adding #f in the middle of the list

passed to ‘register-services’ (see below).

This REPL-on-error “feature” comes from Guix System, not Shepherd, in

the config file generated from (gnu services shepherd):

;; Arrange to spawn a REPL if something goes wrong. This is better

;; than a kernel panic.

(call-with-error-handling

(lambda ()

(register-services

(parameterize ((current-warning-port

(%make-void-port "w")))

(map (lambda (file)

(save-module-excursion

(lambda ()

(set-current-module (make-user-module))

(load-compiled file))))

'#$(map scm->go files))))))

The rationale mentioned in the comment no longer holds: starting from

Shepherd 0.10.2, the config file is loaded in the background; if it’s

evaluation fails, shepherd keeps running (see

‘tests/config-failure.sh’, which tests this behavior).

I think we should change the above to log and gracefully handle failure

to load an individual service file.

Ludo’.

¹ https://lists.gnu.org/archive/html/info-guix/2024-05/msg00000.html

Ludovic Courtès wrote on 25 May 10:56 +0200

control message for bug #71144

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87jzji46e5.fsf@gnu.org

severity 71144 important

quit

Ludovic Courtès wrote on 25 May 11:11 +0200

Re: bug#71144: Interactive prompt opened upon shepherd config file error

Recipients:(address . 71144@debbugs.gnu.org)

Message-ID:878qzy45p3.fsf@gnu.org

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

Toggle quote (3 lines)

> I think we should change the above to log and gracefully handle failure

> to load an individual service file.

With the change below, every service except the offending one is loaded

and started as expected:

Toggle snippet (32 lines)[   22.450515] shepherd[1]: Service root running with value #t.
[   22.454624] shepherd[1]: Service root has been started.
[   22.711738] shepherd[1]: Exception caught while loading '/gnu/store/fjis6iqpjfcnr90fy8rsg9v4j828jslv-shepherd-gwl-web.go': #<&compound-exception components: (#<&undefined-variable> #<&origin origin: #f> #<&message message: "Unbound variable: ~S"> #<&irritants irri
[   22.711839] tants: (make-forkexec-constructor/container)> #<&exception-with-kind-and-args kind: unbound-variable args: (#f "Unbound variable: ~S" (make-forkexec-constructor/container) #f)>)>
[   22.755146] shepherd[1]: starting services...
[   22.756491] shepherd[1]: Configuration successfully loaded from '/gnu/store/mq7y31xnjcjwjkyf6w7qiaq61g6n9f5x-shepherd.conf'.
Uncaught exception in task:
In fibers.scm:
    172:8  7 (_)
In ice-9/exceptions.scm:
   406:15  6 (_)
In ice-9/boot-9.scm:
  1752:10  5 (with-exception-handler _ _ #:unwind? _ # _)
In shepherd/service.scm:
   824:39  4 (_)
In oop/goops.scm:
  1567:11  3 (cache-miss #f)
   1585:2  2 (_ _ _)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1683:16  0 (raise-exception _ #:continuable? _)
ice-9/boot-9.scm:1683:16: In procedure raise-exception:
No applicable method for #<<generic> one-shot-service? (1)> in call (one-shot-service? #f)
[   22.798737] shepherd[1]: Starting service user-file-systems...
[   22.800361] shepherd[1]: Starting service root-file-system...
[   22.802015] shepherd[1]: Starting service host-name...
[   22.803688] shepherd[1]: Starting service pam...
[   22.805372] shepherd[1]: Starting service sysctl...
[   22.806926] shepherd[1]: Starting service loopback...
[   22.808225] shepherd[1]: Starting service firewall...

(There’s still this scary-looking but harmless backtrace in the middle:

that’s because (start-in-the-background '(something-that-does-not-exist))

throws like that as of 0.10.4.)

Once booted, shepherd is fine and you can interact normally with it; the

only thing missing is, in this case, the ‘gwl-web’ service, which we

failed to load.

I think that’s a significant improvement.

Thoughts?

Ludo’.

Toggle diff (48 lines)diff --git a/gnu/services/shepherd.scm b/gnu/services/shepherd.scm
index 455e972535d..f13c52c37ba 100644
--- a/gnu/services/shepherd.scm
+++ b/gnu/services/shepherd.scm
@@ -380,8 +380,7 @@ (define (shepherd-configuration-file services shepherd)
         (scm->go (cute scm->go <> shepherd)))
     (define config
       #~(begin
-          (use-modules (srfi srfi-34)
-                       (system repl error-handling))
+          (use-modules (srfi srfi-1))
 
           (define (make-user-module)
             ;; Copied from (shepherd support), where it's private.
@@ -417,17 +416,22 @@ (define (shepherd-configuration-file services shepherd)
 
           ;; Arrange to spawn a REPL if something goes wrong.  This is better
           ;; than a kernel panic.
-          (call-with-error-handling
-            (lambda ()
-              (register-services
-               (parameterize ((current-warning-port
-                               (%make-void-port "w")))
-                 (map (lambda (file)
-                        (save-module-excursion
-                         (lambda ()
-                           (set-current-module (make-user-module))
-                           (load-compiled file))))
-                      '#$(map scm->go files))))))
+          (register-services
+           (parameterize ((current-warning-port (%make-void-port "w")))
+             (filter-map (lambda (file)
+                           (with-exception-handler
+                               (lambda (exception)
+                                 (format #t "Exception caught \
+while loading '~a': ~s~%"
+                                         file exception)
+                                 #f)
+                             (lambda ()
+                               (save-module-excursion
+                                (lambda ()
+                                  (set-current-module (make-user-module))
+                                  (load-compiled file))))
+                             #:unwind? #t))
+                         '#$(map scm->go files))))
 
           (format #t "starting services...~%")
           (let ((services-to-start

Ludovic Courtès wrote on 25 May 11:33 +0200

control message for bug #71144

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87ttim2q2s.fsf@gnu.org

tags 71144 + patch

quit

Christopher Baines wrote on 25 May 14:02 +0200

Re: bug#71144: Interactive prompt opened upon shepherd config file error

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:87y17yf6bm.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (53 lines)> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>
>> I think we should change the above to log and gracefully handle failure
>> to load an individual service file.
>
> With the change below, every service except the offending one is loaded
> and started as expected:
>
> --8<---------------cut here---------------start------------->8---
> [   22.450515] shepherd[1]: Service root running with value #t.
> [   22.454624] shepherd[1]: Service root has been started.
> [   22.711738] shepherd[1]: Exception caught while loading '/gnu/store/fjis6iqpjfcnr90fy8rsg9v4j828jslv-shepherd-gwl-web.go': #<&compound-exception components: (#<&undefined-variable> #<&origin origin: #f> #<&message message: "Unbound variable: ~S"> #<&irritants irri
> [   22.711839] tants: (make-forkexec-constructor/container)> #<&exception-with-kind-and-args kind: unbound-variable args: (#f "Unbound variable: ~S" (make-forkexec-constructor/container) #f)>)>
> [   22.755146] shepherd[1]: starting services...
> [   22.756491] shepherd[1]: Configuration successfully loaded from '/gnu/store/mq7y31xnjcjwjkyf6w7qiaq61g6n9f5x-shepherd.conf'.
> Uncaught exception in task:
> In fibers.scm:
>     172:8  7 (_)
> In ice-9/exceptions.scm:
>    406:15  6 (_)
> In ice-9/boot-9.scm:
>   1752:10  5 (with-exception-handler _ _ #:unwind? _ # _)
> In shepherd/service.scm:
>    824:39  4 (_)
> In oop/goops.scm:
>   1567:11  3 (cache-miss #f)
>    1585:2  2 (_ _ _)
> In ice-9/boot-9.scm:
>   1685:16  1 (raise-exception _ #:continuable? _)
>   1683:16  0 (raise-exception _ #:continuable? _)
> ice-9/boot-9.scm:1683:16: In procedure raise-exception:
> No applicable method for #<<generic> one-shot-service? (1)> in call (one-shot-service? #f)
> [   22.798737] shepherd[1]: Starting service user-file-systems...
> [   22.800361] shepherd[1]: Starting service root-file-system...
> [   22.802015] shepherd[1]: Starting service host-name...
> [   22.803688] shepherd[1]: Starting service pam...
> [   22.805372] shepherd[1]: Starting service sysctl...
> [   22.806926] shepherd[1]: Starting service loopback...
> [   22.808225] shepherd[1]: Starting service firewall...
> --8<---------------cut here---------------end--------------->8---
>
> (There’s still this scary-looking but harmless backtrace in the middle:
> that’s because (start-in-the-background '(something-that-does-not-exist))
> throws like that as of 0.10.4.)
>
> Once booted, shepherd is fine and you can interact normally with it; the
> only thing missing is, in this case, the ‘gwl-web’ service, which we
> failed to load.
>
> I think that’s a significant improvement.
>
> Thoughts?

That looks good to me, the "Arrange to spawn a REPL if something goes
wrong" comment needs removing/updating, but that's the only thing I
spotted.

Ludovic Courtès wrote on 25 May 16:59 +0200

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87sey6ym2j.fsf@gnu.org

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (4 lines)

> That looks good to me, the "Arrange to spawn a REPL if something goes

> wrong" comment needs removing/updating, but that's the only thing I

> spotted.

Cool. I updated the comment and pushed it as

cca25a67693bb68a1884a081b415a43fad1e8641.

Thanks!

Ludo’.

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 71144@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 71144

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date