'herd status' hangs with daemonized shepherd

Open

Details

2 participants

Simen Endsjø
Ludovic Courtès

Owner: unassigned

Submitted by: Simen Endsjø

Severity: normal

Simen Endsjø wrote 5 days ago

herd status hangs forever

Recipients:(address . bug-guix@gnu.org)

Message-ID:87a5c0jjno.fsf@simendsjo.me

On one of my systems, `herd status` just hangs forever.
`sudo herd status` works as expected, and I'm able to start and stop
services as normal, just not list them.

I understand this is not much to debug based on -- any idea how I can
get more detailed information?

Ludovic Courtès wrote 3 days ago

Recipients:(name . Simen Endsjø)(address . contact@simendsjo.me)(address . 75460@debbugs.gnu.org)

Message-ID:874j25lzbd.fsf@gnu.org

Hi,

Simen Endsjø <contact@simendsjo.me> skribis:

Toggle quote (4 lines)

> On one of my systems, `herd status` just hangs forever.

> `sudo herd status` works as expected, and I'm able to start and stop

> services as normal, just not list them.

The former talks to your user shepherd (perhaps you’re using Guix Home?)

whereas the latter talks to the system shepherd (PID�1).

What you seem to be saying here is that your user shepherd is buggy.

Toggle quote (3 lines)

> I understand this is not much to debug based on -- any idea how I can

> get more detailed information?

Can you say which shepherd version you’re using, for your user shepherd?

You can find out by running “cat /proc/PID/cmdline | xargs -0” for its

PID, or by checking ~/.local/state/shepherd/shepherd.log (version 1.0.x

prints its version string there at startup but previous versions

didn’t.)

Then please check precisely what hangs and what doesn’t. You say “herd

status” hangs; what about “herd status X”, where X is a service defined

for your user shepherd?

Last, could you share the config file of your user shepherd, or maybe

the Guix Home config if that’s what you’re using?

Thanks in advance,

Ludo’.

Simen Endsjø wrote 2 days ago

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 75460@debbugs.gnu.org)

Message-ID:87plks1mv0.fsf@simendsjo.me

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (7 lines)

> Can you say which shepherd version you’re using, for your user shepherd?

> You can find out by running “cat /proc/PID/cmdline | xargs -0” for its

> PID, or by checking ~/.local/state/shepherd/shepherd.log (version 1.0.x

> prints its version string there at startup but previous versions

> didn’t.)

/gnu/store/vl55r8m4h0a2zw0hpfziijybvji21062-guile-3.0.9/bin/guile --no-auto-compile /gnu/store/kmv7ny45x2xirqz0y5pha8jm4v02h0lq-shepherd-1.0.0/bin/shepherd --silent --config /gnu/store/lwjnnpjca0kplrsm82vf1pjq7x4d8lq7-shepherd.conf

Toggle quote (5 lines)

> Then please check precisely what hangs and what doesn’t. You say “herd

> status” hangs; what about “herd status X”, where X is a service defined

> for your user shepherd?

Not sure how I can find out precicely what hangs. Operations on

individual services seems to work fine:

$ herd status emacs-daemon

? Status of emacs-daemon:

It is running since Sat 11 Jan 2025 07:35:36 PM CET (13 hours ago).

# (... snip ...)

Toggle quote (3 lines)

> Last, could you share the config file of your user shepherd, or maybe

> the Guix Home config if that’s what you’re using?

https://git.sr.ht/~simendsjo/dotfiles/tree/main/item/sijo/home/simendsjo/home-environment.scm

And here is the shepherd configuration loaded (as per the command line
arguments above):

(begin
  (use-modules
   (srfi srfi-34)
   (system repl error-handling))
  (define
    (make-user-module)
    (let
        ((m
          (make-fresh-user-module)))
      (module-use! m
                   (resolve-interface
                    (quote
                     (shepherd service))))
      m))
  (register-services
   (map
    (lambda
        (file)
      (save-module-excursion
       (lambda
           ()
         (set-current-module
          (make-user-module))
         (load file))))
    (quote
     ("/gnu/store/d5kay849g8qsxa324h0bxclxvczkv4iw-shepherd-xss-lock.scm" "/gnu/store/b1i400q6wywi3zj2ij4gnrmndf2259yx-shepherd-volctl.scm" "/gnu/store/yhfgc8djxmavfx1k90yg0srs4iykd3ck-shepherd-unclutter.scm" "/gnu/store/6527xv4yp3yagkdz3vszcqv66xrkcg2w-shepherd-redshift.scm" "/gnu/store/jssmizyhlv66a441nwrz57hggbqsssq9-shepherd-picom.scm" "/gnu/store/wn3q5w0li0hn7574cidkz3fs6iwjjl7d-shepherd-network-manager-applet.scm" "/gnu/store/k8w0vgcffya47dsvbcad0ks3yfwgcm6z-shepherd-dunst.scm" "/gnu/store/2y6hkkw9dfynb45y9win9z41p6sn4sa2-shepherd-cbatticon.scm" "/gnu/store/6db7qbavk5q3vi50hnsz80sqzmz3vmad-shepherd-blueman-applet.scm" "/gnu/store/v27fvdki9x0nq8aay6a2dd6rhfpb2sqh-shepherd-emacs-daemon.scm" "/gnu/store/rjwyyq6s9xlm87jjvnhhzd886zdfi6l2-shepherd-ssh-agent.scm" "/gnu/store/ybxjna1g6m50hndl60hf3ybm6clzh4qa-shepherd-dbus.scm" "/gnu/store/s6dpkii7xsr4wa2jqzglfprz8cb0bb8h-shepherd-x11-display.scm"))))
  (perform-service-action root-service
                          (quote daemonize))
  (format #t "Starting services...~%")
  (let
      ((services-to-start
        (quote
         (xss-lock volctl unclutter redshift picom network-manager-applet dunst cbatticon blueman-applet emacs-daemon ssh-agent dbus x11-display))))
    (start-in-the-background services-to-start)
    (redirect-port
     (open-input-file "/dev/null")
     (current-input-port))))

Ludovic Courtès wrote 46 hours ago

Recipients:(name . Simen Endsjø)(address . contact@simendsjo.me)(address . 75460@debbugs.gnu.org)

Message-ID:87bjwcktnb.fsf@gnu.org

Hello,

Simen Endsjø <contact@simendsjo.me> skribis:

Toggle quote (12 lines)>> Then please check precisely what hangs and what doesn’t.  You say “herd
>> status” hangs; what about “herd status X”, where X is a service defined
>> for your user shepherd?
>
> Not sure how I can find out precicely what hangs. Operations on
> individual services seems to work fine:
>
> $ herd status emacs-daemon
> ? Status of emacs-daemon:
>   It is running since Sat 11 Jan 2025 07:35:36 PM CET (13 hours ago).
>   # (... snip ...)

Can you try something like:

Toggle snippet (8 lines)

for s in xss-lock volctl unclutter redshift picom \

network-manager-applet dunst cbatticon blueman-applet \

emacs-daemon ssh-agent dbus x11-display

echo "trying $s"; herd status "$s"

done

to try and determine whether one of these is hanging?

If you don’t mind, could you share ~/.local/state/shepherd/shepherd.log,

privately if you prefer? (Preferably remove everything that comes

before its last boot.)

Toggle quote (5 lines)

>> Last, could you share the config file of your user shepherd, or maybe

>> the Guix Home config if that’s what you’re using?

> https://git.sr.ht/~simendsjo/dotfiles/tree/main/item/sijo/home/simendsjo/home-environment.scm

[...]

Toggle quote (2 lines)

> (xss-lock volctl unclutter redshift picom network-manager-applet dunst cbatticon blueman-applet emacs-daemon ssh-agent dbus x11-display))))

I checked these and the only one that looks fishy is ‘unclutter’, which

I would change like this:

Toggle diff (22 lines)diff --git a/gnu/home/services/desktop.scm b/gnu/home/services/desktop.scm
index fc96ce9295..859dba776a 100644
--- a/gnu/home/services/desktop.scm
+++ b/gnu/home/services/desktop.scm
@@ -353,7 +353,6 @@ (define (home-unclutter-shepherd-service config)
     (modules '((shepherd support) ;for %user-log-dir
                (srfi srfi-1)
                (srfi srfi-26)))
-    (one-shot? #t)
     (start #~(lambda _
                (fork+exec-command
                 (list
@@ -369,7 +368,8 @@ (define (home-unclutter-shepherd-service config)
                       (remove (cut string-prefix? "DISPLAY=" <>)
                               (default-environment-variables)))
                 #:log-file
-                (string-append %user-log-dir "/unclutter.log")))))))
+                (string-append %user-log-dir "/unclutter.log"))))
+    (stop #~(make-kill-destructor)))))
 
 (define home-unclutter-service-type
   (service-type

That said, it cannot explain a hang.

Ludo’.

Simen Endsjø wrote 45 hours ago

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 75460@debbugs.gnu.org)

Message-ID:87y0zgjakm.fsf@simendsjo.me

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (13 lines)> Can you try something like:
>
> --8<---------------cut here---------------start------------->8---
> for s in xss-lock volctl unclutter redshift picom \
>   network-manager-applet dunst cbatticon blueman-applet \
>   emacs-daemon ssh-agent dbus x11-display
> do
>   echo "trying $s"; herd status "$s"
> done
> --8<---------------cut here---------------end--------------->8---
>
> to try and determine whether one of these is hanging?

Oh, I only checked some services and assumed all was working. All the
following services hang. The common denominator is that I have written
these myself, so I might be doing something odd here.

• xss-lock
• volctl
• picom
• network-manager-applet
• dunst
• cbatticon
• blueman-applet

For example xss-lock:
https://git.sr.ht/~simendsjo/dotfiles/tree/main/item/sijo/home/services/xss-lock.scm#L7-30

?????
? (define (home-xss-lock-shepherd-service config)
?   (list
?    (shepherd-service
?     (provision '(xss-lock))
?     (requirement '(x11-display))
?     (start #~(make-forkexec-constructor
?               (list #$(file-append xss-lock "/bin/xss-lock")
?                     "-l" "xsecurelock")
?               #:log-file (string-append
?                           (or (getenv "XDG_STATE_HOME")
?                               (format #f "~a/.local/state"
?                                       (getenv "HOME")))
?                           "/log/xss-lock.log")))
?     (stop #~(make-kill-destructor)))))
?
? (define-public home-xss-lock-service-type
?   (service-type
?    (name 'home-xss-lock)
?    (extensions
?     (list
?      (service-extension home-shepherd-service-type
?                         home-xss-lock-shepherd-service)))
?    (default-value '())
?    (description "Screen locker")))
?????

Toggle quote (4 lines)

> If you don’t mind, could you share ~/.local/state/shepherd/shepherd.log,

> privately if you prefer? (Preferably remove everything that comes

> before its last boot.)

?????
? 2025-01-12 16:39:26 GNU Shepherd 1.0.0 (Guile 3.0.9, x86_64-unknown-linux-gnu)
? 2025-01-12 16:39:26 Starting service root...
? 2025-01-12 16:39:26 Service root started.
? 2025-01-12 16:39:26 Service root running with value #<<process> id: 2377 command: #f>.
? 2025-01-12 16:39:26 Service root has been started.
? 2025-01-12 16:39:26 WARNING: Use of `load' in declarative module (#{ g109}#).  Add #:declarative? #f to your define-module invocation.
? 2025-01-12 16:39:26 Daemonizing...
? 2025-01-12 16:39:26 Restarting signal handler.
? 2025-01-12 16:39:26 Now running as process 2396.
? 2025-01-12 16:39:26 Starting services...
? 2025-01-12 16:39:26 Configuration successfully loaded from '/gnu/store/lwjnnpjca0kplrsm82vf1pjq7x4d8lq7-shepherd.conf'.
? 2025-01-12 16:39:26 Running value of service root changed to #<<process> id: 2396 command: #f>.
? 2025-01-12 16:39:26 Starting service emacs-daemon...
? 2025-01-12 16:39:26 Starting service ssh-agent...
? 2025-01-12 16:39:26 Starting service dbus...
? 2025-01-12 16:39:26 Starting service x11-display...
? 2025-01-12 16:39:26 Service emacs-daemon has been started.
? 2025-01-12 16:39:26 Service ssh-agent has been started.
? 2025-01-12 16:39:26 Service dbus has been started.
? 2025-01-12 16:39:26 Service x11-display has been started.
? 2025-01-12 16:39:26 Service emacs-daemon started.
? 2025-01-12 16:39:26 Service ssh-agent started.
? 2025-01-12 16:39:26 Service dbus started.
? 2025-01-12 16:39:26 Service x11-display started.
? 2025-01-12 16:39:26 Service x11-display running with value ":0".
? 2025-01-12 16:39:26 Service emacs-daemon running with value #<<process> id: 2397 command: #f>.
? 2025-01-12 16:39:26 Service ssh-agent running with value #<<process> id: 2398 command: #f>.
? 2025-01-12 16:39:26 Service dbus running with value #<<process> id: 2399 command: ("/gnu/store/r799iglam0gk9q35j56239wcdg9xzhwg-dbus-1.15.8/bin/dbus-daemon" "--nofork" "--session" "--address=unix:path=/run/user/1000/bus")>.
? 2025-01-12 16:39:26 Starting service xss-lock...
? 2025-01-12 16:39:26 Service xss-lock has been started.
? 2025-01-12 16:39:26 Starting service volctl...
? 2025-01-12 16:39:26 Service xss-lock started.
? 2025-01-12 16:39:26 Service volctl has been started.
? 2025-01-12 16:39:26 Service xss-lock running with value #<<process> id: 2400 command: ("/gnu/store/a20h8scrp7jl3ildfxmj7c4mnc5s6nl2-xss-lock-0.3.0-1.1e158fb/bin/xss-lock" "-l" "xsecurelock")>.
? 2025-01-12 16:39:26 Service volctl started.
? 2025-01-12 16:39:26 Service volctl running with value #<<process> id: 2404 command: ("/gnu/store/jjaz1i6kzx7phwjp85gyggf0s85irxv2-volctl-0.9.4/bin/volctl")>.
? 2025-01-12 16:39:26 Starting service unclutter...
? 2025-01-12 16:39:26 Service unclutter has been started.
? 2025-01-12 16:39:26 Service unclutter started.
? 2025-01-12 16:39:26 Service unclutter running with value #<<process> id: 2407 command: #f>.
? 2025-01-12 16:39:26 Starting service redshift...
? 2025-01-12 16:39:26 Service redshift has been started.
? 2025-01-12 16:39:26 Service redshift started.
? 2025-01-12 16:39:26 Service redshift running with value #<<process> id: 2409 command: #f>.
? 2025-01-12 16:39:26 Starting service picom...
? 2025-01-12 16:39:26 Service picom has been started.
? 2025-01-12 16:39:26 Service picom started.
? 2025-01-12 16:39:26 Service picom running with value #<<process> id: 2410 command: ("/gnu/store/0yj6akbzrrvsrnnszkf73pp3a7n9pc45-picom-11.2/bin/picom" "--vsync" "--backend" "glx")>.
? 2025-01-12 16:39:26 Starting service network-manager-applet...
? 2025-01-12 16:39:26 Service network-manager-applet has been started.
? 2025-01-12 16:39:26 Service network-manager-applet started.
? 2025-01-12 16:39:26 Service network-manager-applet running with value #<<process> id: 2411 command: ("/gnu/store/931akl92zv8hjbq88v5xzkww56b433bh-network-manager-applet-1.36.0/bin/nm-applet")>.
? 2025-01-12 16:39:26 Starting service cbatticon...
? 2025-01-12 16:39:26 Service cbatticon has been started.
? 2025-01-12 16:39:26 Starting service blueman-applet...
? 2025-01-12 16:39:26 Service cbatticon started.
? 2025-01-12 16:39:26 Service blueman-applet has been started.
? 2025-01-12 16:39:26 Service cbatticon running with value #<<process> id: 2419 command: ("/gnu/store/mhgq9x4pq7vsgrj3fvwxnirc33likw73-cbatticon-1.6.13/bin/cbatticon")>.
? 2025-01-12 16:39:26 Starting service dunst...
? 2025-01-12 16:39:26 Service blueman-applet started.
? 2025-01-12 16:39:26 redshift[2409] Waiting for initial location to become available...
? 2025-01-12 16:39:26 Service dunst has been started.
? 2025-01-12 16:39:26 Service blueman-applet running with value #<<process> id: 2420 command: ("/gnu/store/xpix8bhcvzhyzzzkvsisc2lig15wlqrg-blueman-2.3.4/bin/blueman-applet")>.
? 2025-01-12 16:39:26 Service dunst started.
? 2025-01-12 16:39:26 Successfully started 13 services in the background.
? 2025-01-12 16:39:26 Service dunst running with value #<<process> id: 2436 command: ("/gnu/store/9iznijpx287bqkwv2cc4g3x30z72vqky-dunst-1.11.0/bin/dunst")>.
? (snip emacs load messages)
?????

Ludovic Courtès wrote 44 hours ago

Recipients:(name . Simen Endsjø)(address . contact@simendsjo.me)(address . 75460@debbugs.gnu.org)

Message-ID:87r058j89b.fsf@gnu.org

Hello,

Simen Endsjø <contact@simendsjo.me> skribis:

Toggle quote (15 lines)> Oh, I only checked some services and assumed all was working. All the
> following services hang. The common denominator is that I have written
> these myself, so I might be doing something odd here.
>
> • xss-lock
> • volctl
> • picom
> • network-manager-applet
> • dunst
> • cbatticon
> • blueman-applet
>
> For example xss-lock:
> <https://git.sr.ht/~simendsjo/dotfiles/tree/main/item/sijo/home/services/xss-lock.scm#L7-30>

Yes, I already looked at all of them just in case :-) and didn’t notice

anything fishy.

One suggestion:

Toggle diff (12 lines)diff --git a/sijo/home/services/emacs-daemon.scm b/sijo/home/services/emacs-daemon.scm
index 5e438b8..32d6ae0 100644
--- a/sijo/home/services/emacs-daemon.scm
+++ b/sijo/home/services/emacs-daemon.scm
@@ -11,7 +11,7 @@
     (modules '((shepherd support)))  ;for '%user-runtime-dir'
     (start #~(lambda ()
               (let ((socket (string-append %user-runtime-dir "/emacs/server")))
-                (unless (file-exists? socket)
+                (and (not (file-exists? socket))
                      (fork+exec-command (list "emacs" "--fg-daemon"))))))
     (stop #~(make-kill-destructor)))))

Otherwise you end up with ‘emacs-daemon’ marked as “started” with

#<unspecified> as its running value, causing an exception (type error)

when attempting to stop it.

Toggle quote (4 lines)

> ?????

> ? 2025-01-12 16:39:26 GNU Shepherd 1.0.0 (Guile 3.0.9, x86_64-unknown-linux-gnu)

> ? 2025-01-12 16:39:26 Starting service root...

Nothing stands out here, but this log is much recent than your initial

report. Is ‘herd status’ still hanging? IOW, is it 100% reproducible?

Thanks,

Ludo’.

Simen Endsjø wrote 40 hours ago

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 75460@debbugs.gnu.org)

Message-ID:87cygraj9p.fsf@simendsjo.me

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (1 lines)

> One suggestion:

(..)

Toggle quote (2 lines)

> - (unless (file-exists? socket)

> + (and (not (file-exists? socket))

(..)

Toggle quote (4 lines)

> Otherwise you end up with ‘emacs-daemon’ marked as “started” with

> #<unspecified> as its running value, causing an exception (type error)

> when attempting to stop it.

Thanks! I just assumed it worked like Common Lisp, returning #f. That

explains the weird issues I have been seeing.

Toggle quote (7 lines)

>> ?????

>> ? 2025-01-12 16:39:26 GNU Shepherd 1.0.0 (Guile 3.0.9, x86_64-unknown-linux-gnu)

>> ? 2025-01-12 16:39:26 Starting service root...

> Nothing stands out here, but this log is much recent than your initial

> report. Is ‘herd status’ still hanging? IOW, is it 100% reproducible?

I rebooted to get a clean log. Yes, it is fully reproducible.

Ludovic Courtès wrote 26 hours ago

Recipients:(name . Simen Endsjø)(address . contact@simendsjo.me)(address . 75460@debbugs.gnu.org)

Message-ID:871px7c7u9.fsf@gnu.org

Hi,

I was able to reproduce a problem (maybe not *the* problem) running

https://git.sr.ht/~simendsjo/dotfiles/tree/main/item/sijo/home/simendsjo/home-environment.scm

in ‘guix home container’, with the changes below to simplify setup.

What I noticed is that, until ‘x11-display’ has failed to start (it

necessarily fails to start in the container), shepherd is unresponsive:

any ‘herd’ command hangs. After that, everything is fine.

This is fixed by

https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=ffcd3745e5d8a3e4134ae76fe94a5ec80f52c465.

It would be great if you could give it a try because I’m not sure this

is exactly the problem you were having, but it may be related at least.

Thanks,

Ludo’.

Toggle diff (49 lines)diff --git a/sijo/home/services/sijo-fonts.scm b/sijo/home/services/sijo-fonts.scm
index ba6f9f6..2fdd922 100644
--- a/sijo/home/services/sijo-fonts.scm
+++ b/sijo/home/services/sijo-fonts.scm
@@ -8,7 +8,7 @@
   #:use-module (gnu packages ghostscript)
   #:use-module (gnu packages javascript)
   #:use-module (gnu packages xorg)
-  #:use-module (nongnu packages fonts))
+)
 
 (define sijo-fonts-service-profile
   (lambda (config)
diff --git a/sijo/home/simendsjo/home-environment.scm b/sijo/home/simendsjo/home-environment.scm
index 8d5f999..469ce67 100644
--- a/sijo/home/simendsjo/home-environment.scm
+++ b/sijo/home/simendsjo/home-environment.scm
@@ -89,9 +89,9 @@ guix home \
   #:use-module (gnu services configuration)
   #:use-module (gnu services)
   #:use-module (guix gexp)
-  #:use-module (nongnu packages chrome)
-  #:use-module (nongnu packages game-client)
-  #:use-module (nongnu packages mozilla)
+  ;; #:use-module (nongnu packages chrome)
+  ;; #:use-module (nongnu packages game-client)
+  ;; #:use-module (nongnu packages mozilla)
   #:use-module (sijo home services blueman-applet)
   #:use-module (sijo home services cbatticon)
   #:use-module (sijo home services dropbox)
@@ -115,7 +115,7 @@ guix home \
 
 (define-public simendsjo-home-environment
   (home-environment
-    (packages (list
+    #;(packages (list
                ;; spell checking
                ispell                 ; interactive spell checking
                hunspell               ; spell checker (used by e.g. libreoffice)
@@ -346,9 +346,7 @@ guix home \
                 (extra-content "allow-loopback-pinentry")))
       (service sijo-gpg-service-type)
       ;; Font configuration
-      (service sijo-fonts-service-type)
       ;; dotnet
-      (service sijo-dotnet-service-type)
       (service sijo-mob-service-type)
       ;; Guix configuration
       ;; TODO: There is a custom service type for channels which we should use instead

Ludovic Courtès wrote 15 hours ago

control message for bug #75460

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87tta29z3m.fsf@gnu.org

retitle 75460 'herd status' hangs with daemonized shepherd

quit

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 75460@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 75460

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date