Can't restart/halt system with shepherd 0.9.3 after upgrading

Done

Details

3 participants

david larsson
Ludovic Courtès
Christopher Baines

Owner: unassigned

Submitted by: Christopher Baines

Severity: normal

Christopher Baines wrote on 24 May 2023 12:27

Recipients:(address . bug-guix@gnu.org)

Message-ID:87h6s2m38h.fsf@cbaines.net

Hey!

On a system running shepherd 0.9.3 [1], I've reconfigured, but now can't

reboot or halt.

root@hamal ~# halt

Service root is not running.

1: /gnu/store/y6w0xix15cq08qasmq75f04yzgbl98jx-shepherd-0.9.3

Ludovic Courtès wrote on 25 May 2023 15:13

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 63678@debbugs.gnu.org)

Message-ID:878rdceeq5.fsf@gnu.org

Hi,

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (6 lines)

> On a system running shepherd 0.9.3 [1], I've reconfigured, but now can't

> reboot or halt.

> root@hamal ~# halt

> Service root is not running.

Hey, why halt it if it’s not running?

Seriously though, any insight from /var/log/messages? I upgraded a

bunch of machines and didn’t hit this particular problem. Bruno

reported a similar problem with 0.9.3, but this had nothing to do with

the upgrade:

https://issues.guix.gnu.org/62619

Could it be the same problem? Do you see:

Assertion (eq? (canonical-name new) (canonical-name old)) failed.

in /var/log/messages?

Ludo’.

Christopher Baines wrote on 25 May 2023 15:20

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 63678@debbugs.gnu.org)

Message-ID:87fs7k4kc1.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (25 lines)> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> On a system running shepherd 0.9.3 [1], I've reconfigured, but now can't
>> reboot or halt.
>>
>> root@hamal ~# halt
>> Service root is not running.
>
> Hey, why halt it if it’s not running?
>
> Seriously though, any insight from /var/log/messages?  I upgraded a
> bunch of machines and didn’t hit this particular problem.  Bruno
> reported a similar problem with 0.9.3, but this had nothing to do with
> the upgrade:
>
>   https://issues.guix.gnu.org/62619
>
> Could it be the same problem?  Do you see:
>
>   Assertion (eq? (canonical-name new) (canonical-name old)) failed.
>
> in /var/log/messages?

I don't see that, but I think these are the relevant log messages:

May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (and (defined? (quote transient?)) (map (# ?) ?)).
May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
May 24 11:17:03 localhost shepherd[1]: Service host-name has been started.
May 24 11:17:03 localhost shepherd[1]: Service user-homes has been started.
May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_hardlinks = 1
May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_symlinks = 1
May 24 11:18:41 localhost shepherd[1]: Exiting shepherd...
May 24 11:18:46 localhost shepherd[1]: Grace period of 5 seconds is over; sending -337 SIGKILL.
May 24 11:23:55 localhost shepherd[1]: Service root is not running.
May 24 11:24:16 localhost last message repeated 2 times
May 24 11:30:49 localhost syslogd (GNU inetutils 2.3): restart
May 24 11:30:49 localhost vmunix: [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
May 24 11:30:49 localhost vmunix: [    0.000000] Linux version 6.3.3-arm64-generic (guix@guix) (gcc (GCC) 11.3.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT 1

Ludovic Courtès wrote on 27 May 2023 19:04

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 63678@debbugs.gnu.org)

Message-ID:87pm6lhfiy.fsf@gnu.org

Hi,

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (10 lines)> May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (and (defined? (quote transient?)) (map (# ?) ?)).
> May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
> May 24 11:17:03 localhost shepherd[1]: Service host-name has been started.
> May 24 11:17:03 localhost shepherd[1]: Service user-homes has been started.
> May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_hardlinks = 1
> May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_symlinks = 1
> May 24 11:18:41 localhost shepherd[1]: Exiting shepherd...
> May 24 11:18:46 localhost shepherd[1]: Grace period of 5 seconds is over; sending -337 SIGKILL.
> May 24 11:23:55 localhost shepherd[1]: Service root is not running.

The grace period expiration thing is probably due to the fact that

shepherd is no longer processing signals, as I described here:

https://issues.guix.gnu.org/63736

Could you share all of /var/log/messages (possibly privately, and

limiting to “shepherd” lines) starting from when the machine booted?

I’d like to see if there are hints of something that went wrong.

Ludo’.

Christopher Baines wrote on 29 May 2023 20:33

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 63678@debbugs.gnu.org)

Message-ID:87leh7oufa.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (23 lines)> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (and (defined? (quote transient?)) (map (# ?) ?)).
>> May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
>> May 24 11:17:03 localhost shepherd[1]: Service host-name has been started.
>> May 24 11:17:03 localhost shepherd[1]: Service user-homes has been started.
>> May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_hardlinks = 1
>> May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_symlinks = 1
>> May 24 11:18:41 localhost shepherd[1]: Exiting shepherd...
>> May 24 11:18:46 localhost shepherd[1]: Grace period of 5 seconds is over; sending -337 SIGKILL.
>> May 24 11:23:55 localhost shepherd[1]: Service root is not running.
>
> The grace period expiration thing is probably due to the fact that
> shepherd is no longer processing signals, as I described here:
>
>   https://issues.guix.gnu.org/63736
>
> Could you share all of /var/log/messages (possibly privately, and
> limiting to “shepherd” lines) starting from when the machine booted?
> I’d like to see if there are hints of something that went wrong.

The machine is hamal (one of the HoneyComb's) and I've added a user for

you now and added the SSH key from maintenance.git.

So you should be able to: ssh ludo@hamal.cbaines.net

Your users password is also in your home directory.

david larsson wrote on 29 May 2023 21:19

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:a527fa97a7c7b9cf6a846b4923c47555@selfhosted.xyz

On 2023-05-24 12:27, Christopher Baines wrote:

Toggle quote (11 lines)> Hey!
> 
> On a system running shepherd 0.9.3 [1], I've reconfigured, but now 
> can't
> reboot or halt.
> 
> root@hamal ~# halt
> Service root is not running.
> 
> 1: /gnu/store/y6w0xix15cq08qasmq75f04yzgbl98jx-shepherd-0.9.3

FWIW, this has happened to me a bunch of times, I just never reported

it. Sometimes I was able to just login as root and run herd start root

to fix it.

I have an impression, from the "bunch of times" I've experienced, that

service root doesn't fail to work because of the system reconfigure, but

for some other reason.

Best regards,

David

Ludovic Courtès wrote on 6 Jun 2023 17:06

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 63678@debbugs.gnu.org)

Message-ID:877csg8wbp.fsf@gnu.org

Hi,

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (30 lines)> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi,
>>
>> Christopher Baines <mail@cbaines.net> skribis:
>>
>>> May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (and (defined? (quote transient?)) (map (# ?) ?)).
>>> May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
>>> May 24 11:17:03 localhost shepherd[1]: Service host-name has been started.
>>> May 24 11:17:03 localhost shepherd[1]: Service user-homes has been started.
>>> May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_hardlinks = 1
>>> May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_symlinks = 1
>>> May 24 11:18:41 localhost shepherd[1]: Exiting shepherd...
>>> May 24 11:18:46 localhost shepherd[1]: Grace period of 5 seconds is over; sending -337 SIGKILL.
>>> May 24 11:23:55 localhost shepherd[1]: Service root is not running.
>>
>> The grace period expiration thing is probably due to the fact that
>> shepherd is no longer processing signals, as I described here:
>>
>>   https://issues.guix.gnu.org/63736
>>
>> Could you share all of /var/log/messages (possibly privately, and
>> limiting to “shepherd” lines) starting from when the machine booted?
>> I’d like to see if there are hints of something that went wrong.
>
> The machine is hamal (one of the HoneyComb's) and I've added a user for
> you now and added the SSH key from maintenance.git.
>
> So you should be able to: ssh ludo@hamal.cbaines.net

Doesn’t work right now; anything in the logs?

Ludo’.

Christopher Baines wrote on 7 Jun 2023 16:09

Recipients:(address . 63678@debbugs.gnu.org)

Message-ID:877csf1hyp.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (36 lines)> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Hi,
>>>
>>> Christopher Baines <mail@cbaines.net> skribis:
>>>
>>>> May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (and (defined? (quote transient?)) (map (# ?) ?)).
>>>> May 24 11:17:02 localhost shepherd[1]: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
>>>> May 24 11:17:03 localhost shepherd[1]: Service host-name has been started.
>>>> May 24 11:17:03 localhost shepherd[1]: Service user-homes has been started.
>>>> May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_hardlinks = 1
>>>> May 24 11:17:03 localhost shepherd[1]: [sysctl] fs.protected_symlinks = 1
>>>> May 24 11:18:41 localhost shepherd[1]: Exiting shepherd...
>>>> May 24 11:18:46 localhost shepherd[1]: Grace period of 5 seconds is over; sending -337 SIGKILL.
>>>> May 24 11:23:55 localhost shepherd[1]: Service root is not running.
>>>
>>> The grace period expiration thing is probably due to the fact that
>>> shepherd is no longer processing signals, as I described here:
>>>
>>>   https://issues.guix.gnu.org/63736
>>>
>>> Could you share all of /var/log/messages (possibly privately, and
>>> limiting to “shepherd” lines) starting from when the machine booted?
>>> I’d like to see if there are hints of something that went wrong.
>>
>> The machine is hamal (one of the HoneyComb's) and I've added a user for
>> you now and added the SSH key from maintenance.git.
>>
>> So you should be able to: ssh ludo@hamal.cbaines.net
>
> Doesn’t work right now; anything in the logs?

I believe I sorted access for Ludo, but nothing was found when looking

at the logs.

Ludovic Courtès wrote on 20 Mar 19:09 +0100

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)(address . 63678-done@debbugs.gnu.org)

Message-ID:87y1acydu8.fsf@gnu.org

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (3 lines)

> I believe I sorted access for Ludo, but nothing was found when looking

> at the logs.

I’m closing it. Let’s reopen if we stumble upon a similar issue.

Ludo’.

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 63678@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 63678

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date