[Shepherd] Non-responding service control fiber

Hello!

I have experienced many instances of Shepherd hanging through my use

of Guix, though I don't have a clear record of when it first happened.

These days I have seen a few reports on the subject. With a quick

search of recent bug reports, I can't find any related, only to find

this thread [1] on help-guix. So I'll start a bug report here, but I

don't know how to debug Shepherd and I haven't found a way to

reproduce it stably.

I'm not sure if Shepherd hangs at usual, but most of the time I find

it already hanging is when doing a reconfiguration. The

reconfiguration becomes unresponsive and it won't accept a ^C, herd

actions also hang. This usually happens with home reconfiguration,

but I can remember once with system reconfiguration when adding and

deleting some services in the configuration file.

I'm not sure how Shepherd hangs either, because in the latter case

(system one) I can still see logs indicating that it's trying to

respawn a process I killed manually, even though that's just a output

and no processes are actually spawned.

And as shown in [1], there are also cases where Shepherd hangs at some

point in the halting process, usually after syslogd has been

terminated but before term-tty*.

(The termination message indicates that Shepherd is still functional,

and no logs after that point shows that that's a real action, but

because of this I can't know anything happened further either. After

that I'm still able to switch ttys so I assume term-tty* are alive.)

Although I don't know how they are related, I have linked my

configurations below:

Thanks

[1]:

(public-inbox mirror on yhetil.org)

On Wed, 09 Aug 2023 20:41:44 +0800,

Hilton Chain wrote:

Today I encountered the home reconfiguration issue. The behavior is

similar to https://issues.guix.gnu.org/54919.

Ending part of output for the hanging reconfiguration:

And `herd status' also hangs:

On Sun, 13 Aug 2023 23:25:59 +0800,

Hilton Chain wrote:

And today Shepherd hung after starting a service [1], the service

itself started successfully (process started, logs available):

[1]: https://codeberg.org/hako/Rosenthal/src/commit/c7dc95c2932d7362673c28cdc2f52e6bb8357c18/rosenthal/services/child-error.scm#L151

Hi!

Hilton Chain <hako@ultrarare.space> scribes:

I’m assuming this is shepherd 0.10.2, right?

Is any of the services you’re using doing “non-standard things” such as

using constructors/destructors other than those provided by shepherd

(‘make-forkexec-constructor’ et al.)?

Is it reproducible, and do you think you could come up with a reduce

test case (for example by removing services from the config until you

reach the minimum)?

Thanks,

Ludo’.

On Sun, 03 Sep 2023 04:49:35 +0800,

Ludovic Courtès wrote:

Yes!

No, I'm unaware of such things.

I still don't know which condition triggers it, so I can't make a test

case.

It's unreproducible. And I don't think it's really related to the

config, since Shepherd won't hang when rebooting to a system

generation which made it hanging at reconfiguration before.

It might be related to bug#65419 ([Shepherd] Non-reponding service

control fiber) you have reported, since there's similar behavior that

`herd status nscd' still works when Shepherd hangs.

merge 65419 65178

quit

severity 65419 important

quit

retitle 65419 [Shepherd] Non-responding service control fiber

quit

After a bit of searching, it looks like 67538, 67230 and 65178 may be the same

issue.

Attila Lendvai wrote:

Same symptoms here again.

For context: this time I was trying to deploy some OCI/Docker containers using

Guix' `oci-container-service-type', specifically a Shepherd service called

"conduit". My code is here:

(Specifically, commits bf94f7872a1df293bd904bbd2c1ef7229f4f98a8 and

c87dcdae79c6266ac3dac70af08fbef5eb21629b.)

This is with Guix commit 1b2505217cf222d98cc960b8510660976a01cfa1.

I first ran "guix system reconfigure -L . tw/system/lud.scm" with commit

bf94f7872a1df293bd904bbd2c1ef7229f4f98a8, which had a bug (an env var was

wrong, so the container failed to start). This worked as expected in that

Shepherd tried to start the service, which failed, so Shepherd disabled it.

Then, I fixed the env var and re-ran "guix system reconfigure -L .

tw/system/lud.scm" with commit c87dcdae79c6266ac3dac70af08fbef5eb21629b.

Shepherd loaded the new "conduit" service fine, as far as I can tell, but

didn't restart it because it was still disabled.

I then enabled and started the service manually. Enabling worked fine, but on

start, I got no terminal output from Shepherd, and it hung.

I still had an error in my setup (directory permissions were wrong), and I got

a message in /var/log/messages to that effect:

...showing that Shepherd had at least tried to start the new container. The

container is not running, though (due to the error shown above), and nothing

with PID 13226 is running.

The "herd start conduit" command did not return, and ^C-ing it did not help.

Afterwards, every "herd" command also hung without any output.

Here are the last four lines of the output of "sudo strace -s1000 herd status"

on such a hung machine:

The "read(10, " call never completes.

At least in this case, Shepherd still seems to be processing inbound inet

connections, so I can open new SSH connections to the machine.

Attaching to PID 1 with strace shows it is stuck in "epoll_wait(13, "

(unsurprisingly, fd 13 points to "anon_inode:[eventpoll]"). Here's a backtrace

of all threads in "gdb -p 1":

Unrelatedly, I also have another Shepherd on a different machine that became

stuck after I ran a bunch of "guix system reconfigure" commands. The

backtraces there, if it helps:

i think i have found the root cause of this, as documented here: https://issues.guix.gnu.org/67839

that issue contains patches for shepherd to reproduce it in its test suite.

--

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

--

“What divides libertarians from everybody else is not a belief about rights or what rights people have, because the judgments libertarians make about the state are the same as the judgments almost everyone makes about private agents. So it's not that we believe in rights that other people don't believe in, or that other people believe in rights that we don't believe in. It's that other people think the state is exempt from the moral principles that apply to non-government agents.”

— Michael Huemer

On Fri Dec 15, 2023 at 8:47 PM CET, Attila Lendvai wrote:

Thank you very much for this, Attila!

Are the patch in 67839 and/or your branch "attila" linked from there in a

state that I could test them locally? Would it be valuable to you if I ran a

patched Shepherd and sent logs and/or backtraces as I encountered them?

you're welcome! :)

it's nice of you, but not really. now that we have a failing test case in shepherd's unit tests that can reproduce it much easier.

with #67839 you would only get you an extra "Assertion failed" message over master, without much useful output.

as for my branch, it would emit a lot of useful log, including backtraces, but i keep force-pushing into it. i'm running my servers with it, though, so if you feel really adventurous, and want to join the debugging, then you can try... otherwise it's too much in flux.

what we need to focus on now is making shepherd's test suite run clean again, one way or another. then i can test it in a real life environment, and report back with any possible findings.

--

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

--

“Ignorance might be bliss for the ignorant, but for the rest of us it's a fucking pain in the ass.”

— Ricky Gervais

Hello,

Attila Lendvai <attila@lendvai.name> skribis:

Yes, it looks like this long-standing and hard-to-debug issue may well

be fixed now, thumbs up Attila!!

We have accumulated quite a few fixes by now so I think I’ll release

0.10.3 hopefully in 2023 and otherwise soon after.

Thanks,

Ludo’.

close 65419

quit