shepherd: sometimes hangs on `guix system reconfigure`

my fellow hackers,

i'm going to attach two patches that is essentially just adding a couple of asserts that trigger a test failure (tests/replacement.sh).

my current codebase (https://codeberg.org/attila-lendvai-patches/shepherd/commits/branch/attila)logs a whole lot more information, and has a more sophisticated error handling. triggering the same error on that codebase shows that the first assert is already failing (the one that is before spawning the new fiber for the controller of the replacement).

maybe the root cause is this: https://github.com/wingo/fibers/issues/29

HTH,

--

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

--

“Angry people want you to see how powerful they are… loving people want you to see how powerful You are.”

— Chief Red Eagle

dynamic extent of with-process-monitor, so that (current-process-monitor) is

valid for the root-service, too.

---

modules/shepherd.scm | 8 ++++----

1 file changed, 4 insertions(+), 4 deletions(-)

This is the bug that causes `guix system reconfigure ...` to sometimes hang,

and subsequently all shepherd commands, because a match-error flies out from

the service-controller of a replaced service, and thus its fiber dies.

---

modules/shepherd/service.scm | 2 ++

1 file changed, 2 insertions(+)

On Fri Dec 15, 2023 at 8:47 PM CET, Attila Lendvai wrote:

Thank you very much for this, Attila!

Are the patch in 67839 and/or your branch "attila" linked from there in a

state that I could test them locally? Would it be valuable to you if I ran a

patched Shepherd and sent logs and/or backtraces as I encountered them?

you're welcome! :)

it's nice of you, but not really. now that we have a failing test case in shepherd's unit tests that can reproduce it much easier.

with #67839 you would only get you an extra "Assertion failed" message over master, without much useful output.

as for my branch, it would emit a lot of useful log, including backtraces, but i keep force-pushing into it. i'm running my servers with it, though, so if you feel really adventurous, and want to join the debugging, then you can try... otherwise it's too much in flux.

what we need to focus on now is making shepherd's test suite run clean again, one way or another. then i can test it in a real life environment, and report back with any possible findings.

--

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

--

“Ignorance might be bliss for the ignorant, but for the rest of us it's a fucking pain in the ass.”

— Ricky Gervais

with-process-monitor. Fix proposed by @emixa-d at

https://github.com/wingo/fibers/issues/29#issuecomment-1858922276. This way

the parameterize of the process monitor covers everything else.

---

modules/shepherd.scm | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)

with-process-monitor. This way the parameterize of the process monitor covers

everything else. This fixes the bug that caused `guix system reconfigure` to

hang in certain situations. Fix proposed by @emixa-d at:

---

modules/shepherd.scm | 10 +++++-----

1 file changed, 5 insertions(+), 5 deletions(-)

This is the bug that causes `guix system reconfigure ...` to sometimes hang,

and subsequently all shepherd commands, because a match-error flies out from

the service-controller of a replaced service, and thus its fiber dies. These

asserts get triggered without the previous commit that fixes the issue.

---

modules/shepherd/service.scm | 2 ++

1 file changed, 2 insertions(+)

Hi Attila,

Attila Lendvai <attila.lendvai@gmail.com> skribis:

[...]

Good catch!!

I pushed these patches with small edits, in particular adding a test

that reproduces the bug without relying on assertion failures:

5dbde1c support: ‘assert’ logs source location information.

0bcf02a Update NEWS.

c07f0a8 service: Add asserts to ensure a process monitor is running.

9be0b7e shepherd: Make sure ‘with-process-monitor’ covers everything needed.

Thanks for the tedious but fruitful debugging work!

Ludo’.

close 67839

quit