dockerd is not started automatically

  • Done
  • quality assurance status badge
Details
4 participants
  • andreoss
  • Danny Milosavljevic
  • Oleg Pykhalov
  • Maxim Cournoyer
Owner
unassigned
Submitted by
andreoss
Severity
normal
Merged with
A
M
M
Maxim Cournoyer wrote on 28 May 2020 15:24
(address . andreoss@SDF.ORG)(address . 38432@debbugs.gnu.org)
87d06o5h05.fsf@gmail.com
Hello!

andreoss@SDF.ORG writes:

Toggle quote (9 lines)
> When defined in /etc/config.scm as the manual suggests[1]
>> (service docker-service-type)
>
> dockerd is not started with other services.
> It could be started manually by
> $ herd start dockerd
>
> [1] https://guix.gnu.org/manual/en/html_node/Miscellaneous-Services.html

FWIW, I started experimenting the same on my Guix System after a recent
guix pull & guix system reconfigure.

Maxim
D
D
Danny Milosavljevic wrote on 28 May 2020 15:42
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
20200528154133.68781722@scratchpost.org
Hi Maxim,

On Thu, 28 May 2020 09:24:58 -0400
Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

Toggle quote (3 lines)
> FWIW, I started experimenting the same on my Guix System after a recent
> guix pull & guix system reconfigure.

Really? Back then I've fixed and replaced a lot of the modprobe stuff in
docker upstream--so now it should start up much more reliably on Guix.

I don't know what could be up with it now. Could you please check logs
in /var/log/containerd.log and /var/log/docker.log (without
manually starting docker beforehand).
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl7Pv6oACgkQ5xo1VCww
uqXZTAgAl3OOPnOyjRPf0FnKC3snrGgTGOq3BouF2hnddOOLIAos39qXKhRcrfdj
TfOfC0N8hSEb+CloX25x/J2P3rem0z7nBxYVH5lxukWAYxVLpi+WHiz+wCqLf+64
UulbohxToqP3VkEhnxqrHaxpWA/WAssgv/ISDfLb4et4IjhngbOInv6DpxsNx3VH
XN7rXGmsgnlvhl25DAda/rSt5VI3RtpYUb+VuZgMhAlGLO4pi5Q9WKdDyrfAN2QD
UbPmJW9raLztaFLy8HmTs0td0TsuXYm/T97431BeD6+Ad37K3E7/aCqem2sdI9ex
r4pE5tX8mffPYOMyknU2IYVEIOhSXQ==
=nB2E
-----END PGP SIGNATURE-----


M
M
Maxim Cournoyer wrote on 1 Jun 2020 05:27
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
87eeqz5uu5.fsf@gmail.com
Hello Danny,

Sorry for the delay.

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (11 lines)
> Hi Maxim,
>
> On Thu, 28 May 2020 09:24:58 -0400
> Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> FWIW, I started experimenting the same on my Guix System after a recent
>> guix pull & guix system reconfigure.
>
> Really? Back then I've fixed and replaced a lot of the modprobe stuff in
> docker upstream--so now it should start up much more reliably on Guix.

Really! :-/. I've reconfigured earlier and just rebooted now, and
dockerd was stopped:

Toggle snippet (11 lines)
$ sudo herd status dockerd
Password:
Status of dockerd:
It is stopped.
It is enabled.
Provides (dockerd).
Requires (containerd dbus-system elogind file-system-/sys/fs/cgroup/blkio file-system-/sys/fs/cgroup/cpu file-system-/sys/fs/cgroup/cpuset file-system-/sys/fs/cgroup/devices file-system-/sys/fs/cgroup/memory file-system-/sys/fs/cgroup/pids networking udev).
Conflicts with ().
Will be respawned.

Toggle quote (4 lines)
> I don't know what could be up with it now. Could you please check logs
> in /var/log/containerd.log and /var/log/docker.log (without
> manually starting docker beforehand).

Something interesting from containerd.log:

Toggle snippet (17 lines)
time="2020-05-31T22:28:15.863091224-04:00" level=info msg="starting containerd" revision=.m version=
time="2020-05-31T22:28:15.883644866-04:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
time="2020-05-31T22:28:15.883714657-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
time="2020-05-31T22:28:15.883975218-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
time="2020-05-31T22:28:15.885343166-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.4.43-gnu\n": exit status 1"
time="2020-05-31T22:28:15.885391522-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
time="2020-05-31T22:28:15.885483513-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
time="2020-05-31T22:28:15.885632392-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
time="2020-05-31T22:28:15.885850392-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
time="2020-05-31T22:28:15.885871090-04:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
time="2020-05-31T22:28:15.885918881-04:00" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.4.43-gnu\n": exit status 1"
time="2020-05-31T22:28:15.885937520-04:00" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
time="2020-05-31T22:28:15.911651119-04:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
time="2020-05-31T22:28:15.911693576-04:00" level=info msg="loading
plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1

The only new lines to appear in docker.log following my last reboot are:

Toggle snippet (4 lines)
time="2020-05-31T22:28:13.579626539-04:00" level=info msg="Starting up"
failed to start containerd: exec: "containerd": executable file not found in $PATH

So, it seems the failure is related to kernel modules not being found
where they are looked for?

I think there were some changes recently in this area, but I haven't
followed closely.

Maxim
D
D
Danny Milosavljevic wrote on 1 Jun 2020 14:24
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
20200601142454.5a45c75c@scratchpost.org
Hi Maxim,

On Sun, 31 May 2020 23:27:30 -0400
Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

Toggle quote (2 lines)
> time="2020-05-31T22:28:15.885343166-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/5.4.43-gnu\n": exit status 1"

We don't have aufs, but it's not mandatory anyway.

Toggle quote (10 lines)
> The only new lines to appear in docker.log following my last reboot are:
>
> --8<---------------cut here---------------start------------->8---
> time="2020-05-31T22:28:13.579626539-04:00" level=info msg="Starting up"
> failed to start containerd: exec: "containerd": executable file not found in $PATH
> --8<---------------cut here---------------end--------------->8---
>
> So, it seems the failure is related to kernel modules not being found
> where they are looked for?

I don't think so this time, at least according to these logs.

Could it be that containerd takes too long to start up or something? And then it
tries to start it on its own?

To find out, try adding sleep to gnu/services/docker.scm docker-shepherd-service
before make-forkexec-constructor, to make it read

#~(begin
(sleep 2)
(make-forkexec-constructor ....
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAl7U85YACgkQ5xo1VCww
uqU4GQf/QyCsOEO9Wb8+mHKRgLn7j0vurgam2/63+xeXb7accStBZI0a3MAPY7QF
jwUXUl2RUNc77QISAXAGg4WSEJJ8GMMe8pxSaz/98R5vXZRfMl3wC2nGZJpykLXI
FkEsj0/+iG0kq+mY40ydHS53PnWRi14wGeBRjzXaSYvqGPKT1LznqKwvynRFjRrm
rwZDq52eb4Qwiz9CpYVEPZrZS0cTMYHeXkJ3Gmt0sbwOePzpEmfX1jt2kdkztVW/
+TfO8nKXPRtzIqUJtlO0WY/qaZAYOuaLClCqSp6BEGzXRpMLzkM4igHhngKgoK6I
kVCrOdu9kjoFyTuss8vprky4RapD9Q==
=eLPJ
-----END PGP SIGNATURE-----


M
M
Maxim Cournoyer wrote on 2 Jun 2020 02:59
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
87h7vunuyh.fsf@gmail.com
Hi Danny!

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (34 lines)
> Hi Maxim,
>
> On Sun, 31 May 2020 23:27:30 -0400
> Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> time="2020-05-31T22:28:15.885343166-04:00" level=warning msg="failed
>> to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe
>> aufs failed: "modprobe: FATAL: Module aufs not found in directory
>> /lib/modules/5.4.43-gnu\n": exit status 1"
>
> We don't have aufs, but it's not mandatory anyway.
>
>> The only new lines to appear in docker.log following my last reboot are:
>>
>> --8<---------------cut here---------------start------------->8---
>> time="2020-05-31T22:28:13.579626539-04:00" level=info msg="Starting up"
>> failed to start containerd: exec: "containerd": executable file not found in $PATH
>> --8<---------------cut here---------------end--------------->8---
>>
>> So, it seems the failure is related to kernel modules not being found
>> where they are looked for?
>
> I don't think so this time, at least according to these logs.
>
> Could it be that containerd takes too long to start up or something? And then it
> tries to start it on its own?
>
> To find out, try adding sleep to gnu/services/docker.scm docker-shepherd-service
> before make-forkexec-constructor, to make it read
>
> #~(begin
> (sleep 2)
> (make-forkexec-constructor ....

It looks like this! Which is weird, because containerd doesn't take much
time to start, even when networking is not yet functional (I suspected
something like this at first).

I wonder what we can do here... looks like we need to poll containerd to
know when it's ready in the start procedure of dockerd (unless I'm
missing something fancier that Shepherd would allow doing). Else we
could let dockerd spawn its own containerd daemon, which it can do if we
tell it where it is.

Thoughts?

I've made the following changes while troubleshooting this. It only
printed a couple more lines, but I guess it's nice to have:
From 80f3812e36cfb738b494343c5a6c20cb65588ad1 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Mon, 1 Jun 2020 20:54:40 -0400
Subject: [PATCH] gnu: services: docker: Add a debug? parameter.

* gnu/services/docker.scm (docker-configuration): Add a debug? field.
(containerd-shepherd-service): Pass the "--log-level=debug" argument when
DEBUG? is true.
(docker-shepherd-service): Pass the "--debug" and "--log-level=debug"
arguments when DEBUG? is true.
---
doc/guix.texi | 9 +++++++++
gnu/services/docker.scm | 19 +++++++++++++++----
2 files changed, 24 insertions(+), 4 deletions(-)

Toggle diff (76 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index beeda34ea2..de958c9cf1 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -26287,6 +26287,15 @@ The Docker package to use.
@item @code{containerd} (default: @var{containerd})
The Containerd package to use.
+@item @code{proxy} (default @var{docker-libnetwork-cmd-proxy})
+The Docker user-land networking proxy package to use.
+
+@item @code{enable-proxy?} (default @code{#f})
+Enable or disable the use of the Docker user-land networking proxy.
+
+@item @code{debug?} (default @code{#f})
+Enable or disable debug output.
+
@end table
@end deftp
diff --git a/gnu/services/docker.scm b/gnu/services/docker.scm
index d6dc792821..ccdb67ed73 100644
--- a/gnu/services/docker.scm
+++ b/gnu/services/docker.scm
@@ -52,7 +52,10 @@
loop-back communications.")
(enable-proxy?
(boolean #t)
- "Enable or disable the user-land proxy (enabled by default)."))
+ "Enable or disable the user-land proxy (enabled by default).")
+ (debug?
+ (boolean #f)
+ "Enable or disable debug output."))
(define %docker-accounts
(list (user-group (name "docker") (system? #t))))
@@ -71,19 +74,24 @@ loop-back communications.")
(mkdir-p #$state-dir))))
(define (containerd-shepherd-service config)
- (let* ((package (docker-configuration-containerd config)))
+ (let* ((package (docker-configuration-containerd config))
+ (debug? (docker-configuration-debug? config)))
(shepherd-service
(documentation "containerd daemon.")
(provision '(containerd))
(start #~(make-forkexec-constructor
- (list (string-append #$package "/bin/containerd"))
+ (list (string-append #$package "/bin/containerd")
+ #$@(if debug?
+ '("--log-level=debug")
+ '()))
#:log-file "/var/log/containerd.log"))
(stop #~(make-kill-destructor)))))
(define (docker-shepherd-service config)
(let* ((docker (docker-configuration-docker config))
(enable-proxy? (docker-configuration-enable-proxy? config))
- (proxy (docker-configuration-proxy config)))
+ (proxy (docker-configuration-proxy config))
+ (debug? (docker-configuration-debug? config)))
(shepherd-service
(documentation "Docker daemon.")
(provision '(dockerd))
@@ -101,6 +109,9 @@ loop-back communications.")
(start #~(make-forkexec-constructor
(list (string-append #$docker "/bin/dockerd")
"-p" "/var/run/docker.pid"
+ #$@(if debug?
+ '("--debug" "--log-level=debug")
+ '())
(if #$enable-proxy? "--userland-proxy" "")
"--userland-proxy-path" (string-append #$proxy
"/bin/proxy"))
--
2.26.2
Maxim
M
M
Maxim Cournoyer wrote on 2 Jun 2020 14:11
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
87lfl54qgr.fsf@gmail.com
Hello Danny!

The service definition here:
uses the systemd notify mechanism to allow the application (containerd)
to message the init (systemd) that it is ready.

leoprikler on #guix had the idea to patch this away and replace it by
some code that'd write a PID file. That's an interesting idea! Even
nicer would be to have Shepherd listen on the socket that the sd_notify
mechanism relies on and natively support the systemd notify thing :-).

Food for thoughts...

Maxim
M
M
Maxim Cournoyer wrote on 24 Jun 2022 07:08
control message for bug #55936
(address . control@debbugs.gnu.org)
87wnd6zm08.fsf@gmail.com
merge 55936 38432
quit
O
O
Oleg Pykhalov wrote on 2 Jul 2022 13:39
(address . control@debbugs.gnu.org)
62c02e8f.1c69fb81.cca20.7706@mx.google.com
merge 55936 38432
quit
?