docker containers stopped when doing guix install or guix shell

  • Open
  • quality assurance status badge
Details
3 participants
  • Maxim Cournoyer
  • Csepp
  • Remco van 't Veer
Owner
unassigned
Submitted by
Remco van 't Veer
Severity
normal
R
R
Remco van 't Veer wrote on 11 May 2022 09:12
(address . bug-guix@gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
87ilqch79l.fsf@remworks.net
On a Guix system host, some running docker containers are stopped when
doing guix install or other guix operations like shell. I noticed this
happing to mysql and postgres containers but an elasticsearch container
just keeps running.

Here's an example session:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker run -d postgres:10.10
..
2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2b52ee072b1f postgres:10.10 "docker-entrypoint.s…" 2 seconds ago Up 1 seconds 5432/tcp blah_blah
$ guix shell xeyes -- xeyes
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0.0 MB will be downloaded
xeyes-1.1.2 11KiB 613KiB/s 00:00 [##################] 100.0%
The following derivation will be built:
/gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv

applying 4 grafts for xeyes-1.1.2 ...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ exit

First we see no docker containers are running, then we start postgres-10
from docker hub, we see its container is running, then we do something
using guix-shell on an application *not already available on this
system*, and now the container died. This does not work the second time
when the "derivation" is already "built".

Cheers,
Remco
M
M
Maxim Cournoyer wrote on 12 Jul 2022 15:48
(name . Remco van 't Veer)(address . remco@remworks.net)
87mtde8mrr.fsf@gmail.com
Hi,

Remco van 't Veer <remco@remworks.net> writes:

Toggle quote (38 lines)
> On a Guix system host, some running docker containers are stopped when
> doing guix install or other guix operations like shell. I noticed this
> happing to mysql and postgres containers but an elasticsearch container
> just keeps running.
>
> Here's an example session:
>
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> $ docker run -d postgres:10.10
> ..
> 2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> 2b52ee072b1f postgres:10.10 "docker-entrypoint.s…" 2 seconds ago Up 1 seconds 5432/tcp blah_blah
> $ guix shell xeyes -- xeyes
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> 0.0 MB will be downloaded
> xeyes-1.1.2 11KiB 613KiB/s 00:00 [##################] 100.0%
> The following derivation will be built:
> /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>
> applying 4 grafts for xeyes-1.1.2 ...
> building CA certificate bundle...
> listing Emacs sub-directories...
> building fonts directory...
> building directory of Info manuals...
> building profile with 1 package...
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> $ exit
>
> First we see no docker containers are running, then we start postgres-10
> from docker hub, we see its container is running, then we do something
> using guix-shell on an application *not already available on this
> system*, and now the container died. This does not work the second time
> when the "derivation" is already "built".

Are you still able to reproduce this using the new version of docker
packaged in Guix?

Thanks,

Maxim
R
R
Remco van 't Veer wrote on 12 Jul 2022 16:37
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87h73m9z3f.fsf@remworks.net
2022/07/12 09:48, Maxim Cournoyer:

Toggle quote (45 lines)
> Hi,
>
> Remco van 't Veer <remco@remworks.net> writes:
>
>> On a Guix system host, some running docker containers are stopped when
>> doing guix install or other guix operations like shell. I noticed this
>> happing to mysql and postgres containers but an elasticsearch container
>> just keeps running.
>>
>> Here's an example session:
>>
>> $ docker ps
>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>> $ docker run -d postgres:10.10
>> ..
>> 2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
>> $ docker ps
>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>> 2b52ee072b1f postgres:10.10 "docker-entrypoint.s…" 2 seconds ago Up 1 seconds 5432/tcp blah_blah
>> $ guix shell xeyes -- xeyes
>> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> 0.0 MB will be downloaded
>> xeyes-1.1.2 11KiB 613KiB/s 00:00 [##################] 100.0%
>> The following derivation will be built:
>> /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>>
>> applying 4 grafts for xeyes-1.1.2 ...
>> building CA certificate bundle...
>> listing Emacs sub-directories...
>> building fonts directory...
>> building directory of Info manuals...
>> building profile with 1 package...
>> $ docker ps
>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>> $ exit
>>
>> First we see no docker containers are running, then we start postgres-10
>> from docker hub, we see its container is running, then we do something
>> using guix-shell on an application *not already available on this
>> system*, and now the container died. This does not work the second time
>> when the "derivation" is already "built".
>
> Are you still able to reproduce this using the new version of docker
> packaged in Guix?

Yes, same problem after a guix pull and guix system reconfigure just now.

$ guix describe
Generation 72 Jul 12 2022 16:11:38 (current)
guix 9173cb5
branch: master
commit: 9173cb522ddc4f31f21948cee3fb214fd67ef616

Cheers,
Remco
R
R
Remco van 't Veer wrote on 9 Feb 2023 13:26
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
875ycb6n3w.fsf@remworks.net
I think I know what is causing the issue. Both the "standard" mysql and
postgres containers use user-id 999 to run the database service (this
seems like a common practice because the redis container is configured
similarly). That user-id is also configured as guixbuilder01 so I guess
the guix daemon is killing those when processes when it finishes doing
builds.

Does that make sense? If so can guix daemon be fixed to be a tad more
gentile to the processes not spawned on its behalf?


2022/07/12 16:37, Remco van 't Veer:

Toggle quote (58 lines)
> 2022/07/12 09:48, Maxim Cournoyer:
>
>> Hi,
>>
>> Remco van 't Veer <remco@remworks.net> writes:
>>
>>> On a Guix system host, some running docker containers are stopped when
>>> doing guix install or other guix operations like shell. I noticed this
>>> happing to mysql and postgres containers but an elasticsearch container
>>> just keeps running.
>>>
>>> Here's an example session:
>>>
>>> $ docker ps
>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>> $ docker run -d postgres:10.10
>>> ..
>>> 2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
>>> $ docker ps
>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>> 2b52ee072b1f postgres:10.10 "docker-entrypoint.s…" 2 seconds ago Up 1 seconds 5432/tcp blah_blah
>>> $ guix shell xeyes -- xeyes
>>> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>>> 0.0 MB will be downloaded
>>> xeyes-1.1.2 11KiB 613KiB/s 00:00 [##################] 100.0%
>>> The following derivation will be built:
>>> /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>>>
>>> applying 4 grafts for xeyes-1.1.2 ...
>>> building CA certificate bundle...
>>> listing Emacs sub-directories...
>>> building fonts directory...
>>> building directory of Info manuals...
>>> building profile with 1 package...
>>> $ docker ps
>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>> $ exit
>>>
>>> First we see no docker containers are running, then we start postgres-10
>>> from docker hub, we see its container is running, then we do something
>>> using guix-shell on an application *not already available on this
>>> system*, and now the container died. This does not work the second time
>>> when the "derivation" is already "built".
>>
>> Are you still able to reproduce this using the new version of docker
>> packaged in Guix?
>
> Yes, same problem after a guix pull and guix system reconfigure just now.
>
> $ guix describe
> Generation 72 Jul 12 2022 16:11:38 (current)
> guix 9173cb5
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: 9173cb522ddc4f31f21948cee3fb214fd67ef616
>
> Cheers,
> Remco
R
R
Remco van 't Veer wrote on 19 May 2023 17:50
(address . 55358@debbugs.gnu.org)
878rdk8gm9.fsf@remworks.net
Hi Maxim and Zimoun,

2023/02/09 13:26, Remco van 't Veer:

Toggle quote (7 lines)
> I think I know what is causing the issue. Both the "standard" mysql and
> postgres containers use user-id 999 to run the database service (this
> seems like a common practice because the redis container is configured
> similarly). That user-id is also configured as guixbuilder01 so I guess
> the guix daemon is killing those when processes when it finishes doing
> builds.

I found a solution / workaround for this problem by using
"userns-remap". This feature allows the remapping of uids and guids to
different ranges. I tried it by hacking the required files into my
etc-directory and it works; guix no long kills my database containers.

I'd like to add this feature to docker-service-type having a new
configuration option named enable-userns-remap? which introduces a new
user and group (both named dockremap) to do the remapping by adding some
configurable number to the uids and guids of the running container. In
/etc/subuid and /etc/subgid it would look like:

dockremap:100000:65536

documentation about this.

WDYT?

Cheers,
Remco


--
C
(name . Remco van 't Veer)(address . remco@remworks.net)
87fs7st0m3.fsf@riseup.net
Remco van 't Veer <remco@remworks.net> writes:

Toggle quote (32 lines)
> Hi Maxim and Zimoun,
>
> 2023/02/09 13:26, Remco van 't Veer:
>
>> I think I know what is causing the issue. Both the "standard" mysql and
>> postgres containers use user-id 999 to run the database service (this
>> seems like a common practice because the redis container is configured
>> similarly). That user-id is also configured as guixbuilder01 so I guess
>> the guix daemon is killing those when processes when it finishes doing
>> builds.
>
> I found a solution / workaround for this problem by using
> "userns-remap". This feature allows the remapping of uids and guids to
> different ranges. I tried it by hacking the required files into my
> etc-directory and it works; guix no long kills my database containers.
>
> I'd like to add this feature to docker-service-type having a new
> configuration option named enable-userns-remap? which introduces a new
> user and group (both named dockremap) to do the remapping by adding some
> configurable number to the uids and guids of the running container. In
> /etc/subuid and /etc/subgid it would look like:
>
> dockremap:100000:65536
>
> See https://docs.docker.com/engine/security/userns-remap/ for
> documentation about this.
>
> WDYT?
>
> Cheers,
> Remco

The rootless podman example that was shared a few months ago could be
relevant to this, since that also adds a subuid/subgid mapping.
R
R
Remco van 't Veer wrote on 23 May 2023 09:49
[PATCH] services: docker: Add 'enable-userns-remap?' argument.
(address . 55358@debbugs.gnu.org)
de9934e88bf492bc64bd6db330646290eff4fd75.1684828084.git.remco@remworks.net
* gnu/services/docker.scm (docker-configuration): Define the argument.
* gnu/services/docker.scm (docker-shepherd-service): Use it.
* doc/guix.texi (Docker Service): Document it.
---
doc/guix.texi | 27 ++++++++++++++++++++++++++-
gnu/services/docker.scm | 28 +++++++++++++++++++++++++++-
2 files changed, 53 insertions(+), 2 deletions(-)

Toggle diff (130 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index f4cca66d76..ae185ced61 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -100,7 +100,7 @@
Copyright @copyright{} 2021 muradm@*
Copyright @copyright{} 2021, 2022 Andrew Tropin@*
Copyright @copyright{} 2021 Sarah Morgensen@*
-Copyright @copyright{} 2022 Remco van 't Veer@*
+Copyright @copyright{} 2022, 2023 Remco van 't Veer@*
Copyright @copyright{} 2022 Aleksandr Vityazev@*
Copyright @copyright{} 2022 Philip M@sup{c}Grath@*
Copyright @copyright{} 2022 Karl Hallsby@*
@@ -38533,6 +38533,31 @@ Miscellaneous Services
@item @code{enable-iptables?} (default @code{#t})
Enable or disable the addition of iptables rules.
+@item @code{enable-userns-remap?} (default @code{#f})
+Enable remapping and subordinate user and group IDs.
+
+A system user account named @code{dockremap} and user group named
+@code{dockremap} will be created. They must be mapped using the
+@file{/etc/subuid} and @file{/etc/subguid} files otherwise docker fail
+to startup.
+
+Here's an example service to setup both files:
+
+@lisp
+(simple-service
+ 'subuid-subgid etc-service-type
+ (list `("subuid"
+ ,(plain-file "subuid"
+ "dockremap:65536:65536\n"))
+ `("subgid"
+ ,(plain-file "subgid"
+ "dockremap:65536:65536\n"))))
+@end lisp
+
+The above will remap to UID 0 (root) to 65536, UID 1 to 65537 etc. For
+more information regarding the format of these files, consult
+@command{man 5 subuid} and @command{man 5 subgid}.
+
@item @code{environment-variables} (default: @code{()})
List of environment variables to set for @command{dockerd}.
diff --git a/gnu/services/docker.scm b/gnu/services/docker.scm
index 741bab5a8c..e138a6be7e 100644
--- a/gnu/services/docker.scm
+++ b/gnu/services/docker.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
;;; Copyright © 2020 Jesse Dowell <jessedowell@gmail.com>
;;; Copyright © 2021 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2023 Remco van 't Veer <remco@remworks.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -29,6 +30,7 @@ (define-module (gnu services docker)
#:use-module (gnu services shepherd)
#:use-module (gnu system setuid)
#:use-module (gnu system shadow)
+ #:use-module (gnu packages admin)
#:use-module (gnu packages docker)
#:use-module (gnu packages linux) ;singularity
#:use-module (guix records)
@@ -62,6 +64,9 @@ (define-configuration docker-configuration
(enable-iptables?
(boolean #t)
"Enable addition of iptables rules (enabled by default).")
+ (enable-userns-remap?
+ (boolean #f)
+ "Enable remapping and subordinate user and group IDs (disabled by default).")
(environment-variables
(list '())
"Environment variables to set for dockerd")
@@ -107,6 +112,7 @@ (define (docker-shepherd-service config)
(let* ((docker (docker-configuration-docker config))
(enable-proxy? (docker-configuration-enable-proxy? config))
(enable-iptables? (docker-configuration-enable-iptables? config))
+ (enable-userns-remap? (docker-configuration-enable-userns-remap? config))
(environment-variables (docker-configuration-environment-variables config))
(proxy (docker-configuration-proxy config))
(debug? (docker-configuration-debug? config)))
@@ -135,6 +141,9 @@ (define (docker-shepherd-service config)
#~(string-append
"--userland-proxy-path=" #$proxy "/bin/proxy"))
'("--userland-proxy=false"))
+ #$@(if enable-userns-remap?
+ '("--userns-remap=dockremap")
+ '())
(if #$enable-iptables?
"--iptables"
"--iptables=false")
@@ -145,6 +154,18 @@ (define (docker-shepherd-service config)
#:log-file "/var/log/docker.log"))
(stop #~(make-kill-destructor)))))
+(define %docker-remap-user-group
+ (user-group (name "dockremap")
+ (system? #t)))
+
+(define %docker-remap-user-account
+ (user-account (name "dockremap")
+ (group "dockremap")
+ (system? #t)
+ (comment "Docker user namespace remap user")
+ (home-directory "/var/empty")
+ (shell (file-append shadow "/sbin/nologin"))))
+
(define docker-service-type
(service-type (name 'docker)
(description "Provide capability to run Docker application
@@ -161,7 +182,12 @@ (define docker-service-type
(list (containerd-shepherd-service config)
(docker-shepherd-service config))))
(service-extension account-service-type
- (const %docker-accounts))))
+ (lambda (config)
+ (if (docker-configuration-enable-userns-remap? config)
+ (cons* %docker-remap-user-group
+ %docker-remap-user-account
+ %docker-accounts)
+ %docker-accounts)))))
(default-value (docker-configuration))))

base-commit: 849286ba66c96534bddc04df1a47d5692cbc977e
--
2.40.1
R
R
Remco van 't Veer wrote on 23 May 2023 09:53
Re: bug#55358: docker containers stopped when doing guix install or guix shell
(name . Csepp)(address . raingloom@riseup.net)
87ilcjmqkg.fsf@remworks.net
Hi Csepp,

2023/05/20 00:29, Csepp:

Toggle quote (37 lines)
> Remco van 't Veer <remco@remworks.net> writes:
>
>> Hi Maxim and Zimoun,
>>
>> 2023/02/09 13:26, Remco van 't Veer:
>>
>>> I think I know what is causing the issue. Both the "standard" mysql and
>>> postgres containers use user-id 999 to run the database service (this
>>> seems like a common practice because the redis container is configured
>>> similarly). That user-id is also configured as guixbuilder01 so I guess
>>> the guix daemon is killing those when processes when it finishes doing
>>> builds.
>>
>> I found a solution / workaround for this problem by using
>> "userns-remap". This feature allows the remapping of uids and guids to
>> different ranges. I tried it by hacking the required files into my
>> etc-directory and it works; guix no long kills my database containers.
>>
>> I'd like to add this feature to docker-service-type having a new
>> configuration option named enable-userns-remap? which introduces a new
>> user and group (both named dockremap) to do the remapping by adding some
>> configurable number to the uids and guids of the running container. In
>> /etc/subuid and /etc/subgid it would look like:
>>
>> dockremap:100000:65536
>>
>> See https://docs.docker.com/engine/security/userns-remap/ for
>> documentation about this.
>>
>> WDYT?
>>
>> Cheers,
>> Remco
>
> The rootless podman example that was shared a few months ago could be
> relevant to this, since that also adds a subuid/subgid mapping.

Thanks! Borrowed that.

For future reference:


Cheers,
Remco
?