docker containers stopped when doing guix install or guix shell

  • Open
  • quality assurance status badge
Details
3 participants
  • Maxim Cournoyer
  • Csepp
  • Remco van 't Veer
Owner
unassigned
Submitted by
Remco van 't Veer
Severity
normal
R
R
Remco van 't Veer wrote on 11 May 2022 09:12
(address . bug-guix@gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
87ilqch79l.fsf@remworks.net
On a Guix system host, some running docker containers are stopped when
doing guix install or other guix operations like shell. I noticed this
happing to mysql and postgres containers but an elasticsearch container
just keeps running.

Here's an example session:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker run -d postgres:10.10
..
2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2b52ee072b1f postgres:10.10 "docker-entrypoint.s…" 2 seconds ago Up 1 seconds 5432/tcp blah_blah
$ guix shell xeyes -- xeyes
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0.0 MB will be downloaded
xeyes-1.1.2 11KiB 613KiB/s 00:00 [##################] 100.0%
The following derivation will be built:
/gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv

applying 4 grafts for xeyes-1.1.2 ...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ exit

First we see no docker containers are running, then we start postgres-10
from docker hub, we see its container is running, then we do something
using guix-shell on an application *not already available on this
system*, and now the container died. This does not work the second time
when the "derivation" is already "built".

Cheers,
Remco
M
M
Maxim Cournoyer wrote on 12 Jul 2022 15:48
(name . Remco van 't Veer)(address . remco@remworks.net)
87mtde8mrr.fsf@gmail.com
Hi,

Remco van 't Veer <remco@remworks.net> writes:

Toggle quote (38 lines)
> On a Guix system host, some running docker containers are stopped when
> doing guix install or other guix operations like shell. I noticed this
> happing to mysql and postgres containers but an elasticsearch container
> just keeps running.
>
> Here's an example session:
>
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> $ docker run -d postgres:10.10
> ..
> 2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> 2b52ee072b1f postgres:10.10 "docker-entrypoint.s…" 2 seconds ago Up 1 seconds 5432/tcp blah_blah
> $ guix shell xeyes -- xeyes
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> 0.0 MB will be downloaded
> xeyes-1.1.2 11KiB 613KiB/s 00:00 [##################] 100.0%
> The following derivation will be built:
> /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>
> applying 4 grafts for xeyes-1.1.2 ...
> building CA certificate bundle...
> listing Emacs sub-directories...
> building fonts directory...
> building directory of Info manuals...
> building profile with 1 package...
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> $ exit
>
> First we see no docker containers are running, then we start postgres-10
> from docker hub, we see its container is running, then we do something
> using guix-shell on an application *not already available on this
> system*, and now the container died. This does not work the second time
> when the "derivation" is already "built".

Are you still able to reproduce this using the new version of docker
packaged in Guix?

Thanks,

Maxim
R
R
Remco van 't Veer wrote on 12 Jul 2022 16:37
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87h73m9z3f.fsf@remworks.net
2022/07/12 09:48, Maxim Cournoyer:

Toggle quote (45 lines)
> Hi,
>
> Remco van 't Veer <remco@remworks.net> writes:
>
>> On a Guix system host, some running docker containers are stopped when
>> doing guix install or other guix operations like shell. I noticed this
>> happing to mysql and postgres containers but an elasticsearch container
>> just keeps running.
>>
>> Here's an example session:
>>
>> $ docker ps
>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>> $ docker run -d postgres:10.10
>> ..
>> 2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
>> $ docker ps
>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>> 2b52ee072b1f postgres:10.10 "docker-entrypoint.s…" 2 seconds ago Up 1 seconds 5432/tcp blah_blah
>> $ guix shell xeyes -- xeyes
>> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> 0.0 MB will be downloaded
>> xeyes-1.1.2 11KiB 613KiB/s 00:00 [##################] 100.0%
>> The following derivation will be built:
>> /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>>
>> applying 4 grafts for xeyes-1.1.2 ...
>> building CA certificate bundle...
>> listing Emacs sub-directories...
>> building fonts directory...
>> building directory of Info manuals...
>> building profile with 1 package...
>> $ docker ps
>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>> $ exit
>>
>> First we see no docker containers are running, then we start postgres-10
>> from docker hub, we see its container is running, then we do something
>> using guix-shell on an application *not already available on this
>> system*, and now the container died. This does not work the second time
>> when the "derivation" is already "built".
>
> Are you still able to reproduce this using the new version of docker
> packaged in Guix?

Yes, same problem after a guix pull and guix system reconfigure just now.

$ guix describe
Generation 72 Jul 12 2022 16:11:38 (current)
guix 9173cb5
branch: master
commit: 9173cb522ddc4f31f21948cee3fb214fd67ef616

Cheers,
Remco
R
R
Remco van 't Veer wrote on 9 Feb 2023 13:26
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
875ycb6n3w.fsf@remworks.net
I think I know what is causing the issue. Both the "standard" mysql and
postgres containers use user-id 999 to run the database service (this
seems like a common practice because the redis container is configured
similarly). That user-id is also configured as guixbuilder01 so I guess
the guix daemon is killing those when processes when it finishes doing
builds.

Does that make sense? If so can guix daemon be fixed to be a tad more
gentile to the processes not spawned on its behalf?


2022/07/12 16:37, Remco van 't Veer:

Toggle quote (58 lines)
> 2022/07/12 09:48, Maxim Cournoyer:
>
>> Hi,
>>
>> Remco van 't Veer <remco@remworks.net> writes:
>>
>>> On a Guix system host, some running docker containers are stopped when
>>> doing guix install or other guix operations like shell. I noticed this
>>> happing to mysql and postgres containers but an elasticsearch container
>>> just keeps running.
>>>
>>> Here's an example session:
>>>
>>> $ docker ps
>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>> $ docker run -d postgres:10.10
>>> ..
>>> 2b52ee072b1f5584cae597afb033cdcc0e560bbe9145b17b41502c204034e60b
>>> $ docker ps
>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>> 2b52ee072b1f postgres:10.10 "docker-entrypoint.s…" 2 seconds ago Up 1 seconds 5432/tcp blah_blah
>>> $ guix shell xeyes -- xeyes
>>> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>>> 0.0 MB will be downloaded
>>> xeyes-1.1.2 11KiB 613KiB/s 00:00 [##################] 100.0%
>>> The following derivation will be built:
>>> /gnu/store/xc002hxl4g8mskqmpm0grsk8s45m91gz-profile.drv
>>>
>>> applying 4 grafts for xeyes-1.1.2 ...
>>> building CA certificate bundle...
>>> listing Emacs sub-directories...
>>> building fonts directory...
>>> building directory of Info manuals...
>>> building profile with 1 package...
>>> $ docker ps
>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>> $ exit
>>>
>>> First we see no docker containers are running, then we start postgres-10
>>> from docker hub, we see its container is running, then we do something
>>> using guix-shell on an application *not already available on this
>>> system*, and now the container died. This does not work the second time
>>> when the "derivation" is already "built".
>>
>> Are you still able to reproduce this using the new version of docker
>> packaged in Guix?
>
> Yes, same problem after a guix pull and guix system reconfigure just now.
>
> $ guix describe
> Generation 72 Jul 12 2022 16:11:38 (current)
> guix 9173cb5
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: 9173cb522ddc4f31f21948cee3fb214fd67ef616
>
> Cheers,
> Remco
R
R
Remco van 't Veer wrote on 19 May 2023 17:50
(address . 55358@debbugs.gnu.org)
878rdk8gm9.fsf@remworks.net
Hi Maxim and Zimoun,

2023/02/09 13:26, Remco van 't Veer:

Toggle quote (7 lines)
> I think I know what is causing the issue. Both the "standard" mysql and
> postgres containers use user-id 999 to run the database service (this
> seems like a common practice because the redis container is configured
> similarly). That user-id is also configured as guixbuilder01 so I guess
> the guix daemon is killing those when processes when it finishes doing
> builds.

I found a solution / workaround for this problem by using
"userns-remap". This feature allows the remapping of uids and guids to
different ranges. I tried it by hacking the required files into my
etc-directory and it works; guix no long kills my database containers.

I'd like to add this feature to docker-service-type having a new
configuration option named enable-userns-remap? which introduces a new
user and group (both named dockremap) to do the remapping by adding some
configurable number to the uids and guids of the running container. In
/etc/subuid and /etc/subgid it would look like:

dockremap:100000:65536

documentation about this.

WDYT?

Cheers,
Remco


--
C
(name . Remco van 't Veer)(address . remco@remworks.net)
87fs7st0m3.fsf@riseup.net
Remco van 't Veer <remco@remworks.net> writes:

Toggle quote (32 lines)
> Hi Maxim and Zimoun,
>
> 2023/02/09 13:26, Remco van 't Veer:
>
>> I think I know what is causing the issue. Both the "standard" mysql and
>> postgres containers use user-id 999 to run the database service (this
>> seems like a common practice because the redis container is configured
>> similarly). That user-id is also configured as guixbuilder01 so I guess
>> the guix daemon is killing those when processes when it finishes doing
>> builds.
>
> I found a solution / workaround for this problem by using
> "userns-remap". This feature allows the remapping of uids and guids to
> different ranges. I tried it by hacking the required files into my
> etc-directory and it works; guix no long kills my database containers.
>
> I'd like to add this feature to docker-service-type having a new
> configuration option named enable-userns-remap? which introduces a new
> user and group (both named dockremap) to do the remapping by adding some
> configurable number to the uids and guids of the running container. In
> /etc/subuid and /etc/subgid it would look like:
>
> dockremap:100000:65536
>
> See https://docs.docker.com/engine/security/userns-remap/ for
> documentation about this.
>
> WDYT?
>
> Cheers,
> Remco

The rootless podman example that was shared a few months ago could be
relevant to this, since that also adds a subuid/subgid mapping.
R
R
Remco van 't Veer wrote on 23 May 2023 09:49
[PATCH] services: docker: Add 'enable-userns-remap?' argument.
(address . 55358@debbugs.gnu.org)
de9934e88bf492bc64bd6db330646290eff4fd75.1684828084.git.remco@remworks.net
* gnu/services/docker.scm (docker-configuration): Define the argument.
* gnu/services/docker.scm (docker-shepherd-service): Use it.
* doc/guix.texi (Docker Service): Document it.
---
doc/guix.texi | 27 ++++++++++++++++++++++++++-
gnu/services/docker.scm | 28 +++++++++++++++++++++++++++-
2 files changed, 53 insertions(+), 2 deletions(-)

Toggle diff (130 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index f4cca66d76..ae185ced61 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -100,7 +100,7 @@
Copyright @copyright{} 2021 muradm@*
Copyright @copyright{} 2021, 2022 Andrew Tropin@*
Copyright @copyright{} 2021 Sarah Morgensen@*
-Copyright @copyright{} 2022 Remco van 't Veer@*
+Copyright @copyright{} 2022, 2023 Remco van 't Veer@*
Copyright @copyright{} 2022 Aleksandr Vityazev@*
Copyright @copyright{} 2022 Philip M@sup{c}Grath@*
Copyright @copyright{} 2022 Karl Hallsby@*
@@ -38533,6 +38533,31 @@ Miscellaneous Services
@item @code{enable-iptables?} (default @code{#t})
Enable or disable the addition of iptables rules.
+@item @code{enable-userns-remap?} (default @code{#f})
+Enable remapping and subordinate user and group IDs.
+
+A system user account named @code{dockremap} and user group named
+@code{dockremap} will be created. They must be mapped using the
+@file{/etc/subuid} and @file{/etc/subguid} files otherwise docker fail
+to startup.
+
+Here's an example service to setup both files:
+
+@lisp
+(simple-service
+ 'subuid-subgid etc-service-type
+ (list `("subuid"
+ ,(plain-file "subuid"
+ "dockremap:65536:65536\n"))
+ `("subgid"
+ ,(plain-file "subgid"
+ "dockremap:65536:65536\n"))))
+@end lisp
+
+The above will remap to UID 0 (root) to 65536, UID 1 to 65537 etc. For
+more information regarding the format of these files, consult
+@command{man 5 subuid} and @command{man 5 subgid}.
+
@item @code{environment-variables} (default: @code{()})
List of environment variables to set for @command{dockerd}.
diff --git a/gnu/services/docker.scm b/gnu/services/docker.scm
index 741bab5a8c..e138a6be7e 100644
--- a/gnu/services/docker.scm
+++ b/gnu/services/docker.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
;;; Copyright © 2020 Jesse Dowell <jessedowell@gmail.com>
;;; Copyright © 2021 Brice Waegeneire <brice@waegenei.re>
+;;; Copyright © 2023 Remco van 't Veer <remco@remworks.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -29,6 +30,7 @@ (define-module (gnu services docker)
#:use-module (gnu services shepherd)
#:use-module (gnu system setuid)
#:use-module (gnu system shadow)
+ #:use-module (gnu packages admin)
#:use-module (gnu packages docker)
#:use-module (gnu packages linux) ;singularity
#:use-module (guix records)
@@ -62,6 +64,9 @@ (define-configuration docker-configuration
(enable-iptables?
(boolean #t)
"Enable addition of iptables rules (enabled by default).")
+ (enable-userns-remap?
+ (boolean #f)
+ "Enable remapping and subordinate user and group IDs (disabled by default).")
(environment-variables
(list '())
"Environment variables to set for dockerd")
@@ -107,6 +112,7 @@ (define (docker-shepherd-service config)
(let* ((docker (docker-configuration-docker config))
(enable-proxy? (docker-configuration-enable-proxy? config))
(enable-iptables? (docker-configuration-enable-iptables? config))
+ (enable-userns-remap? (docker-configuration-enable-userns-remap? config))
(environment-variables (docker-configuration-environment-variables config))
(proxy (docker-configuration-proxy config))
(debug? (docker-configuration-debug? config)))
@@ -135,6 +141,9 @@ (define (docker-shepherd-service config)
#~(string-append
"--userland-proxy-path=" #$proxy "/bin/proxy"))
'("--userland-proxy=false"))
+ #$@(if enable-userns-remap?
+ '("--userns-remap=dockremap")
+ '())
(if #$enable-iptables?
"--iptables"
"--iptables=false")
@@ -145,6 +154,18 @@ (define (docker-shepherd-service config)
#:log-file "/var/log/docker.log"))
(stop #~(make-kill-destructor)))))
+(define %docker-remap-user-group
+ (user-group (name "dockremap")
+ (system? #t)))
+
+(define %docker-remap-user-account
+ (user-account (name "dockremap")
+ (group "dockremap")
+ (system? #t)
+ (comment "Docker user namespace remap user")
+ (home-directory "/var/empty")
+ (shell (file-append shadow "/sbin/nologin"))))
+
(define docker-service-type
(service-type (name 'docker)
(description "Provide capability to run Docker application
@@ -161,7 +182,12 @@ (define docker-service-type
(list (containerd-shepherd-service config)
(docker-shepherd-service config))))
(service-extension account-service-type
- (const %docker-accounts))))
+ (lambda (config)
+ (if (docker-configuration-enable-userns-remap? config)
+ (cons* %docker-remap-user-group
+ %docker-remap-user-account
+ %docker-accounts)
+ %docker-accounts)))))
(default-value (docker-configuration))))

base-commit: 849286ba66c96534bddc04df1a47d5692cbc977e
--
2.40.1
R
R
Remco van 't Veer wrote on 23 May 2023 09:53
Re: bug#55358: docker containers stopped when doing guix install or guix shell
(name . Csepp)(address . raingloom@riseup.net)
87ilcjmqkg.fsf@remworks.net
Hi Csepp,

2023/05/20 00:29, Csepp:

Toggle quote (37 lines)
> Remco van 't Veer <remco@remworks.net> writes:
>
>> Hi Maxim and Zimoun,
>>
>> 2023/02/09 13:26, Remco van 't Veer:
>>
>>> I think I know what is causing the issue. Both the "standard" mysql and
>>> postgres containers use user-id 999 to run the database service (this
>>> seems like a common practice because the redis container is configured
>>> similarly). That user-id is also configured as guixbuilder01 so I guess
>>> the guix daemon is killing those when processes when it finishes doing
>>> builds.
>>
>> I found a solution / workaround for this problem by using
>> "userns-remap". This feature allows the remapping of uids and guids to
>> different ranges. I tried it by hacking the required files into my
>> etc-directory and it works; guix no long kills my database containers.
>>
>> I'd like to add this feature to docker-service-type having a new
>> configuration option named enable-userns-remap? which introduces a new
>> user and group (both named dockremap) to do the remapping by adding some
>> configurable number to the uids and guids of the running container. In
>> /etc/subuid and /etc/subgid it would look like:
>>
>> dockremap:100000:65536
>>
>> See https://docs.docker.com/engine/security/userns-remap/ for
>> documentation about this.
>>
>> WDYT?
>>
>> Cheers,
>> Remco
>
> The rootless podman example that was shared a few months ago could be
> relevant to this, since that also adds a subuid/subgid mapping.

Thanks! Borrowed that.

For future reference:


Cheers,
Remco
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 55358@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 55358
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch