Really relocatable binaries crash with Permission denied

The manual gives the following example of guix pack -RR:

guix pack -RR -S /mybin=bin bash

tar xf pack.tar.gz

./mybin/sh

This fails on my university’s server for students which uses Linux

container “VMs” with Ubuntu and has no user namespace support and Guix

is not installed. This single line is all output:

$ ./mybin/sh

sh: run.c:162: bind_mount: Unexpected error: Permission denied.

Note that

PROOT_NO_SECCOMP=1 ~/gnu/store/iyd2ikxadcp89j5919pwja6swnx00493-proot-static-5.1.0/bin/proot -w $(pwd | sed 's/${HOME}//') -r ${HOME} -b /proc /mybin/sh

works just fine (inspired by

https://guix-hpc.bordeaux.inria.fr/blog/2017/10/using-guix-without-being-root/).

For testing purposes, I compile the wrapper

gnu/packages/aux-files/run-in-namespace.c:

sed -i 's|@STORE_DIRECTORY@|/gnu/store|g' run-in-namespace.c

sed -i 's|@WRAPPED_PROGRAM@|/mybin/sh|g' run-in-namespace.c

gcc -std=gnu99 -static -O0 -g -Wall run-in-namespace.c

scp run-in-namespace.c a.out … # upload it to the university server

ssh …

gdb a.out

[…]

(gdb) break main

Breakpoint 1 at 0x401ea1: file run-in-namespace.c, line 260.

(gdb) run

Starting program: /home/f_pelz12/a.out

Breakpoint 1, main (argc=1, argv=0x7fffffffe818) at run-in-namespace.c:260

260 size = readlink ("/proc/self/exe", self, sizeof self - 1);

(gdb) next

261 assert (size > 0);

(gdb)

265 size_t index = strlen (self)

(gdb)

268 char *store = strdup (self);

(gdb)

269 store[index] = '\0';

(gdb)

277 if (strcmp (store, "/gnu/store") != 0

(gdb)

278 && lstat ("/mybin/sh", &statbuf) != 0)

(gdb)

283 char *new_root = mkdtemp (strdup ("/tmp/guix-exec-XXXXXX"));

(gdb)

284 char *new_store = concat (new_root, "/gnu/store");

(gdb)

285 char *cwd = get_current_dir_name ();

(gdb)

292 pid_t child = syscall (SYS_clone, SIGCHLD | CLONE_NEWNS | CLONE_NEWUSER,

(gdb)

[Detaching after fork from child process 12748]

294 switch (child)

(gdb) a.out: run-in-namespace.c:162: bind_mount: Unexpected error: Permission denied.

337 disallow_setgroups (child);

(gdb)

a.out: run-in-namespace.c:205: disallow_setgroups: Unexpected error: Permission denied.

Program received signal SIGABRT, Aborted.

0x000000000040796f in raise ()

I do not know how to break into the detached child’s bind_mount call,

so I am unable to give details on this bind_mount error (I do not know

if the bind_mount really is the cause of the crash; it is futile

anyway and the binary should just try proot after all and not crash

before). A breakpoint from `break bind_mount` is ignored. Can I get

more information out of this somehow?

For completeness:

$ uname -a

Linux tux6 4.15.18-14-pve #1 SMP PVE 4.15.18-38 (Tue, 30 Apr 2019 10:51:33 +0200) x86_64 x86_64 x86_64 GNU/Linux

Regards,

Florian

On Fri, May 10, 2019 at 12:01:36AM +0200, pelzflorian (Florian Pelz) wrote:

I think it should have been

sed -i 's|@STORE_DIRECTORY@|/gnu/store|g' run-in-namespace.c

sed -i 's|@WRAPPED_PROGRAM@|/gnu/store/qn1ax1fkj16x280m1rv7mcimfmn9l2pf-bash-4.4.23/bin/sh|g' run-in-namespace.c

echo '#define PROOT_PROGRAM "iyd2ikxadcp89j5919pwja6swnx00493-proot-static-5.1.0/bin/proot"' > new

cat run-in-namespace.c >> new

mv new run-in-namespace.c

gcc -std=gnu99 -static -O0 -g -Wall run-in-namespace.c

but it does not make a difference to the gdb output except the line

Regards,

Florian

Hello,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

That suggests the wrapper chose the user namespace method (not PRoot),

but that didn’t quite work.

Could you post the output of:

strace ./mybin/sh

?

TIA!

Ludo’.

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

My bad, this should be:

strace -f -o log ./mybin/sh

and then post the ‘log’ file (we need ‘-f’ because the problem happens

in the child process.)

Thanks in advance,

Ludo’.

On Mon, May 13, 2019 at 09:49:40AM +0200, Ludovic Courtès wrote:

Oh I did not know there is -f.

[f_pelz12@tux6 ~]$ strace -f -o log ./mybin/sh

sh: run.c:162: bind_mount: Unexpected error: Permission denied.

The log file is attached.

When I do not use -o log, the unexpected error is here:

[pid 36622] mount("//sys", "/tmp/guix-exec-85li6j/sys", 0x47e0c5, MS_RDONLY|MS_BIND|MS_REC, NULL) = -1 EACCES (Permission denied)

[pid 36622] openat(AT_FDCWD, "/tmp/guix-exec-85li6j/core", O_WRONLY|O_CREAT, 056306) = 4

[pid 36622] close(4) = 0

[pid 36622] mount("//core", "/tmp/guix-exec-85li6j/core", 0x47e0c5, MS_RDONLY|MS_BIND|MS_REC, NULL) = -1 EACCES (Permission denied)

[pid 36622] write(2, "sh: run.c:162: bind_mount: Unexp"..., 64sh: run.c:162: bind_mount: Unexpected error: Permission denied.

) = 64

[pid 36622] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4b024f4000

[pid 36622] rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0

[pid 36622] rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0

[pid 36622] getpid() = 36622

[pid 36622] gettid() = 36622

[pid 36622] tgkill(36622, 36622, SIGABRT) = 0

[pid 36622] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

[pid 36622] --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=36622, si_uid=24038} ---

[pid 36622] +++ killed by SIGABRT +++

Regards,

Florian

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

[...]

This is weird. On a machine without Guix and with “proper” user

namespace support, I see:

That is, all bind-mount operations in the child process, which lives in

a separate namespace, succeed.

Can you show the mount options of you root file system?

mount | grep 'on / '

What’s the exit code of this command:

guile -c '((@@ (guix scripts environment) assert-container-features))'

?

Thanks for helping out!

Ludo’.

On Mon, May 13, 2019 at 03:54:11PM +0200, Ludovic Courtès wrote:

[f_pelz12@tux6 ~]$ mount | grep 'on / '

rpool/data/subvol-161199-disk-0 on / type zfs (rw,noatime,xattr,posixacl)

Guix is not installed. Using a Guix git repository in ~/guix:

[f_pelz12@tux6 guix]$ guile -c '((@@ (guix scripts environment) assert-container-features))'

[…]

;;; In procedure scm_lreadr: guix/packages.scm:534:11: Unknown # object: #\~

ERROR: In procedure primitive-load-path:

In procedure scm_lreadr: guix/packages.scm:534:11: Unknown # object: #\~

The line in question is:

#~(begin

(use-modules (ice-9 ftw)

I do not see how to make it recognize gexps.

If I wanted to compile Guix myself, the configure script reports

various missing dependencies (guile-gnutls is among them). I could

ask the admin tomorrow if they could set up guix on a test “virtual

machine”/container.

I will instead now try this from gnu/build/linux-container.scm:

scheme@(guile-user)> (define (user-namespace-supported?)

"Return #t if user namespaces are supported on this system."

(file-exists? "/proc/self/ns/user"))

(define (unprivileged-user-namespace-supported?)

"Return #t if user namespaces can be created by unprivileged users."

(let ((userns-file "/proc/sys/kernel/unprivileged_userns_clone"))

(if (file-exists? userns-file)

(eqv? #\1 (call-with-input-file userns-file read-char))

#t)))

(define (setgroups-supported?)

"Return #t if the setgroups proc file, introduced in Linux-libre 3.19,

exists."

(file-exists? "/proc/self/setgroups"))

scheme@(guile-user)> (user-namespace-supported?)

$1 = #t

scheme@(guile-user)> (unprivileged-user-namespace-supported?)

$2 = #t

scheme@(guile-user)> (setgroups-supported?)

$3 = #t

Regards,

Florian

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

I suspect ZFS-on-Linux (right?) is doing something unusual here:

mount(2) specifies the following reasons for EACCESS, and I don’t see

anything that would apply:

What do the following commands do on this system?

Thanks,

Ludo’.

On Mon, May 13, 2019 at 10:39:21PM +0200, Ludovic Court�s wrote:

I suppose it is ZFS on Linux; it is Linux, I can ask the admins if it

could be something else.

[f_pelz12@tux6 ~]$ mkdir -p /tmp/test/lib

[f_pelz12@tux6 ~]$ unshare -mrf mount /lib /tmp/test/lib -o bind,readonly

unshare: cannot change root filesystem propagation: Permission denied

Thank *you*, Ludo! A working guix pack would be helpful for me.

Regards,

Florian

On Mon, May 13, 2019 at 10:45:24PM +0200, pelzflorian (Florian Pelz) wrote:

The admins have confirmed that they use “Proxmox on ZFS” (judging from

https://pve.proxmox.com/wiki/ZFS_on_Linux it is ZFS on Linux) and

they have confirmed that they have disabled user namespaces in their

Proxmox settings.

Regards,

Florian

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

User namespaces are orthogonal to file systems, but anyway it looks like

ZFS is refusing to let us do these things.

I don’t have any great option to offer. You could perhaps modify

run-in-namespace.c so that it doesn’t even try user namespaces and

instead goes directly to the PRoot option?

However working around this behavior of ZFS it not completely trivial

and I’m not sure we should put much energy to paper over non-standard

file system behavior.

Thoughts?

Ludo’.

On Tue, May 14, 2019 at 10:43:56PM +0200, Ludovic Courtès wrote:

Do I understand correctly that user namespaces are not really disabled

(?) but fail on ZFS? This seems strange, but a Web search for “zfs

user namespaces” shows other people having trouble with this

combination. The admins told me they had to disable user namespaces

because it caused some kind of trouble.

If ZFS makes user namespaces fail, then could run-un-namespace.c fall

back to PRoot when detecting ZFS, somehow?

Regards,

Florian

Hello Ludovic and Florian,

I cannot help here, just some thoughts

as you probably already know, Florian, ZFS is not supported in Linux for

various reasons, above all for a controversial licensing problem [1]

so using zfsonlinux (the ZFS Linux unofficial kernel module) is

basically calling for problems

Ludovic Courtès <ludo@gnu.org> writes:

[...]

it's not clearly stated there, I guess it's

I do not understand what this means: if namespaces are disabled **in

kernel** that whould be detected and guix relocatable binaries should

use PRoot by default: am I wrong?

If "disabled user namespace in Promox settings" means it have something

to do with ZFS filesystem settings, well: it's unorthodox at least :-)

I don't know if this have something to do with this bug, but:

ZFS is confused by user namespaces (uid/gid mapping) when used with acltype=posixacl

Florian: it should be solved but AFAIU it depends on the

kernel/zfsonlinux combination

Ludovic (and others): is it possible to add an option to "guix pack -RR"

(-RRF?!?) to force the use of PRoot for resulting relocated binaries?

I agree, this seems a zfsonlinux bug: Florian please can you report it

upstream to zfsonlinux?

[...]

HTH! Gio'

[1] https://www.fsf.org/licensing/zfs-and-linux

https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/

--

Giovanni Biscuolo

Xelera IT Infrastructures

Hi,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

Correct. Specifically, read-only bind mounts of (and to?) files that

reside on ZFS fail with EACCESS, which is normally “impossible.”

It would be great if you could ask the admins specifically what they did

in relation to user namespaces.

It’s code, so everything is possible :-), but like I wrote it’s a bit of

work, and it’s something that cannot happen (AFAIK) with file systems

that are part of Linux.

Thanks,

Ludo’.

On Wed, May 15, 2019 at 05:20:25PM +0200, Giovanni Biscuolo wrote:

I had forgotten. I remember now that I heard about this.

From a Guix point of view, I believe this maybe should be a

WONT-FIX/NOT-OUR-BUG. I will try and set up current ZFS 0.7.13 and

test if guix pack -RR works there in a week.

Feel free to skip this unless you are interested:

I asked the admins again. They are using Proxmox 5.4. They say they

have disabled user namespaces by commenting the corresponding line in

the Proxmox config file (but I am unsure if this just disables Linux

Container use of user namespaces or something). They use the ZFS from

Proxmox. I looked and found confirmation that this Proxmox uses

current ZFS 0.7.13 at:

I will try to reproduce on a private PC in a week, then I can report.

Regards,

Florian

Hello,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

Sounds good to me. :-)

Thanks,

Ludo’.

tags 35662 wontfix

close 35662