Python unable to find modules within a Singularity container created with guix pack

OpenSubmitted by Marek Felšöci.
Details
2 participants
  • Ludovic Courtès
  • Marek Felšöci
Owner
unassigned
Severity
normal
M
M
Marek Felšöci wrote on 14 Jan 15:20 +0100
(address . bug-guix@gnu.org)
67e7d225-f824-b691-7125-6b37abef8bb0@felsoci.sk
Hello folks,

Recently, I have come around this issue. I created a Singularity container using
the following `guix pack` command:

`guix pack -f squashfs bash coreutils python python-numpy`

Then, I run a bash shell within the container:

`singularity exec test.gz.squashfs bash`

Now, if I want to use numpy,

`python3 -m numpy`

it gives me the following error and the numpy module IS NOT found:

Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError:
'/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'
is not in list
/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No module named
numpy

However, if I do,

`export PYTHONPATH=$GUIX_PYTHONPATH`

the numpy module IS found:

Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError:
'/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'
is not in list
/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No module named
numpy.__main__; 'numpy' is a package and cannot be directly executed

Still, the sitecustomize error persists.

See the output of `guix describe` below (only the guix channel):

Pokolenie 3    14. január 2022 14:33:06    (súčasné)
  guix 5c5d9e5
    vetva: master
    úprava: 5c5d9e5a3208fa51351c6ad8267eba3d96fa2232

Thanks,

Marek
L
L
Ludovic Courtès wrote on 14 Jan 18:45 +0100
(name . Marek Felšöci)(address . marek@felsoci.sk)
87wnj22ouh.fsf@gnu.org
Hi Marek,

Marek Felšöci <marek@felsoci.sk> skribis:

Toggle quote (22 lines)
> Recently, I have come around this issue. I created a Singularity
> container using the following `guix pack` command:
>
> `guix pack -f squashfs bash coreutils python python-numpy`
>
> Then, I run a bash shell within the container:
>
> `singularity exec test.gz.squashfs bash`
>
> Now, if I want to use numpy,
>
> `python3 -m numpy`
>
> it gives me the following error and the numpy module IS NOT found:
>
> Error in sitecustomize; set PYTHONVERBOSE for traceback:
> ValueError:
> '/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'
> is not in list
> /gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No
> module named numpy

I can kinda reproduce via a relocatable pack (relocatable packs and
Singularity packs have in common that they use relative symlinks in
their profile, to allow for relocation):

guix pack -RR python python-numpy -S /bin=bin -S /etc=etc

Then I unpack the thing:

mkdir /tmp/pack; cd /tmp/pack; tar xf /gnu/store/…-pack.tar.gz

And from there I use the same trick as in
‘tests/guix-pack-relocatable.sh’ to “hide” /gnu/store:

Toggle snippet (16 lines)
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; . /tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan 1 1970, 00:00:01)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>> import os
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages")
['setuptools-56.0.0.dist-info', 'distutils-precedence.pth', '_distutils_hack', 'setuptools', 'pip-21.1.3.dist-info', 'sitecustomize.py', 'README.txt', 'pip', 'numpy-1.20.3-py3.9.egg-info', 'numpy', 'pkg_resources', '__pycache__']
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/numpy")
['__init__.pyi', 'doc', 'matlib.py', 'ctypeslib.pyi', 'linalg', 'ctypeslib.py', 'compat', 'typing', '__init__.pxd', 'lib', 'char.pyi', 'version.py', '__config__.py', 'core', '_distributor_init.py', 'distutils', 'tests', 'emath.pyi', '_pytesttester.py', 'fft', '__init__.cython-30.pxd', 'conftest.py', '__init__.py', 'py.typed', '_globals.py', 'rec.pyi', 'testing', 'LICENSE.txt', 'polynomial', 'f2py', 'matrixlib', 'random', 'ma', 'dual.py', 'setup.py', '__pycache__']

So in this case ‘GUIX_PYTHONPATH’ is correctly set, but it looks as
though ‘sitecustomize.py’ wasn’t loaded. (I don’t get the “Error in
sitecustomize” message that Marek sees though, weird!)

From Marek’s message, it would seem that somewhere file names are
constructed by appending “..”. Given Unix’s weird dot-dot semantics,
relative symlinks could be interfering:

Toggle snippet (16 lines)
$ ls -l /tmp/pack/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/
totalo 48
lrwxrwxrwx 1 ludo users 102 Jan 1 1970 _distutils_hack -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/_distutils_hack
lrwxrwxrwx 1 ludo users 111 Jan 1 1970 distutils-precedence.pth -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/distutils-precedence.pth
lrwxrwxrwx 1 ludo users 99 Jan 1 1970 numpy -> ../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy
lrwxrwxrwx 1 ludo users 121 Jan 1 1970 numpy-1.20.3-py3.9.egg-info -> ../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy-1.20.3-py3.9.egg-info
lrwxrwxrwx 1 ludo users 90 Jan 1 1970 pip -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip
lrwxrwxrwx 1 ludo users 107 Jan 1 1970 pip-21.1.3.dist-info -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip-21.1.3.dist-info
lrwxrwxrwx 1 ludo users 100 Jan 1 1970 pkg_resources -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pkg_resources
lrwxrwxrwx 1 ludo users 98 Jan 1 1970 __pycache__ -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/__pycache__
lrwxrwxrwx 1 ludo users 97 Jan 1 1970 README.txt -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/README.txt
lrwxrwxrwx 1 ludo users 97 Jan 1 1970 setuptools -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools
lrwxrwxrwx 1 ludo users 114 Jan 1 1970 setuptools-56.0.0.dist-info -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools-56.0.0.dist-info
lrwxrwxrwx 1 ludo users 103 Jan 1 1970 sitecustomize.py -> ../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/sitecustomize.py

Now look below how dot-dot resolution whereas “lexical” dot-dot
resolution dose the right thing:

Toggle snippet (16 lines)
$ ls -l /tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages
ls: ne eblas atingi '/tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages': Dosiero aŭ dosierujo ne ekzistas
$ ls -l /tmp/pack/gnu/store/p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages
totalo 40
dr-xr-xr-x 3 ludo users 4096 Jan 1 1970 _distutils_hack/
-r--r--r-- 1 ludo users 152 Jan 1 1970 distutils-precedence.pth
dr-xr-xr-x 5 ludo users 4096 Jan 1 1970 pip/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 pip-21.1.3.dist-info/
dr-xr-xr-x 6 ludo users 4096 Jan 1 1970 pkg_resources/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 __pycache__/
-r--r--r-- 1 ludo users 119 Jan 1 1970 README.txt
dr-xr-xr-x 7 ludo users 4096 Jan 1 1970 setuptools/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 setuptools-56.0.0.dist-info/
-r--r--r-- 1 ludo users 2172 Jan 1 1970 sitecustomize.py

In Python’s ‘site.py’, the ‘addsitedir’ method indirectly calls
‘os.path.abspath’, not sure if that is at fault.

I also see that ‘sys.path’ is in fact completely bogus, with “pack”
missing from every directory name:

Toggle snippet (10 lines)
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; . /tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan 1 1970, 00:00:01)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python39.zip', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/lib-dynload', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages']

So I think something somewhere is manipulating file names and gets
confused.

Does that ring a bell, Maxim?

Thanks,
Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 53258@debbugs.gnu.org