python can't find internal pip modules in environment from manifest

  • Done
  • quality assurance status badge
Details
2 participants
  • Jelle Licht
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Jelle Licht
Severity
normal
J
J
Jelle Licht wrote on 16 Feb 2021 21:58
(address . bug-guix@gnu.org)
86czwz7ms3.fsf@fsfe.org
Python can not load all pip-related python modules when loaded from a
manifest file, yet seems to work fine when loaded 'directly' as part of
a `guix environment'-invocation.


Provided that we have a file in `hi.py':
Toggle snippet (6 lines)
import pip
print("halfwaypoint")
import pip._internal.index.package_finder
print("I made it!")

and a `manifest.scm' with:

Toggle snippet (7 lines)
(use-modules (gnu) (guix packages)
(gnu packages python)
(gnu packages python-xyz))

(packages->manifest (list python python-pip))

I can get the expected output by running:
`guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
Toggle quote (3 lines)
> halfwaypoint
> I made it!

Yet, when I run the following:
`guix environment --pure -m manifest.scm -- python3 hi.py' =>

Toggle snippet (19 lines)
halfwaypoint
Traceback (most recent call last):
File "hi.py", line 4, in <module>
import pip._internal.index.package_finder
File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/__init__.py", line 40, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
from pip._internal.commands import (
File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
from pip._internal.commands.completion import CompletionCommand
File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
from pip._internal.cli.base_command import Command
File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 26, in <module>
from pip._internal.index import PackageFinder
ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' (/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/index/__init__.py)

Why is there a difference in the first place? Shouldn't both approaches work?

- Jelle
J
J
Jelle Licht wrote on 18 Feb 2021 14:49
(address . 46569@debbugs.gnu.org)
86zh015vvb.fsf@fsfe.org
Jelle Licht <jlicht@fsfe.org> writes:

Toggle quote (53 lines)
> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
> --8<---------------cut here---------------start------------->8---
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")
> --8<---------------cut here---------------end--------------->8---
>
> and a `manifest.scm' with:
>
> --8<---------------cut here---------------start------------->8---
> (use-modules (gnu) (guix packages)
> (gnu packages python)
> (gnu packages python-xyz))
>
> (packages->manifest (list python python-pip))
> --8<---------------cut here---------------end--------------->8---
>
> I can get the expected output by running:
> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>> halfwaypoint
>> I made it!
>
> Yet, when I run the following:
> `guix environment --pure -m manifest.scm -- python3 hi.py' =>
>
> --8<---------------cut here---------------start------------->8---
> halfwaypoint
> Traceback (most recent call last):
> File "hi.py", line 4, in <module>
> import pip._internal.index.package_finder
> File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/__init__.py", line 40, in <module>
> from pip._internal.cli.autocompletion import autocomplete
> File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
> from pip._internal.cli.main_parser import create_main_parser
> File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
> from pip._internal.commands import (
> File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
> from pip._internal.commands.completion import CompletionCommand
> File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
> from pip._internal.cli.base_command import Command
> File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 26, in <module>
> from pip._internal.index import PackageFinder
> ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' (/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/index/__init__.py)
> --8<---------------cut here---------------end--------------->8---
>
> Why is there a difference in the first place? Shouldn't both approaches work?

Using diffoscope, I found the (practical) difference:

$GUIX_ENVIRONMENT/bin/pip3 is
? ? -destination: /gnu/store/rz42ba0my9vrgbkjpkzr2drmnjk5ah50-python-3.8.2/bin/pip3
? ? +destination: /gnu/store/d75bwzbla5ybhs0mdw80qy94mawnhhsw-python-pip-20.2.4/bin/pip3

If you make sure that `python-pip' precedes the `python' package in the
manifest, things work out fine. Likewise for the following --ad-hoc
example:

This fails:
guix environment --ad-hoc python-pip python -- python3 hi.py

This works:
guix environment --ad-hoc python python-pip -- python3 hi.py

So it seems the order in which guix sees packages when building profiles
determines how collisions are resolved, good to know :-).

Is the python-included pip package supposed to be broken like this, though?
- Jelle
J
J
Jelle Licht wrote on 18 Feb 2021 17:37
(address . 46569@debbugs.gnu.org)
86o8gh5o2r.fsf@fsfe.org
Jelle Licht <jlicht@fsfe.org> writes:

Toggle quote (7 lines)
> Jelle Licht <jlicht@fsfe.org> writes:
> This fails:
> guix environment --ad-hoc python-pip python -- python3 hi.py
>
> This works:
> guix environment --ad-hoc python python-pip -- python3 hi.py

This should be the other way around, pasting error on my end.
M
M
Maxim Cournoyer wrote on 18 Feb 2021 19:05
(name . Jelle Licht)(address . jlicht@fsfe.org)(address . 46569@debbugs.gnu.org)
87sg5tuu84.fsf@gmail.com
Hi Jelle,

Jelle Licht <jlicht@fsfe.org> writes:

Toggle quote (27 lines)
> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
>
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")
>
>
> and a `manifest.scm' with:
>
> (use-modules (gnu) (guix packages)
> (gnu packages python)
> (gnu packages python-xyz))
>
> (packages->manifest (list python python-pip))
>
>
> I can get the expected output by running:
> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>> halfwaypoint
>> I made it!

It seems that two pip versions don't cohabit very well. The same
happens if you install pip via 'pip install --user pip' (e.g., to
~/.local/bin) and try to use it.

You could try using the pip version that comes bundled with Python
itself (e.g, not installing python-pip in the same profile); if it's the
only one, it should work.

HTH,

Maxim
J
J
Jelle Licht wrote on 18 Feb 2021 20:45
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 46569@debbugs.gnu.org)
86lfbl5fd9.fsf@fsfe.org
Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (39 lines)
> Hi Jelle,
>
> Jelle Licht <jlicht@fsfe.org> writes:
>
>> Python can not load all pip-related python modules when loaded from a
>> manifest file, yet seems to work fine when loaded 'directly' as part of
>> a `guix environment'-invocation.
>>
>>
>> Provided that we have a file in `hi.py':
>>
>> import pip
>> print("halfwaypoint")
>> import pip._internal.index.package_finder
>> print("I made it!")
>>
>>
>> and a `manifest.scm' with:
>>
>> (use-modules (gnu) (guix packages)
>> (gnu packages python)
>> (gnu packages python-xyz))
>>
>> (packages->manifest (list python python-pip))
>>
>>
>> I can get the expected output by running:
>> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>>> halfwaypoint
>>> I made it!
>
> It seems that two pip versions don't cohabit very well. The same
> happens if you install pip via 'pip install --user pip' (e.g., to
> ~/.local/bin) and try to use it.
>
> You could try using the pip version that comes bundled with Python
> itself (e.g, not installing python-pip in the same profile); if it's the
> only one, it should work.

It doesn't, though, with the exact same error message as above.

Does my example (hi.py) work for you?
M
M
Maxim Cournoyer wrote on 18 Feb 2021 22:18
(name . Jelle Licht)(address . jlicht@fsfe.org)(address . 46569-done@debbugs.gnu.org)
87h7m9ulag.fsf@gmail.com
Hello Jelle,

Jelle Licht <jlicht@fsfe.org> writes:

Toggle quote (12 lines)
> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
>
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")

The problem seems to be related to the version of pip used. The one
bundled with our Python on master is:

$ guix environment --pure --ad-hoc python -- pip3 --version
pip 19.2.3 from
/gnu/store/pw65c67dmzr2m9wbkj5mskhm6ljk6xal-profile/lib/python3.8/site-packages/pip
(python 3.8)

and

$ guix environment --pure --ad-hoc python -- python3 -c 'import pip._internal.index.package_finder'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named
'pip._internal.index.package_finder'; 'pip._internal.index' is not a
package

but

$ guix environment --pure --ad-hoc python python-pip -- pip3 --version
pip 20.2.4 from
/gnu/store/d75bwzbla5ybhs0mdw80qy94mawnhhsw-python-pip-20.2.4/lib/python3.8/site-packages/pip
(python 3.8)

and

$ guix environment --pure --ad-hoc python python-pip -- python3 -c 'import pip._internal.index.package_finder'
$ echo $?
0

HTH!

Closing.

Maxim
Closed
?