Python .egg files must not be compressed

  • Done
  • quality assurance status badge
Details
5 participants
  • Andreas Enge
  • Hartmut Goebel
  • Leo Famulari
  • Ludovic Courtès
  • Ricardo Wurmus
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
serious
L
L
Ludovic Courtès wrote on 7 Jun 2015 22:37
(address . bug-guix@gnu.org)
87zj4btfuo.fsf@gnu.org
The other day on IRC Ricardo and ??? noted that python-pillow
currently comes with a compressed egg. Because it is compressed, the
daemon’s conservative scanning fails to see what store items it refers
to; in particular Ricardo noted that on his machine, python-pillow
refers to a non-existent store item for OpenJPEG.

To fix that, python-build-system must be tweaked to ask for uncompressed
eggs. I tried the attached patch, which uses the ‘--always-unzip’
option of easyinstall.

Unfortunately, that option is unsupported by some setup.py, such as that
of setuptools itself.

What would be the right way to do that?

TIA. :-)

Ludo’.
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -60,14 +60,20 @@
#:allow-other-keys)
"Install a given Python package."
(let* ((out (assoc-ref outputs "out"))
- (params (append (list (string-append "--prefix=" out))
+ (params (cons* (string-append "--prefix=" out)
+
+ ;; Make sure the .egg files are kept unzipped,
+ ;; otherwise the daemon's conservative scanning may
+ ;; not find store references embedded in it.
+ "--always-unzip"
+
configure-flags))
(python-version (get-python-version (assoc-ref inputs "python")))
(old-path (getenv "PYTHONPATH"))
(add-path (string-append out "/lib/python" python-version
"/site-packages/")))
- ;; create the module installation directory and add it to PYTHONPATH
- ;; to make setuptools happy
+ ;; Create the module installation directory and add it to PYTHONPATH
+ ;; to make setuptools happy.
(mkdir-p add-path)
(setenv "PYTHONPATH"
(string-append (if old-path
L
L
Ludovic Courtès wrote on 1 Jul 2015 10:23
control message for bug #20765
(address . control@debbugs.gnu.org)
87h9poxp4b.fsf@gnu.org
severity 20765 serious
L
L
Ludovic Courtès wrote on 20 Jul 2015 23:22
Re: bug#20765: Python .egg files must not be compressed
(address . 20765@debbugs.gnu.org)
87zj2qcyn8.fsf@gnu.org
Commit e5358a6 disables egg compression by augmenting ‘setup.cfg’ for
python-pillow only.

After:

Toggle snippet (18 lines)
$ guix gc --references /gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/29s2a1hfc47qa292pf9kd0k8m9djd3hf-libwebp-0.4.3
/gnu/store/61c1h5zlxcjq55z13i1n7inm0d1nfv0w-libtiff-4.0.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/fsy5s8wn2l542q05jn54lzp9wx5l9fdj-freetype-2.6
/gnu/store/j4w3bddy32p14mrd1a9d80435y50s3yz-zlib-1.2.7
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/jyk2nxd922wmls7xarx0npy11cvi2848-lcms-2.6
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/mq8kb3z2qga5jb8vc3s832yp9pr4knm6-openjpeg-2.1.0
/gnu/store/n3xi1hzh43hz2i1qq78ajr1mrapif2pa-libjpeg-9a
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
/gnu/store/w29667jfv02s1hgmv0yp7nqyywvdv1fz-glibc-2.21
/gnu/store/z8xw06ns2xjc9v5iza2n7gprjhi7dda0-gcc-4.9.3-lib

Before:

Toggle snippet (9 lines)
$ guix gc --references $(guix build python-pillow)
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/imi7gz1779h6559d5kh8w94p7kqx5fqv-python-pillow-2.8.1
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4

So clearly, I’d expect python-pillow to be more reliable now.

Now the question is how many packages end up installing compressed eggs,
and whether we can generalize that solution.

I think we should try doing the ‘setup.cfg’ dance in
python-build-system.scm after 0.8.3.

Thoughts?

Ludo’.
R
R
Ricardo Wurmus wrote on 13 Oct 2015 15:38
Python .egg files must not be compressed
(address . 20765@debbugs.gnu.org)(address . ludo@gnu.org)
idjpp0izy5d.fsf@bimsb-sys02.mdc-berlin.net
I stumbled upon a similar problem when building other Python packages.
For ‘python-patsy’, for example, I found that an egg archive is produced
no matter what I put in ‘setup.cfg’. As far as I can tell, “zip_ok”
defaults to “0” in recent versions of setuptools, so I don’t think the
fix you used on ‘python-pillow’ would have much effect in general.

I could only coerce the build system to install plain files by adding a
build phase like this:

(add-after 'unpack 'prevent-generation-of-egg-archive
(lambda _
(substitute* "setup.py"
(("from setuptools import setup")
"from distutils.core import setup"))
#t))

It turns out that when ‘setuptools.setup’ is used egg archives are
built, when ‘distutils.core.setup’ is used, however, this does not
happen.

~~ Ricardo
L
L
Ludovic Courtès wrote on 13 Oct 2015 16:10
(name . Ricardo Wurmus)(address . ricardo.wurmus@mdc-berlin.de)(address . 20765@debbugs.gnu.org)
87612aq2pl.fsf@gnu.org
Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

Toggle quote (6 lines)
> I stumbled upon a similar problem when building other Python packages.
> For ‘python-patsy’, for example, I found that an egg archive is produced
> no matter what I put in ‘setup.cfg’. As far as I can tell, “zip_ok”
> defaults to “0” in recent versions of setuptools, so I don’t think the
> fix you used on ‘python-pillow’ would have much effect in general.

OK.

Toggle quote (14 lines)
> I could only coerce the build system to install plain files by adding a
> build phase like this:
>
> (add-after 'unpack 'prevent-generation-of-egg-archive
> (lambda _
> (substitute* "setup.py"
> (("from setuptools import setup")
> "from distutils.core import setup"))
> #t))
>
> It turns out that when ‘setuptools.setup’ is used egg archives are
> built, when ‘distutils.core.setup’ is used, however, this does not
> happen.

But are distutils.core and setuptools the same thing? Replacing one by
the other sounds a bit scary, no?

Still it would be nice to have a generic solution. Maybe we should
patch setuptools itself to change its defaults?

Ludo’.
R
R
Ricardo Wurmus wrote on 13 Oct 2015 16:32
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 20765@debbugs.gnu.org)
idjk2qqzvnw.fsf@bimsb-sys02.mdc-berlin.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (17 lines)
>> I could only coerce the build system to install plain files by adding a
>> build phase like this:
>>
>> (add-after 'unpack 'prevent-generation-of-egg-archive
>> (lambda _
>> (substitute* "setup.py"
>> (("from setuptools import setup")
>> "from distutils.core import setup"))
>> #t))
>>
>> It turns out that when ‘setuptools.setup’ is used egg archives are
>> built, when ‘distutils.core.setup’ is used, however, this does not
>> happen.
>
> But are distutils.core and setuptools the same thing? Replacing one by
> the other sounds a bit scary, no?

I’m not familiar with Python packaging. From what I understand,
distutils has much fewer features than setuptools; but in the few cases
that I tried replacing it did work out just fine.

Toggle quote (3 lines)
> Still it would be nice to have a generic solution. Maybe we should
> patch setuptools itself to change its defaults?

This would probably be best, but I don’t understand setuptools enough to
propose a patch.

~~ Ricardo
L
L
Leo Famulari wrote on 9 Feb 2016 05:54
(no subject)
(address . 20765@debbugs.gnu.org)
20160209045442.GA29136@jasmine
Another reason to not compress Python eggs is that the contents of the
zip archives introduce non-determinism through their timestamps.

You can see it in action:
$ git checkout fe17fb4a2c897fd9186f91887f5af63dd00d227a
$ ./pre-inst-env guix build --rounds=2 acme

If you save the output between rounds, you can use diffoscope to confirm
that the differences between the rounds are the timestamps of the
contents of the eggs.
L
L
Ludovic Courtès wrote on 9 Feb 2016 22:37
(name . Leo Famulari)(address . leo@famulari.name)(address . 20765@debbugs.gnu.org)
87oabppnts.fsf@gnu.org
Leo Famulari <leo@famulari.name> skribis:

Toggle quote (3 lines)
> Another reason to not compress Python eggs is that the contents of the
> zip archives introduce non-determinism through their timestamps.

Indeed, good catch!

Ludo’.
A
A
Andreas Enge wrote on 6 Apr 2016 20:17
(name . Ludovic Courtès)(address . ludo@gnu.org)
20160406181724.GA18933@bugis
I have stumbled over the same issue for another python package:
python-pkgconfig (just committed) makes a system call to pkg-config.
I rewrote the system call to use an absolute path, but the non-scanning
of the compressed egg means that no reference to pkg-config is returned.

Spending a week with Python wizards, I asked around and got the following
advice, implemented in the package as
#:configure-flags '("--single-version-externally-managed" "--root=/")
Our Python build system adds these flags only to the install phase.

We could do this automatically in our Python build system, but we would
need to make sure to add these flags only when python-setuptools or
python2-setuptools are part of the input, since the flags are not recognised
by the setup.py from distutils.

Another advice was to replace the call to "python setup.py install"
by "pip install .", which apparently is becoming the standard approach
of installing Python packages, not only over the Internet, but also locally
(for which the "." stands).

What do you think?

Andreas
L
L
Ludovic Courtès wrote on 6 Apr 2016 22:52
Compressed eggs (Python)
(name . Andreas Enge)(address . andreas@enge.fr)
87oa9m4g0b.fsf_-_@gnu.org
Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (4 lines)
> Spending a week with Python wizards, I asked around and got the following
> advice, implemented in the package as
> #:configure-flags '("--single-version-externally-managed" "--root=/")

Woow, I wouldn’t have guessed that this has something to do with .egg
compression. :-)

Toggle quote (7 lines)
> Our Python build system adds these flags only to the install phase.
>
> We could do this automatically in our Python build system, but we would
> need to make sure to add these flags only when python-setuptools or
> python2-setuptools are part of the input, since the flags are not recognised
> by the setup.py from distutils.

OK. If the above flags do the trick, then why not. The difficulty will
be to find a nice way to determine whether those flags can be passed.

Toggle quote (5 lines)
> Another advice was to replace the call to "python setup.py install"
> by "pip install .", which apparently is becoming the standard approach
> of installing Python packages, not only over the Internet, but also locally
> (for which the "." stands).

Dunno, but that’s a different story. :-)

Thanks!

Ludo’.
H
H
Hartmut Goebel wrote on 18 Apr 2016 21:21
(address . 20765@debbugs.gnu.org)
571533AF.3040801@crazy-compilers.com
Hi,

I support the proposal for switching to pip.

Switching to pip should be save, since this is the proposed standard way
for installing - and thus it should be save in the long run, too.

I suggest calling pip with this options (valid as of pip 8.1.1)

--no-deps Don't install package dependencies.
--no-binary :all: Do not use binary packages.
--no-index Ignore package index
--disable-pip-version-check

Since in the internet one can find issues with
--single-version-externally-managed I verified that pip will even work
for packages using distutils instead of setuptools. I did a test and
inspecting the software.

Here is what I did: I used a simply Python package which imports
distutils instead of setuptools. Installing the via pip in verbose-mode
shows this line (you do not need to look at the details):

Running command /tmp/myvenv/bin/python2.7 -u -c "import setuptools,
tokenize;__file__='/tmp/pip-Biwi3y-build/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-KcE_9O-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/tmp/myvenv/include/site/python2.7/sample

This basically runs the setup.py “wrapped” into setuptools [1]. The
source code of setuptools reviled that it is monkey-pathing distutils
(esp. distutils.dist.Distribution) to objects from setuptools. So the
more elaborate stuff of setuptools is used even if the setup.py only
import distutils.

Additionally I checked what --single-version-externally-managed does
(since the documentation [2] is not that clear): it simply makes
`install` use the "original" distutils install command, which was/is not
able to create zipped eggs. So here we are on the save side, too.

Tested with pip 8.1.1 and setuptools 20.3, but this same code is in pip
6.0 and setuptools 18.0 already (which are older than what is in guix
0.10.0)


[1] Technically this line is executing a command which import setuptools
and then executes setup.py the it the same scope/context.
[2]


--
Regards
Hartmut Goebel

| Hartmut Goebel | h.goebel@crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
L
L
Leo Famulari wrote on 17 Dec 2016 19:43
(name . Hartmut Goebel)(address . h.goebel@crazy-compilers.com)(address . 20765-done@debbugs.gnu.org)
20161217184334.GA1726@jasmine
From what I can see, this bug is fixed by the new Python build system
introduced in commit 03e856ddf5d28bc61144effb1a393b73cb4a2d9f.

Please re-open the bug if I am mistaken.
Closed
?