[core-updates] gnu: python: Remove unnecessary files

DoneSubmitted by Julien Lepiller.
Details
3 participants
  • Efraim Flashner
  • Julien Lepiller
  • Ludovic Courtès
Owner
unassigned
Severity
normal
J
J
Julien Lepiller wrote on 17 Mar 2021 13:41
(address . guix-patches@gnu.org)
20210317133933.5eb6d325@tachikoma.lepiller.eu
Hi Guix!

this patch (going to core-updates) removes files from the output of
python2 and python3, in an attempt to save some space. First, I noticed
that python3 shipped windows binaries in its source distribution and
installs them to the output, in lib/python3.8/distutils/command/*.exe,
lib/python3.8/site-packages/*.exe and
lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
from the snippet, so they should not be installed anymore.

I also noticed that python installs files that were built during the
test phase, so I remove them, but we need to check they are not
actually needed. Other distros ship them in a separate package, so
maybe we could have a separate test output if this is causing issues?

I haven't yet tested the patch, because it needs to rebuild part of the
bootstrap chain. I'll report progress and update my patch depending
on the results!
From 63ebe5c610ff8587a73641e4139dc14a4d3e379e Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Wed, 17 Mar 2021 13:33:22 +0100
Subject: [PATCH] gnu: python: Remove unnecessary files.

* gnu/packages/python.scm (python2)[arguments]: Remove files built
during the test phase.
(python3)[arguments]: Also remove windows binaries shipped with the
sources.
---
gnu/packages/python.scm | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)

Toggle diff (34 lines)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 49d43ab008..3bb30ca011 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -294,7 +294,16 @@
                                             (not
                                              (string-prefix? "test_support."
                                                              file))))))
-                      (call-with-output-file "__init__.py" (const #t)))))))))
+                      (call-with-output-file "__init__.py" (const #t))))
+                  (let ((libdir (string-append out "/lib/" pythonX.Y)))
+                    (for-each
+                     (lambda (directory)
+                       (let ((dir (string-append libdir "/" directory)))
+                         (when (file-exists? dir)
+                           (delete-file-recursively dir))))
+                     '("email/test" "ctypes/test" "unittest/test" "tkinter/test"
+                       "sqlite3/test" "bsddb/test" "lib-tk/test" "lib2to3/tests"
+                       "json/tests" "distutils/tests"))))))))
          (add-after 'remove-tests 'rebuild-bytecode
            (lambda* (#:key outputs #:allow-other-keys)
              (let ((out (assoc-ref outputs "out")))
@@ -424,6 +433,9 @@ data types.")
                   (substitute* "Modules/Setup"
                     ;; Link Expat instead of embedding the bundled one.
                     (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
+                  ;; Delete windows binaries
+                  (for-each delete-file
+                            (find-files "Lib/distutils/command" ".*.exe$"))
                   #t))))
     (arguments
      (substitute-keyword-arguments (package-arguments python-2)
-- 
2.30.0
E
E
Efraim Flashner wrote on 17 Mar 2021 20:22
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47214@debbugs.gnu.org)
YFJW9qZYJ1XnB/GA@3900XT
On Wed, Mar 17, 2021 at 01:41:02PM +0100, Julien Lepiller wrote:
Toggle quote (19 lines)
> Hi Guix!
>
> this patch (going to core-updates) removes files from the output of
> python2 and python3, in an attempt to save some space. First, I noticed
> that python3 shipped windows binaries in its source distribution and
> installs them to the output, in lib/python3.8/distutils/command/*.exe,
> lib/python3.8/site-packages/*.exe and
> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
> from the snippet, so they should not be installed anymore.
>
> I also noticed that python installs files that were built during the
> test phase, so I remove them, but we need to check they are not
> actually needed. Other distros ship them in a separate package, so
> maybe we could have a separate test output if this is causing issues?
>
> I haven't yet tested the patch, because it needs to rebuild part of the
> bootstrap chain. I'll report progress and update my patch depending
> on the results!

I got stuck on python-minimal with test-asyncio but then it didn't hang
the second time I ran the build. I don't have a good pure python package
I can think of to test the change.

--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmBSVvYACgkQQarn3Mo9
g1FO4Q//aDeA/4fEkDC7zR3Epu4Dm8xZcrjJvMVVtEDBthdUqB27AskQBGGl2ADd
g0L7gF1bQdATpOBcbC0UCoiiLdTeaSIPzfL/+4rlcr4o7ebO1gSbLrlJPf9RzFdN
3ujkyaUWTfT5cUs/m5Xsh9J321Nl7/9/ii0hnoSE4t+DnMElHIJB5RFYHcLbTy7F
CS6BiPRzlcCHKt3c/jeGVpM+yGRCg0Et86GlvBIeTJ7LYHKY0LEOYwBnRcyogLHG
9WQfj6oNcIvdjjNf1KgGv8waW4mYQ3/kyhRB2D4epiZh8pGezgdKfnYOeWGcr6OE
J6QMeANx4tXI/fBYoPBpmRUiMpf0fNP3Zz9RiWK1OFUsejBvrMAZ/FPGCxgSAjk9
4vAd9IeBXnBQwqBokwS36r29Nz3jLAo4cY9U3vUjrPgrSM3REOBGmrvdpfhUZw9O
MvnMsW9HCyq7IpYJZKKovQHu2LSIoIsh+oU538sypWFyss/ZE9F1HSeBYDW9uDHQ
ijKv8wdX472vGRP6V4T8o6S99gi8QcZdhIRaczDwWIH+9C6rxnALmTvBoQITJJZR
JBESDoN2bcRQ1/Ty2l7UB/d3mEmWrOWbAdmyovCuiQzw7WxKhiWiNMJrurXABRSK
vk96dq+onqvyQpiHo2zw9wRiP0olBClIC7u6CEk6krK+gLNrcPg=
=bPap
-----END PGP SIGNATURE-----


J
J
Julien Lepiller wrote on 19 Mar 2021 00:08
(address . 47214@debbugs.gnu.org)
20210319000830.23926447@tachikoma.lepiller.eu
Le Wed, 17 Mar 2021 13:41:02 +0100,
Julien Lepiller <julien@lepiller.eu> a écrit :

Toggle quote (20 lines)
> Hi Guix!
>
> this patch (going to core-updates) removes files from the output of
> python2 and python3, in an attempt to save some space. First, I
> noticed that python3 shipped windows binaries in its source
> distribution and installs them to the output, in
> lib/python3.8/distutils/command/*.exe,
> lib/python3.8/site-packages/*.exe and
> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
> from the snippet, so they should not be installed anymore.
>
> I also noticed that python installs files that were built during the
> test phase, so I remove them, but we need to check they are not
> actually needed. Other distros ship them in a separate package, so
> maybe we could have a separate test output if this is causing issues?
>
> I haven't yet tested the patch, because it needs to rebuild part of
> the bootstrap chain. I'll report progress and update my patch
> depending on the results!

I discovered other .exe in python3, so this updated patch deletes them.
Since I modify the python from commencement.scm by changing a phase in
python-2.7, I decided to also change the definition there to remove the
offending files.

I managed to build up to python-3.9 without any issue, but haven't
tested building more than a handful of python packages.
From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Wed, 17 Mar 2021 13:33:22 +0100
Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.

* gnu/packages/python.scm (python2)[arguments]: Remove files built
during the test phase.
(python3)[arguments]: Also remove windows binaries shipped with the
sources.
---
gnu/packages/commencement.scm | 9 ++++++++-
gnu/packages/python.scm | 35 +++++++++++++++++++++++++++++++++--
2 files changed, 41 insertions(+), 3 deletions(-)

Toggle diff (110 lines)
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index b2921a4a48..b2b9651444 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -10,6 +10,7 @@
 ;;; Copyright © 2020 Timothy Sample <samplet@ngyro.com>
 ;;; Copyright © 2020 Guy Fleury Iteriteka <gfleury@disroot.org>
 ;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -3098,7 +3099,12 @@ memoized as a function of '%current-system'."
                    (delete-file-recursively "Modules/expat")
                    (substitute* "Modules/Setup.dist"
                      ;; Link Expat instead of embedding the bundled one.
-                     (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n")))))))
+                     (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
+                   ;; Delete windows binaries
+                   (for-each delete-file
+                             (find-files "Lib/distutils/command" ".*.exe$"))
+                   (for-each delete-file
+                             (find-files "Lib/ensurepip" ".*.whl$")))))))
     (inputs
      `(,@(%boot0-inputs)
        ("expat" ,expat-sans-tests)))              ;remove OpenSSL, zlib, etc.
@@ -3129,6 +3135,7 @@ memoized as a function of '%current-system'."
                                          ('add-after unpack apply-alignment-patch _))
                                        `(modify-phases ,original-phases ,@changes))
                                       (_ phases)))
+               (delete 'remove-windows-binaries)
                (add-before 'configure 'disable-modules
                  (lambda _
                    (substitute* "setup.py"
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 49d43ab008..da9859b11a 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -31,7 +31,7 @@
 ;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org>
 ;;; Copyright © 2016, 2017 Alex Vong <alexvong1995@gmail.com>
 ;;; Copyright © 2016, 2017, 2018 Arun Isaac <arunisaac@systemreboot.net>
-;;; Copyright © 2016, 2017, 2018 Julien Lepiller <julien@lepiller.eu>
+;;; Copyright © 2016, 2017, 2018, 2021 Julien Lepiller <julien@lepiller.eu>
 ;;; Copyright © 2016, 2017, 2018 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2016, 2017 Thomas Danckaert <post@thomasdanckaert.be>
 ;;; Copyright © 2017 Carlo Zancanaro <carlo@zancanaro.id.au>
@@ -294,7 +294,16 @@
                                             (not
                                              (string-prefix? "test_support."
                                                              file))))))
-                      (call-with-output-file "__init__.py" (const #t)))))))))
+                      (call-with-output-file "__init__.py" (const #t))))
+                  (let ((libdir (string-append out "/lib/" pythonX.Y)))
+                    (for-each
+                     (lambda (directory)
+                       (let ((dir (string-append libdir "/" directory)))
+                         (when (file-exists? dir)
+                           (delete-file-recursively dir))))
+                     '("email/test" "ctypes/test" "unittest/test" "tkinter/test"
+                       "sqlite3/test" "bsddb/test" "lib-tk/test" "lib2to3/tests"
+                       "json/tests" "distutils/tests"))))))))
          (add-after 'remove-tests 'rebuild-bytecode
            (lambda* (#:key outputs #:allow-other-keys)
              (let ((out (assoc-ref outputs "out")))
@@ -424,6 +433,9 @@ data types.")
                   (substitute* "Modules/Setup"
                     ;; Link Expat instead of embedding the bundled one.
                     (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
+                  ;; Delete windows binaries
+                  (for-each delete-file
+                            (find-files "Lib/distutils/command" ".*.exe$"))
                   #t))))
     (arguments
      (substitute-keyword-arguments (package-arguments python-2)
@@ -469,6 +481,23 @@ data types.")
            ,@(if (hurd-system?)
                  `((delete 'patch-regen-for-hurd)) ;regen was removed after 3.5.9
                  '())
+           (add-after 'unpack 'remove-windows-binaries
+             (lambda _
+               ;; Delete .exe from embedded .whl (zip) files
+               (for-each
+                (lambda (whl)
+                  (let ((dir "whl-content"))
+                    (mkdir-p dir)
+                    (with-directory-excursion dir
+                      (let ((whl (string-append "../" whl)))
+                        (invoke "unzip" whl)
+                        (for-each delete-file
+                                  (find-files "." ".*.exe$"))
+                        (delete-file whl)
+                        (apply invoke "zip" "-X" whl
+                               (find-files "." ".*" #:directories? #t))))
+                    (delete-file-recursively dir)))
+                (find-files "Lib/ensurepip" ".*.whl$"))))
            (add-before 'check 'set-TZDIR
              (lambda* (#:key inputs native-inputs #:allow-other-keys)
                ;; test_email requires the Olson time zone database.
@@ -503,6 +532,8 @@ data types.")
              ,(customize-site version))))))
     (native-inputs
      `(("tzdata" ,tzdata-for-tests)
+       ("unzip" ,unzip)
+       ("zip" ,(@ (gnu packages compression) zip))
        ,@(if (%current-target-system)
              `(("python3" ,this-package))
              '())
-- 
2.30.0
L
L
Ludovic Courtès wrote on 19 Mar 2021 13:27
Re: bug#47214: [core-updates] gnu: python: Remove unnecessary files
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47214@debbugs.gnu.org)
87k0q3fhwl.fsf_-_@gnu.org
Hi Julien,

Julien Lepiller <julien@lepiller.eu> skribis:

Toggle quote (28 lines)
> Le Wed, 17 Mar 2021 13:41:02 +0100,
> Julien Lepiller <julien@lepiller.eu> a écrit :
>
>> Hi Guix!
>>
>> this patch (going to core-updates) removes files from the output of
>> python2 and python3, in an attempt to save some space. First, I
>> noticed that python3 shipped windows binaries in its source
>> distribution and installs them to the output, in
>> lib/python3.8/distutils/command/*.exe,
>> lib/python3.8/site-packages/*.exe and
>> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
>> from the snippet, so they should not be installed anymore.
>>
>> I also noticed that python installs files that were built during the
>> test phase, so I remove them, but we need to check they are not
>> actually needed. Other distros ship them in a separate package, so
>> maybe we could have a separate test output if this is causing issues?
>>
>> I haven't yet tested the patch, because it needs to rebuild part of
>> the bootstrap chain. I'll report progress and update my patch
>> depending on the results!
>
> I discovered other .exe in python3, so this updated patch deletes them.
> Since I modify the python from commencement.scm by changing a phase in
> python-2.7, I decided to also change the definition there to remove the
> offending files.

Woow, good catch.

Toggle quote (10 lines)
> From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien@lepiller.eu>
> Date: Wed, 17 Mar 2021 13:33:22 +0100
> Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
>
> * gnu/packages/python.scm (python2)[arguments]: Remove files built
> during the test phase.
> (python3)[arguments]: Also remove windows binaries shipped with the
> sources.

Unless it causes bootstrapping issues, I recommend removing those files
from a snippet so that ‘guix build -S python’ returns the cleaned-up
source.

WDYT?

Thanks,
Ludo’.
J
J
Julien Lepiller wrote on 3 Jun 2021 03:56
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47214@debbugs.gnu.org)
20210603035638.6e6fe473@tachikoma.lepiller.eu
Le Fri, 19 Mar 2021 13:27:06 +0100,
Ludovic Courtès <ludo@gnu.org> a écrit :

Toggle quote (19 lines)
> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
> > 2001 From: Julien Lepiller <julien@lepiller.eu>
> > Date: Wed, 17 Mar 2021 13:33:22 +0100
> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
> >
> > * gnu/packages/python.scm (python2)[arguments]: Remove files built
> > during the test phase.
> > (python3)[arguments]: Also remove windows binaries shipped with the
> > sources.
>
> Unless it causes bootstrapping issues, I recommend removing those
> files from a snippet so that ‘guix build -S python’ returns the
> cleaned-up source.
>
> WDYT?
>
> Thanks,
> Ludo’.

Hi, sorry for the late reply. I need to use unzip on these files to
remove some of their content, and rezip them later. How can I do that
from a snippet?
L
L
Ludovic Courtès wrote on 3 Jun 2021 11:46
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47214@debbugs.gnu.org)
87a6o7wa9n.fsf@gnu.org
Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

Toggle quote (17 lines)
> Le Fri, 19 Mar 2021 13:27:06 +0100,
> Ludovic Courtès <ludo@gnu.org> a écrit :
>
>> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
>> > 2001 From: Julien Lepiller <julien@lepiller.eu>
>> > Date: Wed, 17 Mar 2021 13:33:22 +0100
>> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
>> >
>> > * gnu/packages/python.scm (python2)[arguments]: Remove files built
>> > during the test phase.
>> > (python3)[arguments]: Also remove windows binaries shipped with the
>> > sources.
>>
>> Unless it causes bootstrapping issues, I recommend removing those
>> files from a snippet so that ‘guix build -S python’ returns the
>> cleaned-up source.

[...]

Toggle quote (4 lines)
> Hi, sorry for the late reply. I need to use unzip on these files to
> remove some of their content, and rezip them later. How can I do that
> from a snippet?

Oh right, the bit that unzips/rezips .whl files cannot easily be done
from a snippet, so better keep it as a phase.

Deleting *.exe can be done from a snippet though.

BTW, the patch reads:

(find-files "Lib/distutils/command" ".*.exe$")

but the regexp should be:

"\\.exe$"

(“.*” is unnecessary, but the dot before “exe” must be protected.)

Likewise for:

(find-files "Lib/ensurepip" ".*.whl$")

Thanks,
Ludo’.
J
J
Julien Lepiller wrote on 11 Jun 2021 03:01
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47214-done@debbugs.gnu.org)
20210611030126.25fa8606@tachikoma.lepiller.eu
Le Thu, 03 Jun 2021 11:46:44 +0200,
Ludovic Courtès <ludo@gnu.org> a écrit :

Toggle quote (49 lines)
> Hi,
>
> Julien Lepiller <julien@lepiller.eu> skribis:
>
> > Le Fri, 19 Mar 2021 13:27:06 +0100,
> > Ludovic Courtès <ludo@gnu.org> a écrit :
> >
> >> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
> >> > 2001 From: Julien Lepiller <julien@lepiller.eu>
> >> > Date: Wed, 17 Mar 2021 13:33:22 +0100
> >> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
> >> >
> >> > * gnu/packages/python.scm (python2)[arguments]: Remove files
> >> > built during the test phase.
> >> > (python3)[arguments]: Also remove windows binaries shipped with
> >> > the sources.
> >>
> >> Unless it causes bootstrapping issues, I recommend removing those
> >> files from a snippet so that ‘guix build -S python’ returns the
> >> cleaned-up source.
>
> [...]
>
> > Hi, sorry for the late reply. I need to use unzip on these files to
> > remove some of their content, and rezip them later. How can I do
> > that from a snippet?
>
> Oh right, the bit that unzips/rezips .whl files cannot easily be done
> from a snippet, so better keep it as a phase.
>
> Deleting *.exe can be done from a snippet though.
>
> BTW, the patch reads:
>
> (find-files "Lib/distutils/command" ".*.exe$")
>
> but the regexp should be:
>
> "\\.exe$"
>
> (“.*” is unnecessary, but the dot before “exe” must be protected.)
>
> Likewise for:
>
> (find-files "Lib/ensurepip" ".*.whl$")
>
> Thanks,
> Ludo’.

Fixed, and pushed as 20d52562e5c943b30e48e310b58b94054c3a54e5 to
core-updates.
Closed
?
Your comment

This issue is archived.

To comment on this conversation send email to 47214@debbugs.gnu.org