[core-updates] [patch] gnu: python: Enable optimizations.

DoneSubmitted by Julien Lepiller.
Details
2 participants
  • Julien Lepiller
  • Ludovic Courtès
Owner
unassigned
Severity
normal
J
J
Julien Lepiller wrote on 19 Mar 2021 00:16
(address . guix-patches@gnu.org)
20210319001622.1c19ad41@tachikoma.lepiller.eu
Hi Guix!

This is the second patch I created in my exploration of our Python
package. Here, I noticed that our Python was very slow compared to
other distros and I tried to find which options they used. I found some
hints from Fedora. In this patch, I enable a few options and add a C/LD
flag to make python run faster.

I also attached a report of timing running the benchmark from
pyperformance on my Fedora machine. For comparison, I have also added
the results for Fedora's python3, which is somewhere in between.

I noticed that this increases the size of the python package by 20MB,
which is due entirely to --with-lto option, which also accounts for
half of the speedup. Overall, this python is almost twice as fast on
some benchmarks, and always faster on all of them. It's about 1.5 times
faster on average.

Strangely, the python package from Alpine is twice as small as this
package, and they use the --with-lto option. I wonder how they achieve
that.

Note that I modified python-2, as its flags are inherited by python-3
(and also the bootstrap python from commencement.scm).
From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Thu, 18 Mar 2021 23:41:40 +0100
Subject: [PATCH 2/2] gnu: python: Enable optimizations.

As a result, python execution time of scripts is decreased by 6 to 40%.

gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
optimize the resulting python.
---
gnu/packages/python.scm | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)

Toggle diff (29 lines)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index da9859b11a..8de0ca343c 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -183,7 +183,10 @@
              "--with-system-expat"      ;for XML support
              "--with-system-ffi"        ;build ctypes
              "--with-ensurepip=install" ;install pip and setuptools
+             "--with-computed-gotos"    ;main interpreter loop optimization
+             "--with-lto"               ;increase size by 20MB, but 15% speedup
              "--enable-unicode=ucs4"
+             "--enable-optimizations"
 
              ;; Prevent the installed _sysconfigdata.py from retaining a reference
              ;; to coreutils.
@@ -197,8 +200,10 @@
                      "ac_cv_file__dev_ptmx=no"
                      "ac_cv_file__dev_ptc=no")
                    '())
+             "CFLAGS=-fno-semantic-interposition"
              (string-append "LDFLAGS=-Wl,-rpath="
-                            (assoc-ref %outputs "out") "/lib"))
+                            (assoc-ref %outputs "out") "/lib"
+                            " -fno-semantic-interposition"))
        ;; With no -j argument tests use all available cpus, so provide one.
        #:make-flags
        (list (string-append
-- 
2.30.0
Attachment: python-report
L
L
Ludovic Courtès wrote on 31 Mar 2021 14:47
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47251@debbugs.gnu.org)
87r1jva3rj.fsf@gnu.org
Hi Julien,

Julien Lepiller <julien@lepiller.eu> skribis:

Toggle quote (10 lines)
>>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien@lepiller.eu>
> Date: Thu, 18 Mar 2021 23:41:40 +0100
> Subject: [PATCH 2/2] gnu: python: Enable optimizations.
>
> As a result, python execution time of scripts is decreased by 6 to 40%.
>
> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
> optimize the resulting python.

LGTM! Great that you found about these options.

I wonder what can be done to reduce the package size. On
‘core-updates’, we’re already stripping ELF binaries a bit more than on
‘master’.

What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in
the executable or in libpython.so as a result of using -flto?

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 10 Apr 2021 22:55
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47251@debbugs.gnu.org)
871rbh2724.fsf_-_@gnu.org
Ping! :-)

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (25 lines)
> Hi Julien,
>
> Julien Lepiller <julien@lepiller.eu> skribis:
>
>>>>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001
>> From: Julien Lepiller <julien@lepiller.eu>
>> Date: Thu, 18 Mar 2021 23:41:40 +0100
>> Subject: [PATCH 2/2] gnu: python: Enable optimizations.
>>
>> As a result, python execution time of scripts is decreased by 6 to 40%.
>>
>> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
>> optimize the resulting python.
>
> LGTM! Great that you found about these options.
>
> I wonder what can be done to reduce the package size. On
> ‘core-updates’, we’re already stripping ELF binaries a bit more than on
> ‘master’.
>
> What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in
> the executable or in libpython.so as a result of using -flto?
>
> Thanks,
> Ludo’.
L
L
Ludovic Courtès wrote on 10 Apr 2021 22:55
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47251@debbugs.gnu.org)
8735vx272x.fsf_-_@gnu.org
Ping! :-)

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (25 lines)
> Hi Julien,
>
> Julien Lepiller <julien@lepiller.eu> skribis:
>
>>>>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001
>> From: Julien Lepiller <julien@lepiller.eu>
>> Date: Thu, 18 Mar 2021 23:41:40 +0100
>> Subject: [PATCH 2/2] gnu: python: Enable optimizations.
>>
>> As a result, python execution time of scripts is decreased by 6 to 40%.
>>
>> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
>> optimize the resulting python.
>
> LGTM! Great that you found about these options.
>
> I wonder what can be done to reduce the package size. On
> ‘core-updates’, we’re already stripping ELF binaries a bit more than on
> ‘master’.
>
> What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in
> the executable or in libpython.so as a result of using -flto?
>
> Thanks,
> Ludo’.
J
J
Julien Lepiller wrote on 3 Jun 2021 04:06
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47251@debbugs.gnu.org)
20210603040635.3b6c7d12@tachikoma.lepiller.eu
Le Wed, 31 Mar 2021 14:47:44 +0200,
Ludovic Courtès <ludo@gnu.org> a écrit :

Toggle quote (28 lines)
> Hi Julien,
>
> Julien Lepiller <julien@lepiller.eu> skribis:
>
> >>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00
> >>2001
> > From: Julien Lepiller <julien@lepiller.eu>
> > Date: Thu, 18 Mar 2021 23:41:40 +0100
> > Subject: [PATCH 2/2] gnu: python: Enable optimizations.
> >
> > As a result, python execution time of scripts is decreased by 6 to
> > 40%.
> >
> > gnu/packages/python (python-2.7)[arguments]: Add configure and make
> > flags to optimize the resulting python.
>
> LGTM! Great that you found about these options.
>
> I wonder what can be done to reduce the package size. On
> ‘core-updates’, we’re already stripping ELF binaries a bit more than
> on ‘master’.
>
> What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in
> the executable or in libpython.so as a result of using -flto?
>
> Thanks,
> Ludo’.

Sorry for the late reply, I didn't see your message, and never received
the pings I see on issues.guix.gnu.org.

Here is the output of size on bin/python:

text data bss dec hex filename
2072 704 8 2784 ae0
/gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/bin/python3

Same, for lib/libpython3.so:

text data bss dec hex filename
1154 576 8 1738 6ca
/gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/lib/libpython3.so

Same for lib/libpython3.9.so.1.0:

text data bss dec hex filename
3614304 254928 139896 4009128 3d2ca8
/gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/lib/libpython3.9.so.1.0

But I noticed another file gained a lot of weight:
lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs 39MB,
whereas on master (for 3.8.2) it weighs 6MB. Here is what size has to
say:

text data bss dec hex
298 0 52 350 15e
2642 0 0 2642 a52
370 0 100 470 1d6
481 0 0 481 1e1
1828 0 0 1828 724
1909 0 0 1909 775
1817 512 0 2329 919
24497 0 16032 40529 9e51
110286 976 0 111262 1b29e
12936 0 0 12936 3288
895 0 0 895 37f
1159 0 32 1191 4a7
4311 0 0 4311 10d7
29747 56 0 29803 746b
39941 344 0 40285 9d5d
1252 0 0 1252 4e4
1209 760 16 1985 7c1
16582 0 0 16582 40c6
56468 3856 1 60325 eba5
62176 3520 2080 67776 108c0
20674 0 0 20674 50c2
3160 408 0 3568 df0
1594 488 0 2082 822
4895 1392 16 6303 189f
17197 1384 16 18597 48a5
12586 1072 0 13658 355a
20052 6096 0 26148 6624
4232 1176 0 5408 1520
33447 28560 56 62063 f26f
4950 1096 0 6046 179e
18895 4928 1344 25167 624f
5088 880 0 5968 1750
30728 1496 452 32676 7fa4
9294 1016 16 10326 2856
11512 2328 8 13848 3618
1661 728 0 2389 955
2620 1040 0 3660 e4c
48674 2280 672 51626 c9aa
102026 2024 1240 105290 19b4a
69538 6296 1352 77186 12d82
14228 3728 0 17956 4624
30454 1968 0 32422 7ea6
6484 1304 0 7788 1e6c
9917 1368 8 11293 2c1d
2185 552 0 2737 ab1
31339 1816 0 33155 8183
19061 900 584 20545 5041
1886 528 0 2414 96e
12991 2240 0 15231 3b7f
46206 3624 8 49838 c2ae
5898 1176 8 7082 1baa
5952 248 0 6200 1838
13769 1264 256 15289 3bb9
111003 11760 98432 221195 3600b
280942 5056 2168 288166 465a6
141203 0 0 141203 22793
16406 2000 8 18414 47ee
13022 944 16 13982 369e
129842 2352 1712 133906 20b12
810 0 0 810 32a
71059 72 8 71139 115e3
24372 40 0 24412 5f5c
8726 0 48 8774 2246
41512 3992 0 45504 b1c0
61876 2460 4 64340 fb54
20340 360 8 20708 50e4
74371 88 80 74539 1232b
10273 2488 24 12785 31f1
48 0 0 48 30
26200 472 408 27080 69c8
1257 0 0 1257 4e9
1404 0 0 1404 57c
55841 0 8 55849 da29
117 0 0 117 75
412 0 0 412 19c
110 0 0 110 6e
192 0 250 442 1ba
6170 25384 0 31554 7b42
10561 3280 16 13857 3621
3590 0 0 3590 e06
30434 1392 72 31898 7c9a
2079 24 0 2103 837
36616 0 96 36712 8f68
27149 384 0 27533 6b8d
22163 0 8 22171 569b
494 0 0 494 1ee
4461 0 0 4461 116d
5822 0 48 5870 16ee
7570 0 0 7570 1d92
7739 0 32 7771 1e5b
1144 0 0 1144 478
1584 0 0 1584 630
113 0 0 113 71
4422 24 24 4470 1176
20465 416 672 21553 5431
402 0 0 402 192
14654 0 0 14654 393e
16034 696 0 16730 415a
7026 0 0 7026 1b72
1754 40 4 1798 706
2920 0 0 2920 b68
28827 768 56 29651 73d3
39065 2568 1792 43425 a9a1
4590 96 452 5138 1412
7585 952 0 8537 2159
1562 52 8 1622 656
316 0 0 316 13c
4609 56 0 4665 1239
1979 0 0 1979 7bb
36752 8 2400 39160 98f8
24453 0 0 24453 5f85
11769 12 0 11781 2e05
973 32 3104 4109 100d
299 528 0 827 33b
11964 0 0 11964 2ebc
6063 24 0 6087 17c7
18488 896 0 19384 4bb8
141834 17384 8 159226 26dfa
10041 160 32 10233 27f9
3542 416 0 3958 f76
81990 4624 0 86614 15256
27982 1600 16 29598 739e
1808 320 0 2128 850
12262 2776 8 15046 3ac6
18682 3320 0 22002 55f2
19569 4368 136 24073 5e09
11529 824 0 12353 3041
44792 11312 0 56104 db28
3189 328 0 3517 dbd
16392 968 1520 18880 49c0
6380 608 0 6988 1b4c
17490 1288 444 19222 4b16
17253 3088 424 20765 511d
7382 1456 0 8838 2286
19480 576 216 20272 4f30
11320 1936 0 13256 33c8
14121 1336 0 15457 3c61
12682 1720 0 14402 3842
29184 6088 8 35280 89d0
39623 4080 0 43703 aab7
10987 1112 0 12099 2f43
10034 1864 176 12074 2f2a
17528 488 248 18264 4758
1197 352 0 1549 60d
1614 192 0 1806 70e
31873 5840 440 38153 9509
1644 1504 0 3148 c4c
91445 317 0 91762 16672

I had to remove the file name it comes from, because of width issue
in email, but I don't really understand what size does, so I don't know
if that's relevant or not. Do we need to keep that static library
around?
L
L
Ludovic Courtès wrote on 3 Jun 2021 11:48
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47251@debbugs.gnu.org)
875yyvwa6f.fsf@gnu.org
Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

Toggle quote (5 lines)
> But I noticed another file gained a lot of weight:
> lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs 39MB,
> whereas on master (for 3.8.2) it weighs 6MB. Here is what size has to
> say:

Yeah, the .o files in there likely contain serialized GIMPLE, as a
result of compiling with ‘-flto’.

How about removing *.a altogether in a separate patch? Or moving it to
a “static” output? This would be consistent with what’s done in other
packages.

Thanks,
Ludo’.
J
J
Julien Lepiller wrote on 5 Jun 2021 17:43
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47251@debbugs.gnu.org)
20210605174308.57c4a324@tachikoma.lepiller.eu
Le Thu, 03 Jun 2021 11:48:40 +0200,
Ludovic Courtès <ludo@gnu.org> a écrit :

Toggle quote (19 lines)
> Hi,
>
> Julien Lepiller <julien@lepiller.eu> skribis:
>
> > But I noticed another file gained a lot of weight:
> > lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs
> > 39MB, whereas on master (for 3.8.2) it weighs 6MB. Here is what
> > size has to say:
>
> Yeah, the .o files in there likely contain serialized GIMPLE, as a
> result of compiling with ‘-flto’.
>
> How about removing *.a altogether in a separate patch? Or moving it
> to a “static” output? This would be consistent with what’s done in
> other packages.
>
> Thanks,
> Ludo’.

Right, this is a good idea. I wrote the attached patches to: enable
optimizations, prevent building and installing the static library and
to install idle to a separate output, as it is not required in general.

With the patch from https://issues.guix.gnu.org/47214,python 3.9 on
coreupdates is ~1.7x faster than python 3.8 on master, and is ~25MB
smaller than 3.8 on master (I haven't measured the speedup for python2,
and it is ~10MB smaller with these changes than on master).
From 0c45d6e82a5f4d140840aef6b0de321122b6c4df Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Thu, 18 Mar 2021 23:41:40 +0100
Subject: [PATCH 2/4] gnu: python: Enable optimizations.

As a result, python execution time of scripts is decreased by 6 to 40%.

gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
optimize the resulting python.
---
gnu/packages/python.scm | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)

Toggle diff (29 lines)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 0f20f8e7d3..031904dff6 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -184,7 +184,10 @@
              "--with-system-expat"      ;for XML support
              "--with-system-ffi"        ;build ctypes
              "--with-ensurepip=install" ;install pip and setuptools
+             "--with-computed-gotos"    ;main interpreter loop optimization
+             "--with-lto"               ;increase size by 20MB, but 15% speedup
              "--enable-unicode=ucs4"
+             "--enable-optimizations"
 
              ;; Prevent the installed _sysconfigdata.py from retaining a reference
              ;; to coreutils.
@@ -198,8 +201,10 @@
                      "ac_cv_file__dev_ptmx=no"
                      "ac_cv_file__dev_ptc=no")
                    '())
+             "CFLAGS=-fno-semantic-interposition"
              (string-append "LDFLAGS=-Wl,-rpath="
-                            (assoc-ref %outputs "out") "/lib"))
+                            (assoc-ref %outputs "out") "/lib"
+                            " -fno-semantic-interposition"))
        ;; With no -j argument tests use all available cpus, so provide one.
        #:make-flags
        (list (string-append
-- 
2.31.1
From 1faeb01becd558f0f4eec16ff32df9dbf1adda7f Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Thu, 3 Jun 2021 22:10:09 +0200
Subject: [PATCH 3/4] gnu: python: Do not install static libraries.

This is something Fedora already does since 2010, so it should be safe,
and saves a lot of space.

* gnu/packages/patches/python-2.7-no-static-lib.patch: New file.
* gnu/packages/patches/python-3-no-static-lib.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add them.
* gnu/packages/python.scm (python-2.7, python-3): Use them.
---
gnu/local.mk | 2 +
.../patches/python-2.7-no-static-lib.patch | 70 +++++++++++++++++
.../patches/python-3-no-static-lib.patch | 75 +++++++++++++++++++
gnu/packages/python.scm | 6 +-
4 files changed, 151 insertions(+), 2 deletions(-)
create mode 100644 gnu/packages/patches/python-2.7-no-static-lib.patch
create mode 100644 gnu/packages/patches/python-3-no-static-lib.patch

Toggle diff (201 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index d5a832146e..8370ac12e2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1551,11 +1551,13 @@ dist_patch_DATA =						\
   %D%/packages/patches/python-2.7-site-prefixes.patch		\
   %D%/packages/patches/python-2.7-source-date-epoch.patch	\
   %D%/packages/patches/python-2.7-CVE-2021-3177.patch		\
+  %D%/packages/patches/python-2.7-no-static-lib.patch		\
   %D%/packages/patches/python-3-arm-alignment.patch		\
   %D%/packages/patches/python-3-deterministic-build-info.patch	\
   %D%/packages/patches/python-3-search-paths.patch		\
   %D%/packages/patches/python-3-fix-tests.patch			\
   %D%/packages/patches/python-3-hurd-configure.patch		\
+  %D%/packages/patches/python-3-no-static-lib.patch		\
   %D%/packages/patches/python-CVE-2018-14647.patch		\
   %D%/packages/patches/python-CVE-2020-26116.patch		\
   %D%/packages/patches/python-aionotify-0.2.0-py3.8.patch	\
diff --git a/gnu/packages/patches/python-2.7-no-static-lib.patch b/gnu/packages/patches/python-2.7-no-static-lib.patch
new file mode 100644
index 0000000000..02529989be
--- /dev/null
+++ b/gnu/packages/patches/python-2.7-no-static-lib.patch
@@ -0,0 +1,70 @@
+From 92e2dcace7a3c02289b314238c5f18caec659ff1 Mon Sep 17 00:00:00 2001
+From: Julien Lepiller <julien@lepiller.eu>
+Date: Sat, 5 Jun 2021 12:41:47 +0200
+Subject: [PATCH] No static lib.
+
+This patch is adapted from Fedora. The original patch did not apply
+because of whitespace changes.
+
+00111 #
+Patch the Makefile.pre.in so that the generated Makefile doesn't try to build
+a libpythonMAJOR.MINOR.a (bug 550692):
+Downstream only: not appropriate for upstream
+---
+ Makefile.pre.in | 26 +-------------------------
+ 1 file changed, 1 insertion(+), 25 deletions(-)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 6239fc3..948f196 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -486,7 +486,7 @@ coverage-report: regen-grammar
+ 
+ 
+ # Build the interpreter
+-$(BUILDPYTHON):	Modules/python.o $(LIBRARY) $(LDLIBRARY)
++$(BUILDPYTHON):	Modules/python.o $(LDLIBRARY)
+ 		$(LINKCC) $(LDFLAGS) $(LINKFORSHARED) -o $@ \
+ 			Modules/python.o \
+ 			$(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)
+@@ -527,18 +527,6 @@ sharedmods: $(BUILDPYTHON) pybuilddir.txt Modules/_math.o
+ 		_TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \
+ 		$(PY_BUILD_ENVIRON) $(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build
+ 
+-# Build static library
+-# avoid long command lines, same as LIBRARY_OBJS
+-$(LIBRARY): $(LIBRARY_OBJS)
+-	-rm -f $@
+-	$(AR) $(ARFLAGS) $@ Modules/getbuildinfo.o
+-	$(AR) $(ARFLAGS) $@ $(PARSER_OBJS)
+-	$(AR) $(ARFLAGS) $@ $(OBJECT_OBJS)
+-	$(AR) $(ARFLAGS) $@ $(PYTHON_OBJS)
+-	$(AR) $(ARFLAGS) $@ $(MODULE_OBJS) $(SIGNAL_OBJS)
+-	$(AR) $(ARFLAGS) $@ $(MODOBJS)
+-	$(RANLIB) $@
+-
+ libpython$(VERSION).so: $(LIBRARY_OBJS)
+ 	if test $(INSTSONAME) != $(LDLIBRARY); then \
+ 		$(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+@@ -1216,18 +1204,6 @@ libainstall:	@DEF_MAKE_RULE@ python-config
+ 		else	true; \
+ 		fi; \
+ 	done
+-	@if test -d $(LIBRARY); then :; else \
+-		if test "$(PYTHONFRAMEWORKDIR)" = no-framework; then \
+-			if test "$(SO)" = .dll; then \
+-				$(INSTALL_DATA) $(LDLIBRARY) $(DESTDIR)$(LIBPL) ; \
+-			else \
+-				$(INSTALL_DATA) $(LIBRARY) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \
+-				$(RANLIB) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \
+-			fi; \
+-		else \
+-			echo Skip install of $(LIBRARY) - use make frameworkinstall; \
+-		fi; \
+-	fi
+ 	$(INSTALL_DATA) Modules/config.c $(DESTDIR)$(LIBPL)/config.c
+ 	$(INSTALL_DATA) Modules/python.o $(DESTDIR)$(LIBPL)/python.o
+ 	$(INSTALL_DATA) $(srcdir)/Modules/config.c.in $(DESTDIR)$(LIBPL)/config.c.in
+-- 
+2.31.1
+
diff --git a/gnu/packages/patches/python-3-no-static-lib.patch b/gnu/packages/patches/python-3-no-static-lib.patch
new file mode 100644
index 0000000000..b44dbc976c
--- /dev/null
+++ b/gnu/packages/patches/python-3-no-static-lib.patch
@@ -0,0 +1,75 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: David Malcolm <dmalcolm@redhat.com>
+Date: Mon, 18 Jan 2010 17:59:07 +0000
+Subject: [PATCH] 00111: Don't try to build a libpythonMAJOR.MINOR.a
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Downstream only: not appropriate for upstream.
+
+See https://bugzilla.redhat.com/show_bug.cgi?id=556092
+
+Co-authored-by: David Malcolm <dmalcolm@redhat.com>
+Co-authored-by: Bohuslav Kabrda <bkabrda@redhat.com>
+Co-authored-by: Matej Stuchlik <mstuchli@redhat.com>
+Co-authored-by: Robert Kuska <rkuska@redhat.com>
+Co-authored-by: Charalampos Stratakis <cstratak@redhat.com>
+Co-authored-by: Miro HronĨok <miro@hroncok.cz>
+---
+ Makefile.pre.in | 21 ++-------------------
+ 1 file changed, 2 insertions(+), 19 deletions(-)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 0c3981c132..5587422ec7 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -589,7 +589,7 @@ clinic: check-clean-src $(srcdir)/Modules/_blake2/blake2s_impl.c
+ 	$(PYTHON_FOR_REGEN) $(srcdir)/Tools/clinic/clinic.py --make --srcdir $(srcdir)
+ 
+ # Build the interpreter
+-$(BUILDPYTHON):	Programs/python.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS)
++$(BUILDPYTHON):	Programs/python.o $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS)
+ 	$(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS)
+ 
+ platform: $(BUILDPYTHON) pybuilddir.txt
+@@ -637,12 +637,6 @@ sharedmods: $(BUILDPYTHON) pybuilddir.txt Modules/_math.o
+ 		_TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \
+ 		$(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build
+ 
+-
+-# Build static library
+-$(LIBRARY): $(LIBRARY_OBJS)
+-	-rm -f $@
+-	$(AR) $(ARFLAGS) $@ $(LIBRARY_OBJS)
+-
+ libpython$(LDVERSION).so: $(LIBRARY_OBJS) $(DTRACE_OBJS)
+ 	if test $(INSTSONAME) != $(LDLIBRARY); then \
+ 		$(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM); \
+@@ -724,7 +718,7 @@ Makefile Modules/config.c: Makefile.pre \
+ 	@echo "The Makefile was updated, you may need to re-run make."
+ 
+ 
+-Programs/_testembed: Programs/_testembed.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS)
++Programs/_testembed: Programs/_testembed.o $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS)
+ 	$(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/_testembed.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS)
+ 
+ ############################################################################
+@@ -1652,17 +1646,6 @@ libainstall:	@DEF_MAKE_RULE@ python-config
+ 		else	true; \
+ 		fi; \
+ 	done
+-	@if test -d $(LIBRARY); then :; else \
+-		if test "$(PYTHONFRAMEWORKDIR)" = no-framework; then \
+-			if test "$(SHLIB_SUFFIX)" = .dll; then \
+-				$(INSTALL_DATA) $(LDLIBRARY) $(DESTDIR)$(LIBPL) ; \
+-			else \
+-				$(INSTALL_DATA) $(LIBRARY) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \
+-			fi; \
+-		else \
+-			echo Skip install of $(LIBRARY) - use make frameworkinstall; \
+-		fi; \
+-	fi
+ 	$(INSTALL_DATA) Modules/config.c $(DESTDIR)$(LIBPL)/config.c
+ 	$(INSTALL_DATA) Programs/python.o $(DESTDIR)$(LIBPL)/python.o
+ 	$(INSTALL_DATA) $(srcdir)/Modules/config.c.in $(DESTDIR)$(LIBPL)/config.c.in
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 031904dff6..4ac4b7c23c 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -150,7 +150,8 @@
                                 "python-2.7-source-date-epoch.patch"
                                 "python-2.7-adjust-tests.patch"
                                 "python-cross-compile.patch"
-                                "python-2.7-CVE-2021-3177.patch"))
+                                "python-2.7-CVE-2021-3177.patch"
+                                "python-2.7-no-static-lib.patch"))
        (modules '((guix build utils)))
        (snippet
         '(begin
@@ -418,7 +419,8 @@ data types.")
                         "python-3-deterministic-build-info.patch"
                         "python-3-fix-tests.patch"
                         "python-3-hurd-configure.patch"
-                        "python-3-search-paths.patch"))
+                        "python-3-search-paths.patch"
+                        "python-3-no-static-lib.patch"))
               (sha256
                (base32
                 "10vdf46q5ldnzkprm8pldvr5a9hrdpxjv7mpzgdw6vj3cl318nhc"))
-- 
2.31.1
From c650ac48be47b373c6c8c0b3495f30d9fe21e68b Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Thu, 3 Jun 2021 22:14:14 +0200
Subject: [PATCH 4/4] gnu: python: Move idle to a separate output.

idle is a programming environment that is not useful for building
packages, but it is still useful for end users. This saves 5MB in the
default output.

* gnu/packages/python.scm (python-2.7)[outputs]: Add `idle'.
[arguments]: Move idle files in the new output.
(wrap-python): Do not wrap idle since it is in a different output.
---
gnu/packages/python.scm | 28 +++++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)

Toggle diff (55 lines)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 4ac4b7c23c..72e05552d8 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -176,7 +176,8 @@
                        "Lib/test/test_subprocess.py"))
            #t))))
     (outputs '("out"
-               "tk"))                   ;tkinter; adds 50 MiB to the closure
+               "tk"                     ;tkinter; adds 50 MiB to the closure
+               "idle"))                 ;programming environment; weighs 5MB
     (build-system gnu-build-system)
     (arguments
      `(#:test-target "test"
@@ -357,6 +358,27 @@
                       (install-file tkinter.so target)
                       (delete-file tkinter.so)))))
                #t)))
+         (add-after 'install 'move-idle
+           (lambda* (#:key outputs #:allow-other-keys)
+             ;; when idle is built, move it to a separate output to save some
+             ;; space (5MB)
+             (let ((out (assoc-ref outputs "out"))
+                   (idle (assoc-ref outputs "idle")))
+               (when idle
+                 (for-each
+                  (lambda (file)
+                    (let ((target (string-append idle "/bin/" (basename file))))
+                      (install-file file (dirname target))
+                      (delete-file file)))
+                  (find-files (string-append out "/bin") "^idle"))
+                 (match (find-files out "^idlelib$" #:directories? #t)
+                   ((idlelib)
+                    (let* ((len (string-length out))
+                           (target (string-append idle "/"
+                                                  (string-drop idlelib len)
+                                                  "/site-packages")))
+                      (mkdir-p (dirname target))
+                      (rename-file idlelib target))))))))
          (add-after 'install 'install-sitecustomize.py
            ,(customize-site version)))))
     (inputs
@@ -620,8 +642,8 @@ for more information.")))
                   (lambda (old new)
                     (symlink (string-append python old)
                              (string-append bin "/" new)))
-                  `("python3" ,"pydoc3" ,"idle3" ,"pip3")
-                  `("python"  ,"pydoc"  ,"idle"  ,"pip"))
+                  `("python3" ,"pydoc3" ,"pip3")
+                  `("python"  ,"pydoc"  ,"pip"))
                 ;; python-config outputs search paths based upon its location,
                 ;; use a bash wrapper to avoid changing its outputs.
                 (let ((bash (string-append (assoc-ref %build-inputs "bash")
-- 
2.31.1
L
L
Ludovic Courtès wrote on 6 Jun 2021 14:34
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47251@debbugs.gnu.org)
87h7ibkw88.fsf@gnu.org
Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

Toggle quote (9 lines)
> Right, this is a good idea. I wrote the attached patches to: enable
> optimizations, prevent building and installing the static library and
> to install idle to a separate output, as it is not required in general.
>
> With the patch from https://issues.guix.gnu.org/47214, python 3.9 on
> coreupdates is ~1.7x faster than python 3.8 on master, and is ~25MB
> smaller than 3.8 on master (I haven't measured the speedup for python2,
> and it is ~10MB smaller with these changes than on master).

Yay!

Toggle quote (10 lines)
> From 0c45d6e82a5f4d140840aef6b0de321122b6c4df Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien@lepiller.eu>
> Date: Thu, 18 Mar 2021 23:41:40 +0100
> Subject: [PATCH 2/4] gnu: python: Enable optimizations.
>
> As a result, python execution time of scripts is decreased by 6 to 40%.
>
> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
> optimize the resulting python.

[...]

Toggle quote (6 lines)
> + "CFLAGS=-fno-semantic-interposition"
> (string-append "LDFLAGS=-Wl,-rpath="
> - (assoc-ref %outputs "out") "/lib"))
> + (assoc-ref %outputs "out") "/lib"
> + " -fno-semantic-interposition"))

Could you add a comment for ‘-fno-semantic-interposition’? Also, please
make sure that setting CFLAGS does not override useful defaults (for
Autoconf-based build systems, CFLAGS defaults to ‘-O2 -g’ and overriding
it means you lost both -O2 and -g; if we lose ‘-g’, ‘--with-debug-info’
won’t have any effect).

Toggle quote (13 lines)
> From 1faeb01becd558f0f4eec16ff32df9dbf1adda7f Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien@lepiller.eu>
> Date: Thu, 3 Jun 2021 22:10:09 +0200
> Subject: [PATCH 3/4] gnu: python: Do not install static libraries.
>
> This is something Fedora already does since 2010, so it should be safe,
> and saves a lot of space.
>
> * gnu/packages/patches/python-2.7-no-static-lib.patch: New file.
> * gnu/packages/patches/python-3-no-static-lib.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Add them.
> * gnu/packages/python.scm (python-2.7, python-3): Use them.

LGTM.

Toggle quote (13 lines)
> From c650ac48be47b373c6c8c0b3495f30d9fe21e68b Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien@lepiller.eu>
> Date: Thu, 3 Jun 2021 22:14:14 +0200
> Subject: [PATCH 4/4] gnu: python: Move idle to a separate output.
>
> idle is a programming environment that is not useful for building
> packages, but it is still useful for end users. This saves 5MB in the
> default output.
>
> * gnu/packages/python.scm (python-2.7)[outputs]: Add `idle'.
> [arguments]: Move idle files in the new output.
> (wrap-python): Do not wrap idle since it is in a different output.

Good idea, LGTM.

Thank you!

Ludo’.
J
J
Julien Lepiller wrote on 11 Jun 2021 03:03
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47251-done@debbugs.gnu.org)
20210611030333.72f69b00@tachikoma.lepiller.eu
Le Sun, 06 Jun 2021 14:34:31 +0200,
Ludovic Courtès <ludo@gnu.org> a écrit :

Toggle quote (14 lines)
>
> > + "CFLAGS=-fno-semantic-interposition"
> > (string-append "LDFLAGS=-Wl,-rpath="
> > - (assoc-ref %outputs "out") "/lib"))
> > + (assoc-ref %outputs "out") "/lib"
> > + " -fno-semantic-interposition"))
>
> Could you add a comment for ‘-fno-semantic-interposition’? Also,
> please make sure that setting CFLAGS does not override useful
> defaults (for Autoconf-based build systems, CFLAGS defaults to ‘-O2
> -g’ and overriding it means you lost both -O2 and -g; if we lose
> ‘-g’, ‘--with-debug-info’ won’t have any effect).
>

With the patch, python still builds with -g -O3.

Pushed as 99d450b085b7bf44d669093f6c9964391ce1b451 to
e4aa346d57f61956626032d0d1dfca6b3c96e06a to core-updates, thanks!
Closed
?
Your comment

This issue is archived.

To comment on this conversation send email to 47251@debbugs.gnu.org