[PATCH] gnu: python: Fix sitecustomize.py

  • Done
  • quality assurance status badge
Details
3 participants
  • Konrad Hinsen
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Konrad Hinsen
Severity
normal
K
K
Konrad Hinsen wrote on 4 Jan 13:44 +0100
(address . guix-patches@gnu.org)
m1sf3djmjg.fsf@fastmail.net
* gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
to deal with situations where it contains "../"

This happens in particular when running Python from a Singularity image

Change-Id: Ibfe13d7c2a14beaa199f599e64bc0b7bfb500fe8
---
gnu/packages/aux-files/python/sitecustomize.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (17 lines)
diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
index e2348e0356..82ea91104d 100644
--- a/gnu/packages/aux-files/python/sitecustomize.py
+++ b/gnu/packages/aux-files/python/sitecustomize.py
@@ -37,7 +37,7 @@ import sys
major_minor = '{}.{}'.format(*sys.version_info)
site_packages_prefix = os.path.join(
'lib', 'python' + major_minor, 'site-packages')
-python_site = os.path.join(sys.prefix, site_packages_prefix)
+python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))
try:
all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)

base-commit: 7b0863f07a113caef26fea13909bd97d250b629e
--
2.41.0
K
K
Konrad Hinsen wrote on 6 Jan 10:32 +0100
(address . 68241@debbugs.gnu.org)
m1edeukdtb.fsf@fastmail.net
Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

Toggle quote (3 lines)
> * gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
> to deal with situations where it contains "../"

This patch leads to massive rebuilds for non-trivial environments. I
will submit an alternative that patches Python via a graft.

Cheers,
Konrad.
K
K
Konrad Hinsen wrote on 6 Jan 10:37 +0100
[PATCH] gnu: python: Fix sitecustomize.py
(address . 68241@debbugs.gnu.org)
m1bk9ykdkc.fsf@fastmail.net
* gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
to deal with situations where it contains "../"

This happens in particular when running Python from a Singularity image

This is a variant of https://issues.guix.gnu.org/68241that implements the
fix as a graft rather than as a patch to the Python package. Since many
packages in Guix depend on Python, the basic patch leads to enormous
build times. A graft should be sufficient since the fix has no impact
on Python being used within Guix.

Change-Id: I3b07f4edf474a12fcabfbf90f37c4c0223278330
---
.../aux-files/python/sitecustomize-fixed.py | 65 +++++++++++++++++++
gnu/packages/python.scm | 22 ++++++-
2 files changed, 85 insertions(+), 2 deletions(-)
create mode 100644 gnu/packages/aux-files/python/sitecustomize-fixed.py

Toggle diff (115 lines)
diff --git a/gnu/packages/aux-files/python/sitecustomize-fixed.py b/gnu/packages/aux-files/python/sitecustomize-fixed.py
new file mode 100644
index 0000000000..82ea91104d
--- /dev/null
+++ b/gnu/packages/aux-files/python/sitecustomize-fixed.py
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import site
+import sys
+
+# Commentary:
+#
+# Site-specific customization for Guix.
+#
+# The program below honors the GUIX_PYTHONPATH environment variable to
+# discover Python packages. File names appearing in this variable that match
+# a predefined versioned installation prefix are added to the sys.path. To be
+# considered, a Python package must be installed under the
+# 'lib/pythonX.Y/site-packages' directory, where X and Y are the major and
+# minor version numbers of the Python interpreter.
+#
+# Code:
+
+major_minor = '{}.{}'.format(*sys.version_info)
+site_packages_prefix = os.path.join(
+ 'lib', 'python' + major_minor, 'site-packages')
+python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))
+
+try:
+ all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)
+except KeyError:
+ all_sites_raw = []
+# Normalize paths, otherwise a trailing slash would cause it to not match.
+all_sites_norm = [os.path.normpath(p) for p in all_sites_raw]
+matching_sites = [p for p in all_sites_norm
+ if p.endswith(site_packages_prefix)]
+
+if matching_sites:
+ # Deduplicate the entries, append them to sys.path, and handle any
+ # .pth files they contain.
+ for s in matching_sites:
+ site.addsitedir(s)
+
+ # Move the entries that were appended to sys.path in front of
+ # Python's own site-packages directory. This enables Guix
+ # packages to override Python's bundled packages, such as 'pip'.
+ python_site_index = sys.path.index(python_site)
+ new_site_start_index = sys.path.index(matching_sites[0])
+ if python_site_index < new_site_start_index:
+ sys.path = (sys.path[:python_site_index]
+ + sys.path[new_site_start_index:]
+ + sys.path[python_site_index:new_site_start_index])
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 51d5f598d7..3cc3c08360 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -414,7 +414,16 @@ (define-public python-2.7
packages; exception-based error handling; and very high level dynamic
data types.")
(properties '((cpe-name . "python")))
- (license license:psfl)))
+ (license license:psfl)
+ (replacement python-2.7-fixed)))
+
+(define-public python-2.7-fixed
+ (package
+ (inherit python-2.7)
+ (native-inputs
+ (modify-inputs (package-native-inputs python-2.7)
+ (replace "sitecustomize.py"
+ (local-file (search-auxiliary-file "python/sitecustomize-fixed.py")))))))
;; Current 2.x version.
(define-public python-2 python-2.7)
@@ -594,7 +603,16 @@ (define-public python-3.10
;; Python 3.9.
(search-path-specification
(variable "PYTHONTZPATH")
- (files (list "share/zoneinfo")))))))
+ (files (list "share/zoneinfo")))))
+ (replacement python-3.10-fixed)))
+
+(define-public python-3.10-fixed
+ (package
+ (inherit python-3.10)
+ (native-inputs
+ (modify-inputs (package-native-inputs python-3.10)
+ (replace "sitecustomize.py"
+ (local-file (search-auxiliary-file "python/sitecustomize-fixed.py")))))))
;; Current 3.x version.
(define-public python-3 python-3.10)

base-commit: 7b0863f07a113caef26fea13909bd97d250b629e
--
2.41.0
L
L
Ludovic Courtès wrote on 8 Jan 09:57 +0100
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
87o7dw6w3w.fsf@gnu.org
Hi Konrad!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

Toggle quote (22 lines)
> * gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
> to deal with situations where it contains "../"
>
> This happens in particular when running Python from a Singularity image
> created by Guix. See https://issues.guix.gnu.org/53258.
>
> Change-Id: Ibfe13d7c2a14beaa199f599e64bc0b7bfb500fe8
> ---
> gnu/packages/aux-files/python/sitecustomize.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
> index e2348e0356..82ea91104d 100644
> --- a/gnu/packages/aux-files/python/sitecustomize.py
> +++ b/gnu/packages/aux-files/python/sitecustomize.py
> @@ -37,7 +37,7 @@ import sys
> major_minor = '{}.{}'.format(*sys.version_info)
> site_packages_prefix = os.path.join(
> 'lib', 'python' + major_minor, 'site-packages')
> -python_site = os.path.join(sys.prefix, site_packages_prefix)
> +python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))

IIRC Josselin had a patch for https://issues.guix.gnu.org/53258, but I
can’t find it. Josselin?

Also, do I get it right that the patch you sent fixes more than #53258?

Thanks!

Ludo’.
K
K
Konrad Hinsen wrote on 8 Jan 10:14 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
m134v8jift.fsf@fastmail.net
Hi Ludo,

Toggle quote (2 lines)
> Also, do I get it right that the patch you sent fixes more than #53258?

#53258 is two things: a weird error message, and a failure to run valid
Python code, the two being possibly related.

My patch fixes the error message. The Python example that Marek gave
already works fine in today's Guix, so it was probably fixed by
something else as a side effect.

Could the bug that causes the error message also mess up valid Python
code? In theory, yes, but I don't have an example at hand.

Cheers,
Konrad.
M
M
Maxim Cournoyer wrote on 21 Jan 05:22 +0100
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
87plxvb9j6.fsf@gmail.com
Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

Toggle quote (11 lines)
> Hi Ludo,
>
>> Also, do I get it right that the patch you sent fixes more than #53258?
>
> #53258 is two things: a weird error message, and a failure to run valid
> Python code, the two being possibly related.
>
> My patch fixes the error message. The Python example that Marek gave
> already works fine in today's Guix, so it was probably fixed by
> something else as a side effect.

Oh? And you tested using the exact same reproducer, using a relocatable
pack?

--
Thanks,
Maxim
K
K
Konrad Hinsen wrote on 21 Jan 12:11 +0100
m11qabvt4h.fsf@fastmail.net
Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (7 lines)
>> My patch fixes the error message. The Python example that Marek gave
>> already works fine in today's Guix, so it was probably fixed by
>> something else as a side effect.
>
> Oh? And you tested using the exact same reproducer, using a relocatable
> pack?

Yes, I tested it in a Singularity container.

Cheers,
konrad.
M
M
Maxim Cournoyer wrote on 21 Jan 19:29 +0100
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
87v87m8rr3.fsf@gmail.com
Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

Toggle quote (11 lines)
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>>> My patch fixes the error message. The Python example that Marek gave
>>> already works fine in today's Guix, so it was probably fixed by
>>> something else as a side effect.
>>
>> Oh? And you tested using the exact same reproducer, using a relocatable
>> pack?
>
> Yes, I tested it in a Singularity container.

Great, it's now on core-updates.

--
Thanks,
Maxim
Closed
K
K
Konrad Hinsen wrote on 22 Jan 08:15 +0100
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
m1v87lvnwz.fsf@fastmail.net
Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (2 lines)
> Great, it's now on core-updates.

Thanks!

Konrad.
Closed
?