[core-updates] Python sitecustomize issue.

  • Done
  • quality assurance status badge
Details
2 participants
  • Maxim Cournoyer
  • Mathieu Othacehe
Owner
unassigned
Submitted by
Mathieu Othacehe
Severity
important
M
M
Mathieu Othacehe wrote on 18 Aug 2021 11:53
(address . bug-guix@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
878s0zoz98.fsf@gnu.org
Hello,
I tried to upgrade glade to the 3.38.2 release on core-update and
encountered the following test issue:
Toggle snippet (13 lines)
2/5 modules FAIL 0.29s killed by signal 5 SIGTRAP
>>> GLADE_MODULE_SEARCH_PATH=/home/mathieu/glade-3.38.2/build/plugins/gtk+:/home/mathieu/glade-3.38.2/build/plugins/python:/home/mathieu/glade-3.38.2/build/plugins/gjs:/home/mathieu/glade-3.38.2/tests/modules MALLOC_PERTURB_=195 GLADE_TESTING=1 GLADE_CATALOG_SEARCH_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+:/home/mathieu/glade-3.38.2/plugins/python:/home/mathieu/glade-3.38.2/plugins/gjs:/home/mathieu/glade-3.38.2/tests/catalogs GLADE_PIXMAP_DIR=/home/mathieu/glade-3.38.2/data/icons GLADE_ICON_THEME_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+/icons/22x22 /home/mathieu/glade-3.38.2/build/tests/modules
?????????????????????????????????????????????????????????????????????????????????????????????????????? ? ???????????????????????????????????????????????????????????????????????????????????????????????????????
stdout:
# random seed: R02Sd222538bc9b76b6e424ec0cb17ee887c
# GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ?gio-vfs?
Bail out! GladeUI-PYTHON-FATAL-WARNING: Error initializing Python interpreter: could not import pygobject
stderr:
Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
(/home/mathieu/glade-3.38.2/build/tests/modules:4898): GladeUI-PYTHON-WARNING **: 09:43:58.605: Error initializing Python interpreter: could not import pygobject
I think the issue here lies in the sitecustomize.py file introduced with
cb72f9a773e0931ee3758c851d96007ded034e4c.
Toggle snippet (2 lines)
python_root = os.path.realpath(sys.executable).split('/bin/')[0]
When using the Python C library, sys.executable is not the expected
python binary but the test binary.
So when we try to find the associated libraries in the Python search
path, this way:
Toggle snippet (3 lines)
index = sys_path_absolute.index(python_site)
sys.path = sys.path[:index] + matching_sites + sys.path[index:]
we get:
Toggle snippet (2 lines)
ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
Running the test with the PYTHONPATH set to:
Toggle snippet (2 lines)
PYTHONPATH=/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages:$PYTHONPATH
is a way to work around this issue.
The fix here would be to replace sys.executable in the sitecustomize.py
with something more adequate, but I have no idea what could it be.
Thanks,
Mathieu
M
M
Maxim Cournoyer wrote on 31 Aug 2021 06:03
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50105@debbugs.gnu.org)
87zgsyi7lg.fsf@gmail.com
Hi Mathieu,

[...]

Toggle quote (30 lines)
> I think the issue here lies in the sitecustomize.py file introduced with
> cb72f9a773e0931ee3758c851d96007ded034e4c.
>
> python_root = os.path.realpath(sys.executable).split('/bin/')[0]
>
>
> When using the Python C library, sys.executable is not the expected
> python binary but the test binary.
>
> So when we try to find the associated libraries in the Python search
> path, this way:
>
> index = sys_path_absolute.index(python_site)
> sys.path = sys.path[:index] + matching_sites + sys.path[index:]
>
>
> we get:
>
> ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
>
>
> Running the test with the PYTHONPATH set to:
>
> PYTHONPATH=/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages:$PYTHONPATH
>
> is a way to work around this issue.
>
> The fix here would be to replace sys.executable in the sitecustomize.py
> with something more adequate, but I have no idea what could it be.

From the doc [0], sys.prefix seems to be more correct. This had been
suggested by Hartmut some time ago and I had this patch ready recently.
Perhaps it already addresses the issue at hand?

You'll find it attached. It hasn't gotten much testing yet.

From e693b38a6d5c370e13f795c6303f50871ec78ae4 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Tue, 3 Aug 2021 20:41:20 -0400
Subject: [PATCH] aux-files: sitecustomize: Cleanup and add explanatory
comments.

* gnu/packages/aux-files/python/sitecustomize.py: Add a comment explaining the
general idea.

(major_minor): Use the unpacking operator (*) to provide the arguments.
(site_packages_prefix): Use os.path.join to form the path.
(python_site): Likewise.
(all_sites_raw): Split on os.path.pathsep.
(sys.path): Directly splice the result in the list.

Suggested-by: Hartmut Goebel <h.goebel@crazy-compilers.com>
---
.../aux-files/python/sitecustomize.py | 28 ++++++++++++++-----
1 file changed, 21 insertions(+), 7 deletions(-)

Toggle diff (49 lines)
diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
index 65d3c7d554..71e328b9ac 100644
--- a/gnu/packages/aux-files/python/sitecustomize.py
+++ b/gnu/packages/aux-files/python/sitecustomize.py
@@ -20,13 +20,26 @@
import os
import sys
-python_root = os.path.realpath(sys.executable).split('/bin/')[0]
-major_minor = '{}.{}'.format(sys.version_info[0], sys.version_info[1])
-site_packages_prefix = 'lib/python' + major_minor + '/site-packages'
-python_site = python_root + '/' + site_packages_prefix
+# Commentary:
+#
+# Site-specific customization for Guix.
+#
+# The program below honors the GUIX_PYTHONPATH environment variable to
+# discover Python packages. File names appearing in this variable that match
+# a predefined versioned installation prefix are added to the sys.path. To be
+# considered, a Python package must be installed under the
+# 'lib/pythonX.Y/site-packages' directory, where X and Y are the major and
+# minor version numbers of the Python interpreter.
+#
+# Code:
+
+major_minor = '{}.{}'.format(*sys.version_info)
+site_packages_prefix = os.path.join(
+ 'lib', 'python' + major_minor, 'site-packages')
+python_site = os.path.join(sys.prefix, site_packages_prefix)
try:
- all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(':')
+ all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)
except KeyError:
all_sites_raw = []
# Normalize paths, otherwise a trailing slash would cause it to not match.
@@ -35,7 +48,8 @@ matching_sites = [p for p in all_sites_norm
if p.endswith(site_packages_prefix)]
# Insert sites matching the current version into sys.path, right before
-# Python's own site.
+# Python's own site. This way, the user can override the libraries provided
+# by Python itself.
sys_path_absolute = [os.path.realpath(p) for p in sys.path]
index = sys_path_absolute.index(python_site)
-sys.path = sys.path[:index] + matching_sites + sys.path[index:]
+sys.path[index:index] = matching_sites
--
2.32.0
M
M
Maxim Cournoyer wrote on 31 Aug 2021 06:03
control message for bug #50105
(address . control@debbugs.gnu.org)
87y28ii7ks.fsf@gmail.com
severity 50105 important
quit
M
M
Maxim Cournoyer wrote on 31 Aug 2021 15:47
(address . control@debbugs.gnu.org)
87wno1iv4x.fsf@gmail.com
tags 50105 + patch
quit
M
M
Maxim Cournoyer wrote on 3 Sep 2021 16:41
Re: bug#50105: [core-updates] Python sitecustomize issue.
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50105@debbugs.gnu.org)
87y28dbu1z.fsf@gmail.com
Hi Mathieu,
Mathieu Othacehe <othacehe@gnu.org> writes:
Toggle quote (21 lines)
> Hello,
>
> I tried to upgrade glade to the 3.38.2 release on core-update and
> encountered the following test issue:
>
> 2/5 modules FAIL 0.29s killed by signal 5 SIGTRAP
>>>> GLADE_MODULE_SEARCH_PATH=/home/mathieu/glade-3.38.2/build/plugins/gtk+:/home/mathieu/glade-3.38.2/build/plugins/python:/home/mathieu/glade-3.38.2/build/plugins/gjs:/home/mathieu/glade-3.38.2/tests/modules MALLOC_PERTURB_=195 GLADE_TESTING=1 GLADE_CATALOG_SEARCH_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+:/home/mathieu/glade-3.38.2/plugins/python:/home/mathieu/glade-3.38.2/plugins/gjs:/home/mathieu/glade-3.38.2/tests/catalogs GLADE_PIXMAP_DIR=/home/mathieu/glade-3.38.2/data/icons GLADE_ICON_THEME_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+/icons/22x22 /home/mathieu/glade-3.38.2/build/tests/modules
> ?????????????????????????????????????????????????????????????????????????????????????????????????????? ? ???????????????????????????????????????????????????????????????????????????????????????????????????????
> stdout:
> # random seed: R02Sd222538bc9b76b6e424ec0cb17ee887c
> # GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ?gio-vfs?
> Bail out! GladeUI-PYTHON-FATAL-WARNING: Error initializing Python interpreter: could not import pygobject
> stderr:
> Error in sitecustomize; set PYTHONVERBOSE for traceback:
> ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
>
> (/home/mathieu/glade-3.38.2/build/tests/modules:4898): GladeUI-PYTHON-WARNING **: 09:43:58.605: Error initializing Python interpreter: could not import pygobject
>
>
> I think the issue here lies in the sitecustomize.py file introduced with
> cb72f9a773e0931ee3758c851d96007ded034e4c.
[...]
Is there a simple reproducer to test my fix? Would just building
current Glade and its test suite trigger it?
Thanks,
Maxim
M
M
Maxim Cournoyer wrote on 3 Sep 2021 17:21
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50105@debbugs.gnu.org)
87tuj1bs6p.fsf@gmail.com
Hi again,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

Toggle quote (3 lines)
> Is there a simple reproducer to test my fix? Would just building
> current Glade and its test suite trigger it?

OK, nevermind, I found that the test case was deleting the 'fix-tests'
build phase from glade3, and building the package. I confirm the fix
works!

I'll send that patch series I've been talking about.

Thanks,

Maxim
M
M
Mathieu Othacehe wrote on 3 Sep 2021 20:46
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 50105@debbugs.gnu.org)
87fsulqyyn.fsf@gnu.org
Hello Maxim,

Toggle quote (4 lines)
> OK, nevermind, I found that the test case was deleting the 'fix-tests'
> build phase from glade3, and building the package. I confirm the fix
> works!

Great, that's exactly what I was trying to test but you beat me to it
:). I'll be afk for a month so I won't be able to confirm that I have
the same result locally. However, this fix looks good to me.

Thanks,

Mathieu
M
M
Maxim Cournoyer wrote on 4 Dec 2021 04:08
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50105-done@debbugs.gnu.org)
87wnkl9ie8.fsf@gmail.com
Hello,

Mathieu Othacehe <othacehe@gnu.org> writes:

Toggle quote (10 lines)
> Hello Maxim,
>
>> OK, nevermind, I found that the test case was deleting the 'fix-tests'
>> build phase from glade3, and building the package. I confirm the fix
>> works!
>
> Great, that's exactly what I was trying to test but you beat me to it
> :). I'll be afk for a month so I won't be able to confirm that I have
> the same result locally. However, this fix looks good to me.

I had forgotten to close this. The fix has been in core-updates-frozen
for a while.

Thank you,

Maxim
Closed
?