[core-updates] Python sitecustomize issue.

  • Done
  • quality assurance status badge
Details
2 participants
  • Maxim Cournoyer
  • Mathieu Othacehe
Owner
unassigned
Submitted by
Mathieu Othacehe
Severity
important
M
M
Mathieu Othacehe wrote on 18 Aug 2021 11:53
(address . bug-guix@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
878s0zoz98.fsf@gnu.org
Hello,
I tried to upgrade glade to the 3.38.2 release on core-update and
encountered the following test issue:
Toggle snippet (13 lines)
2/5 modules FAIL 0.29s killed by signal 5 SIGTRAP
>>> GLADE_MODULE_SEARCH_PATH=/home/mathieu/glade-3.38.2/build/plugins/gtk+:/home/mathieu/glade-3.38.2/build/plugins/python:/home/mathieu/glade-3.38.2/build/plugins/gjs:/home/mathieu/glade-3.38.2/tests/modules MALLOC_PERTURB_=195 GLADE_TESTING=1 GLADE_CATALOG_SEARCH_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+:/home/mathieu/glade-3.38.2/plugins/python:/home/mathieu/glade-3.38.2/plugins/gjs:/home/mathieu/glade-3.38.2/tests/catalogs GLADE_PIXMAP_DIR=/home/mathieu/glade-3.38.2/data/icons GLADE_ICON_THEME_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+/icons/22x22 /home/mathieu/glade-3.38.2/build/tests/modules
?????????????????????????????????????????????????????????????????????????????????????????????????????? ? ???????????????????????????????????????????????????????????????????????????????????????????????????????
stdout:
# random seed: R02Sd222538bc9b76b6e424ec0cb17ee887c
# GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ?gio-vfs?
Bail out! GladeUI-PYTHON-FATAL-WARNING: Error initializing Python interpreter: could not import pygobject
stderr:
Error in sitecustomize; set PYTHONVERBOSE for traceback:
ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
(/home/mathieu/glade-3.38.2/build/tests/modules:4898): GladeUI-PYTHON-WARNING **: 09:43:58.605: Error initializing Python interpreter: could not import pygobject
I think the issue here lies in the sitecustomize.py file introduced with
cb72f9a773e0931ee3758c851d96007ded034e4c.
Toggle snippet (2 lines)
python_root = os.path.realpath(sys.executable).split('/bin/')[0]
When using the Python C library, sys.executable is not the expected
python binary but the test binary.
So when we try to find the associated libraries in the Python search
path, this way:
Toggle snippet (3 lines)
index = sys_path_absolute.index(python_site)
sys.path = sys.path[:index] + matching_sites + sys.path[index:]
we get:
Toggle snippet (2 lines)
ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
Running the test with the PYTHONPATH set to:
Toggle snippet (2 lines)
PYTHONPATH=/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages:$PYTHONPATH
is a way to work around this issue.
The fix here would be to replace sys.executable in the sitecustomize.py
with something more adequate, but I have no idea what could it be.
Thanks,
Mathieu
M
M
Maxim Cournoyer wrote on 31 Aug 2021 06:03
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50105@debbugs.gnu.org)
87zgsyi7lg.fsf@gmail.com
Hi Mathieu,

[...]

Toggle quote (30 lines)
> I think the issue here lies in the sitecustomize.py file introduced with
> cb72f9a773e0931ee3758c851d96007ded034e4c.
>
> python_root = os.path.realpath(sys.executable).split('/bin/')[0]
>
>
> When using the Python C library, sys.executable is not the expected
> python binary but the test binary.
>
> So when we try to find the associated libraries in the Python search
> path, this way:
>
> index = sys_path_absolute.index(python_site)
> sys.path = sys.path[:index] + matching_sites + sys.path[index:]
>
>
> we get:
>
> ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
>
>
> Running the test with the PYTHONPATH set to:
>
> PYTHONPATH=/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages:$PYTHONPATH
>
> is a way to work around this issue.
>
> The fix here would be to replace sys.executable in the sitecustomize.py
> with something more adequate, but I have no idea what could it be.

From the doc [0], sys.prefix seems to be more correct. This had been
suggested by Hartmut some time ago and I had this patch ready recently.
Perhaps it already addresses the issue at hand?

You'll find it attached. It hasn't gotten much testing yet.

From e693b38a6d5c370e13f795c6303f50871ec78ae4 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date: Tue, 3 Aug 2021 20:41:20 -0400
Subject: [PATCH] aux-files: sitecustomize: Cleanup and add explanatory
comments.

* gnu/packages/aux-files/python/sitecustomize.py: Add a comment explaining the
general idea.

(major_minor): Use the unpacking operator (*) to provide the arguments.
(site_packages_prefix): Use os.path.join to form the path.
(python_site): Likewise.
(all_sites_raw): Split on os.path.pathsep.
(sys.path): Directly splice the result in the list.

Suggested-by: Hartmut Goebel <h.goebel@crazy-compilers.com>
---
.../aux-files/python/sitecustomize.py | 28 ++++++++++++++-----
1 file changed, 21 insertions(+), 7 deletions(-)

Toggle diff (49 lines)
diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
index 65d3c7d554..71e328b9ac 100644
--- a/gnu/packages/aux-files/python/sitecustomize.py
+++ b/gnu/packages/aux-files/python/sitecustomize.py
@@ -20,13 +20,26 @@
import os
import sys
-python_root = os.path.realpath(sys.executable).split('/bin/')[0]
-major_minor = '{}.{}'.format(sys.version_info[0], sys.version_info[1])
-site_packages_prefix = 'lib/python' + major_minor + '/site-packages'
-python_site = python_root + '/' + site_packages_prefix
+# Commentary:
+#
+# Site-specific customization for Guix.
+#
+# The program below honors the GUIX_PYTHONPATH environment variable to
+# discover Python packages. File names appearing in this variable that match
+# a predefined versioned installation prefix are added to the sys.path. To be
+# considered, a Python package must be installed under the
+# 'lib/pythonX.Y/site-packages' directory, where X and Y are the major and
+# minor version numbers of the Python interpreter.
+#
+# Code:
+
+major_minor = '{}.{}'.format(*sys.version_info)
+site_packages_prefix = os.path.join(
+ 'lib', 'python' + major_minor, 'site-packages')
+python_site = os.path.join(sys.prefix, site_packages_prefix)
try:
- all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(':')
+ all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)
except KeyError:
all_sites_raw = []
# Normalize paths, otherwise a trailing slash would cause it to not match.
@@ -35,7 +48,8 @@ matching_sites = [p for p in all_sites_norm
if p.endswith(site_packages_prefix)]
# Insert sites matching the current version into sys.path, right before
-# Python's own site.
+# Python's own site. This way, the user can override the libraries provided
+# by Python itself.
sys_path_absolute = [os.path.realpath(p) for p in sys.path]
index = sys_path_absolute.index(python_site)
-sys.path = sys.path[:index] + matching_sites + sys.path[index:]
+sys.path[index:index] = matching_sites
--
2.32.0
M
M
Maxim Cournoyer wrote on 31 Aug 2021 06:03
control message for bug #50105
(address . control@debbugs.gnu.org)
87y28ii7ks.fsf@gmail.com
severity 50105 important
quit
M
M
Maxim Cournoyer wrote on 31 Aug 2021 15:47
(address . control@debbugs.gnu.org)
87wno1iv4x.fsf@gmail.com
tags 50105 + patch
quit
M
M
Maxim Cournoyer wrote on 3 Sep 2021 16:41
Re: bug#50105: [core-updates] Python sitecustomize issue.
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50105@debbugs.gnu.org)
87y28dbu1z.fsf@gmail.com
Hi Mathieu,
Mathieu Othacehe <othacehe@gnu.org> writes:
Toggle quote (21 lines)
> Hello,
>
> I tried to upgrade glade to the 3.38.2 release on core-update and
> encountered the following test issue:
>
> 2/5 modules FAIL 0.29s killed by signal 5 SIGTRAP
>>>> GLADE_MODULE_SEARCH_PATH=/home/mathieu/glade-3.38.2/build/plugins/gtk+:/home/mathieu/glade-3.38.2/build/plugins/python:/home/mathieu/glade-3.38.2/build/plugins/gjs:/home/mathieu/glade-3.38.2/tests/modules MALLOC_PERTURB_=195 GLADE_TESTING=1 GLADE_CATALOG_SEARCH_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+:/home/mathieu/glade-3.38.2/plugins/python:/home/mathieu/glade-3.38.2/plugins/gjs:/home/mathieu/glade-3.38.2/tests/catalogs GLADE_PIXMAP_DIR=/home/mathieu/glade-3.38.2/data/icons GLADE_ICON_THEME_PATH=/home/mathieu/glade-3.38.2/plugins/gtk+/icons/22x22 /home/mathieu/glade-3.38.2/build/tests/modules
> ?????????????????????????????????????????????????????????????????????????????????????????????????????? ? ???????????????????????????????????????????????????????????????????????????????????????????????????????
> stdout:
> # random seed: R02Sd222538bc9b76b6e424ec0cb17ee887c
> # GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ?gio-vfs?
> Bail out! GladeUI-PYTHON-FATAL-WARNING: Error initializing Python interpreter: could not import pygobject
> stderr:
> Error in sitecustomize; set PYTHONVERBOSE for traceback:
> ValueError: '/home/mathieu/glade-3.38.2/build/lib/python3.9/site-packages' is not in list
>
> (/home/mathieu/glade-3.38.2/build/tests/modules:4898): GladeUI-PYTHON-WARNING **: 09:43:58.605: Error initializing Python interpreter: could not import pygobject
>
>
> I think the issue here lies in the sitecustomize.py file introduced with
> cb72f9a773e0931ee3758c851d96007ded034e4c.
[...]
Is there a simple reproducer to test my fix? Would just building
current Glade and its test suite trigger it?
Thanks,
Maxim
M
M
Maxim Cournoyer wrote on 3 Sep 2021 17:21
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50105@debbugs.gnu.org)
87tuj1bs6p.fsf@gmail.com
Hi again,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

Toggle quote (3 lines)
> Is there a simple reproducer to test my fix? Would just building
> current Glade and its test suite trigger it?

OK, nevermind, I found that the test case was deleting the 'fix-tests'
build phase from glade3, and building the package. I confirm the fix
works!

I'll send that patch series I've been talking about.

Thanks,

Maxim
M
M
Mathieu Othacehe wrote on 3 Sep 2021 20:46
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 50105@debbugs.gnu.org)
87fsulqyyn.fsf@gnu.org
Hello Maxim,

Toggle quote (4 lines)
> OK, nevermind, I found that the test case was deleting the 'fix-tests'
> build phase from glade3, and building the package. I confirm the fix
> works!

Great, that's exactly what I was trying to test but you beat me to it
:). I'll be afk for a month so I won't be able to confirm that I have
the same result locally. However, this fix looks good to me.

Thanks,

Mathieu
M
M
Maxim Cournoyer wrote on 4 Dec 2021 04:08
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50105-done@debbugs.gnu.org)
87wnkl9ie8.fsf@gmail.com
Hello,

Mathieu Othacehe <othacehe@gnu.org> writes:

Toggle quote (10 lines)
> Hello Maxim,
>
>> OK, nevermind, I found that the test case was deleting the 'fix-tests'
>> build phase from glade3, and building the package. I confirm the fix
>> works!
>
> Great, that's exactly what I was trying to test but you beat me to it
> :). I'll be afk for a month so I won't be able to confirm that I have
> the same result locally. However, this fix looks good to me.

I had forgotten to close this. The fix has been in core-updates-frozen
for a while.

Thank you,

Maxim
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 50105@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 50105
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch