‘guix import pypi’ misses package dependencies

  • Open
  • quality assurance status badge
Details
5 participants
  • Leo Prikler
  • Ludovic Courtès
  • Ludovic Courtès
  • Maxim Cournoyer
  • zimoun
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 22 Jun 2021 09:20
(address . bug-guix@gnu.org)
877dim5psf.fsf@inria.fr
Hi!

‘guix import pypi -r django-import-export’ does not include all the
inputs:

Toggle snippet (93 lines)
$ guix build -L /tmp/mods python-django-import-export

[...]

starting phase `build'
running "python setup.py" with command "build" and parameters ()
Download error on https://pypi.org/simple/setuptools_scm/: [Errno -2] Name or service not known -- Some packages may not be found!
Download error on https://pypi.org/simple/setuptools-scm/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'setuptools_scm' (maybe misspelled?)
Download error on https://pypi.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for setuptools_scm
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "setup.py", line 5, in <module>
setup(
File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
_install_setup_requires(attrs)
File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/dist.py", line 716, in fetch_build_eggs
resolved_dists = pkg_resources.working_set.resolve(
File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 780, in resolve
dist = best[req.key] = env.best_match(
File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1065, in best_match
return self.obtain(req, installer)
File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1077, in obtain
return installer(requirement)
File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/dist.py", line 786, in fetch_build_egg
return cmd.easy_install(req)
File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-3.8.2/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('setuptools_scm')
command "python" "-c" "import setuptools, tokenize;__file__='setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\\r\\n', '\\n');f.close();exec(compile(code, __file__, 'exec'))" "build" failed with status 1
builder for `/gnu/store/74ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-3.0.0.drv' failed with exit code 1
build of /gnu/store/74ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-3.0.0.drv failed
View build log at '/var/log/guix/drvs/74/ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-3.0.0.drv.bz2'.
cannot build derivation `/gnu/store/6xk0jk0qnjsazqgjihmpafsx3wfmxjf0-python-django-import-export-2.5.0.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/6xk0jk0qnjsazqgjihmpafsx3wfmxjf0-python-django-import-export-2.5.0.drv' failed
$ cat /tmp/mods/django.scm
(define-module (django)
#:use-module (guix)
#:use-module (guix build-system python)
#:use-module (gnu packages python)
#:use-module (gnu packages python-xyz)
#:use-module (gnu packages django)
#:use-module ((guix licenses) #:prefix license:))

(define-public python-tablib
(package
(name "python-tablib")
(version "3.0.0")
(source
(origin
(method url-fetch)
(uri (pypi-uri "tablib" version))
(sha256
(base32
"03f1z6jq6rf67gwhbm9ma4rydm8h447a5nh5lcs5l8jg8l4aqg7q"))))
(build-system python-build-system)
(home-page "https://tablib.readthedocs.io")
(synopsis
"Format agnostic tabular data library (XLS, JSON, YAML, CSV)")
(description
"Format agnostic tabular data library (XLS, JSON, YAML, CSV)")
(license license:expat)))

(define-public python-django-import-export
(package
(name "python-django-import-export")
(version "2.5.0")
(source
(origin
(method url-fetch)
(uri (pypi-uri "django-import-export" version))
(sha256
(base32
"0j0gsj18bkkmbn262qj2syla80vncvqn49bllwxvcgw0zhxh1763"))))
(build-system python-build-system)
(propagated-inputs
`(("python-diff-match-patch"
,python-diff-match-patch)
("python-django" ,python-django)
("python-tablib" ,python-tablib)))
(home-page
"https://github.com/django-import-export/django-import-export")
(synopsis
"Django application and library for importing and exporting data with included admin integration.")
(description
"Django application and library for importing and exporting data with included admin integration.")
(license license:bsd-3)))


Here, some of tablib’s dependencies are missing; the same problem seems
to happen with python-django-import-export itself.

Are we failing to parse some of the PyPI metadata?

Thanks,
Ludo’.
L
L
Leo Prikler wrote on 22 Jun 2021 10:34
992e746ed43c5ae7def7b35996ed44743fef85bf.camel@student.tugraz.at
Am Dienstag, den 22.06.2021, 09:20 +0200 schrieb Ludovic Courtès:
Toggle quote (141 lines)
> Hi!
>
> ‘guix import pypi -r django-import-export’ does not include all the
> inputs:
>
> --8<---------------cut here---------------start------------->8---
> $ guix build -L /tmp/mods python-django-import-export
>
> [...]
>
> starting phase `build'
> running "python setup.py" with command "build" and parameters ()
> Download error on https://pypi.org/simple/setuptools_scm/: [Errno -2]
> Name or service not known -- Some packages may not be found!
> Download error on https://pypi.org/simple/setuptools-scm/: [Errno -2]
> Name or service not known -- Some packages may not be found!
> Couldn't find index page for 'setuptools_scm' (maybe misspelled?)
> Download error on https://pypi.org/simple/: [Errno -2] Name or
> service not known -- Some packages may not be found!
> No local packages or working download links found for setuptools_scm
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> File "setup.py", line 5, in <module>
> setup(
> File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/setuptools/__init__.py", line 144,
> in setup
> _install_setup_requires(attrs)
> File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/setuptools/__init__.py", line 139,
> in _install_setup_requires
> dist.fetch_build_eggs(dist.setup_requires)
> File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/setuptools/dist.py", line 716, in
> fetch_build_eggs
> resolved_dists = pkg_resources.working_set.resolve(
> File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line
> 780, in resolve
> dist = best[req.key] = env.best_match(
> File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line
> 1065, in best_match
> return self.obtain(req, installer)
> File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line
> 1077, in obtain
> return installer(requirement)
> File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-packages/setuptools/dist.py", line 786, in
> fetch_build_egg
> return cmd.easy_install(req)
> File "/gnu/store/9w9jvy3bgjg4qaqmrij01nbppiccqr7c-python-
> 3.8.2/lib/python3.8/site-
> packages/setuptools/command/easy_install.py", line 673, in
> easy_install
> raise DistutilsError(msg)
> distutils.errors.DistutilsError: Could not find suitable distribution
> for Requirement.parse('setuptools_scm')
> command "python" "-c" "import setuptools,
> tokenize;__file__='setup.py';f=getattr(tokenize, 'open',
> open)(__file__);code=f.read().replace('\\r\\n',
> '\\n');f.close();exec(compile(code, __file__, 'exec'))" "build"
> failed with status 1
> builder for `/gnu/store/74ph75dfy1i9hrx9n327j32ma2zv4hhd-python-
> tablib-3.0.0.drv' failed with exit code 1
> build of /gnu/store/74ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-
> 3.0.0.drv failed
> View build log at
> '/var/log/guix/drvs/74/ph75dfy1i9hrx9n327j32ma2zv4hhd-python-tablib-
> 3.0.0.drv.bz2'.
> cannot build derivation `/gnu/store/6xk0jk0qnjsazqgjihmpafsx3wfmxjf0-
> python-django-import-export-2.5.0.drv': 1 dependencies couldn't be
> built
> guix build: error: build of
> `/gnu/store/6xk0jk0qnjsazqgjihmpafsx3wfmxjf0-python-django-import-
> export-2.5.0.drv' failed
> $ cat /tmp/mods/django.scm
> (define-module (django)
> #:use-module (guix)
> #:use-module (guix build-system python)
> #:use-module (gnu packages python)
> #:use-module (gnu packages python-xyz)
> #:use-module (gnu packages django)
> #:use-module ((guix licenses) #:prefix license:))
>
> (define-public python-tablib
> (package
> (name "python-tablib")
> (version "3.0.0")
> (source
> (origin
> (method url-fetch)
> (uri (pypi-uri "tablib" version))
> (sha256
> (base32
> "03f1z6jq6rf67gwhbm9ma4rydm8h447a5nh5lcs5l8jg8l4aqg7q")))
> )
> (build-system python-build-system)
> (home-page "https://tablib.readthedocs.io")
> (synopsis
> "Format agnostic tabular data library (XLS, JSON, YAML, CSV)")
> (description
> "Format agnostic tabular data library (XLS, JSON, YAML, CSV)")
> (license license:expat)))
>
> (define-public python-django-import-export
> (package
> (name "python-django-import-export")
> (version "2.5.0")
> (source
> (origin
> (method url-fetch)
> (uri (pypi-uri "django-import-export" version))
> (sha256
> (base32
> "0j0gsj18bkkmbn262qj2syla80vncvqn49bllwxvcgw0zhxh1763")))
> )
> (build-system python-build-system)
> (propagated-inputs
> `(("python-diff-match-patch"
> ,python-diff-match-patch)
> ("python-django" ,python-django)
> ("python-tablib" ,python-tablib)))
> (home-page
> "https://github.com/django-import-export/django-import-export")
> (synopsis
> "Django application and library for importing and exporting
> data with included admin integration.")
> (description
> "Django application and library for importing and exporting
> data with included admin integration.")
> (license license:bsd-3)))
>
> --8<---------------cut here---------------end--------------->8---
>
> Here, some of tablib’s dependencies are missing; the same problem
> seems
> to happen with python-django-import-export itself.
>
> Are we failing to parse some of the PyPI metadata?
I don't think so, rather it appears as though the requires.txt is
somehow malformed or ill-suited for importing. It basically marks all
inputs as optional.

Perhaps we could instruct the importer to emit comments for optional
inputs like so
;; Optional inputs by section
;; -- [SECTION]
;; ("python-some-package" ,python-some-package)
;; ("python-some-other-package" ,python-some-other-package)
Maybe add a command line switch to toggle these comments or a way of
assuming some section, e.g. [cli] to be the thing we want to build for
a package.

Regards,
Leo
L
L
Ludovic Courtès wrote on 22 Jun 2021 14:33
(name . Leo Prikler)(address . leo.prikler@student.tugraz.at)(address . 49168@debbugs.gnu.org)
87eecu3wqh.fsf@inria.fr
Hi,

Leo Prikler <leo.prikler@student.tugraz.at> skribis:

Toggle quote (9 lines)
>> Here, some of tablib’s dependencies are missing; the same problem
>> seems
>> to happen with python-django-import-export itself.
>>
>> Are we failing to parse some of the PyPI metadata?
> I don't think so, rather it appears as though the requires.txt is
> somehow malformed or ill-suited for importing. It basically marks all
> inputs as optional.

Oh. So that’s first and foremost a bug of the PyPI package, right?

Actually
does not have a ‘requirements.txt’ file, and
doesn’t have much metadata, so I don’t even get where were get that
info.

Toggle quote (10 lines)
> Perhaps we could instruct the importer to emit comments for optional
> inputs like so
> ;; Optional inputs by section
> ;; -- [SECTION]
> ;; ("python-some-package" ,python-some-package)
> ;; ("python-some-other-package" ,python-some-other-package)
> Maybe add a command line switch to toggle these comments or a way of
> assuming some section, e.g. [cli] to be the thing we want to build for
> a package.

A CLI switch to include optional dependencies would be nice.

As for emitting comments, the (not-so-) pretty printer written for ‘guix
style’¹ could come in handy for importers and more generally in any
place where we need to emit comments.

Thanks,
Ludo’.

L
L
Leo Prikler wrote on 22 Jun 2021 14:37
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 49168@debbugs.gnu.org)
ab46529b6773ef0a1b73504470be29a757a3dc3a.camel@student.tugraz.at
Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:
Toggle quote (26 lines)
> Hi,
>
> Leo Prikler <leo.prikler@student.tugraz.at> skribis:
>
> > > Here, some of tablib’s dependencies are missing; the same problem
> > > seems
> > > to happen with python-django-import-export itself.
> > >
> > > Are we failing to parse some of the PyPI metadata?
> > I don't think so, rather it appears as though the requires.txt is
> > somehow malformed or ill-suited for importing. It basically marks
> > all
> > inputs as optional.
>
> Oh. So that’s first and foremost a bug of the PyPI package, right?
>
> Actually
> <
> https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz
> >
> does not have a ‘requirements.txt’ file, and
> <
> https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl
> >
> doesn’t have much metadata, so I don’t even get where were get that
> info.
It does, but it's well hidden in the src tree. I peeked into the guix
import code to find it.

Toggle quote (19 lines)
> > Perhaps we could instruct the importer to emit comments for
> > optional
> > inputs like so
> > ;; Optional inputs by section
> > ;; -- [SECTION]
> > ;; ("python-some-package" ,python-some-package)
> > ;; ("python-some-other-package" ,python-some-other-package)
> > Maybe add a command line switch to toggle these comments or a way
> > of
> > assuming some section, e.g. [cli] to be the thing we want to build
> > for
> > a package.
>
> A CLI switch to include optional dependencies would be nice.
>
> As for emitting comments, the (not-so-) pretty printer written for
> ‘guix
> style’¹ could come in handy for importers and more generally in any
> place where we need to emit comments.
Indeed, I wasn't thinking about that when I wrote this reply, but I see
what you mean. Should we strive to have such a pretty-printer in Guile
as well or do we keep it to ourselves for now? :)

Toggle quote (4 lines)
> Thanks,
> Ludo’.
>
> ¹ https://issues.guix.gnu.org/49169
M
M
Maxim Cournoyer wrote on 22 Jun 2021 21:01
Re: bug#49168: ‘guix import pypi ’ misses package dependencies
(name . Leo Prikler)(address . leo.prikler@student.tugraz.at)
875yy53erq.fsf@gmail.com
Hello,

Leo Prikler <leo.prikler@student.tugraz.at> writes:

Toggle quote (17 lines)
> Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:
>> Hi,
>>
>> Leo Prikler <leo.prikler@student.tugraz.at> skribis:
>>
>> > > Here, some of tablib’s dependencies are missing; the same problem
>> > > seems
>> > > to happen with python-django-import-export itself.
>> > >
>> > > Are we failing to parse some of the PyPI metadata?
>> > I don't think so, rather it appears as though the requires.txt is
>> > somehow malformed or ill-suited for importing. It basically marks
>> > all
>> > inputs as optional.
>>
>> Oh. So that’s first and foremost a bug of the PyPI package, right?

Sounds like so, if the inputs are required but marked as optional.

Toggle quote (15 lines)
>> > Perhaps we could instruct the importer to emit comments for
>> > optional
>> > inputs like so
>> > ;; Optional inputs by section
>> > ;; -- [SECTION]
>> > ;; ("python-some-package" ,python-some-package)
>> > ;; ("python-some-other-package" ,python-some-other-package)
>> > Maybe add a command line switch to toggle these comments or a way
>> > of
>> > assuming some section, e.g. [cli] to be the thing we want to build
>> > for
>> > a package.
>>
>> A CLI switch to include optional dependencies would be nice.

+1.

Toggle quote (8 lines)
>> As for emitting comments, the (not-so-) pretty printer written for
>> ‘guix
>> style’¹ could come in handy for importers and more generally in any
>> place where we need to emit comments.
> Indeed, I wasn't thinking about that when I wrote this reply, but I see
> what you mean. Should we strive to have such a pretty-printer in Guile
> as well or do we keep it to ourselves for now? :)

This sounds good too, although I haven't read about that new fancy
printer yet.

Maxim
L
L
Ludovic Courtès wrote on 23 Jun 2021 15:20
(name . Leo Prikler)(address . leo.prikler@student.tugraz.at)(address . 49168@debbugs.gnu.org)
87mtrgyaxe.fsf@gnu.org
Hi,

Leo Prikler <leo.prikler@student.tugraz.at> skribis:

Toggle quote (2 lines)
> Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:

[...]

Toggle quote (13 lines)
>> Actually
>> <
>> https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz
>> >
>> does not have a ‘requirements.txt’ file, and
>> <
>> https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl
>> >
>> doesn’t have much metadata, so I don’t even get where were get that
>> info.
> It does, but it's well hidden in the src tree. I peeked into the guix
> import code to find it.

Indeed. The tarball above has ‘tests/requirements.txt’:

Toggle snippet (12 lines)
pytest
pytest-cov
MarkupPy
odfpy
openpyxl>=2.6.0
pandas
pyyaml
tabulate
xlrd
xlwt

There are no optional dependencies in that file, though. Or were you
looking at something else?

Toggle quote (8 lines)
>> As for emitting comments, the (not-so-) pretty printer written for
>> ‘guix
>> style’¹ could come in handy for importers and more generally in any
>> place where we need to emit comments.
> Indeed, I wasn't thinking about that when I wrote this reply, but I see
> what you mean. Should we strive to have such a pretty-printer in Guile
> as well or do we keep it to ourselves for now? :)

Let’s nurture it in Guix and move it to Guile when it looks better.
Likewise, we should have support for comments directly in (ice-9 read).

Ludo’.
L
L
Leo Prikler wrote on 23 Jun 2021 16:17
Re: bug#49168: ‘guix import pypi’ misses package dependencies
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 49168@debbugs.gnu.org)
7da007493c9bcea075a76dc01d6e85b457c5d4e3.camel@student.tugraz.at
Am Mittwoch, den 23.06.2021, 15:20 +0200 schrieb Ludovic Courtès:
Toggle quote (38 lines)
> Hi,
>
> Leo Prikler <leo.prikler@student.tugraz.at> skribis:
>
> > Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:
>
> [...]
>
> > > Actually
> > > <
> > > https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz
> > > does not have a ‘requirements.txt’ file, and
> > > <
> > > https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl
> > > doesn’t have much metadata, so I don’t even get where were get
> > > that
> > > info.
> > It does, but it's well hidden in the src tree. I peeked into the
> > guix
> > import code to find it.
>
> Indeed. The tarball above has ‘tests/requirements.txt’:
>
> --8<---------------cut here---------------start------------->8---
> pytest
> pytest-cov
> MarkupPy
> odfpy
> openpyxl>=2.6.0
> pandas
> pyyaml
> tabulate
> xlrd
> xlwt
> --8<---------------cut here---------------end--------------->8---
>
> There are no optional dependencies in that file, though. Or were you
> looking at something else?
The importer and I are looking at something else:

Toggle snippet (37 lines)
$ tar xfv tablib-3.0.0.tar.gz tablib-3.0.0/src/tablib.egg-
info/requires.txt | xargs cat

[all]
markuppy
odfpy
openpyxl>=2.6.0
pandas
pyyaml
tabulate
xlrd
xlwt

[cli]
tabulate

[html]
markuppy

[ods]
odfpy

[pandas]
pandas

[xls]
xlrd
xlwt

[xlsx]
openpyxl>=2.6.0

[yaml]
pyyaml


Toggle quote (15 lines)
> > > As for emitting comments, the (not-so-) pretty printer written
> > > for
> > > ‘guix
> > > style’¹ could come in handy for importers and more generally in
> > > any
> > > place where we need to emit comments.
> > Indeed, I wasn't thinking about that when I wrote this reply, but I
> > see
> > what you mean. Should we strive to have such a pretty-printer in
> > Guile
> > as well or do we keep it to ourselves for now? :)
>
> Let’s nurture it in Guix and move it to Guile when it looks better.
> Likewise, we should have support for comments directly in (ice-9
> read).
Sounds good to me.
L
L
Ludovic Courtès wrote on 24 Jun 2021 11:57
Re: bug#49168: ‘guix import pypi ’ misses package dependencies
(name . Leo Prikler)(address . leo.prikler@student.tugraz.at)(address . 49168@debbugs.gnu.org)
87a6nftwio.fsf@gnu.org
Hi,

Leo Prikler <leo.prikler@student.tugraz.at> skribis:

Toggle quote (76 lines)
> Am Mittwoch, den 23.06.2021, 15:20 +0200 schrieb Ludovic Courtès:
>> Hi,
>>
>> Leo Prikler <leo.prikler@student.tugraz.at> skribis:
>>
>> > Am Dienstag, den 22.06.2021, 14:33 +0200 schrieb Ludovic Courtès:
>>
>> [...]
>>
>> > > Actually
>> > > <
>> > > https://files.pythonhosted.org/packages/fe/9d/4e15b2e74044ee051b6939c1b3ff716b0106e8f72d78eab8e08212eab44c/tablib-3.0.0.tar.gz
>> > > does not have a ‘requirements.txt’ file, and
>> > > <
>> > > https://files.pythonhosted.org/packages/16/85/078fc037b15aa1120d6a0287ec9d092d93d632ab01a0e7a3e69b4733da5e/tablib-3.0.0-py3-none-any.whl
>> > > doesn’t have much metadata, so I don’t even get where were get
>> > > that
>> > > info.
>> > It does, but it's well hidden in the src tree. I peeked into the
>> > guix
>> > import code to find it.
>>
>> Indeed. The tarball above has ‘tests/requirements.txt’:
>>
>> --8<---------------cut here---------------start------------->8---
>> pytest
>> pytest-cov
>> MarkupPy
>> odfpy
>> openpyxl>=2.6.0
>> pandas
>> pyyaml
>> tabulate
>> xlrd
>> xlwt
>> --8<---------------cut here---------------end--------------->8---
>>
>> There are no optional dependencies in that file, though. Or were you
>> looking at something else?
> The importer and I are looking at something else:
>
> $ tar xfv tablib-3.0.0.tar.gz tablib-3.0.0/src/tablib.egg-
> info/requires.txt | xargs cat
>
> [all]
> markuppy
> odfpy
> openpyxl>=2.6.0
> pandas
> pyyaml
> tabulate
> xlrd
> xlwt
>
> [cli]
> tabulate
>
> [html]
> markuppy
>
> [ods]
> odfpy
>
> [pandas]
> pandas
>
> [xls]
> xlrd
> xlwt
>
> [xlsx]
> openpyxl>=2.6.0
>
> [yaml]
> pyyaml

Oooh, I see.

So I came up with the following patch, which adds a flag for optional
dependencies, based on ‘requires.txt’.

Unfortunately, it has no effect for ‘guix import pypi tablib’ because we
only look at Wheel info in that case, as per:

;; First, try to compute the requirements using the wheel, else, fallback to
;; reading the "requires.txt" from the egg-info directory from the source
;; archive.
(or (guess-requirements-from-wheel)
(guess-requirements-from-source))

AFAICS, wheels don’t provide that info, do they?

Why does the importer favor .whl in the first place? Is it supposed to
be more accurate or more widespread or something?

Thoughts?

Thanks,
Ludo’.
Attachment: file
M
M
Maxim Cournoyer wrote on 24 Jun 2021 21:04
(name . Ludovic Courtès)(address . ludo@gnu.org)
87pmwbythg.fsf@gmail.com
Hi,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

Toggle quote (17 lines)
> So I came up with the following patch, which adds a flag for optional
> dependencies, based on ‘requires.txt’.
>
> Unfortunately, it has no effect for ‘guix import pypi tablib’ because we
> only look at Wheel info in that case, as per:
>
> ;; First, try to compute the requirements using the wheel, else, fallback to
> ;; reading the "requires.txt" from the egg-info directory from the source
> ;; archive.
> (or (guess-requirements-from-wheel)
> (guess-requirements-from-source))
>
> AFAICS, wheels don’t provide that info, do they?
>
> Why does the importer favor .whl in the first place? Is it supposed to
> be more accurate or more widespread or something?

Yes, the METADATA file from the binary wheel is a better place to look
than the source egg-info requires.txt file. In my commit 01589acc5e1, I
simplified a comment that used used to read as:

- ;; First, try to compute the requirements using the wheel, since that is the
- ;; most reliable option. If a wheel is not provided for this package, try
- ;; getting them by reading either the "requirements.txt" file or the
- ;; "requires.txt" from the egg-info directory from the source tarball. Note
- ;; that "requirements.txt" is not mandatory, so this is likely to fail.
+ ;; First, try to compute the requirements using the wheel, else, fallback to
+ ;; reading the "requires.txt" from the egg-info directory from the source
+ ;; tarball.

The wheel (.whl) binary format is well specified as PEP 427 [0] and is
what pip primarily uses for installing Python packages, making it a very
reliable source of metadata. The Python egg is the predecessor of the
wheel, and can be considered obsolete, which explains why it's used as a
fallback.

Perhaps it'd be best to raise the issue to the package maintainers and
have them specify their metadata correctly?

Having the code you wrote to allow importing optional dependencies is
still a nice (optional) option to have though. It was originally left
out based on comments from Ricardo that it wouldn't make a good default
due to raising the packaging effort.

HTH,

Maxim

L
L
Ludovic Courtès wrote on 25 Jun 2021 16:51
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87mtre2e1e.fsf@gnu.org
Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (2 lines)
> Ludovic Courtès <ludo@gnu.org> writes:

[...]

Toggle quote (22 lines)
>> Why does the importer favor .whl in the first place? Is it supposed to
>> be more accurate or more widespread or something?
>
> Yes, the METADATA file from the binary wheel is a better place to look
> than the source egg-info requires.txt file. In my commit 01589acc5e1, I
> simplified a comment that used used to read as:
>
> - ;; First, try to compute the requirements using the wheel, since that is the
> - ;; most reliable option. If a wheel is not provided for this package, try
> - ;; getting them by reading either the "requirements.txt" file or the
> - ;; "requires.txt" from the egg-info directory from the source tarball. Note
> - ;; that "requirements.txt" is not mandatory, so this is likely to fail.
> + ;; First, try to compute the requirements using the wheel, else, fallback to
> + ;; reading the "requires.txt" from the egg-info directory from the source
> + ;; tarball.
>
> The wheel (.whl) binary format is well specified as PEP 427 [0] and is
> what pip primarily uses for installing Python packages, making it a very
> reliable source of metadata. The Python egg is the predecessor of the
> wheel, and can be considered obsolete, which explains why it's used as a
> fallback.

Oh, I see.

Toggle quote (3 lines)
> Perhaps it'd be best to raise the issue to the package maintainers and
> have them specify their metadata correctly?

Going back to the example at the beginning of this thread, what ‘guix
import pypi tablib’ produces is missing ‘python-setuptools-scm’.
Indeed, ‘METADATA’ doesn’t mention it.

Is it really a bug on their side, or is it something peculiar about Guix
packaging? Perhaps ‘python-setuptools-scm’ should be provided more or
less by default?

Toggle quote (5 lines)
> Having the code you wrote to allow importing optional dependencies is
> still a nice (optional) option to have though. It was originally left
> out based on comments from Ricardo that it wouldn't make a good default
> due to raising the packaging effort.

‘METADATA’ format apparently doesn’t support optional dependencies
anyway (which makes sense, because Wheels are a binary format), so
perhaps that idea was misguided.

(However ‘METADATA’ defines ‘Requires-External’, which the importer
could usefully interpret!)

WDYT?

Ludo’.
L
L
Leo Prikler wrote on 25 Jun 2021 17:25
Re: bug#49168: ‘guix import pypi’ misses package dependencies
(address . 49168@debbugs.gnu.org)
18276660c96637b7a17db80ca0712f2ea308aa2d.camel@student.tugraz.at
Am Freitag, den 25.06.2021, 16:51 +0200 schrieb Ludovic Courtès:
Toggle quote (59 lines)
> Hello,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
> > Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
> > > Why does the importer favor .whl in the first place? Is it
> > > supposed to
> > > be more accurate or more widespread or something?
> >
> > Yes, the METADATA file from the binary wheel is a better place to
> > look
> > than the source egg-info requires.txt file. In my commit
> > 01589acc5e1, I
> > simplified a comment that used used to read as:
> >
> > - ;; First, try to compute the requirements using the wheel, since
> > that is the
> > - ;; most reliable option. If a wheel is not provided for this
> > package, try
> > - ;; getting them by reading either the "requirements.txt" file or
> > the
> > - ;; "requires.txt" from the egg-info directory from the source
> > tarball. Note
> > - ;; that "requirements.txt" is not mandatory, so this is likely
> > to fail.
> > + ;; First, try to compute the requirements using the wheel, else,
> > fallback to
> > + ;; reading the "requires.txt" from the egg-info directory from
> > the source
> > + ;; tarball.
> >
> > The wheel (.whl) binary format is well specified as PEP 427 [0] and
> > is
> > what pip primarily uses for installing Python packages, making it a
> > very
> > reliable source of metadata. The Python egg is the predecessor of
> > the
> > wheel, and can be considered obsolete, which explains why it's used
> > as a
> > fallback.
>
> Oh, I see.
>
> > Perhaps it'd be best to raise the issue to the package maintainers
> > and
> > have them specify their metadata correctly?
>
> Going back to the example at the beginning of this thread, what ‘guix
> import pypi tablib’ produces is missing ‘python-setuptools-scm’.
> Indeed, ‘METADATA’ doesn’t mention it.
>
> Is it really a bug on their side, or is it something peculiar about
> Guix
> packaging? Perhaps ‘python-setuptools-scm’ should be provided more
> or
> less by default?
Perhaps both? It probably depends on how setuptools are used, but this
use appears internal to the mechanisms of setuptools itself and not
something, that requires.txt is concerned about. In particular, I
think it is only relevant to the setup.py script and can probably be
detected by the presence of a certain form within it.

I'm not sure on the number of packages, that use this package vs. those
that don't, but I personally don't think we should make it an input in
every package. It is probably already a native-input to those that
need it (or at least probably should be if it's propagated instead).

Of course we would also have to look at the METADATA of other packages
and check whether they actually mention python-setuptools-scm or are
also unaware of the dependency.

Regards,
Leo
M
M
Maxim Cournoyer wrote on 25 Jun 2021 18:39
Re: bug#49168: ‘guix import pypi ’ misses package dependencies
(name . Ludovic Courtès)(address . ludo@gnu.org)
87pmw9yk3h.fsf@gmail.com
Hello,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

Toggle quote (11 lines)
>> Perhaps it'd be best to raise the issue to the package maintainers and
>> have them specify their metadata correctly?
>
> Going back to the example at the beginning of this thread, what ‘guix
> import pypi tablib’ produces is missing ‘python-setuptools-scm’.
> Indeed, ‘METADATA’ doesn’t mention it.
>
> Is it really a bug on their side, or is it something peculiar about Guix
> packaging? Perhaps ‘python-setuptools-scm’ should be provided more or
> less by default?

Hmm, sorry, I had lost track of what the original missing dependencies
were. In this case (setuptools-scm), it is special in that it extends
the setuptools build system; it must be loaded early by setuptools, so
perhaps that explains why the dependency doesn't show as required in
METADATA. I think going forward with the revised PEP 517 build system [0]
will alleviate such issues, by allowing to declare the build system and
its dependencies in a TOML file.

In the mean time, the statu quo seems the best option to me (manually
specify the setuptools extensions as native-inputs for the python
packages that need it). We should look forward adjusting our importer
to the latest PEP 517 trends (along the PEP 517 build system patches
awaiting review for core-updates!), which are quickly becoming
ubiquitous in the Python world.


HTH,

Maxim
L
L
Ludovic Courtès wrote on 28 Jun 2021 11:39
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87v95yxr93.fsf@gnu.org
Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (17 lines)
> Hmm, sorry, I had lost track of what the original missing dependencies
> were. In this case (setuptools-scm), it is special in that it extends
> the setuptools build system; it must be loaded early by setuptools, so
> perhaps that explains why the dependency doesn't show as required in
> METADATA. I think going forward with the revised PEP 517 build system [0]
> will alleviate such issues, by allowing to declare the build system and
> its dependencies in a TOML file.
>
> In the mean time, the statu quo seems the best option to me (manually
> specify the setuptools extensions as native-inputs for the python
> packages that need it). We should look forward adjusting our importer
> to the latest PEP 517 trends (along the PEP 517 build system patches
> awaiting review for core-updates!), which are quickly becoming
> ubiquitous in the Python world.
>
> [0] https://www.python.org/dev/peps/pep-0517/

Alright, makes sense to me!

I’m not sure I have much to contribute to the review of the PEP 517
patches, being ignorant about these things, but I fully support the
effort. :-)

In the meantime I guess we can close this bug?

Thanks,
Ludo’.
Z
Z
zimoun wrote on 8 Mar 2022 10:28
(name . Ludovic Courtès)(address . ludo@gnu.org)
87mti04wuw.fsf@gmail.com
Hi,

On lun., 28 juin 2021 at 11:39, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (11 lines)
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>> In the mean time, the statu quo seems the best option to me (manually
>> specify the setuptools extensions as native-inputs for the python
>> packages that need it). We should look forward adjusting our importer
>> to the latest PEP 517 trends (along the PEP 517 build system patches
>> awaiting review for core-updates!), which are quickly becoming
>> ubiquitous in the Python world.

> In the meantime I guess we can close this bug?

Do we wait the merge of the branch 'wip-python-pep517' to close this
report? Or can we close it now?


Cheers,
simon
?