Should ‘guix import pypi’ get dependencies from pyproject files?

  • Open
  • quality assurance status badge
Details
4 participants
  • Lars-Dominik Braun
  • Ludovic Courtès
  • Sharlatan Hellseher
  • Tanguy LE CARROUR
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 25 Mar 12:06 +0100
(address . bug-guix@gnu.org)
877chqtvsk.fsf@inria.fr
Hello Python team!

Should ‘guix import pypi’ attempt to get dependency information from
‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
as it already does?

It might be more complicated than we’d like: in some cases, that file
seems to be used as a “trampoline” to Poetry. For instance, in
python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

Toggle snippet (26 lines)
[tool.poetry]
name = "pypugjs"
version = "5.9.4"
description = ""
authors = ["Andy Grabow <andy@freilandkiwis.de>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.8"
Jinja2 = "^3.1.1"
Mako = "^1.1.3"
tornado = "^6.0.4"
six = "^1.15.0"
coverage = "^6.3.2"
nose = "^1.3.7"
Flask = "^2.1.1"
charset-normalizer = "^2.1.0"
flake8 = "^4.0.1"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Thoughts?

Ludo’.
S
S
Sharlatan Hellseher wrote on 25 Mar 20:28 +0100
Re: bug#69997: Should ‘guix import pypi’ get dep endencies from pyproject files?
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 69997@debbugs.gnu.org)
CAO+9K5o4xpKnhXcBX81LQ3ct6oS+1y-7mr5JVT=4fVwLE9QkMA@mail.gmail.com
Hi Lido!

Toggle quote (4 lines)
> Should ‘guix import pypi’ attempt to get dependency
> information fromn‘pyproject.toml’, in addition to
> ‘requirements.txt’ and wheel ‘METADATA’ as it already does?

It's quite a common practice in modern Python just to include
pyproject.toml, that fact makes importing long chains problematic.

It would be nice to have common yaml/toml parser for that task.

Oleg
Attachment: file
T
T
Tanguy LE CARROUR wrote on 26 Mar 08:54 +0100
Re: bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?
171143965734.4362.8405481724513962855@bioneland.org
Hi Ludo’,


Quoting Ludovic Courtès (2024-03-25 12:06:51)
Toggle quote (8 lines)
> Should ‘guix import pypi’ attempt to get dependency information from
> ‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
> as it already does?
>
> It might be more complicated than we’d like: in some cases, that file
> seems to be used as a “trampoline” to Poetry. For instance, in
> python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

Short answer: no! ?

I’m pretty sure you know everything that I’m about to write, but better
say it out loud…

For a "standard modern" project managed with Poetry, the Python source
package contains `PKG-INFO` and `pyproject.toml ` that both contain
the run time dependencies. The wheel package only contains `METADATA` that
lists the dependencies. The source only contains a `pyproject.toml`.
To make the installed package as small as possible, tests files and
uncompiled assets are not (should not be) included.
From a Guix stand point, it’s better to build from source to be able to
run the test suite.

For the `python-pypugjs` you used as an example, we build from source,
so I guess the question does not arise. If we were to use the packages
available on PyPI, what I said above is *NOT* confirmed ?:
- wheel (`.whl`) only contains `METADATA` with the dependencies; **BUT**
- source (`.tar.gz`) contains `PKG-INFO` (without dependency information),
`pyproject.toml` (with dep’) and `setup.py` (also with dep’).

… "fun" fact, the information in `pyproject.toml` are **NOT** the same as
the one in `setup.py`!? ? `pyproject.toml` says that `nose` is a run time
dependency (which it is not), but `setup.py` properly lists it in `tests_require`.

So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! ?
But if you have to, rely on the wheel’s `METADATA` file.

I hope this make sense. … I’m not really sure any more! ?

Regards,

--
Tanguy
L
L
Ludovic Courtès wrote on 26 Mar 17:04 +0100
(name . Tanguy LE CARROUR)(address . tanguy@bioneland.org)
87o7b1ou6z.fsf@inria.fr
Hello!

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

Toggle quote (3 lines)
> I’m pretty sure you know everything that I’m about to write, but better
> say it out loud…

Nope, I know nothing (I’ve just been told about using ‘pyproject.toml’
and it seemed to kinda make sense. :-))

Toggle quote (20 lines)
> For a "standard modern" project managed with Poetry, the Python source
> package contains `PKG-INFO` and `pyproject.toml ` that both contain
> the run time dependencies. The wheel package only contains `METADATA` that
> lists the dependencies. The source only contains a `pyproject.toml`.
> To make the installed package as small as possible, tests files and
> uncompiled assets are not (should not be) included.
> From a Guix stand point, it’s better to build from source to be able to
> run the test suite.
>
> For the `python-pypugjs` you used as an example, we build from source,
> so I guess the question does not arise. If we were to use the packages
> available on PyPI, what I said above is *NOT* confirmed ?:
> - wheel (`.whl`) only contains `METADATA` with the dependencies; **BUT**
> - source (`.tar.gz`) contains `PKG-INFO` (without dependency information),
> `pyproject.toml` (with dep’) and `setup.py` (also with dep’).
>
> … "fun" fact, the information in `pyproject.toml` are **NOT** the same as
> the one in `setup.py`!? ? `pyproject.toml` says that `nose` is a run time
> dependency (which it is not), but `setup.py` properly lists it in `tests_require`.

Oh my, such a mess.

Toggle quote (5 lines)
> So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! ?
> But if you have to, rely on the wheel’s `METADATA` file.
>
> I hope this make sense. … I’m not really sure any more! ?

It does!

But then I mean, we could offer, say, ‘guix import upstream https://…’,
and that thing could parse ‘setup.py’ or similar to produce a package
definition from that. Maybe that’s what you had in mind: import
straight from upstream rather than via PyPI?

Thanks,
Ludo’.
T
T
Tanguy LE CARROUR wrote on 26 Mar 17:55 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
171147212395.13452.3553266600176809571@bioneland.org
Hi,


Quoting Ludovic Courtès (2024-03-26 17:04:52)
Toggle quote (12 lines)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> > So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! ?
> > But if you have to, rely on the wheel’s `METADATA` file.
> >
> > I hope this make sense. … I’m not really sure any more! ?
>
> It does!
>
> But then I mean, we could offer, say, ‘guix import upstream https://…’,
> and that thing could parse ‘setup.py’ or similar to produce a package
> definition from that.

I’m not against a good-old-generic-solution®, but this one might be
a bit… too generic. It contains no mention to Python, so the next logical step
would be `guix import URL`. Not that I would not like it, though! ?

So I would say… let’s wait and see what the others think. In the
meantime, I’ll have to dive deeper in the PEP and the actual importer
code.


Toggle quote (2 lines)
> Maybe that’s what you had in mind: import straight from upstream rather than via PyPI?

I actually had nothing in mind! I blacked out after sending this email.
Too many emotions at once, I guess! I had no clue it was such a mess. ?

--
Tanguy
T
T
Tanguy LE CARROUR wrote on 26 Mar 18:14 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
171147325018.15415.4526491903307851998@bioneland.org
Hi,

Quoting Tanguy LE CARROUR (2024-03-26 17:55:23)
Toggle quote (17 lines)
> Quoting Ludovic Courtès (2024-03-26 17:04:52)
> > Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
> > > So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! ?
> > > But if you have to, rely on the wheel’s `METADATA` file.
> > >
> > > I hope this make sense. … I’m not really sure any more! ?
> >
> > It does!
> >
> > But then I mean, we could offer, say, ‘guix import upstream https://…’,
> > and that thing could parse ‘setup.py’ or similar to produce a package
> > definition from that.
> […]
> So I would say… let’s wait and see what the others think. In the
> meantime, I’ll have to dive deeper in the PEP and the actual importer
> code.

According to PEP 427 [1] a.k.a. Binary distribution format [2], if you
go for packaged/PyPI then we should go for `METADATA`.


But, as stated earlier, we should build from source, to make sure we can
run the test suite. Active projects should slowly migrate to PEP 517 [3]
`pyproject.toml`. But, this is not a solution! ? This is actually yet
another problem! ?


Each build system relies on it’s own file organization. For instance, Poetry
looks for a `[tool.poetry.dependencies]` section in the file. So the
importer should be "build system aware", which leads us to… `guix import poetry URL`!?
Not really generic any more! ?

I guess we should sleep on it…

--
Tanguy
L
L
Lars-Dominik Braun wrote on 27 Mar 07:49 +0100
Re: bug#69997: Should ‘gui x import pypi’ get dependencies from pyproject files?
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
ZgPBd4TjMRz8q6lq@noor.fritz.box
Hey Ludo,

Toggle quote (4 lines)
> Should ‘guix import pypi’ attempt to get dependency information from
> ‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’
> as it already does?

yes it should. It’s the next logical step after having a
pyproject-build-system. The python-team branch (not sure whether Ricardo
had time to merge it yet) adds a TOML parser to Guix as a first step.

Toggle quote (4 lines)
> It might be more complicated than we’d like: in some cases, that file
> seems to be used as a “trampoline” to Poetry. For instance, in
> python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

The file is always a trampoline to other build systems, like setuptools
or poetry. That’s (unfortunately) by design, see the [build-system]
section.

Toggle quote (12 lines)
> [tool.poetry.dependencies]
> python = "^3.8"
> Jinja2 = "^3.1.1"
> Mako = "^1.1.3"
> tornado = "^6.0.4"
> six = "^1.15.0"
> coverage = "^6.3.2"
> nose = "^1.3.7"
> Flask = "^2.1.1"
> charset-normalizer = "^2.1.0"
> flake8 = "^4.0.1"

That’s unfortunate, because the
specification includes a common field for dependencies:
I guess we’ll have to deal with these idiosyncracies as well :(

Lars
L
L
Ludovic Courtès wrote on 28 Mar 19:09 +0100
Re: bug#69997: Should ‘guix import pypi’ get dependencies from pyproject files?
(name . Tanguy LE CARROUR)(address . tanguy@bioneland.org)
87msqikz2q.fsf@inria.fr
Hi,

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

Toggle quote (2 lines)
> Quoting Ludovic Courtès (2024-03-26 17:04:52)

[...]

Toggle quote (8 lines)
>> But then I mean, we could offer, say, ‘guix import upstream https://…’,
>> and that thing could parse ‘setup.py’ or similar to produce a package
>> definition from that.
>
> I’m not against a good-old-generic-solution®, but this one might be
> a bit… too generic. It contains no mention to Python, so the next logical step
> would be `guix import URL`. Not that I would not like it, though! ?

Well, this has been on my mind for a long time. Such a tool could
provide at least a useful package skeleton even for software using CMake
or Autotools.

Toggle quote (4 lines)
> So I would say… let’s wait and see what the others think. In the
> meantime, I’ll have to dive deeper in the PEP and the actual importer
> code.

Looks like consensus among you Pythonistas has yet to be reached
regarding whether ‘pyproject.toml’ data would be a useful addition. :-)

We’ll see!

Ludo’.

PS: I hear more and more long-time Python developers dismayed by the sad
state of packaging and code evolution in Python. In Guile land, we
say: refugees welcome! Come discover a great language and a great
community (together with their own set of problems).
T
T
Tanguy LE CARROUR wrote on 29 Mar 08:46 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
171169836168.1828.14021738276113751443@bioneland.org
Hi Ludo’,


Quoting Ludovic Courtès (2024-03-28 19:09:49)
Toggle quote (25 lines)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
>
> > Quoting Ludovic Courtès (2024-03-26 17:04:52)
>
> [...]
>
> >> But then I mean, we could offer, say, ‘guix import upstream https://…’,
> >> and that thing could parse ‘setup.py’ or similar to produce a package
> >> definition from that.
> >
> > I’m not against a good-old-generic-solution®, but this one might be
> > a bit… too generic. It contains no mention to Python, so the next logical step
> > would be `guix import URL`. Not that I would not like it, though! ?
>
> Well, this has been on my mind for a long time. Such a tool could
> provide at least a useful package skeleton even for software using CMake
> or Autotools.
>
> > So I would say… let’s wait and see what the others think. In the
> > meantime, I’ll have to dive deeper in the PEP and the actual importer
> > code.
>
> Looks like consensus among you Pythonistas has yet to be reached
> regarding whether ‘pyproject.toml’ data would be a useful addition. :-)

I have to admit that the Zen of Python [1] "There should be one-- and preferably only one --
obvious way to do it." has been recently difficult to follow packaging-wise!


Even Poetry, the one I’ve been using for years, as made questionable
decisions [2] and I have to admit that I had a look at the competitors.



Toggle quote (5 lines)
> PS: I hear more and more long-time Python developers dismayed by the sad
> state of packaging and code evolution in Python. In Guile land, we
> say: refugees welcome! Come discover a great language and a great
> community (together with their own set of problems).

Thanks for the kind invitation, but… I have an acute form of parens-itis.
Seeing more that a pair of parenthesis on a single line make my eyes bleed! ?
And… `#` is for comments, every other use is complete heresy and those
who go against the creed should suffer! … isn’t that what the parentheses
are for?! ?

Any way, thanks again for caring about Python!

Regards,

--
Tanguy
L
L
Ludovic Courtès wrote on 29 Mar 10:06 +0100
(name . Tanguy LE CARROUR)(address . tanguy@bioneland.org)
87wmpliezs.fsf@inria.fr
Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

Toggle quote (11 lines)
>> PS: I hear more and more long-time Python developers dismayed by the sad
>> state of packaging and code evolution in Python. In Guile land, we
>> say: refugees welcome! Come discover a great language and a great
>> community (together with their own set of problems).
>
> Thanks for the kind invitation, but… I have an acute form of parens-itis.
> Seeing more that a pair of parenthesis on a single line make my eyes bleed! ?
> And… `#` is for comments, every other use is complete heresy and those
> who go against the creed should suffer! … isn’t that what the parentheses
> are for?! ?

Worry not! As part of our refugees-welcome effort, “we” have put
together tools and guides to help you feel at home:


:-)

Ludo’.
T
T
Tanguy LE CARROUR wrote on 29 Mar 11:11 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
171170706932.12538.2476080876569348495@bioneland.org
Quoting Ludovic Courtès (2024-03-29 10:06:31)
Toggle quote (19 lines)
> Tanguy LE CARROUR <tanguy@bioneland.org> skribis:
>
> >> PS: I hear more and more long-time Python developers dismayed by the sad
> >> state of packaging and code evolution in Python. In Guile land, we
> >> say: refugees welcome! Come discover a great language and a great
> >> community (together with their own set of problems).
> >
> > Thanks for the kind invitation, but… I have an acute form of parens-itis.
> > Seeing more that a pair of parenthesis on a single line make my eyes bleed! ?
> > And… `#` is for comments, every other use is complete heresy and those
> > who go against the creed should suffer! … isn’t that what the parentheses
> > are for?! ?
>
> Worry not! As part of our refugees-welcome effort, “we” have put
> together tools and guides to help you feel at home:
>
> https://hg.sr.ht/~arnebab/wisp
> https://www.draketo.de/py2guile

?… ?… ?!

--
Tanguy
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 69997@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 69997
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch