Should ‘guix import pypi’ get dependencies from pyproject files?

Hello Python team!

Should ‘guix import pypi’ attempt to get dependency information from

‘pyproject.toml’, in addition to ‘requirements.txt’ and wheel ‘METADATA’

as it already does?

It might be more complicated than we’d like: in some cases, that file

seems to be used as a “trampoline” to Poetry. For instance, in

python-pypugjs, the ‘requires’ bit delegates everything to Poetry:

Thoughts?

Ludo’.

Hi Lido!

It's quite a common practice in modern Python just to include

pyproject.toml, that fact makes importing long chains problematic.

It would be nice to have common yaml/toml parser for that task.

Oleg

Hi Ludo’,

Quoting Ludovic Courtès (2024-03-25 12:06:51)

Short answer: no! ?

I’m pretty sure you know everything that I’m about to write, but better

say it out loud…

For a "standard modern" project managed with Poetry, the Python source

package contains `PKG-INFO` and `pyproject.toml ` that both contain

the run time dependencies. The wheel package only contains `METADATA` that

lists the dependencies. The source only contains a `pyproject.toml`.

To make the installed package as small as possible, tests files and

uncompiled assets are not (should not be) included.

From a Guix stand point, it’s better to build from source to be able to

run the test suite.

For the `python-pypugjs` you used as an example, we build from source,

so I guess the question does not arise. If we were to use the packages

available on PyPI, what I said above is *NOT* confirmed ?:

- wheel (`.whl`) only contains `METADATA` with the dependencies; **BUT**

- source (`.tar.gz`) contains `PKG-INFO` (without dependency information),

`pyproject.toml` (with dep’) and `setup.py` (also with dep’).

… "fun" fact, the information in `pyproject.toml` are **NOT** the same as

the one in `setup.py`!? ? `pyproject.toml` says that `nose` is a run time

dependency (which it is not), but `setup.py` properly lists it in `tests_require`.

So, my answer would be: do not import from PyPI! Yes, I know, it’s radical! ?

But if you have to, rely on the wheel’s `METADATA` file.

I hope this make sense. … I’m not really sure any more! ?

Regards,

--

Tanguy

Hello!

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

Nope, I know nothing (I’ve just been told about using ‘pyproject.toml’

and it seemed to kinda make sense. :-))

Oh my, such a mess.

It does!

But then I mean, we could offer, say, ‘guix import upstream https://…’,

and that thing could parse ‘setup.py’ or similar to produce a package

definition from that. Maybe that’s what you had in mind: import

straight from upstream rather than via PyPI?

Thanks,

Ludo’.

Hi,

Quoting Ludovic Courtès (2024-03-26 17:04:52)

I’m not against a good-old-generic-solution®, but this one might be

a bit… too generic. It contains no mention to Python, so the next logical step

would be `guix import URL`. Not that I would not like it, though! ?

So I would say… let’s wait and see what the others think. In the

meantime, I’ll have to dive deeper in the PEP and the actual importer

code.

I actually had nothing in mind! I blacked out after sending this email.

Too many emotions at once, I guess! I had no clue it was such a mess. ?

--

Tanguy

Hi,

Quoting Tanguy LE CARROUR (2024-03-26 17:55:23)

According to PEP 427 [1] a.k.a. Binary distribution format [2], if you

go for packaged/PyPI then we should go for `METADATA`.

[1]: https://peps.python.org/pep-0427/

[2]: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-dist-info-directory

But, as stated earlier, we should build from source, to make sure we can

run the test suite. Active projects should slowly migrate to PEP 517 [3]

`pyproject.toml`. But, this is not a solution! ? This is actually yet

another problem! ?

[3]: https://peps.python.org/pep-0517/

Each build system relies on it’s own file organization. For instance, Poetry

looks for a `[tool.poetry.dependencies]` section in the file. So the

importer should be "build system aware", which leads us to… `guix import poetry URL`!?

Not really generic any more! ?

I guess we should sleep on it…

--

Tanguy

Hey Ludo,

yes it should. It’s the next logical step after having a

pyproject-build-system. The python-team branch (not sure whether Ricardo

had time to merge it yet) adds a TOML parser to Guix as a first step.

The file is always a trampoline to other build systems, like setuptools

or poetry. That’s (unfortunately) by design, see the [build-system]

section.

That’s unfortunate, because the

specification includes a common field for dependencies:

I guess we’ll have to deal with these idiosyncracies as well :(

Lars

Hi,

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

[...]

Well, this has been on my mind for a long time. Such a tool could

provide at least a useful package skeleton even for software using CMake

or Autotools.

Looks like consensus among you Pythonistas has yet to be reached

regarding whether ‘pyproject.toml’ data would be a useful addition. :-)

We’ll see!

Ludo’.

PS: I hear more and more long-time Python developers dismayed by the sad

state of packaging and code evolution in Python. In Guile land, we

say: refugees welcome! Come discover a great language and a great

community (together with their own set of problems).

Hi Ludo’,

Quoting Ludovic Courtès (2024-03-28 19:09:49)

I have to admit that the Zen of Python [1] "There should be one-- and preferably only one --

obvious way to do it." has been recently difficult to follow packaging-wise!

[1]: https://peps.python.org/pep-0020

Even Poetry, the one I’ve been using for years, as made questionable

decisions [2] and I have to admit that I had a look at the competitors.

[2]: https://python-poetry.org/docs/faq/#why-does-poetry-not-adhere-to-semantic-versioning

Thanks for the kind invitation, but… I have an acute form of parens-itis.

Seeing more that a pair of parenthesis on a single line make my eyes bleed! ?

And… `#` is for comments, every other use is complete heresy and those

who go against the creed should suffer! … isn’t that what the parentheses

are for?! ?

Any way, thanks again for caring about Python!

Regards,

--

Tanguy

Tanguy LE CARROUR <tanguy@bioneland.org> skribis:

Worry not! As part of our refugees-welcome effort, “we” have put

together tools and guides to help you feel at home:

https://hg.sr.ht/~arnebab/wisp

https://www.draketo.de/py2guile

:-)

Ludo’.

Quoting Ludovic Courtès (2024-03-29 10:06:31)

?… ?… ?!

--

Tanguy