Request for merging "python-team" branch

I haven't been involved in this branch, but I'm opening the request for

merge issue just to keep track of things.

Hello,

(Cc’ing the Python team.)

Christopher Baines <mail@cbaines.net> skribis:

I haven’t been involved either. My understanding is that the main

blocker before merging is assessing the amount of breakage introduced in

this branch¹.

Right now it’s built at 89% on x86_64-linux only, which is a bit less

than on ‘master’ but there are still pending builds. Interested readers

can browse failed builds at:

https://ci.guix.gnu.org/eval/latest/dashboard?spec=python-team

(If you find builds that failed due to transient errors like the

infamous “missing derivation”, you can click on “Action -> Restart”—this

is now available in most cases without privileges.)

An important issue seems to be test failures of ‘python-responses’:

https://ci.guix.gnu.org/build/4738398/details

Excerpt:

Ideas, Pythonistas? :-)

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2024-05/msg00227.html

Hi,

All failed tests have the same reason:

It looks like the bump to the latest version would resolve it 0.22.0 -> 0.25.2

There are some mentioned in GitHub issues with the same simptoms:

On Sat, 8 Jun 2024 at 15:27, Ludovic Courtès <ludo@gnu.org> wrote:

--

VCS: https://github.incerto.xyz/;https://git.sr.ht/~hellseher/

GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… ??? ????? - ???????????? ?????????????? ?????? ??????? ????????

????? ????? ????? ? ??? ??????, ??????????? ????? ???????, ??

?????????? ?? ? ????????? ??????? ????? ? ?????????????????.

Hi Python Team, Guix Team at large, and Parenphilic Pythonistas,

This long lived python-team branch as of today has a lot of git conflicts if you try to rebase and/or merge it on to master.

What do you think if we discuss an alternative team branch policy for the future for feature branches that target master?

Here's a tentative proposal, with an example:

Instead of having a single python-team branch, with a wide variety of new Python features, what if we had, a python-team feature branch that we work on relatively quickly?

In other words, we avoid long lived branches but try to merge for example, a new python-team-sphinx branch as soon as the "Sphinx feature" is ready. This python-team-sphinx branch will only contain the work required to bump sphinx to the latest version that we'd like to support. The reason I use python-sphinx as an example is because the python-sphinx package requires a lot of rebuilds across many language ecosystems that use Sphinx for documentation purposes.

I think that keeping the team branches focused on a particular team sub-feature within that team's scope and not using long-lived and largely scoped branches will avoid a ton of frustration trying to fix merge conflicts when/before we announce a request to merge.

We can then focus our focused efforts on iterating over preparing major Python features and packages that require large amounts rebuilds.

WDYT?

jgart

Hi jgart,

Quoting jgart (2024-06-18 22:46:47)

IM(V)HO… the smaller and the more-frequently-rebased the branches, the better!

I still haven’t wrapped my head around the consequences for the CI of rebasing

often —can a simple rebase trigger a world rebuild?—, but this is definitively

something that I do —quite successfully!— on a daily basis in my professional life.

I guess it’s just a matter of giving it a try and see? It’s always possible to

go back to the one-branch-to-rule-them-all `python-team` branch if managing

multiple branches ends up being a pain, right?

I also have no clue how the other teams are managing their branch and their

rebase/merge policy?

--

Tanguy

"jgart" <jgart@dismail.de> writes:

This aligns with the current (and previous) guidance on managing

branches [1]. Providing there's a consistent topic for the branch, any

name is fine (e.g. python-team-sphinx is fine).

1: https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html

I'm also hoping the requirement to create a guix-patches issue

requesting to merge the branch when it's created will help avoid the

long lived branches that we've had over the last year.

Hi Christopher,

That's great that my proposal already aligns with the current guidance.

I am going to try to start cherry picking uncontroversial and non failing patches on the python-team branch on to master and removing them from the python-team branch in order to reduce the size of the branch. I can then remove those commits from the python-team branch and force push to update it. What do you think of this approach before I attempt it? I'll wait on your guidance before doing this.

Another approach I can take is to break the branch apart into 2 or 3 smaller topic branches and make requests for those branches to merge one at a time.

Note that I have an underpowered old Thinkpad X230 laptop and will only be able to test branches that are not too large. I tried using QA/CI but it seemed to be in an unknown state the last time that I tried. Any suggestions here for how to more effectively use that QA/CI infrastructure would be much appreciated.

I tried spending a bit of time on trying to resolve the merge conflicts but it quickly lead down a long rabbit hole and I only have a limited time to fix so many merge conflicts. The process is also currently very tedious to do with high amounts of accuracy due to their being

1. Lots of commits that I am not familiar

2. Lots of commits

3. The branch has diverged alot since latest master so things don't cleanly apply anymore

What do you think of this current approach of cherry picking good commits off of the python-team branch in order to reduce the size?

Another issue that I ran into on the branch is that there are commits that are failing. For example, at commit fdef5662cce6a57aa6bfa682c7260b65ce3669b8 (the borgmatic changes), borgmatic fails. Should I remove this commit since it is broken?

all the best,

jgart

Hi,

@jgart Please let me know if you need a quick test of changes, I've got some

computer recourses at home.

Thanks

Oleg

On Tue, 25 Jun 2024 at 18:04, jgart <jgart@dismail.de> wrote:

--

VCS: https://github.incerto.xyz/;https://git.sr.ht/~hellseher/

GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… ??? ????? - ???????????? ?????????????? ?????? ??????? ????????

????? ????? ????? ? ??? ??????, ??????????? ????? ???????, ??

?????????? ?? ? ????????? ??????? ????? ? ?????????????????.

Hi,

please go ahead. Divide and conquer is imo a good approach to get this branch merged.

I’m splitting of the build-system-related changes into the

pyproject-build-system branch (again), because these are relatively

self-contained and thus easy to merge into master.

Lars

"jgart" <jgart@dismail.de> writes:

If there are commits on the branch that have been submitted as patches,

you might just be able to look at pushing them to master. If there are

commits that haven't been submitted as patches yet, but could be pushed

to master, it would be good to send them as patches so that QA can take

a look.

And splitting off some commits in to a more specific branch is fine as

well.

QA has been performing recently as it's taking too long to process

revisions. Let me know if things aren't working or if you have questions

though and I can try and help.

I think making sure commits that can go to master, do go to master, is

really helpful to avoid merge conflicts. It's so much more likely to

have conflicts if there are changes that can be made on master, but are

made on a branch, or changes that should be made on a branch but are

pushed to master.

While ideally we'd care about the state at each commit, this doesn't

really happen so it's up to you what you do there.

This branch is first now on the waiting list of QA:

https://qa.guix.gnu.org/

but it has not been touched since May, and in particular not been

rebased on master after the core-updates merge. This is reflected

in the QA message:

https://qa.guix.gnu.org/branch/python-team

"Unable to check changes between branch and master.

Merge base is not a commit known to the data service."

Should we close this issue now and let the branch take the back of the

queue? I see build farm activity around commit e4ad2f178af65be6c8ca6e98a667787674cd8c82,

the latest one in this branch, which looks like a waste of scarce build

power (especially on aarch64) to me. And it holds up the r-team branch

described as ready to merge.

Andreas

On 25 Jun, Sharlatan Hellseher wrote:

(...)

Hi,

I started breaking the branch down into small parts. Oleg (or someone else) can you review and push them for me please?

They all build against current master (b7c94d528875415ea7ec6225d88a6b3d55fa2e14)

I pushed it to a branch (python-team-71408-pt1) here:

git clone https://git.sdf.org/futurile/guix-play/--branch python-team-71408-pt1

Stack is:

python-pygments@2.15.1: 1ff7dfeab65c27bff1ddd0694b5c16c648a0cd0b

python-imagesizei@1.4.1: 93d627adccf07e57a36a8b0a5ad6ba3878814820

python-sphinx-alabaster-theme@0.7.13 126d3223f4687c8319fc386481714c5c17227c42

python-typing-extensions@4.10.0 175eea0d091d80f36a468c2154196a79c7d83148

Add setuptools/wheels to packages 549efcb63714a8715a81e3944de94e7a7d337d7d

python-sphinx@6.2.1 fd577e065c48c06083b8e1eeebc6cc86a7238ffb

- please look at this one as I had to adjust the tests a lot

- might be a mistake

python-markdown-it-py@3.0.0 160f89e8a18c0bf21b232df2d10ff725068a54b2

python-mdit-py-plugins@0.4.0 717ba970f2924c41a315c8bdc7994fbbba98475a

python-argcomplete@3.2.2 f5fa5f549cd88c5b31262e30bf90f426b89793f4

python-traitlets@5.14.1 839057dcfed27a109d3c9daadf320d3a7bff2ae6

python-ipython@8.22.1 406e3b159cccb7a79370205767e55e5c27a395df

- requires python-trailets to work

python-nbconvert 569256f3e6b7e5bcca891123039db3a26c23b0ce

python-jupytext@1.15.0 59ee791c6223761dd467642a0236c6a99200e32d

python-sphinx-pytest@0.2.0 99e8317bce68fd001f1a20980d8489d7b027be60

python-eventlet@0.35.2 a7ee6d91c81a1b9144811b175aad0e758a9fb03c

python-argcomplete 1b588a1baf5d78713d35fd074c4792c972164f95

- fix-up of earlier commit

python-pynacl 7d11cf28bbc6158b971bedb03fa77e3d7cf41940

python-pytest-arraydiff 86144159f36602f17b5808a2276f6fd23fcd7db9

python-itemloaders b548cfd67124fcc75d2f0f4f19d1ebbc3e1ad884

python-misteltoe 25d27be8a937d4fef31af8c5ca87286fa8599485

python-vine 484ad0ee5f3e3a77454aa816585e527b70b41a09

python-joblib 2499d3143cae865839231113012642a60272c2a8

Not working for me - would love help with:

- meson-python 86631897caa19606c407729818c21229f44ed947

- failing for me on master and this fails

- python-rich@13.7.0 b8e3e0e4d42cdaf9b48aecb0e0660493fa4f8b1b

- it needed python-nbconvert.

- still fails but now on a bunch of API tests

- hmm python-ipython master is failing to build for me

- python-textual@0.50.1 d3604f1c9898e79d9d6c5cad06bb06358f78a25f

- needs python-rich

- python-myst-parser ab9c7c535403b7e813a44e56d8cce63afd5b8588

- it fails test_basic.py - needs a python build system change from earlier?

- python-lightning-cloud 2ef5b0b85bba0cda2be8f0b6ab6146e9bc194922

- fails due to python-rich failing

- python-rapidfizz@3.6.1 fa6755520c8e09969aef693b0ab884deac6a0343

- fails - unclear why

- python-cleo@2.1.0 082a2d43a8b850e73e07d07a61f70cae3a3a0428

- requires python-rapid-fuzz

- python-packaging-bootstrap 39fc22f3dccc1f34c83c7469ded91bcd5d32d35d

- fails - complains about a dependency cycle

- python-pycurl 94f54d29eb104d750d6d77b93d9e00e8f0937350

- fails a test

Python build system changes - not cherry-picked:

- 2a06463c6ad41b2d37285988f05e9d7b5fad2d32

- 4ce43dd688dc0d6471ce9b2ed10732e235a5f12c

- fb70a545ad2b494f39ae7d22e205759702006e10

- d383150bd9ded7cc2f1b976ae4a833d6b05924f5

- 0a20b52b44600fed9948865ae45a06a52b8386b8

- cb5f963e7a270a973bcb1850136df2e358a0e75a

- a688230bc25dea5b3c0ff1f1de80dc02e6afd80b

- ee1cacdda4627798930a1ffe8eb4030b940b4030

- c955dd1e732c7abee0a8a598e57f11e564735383

- 6c2090169ba9495deb15734227925b8faf9bbca9

- a436de1fb9dd063f3c61aa5b36fe4e31cbef9048

- fd577e065c48c06083b8e1eeebc6cc86a7238ffb

- df580a54f49ddc5d7aa99992916f4537f48df48d

Thanks,

Futurile / Steve

Hello!

I am interested in merging this branch. Unfortunately, I've got near zero

experience contributing to guix.

Is there any task not on the critical path that I could help with?

--

Respectfully,

Arseniy (a.k.a necto)

Hi,

I've pushed 31 commits to:

git clone https://git.sdf.org/futurile/guix-play/--branch python-team-71408-pt1

As I don't think anyone has pulled from it I force pushed it an hour ago.

The cherry-picked commits are:

Python-pygments@2.15.1: 1ff7dfeab65c27bff1ddd0694b5c16c648a0cd0b

python-imagesizei@1.4.1: 93d627adccf07e57a36a8b0a5ad6ba3878814820

python-sphinx-alabaster-theme@0.7.13 126d3223f4687c8319fc386481714c5c17227c42

python-typing-extensions@4.10.0 175eea0d091d80f36a468c2154196a79c7d83148

Add setuptools/wheels to packages 549efcb63714a8715a81e3944de94e7a7d337d7d

python-sphinx@6.2.1 fd577e065c48c06083b8e1eeebc6cc86a7238ffb

python-markdown-it-py@3.0.0 160f89e8a18c0bf21b232df2d10ff725068a54b2

python-mdit-py-plugins@0.4.0 717ba970f2924c41a315c8bdc7994fbbba98475a

python-pytest-mypy-testing@0.1.2 e2745702b79702a928241545b83169bb748aa9a9

python-argcomplete@3.2.2 f5fa5f549cd88c5b31262e30bf90f426b89793f4

python-traitlets@5.14.1 839057dcfed27a109d3c9daadf320d3a7bff2ae6

python-prompt-toolkit@3.0.43 557aa4792136e860e3f8e28dc25527591d28b3ab

python-ipython@8.22.1 406e3b159cccb7a79370205767e55e5c27a395df

- requires traitlets

python-nbconvert 569256f3e6b7e5bcca891123039db3a26c23b0ce

- was higher up; needed by python-rich

python-jupytext@1.15.0 59ee791c6223761dd467642a0236c6a99200e32d

python-sphinx-pytest@0.2.0 99e8317bce68fd001f1a20980d8489d7b027be60

- was higher up;

python-readme-render@37.2 b1fbfe5b55b85e44d4f3e60c3ae6ec9bff71bf11

python-eventlet@0.35.2 a7ee6d91c81a1b9144811b175aad0e758a9fb03c

python-argcomplete 1b588a1baf5d78713d35fd074c4792c972164f95

python-pynacl 7d11cf28bbc6158b971bedb03fa77e3d7cf41940

python-pytest-arraydiff 86144159f36602f17b5808a2276f6fd23fcd7db9

python-itemloaders b548cfd67124fcc75d2f0f4f19d1ebbc3e1ad884

python-misteltoe 25d27be8a937d4fef31af8c5ca87286fa8599485

python-vine 484ad0ee5f3e3a77454aa816585e527b70b41a09

python-joblib 2499d3143cae865839231113012642a60272c2a8

python-box dd8f10fd79946e8f6f2f1613317a5a8d9e527d32

futurile-python-pytdata-sphinx-theme-fix 071e5e1ca795b2a5dacd46c0da05f57385220046

futurile-python-certauth-fix c390ac8aeb13e4ed565d52d521d09752ce318e07

- need help with this as I can't figure out why it's failing

gnu-python-attrs-Update-to fabbb32638813b944b9da3749599fac6eed4221c

- 2130 package rebuilds

- have built about 100 - better to rebuild after python-rich

python-rich@13.8.1 611b7257463e9a0b757aee492ab54dda33a5ebe8

- NEW - combines two patches

- b8e3e0e4d42cdaf9b48aecb0e0660493fa4f8b1b

- bf18849eda26f83fec42124c95b652c1aaa77f15

python-textual@0.50.1 d3604f1c9898e79d9d6c5cad06bb06358f78a25f

- python-rich wants 350 packages rebuild - done about 30

Not working (for me):

---------------------

- meson-python 86631897caa19606c407729818c21229f44ed947

- python-pycurl 94f54d29eb104d750d6d77b93d9e00e8f0937350

- another ceert fail, as is certauth

- python-packaging-bootstrap 39fc22f3dccc1f34c83c7469ded91bcd5d32d35d

- complains about a dependency cycle

- https://ci.guix.gnu.org/build/4322844/details- SUCCESS on python-team!

- fails for me

- python-cleo@2.1.0 082a2d43a8b850e73e07d07a61f70cae3a3a0428

- requires rapidfuzz

- python-rapidfuzz@3.6.1 fa6755520c8e09969aef693b0ab884deac6a0343

- https://ci.guix.gnu.org/build/4738134/details- python-scikit-build

- python-myst-parser ab9c7c535403b7e813a44e56d8cce63afd5b8588

- https://ci.guix.gnu.org/build/4737838/details

- fails

Pyproject build-system (not cherrypicked)

------------------------------------------

- 2a06463c6ad41b2d37285988f05e9d7b5fad2d32

- 4ce43dd688dc0d6471ce9b2ed10732e235a5f12c

- fb70a545ad2b494f39ae7d22e205759702006e10

- d383150bd9ded7cc2f1b976ae4a833d6b05924f5

- 0a20b52b44600fed9948865ae45a06a52b8386b8

- cb5f963e7a270a973bcb1850136df2e358a0e75a

- a688230bc25dea5b3c0ff1f1de80dc02e6afd80b

- ee1cacdda4627798930a1ffe8eb4030b940b4030

- c955dd1e732c7abee0a8a598e57f11e564735383

- 6c2090169ba9495deb15734227925b8faf9bbca9

- a436de1fb9dd063f3c61aa5b36fe4e31cbef9048

- fd577e065c48c06083b8e1eeebc6cc86a7238ffb

- df580a54f49ddc5d7aa99992916f4537f48df48d

Next steps

-----------

The ones I've cherry-picked are all from the 'start' of the branch.

I'm still rebuilding the python-attrs & python-rich dependents.

For now, it doesn't look like these are making things worse.

I will create another small branch 'stacked' on top of this one, to continue testing.

Futurile / Steve

Hello,

Am Wed, Sep 18, 2024 at 11:09:23AM +0200 schrieb Andreas Enge:

the Bordeaux build farm is mostly idle now, so I will go ahead and close

this bug, in the hope that the next waiting branch will be started.

If this does not work, we can still reopen it.

And I think you can continue discussion in this bug even if it is closed.

Andreas