Request for merging "python-team" branch

Done

Details

7 participants

???
jgart
Lars-Dominik Braun
Leo Famulari
Christopher Baines
Ricardo Wurmus
Sharlatan Hellseher

Owner: unassigned

Submitted by: Ricardo Wurmus

Severity: normal

Ricardo Wurmus wrote on 1 Jan 18:58 +0100

Recipients:(address . guix-patches@gnu.org)

Message-ID:87bjwq76vx.fsf@elephly.net

The python-team branch is currently tracking the rust-team branch,
because it depends on changes there.  It is shaping up nicely and we
hope to be able to merge it some time soon.

-- 
Ricardo

Lars-Dominik Braun wrote on 5 Jan 11:34 +0100

Recipients:(name . Ricardo Wurmus)(address . rekado@elephly.net)

Message-ID:Z3pgSd60hE8LN1uE@noor.fritz.box

Hi Ricardo,

Toggle quote (4 lines)

> The python-team branch is currently tracking the rust-team branch,

> because it depends on changes there. It is shaping up nicely and we

> hope to be able to merge it some time soon.

I recall you wanted to keep python-team small, but do you mind if I
push https://issues.guix.gnu.org/74993there (it’s a small, but
world-rebuilding change)?

Thanks,
Lars

Ricardo Wurmus wrote on 5 Jan 13:22 +0100

Recipients:(name . Lars-Dominik Braun)(address . lars@6xq.net)

Message-ID:87a5c55u20.fsf@elephly.net

Hi Lars,

Toggle quote (8 lines)

>> The python-team branch is currently tracking the rust-team branch,

>> because it depends on changes there. It is shaping up nicely and we

>> hope to be able to merge it some time soon.

> I recall you wanted to keep python-team small, but do you mind if I

> push https://issues.guix.gnu.org/74993 there (it’s a small, but

> world-rebuilding change)?

Looks fine to me. Would this fix the build failure we see in

python-fastapi-pagination-minimal, which seems to be due to a parse

error?

I know I said I wanted to merge this branch quickly and keep the changes

restricted to important fixes, but it turned out that many of the

required fixes are wide-ranging anyway. I had to upgrade a lot of

packages that led to thousands of rebuilds. At this point I'd still

like to keep *experimental* changes out, but almost all other

improvements are welcome.

Thank you!

Ricardo

Lars-Dominik Braun wrote on 5 Jan 14:00 +0100

Recipients:(name . Ricardo Wurmus)(address . rekado@elephly.net)

Message-ID:Z3qCgfkzBnDm5ZnJ@noor.fritz.box

Hi,

Toggle quote (4 lines)

> Looks fine to me. Would this fix the build failure we see in

> python-fastapi-pagination-minimal, which seems to be due to a parse

> error?

yes, that looks like the error this patch tries to fix.

Thanks,

Lars

Ricardo Wurmus wrote on 5 Jan 14:02 +0100

Recipients:(address . guix-patches@gnu.org)

Message-ID:87y0zp4dmp.fsf@elephly.net

Ricardo Wurmus <rekado@elephly.net> writes:

Toggle quote (4 lines)

> The python-team branch is currently tracking the rust-team branch,

> because it depends on changes there. It is shaping up nicely and we

> hope to be able to merge it some time soon.

Looking at the dashboard for any recent evaluation of the python-team

branch, we still see a lot of failures, e.g. here:

https://ci.guix.gnu.org/eval/1967649/dashboard

Is anyone here interested in helping to get most of the packages

prefixed with "python-" to build successfully?

Ricardo

jgart wrote on 5 Jan 21:18 +0100

Recipients:

Message-ID:993b9f1b6faec7a9735a1695679cc9e46ccbba4b@dismail.de

I probably won't have time to work on that in the coming weeks.

Could you cherry-pick out the packages that are introducing failures into a WIP branch and make the scope of this request smaller so that it can be merged?

Someone can then address the failures in the next python-team merge request.

WDYT

all the best,

jgart

Attachment: file

Ricardo Wurmus wrote on 13 Jan 14:10 +0100

Last minute python-team fixes: help needed!

Recipients:(address . guix-devel@gnu.org)

Message-ID:87v7uikggc.fsf@elephly.net

Hi,

I've spent the last weeks whipping the python-team branch into shape,

but I'm running out of steam. There are still a number of failures, and

I'm about to lose motivation to continue fixing the remaining build

failures.

Here are some of the failures that may be of interest:

ansible

python-libcst (needs rust)

python-pandas 1.5

python-pytorch 2.0.1

python-django-q

python-pyodbc-c (3.1.5 fails with: undefined symbol: PyUnicode_GetMax)

python-agate (whole wireservice module needs upgrades)

I won't be working on these packages. Instead I'll focus on preparing

the guix-science channel for the merge. Please view the dashboard for

the latest evaluation of python-team at

https://ci.guix.gnu.org/jobset/python-teamto see if you could

contribute fixes for the remaining failures. Thanks!

The "python-team" branch has just been rebased on top of "master". It

will soon be merged --- even with the above failures --- to avoid

falling into our old habits of delaying merges and encouraging the

proliferation of increasingly divergent fixes across several branches.

It is next in line after "libgcrypt-gpg-update"

(https://issues.guix.gnu.org/75180).

(If you find yourself in Cc, you are listed as a member of the python

team. If you consider this to be outdated, please submit a change to

etc/teams.scm.)

Ricardo

??? wrote on 14 Jan 03:38 +0100

Recipients:(name . Ricardo Wurmus)(address . rekado@elephly.net)

Message-ID:87msfum85v.fsf@envs.net

Ricardo Wurmus <rekado@elephly.net> writes:

Toggle quote (7 lines)

> Hi,

> I've spent the last weeks whipping the python-team branch into shape,

> but I'm running out of steam. There are still a number of failures, and

> I'm about to lose motivation to continue fixing the remaining build

> failures.

Thank you for the work!

Toggle quote (16 lines)> Here are some of the failures that may be of interest:
>
>   ansible
>   python-libcst (needs rust)
>   python-pandas 1.5
>   python-pytorch 2.0.1
>   python-django-q
>   python-pyodbc-c (3.1.5 fails with: undefined symbol: PyUnicode_GetMax)
>   python-agate (whole wireservice module needs upgrades)
>
> I won't be working on these packages.  Instead I'll focus on preparing
> the guix-science channel for the merge.  Please view the dashboard for
> the latest evaluation of python-team at
> https://ci.guix.gnu.org/jobset/python-team to see if you could
> contribute fixes for the remaining failures.  Thanks!

I can contribute some fixes today, it seems many failures are due to
missing python-setuptools and python-wheel in its native-inputs when
using the pyproject-build-system.

Sharlatan Hellseher wrote on 14 Jan 21:47 +0100

Request for merging "python-team" branch

Recipients:(address . 75246@debbugs.gnu.org)(address . rekado@elephly.net)

Message-ID:874j21jf6b.fsf@gmail.com

Hi Ricardo,

I found 2 duplicated packages with different names in bioinformatics:

this

Toggle snippet (23 lines)(define-public fanc
  (let ((commit "354401e52ba2320e6b1ba0d3b5aab3541d31c9f3")
	(revision "1"))
    (package
      (name "fanc")
      (version (git-version "0" revision commit))
      (source (origin
		(method git-fetch)
		(uri (git-reference
		      (url "https://github.com/vaquerizaslab/fanc.git")
		      (commit commit)))
		(file-name (git-file-name name version))
		(sha256
		 (base32
		  "0vp2cak5snla4j2q23d3ixx016bwzvxfiv4z6yp7anl0xaksd5bl"))))
...
      (home-page "https://github.com/vaquerizaslab/fanc")
      (synopsis "Framework for the analysis of C-like data")
      (description "FAN-C provides a pipeline for analysing Hi-C data starting
at mapped paired-end sequencing reads.")
      (license license:gpl3+))))

and this

Toggle snippet (24 lines)(define-public python-fanc
  (package
    (name "python-fanc")
    (version "0.9.25")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/vaquerizaslab/fanc")
                    ;; There are no tags.  This commit corresponds to
                    ;; version 0.9.25.
                    (commit "e2205346c13ea5349681dff21adeb271d4ea5261")))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "0rxq24p852iiayi0083fyigvc30as695rha71q6xd4s2ij1k9mqi"))))
...
    (home-page "https://github.com/vaquerizaslab/fanc")
    (synopsis "Framework for the analysis of C-data")
    (description
     "FAN-C provides a pipeline for analysing Hi-C data starting at
mapped paired-end sequencing reads.")
    (license license:gpl3+)))

Failing and took too much time on tests I've gave up on 20min :-).

Oleg

Ricardo Wurmus wrote on 16 Jan 09:44 +0100

Recipients:(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)(address . 75246@debbugs.gnu.org)

Message-ID:87ldvbi1wt.fsf@elephly.net

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

Toggle quote (3 lines)

> I found 2 duplicated packages with different names in bioinformatics:

> [...]

Thanks for letting me know. I fixed this yesterday and upgraded

python-fanc.

Unfortunately, ci.guix.gnu.org seems to have stopped following our

progress.

https://qa.guix.gnu.org/branch/python-teamdoesn't know the merge base,

so I will rebase python-team once more on top of master some time today.

Ricardo

Ricardo Wurmus wrote on 16 Jan 10:51 +0100

Recipients:(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)(address . 75246@debbugs.gnu.org)

Message-ID:87v7ufgk88.fsf@elephly.net

Ricardo Wurmus <rekado@elephly.net> writes:

Toggle quote (3 lines)

> Unfortunately, ci.guix.gnu.org seems to have stopped following our

> progress.

I've fixed this by deleting a stray lock file.

Toggle quote (3 lines)

> https://qa.guix.gnu.org/branch/python-teamdoesn't know the merge base,

> so I will rebase python-team once more on top of master some time today.

I just pushed the rebased branch.

Ricardo

Sharlatan Hellseher wrote on 16 Jan 11:18 +0100

Request for merging "python-team" branch

Recipients:(address . 75246@debbugs.gnu.org)(name . Ricardo Wurmus)(address . rekado@elephly.net)

Message-ID:87y0zbhxik.fsf@gmail.com

Hi Ricardo,

Do you have a list of packages which you would like to fix/update before

merge?

I try to fix the build of python-stcal which was affected by numpy

upgrade, have no solution yet.

Oleg

Ricardo Wurmus wrote on 16 Jan 22:22 +0100

Recipients:(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)(address . 75246@debbugs.gnu.org)

Message-ID:87jzauh2sm.fsf@elephly.net

Toggle quote (3 lines)

> Do you have a list of packages which you would like to fix/update before

> merge?

I don't have a list, but I have some ongoing work that I'd like to

complete first. I'm afraid one of the fixes will cause yet another

world rebuild, but it's best to get it over with now than to introduce

more workarounds.

Toggle quote (3 lines)

> I try to fix the build of python-stcal which was affected by numpy

> upgrade, have no solution yet.

I have these things still going on before I would be comfortable with a

merge:

- python-pysnptools (almost done, with lots of Rust)

- python-modin (tests don't work yet)

- python-pynetdicom (lots of tests fail)

- python-asyncua (freezes)

In guix-science I'd like to fix python-tensorflow, but that seems to be

a small problem only -- we could relax requirements there, because it

complains about a patch-level mismatch in the numpy version.

(I noticed this time around that quite a few builds have a failing test suite

only when run on my laptop with more than 4 cores. botocore is one of

them. A bit strange.)

Ricardo

Ricardo Wurmus wrote on 16 Jan 22:33 +0100

Recipients:(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)(address . 75246@debbugs.gnu.org)

Message-ID:87bjw6h2a9.fsf@elephly.net

Ricardo Wurmus <rekado@elephly.net> writes:

Toggle quote (8 lines)

> I have these things still going on before I would be comfortable with a

> merge:

> - python-pysnptools (almost done, with lots of Rust)

> - python-modin (tests don't work yet)

> - python-pynetdicom (lots of tests fail)

> - python-asyncua (freezes)

Another important one: python-psycopg-3.1.10 appears to be failing its

test suite now. I haven't looked into it yet.

Ricardo

Leo Famulari wrote on 17 Jan 20:57 +0100

Re: [bug#75246] Request for merging "python-team" branch

Recipients:(name . Ricardo Wurmus)(address . rekado@elephly.net)

Message-ID:Z4q2OBbSGal-fBtD@jasmine.lan

On Wed, Jan 01, 2025 at 06:58:58PM +0100, Ricardo Wurmus wrote:

Toggle quote (4 lines)

> The python-team branch is currently tracking the rust-team branch,

> because it depends on changes there. It is shaping up nicely and we

> hope to be able to merge it some time soon.

I think that we should be bold in merging the branch if there are only
scattered problems.

Already we have had large parts of the Python package graph failing to
build on master (including most of the scientific and numerical
packages), and it was at least partly fixed by cherry-picking from
python-team.

I think it's okay to trade "new with some failures" for "old with some
failures" on the master branch. At least there will be more help to fix
them.

jgart wrote on 17 Jan 21:42 +0100

Recipients:

Message-ID:105c89248e36c2e3baa57eecd42f0a8b9dd9a340@dismail.de

Toggle quote (5 lines)> 
> I think it's okay to trade "new with some failures" for "old with some
> failures" on the master branch. At least there will be more help to fix
> them.
> 

If this the state that we are already in on master then sounds good to me.

In the future, it would be great to have some hooks in place to prevent people from pushing updates to master that break packages or cause too many rebuilds.

In other words, I don't think we should be allowed to push commits to master that break the package graph accidentally.

How can we make such a thing happen? I don't have an answer at the moment :)

all best,

jgart

Attachment: file

Leo Famulari wrote on 18 Jan 00:33 +0100

Recipients:(name . jgart)(address . jgart@dismail.de)

Message-ID:Z4ro2zzOgbvQGmN5@jasmine.lan

On Fri, Jan 17, 2025 at 08:42:41PM +0000, jgart wrote:

Toggle quote (6 lines)

> In the future, it would be great to have some hooks in place to prevent people from pushing updates to master that break packages or cause too many rebuilds.

> In other words, I don't think we should be allowed to push commits to master that break the package graph accidentally.

> How can we make such a thing happen? I don't have an answer at the moment :)

That's basically the premise of qa.guix.gnu.org, and continuous

integration in general.

We "just have to" use those resources in a more disciplined way.

jgart wrote on 18 Jan 01:15 +0100

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:51d3b0baaa9bd5d9c4ecfdad1aa5ee212e4fe471@dismail.de

One issue with QA that confuses me though is when it says unknown. For example, here:

https://issues.guix.gnu.org/75635

What does "unknown" mean? Abort looking at QA?

If the correct response, then, now there's no QA for 75635? ;()

Attachment: file

Christopher Baines wrote on 18 Jan 05:24 +0100

Recipients:(name . jgart via Guix-patches via)(address . guix-patches@gnu.org)

Message-ID:87o704agw9.fsf@cbaines.net

jgart via Guix-patches via <guix-patches@gnu.org> writes:

Toggle quote (8 lines)

> One issue with QA that confuses me though is when it says unknown. For example, here:

> https://issues.guix.gnu.org/75635

> What does "unknown" mean? Abort looking at QA?

> If the correct response, then, now there's no QA for 75635? ;()

The overall status is a summary, there should be more information on the
relevant QA page for that issue. Currently on that page [1] the thing
highlighted in red reads:

  Builds for this patch series not yet submitted as master branch
  substitute availability is low for: aarch64-linux armhf-linux
  i686-linux x86_64-linux

1: https://qa.guix.gnu.org/issue/75635

If you look at the page, QA does actually have some information,
e.g. how many rebuilds there are and some information on lint warnings
to, but it can't give an overall status because it is missing build
information for both the master branch and the state after applying the
patches (the "Unknown" column in the "Package changes" table).

Currently my approach so far has been that QA shouldn't be a blocker,
e.g. if it doesn't give the OK then that's not something you have to
wait for, but given how the bordeaux build farm has been struggling to
keep up recently I'm starting to think that if people don't start
waiting for QA, substitute availability might never catch up so the
bordeaux build farm is always going to be behind and therefore QA will
be unable to actually provide any information.

Ricardo Wurmus wrote 7 days ago

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87cygjeyub.fsf@elephly.net

Leo Famulari <leo@famulari.name> writes:

Toggle quote (8 lines)

> On Wed, Jan 01, 2025 at 06:58:58PM +0100, Ricardo Wurmus wrote:

>> The python-team branch is currently tracking the rust-team branch,

>> because it depends on changes there. It is shaping up nicely and we

>> hope to be able to merge it some time soon.

> I think that we should be bold in merging the branch if there are only

> scattered problems.

We had not been able to figure out if there really are only scattered
problems.  Just a few days ago we had to do another world rebuild due to
an upgrade of a small package close to the root of the Python test
graph.  But from what I can tell in test builds of real-world
environments we're in good shape.

Toggle quote (5 lines)

> Already we have had large parts of the Python package graph failing to

> build on master (including most of the scientific and numerical

> packages), and it was at least partly fixed by cherry-picking from

> python-team.

I'm pretty much done with what I set out to fix. Now it's up to the

build farms to successfully complete the evaluations. I'd like to merge

this all on Monday if the build farms cooperate.

...and then I'd like to start a new shortlived python-team branch with

an upgrade from Python 3.10 to 3.11.

Ricardo

Leo Famulari wrote 5 days ago

Recipients:(name . Ricardo Wurmus)(address . rekado@elephly.net)

Message-ID:Z47SOjyYdHWDvfAg@jasmine.lan

On Mon, Jan 20, 2025 at 09:56:39PM +0100, Ricardo Wurmus wrote:

Toggle quote (3 lines)

> Commit 69d657f9c4ffa59d708356baacbefafb58f09308 is now the tip of the

> "master" branch. The "python-team" branch has been merged.

Hooray! Thanks to the whole team for their work.

Closed

Sharlatan Hellseher wrote 5 days ago

Request for merging "python-team" branch

Recipients:(address . 75246@debbugs.gnu.org)(address . rekado@elephly.net)

Message-ID:87plkhi3np.fsf@gmail.com

Hi,

Thank you for merging and leading this refresh quest!

Toggle quote (5 lines)

> I see that we have Python 3.12, but let's please target 3.11 first.

> It's a small step up from 3.10, and we already know of packages that

> won't work with 3.12. Let's keep the next iteration of this branch

> simple, please.

While refreshing (now) packages from astronomy module I've faced wit few

failing to build due to hard requirement of Python3.11+ and NumPy 1.27+

or 2.0+ which don't have yet.

I've got a chance to to refresh NumPy to 1.24 but any farther version

failed to build (see notes on the package).

May we give it one more go with NumPy and friends?

Oleg

Ricardo Wurmus wrote 5 days ago

Recipients:(address . 75246-done@debbugs.gnu.org)

Message-ID:87ed0xdx14.fsf@elephly.net

Toggle quote (3 lines)

> build farms to successfully complete the evaluations. I'd like to merge

> this all on Monday if the build farms cooperate.

https://qa.guix.gnu.org/branch/python-teamsadly doesn't show *anything*

other than "Information unavailable", but ci.guix.gnu.org is happy. The

dashboard looks much greener than what we currently have for "master",

so I'm feeling good about merging this.

Commit 69d657f9c4ffa59d708356baacbefafb58f09308 is now the tip of the

"master" branch. The "python-team" branch has been merged.

Toggle quote (3 lines)

> ...and then I'd like to start a new shortlived python-team branch with

> an upgrade from Python 3.10 to 3.11.

I see that we have Python 3.12, but let's please target 3.11 first.

It's a small step up from 3.10, and we already know of packages that

won't work with 3.12. Let's keep the next iteration of this branch

simple, please.

Thanks to everyone who contributed to this branch!

Ricardo

Closed

Ricardo Wurmus wrote 5 days ago

Re: Request for merging "python-team" branch

Recipients:(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)(address . 75246@debbugs.gnu.org)

Message-ID:8734hce9u4.fsf@elephly.net

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

Toggle quote (14 lines)>> I see that we have Python 3.12, but let's please target 3.11 first.
>> It's a small step up from 3.10, and we already know of packages that
>> won't work with 3.12.  Let's keep the next iteration of this branch
>> simple, please.
>
> While refreshing (now) packages from astronomy module I've faced wit few
> failing to build due to hard requirement of Python3.11+ and NumPy 1.27+
> or 2.0+ which don't have yet.
>
> I've got a chance to to refresh NumPy to 1.24 but any farther version
> failed to build (see notes on the package).
>
> May we give it one more go with NumPy and friends?

I'd like to avoid the kind of sprawling upgrade chain reactions that

we've encountered in the most recent iteration. My hope is that we've

already upgraded the most important and deeply rooted packages so that

an upgrade to numpy won't lead to an upgrade storm.

Please keep it at Numpy 1.x, though. As with Pandas I think it makes

sense to first add an extra package for 2.x so that we can use package

transformations to test the impact before switching the default.

Let's open a new issue for the next iteration.

Ricardo

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 75246@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 75246

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date