[python-team] Respect --cores build argument.

Greg Hogan wrote on 20 Nov 16:10 +0100

Python respect --cores build argument

Recipients:(address . bug-guix@gnu.org)

Message-ID:CA+3U0Zk+KGiRJCPymMkPE6AYU5PWgZ+G1srgSXfLMGcYN7N7Sg@mail.gmail.com

#74436 fixed two instances of "--numprocesses" "auto" but a simple

grep shows 18 additional cases (one of which is commented out).

Sharlatan Hellseher wrote on 20 Nov 19:47 +0100

Recipients:(address . 74445@debbugs.gnu.org)(name . Greg Hogan)(address . code@greghogan.com)

Message-ID:CAO+9K5qB_q3CLdmCK2X44eQX4OVmbMGmxDYGm1GwtQzhnU+urQ@mail.gmail.com

Hi,

There a lot in astronomy module I guess ^.^

Would you like to send a patch adjusting them?

They could be "-n" "auto" as well.

Thanks,

Oleg

Attachment: file

Greg Hogan wrote on 20 Nov 20:16 +0100

retitle 74445

Recipients:(address . control@debbugs.gnu.org)

Message-ID:CA+3U0Z=oGLE1=z6+rJuNHpOipLt=jpt=qKp8p+d_mWTpU5PzkA@mail.gmail.com

retitle 74445 [python-team] Respect --cores build argument.

Sharlatan Hellseher wrote on 20 Nov 21:54 +0100

[PATCH 0/1] gnu: Respect --cores build argument in some python packages.

Recipients:(address . 74445@debbugs.gnu.org)(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)

Message-ID:cover.1732135585.git.sharlatanus@gmail.com

Hi Greg,

Thanks for highlighting that, I was too much keen to make tests faster.

This patch adjust all "-n" or "--numprocesses" from "auto" to

"parallel-job-count" parameter.

This quite save to be pushed to master directly:

Toggle snippet (28 lines)Building the following 77 packages would ensure 180 dependent packages are
rebuilt: python-yt-astro-analysis@1.1.3 variant-tools@3.1.2 osc@0.172.0
h-client@0.0a0-1.e6c78b1 python-urlgrabber@4.1.0 xapers@0.8.2 ganeti@3.0.2
awscli@1.36.0 instrain@1.5.4 python-osmnx@1.9.3 conda@22.9.0
python-awscrt@0.23.0 awscli@2.20.0 python-spacy@3.5.3 snakemake@6.15.5
python-scm-sr-ht@0.22.23 awslogs@0.14.0 python-tibanna@5.4.3
python-hicexplorer@3.7.4 python-vaex-core@4.13.0 python-kanon@0.6.6
python-zodipy@1.1.0 python-coolest@0.1.9 python-sncosmo@2.11.1
python-halotools@0.9.1 python-healpy@1.16.6 scamp@2.10.0
python-asdf-compression@0.0.1-0.57cc7e7 python-asdf-zarr@0.0.4
python-regularizepsf@0.4.0 python-statmorph@0.5.7 python-pyxsim@4.4.2
python-viresclient@0.12.0 python-naima@0.10.0 python-aplpy@2.1.0
python-astroalign@2.6.0 ginga-qt5@5.1.0 python-poliastro@0.17.0
python-astroplan@0.10.1 python-bulkvis@2.0.0-2.00a82a9 python-cooltools@0.7.0
fanc@0-1.354401e python-fanc@0.9.25 python-pytorch-geometric@2.4.0 cura@4.13.1
python-sunkit-image@0.5.1 python-radiospectra@0.6.0 python-webbpsf@1.4.0
python-acstools@3.7.2 python-jwst-reffiles@1.0.1 python-sbpy@0.5.0
python-glue-qt@0.3.1 python-glue-astronomy@0.10.0 python-pyrodigal@3.3.0
python-verde@1.8.0 python-scikit-allel@1.3.5 python-pandera@0.18.0
python-modin@0.15.1 python-cesium@0.12.1 synapse@1.29.0 python-sparse@0.14.0
python-demuxem@0.1.7 python-cellbender@0.2.2 python-scikit-bio@0.6.0
python-ikarus@0.0.2 python-muon@0.1.6 python-liana-py@1.1.0
r-doubletcollection@1.1.0-1.c0d62f1 python-episcanpy@0.4.0
python-metacells@0.9.4 python-celltypist@1.6.2 python-hotspotsc@1.1.1
python-multivelo@0.1.2 python-baltica@1.1.2 scregseg@0.1.3
python-magic-impute@1.2.1 python-multiscale-spatial-image@1.0.0

Thanks,

Oleg

Sharlatan Hellseher (1):

gnu: Respect --cores build argument in some python packages.

gnu/packages/astronomy.scm | 31 ++++++++++++++++---------------

gnu/packages/check.scm | 2 +-

gnu/packages/databases.scm | 2 +-

gnu/packages/python-web.scm | 6 +++---

gnu/packages/python-xyz.scm | 6 +++---

5 files changed, 24 insertions(+), 23 deletions(-)

base-commit: da8afaa65fe30ae59e1bedbbb231490ad01c013c

2.46.0

Sharlatan Hellseher wrote on 20 Nov 21:54 +0100

[PATCH 1/1] gnu: Respect --cores build argument in some python packages.

Recipients:(address . 74445@debbugs.gnu.org)(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)

Message-ID:7e70f59d92f55b3ad70baa428a2e301a20d967d3.1732135585.git.sharlatanus@gmail.com

Reported by Greg Hogan <code@greghogan.com> in #74445.

* gnu/packages/astronomy.scm (python-asdf-astropy, python-astropy,

python-photutils, python-poppy, python-regions, python-reproject,

python-sunpy, python-spectral-cube, python-stdatamodels, python-pysiaf,

python-sbpy, python-asdf-coordinates-schemas, python-roman-datamodels,

python-webbpsf, python-yt):

[arguments]<test-flags>: Adjust "-n" or "--numprocess" to respect

"--cores" build argument.

* gnu/packages/check.scm (python-crosshair): Likewise.

* gnu/packages/databases.scm (python-fastparquet, python-pycurl, awscli,

python-s3transfer): Likewise.

* gnu/packages/python-xyz.scm (python-glymur, python-zarr, python-dask):

Likewise.

Change-Id: Ifbc6435e4ad22b0ae822b485bccca41eaa165cc5

---

gnu/packages/astronomy.scm | 31 ++++++++++++++++---------------

gnu/packages/check.scm | 2 +-

gnu/packages/databases.scm | 2 +-

gnu/packages/python-web.scm | 6 +++---

gnu/packages/python-xyz.scm | 6 +++---

5 files changed, 24 insertions(+), 23 deletions(-)

Toggle diff (230 lines)diff --git a/gnu/packages/astronomy.scm b/gnu/packages/astronomy.scm
index 6cc5142370..296ac44dbf 100644
--- a/gnu/packages/astronomy.scm
+++ b/gnu/packages/astronomy.scm
@@ -1653,7 +1653,7 @@ (define-public python-asdf-astropy
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases #~(modify-phases %standard-phases
                    (add-before 'check 'set-home-env
                      (lambda _ (setenv "HOME" "/tmp"))))))
@@ -3034,7 +3034,7 @@ (define-public python-astropy
      (list
       #:test-flags
       #~(list "--pyargs" "astropy"
-              "--numprocesses" "auto"
+              "--numprocesses" (number->string (parallel-job-count))
               "-k" (string-append
                     ;; Skip tests that need remote data.
                     "not remote_data"
@@ -3924,7 +3924,7 @@ (define-public python-photutils
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
          (add-after 'unpack 'relax-requirements
@@ -4085,7 +4085,7 @@ (define-public python-poppy
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (propagated-inputs
      ;; XXX: With python-synphot (marked as optional) package added to the list
      ;; it tries to download from remote host during tests and fails. Overall
@@ -4191,7 +4191,7 @@ (define-public python-regions
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           ;; setup.py was removed in 84c80a280431adda00641cda5264c7de18b43b2f
@@ -4295,7 +4295,7 @@ (define-public python-reproject
       #:test-flags
       #~(list "--arraydiff"
               "--arraydiff-default-format=fits"
-              "--numprocesses" "auto"
+              "--numprocesses" (number->string (parallel-job-count))
               "--pyargs" "reproject")
       #:phases
       #~(modify-phases %standard-phases
@@ -4399,7 +4399,7 @@ (define-public python-sunpy
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Requries SpicePy not packed in Guix yet.
               "--ignore=sunpy/coordinates/tests/test_spice.py")
       #:phases
@@ -4589,7 +4589,7 @@ (define-public python-spectral-cube
       ;; See <https://github.com/radio-astro-tools/radio-beam/issues/129>.
       #:tests? #f
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (propagated-inputs
      (list python-astropy
            ;; XXX: Currently failing in upstream as it's optional silent
@@ -4981,7 +4981,7 @@ (define-public python-stdatamodels
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Disable tests requiring access to CRDS servers to download
               ;; ~500MiB of data.
               "-k" "not test_crds_selectors_vs_datamodel")
@@ -5450,7 +5450,7 @@ (define-public python-pysiaf
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Disable 2 failing tests, see
               ;; <https://github.com/spacetelescope/pysiaf/issues/338>
               "-k" (string-append "not test_write_jwst_siaf_xlsx"
@@ -5543,7 +5543,8 @@ (define-public python-sbpy
     (build-system pyproject-build-system)
     (arguments
      (list
-      #:test-flags #~(list "--numprocesses" "auto")
+      #:test-flags
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-before 'check 'set-home-env
@@ -5835,7 +5836,7 @@ (define python-asdf-coordinates-schemas
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (native-inputs
      (list python-pytest
            python-pytest-xdist
@@ -6137,7 +6138,7 @@ (define-public python-roman-datamodels
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-after 'unpack 'set-env
@@ -6295,7 +6296,7 @@ (define-public python-webbpsf
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               "-k" (string-append
                     ;; Test requiring network access
                     "not test_monthly_trending_plot_auto_opdtable"
@@ -6383,7 +6384,7 @@ (define-public python-yt
      (list
       #:build-backend "setuptools.build_meta"
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
          (add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 6593eadf6a..ef1c08f8f3 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -2744,7 +2744,7 @@ (define-public python-crosshair
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; check_examples_test.py contains failing tests that
               ;; show what happens if a counterexample is found.
               "--ignore=crosshair/examples/check_examples_test.py"
diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 5f762eb5be..142f9090c5 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -5203,7 +5203,7 @@ (define-public python-fastparquet
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index c030deeb12..cf3c15857c 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -2026,7 +2026,7 @@ (define-public python-pycurl
     (build-system pyproject-build-system)
     (arguments
      '(#:test-flags
-       (list "-n" "auto"
+       (list "--n" (number->string (parallel-job-count))
              "-k" (string-append
                    ;; Disable hanginging tests
                    "not test_multi_socket_select"
@@ -4235,7 +4235,7 @@ (define-public awscli
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Tests require networking.
               "--ignore" "tests/integration"
               ;; It strugles to set PYTHONPATH.
@@ -4909,7 +4909,7 @@ (define-public python-s3transfer
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Tests require networking.
               "--ignore" "tests/integration")))
     (native-inputs
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index ceb122e08d..3872b511a1 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -2296,7 +2296,7 @@ (define-public python-glymur
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Failing test due to inability of ctypes.util.find_library()
               ;; to determine library path, which is patched above.
               "--ignore=tests/test_config.py")
@@ -28320,7 +28320,7 @@ (define-public python-zarr
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; This tests are flaky.  The pass several times on my laptop
               ;; but occasionally fail.  They fail pretty reliably on the
               ;; build farm.
@@ -28845,7 +28845,7 @@ (define-public python-dask
      (list
       ;; Avoid coverage
       #:test-flags
-      #~(list "-n" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               "-m" "not gpu and not slow and not network"
               ;; These all fail with different hashes.  Doesn't seem
               ;; problematic.
-- 
2.46.0

Sharlatan Hellseher wrote on 23 Nov 15:54 +0100

[python-team] Respect --cores build argument.

Recipients:(address . 74445-done@debbugs.gnu.org)

Message-ID:87ed32ngp8.fsf@gmail.com

Hi,

Pushed as 2d1a7ec92d91b8346d257e5352b65fdce35070a5 to master.

Thanks,

Oleg

Closed

Sharlatan Hellseher wrote on 23 Nov 17:00 +0100

Recipients:(address . 74445@debbugs.gnu.org)

Message-ID:877c8undn8.fsf@gmail.com

Correct commit: origin/master d3b010c38c449ea17bf7beba48de39acec26ae04

Oleg

Your comment

This issue is archived.

To comment on this conversation send an email to 74445@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 74445

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date