[python-team] Respect --cores build argument.

  • Open
  • quality assurance status badge
Details
2 participants
  • Greg Hogan
  • Sharlatan Hellseher
Owner
unassigned
Submitted by
Greg Hogan
Severity
normal
G
G
Greg Hogan wrote 16 hours ago
Python respect --cores build argument
(address . bug-guix@gnu.org)
CA+3U0Zk+KGiRJCPymMkPE6AYU5PWgZ+G1srgSXfLMGcYN7N7Sg@mail.gmail.com
#74436 fixed two instances of "--numprocesses" "auto" but a simple
grep shows 18 additional cases (one of which is commented out).
S
S
Sharlatan Hellseher wrote 13 hours ago
(address . 74445@debbugs.gnu.org)(name . Greg Hogan)(address . code@greghogan.com)
CAO+9K5qB_q3CLdmCK2X44eQX4OVmbMGmxDYGm1GwtQzhnU+urQ@mail.gmail.com
Hi,

There a lot in astronomy module I guess ^.^

Would you like to send a patch adjusting them?

They could be "-n" "auto" as well.

Thanks,
Oleg
Attachment: file
G
G
Greg Hogan wrote 12 hours ago
retitle 74445
(address . control@debbugs.gnu.org)
CA+3U0Z=oGLE1=z6+rJuNHpOipLt=jpt=qKp8p+d_mWTpU5PzkA@mail.gmail.com
retitle 74445 [python-team] Respect --cores build argument.
S
S
Sharlatan Hellseher wrote 11 hours ago
[PATCH 0/1] gnu: Respect --cores build argument in some python packages.
(address . 74445@debbugs.gnu.org)(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)
cover.1732135585.git.sharlatanus@gmail.com
Hi Greg,

Thanks for highlighting that, I was too much keen to make tests faster.

This patch adjust all "-n" or "--numprocesses" from "auto" to
"parallel-job-count" parameter.
This quite save to be pushed to master directly:

Toggle snippet (28 lines)
Building the following 77 packages would ensure 180 dependent packages are
rebuilt: python-yt-astro-analysis@1.1.3 variant-tools@3.1.2 osc@0.172.0
h-client@0.0a0-1.e6c78b1 python-urlgrabber@4.1.0 xapers@0.8.2 ganeti@3.0.2
awscli@1.36.0 instrain@1.5.4 python-osmnx@1.9.3 conda@22.9.0
python-awscrt@0.23.0 awscli@2.20.0 python-spacy@3.5.3 snakemake@6.15.5
python-scm-sr-ht@0.22.23 awslogs@0.14.0 python-tibanna@5.4.3
python-hicexplorer@3.7.4 python-vaex-core@4.13.0 python-kanon@0.6.6
python-zodipy@1.1.0 python-coolest@0.1.9 python-sncosmo@2.11.1
python-halotools@0.9.1 python-healpy@1.16.6 scamp@2.10.0
python-asdf-compression@0.0.1-0.57cc7e7 python-asdf-zarr@0.0.4
python-regularizepsf@0.4.0 python-statmorph@0.5.7 python-pyxsim@4.4.2
python-viresclient@0.12.0 python-naima@0.10.0 python-aplpy@2.1.0
python-astroalign@2.6.0 ginga-qt5@5.1.0 python-poliastro@0.17.0
python-astroplan@0.10.1 python-bulkvis@2.0.0-2.00a82a9 python-cooltools@0.7.0
fanc@0-1.354401e python-fanc@0.9.25 python-pytorch-geometric@2.4.0 cura@4.13.1
python-sunkit-image@0.5.1 python-radiospectra@0.6.0 python-webbpsf@1.4.0
python-acstools@3.7.2 python-jwst-reffiles@1.0.1 python-sbpy@0.5.0
python-glue-qt@0.3.1 python-glue-astronomy@0.10.0 python-pyrodigal@3.3.0
python-verde@1.8.0 python-scikit-allel@1.3.5 python-pandera@0.18.0
python-modin@0.15.1 python-cesium@0.12.1 synapse@1.29.0 python-sparse@0.14.0
python-demuxem@0.1.7 python-cellbender@0.2.2 python-scikit-bio@0.6.0
python-ikarus@0.0.2 python-muon@0.1.6 python-liana-py@1.1.0
r-doubletcollection@1.1.0-1.c0d62f1 python-episcanpy@0.4.0
python-metacells@0.9.4 python-celltypist@1.6.2 python-hotspotsc@1.1.1
python-multivelo@0.1.2 python-baltica@1.1.2 scregseg@0.1.3
python-magic-impute@1.2.1 python-multiscale-spatial-image@1.0.0

Thanks,
Oleg

Sharlatan Hellseher (1):
gnu: Respect --cores build argument in some python packages.

gnu/packages/astronomy.scm | 31 ++++++++++++++++---------------
gnu/packages/check.scm | 2 +-
gnu/packages/databases.scm | 2 +-
gnu/packages/python-web.scm | 6 +++---
gnu/packages/python-xyz.scm | 6 +++---
5 files changed, 24 insertions(+), 23 deletions(-)


base-commit: da8afaa65fe30ae59e1bedbbb231490ad01c013c
--
2.46.0
S
S
Sharlatan Hellseher wrote 11 hours ago
[PATCH 1/1] gnu: Respect --cores build argument in some python packages.
(address . 74445@debbugs.gnu.org)(name . Sharlatan Hellseher)(address . sharlatanus@gmail.com)
7e70f59d92f55b3ad70baa428a2e301a20d967d3.1732135585.git.sharlatanus@gmail.com
Reported by Greg Hogan <code@greghogan.com> in #74445.

* gnu/packages/astronomy.scm (python-asdf-astropy, python-astropy,
python-photutils, python-poppy, python-regions, python-reproject,
python-sunpy, python-spectral-cube, python-stdatamodels, python-pysiaf,
python-sbpy, python-asdf-coordinates-schemas, python-roman-datamodels,
python-webbpsf, python-yt):
[arguments]<test-flags>: Adjust "-n" or "--numprocess" to respect
"--cores" build argument.

* gnu/packages/check.scm (python-crosshair): Likewise.

* gnu/packages/databases.scm (python-fastparquet, python-pycurl, awscli,
python-s3transfer): Likewise.

* gnu/packages/python-xyz.scm (python-glymur, python-zarr, python-dask):
Likewise.

Change-Id: Ifbc6435e4ad22b0ae822b485bccca41eaa165cc5
---
gnu/packages/astronomy.scm | 31 ++++++++++++++++---------------
gnu/packages/check.scm | 2 +-
gnu/packages/databases.scm | 2 +-
gnu/packages/python-web.scm | 6 +++---
gnu/packages/python-xyz.scm | 6 +++---
5 files changed, 24 insertions(+), 23 deletions(-)

Toggle diff (230 lines)
diff --git a/gnu/packages/astronomy.scm b/gnu/packages/astronomy.scm
index 6cc5142370..296ac44dbf 100644
--- a/gnu/packages/astronomy.scm
+++ b/gnu/packages/astronomy.scm
@@ -1653,7 +1653,7 @@ (define-public python-asdf-astropy
(arguments
(list
#:test-flags
- #~(list "-n" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases #~(modify-phases %standard-phases
(add-before 'check 'set-home-env
(lambda _ (setenv "HOME" "/tmp"))))))
@@ -3034,7 +3034,7 @@ (define-public python-astropy
(list
#:test-flags
#~(list "--pyargs" "astropy"
- "--numprocesses" "auto"
+ "--numprocesses" (number->string (parallel-job-count))
"-k" (string-append
;; Skip tests that need remote data.
"not remote_data"
@@ -3924,7 +3924,7 @@ (define-public python-photutils
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'relax-requirements
@@ -4085,7 +4085,7 @@ (define-public python-poppy
(arguments
(list
#:test-flags
- #~(list "-n" "auto")))
+ #~(list "--numprocesses" (number->string (parallel-job-count)))))
(propagated-inputs
;; XXX: With python-synphot (marked as optional) package added to the list
;; it tries to download from remote host during tests and fails. Overall
@@ -4191,7 +4191,7 @@ (define-public python-regions
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
;; setup.py was removed in 84c80a280431adda00641cda5264c7de18b43b2f
@@ -4295,7 +4295,7 @@ (define-public python-reproject
#:test-flags
#~(list "--arraydiff"
"--arraydiff-default-format=fits"
- "--numprocesses" "auto"
+ "--numprocesses" (number->string (parallel-job-count))
"--pyargs" "reproject")
#:phases
#~(modify-phases %standard-phases
@@ -4399,7 +4399,7 @@ (define-public python-sunpy
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Requries SpicePy not packed in Guix yet.
"--ignore=sunpy/coordinates/tests/test_spice.py")
#:phases
@@ -4589,7 +4589,7 @@ (define-public python-spectral-cube
;; See <https://github.com/radio-astro-tools/radio-beam/issues/129>.
#:tests? #f
#:test-flags
- #~(list "-n" "auto")))
+ #~(list "--numprocesses" (number->string (parallel-job-count)))))
(propagated-inputs
(list python-astropy
;; XXX: Currently failing in upstream as it's optional silent
@@ -4981,7 +4981,7 @@ (define-public python-stdatamodels
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Disable tests requiring access to CRDS servers to download
;; ~500MiB of data.
"-k" "not test_crds_selectors_vs_datamodel")
@@ -5450,7 +5450,7 @@ (define-public python-pysiaf
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Disable 2 failing tests, see
;; <https://github.com/spacetelescope/pysiaf/issues/338>
"-k" (string-append "not test_write_jwst_siaf_xlsx"
@@ -5543,7 +5543,8 @@ (define-public python-sbpy
(build-system pyproject-build-system)
(arguments
(list
- #:test-flags #~(list "--numprocesses" "auto")
+ #:test-flags
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-before 'check 'set-home-env
@@ -5835,7 +5836,7 @@ (define python-asdf-coordinates-schemas
(arguments
(list
#:test-flags
- #~(list "-n" "auto")))
+ #~(list "--numprocesses" (number->string (parallel-job-count)))))
(native-inputs
(list python-pytest
python-pytest-xdist
@@ -6137,7 +6138,7 @@ (define-public python-roman-datamodels
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'set-env
@@ -6295,7 +6296,7 @@ (define-public python-webbpsf
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
"-k" (string-append
;; Test requiring network access
"not test_monthly_trending_plot_auto_opdtable"
@@ -6383,7 +6384,7 @@ (define-public python-yt
(list
#:build-backend "setuptools.build_meta"
#:test-flags
- #~(list "--numprocesses" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 6593eadf6a..ef1c08f8f3 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -2744,7 +2744,7 @@ (define-public python-crosshair
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; check_examples_test.py contains failing tests that
;; show what happens if a counterexample is found.
"--ignore=crosshair/examples/check_examples_test.py"
diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 5f762eb5be..142f9090c5 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -5203,7 +5203,7 @@ (define-public python-fastparquet
(arguments
(list
#:test-flags
- #~(list "-n" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index c030deeb12..cf3c15857c 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -2026,7 +2026,7 @@ (define-public python-pycurl
(build-system pyproject-build-system)
(arguments
'(#:test-flags
- (list "-n" "auto"
+ (list "--n" (number->string (parallel-job-count))
"-k" (string-append
;; Disable hanginging tests
"not test_multi_socket_select"
@@ -4235,7 +4235,7 @@ (define-public awscli
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Tests require networking.
"--ignore" "tests/integration"
;; It strugles to set PYTHONPATH.
@@ -4909,7 +4909,7 @@ (define-public python-s3transfer
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Tests require networking.
"--ignore" "tests/integration")))
(native-inputs
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index ceb122e08d..3872b511a1 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -2296,7 +2296,7 @@ (define-public python-glymur
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Failing test due to inability of ctypes.util.find_library()
;; to determine library path, which is patched above.
"--ignore=tests/test_config.py")
@@ -28320,7 +28320,7 @@ (define-public python-zarr
(arguments
(list
#:test-flags
- #~(list "-n" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; This tests are flaky. The pass several times on my laptop
;; but occasionally fail. They fail pretty reliably on the
;; build farm.
@@ -28845,7 +28845,7 @@ (define-public python-dask
(list
;; Avoid coverage
#:test-flags
- #~(list "-n" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
"-m" "not gpu and not slow and not network"
;; These all fail with different hashes. Doesn't seem
;; problematic.
--
2.46.0
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74445@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74445
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch