[PATCHES] [core-updates] PEP 517 python-build-system

> Sorry if I'm (very) late, but apprently this hasn't made it to master

> yet, so… What the status? Do you still need a willing-but-maybe-not-qualified

> person to review or discuss your patch?

> 1) Validate the general idea of using pypa-build is viable and

> sustainable in the long run – ideally through review by someone else

> than me. We can’t touch python-build-system every week to solve

> structural issues, so it needs to be bullet-proof.

> 2) Figure out how to run testing code. Currently python-build-system

> just picks pytest, if available – not sure this is the best option we

> have. How do we deal with other test systems? How do we pass options?

> 4) Iron out minor details like including pip in the python package or

> create a new python-toolchain package? What do we include in that

> meta-package? pip? virtualenv? …?

> 5) Fix my awkward Scheme code, especially regarding unpacking of the

> built wheels. Should we be using Python’s unzip module or can be

> assumed unzip is available in the build environment? (Should we add

> it?)

> sorry for being late for commenting on this (the time I can spend on

> guix is rather limited atm).

> *

>

> Not installing pip by default might break some user's environments.

> Anyhow, since using pip in guix is not such a good idea anyway, this

> should be okay.

> *

>

> "use-setuptools" is gone. There are still about 10 packages with

> "#:use-setuptools #f" - which means they are (expected to be)

> incompatible with setuptools for some reason. You might want to

> check whether these packages actually still can't be packages with

> setuptools.

> *

>

> setuptools-shim has been removed. I don't think this is a good idea,

> since this peace of code enforces packages to be actually build with

> setuptools instead of old distutils. This code is still in current

> pip, so I assume it is still required.

>

> (This shim ensures setuptools is used, even if setup.py only imports

> distutils. And setuptools is required for some options like

> ""--single-version-externally-managed" - as the comment for the shim

> says.)

> *

>

> set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much

> more helpful than the new one-line comment.

> *

>

> set-SOURCE-DATE-EPOCH: This implementation makes the code depend on

> wheel and wheel being used for installation.

> *

>

> Why has rename-pth-file been removed? Are you sure .pth-files are

> never created anymore nowerdays?

> *

>

> python-hashbang: Isn't this done already by the normal

> "patch-shebangs" phase after install in  gnu-build-system? (BTW:

> these are called *she*bangs).

> *

>

> I suggest to have phase compile-bytecode still honor older versions

> of python

> pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as

> far as I can see, adoption is low.

> AFAIK fhere is no standard way for running tests in python. pytest seems

> to be the most modern test-system. Anyhow packages still use nose or tox

> (which again might run pytest or nose, with parameters fetched from

> tox.ini). So I'm afraid, there is no general rule.

>

> Did the PyPA publish some recommendations or PEP on this?

> As I Python developer I nowerdays would expect pip and venv (which is

> part of the std-lib - but not the virualenv, which is a separate module)

> to be availalbe when installing "python". Anyhow I could live with pip

> being a separate package.

> "python-toolchain" sounds oversized for me. Would this include the

> C-compiler, too (which one? maybe I want to build cross). I'd rather not

> have such a package.

> The gnu-build-system already provides the "unzip" binary (used in phase

> "unpack"). So we could simply use this. Otherwise I recommend using the

> Python zip module, as this is what is used for creating the zip-archives

> :-)

>> Not installing pip by default might break some user's environments.

>> Anyhow, since using pip in guix is not such a good idea anyway, this

>> should be okay.

> True. We could rename python→python-minimal-interpreteronly (or similar;

> given that python-minimal already exists) and python-toolchain→python to

> work around that.

>> [setuptools-shim has been removed]

> Is this relevant though? I doubt many packages are still importing

> distutils and the few that do can be patched.

>> set-SOURCE-DATE-EPOCH: This implementation makes the code depend on

>> wheel and wheel being used for installation.

> Technically it depends on the wheel builder understanding

> SOURCE_DATE_EPOCH (not necessarily python-wheel). I’d say that’s

> acceptable and it’d be preferable to fix build systems not respecting

> this variable imo.

>

>> Why has rename-pth-file been removed? Are you sure .pth-files are

>> never created anymore nowerdays?

> Given that easy-install has been deprecated I think it’s safe to remove

> this phase and flag any packages creating this easy-install.pth as

> broken. (There are, however, legitimate packages creating files like

> ruamel.yaml-0.15.83-py3.8-nspkg.pth.)

>

>> python-hashbang: Isn't this done already by the normal

>> "patch-shebangs" phase after install in  gnu-build-system? (BTW:

>> these are called *she*bangs).

> Afaik the function patch-shebang expects a leading slash and thus it

> does not replace this “special” shebang (see

> https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl;

> Spread, point 3).

>> *

>>

>> I suggest to have phase compile-bytecode still honor older versions

>> of python

> I’m not sure what you mean. compileall is also part of Python 2.

>> As I Python developer I nowerdays would expect pip and venv (which is

>> part of the std-lib - but not the virualenv, which is a separate module)

>> to be availalbe when installing "python". Anyhow I could live with pip

>> being a separate package.

> If we keep setuptools/pip bundled, we don’t have to do any of this

> pypa-build dance. We could also modernize python-build-system around

> `pip install` and just be done with it. (I don’t have a proof-of-concept

> for that yet.)

>

>> The gnu-build-system already provides the "unzip" binary (used in phase

>> "unpack"). So we could simply use this. Otherwise I recommend using the

>> Python zip module, as this is what is used for creating the zip-archives

>> :-)

> I’m using Python’s zipfile module already.

> What should be the use of having a package without pip? Anything else

> than saving a few KB?

> AFAIK this might not be true if other build systems not using setuptools

> at all might show up. And isn't this the main reason for all your work?

> No, try

>

> git clone https://github.com/pypa/pep517.git

> cd pep517

> pip wheel --use-pep517 -v .

>

> which has no setup.py and uses flit instead. pip can build it, because

> it supports PEP 517-based builds. As I said, if we decide to keep it

> bundled with our python package, there’s no good reason to choose

> pypa-build.

>> AFAIK this might not be true if other build systems not using setuptools

>> at all might show up. And isn't this the main reason for all your work?

> No, try

> > What should be the use of having a package without pip? Anything else

> > than saving a few KB?

> saving some space and unvendoring components that we also have separate

> packages for. (As I understand it, ensurepip, which installs both pip and

> setuptools, is merely a convenience tool if you install Python from

> source yourself – not for distributions.)

> I pushed the 3rd version of my new PEP 517-based python-build-system

> into wip-python-pep517. I removed the dependency on python-pypa-build

> for building packages and instead implemented building with a single

> python invokation. This simplifies bootstrapping Python. The 'install

> phase also learned how to create scripts in bin/ from an entry points

> specification.

> Currently 600 of 2212 packages using python-build-system are failing to

> build. That’s alot, but most failures so far were related to testsuites,

> which were not enabled correctly previously because 'check was not

> replaced manually. And some fail because upstream does not separate

> sources and tests and a name clash occurs. This also happens when C

> extensions are build and there does not seem to be a workaround.

> This number does not include Python 2 packages, which will probably

> break with this new build system. Since Python 2 is EOL anyway I suggest

> to entirely remove Python 2 support before merging this changeset. After

> merging we should upgrade the entire Python ecosystem, because alot of

> packages are already years old.

> Can you elaborate on the name clash issue?

> Unfortunately we need Python 2 for some time still. It is used to

> bootstrap old versions of GHC, Node, and probably other things.

> Do you think it is feasible to provide this as a new build system, say

> pep517-build-system, during a transitional period? Then we can a) start

> using it right away for the increasing amount of packages that lack a

> setup.py; and b) flesh out most bugs before eventually merging it back

> (possibly piecemeal) into python-build-system.

> * Zipping a wheel just to unpack it afterwards is weird, but there seems

> to be no way around it.

> * I also think trying "python setup.py test" is unnecessary.

> * It would be nice to support the "no tests" case without having to add

> explicit #:tests? everywhere. Perhaps along the lines of...

> (_ (apply invoke `("python" "-m" "unittest" ,@test-flags))))))

> Hi Marius,

>

>> Can you elaborate on the name clash issue?

> the problem seems to be that most packages do not put the source code

> into a subdirectory in the sdist. I.e. package foobar will have a foobar/

> directory, instead of src/foobar. Thus Python tries to load the module

> foobar.barbaz from that directory, instead of the store. Thus it’ll

> also look there for C extensions and fail. I don’t know why it does

> that, even when *not* using `python -m` – according to the documentation

> it should not. If anyone has any pointers I’d like to have a look again.

>> Unfortunately we need Python 2 for some time still. It is used to

>> bootstrap old versions of GHC, Node, and probably other things.

> But as far as I see these specific examples do not use

> python-build-system. Only if some package using `#:python python-2` is

> required during bootstrapping, then we’d need a way to support Python

> 2, right?

>> Do you think it is feasible to provide this as a new build system, say

>> pep517-build-system, during a transitional period? Then we can a) start

>> using it right away for the increasing amount of packages that lack a

>> setup.py; and b) flesh out most bugs before eventually merging it back

>> (possibly piecemeal) into python-build-system.

> Actually, I think that’s a good idea. I tried, but I cannot fix all

> packages broken by this change on my own, so smoothing the transition

> could help and – as you said – we could catch bugs early on.

>

> I could prepare a patch, if there is interest in this path.

>> * I also think trying "python setup.py test" is unnecessary.

> It still works quite often, although its usefulness will decrease in

> the future I guess. Another problem I see is that this command will not

> fail if there are no tests.

>

>> * It would be nice to support the "no tests" case without having to add

>> explicit #:tests? everywhere. Perhaps along the lines of...

> My idea was to force packagers to make and explain this decision

> explicitly. If you don’t run tests, you have to add `#:tests? #f`

> and leave a comment why they are disabled. I do see this could become

> a hassle with packages that simply don’t have any tests. But my hope

> is that it increases package quality.

>> (_ (apply invoke `("python" "-m" "unittest" ,@test-flags))))))

> If I remember correctly I tried this and it did not work for some

> reason. I’ll have a look again.

> the attached patches switch python-build-system to a PEP 517-based build

> system using python-pypa-build.

> One downside is that this tool is not self-contained and has a few

> dependencies. Thus first I bootstrap setuptools using itself (possible

> because it bundles all of its own dependencies), then build

> python-pypa-build’s dependencies using setuptools (which is fortunately

> still possible) and then combine everything into a

> python-toolchain(-for-build), which is then used by the build-process.

>

> I can successfully build packages like python-pypa-build and

> python-pytest and python-pep517-bootstrap. The latter is using flit as

> its build backend. But other packages currently fail because I removed

> some arguments.

> Lars

>

>

>

>

>>From 61313d8ddba30772e2587e3e16ca30d1565d3c7e Mon Sep 17 00:00:00 2001

> From: Lars-Dominik Braun <lars@6xq.net>

> Date: Sun, 28 Feb 2021 13:05:51 +0100

> Subject: [PATCH 04/12] gnu: python-setuptools: Bootstrap using itself

>

> * gnu/packages/python-xyz.scm (python-setuptools) [arguments]: Add phase

> setting GUIX_PYTHONPATH to source directory.

> ---

> gnu/packages/python-xyz.scm | 13 ++++++++++++-

> 1 file changed, 12 insertions(+), 1 deletion(-)

>

> diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm

> index f8afa13f33..79d01f700a 100644

> --- a/gnu/packages/python-xyz.scm

> +++ b/gnu/packages/python-xyz.scm

> @@ -1144,7 +1144,18 @@ other machines, such as over the network.")

> ;; FIXME: Tests require pytest, which itself relies on setuptools.

> ;; One could bootstrap with an internal untested setuptools.

> (arguments

> - `(#:tests? #f))

> + `(#:tests? #f

> + #:python ,python-wrapper

> + #:phases (modify-phases %standard-phases

> + ;; Use this setuptools’ sources to bootstrap themselves.

> + (add-before 'build 'set-PYTHONPATH

> + (lambda _

> + (format #t "current working dir ~s~%" (getcwd))

> + (setenv "GUIX_PYTHONPATH"

> + (string-append ".:" (getenv "GUIX_PYTHONPATH")))

> + #t)))))

> + ;; Not required when not building a wheel

> + ;(propagated-inputs `(("python-wheel" ,python-wheel)))

> (home-page "https://pypi.org/project/setuptools/")

> (synopsis

> "Library designed to facilitate packaging Python projects")

> --

> 2.26.2

>

>

>>From 7a99aaa40e65fde58ee2e78ad7d3e0ccd6d169ae Mon Sep 17 00:00:00 2001

> From: Lars-Dominik Braun <lars@6xq.net>

> Date: Sun, 28 Feb 2021 13:08:58 +0100

> Subject: [PATCH 05/12] python-build: Switch to PEP 517-based build

>

> * gnu/packages/python-commencement.scm: New file, containing

> python-toolchain.

> * gnu/local.mk: Add it.

> * gnu/packages/python.scm (python): Disable installing bundled

> pip/setuptools.

> * guix/build/python-build-system.scm: Rewrite using python-pypa-build.

> * guix/build-system/python.scm (default-python): Switch to

> python-toolchain

> (lower): Remove unused parameter.

>

> XXX: rationale

> ---

> gnu/local.mk | 1 +

> gnu/packages/python-commencement.scm | 175 +++++++++++++++++++

> gnu/packages/python.scm | 2 +-

> guix/build-system/python.scm | 8 +-

> guix/build/python-build-system.scm | 249 ++++++++++++++++++---------

> 5 files changed, 342 insertions(+), 93 deletions(-)

> create mode 100644 gnu/packages/python-commencement.scm

> + (use-modules (ice-9 match)

> + (srfi srfi-1)

> + (srfi srfi-26)

> + (guix build union))

> +

> + (let ((out (assoc-ref %outputs "out")))

> + (union-build out (filter-map (match-lambda

> + ((_ . directory) directory))

> + %build-inputs))

> + #t))))

> + (inputs

> + `(("python" ,python-wrapper)

> + ("python-setuptools" ,python-setuptools)

> + ("python-pip" ,python-pip))) ; XXX Maybe virtualenv/venv too? It kind of

> + ; defeats the purpose of guix, but is used

> + ; alot in local development.

> + (native-search-paths

> + (package-native-search-paths python))

> + (search-paths

> + (package-search-paths python))

> + (license (package-license python)) ; XXX

> + (synopsis "Python toolchain")

> + (description

> + "Python toolchain including Python itself, setuptools and pip. Use this

> +package if you need a fully-fledged Python toolchain instead of just the

> +interpreter.")

> + (home-page (package-home-page python))))

> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm

> index 8e8f46467b..ca5ce667ef 100644

> --- a/gnu/packages/python.scm

> +++ b/gnu/packages/python.scm

> @@ -182,7 +182,7 @@

> (list "--enable-shared" ;allow embedding

> "--with-system-expat" ;for XML support

> "--with-system-ffi" ;build ctypes

> - "--with-ensurepip=install" ;install pip and setuptools

> + "--with-ensurepip=no" ;do not install pip and setuptools

> "--enable-unicode=ucs4"

>

> ;; Prevent the installed _sysconfigdata.py from retaining a reference

> diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm

> index 2bb6fa87ca..998ea9323d 100644

> --- a/guix/build-system/python.scm

> +++ b/guix/build-system/python.scm

> @@ -65,8 +65,8 @@ extension, such as '.tar.gz'."

> (define (default-python)

> "Return the default Python package."

> ;; Lazily resolve the binding to avoid a circular dependency.

> - (let ((python (resolve-interface '(gnu packages python))))

> - (module-ref python 'python-wrapper)))

> + (let ((python (resolve-interface '(gnu packages python-commencement))))

> + (module-ref python 'python-toolchain-for-build)))

>

> (define (default-python2)

> "Return the default Python 2 package."

> @@ -172,8 +172,6 @@ pre-defined variants."

> (define* (python-build store name inputs

> #:key

> (tests? #t)

> - (test-target "test")

> - (use-setuptools? #t)

> (configure-flags ''())

> (phases '(@ (guix build python-build-system)

> %standard-phases))

> @@ -199,9 +197,7 @@ provides a 'setup.py' file as its build system."

> source))

> #:configure-flags ,configure-flags

> #:system ,system

> - #:test-target ,test-target

> #:tests? ,tests?

> - #:use-setuptools? ,use-setuptools?

> #:phases ,phases

> #:outputs %outputs

> #:search-paths ',(map search-path-specification->sexp

> diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm

> index 8ade1d5911..a5731511a9 100644

> --- a/guix/build/python-build-system.scm

> +++ b/guix/build/python-build-system.scm

> @@ -34,6 +34,7 @@

> #:use-module (ice-9 format)

> #:use-module (srfi srfi-1)

> #:use-module (srfi srfi-26)

> + #:use-module (srfi srfi-35)

> #:export (%standard-phases

> add-installed-pythonpath

> site-packages

> @@ -108,30 +109,17 @@

> ;; "--single-version-externally-managed" is set, thus the .egg-info directory

> ;; and the scripts defined in entry-points will always be created.

>

> +;; Base error type.

> +(define-condition-type &python-build-error &error

> + python-build-error?)

>

> -(define setuptools-shim

> - ;; Run setup.py with "setuptools" being imported, which will patch

> - ;; "distutils". This is needed for packages using "distutils" instead of

> - ;; "setuptools" since the former does not understand the

> - ;; "--single-version-externally-managed" flag.

> - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py

> - (string-append

> - "import setuptools, tokenize;__file__='setup.py';"

> - "f=getattr(tokenize, 'open', open)(__file__);"

> - "code=f.read().replace('\\r\\n', '\\n');"

> - "f.close();"

> - "exec(compile(code, __file__, 'exec'))"))

> -

> -(define (call-setuppy command params use-setuptools?)

> - (if (file-exists? "setup.py")

> - (begin

> - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%"

> - command params)

> - (if use-setuptools?

> - (apply invoke "python" "-c" setuptools-shim

> - command params)

> - (apply invoke "python" "./setup.py" command params)))

> - (error "no setup.py found")))

> +;; Raised when 'check cannot find a valid test system in the inputs.

> +(define-condition-type &test-system-not-found &python-build-error

> + test-system-not-found?)

> +

> +;; Raised when multiple wheels are created by 'build.

> +(define-condition-type &cannot-extract-multiple-wheels &python-build-error

> + cannot-extract-multiple-wheels?)

>

> (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys)

> "Ensure packages depending on this package via setuptools work properly,

> @@ -142,23 +130,51 @@ without errors."

> (with-directory-excursion "/tmp"

> (invoke "python" sanity-check.py (site-packages inputs outputs)))))

>

> -(define* (build #:key use-setuptools? #:allow-other-keys)

> +(define* (build #:key outputs #:allow-other-keys)

> "Build a given Python package."

> - (call-setuppy "build" '() use-setuptools?)

> +

> + (define pyproject-build (which "pyproject-build"))

> +

> + (define (build-pep517)

> + ;; XXX: should probably use a different path, outside of source directory,

> + ;; maybe secondary output “wheel”?

> + (mkdir-p "dist")

> + (invoke pyproject-build "--outdir" "dist" "--no-isolation" "--wheel" "."))

> +

> + ;; XXX Would be nice, if we could use bdist_wheel here to remove extra

> + ;; code path in 'install, but that depends on python-wheel.

> + (define (build-setuptools)

> + (invoke "python" "setup.py" "build"))

> +

> + (if pyproject-build

> + (build-pep517)

> + (build-setuptools))

> #t)

>

> -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys)

> +(define* (check #:key inputs outputs tests? #:allow-other-keys)

> "Run the test suite of a given Python package."

> (if tests?

> - ;; Running `setup.py test` creates an additional .egg-info directory in

> - ;; build/lib in some cases, e.g. if the source is in a sub-directory

> - ;; (given with `package_dir`). This will by copied to the output, too,

> - ;; so we need to remove.

> - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t)))

> - (call-setuppy test-target '() use-setuptools?)

> - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t))

> - (inter (lset-difference string=? after before)))

> - (for-each delete-file-recursively inter)))

> + ;; Unfortunately with PEP 517 there is no common method to specify test

> + ;; systems. Guess test system based on inputs instead.

> + (let ((pytest (which "pytest"))

> + (have-setup-py (file-exists? "setup.py")))

> + ;; Prefer pytest

> + ;; XXX: support nose

> + (cond

> + (pytest

> + (begin

> + (format #t "using pytest~%")

> + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments?

> + ;; But fall back to setup.py, which should work for most

> + ;; packages. XXX: would be nice not to depend on setup.py here? fails

> + ;; more often than not to find any tests at all. Maybe we can run

> + ;; `python -m unittest`?

> + (have-setup-py

> + (begin

> + (format #t "using setup.py~%")

> + (invoke "python" "setup.py" "test" "-v")))

> + ;; The developer should explicitly disable tests in this case.

> + (#t (raise (condition (&test-system-not-found))))))

> (format #t "test suite not run~%"))

> #t)

>

> @@ -195,31 +211,109 @@ running checks after installing the package."

> "/bin:"

> (getenv "PATH"))))

>

> -(define* (install #:key inputs outputs (configure-flags '()) use-setuptools?

> - #:allow-other-keys)

> - "Install a given Python package."

> - (let* ((out (python-output outputs))

> - (python (assoc-ref inputs "python"))

> - (major-minor (map string->number

> - (take (string-split (python-version python) #\.) 2)))

> - (<3.7? (match major-minor

> - ((major minor)

> - (or (< major 3) (and (= major 3) (< minor 7))))))

> - (params (append (list (string-append "--prefix=" out)

> - "--no-compile")

> - (if use-setuptools?

> - ;; distutils does not accept these flags

> - (list "--single-version-externally-managed"

> - "--root=/")

> - '())

> - configure-flags)))

> - (call-setuppy "install" params use-setuptools?)

> - ;; Rather than produce potentially non-reproducible .pyc files on Pythons

> - ;; older than 3.7, whose 'compileall' module lacks the

> - ;; '--invalidation-mode' option, do not generate any.

> - (unless <3.7?

> - (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash"

> - out))))

> +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys)

> + "Install a wheel file according to PEP 427"

> + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl

> + (let* ((site-dir (site-packages inputs outputs))

> + (out (assoc-ref outputs "out")))

> + (define (extract file)

> + "Extract wheel (ZIP file) into site-packages directory"

> + ;; Use Python’s zipfile to avoid extra dependency

> + (invoke "python" "-m" "zipfile" "-e" file site-dir))

> +

> + (define python-hashbang

> + (string-append "#!" (assoc-ref inputs "python") "/bin/python"))

> +

> + (define (move-data source destination)

> + (mkdir-p (dirname destination))

> + (rename-file source destination))

> +

> + (define (move-script source destination)

> + "Move executable script file from .data/scripts to out/bin and replace

> +temporary hashbang"

> + (move-data source destination)

> + ;; ZIP does not save/restore permissions, make executable

> + ;; XXX: might not be a file, but directory with subdirectories

> + (chmod destination #o755)

> + (substitute* destination (("#!python") python-hashbang)))

> + ;; Python’s distutils.command.install defines this mapping from source to

> + ;; destination mapping.

> + (define install-schemes

> + `(("scripts" "bin" ,move-script)

> + ;; XXX: Why does Python not use share/ here?

> + ("data" "share" ,move-data)))

> +

> + (define (expand-data-directory directory)

> + "Move files from all .data subdirectories to their respective

> +destinations."

> + (for-each

> + (match-lambda ((source destination function)

> + (let ((source-path (string-append directory "/" source))

> + (destination-path (string-append out "/" destination)))

> + (when (file-exists? source-path)

> + (begin

> + ;; This assumes only files exist in the scripts/ directory.

> + (for-each

> + (lambda (file)

> + (apply

> + function

> + (list

> + (string-append source-path "/" file)

> + (string-append destination-path "/" file))))

> + (scandir source-path (negate (cut member <> '("." "..")))))

> + (rmdir source-path))))))

> + install-schemes))

> +

> + (define pyproject-build (which "pyproject-build"))

> +

> + (define (list-directories base predicate)

> + ;; Cannot use find-files here, because it’s recursive.

> + (scandir

> + base

> + (lambda (name)

> + (let ((stat (lstat (string-append base "/" name))))

> + (and

> + (not (member name '("." "..")))

> + (eq? (stat:type stat) 'directory)

> + (predicate name stat))))))

> +

> + (define (install-pep517)

> + "Install a wheel generated by a PEP 517-compatible builder."

> + (let ((wheels (find-files "dist" "\\.whl$"))) ; XXX: do not recurse

> + (when (> (length wheels) 1) ; This code does not support multiple wheels

> + ; yet, because their outputs would have to be

> + ; merged properly.

> + (raise (condition (&cannot-extract-multiple-wheels))))

> + (for-each extract wheels))

> + (let ((datadirs (map

> + (cut string-append site-dir "/" <>)

> + (list-directories site-dir (file-name-predicate "\\.data$")))))

> + (for-each (lambda (directory)

> + (expand-data-directory directory)

> + (rmdir directory))

> + datadirs)))

> +

> + (define (install-setuptools)

> + "Install using setuptools."

> + (let ((out (assoc-ref outputs "out")))

> + (invoke "python" "setup.py"

> + "install"

> + "--prefix" out

> + "--single-version-externally-managed"

> + "--root=/")))

> +

> + (if pyproject-build

> + (install-pep517)

> + (install-setuptools))

> + #t))

> +

> +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys)

> + "Compile installed byte-code in site-packages."

> + (let ((site-dir (site-packages inputs outputs)))

> + (invoke "python" "-m" "compileall" site-dir)

> + ;; XXX: We could compile with -O and -OO too here, at the cost of more space.

> + #t))

> Here's a review of patches 1 to 6.

> > + ;; Prefer pytest

> > + ;; XXX: support nose

>

> You can remove this; nose is stale/deprecated.

>

> > + (cond

> > + (pytest

> > + (begin

> > + (format #t "using pytest~%")

> > + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments?

>

> We could have a #:test-command argument to specify an arbitrary command

> as a list of strings, such as used by the emacs-build-system; that'd

> allow us to avoid overriding the phase just to add a '-k "not

> this-test"' or similar.

> > + ;; But fall back to setup.py, which should work for most

> > + ;; packages. XXX: would be nice not to depend on setup.py here? fails

> > + ;; more often than not to find any tests at all. Maybe we can run

> > + ;; `python -m unittest`?

> > + (have-setup-py

> > + (begin

> > + (format #t "using setup.py~%")

> > + (invoke "python" "setup.py" "test" "-v")))

>

> As Marius noted, falling back to 'python setup.py test' is not

> desirable; it's scheduled to be removed already.

> > + (define (move-script source destination)

> > + "Move executable script file from .data/scripts to out/bin and replace

> > +temporary hashbang"

> > + (move-data source destination)

> > + ;; ZIP does not save/restore permissions, make executable

> > + ;; XXX: might not be a file, but directory with subdirectories

> > + (chmod destination #o755)

> > + (substitute* destination (("#!python") python-hashbang)))

>

> It seems the directory case should be handled? Otherwise the

> substitute* call would error out upon encountering it.

> So, IIUC, this complicated install phase is because we no longer take

> 'pip' for granted and is only later available, built from this very

> build system, right? Otherwise installing a wheel with pip would be

> trivial (c.f. python-isort).

> When you judge the branch ready to merge, could you please send a subset

> of the patches (at least the ones touching the python-build-system

> directly) to this issue (marked as v2 -- git send-email -v2) so that

> they can be more easily commented?

> Hi Maxim,

>

>> When you judge the branch ready to merge, could you please send a subset

>> of the patches (at least the ones touching the python-build-system

>> directly) to this issue (marked as v2 -- git send-email -v2) so that

>> they can be more easily commented?

>

> as soon as time permits I can do that. Which route do we want to

> take? Replace python-build-system entirely in one big merge or add a

> new one and slowly migrate?

> When you judge the branch ready to merge, could you please send a subset

> of the patches (at least the ones touching the python-build-system

> directly) to this issue (marked as v2 -- git send-email -v2) so that

> they can be more easily commented?

> With the pyproject build system now in master, can we close this issue?