From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 01 08:44:07 2021 Received: (at submit) by debbugs.gnu.org; 1 Mar 2021 13:44:07 +0000 Received: from localhost ([127.0.0.1]:48521 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lGiq5-00040s-W0 for submit@debbugs.gnu.org; Mon, 01 Mar 2021 08:44:07 -0500 Received: from lists.gnu.org ([209.51.188.17]:34676) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lGipz-00040g-09 for submit@debbugs.gnu.org; Mon, 01 Mar 2021 08:43:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59510) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lGipy-0004en-Oi for guix-patches@gnu.org; Mon, 01 Mar 2021 08:43:50 -0500 Received: from mout-p-102.mailbox.org ([2001:67c:2050::465:102]:16092) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1lGipt-0004lQ-1k for guix-patches@gnu.org; Mon, 01 Mar 2021 08:43:49 -0500 Received: from smtp2.mailbox.org (smtp2.mailbox.org [80.241.60.241]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4Dq1gR1VCBzQkJK for ; Mon, 1 Mar 2021 14:43:31 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1614606209; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=gszE9nJa3Nr0ww6hptcwp9mkhmP+fxtZ+I4I9OWz0K8=; b=JlGtx7QHCUxeel9+2zXBXXgRkdkROe6GH/85isarZMdvW8j6kF7zTtIi2VrEK/HlK2BvKG hNyioSoWFYXYwJFCpDdE8EMo3DuCRh3+JCd31EI+hlt+p71GpaMuK9w3vpNtAkmxwMs9yM PNEZFF3gcJjICXOJDkDd8CK1L36pxL/bcQWP7yM2O1sWQEIiixPnWiGomfjgOfO6u0s7t6 gR3hdvj65shvfV/zYx6IZaMVMIcGqzO8aU0ZkJpnIUkwIboNYrfs2uYFQ7KsfHrLnWuN9L rKCa/qZhCstPWsQLOJ1miUb/Cnn0mQE5tAd9pmsgZoqoWwrFPg347yZWQDtESw== Received: from smtp2.mailbox.org ([80.241.60.241]) by spamfilter03.heinlein-hosting.de (spamfilter03.heinlein-hosting.de [80.241.56.117]) (amavisd-new, port 10030) with ESMTP id jjDxxdmqfzvB for ; Mon, 1 Mar 2021 14:43:24 +0100 (CET) Date: Mon, 1 Mar 2021 14:43:22 +0100 From: Lars-Dominik Braun To: guix-patches@gnu.org Subject: [PATCHES] [core-updates] PEP 517 python-build-system Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="9EzuTbC0hpH3Zvh5" Content-Disposition: inline Content-Transfer-Encoding: 8bit X-MBO-SPAM-Probability: X-Rspamd-Score: -8.95 / 15.00 / 15.00 X-Rspamd-Queue-Id: C071C17B4 X-Rspamd-UID: 3d4170 Received-SPF: pass client-ip=2001:67c:2050::465:102; envelope-from=lars@6xq.net; helo=mout-p-102.mailbox.org X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_SBL_A=0.1 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) --9EzuTbC0hpH3Zvh5 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Hi everyone, the attached patches switch python-build-system to a PEP 517-based build system using python-pypa-build. As discussed previously Python is currently in the process of opening up for build systems other than setuptools using the API specified in PEP 517. python-pypa-build is a simple tool for building packages using that new API. It supports setup.py-based builds as a fallback, if no pyproject.toml is present. One downside is that this tool is not self-contained and has a few dependencies. Thus first I bootstrap setuptools using itself (possible because it bundles all of its own dependencies), then build python-pypa-build’s dependencies using setuptools (which is fortunately still possible) and then combine everything into a python-toolchain(-for-build), which is then used by the build-process. I can successfully build packages like python-pypa-build and python-pytest and python-pep517-bootstrap. The latter is using flit as its build backend. But other packages currently fail because I removed some arguments. Cheers, Lars --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-build-python-Handle-missing-setuptools-in-sanity-che.patch" From 7ace01faef72f3a48b18fe241fe0524445a154fb Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 19 Feb 2021 17:22:35 +0100 Subject: [PATCH 01/12] build/python: Handle missing setuptools in sanity-check.py Just skip testing if required dependencies (setuptools) are not available. * gnu/packages/aux-files/python/sanity-check.py: Handle ImportError. --- gnu/packages/aux-files/python/sanity-check.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/gnu/packages/aux-files/python/sanity-check.py b/gnu/packages/aux-files/python/sanity-check.py index 83b6d583ca..240155cecc 100644 --- a/gnu/packages/aux-files/python/sanity-check.py +++ b/gnu/packages/aux-files/python/sanity-check.py @@ -19,9 +19,13 @@ from __future__ import print_function # Python 2 support. import importlib -import pkg_resources import sys import traceback +try: + import pkg_resources +except ImportError: + print('Warning: Skipping, because python-setuptools are not available.') + sys.exit(0) try: from importlib.machinery import PathFinder -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-gnu-python-pypa-build-Update-to-0.3.0.patch" From 66033f14455456d465a3c9f5a2d1f4961cd3b989 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 12:50:42 +0100 Subject: [PATCH 02/12] gnu: python-pypa-build: Update to 0.3.0. * gnu/packages/python-build.scm (python-pypa-build): Update to 0.3.0. --- gnu/packages/python-build.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 140629ca43..92dbda314f 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -118,13 +118,13 @@ Language (TOML) configuration files.") (define-public python-pypa-build (package (name "python-pypa-build") - (version "0.1.0") + (version "0.3.0") (source (origin (method url-fetch) (uri (pypi-uri "build" version)) (sha256 (base32 - "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8")))) + "1pazq66c35whrqd5b0zcydjvy2rmghi8riljkd67q3bpiln5pf8f")))) (build-system python-build-system) (arguments `(#:tests? #f ;to tests in the PyPI release -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-gnu-python-wheel-Install-entrypoint-scripts.patch" From 00ef998413dc0fe6605653ae16f65d6377c2ec77 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:02:15 +0100 Subject: [PATCH 03/12] gnu: python-wheel: Install entrypoint scripts * gnu/packages/python-build.scm (pythont-wheel) [arguments]: Add phase 'patch-enable-entrypoints. --- gnu/packages/python-build.scm | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 92dbda314f..232e24f470 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -47,10 +47,17 @@ "0ii6f34rvpjg3nmw4bc2h7fhdsy38y1h93hghncfs5akfrldmj8h")))) (build-system python-build-system) (arguments - ;; FIXME: The test suite runs "python setup.py bdist_wheel", which in turn - ;; fails to find the newly-built bdist_wheel library, even though it is - ;; available on PYTHONPATH. What search path is consulted by setup.py? - '(#:tests? #f)) + `(#:phases + (modify-phases %standard-phases + (add-after 'unpack 'patch-enable-entrypoints + (lambda _ + ;; python-wheel tells setuptools to not install entry point scripts + ;; by default. Stop doing that, so wheels built contain all data + ;; required. + (substitute* "wheel/bdist_wheel.py" + (("(install_scripts\\.no_ep = )True" all assignment) + (string-append assignment "False"))) + #t))))) (home-page "https://bitbucket.org/pypa/wheel/") (synopsis "Format for built Python packages") (description -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0004-gnu-python-setuptools-Bootstrap-using-itself.patch" Content-Transfer-Encoding: 8bit From 61313d8ddba30772e2587e3e16ca30d1565d3c7e Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:05:51 +0100 Subject: [PATCH 04/12] gnu: python-setuptools: Bootstrap using itself * gnu/packages/python-xyz.scm (python-setuptools) [arguments]: Add phase setting GUIX_PYTHONPATH to source directory. --- gnu/packages/python-xyz.scm | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index f8afa13f33..79d01f700a 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -1144,7 +1144,18 @@ other machines, such as over the network.") ;; FIXME: Tests require pytest, which itself relies on setuptools. ;; One could bootstrap with an internal untested setuptools. (arguments - `(#:tests? #f)) + `(#:tests? #f + #:python ,python-wrapper + #:phases (modify-phases %standard-phases + ;; Use this setuptools’ sources to bootstrap themselves. + (add-before 'build 'set-PYTHONPATH + (lambda _ + (format #t "current working dir ~s~%" (getcwd)) + (setenv "GUIX_PYTHONPATH" + (string-append ".:" (getenv "GUIX_PYTHONPATH"))) + #t))))) + ;; Not required when not building a wheel + ;(propagated-inputs `(("python-wheel" ,python-wheel))) (home-page "https://pypi.org/project/setuptools/") (synopsis "Library designed to facilitate packaging Python projects") -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0005-python-build-Switch-to-PEP-517-based-build.patch" Content-Transfer-Encoding: 8bit From 7a99aaa40e65fde58ee2e78ad7d3e0ccd6d169ae Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:08:58 +0100 Subject: [PATCH 05/12] python-build: Switch to PEP 517-based build * gnu/packages/python-commencement.scm: New file, containing python-toolchain. * gnu/local.mk: Add it. * gnu/packages/python.scm (python): Disable installing bundled pip/setuptools. * guix/build/python-build-system.scm: Rewrite using python-pypa-build. * guix/build-system/python.scm (default-python): Switch to python-toolchain (lower): Remove unused parameter. XXX: rationale --- gnu/local.mk | 1 + gnu/packages/python-commencement.scm | 175 +++++++++++++++++++ gnu/packages/python.scm | 2 +- guix/build-system/python.scm | 8 +- guix/build/python-build-system.scm | 249 ++++++++++++++++++--------- 5 files changed, 342 insertions(+), 93 deletions(-) create mode 100644 gnu/packages/python-commencement.scm diff --git a/gnu/local.mk b/gnu/local.mk index 202677fed1..ef2532cb5d 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -463,6 +463,7 @@ GNU_SYSTEM_MODULES = \ %D%/packages/python.scm \ %D%/packages/python-build.scm \ %D%/packages/python-check.scm \ + %D%/packages/python-commencement.scm \ %D%/packages/python-compression.scm \ %D%/packages/python-crypto.scm \ %D%/packages/python-science.scm \ diff --git a/gnu/packages/python-commencement.scm b/gnu/packages/python-commencement.scm new file mode 100644 index 0000000000..2ced3079bc --- /dev/null +++ b/gnu/packages/python-commencement.scm @@ -0,0 +1,175 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès +;;; Copyright © 2014 Andreas Enge +;;; Copyright © 2012 Nikita Karetnikov +;;; Copyright © 2014, 2015, 2017 Mark H Weaver +;;; Copyright © 2017, 2018, 2019, 2021 Efraim Flashner +;;; Copyright © 2018 Tobias Geerinckx-Rice +;;; Copyright © 2018, 2019, 2020 Jan (janneke) Nieuwenhuizen +;;; Copyright © 2019, 2020 Marius Bakke +;;; Copyright © 2020 Timothy Sample +;;; Copyright © 2020 Guy Fleury Iteriteka +;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2021 Lars-Dominik Braun +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (gnu packages python-commencement) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix download) + #:use-module (guix packages) + #:use-module (guix build-system trivial) + #:use-module (guix build-system python) + #:use-module (gnu packages) + #:use-module (gnu packages python) + #:use-module (gnu packages python-build) + #:use-module (gnu packages python-xyz) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26)) + +;; Python toolchain and all packages required to bootstrap it. + +(define-public python-toolchain + (package + (name "python-toolchain") + (version (package-version python)) + (source #f) + (build-system trivial-build-system) + (arguments + '(#:modules ((guix build union)) + #:builder (begin + (use-modules (ice-9 match) + (srfi srfi-1) + (srfi srfi-26) + (guix build union)) + + (let ((out (assoc-ref %outputs "out"))) + (union-build out (filter-map (match-lambda + ((_ . directory) directory)) + %build-inputs)) + #t)))) + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pip" ,python-pip))) ; XXX Maybe virtualenv/venv too? It kind of + ; defeats the purpose of guix, but is used + ; alot in local development. + (native-search-paths + (package-native-search-paths python)) + (search-paths + (package-search-paths python)) + (license (package-license python)) ; XXX + (synopsis "Python toolchain") + (description + "Python toolchain including Python itself, setuptools and pip. Use this +package if you need a fully-fledged Python toolchain instead of just the +interpreter.") + (home-page (package-home-page python)))) + +;; Python 3 toolchain for python-build-system. We cannot use python-toolchain +;; here, since we’d need to bootstrap python-pip somehow. +(define-public python-toolchain-for-build + (package + (inherit python-toolchain) + (name "python-toolchain-for-build") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pypa-build" ,python-pypa-build-from-setuptools))))) + +;; Python 3 toolchain to bootstrap python-pypa-build +(define-public python-toolchain-only-setuptools + (package + (inherit python-toolchain) + (name "python-toolchain-only-setuptools") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools))))) + +(define-public python-pypa-build-from-setuptools + (package + (inherit python-pypa-build) + (name "python-pypa-build-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools)) + (propagated-inputs + `(("python-pep517" ,python-pep517-from-setuptools) + ("python-packaging" ,python-packaging-from-setuptools))))) + +(define-public python-pep517-from-setuptools + (package + (inherit python-pep517-bootstrap) + (name "python-pep517-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + #:phases (modify-phases %standard-phases + (add-after 'unpack 'patch + (lambda _ + (substitute* "setup.py" + (("distutils\\.core") "setuptools")) + #t))))) + (propagated-inputs + `(("python-toml" ,python-toml-from-setuptools) + ("python-wheel" ,python-wheel-from-setuptools))) + ;; Drop cyclic dependency. + (native-inputs '()))) + +(define-public python-toml-from-setuptools + (package + (inherit python-toml) + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + ,@(package-arguments python-toml))))) + +(define-public python-packaging-from-setuptools + (package + (inherit python-packaging-bootstrap) + (name "python-packaging-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-packaging-bootstrap))) + (propagated-inputs + `(("python-pyparsing" ,python-pyparsing-from-setuptools) + ("python-six" ,python-six-from-setuptools))))) + +(define-public python-pyparsing-from-setuptools + (package + (inherit python-pyparsing) + (name "python-pyparsing-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + ,@(package-arguments python-pyparsing))))) + +(define-public python-six-from-setuptools + (package + (inherit python-six-bootstrap) + (name "python-six-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-six-bootstrap))))) + +(define-public python-wheel-from-setuptools + (package + (inherit python-wheel) + (name "python-wheel-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-wheel))))) + diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 8e8f46467b..ca5ce667ef 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -182,7 +182,7 @@ (list "--enable-shared" ;allow embedding "--with-system-expat" ;for XML support "--with-system-ffi" ;build ctypes - "--with-ensurepip=install" ;install pip and setuptools + "--with-ensurepip=no" ;do not install pip and setuptools "--enable-unicode=ucs4" ;; Prevent the installed _sysconfigdata.py from retaining a reference diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index 2bb6fa87ca..998ea9323d 100644 --- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -65,8 +65,8 @@ extension, such as '.tar.gz'." (define (default-python) "Return the default Python package." ;; Lazily resolve the binding to avoid a circular dependency. - (let ((python (resolve-interface '(gnu packages python)))) - (module-ref python 'python-wrapper))) + (let ((python (resolve-interface '(gnu packages python-commencement)))) + (module-ref python 'python-toolchain-for-build))) (define (default-python2) "Return the default Python 2 package." @@ -172,8 +172,6 @@ pre-defined variants." (define* (python-build store name inputs #:key (tests? #t) - (test-target "test") - (use-setuptools? #t) (configure-flags ''()) (phases '(@ (guix build python-build-system) %standard-phases)) @@ -199,9 +197,7 @@ provides a 'setup.py' file as its build system." source)) #:configure-flags ,configure-flags #:system ,system - #:test-target ,test-target #:tests? ,tests? - #:use-setuptools? ,use-setuptools? #:phases ,phases #:outputs %outputs #:search-paths ',(map search-path-specification->sexp diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm index 8ade1d5911..a5731511a9 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -34,6 +34,7 @@ #:use-module (ice-9 format) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-35) #:export (%standard-phases add-installed-pythonpath site-packages @@ -108,30 +109,17 @@ ;; "--single-version-externally-managed" is set, thus the .egg-info directory ;; and the scripts defined in entry-points will always be created. +;; Base error type. +(define-condition-type &python-build-error &error + python-build-error?) -(define setuptools-shim - ;; Run setup.py with "setuptools" being imported, which will patch - ;; "distutils". This is needed for packages using "distutils" instead of - ;; "setuptools" since the former does not understand the - ;; "--single-version-externally-managed" flag. - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py - (string-append - "import setuptools, tokenize;__file__='setup.py';" - "f=getattr(tokenize, 'open', open)(__file__);" - "code=f.read().replace('\\r\\n', '\\n');" - "f.close();" - "exec(compile(code, __file__, 'exec'))")) - -(define (call-setuppy command params use-setuptools?) - (if (file-exists? "setup.py") - (begin - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%" - command params) - (if use-setuptools? - (apply invoke "python" "-c" setuptools-shim - command params) - (apply invoke "python" "./setup.py" command params))) - (error "no setup.py found"))) +;; Raised when 'check cannot find a valid test system in the inputs. +(define-condition-type &test-system-not-found &python-build-error + test-system-not-found?) + +;; Raised when multiple wheels are created by 'build. +(define-condition-type &cannot-extract-multiple-wheels &python-build-error + cannot-extract-multiple-wheels?) (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) "Ensure packages depending on this package via setuptools work properly, @@ -142,23 +130,51 @@ without errors." (with-directory-excursion "/tmp" (invoke "python" sanity-check.py (site-packages inputs outputs))))) -(define* (build #:key use-setuptools? #:allow-other-keys) +(define* (build #:key outputs #:allow-other-keys) "Build a given Python package." - (call-setuppy "build" '() use-setuptools?) + + (define pyproject-build (which "pyproject-build")) + + (define (build-pep517) + ;; XXX: should probably use a different path, outside of source directory, + ;; maybe secondary output “wheel”? + (mkdir-p "dist") + (invoke pyproject-build "--outdir" "dist" "--no-isolation" "--wheel" ".")) + + ;; XXX Would be nice, if we could use bdist_wheel here to remove extra + ;; code path in 'install, but that depends on python-wheel. + (define (build-setuptools) + (invoke "python" "setup.py" "build")) + + (if pyproject-build + (build-pep517) + (build-setuptools)) #t) -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys) +(define* (check #:key inputs outputs tests? #:allow-other-keys) "Run the test suite of a given Python package." (if tests? - ;; Running `setup.py test` creates an additional .egg-info directory in - ;; build/lib in some cases, e.g. if the source is in a sub-directory - ;; (given with `package_dir`). This will by copied to the output, too, - ;; so we need to remove. - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t))) - (call-setuppy test-target '() use-setuptools?) - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t)) - (inter (lset-difference string=? after before))) - (for-each delete-file-recursively inter))) + ;; Unfortunately with PEP 517 there is no common method to specify test + ;; systems. Guess test system based on inputs instead. + (let ((pytest (which "pytest")) + (have-setup-py (file-exists? "setup.py"))) + ;; Prefer pytest + ;; XXX: support nose + (cond + (pytest + (begin + (format #t "using pytest~%") + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments? + ;; But fall back to setup.py, which should work for most + ;; packages. XXX: would be nice not to depend on setup.py here? fails + ;; more often than not to find any tests at all. Maybe we can run + ;; `python -m unittest`? + (have-setup-py + (begin + (format #t "using setup.py~%") + (invoke "python" "setup.py" "test" "-v"))) + ;; The developer should explicitly disable tests in this case. + (#t (raise (condition (&test-system-not-found)))))) (format #t "test suite not run~%")) #t) @@ -195,31 +211,109 @@ running checks after installing the package." "/bin:" (getenv "PATH")))) -(define* (install #:key inputs outputs (configure-flags '()) use-setuptools? - #:allow-other-keys) - "Install a given Python package." - (let* ((out (python-output outputs)) - (python (assoc-ref inputs "python")) - (major-minor (map string->number - (take (string-split (python-version python) #\.) 2))) - (<3.7? (match major-minor - ((major minor) - (or (< major 3) (and (= major 3) (< minor 7)))))) - (params (append (list (string-append "--prefix=" out) - "--no-compile") - (if use-setuptools? - ;; distutils does not accept these flags - (list "--single-version-externally-managed" - "--root=/") - '()) - configure-flags))) - (call-setuppy "install" params use-setuptools?) - ;; Rather than produce potentially non-reproducible .pyc files on Pythons - ;; older than 3.7, whose 'compileall' module lacks the - ;; '--invalidation-mode' option, do not generate any. - (unless <3.7? - (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" - out)))) +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Install a wheel file according to PEP 427" + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl + (let* ((site-dir (site-packages inputs outputs)) + (out (assoc-ref outputs "out"))) + (define (extract file) + "Extract wheel (ZIP file) into site-packages directory" + ;; Use Python’s zipfile to avoid extra dependency + (invoke "python" "-m" "zipfile" "-e" file site-dir)) + + (define python-hashbang + (string-append "#!" (assoc-ref inputs "python") "/bin/python")) + + (define (move-data source destination) + (mkdir-p (dirname destination)) + (rename-file source destination)) + + (define (move-script source destination) + "Move executable script file from .data/scripts to out/bin and replace +temporary hashbang" + (move-data source destination) + ;; ZIP does not save/restore permissions, make executable + ;; XXX: might not be a file, but directory with subdirectories + (chmod destination #o755) + (substitute* destination (("#!python") python-hashbang))) + + ;; Python’s distutils.command.install defines this mapping from source to + ;; destination mapping. + (define install-schemes + `(("scripts" "bin" ,move-script) + ;; XXX: Why does Python not use share/ here? + ("data" "share" ,move-data))) + + (define (expand-data-directory directory) + "Move files from all .data subdirectories to their respective +destinations." + (for-each + (match-lambda ((source destination function) + (let ((source-path (string-append directory "/" source)) + (destination-path (string-append out "/" destination))) + (when (file-exists? source-path) + (begin + ;; This assumes only files exist in the scripts/ directory. + (for-each + (lambda (file) + (apply + function + (list + (string-append source-path "/" file) + (string-append destination-path "/" file)))) + (scandir source-path (negate (cut member <> '("." ".."))))) + (rmdir source-path)))))) + install-schemes)) + + (define pyproject-build (which "pyproject-build")) + + (define (list-directories base predicate) + ;; Cannot use find-files here, because it’s recursive. + (scandir + base + (lambda (name) + (let ((stat (lstat (string-append base "/" name)))) + (and + (not (member name '("." ".."))) + (eq? (stat:type stat) 'directory) + (predicate name stat)))))) + + (define (install-pep517) + "Install a wheel generated by a PEP 517-compatible builder." + (let ((wheels (find-files "dist" "\\.whl$"))) ; XXX: do not recurse + (when (> (length wheels) 1) ; This code does not support multiple wheels + ; yet, because their outputs would have to be + ; merged properly. + (raise (condition (&cannot-extract-multiple-wheels)))) + (for-each extract wheels)) + (let ((datadirs (map + (cut string-append site-dir "/" <>) + (list-directories site-dir (file-name-predicate "\\.data$"))))) + (for-each (lambda (directory) + (expand-data-directory directory) + (rmdir directory)) + datadirs))) + + (define (install-setuptools) + "Install using setuptools." + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" + "install" + "--prefix" out + "--single-version-externally-managed" + "--root=/"))) + + (if pyproject-build + (install-pep517) + (install-setuptools)) + #t)) + +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Compile installed byte-code in site-packages." + (let ((site-dir (site-packages inputs outputs))) + (invoke "python" "-m" "compileall" site-dir) + ;; XXX: We could compile with -O and -OO too here, at the cost of more space. + #t)) (define* (wrap #:key inputs outputs #:allow-other-keys) (define (list-of-files dir) @@ -243,29 +337,12 @@ running checks after installing the package." files))) bindirs))) -(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys) - "Rename easy-install.pth to NAME.pth to avoid conflicts between packages -installed with setuptools." - ;; Even if the "easy-install.pth" is not longer created, we kept this phase. - ;; There still may be packages creating an "easy-install.pth" manually for - ;; some good reason. - (let* ((site-packages (site-packages inputs outputs)) - (easy-install-pth (string-append site-packages "/easy-install.pth")) - (new-pth (string-append site-packages "/" name ".pth"))) - (when (file-exists? easy-install-pth) - (rename-file easy-install-pth new-pth)))) - -(define* (ensure-no-mtimes-pre-1980 #:rest _) - "Ensure that there are no mtimes before 1980-01-02 in the source tree." - ;; Rationale: patch-and-repack creates tarballs with timestamps at the POSIX - ;; epoch, 1970-01-01 UTC. This causes problems with Python packages, - ;; because Python eggs are ZIP files, and the ZIP format does not support - ;; timestamps before 1980. - (let ((early-1980 315619200)) ; 1980-01-02 UTC - (ftw "." (lambda (file stat flag) - (unless (<= early-1980 (stat:mtime stat)) - (utime file early-1980 early-1980)) - #t)))) +(define* (set-SOURCE-DATE-EPOCH #:rest _) + "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools +that incorporate timestamps as a way to tell them to use a fixed timestamp. +See https://reproducible-builds.org/specs/source-date-epoch/." + (setenv "SOURCE_DATE_EPOCH" "315619200") ;; python-wheel respects this variable and sets pre-1980 times on files in zip files, which is unsupported + #t) (define* (enable-bytecode-determinism #:rest _) "Improve determinism of pyc files." @@ -292,11 +369,11 @@ by Cython." ;; prefix directory. The check phase is moved after the installation phase ;; to ease testing the built package. (modify-phases gnu:%standard-phases - (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980) - (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism + (add-after 'unpack 'enable-bytecode-determinism enable-bytecode-determinism) (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files ensure-no-cythonized-files) + (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH) (delete 'bootstrap) (delete 'configure) ;not needed (replace 'build build) @@ -308,7 +385,7 @@ by Cython." (add-after 'add-install-to-path 'wrap wrap) (add-after 'wrap 'check check) (add-after 'check 'sanity-check sanity-check) - (add-before 'strip 'rename-pth-file rename-pth-file))) + (add-before 'check 'compile-bytecode compile-bytecode))) (define* (python-build #:key inputs (phases %standard-phases) #:allow-other-keys #:rest args) -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0006-gnu-Add-python-pytoml.patch" From 14b70dea3d21684e7fdb66dd072031cef3214b07 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:17:10 +0100 Subject: [PATCH 06/12] gnu: Add python-pytoml. * gnu/packages/python-build.scm (python-pytoml): Add new variable. --- gnu/packages/python-build.scm | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 232e24f470..a6a310177c 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -173,3 +173,20 @@ implementation developed for Poetry. This project is intended to be a light weight, fully compliant, self-contained package allowing PEP 517 compatible build front-ends to build Poetry managed projects.") (license license:expat))) + +(define-public python-pytoml + (package + (name "python-pytoml") + (version "0.1.21") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytoml" version)) + (sha256 + (base32 + "1rv1byiw82k7mj6aprcrqi2vdabs801y97xhfnrz7kxds34ggv4f")))) + (build-system python-build-system) + (home-page "https://github.com/avakar/pytoml") + (synopsis "A parser for TOML-0.4.0") + (description "A parser for TOML-0.4.0") + (license license:expat))) -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0007-gnu-Add-python-flit-core.patch" From e8b1f40157b3d884bb2fc3e3d10bbfbd469c67b9 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:18:17 +0100 Subject: [PATCH 07/12] gnu: Add python-flit-core. * gnu/packages/python-build.scm (python-flit-core): New variable. --- gnu/packages/python-build.scm | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index a6a310177c..f767704a78 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -190,3 +190,30 @@ compatible build front-ends to build Poetry managed projects.") (synopsis "A parser for TOML-0.4.0") (description "A parser for TOML-0.4.0") (license license:expat))) + +(define-public python-flit-core + (package + (name "python-flit-core") + (version "3.0.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "flit_core" version)) + (sha256 + (base32 + "0bbw84r33gwi0xyp7m8dzp2dzpjs4harj3l5wrbxkmp2awh0ard4")))) + (build-system python-build-system) + (arguments + `(;; No tests. + #:tests? #f)) + (propagated-inputs + `(("python-pytoml" ,python-pytoml))) + (home-page "https://github.com/takluyver/flit") + (synopsis + "Simplified packaging of Python modules, distribution-building parts") + (description + "Flit is a simple way to put Python packages and modules on PyPI. It +tries to require less thought about packaging and help you avoid common +mistakes. Distribution-building parts of Flit.") + (license license:bsd-3))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0008-gnu-python-pep517-bootstrap-Build-using-flit-core.patch" From 316c25f310cad4dd8f0cb73a914a4776b1b1375c Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:20:48 +0100 Subject: [PATCH 08/12] gnu: python-pep517-bootstrap: Build using flit-core. * gnu/packages/python-build.scm (python-pep517-bootstrap) [arguments]: Relax dependency on flit-core version. [native-inputs]: Add flit-core. --- gnu/packages/python-build.scm | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index f767704a78..f74a3ee49e 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -110,10 +110,21 @@ Language (TOML) configuration files.") "0zqidxah03qpnp6zkg3zd1kmd5f79hhdsfmlc0cldaniy80qddxf")))) (build-system python-build-system) (arguments - `(#:tests? #f)) ;to avoid circular dependencies + `(#:tests? #f ; To avoid circular dependencies. + #:phases + (modify-phases %standard-phases + (add-after 'unpack 'relax-dependency + (lambda _ + (substitute* "pyproject.toml" + (("flit_core >=2,<3") + "flit_core >=2,<4")) + #t))))) (propagated-inputs `(("python-toml" ,python-toml) ("python-wheel" ,python-wheel))) + (native-inputs + `(;; Build system. + ("python-flit-core" ,python-flit-core))) (home-page "https://github.com/pypa/pep517") (synopsis "Wrappers to build Python packages using PEP 517 hooks") (description -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0009-gnu-python-iniconfig-Add-missing-build-input.patch" From c8898a6a282daaa2cceabfac96e417f7b09e8d5f Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:23:53 +0100 Subject: [PATCH 09/12] gnu: python-iniconfig: Add missing build input. * gnu/packages/python-xyz.scm (python-iniconfig) [native-inputs] Add setuptools-scm. --- gnu/packages/python-xyz.scm | 1 + 1 file changed, 1 insertion(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 79d01f700a..d598a380a9 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -15704,6 +15704,7 @@ in other versions.") (base32 "0ckzngs3scaa1mcfmsi1w40a1l8cxxnncscrxzjjwjyisx8z0fmw")))) (build-system python-build-system) + (native-inputs `(("python-setuptools-scm" ,python-setuptools-scm))) (home-page "https://github.com/RonnyPfannschmidt/iniconfig") (synopsis "Simple INI-file parser") (description "The @code{iniconfig} package provides a small and simple -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0010-gnu-Add-python-u-msgpack.patch" From 8f5c9398ac1a6ac1871d5cf8a1efbe6a70ed4a1b Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:16:07 +0100 Subject: [PATCH 10/12] gnu: Add python-u-msgpack. * gnu/packages/python-xyz.scm (python-u-msgpack): New variable. The redundant -python postfix from the original package name has been removed. --- gnu/packages/python-xyz.scm | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index d598a380a9..ab3c6d301e 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -23659,3 +23659,27 @@ Application Programming Interface based on the Open Inventor 2.1 API.") Crayons automatically wraps a given string in the foreground color and restores the original state after the string is printed.") (license license:expat))) + +(define-public python-u-msgpack + (package + (name "python-u-msgpack") + (version "2.7.1") + (source + (origin + (method url-fetch) + (uri (pypi-uri "u-msgpack-python" version)) + (sha256 + (base32 + "0lcmlr7gc4dydpxn6l5bdcq40c3ghf8mv1sjqyj72wdpr8rx9rxp")))) + (build-system python-build-system) + (home-page + "https://github.com/vsergeev/u-msgpack-python") + (synopsis + "Portable, lightweight MessagePack serializer and deserializer") + (description + "A portable, lightweight MessagePack serializer and deserializer written +in pure Python. u-msgpack-python is fully compliant with the latest MessagePack +specification. In particular, it supports the new binary, UTF-8 string, +application-defined ext, and timestamp types.") + (license license:expat))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0011-gnu-Add-python-pytest-expect.patch" From 39eef77658f400ccfceb65b6fcd3f4996ae90807 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:20:03 +0100 Subject: [PATCH 11/12] gnu: Add python-pytest-expect. * gnu/packages/python-check.scm (python-pytest-expect): New variable. --- gnu/packages/python-check.scm | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/gnu/packages/python-check.scm b/gnu/packages/python-check.scm index 9849b16685..4139f3cdde 100644 --- a/gnu/packages/python-check.scm +++ b/gnu/packages/python-check.scm @@ -1251,3 +1251,31 @@ help in debugging failures and optimizing the scheduler to improve speed.") (description "A pytest plugin for Sanic. It helps you to test your code asynchronously.") (license license:expat))) + +(define-public python-pytest-expect + (package + (name "python-pytest-expect") + (version "1.1.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytest-expect" version)) + (sha256 + (base32 + "0iyq3zd1g5ffaz2wv6mskjfn84sfbyh0j209glcrh1s50hkldd1n")))) + (build-system python-build-system) + (arguments `(#:tests? #f)) ; no tests via pypi + (propagated-inputs + `(("python-pytest" ,python-pytest) ; package declares this dependency + ("python-u-msgpack" ,python-u-msgpack))) + (home-page + "https://github.com/gsnedders/pytest-expect") + (synopsis + "Py.test plugin to store test expectations and mark tests based on them") + (description + "A py.test plugin that stores test expectations by saving the set of +failing tests, allowing them to be marked as xfail when running them in future. +The tests expectations are stored such that they can be distributed alongside +the tests.") + (license license:expat))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0012-gnu-python-html5lib-Fix-tests-with-pytest-6.patch" From 4ed7eba12c80dd33f20626ddb81abc34dd667ab3 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:23:07 +0100 Subject: [PATCH 12/12] gnu: python-html5lib: Fix tests with pytest 6. * gnu/packages/python-web.scm (python-html5lib) [source]: Add upstream patch. [native-inputs]: Add test dependencies. [arguments]: Remove unsupported #:test-target. --- gnu/packages/python-web.scm | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm index 947c200253..c484d7ba36 100644 --- a/gnu/packages/python-web.scm +++ b/gnu/packages/python-web.scm @@ -1020,13 +1020,27 @@ storage.") (uri (pypi-uri "html5lib" version)) (sha256 (base32 - "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj")))) + "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj")) + (patches + (list + ;; Adds Pytest 6 support. + (origin + (method url-fetch) + (uri (string-append + "https://github.com/html5lib/" + "html5lib-python/commit/" + "2c19b9899ab3a3e8bd0ca35e5d78544334204169.patch")) + (file-name "python-html5lib-support-pytest6.patch") + (sha256 + (base32 + "0jg2ry0439q8n7j1mf4p2hdq54i704pq9scv4wwa2pp3cwvb6dvg"))))))) (build-system python-build-system) (propagated-inputs `(("python-six" ,python-six) ("python-webencodings" ,python-webencodings))) - (arguments - `(#:test-target "check")) + (native-inputs + `(("python-pytest" ,python-pytest) + ("python-pytest-expect" ,python-pytest-expect))) (home-page "https://github.com/html5lib/html5lib-python") (synopsis -- 2.26.2 --9EzuTbC0hpH3Zvh5--