[PATCH] Enable cross-compilation for the pyproject-build-system.

  • Open
  • quality assurance status badge
Details
4 participants
  • jgart
  • Ludovic Courtès
  • Christopher Baines
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
normal
M
M
Maxim Cournoyer wrote on 16 Jan 2023 05:04
[PATCH 0/5] Enable cross-compilation for the pyproject-build-system
(address . guix-patches@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
20230116040402.7350-1-maxim.cournoyer@gmail.com
This change enables cross-compilation for the pyproject-build-system, at the
same time paving the way to fix the longstanding bug #25235 (Wrapped python
programs get native-inputs in PYTHONPATH), by keeping the distinction between
native and host (regular) inputs even when not cross-compiling. A unified
builder is used, which departs from the other build systems (which use one
builder for the native build and another one for the cross build, where
implemented).

If this change is accepted, there would be adjustments needed to some of the
~150 Python packages using pyproject (looking for a native inputs now needs to
be done using the 'native-inputs' instead of 'inputs' argument even in the
native compilation mode). Over time I think it'd make sense to migrate more
build systems to use this scheme, since it gives more control as to what
inputs go into the wrapper.

A demo package, python-pycryptodome, that contains C extensions is adjusted so
that it can be cross-compiled using this new build system.


Maxim Cournoyer (5):
gnu: libtommath: Update to 1.2.0-0.03de03d.
gnu: libtomcrypt: Update to 1.18.2-0.29986d0.
build: Enable cross-compilation for pyproject-build-system.
gnu: python-pycryptodome: Fix build and enable cross-compilation.
gnu: python-coverage: Switch to pyproject-build-system.

gnu/packages/check.scm | 2 +-
gnu/packages/multiprecision.scm | 179 +++++++++-----------------
gnu/packages/python-crypto.scm | 20 ++-
guix/build-system/pyproject.scm | 115 ++++++++++-------
guix/build/pyproject-build-system.scm | 126 ++++++++++++++++--
guix/packages.scm | 46 +++----
6 files changed, 288 insertions(+), 200 deletions(-)


base-commit: 5c921977179489caef4a9e54ada6696fc86d2f0b
--
2.38.1
M
M
Maxim Cournoyer wrote on 16 Jan 2023 05:58
control message for bug #60847
(address . control@debbugs.gnu.org)
877cxncbxs.fsf@gmail.com
retitle 60847 [PATCH] Enable cross-compilation for the pyproject-build-system.
quit
M
M
Maxim Cournoyer wrote on 16 Jan 2023 06:01
[PATCH 0/1] Enable cross-compilation for the pyproject-build-system
(address . 60847@debbugs.gnu.org)
20230116050132.24313-1-maxim.cournoyer@gmail.com
This change enables cross-compilation for the pyproject-build-system, at the
same time paving the way to fix the longstanding bug #25235 (Wrapped python
programs get native-inputs in PYTHONPATH), by keeping the distinction between
native and host (regular) inputs even when not cross-compiling. A unified
builder is used, which departs from the other build systems (which use one
builder for the native build and another one for the cross build, where
implemented).

If this change is accepted, there would be adjustments needed to some of the
~120 Python packages using pyproject (looking for a native inputs now needs to
be done using the 'native-inputs' instead of 'inputs' argument even in the
native compilation mode). Over time I think it'd make sense to migrate more
build systems to use this scheme, since it gives more control as to what
inputs go into the wrapper.

A demo package, python-pycryptodome, that contains C extensions is adjusted so
that it can be cross-compiled using this new build system. It is provided
separately as it would need to go to core-updates.


Maxim Cournoyer (1):
build: Enable cross-compilation for pyproject-build-system.

guix/build-system/pyproject.scm | 115 ++++++++++++++---------
guix/build/pyproject-build-system.scm | 126 +++++++++++++++++++++++---
guix/packages.scm | 46 +++++-----
3 files changed, 210 insertions(+), 77 deletions(-)


base-commit: 5c921977179489caef4a9e54ada6696fc86d2f0b
--
2.38.1
M
M
Maxim Cournoyer wrote on 16 Jan 2023 06:01
[PATCH 1/1] build: Enable cross-compilation for pyproject-build-system.
(address . 60847@debbugs.gnu.org)
20230116050132.24313-2-maxim.cournoyer@gmail.com
* guix/build-system/pyproject.scm (lower): Do not return #f when TARGET is
set. Add the target field. Separate build-inputs from target-inputs, and
extend these with the standard-packages or standard-cross-packages procedures.
(pyproject-build): Add the target, build-inputs, target-inputs, host-inputs
and native-search-paths arguments. Remove the inputs argument. Adjust doc.
Adjust the call to the build-side pyproject-build procedure. Always pass the
target to the gexp->derivation call.
* guix/packages.scm (bag->derivation): Special case the pyproject-build
builder to always use cross-derivation bags
* guix/build/pyproject-build-system.scm (set-paths): New procedure, overriding
that provided by gnu-build-system.
(check) [TARGET]: New argument. Disable the phase when it is set.
(install): Adjust to look for Python in the native-inputs instead of inputs.
(compile-bytecode): Likewise.
(create-entrypoints): Likewise.
(set-setuptools-env): New procedure.
(sanity-check): Adjust to look for Python in the native-inputs instead of
inputs.
(add-install-to-pythonpath, wrap, rename-pth-file): Likewise.
(%standard-phases): Override set-paths, add-install-to-pythonpath, wrap,
sanity-check and rename-pth-file. Register set-setuptools-env.

---

guix/build-system/pyproject.scm | 115 ++++++++++++++---------
guix/build/pyproject-build-system.scm | 126 +++++++++++++++++++++++---
guix/packages.scm | 46 +++++-----
3 files changed, 210 insertions(+), 77 deletions(-)

Toggle diff (391 lines)
diff --git a/guix/build-system/pyproject.scm b/guix/build-system/pyproject.scm
index 8f3b562ca3..74d739023f 100644
--- a/guix/build-system/pyproject.scm
+++ b/guix/build-system/pyproject.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
+;;; Copyright © 2023 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -69,28 +70,37 @@ (define* (lower name
#:rest arguments)
"Return a bag for NAME."
(define private-keywords
- '(#:target #:python #:inputs #:native-inputs))
-
- (and (not target) ;XXX: no cross-compilation
- (bag
- (name name)
- (system system)
- (host-inputs `(,@(if source
- `(("source" ,source))
- '())
- ,@inputs
-
- ;; Keep the standard inputs of 'gnu-build-system'.
- ,@(standard-packages)))
- (build-inputs `(("python" ,python)
- ("sanity-check.py" ,(local-file sanity-check.py))
- ,@native-inputs))
- (outputs (append outputs '(wheel)))
- (build pyproject-build)
- (arguments (strip-keyword-arguments private-keywords arguments)))))
-
-(define* (pyproject-build name inputs
- #:key source
+ `(#:python #:inputs #:native-inputs
+ ,@(if target '() '(#:target))))
+
+ (bag
+ (name name)
+ (system system)
+ (target target)
+ (build-inputs `(,@(if source
+ `(("source" ,source))
+ '())
+ ("python" ,python)
+ ("sanity-check.py" ,(local-file sanity-check.py))
+ ,@native-inputs
+ ,@(if target
+ (standard-cross-packages target 'host)
+ '())
+ ;; Keep the standard inputs of 'gnu-build-system'.
+ ,@(standard-packages)))
+ (host-inputs inputs)
+ (target-inputs (if target
+ (standard-cross-packages target 'target)
+ '()))
+ (outputs (append outputs '(wheel)))
+ (build pyproject-build)
+ (arguments (strip-keyword-arguments private-keywords arguments))))
+
+(define* (pyproject-build name
+ #:key
+ target
+ build-inputs target-inputs host-inputs
+ source
(tests? #t)
(configure-flags ''())
(build-backend #f)
@@ -98,44 +108,63 @@ (define* (pyproject-build name inputs
(test-flags ''())
(phases '%standard-phases)
(outputs '("out" "wheel"))
+ (native-search-paths '())
(search-paths '())
(system (%current-system))
(guile #f)
(imported-modules %pyproject-build-system-modules)
(modules '((guix build pyproject-build-system)
(guix build utils))))
- "Build SOURCE using PYTHON, and with INPUTS."
+ "Build SOURCE using PYTHON, and with BUILD-INPUTS, HOST-INPUTS and
+TARGET-INPUTS (if available)."
(define build
(with-imported-modules imported-modules
#~(begin
(use-modules #$@(sexp->gexp modules))
- #$(with-build-variables inputs outputs
- #~(pyproject-build
- #:name #$name
- #:source #+source
- #:configure-flags #$configure-flags
- #:system #$system
- #:build-backend #$build-backend
- #:test-backend #$test-backend
- #:test-flags #$test-flags
- #:tests? #$tests?
- #:phases #$(if (pair? phases)
- (sexp->gexp phases)
- phases)
- #:outputs %outputs
- #:search-paths '#$(sexp->gexp
- (map search-path-specification->sexp
- search-paths))
- #:inputs %build-inputs)))))
+ (define %build-host-inputs
+ #+(input-tuples->gexp build-inputs))
+
+ (define %build-target-inputs
+ (append #$(input-tuples->gexp host-inputs)
+ #+(input-tuples->gexp target-inputs)))
+
+ (define %build-inputs
+ (append %build-host-inputs %build-target-inputs))
+
+ (define %outputs
+ #$(outputs->gexp outputs))
+ (pyproject-build #:name #$name
+ #:source #+source
+ #:configure-flags #$configure-flags
+ #:system #$system
+ #:target #$target
+ #:build-backend #$build-backend
+ #:test-backend #$test-backend
+ #:test-flags #$test-flags
+ #:tests? #$tests?
+ #:phases #$(if (pair? phases)
+ (sexp->gexp phases)
+ phases)
+ #:outputs %outputs
+ #:search-paths
+ '#$(sexp->gexp
+ (map search-path-specification->sexp
+ search-paths))
+ #:native-search-paths
+ '#$(sexp->gexp
+ (map search-path-specification->sexp
+ native-search-paths))
+ #:native-inputs %build-host-inputs
+ #:inputs %build-target-inputs))))
(mlet %store-monad ((guile (package->derivation (or guile (default-guile))
system #:graft? #f)))
(gexp->derivation name build
#:system system
- #:graft? #f ;consistent with 'gnu-build'
- #:target #f
+ #:target target
+ #:graft? #f ;consistent with 'gnu-build'
#:guile-for-build guile)))
(define pyproject-build-system
diff --git a/guix/build/pyproject-build-system.scm b/guix/build/pyproject-build-system.scm
index c69ccc9d64..e51b5cfc43 100644
--- a/guix/build/pyproject-build-system.scm
+++ b/guix/build/pyproject-build-system.scm
@@ -86,6 +86,58 @@ (define-condition-type &cannot-extract-multiple-wheels &python-build-error
;; Raised, when no wheel has been built by the build system.
(define-condition-type &no-wheels-built &python-build-error no-wheels-built?)
+;;; XXX: This is the same as in (guix build gnu-build-system), except adjusted
+;;; for the fact that native-inputs always exist now, whether cross-compiling
+;;; or not. When not cross-compiling, input-directories are appended to
+;;; native-input-directories to so that native-search-paths are computed for
+;;; all inputs.
+(define* (set-paths #:key target inputs native-inputs
+ search-paths native-search-paths
+ #:allow-other-keys)
+ (define input-directories
+ ;; The "source" input can be a directory, but we don't want it for search
+ ;; paths. See <https://issues.guix.gnu.org/44924>.
+ (match (alist-delete "source" inputs)
+ (((_ . dir) ...)
+ dir)))
+
+ (define native-input-directories
+ (match (alist-delete "source" native-inputs)
+ (((_ . dir) ...)
+ dir)))
+
+ ;; Tell 'ld-wrapper' to disallow non-store libraries.
+ (setenv "GUIX_LD_WRAPPER_ALLOW_IMPURITIES" "no")
+
+ ;; When cross building, $PATH must refer only to native (host) inputs since
+ ;; target inputs are not executable.
+ (set-path-environment-variable "PATH" '("bin" "sbin")
+ (append native-input-directories
+ (if target
+ '()
+ input-directories)))
+
+ (for-each (match-lambda
+ ((env-var (files ...) separator type pattern)
+ (set-path-environment-variable env-var files
+ input-directories
+ #:separator separator
+ #:type type
+ #:pattern pattern)))
+ search-paths)
+
+ (for-each (match-lambda
+ ((env-var (files ...) separator type pattern)
+ (set-path-environment-variable env-var files
+ (append native-input-directories
+ (if target
+ '()
+ input-directories))
+ #:separator separator
+ #:type type
+ #:pattern pattern)))
+ native-search-paths))
+
(define* (build #:key outputs build-backend configure-flags #:allow-other-keys)
"Build a given Python package."
@@ -135,9 +187,9 @@ (define (pyproject.toml->build-backend file)
wheel-dir
config-settings)))
-(define* (check #:key tests? test-backend test-flags #:allow-other-keys)
+(define* (check #:key target tests? test-backend test-flags #:allow-other-keys)
"Run the test suite of a given Python package."
- (if tests?
+ (if (and tests? (not target))
;; Unfortunately with PEP 517 there is no common method to specify test
;; systems. Guess test system based on inputs instead.
(let* ((pytest (which "pytest"))
@@ -172,11 +224,11 @@ (define* (check #:key tests? test-backend test-flags #:allow-other-keys)
(else (raise (condition (&test-system-not-found))))))
(format #t "test suite not run~%")))
-(define* (install #:key inputs outputs #:allow-other-keys)
+(define* (install #:key native-inputs outputs #:allow-other-keys)
"Install a wheel file according to PEP 427"
;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl
- (let ((site-dir (site-packages inputs outputs))
- (python (assoc-ref inputs "python"))
+ (let ((site-dir (site-packages native-inputs outputs))
+ (python (assoc-ref native-inputs "python"))
(out (assoc-ref outputs "out")))
(define (extract file)
"Extract wheel (ZIP file) into site-packages directory"
@@ -262,10 +314,10 @@ (define (list-directories base predicate)
(expand-data-directory directory)
(rmdir directory)) datadirs))))
-(define* (compile-bytecode #:key inputs outputs #:allow-other-keys)
+(define* (compile-bytecode #:key native-inputs outputs #:allow-other-keys)
"Compile installed byte-code in site-packages."
- (let* ((site-dir (site-packages inputs outputs))
- (python (assoc-ref inputs "python"))
+ (let* ((site-dir (site-packages native-inputs outputs))
+ (python (assoc-ref native-inputs "python"))
(major-minor (map string->number
(take (string-split (python-version python) #\.) 2)))
(<3.7? (match major-minor
@@ -281,7 +333,7 @@ (define* (compile-bytecode #:key inputs outputs #:allow-other-keys)
(invoke "python" "-m" "compileall"
"--invalidation-mode=unchecked-hash" site-dir))))
-(define* (create-entrypoints #:key inputs outputs #:allow-other-keys)
+(define* (create-entrypoints #:key native-inputs outputs #:allow-other-keys)
"Implement Entry Points Specification
(https://packaging.python.org/specifications/entry-points/) by PyPa,
which creates runnable scripts in bin/ from entry point specification
@@ -337,8 +389,7 @@ (define (create-script path name module function)
import ~a as mod
sys.exit (mod.~a ())~%" interpreter module function)))
(chmod file-path #o755)))
-
- (let* ((site-dir (site-packages inputs outputs))
+ (let* ((site-dir (site-packages native-inputs outputs))
(out (assoc-ref outputs "out"))
(bin-dir (string-append out "/bin"))
(entry-point-files (find-files site-dir "^entry_points.txt$")))
@@ -358,6 +409,51 @@ (define* (set-SOURCE-DATE-EPOCH* #:rest _)
;; not support timestamps before 1980.
(setenv "SOURCE_DATE_EPOCH" "315619200"))
+(define* (set-setuptools-env #:key target #:allow-other-keys)
+ "Set environment variables such as LDSHARED, LDXXSHARED, etc. used by
+setuptools when building native extensions. This is particularly useful for
+cross-compilation."
+ (define cc-for-target (if target
+ (string-append target "-gcc")
+ "gcc"))
+ (define cxx-for-target (if target
+ (string-append target "-g++")
+ "g++"))
+ ;; The variables defined here are taken from CPython's configure.ac file.
+ ;; The explanations are those of the Poky (from Yocto) project.
+ (setenv "CC" cc-for-target)
+ (setenv "CXX" cxx-for-target)
+ ;; LDSHARED is the ld *command* used to create shared library.
+ (setenv "LDSHARED" (string-append cc-for-target " -shared"))
+ ;; LDXXSHARED is the ld *command* used to create shared library of C++
+ ;; objects.
+ (setenv "LDCXXSHARED" (string-append cxx-for-target " -shared"))
+ ;; CCSHARED are the C *flags* used to create objects to go into a shared
+ ;; library (module).
+ (setenv "CCSHARED" "-fPIC")
+ ;; LINKFORSHARED are the flags passed to the $(CC) command that links the
+ ;; python executable.
+ (setenv "LINKFORSHARED" "-Xlinker -export-dynamic"))
+
+(define* (sanity-check #:key tests? native-inputs outputs #:allow-other-keys
+ #:rest args)
+ (apply (assoc-ref python:%standard-phases 'sanity-check)
+ (append args (list #:inputs native-inputs))))
+
+(define* (add-install-to-pythonpath #:key native-inputs outputs
+ #:allow-other-keys)
+ "A phase that just wraps the 'add-installed-pythonpath' procedure."
+ (add-installed-pythonpath native-inputs outputs))
+
+(define* (wrap #:key native-inputs outputs #:allow-other-keys #:rest args)
+ (apply (assoc-ref python:%standard-phases 'wrap)
+ (append args (list #:inputs native-inputs))))
+
+(define* (rename-pth-file #:key name native-inputs outputs #:allow-other-keys
+ #:rest args)
+ (apply (assoc-ref python:%standard-phases 'rename-pth-file)
+ (append args (list #:inputs native-inputs))))
+
(define %standard-phases
;; The build phase only builds C extensions and copies the Python sources,
;; while the install phase copies then byte-compiles the sources to the
@@ -365,13 +461,19 @@ (define %standard-phases
;; to ease testing the built package.
(modify-phases python:%standard-phases
(replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH*)
+ (replace 'set-paths set-paths)
+ (add-after 'set-paths 'set-setuptools-env set-setuptools-env)
(replace 'build build)
(replace 'install install)
+ (replace 'add-install-to-pythonpath add-install-to-pythonpath)
+ (replace 'wrap wrap)
(delete 'check)
;; Must be before tests, so they can use installed packages’ entry points.
(add-before 'wrap 'create-entrypoints create-entrypoints)
(add-after 'wrap 'check check)
- (add-before 'check 'compile-bytecode compile-bytecode)))
+ (add-before 'check 'compile-bytecode compile-bytecode)
+ (replace 'sanity-check sanity-check)
+ (replace 'rename-pth-file rename-pth-file)))
(define* (pyproject-build #:key inputs (phases %standard-phases)
#:allow-other-keys #:rest args)
diff --git a/guix/packages.scm b/guix/packages.scm
index 041a872f9d..6d7df17fc3 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)
"Return the derivation to build BAG for SYSTEM. Optionally, CONTEXT can be
a package object describing the context in which the call occurs, for improved
error reporting."
- (if (bag-target bag)
- (bag->cross-derivation bag)
- (mlet* %store-monad ((system -> (bag-system bag))
- (inputs -> (bag-transitive-inputs bag))
- (input-drvs (mapm %store-monad
- (cut expand-input context <> system)
- inputs))
- (paths -> (delete-duplicates
- (append-map (match-lambda
- ((_ (? package? p) _ ...)
- (package-native-search-paths
- p))
- (_ '()))
- inputs))))
- ;; It's possible that INPUTS contains packages that are not 'eq?' but
- ;; that lead to the same derivation. Delete those duplicates to avoid
- ;; issues down the road, such as duplicate entries in '%build-inputs'.
- (apply (bag-build bag) (bag-name bag)
- (delete-duplicates input-drvs input=?)
- #:search-paths paths
- #:outputs (bag-outputs bag) #:system system
- (bag-arguments bag)))))
+ (let ((builder-name (procedure-name (bag-build bag))))
+ (if (or (bag-target bag)
+ (eq? 'pyproject-build builder-name))
+ (bag->cross-derivation bag)
+ (mlet* %store-monad ((system -> (bag-system bag))
+ (inputs -> (bag-transitive-inputs bag))
+ (input-drvs (mapm %store-monad
+
This message was truncated. Download the full message here.
M
M
Maxim Cournoyer wrote on 16 Jan 2023 06:16
control message for bug #60849
(address . control@debbugs.gnu.org)
875yd7cb48.fsf@gmail.com
block 60849 by 60847
quit
M
M
Maxim Cournoyer wrote on 17 Jan 2023 16:49
control message for bug #25235
(address . control@debbugs.gnu.org)
875yd52mb7.fsf@gmail.com
block 25235 by 60847
quit
M
M
Maxim Cournoyer wrote on 23 Jan 2023 14:32
[PATCH v2 0/1] Enable cross-compilation for the pyproject-build-system
(address . 60847@debbugs.gnu.org)
20230123133217.318-1-maxim.cournoyer@gmail.com
This change enables cross-compilation for the pyproject-build-system, at the
same time paving the way to fix the longstanding bug #25235 (Wrapped python
programs get native-inputs in PYTHONPATH), by keeping the distinction between
native and host (regular) inputs even when not cross-compiling. A unified
builder is used, which departs from the other build systems (which use one
builder for the native build and another one for the cross build, where
implemented).

If this change is accepted, there would be adjustments needed to some of the
~120 Python packages using pyproject (looking for a native inputs now needs to
be done using the 'native-inputs' instead of 'inputs' argument even in the
native compilation mode). Over time I think it'd make sense to migrate more
build systems to use this scheme, since it gives more control as to what
inputs go into the wrapper.

A demo package, python-pycryptodome, that contains C extensions is adjusted so
that it can be cross-compiled using this new build system. It is provided
separately as it would need to go to core-updates.

Changes in v2:
- Rebase

Maxim Cournoyer (1):
build: Enable cross-compilation for pyproject-build-system.

guix/build-system/pyproject.scm | 115 ++++++++++++++---------
guix/build/pyproject-build-system.scm | 126 +++++++++++++++++++++++---
guix/packages.scm | 46 +++++-----
3 files changed, 210 insertions(+), 77 deletions(-)


base-commit: 3a1b18aa4540e6f96ded0a98dd907a8033262582
--
2.39.1
M
M
Maxim Cournoyer wrote on 23 Jan 2023 14:32
[PATCH v2 1/1] build: Enable cross-compilation for pyproject-build-system.
(address . 60847@debbugs.gnu.org)
20230123133217.318-2-maxim.cournoyer@gmail.com
* guix/build-system/pyproject.scm (lower): Do not return #f when TARGET is
set. Add the target field. Separate build-inputs from target-inputs, and
extend these with the standard-packages or standard-cross-packages procedures.
(pyproject-build): Add the target, build-inputs, target-inputs, host-inputs
and native-search-paths arguments. Remove the inputs argument. Adjust doc.
Adjust the call to the build-side pyproject-build procedure. Always pass the
target to the gexp->derivation call.
* guix/packages.scm (bag->derivation): Special case the pyproject-build
builder to always use cross-derivation bags
* guix/build/pyproject-build-system.scm (set-paths): New procedure, overriding
that provided by gnu-build-system.
(check) [TARGET]: New argument. Disable the phase when it is set.
(install): Adjust to look for Python in the native-inputs instead of inputs.
(compile-bytecode): Likewise.
(create-entrypoints): Likewise.
(set-setuptools-env): New procedure.
(sanity-check): Adjust to look for Python in the native-inputs instead of
inputs.
(add-install-to-pythonpath, wrap, rename-pth-file): Likewise.
(%standard-phases): Override set-paths, add-install-to-pythonpath, wrap,
sanity-check and rename-pth-file. Register set-setuptools-env.

---

Changes in v2:
- Rebase

guix/build-system/pyproject.scm | 115 ++++++++++++++---------
guix/build/pyproject-build-system.scm | 126 +++++++++++++++++++++++---
guix/packages.scm | 46 +++++-----
3 files changed, 210 insertions(+), 77 deletions(-)

Toggle diff (391 lines)
diff --git a/guix/build-system/pyproject.scm b/guix/build-system/pyproject.scm
index 8f3b562ca3..74d739023f 100644
--- a/guix/build-system/pyproject.scm
+++ b/guix/build-system/pyproject.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
;;; Copyright © 2022 Marius Bakke <marius@gnu.org>
+;;; Copyright © 2023 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -69,28 +70,37 @@ (define* (lower name
#:rest arguments)
"Return a bag for NAME."
(define private-keywords
- '(#:target #:python #:inputs #:native-inputs))
-
- (and (not target) ;XXX: no cross-compilation
- (bag
- (name name)
- (system system)
- (host-inputs `(,@(if source
- `(("source" ,source))
- '())
- ,@inputs
-
- ;; Keep the standard inputs of 'gnu-build-system'.
- ,@(standard-packages)))
- (build-inputs `(("python" ,python)
- ("sanity-check.py" ,(local-file sanity-check.py))
- ,@native-inputs))
- (outputs (append outputs '(wheel)))
- (build pyproject-build)
- (arguments (strip-keyword-arguments private-keywords arguments)))))
-
-(define* (pyproject-build name inputs
- #:key source
+ `(#:python #:inputs #:native-inputs
+ ,@(if target '() '(#:target))))
+
+ (bag
+ (name name)
+ (system system)
+ (target target)
+ (build-inputs `(,@(if source
+ `(("source" ,source))
+ '())
+ ("python" ,python)
+ ("sanity-check.py" ,(local-file sanity-check.py))
+ ,@native-inputs
+ ,@(if target
+ (standard-cross-packages target 'host)
+ '())
+ ;; Keep the standard inputs of 'gnu-build-system'.
+ ,@(standard-packages)))
+ (host-inputs inputs)
+ (target-inputs (if target
+ (standard-cross-packages target 'target)
+ '()))
+ (outputs (append outputs '(wheel)))
+ (build pyproject-build)
+ (arguments (strip-keyword-arguments private-keywords arguments))))
+
+(define* (pyproject-build name
+ #:key
+ target
+ build-inputs target-inputs host-inputs
+ source
(tests? #t)
(configure-flags ''())
(build-backend #f)
@@ -98,44 +108,63 @@ (define* (pyproject-build name inputs
(test-flags ''())
(phases '%standard-phases)
(outputs '("out" "wheel"))
+ (native-search-paths '())
(search-paths '())
(system (%current-system))
(guile #f)
(imported-modules %pyproject-build-system-modules)
(modules '((guix build pyproject-build-system)
(guix build utils))))
- "Build SOURCE using PYTHON, and with INPUTS."
+ "Build SOURCE using PYTHON, and with BUILD-INPUTS, HOST-INPUTS and
+TARGET-INPUTS (if available)."
(define build
(with-imported-modules imported-modules
#~(begin
(use-modules #$@(sexp->gexp modules))
- #$(with-build-variables inputs outputs
- #~(pyproject-build
- #:name #$name
- #:source #+source
- #:configure-flags #$configure-flags
- #:system #$system
- #:build-backend #$build-backend
- #:test-backend #$test-backend
- #:test-flags #$test-flags
- #:tests? #$tests?
- #:phases #$(if (pair? phases)
- (sexp->gexp phases)
- phases)
- #:outputs %outputs
- #:search-paths '#$(sexp->gexp
- (map search-path-specification->sexp
- search-paths))
- #:inputs %build-inputs)))))
+ (define %build-host-inputs
+ #+(input-tuples->gexp build-inputs))
+
+ (define %build-target-inputs
+ (append #$(input-tuples->gexp host-inputs)
+ #+(input-tuples->gexp target-inputs)))
+
+ (define %build-inputs
+ (append %build-host-inputs %build-target-inputs))
+
+ (define %outputs
+ #$(outputs->gexp outputs))
+ (pyproject-build #:name #$name
+ #:source #+source
+ #:configure-flags #$configure-flags
+ #:system #$system
+ #:target #$target
+ #:build-backend #$build-backend
+ #:test-backend #$test-backend
+ #:test-flags #$test-flags
+ #:tests? #$tests?
+ #:phases #$(if (pair? phases)
+ (sexp->gexp phases)
+ phases)
+ #:outputs %outputs
+ #:search-paths
+ '#$(sexp->gexp
+ (map search-path-specification->sexp
+ search-paths))
+ #:native-search-paths
+ '#$(sexp->gexp
+ (map search-path-specification->sexp
+ native-search-paths))
+ #:native-inputs %build-host-inputs
+ #:inputs %build-target-inputs))))
(mlet %store-monad ((guile (package->derivation (or guile (default-guile))
system #:graft? #f)))
(gexp->derivation name build
#:system system
- #:graft? #f ;consistent with 'gnu-build'
- #:target #f
+ #:target target
+ #:graft? #f ;consistent with 'gnu-build'
#:guile-for-build guile)))
(define pyproject-build-system
diff --git a/guix/build/pyproject-build-system.scm b/guix/build/pyproject-build-system.scm
index c69ccc9d64..e51b5cfc43 100644
--- a/guix/build/pyproject-build-system.scm
+++ b/guix/build/pyproject-build-system.scm
@@ -86,6 +86,58 @@ (define-condition-type &cannot-extract-multiple-wheels &python-build-error
;; Raised, when no wheel has been built by the build system.
(define-condition-type &no-wheels-built &python-build-error no-wheels-built?)
+;;; XXX: This is the same as in (guix build gnu-build-system), except adjusted
+;;; for the fact that native-inputs always exist now, whether cross-compiling
+;;; or not. When not cross-compiling, input-directories are appended to
+;;; native-input-directories to so that native-search-paths are computed for
+;;; all inputs.
+(define* (set-paths #:key target inputs native-inputs
+ search-paths native-search-paths
+ #:allow-other-keys)
+ (define input-directories
+ ;; The "source" input can be a directory, but we don't want it for search
+ ;; paths. See <https://issues.guix.gnu.org/44924>.
+ (match (alist-delete "source" inputs)
+ (((_ . dir) ...)
+ dir)))
+
+ (define native-input-directories
+ (match (alist-delete "source" native-inputs)
+ (((_ . dir) ...)
+ dir)))
+
+ ;; Tell 'ld-wrapper' to disallow non-store libraries.
+ (setenv "GUIX_LD_WRAPPER_ALLOW_IMPURITIES" "no")
+
+ ;; When cross building, $PATH must refer only to native (host) inputs since
+ ;; target inputs are not executable.
+ (set-path-environment-variable "PATH" '("bin" "sbin")
+ (append native-input-directories
+ (if target
+ '()
+ input-directories)))
+
+ (for-each (match-lambda
+ ((env-var (files ...) separator type pattern)
+ (set-path-environment-variable env-var files
+ input-directories
+ #:separator separator
+ #:type type
+ #:pattern pattern)))
+ search-paths)
+
+ (for-each (match-lambda
+ ((env-var (files ...) separator type pattern)
+ (set-path-environment-variable env-var files
+ (append native-input-directories
+ (if target
+ '()
+ input-directories))
+ #:separator separator
+ #:type type
+ #:pattern pattern)))
+ native-search-paths))
+
(define* (build #:key outputs build-backend configure-flags #:allow-other-keys)
"Build a given Python package."
@@ -135,9 +187,9 @@ (define (pyproject.toml->build-backend file)
wheel-dir
config-settings)))
-(define* (check #:key tests? test-backend test-flags #:allow-other-keys)
+(define* (check #:key target tests? test-backend test-flags #:allow-other-keys)
"Run the test suite of a given Python package."
- (if tests?
+ (if (and tests? (not target))
;; Unfortunately with PEP 517 there is no common method to specify test
;; systems. Guess test system based on inputs instead.
(let* ((pytest (which "pytest"))
@@ -172,11 +224,11 @@ (define* (check #:key tests? test-backend test-flags #:allow-other-keys)
(else (raise (condition (&test-system-not-found))))))
(format #t "test suite not run~%")))
-(define* (install #:key inputs outputs #:allow-other-keys)
+(define* (install #:key native-inputs outputs #:allow-other-keys)
"Install a wheel file according to PEP 427"
;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl
- (let ((site-dir (site-packages inputs outputs))
- (python (assoc-ref inputs "python"))
+ (let ((site-dir (site-packages native-inputs outputs))
+ (python (assoc-ref native-inputs "python"))
(out (assoc-ref outputs "out")))
(define (extract file)
"Extract wheel (ZIP file) into site-packages directory"
@@ -262,10 +314,10 @@ (define (list-directories base predicate)
(expand-data-directory directory)
(rmdir directory)) datadirs))))
-(define* (compile-bytecode #:key inputs outputs #:allow-other-keys)
+(define* (compile-bytecode #:key native-inputs outputs #:allow-other-keys)
"Compile installed byte-code in site-packages."
- (let* ((site-dir (site-packages inputs outputs))
- (python (assoc-ref inputs "python"))
+ (let* ((site-dir (site-packages native-inputs outputs))
+ (python (assoc-ref native-inputs "python"))
(major-minor (map string->number
(take (string-split (python-version python) #\.) 2)))
(<3.7? (match major-minor
@@ -281,7 +333,7 @@ (define* (compile-bytecode #:key inputs outputs #:allow-other-keys)
(invoke "python" "-m" "compileall"
"--invalidation-mode=unchecked-hash" site-dir))))
-(define* (create-entrypoints #:key inputs outputs #:allow-other-keys)
+(define* (create-entrypoints #:key native-inputs outputs #:allow-other-keys)
"Implement Entry Points Specification
(https://packaging.python.org/specifications/entry-points/) by PyPa,
which creates runnable scripts in bin/ from entry point specification
@@ -337,8 +389,7 @@ (define (create-script path name module function)
import ~a as mod
sys.exit (mod.~a ())~%" interpreter module function)))
(chmod file-path #o755)))
-
- (let* ((site-dir (site-packages inputs outputs))
+ (let* ((site-dir (site-packages native-inputs outputs))
(out (assoc-ref outputs "out"))
(bin-dir (string-append out "/bin"))
(entry-point-files (find-files site-dir "^entry_points.txt$")))
@@ -358,6 +409,51 @@ (define* (set-SOURCE-DATE-EPOCH* #:rest _)
;; not support timestamps before 1980.
(setenv "SOURCE_DATE_EPOCH" "315619200"))
+(define* (set-setuptools-env #:key target #:allow-other-keys)
+ "Set environment variables such as LDSHARED, LDXXSHARED, etc. used by
+setuptools when building native extensions. This is particularly useful for
+cross-compilation."
+ (define cc-for-target (if target
+ (string-append target "-gcc")
+ "gcc"))
+ (define cxx-for-target (if target
+ (string-append target "-g++")
+ "g++"))
+ ;; The variables defined here are taken from CPython's configure.ac file.
+ ;; The explanations are those of the Poky (from Yocto) project.
+ (setenv "CC" cc-for-target)
+ (setenv "CXX" cxx-for-target)
+ ;; LDSHARED is the ld *command* used to create shared library.
+ (setenv "LDSHARED" (string-append cc-for-target " -shared"))
+ ;; LDXXSHARED is the ld *command* used to create shared library of C++
+ ;; objects.
+ (setenv "LDCXXSHARED" (string-append cxx-for-target " -shared"))
+ ;; CCSHARED are the C *flags* used to create objects to go into a shared
+ ;; library (module).
+ (setenv "CCSHARED" "-fPIC")
+ ;; LINKFORSHARED are the flags passed to the $(CC) command that links the
+ ;; python executable.
+ (setenv "LINKFORSHARED" "-Xlinker -export-dynamic"))
+
+(define* (sanity-check #:key tests? native-inputs outputs #:allow-other-keys
+ #:rest args)
+ (apply (assoc-ref python:%standard-phases 'sanity-check)
+ (append args (list #:inputs native-inputs))))
+
+(define* (add-install-to-pythonpath #:key native-inputs outputs
+ #:allow-other-keys)
+ "A phase that just wraps the 'add-installed-pythonpath' procedure."
+ (add-installed-pythonpath native-inputs outputs))
+
+(define* (wrap #:key native-inputs outputs #:allow-other-keys #:rest args)
+ (apply (assoc-ref python:%standard-phases 'wrap)
+ (append args (list #:inputs native-inputs))))
+
+(define* (rename-pth-file #:key name native-inputs outputs #:allow-other-keys
+ #:rest args)
+ (apply (assoc-ref python:%standard-phases 'rename-pth-file)
+ (append args (list #:inputs native-inputs))))
+
(define %standard-phases
;; The build phase only builds C extensions and copies the Python sources,
;; while the install phase copies then byte-compiles the sources to the
@@ -365,13 +461,19 @@ (define %standard-phases
;; to ease testing the built package.
(modify-phases python:%standard-phases
(replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH*)
+ (replace 'set-paths set-paths)
+ (add-after 'set-paths 'set-setuptools-env set-setuptools-env)
(replace 'build build)
(replace 'install install)
+ (replace 'add-install-to-pythonpath add-install-to-pythonpath)
+ (replace 'wrap wrap)
(delete 'check)
;; Must be before tests, so they can use installed packages’ entry points.
(add-before 'wrap 'create-entrypoints create-entrypoints)
(add-after 'wrap 'check check)
- (add-before 'check 'compile-bytecode compile-bytecode)))
+ (add-before 'check 'compile-bytecode compile-bytecode)
+ (replace 'sanity-check sanity-check)
+ (replace 'rename-pth-file rename-pth-file)))
(define* (pyproject-build #:key inputs (phases %standard-phases)
#:allow-other-keys #:rest args)
diff --git a/guix/packages.scm b/guix/packages.scm
index 041a872f9d..6d7df17fc3 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)
"Return the derivation to build BAG for SYSTEM. Optionally, CONTEXT can be
a package object describing the context in which the call occurs, for improved
error reporting."
- (if (bag-target bag)
- (bag->cross-derivation bag)
- (mlet* %store-monad ((system -> (bag-system bag))
- (inputs -> (bag-transitive-inputs bag))
- (input-drvs (mapm %store-monad
- (cut expand-input context <> system)
- inputs))
- (paths -> (delete-duplicates
- (append-map (match-lambda
- ((_ (? package? p) _ ...)
- (package-native-search-paths
- p))
- (_ '()))
- inputs))))
- ;; It's possible that INPUTS contains packages that are not 'eq?' but
- ;; that lead to the same derivation. Delete those duplicates to avoid
- ;; issues down the road, such as duplicate entries in '%build-inputs'.
- (apply (bag-build bag) (bag-name bag)
- (delete-duplicates input-drvs input=?)
- #:search-paths paths
- #:outputs (bag-outputs bag) #:system system
- (bag-arguments bag)))))
+ (let ((builder-name (procedure-name (bag-build bag))))
+ (if (or (bag-target bag)
+ (eq? 'pyproject-build builder-name))
+ (bag->cross-derivation bag)
+ (mlet* %store-monad ((system -> (bag-system bag))
+ (inputs -> (bag-transitive-inputs bag))
+ (input-drvs (mapm %store-monad
+
This message was truncated. Download the full message here.
J
065abd9e926ff892b97b45662685943e@dismail.de
Hi Maxim,

Sorry, This week is still too busy for me with work to review this. Looking forward to reviewing future patches or these if still needed.

all best,

jgart
L
L
Ludovic Courtès wrote on 6 Mar 2023 18:04
Re: bug#60847: [PATCH] Enable cross-compilation for the pyproject-build-system.
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87zg8p7qvw.fsf_-_@gnu.org
Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (3 lines)
> +++ b/guix/packages.scm
> @@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)

[…]

Toggle quote (5 lines)
> + (let ((builder-name (procedure-name (bag-build bag))))
> + (if (or (bag-target bag)
> + (eq? 'pyproject-build builder-name))
> + (bag->cross-derivation bag)

This one part is a showstopper to me, for two reasons:

1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
not guaranteed to return something useful).

2. Special-casing build systems here is not okay: the bag and build
system abstractions exist to maintain separation of concerns.

I understand there’s an actual bug to fix and the desire to fix a more
common issue, but I think this one approach is not the way forward.

I hope that makes sense!

Ludo’.
J
J
jgart wrote on 6 Mar 2023 23:56
6c904311d43b36f146475d8722295270@dismail.de
Hi Ludo,

Toggle quote (2 lines)
> I understand there’s an actual bug to fix ...

Is the bug you are referencing exist as an open ticket somewhere on our tracker?

Just trying to get more context on that if it does.

all best,

jgart
J
J
jgart wrote on 7 Mar 2023 01:25
69a3a526d296323461478f150bab5325@dismail.de
Toggle quote (2 lines)
> Is the bug you are referencing exist as an open ticket somewhere on our tracker?

Oooops, Ignore, found it: https://issues.guix.gnu.org/issue/25235

Reread Maxim's original message now ;()
M
M
Maxim Cournoyer wrote on 7 Mar 2023 15:08
(name . Ludovic Courtès)(address . ludo@gnu.org)
87jzzsy7qt.fsf@gmail.com
Hi Ludo!

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (27 lines)
> Hello,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> +++ b/guix/packages.scm
>> @@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)
>
> […]
>
>> + (let ((builder-name (procedure-name (bag-build bag))))
>> + (if (or (bag-target bag)
>> + (eq? 'pyproject-build builder-name))
>> + (bag->cross-derivation bag)
>
> This one part is a showstopper to me, for two reasons:
>
> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
> not guaranteed to return something useful).
>
> 2. Special-casing build systems here is not okay: the bag and build
> system abstractions exist to maintain separation of concerns.
>
> I understand there’s an actual bug to fix and the desire to fix a more
> common issue, but I think this one approach is not the way forward.
>
> I hope that makes sense!

I agree this is not "pretty", but it would be a "temporary" kludge until
all the build systems can be migrated (and the package adjusted for) the
"new" way, which is: native-inputs and inputs always co-exist, whether
the build is a native one or a cross one.

In light of this, it seems OK to test the water with a not so
significant build system (only a handful of package relies on
pyproject-build-system thus far). When all the build systems will have
been migrated to the new way (a too big undertaking to be done in one
shot), this kludge can be removed.

Otherwise, could you offer a concrete suggestion as the way forward? I
appreciate the "that's not the way", but stopping short of suggesting a
better alternative leaves me wanting more :-).

--
Thanks,
Maxim
C
C
Christopher Baines wrote on 7 Mar 2023 16:05
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 60847@debbugs.gnu.org)
87jzzsr3tx.fsf@cbaines.net
Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (46 lines)
> Hi Ludo!
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello,
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>
>>> +++ b/guix/packages.scm
>>> @@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)
>>
>> […]
>>
>>> + (let ((builder-name (procedure-name (bag-build bag))))
>>> + (if (or (bag-target bag)
>>> + (eq? 'pyproject-build builder-name))
>>> + (bag->cross-derivation bag)
>>
>> This one part is a showstopper to me, for two reasons:
>>
>> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
>> not guaranteed to return something useful).
>>
>> 2. Special-casing build systems here is not okay: the bag and build
>> system abstractions exist to maintain separation of concerns.
>>
>> I understand there’s an actual bug to fix and the desire to fix a more
>> common issue, but I think this one approach is not the way forward.
>>
>> I hope that makes sense!
>
> I agree this is not "pretty", but it would be a "temporary" kludge until
> all the build systems can be migrated (and the package adjusted for) the
> "new" way, which is: native-inputs and inputs always co-exist, whether
> the build is a native one or a cross one.
>
> In light of this, it seems OK to test the water with a not so
> significant build system (only a handful of package relies on
> pyproject-build-system thus far). When all the build systems will have
> been migrated to the new way (a too big undertaking to be done in one
> shot), this kludge can be removed.
>
> Otherwise, could you offer a concrete suggestion as the way forward? I
> appreciate the "that's not the way", but stopping short of suggesting a
> better alternative leaves me wanting more :-).

I think it would be clearer to see other potential ways forward if the
end goal was more clearly articulated.

Guessing at that here from the changes proposed to guix/packages.scm,
once all build systems are adjusted, cross derivations will be produced
for all bags, regardless of whether there's a target.

That doesn't make much sense to me. One explaination is that the current
naming is confusing when thinking about this goal, so maybe
bag->cross-derivation happens to do what you want it to do in all
circumstances, even when target is #f?
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmQHVNpfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xd0JhAAuSbus21DgEoJu/nQUiNKRyTenkCjOrlz
Ea6YVd2RXZbtwZJI5dTGHqTa4S50959uDYx+TSAejB3hUMoj5cMm9gdp1/JQ6MdG
FWndCf5SI6C53L2GwN9g6Cfx9gfeMLcvlu5+CHR8V/GRxtMRBLuqwVRgB5xCm1Uk
n91eKUjv7GNmzLBH5KW7WLxIvG3a0tzegMdHFnrHKTWA5VuRkCv/ebDULjDmUcz1
SnF/j7SWMAJZgbv5EMqG+7+PRhkpas9j42ETLSw0uCbv+0xRguJ514V0sh4UbsD3
i1ZuOeEzFJqoncx1fiWyvGXy3FyL1h2zyeB9mKU1M2Los9OJvQ+8M1Wrh4L/IDK4
RGDRWLTkAzxkztXtQBM053wzCVBAqCpZ62afbvoPsiyCVkvO1qNn8cV54l01bVPQ
pK6RPLsQXTPete9f2P1qxAm7i7ynqcq8/+Naz8vxwDf24A1U88PKgyyGq9MqtMZe
X9qGF0CdsbBnu8+L24gaNLF0mpEZLyxqgv8SM1wR5jnG0x+jGyht5OiL7JKFrAmX
SFC9alYBxAOvJTh/hLcS2DMiQuI6dih3CUVpGPn4Aj1WSQtmnM4opaqevlfREu05
v/l0ptwdR/Gdvv2/3nC/ADGGZeOqXvuRSyrz0isYoyaGI9aoUJmfRs5nz2kPNf0h
1/c33pzron4=
=TM8f
-----END PGP SIGNATURE-----

M
M
Maxim Cournoyer wrote on 7 Mar 2023 20:03
(name . Christopher Baines)(address . mail@cbaines.net)(address . 60847@debbugs.gnu.org)
87a60owfi8.fsf@gmail.com
Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

Toggle quote (60 lines)
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Hi Ludo!
>>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Hello,
>>>
>>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>>
>>>> +++ b/guix/packages.scm
>>>> @@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)
>>>
>>> […]
>>>
>>>> + (let ((builder-name (procedure-name (bag-build bag))))
>>>> + (if (or (bag-target bag)
>>>> + (eq? 'pyproject-build builder-name))
>>>> + (bag->cross-derivation bag)
>>>
>>> This one part is a showstopper to me, for two reasons:
>>>
>>> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
>>> not guaranteed to return something useful).
>>>
>>> 2. Special-casing build systems here is not okay: the bag and build
>>> system abstractions exist to maintain separation of concerns.
>>>
>>> I understand there’s an actual bug to fix and the desire to fix a more
>>> common issue, but I think this one approach is not the way forward.
>>>
>>> I hope that makes sense!
>>
>> I agree this is not "pretty", but it would be a "temporary" kludge until
>> all the build systems can be migrated (and the package adjusted for) the
>> "new" way, which is: native-inputs and inputs always co-exist, whether
>> the build is a native one or a cross one.
>>
>> In light of this, it seems OK to test the water with a not so
>> significant build system (only a handful of package relies on
>> pyproject-build-system thus far). When all the build systems will have
>> been migrated to the new way (a too big undertaking to be done in one
>> shot), this kludge can be removed.
>>
>> Otherwise, could you offer a concrete suggestion as the way forward? I
>> appreciate the "that's not the way", but stopping short of suggesting a
>> better alternative leaves me wanting more :-).
>
> I think it would be clearer to see other potential ways forward if the
> end goal was more clearly articulated.
>
> Guessing at that here from the changes proposed to guix/packages.scm,
> once all build systems are adjusted, cross derivations will be produced
> for all bags, regardless of whether there's a target.
>
> That doesn't make much sense to me. One explaination is that the current
> naming is confusing when thinking about this goal, so maybe
> bag->cross-derivation happens to do what you want it to do in all
> circumstances, even when target is #f?

Thanks for tipping in. The end goal is to avoid loosing the information
of which inputs are native (build inputs) vs regular in the bag, and yes
bag->cross-derivation allows that. It appears to me the distinction in
the bag representations (native vs cross) was originally perceived
useful as some kind of optimization (there's less variables to worry
about, and we can squash the inputs/search-paths together, since they're
all native anyway), but this information (currently discarded) ends up
being very useful even on the build side (to wrap only the target
inputs, say, and not all the native/build inputs).

So yes, the change long term would be to integrate the
bag->cross-derivation logic into bag->derivation, at which point it
would be unified for any type of build (the bag representation would be
shared between native and cross builds).

When authoring packages, we'd no longer see the odd idiom '(or
native-inputs inputs)' which results from this implementation detail;
instead we'd always have access to both 'native-inputs' and 'inputs'.

I hope that helps understanding the rationale.

--
Thanks,
Maxim
C
C
Christopher Baines wrote on 7 Mar 2023 20:26
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 60847@debbugs.gnu.org)
87fsagqr67.fsf@cbaines.net
Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (79 lines)
> Hi Christopher,
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>
>>> Hi Ludo!
>>>
>>> Ludovic Courtès <ludo@gnu.org> writes:
>>>
>>>> Hello,
>>>>
>>>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>>>
>>>>> +++ b/guix/packages.scm
>>>>> @@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)
>>>>
>>>> […]
>>>>
>>>>> + (let ((builder-name (procedure-name (bag-build bag))))
>>>>> + (if (or (bag-target bag)
>>>>> + (eq? 'pyproject-build builder-name))
>>>>> + (bag->cross-derivation bag)
>>>>
>>>> This one part is a showstopper to me, for two reasons:
>>>>
>>>> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
>>>> not guaranteed to return something useful).
>>>>
>>>> 2. Special-casing build systems here is not okay: the bag and build
>>>> system abstractions exist to maintain separation of concerns.
>>>>
>>>> I understand there’s an actual bug to fix and the desire to fix a more
>>>> common issue, but I think this one approach is not the way forward.
>>>>
>>>> I hope that makes sense!
>>>
>>> I agree this is not "pretty", but it would be a "temporary" kludge until
>>> all the build systems can be migrated (and the package adjusted for) the
>>> "new" way, which is: native-inputs and inputs always co-exist, whether
>>> the build is a native one or a cross one.
>>>
>>> In light of this, it seems OK to test the water with a not so
>>> significant build system (only a handful of package relies on
>>> pyproject-build-system thus far). When all the build systems will have
>>> been migrated to the new way (a too big undertaking to be done in one
>>> shot), this kludge can be removed.
>>>
>>> Otherwise, could you offer a concrete suggestion as the way forward? I
>>> appreciate the "that's not the way", but stopping short of suggesting a
>>> better alternative leaves me wanting more :-).
>>
>> I think it would be clearer to see other potential ways forward if the
>> end goal was more clearly articulated.
>>
>> Guessing at that here from the changes proposed to guix/packages.scm,
>> once all build systems are adjusted, cross derivations will be produced
>> for all bags, regardless of whether there's a target.
>>
>> That doesn't make much sense to me. One explaination is that the current
>> naming is confusing when thinking about this goal, so maybe
>> bag->cross-derivation happens to do what you want it to do in all
>> circumstances, even when target is #f?
>
> Thanks for tipping in. The end goal is to avoid loosing the information
> of which inputs are native (build inputs) vs regular in the bag, and yes
> bag->cross-derivation allows that. It appears to me the distinction in
> the bag representations (native vs cross) was originally perceived
> useful as some kind of optimization (there's less variables to worry
> about, and we can squash the inputs/search-paths together, since they're
> all native anyway), but this information (currently discarded) ends up
> being very useful even on the build side (to wrap only the target
> inputs, say, and not all the native/build inputs).
>
> So yes, the change long term would be to integrate the
> bag->cross-derivation logic into bag->derivation, at which point it
> would be unified for any type of build (the bag representation would be
> shared between native and cross builds).

Thanks for the explanation, so maybe an alternative to trying to get
bag->derivation to function differently for different build systems
would be to push combining the inputs down in to each build system.

Take the gnu-build-system as an example, gnu-build in (guix build-system
gnu) would be changed to take multiple lists of inputs, rather than a
single list. It can then combine the lists of inputs as is done in
bag->derivation, to avoid affecting any packages.

While this does require changing all the build systems, I think it's a
bit more forward thinking compared to trying to add a kludge in to
bag->derivation, since hopefully the change there can be the longer term
one.

Does that make sense?
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmQHlPFfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XcLrBAAoFT5BfSDlpaQqOx1NctpOIqTrX48SG0l
RS1P8YKtucY1yqYWDS9U9IB640fhfjLKP0rYtXcucFHGkMvqOgVnakKf2Dl9KDwu
j51oNMSzMkw44l4yOvh0hc7L10Crb77NymS/n+JXQnnRDbqDnX+X6IEkZjvzsSaW
+aDUw3b+Zf75IKcrrCnoTJRW8cF01M80lMC1oDR0zJg21ctmSU9NlVO1WRIik2wg
vCw7Z8Y6SfivqDXhK8OfogKVFqHUbveRHZJ1BQO115/YrUqN4gO0oCX9XUPVWyJ1
0r0IqIsPaaziw4TwmWS9kmXfJnqKwwYmSwVbXPRpGVbKm08uEoGftXI2IcFKKXK8
9Lwk79lxYG3mzsOMN49r4Zh7HrY0p3aFbf9IojxZWS1SMf9Ffw3xC4jyTN49lAdD
r8rNFjXpkaIUN9hYfiWWW6/s4wlrMIi1FPnGCgZVh4erHNfLNpJi5cr2NuNnIV3A
jDt/lQR4wowbeKrh2Fm0WfJj/nEw3wEUOvacwj0sRALtXNrg2xUs2LjluWPtvjNX
OYMQ1TMpsWy2N2nUJhXvB9Mgwj0c1fSvXtxflY2RLo8XF0VqHv/tQcmE4tdlwFof
06NbGJVVOuv9PYtbaQtxCK2cZGDI5rDW+cnJr2Tu257ItcUndcbJZhZgkcJeOAbb
F2kWX7NBdgE=
=N7ZY
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 10 Mar 2023 09:57
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87o7p19e6w.fsf@gnu.org
Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (31 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hello,
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>
>>> +++ b/guix/packages.scm
>>> @@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)
>>
>> […]
>>
>>> + (let ((builder-name (procedure-name (bag-build bag))))
>>> + (if (or (bag-target bag)
>>> + (eq? 'pyproject-build builder-name))
>>> + (bag->cross-derivation bag)
>>
>> This one part is a showstopper to me, for two reasons:
>>
>> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
>> not guaranteed to return something useful).
>>
>> 2. Special-casing build systems here is not okay: the bag and build
>> system abstractions exist to maintain separation of concerns.
>>
>> I understand there’s an actual bug to fix and the desire to fix a more
>> common issue, but I think this one approach is not the way forward.
>>
>> I hope that makes sense!
>
> I agree this is not "pretty", but it would be a "temporary" kludge until

To make sure there’s no misunderstanding, I think that’s not okay even
as a “temporary kludge” (I’m not against temporary kludges in general,
I’ve done my share of that ;-), but this would be way too brittle.)

Toggle quote (4 lines)
> all the build systems can be migrated (and the package adjusted for) the
> "new" way, which is: native-inputs and inputs always co-exist, whether
> the build is a native one or a cross one.

Again, I’m not sure about the “new way”. I’m sorry I lack the bandwidth
to review this quickly, especially a wide-ranging change like
inputs/native-inputs.

(Just yesterday I was fixing cross-compilation issues on ‘core-updates’;
that gives a lot of insight as to how native-inputs/inputs play out in
practice.)

[...]

Toggle quote (4 lines)
> Otherwise, could you offer a concrete suggestion as the way forward? I
> appreciate the "that's not the way", but stopping short of suggesting a
> better alternative leaves me wanting more :-).

Yup, I’m sorry I don’t have any suggestions! Perhaps one suggestion
would be: start the native-inputs/inputs change on a new branch, for all
the build systems (or at least the main ones), and then try to build the
‘core’ subset (which includes cross-compilation) to get an idea of the
kind of code simplification opportunities as well as issues that arise.
I feel like it’s hard to anticipate the impact of such a change without
actually trying.

Thanks,
Ludo’.
M
M
Maxim Cournoyer wrote on 10 Mar 2023 15:13
(name . Christopher Baines)(address . mail@cbaines.net)(address . 60847@debbugs.gnu.org)
87zg8kr8xz.fsf@gmail.com
Hello Christopher,

Christopher Baines <mail@cbaines.net> writes:

[...]

Toggle quote (29 lines)
>> Thanks for tipping in. The end goal is to avoid loosing the information
>> of which inputs are native (build inputs) vs regular in the bag, and yes
>> bag->cross-derivation allows that. It appears to me the distinction in
>> the bag representations (native vs cross) was originally perceived
>> useful as some kind of optimization (there's less variables to worry
>> about, and we can squash the inputs/search-paths together, since they're
>> all native anyway), but this information (currently discarded) ends up
>> being very useful even on the build side (to wrap only the target
>> inputs, say, and not all the native/build inputs).
>>
>> So yes, the change long term would be to integrate the
>> bag->cross-derivation logic into bag->derivation, at which point it
>> would be unified for any type of build (the bag representation would be
>> shared between native and cross builds).
>
> Thanks for the explanation, so maybe an alternative to trying to get
> bag->derivation to function differently for different build systems
> would be to push combining the inputs down in to each build system.
>
> Take the gnu-build-system as an example, gnu-build in (guix build-system
> gnu) would be changed to take multiple lists of inputs, rather than a
> single list. It can then combine the lists of inputs as is done in
> bag->derivation, to avoid affecting any packages.
>
> While this does require changing all the build systems, I think it's a
> bit more forward thinking compared to trying to add a kludge in to
> bag->derivation, since hopefully the change there can be the longer term
> one.

I'll see if I have a good enough understanding of the code to make sense
of your suggestion. I'll definitely try it!

Thanks for suggesting.

--
Thanks,
Maxim
M
M
Maxim Cournoyer wrote on 10 Mar 2023 15:21
(name . Ludovic Courtès)(address . ludo@gnu.org)
87v8j8r8jt.fsf@gmail.com
Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (39 lines)
> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Hello,
>>>
>>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>>
>>>> +++ b/guix/packages.scm
>>>> @@ -1864,28 +1864,30 @@ (define* (bag->derivation bag #:optional context)
>>>
>>> […]
>>>
>>>> + (let ((builder-name (procedure-name (bag-build bag))))
>>>> + (if (or (bag-target bag)
>>>> + (eq? 'pyproject-build builder-name))
>>>> + (bag->cross-derivation bag)
>>>
>>> This one part is a showstopper to me, for two reasons:
>>>
>>> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
>>> not guaranteed to return something useful).
>>>
>>> 2. Special-casing build systems here is not okay: the bag and build
>>> system abstractions exist to maintain separation of concerns.
>>>
>>> I understand there’s an actual bug to fix and the desire to fix a more
>>> common issue, but I think this one approach is not the way forward.
>>>
>>> I hope that makes sense!
>>
>> I agree this is not "pretty", but it would be a "temporary" kludge until
>
> To make sure there’s no misunderstanding, I think that’s not okay even
> as a “temporary kludge” (I’m not against temporary kludges in general,
> I’ve done my share of that ;-), but this would be way too brittle.)

"way too brittle" is a strong warning :-). I don't understand what is
so brittle about it? It's not like we change the build system names
often.

Toggle quote (20 lines)
>> all the build systems can be migrated (and the package adjusted for) the
>> "new" way, which is: native-inputs and inputs always co-exist, whether
>> the build is a native one or a cross one.
>
> Again, I’m not sure about the “new way”. I’m sorry I lack the bandwidth
> to review this quickly, especially a wide-ranging change like
> inputs/native-inputs.
>
> (Just yesterday I was fixing cross-compilation issues on ‘core-updates’;
> that gives a lot of insight as to how native-inputs/inputs play out in
> practice.)
>
> [...]
>
>> Otherwise, could you offer a concrete suggestion as the way forward? I
>> appreciate the "that's not the way", but stopping short of suggesting a
>> better alternative leaves me wanting more :-).
>
> Yup, I’m sorry I don’t have any suggestions!

Thanks, it's useful to state that up front. Otherwise the other side of
the line goes like "... so?" :-).

Toggle quote (7 lines)
> Perhaps one suggestion would be: start the native-inputs/inputs change
> on a new branch, for all the build systems (or at least the main
> ones), and then try to build the ‘core’ subset (which includes
> cross-compilation) to get an idea of the kind of code simplification
> opportunities as well as issues that arise. I feel like it’s hard to
> anticipate the impact of such a change without actually trying.

I can do this; the exercise would be similar to what I've done for
pyproject-build-system, where I was able to build complex packages (both
natively and cross-compiled) with the new same-bag representation
change, but to a larger scale. It's more work, which is why I would
have preferred to contain its scope to get started, but the changes were
easy, IIRC.

Basically, it forces us to review all the (assoc-ref inputs
"package-name") or (search-input-file inputs "file") procedure calls and
remove the assumption that native-inputs are also inputs for native
builds (that'll fix multiple cross-compilation bugs at the same time, so
it's a good thing to do anyway).

To be continued!

--
Thanks,
Maxim
L
L
Ludovic Courtès wrote on 10 Mar 2023 18:00
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87a60kh77y.fsf@gnu.org
Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

[...]

Toggle quote (26 lines)
>>>>> + (let ((builder-name (procedure-name (bag-build bag))))
>>>>> + (if (or (bag-target bag)
>>>>> + (eq? 'pyproject-build builder-name))
>>>>> + (bag->cross-derivation bag)
>>>>
>>>> This one part is a showstopper to me, for two reasons:
>>>>
>>>> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
>>>> not guaranteed to return something useful).
>>>>
>>>> 2. Special-casing build systems here is not okay: the bag and build
>>>> system abstractions exist to maintain separation of concerns.
>>>>
>>>> I understand there’s an actual bug to fix and the desire to fix a more
>>>> common issue, but I think this one approach is not the way forward.
>>>>
>>>> I hope that makes sense!
>>>
>>> I agree this is not "pretty", but it would be a "temporary" kludge until
>>
>> To make sure there’s no misunderstanding, I think that’s not okay even
>> as a “temporary kludge” (I’m not against temporary kludges in general,
>> I’ve done my share of that ;-), but this would be way too brittle.)
>
> "way too brittle" is a strong warning :-).

It is; I’m objecting to this change.

Toggle quote (4 lines)
> I don't understand what is
> so brittle about it? It's not like we change the build system names
> often.

I have little to add to what I wrote above: procedure names don’t mean
anything, especially with higher-order functions, and separation of
concerns is an important design principle.

I’m speaking with 15+ years of experience with Guile; I don’t mind being
challenged, it’s healthy, but I think we also need to recognize the
expertise of each other and encounter each other halfway.

[...]

Toggle quote (14 lines)
>> Perhaps one suggestion would be: start the native-inputs/inputs change
>> on a new branch, for all the build systems (or at least the main
>> ones), and then try to build the ‘core’ subset (which includes
>> cross-compilation) to get an idea of the kind of code simplification
>> opportunities as well as issues that arise. I feel like it’s hard to
>> anticipate the impact of such a change without actually trying.
>
> I can do this; the exercise would be similar to what I've done for
> pyproject-build-system, where I was able to build complex packages (both
> natively and cross-compiled) with the new same-bag representation
> change, but to a larger scale. It's more work, which is why I would
> have preferred to contain its scope to get started, but the changes were
> easy, IIRC.

Yes, I agree it’s more work, but it’s dissimilar in that you’ll be
confronted with tons of weird cases, some of which we can already
anticipate, but probably others we’ll probably be surprised to discover.

Toggle quote (6 lines)
> Basically, it forces us to review all the (assoc-ref inputs
> "package-name") or (search-input-file inputs "file") procedure calls and
> remove the assumption that native-inputs are also inputs for native
> builds (that'll fix multiple cross-compilation bugs at the same time, so
> it's a good thing to do anyway).

I do hope it’ll be this simple, but I just can’t tell.

Thank you,
Ludo’.
M
M
Maxim Cournoyer wrote on 12 Mar 2023 05:05
(name . Ludovic Courtès)(address . ludo@gnu.org)
87pm9epqbo.fsf@gmail.com
Hi,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (34 lines)
> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
> [...]
>
>>>>>> + (let ((builder-name (procedure-name (bag-build bag))))
>>>>>> + (if (or (bag-target bag)
>>>>>> + (eq? 'pyproject-build builder-name))
>>>>>> + (bag->cross-derivation bag)
>>>>>
>>>>> This one part is a showstopper to me, for two reasons:
>>>>>
>>>>> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
>>>>> not guaranteed to return something useful).
>>>>>
>>>>> 2. Special-casing build systems here is not okay: the bag and build
>>>>> system abstractions exist to maintain separation of concerns.
>>>>>
>>>>> I understand there’s an actual bug to fix and the desire to fix a more
>>>>> common issue, but I think this one approach is not the way forward.
>>>>>
>>>>> I hope that makes sense!
>>>>
>>>> I agree this is not "pretty", but it would be a "temporary" kludge until
>>>
>>> To make sure there’s no misunderstanding, I think that’s not okay even
>>> as a “temporary kludge” (I’m not against temporary kludges in general,
>>> I’ve done my share of that ;-), but this would be way too brittle.)
>>
>> "way too brittle" is a strong warning :-).
>
> It is; I’m objecting to this change.

OK.

Toggle quote (12 lines)
>> I don't understand what is
>> so brittle about it? It's not like we change the build system names
>> often.
>
> I have little to add to what I wrote above: procedure names don’t mean
> anything, especially with higher-order functions, and separation of
> concerns is an important design principle.
>
> I’m speaking with 15+ years of experience with Guile; I don’t mind being
> challenged, it’s healthy, but I think we also need to recognize the
> expertise of each other and encounter each other halfway.

When I don't understand something, I ask for an explanation. I'm happy
(and greatly value) that my correspondent has 15 years of experience
with Guile, but I'll still ask if something is unclear. I hope you
don't mind.

Toggle quote (20 lines)
> [...]
>
>>> Perhaps one suggestion would be: start the native-inputs/inputs change
>>> on a new branch, for all the build systems (or at least the main
>>> ones), and then try to build the ‘core’ subset (which includes
>>> cross-compilation) to get an idea of the kind of code simplification
>>> opportunities as well as issues that arise. I feel like it’s hard to
>>> anticipate the impact of such a change without actually trying.
>>
>> I can do this; the exercise would be similar to what I've done for
>> pyproject-build-system, where I was able to build complex packages (both
>> natively and cross-compiled) with the new same-bag representation
>> change, but to a larger scale. It's more work, which is why I would
>> have preferred to contain its scope to get started, but the changes were
>> easy, IIRC.
>
> Yes, I agree it’s more work, but it’s dissimilar in that you’ll be
> confronted with tons of weird cases, some of which we can already
> anticipate, but probably others we’ll probably be surprised to discover.

OK. I'll put this to the test, when I'm done with the current Ruby
rabbit-hole I've been pursuing. Thanks for the feedback!

--
Thanks,
Maxim
M
M
Maxim Cournoyer wrote on 12 Mar 2023 05:05
(name . Ludovic Courtès)(address . ludo@gnu.org)
87o7oypqb9.fsf@gmail.com
Hi,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (34 lines)
> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
> [...]
>
>>>>>> + (let ((builder-name (procedure-name (bag-build bag))))
>>>>>> + (if (or (bag-target bag)
>>>>>> + (eq? 'pyproject-build builder-name))
>>>>>> + (bag->cross-derivation bag)
>>>>>
>>>>> This one part is a showstopper to me, for two reasons:
>>>>>
>>>>> 1. We cannot rely on ‘procedure-name’ (it’s a debugging aid and it’s
>>>>> not guaranteed to return something useful).
>>>>>
>>>>> 2. Special-casing build systems here is not okay: the bag and build
>>>>> system abstractions exist to maintain separation of concerns.
>>>>>
>>>>> I understand there’s an actual bug to fix and the desire to fix a more
>>>>> common issue, but I think this one approach is not the way forward.
>>>>>
>>>>> I hope that makes sense!
>>>>
>>>> I agree this is not "pretty", but it would be a "temporary" kludge until
>>>
>>> To make sure there’s no misunderstanding, I think that’s not okay even
>>> as a “temporary kludge” (I’m not against temporary kludges in general,
>>> I’ve done my share of that ;-), but this would be way too brittle.)
>>
>> "way too brittle" is a strong warning :-).
>
> It is; I’m objecting to this change.

OK.

Toggle quote (12 lines)
>> I don't understand what is
>> so brittle about it? It's not like we change the build system names
>> often.
>
> I have little to add to what I wrote above: procedure names don’t mean
> anything, especially with higher-order functions, and separation of
> concerns is an important design principle.
>
> I’m speaking with 15+ years of experience with Guile; I don’t mind being
> challenged, it’s healthy, but I think we also need to recognize the
> expertise of each other and encounter each other halfway.

When I don't understand something, I ask for an explanation. I'm happy
(and greatly value) that my correspondent has 15 years of experience
with Guile, but I'll still ask if something is unclear. I hope you
don't mind.

Toggle quote (20 lines)
> [...]
>
>>> Perhaps one suggestion would be: start the native-inputs/inputs change
>>> on a new branch, for all the build systems (or at least the main
>>> ones), and then try to build the ‘core’ subset (which includes
>>> cross-compilation) to get an idea of the kind of code simplification
>>> opportunities as well as issues that arise. I feel like it’s hard to
>>> anticipate the impact of such a change without actually trying.
>>
>> I can do this; the exercise would be similar to what I've done for
>> pyproject-build-system, where I was able to build complex packages (both
>> natively and cross-compiled) with the new same-bag representation
>> change, but to a larger scale. It's more work, which is why I would
>> have preferred to contain its scope to get started, but the changes were
>> easy, IIRC.
>
> Yes, I agree it’s more work, but it’s dissimilar in that you’ll be
> confronted with tons of weird cases, some of which we can already
> anticipate, but probably others we’ll probably be surprised to discover.

OK. I'll put this to the test, when I'm done with the current Ruby
rabbit-hole I've been pursuing. Thanks for the feedback!

--
Thanks,
Maxim
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 60847@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 60847
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch