[PATCH 0/3] Generalized wrap phase for perl, python.

  • Done
  • quality assurance status badge
Details
6 participants
  • Alex Sassmannshausen
  • Alex Sassmannshausen
  • Arun Isaac
  • Brice Waegeneire
  • Ludovic Courtès
  • Marius Bakke
Owner
unassigned
Submitted by
Alex Sassmannshausen
Severity
normal
A
A
Alex Sassmannshausen wrote on 20 May 2017 11:37
(address . bug-guix@gnu.org)(name . Alex Sassmannshausen)(address . alex@pompo.co)
20170520093742.2115-1-alex@pompo.co
Hello,

This patch series goes back some time, but the need for it has not
disappeared.

It implements a general `wrap` phase that can be used in individual build
systems to make sure binaries in those languages have access to all their
propagated inputs at runtime, by setting an appropriate environment variable.

I tested it against perl and python packages.

It will need to go in core-updates due to the number of packages that will
need to be rebuilt.

Best wishes,

Alex

Alex Sassmannshausen (3):
build/utils: Add 'program-wrapper'.
build/perl-build-system: Add 'wrap' phase.
build/python-build-system: Refactor 'wrap'.

guix/build/perl-build-system.scm | 15 +++++++++++++--
guix/build/python-build-system.scm | 36 ++++++++----------------------------
guix/build/utils.scm | 36 ++++++++++++++++++++++++++++++++++++
3 files changed, 57 insertions(+), 30 deletions(-)

--
2.12.2
A
A
Alex Sassmannshausen wrote on 20 May 2017 11:40
[PATCH 1/3] build/utils: Add 'program-wrapper'.
(address . 27003@debbugs.gnu.org)(name . Alex Sassmannshausen)(address . alex@pompo.co)
20170520094030.2471-1-alex@pompo.co
* guix/build/utils.scm (program-wrapper): New procedure.
---
guix/build/utils.scm | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)

Toggle diff (56 lines)
diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index e8efb0653..af5583651 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -78,6 +78,7 @@
patch-/usr/bin/file
fold-port-matches
remove-store-references
+ program-wrapper
wrap-program
locale-category->string))
@@ -956,6 +957,41 @@ known as `nuke-refs' in Nixpkgs."
(put-u8 out (char->integer char))
result))))))
+(define (program-wrapper path-proc env-var)
+ "Return a procedure, which, invoked as part of a 'wrap' phase, is capable of
+wrapping executables inside an environment in which ENV-VAR is correctly set.
+
+The string ENV-VAR is the name of the environmental variable we are setting
+for the executable we are wrapping. PATH-PROC is a procedure of 2 arguments,
+'inputs' and 'outputs', returning the value that we should set ENV-VAR to.
+
+This is a specialized version of 'wrap-program' below, intended specifically
+to grant all executables that are part of our output access to all libraries
+that were declared in our inputs. This is of use for languages such as Perl,
+Python and Guile."
+ (define (list-of-files dir)
+ (map (cut string-append dir "/" <>)
+ (or (scandir dir (lambda (f)
+ (let ((s (stat (string-append dir "/" f))))
+ (eq? 'regular (stat:type s)))))
+ '())))
+ (lambda* (#:key inputs outputs #:allow-other-keys)
+ (define bindirs
+ (append-map (match-lambda
+ ((_ . dir)
+ (list (string-append dir "/bin")
+ (string-append dir "/sbin"))))
+ outputs))
+ (define vars
+ `(,env-var prefix ,(cons (path-proc inputs outputs)
+ (search-path-as-string->list
+ (or (getenv env-var) "")))))
+ (for-each (lambda (dir)
+ (let ((files (list-of-files dir)))
+ (for-each (cut wrap-program <> vars)
+ files)))
+ bindirs)))
+
(define* (wrap-program prog #:rest vars)
"Make a wrapper for PROG. VARS should look like this:
--
2.12.2
A
A
Alex Sassmannshausen wrote on 20 May 2017 11:40
[PATCH 2/3] build/perl-build-system: Add 'wrap' phase.
(address . 27003@debbugs.gnu.org)(name . Alex Sassmannshausen)(address . alex@pompo.co)
20170520094030.2471-2-alex@pompo.co
* guix/build/perl-build-system.scm (wrap): New procedure.
(%standard-phases): Add 'wrap' phase.
---
guix/build/perl-build-system.scm | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)

Toggle diff (44 lines)
diff --git a/guix/build/perl-build-system.scm b/guix/build/perl-build-system.scm
index b2024e440..5bf5b2523 100644
--- a/guix/build/perl-build-system.scm
+++ b/guix/build/perl-build-system.scm
@@ -19,7 +19,7 @@
(define-module (guix build perl-build-system)
#:use-module ((guix build gnu-build-system) #:prefix gnu:)
#:use-module (guix build utils)
- #:use-module (ice-9 match)
+ #:use-module (srfi srfi-1)
#:export (%standard-phases
perl-build))
@@ -51,6 +51,15 @@
(format #t "running `perl' with arguments ~s~%" args)
(zero? (apply system* "perl" args))))
+;; Use 'program-wrapper' to return an executable wrapper for perl.
+(define wrap
+ (program-wrapper
+ (lambda (inputs outputs)
+ (string-append (assoc-ref outputs "out") "/lib/perl5/site_perl/"
+ ;; As in python, assume version at end of `perl' string.
+ (last (string-split (assoc-ref inputs "perl") #\-))))
+ "PERL5LIB"))
+
(define-syntax-rule (define-w/gnu-fallback* (name args ...) body ...)
(define* (name args ... #:rest rest)
(if (access? "Build" X_OK)
@@ -74,9 +83,11 @@
(define %standard-phases
;; Everything is as with the GNU Build System except for the `configure',
- ;; `build', `check', and `install' phases.
+ ;; `build', `check', and `install' phases. We also add a `wrap' phase to
+ ;; wrap perl binaries with a complete PERL5LIB path.
(modify-phases gnu:%standard-phases
(replace 'install install)
+ (add-after 'install 'wrap wrap)
(replace 'check check)
(replace 'build build)
(replace 'configure configure)))
--
2.12.2
A
A
Alex Sassmannshausen wrote on 20 May 2017 11:40
[PATCH 3/3] build/python-build-system: Refactor 'wrap'.
(address . 27003@debbugs.gnu.org)(name . Alex Sassmannshausen)(address . alex@pompo.co)
20170520094030.2471-3-alex@pompo.co
* guix/build/python-build-system.scm (wrap): Use
'wrap-language-program'.
---
guix/build/python-build-system.scm | 36 ++++++++----------------------------
1 file changed, 8 insertions(+), 28 deletions(-)

Toggle diff (49 lines)
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index dd07986b9..30d01f8cb 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -184,34 +184,14 @@ when running checks after installing the package."
configure-flags)))
(call-setuppy "install" params use-setuptools?)))
-(define* (wrap #:key inputs outputs #:allow-other-keys)
- (define (list-of-files dir)
- (map (cut string-append dir "/" <>)
- (or (scandir dir (lambda (f)
- (let ((s (stat (string-append dir "/" f))))
- (eq? 'regular (stat:type s)))))
- '())))
-
- (define bindirs
- (append-map (match-lambda
- ((_ . dir)
- (list (string-append dir "/bin")
- (string-append dir "/sbin"))))
- outputs))
-
- (let* ((out (assoc-ref outputs "out"))
- (python (assoc-ref inputs "python"))
- (var `("PYTHONPATH" prefix
- ,(cons (string-append out "/lib/python"
- (get-python-version python)
- "/site-packages")
- (search-path-as-string->list
- (or (getenv "PYTHONPATH") ""))))))
- (for-each (lambda (dir)
- (let ((files (list-of-files dir)))
- (for-each (cut wrap-program <> var)
- files)))
- bindirs)))
+;; Use 'program-wrapper' to return an executable wrapper for python.
+(define wrap
+ (program-wrapper
+ (lambda (inputs outputs)
+ (string-append (assoc-ref outputs "out") "/lib/python"
+ (get-python-version (assoc-ref inputs "python"))
+ "/site-packages"))
+ "PYTHONPATH"))
(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys)
"Rename easy-install.pth to NAME.pth to avoid conflicts between packages
--
2.12.2
M
M
Marius Bakke wrote on 20 May 2017 22:23
Re: bug#27003: [PATCH 0/3] Generalized wrap phase for perl, python.
(name . Arun Isaac)(address . arunisaac@systemreboot.net)
87pof35lc6.fsf@fastmail.com
Alex Sassmannshausen <alex.sassmannshausen@gmail.com> writes:

Toggle quote (9 lines)
> Hello,
>
> This patch series goes back some time, but the need for it has not
> disappeared.
>
> It implements a general `wrap` phase that can be used in individual build
> systems to make sure binaries in those languages have access to all their
> propagated inputs at runtime, by setting an appropriate environment variable.

Cool, thanks a lot for working on this. CCing Arun who was working on
something similar.

AFAICT this is also subject to https://bugs.gnu.org/25235. Can you
have a look at it?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlkgpbkACgkQoqBt8qM6
VPob1ggAqdMQUyi8DyP1wrgs5r5PpFBgO7Lp5hIsSxuzhrO/h8D9Aw5r5twfxmSn
KUJ9hbEMSP0MQROy5yYj+v2ZVJ5W1n1Q9emAvmoNs5HJvC9cz03ygoXbz/AWlGJr
t6iJfW2p8hUuQ6KF7WnYSpPN5aILwLPLAygdy2FLLT6sRByuUgGQpATS6wKPTTqg
porcDqHReYbRmVVfKdnS3/BSVFQ/GM8tnYTw+g6N0J/RWtNcPn/36yTQZt4l7e66
045dqu7BMHsQsgDAGU1jkI2RhIiKjH5m2gb3O8rldZdyAu3AyknSfuGJKt/4/47E
9CE2qzTZExUv3DsW3oHENukOVIvRMw==
=9coi
-----END PGP SIGNATURE-----

A
A
Arun Isaac wrote on 27 May 2017 21:04
(name . Marius Bakke)(address . mbakke@fastmail.com)
2703649c.AEUAKjf8hTsAAAAAAAAAAAPFd4YAAAACwQwAAAAAAAW9WABZKc3G@mailjet.com
Toggle quote (6 lines)
>> It implements a general `wrap` phase that can be used in individual build
>> systems to make sure binaries in those languages have access to all their
>> propagated inputs at runtime, by setting an appropriate environment variable.
>
> CCing Arun who was working on something similar.

I haven't actually worked on this problem for a long while. I'll test
Alex's patches, and get back to you soon.

Toggle quote (3 lines)
> AFAICT this is also subject to https://bugs.gnu.org/25235. Can you
> have a look at it?

Yes, that's the python wrap phase bug report. And this is the one for
A
A
Arun Isaac wrote on 28 May 2017 21:08
(name . Marius Bakke)(address . mbakke@fastmail.com)
873a2319.AEEALFrChFQAAAAAAAAAAAPFd4YAAAACwQwAAAAAAAW9WABZKyA9@mailjet.com
Building on my system is taking a long time. I'll need some more time to
properly test these patches. But, if I understand correctly, these
patches still wrap binaries with paths of native-inputs. That problem
remains to be solved.
A
A
Alex Sassmannshausen wrote on 29 May 2017 15:45
(name . Arun Isaac)(address . arunisaac@systemreboot.net)
87h90322vn.fsf@pompo.co
Hi Arun,

Thank you for testing my patches.

Arun Isaac writes:

Toggle quote (5 lines)
> Building on my system is taking a long time. I'll need some more time to
> properly test these patches. But, if I understand correctly, these
> patches still wrap binaries with paths of native-inputs. That problem
> remains to be solved.

Yes, I certainly did nothing to resolve that particular issue, so indeed
it remains to be solved.

I cannot currently make guarantees on when I'll be able to wrap my head
around the entire problem, so if you come to conclusions for next steps
on the basis of your testing, we should go with those.

Else I will revisit this issue when I have time (which might be a
while).

Best wishes,

Alex
A
A
Arun Isaac wrote on 30 May 2017 15:45
(address . alex@pompo.co)
414919d5.AEMAK2iO_jcAAAAAAAAAAAPFd4YAAAACwQwAAAAAAAW9WABZLXea@mailjet.com
Toggle quote (7 lines)
> I cannot currently make guarantees on when I'll be able to wrap my head
> around the entire problem, so if you come to conclusions for next steps
> on the basis of your testing, we should go with those.
>
> Else I will revisit this issue when I have time (which might be a
> while).

I don't have any ideas on how to fix this. And, I don't understand Guix
internals very well. So, I'll wait for you or someone else to come up
with a solution. But, I can help with testing patches.
M
M
Marius Bakke wrote on 30 May 2017 17:17
(address . 27003@debbugs.gnu.org)
87poeqe5lo.fsf@fastmail.com
Arun Isaac <arunisaac@systemreboot.net> writes:

Toggle quote (11 lines)
>> I cannot currently make guarantees on when I'll be able to wrap my head
>> around the entire problem, so if you come to conclusions for next steps
>> on the basis of your testing, we should go with those.
>>
>> Else I will revisit this issue when I have time (which might be a
>> while).
>
> I don't have any ideas on how to fix this. And, I don't understand Guix
> internals very well. So, I'll wait for you or someone else to come up
> with a solution. But, I can help with testing patches.

I'm not sure if #25235 should block this issue though, since it's a
useful change regardless. But, not handling it adds "technical debt",
and a (minor) problem that is not currently present in perl modules.

We do have until the next 'core-updates' to think about it though ;-)
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlktjSMACgkQoqBt8qM6
VPprvAf/YnXs7TMTLKDriiK83gB5E4rMtFa6dKKk8kVl2YtIAIqBZRtCQ3LtN03o
jPrkBMoz037M+wy96xRFtmXtt0zHfyW6ajImJ1+sFLhv1rNC2+XYYBk5dV8mmlfv
AgP+tDlBFVWn8rNtKwEkkwPSqippjVR4TMb9wNjx5QsAQJMc5a7pFmoiUDt+yhfI
1iw/CBe8ueTr0AQcXk4WOfF401nYB2gfjM/hcjnWQInKtp/12ee9MmKPRAm2ut7N
3U8K38xO0lsTO5pGqVn5zIq+v0Y9QW3Hk4C9m02a2OHQHQJDuGA4KVb89GaDfokx
1wM7segtBH3Hlt69C1ihiK+sik1xPA==
=uRLK
-----END PGP SIGNATURE-----

B
B
Brice Waegeneire wrote on 21 Mar 2020 15:23
(address . 27003@debbugs.gnu.org)
96269db19c95345f380158f1fbb0ed4c@waegenei.re
Hello,

ludo@gnu.org (Ludovic Courtès) writes:
Toggle quote (6 lines)
> BTW, there’s also this patch series on this topic:
>
> https://issues.guix.info/issue/27003
>
> What to do?

A `wrap-script' procedure has been merged as of
0fb9a8df429a7b9f40610ff15baaff0d8e31e8cf
thanks to Ricardo's patch from https://issues.guix.info/issue/27003,it
seems to
provide similar functionality as the `program-wrapper' suggested here.
Should we close
this issue?

Brice.
L
L
Ludovic Courtès wrote on 21 Mar 2020 17:32
(name . Brice Waegeneire)(address . brice@waegenei.re)(address . 27003@debbugs.gnu.org)
87k13dzn0n.fsf@gnu.org
Hi,

Brice Waegeneire <brice@waegenei.re> skribis:

Toggle quote (15 lines)
> ludo@gnu.org (Ludovic Courtès) writes:
>> BTW, there’s also this patch series on this topic:
>>
>> https://issues.guix.info/issue/27003
>>
>> What to do?
>
> A `wrap-script' procedure has been merged as of
> 0fb9a8df429a7b9f40610ff15baaff0d8e31e8cf
> thanks to Ricardo's patch from https://issues.guix.info/issue/27003,
> it seems to
> provide similar functionality as the `program-wrapper' suggested
> here. Should we close
> this issue?

Yes, it seems that the patch that Alex initially submitted shares the
same goals as Ricardo’s ‘wrap-script’, so I guess we can close it.

Thanks for your work on bug triage!

Ludo’.
B
B
Brice Waegeneire wrote on 21 Mar 2020 21:26
close 27003
(address . control@debbugs.gnu.org)
90eab5a8aefbc5a59ffe37f594a6f3ef@waegenei.re
close 27003
?