[PATCH core-updates 1/?] build: substitute: Don't fail silently.

  • Open
  • quality assurance status badge
Details
One participant
  • Jakub Kądziołka
Owner
unassigned
Submitted by
Jakub Kądziołka
Severity
normal
J
J
Jakub Kądziołka wrote on 1 Jul 2020 00:09
(address . guix-patches@gnu.org)(address . mbakke@fastmail.com)
20200630220913.14665-1-kuba@kadziolka.net
* guix/build/utils.scm (substitute, substitute*)[require-matches?]: New
argument.
---
Of course, this will require some changes in the build recipes. However,
many of the bugs I have seen before could've been prevented by this
behavior, so I believe it is worth it.

I am currently running 'guix build hello' with these changes; I will
send any follow-up fixes needed to this bug#.

guix/build/utils.scm | 47 +++++++++++++++++++++++++++++++++-----------
1 file changed, 35 insertions(+), 12 deletions(-)

Toggle diff (96 lines)
diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index dc55c6745d..1bfb774c60 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -6,6 +6,7 @@
;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net>
;;; Copyright © 2018, 2019 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2020 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -747,29 +748,42 @@ PROC's result is returned."
(lambda (key . args)
(false-if-exception (delete-file template))))))
-(define (substitute file pattern+procs)
+(define* (substitute file pattern+procs #:key (require-matches? #t))
"PATTERN+PROCS is a list of regexp/two-argument-procedure pairs. For each
line of FILE, and for each PATTERN that it matches, call the corresponding
PROC as (PROC LINE MATCHES); PROC must return the line that will be written as
a substitution of the original line. Be careful about using '$' to match the
-end of a line; by itself it won't match the terminating newline of a line."
- (let ((rx+proc (map (match-lambda
- (((? regexp? pattern) . proc)
- (cons pattern proc))
- ((pattern . proc)
- (cons (make-regexp pattern regexp/extended)
- proc)))
- pattern+procs)))
+end of a line; by itself it won't match the terminating newline of a line.
+
+By default, SUBSTITUTE will raise a &message condition if one of the patterns
+fails to match. If a lack of matches is acceptable, pass #:require-matches? #f
+to disable this check."
+ (let ((rx+proc (map (match-lambda
+ (((? regexp? pattern) . proc)
+ (cons* #f "<unknown>" pattern proc))
+ ((pattern . proc)
+ (cons* #f pattern (make-regexp pattern regexp/extended)
+ proc)))
+ pattern+procs)))
(with-atomic-file-replacement file
(lambda (in out)
(let loop ((line (read-line in 'concat)))
(if (eof-object? line)
- #t
+ (when require-matches?
+ (for-each
+ (match-lambda
+ ((#f pat . _)
+ (raise (condition
+ (&message
+ (message (format #f "substitute: ~a: pattern failed to match: ~a" file pat))))))
+ ((#t . _) #t))
+ rx+proc))
(let ((line (fold (lambda (r+p line)
(match r+p
- ((regexp . proc)
+ ((_ _ regexp . proc)
(match (list-matches regexp line)
((and m+ (_ _ ...))
+ (set-car! r+p #t)
(proc line m+))
(_ line)))))
line
@@ -814,9 +828,17 @@ match substring.
Alternatively, FILE may be a list of file names, in which case they are
all subject to the substitutions.
+By default, SUBSTITUTE* will raise a &message condition if one of the patterns
+fails to match on one of the files. If a lack of matches is acceptable,
+add #:require-matches? #f after FILE to disable this check.
+
Be careful about using '$' to match the end of a line; by itself it won't
match the terminating newline of a line."
((substitute* file ((regexp match-var ...) body ...) ...)
+ (substitute* file #:require-matches? #t
+ ((regexp match-var ...) body ...) ...))
+ ((substitute* file #:require-matches? require-matches?
+ ((regexp match-var ...) body ...) ...)
(let ()
(define (substitute-one-file file-name)
(substitute
@@ -840,7 +862,8 @@ match the terminating newline of a line."
(begin body ...)
(substring l o (match:start m))
r))))))))
- ...)))
+ ...)
+ #:require-matches? require-matches?))
(match file
((files (... ...))
--
2.26.2
J
J
Jakub Kądziołka wrote on 1 Jul 2020 00:11
[PATCH 2/?] build: bootstrap-configure: Allow lack of matches in substitute.
(address . 42146@debbugs.gnu.org)
20200630221121.16086-1-kuba@kadziolka.net
Matches are not required here, as not every file will use every
variable.

* guix/build/gnu-bootstrap.scm (bootstrap-configure): Pass
#:require-matches? #f to substitute*.
---
guix/build/gnu-bootstrap.scm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (31 lines)
diff --git a/guix/build/gnu-bootstrap.scm b/guix/build/gnu-bootstrap.scm
index 1cb9dc5512..6ee520e301 100644
--- a/guix/build/gnu-bootstrap.scm
+++ b/guix/build/gnu-bootstrap.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -49,7 +50,7 @@ and object directories."
(format #t "Configuring ~a~%" template)
(let ((target (string-drop-right template 3)))
(copy-file template target)
- (substitute* target
+ (substitute* target #:require-matches? #f
(("@VERSION@") version))))
(find-files modules
(lambda (fn st)
@@ -58,7 +59,7 @@ and object directories."
(format #t "Configuring ~a~%" template)
(let ((target (string-drop-right template 3)))
(copy-file template target)
- (substitute* target
+ (substitute* target #:require-matches? #f
(("@GUILE@") guile)
(("@MODDIR@") moddir)
(("@GODIR@") godir))
--
2.26.2
J
J
Jakub Kądziołka wrote on 1 Jul 2020 00:11
[PATCH 3/?] gnu: commencement: Build fix for %bootstrap-mes-rewired.
(address . 42146@debbugs.gnu.org)
20200630221134.16350-1-kuba@kadziolka.net
Not every file has to refer to all dependencies.

* gnu/packages/commencement.scm (%bootstrap-mes-rewired)[builder]: Allow
lack of matches in substitute*.
---
gnu/packages/commencement.scm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

Toggle diff (23 lines)
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index ef250e037b..1c7cf3d810 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2018, 2019, 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;; Copyright © 2019, 2020 Marius Bakke <mbakke@fastmail.com>
;;; Copyright © 2020 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -283,7 +284,7 @@ pure Scheme to Tar and decompression in one easy step.")
(mescc (string-append bin "/mescc"))
(module (string-append out "/share/mes/module")))
(define (rewire file)
- (substitute* file
+ (substitute* file #:require-matches? #f
((mes) out)
(("/gnu/store[^ ]+mes-minimal-[^/)}\"]*") out)
(("/gnu/store[^ ]+guile-[^/]*/bin/guile") guile)
--
2.26.2
J
J
Jakub Kądziołka wrote on 1 Jul 2020 00:50
Re: [PATCH core-updates 1/?] build: substitute: Don't fail silently.
(address . 42146@debbugs.gnu.org)
20200630225007.tw2rcfbmzh3wvthf@gravity
The error messages I'm getting from this are quite indirect - there are
wrapped in quite a few layers. Is there a better way to report errors
inside a build utility?

ice-9/eval.scm:387:11: In procedure eval:
ice-9/eval.scm:387:11: Throw to key `srfi-34' with args `(#<condition &message [message: "substit ute: install.sh: pattern failed to match: -xf"] 11aa480>)'.
builder for `/gnu/store/wdyzzh7rkg47hfp434w72ly9nay1yva1-mes-boot-0.22.drv' failed with exit code 1

Regards,
Jakub Kądziołka
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEE5Xa/ss9usT31cTO54xWnWEYTFWQFAl77wZ8ACgkQ4xWnWEYT
FWRIhxAAhBBY1dtp5/7VfFJ2cKcDK8X1xTJucuTGpM5AflHMVarCBdba/7pofenb
/cpRR52LfhwQU2HMornopoRIeBnm7DR81vSE4A/V4XO7Z6+BhG2AXAU8biQ6Mzqu
1SH0ykadjyxZSOC+JZ4XfqSc/SDb79aIUh5O4Mw22xCir+2ed1zJcHrhjm4jOhMW
e3ZPoL1+WDhe26LYdhcjUXciBTf0TEhkb4ZCmHgdbe9cVaLq97gA25MsJRiV8Q1d
IoSgf7Fg5FUktl4xCZdMSQ/N3f5mkpS94P6X0EGZQRu3V2KaQH4Yut661W8X00Yr
F2OtlTu4RHivHk/sdkGWETb9W/EqAqquoSg+KENRzkbbASn2upS79/fG74kPooIq
ULNcSW0qCqbSUuELSBiRvP/G+AvSlrQ6w0Tfsur8KF2Rkh4f8m4hzCHkccUSP0FZ
DRsF1sjILlBDjjtb/dgm4uuWr7NOBUddp9s0DvCKpSdHVq9ms7Q9QNyRyZ7sEgzr
eqFkmF2Mv4IZWISy5K1XITWmL+ypctm8chceoQmwrhSU2YQ+2peiLykB7GxcymrB
UkkeULOJ2OZDE5Nl0Mgrz5WOxzN7AuH5C9FAE8JnNJ32mgZsgtPJ1GN4YpL4tmBv
KxwHdl55Rbc/PV3+oiVFnsZ6UOt/bL4wF9DqmPV25KXog63pNQ0=
=BRqx
-----END PGP SIGNATURE-----


J
J
Jakub Kądziołka wrote on 1 Jul 2020 02:42
[PATCH 4/?] gnu: mes-boot: Use verbosity settings integrated into buildsystem
(address . 42146@debbugs.gnu.org)
20200701004234.18050-1-kuba@kadziolka.net
* gnu/packages/commencement.scm (mes-boot): Set the V environment
variable. Don't substitute in the install script.
---
gnu/packages/commencement.scm | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)

Toggle diff (26 lines)
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 1c7cf3d810..295a378d11 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -407,6 +407,7 @@ $MES -e '(mescc)' module/mescc.scm -- \"$@\"
(string-append
mes "/share/mes/module"
":" dir "/nyacc-0.99.0/module"))
+ (setenv "V" "1")
(invoke "gash" "configure.sh"
(string-append "--prefix=" out)
(string-append "--host=i686-linux-gnu")))))
@@ -416,11 +417,6 @@ $MES -e '(mescc)' module/mescc.scm -- \"$@\"
(delete 'check)
(replace 'install
(lambda _
- (substitute* "install.sh" ; show some progress
- ((" -xf") " -xvf")
- (("^( *)((cp|mkdir|tar) [^']*[^\\])\n" all space cmd)
- (string-append space "echo '" cmd "'\n"
- space cmd "\n")))
(invoke "sh" "install.sh")
;; Keep ASCII output, for friendlier comparison and bisection
(let* ((out (assoc-ref %outputs "out"))
--
2.26.2
J
J
Jakub Kądziołka wrote on 1 Jul 2020 02:42
[PATCH 5/?] build-system/gnu: Allow lack of matches in substitution phases.
(address . 42146@debbugs.gnu.org)
20200701004242.18245-1-kuba@kadziolka.net
* guix/build/utils.scm (patch-makefile-SHELL, patch-/usr/bin/file),
guix/build/gnu-build-system.scm (patch-dot-desktop-files):
Allow lack of matches in substitute*.
---
guix/build/gnu-build-system.scm | 3 ++-
guix/build/utils.scm | 4 ++--
2 files changed, 4 insertions(+), 3 deletions(-)

Toggle diff (45 lines)
diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm
index 2e7dff2034..4b96761233 100644
--- a/guix/build/gnu-build-system.scm
+++ b/guix/build/gnu-build-system.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2020 Brendan Tildesley <mail@brendan.scot>
+;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -724,7 +725,7 @@ which cannot be found~%"
;; '.desktop' files contain translations and are always
;; UTF-8-encoded.
(with-fluids ((%default-port-encoding "UTF-8"))
- (substitute* files
+ (substitute* files #:require-matches? #f
(("^Exec=([^/[:blank:]\r\n]*)(.*)$" _ binary rest)
(string-append "Exec=" (which binary) rest))
(("^TryExec=([^/[:blank:]\r\n]*)(.*)$" _ binary rest)
diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index 1bfb774c60..a8218a7743 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -1005,7 +1005,7 @@ When KEEP-MTIME? is true, the atime/mtime of FILE are kept unchanged."
(let ((st (stat file)))
;; Consider FILE is using an 8-bit encoding to avoid errors.
(with-fluids ((%default-port-encoding #f))
- (substitute* file
+ (substitute* file #:require-matches? #f
(("^ *SHELL[[:blank:]]*:?=[[:blank:]]*([[:graph:]]*/)([[:graph:]]+)(.*)$"
_ dir shell args)
(let* ((old (string-append dir shell))
@@ -1033,7 +1033,7 @@ no replacement 'file' command, doing nothing~%")
(let ((st (stat file)))
;; Consider FILE is using an 8-bit encoding to avoid errors.
(with-fluids ((%default-port-encoding #f))
- (substitute* file
+ (substitute* file #:require-matches? #f
(("/usr/bin/file")
(begin
(format (current-error-port)
--
2.26.2
J
J
Jakub Kądziołka wrote on 1 Jul 2020 11:55
[PATCH 6/6] gnu: glibc-mesboot0: Fixup the fixup-configure phase.
(address . 42146@debbugs.gnu.org)
20200701095523.9501-1-kuba@kadziolka.net
* gnu/packages/commencement.scm (glibc-mesboot0)[fixup-configure]: Don't
repeat substitution. Fix indentation.
---
This is an interesting cleanup commit. I believe this concludes a nice
sampling of the consequences of this patch, and thus subsequent fixups
are unlikely to benefit from a review. I will be pushing my work to


gnu/packages/commencement.scm | 24 +++++++++++-------------
1 file changed, 11 insertions(+), 13 deletions(-)

Toggle diff (37 lines)
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 295a378d11..837f4e2ecc 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -1418,19 +1418,17 @@ ac_cv_c_float_format='IEEE (little-endian)'
(format (current-error-port)
"running ./configure ~a\n" (string-join configure-flags))
(apply invoke "./configure" configure-flags)))
- (add-after 'configure 'fixup-configure
- (lambda _
- (let* ((out (assoc-ref %outputs "out"))
- (bash (assoc-ref %build-inputs "bash"))
- (shell (string-append bash "/bin/bash")))
- (substitute* "config.make"
- (("INSTALL = scripts/") "INSTALL = $(..)./scripts/"))
- (substitute* "config.make"
- (("INSTALL = scripts/") "INSTALL = $(..)./scripts/")
- (("BASH = ") (string-append
- "SHELL = " shell "
- BASH = ")))
- #t))))))))
+ (add-after 'configure 'fixup-configure
+ (lambda _
+ (let* ((out (assoc-ref %outputs "out"))
+ (bash (assoc-ref %build-inputs "bash"))
+ (shell (string-append bash "/bin/bash")))
+ (substitute* "config.make"
+ (("INSTALL = scripts/") "INSTALL = $(..)./scripts/")
+ (("BASH = ") (string-append
+ "SHELL = " shell "
+BASH = ")))
+ #t))))))))
(define gcc-mesboot0
(package
--
2.26.2
?