[PATCH 0/2] Optimize profile hooks to avoid unnecessary reruns.

  • Open
  • quality assurance status badge
Details
2 participants
  • iyzsong
  • Ludovic Courtès
Owner
unassigned
Submitted by
iyzsong
Severity
normal
I
I
iyzsong wrote on 18 Jan 12:37 +0100
(address . guix-patches@gnu.org)(name . ???)(address . iyzsong@member.fsf.org)
cover.1737199694.git.iyzsong@member.fsf.org
From: ??? <iyzsong@member.fsf.org>

Hello, these patches make each profile hook run upon its specified interested
inputs, eg: the 'info-dir-file' hook only get inputs with info manuals,
install a package without info files won't trigger it. Thus reduce the chance
and time to rerun them when your profile changed.

Years ago tried in https://issues.guix.gnu.org/29928, now it seems I don't
need some hack like 'eval-gexp', just need all manifest entries built before
hooks.

Sou Bunnbu (???) (2):
profiles: Add #:build? argument to lower-manifest-entry.
profiles: Filter out unwanted manifest entries for profile hooks.

guix/profiles.scm | 562 +++++++++++++++++++++++++---------------------
1 file changed, 309 insertions(+), 253 deletions(-)


base-commit: 87045f0982bd7aebb07b380cbf322651227546f4
prerequisite-patch-id: c01aefda02910c494da59b6124e517aeafdb6803
--
2.47.1
I
I
iyzsong wrote on 18 Jan 12:41 +0100
[PATCH 1/2] profiles: Add #:build? argument to lower-manifest-entry.
(address . 75647@debbugs.gnu.org)(name . ???)(address . iyzsong@member.fsf.org)
11916b9f37612909f6ba4e07ca06ab903b7679ac.1737200329.git.iyzsong@member.fsf.org
From: ??? <iyzsong@member.fsf.org>

* guix/profiles.scm (lower-manifest-entry): Add #:build? keyword argument.

Change-Id: Iab2832d1bac1b28f6124e0c4e78e9284daf9a2ea
---
guix/profiles.scm | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)

Toggle diff (42 lines)
diff --git a/guix/profiles.scm b/guix/profiles.scm
index a28cf872cf..a05b90d685 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -308,9 +308,10 @@ (define (manifest-entry-lookup manifest)
((_ . entry) entry)
(#f #f))))
-(define* (lower-manifest-entry entry system #:key target)
+(define* (lower-manifest-entry entry system #:key target
+ (build? #f))
"Lower ENTRY for SYSTEM and TARGET such that its 'item' field is a store
-file name."
+file name. When BUILD? is true, build the entry before returning."
(define (recurse entry)
(mapm/accumulate-builds (lambda (entry)
(lower-manifest-entry entry system
@@ -319,12 +320,19 @@ (define* (lower-manifest-entry entry system #:key target)
(let ((item (manifest-entry-item entry)))
(if (string? item)
- (with-monad %store-monad
+ (mbegin %store-monad
+ (if (build?)
+ (build (list item))
+ (return #f))
(return entry))
- (mlet %store-monad ((drv (lower-object item system
+ (mlet* %store-monad ((drv (lower-object item system
#:target target))
+
(dependencies (recurse entry))
- (output -> (manifest-entry-output entry)))
+ (output -> (manifest-entry-output entry))
+ (built (if build?
+ (built-derivations (list (cons drv output)))
+ (return #f))))
(return (manifest-entry
(inherit entry)
(item (derivation->output-path drv output))
--
2.47.1
I
I
iyzsong wrote on 18 Jan 12:41 +0100
[PATCH 2/2] profiles: Filter out unwanted manifest entries for profile hooks.
(address . 75647@debbugs.gnu.org)(name . ???)(address . iyzsong@member.fsf.org)
3a82d367eafa61be9603a46d149f99d9867f6353.1737200329.git.iyzsong@member.fsf.org
From: ??? <iyzsong@member.fsf.org>

Before we run profile hooks for all manifest inputs, so if you install a new
package to your profile, all profile hooks will be run again, even if the new
package doesn't provide info manuals, man pages, etc.

After this commit every profile hook will be run with its hook related inputs,
avoid unneccessary reruns.

* guix/profiles.scm (manifest-lookup-package): Remove procedure.
(find-entry): New procedure.
(profile-derivation): Build manifest entries before running hooks.
(manual-database/optional): Remove procedure.
(%default-profile-hooks): Replace 'manual-database/optional' with 'manual-database'.
(info-dir-file, manual-database, fonts-dir-file, ghc-package-cache-file)
(ca-certificate-bundle, emacs-subdir, gdk-pixbuf-loaders-cache-file)
(glib-schemas, gtk-icon-themes, gtk-im-modules, texlive-font-maps)
(xdg-desktop-database, xdg-mime-database): Only run the hook with entries that
contains hook related files.

Change-Id: I19fb172f9ad8d98af9037f0cf5663589955dee2d
---
guix/profiles.scm | 552 +++++++++++++++++++++++++---------------------
1 file changed, 300 insertions(+), 252 deletions(-)

Toggle diff (403 lines)
diff --git a/guix/profiles.scm b/guix/profiles.scm
index a05b90d685..eeec76bf22 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -3,7 +3,7 @@
;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
;;; Copyright © 2014, 2016 Alex Kost <alezost@gmail.com>
;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
-;;; Copyright © 2015 Sou Bunnbu <iyzsong@gmail.com>
+;;; Copyright © 2015, 2025 ??? <iyzsong@envs.net>
;;; Copyright © 2016, 2017, 2018, 2019, 2021, 2022 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2016 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2017 Huang Ying <huang.ying.caritas@gmail.com>
@@ -321,9 +321,10 @@ (define* (lower-manifest-entry entry system #:key target
(let ((item (manifest-entry-item entry)))
(if (string? item)
(mbegin %store-monad
- (if (build?)
+ (build (list item))
+ (if build?
(build (list item))
- (return #f))
+ (return #t))
(return entry))
(mlet* %store-monad ((drv (lower-object item system
#:target target))
@@ -332,7 +333,7 @@ (define* (lower-manifest-entry entry system #:key target
(output -> (manifest-entry-output entry))
(built (if build?
(built-derivations (list (cons drv output)))
- (return #f))))
+ (return #t))))
(return (manifest-entry
(inherit entry)
(item (derivation->output-path drv output))
@@ -957,55 +958,18 @@ (define (manifest-inputs manifest)
(append-map entry->input (manifest-entries manifest)))
-(define* (manifest-lookup-package manifest name #:optional version)
- "Return as a monadic value the first package or store path referenced by
-MANIFEST that is named NAME and optionally has the given VERSION prefix, or #f
-if not found."
- ;; Return as a monadic value the package or store path referenced by the
- ;; manifest ENTRY, or #f if not referenced.
- (define (entry-lookup-package entry)
- (define (find-among-inputs inputs)
- (find (lambda (input)
- (and (package? input)
- (equal? name (package-name input))
- (if version
- (string-prefix? version (package-version input))
- #t)))
- inputs))
- (define (find-among-store-items items)
- (find (lambda (item)
- (let-values (((name* version*)
- (package-name->name+version
- (store-path-package-name item))))
- (and (string=? name name*)
- (if version
- (string-prefix? version version*)
- #t))))
- items))
-
- (with-monad %store-monad
- (match (manifest-entry-item entry)
- ((? package? package)
- (match (cons (list (package-name package) package)
- (package-transitive-inputs package))
- (((labels inputs . _) ...)
- (return (find-among-inputs inputs)))))
- ((? string? item)
- (mlet %store-monad ((refs (references* item)))
- (return (find-among-store-items refs))))
- (item
- ;; XXX: ITEM might be a 'computed-file' or anything like that, in
- ;; which case we don't know what to do. The fix may be to check
- ;; references once ITEM is compiled, as proposed at
- ;; <https://bugs.gnu.org/29927>.
- (return #f)))))
-
- (anym %store-monad
- entry-lookup-package (manifest-entries manifest)))
-
-(define* (info-dir-file manifest #:optional system)
- "Return a derivation that builds the 'dir' file for all the entries of
-MANIFEST."
+(define* (find-entry entries name #:optional version)
+ "Return the first manifest entry from ENTRIES that is named NAME and
+optionally has the given VERSION prefix, or #f if not found."
+ (find (lambda (entry)
+ (and (equal? name (manifest-entry-name entry))
+ (if version
+ (string-prefix? version (manifest-entry-version entry))
+ #t)))
+ entries))
+
+(define* (info-dir-file entries #:optional system)
+ "Return a derivation that builds the 'dir' file for all the manifest ENTRIES."
(define texinfo ;lazy reference
(module-ref (resolve-interface '(gnu packages texinfo)) 'texinfo))
(define gzip ;lazy reference
@@ -1014,7 +978,7 @@ (define* (info-dir-file manifest #:optional system)
(module-ref (resolve-interface '(gnu packages base))
'libc-utf8-locales-for-target))
- (define build
+ (define (build items)
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils)
@@ -1059,24 +1023,33 @@ (define* (info-dir-file manifest #:optional system)
(mkdir-p (string-append #$output "/share/info"))
(exit (every install-info
- (append-map info-files
- '#$(manifest-inputs manifest)))))))
-
- (gexp->derivation "info-dir" build
- #:system system
- #:local-build? #t
- #:substitutable? #f
- #:properties
- `((type . profile-hook)
- (hook . info-dir))))
+ (append-map info-files '#$items))))))
+
+ (mlet %store-monad
+ ;; Only run this hook for entries which contains info files.
+ ((interested -> (filter (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/share/info")))
+ entries)))
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "info-dir"
+ (build (map manifest-entry-item interested))
+ #:system system
+ #:local-build? #t
+ #:substitutable? #f
+ #:properties
+ `((type . profile-hook)
+ (hook . info-dir))))))
-(define* (ghc-package-cache-file manifest #:optional system)
+(define* (ghc-package-cache-file entries #:optional system)
"Return a derivation that builds the GHC 'package.cache' file for all the
-entries of MANIFEST, or #f if MANIFEST does not have any GHC packages."
+ENTRIES of manifest, or #f if ENTRIES does not have any GHC packages."
(define ghc ;lazy reference
(module-ref (resolve-interface '(gnu packages haskell)) 'ghc))
- (define build
+ (define (build items)
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils)
@@ -1107,8 +1080,7 @@ (define* (ghc-package-cache-file manifest #:optional system)
(system* (string-append #+ghc "/bin/ghc-pkg") "init" db-dir)
(for-each copy-conf-file
(append-map conf-files
- (delete-duplicates
- '#$(manifest-inputs manifest))))
+ (delete-duplicates '#$items)))
(let ((success
(zero?
(system* (string-append #+ghc "/bin/ghc-pkg") "recache"
@@ -1116,26 +1088,32 @@ (define* (ghc-package-cache-file manifest #:optional system)
(for-each delete-file (find-files db-dir "\\.conf$"))
(exit success)))))
- (with-monad %store-monad
- ;; Don't depend on GHC when there's nothing to do.
- (if (any (cut string-prefix? "ghc" <>)
- (map manifest-entry-name (manifest-entries manifest)))
- (gexp->derivation "ghc-package-cache" build
+ (mlet %store-monad
+ ;; Don't depend on GHC when there's nothing to do.
+ ((interested -> (filter (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/lib/ghc-"
+ (package-version ghc))))
+ entries)))
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "ghc-package-cache"
+ (build (map manifest-entry-item interested))
#:system system
#:local-build? #t
#:substitutable? #f
#:properties
`((type . profile-hook)
- (hook . ghc-package-cache)))
- (return #f))))
+ (hook . ghc-package-cache))))))
-(define* (ca-certificate-bundle manifest #:optional system)
+(define* (ca-certificate-bundle entries #:optional system)
"Return a derivation that builds a single-file bundle containing the CA
-certificates in the /etc/ssl/certs sub-directories of the packages in
-MANIFEST. Single-file bundles are required by programs such as Git and Lynx."
+certificates in the /etc/ssl/certs sub-directories of the packages for manifest
+ENTRIES. Single-file bundles are required by programs such as Git and Lynx."
;; See <http://lists.gnu.org/archive/html/guix-devel/2015-02/msg00429.html>
;; for a discussion.
- (define build
+ (define (build items)
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils)
@@ -1168,7 +1146,7 @@ (define* (ca-certificate-bundle manifest #:optional system)
;; install a UTF-8 locale.
(setlocale LC_ALL "C.UTF-8")
- (match (append-map ca-files '#$(manifest-inputs manifest))
+ (match (append-map ca-files '#$items)
(()
;; Since there are no CA files, just create an empty directory. Do
;; not create the etc/ssl/certs sub-directory, since that would
@@ -1184,16 +1162,26 @@ (define* (ca-certificate-bundle manifest #:optional system)
"/ca-certificates.crt"))
#t))))))
- (gexp->derivation "ca-certificate-bundle" build
+ (mlet %store-monad
+ ((interested -> (filter
+ (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/etc/ssl/certs")))
+ entries)))
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "ca-certificate-bundle"
+ (build (map manifest-entry-item interested))
#:system system
#:local-build? #t
#:substitutable? #f
#:properties
`((type . profile-hook)
- (hook . ca-certificate-bundle))))
+ (hook . ca-certificate-bundle))))))
-(define* (emacs-subdirs manifest #:optional system)
- (define build
+(define* (emacs-subdirs entries #:optional system)
+ (define (build items)
(with-imported-modules (source-module-closure
'((guix build profiles)
(guix build utils)))
@@ -1212,9 +1200,8 @@ (define* (emacs-subdirs manifest #:optional system)
file-is-directory?
(map (cute string-append dir "/" <>)
(scandir dir (negate (cute member <> '("." "..")))))))
- (filter file-exists?
- (map (cute string-append <> "/share/emacs/site-lisp")
- '#$(manifest-inputs manifest))))))
+ (map (cute string-append <> "/share/emacs/site-lisp")
+ '#$items))))
(mkdir-p destdir)
(with-directory-excursion destdir
(call-with-output-file "subdirs.el"
@@ -1225,77 +1212,82 @@ (define* (emacs-subdirs manifest #:optional system)
port)
(newline port)
#t)))))))
- (gexp->derivation "emacs-subdirs" build
- #:system system
- #:local-build? #t
- #:substitutable? #f
- #:properties
- `((type . profile-hook)
- (hook . emacs-subdirs))))
-(define* (gdk-pixbuf-loaders-cache-file manifest #:optional system)
+ (mlet %store-monad
+ ((interested -> (filter
+ (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/share/emacs/site-lisp")))
+ entries)))
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "emacs-subdirs"
+ (build (map manifest-entry-item interested))
+ #:system system
+ #:local-build? #t
+ #:substitutable? #f
+ #:properties
+ `((type . profile-hook)
+ (hook . emacs-subdirs))))))
+
+(define* (gdk-pixbuf-loaders-cache-file entries #:optional system)
"Return a derivation that produces a loaders cache file for every gdk-pixbuf
-loaders discovered in MANIFEST."
+loaders discovered in manifest ENTRIES."
(define gdk-pixbuf ;lazy reference
(module-ref (resolve-interface '(gnu packages gtk)) 'gdk-pixbuf))
(mlet* %store-monad
- ((gdk-pixbuf (manifest-lookup-package manifest "gdk-pixbuf"))
- (librsvg (manifest-lookup-package manifest "librsvg"))
- (gdk-pixbuf-bin -> (if (string? gdk-pixbuf)
- (string-append gdk-pixbuf "/bin")
- (file-append gdk-pixbuf "/bin"))))
-
- (define build
+ ((interested -> (filter
+ (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/lib/gdk-pixbuf-2.0")))
+ entries))
+ (gdk-pixbuf -> (or (and=> (find-entry entries "gdk-pixbuf")
+ manifest-entry-item)
+ (file-append gdk-pixbuf))))
+
+ (define (build items)
(with-imported-modules (source-module-closure
'((guix build glib-or-gtk-build-system)))
#~(begin
(use-modules (guix build glib-or-gtk-build-system))
- (setenv "PATH" (string-append #$gdk-pixbuf-bin ":" (getenv "PATH")))
+ (setenv "PATH" (string-append #$gdk-pixbuf "/bin:" (getenv "PATH")))
(generate-gdk-pixbuf-loaders-cache
- ;; XXX: MANIFEST-LOOKUP-PACKAGE transitively searches through
- ;; every input referenced by the manifest, while MANIFEST-INPUTS
- ;; only retrieves the immediate inputs as well as their
- ;; propagated inputs; to avoid causing an empty output derivation
- ;; we must ensure that the inputs contain at least one
- ;; loaders.cache file. This is why we include gdk-pixbuf or
- ;; librsvg when they are transitively found.
- (list #$@(if gdk-pixbuf
- (list gdk-pixbuf)
- '())
- #$@(if librsvg
- (list librsvg)
- '())
- #$@(manifest-inputs manifest))
+ (list #$gdk-pixbuf #$@items)
(list #$output)))))
- (if gdk-pixbuf
- (gexp->derivation "gdk-pixbuf-loaders-cache-file" build
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "gdk-pixbuf-loaders-cache-file"
+ (build (map manifest-entry-item interested))
#:system system
#:local-build? #t
#:substitutable? #f
#:properties
'((type . profile-hook)
- (hook . gdk-pixbuf-loaders-cache-file)))
- (return #f))))
+ (hook . gdk-pixbuf-loaders-cache-file))))))
-(define* (glib-schemas manifest #:optional system)
- "Return a derivation that unions all schemas from manifest entries and
+(define* (glib-schemas entries #:optional system)
+ "Return a derivation that unions all schemas from manifest ENTRIES and
creates the Glib 'gschemas.compiled' file."
(define glib ; lazy reference
(module-ref (resolve-interface '(gnu packages glib)) 'glib))
- (mlet %store-monad ((%glib (manifest-lookup-package manifest "glib"))
- ;; XXX: Can't use glib-compile-schemas corresponding
- ;; to the glib referenced by 'manifest'. Because
- ;; '%glib' can be either a package or store path, and
- ;; there's no way to get the "bin" output for the later.
- (glib-compile-schemas
+ (mlet %store-monad ((glib-compile-schemas
-> #~(string-append #+glib:bin
- "/bin/glib-compile-schemas")))
-
- (define build
+ "/bin/glib-compile-schemas"))
+ (interested
+ -> (filter
+ (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/share/glib-2.0/schemas")))
+ entries)))
+
+ (define (build items)
(with-imported-modules '((guix build utils)
(guix build union)
(guix build profiles)
@@ -1308,9 +1300,8 @@ (define* (glib-schemas manifest #:optional system)
(srfi srfi-26))
(let* ((destdir (string-append #$output "/share/glib-2.0/schemas"))
- (schemadirs (filter file-exists?
- (map (cut string-append <> "/share/glib-2.0/schemas")
- '#$(manifest-inputs manifest)))))
+ (schemadirs (map (cut string-append <> "/share/glib-2.0/schemas")
+
This message was truncated. Download the full message here.
I
I
iyzsong wrote on 18 Jan 13:07 +0100
[PATCH v2 1/2] profiles: Add #:build? argument to lower-manifest-entry.
(address . 75647@debbugs.gnu.org)(name . ???)(address . iyzsong@member.fsf.org)
367d0f5cffef102e3e4cdbaabcc093e94a74b180.1737201981.git.iyzsong@member.fsf.org
From: ??? <iyzsong@member.fsf.org>

* guix/profiles.scm (lower-manifest-entry): Add #:build? keyword argument.

Change-Id: Ifb86d581156034897377f3614fac67b7748e0ec3
---
guix/profiles.scm | 21 +++++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)

Toggle diff (54 lines)
diff --git a/guix/profiles.scm b/guix/profiles.scm
index a28cf872cf..0f47268541 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -3,7 +3,7 @@
;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
;;; Copyright © 2014, 2016 Alex Kost <alezost@gmail.com>
;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
-;;; Copyright © 2015 Sou Bunnbu <iyzsong@gmail.com>
+;;; Copyright © 2015, 2025 ??? <iyzsong@envs.net>
;;; Copyright © 2016, 2017, 2018, 2019, 2021, 2022 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2016 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2017 Huang Ying <huang.ying.caritas@gmail.com>
@@ -308,9 +308,10 @@ (define (manifest-entry-lookup manifest)
((_ . entry) entry)
(#f #f))))
-(define* (lower-manifest-entry entry system #:key target)
+(define* (lower-manifest-entry entry system #:key target
+ (build? #f))
"Lower ENTRY for SYSTEM and TARGET such that its 'item' field is a store
-file name."
+file name. When BUILD? is true, build the entry before returning."
(define (recurse entry)
(mapm/accumulate-builds (lambda (entry)
(lower-manifest-entry entry system
@@ -319,12 +320,20 @@ (define* (lower-manifest-entry entry system #:key target)
(let ((item (manifest-entry-item entry)))
(if (string? item)
- (with-monad %store-monad
+ (mbegin %store-monad
+ (build (list item))
+ (if build?
+ (build (list item))
+ (return #t))
(return entry))
- (mlet %store-monad ((drv (lower-object item system
+ (mlet* %store-monad ((drv (lower-object item system
#:target target))
+
(dependencies (recurse entry))
- (output -> (manifest-entry-output entry)))
+ (output -> (manifest-entry-output entry))
+ (built (if build?
+ (built-derivations (list (cons drv output)))
+ (return #t))))
(return (manifest-entry
(inherit entry)
(item (derivation->output-path drv output))

base-commit: 87045f0982bd7aebb07b380cbf322651227546f4
--
2.47.1
I
I
iyzsong wrote on 18 Jan 13:07 +0100
[PATCH v2 2/2] rofiles: Filter out unwanted manifest entries for profile hooks.
(address . 75647@debbugs.gnu.org)(name . ???)(address . iyzsong@member.fsf.org)
a8b17f8bad10e56e00cb2944bb3b4fca6c5d2d87.1737201981.git.iyzsong@member.fsf.org
From: ??? <iyzsong@member.fsf.org>

Before we run profile hooks for all manifest inputs, so if you install a new
package to your profile, all profile hooks will be run again, even if the new
package doesn't provide info manuals, man pages, etc.

After this commit every profile hook will be run with its hook related inputs,
avoid unneccessary reruns.

* guix/profiles.scm (manifest-lookup-package): Remove procedure.
(find-entry): New procedure.
(profile-derivation): Build manifest entries before running hooks.
(manual-database/optional): Remove procedure.
(%default-profile-hooks): Replace 'manual-database/optional' with 'manual-database'.
(info-dir-file, manual-database, fonts-dir-file, ghc-package-cache-file)
(ca-certificate-bundle, emacs-subdir, gdk-pixbuf-loaders-cache-file)
(glib-schemas, gtk-icon-themes, gtk-im-modules, texlive-font-maps)
(xdg-desktop-database, xdg-mime-database): Only run the hook with entries that
contains hook related files.

Change-Id: I6ee8c44cb1e625ced711cd0fc75d1762daa0dc72
---
guix/profiles.scm | 543 +++++++++++++++++++++++++---------------------
1 file changed, 295 insertions(+), 248 deletions(-)

Toggle diff (405 lines)
diff --git a/guix/profiles.scm b/guix/profiles.scm
index 0f47268541..eeec76bf22 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -958,55 +958,18 @@ (define (manifest-inputs manifest)
(append-map entry->input (manifest-entries manifest)))
-(define* (manifest-lookup-package manifest name #:optional version)
- "Return as a monadic value the first package or store path referenced by
-MANIFEST that is named NAME and optionally has the given VERSION prefix, or #f
-if not found."
- ;; Return as a monadic value the package or store path referenced by the
- ;; manifest ENTRY, or #f if not referenced.
- (define (entry-lookup-package entry)
- (define (find-among-inputs inputs)
- (find (lambda (input)
- (and (package? input)
- (equal? name (package-name input))
- (if version
- (string-prefix? version (package-version input))
- #t)))
- inputs))
- (define (find-among-store-items items)
- (find (lambda (item)
- (let-values (((name* version*)
- (package-name->name+version
- (store-path-package-name item))))
- (and (string=? name name*)
- (if version
- (string-prefix? version version*)
- #t))))
- items))
-
- (with-monad %store-monad
- (match (manifest-entry-item entry)
- ((? package? package)
- (match (cons (list (package-name package) package)
- (package-transitive-inputs package))
- (((labels inputs . _) ...)
- (return (find-among-inputs inputs)))))
- ((? string? item)
- (mlet %store-monad ((refs (references* item)))
- (return (find-among-store-items refs))))
- (item
- ;; XXX: ITEM might be a 'computed-file' or anything like that, in
- ;; which case we don't know what to do. The fix may be to check
- ;; references once ITEM is compiled, as proposed at
- ;; <https://bugs.gnu.org/29927>.
- (return #f)))))
-
- (anym %store-monad
- entry-lookup-package (manifest-entries manifest)))
-
-(define* (info-dir-file manifest #:optional system)
- "Return a derivation that builds the 'dir' file for all the entries of
-MANIFEST."
+(define* (find-entry entries name #:optional version)
+ "Return the first manifest entry from ENTRIES that is named NAME and
+optionally has the given VERSION prefix, or #f if not found."
+ (find (lambda (entry)
+ (and (equal? name (manifest-entry-name entry))
+ (if version
+ (string-prefix? version (manifest-entry-version entry))
+ #t)))
+ entries))
+
+(define* (info-dir-file entries #:optional system)
+ "Return a derivation that builds the 'dir' file for all the manifest ENTRIES."
(define texinfo ;lazy reference
(module-ref (resolve-interface '(gnu packages texinfo)) 'texinfo))
(define gzip ;lazy reference
@@ -1015,7 +978,7 @@ (define* (info-dir-file manifest #:optional system)
(module-ref (resolve-interface '(gnu packages base))
'libc-utf8-locales-for-target))
- (define build
+ (define (build items)
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils)
@@ -1060,24 +1023,33 @@ (define* (info-dir-file manifest #:optional system)
(mkdir-p (string-append #$output "/share/info"))
(exit (every install-info
- (append-map info-files
- '#$(manifest-inputs manifest)))))))
-
- (gexp->derivation "info-dir" build
- #:system system
- #:local-build? #t
- #:substitutable? #f
- #:properties
- `((type . profile-hook)
- (hook . info-dir))))
+ (append-map info-files '#$items))))))
+
+ (mlet %store-monad
+ ;; Only run this hook for entries which contains info files.
+ ((interested -> (filter (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/share/info")))
+ entries)))
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "info-dir"
+ (build (map manifest-entry-item interested))
+ #:system system
+ #:local-build? #t
+ #:substitutable? #f
+ #:properties
+ `((type . profile-hook)
+ (hook . info-dir))))))
-(define* (ghc-package-cache-file manifest #:optional system)
+(define* (ghc-package-cache-file entries #:optional system)
"Return a derivation that builds the GHC 'package.cache' file for all the
-entries of MANIFEST, or #f if MANIFEST does not have any GHC packages."
+ENTRIES of manifest, or #f if ENTRIES does not have any GHC packages."
(define ghc ;lazy reference
(module-ref (resolve-interface '(gnu packages haskell)) 'ghc))
- (define build
+ (define (build items)
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils)
@@ -1108,8 +1080,7 @@ (define* (ghc-package-cache-file manifest #:optional system)
(system* (string-append #+ghc "/bin/ghc-pkg") "init" db-dir)
(for-each copy-conf-file
(append-map conf-files
- (delete-duplicates
- '#$(manifest-inputs manifest))))
+ (delete-duplicates '#$items)))
(let ((success
(zero?
(system* (string-append #+ghc "/bin/ghc-pkg") "recache"
@@ -1117,26 +1088,32 @@ (define* (ghc-package-cache-file manifest #:optional system)
(for-each delete-file (find-files db-dir "\\.conf$"))
(exit success)))))
- (with-monad %store-monad
- ;; Don't depend on GHC when there's nothing to do.
- (if (any (cut string-prefix? "ghc" <>)
- (map manifest-entry-name (manifest-entries manifest)))
- (gexp->derivation "ghc-package-cache" build
+ (mlet %store-monad
+ ;; Don't depend on GHC when there's nothing to do.
+ ((interested -> (filter (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/lib/ghc-"
+ (package-version ghc))))
+ entries)))
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "ghc-package-cache"
+ (build (map manifest-entry-item interested))
#:system system
#:local-build? #t
#:substitutable? #f
#:properties
`((type . profile-hook)
- (hook . ghc-package-cache)))
- (return #f))))
+ (hook . ghc-package-cache))))))
-(define* (ca-certificate-bundle manifest #:optional system)
+(define* (ca-certificate-bundle entries #:optional system)
"Return a derivation that builds a single-file bundle containing the CA
-certificates in the /etc/ssl/certs sub-directories of the packages in
-MANIFEST. Single-file bundles are required by programs such as Git and Lynx."
+certificates in the /etc/ssl/certs sub-directories of the packages for manifest
+ENTRIES. Single-file bundles are required by programs such as Git and Lynx."
;; See <http://lists.gnu.org/archive/html/guix-devel/2015-02/msg00429.html>
;; for a discussion.
- (define build
+ (define (build items)
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils)
@@ -1169,7 +1146,7 @@ (define* (ca-certificate-bundle manifest #:optional system)
;; install a UTF-8 locale.
(setlocale LC_ALL "C.UTF-8")
- (match (append-map ca-files '#$(manifest-inputs manifest))
+ (match (append-map ca-files '#$items)
(()
;; Since there are no CA files, just create an empty directory. Do
;; not create the etc/ssl/certs sub-directory, since that would
@@ -1185,16 +1162,26 @@ (define* (ca-certificate-bundle manifest #:optional system)
"/ca-certificates.crt"))
#t))))))
- (gexp->derivation "ca-certificate-bundle" build
+ (mlet %store-monad
+ ((interested -> (filter
+ (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/etc/ssl/certs")))
+ entries)))
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "ca-certificate-bundle"
+ (build (map manifest-entry-item interested))
#:system system
#:local-build? #t
#:substitutable? #f
#:properties
`((type . profile-hook)
- (hook . ca-certificate-bundle))))
+ (hook . ca-certificate-bundle))))))
-(define* (emacs-subdirs manifest #:optional system)
- (define build
+(define* (emacs-subdirs entries #:optional system)
+ (define (build items)
(with-imported-modules (source-module-closure
'((guix build profiles)
(guix build utils)))
@@ -1213,9 +1200,8 @@ (define* (emacs-subdirs manifest #:optional system)
file-is-directory?
(map (cute string-append dir "/" <>)
(scandir dir (negate (cute member <> '("." "..")))))))
- (filter file-exists?
- (map (cute string-append <> "/share/emacs/site-lisp")
- '#$(manifest-inputs manifest))))))
+ (map (cute string-append <> "/share/emacs/site-lisp")
+ '#$items))))
(mkdir-p destdir)
(with-directory-excursion destdir
(call-with-output-file "subdirs.el"
@@ -1226,77 +1212,82 @@ (define* (emacs-subdirs manifest #:optional system)
port)
(newline port)
#t)))))))
- (gexp->derivation "emacs-subdirs" build
- #:system system
- #:local-build? #t
- #:substitutable? #f
- #:properties
- `((type . profile-hook)
- (hook . emacs-subdirs))))
-(define* (gdk-pixbuf-loaders-cache-file manifest #:optional system)
+ (mlet %store-monad
+ ((interested -> (filter
+ (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/share/emacs/site-lisp")))
+ entries)))
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "emacs-subdirs"
+ (build (map manifest-entry-item interested))
+ #:system system
+ #:local-build? #t
+ #:substitutable? #f
+ #:properties
+ `((type . profile-hook)
+ (hook . emacs-subdirs))))))
+
+(define* (gdk-pixbuf-loaders-cache-file entries #:optional system)
"Return a derivation that produces a loaders cache file for every gdk-pixbuf
-loaders discovered in MANIFEST."
+loaders discovered in manifest ENTRIES."
(define gdk-pixbuf ;lazy reference
(module-ref (resolve-interface '(gnu packages gtk)) 'gdk-pixbuf))
(mlet* %store-monad
- ((gdk-pixbuf (manifest-lookup-package manifest "gdk-pixbuf"))
- (librsvg (manifest-lookup-package manifest "librsvg"))
- (gdk-pixbuf-bin -> (if (string? gdk-pixbuf)
- (string-append gdk-pixbuf "/bin")
- (file-append gdk-pixbuf "/bin"))))
-
- (define build
+ ((interested -> (filter
+ (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/lib/gdk-pixbuf-2.0")))
+ entries))
+ (gdk-pixbuf -> (or (and=> (find-entry entries "gdk-pixbuf")
+ manifest-entry-item)
+ (file-append gdk-pixbuf))))
+
+ (define (build items)
(with-imported-modules (source-module-closure
'((guix build glib-or-gtk-build-system)))
#~(begin
(use-modules (guix build glib-or-gtk-build-system))
- (setenv "PATH" (string-append #$gdk-pixbuf-bin ":" (getenv "PATH")))
+ (setenv "PATH" (string-append #$gdk-pixbuf "/bin:" (getenv "PATH")))
(generate-gdk-pixbuf-loaders-cache
- ;; XXX: MANIFEST-LOOKUP-PACKAGE transitively searches through
- ;; every input referenced by the manifest, while MANIFEST-INPUTS
- ;; only retrieves the immediate inputs as well as their
- ;; propagated inputs; to avoid causing an empty output derivation
- ;; we must ensure that the inputs contain at least one
- ;; loaders.cache file. This is why we include gdk-pixbuf or
- ;; librsvg when they are transitively found.
- (list #$@(if gdk-pixbuf
- (list gdk-pixbuf)
- '())
- #$@(if librsvg
- (list librsvg)
- '())
- #$@(manifest-inputs manifest))
+ (list #$gdk-pixbuf #$@items)
(list #$output)))))
- (if gdk-pixbuf
- (gexp->derivation "gdk-pixbuf-loaders-cache-file" build
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "gdk-pixbuf-loaders-cache-file"
+ (build (map manifest-entry-item interested))
#:system system
#:local-build? #t
#:substitutable? #f
#:properties
'((type . profile-hook)
- (hook . gdk-pixbuf-loaders-cache-file)))
- (return #f))))
+ (hook . gdk-pixbuf-loaders-cache-file))))))
-(define* (glib-schemas manifest #:optional system)
- "Return a derivation that unions all schemas from manifest entries and
+(define* (glib-schemas entries #:optional system)
+ "Return a derivation that unions all schemas from manifest ENTRIES and
creates the Glib 'gschemas.compiled' file."
(define glib ; lazy reference
(module-ref (resolve-interface '(gnu packages glib)) 'glib))
- (mlet %store-monad ((%glib (manifest-lookup-package manifest "glib"))
- ;; XXX: Can't use glib-compile-schemas corresponding
- ;; to the glib referenced by 'manifest'. Because
- ;; '%glib' can be either a package or store path, and
- ;; there's no way to get the "bin" output for the later.
- (glib-compile-schemas
+ (mlet %store-monad ((glib-compile-schemas
-> #~(string-append #+glib:bin
- "/bin/glib-compile-schemas")))
-
- (define build
+ "/bin/glib-compile-schemas"))
+ (interested
+ -> (filter
+ (lambda (entry)
+ (file-exists?
+ (string-append (manifest-entry-item entry)
+ "/share/glib-2.0/schemas")))
+ entries)))
+
+ (define (build items)
(with-imported-modules '((guix build utils)
(guix build union)
(guix build profiles)
@@ -1309,9 +1300,8 @@ (define* (glib-schemas manifest #:optional system)
(srfi srfi-26))
(let* ((destdir (string-append #$output "/share/glib-2.0/schemas"))
- (schemadirs (filter file-exists?
- (map (cut string-append <> "/share/glib-2.0/schemas")
- '#$(manifest-inputs manifest)))))
+ (schemadirs (map (cut string-append <> "/share/glib-2.0/schemas")
+ '#$items)))
;; Union all the schemas.
(mkdir-p (string-append #$output "/share/glib-2.0"))
@@ -1326,32 +1316,35 @@ (define* (glib-schemas manifest #:optional system)
dir)))))))
;; Don't run the hook when there's nothing to do.
- (if %glib
- (gexp->derivation "glib-schemas" build
+ (if (null? interested)
+ (return #f)
+ (gexp->derivation "glib-schemas"
+ (build (map manifest-entry-item interested))
#:system system
#:local-build? #t
#:substitutable? #f
#:properties
`((type . profile-hook)
- (hook . glib-schemas)))
- (return #f))))
+ (hook . glib-schemas))))))
-(define* (gtk-icon-themes manifest #:optional system)
- "Return a derivation that unions all icon themes from manifest entries and
+(define* (gtk-icon-themes entries #:optional system)
+ "Return a derivation that unions all icon themes from manifest ENTRIES and
creates the GTK+ 'icon-theme.cache' file for each theme."
(define gtk+ ; lazy reference
(module-ref (resolve-interface '(gnu packages gtk)) 'gtk+))
- (mlet %store-monad ((%gtk+ (manifest-lookup-package manifest "gtk+"))
- ;; XXX: Can't use gtk-update-icon-cache corresponding
- ;; to the gtk+ referenced by 'manifest'. Beca
This message was truncated. Download the full message here.
L
L
Ludovic Courtès wrote 2 days ago
Re: [bug#75647] [PATCH v2 1/2] profiles: Add #:build? argument to lower-manifest-entry.
(address . iyzsong@envs.net)
87plk8kyg6.fsf@gnu.org
Hi,

iyzsong@envs.net skribis:

Toggle quote (6 lines)
> From: ??? <iyzsong@member.fsf.org>
>
> * guix/profiles.scm (lower-manifest-entry): Add #:build? keyword argument.
>
> Change-Id: Ifb86d581156034897377f3614fac67b7748e0ec3

[...]

Toggle quote (21 lines)
> @@ -319,12 +320,20 @@ (define* (lower-manifest-entry entry system #:key target)
>
> (let ((item (manifest-entry-item entry)))
> (if (string? item)
> - (with-monad %store-monad
> + (mbegin %store-monad
> + (build (list item))
> + (if build?
> + (build (list item))
> + (return #t))
> (return entry))
> - (mlet %store-monad ((drv (lower-object item system
> + (mlet* %store-monad ((drv (lower-object item system
> #:target target))
> +
> (dependencies (recurse entry))
> - (output -> (manifest-entry-output entry)))
> + (output -> (manifest-entry-output entry))
> + (built (if build?
> + (built-derivations (list (cons drv output)))

Overall, I think that building things individually is a bad idea because
it prevents parallelism and makes things overall slower (even if the
“build handler” mechanism mitigates that¹).

You wrote:

Toggle quote (5 lines)
> Hello, these patches make each profile hook run upon its specified interested
> inputs, eg: the 'info-dir-file' hook only get inputs with info manuals,
> install a package without info files won't trigger it. Thus reduce the chance
> and time to rerun them when your profile changed.

The ‘info-dir-file’ hooks runs instantly when there are no Info files,
so that’s probably not a good example.

Toggle quote (4 lines)
> Years ago tried in https://issues.guix.gnu.org/29928, now it seems I don't
> need some hack like 'eval-gexp', just need all manifest entries built before
> hooks.

Yeah, I’m not sure how to move forward.

Perhaps you could identify specific cases of profile hooks that may
often run and use resources for no good reason?

Thanks,
Ludo’.

?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 75647@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 75647
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch