[PATCH] profiles: Optimise 'fonts-dir-file'.

OpenSubmitted by Maxime Devos.
Details
2 participants
  • Ludovic Courtès
  • Maxime Devos
Owner
unassigned
Severity
normal
M
M
Maxime Devos wrote on 5 Jul 21:06 +0200
(address . guix-patches@gnu.org)
e1af6b02effdeaf4a05f1ddb602003d57961ec62.camel@telenet.be
Hi guix,
These two patches should speed up profile generationby optimising the 'fonts-dir-file' hook. The first patchis the most important; from the commit message:
‘Only let the build G-exp refer to inputs that might actuallyhave fonts. That way, if the list of fonts in the manifestdidn't change and the new manifest is built, the font fileswill not be rebuilt.’
The second patch doesn't bring much.
To test, you can test whether things like ...
$ ./pre-inst-env guix environment --ad-hoc --pure lagrange -- lagrange
still work (lagrange is a graphical application using fonts).Not sure what a good ‘benchmark’ would be.
Unfortunately, this does not help with the "guix package -i" case,as in that case, the code doesn't have access to all the package objects,and will have to satisfy itself with the store paths, in which case theprofile code pessimistically assumes the store item has fonts ...
I suppose this restriction could be lifted if/when the gs-fonts packageis renamed to font-ghostscript or something like that ...
However, the optimisation should work in the "guix environment ... --ad-hoc ...","guix system reconfigure ..." and (I'd presume) the guix home-manager cases,though I only tested the first.
Greetings,Maxime.
From 4fe1e30e33c01be9fd17cf240732b3351c7b0fa4 Mon Sep 17 00:00:00 2001From: Maxime Devos <maximedevos@telenet.be>Date: Mon, 5 Jul 2021 18:55:31 +0200Subject: [PATCH 1/2] profiles: Optimise 'fonts-dir-file'.
Only let the build G-exp refer to inputs that might actuallyhave fonts. That way, if the list of fonts in the manifestdidn't change and the new manifest is built, the font fileswill not be rebuilt.
* guix/profiles.scm (fonts-dir-file)[has-fonts?]: New predicate. (fonts-dir-file)[relevant-inputs]: New variable. (fonts-dir-file)[build]: Use 'relevant-inputs' instead of 'manifest-inputs'.* doc/contributing.texi (Fonts): Note the 'fonts-' naming convention is technically important now.* gnu/packages/ghostscript.scm (gs-fonts): Work-around the package name contravening the convention.--- doc/contributing.texi | 8 ++++++++ gnu/packages/ghostscript.scm | 10 +++++++++- guix/profiles.scm | 22 +++++++++++++++++++++- 3 files changed, 38 insertions(+), 2 deletions(-)
Toggle diff (96 lines)diff --git a/doc/contributing.texi b/doc/contributing.texiindex 1f0160a707..903cfd390b 100644--- a/doc/contributing.texi+++ b/doc/contributing.texi@@ -886,6 +886,14 @@ is added to the package name. We use @code{-ttf} for TrueType fonts, @code{-otf} for OpenType fonts and @code{-type1} for PostScript Type 1 fonts. +There are important technical reasons for this naming convention as well:+for efficiency reasons, the profile generation code needs to know if a+package has fonts and looks at the package name to determine this.++@c ^ There is an escape hatch (the 'has-fonts?' package property),+@c but let's keep it undocumented until it turns out to be actually+@c needed somewhere else than in the incorrectly-named 'gs-fonts'+@c package ... @node Coding Style @section Coding Stylediff --git a/gnu/packages/ghostscript.scm b/gnu/packages/ghostscript.scmindex 03a516dc52..a9f1c52c66 100644--- a/gnu/packages/ghostscript.scm+++ b/gnu/packages/ghostscript.scm@@ -384,7 +384,15 @@ architecture.") "Ghostscript fonts provides fonts and font metrics customarily distributed with Ghostscript. It currently includes the 35 standard PostScript fonts.") (license license:gpl2)- (home-page "https://sourceforge.net/projects/gs-fonts/")))+ (home-page "https://sourceforge.net/projects/gs-fonts/")+ (properties+ ;; TODO: explicitely tell 'fonts-dir-file' that this is a font package,+ ;; as this package violates the convention that font package names are+ ;; prefixed with 'font-'.+ ;;+ ;; TODO(core-updates): Rename package to 'fonts-ghostscript' or something+ ;; like that.+ `((has-fonts? . #t))))) (define-public libspectre (packagediff --git a/guix/profiles.scm b/guix/profiles.scmindex 8c02149c6f..0f9df68f42 100644--- a/guix/profiles.scm+++ b/guix/profiles.scm@@ -12,6 +12,7 @@ ;;; Copyright © 2019 Mathieu Othacehe <m.othacehe@gmail.com> ;;; Copyright © 2020 Danny Milosavljevic <dannym@scratchpost.org> ;;; Copyright © 2014 David Thompson <davet@gnu.org>+;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be> ;;; ;;; This file is part of GNU Guix. ;;;@@ -33,6 +34,8 @@ #:use-module ((guix utils) #:hide (package-name->name+version)) #:use-module ((guix build utils) #:select (package-name->name+version mkdir-p))+ #:use-module ((guix build-system)+ #:select (build-system-name)) #:use-module ((guix diagnostics) #:select (&fix-hint)) #:use-module (guix i18n) #:use-module (guix records)@@ -1514,6 +1517,23 @@ files for the fonts of the @var{manifest} entries." (define mkfontdir (module-ref (resolve-interface '(gnu packages xorg)) 'mkfontdir)) + (define (has-fonts? input)+ (define thing (gexp-input-thing input))+ (if (package? thing)+ (or (string-prefix? "font-" (package-name thing))+ ;; In the upstream 'guix' channel, font packages should+ ;; be named font-SOMETHING. But if another channel+ ;; names its fonts differently but uses font-build-system,+ ;; accepting that seems friendly.+ (eq? 'font (build-system-name (package-build-system thing)))+ ;; FIXME(core-updates) escape hatch for the incorrectly-named+ ;; 'gs-fonts' package+ (assq-ref (package-properties thing) 'has-fonts?))+ ;; Pessimistically assume the input might have fonts.+ #t))++ (define relevant-inputs (filter has-fonts? (manifest-inputs manifest)))+ (define build #~(begin (use-modules (srfi srfi-26)@@ -1522,7 +1542,7 @@ files for the fonts of the @var{manifest} entries." (let ((fonts-dirs (filter file-exists? (map (cut string-append <> "/share/fonts")- '#$(manifest-inputs manifest)))))+ '#$relevant-inputs)))) (mkdir #$output) (if (null? fonts-dirs) (exit #t)-- 2.32.0
From a706b8be4f54530b1cd12c03a1bf3941be43be3c Mon Sep 17 00:00:00 2001From: Maxime Devos <maximedevos@telenet.be>Date: Mon, 5 Jul 2021 20:16:22 +0200Subject: [PATCH 2/2] profiles: Avoid dependency on 'mkfontdir' and friend when unused.
Avoid depending on on "mkfontdir" and "mkfontscale" ifthey won't actually be used, to avoid building theaforementioned packages when they are updated.
* guix/profiles.scm (fonts-dir-file)[build]: Note why the '(null? fonts-dir)' check cannot be removed. (fonts-dir-file): When the 'relevant-inputs' list is empty, just build an empty directory.--- guix/profiles.scm | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
Toggle diff (45 lines)diff --git a/guix/profiles.scm b/guix/profiles.scmindex 0f9df68f42..6cf480ddf2 100644--- a/guix/profiles.scm+++ b/guix/profiles.scm@@ -1544,6 +1544,8 @@ files for the fonts of the @var{manifest} entries." "/share/fonts") '#$relevant-inputs)))) (mkdir #$output)+ ;; has-fonts? can have false positives,+ ;; so this check is necessary. (if (null? fonts-dirs) (exit #t) (let* ((share-dir (string-append #$output "/share"))@@ -1585,15 +1587,20 @@ files for the fonts of the @var{manifest} entries." (delete-file fonts-dir-file)))) directories))))))) - (gexp->derivation "fonts-dir" build- #:modules '((guix build utils)- (guix build union)- (srfi srfi-26))- #:local-build? #t- #:substitutable? #f- #:properties- `((type . profile-hook)- (hook . fonts-dir))))+ (if (null? relevant-inputs)+ ;; Avoid depending on on "mkfontdir" and "mkfontscale" if+ ;; they won't actually be used, to avoid building the aforementioned+ ;; packages when they are updated.+ (lower-object (file-union "fonts-dir" '()))+ (gexp->derivation "fonts-dir" build+ #:modules '((guix build utils)+ (guix build union)+ (srfi srfi-26))+ #:local-build? #t+ #:substitutable? #f+ #:properties+ `((type . profile-hook)+ (hook . fonts-dir))))) (define (manual-database manifest) "Return a derivation that builds the manual page database (\"mandb\") for-- 2.32.0
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYONYOBccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7lJtAP9nvaqaE+zY3PdMOWeNm3KRB0gnJg7olk/UOUwGsYucgAEA2flnKzsocky7i3yjttW1kFKdfasoNfjiUnMOziukKw0==6xxl-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 7 Jul 18:52 +0200
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 49421@debbugs.gnu.org)
87a6myoyl5.fsf@gnu.org
Hi,
Maxime Devos <maximedevos@telenet.be> skribis:
Toggle quote (26 lines)> These two patches should speed up profile generation> by optimising the 'fonts-dir-file' hook. The first patch> is the most important; from the commit message:>> ‘Only let the build G-exp refer to inputs that might actually> have fonts. That way, if the list of fonts in the manifest> didn't change and the new manifest is built, the font files> will not be rebuilt.’>> The second patch doesn't bring much.>> To test, you can test whether things like ...>> $ ./pre-inst-env guix environment --ad-hoc --pure lagrange -- lagrange>> still work (lagrange is a graphical application using fonts).> Not sure what a good ‘benchmark’ would be.>> Unfortunately, this does not help with the "guix package -i" case,> as in that case, the code doesn't have access to all the package objects,> and will have to satisfy itself with the store paths, in which case the> profile code pessimistically assumes the store item has fonts ...>> I suppose this restriction could be lifted if/when the gs-fonts package> is renamed to font-ghostscript or something like that ...
Do you know what’s taking time in the ‘fonts-dir-file’ hook?(On my x86_64 laptop with SSD, it runs in 4.4s on a cold cache for aprofile containing 14 font packages, and in 1.2s on a warm cache.)
IIUC, the hook completes almost instantaneously when creating a profilewithout fonts because (null? fonts-dirs) is true.
In other cases, one option to speed things up (and possibly remove thehook’s dependency on mkfontdir) would be to create ‘fonts.dir’ and‘fonts.scale’ in a build phase of each font package, *provided* there’sa cheap way to assemble several such files, such as mere concatenation.That way, the profile hook would only need to assemble these files asopposed to traversing all the font files.
Given that ‘fonts.dir’ and ‘fonts.scale’ are mere text files with oneline per font, this approach looks doable.
WDYT?
Toggle quote (20 lines)> From 4fe1e30e33c01be9fd17cf240732b3351c7b0fa4 Mon Sep 17 00:00:00 2001> From: Maxime Devos <maximedevos@telenet.be>> Date: Mon, 5 Jul 2021 18:55:31 +0200> Subject: [PATCH 1/2] profiles: Optimise 'fonts-dir-file'.>> Only let the build G-exp refer to inputs that might actually> have fonts. That way, if the list of fonts in the manifest> didn't change and the new manifest is built, the font files> will not be rebuilt.>> * guix/profiles.scm> (fonts-dir-file)[has-fonts?]: New predicate.> (fonts-dir-file)[relevant-inputs]: New variable.> (fonts-dir-file)[build]: Use 'relevant-inputs' instead of> 'manifest-inputs'.> * doc/contributing.texi (Fonts): Note the 'fonts-' naming> convention is technically important now.> * gnu/packages/ghostscript.scm (gs-fonts): Work-around the> package name contravening the convention.
To me this approach should be a last resort because it relies onconventions and manual annotations, and it’s easy to forget these andget incorrect results (we could end up having to deal with reports like“I installed font X, why doesn’t in show up in the font list?”).
Toggle quote (6 lines)> + ;; In the upstream 'guix' channel, font packages should> + ;; be named font-SOMETHING. But if another channel> + ;; names its fonts differently but uses font-build-system,> + ;; accepting that seems friendly.> + (eq? 'font (build-system-name (package-build-system thing)))
In general ‘build-system-name’ shouldn’t be used; the name is meant tobe a debugging aid.
Thanks,Ludo’.
M
M
Maxime Devos wrote on 15 Jul 18:28 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 49421@debbugs.gnu.org)
5ec75ed429133cbc73dc7916959302033fe70a13.camel@telenet.be
Ludovic Courtès schreef op wo 07-07-2021 om 18:52 [+0200]:
Toggle quote (34 lines)> Hi,> > Maxime Devos <maximedevos@telenet.be> skribis:> > > These two patches should speed up profile generation> > by optimising the 'fonts-dir-file' hook. The first patch> > is the most important; from the commit message:> > > > ‘Only let the build G-exp refer to inputs that might actually> > have fonts. That way, if the list of fonts in the manifest> > didn't change and the new manifest is built, the font files> > will not be rebuilt.’> > > > The second patch doesn't bring much.> > > > To test, you can test whether things like ...> > > > $ ./pre-inst-env guix environment --ad-hoc --pure lagrange -- lagrange> > > > still work (lagrange is a graphical application using fonts).> > Not sure what a good ‘benchmark’ would be.> > > > Unfortunately, this does not help with the "guix package -i" case,> > as in that case, the code doesn't have access to all the package objects,> > and will have to satisfy itself with the store paths, in which case the> > profile code pessimistically assumes the store item has fonts ...> > > > I suppose this restriction could be lifted if/when the gs-fonts package> > is renamed to font-ghostscript or something like that ...> > Do you know what’s taking time in the ‘fonts-dir-file’ hook?> (On my x86_64 laptop with SSD, it runs in 4.4s on a cold cache for a> profile containing 14 font packages, and in 1.2s on a warm cache.)
I noticed ‘building fonts directories’ was taking long (9 minutes or so,when using "guix package -r two packages that don't have fonts"),so I took the opportunity to "strace" the process, though I was a little late.Some statistics:
% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ---------------- 27,14 1,676551 117 14297 rename 19,60 1,211012 13 90829 8 lstat 18,34 1,133182 63 17787 15 statx 11,80 0,729185 51 14297 link 6,65 0,411065 68510 6 wait4 5,08 0,313632 10 28651 chmod 3,66 0,226258 14 15114 readlink 2,18 0,134913 8 15193 utimensat 1,78 0,110257 32 3431 rmdir 0,91 0,056504 7 7370 getdents64 0,82 0,050817 3 14295 getpid 0,47 0,029074 7 3736 openat 0,46 0,028547 8 3479 3428 unlink 0,36 0,021940 5 3747 close 0,26 0,016355 4 3704 fstat 0,16 0,009752 10 898 lchown 0,12 0,007242 4 1800 geteuid 0,07 0,004203 62 67 read 0,06 0,003512 3 900 getegid 0,03 0,001959 326 6 clone 0,02 0,001133 18 62 write 0,01 0,000488 3 127 1 fcntl 0,01 0,000451 3 136 lseek 0,00 0,000197 14 14 mkdir 0,00 0,000065 13 5 select 0,00 0,000045 5 8 chown 0,00 0,000044 7 6 brk 0,00 0,000027 13 2 munmap 0,00 0,000023 11 2 pipe 0,00 0,000013 6 2 stat 0,00 0,000007 7 1 futex 0,00 0,000004 2 2 getuid 0,00 0,000000 0 1 rt_sigreturn------ ----------- ----------- --------- --------- ----------------100,00 6,178457 25 239975 3452 total
And a part of the log:
<start of log>lstat("/gnu/store/.links/[...]2i", {st_mode=S_IFLNK|0777, st_size=124, ...}) = 0lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", {st_mode=S_IFDIR|0555, st_size=4112, ...>chmod("/gnu/store/[..]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", 040755) = 0getpid() = 26371link("/gnu/store/.links/[..]2i", "/gnu/store/.tmp-link-26371-632297102") = 0rename("/gnu/store/.tmp-link-26371-632297102", "/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/2>lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", {st_mode=S_IFDIR|0755, st_size=4112, ...>chmod("/gnu/store/[...]-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", 040555) = 0utimensat(AT_FDCWD, "/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", [{tv_sec=1626363034, tv_ns>lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/2223.png", {st_mode=S_IFLNK|0777, st_size=>lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/2223.png", {st_mode=S_IFLNK|0777, st_size=>lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/2223.png", {st_mode=S_IFLNK|0777, st_size=>readlink("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/2223.png", "/gnu/store/bf582s56ldb3y34p>statx(AT_FDCWD, "/gnu/store/.links/[...]v3", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, 0, {stx_mask=STATX_ALL|>lstat("/gnu/store/.links/[...]v3", {st_mode=S_IFLNK|0777, st_size=124, ...}) = 0lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", {st_mode=S_IFDIR|0555, st_size=4112, ...>chmod("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", 040755) = 0getpid() = 26371link("/gnu/store/.links/[...]v3", "/gnu/store/.tmp-link-26371-114005693") = 0rename("/gnu/store/.tmp-link-26371-114005693", "/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/2>lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", {st_mode=S_IFDIR|0755, st_size=4112, ...>chmod("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", 040555) = 0utimensat(AT_FDCWD, "/gnu/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", [{tv_sec=1626363034, tv_ns>lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/22CC.png", {st_mode=S_IFLNK|0777, st_size=>lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/22CC.png", {st_mode=S_IFLNK|0777, st_size=>lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/22CC.png", {st_mode=S_IFLNK|0777, st_size=>readlink("/gnu/store/h6681qq9s5kqjvdzkhqhddqbjj3q48nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283/22CC.png", "/gnu/store/bf582s56ldb3y34p>statx(AT_FDCWD, "/gnu/store/.links/[...]my", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, 0, {stx_mask=STATX_ALL|>lstat("/gnu/store/.links/[...]my", {st_mode=S_IFLNK|0777, st_size=124, ...}) = 0lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", {st_mode=S_IFDIR|0555, st_size=4112, ...>chmod("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", 040755) = 0getpid() = 26371<end of log>
It seems like ‘duplication’ is taking a lot of time!(See nix/libstore/optimise-store.cc and (guix store deduplication).)
Toggle quote (10 lines)> IIUC, the hook completes almost instantaneously when creating a profile> without fonts because (null? fonts-dirs) is true.> > In other cases, one option to speed things up (and possibly remove the> hook’s dependency on mkfontdir) would be to create ‘fonts.dir’ and> ‘fonts.scale’ in a build phase of each font package, *provided* there’s> a cheap way to assemble several such files, such as mere concatenation.> That way, the profile hook would only need to assemble these files as> opposed to traversing all the font files.
Looking at ~/.guix-profile/share/fonts/truetype/fonts.dir,it seems to be simply a list of entries lke
DejaVuSans-Bold.ttf -misc-dejavu sans-bold-r-normal--0-0-0-0-p-0-koi8-ru
with additionally as first line the number of fonts. It appears to besorted by font file name (DejaVuSans-Bold.ttf) though, though that onlyseems to be to not depend on file system ordering. Not quite _mere_concatenation, but should be easy enough, I think.
(Likewise for fonts.scaale)
I would need to look at the code of "mkfontscale" and "mkfontdir" to be surethough.
Toggle quote (5 lines)> Given that ‘fonts.dir’ and ‘fonts.scale’ are mere text files with one> line per font, this approach looks doable.> > WDYT?
That seems a good idea, reducing the amount of 'stat'ing, 'readdir'ing,and 'read'ing (mkfontscale needs to actually read the font files)which should help on spinning disks (and other disks, but on SSDs it's probablyless noticable).
Toggle quote (25 lines)> > From 4fe1e30e33c01be9fd17cf240732b3351c7b0fa4 Mon Sep 17 00:00:00 2001> > From: Maxime Devos <maximedevos@telenet.be>> > Date: Mon, 5 Jul 2021 18:55:31 +0200> > Subject: [PATCH 1/2] profiles: Optimise 'fonts-dir-file'.> > > > Only let the build G-exp refer to inputs that might actually> > have fonts. That way, if the list of fonts in the manifest> > didn't change and the new manifest is built, the font files> > will not be rebuilt.> > > > * guix/profiles.scm> > (fonts-dir-file)[has-fonts?]: New predicate.> > (fonts-dir-file)[relevant-inputs]: New variable.> > (fonts-dir-file)[build]: Use 'relevant-inputs' instead of> > 'manifest-inputs'.> > * doc/contributing.texi (Fonts): Note the 'fonts-' naming> > convention is technically important now.> > * gnu/packages/ghostscript.scm (gs-fonts): Work-around the> > package name contravening the convention.> > To me this approach should be a last resort because it relies on> conventions and manual annotations, and it’s easy to forget these and> get incorrect results (we could end up having to deal with reports like> “I installed font X, why doesn’t in show up in the font list?”).
AFAIK, all font packages in Guix (except gs-fonts) are currentlycorrectly-named. They aren't going to spontanuously rename theirselves.So to me, the only potential problem seems to be _new_ font packagesignoring the conventions. But new packages should always be reviewed andtested, so such reports don't seem plausible to me.
(Also, the manual annotation would go away once 'gs-fonts' is renamed.I'll run "echo /gnu/store/*/share/fonts > atchie-fonts" though to seeif there are other packages not following the convention.)
Toggle quote (9 lines)> > + ;; In the upstream 'guix' channel, font packages should> > + ;; be named font-SOMETHING. But if another channel> > + ;; names its fonts differently but uses font-build-system,> > + ;; accepting that seems friendly.> > + (eq? 'font (build-system-name (package-build-system thing)))> > In general ‘build-system-name’ shouldn’t be used; the name is meant to> be a debugging aid.
I could do (eq? font-build-system (packag-build-system thing)) insteador drop this clause.
Toggle quote (2 lines)> Thanks,> Ludo’.
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYPBiIBccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7vYQAP4p5PTgHgfrNPzE6lB/cIAg8WQP2UTmyDLOQNOQ8qnVLwEAvkqoSME3w0PP5KnZloAY5BsT0wd2rzaWpBkRzWE4NA4==1Vm6-----END PGP SIGNATURE-----

M
M
Maxime Devos wrote on 15 Jul 20:48 +0200
Re: [bug#49421] [PATCH] profiles: Optimise 'fonts-dir-file'.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 49421@debbugs.gnu.org)
369206a276e225478ed1bbce0b883255dccef369.camel@telenet.be
Toggle quote (4 lines)> (Also, the manual annotation would go away once 'gs-fonts' is renamed.> I'll run "echo /gnu/store/*/share/fonts > atchie-fonts" though to see> if there are other packages not following the convention.)
Scheme code:
(use-modules (ice-9 ftw) (srfi srfi-1))(define (has-share/fonts? o) (file-exists? (string-append "/gnu/store/" o "/share/fonts")))(define t (scandir "/gnu/store" has-share/fonts?))(define t2 (map (lambda (x) (substring x 33)) t))(define t3 (delete-duplicates t2))(define (fonty? x) (string-prefix? "font-" x))(partition (lambda (x) (string-prefix? "font-" x)) t3)
Output on my system:
$1 = ("font-alias-1.0.3" [...])$2 = ("fonts-dir" "profile" "mate-1.24.1" "denemo-2.5.0" "mate-1.24.0" "gs-fonts-8.11")
"fonts-dir", "profile" and "gs-fonts" are expected.
"mate" uses "union-build" and has "font-cantarell" in 'inputs'.Apparently "denemo" has some font files as well.Something will need to be figured out for these cases,if the approach of my patch is followed.
I think it's better to optimise 'union-build' and the store deduplicationcode though (maybe subdirectories can be process concurrently, to maximise I/Outilisation?). That should improve performance outside the 'fonts-dir-file' hookas well.
Greetings,Maxime.
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYPCC6RccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7sFIAQCihDN5aMkr4uoKyseSCramBWrf1uRwf2AHfKk6o2Qx5AD/TJygfcxSi/UZQimJKTtBuduiomtY1ouxpVaqkYDrkwY==iA5C-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 16 Jul 18:05 +0200
Re: bug#49421: [PATCH] profiles: Optimise 'fonts-dir-file'.
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 49421@debbugs.gnu.org)
87pmviw8f6.fsf@gnu.org
Hi!
Maxime Devos <maximedevos@telenet.be> skribis:
Toggle quote (2 lines)> Ludovic Courtès schreef op wo 07-07-2021 om 18:52 [+0200]:
[...]
Toggle quote (7 lines)>> Do you know what’s taking time in the ‘fonts-dir-file’ hook?>> (On my x86_64 laptop with SSD, it runs in 4.4s on a cold cache for a>> profile containing 14 font packages, and in 1.2s on a warm cache.)>> I noticed ‘building fonts directories’ was taking long (9 minutes or so,> when using "guix package -r two packages that don't have fonts"),
Ouch, that’s way more than what I’m observing. :-) Do you have amanifest or package list to reproduce that? Were those timings on aspinning disk?
Toggle quote (10 lines)> so I took the opportunity to "strace" the process, though I was a little late.> Some statistics:>> % time seconds usecs/call calls errors syscall> ------ ----------- ----------- --------- --------- ----------------> 27,14 1,676551 117 14297 rename> 19,60 1,211012 13 90829 8 lstat> 18,34 1,133182 63 17787 15 statx> 11,80 0,729185 51 14297 link
[...]
Toggle quote (9 lines)> statx(AT_FDCWD, "/gnu/store/.links/[...]my", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, 0, {stx_mask=STATX_ALL|>> lstat("/gnu/store/.links/[...]my", {st_mode=S_IFLNK|0777, st_size=124, ...}) = 0> lstat("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", {st_mode=S_IFDIR|0555, st_size=4112, ...>> chmod("/gnu/store/[...]nm-fonts-dir/share/fonts/mathjax/HTML-CSS/TeX/png/AMS/Regular/283", 040755) = 0> getpid() = 26371> <end of log>>> It seems like ‘duplication’ is taking a lot of time!
It surely does but we need more data before we can draw a conclusion(the system time above accounts for ~6 seconds, not 9 minutes).
Anyway, it looks like we should be able to push most of the I/O work topackages. Let’s see!
Thanks,Ludo’.
M
M
Maxime Devos wrote on 17 Jul 16:42 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 49421@debbugs.gnu.org)
5af3b4c9975873800e371da77c6ba929f7bd90d7.camel@telenet.be
Ludovic Courtès schreef op vr 16-07-2021 om 18:05 [+0200]:
Toggle quote (4 lines)> Ouch, that’s way more than what I’m observing. :-) Do you have a> manifest or package list to reproduce that? Were those timings on a> spinning disk?
It is on a spinning disk. I attached the output of "guix package --export-manifest".
Greetings,Maxime.
;; Dit ‘manifest’ kan worden doorgegeven aan ‘guix package -m’ om de inhoud;; van je profiel te reproduceren. Dit is ‘symbolisch’: het verwijst enkel;; naar de pakketnamen. Om precies hetzelfde profiel te reproduceren, moet;; je ook de lijst van gebruikte kanalen bewaren, zoals teruggegeven door ‘guix describe’.;; Zie de sectie ‘Guix reproduceren’ in de handleiding.
(specifications->manifest (list "jami-gnome" "texmacs" "git:send-email" "lagrange" "vlc" "blueman" "bitlbee-purple" "network-manager-openconnect" "audacity" "evolution" "kiwix-desktop" "icecat" "simple-scan" "guile-studio" "ungoogled-chromium" "pidgin-otr" "emacs-calibredb" "calibre" "btrfs-progs" "tiled" "cheese" "seahorse" "wireshark" "git" "reuse" "gzochi" "zbar" "texmaker" "gnumeric" "gpa" "texlive-base" "gimp" "pinentry" "emacs-exwm" "geany" "sky" "texlive" "gnurobots" "graphviz" "guile-sdl" "guile-chickadee" "emacs" "emacs-magit-org-todos-el" "emacs-magit-annex" "emacs-magit-gerrit" "emacs-magit" "tome4" "emacs-guix" "zile-on-guile" "emacs-debbugs" "emacs-helm-bibtex" "emacs-ebdb" "emacs-helm-firefox" "emacs-helm-eww" "emacs-helm" "emacs-helm-wikipedia" "man-pages" "emacs-nov-el" "gnupg" "rr" "openssh" "irssi" "dee" "emacs-bluetooth" "guile-cairo" "xdg-dbus-proxy" "fetchmail" "libimobiledevice" "emacs-ledger-mode" "gdb" "valgrind" "plotutils" "xdg-utils" "artanis" "ghostscript" "dbus:doc" "dbus" "haunt" "nethack" "gcc-toolchain" "emacs-company-coq" "emacs-flycheck" "emacs-flycheck-guile" "emacs-ghub" "emacs-build-farm" "emacs-ebib" "emacs-company-math" "cryptsetup" "emacs-paredit" "emacs-ac-geiser" "emacs-geiser" "emacs-gcmh" "emacs-julia-mode" "emacs-yasnippet" "emacs-ox-epub" "emacs-bongo" "emacs-elf-mode" "emacs-js2-mode" "emacs-company-ebdb" "emacs-bbdb" "emacs-ffap-rfc-space" "z3" "texlive-latex-babel" "texlive-latex-geometry" "texlive-latex-hyperref" "texlive-latex-pgf" "texlive-latex-graphics" "glibc:debug" "glibc" "guile:debug" "guile" "aspell" "aspell-dict-de" "aspell-dict-nl" "aspell-dict-fr" "aspell-dict-en" "xxd" "go-ipfs" "unzip" "screen" "libreoffice" "htop" "scsh" "guile-miniadapton" "sicp" "mig" "texlive-epsf" "font-mathjax" "guile-ac-d-bus" "gash" "zip" "jacal" "gambit-c" "strace" "font-dejavu" "guile-fibers" "pkg-config" "ncurses"))
L
L
Ludovic Courtès wrote on 21 Jul 16:48 +0200
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 49421@debbugs.gnu.org)
875yx3hgdw.fsf_-_@gnu.org
Hello,
Maxime Devos <maximedevos@telenet.be> skribis:
Toggle quote (7 lines)> Ludovic Courtès schreef op vr 16-07-2021 om 18:05 [+0200]:>> Ouch, that’s way more than what I’m observing. :-) Do you have a>> manifest or package list to reproduce that? Were those timings on a>> spinning disk?>> It is on a spinning disk. I attached the output of "guix package --export-manifest".
I isolated ‘fonts-dir.drv’ for that profile and here’s what I get with a~5yo SSD:
Toggle snippet (24 lines)$ time guix build /gnu/store/nmr6clkx4n554a0gynr6sb4a43dj6mfa-fonts-dir.drv --checkThe following profile hook will be built: /gnu/store/nmr6clkx4n554a0gynr6sb4a43dj6mfa-fonts-dir.drvbuilding fonts directory...successfully built /gnu/store/nmr6clkx4n554a0gynr6sb4a43dj6mfa-fonts-dir.drvsuccessfully built /gnu/store/nmr6clkx4n554a0gynr6sb4a43dj6mfa-fonts-dir.drv/gnu/store/fgjmwpgrmp9m83qxvqrs1ghwasjrjxpc-fonts-dir
real 0m24.651suser 0m2.120ssys 0m0.164s$ time guix build /gnu/store/nmr6clkx4n554a0gynr6sb4a43dj6mfa-fonts-dir.drv --checkThe following profile hook will be built: /gnu/store/nmr6clkx4n554a0gynr6sb4a43dj6mfa-fonts-dir.drvbuilding fonts directory...successfully built /gnu/store/nmr6clkx4n554a0gynr6sb4a43dj6mfa-fonts-dir.drvsuccessfully built /gnu/store/nmr6clkx4n554a0gynr6sb4a43dj6mfa-fonts-dir.drv/gnu/store/fgjmwpgrmp9m83qxvqrs1ghwasjrjxpc-fonts-dir
real 0m12.025suser 0m1.924ssys 0m0.036s
That’s fairly significant (it baffles me that it’s still 20x less thanwhat you see with a spinning disk!).
There are only 327 fonts in there:
Toggle snippet (6 lines)$ wc -l /gnu/store/fgjmwpgrmp9m83qxvqrs1ghwasjrjxpc-fonts-dir/share/fonts/truetype/fonts.* 328 /gnu/store/fgjmwpgrmp9m83qxvqrs1ghwasjrjxpc-fonts-dir/share/fonts/truetype/fonts.dir 328 /gnu/store/fgjmwpgrmp9m83qxvqrs1ghwasjrjxpc-fonts-dir/share/fonts/truetype/fonts.scale 656 totalo
Thanks,Ludo’.
L
L
Ludovic Courtès wrote on 21 Jul 16:58 +0200
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 49421@debbugs.gnu.org)
87y29zg1ct.fsf_-_@gnu.org
Hi,
Maxime Devos <maximedevos@telenet.be> skribis:
Toggle quote (5 lines)> I think it's better to optimise 'union-build' and the store deduplication> code though (maybe subdirectories can be process concurrently, to maximise I/O> utilisation?). That should improve performance outside the 'fonts-dir-file' hook> as well.
I’m afraid there’s little room for improvement on ‘union-build’ (Markput quite some effort on it back in commit12129998689648923b58c426362a1bc875da75f9), though with a fresh eyeperhaps you’ll find optimization opportunities.
As for deduplication, there are two cases: deduplication forsubstitutes, and deduplication upon build completion (what we seehere).
In the former case, deduplication is part of the substitute unpackingpipeline (seehttps://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/)and I think syscalls are reduced to the minimum necessary (that’s (guixstore deduplication)).
The strace snippet you sent suggests that, in the latter case, thedaemon’s code stats the same file several times, so maybe there’s roomfor improvement here.
Thanks,Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 49421@debbugs.gnu.org