[PATCH] maint: Factorize po xref translation.

DoneSubmitted by Julien Lepiller.
Details
2 participants
  • Julien Lepiller
  • pelzflorian (Florian Pelz)
Owner
unassigned
Severity
normal
J
J
Julien Lepiller wrote on 10 Oct 23:32 +0200
(address . guix-patches@gnu.org)
8a210e8ceaadfa600e60b0b97a1e7aba04b2bb02.1633901513.git.julien@lepiller.eu
This ensures we use the same method in "make" as in "guix/self.scm".
* Makefile.am: Build guix/build/po.scm.* build-aux/convert-xref.scm: New file.* doc/local.mk (xref_command): Use it.* guix/self.scm (translate-cross-references): Move it...* guix/build/po.scm (translate-cross-references): ...here.--- Makefile.am | 3 +- build-aux/convert-xref.scm | 26 +++++++++ doc/local.mk | 29 +++------- guix/build/po.scm | 115 +++++++++++++++++++++++++++++++++++-- guix/self.scm | 80 ++++---------------------- 5 files changed, 155 insertions(+), 98 deletions(-) create mode 100644 build-aux/convert-xref.scm
Toggle diff (369 lines)diff --git a/Makefile.am b/Makefile.amindex 635147efc1..dc53b2c810 100644--- a/Makefile.am+++ b/Makefile.am@@ -69,7 +69,6 @@ nodist_noinst_SCRIPTS = \ # Modules that are not compiled but are installed nonetheless, such as # build-side modules with unusual dependencies. MODULES_NOT_COMPILED = \- guix/build/po.scm \ guix/man-db.scm include gnu/local.mk@@ -227,6 +226,7 @@ MODULES = \ guix/build/pack.scm \ guix/build/utils.scm \ guix/build/union.scm \+ guix/build/po.scm \ guix/build/profiles.scm \ guix/build/compile.scm \ guix/build/rpath.scm \@@ -640,6 +640,7 @@ EXTRA_DIST += \ build-aux/check-final-inputs-self-contained.scm \ build-aux/check-channel-news.scm \ build-aux/compile-as-derivation.scm \+ build-aux/convert-xref.scm \ build-aux/generate-authors.scm \ build-aux/test-driver.scm \ build-aux/update-guix-package.scm \diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scmnew file mode 100644index 0000000000..47c8828857--- /dev/null+++ b/build-aux/convert-xref.scm@@ -0,0 +1,26 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++;; Translate cross-references in a translated .texi manual.++(use-modules (guix build po)+ (ice-9 match))++(match (command-line)+ ((program texi pofile)+ (translate-cross-references texi pofile)))diff --git a/doc/local.mk b/doc/local.mkindex 8340b75a87..88f7e67155 100644--- a/doc/local.mk+++ b/doc/local.mk@@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete PO4A_PARAMS += -f texinfo # texinfo format # When a change to guix.texi occurs, it is not translated immediately.-# Because @pxref and @xref commands are reference to a section by name, they+# Because @pxref and @xref commands are references to sections by name, they # should be translated. If a modification adds a reference to a section, this # reference is not translated, which means it references a section that does not # exist.-# This command loops through the translated files looking for references. For-# each of these references, it tries to find the translation and replaces the-# reference name, even in untranslated strings.-# The last sed is a multiline sed because some references span multiple lines. define xref_command-cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \- tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \- sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \- if [ -n "$$e" ]; then \- line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\- ((line++)) ;\- if [ "$$line" != "1" ]; then \- translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\- if [ "$$translation" != "" ]; then \- sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\- fi ;\- fi ;\- fi ;\-done+$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \+ "$(top_srcdir)/build-aux/convert-xref.scm" \+ $@.tmp $< endef -$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi+$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po+$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po+$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@"diff --git a/guix/build/po.scm b/guix/build/po.scmindex eb9690ad1a..a167252244 100644--- a/guix/build/po.scm+++ b/guix/build/po.scm@@ -20,17 +20,23 @@ (define-module (guix build po) #:use-module (ice-9 match) #:use-module (ice-9 peg)+ #:use-module (ice-9 regex) #:use-module (ice-9 textual-ports)- #:export (read-po-file))+ #:use-module (ice-9 vlist)+ #:use-module (srfi srfi-1)+ #:export (read-po-file+ translate-cross-references)) ;; A small parser for po files-(define-peg-pattern po-file body (* (or comment entry whitespace)))+(define-peg-pattern po-file body (* (or entry whitespace))) (define-peg-pattern whitespace body (or " " "\t" "\n")) (define-peg-pattern comment-chr body (range #\space #\頋)) (define-peg-pattern comment none (and "#" (* comment-chr) "\n"))+(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n"))) (define-peg-pattern entry all- (and (ignore (* whitespace)) (ignore "msgid ") msgid- (ignore (* whitespace)) (ignore "msgstr ") msgstr))+ (and (* (or flags comment (ignore (* whitespace))))+ (ignore "msgid ") msgid (ignore (* whitespace))+ (ignore "msgstr ") msgstr)) (define-peg-pattern escape body (or "\\\\" "\\\"" "\\n")) (define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"") "\\n" (and (ignore "\\") "\\")@@ -53,7 +59,24 @@ (append (list "\n" prefix) result))))))) (define (parse-tree->assoc parse-tree)- "Converts a po PARSE-TREE to an association list."+ "Converts a po PARSE-TREE to an association list, where the key is the msgid+and the value is the msgstr. The result only contains non fuzzy strings."+ (define (comments->flags comments)+ (match comments+ (('flags flags)+ (map (lambda (flag) (string->symbol (string-trim-both flag #\space)))+ (string-split flags #\,)))+ ((? list? comments)+ (fold+ (lambda (comment res)+ (match comment+ ((? string? _) res)+ (flags+ (append (comments->flags flags)+ res))))+ '()+ comments))))+ (match parse-tree (() '()) ((entry . parse-tree)@@ -66,10 +89,22 @@ ;; empty msgstr (('entry ('msgid msgid) 'msgstr) (parse-tree->assoc parse-tree))+ (('entry _ ('msgid msgid) 'msgstr)+ (parse-tree->assoc parse-tree)) (('entry ('msgid msgid) ('msgstr msgstr)) (acons (interpret-newline-escape msgid) (interpret-newline-escape msgstr)- (parse-tree->assoc parse-tree)))))))+ (parse-tree->assoc parse-tree)))+ (('entry ('msgid msgid) ('msgstr msgstr))+ (acons (interpret-newline-escape msgid)+ (interpret-newline-escape msgstr)+ (parse-tree->assoc parse-tree)))+ (('entry comments ('msgid msgid) ('msgstr msgstr))+ (if (member 'fuzzy (comments->flags comments))+ (parse-tree->assoc parse-tree)+ (acons (interpret-newline-escape msgid)+ (interpret-newline-escape msgstr)+ (parse-tree->assoc parse-tree)))))))) (define (read-po-file port) "Read a .po file from PORT and return an alist of msgid and msgstr."@@ -77,3 +112,71 @@ po-file (get-string-all port))))) (parse-tree->assoc tree)))++(define (canonicalize-whitespace str)+ "Change whitespace (newlines, etc.) in STR to @code{#\\space}."+ (string-map (lambda (chr)+ (if (char-set-contains? char-set:whitespace chr)+ #\space+ chr))+ str))++(define xref-regexp+ ;; Texinfo cross-reference regexp.+ (make-regexp "@(px|x)?ref\\{([^,}]+)"))++(define (translate-cross-references texi pofile)+ "Translate the cross-references that appear in @var{texi}, the initial+translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}."+ (define translations+ (call-with-input-file pofile read-po-file))++ (define content+ (call-with-input-file texi get-string-all))++ (define matches+ (list-matches xref-regexp content))++ (define translation-map+ (fold (match-lambda*+ (((msgid . str) result)+ (vhash-cons msgid str result)))+ vlist-null+ translations))++ (define translated+ ;; Iterate over MATCHES and replace cross-references with their+ ;; translation found in TRANSLATION-MAP. (We can't use+ ;; 'substitute*' because matches can span multiple lines.)+ (let loop ((matches matches)+ (offset 0)+ (result '()))+ (match matches+ (()+ (string-concatenate-reverse+ (cons (string-drop content offset) result)))+ ((head . tail)+ (let ((prefix (match:substring head 1))+ (ref (canonicalize-whitespace (match:substring head 2))))+ (define translated+ (string-append "@" (or prefix "")+ "ref{"+ (match (vhash-assoc ref translation-map)+ (#f ref)+ ((_ . str) str))))++ (loop tail+ (match:end head)+ (append (list translated+ (string-take+ (string-drop content offset)+ (- (match:start head) offset)))+ result)))))))++ (format (current-error-port)+ "translated ~a cross-references in '~a'~%"+ (length matches) texi)++ (call-with-output-file texi+ (lambda (port)+ (display translated port))))diff --git a/guix/self.scm b/guix/self.scmindex 61ff423086..769e0e8a7c 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -316,81 +316,23 @@ the result to OUTPUT." chr)) str)) - (define xref-regexp- ;; Texinfo cross-reference regexp.- (make-regexp "@(px|x)?ref\\{([^,}]+)"))-- (define (translate-cross-references texi translations)- ;; Translate the cross-references that appear in TEXI, a Texinfo- ;; file, using the msgid/msgstr pairs from TRANSLATIONS.- (define content- (call-with-input-file texi get-string-all))-- (define matches- (list-matches xref-regexp content))-- (define translation-map- (fold (match-lambda*- (((msgid . str) result)- (vhash-cons msgid str result)))- vlist-null- translations))-- (define translated- ;; Iterate over MATCHES and replace cross-references with their- ;; translation found in TRANSLATION-MAP. (We can't use- ;; 'substitute*' because matches can span multiple lines.)- (let loop ((matches matches)- (offset 0)- (result '()))- (match matches- (()- (string-concatenate-reverse- (cons (string-drop content offset) result)))- ((head . tail)- (let ((prefix (match:substring head 1))- (ref (canonicalize-whitespace (match:substring head 2))))- (define translated- (string-append "@" (or prefix "")- "ref{"- (match (vhash-assoc ref translation-map)- (#f ref)- ((_ . str) str))))-- (loop tail- (match:end head)- (append (list translated- (string-take- (string-drop content offset)- (- (match:start head) offset)))- result)))))))-- (format (current-error-port)- "translated ~a cross-references in '~a'~%"- (length matches) texi)- (call-with-output-file texi- (lambda (port)- (display translated port))))- (define* (translate-texi prefix po lang #:key (extras '())) "Translate the manual for one language LANG using the PO file. PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is a list of extra files, such as '(\"contributing\")."- (let ((translations (call-with-input-file po read-po-file)))- (for-each (lambda (file)- (translate-tmp-texi po (string-append file ".texi")- (string-append file "." lang- ".texi.tmp")))- (cons prefix extras))+ (for-each (lambda (file)+ (translate-tmp-texi po (string-append file ".texi")+ (string-append file "." lang+ ".texi.tmp")))+ (cons prefix extras)) - (for-each (lambda (file)- (let* ((texi (string-append file "." lang ".texi"))- (tmp (string-append texi ".tmp")))- (copy-file tmp texi)- (translate-cross-references texi- translations)))- (cons prefix extras))))+ (for-each (lambda (file)+ (let* ((texi (string-append file "." lang ".texi"))+ (tmp (string-append texi ".tmp")))+ (copy-file tmp texi)+ (translate-cross-references texi po)))+ (cons prefix extras))) (define (available-translations directory domain) ;; Return the list of available translations under DIRECTORY for-- 2.33.0
J
J
Julien Lepiller wrote on 10 Oct 23:37 +0200
(address . 51122@debbugs.gnu.org)
20211010233711.23bb1e9e@tachikoma.lepiller.eu
For info, I built "make" and "make as-derivation" successfully withthis patch. I need it because with recent changes in guix.texi, a lotof text has become fuzzy, including node names, which are translated bythe current xref_command (in doc/local.mk), when they shouldn't. Iimproved the method we used for guix pull, and used it instead.
P
P
pelzflorian (Florian Pelz) wrote on 11 Oct 10:45 +0200
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 51122@debbugs.gnu.org)
20211011082415.j6wera4b5t4mtnxa@pelzflorian.localdomain
Hi Julien! Thank you for replacing that hard-to-read sed script.65;6003;1cOn Sun, Oct 10, 2021 at 11:37:11PM +0200, Julien Lepiller wrote:
Toggle quote (6 lines)> For info, I built "make" and "make as-derivation" successfully with> this patch. I need it because with recent changes in guix.texi, a lot> of text has become fuzzy, including node names, which are translated by> the current xref_command (in doc/local.mk), when they shouldn't. I> improved the method we used for guix pull, and used it instead.
I think you should add to the commit message that your patch Fixesfuzzy translations being inserted.
Your changes to guix/build/po.scm do more than just move code (whichyou mentioned in the commit message): you added parsing of “#, fuzzy”flags.
You should update the copyright header for guix/build/po.scm.
Otherwise LGTM, but I haven’t tested yet. (make now converts the xrefafter scheme modules are compiled, I think(?), so testing PO filestakes longer for me on a clean checkout. But it is fine.)
Regards,Florian
J
J
Julien Lepiller wrote on 11 Oct 14:46 +0200
Re: [bug#51122] [PATCH v2] maint: Factorize po xref translation.
(name . pelzflorian (Florian Pelz))(address . pelzflorian@pelzflorian.de)(address . 51122@debbugs.gnu.org)
20211011144626.0049af43@tachikoma.lepiller.eu
Le Mon, 11 Oct 2021 10:45:45 +0200,"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
Toggle quote (26 lines)> Hi Julien! Thank you for replacing that hard-to-read sed script.> 65;6003;1c> On Sun, Oct 10, 2021 at 11:37:11PM +0200, Julien Lepiller wrote:> > For info, I built "make" and "make as-derivation" successfully with> > this patch. I need it because with recent changes in guix.texi, a> > lot of text has become fuzzy, including node names, which are> > translated by the current xref_command (in doc/local.mk), when they> > shouldn't. I improved the method we used for guix pull, and used it> > instead. > > I think you should add to the commit message that your patch Fixes> fuzzy translations being inserted.> > Your changes to guix/build/po.scm do more than just move code (which> you mentioned in the commit message): you added parsing of “#, fuzzy”> flags.> > You should update the copyright header for guix/build/po.scm.> > Otherwise LGTM, but I haven’t tested yet. (make now converts the xref> after scheme modules are compiled, I think(?), so testing PO files> takes longer for me on a clean checkout. But it is fine.)> > Regards,> Florian
Thanks for the quick review! I've updated the commit message andupdated the copyright headers in guix/build/po.scm and guix/self.scm.
I had to make the texi files depend on make-go because otherwise, I getmessages like "guix/build/po.scm is newer than compiled.../guix/build/po.go" and it takes forever to build that file, once foreach texi. For some reason, it's almost instantaneous when building inmake-go.
From 75198f83fccf27b447e5eaaa6fb6caeed3b29408 Mon Sep 17 00:00:00 2001Message-Id: <75198f83fccf27b447e5eaaa6fb6caeed3b29408.1633955976.git.julien@lepiller.eu>From: Julien Lepiller <julien@lepiller.eu>Date: Sun, 10 Oct 2021 22:07:51 +0200Subject: [PATCH] maint: Factorize po xref translation.
This ensures we use the same method in "make" as in "guix/self.scm".
* Makefile.am: Build guix/build/po.scm.* build-aux/convert-xref.scm: New file.* doc/local.mk (xref_command): Use it.* guix/self.scm (translate-cross-references): Move it...* guix/build/po.scm: Parse comments and flags separately to find fuzzyflags.(translate-cross-references): ...here.(parse-tree->assoc): Ignore fuzzy entries.--- Makefile.am | 3 +- build-aux/convert-xref.scm | 26 +++++++++ doc/local.mk | 29 +++------ guix/build/po.scm | 117 ++++++++++++++++++++++++++++++++++--- guix/self.scm | 81 ++++--------------------- 5 files changed, 157 insertions(+), 99 deletions(-) create mode 100644 build-aux/convert-xref.scm
Toggle diff (384 lines)diff --git a/Makefile.am b/Makefile.amindex 635147efc1..dc53b2c810 100644--- a/Makefile.am+++ b/Makefile.am@@ -69,7 +69,6 @@ nodist_noinst_SCRIPTS = \ # Modules that are not compiled but are installed nonetheless, such as # build-side modules with unusual dependencies. MODULES_NOT_COMPILED = \- guix/build/po.scm \ guix/man-db.scm include gnu/local.mk@@ -227,6 +226,7 @@ MODULES = \ guix/build/pack.scm \ guix/build/utils.scm \ guix/build/union.scm \+ guix/build/po.scm \ guix/build/profiles.scm \ guix/build/compile.scm \ guix/build/rpath.scm \@@ -640,6 +640,7 @@ EXTRA_DIST += \ build-aux/check-final-inputs-self-contained.scm \ build-aux/check-channel-news.scm \ build-aux/compile-as-derivation.scm \+ build-aux/convert-xref.scm \ build-aux/generate-authors.scm \ build-aux/test-driver.scm \ build-aux/update-guix-package.scm \diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scmnew file mode 100644index 0000000000..47c8828857--- /dev/null+++ b/build-aux/convert-xref.scm@@ -0,0 +1,26 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++;; Translate cross-references in a translated .texi manual.++(use-modules (guix build po)+ (ice-9 match))++(match (command-line)+ ((program texi pofile)+ (translate-cross-references texi pofile)))diff --git a/doc/local.mk b/doc/local.mkindex 8340b75a87..88f7e67155 100644--- a/doc/local.mk+++ b/doc/local.mk@@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete PO4A_PARAMS += -f texinfo # texinfo format # When a change to guix.texi occurs, it is not translated immediately.-# Because @pxref and @xref commands are reference to a section by name, they+# Because @pxref and @xref commands are references to sections by name, they # should be translated. If a modification adds a reference to a section, this # reference is not translated, which means it references a section that does not # exist.-# This command loops through the translated files looking for references. For-# each of these references, it tries to find the translation and replaces the-# reference name, even in untranslated strings.-# The last sed is a multiline sed because some references span multiple lines. define xref_command-cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \- tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \- sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \- if [ -n "$$e" ]; then \- line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\- ((line++)) ;\- if [ "$$line" != "1" ]; then \- translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\- if [ "$$translation" != "" ]; then \- sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\- fi ;\- fi ;\- fi ;\-done+$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \+ "$(top_srcdir)/build-aux/convert-xref.scm" \+ $@.tmp $< endef -$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi+$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po+$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po+$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po make-go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@"diff --git a/guix/build/po.scm b/guix/build/po.scmindex eb9690ad1a..7f88164cd8 100644--- a/guix/build/po.scm+++ b/guix/build/po.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2019 Julien Lepiller <julien@lepiller.eu>+;;; Copyright © 2019, 2021 Julien Lepiller <julien@lepiller.eu> ;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org> ;;; ;;; This file is part of GNU Guix.@@ -20,17 +20,23 @@ (define-module (guix build po) #:use-module (ice-9 match) #:use-module (ice-9 peg)+ #:use-module (ice-9 regex) #:use-module (ice-9 textual-ports)- #:export (read-po-file))+ #:use-module (ice-9 vlist)+ #:use-module (srfi srfi-1)+ #:export (read-po-file+ translate-cross-references)) ;; A small parser for po files-(define-peg-pattern po-file body (* (or comment entry whitespace)))+(define-peg-pattern po-file body (* (or entry whitespace))) (define-peg-pattern whitespace body (or " " "\t" "\n")) (define-peg-pattern comment-chr body (range #\space #\頋)) (define-peg-pattern comment none (and "#" (* comment-chr) "\n"))+(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n"))) (define-peg-pattern entry all- (and (ignore (* whitespace)) (ignore "msgid ") msgid- (ignore (* whitespace)) (ignore "msgstr ") msgstr))+ (and (* (or flags comment (ignore (* whitespace))))+ (ignore "msgid ") msgid (ignore (* whitespace))+ (ignore "msgstr ") msgstr)) (define-peg-pattern escape body (or "\\\\" "\\\"" "\\n")) (define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"") "\\n" (and (ignore "\\") "\\")@@ -53,7 +59,24 @@ (append (list "\n" prefix) result))))))) (define (parse-tree->assoc parse-tree)- "Converts a po PARSE-TREE to an association list."+ "Converts a po PARSE-TREE to an association list, where the key is the msgid+and the value is the msgstr. The result only contains non fuzzy strings."+ (define (comments->flags comments)+ (match comments+ (('flags flags)+ (map (lambda (flag) (string->symbol (string-trim-both flag #\space)))+ (string-split flags #\,)))+ ((? list? comments)+ (fold+ (lambda (comment res)+ (match comment+ ((? string? _) res)+ (flags+ (append (comments->flags flags)+ res))))+ '()+ comments))))+ (match parse-tree (() '()) ((entry . parse-tree)@@ -66,10 +89,22 @@ ;; empty msgstr (('entry ('msgid msgid) 'msgstr) (parse-tree->assoc parse-tree))+ (('entry _ ('msgid msgid) 'msgstr)+ (parse-tree->assoc parse-tree)) (('entry ('msgid msgid) ('msgstr msgstr)) (acons (interpret-newline-escape msgid) (interpret-newline-escape msgstr)- (parse-tree->assoc parse-tree)))))))+ (parse-tree->assoc parse-tree)))+ (('entry ('msgid msgid) ('msgstr msgstr))+ (acons (interpret-newline-escape msgid)+ (interpret-newline-escape msgstr)+ (parse-tree->assoc parse-tree)))+ (('entry comments ('msgid msgid) ('msgstr msgstr))+ (if (member 'fuzzy (comments->flags comments))+ (parse-tree->assoc parse-tree)+ (acons (interpret-newline-escape msgid)+ (interpret-newline-escape msgstr)+ (parse-tree->assoc parse-tree)))))))) (define (read-po-file port) "Read a .po file from PORT and return an alist of msgid and msgstr."@@ -77,3 +112,71 @@ po-file (get-string-all port))))) (parse-tree->assoc tree)))++(define (canonicalize-whitespace str)+ "Change whitespace (newlines, etc.) in STR to @code{#\\space}."+ (string-map (lambda (chr)+ (if (char-set-contains? char-set:whitespace chr)+ #\space+ chr))+ str))++(define xref-regexp+ ;; Texinfo cross-reference regexp.+ (make-regexp "@(px|x)?ref\\{([^,}]+)"))++(define (translate-cross-references texi pofile)+ "Translate the cross-references that appear in @var{texi}, the initial+translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}."+ (define translations+ (call-with-input-file pofile read-po-file))++ (define content+ (call-with-input-file texi get-string-all))++ (define matches+ (list-matches xref-regexp content))++ (define translation-map+ (fold (match-lambda*+ (((msgid . str) result)+ (vhash-cons msgid str result)))+ vlist-null+ translations))++ (define translated+ ;; Iterate over MATCHES and replace cross-references with their+ ;; translation found in TRANSLATION-MAP. (We can't use+ ;; 'substitute*' because matches can span multiple lines.)+ (let loop ((matches matches)+ (offset 0)+ (result '()))+ (match matches+ (()+ (string-concatenate-reverse+ (cons (string-drop content offset) result)))+ ((head . tail)+ (let ((prefix (match:substring head 1))+ (ref (canonicalize-whitespace (match:substring head 2))))+ (define translated+ (string-append "@" (or prefix "")+ "ref{"+ (match (vhash-assoc ref translation-map)+ (#f ref)+ ((_ . str) str))))++ (loop tail+ (match:end head)+ (append (list translated+ (string-take+ (string-drop content offset)+ (- (match:start head) offset)))+ result)))))))++ (format (current-error-port)+ "translated ~a cross-references in '~a'~%"+ (length matches) texi)++ (call-with-output-file texi+ (lambda (port)+ (display translated port))))diff --git a/guix/self.scm b/guix/self.scmindex 61ff423086..fc581becd4 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu> ;;; ;;; This file is part of GNU Guix. ;;;@@ -316,81 +317,23 @@ the result to OUTPUT." chr)) str)) - (define xref-regexp- ;; Texinfo cross-reference regexp.- (make-regexp "@(px|x)?ref\\{([^,}]+)"))-- (define (translate-cross-references texi translations)- ;; Translate the cross-references that appear in TEXI, a Texinfo- ;; file, using the msgid/msgstr pairs from TRANSLATIONS.- (define content- (call-with-input-file texi get-string-all))-- (define matches- (list-matches xref-regexp content))-- (define translation-map- (fold (match-lambda*- (((msgid . str) result)- (vhash-cons msgid str result)))- vlist-null- translations))-- (define translated- ;; Iterate over MATCHES and replace cross-references with their- ;; translation found in TRANSLATION-MAP. (We can't use- ;; 'substitute*' because matches can span multiple lines.)- (let loop ((matches matches)- (offset 0)- (result '()))- (match matches- (()- (string-concatenate-reverse- (cons (string-drop content offset) result)))- ((head . tail)- (let ((prefix (match:substring head 1))- (ref (canonicalize-whitespace (match:substring head 2))))- (define translated- (string-append "@" (or prefix "")- "ref{"- (match (vhash-assoc ref translation-map)- (#f ref)- ((_ . str) str))))-- (loop tail- (match:end head)- (append (list translated- (string-take- (string-drop content offset)- (- (match:start head) offset)))- result)))))))-- (format (current-error-port)- "translated ~a cross-references in '~a'~%"- (length matches) texi)- (call-with-output-file texi- (lambda (port)- (display translated port))))- (define* (translate-texi prefix po lang #:key (extras '())) "Translate the manual for one language LANG using the PO file. PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is a list of extra files, such as '(\"contributing\")."- (let ((translations (call-with-input-file po read-po-file)))- (for-each (lambda (file)- (translate-tmp-texi po (string-append file ".texi")- (string-append file "." lang- ".texi.tmp")))- (cons prefix extras))+ (for-each (lambda (file)+ (translate-tmp-texi po (string-append file ".texi")+ (string-append file "." lang+ ".texi.tmp")))+ (cons prefix extras)) - (for-each (lambda (file)- (let* ((texi (string-append file "." lang ".texi"))- (tmp (string-append texi ".tmp")))- (copy-file tmp texi)- (translate-cross-references texi- translations)))- (cons prefix extras))))+ (for-each (lambda (file)+ (let* ((texi (string-append file "." lang ".texi"))+ (tmp (string-append texi ".tmp")))+ (copy-file tmp texi)+ (translate-cross-references texi po)))+ (cons prefix extras))) (define (available-translations directory domain) ;; Return the list of available translations under DIRECTORY for-- 2.33.0
P
P
pelzflorian (Florian Pelz) wrote on 11 Oct 19:17 +0200
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 51122@debbugs.gnu.org)
20211011171603.3m7spi6bwlhpgjiv@pelzflorian.localdomain
Thanks Julien!
On Mon, Oct 11, 2021 at 02:46:26PM +0200, Julien Lepiller wrote:
Toggle quote (3 lines)> Thanks for the quick review! I've updated the commit message and> updated the copyright headers in guix/build/po.scm and guix/self.scm.
In guix/self.scm you only removed code. I don’t think a copyrightheader is needed there.

Toggle quote (6 lines)> I had to make the texi files depend on make-go because otherwise, I get> messages like "guix/build/po.scm is newer than compiled> .../guix/build/po.go" and it takes forever to build that file, once for> each texi. For some reason, it's almost instantaneous when building in> make-go.
make-core-go instead of make-go is enough in my testing, but somethingmore fine-grained would save time. Also POXREF is run twice for eachguix.??.texi file. Maybe I did something wrong, maybe it is yourpatch. Not sure. Testing takes long anyway.
Regards,Florian
J
J
Julien Lepiller wrote on 12 Oct 00:06 +0200
(name . pelzflorian (Florian Pelz))(address . pelzflorian@pelzflorian.de)(address . 51122@debbugs.gnu.org)
20211012000657.11daa7a3@tachikoma.lepiller.eu
Le Mon, 11 Oct 2021 19:17:28 +0200,"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
Toggle quote (10 lines)> Thanks Julien!> > On Mon, Oct 11, 2021 at 02:46:26PM +0200, Julien Lepiller wrote:> > Thanks for the quick review! I've updated the commit message and> > updated the copyright headers in guix/build/po.scm and> > guix/self.scm. > > In guix/self.scm you only removed code. I don’t think a copyright> header is needed there.
Alright, I removed the copyright line on my side
Toggle quote (12 lines)> > > I had to make the texi files depend on make-go because otherwise, I> > get messages like "guix/build/po.scm is newer than compiled> > .../guix/build/po.go" and it takes forever to build that file, once> > for each texi. For some reason, it's almost instantaneous when> > building in make-go. > > make-core-go instead of make-go is enough in my testing, but something> more fine-grained would save time. Also POXREF is run twice for each> guix.??.texi file. Maybe I did something wrong, maybe it is your> patch. Not sure. Testing takes long anyway.
It only runs it once here, but there are three files to translate:guix.xx.texi, guix-cookbook.xx.texi and contributing.xx.texi.
It looks like depending on make-go or make-core-go means we rebuild thetexi files everytime we run "make", so that's not great :/. I'll see ifI can isolate guix/build/po.scm (any ideas?).
Toggle quote (2 lines)> Regards,> Florian
P
P
pelzflorian (Florian Pelz) wrote on 12 Oct 17:56 +0200
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 51122@debbugs.gnu.org)
20211012155611.vbhkyxjmqlc3ja77@pelzflorian.localdomain
On Tue, Oct 12, 2021 at 12:06:57AM +0200, Julien Lepiller wrote:
Toggle quote (8 lines)> Le Mon, 11 Oct 2021 19:17:28 +0200,> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :> > Also POXREF is run twice for each> > guix.??.texi file. Maybe I did something wrong, maybe it is your> > patch. Not sure. Testing takes long anyway.> It only runs it once here, but there are three files to translate:> guix.xx.texi, guix-cookbook.xx.texi and contributing.xx.texi.
At least when depending on make-core-go instead of make-go, it doesPOXREF each file twice, in between printing “Updatingdoc/version.texi”. Anyway, this is more important:
Toggle quote (3 lines)> It looks like depending on make-go or make-core-go means we rebuild the> texi files everytime we run "make", so that's not great :/.
We definitely want to compile guix/build/po.scm, I think, because itshould be run each time a guix-manual.LL.po or guix-cookbook.LL.pofile changes.
GNU Make manual sayshttps://www.gnu.org/software/make/manual/html_node/Phony-Targets.html#Phony-Targets:
Toggle quote (4 lines)> A phony target should not be a prerequisite of a real target file; if> it is, its recipe will be run every time make goes to update that> file.
But Makefile.am defines guile-compilation-rule to create a Phony target.
This is bad for us because likely there should not be duplicate codeto compile guix/build/po.scm and to compile the rest.
So I presume part of guile-compilation-rule must be split off?
Regards,Florian
P
P
pelzflorian (Florian Pelz) wrote on 12 Oct 20:09 +0200
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 51122@debbugs.gnu.org)
20211012180941.i7c26gyz3yyqsdd6@pelzflorian.localdomain
On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz) wrote:
Toggle quote (4 lines)> We definitely want to compile guix/build/po.scm, I think, because it> should be run each time a guix-manual.LL.po or guix-cookbook.LL.po> file changes.
Then again, is calling guix without compiling necessarily slower thanthe sed script?
Regards,Florian
J
J
Julien Lepiller wrote on 12 Oct 21:25 +0200
(name . pelzflorian (Florian Pelz))(address . pelzflorian@pelzflorian.de)(address . 51122@debbugs.gnu.org)
5EFA87E6-D0D0-4D28-99D4-2CCC7E00EDD0@lepiller.eu
Le 12 octobre 2021 14:09:41 GMT-04:00, "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
Toggle quote (11 lines)>On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz) wrote:>> We definitely want to compile guix/build/po.scm, I think, because it>> should be run each time a guix-manual.LL.po or guix-cookbook.LL.po>> file changes.>>Then again, is calling guix without compiling necessarily slower than>the sed script?>>Regards,>Florian
Yes, each time I see the "newer than compiled" message, it takes several minutes.
J
J
Julien Lepiller wrote on 16 Oct 04:12 +0200
(name . pelzflorian (Florian Pelz))(address . pelzflorian@pelzflorian.de)(address . 51122@debbugs.gnu.org)
20211016041215.1005a12b@tachikoma.lepiller.eu
Le Tue, 12 Oct 2021 15:25:37 -0400,Julien Lepiller <julien@lepiller.eu> a écrit :
Toggle quote (20 lines)> Le 12 octobre 2021 14:09:41 GMT-04:00, "pelzflorian (Florian Pelz)"> <pelzflorian@pelzflorian.de> a écrit :> >On Tue, Oct 12, 2021 at 05:56:23PM +0200, pelzflorian (Florian Pelz)> >wrote: > >> We definitely want to compile guix/build/po.scm, I think, because> >> it should be run each time a guix-manual.LL.po or> >> guix-cookbook.LL.po file changes. > >> >Then again, is calling guix without compiling necessarily slower than> >the sed script?> >> >Regards,> >Florian > > Yes, each time I see the "newer than compiled" message, it takes> several minutes.> > >
I managed to change the dependency to only guix/po.go, and to make itnot phony, so the translated manuals don't get rebuilt on every makeinvocation anymore. I checked again that "make" from a clean checkoutand "make as-derivation" still work.
From 96d7096ac46f0fa8a97db76fefdf751a0584975b Mon Sep 17 00:00:00 2001From: Julien Lepiller <julien@lepiller.eu>Date: Sun, 10 Oct 2021 22:07:51 +0200Subject: [PATCH] maint: Factorize po xref translation.
This ensures we use the same method in "make" as in "guix/self.scm".
* Makefile.am: Build guix/build/po.scm.* build-aux/convert-xref.scm: New file.* doc/local.mk (xref_command): Use it.* guix/self.scm (translate-cross-references): Move it...* guix/build/po.scm: Parse comments and flags separately to find fuzzyflags.(translate-cross-references): ...here.(parse-tree->assoc): Ignore fuzzy entries.--- Makefile.am | 12 +++- build-aux/convert-xref.scm | 26 +++++++++ doc/local.mk | 29 +++------- guix/build/po.scm | 115 +++++++++++++++++++++++++++++++++++-- guix/self.scm | 81 ++++---------------------- 5 files changed, 164 insertions(+), 99 deletions(-) create mode 100644 build-aux/convert-xref.scm
Toggle diff (402 lines)diff --git a/Makefile.am b/Makefile.amindex 635147efc1..41ec19eb89 100644--- a/Makefile.am+++ b/Makefile.am@@ -640,6 +640,7 @@ EXTRA_DIST += \ build-aux/check-final-inputs-self-contained.scm \ build-aux/check-channel-news.scm \ build-aux/compile-as-derivation.scm \+ build-aux/convert-xref.scm \ build-aux/generate-authors.scm \ build-aux/test-driver.scm \ build-aux/update-guix-package.scm \@@ -699,8 +700,6 @@ $(1): $(2) --completed $(3) \ $$(filter %.scm,$$^) -.PHONY: $(1)- endef # Split compilation in several steps, each of which building a subset of@@ -712,22 +711,31 @@ MODULES_CORE = guix.scm $(filter-out guix/scripts/%,$(filter guix/%,$(MODULE MODULES_PACKAGES = $(filter gnu/packages/%,$(MODULES)) MODULES_SYSTEM = gnu.scm $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES))) MODULES_CLI = $(filter guix/scripts/%,$(MODULES))+MODULES_PO = guix/build/po.scm $(eval $(call guile-compilation-rule,make-core-go, \ $(MODULES_CORE) guix/config.scm $(dist_noinst_DATA), \ 0))+.PHONY: make-core-go $(eval $(call guile-compilation-rule,make-packages-go, \ $(MODULES_PACKAGES) make-core-go, \ $(words $(MODULES_CORE))))+.PHONY: make-packages-go $(eval $(call guile-compilation-rule,make-system-go, \ $(MODULES_SYSTEM) make-packages-go make-core-go, \ $(words $(MODULES_CORE) $(MODULES_PACKAGES))))+.PHONY: make-system-go $(eval $(call guile-compilation-rule,make-cli-go, \ $(MODULES_CLI) make-system-go make-packages-go make-core-go, \ $(words $(MODULES_CORE) $(MODULES_PACKAGES) $(MODULES_SYSTEM))))+.PHONY: make-cli-go++$(eval $(call guile-compilation-rule,guix/build/po.go, \+ $(MODULES_PO), \+ 0)) SUFFIXES = .go diff --git a/build-aux/convert-xref.scm b/build-aux/convert-xref.scmnew file mode 100644index 0000000000..47c8828857--- /dev/null+++ b/build-aux/convert-xref.scm@@ -0,0 +1,26 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++;; Translate cross-references in a translated .texi manual.++(use-modules (guix build po)+ (ice-9 match))++(match (command-line)+ ((program texi pofile)+ (translate-cross-references texi pofile)))diff --git a/doc/local.mk b/doc/local.mkindex 8340b75a87..fff11f8183 100644--- a/doc/local.mk+++ b/doc/local.mk@@ -97,44 +97,29 @@ PO4A_PARAMS += -k 0 # produce an output even if the translation is not complete PO4A_PARAMS += -f texinfo # texinfo format # When a change to guix.texi occurs, it is not translated immediately.-# Because @pxref and @xref commands are reference to a section by name, they+# Because @pxref and @xref commands are references to sections by name, they # should be translated. If a modification adds a reference to a section, this # reference is not translated, which means it references a section that does not # exist.-# This command loops through the translated files looking for references. For-# each of these references, it tries to find the translation and replaces the-# reference name, even in untranslated strings.-# The last sed is a multiline sed because some references span multiple lines. define xref_command-cat "$@.tmp" | egrep '@p?x?ref' -A1 | sed 'N;s|--\n||g;P;D' | sed 's|^| |g' | \- tr -d '\012' | sed 's|\(@p\?x\?ref\)|\n\1|g' | egrep '@p?x?ref' | \- sed 's|^.*@p\?x\?ref{\([^,}]*\).*$$|\1|g' | sort | uniq | while read e; do \- if [ -n "$$e" ]; then \- line=$$(grep -n "^msgid \"$$e\"" "$<" | cut -f1 --delimiter=":") ;\- ((line++)) ;\- if [ "$$line" != "1" ]; then \- translation=$$(head -n "$$line" "$<" | tail -1 | grep msgstr | sed 's|msgstr "\([^"]*\)"|\1|') ;\- if [ "$$translation" != "" ]; then \- sed "N;s@\(p\?x\?ref\){$$(echo $$e | sed 's| |[\\n ]|g')\(,\|}\)@\1{$$translation\2@g;P;D" -i "$@.tmp" ;\- fi ;\- fi ;\- fi ;\-done+$(top_srcdir)/pre-inst-env $(GUILE) --no-auto-compile \+ "$(top_srcdir)/build-aux/convert-xref.scm" \+ $@.tmp $< endef -$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi+$(srcdir)/%D%/guix.%.texi: po/doc/guix-manual.%.po $(srcdir)/%D%/contributing.%.texi guix/build/po.go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po+$(srcdir)/%D%/guix-cookbook.%.texi: po/doc/guix-cookbook.%.po guix/build/po.go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/guix-cookbook.texi" -p "$<" -l "$@.tmp" -sed -i "s|guix-cookbook\.info|$$(basename "$@" | sed 's|texi$$|info|')|" "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@" -$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po+$(srcdir)/%D%/contributing.%.texi: po/doc/guix-manual.%.po guix/build/po.go -$(AM_V_PO4A)$(PO4A_TRANSLATE) $(PO4A_PARAMS) -m "%D%/contributing.texi" -p "$<" -l "$@.tmp" -$(AM_V_POXREF)$(xref_command) -mv "$@.tmp" "$@"diff --git a/guix/build/po.scm b/guix/build/po.scmindex eb9690ad1a..a167252244 100644--- a/guix/build/po.scm+++ b/guix/build/po.scm@@ -20,17 +20,23 @@ (define-module (guix build po) #:use-module (ice-9 match) #:use-module (ice-9 peg)+ #:use-module (ice-9 regex) #:use-module (ice-9 textual-ports)- #:export (read-po-file))+ #:use-module (ice-9 vlist)+ #:use-module (srfi srfi-1)+ #:export (read-po-file+ translate-cross-references)) ;; A small parser for po files-(define-peg-pattern po-file body (* (or comment entry whitespace)))+(define-peg-pattern po-file body (* (or entry whitespace))) (define-peg-pattern whitespace body (or " " "\t" "\n")) (define-peg-pattern comment-chr body (range #\space #\頋)) (define-peg-pattern comment none (and "#" (* comment-chr) "\n"))+(define-peg-pattern flags all (and (ignore "#, ") (* comment-chr) (ignore "\n"))) (define-peg-pattern entry all- (and (ignore (* whitespace)) (ignore "msgid ") msgid- (ignore (* whitespace)) (ignore "msgstr ") msgstr))+ (and (* (or flags comment (ignore (* whitespace))))+ (ignore "msgid ") msgid (ignore (* whitespace))+ (ignore "msgstr ") msgstr)) (define-peg-pattern escape body (or "\\\\" "\\\"" "\\n")) (define-peg-pattern str-chr body (or " " "!" (and (ignore "\\") "\"") "\\n" (and (ignore "\\") "\\")@@ -53,7 +59,24 @@ (append (list "\n" prefix) result))))))) (define (parse-tree->assoc parse-tree)- "Converts a po PARSE-TREE to an association list."+ "Converts a po PARSE-TREE to an association list, where the key is the msgid+and the value is the msgstr. The result only contains non fuzzy strings."+ (define (comments->flags comments)+ (match comments+ (('flags flags)+ (map (lambda (flag) (string->symbol (string-trim-both flag #\space)))+ (string-split flags #\,)))+ ((? list? comments)+ (fold+ (lambda (comment res)+ (match comment+ ((? string? _) res)+ (flags+ (append (comments->flags flags)+ res))))+ '()+ comments))))+ (match parse-tree (() '()) ((entry . parse-tree)@@ -66,10 +89,22 @@ ;; empty msgstr (('entry ('msgid msgid) 'msgstr) (parse-tree->assoc parse-tree))+ (('entry _ ('msgid msgid) 'msgstr)+ (parse-tree->assoc parse-tree)) (('entry ('msgid msgid) ('msgstr msgstr)) (acons (interpret-newline-escape msgid) (interpret-newline-escape msgstr)- (parse-tree->assoc parse-tree)))))))+ (parse-tree->assoc parse-tree)))+ (('entry ('msgid msgid) ('msgstr msgstr))+ (acons (interpret-newline-escape msgid)+ (interpret-newline-escape msgstr)+ (parse-tree->assoc parse-tree)))+ (('entry comments ('msgid msgid) ('msgstr msgstr))+ (if (member 'fuzzy (comments->flags comments))+ (parse-tree->assoc parse-tree)+ (acons (interpret-newline-escape msgid)+ (interpret-newline-escape msgstr)+ (parse-tree->assoc parse-tree)))))))) (define (read-po-file port) "Read a .po file from PORT and return an alist of msgid and msgstr."@@ -77,3 +112,71 @@ po-file (get-string-all port))))) (parse-tree->assoc tree)))++(define (canonicalize-whitespace str)+ "Change whitespace (newlines, etc.) in STR to @code{#\\space}."+ (string-map (lambda (chr)+ (if (char-set-contains? char-set:whitespace chr)+ #\space+ chr))+ str))++(define xref-regexp+ ;; Texinfo cross-reference regexp.+ (make-regexp "@(px|x)?ref\\{([^,}]+)"))++(define (translate-cross-references texi pofile)+ "Translate the cross-references that appear in @var{texi}, the initial+translation of a Texinfo file, using the msgid/msgstr pairs from @var{pofile}."+ (define translations+ (call-with-input-file pofile read-po-file))++ (define content+ (call-with-input-file texi get-string-all))++ (define matches+ (list-matches xref-regexp content))++ (define translation-map+ (fold (match-lambda*+ (((msgid . str) result)+ (vhash-cons msgid str result)))+ vlist-null+ translations))++ (define translated+ ;; Iterate over MATCHES and replace cross-references with their+ ;; translation found in TRANSLATION-MAP. (We can't use+ ;; 'substitute*' because matches can span multiple lines.)+ (let loop ((matches matches)+ (offset 0)+ (result '()))+ (match matches+ (()+ (string-concatenate-reverse+ (cons (string-drop content offset) result)))+ ((head . tail)+ (let ((prefix (match:substring head 1))+ (ref (canonicalize-whitespace (match:substring head 2))))+ (define translated+ (string-append "@" (or prefix "")+ "ref{"+ (match (vhash-assoc ref translation-map)+ (#f ref)+ ((_ . str) str))))++ (loop tail+ (match:end head)+ (append (list translated+ (string-take+ (string-drop content offset)+ (- (match:start head) offset)))+ result)))))))++ (format (current-error-port)+ "translated ~a cross-references in '~a'~%"+ (length matches) texi)++ (call-with-output-file texi+ (lambda (port)+ (display translated port))))diff --git a/guix/self.scm b/guix/self.scmindex a0d448742a..8ada1d85a4 100644--- a/guix/self.scm+++ b/guix/self.scm@@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu> ;;; ;;; This file is part of GNU Guix. ;;;@@ -316,81 +317,23 @@ the result to OUTPUT." chr)) str)) - (define xref-regexp- ;; Texinfo cross-reference regexp.- (make-regexp "@(px|x)?ref\\{([^,}]+)"))-- (define (translate-cross-references texi translations)- ;; Translate the cross-references that appear in TEXI, a Texinfo- ;; file, using the msgid/msgstr pairs from TRANSLATIONS.- (define content- (call-with-input-file texi get-string-all))-- (define matches- (list-matches xref-regexp content))-- (define translation-map- (fold (match-lambda*- (((msgid . str) result)- (vhash-cons msgid str result)))- vlist-null- translations))-- (define translated- ;; Iterate over MATCHES and replace cross-references with their- ;; translation found in TRANSLATION-MAP. (We can't use- ;; 'substitute*' because matches can span multiple lines.)- (let loop ((matches matches)- (offset 0)- (result '()))- (match matches- (()- (string-concatenate-reverse- (cons (string-drop content offset) result)))- ((head . tail)- (let ((prefix (match:substring head 1))- (ref (canonicalize-whitespace (match:substring head 2))))- (define translated- (string-append "@" (or prefix "")- "ref{"- (match (vhash-assoc ref translation-map)- (#f ref)- ((_ . str) str))))-- (loop tail- (match:end head)- (append (list translated- (string-take- (string-drop content offset)- (- (match:start head) offset)))- result)))))))-- (format (current-error-port)- "translated ~a cross-references in '~a'~%"- (length matches) texi)- (call-with-output-file texi- (lambda (port)- (display translated port))))- (define* (translate-texi prefix po lang #:key (extras '())) "Translate the manual for one language LANG using the PO file. PREFIX must be the prefix of the manual, 'guix' or 'guix-cookbook'. EXTRAS is a list of extra files, such as '(\"contributing\")."- (let ((translations (call-with-input-file po read-po-file)))- (for-each (lambda (file)- (translate-tmp-texi po (string-append file ".texi")- (string-append file "." lang- ".texi.tmp")))- (cons prefix extras))+ (for-each (lambda (file)+ (translate-tmp-texi po (string-append file ".texi")+ (string-append file "." lang+ ".texi.tmp")))+ (cons prefix extras)) - (for-each (lambda (file)- (let* ((texi (string-append file "." lang ".texi"))- (tmp (string-append texi ".tmp")))- (copy-file tmp texi)- (translate-cross-references texi- translations)))- (cons prefix extras))))+ (for-each (lambda (file)+ (let* ((texi (string-append file "." lang ".texi"))+ (tmp (string-append texi ".tmp")))+ (copy-file tmp texi)+ (translate-cross-references texi po)))+ (cons prefix extras))) (define (available-translations directory domain) ;; Return the list of available translations under DIRECTORY for-- 2.31.1
P
P
pelzflorian (Florian Pelz) wrote on 16 Oct 20:45 +0200
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 51122@debbugs.gnu.org)
20211016184502.x3gxejejudogx5l5@pelzflorian.localdomain
On Sat, Oct 16, 2021 at 04:12:15AM +0200, Julien Lepiller wrote:
Toggle quote (5 lines)> I managed to change the dependency to only guix/po.go, and to make it> not phony, so the translated manuals don't get rebuilt on every make> invocation anymore. I checked again that "make" from a clean checkout> and "make as-derivation" still work.
Good, thank you! I had thought it would be more complicated thanmoving out the PHONY directive, but all works OK:
Without patch, the fuzzy references cause make to fail for Weblate’sversion of the Spanish manual. With patch, make runs through fine asyou write. GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=yes makeupdate-guix-package && git commit and guix pull and still works, guixbuild guix failed in check but for unrelated reasons, so the patchLGTM. Except:
As we discussed before, you only remove code from guix/self.scm, soyou should not add a copyright line.
Regards,Florian
J
J
Julien Lepiller wrote on 17 Oct 22:12 +0200
(name . pelzflorian (Florian Pelz))(address . pelzflorian@pelzflorian.de)(address . 51122-done@debbugs.gnu.org)
20211017221229.071626be@tachikoma.lepiller.eu
Le Sat, 16 Oct 2021 20:45:02 +0200,"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> a écrit :
Toggle quote (22 lines)> On Sat, Oct 16, 2021 at 04:12:15AM +0200, Julien Lepiller wrote:> > I managed to change the dependency to only guix/po.go, and to make> > it not phony, so the translated manuals don't get rebuilt on every> > make invocation anymore. I checked again that "make" from a clean> > checkout and "make as-derivation" still work. > > Good, thank you! I had thought it would be more complicated than> moving out the PHONY directive, but all works OK:> > Without patch, the fuzzy references cause make to fail for Weblate’s> version of the Spanish manual. With patch, make runs through fine as> you write. GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=yes make> update-guix-package && git commit and guix pull and still works, guix> build guix failed in check but for unrelated reasons, so the patch> LGTM. Except:> > As we discussed before, you only remove code from guix/self.scm, so> you should not add a copyright line.> > Regards,> Florian
Thanks, pushed as 0623138ffa5b066afc25547ffdeb97753cb0ee9a, andfollowed by an nls update.
Closed
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 51122@debbugs.gnu.org