[PATCH] Makefile: Reimplement download-po target

DoneSubmitted by Julien Lepiller.
Details
2 participants
  • Julien Lepiller
  • Ludovic Courtès
Owner
unassigned
Severity
normal
J
J
Julien Lepiller wrote on 19 Apr 00:09 +0200
(address . guix-patches@gnu.org)
20210419000958.49968055@tachikoma.lepiller.eu
Hi Guix!
The attached patch reimplements the download-po target in Makefile.am.The previous implementation was based on the implementation for the TPand would use the API of Weblate to get files.
However, the rate-limit is very close to the number of files we need todownload, which was a big limitation, and this target would onlydownload existing files, ignoring new translations.
With this, new translations and existing ones are added/updated. Thetarget will first do a shallow clone of the repository behindWeblate. Then, it will check that files are not empty (contain at leastone translated string) and correct (according to scheme-format atleast). If they are correct, it will normalize them and copy them tothe po/ subdir in the repository.
Here is an example output:
copied po/doc/guix-manual.ru.po./tmp/tmp.dWvtATfWQA/translations/po/doc/guix-manual.si.po:6:AVERTISSEMENT¬†: Le champ d'en-tête « PO-Revision-Date » a encore savaleur initiale par défautWARN: po/doc/guix-manual.si.po (0 translated messages) was notadded/updated./tmp/tmp.dWvtATfWQA/translations/po/guix/nl.po:5135: Les spécificationsde format dans « msgid_plural » et «¬†msgstr[1]¬†» ne sont paséquivalentesmsgfmt: 1 erreur fatale trouvéeWARN: po/guix/nl.po (1001 translated messages) was not added/updated.
In this output, we can see that the Russian manual was copied, theSinhala manual was empty, so not copied, and the Dutch translation hadan issue in one string that needs to be fixed, and was not copied.
I chose not to copy the file because an issue in msgfmt would breakguix pull. Unfortunately, this is not enough testing, as the manual andcookbook are not checked. Checking them requires building the manual,so before commiting changes, please run "make".
To document the new process:
make download-pofor any new file in po/guix and po/packages, add the language in asingle line in po/guix/LINGUAS or po/packages/LINGUAS.for any new file in po/doc, add the file name to po/doc/local.mk, inDOC_PO_FILES or DOC_COOKBOOK_PO_FILES depending on the translationtype. Add the texi file name to doc/local.mk too, in info_TEXINFOS.
From 5c14506d6af24b5307e03604cabf8ca10af56067 Mon Sep 17 00:00:00 2001From: Julien Lepiller <julien@lepiller.eu>Date: Sun, 18 Apr 2021 23:56:48 +0200Subject: [PATCH] Makefile: Reimplement `download-po` target.
The weblate API rate limit is very close to the number of files we needto download. The previous implementation did not add new translations.
* Makefile.am (download-po): Update target.(make-download-po, make-check-po): Remove functions.--- Makefile.am | 87 ++++++++++++++--------------------------------------- 1 file changed, 22 insertions(+), 65 deletions(-)
Toggle diff (100 lines)diff --git a/Makefile.am b/Makefile.amindex 17ad236655..811e47dbfc 100644--- a/Makefile.am+++ b/Makefile.am@@ -969,71 +969,28 @@ cuirass-jobs: $(GOBJECTS) # Downloading up-to-date PO files. -# make-download-po-rule DOMAIN DIRECTORY [FILE-NAME-PREFIX]-define make-download-po-rule--download-po.$(1):- if [ -f "$(top_srcdir)/$(2)/LINGUAS" ]; then \- LINGUAS="`grep -v '^[[:blank:]]*#' < $(top_srcdir)/$(2)/LINGUAS`" ; \- else \- LINGUAS="`(cd $(top_srcdir)/$(2); \- for i in *.po; do echo $$$$i; done) | cut -d . -f 2`" ; \- fi ; \- for lang in $$$$LINGUAS; do \- if wget -nv -O "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" \- "https://translate.fedoraproject.org/api/translations/guix/$(1)/$$$$lang/file/" ; \- then \- msgfilter --no-wrap -i "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" \- cat > "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp2" ; \- rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ; \- mv "$(top_srcdir)/$(2)/$(3)$$$$lang.po"{.tmp2,} ; \- else \- rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ; \- fi ; \- done--.PHONY: download-po.$(1)--endef--# Checking po files for issues. This is useful to run after downloading new-# po files.--# make-check-po-rule DOMAIN DIRECTORY [FILE-NAME-PREFIX]-define make-check-po-rule--check-po.$(1):- if [ -f "$(top_srcdir)/$(2)/LINGUAS" ]; then \- LINGUAS="`grep -v '^[[:blank:]]*#' < $(top_srcdir)/$(2)/LINGUAS`" ; \- else \- LINGUAS="`(cd $(top_srcdir)/$(2); \- for i in *.po; do echo $$$$i; done) | cut -d . -f 2`" ; \- fi ; \- for lang in $$$$LINGUAS; do \- if [ -f "$(top_srcdir)/$(2)/$(3)$$$$lang.po" ]; \- then \- if ! msgfmt -c "$(top_srcdir)/$(2)/$(3)$$$$lang.po" ; \- then \- exit 1 ; \- fi ; \- fi ; \- done--.PHONY: check-po.$(1)--endef--$(eval $(call make-download-po-rule,documentation-cookbook,po/doc,guix-cookbook.))-$(eval $(call make-download-po-rule,documentation-manual,po/doc,guix-manual.))-$(eval $(call make-download-po-rule,guix,po/guix))-$(eval $(call make-download-po-rule,packages,po/packages))--$(eval $(call make-check-po-rule,documentation-cookbook,po/doc,guix-cookbook.))-$(eval $(call make-check-po-rule,documentation-manual,po/doc,guix-manual.))-$(eval $(call make-check-po-rule,guix,po/guix))-$(eval $(call make-check-po-rule,packages,po/packages))--download-po: $(foreach domain,guix packages documentation-manual documentation-cookbook,download-po.$(domain))+WEBLATE_REPO="https://framagit.org/tyreunom/guix-translations"++# shallow clone the git repository behind weblate and copy files from it if+# they contain at least one translation, and they are well-formed (scheme format+# only), warn otherwise. Copied files are converted to a canonical form.+download-po:+ dir=$$(mktemp -d); \+ git clone --depth 1 "$(WEBLATE_REPO)" "$$dir/translations"; \+ for domain in po/doc po/guix po/packages; do \+ for po in "$$dir/translations/$$domain"/*.po; do \+ translated=$$(LANG=en_US.UTF-8 msgfmt --statistics "$$po" 2>&1 | cut -f1 -d' '); \+ target=$$(basename "$$po"); \+ target="$$domain/$$target"; \+ if msgfmt -c "$$po" && [ "$$translated" != "0" ]; then \+ msgfilter --no-wrap -i "$$po" cat > "$$target"; \+ echo "copied $$target."; \+ else \+ echo "WARN: $$target ($$translated translated messages) was not added/updated."; \+ fi; \+ done; \+ done; \+ rm -rf "$$dir" .PHONY: download-po check-po: $(foreach domain,guix packages documentation-manual documentation-cookbook,check-po.$(domain))-- 2.31.1
L
L
Ludovic Courtès wrote on 30 Apr 21:57 +0200
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47882@debbugs.gnu.org)
87im43a6lq.fsf@gnu.org
Hi!
Julien Lepiller <julien@lepiller.eu> skribis:
Toggle quote (8 lines)> The attached patch reimplements the download-po target in Makefile.am.> The previous implementation was based on the implementation for the TP> and would use the API of Weblate to get files.>> However, the rate-limit is very close to the number of files we need to> download, which was a big limitation, and this target would only> download existing files, ignoring new translations.
OK.
Toggle quote (9 lines)> To document the new process:>> make download-po> for any new file in po/guix and po/packages, add the language in a> single line in po/guix/LINGUAS or po/packages/LINGUAS.> for any new file in po/doc, add the file name to po/doc/local.mk, in> DOC_PO_FILES or DOC_COOKBOOK_PO_FILES depending on the translation> type. Add the texi file name to doc/local.mk too, in info_TEXINFOS.
Alright. This looks like a nice improvement! Some nitpicking follows,but it LGTM overall.
Toggle quote (11 lines)>>From 5c14506d6af24b5307e03604cabf8ca10af56067 Mon Sep 17 00:00:00 2001> From: Julien Lepiller <julien@lepiller.eu>> Date: Sun, 18 Apr 2021 23:56:48 +0200> Subject: [PATCH] Makefile: Reimplement `download-po` target.>> The weblate API rate limit is very close to the number of files we need> to download. The previous implementation did not add new translations.>> * Makefile.am (download-po): Update target.> (make-download-po, make-check-po): Remove functions.
Should be: (make-download-po-rule, make-check-po-rule)
Toggle quote (3 lines)> - rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ; \> - mv "$(top_srcdir)/$(2)/$(3)$$$$lang.po"{.tmp2,} ; \
That was kinda crazy, I’m glad you divided the number of dollars bytwo. :-)
Toggle quote (2 lines)> +WEBLATE_REPO="https://framagit.org/tyreunom/guix-translations"
Since it’s not shell, rather:
WEBLATE_REPO = https://framagit.org/tyreunom/guix-translations
BTW, I vaguely remember we asked for a Savannah repo but I’m not surewhat the outcome was; did it eventually stall for some obscure reason?
Toggle quote (4 lines)> +# shallow clone the git repository behind weblate and copy files from it if> +# they contain at least one translation, and they are well-formed (scheme format> +# only), warn otherwise. Copied files are converted to a canonical form.
Please capitalize sentences and proper names.
Toggle quote (11 lines)> +download-po:> + dir=$$(mktemp -d); \> + git clone --depth 1 "$(WEBLATE_REPO)" "$$dir/translations"; \> + for domain in po/doc po/guix po/packages; do \> + for po in "$$dir/translations/$$domain"/*.po; do \> + translated=$$(LANG=en_US.UTF-8 msgfmt --statistics "$$po" 2>&1 | cut -f1 -d' '); \> + target=$$(basename "$$po"); \> + target="$$domain/$$target"; \> + if msgfmt -c "$$po" && [ "$$translated" != "0" ]; then \> + msgfilter --no-wrap -i "$$po" cat > "$$target"; \
Maybe write to $$target.tmp and then do “mv $$target.tmp $$target” soit’s atomic and we don’t end up with corrupt files if ‘msgfilter’ fails.
I would use spaces rather than tabs to indent the ‘for’ loops.
Thanks!
Ludo’.
J
J
Julien Lepiller wrote on 1 May 16:42 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47882-done@debbugs.gnu.org)
20210501164148.79449a60@tachikoma.lepiller.eu
Le Fri, 30 Apr 2021 21:57:05 +0200,Ludovic Courtès <ludo@gnu.org> a écrit :
Toggle quote (5 lines)> Hi!> > Alright. This looks like a nice improvement! Some nitpicking> follows, but it LGTM overall.
Thank you! I fixed your comments and pushed to master as45549e9f31e462d0d1519fd678dd782da6ed5cb9. Thank you!
Toggle quote (3 lines)> BTW, I vaguely remember we asked for a Savannah repo but I’m not sure> what the outcome was; did it eventually stall for some obscure reason?
Yeah, there were still some issues related to copyright. To be honest Ilost my energy to push on this front...
Closed
L
L
Ludovic Courtès wrote on 1 May 17:08 +0200
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 47882-done@debbugs.gnu.org)
87h7jm7aqf.fsf@gnu.org
Julien Lepiller <julien@lepiller.eu> skribis:
Toggle quote (2 lines)> Le Fri, 30 Apr 2021 21:57:05 +0200,
[...]
Toggle quote (6 lines)>> Alright. This looks like a nice improvement! Some nitpicking>> follows, but it LGTM overall.>> Thank you! I fixed your comments and pushed to master as> 45549e9f31e462d0d1519fd678dd782da6ed5cb9. Thank you!
Cool!
Toggle quote (6 lines)>> BTW, I vaguely remember we asked for a Savannah repo but I’m not sure>> what the outcome was; did it eventually stall for some obscure reason?>> Yeah, there were still some issues related to copyright. To be honest I> lost my energy to push on this front...
Yeah, I sympathize. It’s fine and more productive to use the repo youset up.
Thank you!
Ludo’.
Closed
?