From debbugs-submit-bounces@debbugs.gnu.org Sun Apr 18 18:10:17 2021 Received: (at submit) by debbugs.gnu.org; 18 Apr 2021 22:10:17 +0000 Received: from localhost ([127.0.0.1]:48005 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lYFcL-0005HU-4h for submit@debbugs.gnu.org; Sun, 18 Apr 2021 18:10:17 -0400 Received: from lists.gnu.org ([209.51.188.17]:42208) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lYFcJ-0005HN-W2 for submit@debbugs.gnu.org; Sun, 18 Apr 2021 18:10:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35516) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYFcJ-0002iB-RY for guix-patches@gnu.org; Sun, 18 Apr 2021 18:10:11 -0400 Received: from lepiller.eu ([2a00:5884:8208::1]:52124) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lYFcG-0007GR-DC for guix-patches@gnu.org; Sun, 18 Apr 2021 18:10:10 -0400 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id 4ffceb64 for ; Sun, 18 Apr 2021 22:10:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:subject:message-id:mime-version:content-type; s=dkim; bh=1HO dbailmMdwe29HPXqzU/LghhW34+gCwdzDdjVSjBM=; b=ObHtJKSx5/B8kLpx/Hm dkImKm1yK7CybjjA5hjZjMgwvM8oSQM7+/xDpHN9fK1jD876vXvQ1WC7m5nCjuqw g9O6q1DX6AWtUoTnp8oeEjZQZGqiHPkIE8PZNiRLcsEBhXVpv63666HNX4OG9wZl Q3AlZCI6LKK96zTGQd938zAgqdQ15AH4IlqJdH3ihZ/BAol83znfrSWOFXJm4zEh M7+GGGKsIXcwKQU9uSarSvWep3gJsnCcKJ3vkrDkjHiB8wp6LczvEnTJU/UFYmY6 Zi82gMNoUCa8ivGpsNAlVggJs7ETkf30Z9LrdhZCSqffxcBRPdJQSai10+l24JxW C+w== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id 562f5a69 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Sun, 18 Apr 2021 22:10:03 +0000 (UTC) Date: Mon, 19 Apr 2021 00:09:58 +0200 From: Julien Lepiller To: guix-patches@gnu.org Subject: [PATCH] Makefile: Reimplement download-po target Message-ID: <20210419000958.49968055@tachikoma.lepiller.eu> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/736Y_gLr=BoJc5_tZy53o=O" Received-SPF: pass client-ip=2a00:5884:8208::1; envelope-from=julien@lepiller.eu; helo=lepiller.eu X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --MP_/736Y_gLr=BoJc5_tZy53o=O Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi Guix! The attached patch reimplements the download-po target in Makefile.am. The previous implementation was based on the implementation for the TP and would use the API of Weblate to get files. However, the rate-limit is very close to the number of files we need to download, which was a big limitation, and this target would only download existing files, ignoring new translations. With this, new translations and existing ones are added/updated. The target will first do a shallow clone of the repository behind Weblate. Then, it will check that files are not empty (contain at least one translated string) and correct (according to scheme-format at least). If they are correct, it will normalize them and copy them to the po/ subdir in the repository. Here is an example output: copied po/doc/guix-manual.ru.po. /tmp/tmp.dWvtATfWQA/translations/po/doc/guix-manual.si.po:6: AVERTISSEMENT=C2=A0: Le champ d'en-t=C3=AAte =C2=AB PO-Revision-Date =C2=BB= a encore sa valeur initiale par d=C3=A9faut WARN: po/doc/guix-manual.si.po (0 translated messages) was not added/updated. /tmp/tmp.dWvtATfWQA/translations/po/guix/nl.po:5135: Les sp=C3=A9cifications de format dans =C2=AB msgid_plural =C2=BB et =C2=AB=C2=A0msgstr[1]=C2=A0=C2= =BB ne sont pas =C3=A9quivalentes msgfmt: 1 erreur fatale trouv=C3=A9e WARN: po/guix/nl.po (1001 translated messages) was not added/updated. In this output, we can see that the Russian manual was copied, the Sinhala manual was empty, so not copied, and the Dutch translation had an issue in one string that needs to be fixed, and was not copied. I chose not to copy the file because an issue in msgfmt would break guix pull. Unfortunately, this is not enough testing, as the manual and cookbook are not checked. Checking them requires building the manual, so before commiting changes, please run "make". To document the new process: make download-po for any new file in po/guix and po/packages, add the language in a single line in po/guix/LINGUAS or po/packages/LINGUAS. for any new file in po/doc, add the file name to po/doc/local.mk, in DOC_PO_FILES or DOC_COOKBOOK_PO_FILES depending on the translation type. Add the texi file name to doc/local.mk too, in info_TEXINFOS. --MP_/736Y_gLr=BoJc5_tZy53o=O Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0001-Makefile-Reimplement-download-po-target.patch From 5c14506d6af24b5307e03604cabf8ca10af56067 Mon Sep 17 00:00:00 2001 From: Julien Lepiller Date: Sun, 18 Apr 2021 23:56:48 +0200 Subject: [PATCH] Makefile: Reimplement `download-po` target. The weblate API rate limit is very close to the number of files we need to download. The previous implementation did not add new translations. * Makefile.am (download-po): Update target. (make-download-po, make-check-po): Remove functions. --- Makefile.am | 87 ++++++++++++++--------------------------------------- 1 file changed, 22 insertions(+), 65 deletions(-) diff --git a/Makefile.am b/Makefile.am index 17ad236655..811e47dbfc 100644 --- a/Makefile.am +++ b/Makefile.am @@ -969,71 +969,28 @@ cuirass-jobs: $(GOBJECTS) # Downloading up-to-date PO files. -# make-download-po-rule DOMAIN DIRECTORY [FILE-NAME-PREFIX] -define make-download-po-rule - -download-po.$(1): - if [ -f "$(top_srcdir)/$(2)/LINGUAS" ]; then \ - LINGUAS="`grep -v '^[[:blank:]]*#' < $(top_srcdir)/$(2)/LINGUAS`" ; \ - else \ - LINGUAS="`(cd $(top_srcdir)/$(2); \ - for i in *.po; do echo $$$$i; done) | cut -d . -f 2`" ; \ - fi ; \ - for lang in $$$$LINGUAS; do \ - if wget -nv -O "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" \ - "https://translate.fedoraproject.org/api/translations/guix/$(1)/$$$$lang/file/" ; \ - then \ - msgfilter --no-wrap -i "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" \ - cat > "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp2" ; \ - rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ; \ - mv "$(top_srcdir)/$(2)/$(3)$$$$lang.po"{.tmp2,} ; \ - else \ - rm "$(top_srcdir)/$(2)/$(3)$$$$lang.po.tmp" ; \ - fi ; \ - done - -.PHONY: download-po.$(1) - -endef - -# Checking po files for issues. This is useful to run after downloading new -# po files. - -# make-check-po-rule DOMAIN DIRECTORY [FILE-NAME-PREFIX] -define make-check-po-rule - -check-po.$(1): - if [ -f "$(top_srcdir)/$(2)/LINGUAS" ]; then \ - LINGUAS="`grep -v '^[[:blank:]]*#' < $(top_srcdir)/$(2)/LINGUAS`" ; \ - else \ - LINGUAS="`(cd $(top_srcdir)/$(2); \ - for i in *.po; do echo $$$$i; done) | cut -d . -f 2`" ; \ - fi ; \ - for lang in $$$$LINGUAS; do \ - if [ -f "$(top_srcdir)/$(2)/$(3)$$$$lang.po" ]; \ - then \ - if ! msgfmt -c "$(top_srcdir)/$(2)/$(3)$$$$lang.po" ; \ - then \ - exit 1 ; \ - fi ; \ - fi ; \ - done - -.PHONY: check-po.$(1) - -endef - -$(eval $(call make-download-po-rule,documentation-cookbook,po/doc,guix-cookbook.)) -$(eval $(call make-download-po-rule,documentation-manual,po/doc,guix-manual.)) -$(eval $(call make-download-po-rule,guix,po/guix)) -$(eval $(call make-download-po-rule,packages,po/packages)) - -$(eval $(call make-check-po-rule,documentation-cookbook,po/doc,guix-cookbook.)) -$(eval $(call make-check-po-rule,documentation-manual,po/doc,guix-manual.)) -$(eval $(call make-check-po-rule,guix,po/guix)) -$(eval $(call make-check-po-rule,packages,po/packages)) - -download-po: $(foreach domain,guix packages documentation-manual documentation-cookbook,download-po.$(domain)) +WEBLATE_REPO="https://framagit.org/tyreunom/guix-translations" + +# shallow clone the git repository behind weblate and copy files from it if +# they contain at least one translation, and they are well-formed (scheme format +# only), warn otherwise. Copied files are converted to a canonical form. +download-po: + dir=$$(mktemp -d); \ + git clone --depth 1 "$(WEBLATE_REPO)" "$$dir/translations"; \ + for domain in po/doc po/guix po/packages; do \ + for po in "$$dir/translations/$$domain"/*.po; do \ + translated=$$(LANG=en_US.UTF-8 msgfmt --statistics "$$po" 2>&1 | cut -f1 -d' '); \ + target=$$(basename "$$po"); \ + target="$$domain/$$target"; \ + if msgfmt -c "$$po" && [ "$$translated" != "0" ]; then \ + msgfilter --no-wrap -i "$$po" cat > "$$target"; \ + echo "copied $$target."; \ + else \ + echo "WARN: $$target ($$translated translated messages) was not added/updated."; \ + fi; \ + done; \ + done; \ + rm -rf "$$dir" .PHONY: download-po check-po: $(foreach domain,guix packages documentation-manual documentation-cookbook,check-po.$(domain)) -- 2.31.1 --MP_/736Y_gLr=BoJc5_tZy53o=O--