[PATCH 0/3] Add Repology updater

  • Open
  • quality assurance status badge
Details
5 participants
  • Ludovic Courtès
  • Nicolas Goaziou
  • Maxime Devos
  • Xinglu Chen
  • zimoun
Owner
unassigned
Submitted by
Xinglu Chen
Severity
normal
X
X
Xinglu Chen wrote on 6 Feb 2022 12:50
(address . guix-patches@gnu.org)
cover.1644147246.git.public@yoctocell.xyz
Hi,

This patchset adds a new updater, which scans Repology[1] for updates.
It should technically support all packages in Guix! :-)

The data on Repology isn’t as detailed as the one on language-specific
repos, e.g., PyPI, so the updater doesn’t support things like ‘input
changes’. If the source URL doesn’t contain the version verbatim[2], it
won’t be able reconstruct the URL of the updated version, meaning that
‘guix refresh -u’ won’t work.

Because of the way ‘%updaters’ in (guix upstream) works, the Repology
updater is the first or second updater that is used (since it
technically works on ever package), but because of the limitations I
mentioned above, the result might not always be the best. The Repology
updater is mostly useful for things that don’t already have an updater,
e.g., ‘maven-dependency-tree’. Would it make sense to hard-code the
‘%updaters’ variable and put the Repology last in the list?

[2]: e.g., the version is “1.0.0” but the URL is

Xinglu Chen (3):
git-download: Export <git-reference>.
import: Add 'repology' updater.
gnu: xorg-server-xwayland: Set 'repology-name' property.

Makefile.am | 3 +
doc/guix.texi | 7 ++
gnu/packages/xorg.scm | 2 +
guix/git-download.scm | 3 +-
guix/import/repology.scm | 226 ++++++++++++++++++++++++++++++++++++++
tests/import-repology.scm | 145 ++++++++++++++++++++++++
6 files changed, 385 insertions(+), 1 deletion(-)
create mode 100644 guix/import/repology.scm
create mode 100644 tests/import-repology.scm


base-commit: 7c9ad54b0616112c7eea6dd02379616aef206490
--
2.34.1
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmH/tgMVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5ueMP+wWlMMv507Da85m9aFArAhY+gZOD
OyKWCwnqAsQguW/Ijh2u6yGyutFhtDMhE/d29Z0Eu5VhXfrztCxqWStf9LWKN5G1
kgZlRgkbdx1mzlJIiIIbFsmsxpEoVCUpmH6T3Vm/hZQpta6xAWjiU+orOBt53rci
3PR2pU+XGSkmnRpFwzaWeW1ki08HvPsAVRX6VRGjep8p7zhpQh2nEW63tgNzeDRX
L8IuxqT4o+ywSIOEiqYjYF5+rA5HodcZ8C9D3LT2BLyPwYoA80HIdthrfyYhqKVP
XZaiy1sP2aeoyXK+I8mF+C/DQXcb9XUkH8kiMLr5noBK3Ork847YrJmBlP7sFEfP
xOW41Ep7S0ZlFdAVkTlTJ44jTyepqgcSnHp8Eg9MVK7oqQMKafGEe7h9rCf4I5xV
jCljfvlLQ748vsSgvGtU3jq88KNDx7KoMD//Y5yPHmY6pWk55KvZMn2LIDxqzZJA
Jzd1d+EsgGLj1ot242te0bM3pzPBPVzQazgH1+F3EMOzLzlWCxkn2XRorKdwi/XP
DlL24j/cjpfzRVZpRLMi91PmmPRZNiB3rRHjq4l2fCfVnjdEy+W/Cl3cm2GLte91
jcooyT8tNWJiWWszeAd85/3AfmQYk82PBHPPh6yj1fIKpSJuwqhJCXvdeQrrr8xh
JCunDMPlLQ+K6BXR
=ab47
-----END PGP SIGNATURE-----

M
M
Maxime Devos wrote on 6 Feb 2022 13:41
2ba4a342b6c2b53f7038a84c8337608e51ecf13a.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 12:50 [+0100]:
Toggle quote (8 lines)
> Because of the way ‘%updaters’ in (guix upstream) works, the Repology
> updater is the first or second updater that is used (since it
> technically works on ever package), but because of the limitations I
> mentioned above, the result might not always be the best. The Repology
> updater is mostly useful for things that don’t already have an updater,
> e.g., ‘maven-dependency-tree’. Would it make sense to hard-code the
> ‘%updaters’ variable and put the Repology last in the list?

I would prefer not to hardcode %updaters and keep the current
discovery mechanism, such that people can experiment with updaters
outside a git checkout of guix and in channels.

FWIW it would be useful to have the same mechanism for importers.

However, it might be a good idea to do some _postprocessing_ on
the discovered list of updaters, e.g. they could be sorted on
'genericity' with 'stable-sort' (*):

(define (genericity x)
(cond ((it is "generic-SOMETHING") 1)
((it is repology) 2)
(#true 0)))

(define (less x y)
(<= (genericity x) (genericity y)))

(*) stable-sort and not sort, to preserve alphabetical ordering
for updaters with the same genericity.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/B+RccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7kwHAP9vamvhIFQUUY9qfgv6X4JT6xjq
QHRdA+w51aw/A0R+SwD9GE/Mpw/22XbZ3y7WwR4mGcSIerIloYFGo/YFrpQORw4=
=4GIv
-----END PGP SIGNATURE-----


X
X
Xinglu Chen wrote on 6 Feb 2022 14:00
[PATCH 1/3] git-download: Export <git-reference>.
(address . 53818@debbugs.gnu.org)
25f4c5212f9461bd14f668b3fbd9ccb80f8bc5a4.1644147246.git.public@yoctocell.xyz
* guix/git-download.scm: Export <git-reference> record.
---
guix/git-download.scm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/guix/git-download.scm b/guix/git-download.scm
index 5e624b9ae9..43424f2f19 100644
--- a/guix/git-download.scm
+++ b/guix/git-download.scm
@@ -44,7 +44,8 @@ (define-module (guix git-download)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-35)
- #:export (git-reference
+ #:export (<git-reference>
+ git-reference
git-reference?
git-reference-url
git-reference-commit
--
2.34.1
X
X
Xinglu Chen wrote on 6 Feb 2022 14:00
[PATCH 2/3] import: Add 'repology' updater.
(address . 53818@debbugs.gnu.org)
88f866aff65c0ec9611b6cc0ee9d680ba711df38.1644147246.git.public@yoctocell.xyz
* guix/import/repology.scm
* tests/import-repology.scm: New files.
* Makefile.am (MODULES): Register them.
* doc/guix.texi (Invoking guix refresh): Document it.
---
Makefile.am | 3 +
doc/guix.texi | 7 ++
guix/import/repology.scm | 226 ++++++++++++++++++++++++++++++++++++++
tests/import-repology.scm | 145 ++++++++++++++++++++++++
4 files changed, 381 insertions(+)
create mode 100644 guix/import/repology.scm
create mode 100644 tests/import-repology.scm

Toggle diff (431 lines)
diff --git a/Makefile.am b/Makefile.am
index 7463606d20..6792917b59 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -16,6 +16,7 @@
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
# Copyright © 2021 Andrew Tropin <andrew@trop.in>
+# Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
#
# This file is part of GNU Guix.
#
@@ -271,6 +272,7 @@ MODULES = \
guix/import/opam.scm \
guix/import/print.scm \
guix/import/pypi.scm \
+ guix/import/repology.scm \
guix/import/stackage.scm \
guix/import/texlive.scm \
guix/import/utils.scm \
@@ -488,6 +490,7 @@ SCM_TESTS = \
tests/home-import.scm \
tests/import-git.scm \
tests/import-github.scm \
+ tests/import-repology.scm \
tests/import-utils.scm \
tests/inferior.scm \
tests/lint.scm \
diff --git a/doc/guix.texi b/doc/guix.texi
index 0cf865a672..15d215dd48 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12932,6 +12932,13 @@
(release-tag-version-delimiter . ":"))))
@end lisp
+@item repology
+an updater that scans @uref{https://repology.org, Repology}, a website
+that tracks packages on various package repositories, for updates.
+
+The name of a package in Guix is not always that same as the name on
+Repology; users can set the @code{repology-name} package property to
+make the updater use the correct name.
@end table
diff --git a/guix/import/repology.scm b/guix/import/repology.scm
new file mode 100644
index 0000000000..1d88022c1d
--- /dev/null
+++ b/guix/import/repology.scm
@@ -0,0 +1,226 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import repology)
+ #:use-module (guix diagnostics)
+ #:use-module (guix diagnostics)
+ #:use-module (guix git-download)
+ #:use-module (guix i18n)
+ #:use-module (guix import json)
+ #:use-module (guix import utils)
+ #:use-module (guix memoization)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (guix utils)
+ #:use-module (ice-9 match)
+ #:use-module (json)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-2)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-43)
+ #:export (repology-latest-release
+ %repology-updater))
+
+;;; Commentary:
+;;;
+;;; This module provides an updater which scans Repology, a site that monitors
+;;; several package repolsitories, for updates. This means that if any other
+;;; package repository has a version of a package that is newer than the
+;;; version in Guix, the package should be able to be updated. The updater
+;;; should in theory work for all packages in Guix, but the names of some
+;;; packages on Repology don't match the name in Guix. The 'repology-name'
+;;; package property can be used to fix this.
+;;;
+;;; Guix already has many different updaters for language-specific packages,
+;;; and these typically provide more accurate data, e.g., input changes,
+;;; signature URLs. The Repology updater should really be treated as a last
+;;; resort for those packages that don't have any other updater to rely on.
+;;;
+;;; See <https://repology.org/api/v1> for the API.
+;;;
+;;; Code:
+
+(define %repology-url
+ "https://repology.org/api/v1/project")
+
+(define* (package-name->repology-name name #:key (attempt 1))
+ "Convert NAME, the name of a Guix package, to the name of the package on
+Repology. It doesn't always guess the correct name on the first attempt, so
+on the second attempt it will try to guess another name."
+ (match attempt
+ (1 (cond
+ ((string-prefix? "ghc-" name)
+ (string-append "haskell:"
+ (string-drop name (string-length "ghc-"))))
+ ((string-prefix? "ocaml-" name)
+ (string-append "ocaml:"
+ (string-drop name (string-length "ocaml-"))))
+ ((string-prefix? "perl-" name)
+ (string-append "perl:"
+ (string-drop name (string-length "perl-"))))
+ ((string-prefix? "emacs-" name)
+ (string-append "emacs:"
+ (string-drop name (string-length "emacs-"))))
+ ((string-prefix? "go-" name)
+ (string-append "go:"
+ (string-drop name (string-length "go-"))))
+ ((string-prefix? "rust-" name)
+ (string-append "rust:"
+ (string-drop name (string-length "rust-"))))
+ ((string-prefix? "lua-" name)
+ (string-append "lua:"
+ (string-drop name (string-length "lua-"))))
+ ((string-prefix? "node-" name)
+ (string-append "node:"
+ (string-drop name (string-length "node-"))))
+ ((string-prefix? "python-" name)
+ (string-append "python:"
+ (string-drop name (string-length "python-"))))
+ ((string-prefix? "java-" name)
+ (string-append "java:"
+ (string-drop name (string-length "java-"))))
+ ((string-prefix? "r-" name)
+ (string-append "r:"
+ (string-drop name (string-length "r-"))))
+ ((string-prefix? "ruby-" name)
+ (string-append "ruby:"
+ (string-drop name (string-length "ruby-"))))
+ ((string-prefix? "xf86-" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-"))))
+ ((string-prefix? "font-" name)
+ (string-append "fonts:"
+ (string-drop name (string-length "font-"))))
+ (else name)))
+ (2 (cond
+ ((string-prefix? "xf86-video" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-video-"))))
+ ((string-prefix? "xf86-input" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-input-"))))
+ (else name)))))
+
+
+;;; JSON mappings.
+
+(define-json-mapping <repology-package> make-repology-package
+ repology-package?
+ json->repology-package
+ (repository repology-package-repository "repo")
+ (src-name repology-package-src-name "srcname")
+ (binary-name repology-package-binary-name "binname")
+ (visible-name repology-package-visible-name "visiblename")
+ (version repology-package-version)
+ (original-version repology-package-original-version "origversion")
+ (status repology-package-status)
+ (summary repology-package-summary)
+ (categories repology-package-categories)
+ (licenses repology-package-licenses)
+ (maintainers repology-package-maintainers))
+
+
+;;; Updater.
+
+(define repology-fetch-info
+ (memoize
+ (lambda (package)
+ "Fetch information about PACKAGE using the Repology API."
+ (define (name->info name )
+ (let ((url (string-append %repology-url "/" name)))
+ (and=> (json-fetch url)
+ (lambda (url)
+ (vector-map (lambda (a b)
+ (json->repology-package b))
+ url)))))
+
+ (let* ((name (or (assoc-ref (package-properties package)
+ 'repology-name)
+ (package-name->repology-name (package-name package))))
+ (info (name->info name)))
+ (if (and info (not (vector-empty? info)))
+ info
+ (let ((info (name->info (package-name->repology-name
+ (package-name package)
+ #:attempt 2))))
+ (if (and info (not (vector-empty? info)))
+ info
+ (begin
+ (warning (G_ "package not found on Repology: ~a\n")
+ (package-name package))
+ #f))))))))
+
+(define (update-url url old-version new-version)
+ "Replace OLD-VERSION in URL with NEW-VERSION."
+ (match (factorize-uri url old-version)
+ ((? string? uri) #f)
+ ((factorized ...)
+ (apply string-append
+ (map (lambda (component)
+ (match component
+ ('version new-version)
+ ((? string?) component)))
+ factorized)))))
+
+(define (package-source-urls package version)
+ "Return a list of URLs for PACKAGE at VERSION. If no URL was successfully constructed, return #f."
+ (let ((old-version (package-version package)))
+ ;; XXX: (guix upstream) only supports tarballs and Git repos for now.
+ (match (origin-uri (package-source package))
+ (($ <git-reference> url commit recursive?)
+ (and-let* ((updated-commit (if (string=? old-version commit)
+ version
+ (update-url commit old-version version))))
+ (list (git-reference
+ (url url)
+ (commit updated-commit)
+ (recursive? recursive?)))))
+ ((? string? url)
+ (list (update-url url old-version version)))
+ ((? list? urls)
+ (map (cut update-url <> old-version version) urls))
+ (_ #f))))
+
+(define (stable-version? repology-package)
+ (and (or (equal? "newest" (repology-package-status repology-package))
+ (equal? "unique" (repology-package-status repology-package)))
+ (repology-package-version repology-package)))
+
+;; XXX: 'package' will clash with the 'package' field of 'upstream-source'.
+(define (repology-latest-release pkg)
+ "Return the latest release of the PKG on Repology named NAME."
+ (and-let* ((packages (repology-fetch-info pkg))
+ (versions (filter-map stable-version?
+ (vector->list packages)))
+ (latest-version (if (null? versions)
+ #f
+ (car versions))))
+ ;; TODO: set 'signature-urls'.
+ (upstream-source
+ (package (package-name pkg))
+ (version latest-version)
+ (urls (package-source-urls pkg latest-version)))))
+
+(define %repology-updater
+(upstream-updater
+ (name 'repology)
+ (description "Updater for packages on Repology")
+ (pred (const #t))
+ (latest repology-latest-release)))
+
+;;; repology.scm ends here
diff --git a/tests/import-repology.scm b/tests/import-repology.scm
new file mode 100644
index 0000000000..f95071ce07
--- /dev/null
+++ b/tests/import-repology.scm
@@ -0,0 +1,145 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-import-repology)
+ #:use-module (guix download)
+ #:use-module (guix git-download)
+ #:use-module (guix import repology)
+ #:use-module (guix packages)
+ #:use-module (guix tests)
+ #:use-module (guix upstream)
+ #:use-module (json)
+ #:use-module (srfi srfi-64))
+
+(test-begin "repology")
+
+(define package-using-git-repository
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://git.example.org/foo")
+ (commit "1.0")))
+ (sha256 #f)))))
+
+(define package-using-tarball
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (string-append "https://example.org/foo-" version ".tar.gz"))
+ (sha256 #f)))))
+
+(define package-using-tarball-multiple-urls
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (list (string-append "https://example.org/foo-"
+ version ".tar.gz")
+ (string-append "https://mirror.example.org/foo-"
+ version ".tar.gz")))
+ (sha256 #f)))))
+
+(define %test-json
+"[
+ {
+ \"repo\": \"aur\",
+ \"srcname\": \"foo\",
+ \"binname\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"1.0.r25.gb86405a\",
+ \"maintainers\": [
+ \"bob@aur\"
+ ],
+ \"licenses\": [
+ \"LGPL3+\"
+ ],
+ \"summary\": \"foo bar\"
+ \"status\": \"rolling\",
+ \"origversion\": \"1.0.r25.gb86405a-1\"
+ },
+ {
+ \"repo\": \"gnuguix\",
+ \"srcname\": \"foo\",
+ \"binname\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"1.0\",
+ \"summary\": \"foo bar\",
+ \"status\": \"outdated\",
+ \"origversion\": null
+ },
+ {
+ \"repo\": \"nix_unstable\",
+ \"name\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"2.0\",
+ \"maintainers\": [
+ \"bob@example.org\"
+ ],
+ \"licenses\": [
+ \"LGPL-3.0-or-later\"
+ ],
+ \"summary\": \"foo bar\",
+ \"status\": \"newest\",
+ \"origversion\": null
+ }
+]")
+
+(define (latest-release package)
+ (mock ((guix import json) json-fetch
+ (lambda (url)
+ (json-string->scm %test-json)))
+ (repology-latest-release package)))
+
+(test-equal "package using Git repo: version"
+ "2.0"
+ (upstream-source-version
+ (latest-release package-using-git-repository)))
+
+(test-equal "package using Git repo: git-reference"
+ (list (git-reference
+ (url "https://git.example.org/foo")
+ (commit "2.0")))
+ (upstream-source-urls
+ (latest-release package-using-git-repository)))
+
+(test-equal "package using tarball: version"
+ "2.0"
+ (upstream-source-version
+ (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: URL"
+ (list "https://example.org/foo-2.0.tar.gz")
+ (upstream-source-urls
+ (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: multiple URLs"
+ (list "https://example.org/foo-2.0.tar.gz"
+ "https://mirror.example.org/foo-2.0.tar.gz")
+ (upstream-source-urls
+ (latest-release package-using-tarball-multiple-urls)))
+
+(test-end "repology")
--
2.34.1
X
X
Xinglu Chen wrote on 6 Feb 2022 14:00
[PATCH 3/3] gnu: xorg-server-xwayland: Set 'repology-name' property.
(address . 53818@debbugs.gnu.org)
c648a7af35de04f863e4ffed1b935b2f486ec869.1644147246.git.public@yoctocell.xyz
* gnu/packages/xorg.scm (xorg-server-xwayland):[properties]: Set
'repology-name'.
---
gnu/packages/xorg.scm | 2 ++
1 file changed, 2 insertions(+)

Toggle diff (15 lines)
diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 14e35d19ae..ee45630596 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -5520,6 +5520,8 @@ (define-public xorg-server-xwayland
(lambda _
(substitute* (find-files "." "\\.c$")
(("/bin/sh") (which "sh"))))))))
+ (properties
+ '((repology-name . "xwayland")))
(synopsis "Xorg server with Wayland backend")
(description "Xwayland is an X server for running X clients under
Wayland.")
--
2.34.1
M
M
Maxime Devos wrote on 6 Feb 2022 14:11
Re: [bug#53818] [PATCH 2/3] import: Add 'repology' updater.
128afdc0bdf163f025cdfaa2876ae7cdea0b5d86.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (7 lines)
> +(define* (package-name->repology-name name #:key (attempt 1))
> +  "Convert NAME, the name of a Guix package, to the name of the package on
> +Repology.  It doesn't always guess the correct name on the first attempt, so
> +on the second attempt it will try to guess another name."
> +  (match attempt
> +    (1 (cond

Could you add 'minetest' to the list? Sometimes they are named
'minetest-mod-foo' (Debian), sometimes 'minetest-foo' (Guix,
Archlinux).

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/I9BccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7kXwAQCIwOYnHU/22IG0R+eLe2a9n4uC
VyvO++Yt12q0rD8TUwD/aMCWwjLMDqek5OdZHRBQ/rSKRZORDwaA3LmlKEnttQc=
=ZVD+
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 6 Feb 2022 14:13
af74ba55be34c098ec2289e2871592a5b5b6d5aa.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (7 lines)
> +             (if (and info (not (vector-empty? info)))
> +                 info
> +                 (begin
> +                   (warning (G_ "package not found on Repology: ~a\n")
> +                            (package-name package))
> +                   #f))))))))

Why a warning here? For other refreshers, we just return '#f' and
don't print a warning.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/JchccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7pgvAP9MDm8gzYpmus13b5o5OWkQBahG
3zLOsWwrXGh2+DYgawD/dwQ7eppXIF+gdxhSyDb1mc5n60V+e6itzveXQOZFQwI=
=kMCU
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 6 Feb 2022 14:13
5b98cea5ddb2ef9ba79a80003cbd79ec2f5c955c.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (2 lines)
> +     (define (name->info name )

Superfluous space.
M
M
Maxime Devos wrote on 6 Feb 2022 14:17
97d4bfcc95de2a5bf5143dc7b178c270c0c8afde.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (5 lines)
> +      (($ <git-reference> url commit recursive?)
> +       (and-let* ((updated-commit (if (string=? old-version commit)
> +                                      version
> +                                      (update-url commit old-version version))))

When does the URL of a git-reference include a version?

Toggle quote (5 lines)
> +         (list (git-reference
> +                (url url)
> +                (commit updated-commit)
> +                (recursive? recursive?)))))

(guix upstream) expects a single git-reference, not a list of git-
references.

Also, depending on the order of fields here doesn't seem great, I' do
something like

((? git-reference? reference)
(git-reference
(inherit reference)
(commit updated-commit)))

instead.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/KfRccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7pdhAQD24DXPEmzQP5Me8ECaQ3sXUH49
cq83mXM8qrpyJQq3fgD/WcaqSTdWk5r6W/16VmKJM2FWUQsVVpec2wC31692Jwg=
=8GI1
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 6 Feb 2022 14:18
dfef21301f4f514357c96d09e1c22adecac09d60.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (5 lines)
> +(define (stable-version? repology-package)
> +  (and (or (equal? "newest" (repology-package-status repology-package))
> +           (equal? "unique" (repology-package-status repology-package)))
> +       (repology-package-version repology-package)))

What does 'stable' mean here? We usually just use the latest version
of a package, unlike, say, Debian stable.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/KuxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7lezAQDYAW0azBB2X1MyLpTZ97awncNE
U0f0sJwVp2Fs64c0rwEAwVCUtjb09dAaRGbomN+/Jb3W37RoobCZhVjdE7dY8Ag=
=MM7o
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 6 Feb 2022 14:19
232779a14d33f69de0665500099f9968fe6d0ca1.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (5 lines)
> +(test-equal "package using Git repo: git-reference"
> +  (list (git-reference
> +         (url "https://git.example.org/foo")
> +         (commit "2.0")))

Drop 'list', (guix upstream) doesn't expect a list here.

Greetings,
Maxime
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/K8RccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7mTWAQCQuKqErajwXrLQSTyhqeU6Iwgl
eeTFpC4jkSkCo6UiYwD/YfIUskfiROHeqVt/3FiqAuQKXuqmnQUgwLpRF1xMCwI=
=z0mH
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 6 Feb 2022 14:23
d23e9b226fb9de027f2dae3b7ba4e72254f72998.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (2 lines)
> +         (and=> (json-fetch url)

Given that we'll be contacting 'repology' for _every_ package
when doing "guix refresh -l", some caching seems in order.
Could 'http-fetch/cached' be used here? json-fetch may
need to be modified to allow overriding the http-fetch procedure
used.

FWIW this could be useful for some other updaters (e.g. the
Minetest/ContentDB updater), to a lesser degree.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/LthccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7tv6AQC4gGDjZ9WlWzsCljnPTXWLpzZG
Sj55w0l5G64qxJU7EQD+J/6N8jvtHLoIVN4zhjjnslERQHhR82tFJroVb32yvwQ=
=LCp2
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 6 Feb 2022 14:23
e5eca0fb66221519769e9b10f1376260872a825e.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (2 lines)
> +             (latest-version (if (null? versions)

Perhaps (and (pair? versions) (car versions)), YMMV.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/L6xccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7kxDAQC3H6dqqIh9CiF9jN6vB5eWj+7c
ueBEUCfEXC+7SbAMcgD6Arw5MpfPxv0HkLcv/fxHgGgEyJiPYjW5WzSSPaWj0Qs=
=+415
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 6 Feb 2022 15:15
Re: [bug#53818] [PATCH 3/3] gnu: xorg-server-xwayland: Set 'repology-name' property.
a83cf38d9cec226310e630c48704e7094eaaedac.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
Toggle quote (4 lines)
>                      (lambda _
>                        (substitute* (find-files "." "\\.c$")
>                          (("/bin/sh") (which "sh"))))))))

FWIW this is wrong when cross-compiling,
this needs to use (search-input-file inputs "bin/sh") instead
such that a (cross-compiled) shell from 'inputs' is used instead of a
(native) bash from 'native-inputs'.

That's a bug in the old package definition though.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/YFxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7j6lAP911CsZ9LKrL3aa3BzQVAairgwJ
Q2ogrsFbQUmXYuKfoAD/QCZTOYgEYqooh4ZuHECkHF+sNRqxXeZJZ+ePPBqRdw0=
=eSwI
-----END PGP SIGNATURE-----


X
X
Xinglu Chen wrote on 6 Feb 2022 16:17
Re: [bug#53818] [PATCH 0/3] Add Repology updater
878ruo2fe5.fsf@yoctocell.xyz
Maxime schrieb am Sonntag der 06. Februar 2022 um 13:41 +01:

Toggle quote (13 lines)
> Xinglu Chen schreef op zo 06-02-2022 om 12:50 [+0100]:
>> Because of the way ‘%updaters’ in (guix upstream) works, the Repology
>> updater is the first or second updater that is used (since it
>> technically works on ever package), but because of the limitations I
>> mentioned above, the result might not always be the best. The Repology
>> updater is mostly useful for things that don’t already have an updater,
>> e.g., ‘maven-dependency-tree’. Would it make sense to hard-code the
>> ‘%updaters’ variable and put the Repology last in the list?
>
> I would prefer not to hardcode %updaters and keep the current
> discovery mechanism, such that people can experiment with updaters
> outside a git checkout of guix and in channels.

Good point.

Toggle quote (17 lines)
> FWIW it would be useful to have the same mechanism for importers.
>
> However, it might be a good idea to do some _postprocessing_ on
> the discovered list of updaters, e.g. they could be sorted on
> 'genericity' with 'stable-sort' (*):
>
> (define (genericity x)
> (cond ((it is "generic-SOMETHING") 1)
> ((it is repology) 2)
> (#true 0)))
>
> (define (less x y)
> (<= (genericity x) (genericity y)))
>
> (*) stable-sort and not sort, to preserve alphabetical ordering
> for updaters with the same genericity.

That looks like a good idea
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmH/5pIVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5zoEP/iTocn9Z57276DXaWFmwwa7uHczT
L+SQbXM9hB0UHR1k2R+1/UOeZcxJMiVAxTcq63tigRpmdzf105Osl0qK+vyASox0
RTDS07T6tmunXNGyosIOXRbftpVhSLZPzFWGb2km1ZT0sawKHmGUjFcAl77cBtW4
PNsclQEm2iBanYx/bC8ElgiW0so6nRzQ4mqERsM1x73XgJqgAUWYN6URk/aW5XVE
/Zeq/hsDAsZRLyDnBhZc1V8w8xX302C4oNlakJb0/BCtHW0fAG5/c3q8Ss54Wbgu
som4LqLgxnC9qD4EM+Xe4PIzsIYG4tJu+35+bnfEpBDbIV1a67D49JxBsOk42/Dy
fib2vcku2ggTbnYCQyJqPEiVfhtWb2PcWQwYVwixylJANoMyE/cUa+MgE1I0rZsZ
O+7aGBZZefmNaDfGw6i2FF+8QpbRoEOBusxGVcZALqT1cSH0SJvWGxKc2cjkymXV
ggExtdjlevgiQllXvv/M10kweOmQD8aAtKenWT8ccfJdxG9PulrZO+u/h53kW/ER
PJmjfVNZH52qr84qxeB+1mL9L41cZGGd3ezJ4Z+qlHdXplZ90R7O2fXvnOmoR9Mb
1tqfdOLlwgEr3Tk3r/OnC2slEqrgtG6JoMhaSROxAZjdcvSXDDldNk7gTBetEM17
SwCZ8ayY1vbTR/f6
=vIQq
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 6 Feb 2022 16:18
Re: [bug#53818] [PATCH 2/3] import: Add 'repology' updater.
875yps2fc9.fsf@yoctocell.xyz
Maxime schrieb am Sonntag der 06. Februar 2022 um 14:11 +01:

Toggle quote (12 lines)
> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +(define* (package-name->repology-name name #:key (attempt 1))
>> +  "Convert NAME, the name of a Guix package, to the name of the package on
>> +Repology.  It doesn't always guess the correct name on the first attempt, so
>> +on the second attempt it will try to guess another name."
>> +  (match attempt
>> +    (1 (cond
>
> Could you add 'minetest' to the list? Sometimes they are named
> 'minetest-mod-foo' (Debian), sometimes 'minetest-foo' (Guix,
> Archlinux).

Sure!
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmH/5tYVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x50PcP/1R8UjjiV2xVRrjWU4NlegNQGfVS
WoIINHlwVecl35vlXmz7gZUi1l3MBNDzbyRr6xC/2KaDGkIBuW1FTLdNmPjI0epi
Uv79cztS9Ls78FiNfWykRAKDpz9snRZ+2qng6LGz+toKTasRrvyS4UJk82E0UPWT
zuSTZT5V5PvNboLaSHPJzXJkbpRVMES3krOcuuH9fNN8ZDJxXu+IfWCnjSnrV+S0
IPTuTc8FOJUx3iuLz6u+exBHMGrZ1ldZPfdy874BhJ2PKR0BN0o3cNJS49whnsH7
RCjJMGHKI1nzdTxVabVUCNPZYtHC1ben/56xXrk+n686bx76RdrgDV5gL+DnDNDl
E5oleHMuJLKtVWXQtTbhDv36LF4m7Pyzpdmm2voM3LYZ6S0v8/+dMTljKwaxZ9uv
1E1a+/MQ2Oc34gTszZWogjPK42+tUmwbSRzzO/4G+/OyiKrdy4ONVAm3yIyc40L7
g2Duf9vbq8q++rR5HohdW5jH/Umy1ll7qFENypRxd7+SVUGZHPha9xssFZuR1Dbj
k4rueH+qMaSrED8dEA2pfEGL+T6cRAyT0C7+wxRMyD1AQ0raxtiR8zM9K5SaAuMg
fNaODFfv3mrlt5BtnwuLkE4JZ8+kRFLF8D07P2SsYQ6u7Ndik/aW2WfJZccdOg3Z
XkuiLWlyJTXfN2bX
=Pk0i
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 6 Feb 2022 16:26
8735kw2ezo.fsf@yoctocell.xyz
Maxime schrieb am Sonntag der 06. Februar 2022 um 14:13 +01:
Toggle quote (10 lines)
> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +             (if (and info (not (vector-empty? info)))
>> +                 info
>> +                 (begin
>> +                   (warning (G_ "package not found on Repology: ~a\n")
>> +                            (package-name package))
>> +                   #f))))))))
>
> Why a warning here? For other refreshers, we just return '#f' and
> don't print a warning.
Well, some actually do print a warning, e.g., opam. The warning makes
it clear that a package wasn’t found when running ‘guix refresh’
(without arguments).
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmH/6JsVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5KSYP/0qdKTGN1CRsDuh1lfE5U8biLIP6
tpmoqNuQEWKZ75HeUfBkOQofek5tGogutbJvuBcgGr/yiF5g8BPn80yJK3mdtj/S
CQHPQ3FrQHl5GmMY0oWTiFC27EGdu3MSeTgZ0+vPXTYVvTdLfmIuBvtdZM8caU9Y
W5NtJIFaEIEpiTRYUgGRnwv08s99XXMPpbYhrASiy83Rul4Mrn0aokM4u405vfg/
5xZUMaQ3OdkiWfQrIzNBaijWyKtliLHllPGjJtdsek5XGIhpzdpHgVj6RfbpcdBW
gc65KM+bA99eRPNdG7z1JeQQgL+5i/mJINMTiqEXCr2UrVBgQrD/AOJX8OonYrpd
YXvP2NWjj+R/S3vhXCqU4++cePnn5OhcMT3xbJUqlTDk4RfazASrHckGHupsoZ+p
w3GmRrJfc8r4RX2ugekaVrRpDaCrNw6S7gJ2GR08cnvG7OSqtzzbMxqa0XuO9BHc
9doQo5ya7pZskMxQvdVREQHkMVioPxZXvbUjzFFuT1KHSr+VpCURZgzD2z0HEhgF
W/3T5uN4Oqug9aFNlpGIIJhJkGdoNfr77JTJMeu9hapjOl2A2GKk14stpU8HZmYl
w+1EmO4v2Zm3i1L7cVoM4bHEajkIv1NKoWmZwRpEjNhWnxcnPAtH3M5JinHDF+Jo
IyMszfCElzVNcje5
=4hEf
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 6 Feb 2022 16:32
87zgn4105p.fsf@yoctocell.xyz
Maxime schrieb am Sonntag der 06. Februar 2022 um 14:17 +01:
Toggle quote (7 lines)
> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +      (($ <git-reference> url commit recursive?)
>> +       (and-let* ((updated-commit (if (string=? old-version commit)
>> +                                      version
>> +                                      (update-url commit old-version version))))
>
> When does the URL of a git-reference include a version?
It doesn’t, but the tag usually does; I should probably rename the
‘update-url’ procedure to something else, maybe ‘update-version’?
Toggle quote (7 lines)
>> +         (list (git-reference
>> +                (url url)
>> +                (commit updated-commit)
>> +                (recursive? recursive?)))))
>
> (guix upstream) expects a single git-reference, not a list of git-
> references.
Oops, noted.
Toggle quote (9 lines)
> Also, depending on the order of fields here doesn't seem great, I' do
> something like
>
> ((? git-reference? reference)
> (git-reference
> (inherit reference)
> (commit updated-commit)))
>
> instead.
Then I can drop the first patch of the series too. :-)
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmH/6fIVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5b0IQAKjEZ36foAKOtys0XF3DzgSwZql6
CD2E+RZLGWoAVZyEWwOIQxsx4EhZd2fyPkSxqcOhUbpIXRLmTPDkO8BL+dcVNtAu
dmCvmwOunHGrKwGuuEuV5/MXgCjTFksJmgTt2lj4VCDDwDVzXWm3dfuQXbKemLL/
5Rna2VbDQ7JqIQvv/9IS9bh+6muSUxOWDC6PDPzD/vjXPcAxiw9VSZVDuy+9xxw+
Y4IjNCEVWn6rWSHnoKKnZL5UiI5PSh72IdsNa8fO/7q5FTYNyGOcXHVcNcjC4Ii/
x91qTpWgiNSJdEqd0mXqtSiVlNcwvA4arY3BANVMYc/DDXup/ABXZofINsYe4V6w
x2R5uqQjaB0q9v6Fgt9tWO/m1DGWAhHn5YkQ9BJr3IHsB1TQ1GNuTnpMMEtKOtqZ
QN0DjxI7E0brvIvxXaSQNHl0eFzNOhnIZV9vt+BuKQB/+f1Lp/7iUYbdp/GXzxXc
50bIDaB8htFjt65iFe/Hdwn24EFQxAbyE43BU+EHVrz3bDOR2rDkLtB9vBU0TIpu
1t7bb4CqFqOfeAowpWZKVGl0Wc+bC5KMnw/oeyCb0uPJ7Hn3qZ2HJzh88xUPaOM8
s+/K07unctQHAARnktF5YZTbJqDZsFTd/YS3LY6cH+MVRB85wlekD4UE8Iq+bSCD
9/Xs2/0+0S8iw34N
=GfKQ
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 6 Feb 2022 16:34
87wni81023.fsf@yoctocell.xyz
Maxime schrieb am Sonntag der 06. Februar 2022 um 14:18 +01:

Toggle quote (9 lines)
> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +(define (stable-version? repology-package)
>> +  (and (or (equal? "newest" (repology-package-status repology-package))
>> +           (equal? "unique" (repology-package-status repology-package)))
>> +       (repology-package-version repology-package)))
>
> What does 'stable' mean here? We usually just use the latest version
> of a package, unlike, say, Debian stable.

“Stable” refers to a stable release, so things like pre-releases will be
ignored. Maybe it should be renamed to ‘latest-release?’ or
‘latest-stable-release?’?
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmH/6nUVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5isUP/1M+4Gj/t2swzkDVHDMq6DpLdSik
0W8fWxy8eDxMmW0NDaazHor03/jhneSQlqvp99jOW3LBx37aV+5ysb1Oo7bhcQdV
KyfMH6SJ9AzQIWBs5Km/wc9rdgBKqQn0Vpl+K6Cpx+oSslh5+WxP/TySrYfMVPwH
DFGOtjp0SRftoM04wAMpJ/vHSc4FUZ5wgIzi7FB6c+6p80iWZ5U5t/XxGEvWufZg
1F/pqLwiE/icG1ggO6VVeZXpe2Dl51ErKMA96YCgldDa3RbFabVewWm7Z0JwPpya
nspwmF0DMYBqic/i9Bm1mr5/ZAvYmlOzbZ4wgyZ8Y3Q/Ds8l6TScG5Sn4H7ppLch
Zb1IE/6XGZl+gvfp6P/xzYFu76EuuIkFVbys04clK8edctA0AAqLGcp42uCBfMsp
j5drXD64pRP++4/EdtDmV9y/mP3+kcYcBNBCTCzcohqHVR1w9WS5cmdkuyYMR/JD
qfuessWh1nYfHXHDlpSe8fVC/SB2XtrO8xtGlbjEq2dY3N1xd78MqKdxRJQn14cW
rCMcsNUxhQ1No/BSSJpRuJuTBSu9HbufBqIyvcReINK2tsRxDL9cPRQQ1MhsORHs
AeuXrNjPxbw9dzt4yvUThRaUXnfJa5KQTedOdZDeZaozGcbAhf3y3wGivfXTu224
fxUEnUkSQ5NqEasN
=0PHK
-----END PGP SIGNATURE-----

M
M
Maxime Devos wrote on 6 Feb 2022 16:36
a3d7121e863f279e2beeb100525cf27c9178ba2f.camel@telenet.be
Xinglu Chen schreef op zo 06-02-2022 om 16:34 [+0100]:
Toggle quote (7 lines)
> > of a package, unlike, say, Debian stable.
>
> “Stable” refers to a stable release, so things like pre-releases will
> be
> ignored.  Maybe it should be renamed to ‘latest-release?’ or
> ‘latest-stable-release?’?

I would go with 'latest-release?' and a comment or docstring explaining
that pre-releases are ignored.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf/rGhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7sDtAP9bgmc8uJAP2ip8D8XAYl0ZZSMd
hnf3/8DTug2k+Mq3XAD7B6nYjCot2NZz2lSNEGsd/TXAv7sDVhDFfYBprYzIBwg=
=lR8o
-----END PGP SIGNATURE-----


X
X
Xinglu Chen wrote on 6 Feb 2022 16:41
87sfsw0zp5.fsf@yoctocell.xyz
Maxime schrieb am Sonntag der 06. Februar 2022 um 14:23 +01:

Toggle quote (9 lines)
> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +         (and=> (json-fetch url)
>
> Given that we'll be contacting 'repology' for _every_ package
> when doing "guix refresh -l", some caching seems in order.
> Could 'http-fetch/cached' be used here? json-fetch may
> need to be modified to allow overriding the http-fetch procedure
> used.

That would be a good idea. It seems like ‘http-fetch/cached’ doesn’t
take a #:headers argument, so it will have to be modified first.
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmH/7EYVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5TyoQAKJoZTiWm/F/zx+lvryIDHqQTXTe
Mri7zJwKEp6PTCVlper+yF63FzBaCvkaFDytjfgycPTMLN9m0fU823XY+yZxcteF
Z84Q7JbZ/YqmhG1oB6gp+ODwabk+mQ2horboXUzAeGVVGzoZwGQo/rnVhZYZdzRK
GhG6G366qY8KjvojHAy9vVHvUTg9myzxbyKN7b8ady95lbhqJC3+bDl0Am/vCPD/
U+1IKCEetOV1dTpY8ZgZTji/H+z+0u36qH6cdwPK5/m70xN7K0w/OqTAkmFD/e1/
x6O/ybbNgMmeOGFqbtRLcywr3LtQDrDwSpLfsgYv//yi7b6MbigHAtW6Bu9ui4bZ
FTkh3fpSQUkisFzwu2vGSZRsP0ngU98FY1XU3WwuuAzJQ4JDfVKVg8iWiZG15X8q
+iRq6mld8s4IFQjPYkR/xKHhVJOaPQx6AqGcDW6DC1jFrazWLB9ULve85Bz0iZSo
aMwJpSNjv3tCfrbnzd60yFhNJb/aX7Cpvpwer5YOGRL+TWVEIIR50k8IKyioZyZc
qzg3qkaSW175NR8S/pC6Mu0PEO7FF+wwUrCmTdSRnfSQtfrmir/2fI/p3fS01KuN
OcjAPukPiCgcEy45yQccMGqKhTYX26iLqEcymkZ1Ew7pPNlCD4aG7Ncira5aa1gL
5FHJw34trYbQ5PIU
=FJZz
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 6 Feb 2022 16:42
87pmo00zo0.fsf@yoctocell.xyz
Maxime schrieb am Sonntag der 06. Februar 2022 um 14:23 +01:

Toggle quote (5 lines)
> Xinglu Chen schreef op zo 06-02-2022 om 14:00 [+0100]:
>> +             (latest-version (if (null? versions)
>
> Perhaps (and (pair? versions) (car versions)), YMMV.

That looks a little cleaner, thanks!
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmH/7G8VHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5XoAQAKuREi34MbqeRnkWtrS4RyiIrzH1
N+l9m/XutPhDWcJoO9dEpNw/ScyYcheAIETLktxOMDPtLF1GK6k8cwBomubgS9nz
L8d9o/QiYqNR+GFbSukQn/WCP3uiijt6xFacO4/EIpSOd78vWQTorBqGHO4pwCVA
Jq31hDEzNFgqtUUR99IJCHPD83g65p+bz61GHhmrlEzLNoqA+A0prgth2Cn4a5+j
KpA0ejjYNjxXSSOyGfCPU+0TXbwhVTkifJ+GNMrBGNtmH3hXwjI4aT5RGaWdvIHM
EwmeR4hLCjW/gpaecVPN/X5Qm8f+1hPxr98dnetON1bt39MtRha3LzXPkmwY/DLv
FNFZayiwBaAgb69aaGllQ7IKz26KLoIh1wWpU37gVps/jdq3mLjd3ccKFQuWdJ5W
kGFARG0i7ek2nIbszmVnbAI/hXtUrjs4erp66XW9DDqpI3lWYofGNmidAuAX9e1J
8HZ8P8lg2RT7JLT3rJ0yf5Y/4kZd9Mb9pVDuvcMn44EKozrzxk8mv+Bm1PdwkG9h
6PHkpOygPZPtUOB9v1NsZEkuDS+fwP0QvOb6dMNE0ZPhwNG/YBUQQpYXUtYkCo1M
6l763bIypK5QTf9qqeKTWMB2JcrElTHvQ/4TrGJdqD8sM0t2Dh3KIYHiMC4QjD+6
VzjyEmUl3pxCh+Y4
=8bzj
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 7 Feb 2022 10:06
[PATCH v2 0/7] Add Repology updater
(address . 53818@debbugs.gnu.org)
cover.1644224421.git.public@yoctocell.xyz
Hi,

The following changes have been made since v1:

* Sort the list of updaters to make generic ones come last.

* Make ‘json-fetch’ use ‘http-fetch/cached’ if the #:cached? argument is
#t.

* Added some additional rules for converting Guix package names to
Repology package names.

* Some typo fixes and more docstrings.

Xinglu Chen (7):
upstream: Sort list of updaters.
http-client: Make 'http-fetch/cached' take '#:headers' argument.
http-client: 'http-fetch/cached' accepts a string or a <uri>.
import: json: Make 'json-fetch' take '#:cached?' argument.
import: Add 'repology' updater.
gnu: xorg-server-xwayland: Set 'repology-name' property.
gnu: xorg-server-xwayland: Prepare for cross-compilation.

Makefile.am | 3 +
doc/guix.texi | 7 ++
gnu/packages/xorg.scm | 8 +-
guix/http-client.scm | 18 ++-
guix/import/json.scm | 14 ++-
guix/import/repology.scm | 235 ++++++++++++++++++++++++++++++++++++++
guix/upstream.scm | 29 ++++-
tests/import-repology.scm | 145 +++++++++++++++++++++++
8 files changed, 441 insertions(+), 18 deletions(-)
create mode 100644 guix/import/repology.scm
create mode 100644 tests/import-repology.scm


base-commit: 7c9ad54b0616112c7eea6dd02379616aef206490
--
2.34.1
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmIA4QEVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5+98P/2E8K0+rbSoyIldhJB0SpAab+35E
W2vBNZMKdzxXe03NmL3Xq83b8MBPzd6jgACppClzM/5Of3kw/xqDNt3uSxSIpqXC
Tobr8pCjycEBMLrmqw07jYFTU+nfT+q/YN1KXY1+FPCmsK8pCpQJpc+yHeuXfmGQ
V2Hub40UtQXdTAEzWhSRMS8nblyLiwOrntpLtHNNh7EhcBBvWLerpb8/4qHhJyNo
I8AXqQ7DZ+CyVRkcBE2Gm6+BshXnnAamJ9PyVXZsB+2/3U0GGNPbYuB1d4FRC1Pf
kJta9Dpay6Y3bUrzyoELvnIUnGV1xBX5kF6lBv07qfesyeYEdX5QraIP2iicTEjo
bXa14K856Q4ffoIq0u1hJFWwMnwhmpdyD9sdzXpfnw069fcwpv0vdknbaPlZgP41
0z/RDtWOu/iWG/g+KMr9DM3wiMA4NZ46vLLQA2hCaLn0WcexcitlYKpzuKMMaPoF
odOxPghHz76rVi3iP8DohCuaFzUf1dLPhUdBZAIj4Xtmm1urpwDIfMJtUpt3GViN
YUMHkRz4l4v3oWbBAJtPiX8VAdWZjlq1tbANHsop1uopU42XBiJxWa4eBufUANDM
LfjbSmeQRgxaPxoPjL+XMFkiTolA6o2RK78ceKZcX5kfRnpV1ElUvlrol7RJt83C
nuj1lNFbrRDWOl0L
=1dnW
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 7 Feb 2022 10:06
[PATCH v2 1/7] upstream: Sort list of updaters.
(address . 53818@debbugs.gnu.org)
c37b9459451716ba887074e31de919494608832f.1644224421.git.public@yoctocell.xyz
The order determines which updater gets used first when running ‘guix refresh’
on a package. We want the most generic updaters to be last since they usually
don’t provide as much or accurate information as language-specific updaters.

* guix/upstream.scm (sort-updaters): New procedure.
(%updaters): Use it.
---
guix/upstream.scm | 29 +++++++++++++++++++++++------
1 file changed, 23 insertions(+), 6 deletions(-)

Toggle diff (44 lines)
diff --git a/guix/upstream.scm b/guix/upstream.scm
index 6666803a92..d5faf9d3ee 100644
--- a/guix/upstream.scm
+++ b/guix/upstream.scm
@@ -250,14 +250,31 @@ (define (importer-modules)
%load-path)
#:warn warn-about-load-error)))
+(define (sort-updaters updaters)
+ "Sort UPDATERS by putting the more generic ones last."
+ (define (genericity updater)
+ (cond
+ ((equal? 'repology (upstream-updater-name updater))
+ 2)
+ ((string-prefix? "generic-"
+ (symbol->string (upstream-updater-name updater)))
+ 1)
+ (else 0)))
+
+ (define (less a b)
+ (<= (genericity a) (genericity b)))
+
+ (stable-sort updaters less))
+
(define %updaters
;; The list of publically-known updaters.
- (delay (fold-module-public-variables (lambda (obj result)
- (if (upstream-updater? obj)
- (cons obj result)
- result))
- '()
- (importer-modules))))
+ (delay (sort-updaters
+ (fold-module-public-variables (lambda (obj result)
+ (if (upstream-updater? obj)
+ (cons obj result)
+ result))
+ '()
+ (importer-modules)))))
;; Tests need to mock this variable so mark it as "non-declarative".
(set! %updaters %updaters)
--
2.34.1
X
X
Xinglu Chen wrote on 7 Feb 2022 10:06
[PATCH v2 3/7] http-client: 'http-fetch/cached' accepts a string or a <uri>.
(address . 53818@debbugs.gnu.org)
00f4de4c2fddf25566c726dde4a7d52ee80f7f71.1644224421.git.public@yoctocell.xyz
This is consistent with the 'http-fetch' procedure.

* guix/http-client.scm (http-fetch/cached): The 'uri' argument can be a string
or a <uri> record.
---
guix/http-client.scm | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

Toggle diff (18 lines)
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 058f09852f..08efdd5e47 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -297,7 +297,10 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
TIMEOUT specifies the timeout in seconds for connection establishment.
Write information about redirects to LOG-PORT."
- (let ((file (cache-file-for-uri uri)))
+ (let* ((uri (if (string? uri)
+ (string->uri uri)
+ uri))
+ (file (cache-file-for-uri uri)))
(define (update-cache cache-port)
(define cache-time
(and cache-port
--
2.34.1
X
X
Xinglu Chen wrote on 7 Feb 2022 10:06
[PATCH v2 2/7] http-client: Make 'http-fetch/cached' take '#:headers' argument.
(address . 53818@debbugs.gnu.org)
fe8af3a502d7b6399834f0426088200936646aed.1644224421.git.public@yoctocell.xyz
* guix/http-client.scm (http-fetch/cached): Take ‘#:headers’ keyword argument.
---
guix/http-client.scm | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)

Toggle diff (48 lines)
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 10bc278023..058f09852f 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2012, 2015 Free Software Foundation, Inc.
;;; Copyright © 2017 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -283,9 +284,11 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
(write-cache dump-port)
(cache-miss (const #t))
(log-port (current-error-port))
- (timeout 10))
+ (timeout 10)
+ (headers '((user-agent . "GNU Guile"))))
"Like 'http-fetch', return an input port, but cache its contents in
-~/.cache/guix. The cache remains valid for TTL seconds.
+~/.cache/guix. The cache remains valid for TTL seconds. HEADERS is an alist
+of extra HTTP headers. The cache time will automatically be added to HEADERS, so there is no need to specify it.
Call WRITE-CACHE with the HTTP input port and the cache output port to write
the data to cache. Call CACHE-MISS with URI just before fetching data from
@@ -300,8 +303,8 @@ (define cache-time
(and cache-port
(stat:mtime (stat cache-port))))
- (define headers
- `((user-agent . "GNU Guile")
+ (define headers*
+ `(,@headers
,@(if cache-time
`((if-modified-since
. ,(time-utc->date (make-time time-utc 0 cache-time))))
@@ -316,7 +319,7 @@ (define headers
(raise c))))
(let ((port (http-fetch uri #:text? text?
#:log-port log-port
- #:headers headers #:timeout timeout)))
+ #:headers headers* #:timeout timeout)))
(cache-miss uri)
(mkdir-p (dirname file))
(when cache-port
--
2.34.1
X
X
Xinglu Chen wrote on 7 Feb 2022 10:07
[PATCH v2 4/7] import: json: Make 'json-fetch' take '#:cached?' argument.
(address . 53818@debbugs.gnu.org)
92ca3095e8a5cf1197afd8883017c6b3729184f3.1644224421.git.public@yoctocell.xyz
* json.scm (json-fetch): Add ‘#:cached?’ keyword argument.
---
guix/import/json.scm | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)

Toggle diff (41 lines)
diff --git a/guix/import/json.scm b/guix/import/json.scm
index 0c98bb25b8..6f88353659 100644
--- a/guix/import/json.scm
+++ b/guix/import/json.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2015, 2016 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2018, 2019 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -37,17 +38,22 @@ (define-module (guix import json)
(define* (json-fetch url
;; Note: many websites returns 403 if we omit a
;; 'User-Agent' header.
- #:key (headers `((user-agent . "GNU Guile")
- (Accept . "application/json"))))
+ #:key
+ (headers `((user-agent . "GNU Guile")
+ (Accept . "application/json")))
+ cached?)
"Return a representation of the JSON resource URL (a list or hash table), or
#f if URL returns 403 or 404. HEADERS is a list of HTTP headers to pass in
-the query."
+the query. If CACHED? is #t, 'http-fetch/cached' will be used to fetch URL."
(guard (c ((and (http-get-error? c)
(let ((error (http-get-error-code c)))
(or (= 403 error)
(= 404 error))))
#f))
- (let* ((port (http-fetch url #:headers headers))
+ (let* ((port ((if cached?
+ http-fetch/cached
+ http-fetch)
+ url #:headers headers))
(result (json->scm port)))
(close-port port)
result)))
--
2.34.1
X
X
Xinglu Chen wrote on 7 Feb 2022 10:07
[PATCH v2 5/7] import: Add 'repology' updater.
(address . 53818@debbugs.gnu.org)
98726379214702d0745f56d9f946e792e803d326.1644224421.git.public@yoctocell.xyz
* guix/import/repology.scm
* tests/import-repology.scm: New files.
* Makefile.am (MODULES): Register them.
* doc/guix.texi (Invoking guix refresh): Document it.
---
Makefile.am | 3 +
doc/guix.texi | 7 ++
guix/import/repology.scm | 235 ++++++++++++++++++++++++++++++++++++++
tests/import-repology.scm | 145 +++++++++++++++++++++++
4 files changed, 390 insertions(+)
create mode 100644 guix/import/repology.scm
create mode 100644 tests/import-repology.scm

Toggle diff (440 lines)
diff --git a/Makefile.am b/Makefile.am
index 7463606d20..6792917b59 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -16,6 +16,7 @@
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
# Copyright © 2021 Andrew Tropin <andrew@trop.in>
+# Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
#
# This file is part of GNU Guix.
#
@@ -271,6 +272,7 @@ MODULES = \
guix/import/opam.scm \
guix/import/print.scm \
guix/import/pypi.scm \
+ guix/import/repology.scm \
guix/import/stackage.scm \
guix/import/texlive.scm \
guix/import/utils.scm \
@@ -488,6 +490,7 @@ SCM_TESTS = \
tests/home-import.scm \
tests/import-git.scm \
tests/import-github.scm \
+ tests/import-repology.scm \
tests/import-utils.scm \
tests/inferior.scm \
tests/lint.scm \
diff --git a/doc/guix.texi b/doc/guix.texi
index 0cf865a672..15d215dd48 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12932,6 +12932,13 @@
(release-tag-version-delimiter . ":"))))
@end lisp
+@item repology
+an updater that scans @uref{https://repology.org, Repology}, a website
+that tracks packages on various package repositories, for updates.
+
+The name of a package in Guix is not always that same as the name on
+Repology; users can set the @code{repology-name} package property to
+make the updater use the correct name.
@end table
diff --git a/guix/import/repology.scm b/guix/import/repology.scm
new file mode 100644
index 0000000000..28f3a3af5f
--- /dev/null
+++ b/guix/import/repology.scm
@@ -0,0 +1,235 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import repology)
+ #:use-module (guix diagnostics)
+ #:use-module (guix diagnostics)
+ #:use-module (guix git-download)
+ #:use-module (guix i18n)
+ #:use-module (guix import json)
+ #:use-module (guix import utils)
+ #:use-module (guix memoization)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (guix utils)
+ #:use-module (ice-9 match)
+ #:use-module (json)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-2)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-43)
+ #:export (repology-latest-release
+ %repology-updater))
+
+;;; Commentary:
+;;;
+;;; This module provides an updater which scans Repology, a site that monitors
+;;; several package repolsitories, for updates. This means that if any other
+;;; package repository has a version of a package that is newer than the
+;;; version in Guix, the package should be able to be updated. The updater
+;;; should in theory work for all packages in Guix, but the names of some
+;;; packages on Repology don't match the name in Guix. The 'repology-name'
+;;; package property can be used to fix this.
+;;;
+;;; Guix already has many different updaters for language-specific packages,
+;;; and these typically provide more accurate data, e.g., input changes,
+;;; signature URLs. The Repology updater should really be treated as a last
+;;; resort for those packages that don't have any other updater to rely on.
+;;;
+;;; See <https://repology.org/api/v1> for the API.
+;;;
+;;; Code:
+
+(define %repology-url
+ "https://repology.org/api/v1/project")
+
+(define* (package-name->repology-name name #:key (attempt 1))
+ "Convert NAME, the name of a Guix package, to the name of the package on
+Repology. It doesn't always guess the correct name on the first attempt, so
+on the second attempt it will try to guess another name."
+ (match attempt
+ (1 (cond
+ ((string-prefix? "ghc-" name)
+ (string-append "haskell:"
+ (string-drop name (string-length "ghc-"))))
+ ((string-prefix? "ocaml-" name)
+ (string-append "ocaml:"
+ (string-drop name (string-length "ocaml-"))))
+ ((string-prefix? "perl-" name)
+ (string-append "perl:"
+ (string-drop name (string-length "perl-"))))
+ ((string-prefix? "emacs-" name)
+ (string-append "emacs:"
+ (string-drop name (string-length "emacs-"))))
+ ((string-prefix? "go-" name)
+ (string-append "go:"
+ (string-drop name (string-length "go-"))))
+ ((string-prefix? "rust-" name)
+ (string-append "rust:"
+ (string-drop name (string-length "rust-"))))
+ ((string-prefix? "lua-" name)
+ (string-append "lua:"
+ (string-drop name (string-length "lua-"))))
+ ((string-prefix? "node-" name)
+ (string-append "node:"
+ (string-drop name (string-length "node-"))))
+ ((string-prefix? "python-" name)
+ (string-append "python:"
+ (string-drop name (string-length "python-"))))
+ ((string-prefix? "java-" name)
+ (string-append "java:"
+ (string-drop name (string-length "java-"))))
+ ((string-prefix? "r-" name)
+ (string-append "r:"
+ (string-drop name (string-length "r-"))))
+ ((string-prefix? "ruby-" name)
+ (string-append "ruby:"
+ (string-drop name (string-length "ruby-"))))
+ ((string-prefix? "xf86-" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-"))))
+ ((string-prefix? "font-" name)
+ (string-append "fonts:"
+ (string-drop name (string-length "font-"))))
+ ((string-suffix? "-minimal" name)
+ (string-drop-right name (string-length "-minimal")))
+ (else name)))
+ (2 (cond
+ ((string-prefix? "xf86-video" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-video-"))))
+ ((string-prefix? "xf86-input" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-input-"))))
+ ((string-prefix? "minetest-" name)
+ (string-append "minetest-mod-"
+ (string-drop name (string-length "minetest-"))))
+ (else name)))))
+
+
+;;; JSON mappings.
+
+(define-json-mapping <repology-package> make-repology-package
+ repology-package?
+ json->repology-package
+ (repository repology-package-repository "repo")
+ (src-name repology-package-src-name "srcname")
+ (binary-name repology-package-binary-name "binname")
+ (visible-name repology-package-visible-name "visiblename")
+ (version repology-package-version)
+ (original-version repology-package-original-version "origversion")
+ (status repology-package-status)
+ (summary repology-package-summary)
+ (categories repology-package-categories)
+ (licenses repology-package-licenses)
+ (maintainers repology-package-maintainers))
+
+
+;;; Updater.
+
+(define repology-fetch-info
+ (memoize
+ (lambda (package)
+ "Fetch information about PACKAGE using the Repology API."
+ (define (name->info name)
+ (let ((url (string-append %repology-url "/" name)))
+ (and=> (json-fetch url #:cached? #t)
+ (lambda (url)
+ (vector-map (lambda (a b)
+ (json->repology-package b))
+ url)))))
+
+ (let* ((name (or (assoc-ref (package-properties package)
+ 'repology-name)
+ (package-name->repology-name (package-name package))))
+ (info (name->info name)))
+ (if (and info (not (vector-empty? info)))
+ info
+ (let ((info (name->info (package-name->repology-name
+ (package-name package)
+ #:attempt 2))))
+ (if (and info (not (vector-empty? info)))
+ info
+ (begin
+ (warning (G_ "package not found on Repology: ~a\n")
+ (package-name package))
+ #f))))))))
+
+(define (update-version string old-version new-version)
+ "Replace OLD-VERSION in STRING with NEW-VERSION. This assumes that STRING
+contains OLD-VERSION verbatim; if it doesn't, #f is returned."
+ (match (factorize-uri string old-version)
+ ((? string?) #f)
+ ((factorized ...)
+ (apply string-append
+ (map (lambda (component)
+ (match component
+ ('version new-version)
+ ((? string?) component)))
+ factorized)))))
+
+(define (package-source-urls package version)
+ "Return a list of URLs for PACKAGE at VERSION. If no URL was successfully constructed, return #f."
+ (let ((old-version (package-version package)))
+ ;; XXX: (guix upstream) only supports tarballs and Git repos for now.
+ (match (origin-uri (package-source package))
+ ((? git-reference? reference)
+ (and-let* ((old-commit (git-reference-commit reference))
+ (new-commit (if (string=? old-version old-commit)
+ version
+ (update-version old-commit
+ old-version
+ version))))
+ (git-reference
+ (inherit reference)
+ (commit new-commit))))
+ ((? string? url)
+ (list (update-version url old-version version)))
+ ((? list? urls)
+ (map (cut update-version <> old-version version) urls))
+ (_ #f))))
+
+(define (latest-version? repology-package)
+ "Return the latest released version of REPOLOGY-PACKAGE. If none are found,
+return #f."
+ (and (or (equal? "newest" (repology-package-status repology-package))
+ (equal? "unique" (repology-package-status repology-package)))
+ (repology-package-version repology-package)))
+
+;; XXX: We use 'pkg' because 'package' will clash with the 'package' field of
+;; 'upstream-source'.
+(define (repology-latest-release pkg)
+ "Return the latest release of the PKG on Repology named NAME."
+ (and-let* ((packages (repology-fetch-info pkg))
+ (versions (filter-map latest-version?
+ (vector->list packages)))
+ (latest-version (and (pair? versions) (car versions))))
+ ;; TODO: set 'signature-urls'.
+ (upstream-source
+ (package (package-name pkg))
+ (version latest-version)
+ (urls (package-source-urls pkg latest-version)))))
+
+(define %repology-updater
+ (upstream-updater
+ (name 'repology)
+ (description "Updater for packages on Repology")
+ (pred (const #t))
+ (latest repology-latest-release)))
+
+;;; repology.scm ends here
diff --git a/tests/import-repology.scm b/tests/import-repology.scm
new file mode 100644
index 0000000000..2d366db283
--- /dev/null
+++ b/tests/import-repology.scm
@@ -0,0 +1,145 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-import-repology)
+ #:use-module (guix download)
+ #:use-module (guix git-download)
+ #:use-module (guix import repology)
+ #:use-module (guix packages)
+ #:use-module (guix tests)
+ #:use-module (guix upstream)
+ #:use-module (json)
+ #:use-module (srfi srfi-64))
+
+(test-begin "repology")
+
+(define package-using-git-repository
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://git.example.org/foo")
+ (commit "1.0")))
+ (sha256 #f)))))
+
+(define package-using-tarball
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (string-append "https://example.org/foo-" version ".tar.gz"))
+ (sha256 #f)))))
+
+(define package-using-tarball-multiple-urls
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (list (string-append "https://example.org/foo-"
+ version ".tar.gz")
+ (string-append "https://mirror.example.org/foo-"
+ version ".tar.gz")))
+ (sha256 #f)))))
+
+(define %test-json
+"[
+ {
+ \"repo\": \"aur\",
+ \"srcname\": \"foo\",
+ \"binname\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"1.0.r25.gb86405a\",
+ \"maintainers\": [
+ \"bob@aur\"
+ ],
+ \"licenses\": [
+ \"LGPL3+\"
+ ],
+ \"summary\": \"foo bar\"
+ \"status\": \"rolling\",
+ \"origversion\": \"1.0.r25.gb86405a-1\"
+ },
+ {
+ \"repo\": \"gnuguix\",
+ \"srcname\": \"foo\",
+ \"binname\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"1.0\",
+ \"summary\": \"foo bar\",
+ \"status\": \"outdated\",
+ \"origversion\": null
+ },
+ {
+ \"repo\": \"nix_unstable\",
+ \"name\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"2.0\",
+ \"maintainers\": [
+ \"bob@example.org\"
+ ],
+ \"licenses\": [
+ \"LGPL-3.0-or-later\"
+ ],
+ \"summary\": \"foo bar\",
+ \"status\": \"newest\",
+ \"origversion\": null
+ }
+]")
+
+(define (latest-release package)
+ (mock ((guix import json) json-fetch
+ (lambda* (url #:key cached?)
+ (json-string->scm %test-json)))
+ (repology-latest-release package)))
+
+(test-equal "package using Git repo: version"
+ "2.0"
+ (upstream-source-version
+ (latest-release package-using-git-repository)))
+
+(test-equal "package using Git repo: git-reference"
+ (git-reference
+ (url "https://git.example.org/foo")
+ (commit "2.0"))
+ (upstream-source-urls
+ (latest-release package-using-git-repository)))
+
+(test-equal "package using tarball: version"
+ "2.0"
+ (upstream-source-version
+ (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: URL"
+ (list "https://example.org/foo-2.0.tar.gz")
+ (upstream-source-urls
+ (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: multiple URLs"
+ (list "https://example.org/foo-2.0.tar.gz"
+ "https://mirror.example.org/foo-2.0.tar.gz")
+ (upstream-source-urls
+ (latest-release package-using-tarball-multiple-urls)))
+
+(test-end "repology")
--
2.34.1
X
X
Xinglu Chen wrote on 7 Feb 2022 10:07
[PATCH v2 6/7] gnu: xorg-server-xwayland: Set 'repology-name' property.
(address . 53818@debbugs.gnu.org)
d6f6d0a9469679d88fdb08124d144339d73655ad.1644224421.git.public@yoctocell.xyz
* gnu/packages/xorg.scm (xorg-server-xwayland):[properties]: Set
'repology-name'.
---
gnu/packages/xorg.scm | 3 +++
1 file changed, 3 insertions(+)

Toggle diff (23 lines)
diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 14e35d19ae..8be7017da7 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -33,6 +33,7 @@
;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
;;; Copyright © 2021 qblade <qblade@protonmail.com>
;;; Copyright © 2021 Lu Hui <luhux76@gmail.com>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -5520,6 +5521,8 @@ (define-public xorg-server-xwayland
(lambda _
(substitute* (find-files "." "\\.c$")
(("/bin/sh") (which "sh"))))))))
+ (properties
+ '((repology-name . "xwayland")))
(synopsis "Xorg server with Wayland backend")
(description "Xwayland is an X server for running X clients under
Wayland.")
--
2.34.1
X
X
Xinglu Chen wrote on 7 Feb 2022 10:07
[PATCH v2 7/7] gnu: xorg-server-xwayland: Prepare for cross-compilation.
(address . 53818@debbugs.gnu.org)
e87678e75f6be71ec5f5b072ff1507e6e903e50b.1644224421.git.public@yoctocell.xyz
* gnu/packages/xorg.scm (xorg-server-xwayland)[arguments]<#:phases>: Use
‘search-input-file’ instead of ‘which’.
---
gnu/packages/xorg.scm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (19 lines)
diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 8be7017da7..dea7268a2e 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -5518,9 +5518,10 @@ (define-public xorg-server-xwayland
"--localstatedir=/var")
#:phases (modify-phases %standard-phases
(add-after 'unpack 'patch-/bin/sh
- (lambda _
+ (lambda* (#:key inputs #:allow-other-keys)
(substitute* (find-files "." "\\.c$")
- (("/bin/sh") (which "sh"))))))))
+ (("/bin/sh")
+ (search-input-file inputs "bin/sh"))))))))
(properties
'((repology-name . "xwayland")))
(synopsis "Xorg server with Wayland backend")
--
2.34.1
M
M
Maxime Devos wrote on 7 Feb 2022 10:44
Re: [bug#53818] [PATCH v2 4/7] import: json: Make 'json-fetch' take '#:cached?' argument.
afbe3757bf6c4031592e1067ae48c82140be1769.camel@telenet.be
Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
Toggle quote (4 lines)
> +    (let* ((port   ((if cached?
> +                        http-fetch/cached
> +                        http-fetch)

Personally, instead of introdicing a cached? argument,
I would introduce a 'http-fetch' argument defaulting to 'http-fetch',
such that users of 'json-fetch' can set all the options of
'http-fetch/cached' (ttl, cache-miss ...).

(json-fetch #:http-fetch
(lambda (uri . rest)
(apply http-fetch/cached uri #:ttl foo #:cache-miss bar
rest)))

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYgDqDBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7sxwAQDNfL5tyjmkkj6ApQ4VqpzMrZo7
Hi93XM6SPbJ4anLh5wEApsPV7vnDzaY0SkEMb46PFyB7fKejHA0vbblafBOMBQ4=
=+w6J
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 7 Feb 2022 10:45
Re: [bug#53818] [PATCH v2 5/7] import: Add 'repology' updater.
39e000a43e690d53f64dad1317bec5468b696b06.camel@telenet.be
Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
Toggle quote (4 lines)
> +The name of a package in Guix is not always that same as the name on
> +Repology; users can set the @code{repology-name} package property to
> +make the updater use the correct name.

Perhaps note that the updater has some heuristics for determining
the Repology name, so setting 'repology-name' often isn't necessary
even when the name in Guix doesn't match the name in Repology?

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYgDqTRccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7gKdAQCa6apx8Ajef3MEZDv8yh08cofH
NxbUh/XKg3HyvE26EgEAuhy6AnD71nwcSHce8wV+WESSDbYg9o0qE4tafgtTRw8=
=FM4X
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 7 Feb 2022 10:50
76a550f4fc82de9491b99786e8086d9d9d8c4c72.camel@telenet.be
Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
Toggle quote (6 lines)
> +(define (latest-release package)
> +  (mock ((guix import json) json-fetch
> +         (lambda* (url #:key cached?)
> +           (json-string->scm %test-json)))
> +        (repology-latest-release package)))

It would be nice to verify the URL.
FWIW, there's a 'with-http-server' macro.
although it might need to be extended a little, see e.g.

Also, you are using 'memoize' in repology-fetch-info,
so the memoisation cache needs to be invalidated
(see e.g. tests/minetest.scm), or not, depending
on how 'repology-fetch-info' is used and tests are written.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYgDrUhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7mdIAP9XBEBiopGDccsznDNFN7/bhNKU
3jS5oGOO0h9qqUUcdgD9EkOUUU0iinGjNIX2MtSjvKwXaxAyx0xhTSCXMQVMjAc=
=AVGv
-----END PGP SIGNATURE-----


X
X
Xinglu Chen wrote on 8 Feb 2022 13:29
87a6f1zgm1.fsf@yoctocell.xyz
Maxime schrieb am Montag der 07. Februar 2022 um 10:50 +01:

Toggle quote (9 lines)
> Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
>> +(define (latest-release package)
>> +  (mock ((guix import json) json-fetch
>> +         (lambda* (url #:key cached?)
>> +           (json-string->scm %test-json)))
>> +        (repology-latest-release package)))
>
> It would be nice to verify the URL.

What do you mean with “verify”?

Toggle quote (9 lines)
> FWIW, there's a 'with-http-server' macro.
> although it might need to be extended a little, see e.g.
> <https://issues.guix.gnu.org/53389>.
>
> Also, you are using 'memoize' in repology-fetch-info,
> so the memoisation cache needs to be invalidated
> (see e.g. tests/minetest.scm), or not, depending
> on how 'repology-fetch-info' is used and tests are written.

Good catch! Invalidating the cache would be a good idea.
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmICYiYVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5mDQQAKTnfhCvES45yP/a0QdlvLBo2EgW
J8ZAlGqEBlMzkYAlaxkH+xiP0gg7aalzMnarNU1KCAUTovZsk6bYhau+nk8mEQ88
QcyjftLBWLWLKaEvucavQ1M8aGoiuQnVDiRs6hJhlLok3EUau+UEdxCT7clsCBG6
tmwyfcGzKuM1UHrq1cAQ2fr1hMkSBW15WN5ksPXG3ug3M2PJBFgHxqFVEzXCZ9QG
JgT0sE4rTSrFASQpEaUKDBLsII3DYiXgF5ntra8+/B+OrCHoSvwYl6lxBITKq/RV
BXmM1Hrqv6Grjq2mLBb8csU72KpjWoId9WHxqqYwEDmgbQea1/r9A7BUBBfmf2Bo
GIW08cdlutc8hW0bua/UNJbxLEGrhU9Xy/3M6f56/LAXQivdegPSEu3G7rsRzyMw
yFZBp5fmw0oBHnMh6Dmf752Jm5H1VGN5phEjl/PMvsjXoCkyw7Fo6w4CPcCGZMIs
jTSK2nnARTCRfVSzsyNfZq6Ev0+lp9GLiStA+GSOeuxc5+ZhrydaexE71plM86Nv
ykaN1gBCXpVNlPG5VOwPJcA1UnYtDToyoi67iaOh6KaZJX/VVTwhbCxQovezGd+A
zHmuhcMKvYTRqAZYiIW7pa4KGaKHsteU/weBtJieSs8QHMBwA5XHBvuFb9T888kk
eGr7o1or6xz8c3rc
=koqK
-----END PGP SIGNATURE-----

M
M
Maxime Devos wrote on 8 Feb 2022 13:49
3841c2c4ab6dd421237b6b29e9fb746be477380e.camel@telenet.be
Xinglu Chen schreef op di 08-02-2022 om 13:29 [+0100]:
Toggle quote (11 lines)
> > Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
> > > +(define (latest-release package)
> > > +  (mock ((guix import json) json-fetch
> > > +         (lambda* (url #:key cached?)
> > > +           (json-string->scm %test-json)))
> > > +        (repology-latest-release package)))
> >
> > It would be nice to verify the URL.
>
> What do you mean with “verify”

Something along the lines of:

(lambda* (url #:key cached?)
(if (equal? url
"http://[hostname]/release/information/of/PACKAGE.json")
(json-string->scm ...)
(error "the refresher contacted the wrong URL!")))

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYgJm0hccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7qGTAQDjW09xDSUrA1D2i4VliqSyNbQy
Lk9HWvwLA4eKTnxLhwD+JV9Ev7v+6vOS/KlZG265AHxFc9jIbjKrthWx2XS/OwU=
=Zb1R
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 8 Feb 2022 23:59
Re: bug#53818: [PATCH 0/3] Add Repology updater
(name . Xinglu Chen)(address . public@yoctocell.xyz)(address . 53818@debbugs.gnu.org)
87pmnx7ynw.fsf@gnu.org
Hi!

Xinglu Chen <public@yoctocell.xyz> skribis:

Toggle quote (3 lines)
> This patchset adds a new updater, which scans Repology[1] for updates.
> It should technically support all packages in Guix! :-)

I wouldn’t want to spoil the party, but I’m mildly enthusiastic.

Repology implements the same functionality as our updaters, so
repology.org is effectively “service as a software substitute” (SaaSS).

My preference would be to keep our existing updaters rather than
effectively ditch them and delegate the work to Repology. It’s tempting
to think we can have both, but I’m not sure this would last long.

WDYT?

Ludo’.
X
X
Xinglu Chen wrote on 9 Feb 2022 13:52
Re: [bug#53818] [PATCH 0/3] Add Repology updater
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 53818@debbugs.gnu.org)
87y22kxkv3.fsf@yoctocell.xyz
Hi,

Ludovic schrieb am Dienstag der 08. Februar 2022 um 23:59 +01:

Toggle quote (13 lines)
> Hi!
>
> Xinglu Chen <public@yoctocell.xyz> skribis:
>
>> This patchset adds a new updater, which scans Repology[1] for updates.
>> It should technically support all packages in Guix! :-)
>
> I wouldn’t want to spoil the party, but I’m mildly enthusiastic.
>
> Repology implements the same functionality as our updaters, so
> repology.org is effectively “service as a software substitute”
> (SaaSS).

Right, but it tracks a lot more repositories than what our updaters do,
so why not take advantage of that.

Toggle quote (4 lines)
> My preference would be to keep our existing updaters rather than
> effectively ditch them and delegate the work to Repology. It’s tempting
> to think we can have both, but I’m not sure this would last long.

The point of the Repology updater is to act as a fallback if none of
the other updaters can update a package, e.g., ‘maven-dependency-tree’.
I already mentioned that language-specific updaters usually provide more
accurate and detailed information, so they should be used when possible;
we aren’t losing anything here.
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmIDuSAVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5SOAP/0aPD1Uk/NPw0Elv1btTEH6obC9R
mxd7fHgZbmOVCcfnD1AXU5WxfdfsgFaDy9tihYPm39YAr21T+3Wpu6tlmRA5hv5Y
CoTGqKYuAaZZO+xuYvtjyV0wlxHjooGn0QJCEa3h0PvrqmwD8XCKJADpywlkRjXu
WEmMQQs6F8cy56kAVnrySLTGw4eVz1BoyCEsIX8SVHBVIk14pw0hBCmqtf1pC5cw
BAO3flGmh1IopyMg05Np+uodI84oHobQu4PZt94IWJyQK2fSnur4QS1C9qcTXf3j
93Ma4hy3KGshhNioaw4zcLXKBngiNyr3JkUq7qyuHaDPj40mzB1gBU1KSNTNPpMb
bYDQgNzp5Zc8rY0/RbLCo84183n52Hxi9MYsGYu67gDCaE4+3NVy3YcdL6lFeCPQ
jwGpkV5nt5/FBat9flc0qGmNyPhXYktASaZQi8YGCzC4EzYnoFsaLD2zmmJtLj2M
n8AefbpQjGo1LUiRpFKpGQJz5zWWG1IuYkDNAesJ1h2AvbW/+r/ywvMnFm7ChbBy
fKoZFSW5AhKC7IqtB4A1yPnpWMDRCvtUPig4u46LTYg8fASAMMHTyYM4z9D1bigH
4HAtXMIDYKxZHCeTlj+4/eNNluNX5tFCeB3BPfq5jB+oqlVvqliU1ptlyLjuM0Od
FenBTIheprFefjiE
=bFVw
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 9 Feb 2022 13:54
Re: [bug#53818] [PATCH v2 5/7] import: Add 'repology' updater.
87wni4xksc.fsf@yoctocell.xyz
Maxime schrieb am Dienstag der 08. Februar 2022 um 13:49 +01:

Toggle quote (20 lines)
> Xinglu Chen schreef op di 08-02-2022 om 13:29 [+0100]:
>> > Xinglu Chen schreef op ma 07-02-2022 om 10:07 [+0100]:
>> > > +(define (latest-release package)
>> > > +  (mock ((guix import json) json-fetch
>> > > +         (lambda* (url #:key cached?)
>> > > +           (json-string->scm %test-json)))
>> > > +        (repology-latest-release package)))
>> >
>> > It would be nice to verify the URL.
>>
>> What do you mean with “verify”
>
> Something along the lines of:
>
> (lambda* (url #:key cached?)
> (if (equal? url
> "http://[hostname]/release/information/of/PACKAGE.json")
> (json-string->scm ...)
> (error "the refresher contacted the wrong URL!")))

Ah, OK, thanks for clarifying!
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmIDuYQVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5bHcP/1LMGxz5R88mpiGvHMXEnpVPadry
eeKrmZfmiLHjHXAlKGFIVwjVvp0F4MaU8oFuRzk+h++9MZoggp2LaISQdAe4AV8f
9ecU9UC1AGaclOmXRQTgi2ugInfLCIA2T3qh9p2Ed4FPw8TGMl6Dq8HLpnxMGIeO
ARAoiu1JH0NDvMKz0qulg2JtDKytAtqIOjLP5sU9682tXfn7HbFF6MnLI5jzqrCC
zCBWZ9rgUMuzLoHofbdG+UxFveGy3IOg7Ps17iTH/1MX8eeDaJvJLC9vkSbzW+0P
PaQPa7resQ9Ye/vjo2pDXTk0d4W2ig1j0lraZW5flMO1j9qfLgXjqgnrGLyaWRI4
Qg2MsS5kRYrPefSCgLHj89qzPGPASfunJEgZvCM9fsrRs+hmNwLA5ZAS9CHWR2Q4
0YYsXhW0SfmUmSVtCKblVlG8/EUz6fTe9C55cqN2QGJoZe+2akjemeXvkd+5w/uG
T1unWuL3XIKj/ub4U0FwW+JjgL87BCIlewBGBAI2EqiQcnQ6oLsrP/ZPUzAHrmX+
MLNQRZyJSzottnFYNkk/DtUEpikbvdq0DJcLreoMJyloXE9yFoo3EaCJ6tWSJWXZ
PylttRjVm0rpBD0hxTYKfj17ggRwau2RVOjkRB8QiCFUJWdSvpY0UMJKgu+nzxYx
B4blmmBo0ST4Kyyj
=ksoR
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 9 Feb 2022 14:22
[PATCH v3 0/7] Add Repology updater
(address . 53818@debbugs.gnu.org)
cover.1644412701.git.public@yoctocell.xyz
Changes since v2:

* Change ‘#:cached?’ argument to ‘#:http-fetch’.

* Invalidate cache in tests and make sure the requested URL is the one
we expect.

* Expound a bit on the documentation in the manual.

* Some more rules for guessing the Repology name of a package.

Xinglu Chen (7):
upstream: Sort list of updaters.
http-client: Make 'http-fetch/cached' take '#:headers' argument.
http-client: 'http-fetch/cached' accepts a string or a <uri>.
import: json: Make 'json-fetch' take '#:http-fetch' argument.
import: Add 'repology' updater.
gnu: xorg-server-xwayland: Set 'repology-name' property.
gnu: xorg-server-xwayland: Prepare for cross-compilation.

Makefile.am | 3 +
doc/guix.texi | 8 ++
gnu/packages/xorg.scm | 8 +-
guix/http-client.scm | 18 ++-
guix/import/json.scm | 9 +-
guix/import/repology.scm | 249 ++++++++++++++++++++++++++++++++++++++
guix/upstream.scm | 29 ++++-
tests/import-repology.scm | 150 +++++++++++++++++++++++
8 files changed, 457 insertions(+), 17 deletions(-)
create mode 100644 guix/import/repology.scm
create mode 100644 tests/import-repology.scm


base-commit: ad9cd004c81a01e33a605221a102f4d20abe442d
--
2.34.1
-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmIDwBMVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5ILkQAKI8fX1+49Gc4kh4Z8evUNFoLmmu
VdbRLD7nXgKyqFN1YZf4ybL4C/EI15lBQOMhgiYvPUtTle1px2QvL7WxHk4/Q+Sw
m0O2A4pv5E9gG38Wg3yu1gpM3g5HGOWa4Q8hAlFo/CZJlJvRRebJghm7QVETq9LM
BjQxuIgxCCr7+t5psF/Vol0WK0tVemxC7PotXjMNWKzWpAcjiDPs6sFhcnX6ezDm
0UN3z8lreQ3isHUwg8+1LcfMZs9iew7/YojJytYepDUEMCVNyC90XP9SqagpIfDv
KA3wDzIep22Xy3KKyEmBVsVbhcIkdCk8ctmPGMqY5VLq3wWNh9VeycTgzZr7ZVW7
DFlGE2gL3f+U4AfQ9AWtV3Xb9vh7xTH9cikrPOxwGYfjcc9T4jyZ8brgZVP6cXe0
+pxph2MkANaqiWEwFW0FRT3x56dAwg4WA04w3AJueuZJVGRL6Jsofi5378yUTmYy
OhDiq5jK5dMlqq+1ianKFTaBKpEFKOCvtONSJ/hgAirNEMrCtUFJnKv72FECOVqC
+Bk5VoB2S3WLANpoAP4LoAqAFGxNjLGqiOlSWlkDInflqN0uklEUYaWypmNuu8xi
8F06QOiK5XiI8Uuo9IReDtFg3IJ+Gtde+jI44NlDRBkSkoswM/D3JET75nnO2ocL
KlbnNA8nXwG2ZE62
=PsR7
-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 9 Feb 2022 14:24
[PATCH v3 1/7] upstream: Sort list of updaters.
(address . 53818@debbugs.gnu.org)
0f119f2aaf1f3d492bdf733136c310a9f1a6d85d.1644412701.git.public@yoctocell.xyz
The order determines which updater gets used first when running ‘guix refresh’
on a package. We want the most generic updaters to be last since they usually
don’t provide as much or accurate information as language-specific updaters.

* guix/upstream.scm (sort-updaters): New procedure.
(%updaters): Use it.
---
guix/upstream.scm | 29 +++++++++++++++++++++++------
1 file changed, 23 insertions(+), 6 deletions(-)

Toggle diff (44 lines)
diff --git a/guix/upstream.scm b/guix/upstream.scm
index 6666803a92..d5faf9d3ee 100644
--- a/guix/upstream.scm
+++ b/guix/upstream.scm
@@ -250,14 +250,31 @@ (define (importer-modules)
%load-path)
#:warn warn-about-load-error)))
+(define (sort-updaters updaters)
+ "Sort UPDATERS by putting the more generic ones last."
+ (define (genericity updater)
+ (cond
+ ((equal? 'repology (upstream-updater-name updater))
+ 2)
+ ((string-prefix? "generic-"
+ (symbol->string (upstream-updater-name updater)))
+ 1)
+ (else 0)))
+
+ (define (less a b)
+ (<= (genericity a) (genericity b)))
+
+ (stable-sort updaters less))
+
(define %updaters
;; The list of publically-known updaters.
- (delay (fold-module-public-variables (lambda (obj result)
- (if (upstream-updater? obj)
- (cons obj result)
- result))
- '()
- (importer-modules))))
+ (delay (sort-updaters
+ (fold-module-public-variables (lambda (obj result)
+ (if (upstream-updater? obj)
+ (cons obj result)
+ result))
+ '()
+ (importer-modules)))))
;; Tests need to mock this variable so mark it as "non-declarative".
(set! %updaters %updaters)
--
2.34.1
X
X
Xinglu Chen wrote on 9 Feb 2022 14:24
[PATCH v3 2/7] http-client: Make 'http-fetch/cached' take '#:headers' argument.
(address . 53818@debbugs.gnu.org)
0ad40732008ce4a9214df154d431741c90e0f128.1644412701.git.public@yoctocell.xyz
* guix/http-client.scm (http-fetch/cached): Take ‘#:headers’ keyword argument.
---
guix/http-client.scm | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)

Toggle diff (48 lines)
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 10bc278023..058f09852f 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2012, 2015 Free Software Foundation, Inc.
;;; Copyright © 2017 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -283,9 +284,11 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
(write-cache dump-port)
(cache-miss (const #t))
(log-port (current-error-port))
- (timeout 10))
+ (timeout 10)
+ (headers '((user-agent . "GNU Guile"))))
"Like 'http-fetch', return an input port, but cache its contents in
-~/.cache/guix. The cache remains valid for TTL seconds.
+~/.cache/guix. The cache remains valid for TTL seconds. HEADERS is an alist
+of extra HTTP headers. The cache time will automatically be added to HEADERS, so there is no need to specify it.
Call WRITE-CACHE with the HTTP input port and the cache output port to write
the data to cache. Call CACHE-MISS with URI just before fetching data from
@@ -300,8 +303,8 @@ (define cache-time
(and cache-port
(stat:mtime (stat cache-port))))
- (define headers
- `((user-agent . "GNU Guile")
+ (define headers*
+ `(,@headers
,@(if cache-time
`((if-modified-since
. ,(time-utc->date (make-time time-utc 0 cache-time))))
@@ -316,7 +319,7 @@ (define headers
(raise c))))
(let ((port (http-fetch uri #:text? text?
#:log-port log-port
- #:headers headers #:timeout timeout)))
+ #:headers headers* #:timeout timeout)))
(cache-miss uri)
(mkdir-p (dirname file))
(when cache-port
--
2.34.1
X
X
Xinglu Chen wrote on 9 Feb 2022 14:24
[PATCH v3 3/7] http-client: 'http-fetch/cached' accepts a string or a <uri>.
(address . 53818@debbugs.gnu.org)
7be659f7ef029ee77df683f3f522f9f51fe40ba5.1644412701.git.public@yoctocell.xyz
This is consistent with the 'http-fetch' procedure.

* guix/http-client.scm (http-fetch/cached): The 'uri' argument can be a string
or a <uri> record.
---
guix/http-client.scm | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

Toggle diff (18 lines)
diff --git a/guix/http-client.scm b/guix/http-client.scm
index 058f09852f..08efdd5e47 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -297,7 +297,10 @@ (define* (http-fetch/cached uri #:key (ttl (%http-cache-ttl)) text?
TIMEOUT specifies the timeout in seconds for connection establishment.
Write information about redirects to LOG-PORT."
- (let ((file (cache-file-for-uri uri)))
+ (let* ((uri (if (string? uri)
+ (string->uri uri)
+ uri))
+ (file (cache-file-for-uri uri)))
(define (update-cache cache-port)
(define cache-time
(and cache-port
--
2.34.1
X
X
Xinglu Chen wrote on 9 Feb 2022 14:25
[PATCH v3 4/7] import: json: Make 'json-fetch' take '#:http-fetch' argument.
(address . 53818@debbugs.gnu.org)
f0eec52af67aa71aa686ec14d5a49b621e95c779.1644412701.git.public@yoctocell.xyz
This will let users of the 'json-fetch' procedure use a custom procedure for
fetching the URL, e.g., 'http-fetch/cached'.

* json.scm (json-fetch): Add ‘#:http-fetch’ keyword argument.

Suggested-by: Maxime Devos <maximedevos@telenet.be>
---
guix/import/json.scm | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)

Toggle diff (31 lines)
diff --git a/guix/import/json.scm b/guix/import/json.scm
index 0c98bb25b8..de7ed60e8d 100644
--- a/guix/import/json.scm
+++ b/guix/import/json.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2015, 2016 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2018, 2019 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -37,11 +38,13 @@ (define-module (guix import json)
(define* (json-fetch url
;; Note: many websites returns 403 if we omit a
;; 'User-Agent' header.
- #:key (headers `((user-agent . "GNU Guile")
- (Accept . "application/json"))))
+ #:key
+ (headers `((user-agent . "GNU Guile")
+ (Accept . "application/json")))
+ (http-fetch http-fetch))
"Return a representation of the JSON resource URL (a list or hash table), or
#f if URL returns 403 or 404. HEADERS is a list of HTTP headers to pass in
-the query."
+the query. HTTP-FETCH is the procedure that is used to fetch the JSON data from URL. It should take the URL as an argument and HEADERS as a keyword argument, and return an input port for the JSON data."
(guard (c ((and (http-get-error? c)
(let ((error (http-get-error-code c)))
(or (= 403 error)
--
2.34.1
X
X
Xinglu Chen wrote on 9 Feb 2022 14:25
[PATCH v3 5/7] import: Add 'repology' updater.
(address . 53818@debbugs.gnu.org)
452009a171299629965ab860eac2a1fdbe8a3554.1644412701.git.public@yoctocell.xyz
* guix/import/repology.scm
* tests/import-repology.scm: New files.
* Makefile.am (MODULES): Register them.
* doc/guix.texi (Invoking guix refresh): Document it.
---
Makefile.am | 3 +
doc/guix.texi | 8 ++
guix/import/repology.scm | 249 ++++++++++++++++++++++++++++++++++++++
tests/import-repology.scm | 150 +++++++++++++++++++++++
4 files changed, 410 insertions(+)
create mode 100644 guix/import/repology.scm
create mode 100644 tests/import-repology.scm

Toggle diff (460 lines)
diff --git a/Makefile.am b/Makefile.am
index 7463606d20..6792917b59 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -16,6 +16,7 @@
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
# Copyright © 2021 Andrew Tropin <andrew@trop.in>
+# Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
#
# This file is part of GNU Guix.
#
@@ -271,6 +272,7 @@ MODULES = \
guix/import/opam.scm \
guix/import/print.scm \
guix/import/pypi.scm \
+ guix/import/repology.scm \
guix/import/stackage.scm \
guix/import/texlive.scm \
guix/import/utils.scm \
@@ -488,6 +490,7 @@ SCM_TESTS = \
tests/home-import.scm \
tests/import-git.scm \
tests/import-github.scm \
+ tests/import-repology.scm \
tests/import-utils.scm \
tests/inferior.scm \
tests/lint.scm \
diff --git a/doc/guix.texi b/doc/guix.texi
index 583ba1c61d..2d7612b09a 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12932,6 +12932,14 @@
(release-tag-version-delimiter . ":"))))
@end lisp
+@item repology
+an updater that scans @uref{https://repology.org, Repology}, a website
+that tracks packages on various package repositories, for updates.
+
+The name of a package in Guix is not always that same as the name on
+Repology. In most cases, the updater will be able to guess the name
+correctly. If it doesn’t, users can set the @code{repology-name}
+package property.
@end table
diff --git a/guix/import/repology.scm b/guix/import/repology.scm
new file mode 100644
index 0000000000..87fbd2ee6f
--- /dev/null
+++ b/guix/import/repology.scm
@@ -0,0 +1,249 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import repology)
+ #:use-module (guix diagnostics)
+ #:use-module (guix git-download)
+ #:use-module (guix http-client)
+ #:use-module (guix i18n)
+ #:use-module (guix import json)
+ #:use-module (guix import utils)
+ #:use-module (guix memoization)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (guix utils)
+ #:use-module (ice-9 match)
+ #:use-module (json)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-2)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-43)
+ #:export (%repology-url
+ repology-fetch-info
+ repology-latest-release
+ %repology-updater))
+
+;;; Commentary:
+;;;
+;;; This module provides an updater which scans Repology, a site that monitors
+;;; several package repolsitories, for updates. This means that if any other
+;;; package repository has a version of a package that is newer than the
+;;; version in Guix, the package should be able to be updated. The updater
+;;; should in theory work for all packages in Guix, but the names of some
+;;; packages on Repology don't match the name in Guix. The 'repology-name'
+;;; package property can be used to fix this.
+;;;
+;;; Guix already has many different updaters for language-specific packages,
+;;; and these typically provide more accurate data, e.g., input changes,
+;;; signature URLs. The Repology updater should really be treated as a last
+;;; resort for those packages that don't have any other updater to rely on.
+;;;
+;;; See <https://repology.org/api/v1> for the API.
+;;;
+;;; Code:
+
+(define %repology-url
+ "https://repology.org/api/v1/project")
+
+(define* (package-name->repology-name name #:key (attempt 1))
+ "Convert NAME, the name of a Guix package, to the name of the package on
+Repology. It doesn't always guess the correct name on the first attempt, so
+on the second attempt it will try to guess another name."
+ (match attempt
+ (1 (cond
+ ((string-prefix? "ghc-" name)
+ (string-append "haskell:"
+ (string-drop name (string-length "ghc-"))))
+ ((string-prefix? "ocaml-" name)
+ (string-append "ocaml:"
+ (string-drop name (string-length "ocaml-"))))
+ ((string-prefix? "perl-" name)
+ (string-append "perl:"
+ (string-drop name (string-length "perl-"))))
+ ((string-prefix? "emacs-" name)
+ (string-append "emacs:"
+ (string-drop name (string-length "emacs-"))))
+ ((string-prefix? "go-" name)
+ (string-append "go:"
+ (string-drop name (string-length "go-"))))
+ ((string-prefix? "rust-" name)
+ (string-append "rust:"
+ (string-drop name (string-length "rust-"))))
+ ((string-prefix? "lua-" name)
+ (string-append "lua:"
+ (string-drop name (string-length "lua-"))))
+ ((string-prefix? "node-" name)
+ (string-append "node:"
+ (string-drop name (string-length "node-"))))
+ ((string-prefix? "python-" name)
+ (string-append "python:"
+ (string-drop name (string-length "python-"))))
+ ((string-prefix? "java-" name)
+ (string-append "java:"
+ (string-drop name (string-length "java-"))))
+ ((string-prefix? "r-" name)
+ (string-append "r:"
+ (string-drop name (string-length "r-"))))
+ ((string-prefix? "ruby-" name)
+ (string-append "ruby:"
+ (string-drop name (string-length "ruby-"))))
+ ((string-prefix? "xf86-" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-"))))
+ ((string-prefix? "font-" name)
+ (string-append "fonts:"
+ (string-drop name (string-length "font-"))))
+ ((string-prefix? "trytond-" name)
+ (string-append "tryton:"
+ (string-drop name (string-length "trytond-"))))
+ ((string-prefix? "python-trytond-" name)
+ (string-append "tryton:"
+ (string-drop name (string-length "python-trytond-"))))
+ ((string-suffix? "-minimal" name)
+ (string-drop-right name (string-length "-minimal")))
+ (else name)))
+ (2 (cond
+ ((string-prefix? "xf86-video" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-video-"))))
+ ((string-prefix? "xf86-input" name)
+ (string-append "xdrv:"
+ (string-drop name (string-length "xf86-input-"))))
+ ((string-prefix? "minetest-" name)
+ (string-append "minetest-mod-"
+ (string-drop name (string-length "minetest-"))))
+ ((string-prefix? "lib" name)
+ (string-drop name (string-length "lib")))
+ ((string-prefix? "vim-" name)
+ (string-append "vim:"
+ (string-drop name (string-length "vim-"))))
+ (else name)))))
+
+
+;;; JSON mappings.
+
+(define-json-mapping <repology-package> make-repology-package
+ repology-package?
+ json->repology-package
+ (repository repology-package-repository "repo")
+ (src-name repology-package-src-name "srcname")
+ (binary-name repology-package-binary-name "binname")
+ (visible-name repology-package-visible-name "visiblename")
+ (version repology-package-version)
+ (original-version repology-package-original-version "origversion")
+ (status repology-package-status)
+ (summary repology-package-summary)
+ (categories repology-package-categories)
+ (licenses repology-package-licenses)
+ (maintainers repology-package-maintainers))
+
+
+;;; Updater.
+
+(define repology-fetch-info
+ (memoize
+ (lambda (package)
+ "Fetch information about PACKAGE using the Repology API."
+ (define (name->info name)
+ (let ((url (string-append %repology-url "/" name)))
+ (and=> (json-fetch url #:http-fetch http-fetch/cached)
+ (lambda (url)
+ (vector-map (lambda (a b)
+ (json->repology-package b))
+ url)))))
+
+ (let* ((name (or (assoc-ref (package-properties package)
+ 'repology-name)
+ (package-name->repology-name (package-name package))))
+ (info (name->info name)))
+ (if (and info (not (vector-empty? info)))
+ info
+ (let ((info (name->info (package-name->repology-name
+ (package-name package)
+ #:attempt 2))))
+ (if (and info (not (vector-empty? info)))
+ info
+ (begin
+ (warning (G_ "package not found on Repology: ~a\n")
+ (package-name package))
+ #f))))))))
+
+(define (update-version string old-version new-version)
+ "Replace OLD-VERSION in STRING with NEW-VERSION. This assumes that STRING
+contains OLD-VERSION verbatim; if it doesn't, #f is returned."
+ (match (factorize-uri string old-version)
+ ((? string?) #f)
+ ((factorized ...)
+ (apply string-append
+ (map (lambda (component)
+ (match component
+ ('version new-version)
+ ((? string?) component)))
+ factorized)))))
+
+(define (package-source-urls package version)
+ "Return a list of URLs for PACKAGE at VERSION. If no URL was successfully constructed, return #f."
+ (and-let* ((old-version (package-version package))
+ (source (package-source package)))
+ ;; XXX: (guix upstream) only supports tarballs and Git repos for now.
+ (match (origin-uri source)
+ ((? git-reference? reference)
+ (and-let* ((old-commit (git-reference-commit reference))
+ (new-commit (if (string=? old-version old-commit)
+ version
+ (update-version old-commit
+ old-version
+ version))))
+ (git-reference
+ (inherit reference)
+ (commit new-commit))))
+ ((? string? url)
+ (list (update-version url old-version version)))
+ ((? list? urls)
+ (map (cut update-version <> old-version version) urls))
+ (_ #f))))
+
+(define (latest-version? repology-package)
+ "Return the latest released version of REPOLOGY-PACKAGE. If none are found,
+return #f."
+ (and (or (equal? "newest" (repology-package-status repology-package))
+ (equal? "unique" (repology-package-status repology-package)))
+ (repology-package-version repology-package)))
+
+;; XXX: We use 'pkg' because 'package' will clash with the 'package' field of
+;; 'upstream-source'.
+(define (repology-latest-release pkg)
+ "Return the latest release of the PKG on Repology named NAME."
+ (and-let* ((packages (repology-fetch-info pkg))
+ (versions (filter-map latest-version?
+ (vector->list packages)))
+ (latest-version (and (pair? versions) (car versions))))
+ ;; TODO: set 'signature-urls'.
+ (upstream-source
+ (package (package-name pkg))
+ (version latest-version)
+ (urls (package-source-urls pkg latest-version)))))
+
+(define %repology-updater
+ (upstream-updater
+ (name 'repology)
+ (description "Updater for packages on Repology")
+ (pred (const #t))
+ (latest repology-latest-release)))
+
+;;; repology.scm ends here
diff --git a/tests/import-repology.scm b/tests/import-repology.scm
new file mode 100644
index 0000000000..4da01a4106
--- /dev/null
+++ b/tests/import-repology.scm
@@ -0,0 +1,150 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-import-repology)
+ #:use-module (guix download)
+ #:use-module (guix git-download)
+ #:use-module (guix import repology)
+ #:use-module (guix memoization)
+ #:use-module (guix packages)
+ #:use-module (guix tests)
+ #:use-module (guix upstream)
+ #:use-module (json)
+ #:use-module (srfi srfi-64))
+
+(test-begin "repology")
+
+(define package-using-git-repository
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://git.example.org/foo")
+ (commit "1.0")))
+ (sha256 #f)))))
+
+(define package-using-tarball
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (string-append "https://example.org/foo-" version ".tar.gz"))
+ (sha256 #f)))))
+
+(define package-using-tarball-multiple-urls
+ (dummy-package
+ "foo"
+ (version "1.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (list (string-append "https://example.org/foo-"
+ version ".tar.gz")
+ (string-append "https://mirror.example.org/foo-"
+ version ".tar.gz")))
+ (sha256 #f)))))
+
+(define %test-json
+"[
+ {
+ \"repo\": \"aur\",
+ \"srcname\": \"foo\",
+ \"binname\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"1.0.r25.gb86405a\",
+ \"maintainers\": [
+ \"bob@aur\"
+ ],
+ \"licenses\": [
+ \"LGPL3+\"
+ ],
+ \"summary\": \"foo bar\"
+ \"status\": \"rolling\",
+ \"origversion\": \"1.0.r25.gb86405a-1\"
+ },
+ {
+ \"repo\": \"gnuguix\",
+ \"srcname\": \"foo\",
+ \"binname\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"1.0\",
+ \"summary\": \"foo bar\",
+ \"status\": \"outdated\",
+ \"origversion\": null
+ },
+ {
+ \"repo\": \"nix_unstable\",
+ \"name\": \"foo\",
+ \"visiblename\": \"foo\",
+ \"version\": \"2.0\",
+ \"maintainers\": [
+ \"bob@example.org\"
+ ],
+ \"licenses\": [
+ \"LGPL-3.0-or-later\"
+ ],
+ \"summary\": \"foo bar\",
+ \"status\": \"newest\",
+ \"origversion\": null
+ }
+]")
+
+(define (latest-release package)
+ (invalidate-memoization! repology-fetch-info)
+ (mock ((guix import json) json-fetch
+ (lambda* (url #:key http-fetch)
+ (if (string=? url
+ (string-append %repology-url "/foo"))
+ (json-string->scm %test-json)
+ (error "the URL is not correct"))))
+ (repology-latest-release package)))
+
+(test-equal "package using Git repo: version"
+ "2.0"
+ (upstream-source-version
+ (latest-release package-using-git-repository)))
+
+(test-equal "package using Git repo: git-reference"
+ (git-reference
+ (url "https://git.example.org/foo")
+ (commit "2.0"))
+ (upstream-source-urls
+ (latest-release package-using-git-repository)))
+
+(test-equal "package using tarball: version"
+ "2.0"
+ (upstream-source-version
+ (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: URL"
+ (list "https://example.org/foo-2.0.tar.gz")
+ (upstream-source-urls
+ (latest-release package-using-tarball)))
+
+(test-equal "package using tarball: multiple URLs"
+ (list "https://example.org/foo-2.0.tar.gz"
+ "https://mirror.example.org/foo-2.0.tar.gz")
+ (upstream-source-urls
+ (latest-release package-using-tarball-multiple-urls)))
+
+(test-end "repology")
--
2.34.1
X
X
Xinglu Chen wrote on 9 Feb 2022 14:25
[PATCH v3 6/7] gnu: xorg-server-xwayland: Set 'repology-name' property.
(address . 53818@debbugs.gnu.org)
683fa3b5867f02268b9e6d09c2f93c749d0f75fb.1644412701.git.public@yoctocell.xyz
* gnu/packages/xorg.scm (xorg-server-xwayland):[properties]: Set
'repology-name'.
---
gnu/packages/xorg.scm | 3 +++
1 file changed, 3 insertions(+)

Toggle diff (23 lines)
diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 14e35d19ae..8be7017da7 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -33,6 +33,7 @@
;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
;;; Copyright © 2021 qblade <qblade@protonmail.com>
;;; Copyright © 2021 Lu Hui <luhux76@gmail.com>
+;;; Copyright © 2022 Xinglu Chen <public@yoctocell.xyz>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -5520,6 +5521,8 @@ (define-public xorg-server-xwayland
(lambda _
(substitute* (find-files "." "\\.c$")
(("/bin/sh") (which "sh"))))))))
+ (properties
+ '((repology-name . "xwayland")))
(synopsis "Xorg server with Wayland backend")
(description "Xwayland is an X server for running X clients under
Wayland.")
--
2.34.1
X
X
Xinglu Chen wrote on 9 Feb 2022 14:25
[PATCH v3 7/7] gnu: xorg-server-xwayland: Prepare for cross-compilation.
(address . 53818@debbugs.gnu.org)
005f249032a707f538768f1b6059706c45e50d24.1644412701.git.public@yoctocell.xyz
* gnu/packages/xorg.scm (xorg-server-xwayland)[arguments]<#:phases>: Use
‘search-input-file’ instead of ‘which’.
---
gnu/packages/xorg.scm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (19 lines)
diff --git a/gnu/packages/xorg.scm b/gnu/packages/xorg.scm
index 8be7017da7..dea7268a2e 100644
--- a/gnu/packages/xorg.scm
+++ b/gnu/packages/xorg.scm
@@ -5518,9 +5518,10 @@ (define-public xorg-server-xwayland
"--localstatedir=/var")
#:phases (modify-phases %standard-phases
(add-after 'unpack 'patch-/bin/sh
- (lambda _
+ (lambda* (#:key inputs #:allow-other-keys)
(substitute* (find-files "." "\\.c$")
- (("/bin/sh") (which "sh"))))))))
+ (("/bin/sh")
+ (search-input-file inputs "bin/sh"))))))))
(properties
'((repology-name . "xwayland")))
(synopsis "Xorg server with Wayland backend")
--
2.34.1
N
N
Nicolas Goaziou wrote on 9 Feb 2022 15:29
Re: [bug#53818] [PATCH 0/3] Add Repology updater
(name . Xinglu Chen)(address . public@yoctocell.xyz)
87k0e4w1to.fsf@nicolasgoaziou.fr
Hello,

Xinglu Chen <public@yoctocell.xyz> writes:

Toggle quote (6 lines)
> The point of the Repology updater is to act as a fallback if none of
> the other updaters can update a package, e.g., ‘maven-dependency-tree’.
> I already mentioned that language-specific updaters usually provide more
> accurate and detailed information, so they should be used when possible;
> we aren’t losing anything here.

One issue is that such an updater will introduce frequent false
positives. It is common for Repology to get the latest release wrong,
because some distribution is doing fancy versioning, or because
different distributions disagree about what is upstream.

I don't think we can rely on Repology's "newest" status. The updater may
need to provide its own version comparison tool, because Repology's tool
and Guix versioning do not play nice, in particular when using
`git-version'.

Regards,
--
Nicolas Goaziou
X
X
Xinglu Chen wrote on 10 Feb 2022 19:17
(name . Nicolas Goaziou)(address . mail@nicolasgoaziou.fr)
87bkzewpqy.fsf@yoctocell.xyz
Nicolas schrieb am Mittwoch der 09. Februar 2022 um 15:29 +01:

Toggle quote (15 lines)
> Hello,
>
> Xinglu Chen <public@yoctocell.xyz> writes:
>
>> The point of the Repology updater is to act as a fallback if none of
>> the other updaters can update a package, e.g., ‘maven-dependency-tree’.
>> I already mentioned that language-specific updaters usually provide more
>> accurate and detailed information, so they should be used when possible;
>> we aren’t losing anything here.
>
> One issue is that such an updater will introduce frequent false
> positives. It is common for Repology to get the latest release wrong,
> because some distribution is doing fancy versioning, or because
> different distributions disagree about what is upstream.

Yeah, I have noticed that it sometimes thinks that a version like
“20080323” is newer than something like “0.1.2-0.a1b2b3d”, even though
it might not necessarily be true. This seems to be the case for a lot
of Common Lisp packages which usually don’t have any proper releases.

Toggle quote (5 lines)
> I don't think we can rely on Repology's "newest" status. The updater may
> need to provide its own version comparison tool, because Repology's tool
> and Guix versioning do not play nice, in particular when using
> `git-version'.

In my testing, the “newest” status does a pretty good job (besides the
problem I mentioned above)

Some other “bad” updates I found[*] are listed below (excluding Common Lisp
packages).

Toggle snippet (14 lines)
guile-ac-d-bus would be upgraded from 1.0.0-beta.0 to 1.0.0-beta0
sic would be upgraded from 1.2 to 1.2+20210506_058547e
tla2tools would be upgraded from 1.7.1-0.6932e19 to 20140313
quickjs would be upgraded from 2021-03-27 to 2021.03.27
stow would be upgraded from 2.3.1 to 2.3.1+5.32
cube would be upgraded from 4.3.5 to 2005.08.29
python-ratelimiter would be upgraded from 1.2.0 to 1.2.0.post0
gr-osmosdr would be upgraded from 0.2.3-0.a100eb0 to 0.2.3.20210128
countdown would be upgraded from 1.0.0 to 20150606
http-parser would be upgraded from 2.9.4-1.ec8b5ee to 2.9.4.20201223
xlsx2csv would be upgraded from 0.7.4 to 20200427211949
keynav would be upgraded from 0.20110708.0 to 20150730+4ae486d

It seems like most of these could be solved by checking if the version
scheme changed from semver to calver. I think that’s a pretty good
result considering how many packages we have.

-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmIFVqUVHHB1YmxpY0B5
b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5keoQALFZ7lM02DNnOm/wWSH1hTDfkmhw
5oSl96b7q4Sk2LlCPy3BX9NxqUYRLJgHR95m7UQU+1nJeFL8GLvguCc2FJJ/06i3
22wAnZglqrdrnpkf2THx3RoDVCZFTgreJFJczpI08DftvdPbDk6WVuMP/vGxiva+
Gmm2Nhf7ldlyTIXJIHmBRzvBNjKc9qFo0qD4HkodTACFHsQ93Bsm1SjXGcxcggyq
6HXcRzXLXQeu8XwLE1fyIZHkoicO2TZddVZxXYQlggkZTZvxhN1U4gOlYRUrzPWz
Ho+4Be0Cy4GBMsM1JCVbietZa/gW62CpTrGTwCEcH35I1TnYbk4MBOXwY1gPdz7v
R9vdPlEotayxevcQNEnI3KUYi0O79fXXbAJX0Mjy0Vk20TdMWfIAlEIrBJXOKLMx
qsrYOUDcpt2jgrKLvyxj2D7OmPuMXnnd/uk4TcXIdMEh5f2+a7Ae4PvBE0sQnaKw
zsnWJ0AucUSi7/WzEOZn2SKj/IGGkLwkHl2iXlGK9onRZjRcBcjE7Oz4sVVIzdVS
gNLWwnV2gEHqTKpURldx0W9hS9cMHwVaomfUcwekNkxrHDEWHfLiu+qrYfBf+UWg
9ef8pTMvKwdEnnNMmggjoM5lBtMIkCLHWe9VjXrBkOcfa0bRME3YoxsiJJehJWB/
lH5IjK61Ndbo8r6g
=oOl3
-----END PGP SIGNATURE-----

N
N
Nicolas Goaziou wrote on 10 Feb 2022 20:30
(name . Xinglu Chen)(address . public@yoctocell.xyz)
87bkzev7sb.fsf@nicolasgoaziou.fr
Hello,

Xinglu Chen <public@yoctocell.xyz> writes:

Toggle quote (5 lines)
> Yeah, I have noticed that it sometimes thinks that a version like
> “20080323” is newer than something like “0.1.2-0.a1b2b3d”, even though
> it might not necessarily be true. This seems to be the case for a lot
> of Common Lisp packages which usually don’t have any proper releases.

[...]

Toggle quote (25 lines)
> In my testing, the “newest” status does a pretty good job (besides the
> problem I mentioned above)
>
> Some other “bad” updates I found[*] are listed below (excluding Common Lisp
> packages).
>
> --8<---------------cut here---------------start------------->8---
> guile-ac-d-bus would be upgraded from 1.0.0-beta.0 to 1.0.0-beta0
> sic would be upgraded from 1.2 to 1.2+20210506_058547e
> tla2tools would be upgraded from 1.7.1-0.6932e19 to 20140313
> quickjs would be upgraded from 2021-03-27 to 2021.03.27
> stow would be upgraded from 2.3.1 to 2.3.1+5.32
> cube would be upgraded from 4.3.5 to 2005.08.29
> python-ratelimiter would be upgraded from 1.2.0 to 1.2.0.post0
> gr-osmosdr would be upgraded from 0.2.3-0.a100eb0 to 0.2.3.20210128
> countdown would be upgraded from 1.0.0 to 20150606
> http-parser would be upgraded from 2.9.4-1.ec8b5ee to 2.9.4.20201223
> xlsx2csv would be upgraded from 0.7.4 to 20200427211949
> keynav would be upgraded from 0.20110708.0 to 20150730+4ae486d
> --8<---------------cut here---------------end--------------->8---
>
> It seems like most of these could be solved by checking if the version
> scheme changed from semver to calver. I think that’s a pretty good
> result considering how many packages we have.

I think this would not cut it.

As I wrote, almost any package using `git-version' is going to create
a version mismatch. This is because we consider

(git-version "X.Y" revision commit)

to be greater than "X.Y" whereas Repology either ignore the version, or
consider it to be a pre-release before "X.Y". See, e.g., "emacs:circe"
project, or "joycond". This, I think, the most prominent category of
comparison failures.

Also, there are versions which are plain wrong, e.g., "emacs:csv-mode",
and disqualify correct and up-to-date version. There are also version
disagreement in, e.g., "colobot", or upstream disagreement, e.g.,
"emacs:scala-mode".

See also "emacs:geiser-racket", "python:folium" or "higan" for other
projects with versioning issues.

Regards,
--
Nicolas Goaziou
L
L
Ludovic Courtès wrote on 10 Feb 2022 21:49
(name . Xinglu Chen)(address . public@yoctocell.xyz)(address . 53818@debbugs.gnu.org)
87leyi4fcv.fsf@gnu.org
Hi,

Xinglu Chen <public@yoctocell.xyz> skribis:

Toggle quote (2 lines)
> Ludovic schrieb am Dienstag der 08. Februar 2022 um 23:59 +01:

[...]

Toggle quote (7 lines)
>> Repology implements the same functionality as our updaters, so
>> repology.org is effectively “service as a software substitute”
>> (SaaSS).
>
> Right, but it tracks a lot more repositories than what our updaters do,
> so why not take advantage of that.

True, but this is kinda self-reinforcing: it’ll sure keep tracking more
if we stop maintaining our own code (IIRC, Repology was started after
‘guix refresh’ and I believe it’s maintained by one person.)

Toggle quote (10 lines)
>> My preference would be to keep our existing updaters rather than
>> effectively ditch them and delegate the work to Repology. It’s tempting
>> to think we can have both, but I’m not sure this would last long.
>
> The point of the Repology updater is to act as a fallback if none of
> the other updaters can update a package, e.g., ‘maven-dependency-tree’.
> I already mentioned that language-specific updaters usually provide more
> accurate and detailed information, so they should be used when possible;
> we aren’t losing anything here.

Hmm yes, could be. OTOH, like Nicolas writes, we would probably need
some filtering or post-processing to reduce false-positives, right?

Do you have examples where our updaters perform poorly and where
Repology does a better job? I wonder if there are lessons to be drawn
and bugs to be fixed.

Thanks,
Ludo’.
N
N
Nicolas Goaziou wrote on 14 Feb 2022 11:40
(name . Ludovic Courtès)(address . ludo@gnu.org)
87pmnprasd.fsf@nicolasgoaziou.fr
Hello,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (4 lines)
> Do you have examples where our updaters perform poorly and where
> Repology does a better job? I wonder if there are lessons to be drawn
> and bugs to be fixed.

As a data point, I'm sorry to say that our updaters are useless to me.

I watch over more than one thousand packages. I would have a hard time
expressing what are those packages to the updater, besides writing and
keeping up-to-date a huge manifest file. Assuming I could manage this,
fetching all version information would take considerable time, and,
since many packages are from GitHub, the party would stop early anyway
with GitHub refusing to proceed and requesting some token I don't have.

OTOH, using Repology API, I get the information I want in about ten
seconds. Sure, I need to eyeball through the results, filtering false
positives (around 4% in my case), but it still is a practical solution.

IMO, to be useful, updaters may need to rely on an external service,
which may, or may not, belong to the Guix ecosystem. They also need
a good UI.

I don't want to sound too negative, though. And current updaters are
certainly good enough when watching over a couple of packages, which
might be the most common use-case.

Cheers,
--
Nicolas Goaziou
M
M
Maxime Devos wrote on 14 Feb 2022 17:07
0c9466812aeafc33df51270feed16f8f16c531e7.camel@telenet.be
Nicolas Goaziou schreef op ma 14-02-2022 om 11:40 [+0100]:
Toggle quote (13 lines)
> [...]. Assuming I could manage this,
> fetching all version information would take considerable time, and,
> since many packages are from GitHub, the party would stop early anyway
> with GitHub refusing to proceed and requesting some token I don't have.
>
> OTOH, using Repology API, I get the information I want in about ten
> seconds. Sure, I need to eyeball through the results, filtering false
> positives (around 4% in my case), but it still is a practical solution.
>
> IMO, to be useful, updaters may need to rely on an external service,
> which may, or may not, belong to the Guix ecosystem. They also need
> a good UI.

To avoid exceeding API limits and reduce network traffic, I suggest the
following change:

Cache HTTP responses, using http-fetch/cached instead of
http-fetch. When something is in the cache and not expired,
this avoids some network traffic and does not bring us closer
to the API limits.

When it is expired (and in the cache), then at least
http-fetch/cached makes a conditional request with
If-Modified-Since, which GitHub does not count against the rate
limit, assuming a ‘304 Not Modified’ response!

That does not address all your concerns but it should help I think.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYgp+KxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rs8AQCPb2Nc7ojg6vJ8KIfvTrVdwhjO
A2RbswK28kuzJzYJVgEA0XaZ4XfcThz6CWWQf9GERfNDQsfmDBW/l1rCGt0pZwo=
=XkiL
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 14 Feb 2022 17:58
(name . Nicolas Goaziou)(address . mail@nicolasgoaziou.fr)
87fsolv109.fsf@gnu.org
Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

Toggle quote (19 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Do you have examples where our updaters perform poorly and where
>> Repology does a better job? I wonder if there are lessons to be drawn
>> and bugs to be fixed.
>
> As a data point, I'm sorry to say that our updaters are useless to me.
>
> I watch over more than one thousand packages. I would have a hard time
> expressing what are those packages to the updater, besides writing and
> keeping up-to-date a huge manifest file. Assuming I could manage this,
> fetching all version information would take considerable time, and,
> since many packages are from GitHub, the party would stop early anyway
> with GitHub refusing to proceed and requesting some token I don't have.
>
> OTOH, using Repology API, I get the information I want in about ten
> seconds. Sure, I need to eyeball through the results, filtering false
> positives (around 4% in my case), but it still is a practical solution.

(I’m confused because my understanding of what you first wrote was that
Repology had too many false positives to be useful.)

You wrote about your feelings and that’s insightful, but can we focus on
specific examples where updaters are not helpful so we can better
understand and improve the situation?

Toggle quote (3 lines)
> IMO, to be useful, updaters may need to rely on an external service,
> which may, or may not, belong to the Guix ecosystem.

All the updaters rely on an external service. Relying on a centralized
SaaSS is different, though.

Toggle quote (2 lines)
> They also need a good UI.

Do you have examples of what’s wrong on the UI side?

To me, the main shortcoming is that ‘guix refresh’ doesn’t tell you that
if you update X, you may also need to update Y and Z. That info is not
always available, but it is available in repos such as PyPI and ELPA.

Thanks,
Ludo’.
N
N
Nicolas Goaziou wrote on 14 Feb 2022 19:42
(name . Ludovic Courtès)(address . ludo@gnu.org)
87leydqohm.fsf@nicolasgoaziou.fr
Hello,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (3 lines)
> (I’m confused because my understanding of what you first wrote was that
> Repology had too many false positives to be useful.)

Repology is okay for my use-case because I've gotten accustomed to its
quirks. I wouldn't recommend it as a fall-back solution for Guix in its
current form, tho, for the reason above. Does that make sense?

Toggle quote (4 lines)
> You wrote about your feelings and that’s insightful, but can we focus on
> specific examples where updaters are not helpful so we can better
> understand and improve the situation?

I wrote about the following facts:
- it is difficult to specify a large number of packages,
- when you have specified a large number of packages, the processing is
slow,
- checking GitHub fails for me.

I don't see any feelings in there.

Toggle quote (6 lines)
>> IMO, to be useful, updaters may need to rely on an external service,
>> which may, or may not, belong to the Guix ecosystem.
>
> All the updaters rely on an external service. Relying on a centralized
> SaaSS is different, though.

Fair enough. I meant an external centralized service.

Toggle quote (2 lines)
> Do you have examples of what’s wrong on the UI side?

It has no Emacs interface. Nuff said. ;)

Again, I don't know how to specify efficiently many packages, e.g., all
Emacs packages, or all games. Also, reading through a massive output in
the terminal is not very user friendly, IMO.

Toggle quote (4 lines)
> To me, the main shortcoming is that ‘guix refresh’ doesn’t tell you that
> if you update X, you may also need to update Y and Z. That info is not
> always available, but it is available in repos such as PyPI and ELPA.

I don't think solving this is realistic. Dependencies are sometimes very
loose.

Regards,
--
Nicolas Goaziou
L
L
Ludovic Courtès wrote on 15 Feb 2022 10:57
Improving updaters and ‘guix refresh’
(name . Nicolas Goaziou)(address . mail@nicolasgoaziou.fr)
87ee44pi4p.fsf_-_@gnu.org
Hi!

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

Toggle quote (9 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> (I’m confused because my understanding of what you first wrote was that
>> Repology had too many false positives to be useful.)
>
> Repology is okay for my use-case because I've gotten accustomed to its
> quirks. I wouldn't recommend it as a fall-back solution for Guix in its
> current form, tho, for the reason above. Does that make sense?

It sure does, thanks for explaining.

Toggle quote (6 lines)
> I wrote about the following facts:
> - it is difficult to specify a large number of packages,
> - when you have specified a large number of packages, the processing is
> slow,
> - checking GitHub fails for me.

Alright, I had missed that.

Regarding “specifying many packages”, do examples like these work for
you:

• guix refresh -t elpa

• guix refresh $(guix package -A ^emacs- | cut -f1)

• guix refresh -r emacs-emms

• guix refresh -s non-core -t generic-git

• guix refresh -m packages-i-care-about.scm

If not, what kind of selection mechanism could help? ‘-s’ currently
accepts only two values, but we could augment it.

Regarding slow processing, it very much depends on the updater. For
example, on a warm cache, ‘guix refresh -t gnu’ is relatively fast
thanks to caching:

Toggle snippet (12 lines)
$ time guix refresh -t gnu
gnu/packages/wget.scm:48:13: wget would be upgraded from 1.21.1 to 1.21.2
gnu/packages/tls.scm:86:13: libtasn1 would be upgraded from 4.17.0 to 4.18.0

[...]


real 0m38.314s
user 0m38.981s
sys 0m0.164s

It could be that some updaters do many HTTP round trips without any
caching, which slows things down.

[...]

Toggle quote (4 lines)
>> Do you have examples of what’s wrong on the UI side?
>
> It has no Emacs interface. Nuff said. ;)

True! :-)

I realize this is going off-topic, but let’s see if we can improve the
existing infrastructure to make it more convenient.

Thanks,
Ludo’.
N
N
Nicolas Goaziou wrote on 16 Feb 2022 13:43
Re: [bug#53818] Improving updaters and ‘guix re fresh’
(name . Ludovic Courtès)(address . ludo@gnu.org)
87a6eroubv.fsf@nicolasgoaziou.fr
Hello,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (5 lines)
> Regarding “specifying many packages”, do examples like these work for
> you:
>
> • guix refresh -t elpa

I don't find it very useful in practice. As a user, the packages I'm
interested in probably rely on more than one updater. I'm not even
supposed to know what updater relates to a given package.

I actually only use this when I know a GNU ELPA package is outdated
already, and I want it to compute the hash for me:

./pre-inst-env guix refresh -t elpa -u emacs-foo

Toggle quote (2 lines)
> • guix refresh $(guix package -A ^emacs- | cut -f1)

This one is interesting. This illustrates that the UI is, from my point
of view, a bit lacking. It would be a nice improvement to add a regexp
mechanism built-in, like in "guix search".

In any case, this fails after reporting status of around 50 packages,
with this time:

real 0m41,881s
user 0m12,155s
sys 0m0,726s

Assuming I don't get the "rate limit exceeded" error, at this rate, it
would take more than 15 minutes to check all the packages in
"emacs-xyz.scm". This is a bit long.

I don't see how this could reasonably be made faster without relying on
an external centralized service doing the checks regularly (e.g., once
a day) before the user actually requests them.

Toggle quote (2 lines)
> • guix refresh -r emacs-emms

It also fails with the "rate limit exceeded". While this sounds
theoretically nice, I wouldn't know how to make use of it yet.

Toggle quote (2 lines)
> • guix refresh -s non-core -t generic-git

See above about "-t elpa".

Toggle quote (2 lines)
> • guix refresh -m packages-i-care-about.scm

Yes, obviously, this is a nice, too. However, it doesn't scale if you
need to specify 1000+ packages.

Toggle quote (3 lines)
> If not, what kind of selection mechanism could help? ‘-s’ currently
> accepts only two values, but we could augment it.

Besides regexp matching, it may be useful to filter packages per module,
or source file name. Package categories is a bit awkward, tho, and
probably not satisfying.

Toggle quote (3 lines)
> I realize this is going off-topic, but let’s see if we can improve the
> existing infrastructure to make it more convenient.

Is it really off-topic?

Anyway, all of this is only one data point, and, as a reminder,
I certainly don't want to disparage either Xinglu Chen's work, or
current "guix refresh" functionality.

HTH,
--
Nicolas Goaziou
L
L
Ludovic Courtès wrote on 17 Feb 2022 11:35
(name . Nicolas Goaziou)(address . mail@nicolasgoaziou.fr)
87mtipok57.fsf@gnu.org
Hi,

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

Toggle quote (11 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Regarding “specifying many packages”, do examples like these work for
>> you:
>>
>> • guix refresh -t elpa
>
> I don't find it very useful in practice. As a user, the packages I'm
> interested in probably rely on more than one updater. I'm not even
> supposed to know what updater relates to a given package.

Right, that’s more for packagers than for users.

As a user, what works better is:

guix refresh -r $(guix package -I |cut -f1) -s non-core

… or simply ‘--with-latest’, if I’m not interested in updating package
definitions.

Toggle quote (6 lines)
>> • guix refresh $(guix package -A ^emacs- | cut -f1)
>
> This one is interesting. This illustrates that the UI is, from my point
> of view, a bit lacking. It would be a nice improvement to add a regexp
> mechanism built-in, like in "guix search".

Makes sense, we can do that.

Toggle quote (7 lines)
> In any case, this fails after reporting status of around 50 packages,
> with this time:
>
> real 0m41,881s
> user 0m12,155s
> sys 0m0,726s

How does it fail? If it’s the GitHub rate limit, then there’s only one
answer: you have to provide a token.

Toggle quote (8 lines)
> Assuming I don't get the "rate limit exceeded" error, at this rate, it
> would take more than 15 minutes to check all the packages in
> "emacs-xyz.scm". This is a bit long.

> I don't see how this could reasonably be made faster without relying on
> an external centralized service doing the checks regularly (e.g., once
> a day) before the user actually requests them.

Maybe you’re right, but before jumping to the conclusion, we have to
investigate a bit. Like I wrote, the ‘gnu’ updater for instance fetches
a single file that remains in cache afterwards—the cost is constant.

We should identify updaters that have linear cost and check what can be
done. ‘github’, ‘generic-html’, and ‘generic-git’ are of that kind.

Now, the command I gave above looks at 1,134 packages, so is it even
something you want to do as a packager?

Toggle quote (14 lines)
>> • guix refresh -r emacs-emms
>
> It also fails with the "rate limit exceeded". While this sounds
> theoretically nice, I wouldn't know how to make use of it yet.
>
>> • guix refresh -s non-core -t generic-git
>
> See above about "-t elpa".
>
>> • guix refresh -m packages-i-care-about.scm
>
> Yes, obviously, this is a nice, too. However, it doesn't scale if you
> need to specify 1000+ packages.

You can use ‘fold-packages’ and have three lines that return a manifest
of 10K packages if you want it.

Honestly, since I mostly rely on others these days :-), I’m no longer
sure what the packager’s workflow is. Also, the level of coupling
varies greatly between, say, a C/C++ package and a set of
Python/Emacs/Rust packages.

I find that ‘guix refresh’ works fine for loosely-coupled C/C++ packages
where often you’d want to upgrade packages individually.

But for Python and Emacs packages, what do we want? Do packagers always
want to check 1K+ packages at once? Or are there other patterns?

Toggle quote (7 lines)
>> If not, what kind of selection mechanism could help? ‘-s’ currently
>> accepts only two values, but we could augment it.
>
> Besides regexp matching, it may be useful to filter packages per module,
> or source file name. Package categories is a bit awkward, tho, and
> probably not satisfying.

We can add options to make it more convenient, but it’s already
possible:

guix refresh $(guix package -A | grep emacs-xyz.scm | cut -f1)

Toggle quote (9 lines)
>> I realize this is going off-topic, but let’s see if we can improve the
>> existing infrastructure to make it more convenient.
>
> Is it really off-topic?
>
> Anyway, all of this is only one data point, and, as a reminder,
> I certainly don't want to disparage either Xinglu Chen's work, or
> current "guix refresh" functionality.

Yup, same here!

I think we have nice infrastructure but you raise important
shortcomings. What Xinglu Chen did might in fact be one way to address
it, and there may also be purely UI issues that we could address.

Thanks,
Ludo’.
Z
Z
zimoun wrote on 17 Feb 2022 12:17
86sfsh4u9x.fsf@gmail.com
Hi,

On Thu, 17 Feb 2022 at 11:35, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (8 lines)
>>> • guix refresh $(guix package -A ^emacs- | cut -f1)
>>
>> This one is interesting. This illustrates that the UI is, from my point
>> of view, a bit lacking. It would be a nice improvement to add a regexp
>> mechanism built-in, like in "guix search".
>
> Makes sense, we can do that.

I agree the UI is not nice. Well, at the command line, I never read the
complete output of “guix package -A” and I always pipe it with “cut
-f1”. Well, I think this complete display is only useful for
third-party; the only one I have in mind is emacs-guix. Therefore, are
we maintaining this CLI for backward compatibility when we could change
both?

Something more useful as output would be:

name version synopsis

Whatever. :-)

Even the internal etc/completion/bash/guix has to pipe:

Toggle snippet (15 lines)
_guix_complete_available_package ()
{
local prefix="$1"
if [ -z "$_guix_available_packages" ]
then
# Cache the complete list because it rarely changes and makes
# completion much faster.
_guix_available_packages="$(${COMP_WORDS[0]} package -A 2> /dev/null \
| cut -f1)"
fi
COMPREPLY+=($(compgen -W "$_guix_available_packages" -- "$prefix"))
}


Last, I am not convinced that “guix search” would be help here.
Because:

1. the output requires to pipe with recsel,
2. it is much slower than “package -A” [1].




Toggle quote (5 lines)
>>> • guix refresh -m packages-i-care-about.scm
>>
>> Yes, obviously, this is a nice, too. However, it doesn't scale if you
>> need to specify 1000+ packages.

[...]

Toggle quote (10 lines)
>> In any case, this fails after reporting status of around 50 packages,
>> with this time:
>>
>> real 0m41,881s
>> user 0m12,155s
>> sys 0m0,726s
>
> How does it fail? If it’s the GitHub rate limit, then there’s only one
> answer: you have to provide a token.

Let mimick a collection if 1000+ packages I care about. Consider this
manifest for packages using r-build-system only…

Toggle snippet (12 lines)
(use-modules (guix packages)
(gnu packages)
(guix build-system r))

(packages->manifest
(fold-packages (lambda (package result)
(if (eq? (package-build-system package) r-build-system)
(cons package result)
result))
'()))

…it hits the issue of Github token…

Toggle snippet (19 lines)
gnu/packages/bioconductor.scm:6034:13: 1.66.0 is already the latest version of r-plgem
gnu/packages/bioconductor.scm:6011:13: 1.22.0 is already the latest version of r-rots
gnu/packages/bioconductor.scm:12614:2: warning: 'bioconductor' updater failed to determine available releases for r-fourcseq
Backtrace:
13 (primitive-load "/home/simon/.config/guix/current/bin/guix")

[...]

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Error downloading release information through the GitHub
API. This may be fixed by using an access token and setting the environment
variable GUIX_GITHUB_TOKEN, for instance one procured from
https://github.com/settings/tokens

real 10m27.306s
user 4m14.077s
sys 0m12.467s

…when most R packages come from CRAN or Bioconductor archives.


Basically, ~5000 packages come from Github which represents ~25% of
overall. Therefore, one needs to be really lucky when updating many
package and not hit the Github rate limit.


Yes, large collection of packages cannot be updated easily. Somehow, it
is an issue from upstream and it is hard to fix… except by
duplicating upstream or provide a token. :-)


Well, using the external centralized Repology service is a first step to
update at scale, no? A second step could be to have this feature
included in the Data Service; but before we have other fishes to fry,
IMHO. :-)


Toggle quote (12 lines)
>> Assuming I don't get the "rate limit exceeded" error, at this rate, it
>> would take more than 15 minutes to check all the packages in
>> "emacs-xyz.scm". This is a bit long.
>>
>> I don't see how this could reasonably be made faster without relying on
>> an external centralized service doing the checks regularly (e.g., once
>> a day) before the user actually requests them.
>
> Maybe you’re right, but before jumping to the conclusion, we have to
> investigate a bit. Like I wrote, the ‘gnu’ updater for instance fetches
> a single file that remains in cache afterwards—the cost is constant.

Repology acts as this “external centralized service”, no? On one hand,
it is a practical solution; especially by being fast enough. On the
other hand, it serves few false positives (say 4% to fix the ideas).

Nicolas, considering the complexity of packages and their origins, do
you think it would be possible to do better (fast and accurate) than
Repology at scale?


Toggle quote (8 lines)
>>> • guix refresh -m packages-i-care-about.scm
>>
>> Yes, obviously, this is a nice, too. However, it doesn't scale if you
>> need to specify 1000+ packages.
>
> You can use ‘fold-packages’ and have three lines that return a manifest
> of 10K packages if you want it.

Yes, see example above.


Toggle quote (10 lines)
>>> If not, what kind of selection mechanism could help? ‘-s’ currently
>>> accepts only two values, but we could augment it.
>>
>> Besides regexp matching, it may be useful to filter packages per module,
>> or source file name. Package categories is a bit awkward, tho, and
>> probably not satisfying.
>
> We can add options to make it more convenient, but it’s already
> possible:

Since these features are advanced, why not keep the CLI simple and
instead on rely manifest files for complex filtering?


Toggle quote (3 lines)
>>> I realize this is going off-topic, but let’s see if we can improve the
>>> existing infrastructure to make it more convenient.

[...]

Toggle quote (4 lines)
> I think we have nice infrastructure but you raise important
> shortcomings. What Xinglu Chen did might in fact be one way to address
> it, and there may also be purely UI issues that we could address.

All the points raised here are important but appears to me orthogonal
with the patch series. :-)


Cheers,
simon
N
N
Nicolas Goaziou wrote on 18 Feb 2022 11:28
(name . zimoun)(address . zimon.toutoune@gmail.com)
87tucwmpsl.fsf@nicolasgoaziou.fr
Hello,

zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (6 lines)
> On Thu, 17 Feb 2022 at 11:35, Ludovic Courtès <ludo@gnu.org> wrote:


>> How does it fail? If it’s the GitHub rate limit, then there’s only one
>> answer: you have to provide a token.

IIUC, I have to register on GitHub to create this token. This is a bit
sad as a prerequisite to use one core feature of Guix.

Toggle quote (16 lines)
> Let mimick a collection if 1000+ packages I care about. Consider this
> manifest for packages using r-build-system only…
>
> --8<---------------cut here---------------start------------->8---
> (use-modules (guix packages)
> (gnu packages)
> (guix build-system r))
>
> (packages->manifest
> (fold-packages (lambda (package result)
> (if (eq? (package-build-system package) r-build-system)
> (cons package result)
> result))
> '()))
> --8<---------------cut here---------------end--------------->8---

I have to learn about fold-packages.

Toggle quote (4 lines)
> Nicolas, considering the complexity of packages and their origins, do
> you think it would be possible to do better (fast and accurate) than
> Repology at scale?

It's not about doing better, but doing differently. We do not need all
of Repology's features.

As far as the updater part is concerned, Repology pokes at various
package repositories, which are usually not upstream, extracts package
versions, applies some heuristics to normalize and compare them, then
decides what is the newest version and which repositories provide
outdated packages. This has a two obvious shortcomings:

1. the version number usually doesn't come from an official source, so
it may be wrong—e.g., our emacs-csv-mode is "outdated" because Funtoo
1.4 chose a non-existing higher version number for the same package.

2. version comparison does not understand every local versioning
scheme—e.g., our emacs-fold-dwim packages is currently at
"1.2-0.c46f4bb", which is, in Guix parlance, after "1.2", yet
Repology thinks this is actually older than "1.2".

Therefore, I think a (theoretical) centralized Guix-centric version
checker could be fast: it would only poke at what our packages consider
to be upstream, and accurate, since it would know about our versioning
rules. Basically it could boil down to calling current "guix refresh" on
every package daily, and serializing the results.

Toggle quote (10 lines)
>>>> • guix refresh -m packages-i-care-about.scm
>>>
>>> Yes, obviously, this is a nice, too. However, it doesn't scale if you
>>> need to specify 1000+ packages.
>>
>> You can use ‘fold-packages’ and have three lines that return a manifest
>> of 10K packages if you want it.
>
> Yes, see example above.

Point taken.

Regards,
--
Nicolas Goaziou
L
L
Ludovic Courtès wrote on 3 Mar 2022 22:28
(name . Nicolas Goaziou)(address . mail@nicolasgoaziou.fr)
87r17ivi7l.fsf@gnu.org
Hi!

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

Toggle quote (11 lines)
> zimoun <zimon.toutoune@gmail.com> writes:
>
>> On Thu, 17 Feb 2022 at 11:35, Ludovic Courtès <ludo@gnu.org> wrote:
>
>
>>> How does it fail? If it’s the GitHub rate limit, then there’s only one
>>> answer: you have to provide a token.
>
> IIUC, I have to register on GitHub to create this token. This is a bit
> sad as a prerequisite to use one core feature of Guix.

I have some good news!


Granted, it’s not a revolution, but it should fix one of the main
annoyances of ‘guix refresh’.

Regarding the “sad prerequisite”, the ‘github’ updater predates the
‘generic-git’ updater by several years, during which it was the only way
to get data for matching packages.

Actually I wonder if it’s still useful to keep. In theory it can
provide more accurate data than the ‘generic-git’ updater; not sure if
this is the case in practice.

Ludo’.
?