* guix/import/utils.scm (find-version): New procedure.
* guix/scripts/refresh.scm (<update-spec>) [partial?]: New field.
(update-spec-partial?): New accessor.
(update-spec): Add a PARTIAL? optional argument.
(update-specification->update-spec) <update-spec>: Call with its new PARTIAL?
optional argument when FALLBACK-VERSION is provided, i.e. when
'--target-version' was used.
(update-package): Remove the PACKAGE and VERSION positional arguments, and
replace them with UPDATE-SPEC. Update doc. Call `package-update' with its
new #:partial-version? argument.
(check-for-package-update) <package-latest-release>: Pass the new
#:partial-version? argument to it.
(guix-refresh) <update-package>: Adjust call accordingly.
(show-help): Udate doc.
* guix/upstream.scm (package-latest-release): Add #:partial-version? argument,
and apply it to the importer call.
(package-update): Add #:partial-version?> argument. Update doc. Pass it to
the `package-latest-release' call.
* guix/gnu-maintenance.scm (rewrite-url): Add #:partial-version? argument.
Update doc. Crawl URL for newer compatible versions when provided.
(import-html-release): Add #:partial-version? argument, and pass it to the
`rewrite-url' call. Use `find-version' to find the best version.
(import-release, import-ftp-release, import-gnu-release)
(import-release*): Add #:partial-version? argument and honor it.
(import-html-updatable-release): Add #:partial-version? argument, and pass it
to the `import-html-release' call.
* guix/import/gnome.scm (import-gnome-release)
<#:partial-version?>: Add new argument and honor it.
* guix/import/texlive.scm (latest-texlive-tag): Rename to...
(texlive-tags): ... this, and have it return all tags.
(texlive->guix-package): Adjust accordingly.
(latest-release): Add a #:partial-version? argument. Update doc.
* guix/import/stackage.scm (latest-lts-release): New #:partial-version?
argument.
* guix/import/pypi.scm (import-release): New #:partial-version? argument; pass
it to `pypi-package->upstream-source'.
* guix/import/opam.scm (latest-release): New #:partial-version? argument.
* guix/import/minetest.scm (latest-minetest-release): New #:partial-version?
argument.
(pypi-package->upstream-source): New #:partial-version? argument. Update doc.
* guix/import/launchpad.scm (latest-released-version): Rename to...
(release-versions): ... this, making it return all versions.
(import-release) <#:partial-version?>: New argument.
* guix/import/kde.scm (import-kde-release)
<#:partial-version?>: New argument. Update doc. Refactor to honor argument.
* guix/import/hexpm.scm (lookup-hexpm): Update doc.
(hexpm-latest-release): Rename to...
(hexpm-releases): ... this; return all release strings.
(hexpm->guix-package): Adjust accordingly.
(import-release): Add and honor a #:partial-version? argument. Update doc.
* guix/import/hackage.scm (import-release): New #:partial-version? argument.
* guix/import/cpan.scm (latest-release): New #:partial-version? argument.
* guix/import/crate.scm (max-crate-version-of-semver): Improve doc.
(import-release): Add a #:partial-version? argument and honor it.
* guix/import/egg.scm (find-latest-version): Rename to...
(get-versions): ... this, returning all versions.
(egg-metadata): Adjust accordingly.
(egg->guix-package): Likewise.
(import-release): Add a new #:partial-version? argument and honor it.
* guix/import/elpa.scm (latest-release): New #:partial-version? argument.
* guix/import/gem.scm (get-versions): New procedure.
(import-release): Add a new #:partial-version? argument and honor it.
* guix/import/git.scm (version-mapping): Update doc; streamline a bit.
(latest-tag): Rename to...
(get-tags): ... this, dropping the #:version keyword and returning the complete
tags alist. Update doc.
(latest-git-tag-version): Rename to...
(get-package-tags): ... this, returning the complete tags alist of the
package. Update doc.
(import-git-release): Add a new #:partial-version? argument and honor it.
Update doc.
* guix/import/github.scm (latest-released-version): Rename to...
(get-package-tags): ... this, returning all tags. Update doc.
(import-release): Add a new #:partial-version? argument and honor it.
* guix/import/cran.scm (latest-cran-release)
(latest-bioconductor-release): Add #:partial-version? argument.
* guix/import/composer.scm (latest-version): Delete procedure.
(composer-fetch): Add #:partial-version? keyword and honor it. Update doc.
(import-release): Likewise.
* guix/import/test.scm (import-release): Add #:partial-version? argument.
* tests/guix-refresh.sh: Add test.
* tests/gem.scm (test-foo-versions-json): New variable.
(package-latest-release): Mock new URL.
* tests/import-git.scm (latest-git-tag-version): New procedure.
* tests/gnu-maintenance.scm (libuv-dist-html)
(libuv-dist-1.46.0-html, libuv-dist-1.44.2-html)
(libuv-html-data): New variables.
(mock-http-fetch/cached): New procedure.
("rewrite-url, without to-version"): Rewrite using the above.
("rewrite-url, partial to-version"): New test.
* doc/guix.texi <"Invoking guix refresh">: Update doc.
Change-Id: I092a58b57ac42e54a2fa55e7761e8c6993af8ad4
---
doc/guix.texi | 12 +++
guix/gnu-maintenance.scm | 120 ++++++++++++++-----------
guix/import/composer.scm | 59 ++++++------
guix/import/cpan.scm | 2 +-
guix/import/cran.scm | 4 +-
guix/import/crate.scm | 51 ++++++-----
guix/import/egg.scm | 33 +++----
guix/import/elpa.scm | 2 +-
guix/import/gem.scm | 29 ++++--
guix/import/git.scm | 103 +++++++++------------
guix/import/github.scm | 114 +++++++++++------------
guix/import/gnome.scm | 50 ++++-------
guix/import/hackage.scm | 2 +-
guix/import/hexpm.scm | 42 +++++----
guix/import/kde.scm | 57 ++++++------
guix/import/launchpad.scm | 36 ++++----
guix/import/minetest.scm | 2 +-
guix/import/opam.scm | 2 +-
guix/import/pypi.scm | 25 ++++--
guix/import/stackage.scm | 2 +-
guix/import/test.scm | 15 ++--
guix/import/texlive.scm | 38 ++++----
guix/import/utils.scm | 20 ++++-
guix/scripts/refresh.scm | 185 ++++++++++++++++++++------------------
guix/upstream.scm | 26 +++---
tests/gem.scm | 22 +++++
tests/gnu-maintenance.scm | 65 +++++++++++---
tests/guix-refresh.sh | 10 ++-
tests/import-git.scm | 4 +
29 files changed, 629 insertions(+), 503 deletions(-)
Toggle diff (285 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 9a53bdcd37..cc03dda4ee 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14961,6 +14961,7 @@ Invoking guix refresh
@dots{}
@end example
+@cindex target version, guix refresh
In some specific cases, you may have many packages specified via a
manifest or a module selection which should all be updated together; for
these cases, the @option{--target-version} option can be provided to have
@@ -14981,6 +14982,17 @@ Invoking guix refresh
@dots{}
@end example
+@cindex partial target version, guix refresh
+The @option{--target-version} option accepts partial version prefixes,
+which can be useful to update to the latest major or major-minor
+prefixed version:
+
+@example
+$ guix refresh qtbase@@5 qtdeclarative@@5 --target-version=5
+gnu/packages/qt.scm:1472:13: qtdeclarative would be upgraded from 5.15.8 to 5.15.10
+gnu/packages/qt.scm:452:13: qtbase would be upgraded from 5.15.8 to 5.15.10
+@end example
+
Sometimes the upstream name differs from the package name used in Guix,
and @command{guix refresh} needs a little help. Most updaters honor the
@code{upstream-name} property in package definitions, which can be used
diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index ee4882326f..f26d8c5fbc 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -3,7 +3,7 @@
;;; Copyright © 2012, 2013 Nikita Karetnikov <nikita@karetnikov.org>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
-;;; Copyright © 2023 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2023, 2025 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -44,7 +44,7 @@ (define-module (guix gnu-maintenance)
#:use-module (guix records)
#:use-module (guix upstream)
#:use-module (guix packages)
- #:autoload (guix import utils) (false-if-networking-error)
+ #:autoload (guix import utils) (false-if-networking-error find-version)
#:autoload (zlib) (call-with-gzip-input-port)
#:autoload (htmlprag) (html->sxml) ;from Guile-Lib
#:export (gnu-package-name
@@ -346,12 +346,15 @@ (define* (releases project
(define* (import-ftp-release project
#:key
- (version #f)
+ version
+ partial-version?
(server "ftp.gnu.org")
(directory (string-append "/gnu/" project))
(file->signature (cut string-append <> ".sig")))
"Return an <upstream-source> for the latest release of PROJECT on SERVER
-under DIRECTORY, or #f. Optionally include a VERSION string to fetch a specific version.
+under DIRECTORY, or #f. Optionally include a VERSION string to fetch a
+specific version, which may be marked as partially specified via
+PARTIAL-VERSION?.
Use FTP-OPEN and FTP-CLOSE to open (resp. close) FTP connections; this can be
useful to reuse connections.
@@ -417,7 +420,9 @@ (define* (import-ftp-release project
(and (release-file? project file)
(file->source directory file)))
(_ #f))
- entries)))
+ entries))
+ (versions (map upstream-source-version releases))
+ (version (find-version versions version partial-version?)))
;; Assume that SUBDIRS correspond to versions, and jump into the
;; one with the highest version number.
@@ -440,14 +445,17 @@ (define* (import-ftp-release project
(define* (import-release package
#:key
- (version #f)
+ version
+ partial-version?
(server "ftp.gnu.org")
(directory (string-append "/gnu/" package)))
"Return the <upstream-source> for the latest version of PACKAGE or #f.
PACKAGE must be the canonical name of a GNU package. Optionally include a
-VERSION string to fetch a specific version."
+VERSION string to fetch a specific version, which may be marked as partially
+specified via PARTIAL-VERSION?."
(import-ftp-release package
#:version version
+ #:partial-version? partial-version?
#:server server
#:directory directory))
@@ -463,7 +471,7 @@ (define-syntax-rule (false-if-ftp-error exp)
(close-port port))
#f)))
-(define* (import-release* package #:key (version #f))
+(define* (import-release* package #:key version partial-version?)
"Like 'import-release', but (1) take a <package> object, and (2) ignore FTP
errors that might occur when PACKAGE is not actually a GNU package, or not
hosted on ftp.gnu.org, or not under that name (this is the case for
@@ -474,6 +482,7 @@ (define* (import-release* package #:key (version #f))
(false-if-ftp-error
(import-release (package-upstream-name package)
#:version version
+ #:partial-version? partial-version?
#:server server
#:directory directory)))))
@@ -561,16 +570,23 @@ (define (strip-trailing-slash s)
;;; TODO: Extend to support the RPM and GNOME version schemes?
(define %version-rx "[0-9.]+")
-(define* (rewrite-url url version #:key to-version)
+(define* (rewrite-url url version #:key to-version partial-version?)
"Rewrite URL so that the URL path components matching the current VERSION or
VERSION-MAJOR.VERSION-MINOR are updated with that of the latest version found
by crawling the corresponding URL directories. Alternatively, when TO-VERSION
-is specified, rewrite version matches directly to it without crawling URL.
+is specified, rewrite version matches directly to it without crawling URL. If
+TO-VERSION is provided and PARTIAL-VERSION? set to #t, then crawl URL to find
+the newest compatible release (one that is prefixed by TO-VERSION).
For example, the URL
\"https://dist.libuv.org/dist/v1.45.0/libuv-v1.45.0.tar.gz\" could be
rewritten to something like
-\"https://dist.libuv.org/dist/v1.46.0/libuv-v1.46.0.tar.gz\"."
+\"https://dist.libuv.org/dist/v1.46.0/libuv-v1.46.0.tar.gz\".
+
+With TO-VERSION set to \"1.49\" and PARTIAL-VERSION? set to #t, the URL
+\"https://dist.libuv.org/dist/v1.45.0/libuv-v1.45.0.tar.gz\" could be
+rewritten to something like
+\"https://dist.libuv.org/dist/v1.49.2/libuv-v1.49.2.tar.gz\"."
;; XXX: major-minor may be #f if version is not a triplet but a single
;; number such as "2".
(let* ((major-minor (false-if-exception (version-major+minor version)))
@@ -590,14 +606,15 @@ (define* (rewrite-url url version #:key to-version)
(reverse
(fold
(lambda (s parents)
- (if to-version
+ (if (and to-version (not partial-version?))
;; Direct rewrite case; the archive is assumed to exist.
(let ((u (string-replace-substring s version to-version)))
(cons (if (and major-minor to-major-minor)
(string-replace-substring u major-minor to-major-minor)
u)
parents))
- ;; More involved HTML crawl case.
+ ;; More involved HTML crawl case to get the latest version or a
+ ;; partial to-version.
(let* ((pattern (if major-minor
(format #f "(~a|~a)" version major-minor)
(format #f "(~a)" version)))
@@ -620,15 +637,14 @@ (define* (rewrite-url url version #:key to-version)
(m (string-match pattern l))
(v (match:substring m 1)))
(cons v l)))
- links)))
- ;; Retrieve the item having the largest version.
- (if (null? candidates)
- parents
- (cons (cdr (first (sort candidates
- (lambda (x y)
- (version>? (car x)
- (car y))))))
- parents)))
+ links))
+ (versions (map car candidates))
+ (version (find-version versions to-version
+ partial-version?)))
+ ;; Retrieve the item having the greatest version.
+ (if version
+ (cons (assoc-ref candidates version) parents)
+ parents)) ;XXX: bogus case; throw an error?
;; No version found in path component; continue.
(cons s parents)))))
(reverse url-prefix-components)
@@ -639,12 +655,14 @@ (define* (import-html-release base-url package
#:key
rewrite-url?
version
+ partial-version?
(directory (string-append
"/" (package-upstream-name package)))
file->signature)
"Return an <upstream-source> for the latest release of PACKAGE under
DIRECTORY at BASE-URL, or #f. Optionally include a VERSION string to fetch a
-specific version.
+specific version, which may be marked as partially specified via
+PARTIAL-VERSION?.
BASE-URL should be the URL of an HTML page, typically a directory listing as
found on 'https://kernel.org/pub'.
@@ -663,7 +681,8 @@ (define* (import-html-release base-url package
base-url
(string-append base-url directory "/")))
(url (if rewrite-url?
- (rewrite-url url current-version #:to-version version)
+ (rewrite-url url current-version #:to-version version
+ #:partial-version? partial-version?)
url))
(links (map (cut canonicalize-url <> url) (url->links url))))
@@ -695,23 +714,18 @@ (define* (import-html-release base-url package
(lambda (url) (list (uri-mirror-rewrite url))))))))))
(define candidates
- (filter-map url->release links))
-
- (match candidates
- (() #f)
- ((first . _)
- (if version
- ;; Find matching release version and return it.
- (find (lambda (upstream)
- (string=? (upstream-source-version upstream) version))
- (coalesce-sources candidates))
- ;; Select the most recent release and return it.
- (reduce (lambda (r1 r2)
- (if (version>? (upstream-source-version r1)
- (upstream-source-version r2))
- r1 r2))
- first
- (coalesce-sources candidates)))))))
+ (coalesce-sources (filter-map url->release links)))
+
+ (define versions
+ (map upstream-source-version candidates))
+
+ (define new-version
+ (find-version versions version partial-version?))
+
+ (and new-version
+ (find (compose (cut string=? new-version <>)
+ upstream-source-version)
+ candidates))))
;;;
@@ -743,7 +757,7 @@ (define ftp.gnu.org-files
(call-with-gzip-input-port port
(compose string->lines get-string-all))))))
-(define* (import-gnu-release package #:key (version #f))
+(define* (import-gnu-release package #:key version partial-version?)
"Return the latest release of PACKAGE, a GNU package available via
ftp.gnu.org. Optionally include a VERSION string to fetch a specific version.
@@ -776,12 +790,15 @@ (define* (import-gnu-release package #:key (version #f))
(string-contains file directory)
(release-file? name (basename file))))
files))
- ;; find latest version
- (version (or version
- (and (not (null? relevant))
- (tarball->version
- (find-latest-tarball-version relevant)))))
- ;; find tarballs matching this version
+ (versions (delay (sort (delete-duplicates
+ (map tarball->version relevant))
+ version>?)))
+ (version (or (and version partial-version?
+ (find (cut version-prefix? version <>)
+ (force versions)))
+ version
+ (first (force versions))))
+ ;; Find tarballs matching this version.
(tarballs (filter (lambda (file)
(string=? version (tarball->version file)))
relevant)))
@@ -998,11 +1015,11 @@ (define (html-updatable-package? package)
(or (assoc-ref (package-properties package) 'release-monitoring-url)
((url-predicate http-url?) package)))
-(define* (import-html-updatable-rele
