[PATCH WIP 0/4] Add upstream updater for git-fetch origins.

OpenSubmitted by Sarah Morgensen.
Details
4 participants
  • Sarah Morgensen
  • Ludovic Courtès
  • Maxime Devos
  • Xinglu Chen
Owner
unassigned
Severity
normal
S
S
Sarah Morgensen wrote on 16 Aug 01:16 +0200
(address . guix-patches@gnu.org)
cover.1629068119.git.iskarian@mgsn.dev
Hello Guix,
This is a proof-of-concept for extending `guix refresh -u` to support packageswith git-fetch origins. The potential impact is large: approximately 4.5kpackages use git-fetch, although only some fraction would be supported.
Currently, this enables update support for (at least) any package where
* github-updater finds an update,* origin-method is 'git-fetch', and* the package version is a suffix of the 'commit' field.
Drawbacks currently include the fact that libgit2 doesn't (yet) supportshallow checkouts [0], so the entire repository must be cloned. There is alsono support for verifying commits.
There should probably also be a few tests added.
WDYT?
[0] https://github.com/libgit2/libgit2/issues/3058
--Sarah Morgensen (4): guix hash: Extract file hashing procedures. import: Factorize file hashing. refresh: Support non-tarball sources. upstream: Support updating git-fetch origins.
guix/git-download.scm | 18 +++++++++++++- guix/hash.scm | 51 ++++++++++++++++++++++++++++++++++++++++ guix/import/cran.scm | 32 +++---------------------- guix/import/elpa.scm | 28 ++++------------------ guix/import/go.scm | 26 +++----------------- guix/scripts/hash.scm | 29 ++++++----------------- guix/scripts/refresh.scm | 10 ++++---- guix/upstream.scm | 41 +++++++++++++++++++++++++++++++- 8 files changed, 130 insertions(+), 105 deletions(-) create mode 100644 guix/hash.scm

base-commit: 12099eac1b161d364be923451d27d7d739d0f14d-- 2.31.1
S
S
Sarah Morgensen wrote on 16 Aug 01:25 +0200
[PATCH WIP 1/4] guix hash: Extract file hashing procedures.
(address . 50072@debbugs.gnu.org)
9ba0b798de2fdf859f94ca7b2f1ee052bceac63a.1629068119.git.iskarian@mgsn.dev
* guix/scripts/hash.scm (guix-hash)[vcs-file?, file-hash]: Extract logicto...* guix/hash.scm: ...here. New file.--- guix/hash.scm | 51 +++++++++++++++++++++++++++++++++++++++++++ guix/scripts/hash.scm | 29 ++++++------------------ 2 files changed, 58 insertions(+), 22 deletions(-) create mode 100644 guix/hash.scm
Toggle diff (120 lines)diff --git a/guix/hash.scm b/guix/hash.scmnew file mode 100644index 0000000000..8c2ab8187f--- /dev/null+++ b/guix/hash.scm@@ -0,0 +1,51 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++(define-module (guix hash)+ #:use-module (gcrypt hash)+ #:use-module (guix serialization)+ #:use-module (srfi srfi-1)+ #:use-module (srfi srfi-11)+ #:export (vcs-file?+ file-hash*))++(define (vcs-file? file stat)+ "Returns true if FILE is a version control system file."+ (case (stat:type stat)+ ((directory)+ (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))+ ((regular)+ ;; Git sub-modules have a '.git' file that is a regular text file.+ (string=? (basename file) ".git"))+ (else+ #f)))++(define* (file-hash* file #:key+ (algorithm (hash-algorithm sha256))+ (recursive? #t)+ (select? (negate vcs-file?)))+ "Compute the hash of FILE with ALGORITHM. If RECURSIVE? is true, recurse+into subdirectories of FILE, computing the combined hash of all files for+which (SELECT? FILE STAT) returns true."+ (if recursive?+ (let-values (((port get-hash)+ (open-hash-port algorithm)))+ (write-file file port #:select? select?)+ (force-output port)+ (get-hash))+ (file-hash algorithm file)))diff --git a/guix/scripts/hash.scm b/guix/scripts/hash.scmindex b8622373cc..353ca30c2c 100644--- a/guix/scripts/hash.scm+++ b/guix/scripts/hash.scm@@ -3,6 +3,7 @@ ;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org> ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org> ;;; Copyright © 2018 Tim Gesthuizen <tim.gesthuizen@yahoo.de>+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev> ;;; ;;; This file is part of GNU Guix. ;;;@@ -23,6 +24,7 @@ #:use-module (gcrypt hash) #:use-module (guix serialization) #:use-module (guix ui)+ #:use-module (guix hash) #:use-module (guix scripts) #:use-module (guix base16) #:use-module (guix base32)@@ -125,16 +127,6 @@ and 'base16' ('hex' and 'hexadecimal' can be used as well).\n")) (parse-command-line args %options (list %default-options) #:build-options? #f)) - (define (vcs-file? file stat)- (case (stat:type stat)- ((directory)- (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))- ((regular)- ;; Git sub-modules have a '.git' file that is a regular text file.- (string=? (basename file) ".git"))- (else- #f)))- (let* ((opts (parse-options)) (args (filter-map (match-lambda (('argument . value)@@ -150,18 +142,11 @@ and 'base16' ('hex' and 'hexadecimal' can be used as well).\n")) ;; Compute the hash of FILE. ;; Catch and gracefully report possible '&nar-error' conditions. (with-error-handling- (if (assoc-ref opts 'recursive?)- (let-values (((port get-hash)- (open-hash-port (assoc-ref opts 'hash-algorithm))))- (write-file file port #:select? select?)- (force-output port)- (get-hash))- (match file- ("-" (port-hash (assoc-ref opts 'hash-algorithm)- (current-input-port)))- (_ (call-with-input-file file- (cute port-hash (assoc-ref opts 'hash-algorithm)- <>)))))))+ (match file+ ("-" (port-hash (assoc-ref opts 'hash-algorithm)+ (current-input-port)))+ (_ (file-hash* #:algorithm (assoc-ref opts 'hash-algorithm)+ #:recursive? (assoc-ref opts 'recursive?)))))) (match args ((file)-- 2.31.1
S
S
Sarah Morgensen wrote on 16 Aug 01:25 +0200
[PATCH WIP 2/4] import: Factorize file hashing.
(address . 50072@debbugs.gnu.org)
82bab33f2d4a03c8e83d1825648577fbae3aee7e.1629068119.git.iskarian@mgsn.dev
* guix/import/cran.scm (vcs-file?, file-hash): Remove procedures.(description->package): Use 'file-hash*' instead.* guix/import/elpa.scm (vcs-file?, file-hash): Remove procedures.(git-repository->origin, elpa-package->sexp): Use 'file-hash* instead'.* guix/import/go.scm (vcs-file?, file-hash): Remove procedures.(git-checkout-hash): Use 'file-hash*' instead.--- guix/import/cran.scm | 32 +++----------------------------- guix/import/elpa.scm | 28 ++++------------------------ guix/import/go.scm | 26 +++----------------------- 3 files changed, 10 insertions(+), 76 deletions(-)
Toggle diff (202 lines)diff --git a/guix/import/cran.scm b/guix/import/cran.scmindex f649928c5a..ac24bc117e 100644--- a/guix/import/cran.scm+++ b/guix/import/cran.scm@@ -3,6 +3,7 @@ ;;; Copyright © 2015, 2016, 2017, 2019, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net>+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev> ;;; ;;; This file is part of GNU Guix. ;;;@@ -34,9 +35,8 @@ #:use-module (web uri) #:use-module (guix memoization) #:use-module (guix http-client)- #:use-module (gcrypt hash)+ #:use-module (guix hash) #:use-module (guix store)- #:use-module ((guix serialization) #:select (write-file)) #:use-module (guix base32) #:use-module ((guix download) #:select (download-to-store)) #:use-module (guix import utils)@@ -194,17 +194,6 @@ bioconductor package NAME, or #F if the package is unknown." (bioconductor-packages-list type)) (cut assoc-ref <> "Version"))) -;; XXX taken from (guix scripts hash)-(define (vcs-file? file stat)- (case (stat:type stat)- ((directory)- (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))- ((regular)- ;; Git sub-modules have a '.git' file that is a regular text file.- (string=? (basename file) ".git"))- (else- #f)))- ;; Little helper to download URLs only once. (define download (memoize@@ -437,16 +426,6 @@ reference the pkg-config tool." (define (needs-knitr? meta) (member "knitr" (listify meta "VignetteBuilder"))) -;; XXX adapted from (guix scripts hash)-(define (file-hash file select? recursive?)- ;; Compute the hash of FILE.- (if recursive?- (let-values (((port get-hash) (open-sha256-port)))- (write-file file port #:select? select?)- (force-output port)- (get-hash))- (call-with-input-file file port-sha256)))- (define (description->package repository meta) "Return the `package' s-expression for an R package published on REPOSITORY from the alist META, which was derived from the R package's DESCRIPTION file."@@ -544,12 +523,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file." (sha256 (base32 ,(bytevector->nix-base32-string- (case repository- ((git)- (file-hash source (negate vcs-file?) #t))- ((hg)- (file-hash source (negate vcs-file?) #t))- (else (file-sha256 source))))))))+ (file-hash* source)))))) ,@(if (not (and git? hg? (equal? (string-append "r-" name) (cran-guix-name name))))diff --git a/guix/import/elpa.scm b/guix/import/elpa.scmindex c0dc5acf51..22c937ca5f 100644--- a/guix/import/elpa.scm+++ b/guix/import/elpa.scm@@ -4,6 +4,7 @@ ;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> ;;; Copyright © 2020 Martin Becze <mjbecze@riseup.net> ;;; Copyright © 2020 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev> ;;; ;;; This file is part of GNU Guix. ;;;@@ -36,10 +37,10 @@ #:use-module (guix import utils) #:use-module (guix http-client) #:use-module (guix git)+ #:use-module (guix hash) #:use-module ((guix serialization) #:select (write-file)) #:use-module (guix store) #:use-module (guix ui)- #:use-module (gcrypt hash) #:use-module (guix base32) #:use-module (guix upstream) #:use-module (guix packages)@@ -226,27 +227,6 @@ keywords to values." (close-port port) (data->recipe (cons ':name data)))) -;; XXX adapted from (guix scripts hash)-(define (file-hash file select? recursive?)- ;; Compute the hash of FILE.- (if recursive?- (let-values (((port get-hash) (open-sha256-port)))- (write-file file port #:select? select?)- (force-output port)- (get-hash))- (call-with-input-file file port-sha256)))--;; XXX taken from (guix scripts hash)-(define (vcs-file? file stat)- (case (stat:type stat)- ((directory)- (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))- ((regular)- ;; Git sub-modules have a '.git' file that is a regular text file.- (string=? (basename file) ".git"))- (else- #f)))- (define (git-repository->origin recipe url) "Fetch origin details from the Git repository at URL for the provided MELPA RECIPE."@@ -268,7 +248,7 @@ RECIPE." (sha256 (base32 ,(bytevector->nix-base32-string- (file-hash directory (negate vcs-file?) #t)))))))+ (file-hash* directory))))))) (define* (melpa-recipe->origin recipe) "Fetch origin details from the MELPA recipe and associated repository for@@ -379,7 +359,7 @@ type '<elpa-package>'." (sha256 (base32 ,(if tarball- (bytevector->nix-base32-string (file-sha256 tarball))+ (bytevector->nix-base32-string (file-hash* tarball)) "failed to download package"))))))) (build-system emacs-build-system) ,@(maybe-inputs 'propagated-inputs dependencies)diff --git a/guix/import/go.scm b/guix/import/go.scmindex 617a0d0e23..c6425667f8 100644--- a/guix/import/go.scm+++ b/guix/import/go.scm@@ -25,6 +25,7 @@ (define-module (guix import go) #:use-module (guix build-system go) #:use-module (guix git)+ #:use-module (guix hash) #:use-module (guix i18n) #:use-module (guix diagnostics) #:use-module (guix import utils)@@ -35,9 +36,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix memoization) #:autoload (htmlprag) (html->sxml) ;from Guile-Lib- #:autoload (guix git) (update-cached-checkout)- #:autoload (gcrypt hash) (open-hash-port hash-algorithm sha256)- #:autoload (guix serialization) (write-file)+ #:autoload (gcrypt hash) (hash-algorithm sha256) #:autoload (guix base32) (bytevector->nix-base32-string) #:autoload (guix build utils) (mkdir-p) #:use-module (ice-9 match)@@ -494,25 +493,6 @@ source." goproxy (module-meta-repo-root meta-data))) -;; XXX: Copied from (guix scripts hash).-(define (vcs-file? file stat)- (case (stat:type stat)- ((directory)- (member (basename file) '(".bzr" ".git" ".hg" ".svn" "CVS")))- ((regular)- ;; Git sub-modules have a '.git' file that is a regular text file.- (string=? (basename file) ".git"))- (else- #f)))--;; XXX: Adapted from 'file-hash' in (guix scripts hash).-(define* (file-hash file #:optional (algorithm (hash-algorithm sha256)))- ;; Compute the hash of FILE.- (let-values (((port get-hash) (open-hash-port algorithm)))- (write-file file port #:select? (negate vcs-file?))- (force-output port)- (get-hash)))- (define* (git-checkout-hash url reference algorithm) "Return the ALGORITHM hash of the checkout of URL at REFERENCE, a commit or tag."@@ -531,7 +511,7 @@ tag." (update-cached-checkout url #:ref `(tag-or-commit . ,reference)))))- (file-hash checkout algorithm)))+ (file-hash* checkout #:algorithm algorithm))) (define (vcs->origin vcs-type vcs-repo-url version) "Generate the `origin' block of a package depending on what type of source-- 2.31.1
S
S
Sarah Morgensen wrote on 16 Aug 01:25 +0200
[PATCH WIP 3/4] refresh: Support non-tarball sources.
(address . 50072@debbugs.gnu.org)
54668cb99babc81db0edfcdf2a8fa870bde96863.1629068119.git.iskarian@mgsn.dev
* guix/scripts/refresh.scm (update-package): Use 'file-hash*' instead of'port-sha256'. Rename TARBALL to OUTPUT.--- guix/scripts/refresh.scm | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
Toggle diff (57 lines)diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scmindex fb6c52a567..abb0c24e96 100644--- a/guix/scripts/refresh.scm+++ b/guix/scripts/refresh.scm@@ -8,6 +8,7 @@ ;;; Copyright © 2018 Efraim Flashner <efraim@flashner.co.il> ;;; Copyright © 2019 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev> ;;; ;;; This file is part of GNU Guix. ;;;@@ -26,7 +27,6 @@ (define-module (guix scripts refresh) #:use-module (guix ui)- #:use-module (gcrypt hash) #:use-module (guix scripts) #:use-module ((guix scripts build) #:select (%standard-build-options)) #:use-module (guix store)@@ -38,6 +38,7 @@ #:use-module (guix scripts graph) #:use-module (guix monads) #:use-module (guix gnupg)+ #:use-module (guix hash) #:use-module (gnu packages) #:use-module ((gnu packages commencement) #:select (%final-inputs)) #:use-module (ice-9 match)@@ -314,14 +315,14 @@ KEY-DOWNLOAD specifies a download policy for missing OpenPGP keys; allowed values: 'interactive' (default), 'always', and 'never'. When WARN? is true, warn about packages that have no matching updater." (if (lookup-updater package updaters)- (let-values (((version tarball source)+ (let-values (((version output source) (package-update store package updaters #:key-download key-download)) ((loc) (or (package-field-location package 'version) (package-location package)))) (when version- (if (and=> tarball file-exists?)+ (if (and=> output file-exists?) (begin (info loc (G_ "~a: updating from version ~a to version ~a...~%")@@ -347,8 +348,7 @@ warn about packages that have no matching updater." (package-name package) (upstream-input-change-name change))) (upstream-source-input-changes source))- (let ((hash (call-with-input-file tarball- port-sha256)))+ (let ((hash (file-hash* output))) (update-package-source package source hash))) (warning (G_ "~a: version ~a could not be \ downloaded and authenticated; not updating~%")-- 2.31.1
S
S
Sarah Morgensen wrote on 16 Aug 01:25 +0200
[PATCH WIP 4/4] upstream: Support updating git-fetch origins.
(address . 50072@debbugs.gnu.org)
8d1ae518b23fac5b15812a30b11df1c360ab3fbf.1629068119.git.iskarian@mgsn.dev
* guix/git-download.scm (checkout-to-store): New procedure.* guix/upstream.scm (guess-version-transform)(package-update/git-fetch): New procedures.(%method-updates): Add GIT-FETCH mapping.--- guix/git-download.scm | 18 +++++++++++++++++- guix/upstream.scm | 41 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 57 insertions(+), 2 deletions(-)
Toggle diff (118 lines)diff --git a/guix/git-download.scm b/guix/git-download.scmindex 5e624b9ae9..a7bdc16718 100644--- a/guix/git-download.scm+++ b/guix/git-download.scm@@ -3,6 +3,7 @@ ;;; Copyright © 2017 Mathieu Lirzin <mthl@gnu.org> ;;; Copyright © 2017 Christopher Baines <mail@cbaines.net> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev> ;;; ;;; This file is part of GNU Guix. ;;;@@ -26,6 +27,7 @@ #:use-module (guix records) #:use-module (guix packages) #:use-module (guix modules)+ #:use-module (guix git) #:autoload (guix build-system gnu) (standard-packages) #:autoload (git bindings) (libgit2-init!) #:autoload (git repository) (repository-open@@ -53,7 +55,9 @@ git-fetch git-version git-file-name- git-predicate))+ git-predicate++ checkout-to-store)) ;;; Commentary: ;;;@@ -287,4 +291,16 @@ absolute file name and STAT is the result of 'lstat'." (#f #f))))) (const #f))) +(define* (checkout-to-store store ref #:key (log (current-error-port)))+ "Checkout REF to STORE. Write progress reports to LOG. RECURSIVE? has the+same effect as the same-named parameter of 'latest-repository-commit'."+ ;; XXX: (guix git) does not use shallow clones, so this will be slow+ ;; for long-running repositories.+ (match-record ref <git-reference>+ (url commit recursive?)+ (latest-repository-commit store url+ #:ref `(tag-or-commit . ,commit)+ #:recursive? recursive?+ #:log-port log)))+ ;;; git-download.scm ends herediff --git a/guix/upstream.scm b/guix/upstream.scmindex 632e9ebc4f..927260cd89 100644--- a/guix/upstream.scm+++ b/guix/upstream.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2015 Alex Kost <alezost@gmail.com> ;;; Copyright © 2019 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev> ;;; ;;; This file is part of GNU Guix. ;;;@@ -24,6 +25,7 @@ #:use-module (guix discovery) #:use-module ((guix download) #:select (download-to-store url-fetch))+ #:use-module (guix git-download) #:use-module (guix gnupg) #:use-module (guix packages) #:use-module (guix diagnostics)@@ -430,9 +432,46 @@ SOURCE, an <upstream-source>." #:key-download key-download))) (values version tarball source)))))) +(define (guess-version-transform commit from-version)+ "Return a one-argument proc that transforms FROM-VERSION to COMMIT, or #f+if no such transformation could be determined."+ ;; Just handle prefixes for now, since that's the most common.+ (if (string-suffix? from-version commit)+ (let* ((version-length (string-length from-version))+ (commit-prefix (string-drop-right commit version-length)))+ (lambda (version)+ (string-append commit-prefix version)))+ #f))++(define* (package-update/git-fetch store package source+ #:key key-download)+ "Return the version, checkout, and SOURCE, to update PACKAGE to+SOURCE, an <upstream-source>."++ (define (uri-update/git old-uri old-version url version)+ (let* ((old-commit (git-reference-commit old-uri))+ (transform (guess-version-transform old-commit old-version)))+ (and transform+ (git-reference+ (inherit old-uri)+ (url url)+ (commit (transform version))))))++ ;; Only use the first element of URLS.+ (match-record source <upstream-source>+ (version urls)+ (let* ((old-uri (origin-uri (package-source package)))+ (old-version (package-version package))+ (new-uri (uri-update/git old-uri old-version+ (first urls) version)))+ (if new-uri+ (values version (checkout-to-store store new-uri) source)+ (values #f #f #f)))))+ (define %method-updates ;; Mapping of origin methods to source update procedures.- `((,url-fetch . ,package-update/url-fetch)))+ `((,url-fetch . ,package-update/url-fetch)+ (,git-fetch . ,package-update/git-fetch))) (define* (package-update store package #:optional (updaters (force %updaters))-- 2.31.1
M
M
Maxime Devos wrote on 16 Aug 12:46 +0200
cbcef388b1df20c24b6615a006c0daaf50f74b1f.camel@telenet.be
Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:
Toggle quote (5 lines)> * guix/git-download.scm (checkout-to-store): New procedure.> * guix/upstream.scm (guess-version-transform)> (package-update/git-fetch): New procedures.> (%method-updates): Add GIT-FETCH mapping.
Does it support packages defined like (a)
(define-public gnash (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4") (revision "0")) (package (name "gnash") (version (git-version "0.8.11" revision commit)) (source (git-reference (url "https://example.org") (commit commit))) [...])))
and (b)
(define-public gnash (package (name "gnash") (version "0.8.11") (source (git-reference (url "https://example.org") (commit commit)) [...]))?
(Maybe (a) and (b) can be used as test cases.)
FWIW, I had a try at supporting git-fetch origins in "--with-latest" and"guix refresh -e" myself, and had to modify 'package-update' to replacecommit strings. IIRC, it supports (b), but not (a). The patch isattached, hopefully it will be useful.
Greetings,Maxime.
Toggle diff (200 lines)diff --git a/guix/import/minetest.scm b/guix/import/minetest.scmindex 4264341d6a..2904c3f94a 100644--- a/guix/import/minetest.scm+++ b/guix/import/minetest.scm@@ -297,7 +297,7 @@ results. The return value is a list of <package/keys> records." (define (make-minetest-sexp author/name version repository commit inputs home-page synopsis description media-license license)- "Return a S-expression for the minetest package with the given author/NAME,+ "Return a S-expression for the minetest package with the given AUTHOR/NAME, VERSION, REPOSITORY, COMMIT, INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, MEDIA-LICENSE and LICENSE." `(package@@ -452,3 +452,37 @@ list of AUTHOR/NAME strings." #:repo->guix-package minetest->guix-package* #:guix-name (compose contentdb->package-name author/name->name)))++#|+(define (minetest-package? pkg)+ (and (string-prefix? "minetest-" (package:package-name pkg))+ (assq-ref (package:package-properties pkg) 'upstream-name)))++(define (latest-minetest-release pkg)+ "Return an <upstream-source> for the latest release of the package PKG."+ (define upstream-name+ (assoc-ref (package:package-properties pkg) 'upstream-name))+ (define contentdb-package (contentdb-fetch upstream-name))+ (define release (latest-release upstream-name))+ (and contentdb-package release+ (and-let* ((old-origin (package:package-source pkg))+ (old-reference (package:origin-uri old-origin))+ (is-git? (download:git-reference? old-reference))+ (commit (release-commit release)))+ (upstream-source+ (package (package:package-name pkg))+ (version (release-title release))+ (urls (download:git-reference+ (url (package-repository contentdb-package))+ (commit commit)))))))++(define %minetest-updater+ (upstream-updater+ (name 'minetest)+ (description "Updater for Minetest packages on ContentDB")+ (pred minetest-package?)+ (latest latest-minetest-release)))+|#+;; #:use-module (guix upstream)+;; #:use-module ((guix git-download) #:prefix download:)+;; #:use-module ((guix packages) #:prefix package:)diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scmindex fb6c52a567..4f3bbbcb94 100644--- a/guix/scripts/refresh.scm+++ b/guix/scripts/refresh.scm@@ -28,8 +28,10 @@ #:use-module (guix ui) #:use-module (gcrypt hash) #:use-module (guix scripts)+ #:use-module (guix serialization) #:use-module ((guix scripts build) #:select (%standard-build-options)) #:use-module (guix store)+ #:use-module (guix build utils) #:use-module (guix utils) #:use-module (guix packages) #:use-module (guix profiles)@@ -307,6 +309,17 @@ update would trigger a complete rebuild." (G_ "no updater for ~a~%") (package-name package))) ++;; XXX adapted from (guix scripts hash)+(define (file-hash file select? recursive?)+ ;; Compute the hash of FILE.+ (if recursive?+ (let-values (((port get-hash) (open-sha256-port)))+ (write-file file port #:select? select?)+ (force-output port)+ (get-hash))+ (call-with-input-file file port-sha256)))+ (define* (update-package store package updaters #:key (key-download 'interactive) warn?) "Update the source file that defines PACKAGE with the new version.@@ -347,8 +360,8 @@ warn about packages that have no matching updater." (package-name package) (upstream-input-change-name change))) (upstream-source-input-changes source))- (let ((hash (call-with-input-file tarball- port-sha256)))+ (let ((hash (file-hash tarball (const #t)+ (directory-exists? tarball)))) (update-package-source package source hash))) (warning (G_ "~a: version ~a could not be \ downloaded and authenticated; not updating~%")diff --git a/guix/upstream.scm b/guix/upstream.scmindex 632e9ebc4f..61f67b57c1 100644--- a/guix/upstream.scm+++ b/guix/upstream.scm@@ -24,6 +24,11 @@ #:use-module (guix discovery) #:use-module ((guix download) #:select (download-to-store url-fetch))+ #:use-module ((guix git-download)+ #:select (git-fetch git-reference?+ git-reference-url+ git-reference-commit+ git-reference-recursive?)) #:use-module (guix gnupg) #:use-module (guix packages) #:use-module (guix diagnostics)@@ -33,6 +38,7 @@ #:use-module (guix store) #:use-module ((guix derivations) #:select (built-derivations derivation->output-path)) #:autoload (gcrypt hash) (port-sha256)+ #:autoload (guix git) (latest-repository-commit) #:use-module (guix monads) #:use-module (srfi srfi-1) #:use-module (srfi srfi-9)@@ -93,7 +99,8 @@ upstream-source? (package upstream-source-package) ;string (version upstream-source-version) ;string- (urls upstream-source-urls) ;list of strings+ ; list of strings or a <git-reference>+ (urls upstream-source-urls) (signature-urls upstream-source-signature-urls ;#f | list of strings (default #f)) (input-changes upstream-source-input-changes@@ -361,6 +368,11 @@ values: 'interactive' (default), 'always', and 'never'." system target) "Download SOURCE from its first URL and lower it as a fixed-output derivation that would fetch it."+ (define url+ (match (upstream-source-urls source)+ ((first . _) first)+ (_ (raise (formatted-message+ (G_ "git origins are unsupported by --with-latest")))))) (mlet* %store-monad ((url -> (first (upstream-source-urls source))) (signature -> (and=> (upstream-source-signature-urls source)@@ -430,9 +442,23 @@ SOURCE, an <upstream-source>." #:key-download key-download))) (values version tarball source)))))) +(define* (package-update/git-fetch store package source #:key key-download)+ "Return the version, source code directory, and SOURCE, to update PACKAGE to+SOURCE, an <upstream-source>."+ (match source+ (($ <upstream-source> _ version ref _)+ (values version+ (latest-repository-commit+ store+ (git-reference-url ref)+ #:ref `(commit . ,(git-reference-commit ref))+ #:recursive? (git-reference-recursive? ref))+ source))))+ (define %method-updates ;; Mapping of origin methods to source update procedures.- `((,url-fetch . ,package-update/url-fetch)))+ `((,url-fetch . ,package-update/url-fetch)+ (,git-fetch . ,package-update/git-fetch))) (define* (package-update store package #:optional (updaters (force %updaters))@@ -492,9 +518,22 @@ new version string if an update was made, and #f otherwise." (origin-hash (package-source package)))) (old-url (match (origin-uri (package-source package)) ((? string? url) url)+ ((? git-reference? ref)+ (git-reference-url ref)) (_ #f))) (new-url (match (upstream-source-urls source)- ((first _ ...) first)))+ ((first _ ...) first)+ ((? git-reference? ref)+ (git-reference-url ref))+ (_ #f)))+ (old-commit (match (origin-uri (package-source package))+ ((? git-reference? ref)+ (git-reference-commit ref))+ (_ #f)))+ (new-commit (match (upstream-source-urls source)+ ((? git-reference? ref)+ (git-reference-commit ref))+ (_ #f))) (file (and=> (location-file loc) (cut search-path %load-path <>)))) (if file@@ -508,6 +547,9 @@ new version string if an update was made, and #f otherwise." 'filename file)) (replacements `((,old-version . ,version) (,old-hash . ,hash)+ ,@(if (and old-commit new-commit)+ `((,old-commit . ,new-commit))+ '()) ,@(if (and old-url new-url) `((,(dirname old-url) . ,(dirname new-url)))
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYRpCAhccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7mtFAQC2K6AYws89maCmlssXfSbym3b29CG2Ima2OaxW4a7k3wD/Z1vXqq7oTCNKbFPVwtQSEH++PEd83p7UKB2LMsw+cgg==ilA3-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 16 Aug 15:02 +0200
87wnol8rw3.fsf@yoctocell.xyz
On Mon, Aug 16 2021, Maxime Devos wrote:
Toggle quote (17 lines)> Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:>> * guix/git-download.scm (checkout-to-store): New procedure.>> * guix/upstream.scm (guess-version-transform)>> (package-update/git-fetch): New procedures.>> (%method-updates): Add GIT-FETCH mapping.>> Does it support packages defined like (a)>> (define-public gnash> (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")> (revision "0"))> (package> (name "gnash")> (version (git-version "0.8.11" revision commit))> (source (git-reference> (url "https://example.org")
IIUC, it only supports GitHub URLs at the moment. I have a WIP patchfor adding support for any arbitrary Git repository[1].
[1]:https://git.yoctocell.xyz/guix/commit/?h=guix-upstream-git-fetch&id=0356c7603a4611d40875b4eb352e3378295f34bc
-----BEGIN PGP SIGNATURE-----
iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmEaYcwVHHB1YmxpY0B5b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5r9oQAI3ATDH9rGTsS0dn7stOIBBvrjqsNAfqeBKClrHTptYn+8G3AZ+XEiaqrbRGIZ8aHkXjg6mD4B7d03mUdqIu3JedYeRYY+/iA5kE9HhohRmUwdUqCBjYYMZOO8SsrRKAUpgTu1dwIK8sb+fkwgZWG1ZYuGLUya1zGkVfeBailLxvfpUFneKLmY0gvEryVfVoYasbiYmCLSAQxiE6aJUSLpv20FUmjdyPWwPcwhLz5n9SkPHIzoNdlBv7e/M3sG63V7wwceeNeEgIOXD/rhwHluj5GgMzqOdcUN+qkHZDB4Nh5RGqHGTahqh2EuG5Vj6pVDOagZR+30chqiMpm3Py4O2E74ulmMw7LeTROBXwDoapdCkY3YWn1LutFKW3asZR7A5OWB6wKMjNn3p9nSlZHWb2bAxpXFvOhu89oVtokQKzbCHyF99v/5/aWNXSmT0FM6tVL9j8tT0SD+7TYwEWReMaPmtAb6XsDk7dpmK6Y9hjPV3uA/FojaVjHWuJim5xQG75NcjtG2mf3A+hXgumFuznpPLYKuJklLd6HjH3BfVxrOQC3ISJa0SJxvUG/lt2LV866QBgJPjCnox9TIr4nKzTm/TqxaIKQOzF/40TWPMZOyH2/ujmNpq2qarY6BOmah6lI/nFM47I/wcbHOqhDBDTV1B0VseqEL6DUR1mKlHd=ViNI-----END PGP SIGNATURE-----
M
M
Maxime Devos wrote on 16 Aug 20:15 +0200
12f985cd431b8b8099f680a3b25ed2eb90e6b26d.camel@telenet.be
Xinglu Chen schreef op ma 16-08-2021 om 15:02 [+0200]:
Toggle quote (22 lines)> On Mon, Aug 16 2021, Maxime Devos wrote:> > > Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:> > > * guix/git-download.scm (checkout-to-store): New procedure.> > > * guix/upstream.scm (guess-version-transform)> > > (package-update/git-fetch): New procedures.> > > (%method-updates): Add GIT-FETCH mapping.> > > > Does it support packages defined like (a)> > > > (define-public gnash> > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")> > (revision "0"))> > (package> > (name "gnash")> > (version (git-version "0.8.11" revision commit))> > (source (git-reference> > (url "https://example.org")> > IIUC, it only supports GitHub URLs at the moment. I have a WIP patch> for adding support for any arbitrary Git repository[1].
This patch series doesn't mention GitHub anywhere (except in the patchseries description) so I don't think it only supports GitHub URLs.Admittedly, only one updater, "github", currently produces git-referenceURLs, but I sent a patch series [2] that adds an importer which producesgit-reference URLs and the corresponding updater (see ‘git-fetch.patch’)produces appropriate git-reference objects.
[2]: https://issues.guix.gnu.org/49828#51.
Greetings,Maxime.
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYRqrKxccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hj2AQDWxLAJXlGpJDkETEBgpnyP1iqCLTkAy45kWUZkOdqY1AD6A2CMB630ENX8F2HOy6f9iMU1h6G/+xqqCy7ltH39AwI==487y-----END PGP SIGNATURE-----

S
S
Sarah Morgensen wrote on 16 Aug 21:56 +0200
Re: bug#50072: [PATCH WIP 0/4] Add upstream updater for git-fetch origins.
(name . Maxime Devos)(address . maximedevos@telenet.be)
86fsv9jh8h.fsf_-_@mgsn.dev
Hi Maxime,
Thanks for taking a look at this. :)
Maxime Devos <maximedevos@telenet.be> writes:
Toggle quote (19 lines)> Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:>> * guix/git-download.scm (checkout-to-store): New procedure.>> * guix/upstream.scm (guess-version-transform)>> (package-update/git-fetch): New procedures.>> (%method-updates): Add GIT-FETCH mapping.>> Does it support packages defined like (a)>> (define-public gnash> (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")> (revision "0"))> (package> (name "gnash")> (version (git-version "0.8.11" revision commit))> (source (git-reference> (url "https://example.org")> (commit commit)))> [...])))
No, it doesn't. Since the commit definition isn't part of the actualpackage definition, the current code has no way of updating it. Itwould require a rewrite of the edit-in-place logic with probably a lotof special-casing.
There are currently ~1250 package which use this format, though, so itcould be worth it... Perhaps what we actually need is a better idiom toexpress this situation. Package properties ('git-commit)? A 'git-version*'?
Toggle snippet (6 lines)(define (git-version* version revision) (let* ((source (package-source this-package)) (commit (git-reference-commit (origin-uri source)))) (git-version version revision commit)))
I'm not sure if binding order would be an issue with that.
Toggle quote (12 lines)> and (b)>> (define-public gnash> (package> (name "gnash")> (version "0.8.11")> (source (git-reference> (url "https://example.org")> (commit commit))> [...]))> ?
Is this missing a definition for commit? If it's like above, the sameapplies. Or if you mean
Toggle snippet (5 lines) (source (git-reference (url "https://example.org") (commit "583ccbc1275c7701dc4843ec12142ff86bb305b"))
Then that wouldn't be too hard to support. There seem to be ~136packages with this idiom.
Toggle quote (51 lines)> (Maybe (a) and (b) can be used as test cases.)>> FWIW, I had a try at supporting git-fetch origins in "--with-latest" and> "guix refresh -e" myself, and had to modify 'package-update' to replace> commit strings. IIRC, it supports (b), but not (a). The patch is> attached, hopefully it will be useful.>> Greetings,> Maxime.>> diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm> index 4264341d6a..2904c3f94a 100644> --- a/guix/import/minetest.scm> +++ b/guix/import/minetest.scm> @@ -297,7 +297,7 @@ results. The return value is a list of <package/keys> records."> (define (make-minetest-sexp author/name version repository commit> inputs home-page synopsis> description media-license license)> - "Return a S-expression for the minetest package with the given author/NAME,> + "Return a S-expression for the minetest package with the given AUTHOR/NAME,> VERSION, REPOSITORY, COMMIT, INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION,> MEDIA-LICENSE and LICENSE."> `(package> @@ -452,3 +452,37 @@ list of AUTHOR/NAME strings."> #:repo->guix-package minetest->guix-package*> #:guix-name> (compose contentdb->package-name author/name->name)))> +> +#|> +(define (minetest-package? pkg)> + (and (string-prefix? "minetest-" (package:package-name pkg))> + (assq-ref (package:package-properties pkg) 'upstream-name)))> +> +(define (latest-minetest-release pkg)> + "Return an <upstream-source> for the latest release of the package PKG."> + (define upstream-name> + (assoc-ref (package:package-properties pkg) 'upstream-name))> + (define contentdb-package (contentdb-fetch upstream-name))> + (define release (latest-release upstream-name))> + (and contentdb-package release> + (and-let* ((old-origin (package:package-source pkg))> + (old-reference (package:origin-uri old-origin))> + (is-git? (download:git-reference? old-reference))> + (commit (release-commit release)))> + (upstream-source> + (package (package:package-name pkg))> + (version (release-title release))> + (urls (download:git-reference> + (url (package-repository contentdb-package))> + (commit commit)))))))
Aha! This is actually what should be done, having the updater put thegit-reference into upstream-source, since the updater is going to knowbetter how to manipulate the uri.
Toggle quote (87 lines)> +> +(define %minetest-updater> + (upstream-updater> + (name 'minetest)> + (description "Updater for Minetest packages on ContentDB")> + (pred minetest-package?)> + (latest latest-minetest-release)))> +|#> +;; #:use-module (guix upstream)> +;; #:use-module ((guix git-download) #:prefix download:)> +;; #:use-module ((guix packages) #:prefix package:)> diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm> index fb6c52a567..4f3bbbcb94 100644> --- a/guix/scripts/refresh.scm> +++ b/guix/scripts/refresh.scm> @@ -28,8 +28,10 @@> #:use-module (guix ui)> #:use-module (gcrypt hash)> #:use-module (guix scripts)> + #:use-module (guix serialization)> #:use-module ((guix scripts build) #:select (%standard-build-options))> #:use-module (guix store)> + #:use-module (guix build utils)> #:use-module (guix utils)> #:use-module (guix packages)> #:use-module (guix profiles)> @@ -307,6 +309,17 @@ update would trigger a complete rebuild."> (G_ "no updater for ~a~%")> (package-name package)))> > +> +;; XXX adapted from (guix scripts hash)> +(define (file-hash file select? recursive?)> + ;; Compute the hash of FILE.> + (if recursive?> + (let-values (((port get-hash) (open-sha256-port)))> + (write-file file port #:select? select?)> + (force-output port)> + (get-hash))> + (call-with-input-file file port-sha256)))> +> (define* (update-package store package updaters> #:key (key-download 'interactive) warn?)> "Update the source file that defines PACKAGE with the new version.> @@ -347,8 +360,8 @@ warn about packages that have no matching updater."> (package-name package)> (upstream-input-change-name change)))> (upstream-source-input-changes source))> - (let ((hash (call-with-input-file tarball> - port-sha256)))> + (let ((hash (file-hash tarball (const #t)> + (directory-exists? tarball))))> (update-package-source package source hash)))> (warning (G_ "~a: version ~a could not be \> downloaded and authenticated; not updating~%")> diff --git a/guix/upstream.scm b/guix/upstream.scm> index 632e9ebc4f..61f67b57c1 100644> --- a/guix/upstream.scm> +++ b/guix/upstream.scm> @@ -24,6 +24,11 @@> #:use-module (guix discovery)> #:use-module ((guix download)> #:select (download-to-store url-fetch))> + #:use-module ((guix git-download)> + #:select (git-fetch git-reference?> + git-reference-url> + git-reference-commit> + git-reference-recursive?))> #:use-module (guix gnupg)> #:use-module (guix packages)> #:use-module (guix diagnostics)> @@ -33,6 +38,7 @@> #:use-module (guix store)> #:use-module ((guix derivations) #:select (built-derivations derivation->output-path))> #:autoload (gcrypt hash) (port-sha256)> + #:autoload (guix git) (latest-repository-commit)> #:use-module (guix monads)> #:use-module (srfi srfi-1)> #:use-module (srfi srfi-9)> @@ -93,7 +99,8 @@> upstream-source?> (package upstream-source-package) ;string> (version upstream-source-version) ;string> - (urls upstream-source-urls) ;list of strings> + ; list of strings or a <git-reference>> + (urls upstream-source-urls)
Is it possible for an updater to want to return a list of<git-reference>? I'm still not sure what the purpose of multiple urlsis, since nearly everthing seems to just take (first urls)...
Toggle quote (19 lines)> (signature-urls upstream-source-signature-urls ;#f | list of strings> (default #f))> (input-changes upstream-source-input-changes> @@ -361,6 +368,11 @@ values: 'interactive' (default), 'always', and 'never'."> system target)> "Download SOURCE from its first URL and lower it as a fixed-output> derivation that would fetch it."> + (define url> + (match (upstream-source-urls source)> + ((first . _) first)> + (_ (raise (formatted-message> + (G_ "git origins are unsupported by --with-latest"))))))> (mlet* %store-monad ((url -> (first (upstream-source-urls source)))> (signature> -> (and=> (upstream-source-signature-urls source)> @@ -430,9 +442,23 @@ SOURCE, an <upstream-source>."> #:key-download key-download)))> (values version tarball source))))))
What is this 'upstream-source-compiler' actually used for? I couldn'tfigure that out, so I just left it untouched.
Toggle quote (57 lines)> > +(define* (package-update/git-fetch store package source #:key key-download)> + "Return the version, source code directory, and SOURCE, to update PACKAGE to> +SOURCE, an <upstream-source>."> + (match source> + (($ <upstream-source> _ version ref _)> + (values version> + (latest-repository-commit> + store> + (git-reference-url ref)> + #:ref `(commit . ,(git-reference-commit ref))> + #:recursive? (git-reference-recursive? ref))> + source))))> +> (define %method-updates> ;; Mapping of origin methods to source update procedures.> - `((,url-fetch . ,package-update/url-fetch)))> + `((,url-fetch . ,package-update/url-fetch)> + (,git-fetch . ,package-update/git-fetch)))> > (define* (package-update store package> #:optional (updaters (force %updaters))> @@ -492,9 +518,22 @@ new version string if an update was made, and #f otherwise."> (origin-hash (package-source package))))> (old-url (match (origin-uri (package-source package))> ((? string? url) url)> + ((? git-reference? ref)> + (git-reference-url ref))> (_ #f)))> (new-url (match (upstream-source-urls source)> - ((first _ ...) first)))> + ((first _ ...) first)> + ((? git-reference? ref)> + (git-reference-url ref))> + (_ #f)))> + (old-commit (match (origin-uri (package-source package))> + ((? git-reference? ref)> + (git-reference-commit ref))> + (_ #f)))> + (new-commit (match (upstream-source-urls source)> + ((? git-reference? ref)> + (git-reference-commit ref))> + (_ #f)))> (file (and=> (location-file loc)> (cut search-path %load-path <>))))> (if file> @@ -508,6 +547,9 @@ new version string if an update was made, and #f otherwise."> 'filename file))> (replacements `((,old-version . ,version)> (,old-hash . ,hash)> + ,@(if (and old-commit new-commit)> + `((,old-commit . ,new-commit))> + '())> ,@(if (and old-url new-url)> `((,(dirname old-url) .> ,(dirname new-url)))
Thanks for sharing your work; it was very helpful!
--Sarah
M
M
Maxime Devos wrote on 17 Aug 12:18 +0200
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)
7986923ce7712dc341e859e62675abee12072922.camel@telenet.be
Sarah Morgensen schreef op ma 16-08-2021 om 12:56 [-0700]:
Toggle quote (30 lines)> Hi Maxime,> > Thanks for taking a look at this. :)> > Maxime Devos <maximedevos@telenet.be> writes:> > > Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:> > > * guix/git-download.scm (checkout-to-store): New procedure.> > > * guix/upstream.scm (guess-version-transform)> > > (package-update/git-fetch): New procedures.> > > (%method-updates): Add GIT-FETCH mapping.> > > > Does it support packages defined like (a)> > > > (define-public gnash> > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")> > (revision "0"))> > (package> > (name "gnash")> > (version (git-version "0.8.11" revision commit))> > (source (git-reference> > (url "https://example.org")> > (commit commit)))> > [...])))> > No, it doesn't. Since the commit definition isn't part of the actual> package definition, the current code has no way of updating it. It> would require a rewrite of the edit-in-place logic with probably a lot> of special-casing.
Perhaps a 'surrounding-expression-location' procedure can be defined?
(define (surrounding-expression-location inner-location) "Determine the location of the S-expression that surrounds the S-expressionat INNER-LOCATION, or #false if the inner S-expression is at the top-level." ??? Something like 'read', but in reverse, maybe? Doesn't need to support every construct, just "string without escapes" and (parentheses other-things) might be good enough in practice for now)
Seems tricky to implement, but it would be more robust than relyingon conventions like ‘the surrounding 'let' can be found by moving two columnsand two lines backwards’. Or see another method (let&) below that is actuallyimplemented ...
Toggle quote (13 lines)> There are currently ~1250 package which use this format, though, so it> could be worth it... Perhaps what we actually need is a better idiom to> express this situation. Package properties ('git-commit)? A 'git-version*'?> > --8<---------------cut here---------------start------------->8---> (define (git-version* version revision)> (let* ((source (package-source this-package))> (commit (git-reference-commit (origin-uri source))))> (git-version version revision commit)))> --8<---------------cut here---------------end--------------->8---> > I'm not sure if binding order would be an issue with that.
The 'file-name' field of 'origin' is not thunked, and refers to the 'version'field of the 'package' (also not thunked). If 'version' would use the 'git-version*'from above, then there would be a loop (I'm having the 'gnash' package in mind,see "guix edit gnash"). And git-version* cannot be a procedure, it must be a macro,as it used 'this-package', which can only be expanded inside a package definition.
Alternatively, what do you think of a let& macro, that adjusts the inner expressionto have the source location of the 'let&' form:
(define-syntax with-source-location (lambda (s) (syntax-case s () ((_ (exp . exp*) source) "Expand to (EXP . EXP*), but with the source location replacedby the source location of SOURCE." (datum->syntax s (cons #'exp #'exp*) #:source (syntax-source #'source))))))
(define-syntax let& (lambda (s) "Like 'let', but let the inner expression have the locationof the 'let&' form when it is expanded. Only a single innerexpression is allowed." (syntax-case s () ((_ bindings exp) #'(let bindings (with-source-location exp s))))))
That way, 'update-package-source' doesn't need to know about the surrounding'let' form; it would simply use 'edit-expression' as usual (though somethinglike
,@(if (and old-commit new-commit) `((,old-commit . ,new-commit)) '())
would need to be added, and something to replace ‘(revision "N")’ with‘(revision "N+1")’.)
A complete example is attached (a.scm). The previous usages of(let ((commit ...) (revision ...)) ...) would need to be adjustedto use let& instead (build-aux/update-guix-package.scm needs tobe adjusted as well).
Personally, I'd go with the 'let&' form
Toggle quote (21 lines)> > and (b)> > > > (define-public gnash> > (package> > (name "gnash")> > (version "0.8.11")> > (source (git-reference> > (url "https://example.org")> > (commit commit))> > [...]))> > ?> > Is this missing a definition for commit? If it's like above, the same> applies. Or if you mean> > --8<---------------cut here---------------start------------->8---> (source (git-reference> (url "https://example.org")> (commit "583ccbc1275c7701dc4843ec12142ff86bb305b"))> --8<---------------cut here---------------end--------------->8---
The latter.
Toggle quote (3 lines)> Then that wouldn't be too hard to support. There seem to be ~136> packages with this idiom.
FWIW, the patch I sent modified 'update-package-source' to replacethe commit in this case (b) (but not case (a)).
Toggle quote (12 lines)> > [the patch Maxime sent]> > > > upstream-source?> > (package upstream-source-package) ;string> > (version upstream-source-version) ;string> > - (urls upstream-source-urls) ;list of strings> > + ; list of strings or a <git-reference>> > + (urls upstream-source-urls)> > Is it possible for an updater to want to return a list of> <git-reference>?
No, 'git-fetch' from (guix git-download) only accepts a single <git-reference>object, it doesn't support lists of <git-reference>. It will throw a typeerror if a list is passed. Compare with 'url-fetch*', which does accept a listof URLs (in which case it will fall-back to the second, the third, the fourth ...entry when the first entry gives a 404 or something).
Toggle quote (3 lines)> I'm still not sure what the purpose of multiple urls> is, since nearly everthing seems to just take (first urls)...
As I understand it, the second, third, fourth ... URL (when using url-fetch)are fall-backs. Also, (guix upstream) sometimes distinguishes between thedifferent URLs, see e.g. package-update/url-fetch, which will try to choose atarball with the same kind of extension (.zip, .tar.gz, .tar.xz, ...) as the originalURI.
Toggle quote (22 lines)> > (signature-urls upstream-source-signature-urls ;#f | list of strings> > (default #f))> > (input-changes upstream-source-input-changes> > @@ -361,6 +368,11 @@ values: 'interactive' (default), 'always', and 'never'."> > system target)> > "Download SOURCE from its first URL and lower it as a fixed-output> > derivation that would fetch it."> > + (define url> > + (match (upstream-source-urls source)> > + ((first . _) first)> > + (_ (raise (formatted-message> > + (G_ "git origins are unsupported by --with-latest"))))))> > (mlet* %store-monad ((url -> (first (upstream-source-urls source)))> > (signature> > -> (and=> (upstream-source-signature-urls source)> > @@ -430,9 +442,23 @@ SOURCE, an <upstream-source>."> > #:key-download key-download)))> > (values version tarball source))))))> > What is this 'upstream-source-compiler' actually used for? I couldn't> figure that out, so I just left it untouched.
It is used to ‘lower’ <upstream-source> objects. More specifically,transform-package-latest from (guix transformations) will sometimesreplace the 'source' of a package with a <upstream-source> object,and 'upstream-source-compiler' is used to turn the <upstream-source>into a (fixed-output) derivation that can be built into a/gnu/store/...-checkout or /gnu/store/...-version.tar.gz file in the store.
Greetings,Maxime
;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be> ;;; ;;; This file is part of GNU Guix. ;;; ;;; GNU Guix is free software; you can redistribute it and/or modify it ;;; under the terms of the GNU General Public License as published by ;;; the Free Software Foundation; either version 3 of the License, or (at ;;; your option) any later version. ;;; ;;; GNU Guix is distributed in the hope that it will be useful, but ;;; WITHOUT ANY WARRANTY; without even the implied warranty of ;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;;; GNU General Public License for more details. ;;; ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. (use-modules (guix packages) (gnu packages animation) (guix git-download)) (define-syntax with-source-location (lambda (s) (syntax-case s () ((_ (exp . exp*) source) "Expand to (EXP . EXP*), but with the source location replaced by the source location of SOURCE." (datum->syntax s (cons #'exp #'exp*) #:source (syntax-source #'source)))))) (define-syntax let& (lambda (s) "Like 'let', but let the inner expression have the location of the 'let&' form when it is expanded. Only a single inner expression is allowed." (syntax-case s () ((_ bindings exp) #'(let bindings (with-source-location exp s)))))) (define-public gnash2 ;; The last tagged release of Gnash was in 2013. (let& ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4") (revision "0")) (package (inherit gnash) (name "gnash2") (version (git-version "0.8.11" revision commit)) (source (origin (method git-fetch) (uri (git-reference (url "https://git.savannah.gnu.org/git/gnash.git/") (commit commit))) (file-name (git-file-name name version)) (patches (search-patches "gnash-fix-giflib-version.patch")) (sha256 (base32 "0fh0bljn0i6ypyh6l99afi855p7ki7lm869nq1qj6k8hrrwhmfry"))))))) (format #t "old: ~a~%" (package-location gnash)) (format #t "new: ~a~%" (package-location gnash2)) ;; ^ it says column 2, which is the column of the let& form.
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYRuM9BccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7sCHAP9xBj75KhXqSDRyCSSQFyATroZ3K22ko8hK0JHF1DtoFAD/U7vAsQq8rQXjBlwMe2f+W0O55HD6OTDRwZ99pKRdwwo==BfS0-----END PGP SIGNATURE-----

X
X
Xinglu Chen wrote on 18 Aug 16:45 +0200
Re: [bug#50072] [PATCH WIP 4/4] upstream: Support updating git-fetch origins.
878s0yke14.fsf@yoctocell.xyz
On Mon, Aug 16 2021, Maxime Devos wrote:
Toggle quote (28 lines)> Xinglu Chen schreef op ma 16-08-2021 om 15:02 [+0200]:>> On Mon, Aug 16 2021, Maxime Devos wrote:>> >> > Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:>> > > * guix/git-download.scm (checkout-to-store): New procedure.>> > > * guix/upstream.scm (guess-version-transform)>> > > (package-update/git-fetch): New procedures.>> > > (%method-updates): Add GIT-FETCH mapping.>> > >> > Does it support packages defined like (a)>> > >> > (define-public gnash>> > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")>> > (revision "0"))>> > (package>> > (name "gnash")>> > (version (git-version "0.8.11" revision commit))>> > (source (git-reference>> > (url "https://example.org")>> >> IIUC, it only supports GitHub URLs at the moment. I have a WIP patch>> for adding support for any arbitrary Git repository[1].>> This patch series doesn't mention GitHub anywhere (except in the patch> series description) so I don't think it only supports GitHub URLs.> Admittedly, only one updater, "github", currently produces git-reference> URLs,
That was what I was referring to, sorry for not making it clearer.
Only the ‘github’ updater can update ‘git-fetch’ origins;=> only GitHub URLs can are recognized by the ‘github’ updater;=> thus, only packages hosted on GitHub can be updated.
Toggle quote (6 lines)> but I sent a patch series [2] that adds an importer which produces> git-reference URLs and the corresponding updater (see ‘git-fetch.patch’)> produces appropriate git-reference objects.>> [2]: <https://issues.guix.gnu.org/49828#51>.
I haven’t looked at the patches yet, but that looks very cool! :-)
-----BEGIN PGP SIGNATURE-----
iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmEdHPcVHHB1YmxpY0B5b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5iSgQAIzTNsUOYtetadiDvv7HCvaXGbG4RSC+bJmKVaayMPBsOcSPOIkiT+YwYambXyGSZqyyuxc6oSbBXtmuUHxD63tpoYIJNMutoH8f4GLsJxLwojZP+OWoduIC8cQtyxr24KJyqN58tIkFgg+QYihGPktg9CsNXI8AurDzCA8DFnT1r6sDM3+5zQV7PTKBvRavPh6bJDvUQgh7m3rNOu7rfC8XGcO7VAOIHni8TW66MKcgXHDVLlSxzG9Ab0OhpaTSOKhcNNc0dGjrp4jR+CMqjR8Yg+bO7JB6fWfEYJRxjHV27NslJ26qn7NgpXEIDoXUSxAT7C0qQ81Xv0RCavV7ZT6USb0RNpQCNcblrV0maoYP2aQPSvKNSTTI9t19Z5jBghQZ0iuIUpu6lhf12BULemR+Pv3O6yy9m3OOLvaMJRzQjs4BmVqHaUMaYrpNODl6f3TrBEWW6eBaV6YU5qm4Ub+u8glMfCiiCexYjMeJZhftS3M5FQnWMMk31Li+11uuixT73kARVW3ukNZothcQAX3KSG0d+jP/gPeunZ0EwAAlVYEmo8A8h6MZtyP8DMlAo0fvc7XuTeMbSnD9Dx/DsTb4niOjSWi7zeqxJS6koXAFd3KnOZ6steMEBZZuBvak74UOMVK0cK/y0Pm3k/gG86VmufF9j/jetFShbiDluYa6=Ct9e-----END PGP SIGNATURE-----
M
M
Maxime Devos wrote on 30 Aug 23:36 +0200
Re: [bug#50072] [PATCH WIP 0/4] Add upstream updater for git-fetch origins.
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)
2c76cd38605f4358df29ed0d464a8f5512b1b662.camel@telenet.be
Maxime Devos schreef op di 17-08-2021 om 12:18 [+0200]:
Toggle quote (14 lines)> [... stuff about let&, let*&, supporting packages like:> > > (define-public gnash> > > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")> > > (revision "0"))> > > (package> > > (name "gnash")> > > (version (git-version "0.8.11" revision commit))> > > (source (git-reference> > > (url "https://example.org")> > > (commit commit)))> > > [...])))> > ...> ... by fudging the source locations ...]
I went ahead and send a patch replacing 'let' with 'let&':https://issues.guix.gnu.org/50286.
Greetings,Maxime.
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYS1PXBccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7h2rAP0VJi/wmuk0zSVG122dfNY7jHOMb27we11DqPJz77yfpgEA8Io5XvYfNa2WMyAt84BkSWyM/IlMlsrnLUtm9ImBcQM==P2nY-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 6 Sep 12:23 +0200
Re: bug#50072: [PATCH WIP 0/4] Add upstream updater for git-fetch origins.
(name . Maxime Devos)(address . maximedevos@telenet.be)
87k0ju80kk.fsf_-_@gnu.org
Hi,
Maxime Devos <maximedevos@telenet.be> skribis:
Toggle quote (45 lines)> Sarah Morgensen schreef op ma 16-08-2021 om 12:56 [-0700]:>> Hi Maxime,>> >> Thanks for taking a look at this. :)>> >> Maxime Devos <maximedevos@telenet.be> writes:>> >> > Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:>> > > * guix/git-download.scm (checkout-to-store): New procedure.>> > > * guix/upstream.scm (guess-version-transform)>> > > (package-update/git-fetch): New procedures.>> > > (%method-updates): Add GIT-FETCH mapping.>> > >> > Does it support packages defined like (a)>> > >> > (define-public gnash>> > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")>> > (revision "0"))>> > (package>> > (name "gnash")>> > (version (git-version "0.8.11" revision commit))>> > (source (git-reference>> > (url "https://example.org")>> > (commit commit)))>> > [...])))>> >> No, it doesn't. Since the commit definition isn't part of the actual>> package definition, the current code has no way of updating it. It>> would require a rewrite of the edit-in-place logic with probably a lot>> of special-casing.>> Perhaps a 'surrounding-expression-location' procedure can be defined?>> (define (surrounding-expression-location inner-location)> "Determine the location of the S-expression that surrounds the S-expression> at INNER-LOCATION, or #false if the inner S-expression is at the top-level."> ??? Something like 'read', but in reverse, maybe?> Doesn't need to support every construct, just "string without escapes" and> (parentheses other-things) might be good enough in practice for now)>> Seems tricky to implement, but it would be more robust than relying> on conventions like ‘the surrounding 'let' can be found by moving two columns> and two lines backwards’. Or see another method (let&) below that is actually> implemented ...
I think we can work incrementally. It wouldn’t be unreasonable to startwith a ‘definition-location’ procedure that would work in a way similarto ‘package-field-location’ (essentially ‘read’ each top-level sexp ofthe file and record the location of the one that immediately precedesthe package location.)
But maybe the discussion in https://issues.guix.gnu.org/50286 willgive us something nice.
Thanks,Ludo’.
L
L
Ludovic Courtès wrote on 6 Sep 12:27 +0200
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)(address . 50072@debbugs.gnu.org)
87fsui80dd.fsf_-_@gnu.org
Hi Sarah,
I like this patch series. :-)
Sarah Morgensen <iskarian@mgsn.dev> skribis:
Toggle quote (5 lines)> * guix/git-download.scm (checkout-to-store): New procedure.> * guix/upstream.scm (guess-version-transform)> (package-update/git-fetch): New procedures.> (%method-updates): Add GIT-FETCH mapping.
This LGTM.
Nitpick:
Toggle quote (7 lines)> +(define* (checkout-to-store store ref #:key (log (current-error-port)))> + "Checkout REF to STORE. Write progress reports to LOG. RECURSIVE? has the> +same effect as the same-named parameter of 'latest-repository-commit'."> + ;; XXX: (guix git) does not use shallow clones, so this will be slow> + ;; for long-running repositories.> + (match-record ref <git-reference>
[...]
Toggle quote (4 lines)> + ;; Only use the first element of URLS.> + (match-record source <upstream-source>> + (version urls)
I’d use the record acceesors in this cases rather than ‘match-record’;currently ‘match-record’ is not super efficient and I find it slightlyless readable when you’re just accessing a couple of fields.
Thanks,Ludo’.
M
M
Maxime Devos wrote on 6 Sep 13:47 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)
10996c63a8690ceea6c88a5fc88ddabad7b000dd.camel@telenet.be
Hi,
Ludovic Courtès schreef op ma 06-09-2021 om 12:23 [+0200]:
Toggle quote (40 lines)> > > > > > > > [...]> > > > Does it support packages defined like (a)> > > > > > > > (define-public gnash> > > > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")> > > > (revision "0"))> > > > (package> > > > (name "gnash")> > > > (version (git-version "0.8.11" revision commit))> > > > (source (git-reference> > > > (url "https://example.org")> > > > (commit commit)))> > > > [...])))> > > > > > No, it doesn't. Since the commit definition isn't part of the actual> > > package definition, the current code has no way of updating it. It> > > would require a rewrite of the edit-in-place logic with probably a lot> > > of special-casing.> > > > Perhaps a 'surrounding-expression-location' procedure can be defined?> > > > (define (surrounding-expression-location inner-location)> > "Determine the location of the S-expression that surrounds the S-expression> > at INNER-LOCATION, or #false if the inner S-expression is at the top-level."> > ??? Something like 'read', but in reverse, maybe?> > Doesn't need to support every construct, just "string without escapes" and> > (parentheses other-things) might be good enough in practice for now)> > > > Seems tricky to implement, but it would be more robust than relying> > on conventions like ‘the surrounding 'let' can be found by moving two columns> > and two lines backwards’. Or see another method (let&) below that is actually> > implemented ...> > I think we can work incrementally. It wouldn’t be unreasonable to start> with a ‘definition-location’ procedure that would work in a way similar> to ‘package-field-location’ (essentially ‘read’ each top-level sexp of> the file and record the location of the one that immediately precedes> the package location.)
‘package-field-location’ (currently) doesn't work like that. Currently,it extracts the location from the package, opens the file, uses a procedure'goto' that works like 'seek' except that it accepts line and column numbersinstead of byte offsets.
What you proposed could work, though it seems a bit inefficient to me.Asking upstream for an update probably takes a lot more time though.
Toggle quote (3 lines)> But maybe the discussion in https://issues.guix.gnu.org/50286 will> give us something nice.
Greetings,Maxime
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYTX/1xccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7ophAQC7GSfWH0C/M+dLr8Jkdg0mny/ryoG85v0kcMpD6jIk/AD+MgzJwEj1oLYkRndZc9Ql4ILaODgpj0tHD0orEEKA/gk==koNy-----END PGP SIGNATURE-----

S
S
Sarah Morgensen wrote on 7 Sep 03:16 +0200
Re: [bug#50072] [PATCH WIP 4/4] upstream: Support updating git-fetch origins.
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 50072@debbugs.gnu.org)
86mtopi3s7.fsf@mgsn.dev
Hi Maxime,
Maxime Devos <maximedevos@telenet.be> writes:
Toggle quote (19 lines)> Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:>> * guix/git-download.scm (checkout-to-store): New procedure.>> * guix/upstream.scm (guess-version-transform)>> (package-update/git-fetch): New procedures.>> (%method-updates): Add GIT-FETCH mapping.>> Does it support packages defined like (a)>> (define-public gnash> (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")> (revision "0"))> (package> (name "gnash")> (version (git-version "0.8.11" revision commit))> (source (git-reference> (url "https://example.org")> (commit commit)))> [...])))
Thinking about this again, since updaters typically returns actualversions (tags) instead of commits, how much would such afeature be used?
OTOH, I could definitely see use for an ability to update packages likethese to proper versions (removing the surrounding 'let') but that'sprobably more rare and may not be worth the implementation effort.
--SEarah
S
S
Sarah Morgensen wrote on 7 Sep 03:59 +0200
Re: [bug#50072] [PATCH WIP 0/4] Add upstream updater for git-fetch origins.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 50072@debbugs.gnu.org)
86k0jti1r4.fsf@mgsn.dev
Hi Ludo,
Ludovic Courtès <ludo@gnu.org> writes:
Toggle quote (4 lines)> Hi Sarah,>> I like this patch series. :-)
Thanks for taking a look!
Toggle quote (10 lines)>> Sarah Morgensen <iskarian@mgsn.dev> skribis:>>> * guix/git-download.scm (checkout-to-store): New procedure.>> * guix/upstream.scm (guess-version-transform)>> (package-update/git-fetch): New procedures.>> (%method-updates): Add GIT-FETCH mapping.>> This LGTM.
Thanks. WDYT about pre-emptively adding support for non-url URIs aswell? That is,
1. change "urls" in <upstream-source> to "uri"
2. in 'git-fetch'
a) if the upstream-source-uri is a git-reference, just use it as-is rather than guessing the tag
b) if it's not, return an 'upstream-source' with a git-reference URI
3. update 'upstream-source-compiler' to work for git-reference URIs.
If there are no objections, I think I'll make those changes and sendthat as a proper patch.
Toggle quote (20 lines)>> Nitpick:>>> +(define* (checkout-to-store store ref #:key (log (current-error-port)))>> + "Checkout REF to STORE. Write progress reports to LOG. RECURSIVE? has the>> +same effect as the same-named parameter of 'latest-repository-commit'.">> + ;; XXX: (guix git) does not use shallow clones, so this will be slow>> + ;; for long-running repositories.>> + (match-record ref <git-reference>>> [...]>>> + ;; Only use the first element of URLS.>> + (match-record source <upstream-source>>> + (version urls)>> I’d use the record acceesors in this cases rather than ‘match-record’;> currently ‘match-record’ is not super efficient and I find it slightly> less readable when you’re just accessing a couple of fields.
Fair. I got a little excited to discover new syntax :)
--Sarah
M
M
Maxime Devos wrote on 7 Sep 12:00 +0200
Re: [bug#50072] [PATCH WIP 4/4] upstream: Support updating git-fetch origins.
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)(address . 50072@debbugs.gnu.org)
470a95f159120d14384c3096cff91e3ce8d3c6c8.camel@telenet.be
Sarah Morgensen schreef op ma 06-09-2021 om 18:16 [-0700]:
Toggle quote (27 lines)> Hi Maxime,> > Maxime Devos <maximedevos@telenet.be> writes:> > > Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:> > > * guix/git-download.scm (checkout-to-store): New procedure.> > > * guix/upstream.scm (guess-version-transform)> > > (package-update/git-fetch): New procedures.> > > (%method-updates): Add GIT-FETCH mapping.> > > > Does it support packages defined like (a)> > > > (define-public gnash> > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")> > (revision "0"))> > (package> > (name "gnash")> > (version (git-version "0.8.11" revision commit))> > (source (git-reference> > (url "https://example.org")> > (commit commit)))> > [...])))> > Thinking about this again, since updaters typically returns actual> versions (tags) instead of commits, how much would such a> feature be used?
The minetest updater returns version numbers.It also returns a git-reference object, which includes the commit.Just returning a version number often isn't sufficient,because many repositories of minetest mods do not keep version tags.
See https://issues.guix.gnu.org/50072#5.
Greetings,Maxime.
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYTc4ShccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7oCxAP9Ij02MwmdqzRHO0untiA8XMwvQ5mIZFI/F4rGBs+bFVAEA/HGbTDWV65zhq7GRhx3iSkL8dQ7TNZLaOhn1KKNnpAc==OfIQ-----END PGP SIGNATURE-----

S
S
Sarah Morgensen wrote on 7 Sep 19:51 +0200
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 50072@debbugs.gnu.org)
86a6koi8ac.fsf@mgsn.dev
Hi,
Maxime Devos <maximedevos@telenet.be> writes:
Toggle quote (33 lines)> Sarah Morgensen schreef op ma 06-09-2021 om 18:16 [-0700]:>> Hi Maxime,>> >> Maxime Devos <maximedevos@telenet.be> writes:>> >> > Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]:>> > > * guix/git-download.scm (checkout-to-store): New procedure.>> > > * guix/upstream.scm (guess-version-transform)>> > > (package-update/git-fetch): New procedures.>> > > (%method-updates): Add GIT-FETCH mapping.>> > >> > Does it support packages defined like (a)>> > >> > (define-public gnash>> > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4")>> > (revision "0"))>> > (package>> > (name "gnash")>> > (version (git-version "0.8.11" revision commit))>> > (source (git-reference>> > (url "https://example.org")>> > (commit commit)))>> > [...])))>> >> Thinking about this again, since updaters typically returns actual>> versions (tags) instead of commits, how much would such a>> feature be used?>> The minetest updater returns version numbers.> It also returns a git-reference object, which includes the commit.> Just returning a version number often isn't sufficient,> because many repositories of minetest mods do not keep version tags.
Thanks for the explanation.
So there is a version number indicated elsewhere than in the tags forsome minetest packages? (Is this data in the package's git repo or ine.g. minetest repo metadata?) That is, the minetest updater always uses"blessed versions" (not just random commits), such that "revision" willalways be "0"?
Are current minetest packages like this formatted like 'gnash' above?
Toggle quote (2 lines)> See https://issues.guix.gnu.org/50072#5.
That's the message I quoted ;)
--Sarah
M
M
Maxime Devos wrote on 7 Sep 22:58 +0200
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)(address . 50072@debbugs.gnu.org)
7905b183a5865bb597b10f8440074efa463ab544.camel@telenet.be
Sarah Morgensen schreef op di 07-09-2021 om 10:51 [-0700]:
Toggle quote (6 lines)> So there is a version number indicated elsewhere than in the tags for> some minetest packages? (Is this data in the package's git repo or in> e.g. minetest repo metadata?) That is, the minetest updater always uses> "blessed versions" (not just random commits), such that "revision" will> always be "0"?
The minetest importer looks at ContentDB. E.g., for Jeija/mesecons:https://content.minetest.net/packages/Jeija/mesecons/. It doesn't lookat git tags at all. It only clones the git repository to compute the hash.
Strictly speaking, ContentDB only has ‘release titles’, and not ‘version numbers’.Release titles are usually version numbers or dates. In the former case, all iswell. In the latter case, there isn't much the importer/updater can do about that,so it will use the date even though it isn't a ‘proper version number’.
Releases on ContentDB are ordered. The importer and refresher always use thelatest release, not some random commit. ContentDB has a mapping from releasesto their commits, which the importer and refresher uses.
So, yes, there are ‘blessed versions’. However, due to particularities of howminetest mods are released, revision won't always be 0, because there areminetest mods that make a new release on ContentDB without a correspondingversion bump (e.g. minetest-ethereal, minetest-mesecons, minetest-throwing,minetest-throwing-arrows).
Toggle quote (2 lines)> Are current minetest packages like this formatted like 'gnash' above?
About a third are formatted like 'gnash' (let ((commit ...) (revision ...)) ...)).
Greetings,Maxime
-----BEGIN PGP SIGNATURE-----
iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYTfSYBccbWF4aW1lZGV2b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7lPAAQCDu0xEAeybas2f1Y3lnGkGYxJK9NbcMQWMRRWX6bmLvgD/Wkkcowr0BIACPDCucoDP4n2KqS/W+PHletLtFJ+xNQ8==mgxY-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 29 Sep 23:28 +0200
Re: bug#50072: [PATCH WIP 0/4] Add upstream updater for git-fetch origins.
(name . Sarah Morgensen)(address . iskarian@mgsn.dev)(address . 50072@debbugs.gnu.org)
87a6jv6pgm.fsf_-_@gnu.org
Hi Sarah,
I just noticed I hadn’t answered this message…
Sarah Morgensen <iskarian@mgsn.dev> skribis:
Toggle quote (2 lines)> Ludovic Courtès <ludo@gnu.org> writes:
[...]
Toggle quote (26 lines)>> Sarah Morgensen <iskarian@mgsn.dev> skribis:>>>>> * guix/git-download.scm (checkout-to-store): New procedure.>>> * guix/upstream.scm (guess-version-transform)>>> (package-update/git-fetch): New procedures.>>> (%method-updates): Add GIT-FETCH mapping.>>>> This LGTM.>> Thanks. WDYT about pre-emptively adding support for non-url URIs as> well? That is,>> 1. change "urls" in <upstream-source> to "uri">> 2. in 'git-fetch'>> a) if the upstream-source-uri is a git-reference, just use it as-is> rather than guessing the tag>> b) if it's not, return an 'upstream-source' with a git-reference URI >> 3. update 'upstream-source-compiler' to work for git-reference URIs.>> If there are no objections, I think I'll make those changes and send> that as a proper patch.
That sounds like a good idea. We’ll need to check users of‘upstream-source-urls’ & co. and see whether/how they can deal withgeneralized “URIs”.
That said, perhaps it can come after this patch series, which I thinkwas mostly waiting on ‘package-definition-location’ initially?
Thanks,Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 50072@debbugs.gnu.org