From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 07 15:13:38 2021 Received: (at 50359) by debbugs.gnu.org; 7 Sep 2021 19:13:38 +0000 Received: from localhost ([127.0.0.1]:57771 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNgXJ-0005LB-6p for submit@debbugs.gnu.org; Tue, 07 Sep 2021 15:13:38 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:51562 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNgXG-0005Kt-4E for 50359@debbugs.gnu.org; Tue, 07 Sep 2021 15:13:35 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631042005; bh=DLb7xkBOXXKVJSPIRcO86Zhb4/KB0Iu8cDZSok1Pkhk=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=i9smTY8/QkOYWKejtO5vC8Fy24djHjLWQYhEKwznR3WHQZ3AwM/8KLYNMJErr2fr+ Ky+4vIx/iRGF9oInQj9zfPgyH8LPq/DqUsIgPqYvEAuLgyiv+Lo+zRVmvSGy+bEVYB 1RzSz3b7JCBn5Mf5yVAZtmUXXEXm9m+8k7qovVW8= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: <86pmtli4hn.fsf@mgsn.dev> References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> <87y28928vh.fsf@yoctocell.xyz> <86pmtli4hn.fsf@mgsn.dev> Date: Tue, 07 Sep 2021 21:13:22 +0200 Message-ID: <87wnnsyzal.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Mon, Sep 06 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> Any trick you used to find all of there weird version numbers? :-) > > This monstrosity: > > rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(versio [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Mon, Sep 06 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> Any trick you used to find all of there weird version numbers? :-) > > This monstrosity: > > rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(versio [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Mon, Sep 06 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> Any trick you used to find all of there weird version numbers? :-) > > This monstrosity: > > rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(version "([^\n"]*[^0-9\.][^"= \n]*)".*(\n.*){0,10}commit.*version' gnu/packages > > and to show just the versions: > > rg -Uor '$2' --pcre2 --no-filename --no-line-number=20 Wow! I will try that and see for myself! :-) >>> IMO, just get rid of the delimiter. If we wanted to be *that* flexible, >>> we could make it so they provide a tag->version proc instead of (prefix, >>> suffix, delimiter). >> >> a =E2=80=98tag->version=E2=80=99 procedure would probably make things a = bit too >> complicated for the people writing package definitions. For example, >> having a delimiter would make it easy to match a tag like >> =E2=80=9C2021-01-01-release=E2=80=9D >> >> Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) >> >> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:pun= ct:]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) >> $28 =3D "2021" >> >> Delimiter is =E2=80=9C-=E2=80=9D >> >> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct= :]]+(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) >> $29 =3D "2021-01-01-release" >> >> And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match j= ust the version >> part. > > Right. I missed that. > > In that vein, should we keep the dashes in "2021-01-01" or convert them > to periods? Having periods would be more consistent, then could have a =E2=80=98date->version=E2=80=99 procedure that replaces the hyphens with do= ts and have (git-reference (url "https://git.example.org") (commit (date->version version))) > What about when a tag has underscores? Hmm, not sure about that, below is a list of packages I could find which had underscores as delimiters gnu/packages/graphics.scm 239: (commit "DIRECTFB_1_7_7"))) gnu/packages/gstreamer.scm 326: (commit "ESOUND_0_2_41"))) gnu/packages/java.scm 13925: (commit "jboss-transaction-api_1.2_spec-1.1.1.F= inal"))) They all seem to use periods in the =E2=80=98version=E2=80=99 field, though= , so I would say that the underscroes, should also be converted to periods. > What if a repo has tags in both formats? Then "3.0.1" would be > considered older than "2011-01-01". That=E2=80=99s tricky, there isn=E2=80=99t really a way to know how old =E2= =80=9C3.0.1=E2=80=9D is, without looking at the metadata of the tag. Maybe this is one of those corner cases which can=E2=80=99t really automatically determine the latest release. Should we have a =E2=80=98no-refresh?=E2=80=99 property to tell t= he refresh to not try to update the package? > Maybe we should just add an extra bit to detect a date format and only > consider it when there's no "proper versions"? That could be a good idea! > Aaaand I fell down a rabbit hole after that :) I've attached a patch > with what I've done. It still has lots of issues--it requires the tag > to contain at least one version delimiter, it requires the first > character of the version to be a number... it might not even be better > than before I touched it, and even so the added complexity might not be > worth it. But if you'd like to take it for a spin, I've attached it (it > applies straight on master). Great! I will try it out and see how it compares to my current WIP version. Not having characters in the first version number probably isn=E2=80=99t su= ch a big deal, most version that contain characters end with a character. E.g., =E2=80=9C1.2.3a=E2=80=9D is not to uncommon, but =E2=80=9Ca1.2.3=E2= =80=9D is rarely seen. > -- > Sarah > > From 08bd59a7fa1aa9735a1794672ce8d1f683d3d6db Mon Sep 17 00:00:00 2001 > Message-Id: <08bd59a7fa1aa9735a1794672ce8d1f683d3d6db.1630975873.git.iska= rian@mgsn.dev> > From: Xinglu Chen > Date: Fri, 3 Sep 2021 17:50:56 +0200 > Subject: [PATCH] import: Add 'generic-git' updater. > > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * Makefile.am (MODULES): Register it. > --- > Makefile.am | 1 + > doc/guix.texi | 27 +++++++ > guix/git.scm | 33 ++++++++ > guix/import/git.scm | 191 ++++++++++++++++++++++++++++++++++++++++++++ > 4 files changed, 252 insertions(+) > create mode 100644 guix/import/git.scm > > diff --git a/Makefile.am b/Makefile.am > index 3c79760734..c4d3a456b1 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -254,6 +254,7 @@ MODULES =3D \ > guix/import/egg.scm \ > guix/import/elpa.scm \ > guix/import/gem.scm \ > + guix/import/git.scm \ > guix/import/github.scm \ > guix/import/gnome.scm \ > guix/import/gnu.scm \ > diff --git a/doc/guix.texi b/doc/guix.texi > index 36a0c7f5ec..26afb1607a 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Lau= nchpad} packages. > @item generic-html > a generic updater that crawls the HTML page where the source tarball of > the package is hosted, when applicable. > +@item generic-git > +a generic updater for packages hosted on Git repositories. It tries to > +be smart about parsing Git tag names, but if it is not able to parse the > +tag name and compare tags correctly, users can define the following > +properties for a package. > + > +@itemize > +@item @code{tag-prefix}: a regular expression for matching a prefix of > +the tag name. > + > +@item @code{tag-suffix}: a regular expression for matching a suffix of > +the tag name. > + > +@item @code{tag-version-delimiter}: a string used as the delimiter in > +the tag name for separating the numbers of the version. > +@end itemize > + > +@lisp > +(package > + (name "foo") > + ;; ... > + (properties > + '((tag-prefix . "^release0-") > + (tag-suffix . "[a-z]?$") > + (tag-version-delimiter . ":")))) > +@end lisp=20=20=20=20=20=20 > + > @end table >=20=20 > For instance, the following command only checks for updates of Emacs > diff --git a/guix/git.scm b/guix/git.scm > index 9c6f326c36..c5d0d2da8e 100644 > --- a/guix/git.scm > +++ b/guix/git.scm > @@ -56,6 +56,8 @@ > commit-difference > commit-relation >=20=20 > + ls-remote-refs > + > git-checkout > git-checkout? > git-checkout-url > @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor = of NEW), 'descendant, or > (if (set-contains? oldest new) > 'descendant > 'unrelated)))))) > + > +;; > +;;; Remote operations. > +;;; > + > +(define* (ls-remote-refs url #:key tags?) > + "Return the list of references advertised at Git repository URL. If T= AGS? > +is true, limit to only refs/tags." > + (define (ref? ref) > + ;; Like `git ls-remote --refs', only show actual references. > + (and (string-prefix? "refs/" ref) > + (not (string-suffix? "^{}" ref)))) > + > + (define (tag? ref) > + (string-prefix? "refs/tags/" ref)) > + > + (define (include? ref) > + (and (ref? ref) > + (or (not tags?) (tag? ref)))) > + > + (with-libgit2 > + (call-with-temporary-directory > + (lambda (cache-directory) > + (let* ((repository (repository-init cache-directory)) > + ;; Create an in-memory remote so we don't touch disk. > + (remote (remote-create-anonymous repository url))) > + (remote-connect remote) > + (remote-disconnect remote) > + (repository-close! repository) > + > + (filter include? (map remote-head-name (remote-ls remote)))))))) >=20=20 > > ;;; > diff --git a/guix/import/git.scm b/guix/import/git.scm > new file mode 100644 > index 0000000000..8568981af2 > --- /dev/null > +++ b/guix/import/git.scm > @@ -0,0 +1,191 @@ > +;;; GNU Guix --- Functional package management for GNU > +;;; Copyright =C2=A9 2021 Xinglu Chen > +;;; Copyright =C2=A9 2021 Sarah Morgensen > +;;; > +;;; This file is part of GNU Guix. > +;;; > +;;; GNU Guix is free software; you can redistribute it and/or modify it > +;;; under the terms of the GNU General Public License as published by > +;;; the Free Software Foundation; either version 3 of the License, or (at > +;;; your option) any later version. > +;;; > +;;; GNU Guix is distributed in the hope that it will be useful, but > +;;; WITHOUT ANY WARRANTY; without even the implied warranty of > +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +;;; GNU General Public License for more details. > +;;; > +;;; You should have received a copy of the GNU General Public License > +;;; along with GNU Guix. If not, see . > + > +(define-module (guix import git) > + #:use-module (guix build utils) > + #:use-module (guix diagnostics) > + #:use-module (guix git) > + #:use-module (guix git-download) > + #:use-module (guix i18n) > + #:use-module (guix packages) > + #:use-module (guix upstream) > + #:use-module (guix utils) > + #:use-module (ice-9 format) > + #:use-module (ice-9 match) > + #:use-module (ice-9 rdelim) > + #:use-module (ice-9 regex) > + #:use-module (srfi srfi-1) > + #:use-module (srfi srfi-26) > + #:use-module (srfi srfi-34) > + #:use-module (srfi srfi-35) > + #:use-module (srfi srfi-71) > + #:export (%generic-git-updater)) > + > +;;; Commentary: > +;;; > +;;; This module provides a generic package updater for packages hosted o= n Git > +;;; repositories. > +;;; > +;;; It tries to be smart about tag names, but if it is not automatically= able > +;;; to parse the tag names correctly, users can set the `tag-prefix', > +;;; `tag-suffix' and `tag-version-delimiter' properties of the package t= o make > +;;; the updater parse the Git tag name correctly. > +;;; > +;;; Code: > + > +;;; Errors & warnings > + > +(define-condition-type &git-no-valid-tags-error &error > + git-no-valid-tags-error?) > + > +(define (git-no-valid-tags-error) > + (raise (condition (&message (message "no valid tags found")) > + (&git-no-valid-tags-error)))) > + > +(define-condition-type &git-no-tags-error &error > + git-no-tags-error?) > + > +(define (git-no-tags-error) > + (raise (condition (&message (message "no tags were found")) > + (&git-no-tags-error)))) > + > + > +;;; Updater > + > +(define* (get-version-mapping tags #:key prefix suffix delim) > + (define (guess-delim) > + (let ((total (length tags)) > + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) > + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) > + (underscores (reduce + 0 (map (cut string-count <> #\_) tags))= )) > + ;; (format #t "total: ~d, dots: ~d, dashes ~d, underscores ~d~%" > + ;; total dots dashes underscores) > + (cond > + ((> dots (* total 0.35)) ".") > + ((> dashes (* total 0.8)) "-") > + ((> underscores (* total 0.8)) "_") > + (else ".")))) These numbers seem rather arbitrary, how did you arrive at them? Also, AFAICS, versions without delimiters won=E2=80=99t be matched. > + (define delim-rx (regexp-quote (or delim (guess-delim)))) > + (define suffix-rx (or suffix "")) > + > + (define prefix-rx > + (make-regexp (string-append "^" (or prefix ".*") "$"))) Why is there a $ here? > + (define tag-rx > + (make-regexp > + (string-append "([[:digit:]][^" delim-rx "[:punct:]]*" > + "(" delim-rx "[^" delim-rx "]+)+" > + ")" suffix-rx "$"))) > + > + (define (get-version tag) > + (let ((tag-match (regexp-exec tag-rx tag))) > + (and tag-match > + (regexp-exec prefix-rx (match:prefix tag-match)) > + (regexp-substitute/global > + #f delim-rx (match:substring tag-match 1) > + 'pre "." 'post)))) > + > + (define (entry + (eq? (version-compare (car a) (car b)) '<)) > + > + (let ((mapping (fold alist-cons '() (map get-version tags) tags))) > + (stable-sort! (filter car mapping) entry + > +(define* (get-latest-tag url #:key prefix suffix delim) > + "Return the latest tag available from the Git repository at URL." > + (define (pre-release? tag) > + (any (cut string-contains tag <>) > + '("alpha" "beta" "rc" "dev" "test"))) This would only match lower-case characters, but sometimes upper-case characters are used. (define (pre-release? tag) (any (lambda (rx) (regexp-exec (make-regexp rx regexp/icase) tag)) '("alpha" "beta" "rc" "dev" "test"))) =20=20=20=20=20=20=20=20=20=20=20=20=20=20 > + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) > + (ls-remote-refs url #:tags? #t))) > + (versions->tags > + (get-version-mapping (filter (negate pre-release?) tags) > + #:prefix prefix > + #:suffix suffix > + #:delim delim))) > + (cond > + ((null? tags) > + (git-no-tags-error)) > + ((null? versions->tags) > + (git-no-valid-tags-error)) > + (else > + (match (last versions->tags) > + ((version . tag) > + (values version tag))))))) > + > +(define (latest-git-tag-version package tag-prefix tag-suffix > + tag-version-delimiter) > + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, and TAG-VERSION-DELIMITER > +properties of PACKAGE, returns the latest version of PACKAGE." > + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "~a for ~a~%") > + (condition-message c) > + (package-name package)) > + #f) > + ((eq? (exception-kind c) 'git-error) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "failed to fetch Git repository for ~a~%") > + (package-name package)) > + #f)) > + (let* ((source (package-source package)) > + (url (git-reference-url (origin-uri source)))) > + ;;(format #t "~a~%" (package-name package)) > + (get-latest-tag url #:prefix tag-prefix #:suffix tag-suffix > + #:delim tag-version-delimiter)))) > + > +(define (git-package? package) > + "Whether the origin of PACKAGE is a Git repostiory." > + (match (package-source package) > + ((? origin? origin) > + (and (eq? (origin-method origin) git-fetch) > + (git-reference? (origin-uri origin)))) > + (_ #f))) > + > +(define (latest-git-release package) > + "Return the latest release of PACKAGE." > + (let* ((name (package-name package)) > + (properties (package-properties package)) > + (tag-prefix (assq-ref properties 'tag-prefix)) > + (tag-suffix (assq-ref properties 'tag-suffix)) > + (tag-version-delimiter (assq-ref properties 'tag-version-delimi= ter)) > + (old-version (package-version package)) > + (url (git-reference-url (origin-uri (package-source package)))) > + (new-version (latest-git-tag-version package > + tag-prefix > + tag-suffix > + tag-version-delimiter))) > + > + (if new-version > + (upstream-source > + (package name) > + (version new-version) > + (urls (list url))) > + ;; No new release or no tags available. > + #f))) > + > +(define %generic-git-updater > + (upstream-updater > + (name 'generic-git) > + (description "Updater for packages hosted on Git repositories") > + (pred git-package?) > + (latest latest-git-release))) > > base-commit: 522a3bf99cbc21a9093f63280b9508cd69b94ff0 > --=20 > 2.32.0 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE3udIVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x59rgP/i9zonrR51YoZ4Dw68xfhW66m2op K7Xf9B45E+pAYKxo2R31HjO2B9CNYUXb5LBLAMvkqaH0KmP7Yv3LpBUpOm6EOmMh gtgTJTBlLetvjxrQmcOHBd6n/vIFp74wlukV17vPrxtlrQVCM50IBYvPcVuq8GtU eXQZhL3HYNc8K5ddKSRGwHNogZg3casFP3O2/x9J3wImFhIqA57IHan4n+mNAOUk ddUMhiypYNTM74gYQZ4tqspJNyWD7l7YNF2CA+mpsn+Vadz+tCALffKZTO6f1mQT ibuOV75ZbtuAcEkR+43o3dtWFamaQOuUZWLMBXFYU8iy91Suyqh3I1QazoKE0gXU 3hJO2x0+TdQV/bI8U/Hoj7nimXDbicJovFH9KCzOp2fPhcFxLCLwIMjc++BlMLx8 0TfkIZfHti962MlnYFgo0I0N/F0SiI6fFQBJ7L7i0nm3/el1wB23IIbjrNCQ9841 fLNw/Vz8QsO1xBNEFhrLhoFerzHLqmy+TggKVCzdlnlwPlrK5Y9cZZ1FIJq5LKli sirHcyfi22YL7B19sTRhqZYBY1Gttucykn1ttuRZg56JhJvPoDunoluZyTtYXUQt QkSpbjD/9VkyODCKgAjvm+3+e68uNlVePPCN5W/AroVPxw8oMjCeLakUTrKavM6w vl+l3fGzkXs14GR0 =d2MH -----END PGP SIGNATURE----- --=-=-=--