From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 06 08:20:53 2021 Received: (at 50359) by debbugs.gnu.org; 6 Sep 2021 12:20:53 +0000 Received: from localhost ([127.0.0.1]:52280 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNDcK-0007kr-Kz for submit@debbugs.gnu.org; Mon, 06 Sep 2021 08:20:53 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:37942 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNDcI-0007ka-Ag for 50359@debbugs.gnu.org; Mon, 06 Sep 2021 08:20:51 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1630930837; bh=rTeAfqqkrZ8uCrLD42TmYnmw6Zfc0mWDeJW9zRplpEc=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=MYYVUOoe+DrZ96dkt00vEYMFEyxSINEtCD0m6tE9lhiRsm8VlNqbqiwK2gv9ooEVL OTFRKDlomoQ8fgVPqa4Aa/k+tTnu2tQFbO4GngHrLonWRvNWg20F1hGOi7bGdIBzS8 hTGdJsdFudoC2F/mSQqDu+WCMDT9WzKcjQD9cP/g= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: <86y28ai7ns.fsf@mgsn.dev> References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> Date: Mon, 06 Sep 2021 14:20:34 +0200 Message-ID: <87y28928vh.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Sun, Sep 05 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >>> There are about 50-60 packages like this. >>> >>> I'm not sure how much effort should be spent including them, and for >>> some of them I'm n [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Sun, Sep 05 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >>> There are about 50-60 packages like this. >>> >>> I'm not sure how much effort should be spent including them, and for >>> some of them I'm n [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Sun, Sep 05 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >>> There are about 50-60 packages like this. >>> >>> I'm not sure how much effort should be spent including them, and for >>> some of them I'm not sure what our ideal behavior *is*. Even if we >>> could reliably detect them, should "alpha" or "dev" packages be returned >>> by the updater? >> >> I don=E2=80=99t think we usually include alpha or rc releases, so updater >> probably shouldn=E2=80=99t return them either. Not sure how we would tr= y to >> detect alpha/beta/rc releases, though, besides running something like >> >> (string-match? "(alpha|beta|rc|dev)" TAG) >> >> On each tag. > > That heuristic is pretty good. (It might miss a few, but I'd rather > accidentally include some alpha/beta/rc releases than risk excluding > real ones.) > > We could then safely sort tags with just the prefix removed -- this takes= care > of "1.1f" coming after "1.1", and so on. > > Actually, it looks like there's only a few packages with a suffix; > instead, we can probably just only use a suffix if they provide > 'tag-suffix. This is all of them (and the "dev" ones probably shouldn't > be suffixes, just part of the version): Yeah, I think that sounds like a reasonable thing to do. > (commit (string-append "v" version "-stable")))) > (commit (string-append version "-stable")))) > (commit (string-append version "-Leia")))) > (commit (string-append "haddock-" version "-release")))) > (commit (string-append "v" version "-8.13")))) > (commit (string-append "v" version "-oss")))) > (commit (string-append "v" version "-stable")))) > (commit (string-append "ddskk-" version "_" code-name)))) > (commit (string-append version "-freebsdport")))) > (commit (string-append version "-dev")))) > (commit (string-append version "-release-20210531143054")))) > (commit (string-append version "-release-20210412001032")))) > (commit (string-append "v" version "-debian")))) > (commit (string-append version "dev")))) > (commit (string-append version "_Linux")) > (commit (string-append version "R")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append version "-opt")))) > (commit (string-append "1.1-" version "-RELEASE")))) > > Additionally, these are all the weird version strings I could find that > are actually used as the commit: > 1.2.2.rc2 > 12-068oasis4 > 1.2-2 > 0.F-2 > 0.2.0-alpha > 5.1.0-b2 > 1.5-11 > 1.9.14-20210407 > 0.9.3b > 2.8-fix-2 > 2020-05-19 > 10-11.0.0 > 1.0.12-2 > 0.9.3+16.04.20160218-0ubuntu1 > 2.12.c > 1.1+11 > 5.1+4.06.0 > 4.2-411 > 1.4.0rc1-450-g2725ef99d > 2.0.0-alpha14 > 60.2.3-2 > 1.21c > 1.0.0rc4 > 2.1.0b1 > 1.02r6 > 0.32-14-gcdfe14e > 3.0.0a3 > 2.00a2.3 > 1.0beta.18 > 2.1b > 2.7.8a > 2.7.3a > 1.1.alpha19 > 1.0.2-rc4 > 0.5.3+git20200502 > 1.0.3-rc3 > 4.0.0.dev8 > 3.0.0beta1-24-g024cc9fa2 > 0.16-2-ge145396 > 2.0b6 > 2.0M10 > 1.0.7+0 > 1.16.0+5 > 0.4.0+1 > 2.2.10+0 > 4.3.1+2 > 2.13.93+0 > 2.10.4+0 > 1.0.5+5 > 0.21.0+0 > 3.3.4+0 > 2.68.1+0 > 0.10.1+1 > 6.9.10-12+3 > 2.0.1+2 > 3.100.0+1 > 0.14.0+2 > 0.1.6+2 > 3.3.0+0 > 1.8.7+0 > 1.3.0+2 > 1.42.0+0 > 1.16.1+0 > 2.35.0+0 > 1.6.37+5 > 4.1.0+1 > 2.36.0+0 > 1.3.6+4 > 2.10.1+0 > 2.26.0+0 > 1.3.4+0 > 1.1.1+2 > 1.3.1+1 > 8.44.0+0 > 0.40.1+0 > 5.15.2+0 > 1.17.0+3 > 1.18.0+3 > 2020.7.14+0 > 3.0.0+1 > 0.9.1+3 > 2.9.12+0 > 0.1.0+2 > 1.6.9+2 > 1.0.9+3 > 1.13.0+2 > 1.2.0+3 > 1.1.3+3 > 1.3.4+2 > 5.0.3+3 > 1.7.10+3 > 1.1.4+3 > 1.1.0+3 > 1.5.2+3 > 0.9.10+3 > 0.4.0+1 > 0.4.0+1 > 0.4.0+1 > 0.3.9+1 > 0.4.1+1 > 1.4.2+3 > 2.27.0+3 > 1.4.0+2 > 1.1.34+0 > 1.2.12+1 > 1.5.0+0 > 2021-06-07 > 1.2.2-5-g20dc8ed > 2.1-20201229 > 3.0-rc1 > 1.32.0-0 > 20200701.154658.b0d6223 > 3.9-0 > R63-10032.B > 0.58.2.a > 0.0.0+git20200527 > 3.028R > 3.001R > 2.57b > 1.0.0-20201130134442-10cb98267c6c > 0.0.0-20161123171359-e6a2ba005892 > 0.0.0-20210615171337-6886f2dfbf5b > 2020-11-10 > 5.2.0-alpha > 2021-01-01 > 2021-02-28 > 2020-11-01 > 4.4-git.1 > 1.217-2 > 3.3.06-1 > 0.2.0-alpha-199-g3e7a475 > 1.0.0-beta.0 > 1.20190621-4 > 1.9.0-147-g61edec1ef > 0.0.9.4f > > Yep, it looks like the above would work for the majority of these. > That's probably Good Enough^tm. Any trick you used to find all of there weird version numbers? :-) >>>> + (define no-delims >>>> + (if (string-match delim-regexp no-suffix) >>>> + (string-split* no-suffix delim*) >>>> + (git-tag-error 'tag-version-delimiter))) >>> >>> This throws an error if the version doesn't have any delimiter. >> >> Setting the =E2=80=98tag-version-delimiter=E2=80=99 prefix to an empty s= tring would >> solve this, right? Or, maybe we should just get rid of the delimiter >> thing since only a few packages use a different delimiter. > > IMO, just get rid of the delimiter. If we wanted to be *that* flexible, > we could make it so they provide a tag->version proc instead of (prefix, > suffix, delimiter). a =E2=80=98tag->version=E2=80=99 procedure would probably make things a bit= too complicated for the people writing package definitions. For example, having a delimiter would make it easy to match a tag like =E2=80=9C2021-01-01-release=E2=80=9D Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:punct:= ]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) $28 =3D "2021" Delimiter is =E2=80=9C-=E2=80=9D scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct:]]= +(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) $29 =3D "2021-01-01-release" And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match just= the version part. On Sun, Sep 05 2021, Sarah Morgensen wrote (again): >>> And this lets us just do something like (untested): >>> >>> (define* (get-version tag #:key prefix suffix delim) >>> (define delim-rx (regexp-quote (or delim "."))) >>> (define prefix-rx (or prefix "[^[:digit:]]*")) >>> (define suffix-rx (or suffix ".*")) >>> (define version-char-rx >>> (string-append "[^" delim-rx "[:punct:]]")) >>> >>> (define tag-rx >>> (string-append "^" prefix "(" version-char-rx "+(" >>> delim-rx version-char-rx ")*)" suffix-rx "$")) >> >> This wouldn=E2=80=99t match anything if the version is just a plain numb= er, >> e.g., 1 or 09. > > It does, Oh, I missed the extra pair of parens, sorry about that. > but I had many errors in the definition. Again, apologies. I > shouldn't send emails that late, haha. This method should read: > > --8<---------------cut here---------------start------------->8--- > (define* (get-version tag #:key prefix suffix delim) > (define delim-rx (regexp-quote (or delim "."))) > (define prefix-rx (or prefix "[^[:digit:]]*")) > (define suffix-rx (or suffix ".*")) > (define version-char-rx > (string-append "[^" delim-rx "[:punct:]]")) > > (define tag-rx > (string-append "^" prefix-rx "(" version-char-rx "+(" > delim-rx version-char-rx "+)*)" suffix-rx "$")) > (and=3D> (string-match tag-rx tag) > (cut match:substring <> 1))) > --8<---------------cut here---------------end--------------->8--- > >> >> With this, something like =E2=80=9C1.4.0rc1-450-g2725ef99d=E2=80=9D will= result in >> =E2=80=9C1.4.0=E2=80=9D being returned, which is incorrect. Changing (c= ut >> match:substring <> 1) to just =E2=80=98match:substring=E2=80=99 would so= lve the issue, >> but then pre-release tags, which we usually don=E2=80=99t want, would a= lso get >> matched. Not sure what the best option would be in this case. >> > > With the fixed method above: > > scheme@(emacs-guix)> (get-version "8") > $16 =3D "8" > scheme@(emacs-guix)> (get-version "1.4.0rc1-450-g2725ef99d") > $17 =3D "1.4.0rc1" > > But, we still get: > > scheme@(emacs-guix)> (get-version "1.4.0-rc1") > $18 =3D "1.4.0" > > which leads us to what you talked about in your other message. Hmm, maybe we could check that if (string-append "1.4.0" suffix) is a suffix on of the tag, or is this too much overhead? This would only work if we set the suffix to the empty string by default. Since (string-match "1.4.0$" "1.4.0-rc1") returns #f, this tag would get filtered out. But then a suffix would have to be specified to match "1.4.0rc1-450-g2725ef99d". Not sure what the best option is here. >>> +(define* (ls-remote-refs url #:key tags?) >>> + "Return the list of references advertised at Git repository URL. If= TAGS? >>> +is true, limit to only refs/tags." >>> + (define (ref? ref) >>> + ;; Like `git ls-remote --refs', only show actual references. >>> + (and (string-prefix? "refs/" ref) >>> + (not (string-suffix? "^{}" ref)))) >>> + >>> + (define (tag? ref) >>> + (string-prefix? "refs/tags/" ref)) >>> + >>> + (define (include? ref) >>> + (and ref? > > This should be: > (and (ref? ref) Ah, problem solved! :-) >>> + (or (not tags?) (tag? ref)))) >>> + >>> + (with-libgit2 >>> + (with-temporary-directory >>> + (lambda (cache-directory) >>> + (let* ((repository (repository-init cache-directory)) >>> + ;; Create an in-memory remote so we don't touch disk. >>> + (remote (remote-create-anonymous repository url))) >>> + (remote-connect remote) >>> + (remote-disconnect remote) >>> + (repository-close! repository) >>> + >>> + (filter include? (map remote-head-name (remote-ls remote))))))= )) >>> >> >> For some reason it seems to include refs that do and don=E2=80=99t end w= ith >> =E2=80=9C^{}=E2=80=9D > > Sorry, another typo I missed. See above. No worries, thanks for taking the time to review this! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE2B5IVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5BgIQAIZHtTauloSjYVC1T6cs8epK0r8B gUw3Dk3ZdBbjd9TfpfjsUnaGN2o6KwBKTnpj5atLPiM2Vo5an8BLmSOTlJQ9+O8p DBiKYY0kKdknJYhboSIKj/ejCRIkdLQy6Wj6ucWmYmxipqk+muBO0c5qoMsIYHFp iLUH0rrT7R0PMkBOSGrGgLi5Y4RRBlM9O4QqMp4RZ5Vw+brn6o3Z/+ja9/wlPSl9 md9W0g3D8d7eTmu/ATW71GDaI3UndGOUeCW4X3uTV0SqPUkyFuLKFFsqiQMxZHe1 iQ+uu1vj6shpxa+tXvpzqT3CdPDRzU8UyCtPNCUc8TaoLLj+SRlsdGv/xe1DSRYa lZhX1kxkUdJxpAjKuyU4i9N2b0Wftbmsfg4PATQJgP/XbY9RiSp6pXNY0uKNIhLF CBEoC5cq3MDjmLMswaipdlcVG4a5gxzhGI37aMBpE/of9hCDlY+rxth02m4nr4Rf 6SMeXcwYlcLNTU8FRw90d6aE+8NJxH4XR5VOW/CvZw6aArzYE3lwUvyYmFD78xfu 9LfehCH0DR3q98GCabyswVxpMrNdv8Z+wVNlpF+kDnCDzcD9vIhjKU6RTkzjp1UN zc5hQv4VjEqTs2hVSjf04X9L0s6gO2TRjpFJtquPq+GPqnozLJBuzIt+v0MRj/AH DF2EcH0wmD9yKpdI =l3fr -----END PGP SIGNATURE----- --=-=-=--