Code stored with Subversion (SVN) cannot be retrieved from SWH

  • Done
  • quality assurance status badge
Details
4 participants
  • Ludovic Courtès
  • Ludovic Courtès
  • Timothy Sample
  • zimoun
Owner
unassigned
Submitted by
zimoun
Severity
important
Z
Z
zimoun wrote on 16 Sep 2020 10:14
[PATCH] Fixes init of #42162: gforge.inria.fr down Dec. 2020
(address . guix-patches@gnu.org)
20200916081411.5801-1-zimon.toutoune@gmail.com
Dear,

The first message in [1] explains the concrete issue. To avoid to pollute the
already long thread discussing long term Tarball Archiving (which will not be
ready before the down), here is sent patches fixing the concrete issue.

Aside, note that the tarballs should be now ingested by Software Heritage via:


but the issue is to reach them; well see [2].


From [1], the non-archived packages (yet) are:

Toggle snippet (9 lines)
scheme@(guile-user)> ,pp (lset-difference eq? $7 $8)
$11 = (
#<package r-spams@2.6-2017-03-22 gnu/packages/statistics.scm:3931 7f632401a640>
#<package mpfi@1.5.4 gnu/packages/multiprecision.scm:158 7f632ee3adc0>
#<package gf2x@1.2 gnu/packages/algebra.scm:103 7f6323ea1280>
#<package gmp-ecm@7.0.4 gnu/packages/algebra.scm:658 7f6323eb4960>
#<package cmh@1.0 gnu/packages/algebra.scm:322 7f6323eb4dc0>)

However, some have migrated to gitlab:

- r-spams
- gf2x
- cmh

Note that repackage 'r-spams' from the Git checkout needs some love and is not
straightforward. (Not tried yet the 2 others).


The aim of switching the 2 packages from 'url-fetch' to 'svn-fetch' is
twofolds, test case for improving:

- "guix lint -c archival" (support 'svn' and 'hg')
- the fallback to SWH (for non-git VCS source)

WDYT?



All the best,
simon

PS:
I am not sure how to deal with <control@debbugs.gnu.org> to "clone" (split)
the bug #42162. That's why this one. :-)


zimoun (2):
gnu: mpfi: Replace 'url-fetch' by 'svn-fetch'.
gnu: gmp-ecm: Replace 'url-fetch' by 'svn-fetch'.

gnu/packages/algebra.scm | 28 +++++++++++++++++++++-------
gnu/packages/multiprecision.scm | 13 ++++++++-----
2 files changed, 29 insertions(+), 12 deletions(-)


base-commit: f5163902d7c3cfed5a97033f38e92d7158b19fa8
--
2.28.0
Z
Z
zimoun wrote on 16 Sep 2020 10:16
[PATCH 2/2] gnu: gmp-ecm: Replace 'url-fetch' by 'svn-fetch'.
(address . 43442@debbugs.gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
20200916081642.6716-2-zimon.toutoune@gmail.com

* gnu/packages/algebra.scm (gmp-ecm)[source]: Replace 'url-fetch' by
'svn-fetch'.
[natice-inputs]: Add requirements.
[arguments]: Adjust 'check phase to substitute correct path.
---
gnu/packages/algebra.scm | 28 +++++++++++++++++++++-------
1 file changed, 21 insertions(+), 7 deletions(-)

Toggle diff (65 lines)
diff --git a/gnu/packages/algebra.scm b/gnu/packages/algebra.scm
index 318d653618..1cf293f17a 100644
--- a/gnu/packages/algebra.scm
+++ b/gnu/packages/algebra.scm
@@ -13,6 +13,7 @@
;;; Copyright © 2020 Jakub K?dzio?ka <kuba@kadziolka.net>
;;; Copyright © 2020 Vincent Legoll <vincent.legoll@gmail.com>
;;; Copyright © 2020 Vinicius Monego <monego@posteo.net>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -66,6 +67,7 @@
#:use-module (guix download)
#:use-module (guix git-download)
#:use-module (guix hg-download)
+ #:use-module (guix svn-download)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
#:use-module (guix utils))
@@ -670,15 +672,20 @@ geometry and singularity theory.")
(define-public gmp-ecm
(package
(name "gmp-ecm")
- (version "7.0.4")
+ (version "7.0.4-3088")
(source (origin
- (method url-fetch)
- ;; Use the ‘Latest version’ link for a stable URI across releases.
- (uri (string-append "https://gforge.inria.fr/frs/download.php/"
- "latestfile/160/ecm-" version ".tar.gz"))
+ (method svn-fetch)
+ (uri
+ (svn-reference
+ (url "https://scm.gforge.inria.fr/anonscm/svn/ecm/trunk")
+ (revision 3088)))
(sha256 (base32
- "0hxs24c2m3mh0nq1zz63z3sb7dhy1rilg2s1igwwcb26x3pb7xqc"))))
+ "0g1jqgp9baqz4x9mzksch69i4kp3207d3wn56g9vnsmanrrb4qvb"))))
(build-system gnu-build-system)
+ (native-inputs
+ `(("autoconf" ,autoconf)
+ ("automake" ,automake)
+ ("libtool" ,libtool)))
(inputs
`(("gmp" ,gmp)))
(arguments
@@ -686,7 +693,14 @@ geometry and singularity theory.")
;; Disable specific assembly routines, which depend
;; on the subarchitecture of the build machine,
;; and use gmp instead.
- "--disable-asm-redc")))
+ "--disable-asm-redc")
+ #:phases (modify-phases %standard-phases
+ (add-before 'check 'replace-/bin/rm
+ (lambda _
+ (substitute*
+ (list "test.ecm")
+ (("/bin/rm") (which "rm")))
+ #t)))))
(synopsis "Integer factorization library using the elliptic curve method")
(description
"GMP-ECM factors integers using the elliptic curve method (ECM) as well
--
2.28.0
Z
Z
zimoun wrote on 16 Sep 2020 10:16
[PATCH 1/2] gnu: mpfi: Replace 'url-fetch' by 'svn-fetch'.
(address . 43442@debbugs.gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
20200916081642.6716-1-zimon.toutoune@gmail.com

* gnu/packages/multiprecision.scm (mpfi)[source]: Replace 'url-fetch' by
'svn-fetch'.
---
gnu/packages/multiprecision.scm | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)

Toggle diff (43 lines)
diff --git a/gnu/packages/multiprecision.scm b/gnu/packages/multiprecision.scm
index b3a5ec5894..f1b152d9aa 100644
--- a/gnu/packages/multiprecision.scm
+++ b/gnu/packages/multiprecision.scm
@@ -7,6 +7,7 @@
;;; Copyright © 2018, 2019 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2018 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2018, 2019 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -32,6 +33,7 @@
#:use-module (gnu packages texinfo)
#:use-module (guix packages)
#:use-module (guix download)
+ #:use-module (guix svn-download)
#:use-module (guix utils)
#:use-module (guix build-system gnu))
@@ -157,14 +159,15 @@ precision and correctly rounds the results.")
(define-public mpfi
(package
(name "mpfi")
- (version "1.5.4")
+ (version "1.5.4-688")
(source
(origin
- (method url-fetch)
- (uri (string-append "https://gforge.inria.fr/frs/download.php"
- "/latestfile/181/mpfi-" version ".tgz"))
+ (method svn-fetch)
+ (uri (svn-reference
+ (url "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
+ (revision 688)))
(sha256
- (base32 "0mismr1ll3wp788dq2n22s5irm0dziy75byyfdwz22kjbmckhf9v"))))
+ (base32 "13pqkdvi8maj0mm76zh8p7jcc4c4569zh4kq336shh9cxlrh8qqx"))))
(build-system gnu-build-system)
(arguments
`(#:tests? #f ;tests are broken in this release
--
2.28.0
L
L
Ludovic Courtès wrote on 17 Sep 2020 10:14
Re: [PATCH] Fixes init of #42162: gforge.inria.fr down Dec. 2020
(name . zimoun)(address . zimon.toutoune@gmail.com)
87imcczuol.fsf@inria.fr
Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (6 lines)
> The aim of switching the 2 packages from 'url-fetch' to 'svn-fetch' is
> twofolds, test case for improving:
>
> - "guix lint -c archival" (support 'svn' and 'hg')
> - the fallback to SWH (for non-git VCS source)

Woow, it’s ambitious given that SWH/SVN integration is currently lacking
on our side. :-)

It’s a good idea to switch to gitlab.inria.fr for those that have
migrated, though.

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 21 Sep 2020 23:19
Re: [bug#43442] [PATCH 1/2] gnu: mpfi: Replace 'url-fetch' by 'svn-fetch'.
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 43442@debbugs.gnu.org)
87o8lyj090.fsf@gnu.org
Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (5 lines)
>
> * gnu/packages/multiprecision.scm (mpfi)[source]: Replace 'url-fetch' by
> 'svn-fetch'.

[...]

Toggle quote (5 lines)
> + (method svn-fetch)
> + (uri (svn-reference
> + (url "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
> + (revision 688)))

Does this have any chance of working with SWH? Does their HTTP API
support looking up svn revision for a given origin URL?

Also:

scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
$6 = #f

So the effect today would be the opposite of what we’re attempting: when
GForge goes down, we’d lose it completely.

Thoughts?

Ludo’.
Z
Z
zimoun wrote on 21 Sep 2020 23:51
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43442@debbugs.gnu.org)
86zh5iol1n.fsf@gmail.com
Hi,

On Mon, 21 Sep 2020 at 23:19, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (8 lines)
>> + (method svn-fetch)
>> + (uri (svn-reference
>> + (url "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
>> + (revision 688)))
>
> Does this have any chance of working with SWH? Does their HTTP API
> support looking up svn revision for a given origin URL?

Well, I have not yet did my homework on the topic, neither asked on
#swh-devel.

Probably not!

In any case, we should find a way because there are a lot of Subversion
sources.


Toggle quote (5 lines)
> Also:
>
> scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
> $6 = #f

Since “guix lint -c archival” I should have forgotten to submit the
request via their Web interface. Now, it is scheduled:



Toggle quote (3 lines)
> So the effect today would be the opposite of what we’re attempting: when
> GForge goes down, we’d lose it completely.

I agree.


Well, the shutdown of the INRIA’s GForge is a concrete deadline to add
the support of Subversion (and Mercurial) sources to:

- fallback to SWH
- guix lint -c archival

I am working on it. :-)


All the best,
simon
L
L
Ludovic Courtès wrote on 23 Sep 2020 18:21
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 43442@debbugs.gnu.org)
87h7rocvkt.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (18 lines)
> On Mon, 21 Sep 2020 at 23:19, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>> + (method svn-fetch)
>>> + (uri (svn-reference
>>> + (url "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
>>> + (revision 688)))
>>
>> Does this have any chance of working with SWH? Does their HTTP API
>> support looking up svn revision for a given origin URL?
>
> Well, I have not yet did my homework on the topic, neither asked on
> #swh-devel.
>
> Probably not!
>
> In any case, we should find a way because there are a lot of Subversion
> sources.

Yes, we should implement it. But until that is done, I’m not sure it’s
worth replacing tarballs with svn references. WDYT?

Toggle quote (10 lines)
>> Also:
>>
>> scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
>> $6 = #f
>
> Since “guix lint -c archival” I should have forgotten to submit the
> request via their Web interface. Now, it is scheduled:
>
> <https://archive.softwareheritage.org/save/#requests>

‘lookup-origin’ still returns #f though.

Toggle quote (14 lines)
>> So the effect today would be the opposite of what we’re attempting: when
>> GForge goes down, we’d lose it completely.
>
> I agree.
>
>
> Well, the shutdown of the INRIA’s GForge is a concrete deadline to add
> the support of Subversion (and Mercurial) sources to:
>
> - fallback to SWH
> - guix lint -c archival
>
> I am working on it. :-)

Awesome.

(Note that it seems possible that the shutdown of gforge.inria.fr will
be delayed a bit or at least be made less brutally that initially
announced.)

Thanks,
Ludo’.
Z
Z
zimoun wrote on 23 Sep 2020 19:07
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43442@debbugs.gnu.org)
CAJ3okZ175Tjr9ZjfatMDaogNT1guMZBfF+U1eGssUj+2_xtvww@mail.gmail.com
On Wed, 23 Sep 2020 at 18:22, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (23 lines)
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > On Mon, 21 Sep 2020 at 23:19, Ludovic Courtès <ludo@gnu.org> wrote:
> >
> >>> + (method svn-fetch)
> >>> + (uri (svn-reference
> >>> + (url "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
> >>> + (revision 688)))
> >>
> >> Does this have any chance of working with SWH? Does their HTTP API
> >> support looking up svn revision for a given origin URL?
> >
> > Well, I have not yet did my homework on the topic, neither asked on
> > #swh-devel.
> >
> > Probably not!
> >
> > In any case, we should find a way because there are a lot of Subversion
> > sources.
>
> Yes, we should implement it. But until that is done, I’m not sure it’s
> worth replacing tarballs with svn references. WDYT?

I do not have a strong opinion.

Toggle quote (12 lines)
> >> Also:
> >>
> >> scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/svn/mpfi/trunk/mpfi")
> >> $6 = #f
> >
> > Since “guix lint -c archival” I should have forgotten to submit the
> > request via their Web interface. Now, it is scheduled:
> >
> > <https://archive.softwareheritage.org/save/#requests>
>
> ‘lookup-origin’ still returns #f though.

Weird because SWH says it is in. Requests at 21/09/2020, 23:47:09 then
click. :-)

- scheduled: 21/09/2020, 23:47:21
- start: 21/09/2020, 23:47:24
- end: 21/09/2020, 23:53:26
- log: [2020-09-21 21:53:26,098: INFO/ForkPoolWorker-1] Task
swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[b13942f6-b415-4831-b25f-e68bd4515cf2]
succeeded in 361.4525023885071s: {'status': 'eventful'}

Well, the issue is:



as you can see here:



Toggle quote (4 lines)
> (Note that it seems possible that the shutdown of gforge.inria.fr will
> be delayed a bit or at least be made less brutally that initially
> announced.)

Good to know. :-)

All the best,
simon
L
L
Ludovic Courtès wrote on 25 Sep 2020 10:56
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 43442@debbugs.gnu.org)
877dsi44lu.fsf@gnu.org
Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (4 lines)
> as you can see here:
>
> <https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://scm.gforge.inria.fr/anonscm/svn/mpfi>

Oh indeed:

Toggle snippet (9 lines)
scheme@(guile-user)> ,use(guix swh)
scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/svn/mpfi")
$14 = #<<origin> visits-url: "https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visits/" type: #f url: "https://scm.gforge.inria.fr/anonscm/svn/mpfi">
scheme@(guile-user)> (origin-visits $14)
$15 = (#<<visit> date: #<date nanosecond: 902765 second: 25 minute: 53 hour: 21 day: 21 month: 9 year: 2020 zone-offset: 0> origin: "https://scm.gforge.inria.fr/anonscm/svn/mpfi" url: "https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visit/1/" snapshot-url: "https://archive.softwareheritage.org/api/1/snapshot/e7fdd4dc6230f710dbc55c1b308804fa1b5f51f0/" status: full number: 1>)
scheme@(guile-user)> (visit-snapshot (car $15))
$16 = #<<snapshot> branches: (#<<branch> name: "HEAD" target-type: revision target-url: "https://archive.softwareheritage.org/api/1/revision/f7b445a6bdc38bf075f29265120ca49824f698ea/">)>

Note that the original svn revision number is discarded, AFAICS.

Ludo’.
Z
Z
zimoun wrote on 1 Oct 2020 22:26
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43442@debbugs.gnu.org)
CAJ3okZ1i4E0iKM1nYCtQ2N+hD2tE7=hcOAUihuay-x0bZQmLeQ@mail.gmail.com
Hi Ludo,

On Fri, 25 Sep 2020 at 10:56, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)
> Note that the original svn revision number is discarded, AFAICS.

No, it is not... but to find it one needs a bit of guidance by SWH folks. :-)

1. With the URL, fetch:

2. From #1 one gets 'snapshot' and fetch:

3. From #3, one gets the 'target' and fetch:

And #3 lists all the revisions in SWH.

Well, this should be enough for the saving request. WDYT?


However, it is not clear how to fetch cooking from the vault. To be
continued...

Cheers,
simon
Z
Z
zimoun wrote on 1 Oct 2020 23:01
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43442@debbugs.gnu.org)
CAJ3okZ1+SZbkqRkAe1Q6uAHoHrAugp99aouRKYFkc=98z1h_-Q@mail.gmail.com
On Thu, 1 Oct 2020 at 22:26, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (17 lines)
> 1. With the URL, fetch:
> https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visit/latest/
>
> 2. From #1 one gets 'snapshot' and fetch:
> https://archive.softwareheritage.org/api/1/snapshot/e7fdd4dc6230f710dbc55c1b308804fa1b5f51f0/
>
> 3. From #3, one gets the 'target' and fetch:
> https://archive.softwareheritage.org/api/1/revision/f7b445a6bdc38bf075f29265120ca49824f698ea/log/
>
> And #3 lists all the revisions in SWH.
>
> Well, this should be enough for the saving request. WDYT?
>

> However, it is not clear how to fetch cooking from the vault. To be
> continued...

From #3, using 'directory' (4193e93b4aea0abeaff715b1a1274c4d7eb4a27d),
it seems possible to cook from the vault and get the tarball (I did
with the web interface). And the integrity field (guix hash -r) is
the same as the one I put in the patch submission.
L
L
Ludovic Courtès wrote on 3 Oct 2020 10:59
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 43442@debbugs.gnu.org)
87y2knhei3.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (17 lines)
> On Fri, 25 Sep 2020 at 10:56, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Note that the original svn revision number is discarded, AFAICS.
>
> No, it is not... but to find it one needs a bit of guidance by SWH folks. :-)
>
> 1. With the URL, fetch:
> https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visit/latest/
>
> 2. From #1 one gets 'snapshot' and fetch:
> https://archive.softwareheritage.org/api/1/snapshot/e7fdd4dc6230f710dbc55c1b308804fa1b5f51f0/
>
> 3. From #3, one gets the 'target' and fetch:
> https://archive.softwareheritage.org/api/1/revision/f7b445a6bdc38bf075f29265120ca49824f698ea/log/
>
> And #3 lists all the revisions in SWH.

Ah yes, under “extra_headers” there’s the SVN revision number. (guix
swh) doesn’t expose “extra_headers” yet, but once it does, we can walk
snapshots until we find the SVN revision we’re looking for.

Toggle snippet (8 lines)
scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/svn/mpfi/")
$3 = #<<origin> visits-url: "https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visits/" type: #f url: "https://scm.gforge.inria.fr/anonscm/svn/mpfi">
scheme@(guile-user)> (origin-visits $3)
$4 = (#<<visit> date: #<date nanosecond: 902765 second: 25 minute: 53 hour: 21 day: 21 month: 9 year: 2020 zone-offset: 0> origin: "https://scm.gforge.inria.fr/anonscm/svn/mpfi" url: "https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visit/1/" snapshot-url: "https://archive.softwareheritage.org/api/1/snapshot/e7fdd4dc6230f710dbc55c1b308804fa1b5f51f0/" status: full number: 1>)
scheme@(guile-user)> (visit-snapshot (car $4))
$5 = #<<snapshot> branches: (#<<branch> name: "HEAD" target-type: revision target-url: "https://archive.softwareheritage.org/api/1/revision/f7b445a6bdc38bf075f29265120ca49824f698ea/">)>

So the next step is to augment (guix swh) with a
‘lookup-subversion-revision’ procedure.

Thanks for investigating!

Ludo’.
L
L
Ludovic Courtès wrote on 20 Mar 2023 14:47
control message for bug #43442
(address . control@debbugs.gnu.org)
87zg87mt5m.fsf@gnu.org
severity 43442 important
quit
L
L
Ludovic Courtès wrote on 20 Mar 2023 15:09
Re: bug#43442: [PATCH] Fixes init of #42162: gforge.inria.fr down Dec. 2020
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 43442@debbugs.gnu.org)
87jzzbms54.fsf_-_@gnu.org
Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (14 lines)
> Ah yes, under “extra_headers” there’s the SVN revision number. (guix
> swh) doesn’t expose “extra_headers” yet, but once it does, we can walk
> snapshots until we find the SVN revision we’re looking for.
>
> scheme@(guile-user)> (lookup-origin "https://scm.gforge.inria.fr/anonscm/svn/mpfi/")
> $3 = #<<origin> visits-url: "https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visits/" type: #f url: "https://scm.gforge.inria.fr/anonscm/svn/mpfi">
> scheme@(guile-user)> (origin-visits $3)
> $4 = (#<<visit> date: #<date nanosecond: 902765 second: 25 minute: 53 hour: 21 day: 21 month: 9 year: 2020 zone-offset: 0> origin: "https://scm.gforge.inria.fr/anonscm/svn/mpfi" url: "https://archive.softwareheritage.org/api/1/origin/https://scm.gforge.inria.fr/anonscm/svn/mpfi/visit/1/" snapshot-url: "https://archive.softwareheritage.org/api/1/snapshot/e7fdd4dc6230f710dbc55c1b308804fa1b5f51f0/" status: full number: 1>)
> scheme@(guile-user)> (visit-snapshot (car $4))
> $5 = #<<snapshot> branches: (#<<branch> name: "HEAD" target-type: revision target-url: "https://archive.softwareheritage.org/api/1/revision/f7b445a6bdc38bf075f29265120ca49824f698ea/">)>
>
> So the next step is to augment (guix swh) with a
> ‘lookup-subversion-revision’ procedure.

The attached patch does that:

Toggle snippet (6 lines)
scheme@(guile-user)> (lookup-subversion-revision "https://scm.gforge.inria.fr/anonscm/svn/mpfi" 680)
$12 = #<<revision> id: "72102de7605a2459ebcb016338ebbf1a997e8c8d" date: #<date nanosecond: 938388 second: 35 minute: 32 hour: 11 day: 6 month: 9 year: 2018 zone-offset: 0> directory: "5c89c025a4cd9d16befdfec12dfc23f7318d0d5b" directory-url: "https://archive.softwareheritage.org/api/1/directory/5c89c025a4cd9d16befdfec12dfc23f7318d0d5b/" parents-ids: ("16da41f1848d77a93aec565320b72b460c429b61") extra-headers: (("svn_repo_uuid" . "e2f78e0c-bb60-4709-9413-9660a31d4696") ("svn_revision" . "680"))>
scheme@(guile-user)> (lookup-subversion-revision "https://scm.gforge.inria.fr/anonscm/svn/mpfi" 666)
$13 = #<<revision> id: "148eb1e7206b111af4075c73c656e54c9efed6cb" date: #<date nanosecond: 654167 second: 2 minute: 52 hour: 12 day: 2 month: 8 year: 2018 zone-offset: 0> directory: "ed7b0bd7019fb85cd86d948a97c23b9d43aa8728" directory-url: "https://archive.softwareheritage.org/api/1/directory/ed7b0bd7019fb85cd86d948a97c23b9d43aa8728/" parents-ids: ("0ba2aa7e0d3fc0a1eb3ba72b32094515415ae47a") extra-headers: (("svn_repo_uuid" . "e2f78e0c-bb60-4709-9413-9660a31d4696") ("svn_revision" . "666"))>

The implementation is pretty bad though, because it walks the revision
history until it finds the right revision number—so you’re likely to
reach the bandwidth rate limit before you’ve found the revision you’re
looking for.

More importantly, most svn origins cannot be found, or at least not
by passing the URL as-is:


This whole hack looks like a dead end.

It would be ideal if SWH would compute nar hashes, as you proposed:


As a stopgap, I wonder if we could use “double hashing” on our side, but
only for svn: we’d store both the nar sha256 as we currently do, plus
the swhid. It still seems to me that it’d be hard to scale and to
maintain that over time, even if it’s limited to svn. Plus, there’d
still be the problem of ‘svn-multi-fetch’, which is what most TeX Live
packages use.

Thoughts?

Ludo’.
Toggle diff (104 lines)
diff --git a/guix/swh.scm b/guix/swh.scm
index c7c1c873a2..a65635b1db 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2018, 2019, 2020, 2021, 2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Jakub K?dzio?ka <kuba@kadziolka.net>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
@@ -75,8 +75,10 @@ (define-module (guix swh)
revision-id
revision-date
revision-directory
+ revision-parents
lookup-revision
lookup-origin-revision
+ lookup-subversion-revision
content?
content-checksums
@@ -207,6 +209,14 @@ (define string*
((? null?) #f) ;Guile-JSON 3.x
('null #f))) ;Guile-JSON 4.x
+(define pair-vector->alist
+ (match-lambda
+ ('null '())
+ ((= vector->list lst)
+ (map (match-lambda
+ (#(key value) (cons key value)))
+ lst))))
+
(define %allow-request?
;; Takes a URL and method (e.g., the 'http-get' procedure) and returns true
;; to keep going. This can be used to disallow requests when
@@ -346,7 +356,14 @@ (define-json-mapping <revision> make-revision revision?
(id revision-id)
(date revision-date "date" (maybe-null string->date*))
(directory revision-directory)
- (directory-url revision-directory-url "directory_url"))
+ (directory-url revision-directory-url "directory_url")
+ (parents-ids revision-parent-ids "parents"
+ (lambda (vector)
+ (map (lambda (alist)
+ (assoc-ref alist "id"))
+ (vector->list vector))))
+ (extra-headers revision-extra-headers ;alist--e.g., with "svn_revision"
+ "extra_headers" pair-vector->alist))
;; <https://archive.softwareheritage.org/api/1/content/>
(define-json-mapping <content> make-content content?
@@ -524,6 +541,50 @@ (define (lookup-origin-revision url tag)
(()
#f)))))
+(define (revision-parents revision)
+ "Return the parent revision(s) of REVISION."
+ (filter-map lookup-revision (revision-parent-ids revision)))
+
+(define (lookup-subversion-revision-in-history revision revision-number)
+ "Look for Subversion REVISION-NUMBER starting from REVISION and going back
+in history."
+ (let loop ((revision revision))
+ (let ((number (and=> (assoc-ref (revision-extra-headers revision)
+ "svn_revision")
+ string->number)))
+ (and number
+ (cond ((= number revision-number)
+ ;; Found it!
+ revision)
+ ((< number revision-number)
+ ;; REVISION is ancestor of REVISION-NUMBER, so stop here.
+ #f)
+ (else
+ ;; Check the parent(s) of REVISION.
+ (any loop (revision-parents revision))))))))
+
+(define (lookup-subversion-revision url revision-number)
+ "Return either #f or the revision of the Subversion repository once
+available at URL with the given REVISION-NUMBER."
+ (match (lookup-origin url)
+ (#f #f)
+ (origin
+ (match (filter (lambda (visit)
+ ;; Return #f if (visit-snapshot VISIT) would return #f.
+ (and (visit-snapshot-url visit)
+ (eq? 'full (visit-status visit))))
+ (origin-visits origin))
+ (()
+ #f)
+ ((visit . _)
+ (any (lambda (branch)
+ (match (branch-target branch)
+ ((? revision? revision)
+ (lookup-subversion-revision-in-history revision
+ revision-number))
+ (_ #f)))
+ (snapshot-branches (visit-snapshot visit))))))))
+
(define (release-target release)
"Return the revision that is the target of RELEASE."
(match (release-target-type release)
L
L
Ludovic Courtès wrote on 20 Mar 2023 15:10
control message for bug #43442
(address . control@debbugs.gnu.org)
87ilevms28.fsf@gnu.org
retitle 43442 Code stored with Subversion (SVN) cannot be retrieved from SWH
quit
T
T
Timothy Sample wrote on 22 Mar 2023 23:42
Re: [bug#43442] [PATCH] Fixes init of #42162: gforge.inria.fr down Dec. 2020
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
87jzz8xva8.fsf@ngyro.com
Hello,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

Toggle quote (15 lines)
> This whole hack looks like a dead end.
>
> It would be ideal if SWH would compute nar hashes, as you proposed:
>
> https://gitlab.softwareheritage.org/swh/meta/-/issues/4538
>
> As a stopgap, I wonder if we could use “double hashing” on our side, but
> only for svn: we’d store both the nar sha256 as we currently do, plus
> the swhid. It still seems to me that it’d be hard to scale and to
> maintain that over time, even if it’s limited to svn. Plus, there’d
> still be the problem of ‘svn-multi-fetch’, which is what most TeX Live
> packages use.
>
> Thoughts?

Not too many, but I do have more bad news. Apologies if this is already
known, but I’m just getting up to speed with how SWH handles Subversion
(for coverage checking) and thought this seemed pretty significant.

I was starting with doing a simple check for the “easy” Subversion
repositories. That is, no externals (‘recursive?’) and no
‘svn-multi-fetch’ [1]. I immediately hit a problem. Guix hashes the
export of the repository with the keywords processed, while SWH hashes
it with unprocessed keywords.

For example, take ‘libsmpeg’. It has a file called “mkinstalldirs”,
which has a keyword in it: “$Id$”. The SWH loader hashes this as

$Id$

while we hash it as

$Id: mkinstalldirs 9 1999-10-21 15:55:01Z hercules $

This is not a big issue in terms of coverage checking, but it will be an
issue for automatic recovery. Even if you know the exact SWH directory
ID, you won’t get a directory that satisfies the daemon’s hash check. I
have no idea how hard it is to process the keywords with only data from
SWH. In this case, you would have to walk revisions to find the last
time “mkinstalldirs” was modified, and then format its metadata.
However, I assume the Subversion properties are gone, so there might be
edge cases like a file with “$Id$” (or whatever) that Subversion
wouldn’t processes.

Again, apologies if this is old news. Actually apologies either way,
’cause this is a bit of a downer!


-- Tim

[1] More precisely, I was going to process recursive ‘svn-fetch’ origins
because a lot of them are needlessly marked as recursive. In some
(many?) cases, the repositories don’t actually have external references,
so the flag does nothing. I was only going to skip the ones where it
makes a difference.
L
L
Ludovic Courtès wrote on 24 Mar 2023 18:22
Subversion keyword substitution
(name . Timothy Sample)(address . samplet@ngyro.com)
871qle6p4x.fsf_-_@inria.fr
Hi Timothy,

Timothy Sample <samplet@ngyro.com> skribis:

Toggle quote (6 lines)
> I was starting with doing a simple check for the “easy” Subversion
> repositories. That is, no externals (‘recursive?’) and no
> ‘svn-multi-fetch’ [1]. I immediately hit a problem. Guix hashes the
> export of the repository with the keywords processed, while SWH hashes
> it with unprocessed keywords.

Ouch. I had forgotten keywords were also a thing in Subversion. :-/

Can we tell Subversion to not expand them? That could be the way
forward in Guix, though it won’t help for past revisions.

How frequent is the use of keywords though? I tried the patch below to
turn off keyword substitution, and then ran

guix build -S --check -v1 --keep-going

for the 407 packages identified by:

Toggle snippet (13 lines)
scheme@(guile-user)> (define svn (fold-packages (lambda (p lst)
(if (and (not (hidden-package? p))
(not (package-superseded p))
(origin? (package-source p))
(memq (origin-method (package-source p))
(list svn-fetch svn-multi-fetch)))
(cons p lst)
lst))
'()))
scheme@(guile-user)> (length svn)
$8 = 407

That led to:

Toggle snippet (3 lines)
guix build: error: build of `/gnu/store/2byn59zmdbc4bz2wknnv0df4n67bdvgr-texlive-pdftex-59745-checkout.drv', `/gnu/store/2gj88z4plmwhraghxj5626zpiir1ck6k-libsmpeg-0.4.5-401-checkout.drv', `/gnu/store/2zygylsb2b333rzrvjyrh4qybw799hl3-ghmm-0.9-rc3-0.2341-checkout.drv', `/gnu/store/4a81qlka5w73rprapzi1w63xzb01n0r8-java-geronimo-xbean-reflect-4.5.drv', `/gnu/store/4mabgwil0ygwm3bkka3nzfbrwg1kk0wz-texlive-kpathsea-59745-checkout.drv', `/gnu/store/5ivk83abj22bs9ka10dk1v67kyczcd80-texlive-dvips-59745-checkout.drv', `/gnu/store/6zhnahylfr1zmpwzb8qzh8qp3yf9yl1p-texlive-tex-plain-59745-checkout.drv', `/gnu/store/f1sjmghs0f4v0y2pnljqaplifq52qbn2-texlive-cm-59745-checkout.drv', `/gnu/store/kd7kahaq71gi8j6zbabr0njqw60sjjxc-libsmpeg-0.4.5-399-checkout.drv', `/gnu/store/q3kip5bxkkdh14kx81afakbmpy7l4lr4-texlive-latexconfig-59745-checkout.drv', `/gnu/store/qskxc2c30fdclmrjp7nk35n1q2sl6sp8-texlive-tetex-59745-checkout.drv', `/gnu/store/wibsxy4kxlpq8lr76wvwgdyc2x5farr4-texlive-hyphen-base-59745-checkout.drv' failed

That’s 11 failures out of 407.

So, how about applying the ‘--ignore-keywords’ change and updating
hashes accordingly?

Toggle quote (6 lines)
> [1] More precisely, I was going to process recursive ‘svn-fetch’ origins
> because a lot of them are needlessly marked as recursive. In some
> (many?) cases, the repositories don’t actually have external references,
> so the flag does nothing. I was only going to skip the ones where it
> makes a difference.

We should remove that recursive flag when it has no effect. Perhaps we
could proceed similarly?

Thanks,
Ludo’.
Toggle diff (23 lines)
diff --git a/guix/build/svn.scm b/guix/build/svn.scm
index 2d960cb364..863c48e46d 100644
--- a/guix/build/svn.scm
+++ b/guix/build/svn.scm
@@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2014, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2020, 2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2014 Sree Harsha Totakura <sreeharsha@totakura.in>
;;; Copyright © 2018 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2020 Simon Tournier <zimon.toutoune@gmail.com>
@@ -47,6 +47,11 @@ (define* (svn-fetch url revision directory
;; verify the checksum later. This can be removed when
;; ca-certificates package is added.
"--trust-server-cert" "-r" (number->string revision)
+
+ ;; Disable keyword substitution (keywords are CVS-like strings
+ ;; like "$Date$", "$Id$", and so on).
+ "--ignore-keywords"
+
`(,@(if (and user-name password)
(list (string-append "--username=" user-name)
(string-append "--password=" password))
T
T
Timothy Sample wrote on 25 Mar 2023 00:31
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
87v8ipwwt4.fsf@ngyro.com
Hey,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

Toggle quote (15 lines)
> Hi Timothy,
>
> Timothy Sample <samplet@ngyro.com> skribis:
>
>> I was starting with doing a simple check for the “easy” Subversion
>> repositories. That is, no externals (‘recursive?’) and no
>> ‘svn-multi-fetch’ [1]. I immediately hit a problem. Guix hashes the
>> export of the repository with the keywords processed, while SWH hashes
>> it with unprocessed keywords.
>
> Ouch. I had forgotten keywords were also a thing in Subversion. :-/
>
> Can we tell Subversion to not expand them? That could be the way
> forward in Guix, though it won’t help for past revisions.

Thinking entirely abstractly, the keywords should be expanded. I’m not
really long enough in the tooth (old enough) to know how people use
keywords, but one might be tempted to do something like:

printf ("This is foo version %s\n", "$Revision$");

If that ever happens, processing the keywords would be very important.

Thinking practically, we’ve never encountered anything like that (see
below), and compatibility with SWH is nice.

It’s not clear to me why SWH passes ‘--ignore-keywords’ to Subversion in
the first place. I guess it saves storage, because having identical
files allows deduplication.

Toggle quote (2 lines)
> How frequent is the use of keywords though?

Well, you found 11 in the current Guix, and I see 30 when I process
everything I have (from version 1.0 to a few weeks ago). Furthermore,
the only usage pattern I see is “$Id” in a comment.

Toggle quote (3 lines)
> So, how about applying the ‘--ignore-keywords’ change and updating
> hashes accordingly?

It’s probably the right default given the circumstances.

It seems like there’s a direct conflict between ease of packaging and
ease of time travel. In the hypothetical case that a keyword mattered,
it would be a nasty surprise to the package author. They would have to
(a) discover the problem and (b) manually do the keyword substitution in
Scheme (or work around it).

Similarly, for recursive checkouts (including Git), it would be easier
to do time travel if we explicitly listed each source and how to compose
them. However, that’s a pain for package authors and maintainers.

Add to those two examples all the discussions about multihashing or
keeping track of the SWHID and something of a theme emerges....

Having said that, it’s nice that the work we are doing is letting us
think very clearly about this trade-off.

Toggle quote (9 lines)
>> [1] More precisely, I was going to process recursive ‘svn-fetch’
>> origins because a lot of them are needlessly marked as recursive. In
>> some (many?) cases, the repositories don’t actually have external
>> references, so the flag does nothing. I was only going to skip the
>> ones where it makes a difference.
>
> We should remove that recursive flag when it has no effect. Perhaps we
> could proceed similarly?

Huh. My scripts tell me that we haven’t needed it at all in the last
three years. That’s a suspicious enough result that I wonder if there’s
a bug in my scripts. The results are looking good so far, but there are
a few things I still need to look over.

I ran your same experiment, passing all the packages through:

(define (make-svn-package-non-recursive pkg)
(package
(inherit pkg)
(source
(origin
(inherit (package-source pkg))
(uri (match (origin-uri (package-source pkg))
((? svn-reference? ref)
(svn-reference
(inherit ref)
(recursive? #f)))
((? svn-multi-reference? ref)
(svn-multi-reference
(inherit ref)
(recursive? #f)))))))))

All of them were fine. Maybe we just haven’t had a truly recursive
Subversion source in recent memory.


-- Tim
L
L
Ludovic Courtès wrote on 27 Mar 2023 11:04
(name . Timothy Sample)(address . samplet@ngyro.com)
87v8imimyw.fsf@inria.fr
Hi,

Timothy Sample <samplet@ngyro.com> skribis:

Toggle quote (8 lines)
> Thinking entirely abstractly, the keywords should be expanded. I’m not
> really long enough in the tooth (old enough) to know how people use
> keywords, but one might be tempted to do something like:
>
> printf ("This is foo version %s\n", "$Revision$");
>
> If that ever happens, processing the keywords would be very important.

“Very” might be an overstatement. :-)

In practice, these were typically used in source file headers, so that
if you exported or copied files around (outside version control), they’d
have a timestamp of sorts at the top.

[...]

Toggle quote (4 lines)
> It’s not clear to me why SWH passes ‘--ignore-keywords’ to Subversion in
> the first place. I guess it saves storage, because having identical
> files allows deduplication.

I asked on #swh-devel and the fine folks there hinted at
non-reproducibility. Looking at
one thing that’s definitely not reproducible is the “local time zone”
bit. From that perspective it makes a lot of sense to disable keyword
substitution.

Toggle quote (6 lines)
>> How frequent is the use of keywords though?
>
> Well, you found 11 in the current Guix, and I see 30 when I process
> everything I have (from version 1.0 to a few weeks ago). Furthermore,
> the only usage pattern I see is “$Id” in a comment.

Interesting.

Toggle quote (5 lines)
>> So, how about applying the ‘--ignore-keywords’ change and updating
>> hashes accordingly?
>
> It’s probably the right default given the circumstances.

OK. I’ll submit a patch to that effect, unless you beat me at it. :-)

Toggle quote (6 lines)
> It seems like there’s a direct conflict between ease of packaging and
> ease of time travel. In the hypothetical case that a keyword mattered,
> it would be a nasty surprise to the package author. They would have to
> (a) discover the problem and (b) manually do the keyword substitution in
> Scheme (or work around it).

My intuition is that the worst “problem” we might have is ‘--version’
showing unexpanded keywords.

[...]

Toggle quote (8 lines)
>> We should remove that recursive flag when it has no effect. Perhaps we
>> could proceed similarly?
>
> Huh. My scripts tell me that we haven’t needed it at all in the last
> three years. That’s a suspicious enough result that I wonder if there’s
> a bug in my scripts. The results are looking good so far, but there are
> a few things I still need to look over.

Looks like it might be easily addressed!

Thanks,
Ludo’.
S
S
Simon Tournier wrote on 3 Apr 2023 14:05
Re: [bug#43442] Subversion keyword substitution
(address . 43442@debbugs.gnu.org)
875yadgohx.fsf@gmail.com
Hi,

On lun., 27 mars 2023 at 11:04, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (6 lines)
>> It seems like there’s a direct conflict between ease of packaging and
>> ease of time travel. In the hypothetical case that a keyword mattered,
>> it would be a nasty surprise to the package author. They would have to
>> (a) discover the problem and (b) manually do the keyword substitution in
>> Scheme (or work around it).

That’s an application of the non-free lunch theorem, no? :-)


Toggle quote (3 lines)
> My intuition is that the worst “problem” we might have is ‘--version’
> showing unexpanded keywords.

Somehow, I would prefer to pay the surprise by package author using
Subversion rather than not being fully reproducible over the time.


Cheers,
simon
S
S
Simon Tournier wrote on 3 Apr 2023 15:34
Re: [bug#43442] [PATCH] Fixes init of #42162: gforge.inria.fr down Dec. 2020
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 43442@debbugs.gnu.org)
871ql1gkcc.fsf@gmail.com
Hi,

On lun., 20 mars 2023 at 15:09, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (7 lines)
> As a stopgap, I wonder if we could use “double hashing” on our side, but
> only for svn: we’d store both the nar sha256 as we currently do, plus
> the swhid. It still seems to me that it’d be hard to scale and to
> maintain that over time, even if it’s limited to svn. Plus, there’d
> still be the problem of ‘svn-multi-fetch’, which is what most TeX Live
> packages use.

As proposed, somehow, in [1], I think we have nothing to loose with
optionally “double hashing” (or even triple or more). I do not
understand your concerns in [1] and I will discuss overthere. :-)

About ’svn-multi-fetch’ and TeX, what is at our advantage is that most
TeX Live packages use ’svn-multi-fetch’ via ’simple-texlive-package’
relying on ’texlive-origin’. And from official TeX documentation [2],
Subversion is not the recommended access, instead they suggest to rely
on ’rsync’. Moreover, some Git mirror is available.

Therefore, about TeX we could ask if relying on Subversion is still the
“good” way and maybe we could switch to something else as url-fetch or
git-fetch. Hum?! Not convinced by my proposal here. :-)

Well, IMHO, there is 2 issues: one about using SWH as fallback for
packages using svn-fetch and another one about TeX Live packages. It
could be nice to solve them both at once but maybe we should start now one
foot, now the other.



Cheers,
simon
T
T
Timothy Sample wrote on 4 Apr 2023 19:16
Re: Subversion keyword substitution
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
87v8iby3d0.fsf@ngyro.com
Hi!

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

Toggle quote (2 lines)
> OK. I’ll submit a patch to that effect, unless you beat me at it. :-)

I’m on it! It might take me another day or two to actually submit the
patch. The keyword expansion change affects ‘texlive-bin’, which means
a lot of rebuilds, so I guess we will need to coordinate a feature
branch or whatever (my understanding is that core-updates is essentially
frozen and deprecated at this point).

Toggle quote (16 lines)
> Timothy Sample <samplet@ngyro.com> skribis:
>
>> Thinking entirely abstractly, the keywords should be expanded. I’m not
>> really long enough in the tooth (old enough) to know how people use
>> keywords, but one might be tempted to do something like:
>>
>> printf ("This is foo version %s\n", "$Revision$");
>>
>> If that ever happens, processing the keywords would be very important.
>
> “Very” might be an overstatement. :-)
>
> In practice, these were typically used in source file headers, so that
> if you exported or copied files around (outside version control), they’d
> have a timestamp of sorts at the top.

I ended up finding 17 origins that make use of keyword expansion. Two
of them indeed do so outside of comments. (1) The “texlive-scripts”
source has some Perl scripts that do the Perl equivalent of my example
above. (2) There’s some Java code (“geronimo”) that uses keywords in
Javadoc comments, which might show up in generated documentation.

So no, definitely not “very” important! :)

Toggle quote (7 lines)
>> Huh. My scripts tell me that we haven’t needed it at all in the last
>> three years. That’s a suspicious enough result that I wonder if there’s
>> a bug in my scripts. The results are looking good so far, but there are
>> a few things I still need to look over.
>
> Looks like it might be easily addressed!

I’ll switch the ‘recursive?’ field to ‘#f’ by default as part of the
series I send in.


-- Tim
L
L
Ludovic Courtès wrote on 7 Apr 2023 18:45
(name . Timothy Sample)(address . samplet@ngyro.com)
87sfdbbq0d.fsf@inria.fr
Hi!

Timothy Sample <samplet@ngyro.com> skribis:

Toggle quote (5 lines)
>> So, how about applying the ‘--ignore-keywords’ change and updating
>> hashes accordingly?
>
> It’s probably the right default given the circumstances.


Ludo’.
L
L
Ludovic Courtès wrote on 9 Mar 23:34 +0100
Re: bug#43442: Code stored with Subversion (SVN) cannot be retrieved from SWH
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 43442-done@debbugs.gnu.org)
87ttlfyqz3.fsf_-_@gnu.org
Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

Toggle quote (23 lines)
> The attached patch does that:
>
> scheme@(guile-user)> (lookup-subversion-revision "https://scm.gforge.inria.fr/anonscm/svn/mpfi" 680)
> $12 = #<<revision> id: "72102de7605a2459ebcb016338ebbf1a997e8c8d" date: #<date nanosecond: 938388 second: 35 minute: 32 hour: 11 day: 6 month: 9 year: 2018 zone-offset: 0> directory: "5c89c025a4cd9d16befdfec12dfc23f7318d0d5b" directory-url: "https://archive.softwareheritage.org/api/1/directory/5c89c025a4cd9d16befdfec12dfc23f7318d0d5b/" parents-ids: ("16da41f1848d77a93aec565320b72b460c429b61") extra-headers: (("svn_repo_uuid" . "e2f78e0c-bb60-4709-9413-9660a31d4696") ("svn_revision" . "680"))>
> scheme@(guile-user)> (lookup-subversion-revision "https://scm.gforge.inria.fr/anonscm/svn/mpfi" 666)
> $13 = #<<revision> id: "148eb1e7206b111af4075c73c656e54c9efed6cb" date: #<date nanosecond: 654167 second: 2 minute: 52 hour: 12 day: 2 month: 8 year: 2018 zone-offset: 0> directory: "ed7b0bd7019fb85cd86d948a97c23b9d43aa8728" directory-url: "https://archive.softwareheritage.org/api/1/directory/ed7b0bd7019fb85cd86d948a97c23b9d43aa8728/" parents-ids: ("0ba2aa7e0d3fc0a1eb3ba72b32094515415ae47a") extra-headers: (("svn_repo_uuid" . "e2f78e0c-bb60-4709-9413-9660a31d4696") ("svn_revision" . "666"))>
>
> The implementation is pretty bad though, because it walks the revision
> history until it finds the right revision number—so you’re likely to
> reach the bandwidth rate limit before you’ve found the revision you’re
> looking for.
>
> More importantly, most svn origins cannot be found, or at least not
> by passing the URL as-is:
>
> https://sympa.inria.fr/sympa/arc/swh-devel/2023-03/msg00009.html
>
> This whole hack looks like a dead end.
>
> It would be ideal if SWH would compute nar hashes, as you proposed:
>
> https://gitlab.softwareheritage.org/swh/meta/-/issues/4538

Happy to report that this is solved with nar-sha256 ExtIDs at SWH, which
commit 0e73f933b291c7e154c7e019b6de1e2f3a97e4c1 uses to implement the
SWH fallback for Subversion checkouts. Wo0t!

Ludo’.
Closed
?