[PATCH] gnu: packages: Add grokmirror.

  • Done
  • quality assurance status badge
Details
2 participants
  • Kyle Meyer
  • Ludovic Courtès
Owner
unassigned
Submitted by
Kyle Meyer
Severity
normal
K
K
Kyle Meyer wrote on 26 Sep 2020 22:48
(address . guix-patches@gnu.org)
20200926204819.30602-1-kyle@kyleam.com
* gnu/packages/version-control.scm (grokmirror): New variable.
---
gnu/packages/version-control.scm | 39 +++++++++++++++++++++++++++++++-
1 file changed, 38 insertions(+), 1 deletion(-)

Toggle diff (61 lines)
diff --git a/gnu/packages/version-control.scm b/gnu/packages/version-control.scm
index 2ee8247071..68971c60f9 100644
--- a/gnu/packages/version-control.scm
+++ b/gnu/packages/version-control.scm
@@ -7,7 +7,7 @@
;;; Copyright © 2014, 2015, 2016 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2014, 2016, 2019 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Efraim Flashner <efraim@flashner.co.il>
-;;; Copyright © 2015, 2018 Kyle Meyer <kyle@kyleam.com>
+;;; Copyright © 2015, 2018, 2020 Kyle Meyer <kyle@kyleam.com>
;;; Copyright © 2015, 2017, 2018, 2020 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2016, 2017 Leo Famulari <leo@famulari.name>
;;; Copyright © 2016, 2017, 2018 Nikita <nikita@n0.is>
@@ -2269,6 +2269,43 @@ (define-public myrepos
Mercurial, Bazaar, Darcs, CVS, Fossil, and Veracity.")
(license license:gpl2+)))
+(define-public grokmirror
+ (package
+ (name "grokmirror")
+ (version "2.0.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url (string-append "https://git.kernel.org/pub/scm/"
+ "utils/grokmirror/grokmirror.git"))
+ (commit (string-append "v" version))))
+ (file-name (string-append name "-" version "-checkout"))
+ (sha256
+ (base32 "1cs43vf87x8x5k5ncgiwiclc92a1dvxpg2z6lh6psaiip808gylp"))))
+ (build-system python-build-system)
+ (arguments
+ `(#:tests? #f ; no test suite
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'install 'install-manpages
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((man (string-append (assoc-ref outputs "out")
+ "/man/man1/")))
+ (mkdir-p man)
+ (for-each (lambda (file) (install-file file man))
+ (find-files "." "\\.1$")))
+ #t)))))
+ (propagated-inputs
+ `(("python-requests" ,python-requests)))
+ (home-page
+ "https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git")
+ (synopsis "Framework to smartly mirror git repositories")
+ (description "Grokmirror enables replicating large git repository
+collections efficiently. Mirrors decide to clone and update repositories
+based on a manifest file published by servers.")
+ (license license:gpl3+)))
+
(define-public git-annex-remote-rclone
(package
(name "git-annex-remote-rclone")

base-commit: 3876ed32f25e160c47ba046d987ad9097b2bf0cb
--
2.28.0.849.g665beee440
K
K
Kyle Meyer wrote on 26 Sep 2020 22:58
(address . 43637@debbugs.gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
878scww903.fsf@kyleam.com
A recent message on bug-guix [1] mentioned cloning the archives at
https://yhetil.org. public-inbox archives, including those at
yhetil.org, publish a grokmirror-compatible manifest. So, if you're
interested in mirroring the Guix-related archives at yhetil.org for any
reason [2], you can use grokmirror to do so in a way that's nicer than
periodically hitting yhetil.org with a git-fetch call for each
Guix-related inbox.

You can clone (and then later update) Guix-related inboxes with

$ grok-pull -vc /path/to/config

and a config file that looks something like this:

Toggle snippet (16 lines)
[core]
toplevel = /path/to/inboxes
manifest = ${toplevel}/manifest.js.gz
log = ${toplevel}/grok.log
loglevel = info
objstore = ${toplevel}/objstore

[pull]
pull_threads = 2
include = /guix-*

[remote]
site = https://yhetil.org
manifest = ${site}/manifest.js.gz

CAJ3okZ0g25+-wsL2OepE+xMR5QWShmS73hSHsjEmEgYBiOWwAg@mail.gmail.com

[2] to have a fast local copy, to publish your own mirror, to feed the
archives into some other tool...
L
L
Ludovic Courtès wrote on 27 Sep 2020 22:51
Re: [bug#43637] [PATCH] gnu: packages: Add grokmirror.
(name . Kyle Meyer)(address . kyle@kyleam.com)(address . 43637-done@debbugs.gnu.org)
87wo0fdjtn.fsf@gnu.org
Hi,

Kyle Meyer <kyle@kyleam.com> skribis:

Toggle quote (2 lines)
> * gnu/packages/version-control.scm (grokmirror): New variable.

Applied, thanks!

Toggle quote (8 lines)
> A recent message on bug-guix [1] mentioned cloning the archives at
> <https://yhetil.org>. public-inbox archives, including those at
> yhetil.org, publish a grokmirror-compatible manifest. So, if you're
> interested in mirroring the Guix-related archives at yhetil.org for any
> reason [2], you can use grokmirror to do so in a way that's nicer than
> periodically hitting yhetil.org with a git-fetch call for each
> Guix-related inbox.

So yhetil.org stores mailing list archives as a Git repo?

Sounds interesting.

Thanks,
Ludo’.
Closed
K
K
Kyle Meyer wrote on 27 Sep 2020 23:14
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43637-done@debbugs.gnu.org)
87wo0eexbn.fsf@kyleam.com
Ludovic Courtès writes:

Toggle quote (8 lines)
> Hi,
>
> Kyle Meyer <kyle@kyleam.com> skribis:
>
>> * gnu/packages/version-control.scm (grokmirror): New variable.
>
> Applied, thanks!

Thanks for reviewing/applying!

Toggle quote (12 lines)
>> A recent message on bug-guix [1] mentioned cloning the archives at
>> <https://yhetil.org>. public-inbox archives, including those at
>> yhetil.org, publish a grokmirror-compatible manifest. So, if you're
>> interested in mirroring the Guix-related archives at yhetil.org for any
>> reason [2], you can use grokmirror to do so in a way that's nicer than
>> periodically hitting yhetil.org with a git-fetch call for each
>> Guix-related inbox.
>
> So yhetil.org stores mailing list archives as a Git repo?
>
> Sounds interesting.

Yep, messages are stored in a Git repo or, for archives greater than
around one gigabyte, a collection of repos [*]. (A freshly gc'd
guix-devel weighs in at ~140M.)

[*] documentation on public-inbox's v2 format:
Closed
?