[PATCH] gnu: packages: Add grokmirror.

  • Done
  • quality assurance status badge
Details
2 participants
  • Kyle Meyer
  • Ludovic Courtès
Owner
unassigned
Submitted by
Kyle Meyer
Severity
normal
K
K
Kyle Meyer wrote on 26 Sep 2020 22:48
(address . guix-patches@gnu.org)
20200926204819.30602-1-kyle@kyleam.com
* gnu/packages/version-control.scm (grokmirror): New variable.
---
gnu/packages/version-control.scm | 39 +++++++++++++++++++++++++++++++-
1 file changed, 38 insertions(+), 1 deletion(-)

Toggle diff (61 lines)
diff --git a/gnu/packages/version-control.scm b/gnu/packages/version-control.scm
index 2ee8247071..68971c60f9 100644
--- a/gnu/packages/version-control.scm
+++ b/gnu/packages/version-control.scm
@@ -7,7 +7,7 @@
;;; Copyright © 2014, 2015, 2016 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2014, 2016, 2019 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Efraim Flashner <efraim@flashner.co.il>
-;;; Copyright © 2015, 2018 Kyle Meyer <kyle@kyleam.com>
+;;; Copyright © 2015, 2018, 2020 Kyle Meyer <kyle@kyleam.com>
;;; Copyright © 2015, 2017, 2018, 2020 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2016, 2017 Leo Famulari <leo@famulari.name>
;;; Copyright © 2016, 2017, 2018 Nikita <nikita@n0.is>
@@ -2269,6 +2269,43 @@ (define-public myrepos
Mercurial, Bazaar, Darcs, CVS, Fossil, and Veracity.")
(license license:gpl2+)))
+(define-public grokmirror
+ (package
+ (name "grokmirror")
+ (version "2.0.0")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url (string-append "https://git.kernel.org/pub/scm/"
+ "utils/grokmirror/grokmirror.git"))
+ (commit (string-append "v" version))))
+ (file-name (string-append name "-" version "-checkout"))
+ (sha256
+ (base32 "1cs43vf87x8x5k5ncgiwiclc92a1dvxpg2z6lh6psaiip808gylp"))))
+ (build-system python-build-system)
+ (arguments
+ `(#:tests? #f ; no test suite
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'install 'install-manpages
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((man (string-append (assoc-ref outputs "out")
+ "/man/man1/")))
+ (mkdir-p man)
+ (for-each (lambda (file) (install-file file man))
+ (find-files "." "\\.1$")))
+ #t)))))
+ (propagated-inputs
+ `(("python-requests" ,python-requests)))
+ (home-page
+ "https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git")
+ (synopsis "Framework to smartly mirror git repositories")
+ (description "Grokmirror enables replicating large git repository
+collections efficiently. Mirrors decide to clone and update repositories
+based on a manifest file published by servers.")
+ (license license:gpl3+)))
+
(define-public git-annex-remote-rclone
(package
(name "git-annex-remote-rclone")

base-commit: 3876ed32f25e160c47ba046d987ad9097b2bf0cb
--
2.28.0.849.g665beee440
K
K
Kyle Meyer wrote on 26 Sep 2020 22:58
(address . 43637@debbugs.gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
878scww903.fsf@kyleam.com
A recent message on bug-guix [1] mentioned cloning the archives at
https://yhetil.org. public-inbox archives, including those at
yhetil.org, publish a grokmirror-compatible manifest. So, if you're
interested in mirroring the Guix-related archives at yhetil.org for any
reason [2], you can use grokmirror to do so in a way that's nicer than
periodically hitting yhetil.org with a git-fetch call for each
Guix-related inbox.

You can clone (and then later update) Guix-related inboxes with

$ grok-pull -vc /path/to/config

and a config file that looks something like this:

Toggle snippet (16 lines)
[core]
toplevel = /path/to/inboxes
manifest = ${toplevel}/manifest.js.gz
log = ${toplevel}/grok.log
loglevel = info
objstore = ${toplevel}/objstore

[pull]
pull_threads = 2
include = /guix-*

[remote]
site = https://yhetil.org
manifest = ${site}/manifest.js.gz

CAJ3okZ0g25+-wsL2OepE+xMR5QWShmS73hSHsjEmEgYBiOWwAg@mail.gmail.com

[2] to have a fast local copy, to publish your own mirror, to feed the
archives into some other tool...
L
L
Ludovic Courtès wrote on 27 Sep 2020 22:51
Re: [bug#43637] [PATCH] gnu: packages: Add grokmirror.
(name . Kyle Meyer)(address . kyle@kyleam.com)(address . 43637-done@debbugs.gnu.org)
87wo0fdjtn.fsf@gnu.org
Hi,

Kyle Meyer <kyle@kyleam.com> skribis:

Toggle quote (2 lines)
> * gnu/packages/version-control.scm (grokmirror): New variable.

Applied, thanks!

Toggle quote (8 lines)
> A recent message on bug-guix [1] mentioned cloning the archives at
> <https://yhetil.org>. public-inbox archives, including those at
> yhetil.org, publish a grokmirror-compatible manifest. So, if you're
> interested in mirroring the Guix-related archives at yhetil.org for any
> reason [2], you can use grokmirror to do so in a way that's nicer than
> periodically hitting yhetil.org with a git-fetch call for each
> Guix-related inbox.

So yhetil.org stores mailing list archives as a Git repo?

Sounds interesting.

Thanks,
Ludo’.
Closed
K
K
Kyle Meyer wrote on 27 Sep 2020 23:14
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43637-done@debbugs.gnu.org)
87wo0eexbn.fsf@kyleam.com
Ludovic Courtès writes:

Toggle quote (8 lines)
> Hi,
>
> Kyle Meyer <kyle@kyleam.com> skribis:
>
>> * gnu/packages/version-control.scm (grokmirror): New variable.
>
> Applied, thanks!

Thanks for reviewing/applying!

Toggle quote (12 lines)
>> A recent message on bug-guix [1] mentioned cloning the archives at
>> <https://yhetil.org>. public-inbox archives, including those at
>> yhetil.org, publish a grokmirror-compatible manifest. So, if you're
>> interested in mirroring the Guix-related archives at yhetil.org for any
>> reason [2], you can use grokmirror to do so in a way that's nicer than
>> periodically hitting yhetil.org with a git-fetch call for each
>> Guix-related inbox.
>
> So yhetil.org stores mailing list archives as a Git repo?
>
> Sounds interesting.

Yep, messages are stored in a Git repo or, for archives greater than
around one gigabyte, a collection of repos [*]. (A freshly gc'd
guix-devel weighs in at ~140M.)

[*] documentation on public-inbox's v2 format:
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 43637@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 43637
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch