[PATCH] gnu: Add bees.

  • Done
  • quality assurance status badge
Details
3 participants
  • Adam Faiz
  • Hilton Chain
  • Ludovic Courtès
Owner
unassigned
Submitted by
Hilton Chain
Severity
normal
H
H
Hilton Chain wrote on 22 Dec 2022 09:19
(address . guix-patches@gnu.org)
87k02j4zq8.wl-hako@ultrarare.space
* gnu/packages/file-systems.scm (bees): New variable.
---
gnu/packages/file-systems.scm | 39 +++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)

Toggle diff (51 lines)
diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 57a25a0d90..cce406a01d 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -1766,3 +1766,42 @@ (define-public udftools
and other optical media. It supports read-only media (DVD/CD-R)
and rewritable media that wears out (DVD/CD-RW).")
(license license:gpl2+)))
+
+(define-public bees
+ (package
+ (name "bees")
+ (version "0.8")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/Zygo/bees")
+ (commit (string-append "v" version))))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ "1kxpz1p9k5ir385kpvmfjawki5vg22hlx768k7835w6n5z5a65y4"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list #:test-target "test"
+ #:make-flags
+ #~(list (string-append "CC=" #$(cc-for-target))
+ (string-append "DESTDIR=" #$output)
+ (string-append "BEES_VERSION=" #$version)
+ "PREFIX=''"
+ "LIBEXEC_PREFIX=/lib/bees")
+ #:phases
+ #~(modify-phases %standard-phases
+ (delete 'configure)
+ (add-after 'install 'fix-path
+ (lambda _
+ (substitute* (string-append #$output "/sbin/beesd")
+ (("/lib/bees/bees" all)
+ (string-append #$output all))))))))
+ (home-page "https://github.com/Zygo/bees")
+ (synopsis "Best-Effort Extent-Same, a btrfs dedupe agent")
+ (description
+ "@code{bees} is a block-oriented userspace deduplication agent designed
+for large btrfs filesystems. It is an offline dedupe combined with an
+incremental data scan capability to minimize time data spends on disk from
+write to dedupe.")
+ (license license:gpl3+)))

base-commit: 0744540d09ddef8dbf25cc5d65da9d029dab338c
--
2.38.1
A
A
Adam Faiz wrote on 22 Dec 2022 19:52
(address . 60250@debbugs.gnu.org)(address . hako@ultrarare.space)
21e09616-e1cc-05bc-5b9b-6b833198945b@disroot.org
Hi,

Toggle quote (5 lines)
> #~(list (string-append "CC=" #$(cc-for-target))
> + (string-append "DESTDIR=" #$output)
> + (string-append "BEES_VERSION=" #$version)
> + "PREFIX=''"
> + "LIBEXEC_PREFIX=/lib/bees"
The LIBEXEC_PREFIX shouldn't be set to this, this is actually a bug with
beesd.in and also the sed invocation(or $TEMPLATE_COMPILER which is what
it's called in the Makefile).

Toggle quote (7 lines)
> #~(modify-phases %standard-phases
> + (delete 'configure)
> + (add-after 'install 'fix-path
> + (lambda _
> + (substitute* (string-append #$output "/sbin/beesd")
> + (("/lib/bees/bees" all)
> + (string-append #$output all))))))))
The 'fix-path phase isn't necessary, and the LIBEXEC_PREFIX doesn't need
to be set. The snippet below fixes the bug in bees:

Toggle quote (10 lines)
> (modules '((guix build utils)))
> (snippet
> #~(begin
> (substitute* "Defines.mk"
> (("^sed.*" all)
> (string-append all
> "\t\t-e's#@DESTDIR@#$(DESTDIR)#' \\\n")))
> (substitute* "scripts/beesd.in"
> (("@LIBEXEC_PREFIX@") "@DESTDIR@/@LIBEXEC_PREFIX@"))))))

The files lib/city.cc and include/crucible/city.h need to be unbundled
and the cityhash included in guix used instead. The cityhash package
needs to be in the inputs, and these files need to be deleted as part of
the snippet above:> (for-each delete-file
Toggle quote (2 lines)
> '("lib/city.cc" "include/crucible/city.h"))

It appears that bees hasn't been built on a non-systemd distribution,
because an optional feature hard fails. This fixes it, the systemd unit
files are installed conditionally anyways:
Toggle quote (8 lines)
> (substitute* "Makefile"
> (("pkg-config systemd --variable=systemdsystemunitdir" all)
> (string-append all " || true")))

> (substitute* "lib/Makefile"
> (("city.o.*") ""))
> (substitute* "src/bees-hash.cc"
> (("#include .crucible/city.h.") "#include <city.h>"))
These substitutions remove references to the bundled cityhash(which is
11 years old!)

Native inputs and inputs for bees:
Toggle quote (5 lines)
> (native-inputs
> (list pkg-config markdown))
> (inputs
> (list cityhash))

All the modifications made as part of the snippet should definitely be
upstreamed, could you do that for me?
A
A
Adam Faiz wrote on 22 Dec 2022 20:22
(address . 60250@debbugs.gnu.org)
08ad30af-0278-fffb-ec55-9c1a3a3dc578@disroot.org
From the docs/install.md, it looks like btrfs-progs and
util-linux(needed for the blkid program) are used at runtime. References
to those programs in the source should probably just be substituted in a
phase.
H
H
Hilton Chain wrote on 23 Dec 2022 07:25
(name . Adam Faiz)(address . adam.faiz@disroot.org)(address . 60250@debbugs.gnu.org)
875ye27i1p.wl-hako@ultrarare.space
Hi Adam,

On Fri, 23 Dec 2022 02:52:38 +0800,
Adam Faiz wrote:
Toggle quote (57 lines)
>
> Hi,
>
> > #~(list (string-append "CC=" #$(cc-for-target))
> > + (string-append "DESTDIR=" #$output)
> > + (string-append "BEES_VERSION=" #$version)
> > + "PREFIX=''"
> > + "LIBEXEC_PREFIX=/lib/bees"
> The LIBEXEC_PREFIX shouldn't be set to this, this is actually a bug with beesd.in and also the sed
> invocation(or $TEMPLATE_COMPILER which is what it's called in the Makefile).
>
> > #~(modify-phases %standard-phases
> > + (delete 'configure)
> > + (add-after 'install 'fix-path
> > + (lambda _
> > + (substitute* (string-append #$output "/sbin/beesd")
> > + (("/lib/bees/bees" all)
> > + (string-append #$output all))))))))
> The 'fix-path phase isn't necessary, and the LIBEXEC_PREFIX doesn't need to be set. The snippet
> below fixes the bug in bees:
>
> > (modules '((guix build utils)))
> > (snippet
> > #~(begin
> > (substitute* "Defines.mk"
> > (("^sed.*" all)
> > (string-append all
> > "\t\t-e's#@DESTDIR@#$(DESTDIR)#' \\\n")))
> > (substitute* "scripts/beesd.in"
> > (("@LIBEXEC_PREFIX@") "@DESTDIR@/@LIBEXEC_PREFIX@"))))))
>
> The files lib/city.cc and include/crucible/city.h need to be unbundled and the cityhash included in
> guix used instead. The cityhash package needs to be in the inputs, and these files need to be
> deleted as part of the snippet above:> (for-each delete-file
> > '("lib/city.cc" "include/crucible/city.h"))
>
> It appears that bees hasn't been built on a non-systemd distribution, because an optional feature
> hard fails. This fixes it, the systemd unit files are installed conditionally anyways:
> > (substitute* "Makefile"
> > (("pkg-config systemd --variable=systemdsystemunitdir" all)
> > (string-append all " || true")))
>
> > (substitute* "lib/Makefile"
> > (("city.o.*") ""))
> > (substitute* "src/bees-hash.cc"
> > (("#include .crucible/city.h.") "#include <city.h>"))
> These substitutions remove references to the bundled cityhash(which is 11 years old!)
>
> Native inputs and inputs for bees:
> > (native-inputs
> > (list pkg-config markdown))
> > (inputs
> > (list cityhash))
>
> All the modifications made as part of the snippet should definitely be upstreamed, could you do that
> for me?

Thank you for the snippet and explanation!

I've made a pull request (of the beesd part) to the upstream:

(And it's merged before I send the mail XDDD)

However I'm not sure about the systemd part, could you explain more about which optional feature
hard fails? And I get same binaries whether-ever this part is used.

As pkg-config is only used for systemd, and markdown for documentations that won't be installed,
I'd rather remove them from native-inputs, WDYT?
H
H
Hilton Chain wrote on 23 Dec 2022 07:35
[PATCH v2] gnu: Add bees.
(address . 60250@debbugs.gnu.org)(name . Adam Faiz)(address . adam.faiz@disroot.org)
874jtm7hm0.wl-hako@ultrarare.space
* gnu/packages/patches/bees-beesd-honor-destdir-on-installation.patch: New
file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/file-systems.scm (bees): New variable.
---
v1 -> v2:
1. Unbundle cityhash.
2. Patch beesd.in for DESTDIR handling.
3. Modify beesd.in to use absolute paths of commands.

gnu/local.mk | 1 +
gnu/packages/file-systems.scm | 73 +++++++++++++++++++
...-beesd-honor-destdir-on-installation.patch | 40 ++++++++++
3 files changed, 114 insertions(+)
create mode 100644 gnu/packages/patches/bees-beesd-honor-destdir-on-installation.patch

Toggle diff (148 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index 45d05de02d..a733d296bd 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -935,6 +935,7 @@ dist_patch_DATA = \
%D%/packages/patches/bsd-games-prevent-name-collisions.patch \
%D%/packages/patches/bsd-games-stdio.h.patch \
%D%/packages/patches/beancount-disable-googleapis-fonts.patch \
+ %D%/packages/patches/bees-beesd-honor-destdir-on-installation.patch \
%D%/packages/patches/beignet-correct-file-names.patch \
%D%/packages/patches/bidiv-update-fribidi.patch \
%D%/packages/patches/binutils-2.37-file-descriptor-leak.patch \
diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 57a25a0d90..b7c681d1fb 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -81,6 +81,7 @@ (define-module (gnu packages file-systems)
#:use-module (gnu packages rsync)
#:use-module (gnu packages sssd)
#:use-module (gnu packages sqlite)
+ #:use-module (gnu packages textutils)
#:use-module (gnu packages time)
#:use-module (gnu packages tls)
#:use-module (gnu packages valgrind)
@@ -1766,3 +1767,75 @@ (define-public udftools
and other optical media. It supports read-only media (DVD/CD-R)
and rewritable media that wears out (DVD/CD-RW).")
(license license:gpl2+)))
+
+(define-public bees
+ (package
+ (name "bees")
+ (version "0.8")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/Zygo/bees")
+ (commit (string-append "v" version))))
+ (file-name (git-file-name name version))
+ (modules '((guix build utils)))
+ (snippet
+ ;; Unbundle cityhash.
+ #~(begin
+ (for-each delete-file
+ '("lib/city.cc" "include/crucible/city.h"))
+ (substitute* "lib/Makefile"
+ (("city.o.*") ""))
+ (substitute* "src/bees-hash.cc"
+ (("#include .crucible/city.h.") "#include <city.h>"))))
+ (patches
+ (search-patches
+ ;; XXX: Cherry-picked from upstream, remove the patch when
+ ;; bumping version.
+ "bees-beesd-honor-destdir-on-installation.patch"))
+ (sha256
+ (base32
+ "1kxpz1p9k5ir385kpvmfjawki5vg22hlx768k7835w6n5z5a65y4"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list #:test-target "test"
+ #:make-flags
+ #~(list (string-append "CC=" #$(cc-for-target))
+ (string-append "DESTDIR=" #$output)
+ (string-append "BEES_VERSION=" #$version)
+ "PREFIX=''")
+ #:phases
+ #~(modify-phases %standard-phases
+ (delete 'configure)
+ (add-after 'unpack 'fixpath
+ (lambda* (#:key inputs #:allow-other-keys)
+ (substitute* "scripts/beesd.in"
+ (((string-append "\\<(" (string-join (list "realpath"
+ "uuidparse"
+ "grep"
+ "false"
+ "sed"
+ "true"
+ "head"
+ "mkdir"
+ "mount"
+ "touch"
+ "du"
+ "cut"
+ "rm"
+ "truncate"
+ "chmod")
+ "|") ")\\>") command)
+ (search-input-file inputs (string-append "/bin/" command)))
+
+ (("btrfs sub")
+ (string-append (search-input-file inputs "/bin/btrfs") " sub"))))))))
+ (inputs (list btrfs-progs cityhash util-linux))
+ (home-page "https://github.com/Zygo/bees")
+ (synopsis "Best-Effort Extent-Same, a btrfs dedupe agent")
+ (description
+ "@code{bees} is a block-oriented userspace deduplication agent designed
+for large btrfs filesystems. It is an offline dedupe combined with an
+incremental data scan capability to minimize time data spends on disk from
+write to dedupe.")
+ (license license:gpl3+)))
diff --git a/gnu/packages/patches/bees-beesd-honor-destdir-on-installation.patch b/gnu/packages/patches/bees-beesd-honor-destdir-on-installation.patch
new file mode 100644
index 0000000000..4800bfc618
--- /dev/null
+++ b/gnu/packages/patches/bees-beesd-honor-destdir-on-installation.patch
@@ -0,0 +1,40 @@
+From 66b00f8a972ebb4da68f7aa0d0656f43ce2a2c3a Mon Sep 17 00:00:00 2001
+From: Hilton Chain <hako@ultrarare.space>
+Date: Fri, 23 Dec 2022 11:04:46 +0800
+Subject: [PATCH] beesd: Honor DESTDIR on installation.
+
+Co-authored-by: Adam Faiz <adam.faiz@disroot.org>
+Signed-off-by: Hilton Chain <hako@ultrarare.space>
+---
+ Defines.mk | 1 +
+ scripts/beesd.in | 2 +-
+ 2 files changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/Defines.mk b/Defines.mk
+index 9e8df40..e5394ba 100644
+--- a/Defines.mk
++++ b/Defines.mk
+@@ -2,6 +2,7 @@ MAKE += PREFIX=$(PREFIX) LIBEXEC_PREFIX=$(LIBEXEC_PREFIX) ETC_PREFIX=$(ETC_PREFI
+
+ define TEMPLATE_COMPILER =
+ sed $< >$@ \
++ -e's#@DESTDIR@#$(DESTDIR)#' \
+ -e's#@PREFIX@#$(PREFIX)#' \
+ -e's#@ETC_PREFIX@#$(ETC_PREFIX)#' \
+ -e's#@LIBEXEC_PREFIX@#$(LIBEXEC_PREFIX)#'
+diff --git a/scripts/beesd.in b/scripts/beesd.in
+index 174bb6c..35d04aa 100755
+--- a/scripts/beesd.in
++++ b/scripts/beesd.in
+@@ -15,7 +15,7 @@ readonly AL128K="$((128*1024))"
+ readonly AL16M="$((16*1024*1024))"
+ readonly CONFIG_DIR=@ETC_PREFIX@/bees/
+
+-readonly bees_bin=$(realpath @LIBEXEC_PREFIX@/bees)
++readonly bees_bin=$(realpath @DESTDIR@/@LIBEXEC_PREFIX@/bees)
+
+ command -v "$bees_bin" &> /dev/null || ERRO "Missing 'bees' agent"
+
+--
+2.38.1
+

base-commit: 1b29fccff2334fb5d6e979a290233a452969e39b
--
2.38.1
A
A
Adam Faiz wrote on 23 Dec 2022 08:57
Re: [PATCH] gnu: Add bees.
(name . Hilton Chain)(address . hako@ultrarare.space)(address . 60250@debbugs.gnu.org)
e86d8a06-0418-ebe5-3b63-288bb6b8de9d@disroot.org
On 12/23/22 14:25, Hilton Chain wrote:
Toggle quote (21 lines)
> Hi Adam,
>
> On Fri, 23 Dec 2022 02:52:38 +0800,
> Adam Faiz wrote:
>>
>> Hi,
>>
>> It appears that bees hasn't been built on a non-systemd distribution, because an optional feature
>> hard fails. This fixes it, the systemd unit files are installed conditionally anyways:
>>> (substitute* "Makefile"
>>> (("pkg-config systemd --variable=systemdsystemunitdir" all)
>>> (string-append all " || true")))
> Thank you for the snippet and explanation!
>
> I've made a pull request (of the beesd part) to the upstream:
> <https://github.com/Zygo/bees/pull/241>
>
> (And it's merged before I send the mail XDDD)
>
> However I'm not sure about the systemd part, could you explain more about which optional feature
> hard fails? And I get same binaries whether-ever this part is used.
I had a problem earlier, but I can't reproduce it now. You can remove
this one.

Toggle quote (2 lines)
> As pkg-config is only used for systemd, and markdown for documentations that won't be installed,
> I'd rather remove them from native-inputs, WDYT?
I agree that it should be done, but it's a shame that there's no
install_docs rule in the Makefile. That should be a feature request/patch.

I already have a bees-remove-bundled-cityhash.patch and
bees-add-cityhash-to-install.md.patch to send upstream. Should I email
them to you?
H
H
Hilton Chain wrote on 24 Dec 2022 12:34
(name . Adam Faiz)(address . adam.faiz@disroot.org)(address . 60250@debbugs.gnu.org)
87cz89xcfr.wl-hako@ultrarare.space
On Fri, 23 Dec 2022 15:57:53 +0800,
Adam Faiz wrote:
Toggle quote (8 lines)
> > As pkg-config is only used for systemd, and markdown for documentations that won't be installed,
> > I'd rather remove them from native-inputs, WDYT?
> I agree that it should be done, but it's a shame that there's no install_docs rule in the
> Makefile. That should be a feature request/patch.
>
> I already have a bees-remove-bundled-cityhash.patch and bees-add-cityhash-to-install.md.patch to
> send upstream. Should I email them to you?

As v2 has unbundled cityhash, there's no need to send the patches to me.

One thing to mention is that, according to Repology[1], cityhash is not widely packaged, as a
result, it might be better to implement the unbundle as an option.

A
A
Adam Faiz wrote on 24 Dec 2022 17:31
(name . Hilton Chain)(address . hako@ultrarare.space)(address . 60250@debbugs.gnu.org)
333e7b0f-0d53-ac85-191d-cedf57094e9a@disroot.org
On 12/24/22 19:34, Hilton Chain wrote:
Toggle quote (16 lines)
> On Fri, 23 Dec 2022 15:57:53 +0800,
> Adam Faiz wrote:
>>> As pkg-config is only used for systemd, and markdown for documentations that won't be installed,
>>> I'd rather remove them from native-inputs, WDYT?
>> I agree that it should be done, but it's a shame that there's no install_docs rule in the
>> Makefile. That should be a feature request/patch.
>>
>> I already have a bees-remove-bundled-cityhash.patch and bees-add-cityhash-to-install.md.patch to
>> send upstream. Should I email them to you?
>
> As v2 has unbundled cityhash, there's no need to send the patches to me.
>
> One thing to mention is that, according to Repology[1], cityhash is not widely packaged, as a
> result, it might be better to implement the unbundle as an option.
>
> [1]: <https://repology.org/project/cityhash/versions>
I don't think that should be done. It's all the more reason these
distributions need cityhash packaged and people to help them.

If they need a reference, they can always look at the cityhash package
in guix or other distributions and how they do it.
L
L
Ludovic Courtès wrote on 17 Jan 2023 16:02
Re: bug#60250: [PATCH] gnu: Add bees.
(name . Hilton Chain)(address . hako@ultrarare.space)
87bkmxgq5s.fsf_-_@gnu.org
Hi,

Hilton Chain <hako@ultrarare.space> skribis:

Toggle quote (10 lines)
> * gnu/packages/patches/bees-beesd-honor-destdir-on-installation.patch: New
> file.
> * gnu/local.mk (dist_patch_DATA): Add it.
> * gnu/packages/file-systems.scm (bees): New variable.
> ---
> v1 -> v2:
> 1. Unbundle cityhash.
> 2. Patch beesd.in for DESTDIR handling.
> 3. Modify beesd.in to use absolute paths of commands.

Applied, thanks!

Ludo’.
Closed
?