[PATCH 0/2] Add DwarFS

  • Done
  • quality assurance status badge
Details
2 participants
  • Pierre Neidhardt
  • Simon Tournier
Owner
unassigned
Submitted by
Pierre Neidhardt
Severity
normal
P
P
Pierre Neidhardt wrote on 29 Dec 2022 15:50
(address . guix-patches@gnu.org)
874jteqn67.fsf@ambrevar.xyz
This adds the dwarfs package.

1. For libdwarf, Guix lint complains that the source archive fails to be
extracted, complaining that 3600 is not a valid date. Another source
archive is fetched from a fallback source instead.

2. For dwarfs, I've disabled the tests because 1 test fails on 'modprobe
fuse'. If someone knows how to disable a specific test in the
'googletest' framework, please let me know.

Pierre Neidhardt (2):
gnu: Add libdwarf.
gnu: Add dwarfs.

gnu/packages/elf.scm | 25 ++++++++-
gnu/packages/file-systems.scm | 100 ++++++++++++++++++++++++++++++++++
2 files changed, 124 insertions(+), 1 deletion(-)


base-commit: 77d4bff94c6918eb0c0ccd20ee610e6dfc823aec
--
2.38.1
-----BEGIN PGP SIGNATURE-----

iQFGBAEBCAAwFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAmOtqTASHG1haWxAYW1i
cmV2YXIueHl6AAoJEJvc9Jeku8x/y68IAKsBQsWdNl5hhoLP+B02T8NTkwwu8ixQ
+YbjEtOwrwktV2+pFRsDalcfzIzq5SRWv+Ojv3woK5AW6YkLqI+gdy6LZCXtv8T7
KMgDA844NNUO2hzt83g6LM7on0kxnBlE3VE00CYpgkg9WT+CzVxyBNuA101NGtFf
Qf1y8fG7lE4M1M9Hi5AWijqfU+rjw+aJIGhpesncmOEYR3fbKhWBlQE757KWrki9
QYy8Ql4gvfET+Px+7lMJtXerCOQI9KEp0F8aNelGKBjwT+y6hpEYk4VybVKlqN7V
fVwXUgYDinMxWMRkqWl5mpDGGpAy+urb7/qyv7knxd1DugakUZB8tCE=
=QRVu
-----END PGP SIGNATURE-----

P
P
Pierre Neidhardt wrote on 29 Dec 2022 15:55
[PATCH 1/2] gnu: Add libdwarf.
(address . guix-patches@gnu.org)(address . 60406@debbugs.gnu.org)
87y1qqp8dz.fsf@ambrevar.xyz
* gnu/packages/elf.scm (libdwarf): New variable.
---
gnu/packages/elf.scm | 25 ++++++++++++++++++++++++-
1 file changed, 24 insertions(+), 1 deletion(-)

Toggle diff (42 lines)
diff --git a/gnu/packages/elf.scm b/gnu/packages/elf.scm
index 8cc2ff3637..7f2865f17f 100644
--- a/gnu/packages/elf.scm
+++ b/gnu/packages/elf.scm
@@ -31,7 +31,7 @@ (define-module (gnu packages elf)
#:use-module (guix packages)
#:use-module (guix download)
#:use-module (guix build-system gnu)
- #:use-module ((guix licenses) #:select (gpl3+ lgpl3+ lgpl2.0+))
+ #:use-module ((guix licenses) #:select (gpl3+ lgpl3+ lgpl2.0+ lgpl2.1 gpl2 bsd-2))
#:use-module (gnu packages)
#:use-module (gnu packages autotools)
#:use-module (gnu packages compression)
@@ -282,3 +282,26 @@ (define-public patchelf
(properties
'((release-monitoring-url . "https://github.com/NixOS/patchelf/releases")))
(license gpl3+)))
+
+(define-public libdwarf
+ (package
+ (name "libdwarf")
+ (version "0.5.0")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append "https://www.prevanders.net/libdwarf-"
+ version ".tar.xz"))
+ (sha256
+ (base32
+ "1a8j05z0rxf5ab9p5av2hqz6kxl8hfgar8h1387a0zric0n85yhi"))))
+ (build-system gnu-build-system)
+ (arguments
+ '(#:configure-flags '("--enable-shared")))
+ (native-inputs (list python))
+ (inputs (list elfutils))
+ (home-page "https://www.prevanders.net/dwarf.html")
+ (synopsis "Handle DWARF debugging information")
+ (description "@code{libdwarf} is a library that handles the DWARF
+debugging information format.")
+ ;; See https://www.prevanders.net/dwarflicense.html:
+ (license (list lgpl2.1 gpl2 bsd-2))))
--
2.38.1
-----BEGIN PGP SIGNATURE-----

iQFGBAEBCAAwFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAmOtqkgSHG1haWxAYW1i
cmV2YXIueHl6AAoJEJvc9Jeku8x/bi4H/0oie7L2EzONOdOthtRClAcZPY71NTGc
g5kWXtVH6wBPJhB+A57qkGWIs2o+YVsESVgv0o9u733HCqVZkW5jEqDA0kgXDDtg
N/TsNA3dW47pUbdNmsj8XOP2vR7fdS7kWmuSzM76mQYoN3+7WEgMHYCP9LuI2uDc
4YnOsjemJFLGlVXup8i/+9ClTR71KarVINeo9PqAb9Epex4vTQrVAH+JVqebPq+i
sbAyFV59iIIVW/K4kVAU8up9mlMxwmHAncysDVSl/wzDsglU48QZl41u/CDYPijA
5ItWKiw53+2NGbpLbbCx8LS/FV3yuO+u/OYdcdhg2SC9rdUFnClYG6Y=
=0pTB
-----END PGP SIGNATURE-----

P
P
Pierre Neidhardt wrote on 29 Dec 2022 15:55
[PATCH 2/2] gnu: Add dwarfs.
(address . guix-patches@gnu.org)(address . 60406@debbugs.gnu.org)
87v8lup8d2.fsf@ambrevar.xyz
* gnu/packages/file-systems.scm (dwarfs): New variable.
---
gnu/packages/file-systems.scm | 100 ++++++++++++++++++++++++++++++++++
1 file changed, 100 insertions(+)

Toggle diff (144 lines)
diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 3d45634169..e268c52f06 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -46,26 +46,36 @@ (define-module (gnu packages file-systems)
#:use-module (gnu packages admin)
#:use-module (gnu packages attr)
#:use-module (gnu packages autotools)
+ #:use-module (gnu packages backup)
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages bison)
+ #:use-module (gnu packages boost)
#:use-module (gnu packages check)
#:use-module (gnu packages compression)
+ #:use-module (gnu packages cpp)
#:use-module (gnu packages crypto)
#:use-module (gnu packages curl)
#:use-module (gnu packages cyrus-sasl)
#:use-module (gnu packages datastructures)
+ #:use-module (gnu packages digest)
#:use-module (gnu packages documentation)
#:use-module (gnu packages docbook)
+ #:use-module (gnu packages elf)
#:use-module (gnu packages flex)
#:use-module (gnu packages gawk)
#:use-module (gnu packages glib)
#:use-module (gnu packages gnupg)
#:use-module (gnu packages golang)
#:use-module (gnu packages guile)
+ #:use-module (gnu packages jemalloc)
#:use-module (gnu packages kerberos)
+ #:use-module (gnu packages libevent)
#:use-module (gnu packages libffi)
+ #:use-module (gnu packages libunwind)
#:use-module (gnu packages linux)
+ #:use-module (gnu packages maths)
+ #:use-module (gnu packages man)
#:use-module (gnu packages nfs)
#:use-module (gnu packages onc-rpc)
#:use-module (gnu packages openldap)
@@ -73,6 +83,8 @@ (define-module (gnu packages file-systems)
#:use-module (gnu packages perl)
#:use-module (gnu packages photo)
#:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages popt)
+ #:use-module (gnu packages pretty-print)
#:use-module (gnu packages python)
#:use-module (gnu packages python-crypto)
#:use-module (gnu packages python-web)
@@ -1790,3 +1802,91 @@ (define-public fuse-overlayfs
(description "This package provides an implementation of overlay+shiftfs
in FUSE for rootless containers.")
(license license:gpl3)))
+
+(define-public dwarfs
+ (package
+ (name "dwarfs")
+ (version "0.6.2")
+ (source (origin
+ (method url-fetch)
+ ;; The release archive is needed so that version.h is included.
+ (uri (string-append "https://github.com/mhx/dwarfs/releases/download/v"
+ version "/dwarfs-" version ".tar.xz"))
+ (sha256
+ (base32
+ "1kncxf85gsj3anck8ccjmxn2azp5ifqbgkiky2kharmvphkbmfcv"))
+ (snippet
+ #~(begin
+ (use-modules (guix build utils))
+ ;; Prefer system libraries instead of submodules.
+ ;; TODO: Package fbthrift.
+ ;; TODO: Can we use Guix own folly? There is no CMake option for it.
+ ;; TODO: Package parallel-hashmap.
+ (for-each delete-file-recursively
+ '(;; "fbthrift"
+ ;; "folly"
+ ;; "parallel-hashmap"
+ "xxHash"
+ "zstd"))))))
+ (build-system cmake-build-system)
+ (arguments
+ '(#:tests? #f ; TODO: 1 test fails because 'modprobe fuse' needs privileged access.
+ #:configure-flags
+ (list "-DPREFER_SYSTEM_ZSTD=ON"
+ "-DPREFER_SYSTEM_XXHASH=ON"
+ "-DPREFER_SYSTEM_GTEST=ON"
+ "-DWITH_TESTS=ON"
+ ;; Disable man pages since ronn fails to run without hpricot.
+ "-DWITH_MAN_PAGES=OFF")))
+ (native-inputs
+ (list
+ ;; FIXME: Building with ronn fail because hpricot is missing.
+ ;; ronn
+ folly googletest libdwarf libevent pkg-config))
+ (inputs
+ (list
+ boost
+ double-conversion
+ fmt
+ fuse-3
+ gflags
+ jemalloc
+ libarchive
+ libunwind
+ lz4
+ openssl
+ xxhash
+ xz
+ `(,zstd "lib")))
+ (home-page "https://github.com/mhx/dwarfs")
+ (synopsis "Fast high compression read-only file system")
+ (description "DwarFS is a read-only file system with a focus on achieving
+very high compression ratios in particular for very redundant data.
+
+DwarFS also doesn't compromise on speed and for some cases it is on par with
+or performs better than SquashFS. For the primary use case, DwarFS
+compression is an order of magnitude better than SquashFS compression, it's 6
+times faster to build the file system, it's typically faster to access files
+on DwarFS and it uses less CPU resources.
+
+Distinct features of DwarFS are:
+
+@itemize
+
+@item Clustering of files by similarity using a similarity hash function. This
+makes it easier to exploit the redundancy across file boundaries.
+
+@item Segmentation analysis across file system blocks in order to reduce the
+size of the uncompressed file system. This saves memory when using the
+compressed file system and thus potentially allows for higher cache hit rates
+as more data can be kept in the cache.
+
+@item Highly multi-threaded implementation. Both the file system creation tool
+as well as the FUSE driver are able to make good use of the many cores of your
+system.
+
+@item Optional experimental Python scripting support to provide custom
+filtering and ordering functionality.
+
+@end itemize\n")
+ (license license:gpl3)))
--
2.38.1
-----BEGIN PGP SIGNATURE-----

iQFGBAEBCAAwFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAmOtqmkSHG1haWxAYW1i
cmV2YXIueHl6AAoJEJvc9Jeku8x/x2UH/i2chSi9E5k2fWYRjgrwUXhcmZOfuCJd
mfxlqDl2nuXUHbUXbwzpzmIK99bG7J8RblFfx1wTx0wl3xGN5gQSGQ49Q3R91okO
UGk8LYjq3UB/RzLBKpQFCmqksStR8oRjLIRnoc4Si5OjwCc0WtIKPV62GS1Vu7WF
p9goE8KQewsNLByhm6IohdTjTYKXsFSgriXgHsCtSVKdGiB71BghSKnJ40bwZywI
qK+wcyFSEHi8HnJBkGHUGKf3uTt4go6RoVcUR+hhT9xX8xlJ/kInxew7ax73rtfU
Z0aMWCRaXK7Cb2Z3I7Nsk6Ud14yIVvcsQL/R73jHGKORmYNbHO2LX+g=
=BDRQ
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 12 Jan 2023 14:58
Re: [bug#60406] [PATCH 0/2] Add DwarFS
87bkn3yhvi.fsf@gmail.com
Hi Pierre,

On jeu., 29 déc. 2022 at 15:50, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

Toggle quote (4 lines)
> 1. For libdwarf, Guix lint complains that the source archive fails to be
> extracted, complaining that 3600 is not a valid date. Another source
> archive is fetched from a fallback source instead.

What is the issue the Git repository?



Because that does not sound good: :-)

Toggle snippet (16 lines)
$ guix download https://www.prevanders.net/libdwarf-0.5.0.tar.xz

Starting download of /tmp/guix-file.oWkq5d
From https://www.prevanders.net/libdwarf-0.5.0.tar.xz...
Bad Date header: 3600


Starting download of /tmp/guix-file.oWkq5d
From https://web.archive.org/web/20230112145352/https://www.prevanders.net/libdwarf-0.5.0.tar.xz...
following redirection to `https://web.archive.org/web/20221224112643/https://www.prevanders.net/libdwarf-0.5.0.tar.xz'...
….5.0.tar.xz 2.4MiB 1.0MiB/s 00:02 [##################] 100.0%
/gnu/store/0xcriql6m20wcdzjv4a5fbcg7h0nmq8h-libdwarf-0.5.0.tar.xz
1a8j05z0rxf5ab9p5av2hqz6kxl8hfgar8h1387a0zric0n85yhi


Cheers,
simon
P
P
Pierre Neidhardt wrote on 13 Jan 2023 13:09
87lem6skj4.fsf@ambrevar.xyz
Oh right, I can just use the GitHub repo.
Thank you Simon!
Problem 1 solved then.

I'll push with a note about the tests.
-----BEGIN PGP SIGNATURE-----

iQFGBAEBCAAwFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAmPBSg8SHG1haWxAYW1i
cmV2YXIueHl6AAoJEJvc9Jeku8x/XDkH/iQ9zPf+Pq3hJG+8hqtTyCD6hB3zTdET
YiFGsndcinrksEbJ35EK8gJgEX2FRRJsDTvKNlSZtPoOedVV3Rh3wmyUzW4qBAHW
mx/Ot3ttvjsKH8MmUPd08NQni+RTiYoaTII3XLHvCTfADhykEGVi++U6E4t7zTBC
6Laa9M7KONno9377QU4bYfdNbQRpaL2ncKXR+3ilp4ZAFDY5eq1SPSKeWMm2nywd
tZ6MEeeA4f0XfRLFa/Z5lhzC/84xPcaeTpRl2Az94G99vcYOBL6hIfqpwyYh+d5U
K3uzhzkSghoppuNI1/axLNqK00bdjj3T74DZthojHpCcngub3JuACIQ=
=68mr
-----END PGP SIGNATURE-----

P
P
Pierre Neidhardt wrote on 23 Jan 2023 14:20
87ilgxbd6k.fsf@ambrevar.xyz
Pushed with

04ade01beb95da45d63ca1f58126508d0491d99d

and

3a1b18aa4540e6f96ded0a98dd907a8033262582

Thanks!
Pierre
-----BEGIN PGP SIGNATURE-----

iQFGBAEBCAAwFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAmPOiYMSHG1haWxAYW1i
cmV2YXIueHl6AAoJEJvc9Jeku8x/u6sH/3o/O1HdHHu+OmmCghbPkHnlsZwT+p+g
kiniN2WGaNJxb7MK5/bBNPWavb+biTo+YP7HifLLEb1sg7qQE0w4ojwwKDcWjSU9
4TbJ8Aq1rsnePufBGx87zaYg6awOeJhTp1qiGMrJGpcBViQrjU0MN7bx99M3dMbP
VRflV3QV9JADo2Vgj63CL8tPiBzUF64ifogaTWcGjVYv6Y2YbdkgBsaO3xZYoNVh
6fnN6g0Zd3VUxLWujEwoTJ7wGm/8gMdB7RnsrUhQr30zUJ5SJ1xsNcg5vW26SALh
7SeQYRmrRDvESfXZ8BZOxikc1Che05xIuuabYVYdnYbJpeVa4cnHYFQ=
=Wwor
-----END PGP SIGNATURE-----

P
P
Pierre Neidhardt wrote on 23 Jan 2023 14:21
control message for bug #60406
(address . control@debbugs.gnu.org)
87h6whbd4e.fsf@ambrevar.xyz
close 60406
quit
?