[PATCH 0/3] Better Support for ZFS on Guix

OpenSubmitted by raid5atemyhomework.
Details
2 participants
  • Ludovic Courtès
  • raid5atemyhomework
Owner
unassigned
Severity
normal
Merged with
R
R
raid5atemyhomework wrote on 4 Jan 02:02 +0100
(name . guix-patches@gnu.org)(address . guix-patches@gnu.org)
KsQRoKgO0OqFaU5n-6nXB2fBwhViqfa-T12IzGcIDCtZFGhGfKz_8cVrnk2IVYQERMjIhw7MV6nOQxMZ0nZn2ddHQFQi4jJJf_spXGkUgkg=@protonmail.com
ZFS package is broken, and we should really recompile ZFS at each kernel update anyway.
This patchset includes:
* Patch 1: * Fixes for "util-linux" dependency which causes problems with mounting ZFS datasets. * Expose a "make-zfs-package" procedure to create kernel-specific ZFS packages as suggested here: https://issues.guix.gnu.org/45403#3 * Add a service "zfs-loader-service-type" that provisions "zfs-loader", that modprobes ZFS and automounts ZFS datasets at bootup. The latter behavior is what is expected of ZFS from other OSs, so it's best to imitate that as well. * Document how to set up and use the above. * This makes ZFS on Guix actually *possible*. Tested in a VM.* Patch 2: * Make the "file-systems" provisioning service a target similar to "user-processes", so that other services can insert themselves into "file-systems" mounting. * Make the "zfs-loader" part of the "file-systems" target if it is instantiated. * This enables /home on ZFS (previous patch is not enough as "user-homes" service is part of the "user-processes"; we need to make ZFS loading and mounting earlier).* Patch 3 * Disable checking of "device" field for declared file-systems that are of type "zfs". * Make file-systems of type "zfs" dependent on "zfs-loader" to ensure that "zfs" module is loaded and all pools are imported and known by the ZFS module. * This enables declaring ZFS datasets as (file-system ...) declarations, if the sysad wants to explicitly document the mountpoints in the configuration file rather than having ZFS manage mountpoints.
R
R
raid5atemyhomework wrote on 4 Jan 02:05 +0100
[PATCH 1/3] gnu, doc: Create and document procedure to compile ZFS for specific kernel.
(name . 45643@debbugs.gnu.org)(address . 45643@debbugs.gnu.org)
_yQ96oMMZ2qR6_Y7zRGocuGc7bzzQl5zF_6x-AEmsOFM4YnJvcpGzfcnHNBNhYl0LxPvPsac1rL6eHG63g5W58W9K76qXYwKa57-hu36AYI=@protonmail.com
From 004a84a38266eb8735f1e127f5794bf6d66b9272 Mon Sep 17 00:00:00 2001From: raid5atemyhomework <raid5atemyhomework@protonmail.com>Date: Fri, 1 Jan 2021 20:26:42 +0800Subject: [PATCH 1/3] gnu, doc: Create and document procedure to compile ZFS for specific kernel.
--- doc/guix.texi | 100 ++++++++++++++++++++++++++++++++++ gnu/local.mk | 1 + gnu/packages/file-systems.scm | 25 ++++++++- gnu/services/file-systems.scm | 72 ++++++++++++++++++++++++ 4 files changed, 196 insertions(+), 2 deletions(-) create mode 100644 gnu/services/file-systems.scm
Toggle diff (245 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex 1081ed26a3..5ad3907dbe 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -13885,6 +13885,106 @@ a file system declaration such as: compress-force=zstd,space_cache=v2")) @end lisp
+@node ZFS file system+@subsection ZFS file system++The ZFS on Linux file system cannot legally be downloaded as part of the+Linux kernel, but you as a user can do anything on hardware you own,+including download ZFS as source code, compile ZFS as a kernel module,+and link it into Linux.++As a large and complex kernel module, ZFS has to be compiled for a+specific version of Linux. Often the latest ZFS package available in Guix+cannot be compiled with the latest Linux kernel available in Guix, so+installing the @code{zfs} package in your system configuration file will+fail.++Instead, you have to check the+@url{https://github.com/openzfs/zfs/releases,OpenZFS release notes} for+the specific version of ZFS that Guix has packaged to determine what+Linux kernels you can use, then check the @code{linux-libre} packages+that Guix has packaged, and select one you can use on your system.++Then, you have to modify your system configuration file, and create a+ZFS package that compiles using the specific Linux version you chose.+Below is a sketch of how you would modify your @code{operating-system}+declaration in order to set up ZFS:++@lisp+(use-modules (gnu))+;; @dots{}+(use-service-modules+ ; @dots{}+ file-systems)+(use-package-modules+ ; @dots{}+ linux+ file-systems)++;; @dots{}++;;; (1) Select a specific kernel.+(define system-kernel linux-libre-5.4)+;;; (2) Define a ZFS package for your kernel.+(define system-zfs (make-zfs-package system-kernel))++;; @dots{}++(operating-system+ ;;; (3) Specify your selected kernel.+ (kernel system-kernel)+ ;;; (4) Add the "module" output of the system ZFS package to+ ;;; the kernel-loadable modules.+ (kernel-loadable-modules (list (list system-zfs "module")))++ ;; @dots{}++ (packages+ ;;; (5) Add the system ZFS package to global packages so that+ ;;; "zfs", "zpool" etc. commands are available.+ (cons* system-zfs+ ; @dots{}+ %base-packages))++ ;; @dots{}++ (services+ ;;; (6) Add a service that loads ZFS and all ZFS pools at boot.+ (cons* (service zfs-loader-service-type+ system-zfs)+ ; @dots{}+ %base-services))+ ;; @dots{}+ )+@end lisp++@deffn (Scheme Procedure) make-zfs-package @var{kernel}+This procedure creates a package which, when included as a package+in your system, can be loaded as a kernel module for the specified+@var{kernel}, a Linux kernel package.+It is intended to create a system-specific ZFS kernel module for+the Linux kernel you will use in your system.+@end deffn++@deffn (Scheme Variable) zfs-loader-service-type+This is the service that loads the ZFS kernel module and imports all+ZFS pools. The value associated with @code{zfs-loader-service-type}+services must be the ZFS package you defined for your system, and+whose @code{"module"} output you have included as a loadable kernel+module.+@end deffn++When you have modified your @code{operating-system} to include ZFS+as in the above and reconfigured and rebooted, you will now be able+to create, mount, and manage ZFS pools. At each boot, ZFS will+automatically open any ZFS pools it detects in your system, and will+mount ZFS pools that have a non-@code{legacy} mountpoint. If ZFS+finds mountable ZFS filesystems that are encrypted by passphrase, it+will prompt for passphrases on the console.++ZFS as root filesystem is not supported yet. ZFS for @code{/home} is+also probably not easily doable yet.+ @node Mapped Devices @section Mapped Devices
diff --git a/gnu/local.mk b/gnu/local.mkindex c03a8b9f51..e59a3bb9f7 100644--- a/gnu/local.mk+++ b/gnu/local.mk@@ -601,6 +601,7 @@ GNU_SYSTEM_MODULES = \ %D%/services/dict.scm \ %D%/services/dns.scm \ %D%/services/docker.scm \+ %D%/services/file-systems.scm \ %D%/services/authentication.scm \ %D%/services/games.scm \ %D%/services/ganeti.scm \diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scmindex 895ad069c5..e8f49a59fa 100644--- a/gnu/packages/file-systems.scm+++ b/gnu/packages/file-systems.scm@@ -944,16 +944,37 @@ APFS.") ("openssl" ,openssl) ("python" ,python) ("python-cffi" ,python-cffi)- ("util-linux" ,util-linux "lib")+ ("util-linux" ,util-linux)+ ("util-linux-lib" ,util-linux "lib") ("zlib" ,zlib))) (home-page "https://zfsonlinux.org/") (synopsis "Native ZFS on Linux") (description "ZFS on Linux is an advanced file system and volume manager which was originally developed for Solaris and is now maintained by the OpenZFS-community.")+community.++DO NOT INSTALL THIS PACKAGE. Instead, refer to the 'ZFS file system' section+of the Guix info manual for how to install ZFS.") (license license:cddl1.0)))
+(define-public (make-zfs-package kernel)+ (package+ (inherit zfs)+ (name (string-append "zfs-for-"+ (package-name kernel)+ "-"+ (package-version kernel)+ "-version"))+ (arguments+ (cons* #:linux kernel (package-arguments zfs)))+ (description+ "ZFS on Linux is an advanced file system and volume manager which was+originally developed for Solaris and is now maintained by the OpenZFS+community.++This package has been compiled for a specific Linux kernel.")))+ (define-public mergerfs (package (name "mergerfs")diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scmnew file mode 100644index 0000000000..bdc33f4028--- /dev/null+++ b/gnu/services/file-systems.scm@@ -0,0 +1,72 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2021 raid5atemyhomework <raid5atemyhomework@protonmail.com>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.++(define-module (gnu services file-systems)+ #:use-module (guix gexp)+ #:use-module (gnu services)+ #:use-module (gnu services shepherd))++;;; ZFS+(define (zfs-loader-shepherd-service system-zfs)+ (let* ((zpool (file-append system-zfs "/sbin/zpool")))+ (list+ (shepherd-service+ (documentation "Load ZFS kernel module and import ZFS pools.")+ (provision '(zfs-loader))+ (requirement '(file-systems))+ (one-shot? #t)+ (modules `((srfi srfi-1)+ (srfi srfi-34)+ (srfi srfi-35)+ (rnrs io ports)+ ,@%default-modules))+ (start #~(lambda _+ (if (not (file-exists? "/proc/sys/kernel/modprobe"))+ (begin (format (current-error-port) "error: ~a~%"+ "Kernel is missing loadable module support.")+ #f)+ (and+ (let ((modprobe (call-with-input-file+ "/proc/sys/kernel/modprobe" get-line)))+ (guard (c ((message-condition? c)+ (format (current-error-port)+ "error loading 'zfs' kernel module: ~a~%"+ (condition-message c))+ #f))+ (invoke/quiet modprobe "--" "zfs")))+ (guard (c ((message-condition? c)+ (format (current-error-port)+ "error importing zpools: ~a~%"+ (condition-message c))+ #f))+ ;; 'current-output-port' is typically connected to /dev/klog+ ;; in PID 1, so redirect it to the console so we are talking+ ;; to the user if zpool sees an encrpyted fs that needs a+ ;; passphrase.+ (with-output-to-port (current-error-port)+ (lambda ()+ (invoke #$zpool "import" "-a" "-l"))))))))))))++(define-public zfs-loader-service-type+ (service-type+ (name 'zfs-loader)+ (description "Load ZFS kernel module and import ZFS pools.")+ (extensions+ (list (service-extension shepherd-root-service-type zfs-loader-shepherd-service)+ (service-extension user-processes-service-type (const '(zfs-loader)))))))+
--2.29.2
R
R
raid5atemyhomework wrote on 4 Jan 02:06 +0100
[PATCH 2/3] gnu: Make 'file-systems' target extensible.
(name . 45643@debbugs.gnu.org)(address . 45643@debbugs.gnu.org)
jEYheQNgKAst_NT1rsSJiKBjWyhGsVPH0eIV0q3pdA7e0-1fPBxiIKzqTXWSGLTNVypyerCiYYN4MqwzaO9vm3q_HmcRn6eSck1MmVQJL6Y=@protonmail.com
From fc4538963960550b678713ec46ac461cfbd6173e Mon Sep 17 00:00:00 2001From: raid5atemyhomework <raid5atemyhomework@protonmail.com>Date: Sun, 3 Jan 2021 21:09:02 +0800Subject: [PATCH 2/3] gnu: Make 'file-systems' target extensible.
This is to support something like ZFS, which has a service that mountsZFS file systems and does not normally use /etc/fstab--- doc/guix.texi | 9 +++++++-- gnu/services/base.scm | 37 +++++++++++++++++++++++++---------- gnu/services/file-systems.scm | 5 +++-- 3 files changed, 37 insertions(+), 14 deletions(-)
Toggle diff (120 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex 5ad3907dbe..4544b481b0 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -13982,8 +13982,13 @@ mount ZFS pools that have a non-@code{legacy} mountpoint. If ZFS finds mountable ZFS filesystems that are encrypted by passphrase, it will prompt for passphrases on the console.
-ZFS as root filesystem is not supported yet. ZFS for @code{/home} is-also probably not easily doable yet.+You can put @code{/home} on a ZFS filesystem by setting it as the+mountpoint of some ZFS filesystem. However, ZFS will refuse to mount+onto a non-empty directory, so if you already have an existing+@code{/home} you have to remove all files in it, then reboot to let+ZFS mount into @code{/home}.++ZFS as root filesystem is not supported yet.
@node Mapped Devices @section Mapped Devicesdiff --git a/gnu/services/base.scm b/gnu/services/base.scmindex 945b546607..deffd49154 100644--- a/gnu/services/base.scm+++ b/gnu/services/base.scm@@ -66,6 +66,7 @@ %default-substitute-urls) #:export (fstab-service-type root-file-system-service+ file-systems-service-type file-system-service-type swap-service host-name-service@@ -362,18 +363,30 @@ FILE-SYSTEM." (gnu system file-systems) ,@%default-modules)))))))
+(define (file-systems-service requirements)+ (list+ (shepherd-service+ (provision '(file-systems))+ (requirement (cons* 'root-file-system 'user-file-systems requirements))+ (documentation "Target for all the initially-mounted file systems")+ (start #~(const #t))+ (stop #~(const #t)))))++(define file-systems-service-type+ (service-type+ (name 'file-systems)+ (extensions (list (service-extension shepherd-root-service-type+ file-systems-service)))+ (compose concatenate)+ (extend append)+ ;; Extensions can add new services to this list.+ (default-value '())+ (description "The @code{file-systems} service is the target that is started+when all file systems have been mounted.")))+ (define (file-system-shepherd-services file-systems) "Return the list of Shepherd services for FILE-SYSTEMS." (let* ((file-systems (filter file-system-mount? file-systems)))- (define sink- (shepherd-service- (provision '(file-systems))- (requirement (cons* 'root-file-system 'user-file-systems- (map file-system->shepherd-service-name- file-systems)))- (documentation "Target for all the initially-mounted file systems")- (start #~(const #t))- (stop #~(const #f))))
(define known-mount-points (map file-system-mount-point file-systems))@@ -403,7 +416,7 @@ FILE-SYSTEM." (filter (negate known?) (mount-points))) #f))))
- (cons* sink user-unmount+ (cons* user-unmount (map file-system-shepherd-service file-systems))))
(define (file-system-fstab-entries file-systems)@@ -431,6 +444,10 @@ FILE-SYSTEM." (service-extension fstab-service-type file-system-fstab-entries)
+ ;; Have 'file-systems' depend on each file-system+ (service-extension file-systems-service-type+ (cut map file-system->shepherd-service-name <>))+ ;; Have 'user-processes' depend on 'file-systems'. (service-extension user-processes-service-type (const '(file-systems)))))diff --git a/gnu/services/file-systems.scm b/gnu/services/file-systems.scmindex bdc33f4028..73a6d52538 100644--- a/gnu/services/file-systems.scm+++ b/gnu/services/file-systems.scm@@ -19,6 +19,7 @@ (define-module (gnu services file-systems) #:use-module (guix gexp) #:use-module (gnu services)+ #:use-module (gnu services base) #:use-module (gnu services shepherd))
;;; ZFS@@ -28,7 +29,7 @@ (shepherd-service (documentation "Load ZFS kernel module and import ZFS pools.") (provision '(zfs-loader))- (requirement '(file-systems))+ (requirement '(root-file-system)) (one-shot? #t) (modules `((srfi srfi-1) (srfi srfi-34)@@ -68,5 +69,5 @@ (description "Load ZFS kernel module and import ZFS pools.") (extensions (list (service-extension shepherd-root-service-type zfs-loader-shepherd-service)- (service-extension user-processes-service-type (const '(zfs-loader)))))))+ (service-extension file-systems-service-type (const '(zfs-loader)))))))
--2.29.2
R
R
raid5atemyhomework wrote on 4 Jan 02:09 +0100
[PATCH 3/3] gnu, guix: Support mounting legacy-mounted ZFS in operating-system form.
(name . 45643@debbugs.gnu.org)(address . 45643@debbugs.gnu.org)
JQwrx0zllfKouhNkLC3SdQXURha-g0cP1d37GCcZZc_Kq1WqlDJzKYyJxBlUsMscWI9cNq6I32GJ46E2JZwZOdRJVNlWjtwrIDZQ4YAOuEU=@protonmail.com
From 59c9bd5642e33962798c01f4dcf30be38ead4ab8 Mon Sep 17 00:00:00 2001From: raid5atemyhomework <raid5atemyhomework@protonmail.com>Date: Mon, 4 Jan 2021 08:22:01 +0800Subject: [PATCH 3/3] gnu, guix: Support mounting legacy-mounted ZFS in operating-system form.
--- doc/guix.texi | 18 ++++++++++++++++++ gnu/build/file-systems.scm | 2 ++ gnu/machine/ssh.scm | 3 +++ gnu/services/base.scm | 5 ++++- guix/scripts/system.scm | 3 +++ 5 files changed, 30 insertions(+), 1 deletion(-)
Toggle diff (86 lines)diff --git a/doc/guix.texi b/doc/guix.texiindex 4544b481b0..2909674302 100644--- a/doc/guix.texi+++ b/doc/guix.texi@@ -13982,6 +13982,24 @@ mount ZFS pools that have a non-@code{legacy} mountpoint. If ZFS finds mountable ZFS filesystems that are encrypted by passphrase, it will prompt for passphrases on the console.
+You should @emph{not} put ZFS datasets with a non-@code{legacy} mountpoint+into the @code{file-systems} field of your @code{operating-sstem}. However,+if you want to declare them in your @code{file-systems} field, you can+set their ZFS mountpoints to @code{legacy} and add a @code{file-system}+declaration for them, using the ZFS pool dataset name as the @code{device}+field:++@example+zfs set mountpoint=legacy pool/filesystem+@end example++@lisp+(file-system+ (device "pool/filesystem")+ (mount-point "/path/to/mountpoint")+ (type "zfs"))+@end lisp+ You can put @code{/home} on a ZFS filesystem by setting it as the mountpoint of some ZFS filesystem. However, ZFS will refuse to mount onto a non-empty directory, so if you already have an existingdiff --git a/gnu/build/file-systems.scm b/gnu/build/file-systems.scmindex ddf6117b67..af75aee2b6 100644--- a/gnu/build/file-systems.scm+++ b/gnu/build/file-systems.scm@@ -828,6 +828,8 @@ containing ':/')." ((string-prefix? "f2fs" type) check-f2fs-file-system) ((string-prefix? "ntfs" type) check-ntfs-file-system) ((string-prefix? "nfs" type) (const 'pass))+ ;; "ZFS doesn't need fsck."+ ((string-prefix? "zfs" type) (const 'pass)) (else #f)))
(if check-procedurediff --git a/gnu/machine/ssh.scm b/gnu/machine/ssh.scmindex 08c653ba17..cdd8913a00 100644--- a/gnu/machine/ssh.scm+++ b/gnu/machine/ssh.scm@@ -175,6 +175,9 @@ exist on the machine." %pseudo-file-system-types)) ;; Don't try to validate network file systems. (not (string-prefix? "nfs" (file-system-type fs)))+ ;; Don't try to validate ZFS file systems.+ ;; XXX We should validate these by 'zfs status'.+ (not (string-prefix? "zfs" (file-system-type fs))) (not (memq 'bind-mount (file-system-flags fs))))) (operating-system-file-systems (machine-operating-system machine))))
diff --git a/gnu/services/base.scm b/gnu/services/base.scmindex deffd49154..dcacc5b4b8 100644--- a/gnu/services/base.scm+++ b/gnu/services/base.scm@@ -322,7 +322,10 @@ FILE-SYSTEM." (shepherd-service (provision (list (file-system->shepherd-service-name file-system))) (requirement `(root-file-system udev- ,@(map dependency->shepherd-service-name dependencies)))+ ,@(map dependency->shepherd-service-name dependencies)+ ,@(if (string-prefix? "zfs" (file-system-type file-system))+ '(zfs-loader)+ '()))) (documentation "Check, mount, and unmount the given file system.") (start #~(lambda args #$(if create?diff --git a/guix/scripts/system.scm b/guix/scripts/system.scmindex 0dcf2b3afe..1515062ada 100644--- a/guix/scripts/system.scm+++ b/guix/scripts/system.scm@@ -564,6 +564,9 @@ any, are available. Raise an error if they're not." %pseudo-file-system-types)) ;; Don't try to validate network file systems. (not (string-prefix? "nfs" (file-system-type fs)))+ ;; Don't try to validate ZFS file systems.+ ;; XXX We should validate these by 'zfs status'.+ (not (string-prefix? "zfs" (file-system-type fs))) (not (memq 'bind-mount (file-system-flags fs))))) file-systems))
--2.29.2
L
L
Ludovic Courtès wrote on 10 Feb 16:47 +0100
control message for bug #45692
(address . control@debbugs.gnu.org)
87wnvf6i1i.fsf@gnu.org
merge 45692 45643quit
?