[PATCH WIP 0/5] Improve on restic-backup-service

  • Open
  • quality assurance status badge
Details
3 participants
  • paul
  • Fabio Natali
  • Richard Sent
Owner
unassigned
Submitted by
Richard Sent
Severity
normal
R
R
Richard Sent wrote on 19 Jun 00:06 +0200
(address . guix-patches@gnu.org)
cover.1718747513.git.richard@freakingpenguin.com
Hi all!

This is the previously promised [1] patch series relating to
restic-backup-service. The highlights:

1. init? was added that bootstraps a new backup.

2. RESTIC_PASSWORD_COMMAND support.

3. Add extra-packages field for jobs.

4. Move restic from restic-backup-job to restic-backup-configuration.

5. Add a basic system-level test suite.

Regarding 4, the previous way felt a bit orthogonal to how services of a
similar nature work. (For example, you don't specify an mcron package for
every job.) I can't think of a use case for mixing-and-matching restic
packages with different jobs. If there is a reason, I think a good compromise
would be a "restic-override" job field.

I'm marking this as WIP because there are a couple more things I want to
investigate, but it feels complete enough to submit for feedback. My main
point of interest is what can be done to improve support for Restic's other
backends, see [2]. I imagine 0.9.6 only supports a subset of those, but still.

In my dream world we could combine this patch with an upgrade to the restic
package itself (which, among other things, supports compression). I asked the
developers about (re)including a vendor directory in their tarball but haven't
heard back [3]. Perhaps this service can spark some action on an old Guix
tracking issue [4].


Richard Sent (5):
services: backup: Support bootstrapping an initial restic backup
services: backup: Add password-command support to restic-service
services: backup: Add extra-packages field to restic-backup-job
services: backup: Move restic package to restic-configuration
tests: Add restic system test.

doc/guix.texi | 21 ++++++-
gnu/services/backup.scm | 131 +++++++++++++++++++++++++++++-----------
gnu/tests/restic.scm | 124 +++++++++++++++++++++++++++++++++++++
3 files changed, 238 insertions(+), 38 deletions(-)
create mode 100644 gnu/tests/restic.scm


base-commit: a575d0f5d5322bac977423b6bd2742c8dc5a14a6
--
2.45.1
R
R
Richard Sent wrote on 19 Jun 00:08 +0200
[PATCH WIP 1/5] services: backup: Support bootstrapping an initial restic backup
(address . 71639@debbugs.gnu.org)
88ce7267a59dfb3d80bd790e99b00a731e56835f.1718747513.git.richard@freakingpenguin.com
* gnu/services/backup.scm: (restic-backup-job): Add init? field.
(restic-backup-job-program): Initialize repository if init? is set and
repository does not already exist.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I71d0cbaac646b9d160e662b69286f229b9a9f64d
---
doc/guix.texi | 4 ++++
gnu/services/backup.scm | 20 ++++++++++++++++++++
2 files changed, 24 insertions(+)

Toggle diff (75 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 0102fd0fad..63c9cbd1a7 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41353,6 +41353,10 @@ Miscellaneous Services
The list of files or directories to be backed up. It must be a list of
values that can be lowered to strings.
+@item @code{init?} (default: @code{#f}) (type: boolean)
+Whether restic-backup-service should check and (if it does not exist)
+initialize the repository before running the backup.
+
@item @code{verbose?} (default: @code{#f}) (type: boolean)
Whether to enable verbose output for the current backup job.
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index 555e9fc959..eeef11eae7 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2024 Giacomo Leidi <goodoldpaul@autistici.org>
+;;; Copyright © 2024 Richard Sent <richard@freakingpenguin.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -38,6 +39,7 @@ (define-module (gnu services backup)
restic-backup-job-password-file
restic-backup-job-schedule
restic-backup-job-files
+ restic-backup-job-init?
restic-backup-job-verbose?
restic-backup-job-extra-flags
@@ -94,6 +96,9 @@ (define-configuration/no-serialization restic-backup-job
(verbose?
(boolean #f)
"Whether to enable verbose output for the current backup job.")
+ (init?
+ (boolean #f)
+ "Whether to attempt to initialize a new repository for automated bootstrap purposes.")
(extra-flags
(list-of-lowerables '())
"A list of values that are lowered to strings. These will be passed as
@@ -118,6 +123,8 @@ (define (restic-backup-job-program config)
(restic-backup-job-files config))
(extra-flags
(restic-backup-job-extra-flags config))
+ (init?
+ (restic-backup-job-init? config))
(verbose
(if (restic-backup-job-verbose? config)
'("--verbose")
@@ -130,6 +137,19 @@ (define (restic-backup-job-program config)
(setenv "RESTIC_PASSWORD"
(with-input-from-file #$password-file read-line))
+ (when #$init?
+ ;; Check if the repository exists. See
+ ;; https://github.com/restic/restic/issues/1690 and
+ ;; https://github.com/NixOS/nixpkgs/pull/307962.
+ ;;
+ ;; XXX: restic returns values other than 1 on failure. Use
+ ;; unless EXIT_SUCCESS instead of when EXIT_FAILURE.
+ (unless (equal? EXIT_SUCCESS (system* #$restic "cat" "config"
+ "-r" #$repository))
+ ;; Initialize it.
+ (unless (equal? EXIT_SUCCESS (system* #$restic "init"
+ "-r" #$repository))
+ (error "Failed to initialize restic repository: " #$repository))))
(execlp #$restic #$restic #$@verbose
"-r" #$repository
#$@extra-flags
--
2.45.1
R
R
Richard Sent wrote on 19 Jun 00:08 +0200
[PATCH WIP 3/5] services: backup: Add extra-packages field to restic-backup-job
(address . 71639@debbugs.gnu.org)
146d362c149771810d673e927b56d3ec44bc9bba.1718747513.git.richard@freakingpenguin.com
* gnu/services/backup.scm (restic-backup-job): Create extra-packages.
(restic-guix-wrapper-package): Add the extra-packages field of every job as
inputs.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I4f0b070bc6dc895553ba69256d14e45898291c02
---
doc/guix.texi | 4 ++++
gnu/services/backup.scm | 33 ++++++++++++++++++++-------------
2 files changed, 24 insertions(+), 13 deletions(-)

Toggle diff (78 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index f22d679023..32ce0c86b9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41367,6 +41367,10 @@ Miscellaneous Services
@item @code{verbose?} (default: @code{#f}) (type: boolean)
Whether to enable verbose output for the current backup job.
+@item @code{extra-packages} (default: @code{'()} (type: list-of-packages)
+The list of extra packages needed for restic to run the current backup
+job.
+
@item @code{extra-flags} (default: @code{'()}) (type: list-of-lowerables)
A list of values that are lowered to strings. These will be passed as
command-line arguments to the current job @command{restic backup}
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index 2471d0ea7b..a6d8404a5a 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -41,6 +41,7 @@ (define-module (gnu services backup)
restic-backup-job-files
restic-backup-job-init?
restic-backup-job-verbose?
+ restic-backup-job-extra-packages
restic-backup-job-extra-flags
restic-backup-configuration
@@ -102,6 +103,9 @@ (define-configuration/no-serialization restic-backup-job
(list-of-lowerables '())
"The list of files or directories to be backed up. It must be a list of
values that can be lowered to strings.")
+ (extra-packages
+ (list-of-packages '())
+ "The list of extra packages needed for restic to run this backup job.")
(verbose?
(boolean #f)
"Whether to enable verbose output for the current backup job.")
@@ -239,22 +243,25 @@ (define (restic-backup-job->mcron-job config)
#:user #$user)))
(define (restic-guix-wrapper-package jobs)
- (package
- (name "restic-backup-service-wrapper")
- (version "0.0.0")
- (source (restic-guix jobs))
- (build-system copy-build-system)
- (arguments
- (list #:install-plan #~'(("./" "/bin"))))
- (home-page "https://restic.net")
- (synopsis
- "Easily interact from the CLI with Guix configured backups")
- (description
- "This package provides a simple wrapper around @code{restic}, handled
+ (let ((extra-packages (append-map restic-backup-job-extra-packages
+ jobs)))
+ (package
+ (name "restic-backup-service-wrapper")
+ (version "0.0.0")
+ (source (restic-guix restic-package jobs))
+ (build-system copy-build-system)
+ (arguments
+ (list #:install-plan #~'(("./" "/bin"))))
+ (home-page "https://restic.net")
+ (synopsis
+ "Easily interact from the CLI with Guix configured backups")
+ (description
+ "This package provides a simple wrapper around @code{restic}, handled
by the @code{restic-backup-service-type}. It allows for easily interacting
with Guix configured backup jobs, for example for manually triggering a backup
without waiting for the scheduled job to run.")
- (license license:gpl3+)))
+ (inputs extra-packages)
+ (license license:gpl3+))))
(define restic-backup-service-profile
(lambda (config)
--
2.45.1
R
R
Richard Sent wrote on 19 Jun 00:08 +0200
[PATCH WIP 2/5] services: backup: Add password-command support to restic-service
(address . 71639@debbugs.gnu.org)
388adecd6bae7d959392b862f1ccc234c0c24a6d.1718747513.git.richard@freakingpenguin.com
* gnu/services/backup.scm (restic-backup-job): Add password-command.
(verify-restic-backup-job-configuration): Create.
(restic-backup-job-program): Set either RESTIC_PASSWORD or
RESTIC_PASSWORD_COMMAND depending on what is configured.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: Ice9cf85d1ee4485a2737f515c63c969918219df0
---
doc/guix.texi | 7 +++++++
gnu/services/backup.scm | 42 ++++++++++++++++++++++++++++++++++++-----
2 files changed, 44 insertions(+), 5 deletions(-)

Toggle diff (109 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 63c9cbd1a7..f22d679023 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41344,6 +41344,13 @@ Miscellaneous Services
that will be used to set the @env{RESTIC_PASSWORD} environment variable
for the current job.
+@item @code{password-command} (type: file-like)
+String path or file-like object representing the executable file that
+prints password to stdout. If a file-like object is used, it is placed
+in the store globally executable and in plain text. The executable
+should be designed such that it does not compromise the password if an
+unauthorized user runs it.
+
@item @code{schedule} (type: gexp-or-string)
A string or a gexp that will be passed as time specification in the
mcron job specification (@pxref{Syntax, mcron job specifications,,
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index eeef11eae7..2471d0ea7b 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -66,6 +66,9 @@ (define (lowerable? value)
(define list-of-lowerables?
(list-of lowerable?))
+(define-maybe/no-serialization string)
+(define-maybe/no-serialization file-like)
+
(define-configuration/no-serialization restic-backup-job
(restic
(package restic)
@@ -80,10 +83,16 @@ (define-configuration/no-serialization restic-backup-job
(string)
"The restic repository target of this job.")
(password-file
- (string)
+ (maybe-string)
"Name of the password file, readable by the configured @code{user}, that
will be used to set the @code{RESTIC_PASSWORD} environment variable for the
current job.")
+ (password-command
+ (maybe-file-like)
+ "Name of the password command that, when run, returns the password over
+stdin. Due to the nature of the store this command will be globally executable
+and should have external protections to ensure other users cannot retrieve the
+password. This overrides password-file.")
(schedule
(gexp-or-string)
"A string or a gexp that will be passed as time specification in the mcron
@@ -104,6 +113,14 @@ (define-configuration/no-serialization restic-backup-job
"A list of values that are lowered to strings. These will be passed as
command-line arguments to the current job @command{restic backup} invokation."))
+(define (verify-restic-backup-job-configuration config)
+ (unless (or (maybe-value-set? (restic-backup-job-password-file config))
+ (maybe-value-set? (restic-backup-job-password-command config)))
+ (error "either password-file or password-command must be configured."))
+ (when (and (maybe-value-set? (restic-backup-job-password-file config))
+ (maybe-value-set? (restic-backup-job-password-command config)))
+ (error "password-file and password-command can not be configured simultaneously.")))
+
(define list-of-restic-backup-jobs?
(list-of restic-backup-job?))
@@ -113,12 +130,22 @@ (define-configuration/no-serialization restic-backup-configuration
"The list of backup jobs for the current system."))
(define (restic-backup-job-program config)
+ (define (maybe-value-or-false maybe)
+ (if (maybe-value-set? maybe)
+ maybe
+ #f))
+
+ ;; TODO: Find a place to also verify restic-backup-configuration. Mainly that jobs >=1
+ (verify-restic-backup-job-configuration config)
+
(let ((restic
(file-append (restic-backup-job-restic config) "/bin/restic"))
(repository
(restic-backup-job-repository config))
(password-file
- (restic-backup-job-password-file config))
+ (maybe-value-or-false (restic-backup-job-password-file config)))
+ (password-command
+ (maybe-value-or-false (restic-backup-job-password-command config)))
(files
(restic-backup-job-files config))
(extra-flags
@@ -134,9 +161,14 @@ (define (restic-backup-job-program config)
#~(begin
(use-modules (ice-9 popen)
(ice-9 rdelim))
- (setenv "RESTIC_PASSWORD"
- (with-input-from-file #$password-file read-line))
-
+ (or (and=> #$password-file (lambda (x)
+ (setenv "RESTIC_PASSWORD"
+ (with-input-from-file x read-line))))
+ (and=> #$password-command (lambda (x)
+ (setenv "RESTIC_PASSWORD_COMMAND" x)))
+ ;; Have a backup error message in case
+ ;; verify-restic-backup-job-configuration is messed with
+ (error "Neither password-file or password-command set"))
(when #$init?
;; Check if the repository exists. See
;; https://github.com/restic/restic/issues/1690 and
--
2.45.1
R
R
Richard Sent wrote on 19 Jun 00:08 +0200
[PATCH WIP 4/5] services: backup: Move restic package to restic-configuration
(address . 71639@debbugs.gnu.org)
7749ca0f06a80cf9accd24dc62fa8896cd35625d.1718747513.git.richard@freakingpenguin.com
* gnu/services/backup.scm (restic-backup-configuration): Add restic field.
(restic-backup-job-program): Add restic package to function signature.
(restic-guix): Ditto.
(restic-guix-wrapper-package): Ditto. Add restic package as package input.
(restic-backup-service-profile): Remove excess lamba.

* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I1e5f63c21cd072354225afe0ee270dca8d9d840b
---
doc/guix.texi | 6 +++---
gnu/services/backup.scm | 38 ++++++++++++++++++++------------------
2 files changed, 23 insertions(+), 21 deletions(-)

Toggle diff (145 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 32ce0c86b9..e4379c5e1c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41311,6 +41311,9 @@ Miscellaneous Services
Available @code{restic-backup-configuration} fields are:
@table @asis
+@item @code{restic} (default: @code{restic}) (type: package)
+The restic package to use for all jobs.
+
@item @code{jobs} (default: @code{'()}) (type: list-of-restic-backup-jobs)
The list of backup jobs for the current system.
@@ -41327,9 +41330,6 @@ Miscellaneous Services
Available @code{restic-backup-job} fields are:
@table @asis
-@item @code{restic} (default: @code{restic}) (type: package)
-The restic package to be used for the current job.
-
@item @code{user} (default: @code{"root"}) (type: string)
The user used for running the current job.
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index a6d8404a5a..a59cf08c71 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -29,10 +29,10 @@ (define-module (gnu services backup)
#:use-module (guix modules)
#:use-module (guix packages)
#:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
#:export (restic-backup-job
restic-backup-job?
restic-backup-job-fields
- restic-backup-job-restic
restic-backup-job-user
restic-backup-job-name
restic-backup-job-repository
@@ -46,6 +46,7 @@ (define-module (gnu services backup)
restic-backup-configuration
restic-backup-configuration?
+ restic-backup-configuration-restic
restic-backup-configuration-fields
restic-backup-configuration-jobs
@@ -71,9 +72,6 @@ (define-maybe/no-serialization string)
(define-maybe/no-serialization file-like)
(define-configuration/no-serialization restic-backup-job
- (restic
- (package restic)
- "The restic package to be used for the current job.")
(user
(string "root")
"The user used for running the current job.")
@@ -129,11 +127,14 @@ (define list-of-restic-backup-jobs?
(list-of restic-backup-job?))
(define-configuration/no-serialization restic-backup-configuration
+ (restic
+ (package restic)
+ "The restic package to be used.")
(jobs
(list-of-restic-backup-jobs '())
"The list of backup jobs for the current system."))
-(define (restic-backup-job-program config)
+(define (restic-backup-job-program restic-package config)
(define (maybe-value-or-false maybe)
(if (maybe-value-set? maybe)
maybe
@@ -143,7 +144,7 @@ (define (restic-backup-job-program config)
(verify-restic-backup-job-configuration config)
(let ((restic
- (file-append (restic-backup-job-restic config) "/bin/restic"))
+ (file-append restic-package "/bin/restic"))
(repository
(restic-backup-job-repository config))
(password-file
@@ -191,7 +192,7 @@ (define (restic-backup-job-program config)
#$@extra-flags
"backup" #$@files)))))
-(define (restic-guix jobs)
+(define (restic-guix restic-package jobs)
(program-file
"restic-guix"
#~(begin
@@ -199,7 +200,7 @@ (define (restic-guix jobs)
(srfi srfi-1))
(define names '#$(map restic-backup-job-name jobs))
- (define programs '#$(map restic-backup-job-program jobs))
+ (define programs '#$(map (cut restic-backup-job-program restic-package <>) jobs))
(define (get-program name)
(define idx
@@ -242,13 +243,13 @@ (define (restic-backup-job->mcron-job config)
#$(string-append "restic-guix backup " name)
#:user #$user)))
-(define (restic-guix-wrapper-package jobs)
+(define (restic-guix-wrapper-package restic jobs)
(let ((extra-packages (append-map restic-backup-job-extra-packages
jobs)))
(package
(name "restic-backup-service-wrapper")
(version "0.0.0")
- (source (restic-guix restic-package jobs))
+ (source (restic-guix restic jobs))
(build-system copy-build-system)
(arguments
(list #:install-plan #~'(("./" "/bin"))))
@@ -260,16 +261,17 @@ (define (restic-guix-wrapper-package jobs)
by the @code{restic-backup-service-type}. It allows for easily interacting
with Guix configured backup jobs, for example for manually triggering a backup
without waiting for the scheduled job to run.")
- (inputs extra-packages)
+ (inputs (cons restic extra-packages))
(license license:gpl3+))))
-(define restic-backup-service-profile
- (lambda (config)
- (define jobs (restic-backup-configuration-jobs config))
- (if (> (length jobs) 0)
- (list
- (restic-guix-wrapper-package jobs))
- '())))
+(define (restic-backup-service-profile config)
+ (let ((jobs (restic-backup-configuration-jobs config))
+ (restic (restic-backup-configuration-restic config)))
+ ;; Even if we provide a wrapper we need to add restic to the profile so
+ ;; the service works in containers and VMs which only see a subset of
+ ;; /gnu/store. https://issues.guix.gnu.org/70553. Ergo pass restic to
+ ;; the wrapper package so it is added as an input.
+ (list (restic-guix-wrapper-package restic jobs))))
(define restic-backup-service-type
(service-type (name 'restic-backup)
--
2.45.1
R
R
Richard Sent wrote on 19 Jun 00:08 +0200
[PATCH WIP 5/5] tests: Add restic system test.
(address . 71639@debbugs.gnu.org)
8efb248a6930d4e82d9f0e1e9cf77369c67780ca.1718747513.git.richard@freakingpenguin.com
* gnu/tests/restic.scm: Create.

Change-Id: Iad5472414c140b133d9b402855bb2f01e96bb0cc
---
gnu/tests/restic.scm | 124 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 124 insertions(+)
create mode 100644 gnu/tests/restic.scm

Toggle diff (132 lines)
diff --git a/gnu/tests/restic.scm b/gnu/tests/restic.scm
new file mode 100644
index 0000000000..f8ea76538e
--- /dev/null
+++ b/gnu/tests/restic.scm
@@ -0,0 +1,124 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Richard Sent <richard@freakingpenguin.com>.
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu tests restic)
+ ;; TODO: Remove what I can from here.
+ #:use-module (gnu bootloader)
+ #:use-module (gnu bootloader grub)
+ #:use-module (gnu packages)
+ #:use-module (gnu packages sync) ;rclone
+ #:use-module (gnu packages xorg)
+ #:use-module (gnu services)
+ #:use-module (gnu services desktop)
+ #:use-module (gnu services backup) ;restic
+ #:use-module (gnu services xorg)
+ #:use-module (gnu system)
+ #:use-module (gnu system vm)
+ #:use-module (gnu tests)
+ #:use-module (guix gexp)
+ #:use-module (guix modules)
+ #:use-module (srfi srfi-1)
+ #:export (%test-restic))
+
+(define password "password")
+
+(define password-file
+ (plain-file "password-file" password))
+
+(define password-command
+ (program-file "password-command" #~(display #$password)))
+
+(define (run-restic-test)
+ "Run tests in %restic-os."
+
+ (define os
+ (marionette-operating-system
+ (simple-operating-system (extra-special-file "/root/.restic-test"
+ (plain-file "restic-test"
+ "Hello world!"))
+ ;; restic-backup-service only takes a string to avoid putting
+ ;; plaintext entries in the store. Ergo, symlink it.
+ (extra-special-file "/root/password-file"
+ password-file)
+ (service restic-backup-service-type
+ (restic-backup-configuration
+ (jobs
+ (list (restic-backup-job
+ (name "password-file-backup")
+ (repository "/root/restic-password-file-repo")
+ (schedule #~'(next-second '(0 15 30 45)))
+ (password-file "/root/password-file")
+ (files '("/root/.restic-test"))
+ (init? #t))
+ (restic-backup-job
+ (name "password-command-backup")
+ (repository "/root/restic-password-command-repo")
+ (schedule #~'(next-second '(0 15 30 45)))
+ (password-command password-command)
+ (files '("/root/.restic-test"))
+ (init? #t)))))))
+ #:imported-modules '((gnu services herd)
+ (guix combinators))))
+
+ (define vm (virtual-machine
+ (operating-system os)
+ ;; TODO: We might be able to lower this.
+ (memory-size 1024)))
+
+ (define test
+ (with-imported-modules (source-module-closure
+ '((gnu build marionette)))
+ #~(begin
+ (use-modules (gnu build marionette)
+ (srfi srfi-26)
+ (srfi srfi-64))
+
+ (let ((marionette (make-marionette (list #$vm))))
+
+ (test-runner-current (system-test-runner #$output))
+ (test-begin "restic")
+
+ (test-assert "backup-file-created"
+ (wait-for-file "/root/.restic-test" marionette))
+
+ (test-assert "mcron running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (start-service 'mcron))
+ marionette))
+
+ (test-assert "password-file backup completed"
+ (wait-for-file "/root/restic-password-file-repo/config" marionette
+ ;; Restic takes a second to run, give it a bit
+ ;; more time.
+ #:timeout 20))
+
+ (test-assert "password-comand backup completed"
+ (wait-for-file "/root/restic-password-file-repo/config" marionette
+ #:timeout 20))
+
+ (test-end)))))
+
+ (gexp->derivation "restic-test" test))
+
+(define %test-restic
+ (system-test
+ (name "restic")
+ (description "Basic tests for the restic service.")
+ (value (run-restic-test))))
--
2.45.1
R
R
Richard Sent wrote on 20 Jun 05:44 +0200
[PATCH v2 0/5] Improve on restic-backup-service
(name . Richard Sent)(address . richard@freakingpenguin.com)
cover.1718854920.git.richard@freakingpenguin.com
Some minor adjustments to the WIP version, mostly cleanup and removing some
TODOs that snuck in. Thinking on it there's no need to worry about marking
this as WIP. If it winds up merged earlier than expected I can just submit
another patch series.

Richard Sent (5):
services: backup: Support bootstrapping an initial restic backup
services: backup: Add password-command support to restic-service
services: backup: Add extra-packages field to restic-backup-job
services: backup: Move restic package to restic-configuration
tests: Add restic system test.

doc/guix.texi | 21 ++++++-
gnu/services/backup.scm | 129 +++++++++++++++++++++++++++++-----------
gnu/tests/restic.scm | 119 ++++++++++++++++++++++++++++++++++++
3 files changed, 231 insertions(+), 38 deletions(-)
create mode 100644 gnu/tests/restic.scm


base-commit: e32e3d0a03dc17c4c54a91aad053c9036998b601
--
2.45.1
R
R
Richard Sent wrote on 20 Jun 05:44 +0200
[PATCH v2 1/5] services: backup: Support bootstrapping an initial restic backup
(name . Richard Sent)(address . richard@freakingpenguin.com)
93ff43dbc77df724597c7c27ae955581a31a851e.1718854920.git.richard@freakingpenguin.com
* gnu/services/backup.scm: (restic-backup-job): Add init? field.
(restic-backup-job-program): Initialize repository if init? is set and
repository does not already exist.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I71d0cbaac646b9d160e662b69286f229b9a9f64d
---
doc/guix.texi | 4 ++++
gnu/services/backup.scm | 19 +++++++++++++++++++
2 files changed, 23 insertions(+)

Toggle diff (74 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 0102fd0fad..63c9cbd1a7 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41353,6 +41353,10 @@ Miscellaneous Services
The list of files or directories to be backed up. It must be a list of
values that can be lowered to strings.
+@item @code{init?} (default: @code{#f}) (type: boolean)
+Whether restic-backup-service should check and (if it does not exist)
+initialize the repository before running the backup.
+
@item @code{verbose?} (default: @code{#f}) (type: boolean)
Whether to enable verbose output for the current backup job.
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index 555e9fc959..1279ece88f 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2024 Giacomo Leidi <goodoldpaul@autistici.org>
+;;; Copyright © 2024 Richard Sent <richard@freakingpenguin.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -38,6 +39,7 @@ (define-module (gnu services backup)
restic-backup-job-password-file
restic-backup-job-schedule
restic-backup-job-files
+ restic-backup-job-init?
restic-backup-job-verbose?
restic-backup-job-extra-flags
@@ -94,6 +96,9 @@ (define-configuration/no-serialization restic-backup-job
(verbose?
(boolean #f)
"Whether to enable verbose output for the current backup job.")
+ (init?
+ (boolean #f)
+ "Whether to attempt to initialize a new repository for automated bootstrap purposes.")
(extra-flags
(list-of-lowerables '())
"A list of values that are lowered to strings. These will be passed as
@@ -118,6 +123,8 @@ (define (restic-backup-job-program config)
(restic-backup-job-files config))
(extra-flags
(restic-backup-job-extra-flags config))
+ (init?
+ (restic-backup-job-init? config))
(verbose
(if (restic-backup-job-verbose? config)
'("--verbose")
@@ -130,6 +137,18 @@ (define (restic-backup-job-program config)
(setenv "RESTIC_PASSWORD"
(with-input-from-file #$password-file read-line))
+ (when #$init?
+ ;; Use cat config to check if the repository exists. See
+ ;; https://github.com/restic/restic/issues/1690 and
+ ;; https://github.com/NixOS/nixpkgs/pull/307962.
+ ;;
+ ;; XXX: restic returns values other than 1 on failure. Check
+ ;; EXIT_SUCCESS instead of EXIT_FAILURE.
+ (unless (or (equal? EXIT_SUCCESS (system* #$restic "cat" "config"
+ "-r" #$repository))
+ (equal? EXIT_SUCCESS (system* #$restic "init"
+ "-r" #$repository)))
+ (error "Failed to initialize restic repository: " #$repository)))
(execlp #$restic #$restic #$@verbose
"-r" #$repository
#$@extra-flags
--
2.45.1
R
R
Richard Sent wrote on 20 Jun 05:44 +0200
[PATCH v2 2/5] services: backup: Add password-command support to restic-service
(name . Richard Sent)(address . richard@freakingpenguin.com)
ceb1cc05b9c66b1f3d0ca59ba5548921d71cb11b.1718854920.git.richard@freakingpenguin.com
* gnu/services/backup.scm (restic-backup-job): Add password-command.
(verify-restic-backup-job-configuration): Create.
(restic-backup-job-program): Set either RESTIC_PASSWORD or
RESTIC_PASSWORD_COMMAND depending on what is configured.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: Ice9cf85d1ee4485a2737f515c63c969918219df0
---
doc/guix.texi | 7 +++++++
gnu/services/backup.scm | 41 ++++++++++++++++++++++++++++++++++++-----
2 files changed, 43 insertions(+), 5 deletions(-)

Toggle diff (108 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 63c9cbd1a7..f22d679023 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41344,6 +41344,13 @@ Miscellaneous Services
that will be used to set the @env{RESTIC_PASSWORD} environment variable
for the current job.
+@item @code{password-command} (type: file-like)
+String path or file-like object representing the executable file that
+prints password to stdout. If a file-like object is used, it is placed
+in the store globally executable and in plain text. The executable
+should be designed such that it does not compromise the password if an
+unauthorized user runs it.
+
@item @code{schedule} (type: gexp-or-string)
A string or a gexp that will be passed as time specification in the
mcron job specification (@pxref{Syntax, mcron job specifications,,
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index 1279ece88f..fd904bc9a9 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -66,6 +66,9 @@ (define (lowerable? value)
(define list-of-lowerables?
(list-of lowerable?))
+(define-maybe/no-serialization string)
+(define-maybe/no-serialization file-like)
+
(define-configuration/no-serialization restic-backup-job
(restic
(package restic)
@@ -80,10 +83,16 @@ (define-configuration/no-serialization restic-backup-job
(string)
"The restic repository target of this job.")
(password-file
- (string)
+ (maybe-string)
"Name of the password file, readable by the configured @code{user}, that
will be used to set the @code{RESTIC_PASSWORD} environment variable for the
current job.")
+ (password-command
+ (maybe-file-like)
+ "An executable file who's path is stored in @code{RESTIC_PASSWORD_COMMAND}.
+When run, the file writes the password to standard output. Due to the nature
+of the store this command will be globally executable and should have external
+protections to ensure unauthorized users cannot retrieve the password.")
(schedule
(gexp-or-string)
"A string or a gexp that will be passed as time specification in the mcron
@@ -104,6 +113,14 @@ (define-configuration/no-serialization restic-backup-job
"A list of values that are lowered to strings. These will be passed as
command-line arguments to the current job @command{restic backup} invokation."))
+(define (verify-restic-backup-job-configuration config)
+ (unless (or (maybe-value-set? (restic-backup-job-password-file config))
+ (maybe-value-set? (restic-backup-job-password-command config)))
+ (error "either password-file or password-command must be configured."))
+ (when (and (maybe-value-set? (restic-backup-job-password-file config))
+ (maybe-value-set? (restic-backup-job-password-command config)))
+ (error "password-file and password-command can not be configured simultaneously.")))
+
(define list-of-restic-backup-jobs?
(list-of restic-backup-job?))
@@ -113,12 +130,21 @@ (define-configuration/no-serialization restic-backup-configuration
"The list of backup jobs for the current system."))
(define (restic-backup-job-program config)
+ (define (maybe-value-or-false maybe)
+ (if (maybe-value-set? maybe)
+ maybe
+ #f))
+
+ (verify-restic-backup-job-configuration config)
+
(let ((restic
(file-append (restic-backup-job-restic config) "/bin/restic"))
(repository
(restic-backup-job-repository config))
(password-file
- (restic-backup-job-password-file config))
+ (maybe-value-or-false (restic-backup-job-password-file config)))
+ (password-command
+ (maybe-value-or-false (restic-backup-job-password-command config)))
(files
(restic-backup-job-files config))
(extra-flags
@@ -134,9 +160,14 @@ (define (restic-backup-job-program config)
#~(begin
(use-modules (ice-9 popen)
(ice-9 rdelim))
- (setenv "RESTIC_PASSWORD"
- (with-input-from-file #$password-file read-line))
-
+ (or (and=> #$password-file (lambda (x)
+ (setenv "RESTIC_PASSWORD"
+ (with-input-from-file x read-line))))
+ (and=> #$password-command (lambda (x)
+ (setenv "RESTIC_PASSWORD_COMMAND" x)))
+ ;; Have a backup error message in case
+ ;; verify-restic-backup-job-configuration is messed with
+ (error "Neither password-file or password-command set"))
(when #$init?
;; Use cat config to check if the repository exists. See
;; https://github.com/restic/restic/issues/1690 and
--
2.45.1
R
R
Richard Sent wrote on 20 Jun 05:44 +0200
[PATCH v2 4/5] services: backup: Move restic package to restic-configuration
(name . Richard Sent)(address . richard@freakingpenguin.com)
56345eb5926246798f52424883caa3fd0e88cee8.1718854920.git.richard@freakingpenguin.com
* gnu/services/backup.scm (restic-backup-configuration): Add restic field.
(restic-backup-job-program): Add restic package to function signature.
(restic-guix): Ditto.
(restic-guix-wrapper-package): Ditto. Add restic package as package input.
(restic-backup-service-profile): Remove excess lamba.

* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I1e5f63c21cd072354225afe0ee270dca8d9d840b
---
doc/guix.texi | 6 +++---
gnu/services/backup.scm | 38 ++++++++++++++++++++------------------
2 files changed, 23 insertions(+), 21 deletions(-)

Toggle diff (145 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 32ce0c86b9..e4379c5e1c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41311,6 +41311,9 @@ Miscellaneous Services
Available @code{restic-backup-configuration} fields are:
@table @asis
+@item @code{restic} (default: @code{restic}) (type: package)
+The restic package to use for all jobs.
+
@item @code{jobs} (default: @code{'()}) (type: list-of-restic-backup-jobs)
The list of backup jobs for the current system.
@@ -41327,9 +41330,6 @@ Miscellaneous Services
Available @code{restic-backup-job} fields are:
@table @asis
-@item @code{restic} (default: @code{restic}) (type: package)
-The restic package to be used for the current job.
-
@item @code{user} (default: @code{"root"}) (type: string)
The user used for running the current job.
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index bfbacfe590..5e2add088e 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -29,10 +29,10 @@ (define-module (gnu services backup)
#:use-module (guix modules)
#:use-module (guix packages)
#:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
#:export (restic-backup-job
restic-backup-job?
restic-backup-job-fields
- restic-backup-job-restic
restic-backup-job-user
restic-backup-job-name
restic-backup-job-repository
@@ -46,6 +46,7 @@ (define-module (gnu services backup)
restic-backup-configuration
restic-backup-configuration?
+ restic-backup-configuration-restic
restic-backup-configuration-fields
restic-backup-configuration-jobs
@@ -71,9 +72,6 @@ (define-maybe/no-serialization string)
(define-maybe/no-serialization file-like)
(define-configuration/no-serialization restic-backup-job
- (restic
- (package restic)
- "The restic package to be used for the current job.")
(user
(string "root")
"The user used for running the current job.")
@@ -129,11 +127,14 @@ (define list-of-restic-backup-jobs?
(list-of restic-backup-job?))
(define-configuration/no-serialization restic-backup-configuration
+ (restic
+ (package restic)
+ "The restic package to be used.")
(jobs
(list-of-restic-backup-jobs '())
"The list of backup jobs for the current system."))
-(define (restic-backup-job-program config)
+(define (restic-backup-job-program restic-package config)
(define (maybe-value-or-false maybe)
(if (maybe-value-set? maybe)
maybe
@@ -142,7 +143,7 @@ (define (restic-backup-job-program config)
(verify-restic-backup-job-configuration config)
(let ((restic
- (file-append (restic-backup-job-restic config) "/bin/restic"))
+ (file-append restic-package "/bin/restic"))
(repository
(restic-backup-job-repository config))
(password-file
@@ -189,7 +190,7 @@ (define (restic-backup-job-program config)
#$@extra-flags
"backup" #$@files)))))
-(define (restic-guix jobs)
+(define (restic-guix restic-package jobs)
(program-file
"restic-guix"
#~(begin
@@ -197,7 +198,7 @@ (define (restic-guix jobs)
(srfi srfi-1))
(define names '#$(map restic-backup-job-name jobs))
- (define programs '#$(map restic-backup-job-program jobs))
+ (define programs '#$(map (cut restic-backup-job-program restic-package <>) jobs))
(define (get-program name)
(define idx
@@ -240,13 +241,13 @@ (define (restic-backup-job->mcron-job config)
#$(string-append "restic-guix backup " name)
#:user #$user)))
-(define (restic-guix-wrapper-package jobs)
+(define (restic-guix-wrapper-package restic jobs)
(let ((extra-packages (append-map restic-backup-job-extra-packages
jobs)))
(package
(name "restic-backup-service-wrapper")
(version "0.0.0")
- (source (restic-guix restic-package jobs))
+ (source (restic-guix restic jobs))
(build-system copy-build-system)
(arguments
(list #:install-plan #~'(("./" "/bin"))))
@@ -258,16 +259,17 @@ (define (restic-guix-wrapper-package jobs)
by the @code{restic-backup-service-type}. It allows for easily interacting
with Guix configured backup jobs, for example for manually triggering a backup
without waiting for the scheduled job to run.")
- (inputs extra-packages)
+ (inputs (cons restic extra-packages))
(license license:gpl3+))))
-(define restic-backup-service-profile
- (lambda (config)
- (define jobs (restic-backup-configuration-jobs config))
- (if (> (length jobs) 0)
- (list
- (restic-guix-wrapper-package jobs))
- '())))
+(define (restic-backup-service-profile config)
+ (let ((jobs (restic-backup-configuration-jobs config))
+ (restic (restic-backup-configuration-restic config)))
+ ;; Even if we provide a wrapper we need to add restic to the profile so
+ ;; the service works in containers and VMs which only see a subset of
+ ;; /gnu/store. https://issues.guix.gnu.org/70553. Ergo pass restic to
+ ;; the wrapper package so it is added as an input.
+ (list (restic-guix-wrapper-package restic jobs))))
(define restic-backup-service-type
(service-type (name 'restic-backup)
--
2.45.1
R
R
Richard Sent wrote on 20 Jun 05:44 +0200
[PATCH v2 3/5] services: backup: Add extra-packages field to restic-backup-job
(name . Richard Sent)(address . richard@freakingpenguin.com)
b0253ce5f357e984716314a191a3f55380455e66.1718854920.git.richard@freakingpenguin.com
* gnu/services/backup.scm (restic-backup-job): Create extra-packages.
(restic-guix-wrapper-package): Add the extra-packages field of every job as
inputs.
* doc/guix.texi (Miscellaneous Services): Document it.

Change-Id: I4f0b070bc6dc895553ba69256d14e45898291c02
---
doc/guix.texi | 4 ++++
gnu/services/backup.scm | 33 ++++++++++++++++++++-------------
2 files changed, 24 insertions(+), 13 deletions(-)

Toggle diff (78 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index f22d679023..32ce0c86b9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -41367,6 +41367,10 @@ Miscellaneous Services
@item @code{verbose?} (default: @code{#f}) (type: boolean)
Whether to enable verbose output for the current backup job.
+@item @code{extra-packages} (default: @code{'()} (type: list-of-packages)
+The list of extra packages needed for restic to run the current backup
+job.
+
@item @code{extra-flags} (default: @code{'()}) (type: list-of-lowerables)
A list of values that are lowered to strings. These will be passed as
command-line arguments to the current job @command{restic backup}
diff --git a/gnu/services/backup.scm b/gnu/services/backup.scm
index fd904bc9a9..bfbacfe590 100644
--- a/gnu/services/backup.scm
+++ b/gnu/services/backup.scm
@@ -41,6 +41,7 @@ (define-module (gnu services backup)
restic-backup-job-files
restic-backup-job-init?
restic-backup-job-verbose?
+ restic-backup-job-extra-packages
restic-backup-job-extra-flags
restic-backup-configuration
@@ -102,6 +103,9 @@ (define-configuration/no-serialization restic-backup-job
(list-of-lowerables '())
"The list of files or directories to be backed up. It must be a list of
values that can be lowered to strings.")
+ (extra-packages
+ (list-of-packages '())
+ "The list of extra packages needed for restic to run this backup job.")
(verbose?
(boolean #f)
"Whether to enable verbose output for the current backup job.")
@@ -237,22 +241,25 @@ (define (restic-backup-job->mcron-job config)
#:user #$user)))
(define (restic-guix-wrapper-package jobs)
- (package
- (name "restic-backup-service-wrapper")
- (version "0.0.0")
- (source (restic-guix jobs))
- (build-system copy-build-system)
- (arguments
- (list #:install-plan #~'(("./" "/bin"))))
- (home-page "https://restic.net")
- (synopsis
- "Easily interact from the CLI with Guix configured backups")
- (description
- "This package provides a simple wrapper around @code{restic}, handled
+ (let ((extra-packages (append-map restic-backup-job-extra-packages
+ jobs)))
+ (package
+ (name "restic-backup-service-wrapper")
+ (version "0.0.0")
+ (source (restic-guix restic-package jobs))
+ (build-system copy-build-system)
+ (arguments
+ (list #:install-plan #~'(("./" "/bin"))))
+ (home-page "https://restic.net")
+ (synopsis
+ "Easily interact from the CLI with Guix configured backups")
+ (description
+ "This package provides a simple wrapper around @code{restic}, handled
by the @code{restic-backup-service-type}. It allows for easily interacting
with Guix configured backup jobs, for example for manually triggering a backup
without waiting for the scheduled job to run.")
- (license license:gpl3+)))
+ (inputs extra-packages)
+ (license license:gpl3+))))
(define restic-backup-service-profile
(lambda (config)
--
2.45.1
R
R
Richard Sent wrote on 20 Jun 05:44 +0200
[PATCH v2 5/5] tests: Add restic system test.
(name . Richard Sent)(address . richard@freakingpenguin.com)
e6e444a7e2a4ec32fca27a5098070947974abcd4.1718854920.git.richard@freakingpenguin.com
* gnu/tests/restic.scm: Create.

Change-Id: Iad5472414c140b133d9b402855bb2f01e96bb0cc
---
gnu/tests/restic.scm | 119 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 119 insertions(+)
create mode 100644 gnu/tests/restic.scm

Toggle diff (127 lines)
diff --git a/gnu/tests/restic.scm b/gnu/tests/restic.scm
new file mode 100644
index 0000000000..8d29ff441b
--- /dev/null
+++ b/gnu/tests/restic.scm
@@ -0,0 +1,119 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Richard Sent <richard@freakingpenguin.com>.
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu tests restic)
+ #:use-module (gnu bootloader)
+ #:use-module (gnu bootloader grub)
+ #:use-module (gnu packages)
+ #:use-module (gnu packages sync) ;rclone
+ #:use-module (gnu services)
+ #:use-module (gnu services backup) ;restic
+ #:use-module (gnu system)
+ #:use-module (gnu system vm)
+ #:use-module (gnu tests)
+ #:use-module (guix gexp)
+ #:use-module (guix modules)
+ #:use-module (srfi srfi-1)
+ #:export (%test-restic))
+
+(define password "password")
+
+(define password-file
+ (plain-file "password-file" password))
+
+(define password-command
+ (program-file "password-command" #~(display #$password)))
+
+(define (run-restic-test)
+ "Run tests in %restic-os."
+
+ (define os
+ (marionette-operating-system
+ (simple-operating-system (extra-special-file "/root/.restic-test"
+ (plain-file "restic-test"
+ "Hello world!"))
+ ;; restic-backup-service only takes a string to avoid putting
+ ;; plaintext entries in the store. Ergo, symlink it.
+ (extra-special-file "/root/password-file"
+ password-file)
+ (service restic-backup-service-type
+ (restic-backup-configuration
+ (jobs
+ (list (restic-backup-job
+ (name "password-file-backup")
+ (repository "/root/restic-password-file-repo")
+ (schedule #~'(next-second '(0 15 30 45)))
+ (password-file "/root/password-file")
+ (files '("/root/.restic-test"))
+ (init? #t))
+ (restic-backup-job
+ (name "password-command-backup")
+ (repository "/root/restic-password-command-repo")
+ (schedule #~'(next-second '(0 15 30 45)))
+ (password-command password-command)
+ (files '("/root/.restic-test"))
+ (init? #t)))))))
+ #:imported-modules '((gnu services herd)
+ (guix combinators))))
+
+ (define vm (virtual-machine
+ (operating-system os)
+ (memory-size 512)))
+
+ (define test
+ (with-imported-modules (source-module-closure
+ '((gnu build marionette)))
+ #~(begin
+ (use-modules (gnu build marionette)
+ (srfi srfi-26)
+ (srfi srfi-64))
+
+ (let ((marionette (make-marionette (list #$vm))))
+
+ (test-runner-current (system-test-runner #$output))
+ (test-begin "restic")
+
+ (test-assert "backup-file-created"
+ (wait-for-file "/root/.restic-test" marionette))
+
+ (test-assert "mcron running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (start-service 'mcron))
+ marionette))
+
+ (test-assert "password-file backup completed"
+ (wait-for-file "/root/restic-password-file-repo/config" marionette
+ ;; Restic takes a second to run, give it a bit
+ ;; more time.
+ #:timeout 20))
+
+ (test-assert "password-comand backup completed"
+ (wait-for-file "/root/restic-password-file-repo/config" marionette
+ #:timeout 20))
+
+ (test-end)))))
+
+ (gexp->derivation "restic-test" test))
+
+(define %test-restic
+ (system-test
+ (name "restic")
+ (description "Basic tests for the restic service.")
+ (value (run-restic-test))))
--
2.45.1
P
P
paul wrote on 25 Jun 00:49 +0200
Re: [PATCHv2 0/5] Improve on restic-backup-service
(address . 71639@debbugs.gnu.org)(name . Richard Sent)(address . richard@freakingpenguin.com)
c4483a64-abf2-62d4-78a2-05c6bf349857@autistici.org
Hi Richard,

thank you for this work, it looks really useful and in general I think
the patches are ok but I have some points I'd like to discuss.

I think it would be nice to have an init action for the restic-guix
command shipped with the service and move there the logic for
initializing repositories. The service then could be adapted in a way
that it would call the restic-guix init before calling restic-guix
backup. Would you be interested in implementing this?

At last, my use case for having a restic package field for each job is
to have some critical jobs that I don't want to touch running with
Guix's bootstrapped restic package and some personal jobs that I run
with a restic 0.16 binary package I have in my personal channel . I'm
not sure this warrants a field in each single job, but they are optional
anyway. Anyway I wouldn't consider this a blocker and if the Guix
project has some guidelines in this sense I'd say follow them.

Cheers,

giacomo
R
R
Richard Sent wrote on 27 Jun 05:56 +0200
(name . paul)(address . goodoldpaul@autistici.org)(address . 71639@debbugs.gnu.org)
87ikxvvy0o.fsf@freakingpenguin.com
Hi giacomo,

Toggle quote (6 lines)
> I think it would be nice to have an init action for the restic-guix
> command shipped with the service and move there the logic for
> initializing repositories. The service then could be adapted in a way
> that it would call the restic-guix init before calling restic-guix
> backup. Would you be interested in implementing this?

I'll take a look at it and see what the code looks like. It might be a
bit of effort to get that working cleanly while avoiding code dupe. For
instance, setting RESTIC_PASSWORD(_COMMAND) in both init and backup
program actions.

Where do you think the appropriate place to check init? and run the init
action is if it's no longer encapsulated in the backup action? A
conditional in restic-backup-job->mcron-job before launching restic-guix
backup? A one-shot shepherd service?

The former will necessitate adding a job string to display when running
$ herd schedule mcron. A downside of the latter is if the restic
repository is deleted for one reason or another that backup job will
fail until the system is rebooted, which isn't immediately obvious.
Personally I prefer the mcron-job conditional.

Toggle quote (8 lines)
> At last, my use case for having a restic package field for each job is
> to have some critical jobs that I don't want to touch running with
> Guix's bootstrapped restic package and some personal jobs that I run
> with a restic 0.16 binary package I have in my personal channel . I'm
> not sure this warrants a field in each single job, but they are optional
> anyway. Anyway I wouldn't consider this a blocker and if the Guix
> project has some guidelines in this sense I'd say follow them.

Ah, I see. To me this is where a per-job restic-override or similarly
named field makes sense. This way we can have a default "global" restic
package configured at a service level while still allowing individual
jobs to use a custom restic package. I imagine this restic-override
field would be a "maybe-file-like" either set to #f or a custom restic
package.

Thanks for the feedback!

--
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.
P
P
paul wrote on 7 Jul 22:40 +0200
(name . Richard Sent)(address . richard@freakingpenguin.com)(address . 71639@debbugs.gnu.org)
18da4124-465c-fa58-6e8f-d9daf751443c@autistici.org
Hi Richard,

On 6/27/24 05:56, Richard Sent wrote:
Toggle quote (4 lines)
> I'll take a look at it and see what the code looks like. It might be a
> bit of effort to get that working cleanly while avoiding code dupe. For
> instance, setting RESTIC_PASSWORD(_COMMAND) in both init and backup
> program actions.
Yes, 100% agree. What I would do is (and probably there is a better way):

- rename restic-backup-job-program to restic-action-job-program or
something similar. the procedure would be the same but it would take two
additional arguments: an action-name argument or similar that would
replace all instances of "backup" and I think this new procedure should
not access the files field of the job record but it instead it would
take an action-arguments argument which would replace #$@files with
#$@action-arguments in the execlp call. This way, I hope, it could be
general enough to support additional actions besides init in the future,
such as purge or restore.

- create a new restic-backup-job-program and a restic-init-job-program
or similar procedures that would call the new generic
restic-action-job-program

- inside the restic-guix procedure I'd rename the programs list to
backup-programs and I would create a similar list for the init action
with #f placeholders for jobs that do not require initializing, and I
would check the conditional inside the backup procedure something like:

(define (get-program name programs)
[...])

(define (init args)
[...])

(define (backup args)
(define name (third args))
(define backup-program (get-program name backup-programs))
(define init-program (get-program name init-programs))
(when init-program
(init args))
(execlp backup-program backup-program))


this requires the init action not to exec the script resulting from the
gexp but I can't think of a reason why it could be problematic right now
so I guess it is ok.

Please take all of the above as a proposal there are for sure things to
make better.

Toggle quote (2 lines)
> Thanks for the feedback!

Thank you very much for your work,

giacomo
Attachment: file
F
F
Fabio Natali wrote on 22 Aug 19:43 +0200
Re: [bug#71639] [PATCHv2 0/5] Improve on restic-backup-service
(address . 71639@debbugs.gnu.org)
87wmk8h2qw.fsf@fabionatali.com
Hi Giacomo,

Thanks for adding the Restic Service to Guix.

Hi Richard,

Thanks for patch request 71639, which includes a few changes I'm looking
forward to. For instance, I like the idea of exposing the
'RESTIC_PASSWORD_COMMAND' variable directly via the service
configuration.

I think I might still be able to use the 'extra-flags' field to specify
a '--password-command' - if I understand it correctly - but direct
access to the option would be neat indeed.

Anything I can do to help with the patch? Could it make sense to break
the patch down into a couple of stages, possibly leaving the most
impactful changes for a later stage?

Thanks, cheers, Fabio.
P
P
paul wrote on 25 Aug 16:12 +0200
Re: [PATCHv2 0/5] Improve on restic-backup-service
(address . 71639@debbugs.gnu.org)
a576c917-2242-ff16-b95e-e789cf7b07ea@autistici.org
Dear Fabio and Richard,

I apologize for blocking this issue with my proposal. I think Fabio's
proposal of splitting the changes in this patch is the best way forward,
if you agree as well Richard. I submitted [0] to refactor the
restic-guix procedure in a way that it can support many different commands.

After it gets in it should be sufficient to add "init" to
%restic-guix-supported-actions to have a working restic-guix init
invokation. it should be then matter of understanding where it is better
to put it. Lately I was thinking that may be best to have initialization
as a one shot Shepherd service that check whether a given job is
supposed to have its repository initialized and if that's the case it
could run restic-guix init name-of the job. Please Richard let me know
what you think of this approach and whether you would still be
interested in implementing it, thank you very much!

I think the following subdivision should match all requirements we
stated until now:

1. Improve the restic backup system service . This can be done in the
current issue #71639:

services: backup: Add password-command support to restic-service
services: backup: Add extra-packages field to restic-backup-job
services: backup: Move restic package to restic-configuration

2. Refactor the restic-guix function to allow for more restic commands
to be wrapped. This can be done in issue #72803:

services: restic-backup: Add more restic commands to the restic-guix
package.

3. Allow for repositories to be initialized with a restic-guix init
command. This commit could be adapted and moved to a new branch based on
#72803:

services: backup: Support bootstrapping an initial restic backup

What do you think? Could this be a suitable action plan?

Thank you very much for your work,

giacomo


Attachment: file
F
F
Fabio Natali wrote on 3 Sep 18:43 +0200
Re: [PATCH WIP 0/5] Improve on restic-backup-service
(address . 71639@debbugs.gnu.org)
87seugelig.fsf@fabionatali.com
Toggle quote (6 lines)
> I apologize for blocking this issue with my proposal. I think Fabio's
> proposal of splitting the changes in this patch is the best way
> forward, if you agree as well Richard. I submitted [0] to refactor the
> restic-guix procedure in a way that it can support many different
> commands.

Hi Giacomo,

Thanks for this!

Toggle quote (6 lines)
> 1. Improve the restic backup system service [...]

> 2. Refactor the restic-guix function [...]

> 3. Allow for repositories to be initialized with a restic-guix init [...]

I think this is a perfectly sensible plan. For what it's worth, I've
provided some little feedback around point 2 as part of #72803.

As a further point, I also think we should manage to update Restic
itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov
2019). With such an out-of-date package, the service also risks to loose
a little bit of its value.

I've just sent a quick follow-up to #63019 in the hope that we might be
able to get this sorted soon.

Thanks, cheers, Fabio.
R
R
Richard Sent wrote on 4 Sep 17:49 +0200
Re: [bug#71639] [PATCH WIP 0/5] Improve on restic-backup-service
(name . Fabio Natali)(address . me@fabionatali.com)
87wmjrqv0k.fsf@freakingpenguin.com
Hi Giacomo and Fabio!

I've been struck by the the law of development: the backlog will always
expand to exceed the time available.

Toggle quote (6 lines)
>> I apologize for blocking this issue with my proposal. I think Fabio's
>> proposal of splitting the changes in this patch is the best way
>> forward, if you agree as well Richard. I submitted [0] to refactor the
>> restic-guix procedure in a way that it can support many different
>> commands.

That sounds fine with me. I'll try to rebase and resubmit as time is
available (hopefully the next few days......). If my schedule is
inconvenient feel free to do with my patches what you will.

Toggle quote (5 lines)
>> 1. Improve the restic backup system service [...]
>> 2. Refactor the restic-guix function [...]
>> 3. Allow for repositories to be initialized with a restic-guix init
>> [...]

Ideally, we'd also move the system tests to 1 so further work can be
trivially verified by running $ make check-system TESTS=restic. That'll
require writing an intermediary bootstrapping function in the test
suite.

Toggle quote (8 lines)
> I think this is a perfectly sensible plan. For what it's worth, I've
> provided some little feedback around point 2 as part of #72803.
>
> As a further point, I also think we should manage to update Restic
> itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov
> 2019). With such an out-of-date package, the service also risks to loose
> a little bit of its value.

That would be lovely! One particular feature I'm hopeful for is
compression support.

Unfortunately restic stopped providing a vendor directory in their
tarball after 0.9.6 and seemingly has no interest in providing it again
[1]. Last time I tried guix import with restic I saw some odd behavior.
Someone with more experience in Go may have better luck.

Toggle quote (5 lines)
>> Lately I was thinking that may be best to have initialization as a
>> one shot Shepherd service that check whether a given job is supposed
>> to have its repository initialized and if that's the case it could
>> run restic-guix init name-of the job.

Is the intent to have one "mega" one-shot service that checks every job,
or a unique one-shot service per job? I greatly prefer the former as $
sudo herd status is crowded as it is.

It's worth noting users will need the ability to specify custom Shepherd
requirements so e.g. backup initialization to a remote file-share isn't
triggered until the file-system-/my/remote/nas symbol is provisioned by
Shepherd.

On an unrelated note, Shepherd timers are interesting and may allow for
better user management/error signaling instead of mcron. I'm not
proposing we refactor to use them now (especially since they're not even
in stable), but just thought I'd mention them. [2]


--
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.
F
F
Fabio Natali wrote on 5 Sep 12:06 +0200
Re: [PATCH WIP 0/5] Improve on restic-backup-service
(address . 71639@debbugs.gnu.org)
87plpict3u.fsf@fabionatali.com
Hi Richard and Giacomo.

Toggle quote (1 lines)
> >> 1. Improve the restic backup system service [...]
[...]
Toggle quote (5 lines)
> Ideally, we'd also move the system tests to 1 so further work can be
> trivially verified by running $ make check-system
> TESTS=restic. That'll require writing an intermediary bootstrapping
> function in the test suite.

+1

Toggle quote (15 lines)
> > As a further point, I also think we should manage to update Restic
> > itself to 0.17.0 (while we're still at 0.9.6 which was released in Nov
> > 2019). With such an out-of-date package, the service also risks to loose
> > a little bit of its value.
>
> That would be lovely! One particular feature I'm hopeful for is
> compression support.
>
> Unfortunately restic stopped providing a vendor directory in their
> tarball after 0.9.6 and seemingly has no interest in providing it
> again [1]. Last time I tried guix import with restic I saw some odd
> behavior. Someone with more experience in Go may have better luck.

> [1]: https://github.com/restic/restic/issues/3945

Thanks for the link to the Restic's issue tracker. I think it's actually
better to have the dependencies unbundled and built/imported
individually - although this is causing us some pain at the moment.

Some of the 'guix import go --recursive restic' errors seem to be caused
by Go projects using a slightly different subfolder scheme? There seem
to be various open issues (and semi-finalised patches) around this, such
as:


I'll follow up on one of the threads above and report back here if I
find anything useful.

Thanks, cheers, Fabio.
P
P
paul wrote 47 hours ago
(name . Richard Sent)(address . richard@freakingpenguin.com)
e78634e4-8d8b-4774-9851-e692eee648a1@autistici.org
Hi Richard and Fabio,

With Shepherd 1.0 we got new shiny timers and I just sent a patch [0] to
migrate the restic-backup-service type to Shepherd timers. It shouldn't
break any existing backup (provided a reboot is done after
configuration) AFAIU. If you don't have objections, I would update the
plan this way:

1. Implement as a Shepherd timer, with issue#75045

2. Improve the restic backup system service . This can be done in the
current issue #71639:

services: backup: Add password-command support to restic-service
services: backup: Add extra-packages field to restic-backup-job
services: backup: Move restic package to restic-configuration

3. Refactor the restic-guix function to allow for more restic commands
to be wrapped. This can be done in issue #72803:

services: restic-backup: Add more restic commands to the restic-guix
package.

4. Allow for repositories to be initialized with a restic-guix init
command. This commit could be adapted and moved to a new branch based on
#72803:

services: backup: Support bootstrapping an initial restic backup




What do you think?



cheers,

giacomo


Attachment: file
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 71639@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 71639
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch