[WIP RFC] services: Add file system monitor service.

  • Open
  • quality assurance status badge
Details
4 participants
  • Clément Lassieur
  • Danny Milosavljevic
  • Ludovic Courtès
  • swedebugia
Owner
unassigned
Submitted by
Danny Milosavljevic
Severity
normal
D
D
Danny Milosavljevic wrote on 5 Nov 2018 04:18
(address . guix-patches@gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20181105031833.7917-1-dannym@scratchpost.org
* gnu/services/monitoring.scm (file-system-monitoring-configuration): New
variable.
(file-system-monitoring-service-type): New variable.
* gnu/tests/monitoring.scm (%test-file-system-monitoring): New variable.
* doc/guix.texi (File System Monitoring Service): New subsubsection.
---
doc/guix.texi | 37 +++++++++++++++
gnu/services/monitoring.scm | 65 +++++++++++++++++++++++++-
gnu/tests/monitoring.scm | 92 ++++++++++++++++++++++++++++++++++++-
3 files changed, 192 insertions(+), 2 deletions(-)

Toggle diff (241 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 3b7fa50d8..8997a0915 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -21496,6 +21496,43 @@ The following is an example @code{dicod-service} configuration.
%dicod-database:gcide))))
@end example
+@cindex file system monitoring
+@subsubheading File System Monitoring Service
+
+The @code{(gnu services monitoring)} module provides a shepherd service to
+monitor file system changes and call a handler procedure on changes.
+
+@defvr {Scheme Variable} file-system-monitoring-service-type
+The service type for @command{fswatch}, which provides the file system
+monitoring capability.
+
+@example
+(service file-system-monitoring-service-type
+ (file-system-monitoring-configuration
+ (monitored-files '("/foo/bar"))
+ (handlers '((lambda args
+ (display "UH OH\n"))))))
+@end example
+@end defvr
+
+@deftp {Data Type} file-system-monitoring-configuration
+The data type representing the configuration of the file-system-monitoring
+service.
+
+@table @asis
+@item @code{package}
+Package containing the actual file system monitor (fswatch).
+
+@item @code{monitored-files}
+List of files to be monitored.
+
+@item @code{handlers}
+List of thunks which will be called once the file system monitor noticed
+changes in the monitored files.
+
+@end table
+@end deftp
+
@node Setuid Programs
@subsection Setuid Programs
diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index aa3b63a0e..94320c7d2 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -26,10 +26,13 @@
#:use-module (guix gexp)
#:use-module (guix records)
#:use-module (ice-9 match)
+ #:use-module (srfi srfi-1)
#:export (darkstat-configuration
prometheus-node-exporter-configuration
darkstat-service-type
- prometheus-node-exporter-service-type))
+ prometheus-node-exporter-service-type
+ file-system-monitoring-configuration
+ file-system-monitoring-service-type))
;;;
@@ -125,3 +128,63 @@ prometheus.")
(list (service-extension
shepherd-root-service-type
(compose list prometheus-node-exporter-shepherd-service))))))
+
+
+;;;
+;;; File System Monitoring
+;;;
+
+(define-record-type* <file-system-monitoring-configuration>
+ file-system-monitoring-configuration
+ make-file-system-monitoring-configuration
+ file-system-monitoring-configuration?
+ (package file-system-monitoring-configuration-package
+ (default fswatch))
+ (monitored-files file-system-monitoring-configuration-monitored-files
+ (default '()))
+ ;; Note: These thunks are called in the shepherd environment.
+ (handlers file-system-monitoring-configuration-handlers
+ (default '())))
+
+(define file-system-monitoring-shepherd-services
+ (match-lambda
+ (($ <file-system-monitoring-configuration> package monitored-files handlers)
+ (list (shepherd-service
+ (provision '(file-system-monitoring))
+ (documentation "File System Monitor")
+ (requirement '(file-systems))
+ (start #~(lambda ()
+ (sleep 1)
+ (for-each (lambda (handler)
+ (handler))
+ handlers)
+ (fork+exec-command
+ `(#$(file-append package "/bin/fswatch")
+ "--one-event"
+ "-l" "1" ; latency: 1 s
+ ; "-d"
+ "--"
+ #$@(if monitored-files
+ monitored-files
+ '())))))
+ (stop #~(make-kill-destructor))
+ (respawn? #t))))))
+
+(define file-system-monitoring-service-type
+ (service-type (name 'monitor-file-system)
+ (extensions
+ (list (service-extension shepherd-root-service-type
+ file-system-monitoring-shepherd-services)))
+ (compose concatenate)
+ (extend (lambda (config monitored-files handlers)
+ (match config
+ (($ <file-system-monitoring-configuration> package initial-monitored-files initial-handlers)
+ (file-system-monitoring-configuration
+ (package package)
+ (monitored-files (append initial-monitored-files monitored-files))
+ (handlers (append initial-handlers handlers))
+)))))
+ (description
+ "Call all @dfn{handlers} once something happens with one of
+the files monitored, and on overflow. Can have false positives. Will call
+all @dfn{handlers} on startup.")))
diff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scm
index 3320a19a7..95fbc7906 100644
--- a/gnu/tests/monitoring.scm
+++ b/gnu/tests/monitoring.scm
@@ -20,10 +20,12 @@
#:use-module (gnu services)
#:use-module (gnu services monitoring)
#:use-module (gnu services networking)
+ ;#:use-module (gnu system)
#:use-module (gnu system vm)
#:use-module (gnu tests)
#:use-module (guix gexp)
- #:export (%test-prometheus-node-exporter))
+ #:export (%test-prometheus-node-exporter
+ %test-file-system-monitoring))
;;;
@@ -95,3 +97,91 @@
(description "Connect to a running prometheus-node-exporter server.")
(value (run-prometheus-node-exporter-server-test
name %prometheus-node-exporter-os))))
+
+
+;;;
+;;; File System Monitoring.
+;;;
+
+(define* (run-file-system-monitoring-test name test-os)
+ "Run tests in TEST-OS, which has file system monitoring running."
+ (define os
+ (marionette-operating-system
+ test-os
+ #:imported-modules '((gnu services herd))))
+
+ (define vm
+ (virtual-machine
+ (operating-system os)))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (srfi srfi-11)
+ (srfi srfi-64)
+ (gnu build marionette)
+ (web client)
+ (web response))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin #$name)
+
+ (test-assert "file system monitor running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (match (start-service 'monitor-file-system)
+ (#f #f)
+ (('service response-parts ...)
+ (match (assq-ref response-parts 'running)
+ ((pid) (number? pid))))))
+ marionette))
+
+ (test-assert "file system monitor notices file system change"
+ (marionette-eval
+ '(begin
+ ;; Not strictly necessary - but we want to test the actual
+ ;; fswatch invocation and not our own synthetic events, so
+ ;; give fswatch the chance to start up.
+ (sleep 2)
+ ;; Now we provide a file change. The monitor should [restart
+ ;; and thus] call all the handlers again.
+ (call-with-output-file "/tmp/notice_me" identity)
+ (and
+ (let loop ((i 0))
+ (if (file-exists? "/tmp/noticed")
+ #t
+ (if (>= i 10)
+ #f
+ (begin
+ (sleep 1)
+ (loop (+ i 1))))))
+ ; assume (file-exists? "/tmp/notice_me")
+ (>= (stat:mtime (stat "/tmp/noticed"))
+ (stat:mtime (stat "/tmp/notice_me")))))
+ marionette))
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+ (gexp->derivation (string-append name "-test") test))
+
+(define %file-system-monitoring-os
+ (simple-operating-system
+ (service file-system-monitoring-service-type
+ (file-system-monitoring-configuration
+ (monitored-files '("/tmp/notice_me"))
+ (handlers '(lambda args
+ (call-with-output-file "/tmp/noticed"
+ identity)))))))
+
+(define %test-file-system-monitoring
+ (system-test
+ (name "file system monitoring")
+ (description "Test file system monitoring event handler.")
+ (value (run-file-system-monitoring-test
+ name %file-system-monitoring-os))))
D
D
Danny Milosavljevic wrote on 5 Nov 2018 04:45
[WIP RFC v2] services: Add file system monitor service.
(address . 33265@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20181105034525.8676-1-dannym@scratchpost.org
* gnu/services/monitoring.scm (file-system-monitoring-configuration): New
variable.
(file-system-monitoring-service-type): New variable.
* gnu/tests/monitoring.scm (%test-file-system-monitoring): New variable.
* doc/guix.texi (File System Monitoring Service): New subsubsection.
---
doc/guix.texi | 37 +++++++++++++++
gnu/services/monitoring.scm | 67 +++++++++++++++++++++++++-
gnu/tests/monitoring.scm | 93 ++++++++++++++++++++++++++++++++++++-
3 files changed, 195 insertions(+), 2 deletions(-)

Toggle diff (244 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 3b7fa50d8..8997a0915 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -21496,6 +21496,43 @@ The following is an example @code{dicod-service} configuration.
%dicod-database:gcide))))
@end example
+@cindex file system monitoring
+@subsubheading File System Monitoring Service
+
+The @code{(gnu services monitoring)} module provides a shepherd service to
+monitor file system changes and call a handler procedure on changes.
+
+@defvr {Scheme Variable} file-system-monitoring-service-type
+The service type for @command{fswatch}, which provides the file system
+monitoring capability.
+
+@example
+(service file-system-monitoring-service-type
+ (file-system-monitoring-configuration
+ (monitored-files '("/foo/bar"))
+ (handlers '((lambda args
+ (display "UH OH\n"))))))
+@end example
+@end defvr
+
+@deftp {Data Type} file-system-monitoring-configuration
+The data type representing the configuration of the file-system-monitoring
+service.
+
+@table @asis
+@item @code{package}
+Package containing the actual file system monitor (fswatch).
+
+@item @code{monitored-files}
+List of files to be monitored.
+
+@item @code{handlers}
+List of thunks which will be called once the file system monitor noticed
+changes in the monitored files.
+
+@end table
+@end deftp
+
@node Setuid Programs
@subsection Setuid Programs
diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index aa3b63a0e..b4bec5648 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -26,10 +26,13 @@
#:use-module (guix gexp)
#:use-module (guix records)
#:use-module (ice-9 match)
+ #:use-module (srfi srfi-1)
#:export (darkstat-configuration
prometheus-node-exporter-configuration
darkstat-service-type
- prometheus-node-exporter-service-type))
+ prometheus-node-exporter-service-type
+ file-system-monitoring-configuration
+ file-system-monitoring-service-type))
;;;
@@ -125,3 +128,65 @@ prometheus.")
(list (service-extension
shepherd-root-service-type
(compose list prometheus-node-exporter-shepherd-service))))))
+
+
+;;;
+;;; File System Monitoring
+;;;
+
+(define-record-type* <file-system-monitoring-configuration>
+ file-system-monitoring-configuration
+ make-file-system-monitoring-configuration
+ file-system-monitoring-configuration?
+ (package file-system-monitoring-configuration-package
+ (default fswatch))
+ (monitored-files file-system-monitoring-configuration-monitored-files
+ (default '()))
+ ;; Note: These thunks are called in the shepherd environment.
+ (handlers file-system-monitoring-configuration-handlers
+ (default '())))
+
+(define file-system-monitoring-shepherd-services
+ (match-lambda
+ (($ <file-system-monitoring-configuration> package monitored-files handlers)
+ (list (shepherd-service
+ (provision '(file-system-monitoring))
+ (documentation "File System Monitor")
+ (requirement '(file-systems))
+ (start #~(lambda ()
+ (sleep 1)
+ (for-each (lambda (handler)
+ (handler))
+ handlers)
+ (fork+exec-command
+ `(#$(file-append package "/bin/fswatch")
+ "--one-event"
+ "-l" "1" ; latency: 1 s
+ ; "-d"
+ "--"
+ #$@(if monitored-files
+ monitored-files
+ '())))))
+ (stop #~(make-kill-destructor))
+ (respawn? #t))))))
+
+(define file-system-monitoring-service-type
+ (service-type (name 'monitor-file-system)
+ (extensions
+ (list (service-extension shepherd-root-service-type
+ file-system-monitoring-shepherd-services)))
+ (compose concatenate)
+ (extend (lambda (config monitored-files-handlers)
+ (let ((monitored-files (map car monitored-files-handlers))
+ (handlers (map cdr monitored-files-handlers)))
+ (match config
+ (($ <file-system-monitoring-configuration> package initial-monitored-files initial-handlers)
+ (file-system-monitoring-configuration
+ (package package)
+ (monitored-files (append initial-monitored-files monitored-files))
+ (handlers (append initial-handlers handlers))
+))))))
+ (description
+ "Call all @dfn{handlers} once something happens with one of
+the files monitored, and on overflow. Can have false positives. Will call
+all @dfn{handlers} on startup.")))
diff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scm
index 3320a19a7..46364f830 100644
--- a/gnu/tests/monitoring.scm
+++ b/gnu/tests/monitoring.scm
@@ -20,10 +20,12 @@
#:use-module (gnu services)
#:use-module (gnu services monitoring)
#:use-module (gnu services networking)
+ ;#:use-module (gnu system)
#:use-module (gnu system vm)
#:use-module (gnu tests)
#:use-module (guix gexp)
- #:export (%test-prometheus-node-exporter))
+ #:export (%test-prometheus-node-exporter
+ %test-file-system-monitoring))
;;;
@@ -95,3 +97,92 @@
(description "Connect to a running prometheus-node-exporter server.")
(value (run-prometheus-node-exporter-server-test
name %prometheus-node-exporter-os))))
+
+
+;;;
+;;; File System Monitoring.
+;;;
+
+(define* (run-file-system-monitoring-test name test-os)
+ "Run tests in TEST-OS, which has file system monitoring running."
+ (define os
+ (marionette-operating-system
+ test-os
+ #:imported-modules '((gnu services herd))))
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ (port-forwardings '())))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (srfi srfi-11)
+ (srfi srfi-64)
+ (gnu build marionette)
+ (web client)
+ (web response))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin #$name)
+
+ (test-assert "file system monitor running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (match (start-service 'monitor-file-system)
+ (#f #f)
+ (('service response-parts ...)
+ (match (assq-ref response-parts 'running)
+ ((pid) (number? pid))))))
+ marionette))
+
+ (test-assert "file system monitor notices file system change"
+ (marionette-eval
+ '(begin
+ ;; Not strictly necessary - but we want to test the actual
+ ;; fswatch invocation and not our own synthetic events, so
+ ;; give fswatch the chance to start up.
+ (sleep 2)
+ ;; Now we provide a file change. The monitor should [restart
+ ;; and thus] call all the handlers again.
+ (call-with-output-file "/tmp/notice_me" identity)
+ (and
+ (let loop ((i 0))
+ (if (file-exists? "/tmp/noticed")
+ #t
+ (if (>= i 10)
+ #f
+ (begin
+ (sleep 1)
+ (loop (+ i 1))))))
+ ; assume (file-exists? "/tmp/notice_me")
+ (>= (stat:mtime (stat "/tmp/noticed"))
+ (stat:mtime (stat "/tmp/notice_me")))))
+ marionette))
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+ (gexp->derivation (string-append name "-test") test))
+
+(define %file-system-monitoring-os
+ (simple-operating-system
+ (service file-system-monitoring-service-type
+ (file-system-monitoring-configuration
+ (monitored-files '("/tmp/notice_me"))
+ (handlers '(lambda args
+ (call-with-output-file "/tmp/noticed"
+ identity)))))))
+
+(define %test-file-system-monitoring
+ (system-test
+ (name "file-system-monitoring")
+ (description "Test file system monitoring event handler.")
+ (value (run-file-system-monitoring-test
+ name %file-system-monitoring-os))))
D
D
Danny Milosavljevic wrote on 5 Nov 2018 04:51
[WIP RFC v3] services: Add file system monitoring service.
(address . 33265@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20181105035122.4359-1-dannym@scratchpost.org
* gnu/services/monitoring.scm (file-system-monitoring-configuration): New
variable.
(file-system-monitoring-entry): New variable.
(file-system-monitoring-service-type): New variable.
* gnu/tests/monitoring.scm (%test-file-system-monitoring): New variable.
* doc/guix.texi (File System Monitoring Service): New subsubsection.
---
doc/guix.texi | 37 +++++++++++++++
gnu/services/monitoring.scm | 77 +++++++++++++++++++++++++++++-
gnu/tests/monitoring.scm | 93 ++++++++++++++++++++++++++++++++++++-
3 files changed, 205 insertions(+), 2 deletions(-)

Toggle diff (254 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 3b7fa50d8..8997a0915 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -21496,6 +21496,43 @@ The following is an example @code{dicod-service} configuration.
%dicod-database:gcide))))
@end example
+@cindex file system monitoring
+@subsubheading File System Monitoring Service
+
+The @code{(gnu services monitoring)} module provides a shepherd service to
+monitor file system changes and call a handler procedure on changes.
+
+@defvr {Scheme Variable} file-system-monitoring-service-type
+The service type for @command{fswatch}, which provides the file system
+monitoring capability.
+
+@example
+(service file-system-monitoring-service-type
+ (file-system-monitoring-configuration
+ (monitored-files '("/foo/bar"))
+ (handlers '((lambda args
+ (display "UH OH\n"))))))
+@end example
+@end defvr
+
+@deftp {Data Type} file-system-monitoring-configuration
+The data type representing the configuration of the file-system-monitoring
+service.
+
+@table @asis
+@item @code{package}
+Package containing the actual file system monitor (fswatch).
+
+@item @code{monitored-files}
+List of files to be monitored.
+
+@item @code{handlers}
+List of thunks which will be called once the file system monitor noticed
+changes in the monitored files.
+
+@end table
+@end deftp
+
@node Setuid Programs
@subsection Setuid Programs
diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index aa3b63a0e..c2a0faa88 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -26,10 +26,14 @@
#:use-module (guix gexp)
#:use-module (guix records)
#:use-module (ice-9 match)
+ #:use-module (srfi srfi-1)
#:export (darkstat-configuration
prometheus-node-exporter-configuration
darkstat-service-type
- prometheus-node-exporter-service-type))
+ prometheus-node-exporter-service-type
+ file-system-monitoring-configuration
+ file-system-monitoring-entry
+ file-system-monitoring-service-type))
;;;
@@ -125,3 +129,74 @@ prometheus.")
(list (service-extension
shepherd-root-service-type
(compose list prometheus-node-exporter-shepherd-service))))))
+
+
+;;;
+;;; File System Monitoring
+;;;
+
+(define-record-type* <file-system-monitoring-entry>
+ file-system-monitoring-entry make-file-system-monitoring-entry
+ file-system-monitoring-entry?
+ (file-name file-system-monitoring-entry-file-name)
+ (handler file-system-monitoring-entry-handler))
+
+(define-record-type* <file-system-monitoring-configuration>
+ file-system-monitoring-configuration
+ make-file-system-monitoring-configuration
+ file-system-monitoring-configuration?
+ (package file-system-monitoring-configuration-package
+ (default fswatch))
+ (monitored-files file-system-monitoring-configuration-monitored-files
+ (default '()))
+ ;; Note: These thunks are called in the shepherd environment.
+ (handlers file-system-monitoring-configuration-handlers
+ (default '())))
+
+(define file-system-monitoring-shepherd-services
+ (match-lambda
+ (($ <file-system-monitoring-configuration> package monitored-files handlers)
+ (list (shepherd-service
+ (provision '(file-system-monitoring))
+ (documentation "File System Monitor")
+ (requirement '(file-systems))
+ (start #~(lambda ()
+ (sleep 1)
+ (for-each (lambda (handler)
+ (handler))
+ handlers)
+ (fork+exec-command
+ `(#$(file-append package "/bin/fswatch")
+ "--one-event"
+ "-l" "1" ; latency: 1 s
+ ; "-d"
+ "--"
+ #$@(if monitored-files
+ monitored-files
+ '())))))
+ (stop #~(make-kill-destructor))
+ (respawn? #t))))))
+
+(define file-system-monitoring-service-type
+ (service-type (name 'monitor-file-system)
+ (extensions
+ (list (service-extension shepherd-root-service-type
+ file-system-monitoring-shepherd-services)))
+ (compose concatenate)
+ (extend (lambda (config monitored-files-handlers)
+ (let ((monitored-files
+ (map file-system-monitoring-entry-file-name
+ monitored-files-handlers))
+ (handlers
+ (map file-system-monitoring-entry-handler
+ monitored-files-handlers)))
+ (match config
+ (($ <file-system-monitoring-configuration> package initial-monitored-files initial-handlers)
+ (file-system-monitoring-configuration
+ (package package)
+ (monitored-files (append initial-monitored-files monitored-files))
+ (handlers (append initial-handlers handlers))))))))
+ (description
+ "Call all @dfn{handlers} once something happens with one of
+the files monitored, and on overflow. Can have false positives. Will call
+all @dfn{handlers} on startup.")))
diff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scm
index 3320a19a7..46364f830 100644
--- a/gnu/tests/monitoring.scm
+++ b/gnu/tests/monitoring.scm
@@ -20,10 +20,12 @@
#:use-module (gnu services)
#:use-module (gnu services monitoring)
#:use-module (gnu services networking)
+ ;#:use-module (gnu system)
#:use-module (gnu system vm)
#:use-module (gnu tests)
#:use-module (guix gexp)
- #:export (%test-prometheus-node-exporter))
+ #:export (%test-prometheus-node-exporter
+ %test-file-system-monitoring))
;;;
@@ -95,3 +97,92 @@
(description "Connect to a running prometheus-node-exporter server.")
(value (run-prometheus-node-exporter-server-test
name %prometheus-node-exporter-os))))
+
+
+;;;
+;;; File System Monitoring.
+;;;
+
+(define* (run-file-system-monitoring-test name test-os)
+ "Run tests in TEST-OS, which has file system monitoring running."
+ (define os
+ (marionette-operating-system
+ test-os
+ #:imported-modules '((gnu services herd))))
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ (port-forwardings '())))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (srfi srfi-11)
+ (srfi srfi-64)
+ (gnu build marionette)
+ (web client)
+ (web response))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin #$name)
+
+ (test-assert "file system monitor running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (match (start-service 'monitor-file-system)
+ (#f #f)
+ (('service response-parts ...)
+ (match (assq-ref response-parts 'running)
+ ((pid) (number? pid))))))
+ marionette))
+
+ (test-assert "file system monitor notices file system change"
+ (marionette-eval
+ '(begin
+ ;; Not strictly necessary - but we want to test the actual
+ ;; fswatch invocation and not our own synthetic events, so
+ ;; give fswatch the chance to start up.
+ (sleep 2)
+ ;; Now we provide a file change. The monitor should [restart
+ ;; and thus] call all the handlers again.
+ (call-with-output-file "/tmp/notice_me" identity)
+ (and
+ (let loop ((i 0))
+ (if (file-exists? "/tmp/noticed")
+ #t
+ (if (>= i 10)
+ #f
+ (begin
+ (sleep 1)
+ (loop (+ i 1))))))
+ ; assume (file-exists? "/tmp/notice_me")
+ (>= (stat:mtime (stat "/tmp/noticed"))
+ (stat:mtime (stat "/tmp/notice_me")))))
+ marionette))
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+ (gexp->derivation (string-append name "-test") test))
+
+(define %file-system-monitoring-os
+ (simple-operating-system
+ (service file-system-monitoring-service-type
+ (file-system-monitoring-configuration
+ (monitored-files '("/tmp/notice_me"))
+ (handlers '(lambda args
+ (call-with-output-file "/tmp/noticed"
+ identity)))))))
+
+(define %test-file-system-monitoring
+ (system-test
+ (name "file-system-monitoring")
+ (description "Test file system monitoring event handler.")
+ (value (run-file-system-monitoring-test
+ name %file-system-monitoring-os))))
S
S
swedebugia wrote on 5 Nov 2018 10:16
1facd4f2-7dad-57f6-cbdd-7726ee5f367d@riseup.net
Hi

On 2018-11-05 04:51, Danny Milosavljevic wrote:
Toggle quote (2 lines)
> +(define file-system-monitoring-service-type
> + (service-type (name 'monitor-file-system)
Is a docstring missing here?

--
Cheers
Swedebugia
C
C
Clément Lassieur wrote on 5 Nov 2018 10:36
Re: [bug#33265] [WIP RFC] services: Add file system monitor service.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 33265@debbugs.gnu.org)
87pnvjn9xb.fsf@lassieur.org
Hi Danny,

I don't have time to do a proper review, but just one comment:

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (11 lines)
> + (test-assert "file system monitor running"
> + (marionette-eval
> + '(begin
> + (use-modules (gnu services herd))
> + (match (start-service 'monitor-file-system)
> + (#f #f)
> + (('service response-parts ...)
> + (match (assq-ref response-parts 'running)
> + ((pid) (number? pid))))))
> + marionette))

How is that better than:

(test-assert "file system monitor running"
(marionette-eval
'(begin
(use-modules (gnu services herd))
(start-service 'monitor-file-system))
marionette))

'start-service' should return #f if the service fails to start.

Cheers,
Clément
D
D
Danny Milosavljevic wrote on 5 Nov 2018 10:41
[WIP RFC v4] services: Add file system monitoring service.
(address . 33265@debbugs.gnu.org)(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
20181105094109.21915-1-dannym@scratchpost.org
* gnu/services/monitoring.scm (file-system-monitoring-configuration): New
variable.
(file-system-monitoring-entry): New variable.
(file-system-monitoring-service-type): New variable.
* gnu/tests/monitoring.scm (%test-file-system-monitoring): New variable.
* doc/guix.texi (File System Monitoring Service): New subsubsection.
---
doc/guix.texi | 37 ++++++++++++++
gnu/services/monitoring.scm | 79 +++++++++++++++++++++++++++++-
gnu/tests/monitoring.scm | 97 ++++++++++++++++++++++++++++++++++++-
3 files changed, 211 insertions(+), 2 deletions(-)

Toggle diff (260 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 3b7fa50d8..8997a0915 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -21496,6 +21496,43 @@ The following is an example @code{dicod-service} configuration.
%dicod-database:gcide))))
@end example
+@cindex file system monitoring
+@subsubheading File System Monitoring Service
+
+The @code{(gnu services monitoring)} module provides a shepherd service to
+monitor file system changes and call a handler procedure on changes.
+
+@defvr {Scheme Variable} file-system-monitoring-service-type
+The service type for @command{fswatch}, which provides the file system
+monitoring capability.
+
+@example
+(service file-system-monitoring-service-type
+ (file-system-monitoring-configuration
+ (monitored-files '("/foo/bar"))
+ (handlers '((lambda args
+ (display "UH OH\n"))))))
+@end example
+@end defvr
+
+@deftp {Data Type} file-system-monitoring-configuration
+The data type representing the configuration of the file-system-monitoring
+service.
+
+@table @asis
+@item @code{package}
+Package containing the actual file system monitor (fswatch).
+
+@item @code{monitored-files}
+List of files to be monitored.
+
+@item @code{handlers}
+List of thunks which will be called once the file system monitor noticed
+changes in the monitored files.
+
+@end table
+@end deftp
+
@node Setuid Programs
@subsection Setuid Programs
diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index aa3b63a0e..a717175c0 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -26,10 +26,14 @@
#:use-module (guix gexp)
#:use-module (guix records)
#:use-module (ice-9 match)
+ #:use-module (srfi srfi-1)
#:export (darkstat-configuration
prometheus-node-exporter-configuration
darkstat-service-type
- prometheus-node-exporter-service-type))
+ prometheus-node-exporter-service-type
+ file-system-monitoring-configuration
+ file-system-monitoring-entry
+ file-system-monitoring-service-type))
;;;
@@ -125,3 +129,76 @@ prometheus.")
(list (service-extension
shepherd-root-service-type
(compose list prometheus-node-exporter-shepherd-service))))))
+
+
+;;;
+;;; File System Monitoring
+;;;
+
+(define-record-type* <file-system-monitoring-entry>
+ file-system-monitoring-entry make-file-system-monitoring-entry
+ file-system-monitoring-entry?
+ (file-name file-system-monitoring-entry-file-name)
+ (handler file-system-monitoring-entry-handler))
+
+(define-record-type* <file-system-monitoring-configuration>
+ file-system-monitoring-configuration
+ make-file-system-monitoring-configuration
+ file-system-monitoring-configuration?
+ (package file-system-monitoring-configuration-package
+ (default fswatch))
+ (monitored-files file-system-monitoring-configuration-monitored-files
+ (default '()))) ; list of <file-system-monitoring-entry>.
+
+(define file-system-monitoring-shepherd-services
+ (match-lambda
+ (($ <file-system-monitoring-configuration> package monitored-files)
+ (list (shepherd-service
+ (provision '(file-system-monitoring))
+ (documentation "File System Monitor")
+ (requirement '(file-systems))
+ (start #~(let ((handlers
+ (list #$@(map file-system-monitoring-entry-handler
+ monitored-files))))
+ (lambda ()
+ (sleep 1)
+ (for-each (lambda (handler)
+ (handler))
+ handlers)
+ (fork+exec-command
+ `(#$(file-append package "/bin/fswatch")
+ "--one-event"
+ "-l" "1" ; latency: 1 s
+ ; "-d"
+ "--"
+ #$@(if monitored-files
+ (map file-system-monitoring-entry-file-name
+ monitored-files)
+ '("/does_not_exist")))))))
+ (stop #~(make-kill-destructor))
+ (respawn? #t))))))
+
+(define file-system-monitoring-service-type
+ (service-type (name 'monitor-file-system)
+ (extensions
+ (list (service-extension shepherd-root-service-type
+ file-system-monitoring-shepherd-services)))
+ (compose concatenate)
+ (extend (lambda (config monitored-entries)
+ (let ((monitored-files
+ (map file-system-monitoring-entry-file-name
+ monitored-entries))
+ (handlers
+ (map file-system-monitoring-entry-handler
+ monitored-entries)))
+ (match config
+ (($ <file-system-monitoring-configuration>
+ package initial-monitored-files)
+ (file-system-monitoring-configuration
+ (package package)
+ (monitored-files (append initial-monitored-files
+ monitored-files))))))))
+ (description
+ "Call all @dfn{handler}s once something happens with one of
+the files monitored, and on overflow. Can have false positives. Will also
+call all @dfn{handlers} on startup.")))
diff --git a/gnu/tests/monitoring.scm b/gnu/tests/monitoring.scm
index 3320a19a7..d2b62374d 100644
--- a/gnu/tests/monitoring.scm
+++ b/gnu/tests/monitoring.scm
@@ -20,10 +20,12 @@
#:use-module (gnu services)
#:use-module (gnu services monitoring)
#:use-module (gnu services networking)
+ ;#:use-module (gnu system)
#:use-module (gnu system vm)
#:use-module (gnu tests)
#:use-module (guix gexp)
- #:export (%test-prometheus-node-exporter))
+ #:export (%test-prometheus-node-exporter
+ %test-file-system-monitoring))
;;;
@@ -95,3 +97,96 @@
(description "Connect to a running prometheus-node-exporter server.")
(value (run-prometheus-node-exporter-server-test
name %prometheus-node-exporter-os))))
+
+
+;;;
+;;; File System Monitoring.
+;;;
+
+(define* (run-file-system-monitoring-test name test-os)
+ "Run tests in TEST-OS, which has file system monitoring running."
+ (define os
+ (marionette-operating-system
+ test-os
+ #:imported-modules '((gnu services herd))))
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ (port-forwardings '())))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (srfi srfi-11)
+ (srfi srfi-64)
+ (gnu build marionette)
+ (web client)
+ (web response))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin #$name)
+
+ (test-assert "file system monitor is running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (match (start-service 'file-system-monitoring)
+ (#f #f)
+ (('service response-parts ...)
+ (match (assq-ref response-parts 'running)
+ ((pid) (number? pid))))))
+ marionette))
+
+ (test-assert "file system monitor notices file system change"
+ (marionette-eval
+ '(begin
+ ;; Not strictly necessary - but we want to test the actual
+ ;; fswatch invocation and not our own synthetic events, so
+ ;; give fswatch the chance to start up.
+ (sleep 2)
+ (mkdir-p "/tmp/glasshouse")
+ ;; Now we provide a file change. The monitor should [restart
+ ;; and thus] call all the handlers again.
+ (call-with-output-file "/tmp/glasshouse/notice_me" identity)
+ (and
+ (let loop ((i 0))
+ (if (file-exists? "/tmp/glasshouse_noticed")
+ #t
+ (if (>= i 10)
+ #f
+ (begin
+ (sleep 1)
+ (loop (+ i 1))))))
+ ; assume (file-exists? "/tmp/notice_me")
+ (>= (stat:mtime (stat "/tmp/glasshouse_noticed"))
+ (stat:mtime (stat "/tmp/glasshouse/notice_me")))))
+ marionette))
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+ (gexp->derivation (string-append name "-test") test))
+
+(define %file-system-monitoring-os
+ (simple-operating-system
+ (service file-system-monitoring-service-type
+ (file-system-monitoring-configuration
+ (monitored-files
+ (list
+ (file-system-monitoring-entry
+ (file-name "/tmp/glasshouse")
+ (handler '(lambda _
+ (call-with-output-file "/tmp/glasshouse_noticed"
+ identity))))))))))
+
+(define %test-file-system-monitoring
+ (system-test
+ (name "file-system-monitoring")
+ (description "Test file system monitoring event handler.")
+ (value (run-file-system-monitoring-test
+ name %file-system-monitoring-os))))
L
L
Ludovic Courtès wrote on 10 Nov 2018 23:19
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 33265@debbugs.gnu.org)
87a7mgwp6e.fsf@gnu.org
Hello!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (7 lines)
> * gnu/services/monitoring.scm (file-system-monitoring-configuration): New
> variable.
> (file-system-monitoring-entry): New variable.
> (file-system-monitoring-service-type): New variable.
> * gnu/tests/monitoring.scm (%test-file-system-monitoring): New variable.
> * doc/guix.texi (File System Monitoring Service): New subsubsection.

Nice! Should we call it ‘fswatch-monitoring-service’, since there may
be other tools to achieve similar results?

Toggle quote (5 lines)
> +@cindex file system monitoring
> +@subsubheading File System Monitoring Service
> +
> +The @code{(gnu services monitoring)} module provides a shepherd service to

s/shepherd/Shepherd/

Toggle quote (4 lines)
> +@defvr {Scheme Variable} file-system-monitoring-service-type
> +The service type for @command{fswatch}, which provides the file system
> +monitoring capability.

Perhaps add a @uref to the fswatch home page.

Toggle quote (7 lines)
> +@example
> +(service file-system-monitoring-service-type
> + (file-system-monitoring-configuration
> + (monitored-files '("/foo/bar"))
> + (handlers '((lambda args
> + (display "UH OH\n"))))))

It would seem more natural to me for ‘handlers’ to be singular,
‘handler’. Also I’d use a gexp there (though in this particular example
it doesn’t make any difference):

(file-system-monitoring-configuration
(monitored-files '("/foo/bar"))
(handler #~(lambda (file whatever)
(format #t "~a modified!~%" file))))

Toggle quote (19 lines)
> + (start #~(let ((handlers
> + (list #$@(map file-system-monitoring-entry-handler
> + monitored-files))))
> + (lambda ()
> + (sleep 1)
> + (for-each (lambda (handler)
> + (handler))
> + handlers)
> + (fork+exec-command
> + `(#$(file-append package "/bin/fswatch")
> + "--one-event"
> + "-l" "1" ; latency: 1 s
> + ; "-d"
> + "--"
> + #$@(if monitored-files
> + (map file-system-monitoring-entry-file-name
> + monitored-files)
> + '("/does_not_exist")))))))

The ‘sleep’ call looks suspicious. :-)

IIUC, the service stops (and is respawned) every time an even occurs, is
that right?

Can’t we instead remove ‘--one-event’ and pass fswatch a script to run?
In that case, we’d pass it something along these lines:

#$(scheme-file "event-handler.scm"
#~(for-each (lambda (handler) …) …))

WDYT?

Toggle quote (14 lines)
> +(define file-system-monitoring-service-type
> + (service-type (name 'monitor-file-system)
> + (extensions
> + (list (service-extension shepherd-root-service-type
> + file-system-monitoring-shepherd-services)))
> + (compose concatenate)
> + (extend (lambda (config monitored-entries)
> + (let ((monitored-files
> + (map file-system-monitoring-entry-file-name
> + monitored-entries))
> + (handlers
> + (map file-system-monitoring-entry-handler
> + monitored-entries)))

So here if we changes ‘handlers’ to ‘handler’, we could do:

(apply compose (map file-system-monitoring-entry-handler entries))

Thanks,
Ludo’.
D
D
Danny Milosavljevic wrote on 11 Nov 2018 01:12
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 33265@debbugs.gnu.org)
20181111011218.00265d2f@scratchpost.org
Hi Ludo!

On Sat, 10 Nov 2018 23:19:53 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (3 lines)
> Nice! Should we call it ‘fswatch-monitoring-service’, since there may
> be other tools to achieve similar results?

Its use is mostly to allow other Guix services to extend it in order to
do stuff on file change (for example restarting a service; starting the
user shepherd - the latter is what caused me to write this).

I was very much tempted to call it "fswatch-monitoring-service", but that's
really an implementation detail - what it does is provide file change
notification services to other Guix services.

It's not meant to be a generic fswatch service. I've not known fswatch
before writing this service - so it's pretty likely that there are tons
more features an fswatch user could want to use which the service doesn't
provide. Also, fswatch isn't really meant to be used as a daemon AFAIK,
so we are not providing fswatch to the user in the way it's supposed to be
used by him.

Toggle quote (2 lines)
> The ‘sleep’ call looks suspicious. :-)

Pretty suspicious :)

Right now, the service has only a very narrow usecase where rapid
notifications is not something one wants. On the contrary, both rapid
notifications and (in a pathological case where a watch couldn't be
registered - see the "/does_not_exist" workaround) rapid fswatch invocations
would waste (potentially enormous amounts of) cpu time and would
forkbomb fswatch, shepherd and/or worse things. On the other hand, using
fswatch like we do now causes us to lose events. But so does starting fswatch
later than the clients. And most fswatch backends (for example the Linux one)
can lose events anyway (since the kernel buffer is limited - Linux will then
drop events). That's why it always calls the handlers when restarting - just
in case.

Toggle quote (3 lines)
> IIUC, the service stops (and is respawned) every time an even[t] occurs, is
> that right?

Yes. It's very simple :)

Toggle quote (2 lines)
> Can’t we instead remove ‘--one-event’ and pass fswatch a script to run?

Not to my knowledge. It would be nice...

What the stuff here does is it waits for events to accumulate in the time
given by "-l" (that is called a "batch" in fswatch).

The first such batch which contains at least one event will cause fswatch
to exit.
Otherwise fswatch will continue waiting (blocking) until the above happens.

It's an explicitly supported way to use fswatch (because it's very resilient).

The service collects all the files from the system into one fswatch
invocation - it's very much supposed to be a tripwire and not something
regularily used. It also collects all the handlers (regardless of monitored
file) into one handler to be called.

Toggle quote (7 lines)
> In that case, we’d pass it something along these lines:
>
> #$(scheme-file "event-handler.scm"
> #~(for-each (lambda (handler) …) …))
>
> WDYT?

I'd like to, but it doesn't support it.

What would be possible is to pipe something at the end (i.e. use open-pipe), but
I've had bad experiences with pipes buffering too much in the past and I'd rather
not hang the entire thing by some stupid buffering or parsing problem.

That said, we could try in a future version (after some serious testing).
But I think that trying to use pipes for us to be remote-controlled by
fswatch it will not work correctly if fswatch doesn't exit.

It would work with a pty, but that's kinda... convoluted. Also, forkpty is not
available in Guile.

There's libfswatch but no Guile bindings yet. Would be cool to have those
because fswatch is cross platform (and it has C bindings).

Toggle quote (18 lines)
> > +(define file-system-monitoring-service-type
> > + (service-type (name 'monitor-file-system)
> > + (extensions
> > + (list (service-extension shepherd-root-service-type
> > + file-system-monitoring-shepherd-services)))
> > + (compose concatenate)
> > + (extend (lambda (config monitored-entries)
> > + (let ((monitored-files
> > + (map file-system-monitoring-entry-file-name
> > + monitored-entries))
> > + (handlers
> > + (map file-system-monitoring-entry-handler
> > + monitored-entries)))
>
> So here if we changes ‘handlers’ to ‘handler’, we could do:
>
> (apply compose (map file-system-monitoring-entry-handler entries))

All good points!
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvnc+IACgkQ5xo1VCww
uqU+qwf/d4DuoZUrTye4TESNL4AuXngUktkQZLQzW2TzIw1a3fsz28fHyIM4AEqY
4RhV09s44RTvtDtpM5Pt7MBfXpZvCgSUhyXD1kd82vq3GPiFRM61yTN6PdEgIkM5
UemzReYEq03NxF3g1hGC5tsxZ/9SctiFyqpQySAQnR/RWsz4ZnkUgYX9eyvLhIwv
Kosxg06JVqTLuBnSyjasPxCTZKKOrmpGZfbZfSgwKL5Y1XaJXSTmJdT/HPOdksm4
GHeZMzy7j+sa8yz1P6Bg1rnBU4RX5sT2IvvuuSQv2XNd5Yk0H2bSZQ/H9XV3hx4J
e07KxHak+RvmRf4Ik1OrydIiqQR/1Q==
=ov4y
-----END PGP SIGNATURE-----


D
D
Danny Milosavljevic wrote on 11 Nov 2018 01:13
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 33265@debbugs.gnu.org)
20181111011350.4b6960a7@scratchpost.org
Toggle quote (3 lines)
> drop events). That's why it always calls the handlers when restarting - just
> in case.

That's why *WE* always calls the handlers when restarting - just in case.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvndD4ACgkQ5xo1VCww
uqXKXQf+KutvLzmQwfWDx2gEdq5z5GGzvBrhss3Hi2LCavQAVqadbYSD8wPhY31/
8K0PkUGlCYVshrFe9Eb38zBuSQNbEs80V5PJMxIPhGz/QK68YcqtH5+MYMhNFYvb
d9JsWl0fXpUI9jPPqQUvZBY+1ieZWNRXPtp5snG4AHFdb8B6NELkcoEhUa4/sGBr
A7d7ndwHjUvJsqZBOXfDkNgOtfqJDOUyRU46H0EqvfnWF6B/B9fyVlGptKWyiX+U
EWpX3W5kmCRxIjUzvsZyi3I/nw54YsQOFLKEP/KKv3Y9+ls55MjGtSj9uvTFttxq
+GgiInUs0Ms6/L/BRDP0IyGf7OG/mA==
=/4fA
-----END PGP SIGNATURE-----


D
D
Danny Milosavljevic wrote on 11 Nov 2018 01:15
Re: [bug#33265] [WIP RFC] services: Add file system monitor service.
(name . Clément Lassieur)(address . clement@lassieur.org)(address . 33265@debbugs.gnu.org)
20181111011559.37e42901@scratchpost.org
Hi Clément,

On Mon, 05 Nov 2018 10:36:00 +0100
Clément Lassieur <clement@lassieur.org> wrote:

Toggle quote (15 lines)
> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
> > + (test-assert "file system monitor running"
> > + (marionette-eval
> > + '(begin
> > + (use-modules (gnu services herd))
> > + (match (start-service 'monitor-file-system)
> > + (#f #f)
> > + (('service response-parts ...)
> > + (match (assq-ref response-parts 'running)
> > + ((pid) (number? pid))))))
> > + marionette))
>
> How is that better than:

It's not! Replaced. Thanks.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvndL8ACgkQ5xo1VCww
uqVx8Af/RxpQwVycvnTG9LzunqYLLDYQtyMMHYxnCxGB1tU2bkhmWnVZALgVMTqb
ZTkqYVwgkwS/TJYUxzO84Mp85RX+QxA2jquRP51yHrE4r7pw1kMXoLiWYz4EwJDs
C8Q5SdfxsfZo05+NROJiZpwIEsXFJuyawdOKujECu83HyBhJI0FL/+aGvNgNTPL8
EMUiSCp1A9QyMn9tlXIpyspF6TWr3eAcn2bQHwRnUeA1Zev1V6TRm0VGZ2q1Jdt7
DUafBAxWsJk0DziR3ehigc74Gll/hfBc40oaY6dAl//KjF0GAyTz1XXxpMsk/bZb
RUmyIL/y9sDhr276vCluIs8bqOnGZw==
=KR8+
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 11 Nov 2018 12:25
Re: [bug#33265] [WIP RFC v4] services: Add file system monitoring service.
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 33265@debbugs.gnu.org)
87wopjua86.fsf@gnu.org
Hello!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (21 lines)
> On Sat, 10 Nov 2018 23:19:53 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Nice! Should we call it ‘fswatch-monitoring-service’, since there may
>> be other tools to achieve similar results?
>
> Its use is mostly to allow other Guix services to extend it in order to
> do stuff on file change (for example restarting a service; starting the
> user shepherd - the latter is what caused me to write this).
>
> I was very much tempted to call it "fswatch-monitoring-service", but that's
> really an implementation detail - what it does is provide file change
> notification services to other Guix services.
>
> It's not meant to be a generic fswatch service. I've not known fswatch
> before writing this service - so it's pretty likely that there are tons
> more features an fswatch user could want to use which the service doesn't
> provide. Also, fswatch isn't really meant to be used as a daemon AFAIK,
> so we are not providing fswatch to the user in the way it's supposed to be
> used by him.

OK, that makes sense.

Toggle quote (16 lines)
>> The ‘sleep’ call looks suspicious. :-)
>
> Pretty suspicious :)
>
> Right now, the service has only a very narrow usecase where rapid
> notifications is not something one wants. On the contrary, both rapid
> notifications and (in a pathological case where a watch couldn't be
> registered - see the "/does_not_exist" workaround) rapid fswatch invocations
> would waste (potentially enormous amounts of) cpu time and would
> forkbomb fswatch, shepherd and/or worse things. On the other hand, using
> fswatch like we do now causes us to lose events. But so does starting fswatch
> later than the clients. And most fswatch backends (for example the Linux one)
> can lose events anyway (since the kernel buffer is limited - Linux will then
> drop events). That's why it always calls the handlers when restarting - just
> in case.

Uh, not very confidence-inspiring. ;-)

Toggle quote (5 lines)
>> IIUC, the service stops (and is respawned) every time an even[t] occurs, is
>> that right?
>
> Yes. It's very simple :)

OK that works, but I’m not very comfortable with the approach: normally
respawns indicate that the service failed unexpectedly, so here we’re
really abusing the mechanism IMO.

Toggle quote (4 lines)
>> Can’t we instead remove ‘--one-event’ and pass fswatch a script to run?
>
> Not to my knowledge. It would be nice...

Another option would be to use Direvent, which supports this and more,
or maybe ‘inotifywatch’ from inotify-tools though it seems to be quite
similar to fswatch feature-wise.

WDYT?

Ludo’.
D
D
Danny Milosavljevic wrote on 11 Nov 2018 13:30
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 33265@debbugs.gnu.org)
20181111133052.4c4dfbe4@scratchpost.org
Hi Ludo,

On Sun, 11 Nov 2018 12:25:45 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

Toggle quote (11 lines)
> > registered - see the "/does_not_exist" workaround) rapid fswatch invocations
> > would waste (potentially enormous amounts of) cpu time and would
> > forkbomb fswatch, shepherd and/or worse things. On the other hand, using
> > fswatch like we do now causes us to lose events. But so does starting fswatch
> > later than the clients. And most fswatch backends (for example the Linux one)
> > can lose events anyway (since the kernel buffer is limited - Linux will then
> > drop events). That's why it always calls the handlers when restarting - just
> > in case.
>
> Uh, not very confidence-inspiring. ;-)

Yeah well, file change notification is surprisingly crappy - especially when
you want it to *always* work. On the other hand typical UNIX :)

Toggle quote (4 lines)
> OK that works, but I’m not very comfortable with the approach: normally
> respawns indicate that the service failed unexpectedly, so here we’re
> really abusing the mechanism IMO.

I agree. In the end it would be nicer to get the pipe thing to work or
better yet to have libfswatch bindings or better yet use direvent, aha.

Toggle quote (2 lines)
> Another option would be to use Direvent, which supports this and more,

Aha, that looks as if it's intended exactly for this use case! Nice!

I'll try that one next.

Toggle quote (3 lines)
> or maybe ‘inotifywatch’ from inotify-tools though it seems to be quite
> similar to fswatch feature-wise.

inotifywatch is Linux-specific. If we want Guix to support the Hurd, BSD,
Windows then it shouldn't be using Linux-specific things. Also, it uses
the same stuff as fswatch under the hood so it has the same limitations.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvoIPwACgkQ5xo1VCww
uqX9vAf8C5cMYvz7JuLNPZF/0swUq1xgGuoBWdRyXDSreNEZ6elMrndI0D/Dahn0
7uTqZQcgNV25kWJ3fjMMCEFW1/ft66Lo08hACuxqWVPKr8VVM+XTPO4enr+SmYcM
WbwXl0CYJYNOe+NZHNJvrvmSZWaXhcEbawL70EPJfPXTJrxRFcrpo8w1qhFjR5Kg
bvF+6VTzcsqzkc05zV7XfLHqlOUvaEBhIYT39AbFychwv0yCY5F0BaGBX8JweXo9
cFRSK7xk3wiKpLMPEL0LYsufJBvcEOkEfg072fUZ7emO/GS7bWF+Ap/3hgP8HcLu
h1h4gjxbiRi6+7RH5Qx4zkQqd6C3tA==
=BGCc
-----END PGP SIGNATURE-----


D
D
Danny Milosavljevic wrote on 11 Nov 2018 13:50
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 33265@debbugs.gnu.org)
20181111134811.7d48b621@scratchpost.org
Toggle quote (2 lines)
> Another option would be to use Direvent

I've read through direvent 5.1 now and:

(1) It supports only watching entire directories. OK I guess, especially
since the client can specify basename filters to be applied, so if you
want to watch individual files you can, using a strange configuration
(something like: path $(dirname $file), file $(escape-regex $(basename $file))).

(2) It doesn't notify the client on inotify queue overflow. *shakes head*
(genev_xlat does not translate IN_Q_OVERFLOW)

(fswatch does notify the client on inotify queue overflow)

Whoever doesn't handle inotify queue overflow uses the kernel API incorrectly -
it's not going to be reliable.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvoJZkACgkQ5xo1VCww
uqUIFAf9GXZsDEUyx8gSF7uanm/xcot8SndxkHK1/Bu6wuBrJZKje74T+qq/GBof
1goXO5HdbYw/x9amL0W989wX7PkzrmsSfjbKDeCZGzT8DnuBkkaUyvu68MGmbj7a
ybGs/0KIqgKopBOqHXLV/DtgmMyOicaufcK8+ZPWrGydKRgTsdvouBZEiohahwcW
BiCREYfFIpcAXCcy/kL5APrBtOdA6yNSwsXxeiOTbQ9BacN1rkUisIXuS29vm6ot
MMeyG/l5VzwStU33ePtPjPDbHdj5asowPMpVwk7vwaWPUTlLxb/pJBMhSgldhmMA
WvYfM5kmEaVAii7x6+2FcP5VFSAhxg==
=s+o2
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 11 Nov 2018 14:22
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 33265@debbugs.gnu.org)
87a7mfu4tc.fsf@gnu.org
Hello Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

Toggle quote (3 lines)
> Whoever doesn't handle inotify queue overflow uses the kernel API incorrectly -
> it's not going to be reliable.

It seems that inotify and reliability don’t go together well anyway:


So I don’t know if what you describe is a showstopper for Direvent
specifically, or if it’s just that things aren’t going to work
“reliably” anyway.

WDYT?

Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 33265@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 33265
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch