[PATCH] services: add cloud-init service

Alexander Joss wrote 6 days ago

Recipients:(address . guix-patches@gnu.org)(name . Alexander Joss)(address . alex@infiniteadaptability.org)

Message-ID:c907e459d6898df885e3aac67c8446a1c15d62b2.1731824739.git.alex@infiniteadaptability.org

* gnu/services/cloud-init.scm: add cloud-init service
* gnu/system/examples: add cloud-init-image.tmpl

Change-Id: I28fe295c1dbbab7ea7df65f6b764c7d795e58d77
---
 gnu/services/cloud-init.scm               | 137 ++++++++++++++++++++++
 gnu/system/examples/cloud-init-image.tmpl |  63 ++++++++++
 2 files changed, 200 insertions(+)
 create mode 100644 gnu/services/cloud-init.scm
 create mode 100644 gnu/system/examples/cloud-init-image.tmpl

Toggle diff (216 lines)diff --git a/gnu/services/cloud-init.scm b/gnu/services/cloud-init.scm
new file mode 100644
index 0000000000..d6362f70a7
--- /dev/null
+++ b/gnu/services/cloud-init.scm
@@ -0,0 +1,137 @@
+(define-module (gnu services cloud-init)
+  #:use-module (gnu packages bash)
+  #:use-module (gnu packages python-web)
+  #:use-module (gnu services)
+  #:use-module (gnu services shepherd)
+  #:use-module (guix gexp)
+  #:use-module (guix records)
+  #:export (cloud-init-configuration cloud-init-service
+                                     cloud-init-service-type))
+
+(define-record-type* <cloud-init-configuration> cloud-init-configuration
+                     make-cloud-init-configuration
+  cloud-init-configuration?
+
+  (cloud-init cloud-init-configuration-cloud-init ;file-like
+              (default python-cloud-init))
+  (init-modules cloud-init-configuration-init-modules ;list of symbols
+                (default '(seed_random growpart
+                                       resizefs
+                                       disk_setup
+                                       mounts
+                                       set_hostname
+                                       update_hostname
+                                       users_groups
+                                       ssh
+                                       set_passwords)))
+  (config-modules cloud-init-configuration-config-modules ;list of symbols
+                  (default '()))
+  (final-modules cloud-init-configuration-final-modules ;list of symbols
+                 (default '(ssh_authkey_fingerprints)))
+  (extra-configuration-files
+   cloud-init-configuration-extra-configuration-files ;list of file-likes
+   (default '())))
+
+(define %cloud-dir
+  "/etc/cloud")
+
+(define %cloud-cfg
+  (string-append %cloud-dir "/cloud.cfg"))
+
+(define %cloud-run
+  (mixed-text-file "run.sh"
+                   "#!"
+                   (file-append bash "/bin/bash")
+                   "\n\nset -euo pipefail\n\n"
+                   (file-append python-cloud-init "/bin/cloud-init")
+                   " init --local\n"
+                   (file-append python-cloud-init "/bin/cloud-init")
+                   " init\n"
+                   (file-append python-cloud-init "/bin/cloud-init")
+                   " modules --mode config\n"
+                   (file-append python-cloud-init "/bin/cloud-init")
+                   " modules --mode final\n"))
+
+(define %cloud-cfg-d
+  (string-append %cloud-dir "/cloud.cfg.d"))
+
+(define (cloud-init-initialization init-modules config-modules final-modules
+                                   extra)
+  "Return the gexp to initialize the cloud-init configuration files"
+  #~(begin
+      (use-modules (srfi srfi-1)
+                   (srfi srfi-2)
+                   (guix build utils))
+
+      (define reduce-modules
+        (lambda (mods)
+          (string-join (map (lambda (mod)
+                              (string-append "\n  - "
+                                             (symbol->string mod))) mods))))
+
+      (mkdir-p #$%cloud-cfg-d)
+
+      (copy-file #$%cloud-run
+                 (string-append #$%cloud-dir "/run.sh"))
+      (chmod (string-append #$%cloud-dir "/run.sh") #o755)
+
+      (unless (null? '(#$@extra))
+        (for-each (lambda (file)
+                    (symlink (cadr file)
+                             (string-append #$%cloud-cfg-d "/"
+                                            (car file))))
+                  '(#$@extra)))
+
+      (call-with-output-file #$%cloud-cfg
+        (lambda (p)
+          (unless (null? '(#$@init-modules))
+            (display (string-append "cloud_init_modules:"
+                                    (reduce-modules '(#$@init-modules)) "\n\n")
+                     p))
+          (unless (null? '(#$@config-modules))
+            (display (string-append "cloud_config_modules:"
+                                    (reduce-modules '(#$@config-modules))
+                                    "\n\n") p))
+          (unless (null? '(#$@final-modules))
+            (display (string-append "cloud_final_modules:"
+                                    (reduce-modules '(#$@final-modules))
+                                    "\n\n") p))))))
+
+(define (cloud-init-activation config)
+  "Return the activation gexp for CONFIG."
+  #~(begin
+      (use-modules (guix build utils))
+      #$(cloud-init-initialization (cloud-init-configuration-init-modules
+                                    config)
+                                   (cloud-init-configuration-config-modules
+                                    config)
+                                   (cloud-init-configuration-final-modules
+                                    config)
+                                   (cloud-init-configuration-extra-configuration-files
+                                    config))))
+
+(define (cloud-init-service config)
+  "Return a <cloud-init-service> for cloud-init with CONFIG."
+  (define cloud-init
+    (cloud-init-configuration-cloud-init config))
+
+  (list (shepherd-service (documentation "cloud-init service")
+                          (provision '(cloud-init))
+                          (requirement '(networking))
+                          (one-shot? #t)
+                          (start #~(fork+exec-command (list (string-append #$%cloud-dir
+                                                             "/run.sh"))
+                                                      #:log-file (string-append
+                                                                  "/var/log/cloud-init.log")
+                                                      #:environment-variables '
+                                                      ("PATH=/run/current-system/profile/bin:/run/current-system/profile/sbin:"))))))
+
+(define cloud-init-service-type
+  (service-type (name 'cloud-init)
+                (default-value (cloud-init-configuration))
+                (description "cloud init")
+                (extensions (list (service-extension
+                                   shepherd-root-service-type
+                                   cloud-init-service)
+                                  (service-extension activation-service-type
+                                                     cloud-init-activation)))))
diff --git a/gnu/system/examples/cloud-init-image.tmpl b/gnu/system/examples/cloud-init-image.tmpl
new file mode 100644
index 0000000000..e2e69e8691
--- /dev/null
+++ b/gnu/system/examples/cloud-init-image.tmpl
@@ -0,0 +1,63 @@
+;; This vm image is meant to be used as an image template
+;; to be deployed on cloud providers that use cloud-init.
+
+(use-modules (gnu)
+             (guix)
+             (guix gexp)
+             (srfi srfi-1))
+(use-service-modules cloud-init base networking ssh)
+(use-package-modules admin bootloaders package-management python-web ssh)
+
+(operating-system
+  (host-name "gnu")
+  (timezone "Etc/UTC")
+  (locale "en_US.utf8")
+  (keyboard-layout (keyboard-layout "us"))
+
+  (firmware '())
+
+  ;; Below we assume /dev/vda is the VM's hard disk.
+  ;; Adjust as needed.
+  (bootloader (bootloader-configuration
+                (bootloader grub-bootloader)
+                (targets '("/dev/vda"))
+                (terminal-outputs '(console))))
+  (file-systems (cons (file-system
+                        (mount-point "/")
+                        (device "/dev/vda1")
+                        (type "ext4")) %base-file-systems))
+
+  ;; The cloud-utils packages provides some utilities to allow
+  ;; us to piggyback off ubuntu's cloud-init modules/integrations
+  ;; without having to write guix specific functionality.
+  ;;
+  ;; The python-cloud-init package is not strictly required to be
+  ;; in system-wide packages.
+  (packages (append (list cloud-utils python-cloud-init) %base-packages))
+
+  (services
+   (append (list (service cloud-init-service-type)
+                 ;; An example of extra configuration files. This specific
+                 ;; file is required for properly running cloud-init on DigitalOcean
+                 ;; (cloud-init-configuration (extra-configuration-files `
+                 ;; (("99-digitalocean.cfg" ,
+                 ;; (plain-file
+                 ;; "99-digitalocean.cfg"
+                 ;; "datasource_list: [ ConfigDrive, DigitalOcean, NoCloud, None ]"))))))
+                 
+                 (service network-manager-service-type)
+                 (service wpa-supplicant-service-type)
+                 (service openssh-service-type
+                          (openssh-configuration (openssh openssh-sans-x)
+                                                 (permit-root-login #t))))
+           %base-services
+           ;; Uncomment the following and replace the above to automatically add your guix
+           ;; signing key to the vm for easy reconfiguration.
+           ;; (modify-services %base-services
+           ;; (guix-service-type config =>
+           ;; (guix-configuration (inherit config)
+           ;; (authorized-keys (append
+           ;; (list (local-file
+           ;; "/etc/guix/signing-key.pub"))
+           ;; %default-authorized-guix-keys)))))))
+           )))

base-commit: 0e1ffbc7f5f060f89c890472377a6102f27f6e9b
-- 
2.46.0

Ricardo Wurmus wrote 47 hours ago

Recipients:(address . 74389@debbugs.gnu.org)

Message-ID:87serkzbo4.fsf@elephly.net

Hi Alexander,

thank you for the patch!

I'm interested in a cloud-init service because I think we need a way to

have Guix System images be configured dynamically when using them in AWS

or in Openstack environments. The dynamic configuration I'm most

interested in is user credentials and networking.

Last I looked into this, the Python library for cloud init had explicit

support for various distributions to modify system configuration files

and run tools (e.g. to configure and bring up networking, or to set the

hostname). It did not generate a central configuration file that could

be processed. It just triggered explicit per-distro actions.

We couldn't actually use any of that stuff for Guix System. What I

investigated a few months back was to see if there was a way to generate

some sort of configuration file that our services could then consult.

Sadly there was no such thing as a central configuration file containing

all necessary bits of information. From what I can tell cloud-utils

does not have what it takes to configure networking or user accounts.

It's probably useful for other bits, though I haven't ever used those.

Could you please expand on what your service does---and what it cannot

or does not aim to do---and how it would be used?

Thanks!

Ricardo

Alex wrote 22 hours ago

Recipients:(name . 74389@debbugs.gnu.org)(address . 74389@debbugs.gnu.org)

Message-ID:bATnGQtQhBzofETx9zjy9A8gqJwfjZAFRLjh4x1zhKmNFUjDQLqzBuEcja-YyB7bkl3esUJDMuGwmRS16QTCGNhXVLueI4BKNKHz-qU_Vmg=@infiniteadaptability.org

Ricardo,

This patch is just a MVP for implementing a cloud-init service. The goal primarily was to get the networking, disk and filesystem mounts, and ssh authentication working properly.

From a high level this patch implements a cloud-init-service which uses cloud-init functionality to gather information from it's run-time environment and then run cloud-init modules in order to properly configure the vm. It provides a configuration interface for choosing which modules to run as well as a mechanism to add extra configuration files.

I took a few major shortcuts in order to get a working image.

The first is that instead of writing guix specific functionality for cloud-init modules, I decided to attempt to use the ubuntu (which I believe is the default) tools. This is why the cloud-utils package as well as the network-manager-service-type are included in the vm image template I created as part of this patch.

The second shortcut is that I didn't see a good way to guix-ify the cloud-init service definitions/runtime environment (i.e. I used the /etc/cloud directory to handle most of the configuration files). I'd prefer if configuration lives in the store, but the configuration files and the gathered data live in locations which I didn't see a way to overwrite easily. I'm sure this exists, but would require more digging into the cloud-init source code and/or patches that would need to be pushed upstream.

The final shortcut is the actual running of the cloud-init-service. I didn't see a good way to *wait* for a process to complete, i.e. to block other services until the current service completes. It looks to me that running a long-running one-shot service (each stage of cloud-init typically takes a few seconds to run) doesn't block any service which requires it's provisions. Maybe there's some shepherd nuances that I'm not aware of. I ended up creating a computed file which produces a shell script which runs each cloud-init stage in sequence as a workaround to a more robust solution.

I added some extra comments with instructions on how to add extra cloud-init configuration files. This is required for cloud-init to run properly on DigitalOcean (the order of the data sources needs to be changed from the default). It might be necessary for other p

I've tested this on digital ocean and gotten it work properly (and then used guix deploy to update the configuration of the image).

NOTE: there are some comments in the template file that would need to be uncommented to build the exact image I used.

Using this patch it should be easy to create an image for testing using:

./pre-inst-env guix system image -t qcow2 gnu/system/examples/cloud-init-image.tmpl

I would appreciate further testers and any suggestions for improvement.

Thanks!

Alex

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74389@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 74389

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date