(address . guix-patches@gnu.org)(name . ()(address . paren@disroot.org)
20230321205749.4974-1-paren@disroot.org
* website/posts/dissecting-guix-3-gexps.md: New blog post.
---
Heya Guix,
Here's the third post in the Dissecting Guix series; this one aims to demystify
g-expressions ;)
-- (
website/posts/dissecting-guix-3-gexps.md | 673 +++++++++++++++++++++++
1 file changed, 673 insertions(+)
create mode 100644 website/posts/dissecting-guix-3-gexps.md
Toggle diff (458 lines)
diff --git a/website/posts/dissecting-guix-3-gexps.md b/website/posts/dissecting-guix-3-gexps.md
new file mode 100644
index 0000000..32f5d51
--- /dev/null
+++ b/website/posts/dissecting-guix-3-gexps.md
@@ -0,0 +1,673 @@
+title: Dissecting Guix, Part 3: G-Expressions
+date: TBC
+author: (
+tags: Dissecting Guix, Functional package management, Programming interfaces, Scheme API
+---
+Welcome back to [Dissecting Guix](https://guix.gnu.org/en/blog/tags/dissecting-guix)!
+Last time, we discussed [monads](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad),
+the functional programming idiom used by Guix to thread a store connection
+through a series of store-related operations.
+
+Today, we'll be talking about a concept rather more specific to Guix:
+_g-expressions_. Being an implementation of the Scheme language, Guile is built
+around [_s-expressions_](https://en.wikipedia.org/wiki/S-expression), which can
+represent, as the saying goes, _code as data_, thanks to the simple structure of
+Scheme forms.
+
+As Guix's package recipes are written in Scheme, it naturally needs some way to
+represent code that is to be run only when the package is built. Additionally,
+there needs to be some way to reference dependencies and retrieve output paths;
+otherwise, you wouldn't be able to, for instance, create a phase to install a
+file in the output directory.
+
+So, how do we implement this "deferred" code? Well, initially Guix used plain
+old s-expressions for this purpose.
+
+# Once Upon a Time
+
+Let's say we want to create a store item that's just a symlink to the
+`bin/irssi` file of the `irssi` package. How would we do that with an
+s-expression? Well, the s-expression itself, which we call the _builder_, is
+fairly simple:
+
+```scheme
+(define sexp-builder
+ `(let* ((out (assoc-ref %outputs "out"))
+ (irssi (assoc-ref %build-inputs "irssi"))
+ (bin/irssi (string-append irssi "/bin/irssi")))
+ (symlink bin/irssi out)))
+```
+
+If you aren't familliar with the "quoting" syntax used to create s-expressions,
+I strongly recommend that you read the excellent Scheme Primer; specifically,
+section 7, [_Lists and
+"cons"_](https://spritely.institute/static/papers/scheme-primer.html#scheme-lists-and-cons)
+and section 11, [_On the extensibility of Scheme (and Lisps in
+general)_](https://spritely.institute/static/papers/scheme-primer.html#scheme-extensibility)
+
+The `%outputs` and `%build-inputs` variables are bound within builder scripts to
+_association lists_, which are lists of pairs that act like key/value stores,
+for instance:
+
+```scheme
+'(("foo" . "bar")
+ ("floob" . "blarb")
+ ("fvoolag" . "bvarlag"))
+```
+
+To retrieve values from association lists, which are often referred to as
+_alists_, we use the `assoc-ref` procedure:
+
+```scheme
+(assoc-ref '(("boing" . "bouncy")
+ ("floing" . "flouncy"))
+ "boing")
+⇒ "bouncy"
+```
+
+`%outputs`, as the name might suggest, maps derivation output names to the paths
+of their respective store items, the default output being `out`, and
+`%build-inputs` maps inputs labels to their store items.
+
+The builder is the easy part; we now need to turn it into a derivation and tell
+it what `"irssi"` actually refers to. For this, we use the
+`build-expression->derivation` procedure from `(guix derivations)`:
+
+```scheme
+(use-modules (guix derivations)
+ (guix packages)
+ (guix store)
+ (gnu packages guile)
+ (gnu packages irc))
+
+(with-store store
+ (let ((guile-3.0-drv (package-derivation store guile-3.0))
+ (irssi-drv (package-derivation store irssi)))
+ (build-expression->derivation store "irssi-symlink" sexp-builder
+ #:guile-for-build guile-3.0-drv
+ #:inputs `(("irssi" ,irssi-drv)))))
+⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>
+```
+
+There are several things to note here:
+
+- The inputs _must_ all be derivations, so we need to first convert the packages
+ using `package-derivation`.
+- We need to explicitly set `#:guile-for-build`; there's no default value.
+- The `build-expression->derivation` and `package-derivation` procedures are
+ _not_ monadic, so we need to explicitly pass them the store connection.
+
+The shortcomings of using s-expressions in this way are numerous: we have to
+convert everything to a derivation before using it, and _inputs are not an
+inherent aspect of the builder_. G-expressions were designed to overcome these
+issues.
+
+# Premortem Examination
+
+A gexp is fundumentally a record of type `<gexp>`, which is, naturally, defined
+in `(guix gexp)`. The two most important fields of this record type, out of a
+total of five, are `proc` and `references`; the former is a procedure that
+returns the equivalent sexp, the latter a list containing everything from the
+"outside world" that's used by the gexp.
+
+When we want to turn the gexp into something that we can actually run as code,
+we combine these two fields by first building any gexp inputs that can become
+derivations (leaving alone those that cannot, such as and then passing the built
+`references` as the arguments of `proc`.
+
+Here's an example gexp that is essentially equivalent to our `sexp-builder`:
+
+```scheme
+(use-modules (guix gexp))
+
+(define gexp-builder
+ #~(symlink #$(file-append irssi "/bin/irssi")
+ #$output))
+```
+
+`gexp-builder` is far more concise than `sexp-builder`; let's examine the syntax
+and the `<gexp>` object we've created. To make a gexp, we use the `#~` syntax,
+equivalent to the `gexp` macro, rather than the `quasiquote` backtick used to
+create sexps.
+
+When we want to embed values from outside as references, we use `#$`, or
+`ungexp`, which is, in appearance if not function, equivalent to `unquote`
+(`,`). `ungexp` can accept any of four reference types:
+
+- Sexps (strings, lists, etc), which will be embedded literally.
+- Other gexps, embedded literally.
+- Expressions returning any sort of object that can be lowered into a
+ derivation, such as `<package>`, embedding that object's `out` store item; if
+ the expression is specifically a symbol bound to a buildable object, you can
+ optionally follow it with a colon and an alternative output name, so
+ `package:lib` is permitted, but `(get-package):lib` isn't.
+- The symbol `output`, embedding an output path. Like symbols bound to
+ buildable objects, this can be followed by a colon and the output name that
+ should be used rather than the default `out`.
+
+All these reference types will be represented by `<gexp-input>` records in the
+`references` field, except for the last kind, which will become `<gexp-output>`
+records. To give an example of each type of reference (with the return value
+output formatted for easier reading):
+
+```scheme
+(use-modules (gnu packages glib))
+
+#~(list #$"foobar" ;s-expression
+ #$#~(string-append "foo" "bar") ;g-expression
+ #$(file-append irssi "/bin/irssi") ;buildable object (expression)
+ #$glib:bin ;buildable object (symbol)
+ #$output:out) ;output
+⇒ #<gexp (list #<gexp-input "foobar":out>
+ #<gexp-input #<gexp (string-append "foo" "bar") …>:out>
+ #<gexp-input #<file-append #<package irssi@1.4.3 …> "/bin/irssi">:out>
+ #<gexp-input #<package glib@2.70.2 …>:bin>
+ #<gexp-output out>) …>
+```
+
+Note the use of `file-append` in both the previous example and `gexp-builder`;
+this procedure produces a `<file-append>` object that builds its first argument
+and is embedded as the concatenation of the first argument's output path and the
+second argument, which should be a string. For instance,
+`(file-append irssi "/bin/irssi")` builds `irssi` and expands to
+`/gnu/store/…-irssi/bin/irssi`, rather than the `/gnu/store/…-irssi` that the
+package alone would be embedded as.
+
+So, now that we have a gexp, how do we turn it into a derivation? This process
+is known as _lowering_; it entails the use of the aptly-named `lower-gexp`
+monadic procedure to combine `proc` and `references` and produce a
+`<lowered-gexp>` record, which acts as a sort of intermediate representation
+between gexps and derivations. We can piece apart this lowered form to get a
+sense of what the final derivation's builder script would look like:
+
+```scheme
+(define lowered-gexp-builder
+ (with-store store
+ (run-with-store store
+ (lower-gexp gexp-builder))))
+
+(lowered-gexp-sexp lowered-gexp-builder)
+⇒ (symlink
+ "/gnu/store/…-irssi-1.4.3/bin/irssi"
+ ((@ (guile) getenv) "out"))
+```
+
+And there you have it: a s-expression compiled from a g-expression, ready to be
+written into a builder script file in the store. So, how exactly do you turn
+this into said derivation?
+
+Well, it turns out that there isn't an interface for turning lowered gexps into
+derivations, only one for turning regular gexps into derivations that first uses
+`lower-gexp`, then implements the aforementioned conversion internally, rather
+than outsourcing it to some other procedure, so that's what we'll use.
+
+Unsurprisingly, that procedure is called `gexp->derivation`, and unlike its sexp
+equivalent, it's monadic. (`build-expression->derivation` and other deprecated
+procedures were in Guix since before the monads system existed.)
+
+```scheme
+(with-store store
+ (run-with-store store
+ (gexp->derivation "irssi-symlink" gexp-builder)))
+⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>
+```
+
+Finally, we have a gexp-based equivalent to the derivation we earlier created
+with `build-expression->derivation`! Here's the code we used for the sexp
+version in full:
+
+```scheme
+(define sexp-builder
+ `(let* ((out (assoc-ref %outputs "out"))
+ (irssi (assoc-ref %build-inputs "irssi"))
+ (bin/irssi (string-append irssi "/bin/irssi")))
+ (symlink bin/irssi out)))
+
+(with-store store
+ (let ((guile-3.0-drv (package-derivation store guile-3.0))
+ (irssi-drv (package-derivation store irssi)))
+ (build-expression->derivation store "irssi-symlink" sexp-builder
+ #:guile-for-build guile-3.0-drv
+ #:inputs `(("irssi" ,irssi-drv)))))
+```
+
+And here's the gexp equivalent:
+
+```scheme
+(define gexp-builder
+ #~(symlink #$(file-append irssi "/bin/irssi")
+ #$output))
+
+(with-store store
+ (run-with-store store
+ (gexp->derivation "irssi-symlink" gexp-builder)))
+```
+
+That's a lot of complexity abstracted away! For more complex packages and
+services, especially, gexps are a lifesaver; you can refer to the output paths
+of inputs just as easily as you would a string constant. You do, however, have
+to watch out for situations where `ungexp-native`, written as `#+`, would be
+preferable over regular `ungexp`, and that's something we'll discuss later.
+
+A brief digression before we continue: if you'd like to look inside a `<gexp>`
+record, but you'd rather not build anything, you can use the
+`gexp->approximate-sexp` procedure, which replaces all references with dummy
+values:
+
+```scheme
+(gexp->approximate-sexp gexp-builder)
+⇒ (symlink (*approximate*) (*approximate*))
+```
+
+# The Lowerable-Object Hardware Shop
+
+We've seen two examples already of records we can turn into derivations, which
+are generally referred to as _lowerable objects_ or _file-like objects_:
+
+- `<package>`, a Guix package.
+- `<file-append>`, which wraps another lowerable object and appends a string to
+ the embedded output path when ungexped.
+
+There are many more available to us. Recall from the previous post,
+[_The Store Monad_](https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad),
+that Guix provides the two monadic procedures `text-file` and `interned-file`,
+which can be used, respectively, to put arbitrary text or files from the
+filesystem in the store, returning the path to the created item.
+
+This doesn't work so well with gexps, though; you'd have to wrap each ungexped
+use of either of them with `(with-store store (run-with-store store …))`, which
+would be quite tedious. Thankfully, `(guix gexp)` provides the `plain-file` and
+`local-file` procedures, which return equivalent lowerable objects. This code
+example builds a directory containing symlinks to files greeting the world:
+
+```scheme
+(use-modules (guix monads)
+ (ice-9 ftw)
+ (ice-9 textual-ports))
+
+(define (build-derivation monadic-drv)
+ (with-store store
+ (run-with-store store
+ (mlet* %store-monad ((drv monadic-drv))
+ (mbegin %store-monad
+ ;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS.
+ (built-derivations (list drv))
+ (return (derivation-output-path
+ (assoc-ref (derivation-outputs drv) "out"))))))))
+
+(define world-greeting-output
+ (build-derivation
+ (gexp->derivation "world-greeting"
+ #~(begin
+ (mkdir #$output)
+ (symlink #$(plain-file "hi-world"
+ "Hi, world!")
+ (string-append #$output "/hi"))
+ (symlink #$(plain-file "hello-world"
+ "Hello, world!")
+ (string-append #$output "/hello"))
+ (symlink #$(plain-file "greetings-world"
+ "Greetings, world!")
+ (string-append #$output "/greetings"))))))
+
+;; We turn the list into multiple values using (APPLY VALUES …).
+(apply values
+ (map (lambda (file-path)
+ (let* ((path (string-append world-greeting-output "/" file-path))
+ (contents (call-with-input-file path get-string-all)))
+ (list path contents)))
+ ;; SCANDIR from (ICE-9 FTW) returns the list of all files in a
+ ;; directory (including ``.'' and ``..'', so we remove them with the
+ ;; second argument, SELECT?, which specifies a predicate).
+ (scandir world-greeting-output
+ (lambda (path)
+ (not (or (string=? path ".")
+ (string=? path "..")))))))
+⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!")
+⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!")
+⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!")
+```
+
+Note that we define a procedure for building the output; we will need to build
+more derivations in a very similar fashion later, so it helps to have this to
+reuse instead of copying the code in `world-greeting-output`.
+
+There are many other useful lowerable objects available as part of the gexp
+library. These include `computed-file`, which accepts a gexp that builds
+the output file, `program-file`, which creates an executable Scheme script in
+the store using a gexp, and `mixed-text-file`, which allows you to, well, mix
+text and lowerable objects; it creates a file from the concatenation of a
+sequence of strings and file-likes. The
+[G-Expressions](https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html)
+manual page has more details.
+
+So, you may be wondering, at this point: there's so many lowerable objects
+included with the gexps library, surely there must be a way to define more?
+Naturally, there is; this is Scheme, after all! We simply need to acquaint
+ourselves with the `define-gexp-compiler` macro.
+
+The most basic usage of `define-gexp-compiler` essentially creates a procedure
+that takes as arguments a record to lower, the host system, and the target
+system, and returns a derivation or store item as a monadic value in
+`%store-monad`.
+
+Let's try implementing a lowerable object representing a file that greets the
+world. First, we'll define the record type:
+
+```scheme
+(use-modules (srfi srfi-9))
+
+(define-record-type <greeting-file>
+ (greeting-file greeting)
+ greeting?
+ (greeting greeting-file-greeting))
+```
+
+Now we use `define-gexp-compiler` like so; note how we can use `lower-object`
+to compile down any sort of lowerable object into the equivalent store item or
+derivation; essentially, `lower-object` is just the procedure for applying the
+right gexp compiler to an object:
+
+```scheme
+(use-modules (ice-9 i18n))
+
+(define-gexp-compiler (greeting-file-compiler
+ (greeting-file <greeting-file>)
+ system target)
+ (lower-object
+ (let ((greeting (greeting-file-greeting greeting-file)))
+ (plain-file (string-append greeting "-greeting")
+ (string-append (string-locale-titlecase greeting) ", world!")))))
+```
+
+Let's try it out now. Here's how we could rewrite our greetings directory
+example from before using `<greeting-file>`:
+
+```scheme
+(define world-greeting-2-output
+ (build-derivation
+ (gexp->derivation "world-greeting-2"
+ #~(begin
+ (mkdir #$output)
+ (symlink #$(greeting-file "hi")
+ (string-append #$output "/hi"))
+ (symlink #$(greeting-file "hello")
+ (string-append #$output "/hello"))
+ (symlink #$(greeting-file "greetings")
+ (string-append #$output "/greetings"))))))
+
+(apply values
+ (map (lambda (file-path)
+ (let* ((path (string-append world-greeting-2-output
+ "/" file-path))
+ (contents (call-with-input-file path get-string-all)))
+ (list path contents)))
+ (scandir world-greeting-2-output
+ (lambda (path)
+ (not (or (string=? path ".")
+ (string=? path "..")))))))
+⇒ ("/gnu/store/…-world-greeting-2/greetings" "Greetings, world!")
+⇒ ("/gnu/store/…-world-greeting-2/hello" "Hello, world!")
+⇒ ("/gnu/store/…-world-greeting-2/hi" "Hi, world!")
+```
+
+Now, this is probably not worth a whole new gexp compiler. How about something
+a bit more complex? Sharp-eyed readers who are trying all this in the REPL may
+have noticed the following output when they used `define-gexp-compiler`
+(formatted for ease of reading):
+
+```scheme
+⇒ #<<gexp-compiler>
+ type: #<record-type <greeting-file>>
+ lower: #<procedure … (greeting-file system target)>
+ expand: #<procedure default-expander (thing obj output)>>
+```
+
+Now, the purpose of `type` and `lower` is self-explanatory, but what's this
+`expand` procedure here? Well, if you recall `file-append`, you may realise
+that the text produced by a gexp compiler for embedding into a gexp doesn't
+necessarily have to be the exact output path of the produced derivation.
+
+There turns out to be another way to write a `define-gexp-compiler` form that
+allows you to specify _both_ the lowering procedure, which produces the
+derivation or store item, and the expanding procedure, which produces the text.
+
+Let's make another record; this one will let us build a store item containing a
+`bin` directory with multiple scripts inside, and expand to the full path to
+that script.
+
+```scheme
+(define-record-type <script-directory>
+ (script-directory scripts)
+ script-directory?
+ (scripts script-directory-scripts))
+```
+
+Here's how we define both a compiler and expander for our new record:
+
+```scheme
+(define-gexp-compiler script-directory-compiler <script-directory>
+ compiler => (lambda (obj system target)
+ (gexp->derivation "script-directory"
+ #~(let ((bindir (string-append #$output "/b
This message was truncated. Download the full message here.