Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

  • Done
  • quality assurance status badge
Details
4 participants
  • jbranso
  • Ludovic Courtès
  • Maxime Devos
  • zimoun
Owner
unassigned
Submitted by
Maxime Devos
Severity
normal
M
M
Maxime Devos wrote on 20 Jun 2022 23:01
(address . bug-guix@gnu.org)
1652a111d71c3e74c61904f89120ea45631c7013.camel@telenet.be
Variants of this appear repeatedly on #guix every now and then,
and also sometimes unasked by me not posted anywhere.

How do I run compile a package, computed-file or more generally a
lowerable object from the REPL? Let's look in the manual:

Toggle quote (9 lines)
> Note that the ‘(guix monad-repl)’ module extends the Guile REPL with
> new “meta-commands” to make it easier to deal with monadic
> procedures: ‘run-in-store’, and ‘enter-store-monad’. The former is
> used to “run” a single monadic value through the store:
>
> scheme@(guile-user)> ,run-in-store (package->derivation hello)
> $1 = #<derivation /gnu/store/...-hello-2.9.drv => ...>
>

Um, I want the actual store item, not the derivation, and I'm looking
into lowering a non-package, let's look elsewhere:

Toggle quote (18 lines)
> As it turns out, the call to ‘package->derivation’ can even be
> omitted since it will take place implicitly, as we will see later
> (*note G-Expressions::):
>
> (define (sh-symlink)
> (gexp->derivation "sh"
> #~(symlink (string-append #$bash
> "/bin/bash")
> #$output)))
>
> Calling the monadic ‘sh-symlink’ has no effect. As someone once
> said, “you exit a monad like you exit a building on fire: by
> running”. So, to exit the monad and get the desired effect, one must
> use ‘run-with-store’:
>
> (run-with-store (open-connection) (sh-symlink))
> ⇒ /gnu/store/...-sh-symlink

Let's try this (using the REPL command):

scheme@(guix-user)> ,run-in-store (plain-file "foo" "bar")
While executing meta-command:
Wrong type to apply: #<<plain-file> name: "foo" content: "bar"
references: ()>

doesn't work, let's try the 'lower-object' procedure which I vaguely
recall to helped me out in the past but doesn't have any actual
examples in the manual:

scheme@(guix-user)> ,run-in-store (lower-object (plain-file "foo" "bar"))
$15 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"

This looks useful, let's try it for mixed-text-file:

scheme@(guix-user)> ,run-in-store (lower-object (mixed-text-file "foo"
"bar"))
$17 = #<derivation /gnu/store/ah3yfw0nvw6jaif56r121sgwjiygmvrk-foo.drv
=> /gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo 7f862e53f820>

nooo there's the #<derivation ...> again how do I get a compiled mixed-
text-file from the REPL?

(I mean I could use 'build-derivations' from (guix)The Store, but that
accepts a list, not an individual derivation, and it doesn't accept strings
according to the documentation so it cannot be used for the 'plain-file'
and aren't derivations a super-low-level thing for which G-exps and such
were intended to make the low level thing almost an implementation detail?)

Seems like we are missing a general & documented & simple to use
'lower-object-completely' (or maybe 'compile-object'?) procedure for
this ... And maybe also a ,build-object REPL command?

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYrDgQhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7lnWAP4t9eKDgA5f5zUxI/tGQAC4cIfQ
/eWet8YBdeH0L8X2WQEArAYhoWZtS7vv3AmJ81EZelUMrJUtmO5Jv5zgYlacOgs=
=Iq/q
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 3 Jul 2022 22:50
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 56114@debbugs.gnu.org)
87edz1zzrz.fsf@gnu.org
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (4 lines)
> Seems like we are missing a general & documented & simple to use
> 'lower-object-completely' (or maybe 'compile-object'?) procedure for
> this ... And maybe also a ,build-object REPL command?

How about the attached patch?

Sample session:

Toggle snippet (17 lines)
scheme@(guile-user)> ,use(gnu packages base)
scheme@(guile-user)> ,build coreutils
$11 = "/gnu/store/y8933036ljrz4ah9zcph09nmvdmmv5sf-coreutils-8.32-debug"
$12 = "/gnu/store/8fpk2cja3f07xls48jfnpgrzrljpqivr-coreutils-8.32"
scheme@(guile-user)> ,lower coreutils
$13 = #<derivation /gnu/store/nc93q3hmlzcgdn6jr7dv3j2m50ivn55f-coreutils-8.32.drv => /gnu/store/y8933036ljrz4ah9zcph09nmvdmmv5sf-coreutils-8.32-debug /gnu/store/8fpk2cja3f07xls48jfnpgrzrljpqivr-coreutils-8.32 7f0ebaf821e0>
scheme@(guile-user)> ,build (computed-file "foo" #~(begin (display "hi!\n")(mkdir #$output)(mkdir #$output:lib)))
$14 = "/gnu/store/axij9bkg56xv3k1z0l20gd6b0swn14sy-foo-lib"
$15 = "/gnu/store/h5s7k7m0fxk9n7q7729l3n4q7vyxnvpy-foo"
scheme@(guile-user)> ,build (computed-file "foo" #~(begin (display "Goeden dag!\n")(mkdir #$output)))
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0%
building /gnu/store/gplhka9g0bwv8b640zavw1z65y9zrvag-foo.drv...
$16 = "/gnu/store/ynvga6gxwj68mmk8ddj3i9sjhavkqah1-foo"

“lower” does what you would expect; “build” yields one value per output.

If that works for you, I’ll update the manual accordingly, and we can
always add more features (like build options) later on. Thoughts?

Thanks,
Ludo’.
Toggle diff (88 lines)
diff --git a/guix/monad-repl.scm b/guix/monad-repl.scm
index aefabdeebb..15c10efe01 100644
--- a/guix/monad-repl.scm
+++ b/guix/monad-repl.scm
@@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2015, 2016, 2022 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -21,6 +21,12 @@ (define-module (guix monad-repl)
#:use-module (guix monads)
#:use-module (guix utils)
#:use-module (guix packages)
+ #:use-module (guix status)
+ #:autoload (guix gexp) (lower-object)
+ #:use-module ((guix derivations)
+ #:select (derivation?
+ derivation->output-paths built-derivations))
+ #:use-module (ice-9 match)
#:use-module (ice-9 pretty-print)
#:use-module (system repl repl)
#:use-module (system repl common)
@@ -69,16 +75,56 @@ (define (store-monad-language store)
#:guile-for-build guile)
'store-monad)))
+(define %build-verbosity 1)
+
+(define* (evaluate/print-with-store mvalue #:key build?)
+ "Run monadic value MVALUE in the store monad and print its value."
+ (with-store store
+ (set-build-options store
+ #:print-build-trace #t
+ #:print-extended-build-trace? #t
+ #:multiplexed-build-output? #t)
+ (with-status-verbosity %build-verbosity
+ (let* ((guile (or (%guile-for-build)
+ (default-guile-derivation store)))
+ (values (run-with-store store
+ (if build?
+ (mlet %store-monad ((obj mvalue))
+ (if (derivation? obj)
+ (mbegin %store-monad
+ (built-derivations (list obj))
+ (return
+ (match (derivation->output-paths obj)
+ (((_ . files) ...) files))))
+ (return (list obj))))
+ (mlet %store-monad ((obj mvalue))
+ (return (list obj))))
+ #:guile-for-build guile)))
+ (for-each (lambda (value)
+ (run-hook before-print-hook value)
+ (pretty-print value))
+ values)))))
+
(define-meta-command ((run-in-store guix) repl (form))
"run-in-store EXP
Run EXP through the store monad."
- (with-store store
- (let* ((guile (or (%guile-for-build)
- (default-guile-derivation store)))
- (value (run-with-store store (repl-eval repl form)
- #:guile-for-build guile)))
- (run-hook before-print-hook value)
- (pretty-print value))))
+ (evaluate/print-with-store (repl-eval repl form)))
+
+(define-meta-command ((verbosity guix) repl (level))
+ "verbosity LEVEL
+Change build verbosity to LEVEL."
+ (set! %build-verbosity level))
+
+(define-meta-command ((lower guix) repl (form))
+ "lower OBJECT
+Lower OBJECT into a derivation and return it."
+ (evaluate/print-with-store (lower-object (repl-eval repl form))))
+
+(define-meta-command ((build guix) repl (form))
+ "build OBJECT
+Lower OBJECT and build it, returning its output file name(s)."
+ (evaluate/print-with-store (lower-object (repl-eval repl form))
+ #:build? #t))
(define-meta-command ((enter-store-monad guix) repl)
"enter-store-monad
Z
Z
zimoun wrote on 4 Jul 2022 17:43
(name . Ludovic Courtès)(address . ludo@gnu.org)
87y1x8nasl.fsf@gmail.com
Hi,

If I read correctly, this snippet from the manual,

Toggle quote (3 lines)
>> scheme@(guile-user)> ,run-in-store (package->derivation hello)
>> $1 = #<derivation /gnu/store/...-hello-2.9.drv => ...>

becomes

scheme@(guile-user)> ,lower hello

and this snippet,

Toggle quote (3 lines)
> scheme@(guix-user)> ,run-in-store (lower-object (plain-file "foo" "bar"))
> $15 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"

becomes

scheme@(guile-user)> ,build (plain-file "foo" "bar")


However, note that,

Toggle snippet (6 lines)
scheme@(guix-user)> ,build (plain-file "foo" "bar")
$1 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"
scheme@(guix-user)> ,lower (plain-file "foo" "bar")
$2 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"

which is confusing, especially when,

Toggle snippet (7 lines)
scheme@(guix-user)> ,lower (mixed-text-file "foo" "bar")
$6 = #<derivation /gnu/store/40982nzm6c7n4yawvs3m39k313jsrl80-foo.drv => /gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo 7f6f9bc5cd20>
scheme@(guix-user)> ,build (mixed-text-file "foo" "bar")
$7 = "/gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo"


The issue – unrelated to the nice patch – is a potential “inconsistency”
between ’<plain-file>’ and ’<computed-file>’; ’plain-file’ refers to
’text-file’ and ’mixed-text-file’ refers to ’text-file*’; and both are
really different despite close names around. :-)

Maybe the docstring of ’plain-file’ and of ’mixed-text-file’ could be
extended. For instance,

Toggle snippet (7 lines)
(define (plain-file name content)
"Return an lowered object representing a text file called NAME with the given
CONTENT (a string) to be added to the store.

This is the declarative counterpart of 'text-file'."

Toggle snippet (13 lines)
(define* (mixed-text-file name #:key guile #:rest text)
"Return an object representing store file NAME containing TEXT. TEXT is a
sequence of strings and file-like objects, as in:

(mixed-text-file \"profile\"
\"export PATH=\" coreutils \"/bin:\" grep \"/bin\")

This is the declarative counterpart of 'text-file*' but note that a
'computed-file' object is returned."



Toggle quote (6 lines)
> diff --git a/guix/monad-repl.scm b/guix/monad-repl.scm
> index aefabdeebb..15c10efe01 100644
> --- a/guix/monad-repl.scm
> +++ b/guix/monad-repl.scm
> @@ -1,5 +1,5 @@

All LGTM.


Toggle quote (3 lines)
> +(define* (evaluate/print-with-store mvalue #:key build?)
> + "Run monadic value MVALUE in the store monad and print its value."

I do not know if it makes sense to have a list of monadic values,
allowing something like:

,build hello coreutils



Cheers,
simon
L
L
Ludovic Courtès wrote on 4 Jul 2022 22:13
(name . zimoun)(address . zimon.toutoune@gmail.com)
8735fg8wlr.fsf@gnu.org
Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (15 lines)
> However, note that,
>
> scheme@(guix-user)> ,build (plain-file "foo" "bar")
> $1 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"
> scheme@(guix-user)> ,lower (plain-file "foo" "bar")
> $2 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"
>
>
> which is confusing, especially when,
>
> scheme@(guix-user)> ,lower (mixed-text-file "foo" "bar")
> $6 = #<derivation /gnu/store/40982nzm6c7n4yawvs3m39k313jsrl80-foo.drv => /gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo 7f6f9bc5cd20>
> scheme@(guix-user)> ,build (mixed-text-file "foo" "bar")
> $7 = "/gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo"

Right, that’s because ‘plain-file’ lowers directly to a store item, not
to a derivation.

I explicitly made “,build” hide that difference; I thought one would
just want “,build” to do the right thing and display the resulting store
item regardless of that kind of detail.

Conversely, “,lower” is for die-hard hackers and it doesn’t hide any
subtlety.

Toggle quote (8 lines)
>> +(define* (evaluate/print-with-store mvalue #:key build?)
>> + "Run monadic value MVALUE in the store monad and print its value."
>
> I do not know if it makes sense to have a list of monadic values,
> allowing something like:
>
> ,build hello coreutils

I don’t think the REPL can support this syntax, but we could add support
for:

,build (list hello coreutils)

Thanks for your feedback!

Ludo’.
Z
Z
zimoun wrote on 4 Jul 2022 23:59
(name . Ludovic Courtès)(address . ludo@gnu.org)
865ykczgh7.fsf@gmail.com
Hi,

On Mon, 04 Jul 2022 at 22:13, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (7 lines)
> I explicitly made “,build” hide that difference; I thought one would
> just want “,build” to do the right thing and display the resulting store
> item regardless of that kind of detail.
>
> Conversely, “,lower” is for die-hard hackers and it doesn’t hide any
> subtlety.

Both make sense. :-)


Toggle quote (3 lines)
> Right, that’s because ‘plain-file’ lowers directly to a store item, not
> to a derivation.

My comment was about an unrelated thing. Well, I do not use every day
G-exp nor ’plain-file’ and the like. The interface appears to me
unexpected each time I use it but hey I can live with it. :-) However,
the documentation (or docstring) does not appear clear, each time I have
my «aaah right!» moment. For instance, the manual says:

The local-file, plain-file, computed-file, program-file, and
scheme-file procedures below return file-like objects. That is,
when unquoted in a G-expression, these objects lead to a file in
the store.

and the docstrings say:

Toggle snippet (7 lines)
(define (plain-file name content)
"Return an object representing a text file called NAME with the given
CONTENT (a string) to be added to the store.

This is the declarative counterpart of 'text-file'."

or

Toggle snippet (10 lines)
(define* (computed-file name gexp
#:key guile (local-build? #t) (options '()))
"Return an object representing the store item NAME, a file or directory
computed by GEXP. When LOCAL-BUILD? is #t (the default), it ensures the
corresponding derivation is built locally. OPTIONS may be used to pass
additional arguments to 'gexp->derivation'.

This is the declarative counterpart of 'gexp->derivation'."

when ’plain-file’ and ’computed-file’ does not return the same type of
“file-like object“. Especially when reading,

Toggle snippet (10 lines)
(define* (mixed-text-file name #:key guile #:rest text)
"Return an object representing store file NAME containing TEXT. TEXT is a
sequence of strings and file-like objects, as in:

(mixed-text-file \"profile\"
\"export PATH=\" coreutils \"/bin:\" grep \"/bin\")

This is the declarative counterpart of 'text-file*'."

it is hard to guess beforehand that ’plain-file’ and ’mixed-text-file’
do not return the same type of file-like object.

Therefore, I was thinking to tweak these docstrings to better underline
their difference. Well, maybe it is only me. :-)


Cheers,
simon


PS:
It remembers me unrelated [1] mentioning confusion about “file or
file-like object”. :-)

L
L
Ludovic Courtès wrote on 5 Jul 2022 09:57
(name . zimoun)(address . zimon.toutoune@gmail.com)
87wncs56w0.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (2 lines)
> My comment was about an unrelated thing.

I don’t take comments about unrelated things. ;-)

Toggle quote (5 lines)
> Well, I do not use every day G-exp nor ’plain-file’ and the like. The
> interface appears to me unexpected each time I use it but hey I can
> live with it. :-) However, the documentation (or docstring) does not
> appear clear, each time I have my «aaah right!» moment.

[...]

Toggle quote (3 lines)
> it is hard to guess beforehand that ’plain-file’ and ’mixed-text-file’
> do not return the same type of file-like object.

As someone who writes gexps and manipulates “file-like objects”, it
doesn’t matter whether a thing lowers to a derivation or to a plain
store item. What matters is that you can insert it in a gexp and it’ll
do the right thing. That’s why the docstrings don’t insist on that
difference.

Hope it makes sense!

Ludo’.
Z
Z
zimoun wrote on 5 Jul 2022 12:13
(name . Ludovic Courtès)(address . ludo@gnu.org)
CAJ3okZ3PRG+C2OgGNFkc8e3poZGrHExKB1f2QDNwhW08ZMw0Ug@mail.gmail.com
Hi,

On Tue, 5 Jul 2022 at 09:57, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)
> I don’t take comments about unrelated things. ;-)

So one more. ;-)


Toggle quote (8 lines)
> As someone who writes gexps and manipulates “file-like objects”, it
> doesn’t matter whether a thing lowers to a derivation or to a plain
> store item. What matters is that you can insert it in a gexp and it’ll
> do the right thing. That’s why the docstrings don’t insist on that
> difference.
>
> Hope it makes sense!

The difference is explained nowhere and to understand why the returned
call is different, then one has to read the code; it is what I did.
:-)

Well, "do the right thing" is an expectation based on an understanding
about what the "right thing" means. :-) When I am manipulating
"file-like objects", I am always confused by what it means and thus it
appears to me that it does not "do the right thing" and it is hard to
understand why.

Yes it makes sense to not insistt on that difference once you have
clear understanding of this very difference. However, when learning
G-Exp -- which is already unusual -- "file-like objects" and how to
manipulate them is hard to grasp because the difference is never
written down. I still think the docstring or the manual should
mention such difference.

Explicit is better than implicit.
In the face of ambiguity, refuse the temptation to guess.

– The Zen of Python, by Tim Peters –

Anyway! That's unrelated. ;-)


Cheers,
simon
J
J
jbranso wrote on 8 Jul 2022 00:25
Re: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.
(address . 56114@debbugs.gnu.org)
f99a1cefe2af96e84e4e7427ebdee7e8@dismail.de
I was playing around with this today, and it is awesome!

I think it should be merged, because it has already helped me with my opensmtpd records project.

Thanks,

Joshua
Attachment: file
J
J
Joshua Branson wrote on 10 Jul 2022 22:03
Re: bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.
(name . zimoun)(address . zimon.toutoune@gmail.com)
87mtdgohuq.fsf@dismail.de
Whoops. I sent my original message to the list, not directly to ya'll.

,build and ,lower are awesome! I reccomend that we add ludo's proposed
patch into guix. It's awesome!

Thanks,

Joshua
L
L
Ludovic Courtès wrote on 15 Jul 2022 17:41
(name . Joshua Branson)(address . jbranso@dismail.de)
87o7xq5qob.fsf@gnu.org
Hi!

Thanks all for your support. :-)

I’ve pushed these new REPL commands as
4ce7f1fb24a111f3e92d5b889d1271bebf109d09, including a new “Using Guix
Interactively” section in the manual to document it. Feedback welcome!

I’m sure we can add more REPL commands as we better see what useful
shorthands would be welcome.

Cheers,
Ludo’.
Closed
M
M
Maxime Devos wrote on 19 Jul 2022 16:10
42461a055f438b7a01d7783a3daaa68c3507dbfb.camel@telenet.be
Ludovic Courtès schreef op vr 15-07-2022 om 17:41 [+0200]:
Toggle quote (14 lines)
> Hi!
>
> Thanks all for your support.  :-)
>
> I’ve pushed these new REPL commands as
> 4ce7f1fb24a111f3e92d5b889d1271bebf109d09, including a new “Using Guix
> Interactively” section in the manual to document it.  Feedback welcome!
>
> I’m sure we can add more REPL commands as we better see what useful
> shorthands would be welcome.
>
> Cheers,
> Ludo’.

Works nicely:

scheme@(guix-user)> ,build (@ (gnu packages base) hello)
[...]
$1 = "/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1"

though an actual documented _procedure_ (not command) seems to be
missing, so maybe the issue needs to be reopened later, don't know yet
...

Greetings,
Maxime.
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 56114@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 56114
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch