Mutable guix shell environments

  • Open
  • quality assurance status badge
Details
5 participants
  • Charles
  • Liliana Marie Prikler
  • Ludovic Courtès
  • Maxime Devos
  • zimoun
Owner
unassigned
Submitted by
Charles
Severity
normal
C
C
Charles wrote on 13 Mar 2022 19:21
(name . guix-patches@gnu.org)(address . guix-patches@gnu.org)
Zycszoymk_U1ui1Ek6NODmj1rphro1ZjtzltZf6rnv7wtFAxmj0DW8IRWerZaW3PkR80EmJ6Fru8R7pp9dzDpPoh-FFzqeGAsB_Ivs4Ipr4=@protonmail.com
sample useage:
$ cd project
$ guix package --manifest=manifest.scm --profile=.guix-profile
$ guix shell # --profile=.guix-profile is implicit

do some stuff. realize that you want to bring in another package.

$ guix install --profile=.guix-profile additional-package

Then additional package is instantly available. This is especially useful to lisp programmers because, currently, bringing in an additional library involves restarting shell, lisp process, reloading source files, regenerating process state.
From 47c4c6f9896c2b4b884ff11063d33f5458cbecf7 Mon Sep 17 00:00:00 2001
From: Charles <charles.b.jackson@protonmail.com>
Date: Sun, 13 Mar 2022 12:58:25 -0500
Subject: [PATCH 2/2] guix: shell: Implicitly use a .guix-profile as --profile
option.

* guix/scripts/shell.scm (auto-detect-manifest): Add ".guix-profile" to
matches as --profile option
---
guix/scripts/shell.scm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

Toggle diff (30 lines)
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 1eab05d737..fca41cc2d4 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2022 Charles Jackson <charles.b.jackson@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -256,7 +257,7 @@ (define disallow-implicit-load?
disallow-implicit-load?
(options-contain-payload? opts))
opts
- (match (find-file-in-parent-directories '("manifest.scm" "guix.scm"))
+ (match (find-file-in-parent-directories '(".guix-profile" "manifest.scm" "guix.scm"))
(#f
(warning (G_ "no packages specified; creating an empty environment~%"))
opts)
@@ -265,6 +266,7 @@ (define disallow-implicit-load?
(begin
(info (G_ "loading environment from '~a'...~%") file)
(match (basename file)
+ (".guix-profile" (alist-cons 'profile file opts))
("guix.scm" (alist-cons 'load `(package ,file) opts))
("manifest.scm" (alist-cons 'manifest file opts))))
(begin
--
2.34.0
L
L
Liliana Marie Prikler wrote on 13 Mar 2022 20:51
68a3e146199e408d50e4ed751e1b7000364704f5.camel@gmail.com
Am Sonntag, dem 13.03.2022 um 18:21 +0000 schrieb Charles:

Toggle quote (5 lines)
> Subject: [PATCH 2/2] guix: shell: Implicitly use a .guix-profile as -
> -profile option.
>
> * guix/scripts/shell.scm (auto-detect-manifest): Add ".guix-profile"
> to matches as --profile option
This one LGTM and it even avoids some problems we're having with
caches. I'll keep it open for bikeshedding suggestions w.r.t. the name
of the directory to detect, though.

Toggle quote (10 lines)
> Subject: [PATCH 1/2] guix: environment: Enable mutable environments.
>
> * guix/scripts/environment.scm (launch-environment launch-
> environment/fork
> launch-environment/container guix-environment*): Add #:set-profile?
> parameter set GUIX_PROFILE
> when --profile option is used.
> * guix/scripts/environment.scm (guix-environment*): Profile could
> point to a
> profile directory instead of a store directory.
This one not so much. You already have GUIX_ENVIRONMENT set, which
should be enough for lookup purposes. If it's about manipulating PATH
and other environment variables, I think it'd better to do this
manually – you could even spawn a lightweight shell on your own by
simply doing the following:

$ $SHELL
$ GUIX_PROFILE=/path/to/profile
$ source $GUIX_PROFILE/etc/profile

If you want mutable environments, that's the easiest and imho best way
of achieving such a thing. Guix is functional by design and we should
not compromise on that.

As far as using $GUIX_PROFILE together with `guix shell' is concerned,
if anything is even holding it back, it's that we don't respect
GUIX_PROFILE in the Scheme code that adds those variables. Which to be
fair might be for the best, as GUIX_PROFILE is not guaranteed to
correspond to this invocation of `guix shell' when people are careless.
That's not to say that support for this couldn't be added, but at the
very least we'd have to be more careful about it.

Cheers
C
C
Charles wrote on 14 Mar 2022 00:38
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)(address . 54375@debbugs.gnu.org)
9QXS8i-Ylzc9pxojB3WT0LMPFqm00N-jwx08sZnPLn7KmSvC85K7jvgGmQTcg138EEa6Aqh7W4n87KXkZvxJyt--ze0rCVnGVG2SMskLgNU=@protonmail.com
Thanks for taking a look Liliana. I'm glad you like guix shell detecting a local profile.

Toggle quote (4 lines)
> $ $SHELL
> $ GUIX_PROFILE=/path/to/profile
> $ source $GUIX_PROFILE/etc/profile

This is basically what I was going for. I just find this useful to be built in. When I think "guix shell", I think: "make an environment where I have everything I need". If I want to adjust what I need, why do I need to restart everything from scratch. If I did not know about guix shell and did all my development using ~/.guix-profile, this functionallity would already be available.

Toggle quote (1 lines)
> Guix is functional by design and we should
not compromise on that.

I totally agree, and I do not think this does compromise it. With my patch, guix shells using guix.scm or manifest, would be unchanged. Currently, guix shell --profile, has strickly less functionallity than using the profile otherwise, this could be confusing (as it is to me). Profiles are not really mutable since they use generations; the subject line is a bit of a misnomer.

Toggle quote (2 lines)
> As far as using $GUIX_PROFILE tog...

Yeah, before my patch, guix shell --profile, would not be set to the profile I'm using. This actually makes sense, because, we are not really using it. Guix shell just happens to be using all the same packages that were available in that profile. Confusingly, If the profile, is updated, the shell is now out of sync with it. My patch should make using guix shell --profile more intuitive.


------- Original Message -------

On Sunday, March 13th, 2022 at 2:51 PM, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

Toggle quote (63 lines)
> Am Sonntag, dem 13.03.2022 um 18:21 +0000 schrieb Charles:
>
> > Subject: [PATCH 2/2] guix: shell: Implicitly use a .guix-profile as -
> >
> > -profile option.
> >
> > * guix/scripts/shell.scm (auto-detect-manifest): Add ".guix-profile"
> >
> > to matches as --profile option
>
> This one LGTM and it even avoids some problems we're having with
>
> caches. I'll keep it open for bikeshedding suggestions w.r.t. the name
>
> of the directory to detect, though.
>
> > Subject: [PATCH 1/2] guix: environment: Enable mutable environments.
> >
> > * guix/scripts/environment.scm (launch-environment launch-
> >
> > environment/fork
> >
> > launch-environment/container guix-environment*): Add #:set-profile?
> >
> > parameter set GUIX_PROFILE
> >
> > when --profile option is used.
> >
> > * guix/scripts/environment.scm (guix-environment*): Profile could
> >
> > point to a
> >
> > profile directory instead of a store directory.
>
> This one not so much. You already have GUIX_ENVIRONMENT set, which
>
> should be enough for lookup purposes. If it's about manipulating PATH
>
> and other environment variables, I think it'd better to do this
>
> manually – you could even spawn a lightweight shell on your own by
>
> simply doing the following:
>
> $ $SHELL
>
> $ GUIX_PROFILE=/path/to/profile
>
> $ source $GUIX_PROFILE/etc/profile
>
> If you want mutable environments, that's the easiest and imho best way
>
> of achieving such a thing. Guix is functional by design and we should
>
> not compromise on that.
>
> As far as using $GUIX_PROFILE together with `guix shell' is concerned, if anything is even holding it back, it's that we don't respect GUIX_PROFILE in the Scheme code that adds those variables. Which to be fair might be for the best, as GUIX_PROFILE is not guaranteed to correspond to this invocation of` guix shell' when people are careless.
>
> That's not to say that support for this couldn't be added, but at the
>
> very least we'd have to be more careful about it.
>
> Cheers
L
L
Liliana Marie Prikler wrote on 14 Mar 2022 06:19
(name . Charles)(address . charles.b.jackson@protonmail.com)(address . 54375@debbugs.gnu.org)
73d3dfc2ceebd68f6afc9af6ae96391933592e07.camel@gmail.com
Hi Charles,

Am Sonntag, dem 13.03.2022 um 23:38 +0000 schrieb Charles:
Toggle quote (13 lines)
> Thanks for taking a look Liliana. I'm glad you like guix shell
> detecting a local profile.
>
> > $ $SHELL
> > $ GUIX_PROFILE=/path/to/profile
> > $ source $GUIX_PROFILE/etc/profile
>
> This is basically what I was going for. I just find this useful to be
> built in. When I think "guix shell", I think: "make an environment
> where I have everything I need". If I want to adjust what I need, why
> do I need to restart everything from scratch. If I did not know about
> guix shell and did all my development using ~/.guix-profile, this
> functionallity would already be available.
Yes, but it'd also be broken af. You are focusing on a very particular
use case and ignoring all others. What if instead of just "a package"
I wanted to add or remove a package that defines a search path, e.g.
Emacs? In the adding case, EMACSLOADPATH would not get defined and in
the removing case it would still linger. What you describe as desired
functionality is not at all well-defined. Using `guix shell --profile'
on the other hand has the same semantics as the other `guix shell'
commands, and is thus imo less confusing once you start to think about
it.

Cheers
C
C
Charles wrote on 14 Mar 2022 07:24
(name . Liliana Marie Prikler)(address . liliana.prikler@gmail.com)(address . 54375@debbugs.gnu.org)
U42QS9XA3iu83SyzCgtOLO_Ody998pm5fmz1JS7nn2tzCjuf8paq5w-Vonz0745nSoRcM7t_kYsL4BEgiERmxl0NxQLIJ33L1k9QliKssgg=@protonmail.com
Toggle quote (2 lines)
> What if instead of just "a package" I wanted to add or remove a package that defines a search path

As with ~/.guix-profile, you will have to . "$GUIX_PROFILE/etc/profile" just like the hint says. For removing, they linger, that is exactly what the default profile does too.

This comes down to us disagreeing which is more important & useful: guix shell semantics vs profile semantics. guix shell --profile could go either way.

Another way to look at it is that I this sequence of 3 commands very useful (so might others). They clearly have something to do with guix profiles, and they spawn a new shell. Thus I thought guix shell --profile was the perfect place for them. You claim they do not belong there. Is there a better place you would suggest?
M
M
Maxime Devos wrote on 14 Mar 2022 18:43
Re: [bug#54375] Mutable guix shell environments
70714172793d0ad63c4c173606e4fe9ae6b4b228.camel@telenet.be
Charles via Guix-patches via schreef op zo 13-03-2022 om 18:21 [+0000]:
Toggle quote (11 lines)
> sample useage:
> $ cd project
> $ guix package --manifest=manifest.scm --profile=.guix-profile
> $ guix shell # --profile=.guix-profile is implicit
>
> do some stuff. realize that you want to bring in another package.
>
> $ guix install --profile=.guix-profile additional-package
>
> Then additional package is instantly available. This is especially useful to lisp programmers because, currently, bringing in an additional library involves restarting shell, lisp process, reloading source files, regenerating process state.

If this is added, then this needs to be documented in the manual,
preferably with some examples on how to use it.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYi9+qhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7tgvAQCth5d7r2ZniVCh2VgNmxhXV67L
sj0ITedZrfJnsFeYJQD8CUkX4qzWa8KQQJAXa6GUXDM79DQywju1U4xjRx4DrAQ=
=eiKs
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 14 Mar 2022 18:54
6aaf3dfaa6c207b2968b756fa86806232227e2be.camel@telenet.be
Charles via Guix-patches via schreef op zo 13-03-2022 om 18:21 [+0000]:
Toggle quote (5 lines)
> sample useage:
> $ cd project
> $ guix package --manifest=manifest.scm --profile=.guix-profile
> $ guix shell # --profile=.guix-profile is implicit

Alternative suggestion, which IMHO fits the non-persistent naturre of
"guix shell" better:

$ cd project
$ guix shell --allow-temporary-modifications # fails on read-only file systems
$ echo $GUIX_ENVIRONMENT
# a temporary (mutable) profile was created
Toggle quote (1 lines)
> $HOME/.cache/.../profile
# install things in the $GUIX_ENVIRONMENT profile
$ guix install foo bar ...

As noted by lilyp, this won't work in all situations due to search path
issues. These could be resolved by always setting a few search paths
(PATH, EMACSLOADPATH, GUILE_LOAD_PATH, INFOPATH, ...) (even in not currently
present in the profile!), at cost of having to make them ‘magical’ in some
sense, which ludo does not seem to like IIUC. (This could also be done
for ~/.guix-profile)

(I don't like it much either, but it seems a very convenient solution that
does not seem to have any concrete downsides and IMHO it seems much better
than telling the user to re-source the profile.)

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYi+BbBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7uDOAQCjicNVh1niTT00xHk+NBo7MyY6
IGiGV+T3+Nu1ov+ZdwD+Lw/9ReUFPNadRiKk98TkqpeXj41ePPHbw3oXJOTa7gM=
=vAKG
-----END PGP SIGNATURE-----


C
C
Charles wrote on 14 Mar 2022 20:41
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 54375@debbugs.gnu.org)
fnqU30fxWa3R4AOgelUn70VfGNWYY6on5nZ402TSQwRKesHShZBvNwzyuOZvOtccCtl47LSv_oQ1KcV-l4rtNlw2x3cPBjzCfM_xpY3_ZxY=@protonmail.com
Thanks for taking a look Maxime.

I actually like Your alternate suggestion better (though the name --allow-temporary-modifications is a bit long). I'm not sure how to implement it though.
L
L
Ludovic Courtès wrote on 14 Mar 2022 23:18
Re: bug#54375: Mutable guix shell environments
(name . Maxime Devos)(address . maximedevos@telenet.be)
87pmmop490.fsf_-_@gnu.org
Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (12 lines)
> Charles via Guix-patches via schreef op zo 13-03-2022 om 18:21 [+0000]:
>> sample useage:
>> $ cd project
>> $ guix package --manifest=manifest.scm --profile=.guix-profile
>> $ guix shell # --profile=.guix-profile is implicit
>
> Alternative suggestion, which IMHO fits the non-persistent naturre of
> "guix shell" better:
>
> $ cd project
> $ guix shell --allow-temporary-modifications # fails on read-only file systems

Or ‘--transient’.

Toggle quote (13 lines)
> $ echo $GUIX_ENVIRONMENT
> # a temporary (mutable) profile was created
>> $HOME/.cache/.../profile
> # install things in the $GUIX_ENVIRONMENT profile
> $ guix install foo bar ...
>
> As noted by lilyp, this won't work in all situations due to search path
> issues. These could be resolved by always setting a few search paths
> (PATH, EMACSLOADPATH, GUILE_LOAD_PATH, INFOPATH, ...) (even in not currently
> present in the profile!), at cost of having to make them ‘magical’ in some
> sense, which ludo does not seem to like IIUC. (This could also be done
> for ~/.guix-profile)

I like this approach.

Charles’ proposal reminds me of ‘module’, a venerable “environment
management” tool widely used in high-performance computing (HPC):


That command allows for an incremental style like Charles proposes:

module load gcc # adds GCC to $PATH
module load libgc # adds libgc to $C_INCLUDE_PATH
module unload libgc # removes libgc from $C_INCLUDE_PATH

For this to work though, ‘module’ is actually a shell function: that
lets it adjust environment variables here and now.

Anyway, I digress…

I thought about this other approach:

1. ‘guix shell’ always builds a new profile, as it already does.

2. When ‘GUIX_ENVIRONMENT’ is defined, it computes search paths as a
combination of the new profile and that pointed to by
$GUIX_ENVIRONMENT.

3. It spawns a new shell, like it already does.

Thus, instead of doing upfront:

guix shell a b c

One could do:

$ guix shell a
[env]$ guix shell b
[env]$ guix shell c

It is more accurate but less flexible than what you propose though,
because only child processes of the nested shells would see the
“changes”. So maybe not a good idea, after all.

Hmm, need more thought…

Thanks,
Ludo’.
Z
Z
zimoun wrote on 15 Mar 2022 10:49
Re: [bug#54375] Mutable guix shell environments
(name . Ludovic Courtès)(address . ludo@gnu.org)
CAJ3okZ0g5PcEmaMRJd6dubMaTXXNXGdY19LeZebzHRmEcOvxKg@mail.gmail.com
Hi,

On Mon, 14 Mar 2022 at 23:19, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (9 lines)
> That command allows for an incremental style like Charles proposes:
>
> module load gcc # adds GCC to $PATH
> module load libgc # adds libgc to $C_INCLUDE_PATH
> module unload libgc # removes libgc from $C_INCLUDE_PATH
>
> For this to work though, ‘module’ is actually a shell function: that
> lets it adjust environment variables here and now.

While I understand the need of Charles's proposal, especially in
development or exploration, I am convinced that this "incremental
style" is the root of many unreproducible computational environments.
The "incremental style" is a quick and dirty approach for creating a
computational environment and because the discipline is wrong then it
lead more than often to hard-to-reproduce computational environment,
therefore Guix should not try to mimick, IMHO. I agree with Liliana's
words: «Guix is functional by design and we should not compromise on
that.». To me, it is as programmers being strongly used to imperative
style complaining about the paradigm shift of the functional style --
or vice-versa... I digress. :-)

To mitigate because we have to work and sadly need to have things done...

Toggle quote (6 lines)
> One could do:
>
> $ guix shell a
> [env]$ guix shell b
> [env]$ guix shell c

..somehow, it would ease if when requiring the package 'b', the
environment is "aware" of the search paths of the environment required
by the package 'a'. For instance,

Toggle snippet (23 lines)
$ guix shell man-db
[env]$ tree $MANPATH
/gnu/store/m3i9pga6rqfg342sajzww4gl8w51q6sz-profile/share/man
??? index.db ->
/gnu/store/pdh5nz4qkg5q243q5rgxkk1hnmaf1plq-manual-database/share/man/index.db
??? it -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/it
??? man1 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man1
??? man5 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man5
??? man8 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man8

4 directories, 1 file

[env]$ guix shell coreutils
[env]$ tree $MANPATH
/gnu/store/m3i9pga6rqfg342sajzww4gl8w51q6sz-profile/share/man
??? index.db ->
/gnu/store/pdh5nz4qkg5q243q5rgxkk1hnmaf1plq-manual-database/share/man/index.db
??? it -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/it
??? man1 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man1
??? man5 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man5
??? man8 -> /gnu/store/vg7g63qddkd8jxlhlm9j1wxxj8wnn2mx-man-db-2.9.4/share/man/man8

Here, 'coreutils' is not "aware" of the parent 'man-db' and thus
MANPATH does not contains the expected 114 files.

Toggle snippet (6 lines)
$ guix shell coreutils man-db
[env]$ tree $MANPATH
[...]
4 directories, 114 files

Using an option, say '--transient', a new profile is created each time
but what this new profile contains would depends of the parent
profile. Last, the manipulation of a "temporary" profile is done via
"guix shell" thus this "transient" profile should be manipulated via
"guix shell" and not "guix package" (or aliases as "guix install").
However, what could appear still annoying is the stack of environment:

guix shell a
guix shell b
guix shell c

then remove the package 'a' from the environment requires to exit 2
environments and then re-create them. Need more thoughts. :-)


Cheers,
simon
C
C
Charles wrote on 16 Mar 2022 00:52
qvaBorHu9_h-bScZCkdB5pnXDKlNl4C6FmGVP_sMHD4YtVl2rG4W7kDaatQh4w4CU2hfoy9brbt_1C1iEptKo_G-XAhVEn-L81Wx-PEHUn0=@protonmail.com
Hello simon; thank you for your input.

Toggle quote (2 lines)
> The "incremental style" is a quick and dirty approach for creating a computational environment

If profiles are used, it can be exported to a manifest.

Toggle quote (2 lines)
> Guix is functional by design and we should not compromise on that.

I want to reiterate that I am not proposing anything less FP than profiles.

Nested shells do not fit my use case because a long running process started in shell "a" does not become aware of packages installed on shell "b".

Toggle quote (2 lines)
> thus this "transient" profile should be manipulated via "guix shell" and not "guix package"

I chose "guix install" for my proposal because it already has the functionality of adding a new generation. I do not think it would be good to change the semantics of nested shells.

I failed to mention earlier that I like the name "--transient"
Attachment: file
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 54375@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 54375
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch