[PATCH 0/2] environment: Add --emulate-fhs option.

  • Done
  • quality assurance status badge
Details
2 participants
  • John Kehayias
  • Ludovic Courtès
Owner
unassigned
Submitted by
John Kehayias
Severity
normal
J
J
John Kehayias wrote on 21 Jul 2022 06:14
(name . Guix-patches)(address . guix-patches@gnu.org)
1LCXD7_zuGflSFovh_mKvhdayNcoWp8ALeguq3i2-XWwWHgzZuPak2gSBN91VLNJ84lubuFAt1dPB282Zy0pBmh_zbiI2qI-7n0LP9F03u4=@protonmail.com
Hello Guix,

As discussed on guix-devel here (please see for more detailed discussion and design aims): https://lists.gnu.org/r/guix-devel/2022-07/msg00161.htmlthis is a patch to add an FHS (Filesystem Hierarchy Standard) emulation option for environments.

The overall goal is to mimic typical GNU/Linux distributions in following FHS (/bin, /etc, and /usr in particular) as well as a glibc that reads a global /etc/ld.so.cache and PATH with /bin, and so on. The idea is that following instructions for setting up a development environment, building software, running something, and so on in "typical" Linux environments, should "just work" with 'guix shell --container --emulate-fhs ...', provided the right inputs and other options are set.

For testers, this can be used by using pre-inst-env (outside of the pure shell used to build a local guix) to run guix shell with this patch. Please see the mailing list discussion for particular examples as well.

For review, in particular:

1. On the mailing list there was discussion about the necessity or not of glibc-for-fhs (added in the first patch). I find this useful and a big piece of making this FHS option work, but open to discussion or if it should be a further option.

2. Right now I used a script written to the containers /tmp/fhs.sh to generate the ld cache, supplement $PATH (somewhat optional, but I found useful for less tinkering), and finally launch the given command or shell. I found that when not providing a command the prompt for /bin/sh is not the same as when not using --emulate-fhs. So I'm not sure if this is the correct way to launch the default /bin/sh if no command is given. Open to ideas of a better way to implement these actions for a container start up as well.

3. This is my first time touching a guix script and the documentation, so please do check the commit message and guix.texi.

4. I decided to link the second level FHS directories, like /usr/bin, as well as optional ones like /lib64 (or /lib32), to the top level /bin, /lib, and so on. These could just be bind mounted to profile/bin and so on as well, but again tried to mimic an FHS distribution like Arch where the files only live in one place. While perhaps making the code a little more involved, I hope this makes the container look tidier.

I may be forgetting other elements in the implementation decisions I made, but I have been testing these patches along the way and have gotten good usage of them. Please test further too!

Thanks,
John
J
J
John Kehayias wrote on 21 Jul 2022 06:19
[PATCH 1/2] gnu: Add glibc-for-fhs.
(name . 56677@debbugs.gnu.org)(address . 56677@debbugs.gnu.org)
X5o4zZg2G3K-B2PwUVdmZKFDQaMjvooCItQJOfL5lJWE9atdn2ufujwLYB2r8rrD4EmL4zZ6mXo37zHMksvVzuyie0huTyYmBT5YU0DOlRY=@protonmail.com
Empty Message
From ac94f4203423c625c06018212677a798ea46b417 Mon Sep 17 00:00:00 2001
From: John Kehayias <john.kehayias@protonmail.com>
Date: Wed, 20 Jul 2022 23:26:53 -0400
Subject: [PATCH 1/2] gnu: Add glibc-for-fhs.

* gnu/packages/base.scm (glibc-for-fhs): New variable. Hidden package for use
in forthcoming --emulate-fhs option for guix containers.
---
gnu/packages/base.scm | 15 +++++++++++++++
1 file changed, 15 insertions(+)

Toggle diff (35 lines)
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index 4bdc3e7792..b566057b41 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -20,6 +20,7 @@
;;; Copyright © 2021 Maxime Devos <maximedevos@telenet.be>
;;; Copyright © 2021 Guillaume Le Vaillant <glv@posteo.net>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2022 John Kehayias <john.kehayias@protonmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -928,6 +929,20 @@ (define-public glibc
(license lgpl2.0+)
(home-page "https://www.gnu.org/software/libc/")))
+;; Define a variation of glibc which uses the default /etc/ld.so.cache, useful
+;; in FHS containers.
+(define-public glibc-for-fhs
+ (hidden-package
+ (package
+ (inherit glibc)
+ (name "glibc-for-fhs")
+ (source (origin (inherit (package-source glibc))
+ ;; Remove Guix's patch to read ld.so.cache from /gnu/store
+ ;; directories, re-enabling the default /etc/ld.so.cache
+ ;; behavior.
+ (patches (delete (car (search-patches "glibc-dl-cache.patch"))
+ (origin-patches (package-source glibc)))))))))
+
;; Below are old libc versions, which we use mostly to build locale data in
;; the old format (which the new libc cannot cope with.)
(define-public glibc-2.32
--
2.37.0
J
J
John Kehayias wrote on 21 Jul 2022 06:19
[PATCH 2/2] environment: Add '--emulate-fhs'.
(name . 56677@debbugs.gnu.org)(address . 56677@debbugs.gnu.org)
iVLKUjDwiUxqX41SXYEzbpi9V9PtW7XJe3DWWgbIxdHH2tnJQRB7asFpjTnbROszyTa1cUkR_WY9fs0sEO7A95Z5maaVG6sY8as0UB3r6vs=@protonmail.com
Empty Message
J
J
John Kehayias wrote on 2 Aug 2022 16:27
Re: [PATCH 2/2 v2] environment: Add '--emulate-fhs'.
(name . 56677@debbugs.gnu.org)(address . 56677@debbugs.gnu.org)
C8_6TSoEzsE5rmSrRu-9MDxx1afrWsrb3a1fDCBFsB5lZP2pwoxw6R9Lmw5-s_zCHUudnUDNWxurMyGk4QwgY3XsFT9RSKXm2R4DNbZNDYg=@protonmail.com
Here is a tiny update to this patch; I noticed after I sent it originally that there was a formatting typo in the guix.texi changes. Update attached.

Has anyone had a chance to try out the patch or take a look at it?

I do use this in some daily work, for a project that uses Python's poetry shell and the Playwright libraries (uses browser binaries it downloads to do web automation). Here is the command I use, where since I need to use browsers in the container just getting the development inputs for ungoogled-chromium is a nice shortcut. The share/expose is to get all graphical/hardware acceleration working.

~/path/to/guix-source/pre-inst-env guix shell -C -F -N poetry coreutils gcc:lib -D ungoogled-chromium --share=$HOME/temphome=$HOME --preserve='^DISPLAY$' --preserve='^XAUTHORITY$' --share=$XAUTHORITY -- "poetry shell"

All works great and is very handy for doing this on my Guix machine.
L
L
Ludovic Courtès wrote on 4 Aug 2022 12:36
Re: bug#56677: [PATCH 0/2] environment: Add --emulate-fhs option.
(name . John Kehayias)(address . john.kehayias@protonmail.com)(address . 56677@debbugs.gnu.org)
87r11wmh2h.fsf@gnu.org
Hello,

John Kehayias <john.kehayias@protonmail.com> skribis:

Toggle quote (2 lines)
> As discussed on guix-devel here (please see for more detailed discussion and design aims): https://lists.gnu.org/r/guix-devel/2022-07/msg00161.htmlthis is a patch to add an FHS (Filesystem Hierarchy Standard) emulation option for environments.

Wo0t!

Toggle quote (2 lines)
> 1. On the mailing list there was discussion about the necessity or not of glibc-for-fhs (added in the first patch). I find this useful and a big piece of making this FHS option work, but open to discussion or if it should be a further option.

I would prefer to keep complexity as low as possible, and thus not have
this glibc variant.

Now, I don’t know for this use case how much it matters that libc honors
/etc/ld.so.cache. Intuitively, like I wrote on guix-devel, I’d think
ld.so.cache doesn’t matter, but you encountered counterexamples.

So I guess that if in practice presence of /etc/ld.so.cache *and* having
glibc honor it is necessary often enough, we can do that.

It seems that ‘glibc-for-fhs’ is merely added to the environment though,
and not actually used?

+ ;; For an FHS-container, add the
+ ;; (hidden) package glibc-for-fhs which
+ ;; uses the global cache at
+ ;; /etc/ld.so.cache.
+ (if emulate-fhs?
+ (alist-cons 'expression
+ '(ad-hoc-package
+ "(@@ (gnu packages base) glibc-for-fhs)")
+ opts)
+ opts)))

Or rather it’s only used when running ‘ldconfig’, right?

Toggle quote (7 lines)
> 2. Right now I used a script written to the containers /tmp/fhs.sh to generate the ld cache, supplement $PATH (somewhat optional, but I found useful for less tinkering), and finally launch the given command or shell. I found that when not providing a command the prompt for /bin/sh is not the same as when not using --emulate-fhs. So I'm not sure if this is the correct way to launch the default /bin/sh if no command is given. Open to ideas of a better way to implement these actions for a container start up as well.
>
> 3. This is my first time touching a guix script and the documentation, so please do check the commit message and guix.texi.
>
> 4. I decided to link the second level FHS directories, like /usr/bin, as well as optional ones like /lib64 (or /lib32), to the top level /bin, /lib, and so on. These could just be bind mounted to profile/bin and so on as well, but again tried to mimic an FHS distribution like Arch where the files only live in one place. While perhaps making the code a little more involved, I hope this makes the container look tidier.
>
> I may be forgetting other elements in the implementation decisions I made, but I have been testing these patches along the way and have gotten good usage of them. Please test further too!
At first sight, I find it pretty cool! I would have two grievances:

1. Can we make the implementation more orthogonal and less entangled
in the already-long ‘launch-environment/container’?

Maybe that can be accomplished by moving all the code conditional
on ‘emulate-fhs?’ out of the way in a separate procedure, and
possibly by adding a generic hook in ‘launch-environment/container’
that would call said procedure.

2. Please add tests. You can probably augment
‘tests/guix-environment-container.sh’ for that. Let us know if
you’re not sure how to do that.

Thanks for all the work, and sorry for the delay: it seems to be summer
time for many of us. :-)

Ludo’.
J
J
John Kehayias wrote on 17 Aug 2022 23:43
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 56677@debbugs.gnu.org)
lRCczsf4p1BjaKh298oj36OxIGgAcLEuAtW6_T9YQxsLmYH03GZC3fEHPFrlskHHhMr8bXT9Sa2D3-i6aCvbCzgP2Hj1h1SBV7wvpwN5400=@protonmail.com
Hello,

Took me a while to respond as well, summer meaning more time to hack on things and then also let them get away...

------- Original Message -------
On Thursday, August 4th, 2022 at 6:36 AM, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (22 lines)
>
> Hello,
>
> John Kehayias john.kehayias@protonmail.com skribis:
>
> > As discussed on guix-devel here (please see for more detailed discussion and design aims): https://lists.gnu.org/r/guix-devel/2022-07/msg00161.html this is a patch to add an FHS (Filesystem Hierarchy Standard) emulation option for environments.
>
> Wo0t!
>
> > 1. On the mailing list there was discussion about the necessity or not of glibc-for-fhs (added in the first patch). I find this useful and a big piece of making this FHS option work, but open to discussion or if it should be a further option.
>
> I would prefer to keep complexity as low as possible, and thus not have
> this glibc variant.
>
> Now, I don’t know for this use case how much it matters that libc honors
> /etc/ld.so.cache. Intuitively, like I wrote on guix-devel, I’d think
> ld.so.cache doesn’t matter, but you encountered counterexamples.
>
> So I guess that if in practice presence of /etc/ld.so.cache and having
> glibc honor it is necessary often enough, we can do that.
>

Right, though as I said, happy to hear of alternatives or what other use cases come up. This seems rather robust to any "usual" assumptions though.

Toggle quote (4 lines)
> It seems that ‘glibc-for-fhs’ is merely added to the environment though,
> and not actually used?
>

Well, it is added to the environment which here means the glibc-for-fhs lib directory ends up in the container's global /lib. This may be useful for anything expecting a more "typical" glibc to be found in the typical location. I can't say I know the particulars here, other than binaries and an example of other nested containers (used in non-free software, but the containers are bwrap and friends) expecting glibc to default to a global ld cache. Again, there may be other workarounds or ways to reduce this, but for now I followed the "emulate" part of the flag :)

Toggle quote (14 lines)
> + ;; For an FHS-container, add the
> + ;; (hidden) package glibc-for-fhs which
> + ;; uses the global cache at
> + ;; /etc/ld.so.cache.
> + (if emulate-fhs?
> + (alist-cons 'expression
> + '(ad-hoc-package
> + "(@@ (gnu packages base) glibc-for-fhs)")
> + opts)
> + opts)))
>
> Or rather it’s only used when running ‘ldconfig’, right?
>

Yes, since that is the glibc in the container. Though actually generating a cache shouldn't matter, right? Guix's glibc will do that as well, as the only patch removed is the one that changed where glibc reads the ld cache from. So the cache could be generated with Guix's glibc, but then likely won't be read otherwise? Sorry, a bit out of what I know for all the details; I'm mostly at "this is what is often expected" and "this makes it work."

Toggle quote (11 lines)
> > 2. Right now I used a script written to the containers /tmp/fhs.sh to generate the ld cache, supplement $PATH (somewhat optional, but I found useful for less tinkering), and finally launch the given command or shell. I found that when not providing a command the prompt for /bin/sh is not the same as when not using --emulate-fhs. So I'm not sure if this is the correct way to launch the default /bin/sh if no command is given. Open to ideas of a better way to implement these actions for a container start up as well.
> >
> > 3. This is my first time touching a guix script and the documentation, so please do check the commit message and guix.texi.
> >
> > 4. I decided to link the second level FHS directories, like /usr/bin, as well as optional ones like /lib64 (or /lib32), to the top level /bin, /lib, and so on. These could just be bind mounted to profile/bin and so on as well, but again tried to mimic an FHS distribution like Arch where the files only live in one place. While perhaps making the code a little more involved, I hope this makes the container look tidier.
> >
> > I may be forgetting other elements in the implementation decisions I made, but I have been testing these patches along the way and have gotten good usage of them. Please test further too!
>
> At first sight, I find it pretty cool! I would have two grievances:
>

Thanks! It is serving me in some day-to-day work nicely.

Toggle quote (9 lines)
> 1. Can we make the implementation more orthogonal and less entangled
> in the already-long ‘launch-environment/container’?
>
> Maybe that can be accomplished by moving all the code conditional
> on ‘emulate-fhs?’ out of the way in a separate procedure, and
> possibly by adding a generic hook in ‘launch-environment/container’
> that would call said procedure.
>

Sure, this sounds like a good idea. I can certainly separate out the FHS setup to a separate function and call it. But I'm not sure what you mean by a "generic hook" here. Do you mean that launch-environment/container would have as an argument say a list of functions it would call?

Toggle quote (5 lines)
> 2. Please add tests. You can probably augment
> ‘tests/guix-environment-container.sh’ for that. Let us know if
> you’re not sure how to do that.
>

Thanks, definitely forgot about that. In looking at that, I've just ran it with "./pre-inst-env sh tests/guix-environment-container.sh" and see that the exit code is 0. Is that the correct way to run these?

Secondly, I'm trying to think of what tests to add. I could of course run the same tests already, but with the --emulate-fhs option, to check that there are no regressions. Other than that, maybe checking that e.g. there's /etc/ld.so.cache, /lib, and so on?

Toggle quote (5 lines)
> Thanks for all the work, and sorry for the delay: it seems to be summer
> time for many of us. :-)
>
> Ludo’.

No worries, summer is a good time to get away, or dig in :)

John
L
L
Ludovic Courtès wrote on 8 Sep 2022 22:58
(name . John Kehayias)(address . john.kehayias@protonmail.com)(address . 56677@debbugs.gnu.org)
87mtb9d1mf.fsf_-_@gnu.org
Howdy,

John Kehayias <john.kehayias@protonmail.com> skribis:

Toggle quote (19 lines)
>> I would prefer to keep complexity as low as possible, and thus not have
>> this glibc variant.
>>
>> Now, I don’t know for this use case how much it matters that libc honors
>> /etc/ld.so.cache. Intuitively, like I wrote on guix-devel, I’d think
>> ld.so.cache doesn’t matter, but you encountered counterexamples.
>>
>> So I guess that if in practice presence of /etc/ld.so.cache and having
>> glibc honor it is necessary often enough, we can do that.
>>
>
> Right, though as I said, happy to hear of alternatives or what other use cases come up. This seems rather robust to any "usual" assumptions though.
>
>> It seems that ‘glibc-for-fhs’ is merely added to the environment though,
>> and not actually used?
>>
>
> Well, it is added to the environment which here means the glibc-for-fhs lib directory ends up in the container's global /lib. This may be useful for anything expecting a more "typical" glibc to be found in the typical location. I can't say I know the particulars here, other than binaries and an example of other nested containers (used in non-free software, but the containers are bwrap and friends) expecting glibc to default to a global ld cache. Again, there may be other workarounds or ways to reduce this, but for now I followed the "emulate" part of the flag :)

Oh I got it; that /lib/libc.so *is* used, but only by binaries that were
built on an FHS distro and that you’d bring in (that’s the whole point,
I guess). It’s not used by Guix packages.

Toggle quote (11 lines)
>> 1. Can we make the implementation more orthogonal and less entangled
>> in the already-long ‘launch-environment/container’?
>>
>> Maybe that can be accomplished by moving all the code conditional
>> on ‘emulate-fhs?’ out of the way in a separate procedure, and
>> possibly by adding a generic hook in ‘launch-environment/container’
>> that would call said procedure.
>>
>
> Sure, this sounds like a good idea. I can certainly separate out the FHS setup to a separate function and call it. But I'm not sure what you mean by a "generic hook" here. Do you mean that launch-environment/container would have as an argument say a list of functions it would call?

Yes, or an argument with a single procedure to call at a specific
point. That would default to a no-op.

Toggle quote (7 lines)
>> 2. Please add tests. You can probably augment
>> ‘tests/guix-environment-container.sh’ for that. Let us know if
>> you’re not sure how to do that.
>>
>
> Thanks, definitely forgot about that. In looking at that, I've just ran it with "./pre-inst-env sh tests/guix-environment-container.sh" and see that the exit code is 0. Is that the correct way to run these?

The correct way is:

make check TESTS=tests/guix-environment-container.sh

Compared to what you wrote, it uses ./test-env (which spawns a daemon
that uses the local store, not /gnu/sore) and sets a bunch of
environment variables.

See

Toggle quote (2 lines)
> Secondly, I'm trying to think of what tests to add. I could of course run the same tests already, but with the --emulate-fhs option, to check that there are no regressions. Other than that, maybe checking that e.g. there's /etc/ld.so.cache, /lib, and so on?

Right, at least you’d want to check for these files/directories.

Note that since the test relies on ‘glibc-for-fhs’, it cannot be done
the “normal way” (that is, using the local store rather than /gnu/store)
because it would end up building the world.

The solution here is to use /gnu/store, if available, and to otherwise
skip the test (return 77). See ‘tests/guix-pack-relocatable.sh’ up to
line 40 on how to do that.

HTH!

Ludo’.
L
L
Ludovic Courtès wrote on 8 Sep 2022 22:58
(name . John Kehayias)(address . john.kehayias@protonmail.com)(address . 56677@debbugs.gnu.org)
87leqtd1m4.fsf_-_@gnu.org
Howdy,

John Kehayias <john.kehayias@protonmail.com> skribis:

Toggle quote (19 lines)
>> I would prefer to keep complexity as low as possible, and thus not have
>> this glibc variant.
>>
>> Now, I don’t know for this use case how much it matters that libc honors
>> /etc/ld.so.cache. Intuitively, like I wrote on guix-devel, I’d think
>> ld.so.cache doesn’t matter, but you encountered counterexamples.
>>
>> So I guess that if in practice presence of /etc/ld.so.cache and having
>> glibc honor it is necessary often enough, we can do that.
>>
>
> Right, though as I said, happy to hear of alternatives or what other use cases come up. This seems rather robust to any "usual" assumptions though.
>
>> It seems that ‘glibc-for-fhs’ is merely added to the environment though,
>> and not actually used?
>>
>
> Well, it is added to the environment which here means the glibc-for-fhs lib directory ends up in the container's global /lib. This may be useful for anything expecting a more "typical" glibc to be found in the typical location. I can't say I know the particulars here, other than binaries and an example of other nested containers (used in non-free software, but the containers are bwrap and friends) expecting glibc to default to a global ld cache. Again, there may be other workarounds or ways to reduce this, but for now I followed the "emulate" part of the flag :)

Oh I got it; that /lib/libc.so *is* used, but only by binaries that were
built on an FHS distro and that you’d bring in (that’s the whole point,
I guess). It’s not used by Guix packages.

Toggle quote (11 lines)
>> 1. Can we make the implementation more orthogonal and less entangled
>> in the already-long ‘launch-environment/container’?
>>
>> Maybe that can be accomplished by moving all the code conditional
>> on ‘emulate-fhs?’ out of the way in a separate procedure, and
>> possibly by adding a generic hook in ‘launch-environment/container’
>> that would call said procedure.
>>
>
> Sure, this sounds like a good idea. I can certainly separate out the FHS setup to a separate function and call it. But I'm not sure what you mean by a "generic hook" here. Do you mean that launch-environment/container would have as an argument say a list of functions it would call?

Yes, or an argument with a single procedure to call at a specific
point. That would default to a no-op.

Toggle quote (7 lines)
>> 2. Please add tests. You can probably augment
>> ‘tests/guix-environment-container.sh’ for that. Let us know if
>> you’re not sure how to do that.
>>
>
> Thanks, definitely forgot about that. In looking at that, I've just ran it with "./pre-inst-env sh tests/guix-environment-container.sh" and see that the exit code is 0. Is that the correct way to run these?

The correct way is:

make check TESTS=tests/guix-environment-container.sh

Compared to what you wrote, it uses ./test-env (which spawns a daemon
that uses the local store, not /gnu/sore) and sets a bunch of
environment variables.

See

Toggle quote (2 lines)
> Secondly, I'm trying to think of what tests to add. I could of course run the same tests already, but with the --emulate-fhs option, to check that there are no regressions. Other than that, maybe checking that e.g. there's /etc/ld.so.cache, /lib, and so on?

Right, at least you’d want to check for these files/directories.

Note that since the test relies on ‘glibc-for-fhs’, it cannot be done
the “normal way” (that is, using the local store rather than /gnu/store)
because it would end up building the world.

The solution here is to use /gnu/store, if available, and to otherwise
skip the test (return 77). See ‘tests/guix-pack-relocatable.sh’ up to
line 40 on how to do that.

HTH!

Ludo’.
J
J
John Kehayias wrote on 4 Oct 2022 20:32
Re: bug#56677: [PATCH v2 1/2] environment: Add --emulate-fhs option.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 56677@debbugs.gnu.org)
87edvntp6f.fsf@protonmail.com
Hi Ludo’!

Apologies for the long delay! I've done some more rewriting than expected, so let me outline some changes in the v2 patch (attached, while path 1/2 for glibc-for-fhs remains unchanged).

On Thu, Sep 08, 2022 at 10:58 PM, Ludovic Courtès wrote:

Toggle quote (4 lines)
> Howdy,
>
> John Kehayias <john.kehayias@protonmail.com> skribis:
>
[snip]
Toggle quote (18 lines)
>>> 1. Can we make the implementation more orthogonal and less entangled
>>> in the already-long ‘launch-environment/container’?
>>>
>>> Maybe that can be accomplished by moving all the code conditional
>>> on ‘emulate-fhs?’ out of the way in a separate procedure, and
>>> possibly by adding a generic hook in ‘launch-environment/container’
>>> that would call said procedure.
>>>
>>
>> Sure, this sounds like a good idea. I can certainly separate out the FHS setup to a
>> separate function and call it. But I'm not sure what you mean by a "generic hook" here.
>> Do you mean that launch-environment/container would have as an argument say a list of
>> functions it would call?
>
> Yes, or an argument with a single procedure to call at a specific
> point. That would default to a no-op.
>

I've done some rearranging to address this and along the way fixed the command passing to the container to work the same as without --emulate-fhs (previously I hacked together a startup script that didn't properly capture the command given by the user).

Unfortunately, since some of the needed directories are bind mounted and others can be linked after creating filesystems, not all of the FHS directory setup can be done in the same place. To be explicit, bind mounts for all filesystems are set up in the early mlet of launch-environment/container, before being able to call a procedure to do other setup needed here.

The end result:

launch-environment/container now takes a key 'setup-hook' which defaults to #f. For now this only handles a single list with the function name and arguments (could extend this to be more general, but didn't have any use cases for it off hand). For the FHS container this is set, by the entry point guix-environment*, to the new function setup-fhs which sets up symlinks and ld.so.conf. Handling arguments was necessary as setup-fhs needs the profile used in the container.

So then launch-environment also gets a new 'emulate-fhs?' option so it can run some additional setup before running the command given by the user. What was the (hacky) /tmp/fhs.sh is now a setenv and invoke, before the final execlp for the given command.

A little more complicated perhaps, but I think functionally separates the different steps (bind mounts, symlinks and file creation, and running in the container) at least.

Toggle quote (21 lines)
>>> 2. Please add tests. You can probably augment
>>> ‘tests/guix-environment-container.sh’ for that. Let us know if
>>> you’re not sure how to do that.
>>>
>>
>> Thanks, definitely forgot about that. In looking at that, I've just ran it with
>> "./pre-inst-env sh tests/guix-environment-container.sh" and see that the exit code is 0.
>> Is that the correct way to run these?
>
> The correct way is:
>
> make check TESTS=tests/guix-environment-container.sh
>
> Compared to what you wrote, it uses ./test-env (which spawns a daemon
> that uses the local store, not /gnu/sore) and sets a bunch of
> environment variables.
>
> See
> <https://guix.gnu.org/manual/devel/en/html_node/Running-the-Test-Suite.html>.
>

Thanks, missed that somehow.

Toggle quote (17 lines)
>> Secondly, I'm trying to think of what tests to add. I could of course run the same tests
>> already, but with the --emulate-fhs option, to check that there are no regressions.
>> Other than that, maybe checking that e.g. there's /etc/ld.so.cache, /lib, and so on?
>
> Right, at least you’d want to check for these files/directories.
>
> Note that since the test relies on ‘glibc-for-fhs’, it cannot be done
> the “normal way” (that is, using the local store rather than /gnu/store)
> because it would end up building the world.
>
> The solution here is to use /gnu/store, if available, and to otherwise
> skip the test (return 77). See ‘tests/guix-pack-relocatable.sh’ up to
> line 40 on how to do that.
>
> HTH!
>

Very helpful, thanks!

I added in two tests to the end of tests/guix-environment-container.sh. One checks for the FHS file structure in the container and the other tries to read the ld cache (using 'ldconfig -p').

If we wanted to run all the non-fhs tests with --emulate-fhs, then maybe we'd want to make it so the FHS specific tests live in a new file and guix-environment-container.sh can be called in a way to enable that option? (A quick guess would be to just set an alias so that guix environment is always called with --emulate-fhs, but not sure if that works in the test environment.)

Wasn't sure if all that is necessary so I just went with the FHS-specific tests for now. I checked that they pass for me.

The commit changelog has gotten a bit more complicated with these changes, hopefully I got everything in there.

Thanks for your help on this and I'll make sure any future revisions are more timely!
John
L
L
Ludovic Courtès wrote on 13 Oct 2022 09:37
Re: bug#56677: [PATCH 0/2] environment: Add --emulate-fhs option.
(name . John Kehayias)(address . john.kehayias@protonmail.com)(address . 56677@debbugs.gnu.org)
877d14tbqy.fsf_-_@gnu.org
Hi John,

John Kehayias <john.kehayias@protonmail.com> skribis:

Toggle quote (21 lines)
> From de7ae9b203a65e1ec1b1429ca4076f3f91b1ae33 Mon Sep 17 00:00:00 2001
> From: John Kehayias <john.kehayias@protonmail.com>
> Date: Wed, 20 Jul 2022 23:46:45 -0400
> Subject: [PATCH] environment: Add '--emulate-fhs'.
>
> * guix/scripts/environment.scm (show-environment-options-help, %options): Add
> '--emulate-fhs'.
> (setup-fhs): New procedure. Setup for the Filesystem Hierarchy Standard (FHS)
> container. Defines and uses FHS-SYMLINKS and LINK-CONTENTS to create FHS
> expected directories and creates /etc/ld.so.conf.
> (launch-environment): Add 'emulate-fhs?' key and implement it to set $PATH and
> generate /etc/ld.so.cache before calling COMMAND.
> (launch-environment/container): Add 'emulate-fhs?' and 'setup-hook' keys and
> implement them. Define and use FHS-MAPPINGS, to set up additional bind mounts
> in the container to follow FHS expectations.
> (guix-environment*): Add glibc-for-fhs to the container packages when
> 'emulate-fhs?' key is in OPTS.
> * doc/guix.texi (Invoking guix shell): Document '--emulate-fhs'.
> (Invoking guix environment): Document '--emulate-fhs'.
> * tests/guix-environment-container.sh: Add tests for '--emulate-fhs'.

This new version of the patch looks great to me!

I applied it and took the liberty to make the changes below; let me know
if anything’s amiss.

Pushed together with a news entry:

2c11e857af news: Add entry for 'guix shell --emulate-fhs'.
c7ba5f38b8 environment: Add '--emulate-fhs'.
3d1d29e440 gnu: Add glibc-for-fhs.

Thanks a lot for your work and for your patience!

Ludo’.
Attachment: file
L
L
Ludovic Courtès wrote on 13 Oct 2022 09:38
control message for bug #56677
(address . control@debbugs.gnu.org)
875ygotbqf.fsf@gnu.org
close 56677
quit
?