~guix shell -C -f guix.scm …~ should not always need ~--rebuild-cache~ option to build the expected environment.

  • Done
  • quality assurance status badge
Details
3 participants
  • Felix Lechner
  • Ludovic Courtès
  • Pierre-Henry Fröhring
Owner
unassigned
Submitted by
Pierre-Henry Fröhring
Severity
normal
P
P
Pierre-Henry Fröhring wrote on 25 Jul 2023 16:01
~guix shell -C -f guix.scm …~ should not always ne ed ~--rebuild-cache~ option to build the expected environmen t.
(address . bug-guix@gnu.org)
CAP84DVVaUGBugE3Xzo-A6noBAO=2kjZnRSrEeC+BFuOfpciQJA@mail.gmail.com
Hello Guix!

As discussed the [[
providing a more detailed description
(see below) of the unexpected behaviour and an archive containing
enough material to reproduce the bug.

* Experiment
** pkgex-1 -> /gnu/store/0yk3xz85…

The Guix package ~pkgex-1~ is built then its path (~/gnu/store/0yk3xz85…~)
is
shown from within a container (~guix shell -C -f guix.scm ripgrep fd
coreutils
emacs~).

#+begin_example
$ make build # equivalent to: guix build -f guix.scm
…
$ guix shell -C -f guix.scm ripgrep fd coreutils emacs
[env]$ ls -al $EMACSLOADPATH/
total 32
dr-xr-xr-x 2 65534 overflow 4096 Jan 1 1970 .
dr-xr-xr-x 3 65534 overflow 4096 Jan 1 1970 ..
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 guix-emacs.el ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/guix-emacs.el
lrwxrwxrwx 1 65534 overflow 91 Jan 1 1970 guix-emacs.elc ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/guix-emacs.elc
lrwxrwxrwx 1 65534 overflow 81 Jan 1 1970 pkgex-1 ->
/gnu/store/0yk3xz85gamig58iska1py6rvn9924ss-pkgex-1/share/emacs/site-lisp/pkgex-1
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 site-start.el ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/site-start.el
lrwxrwxrwx 1 65534 overflow 91 Jan 1 1970 site-start.elc ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/site-start.elc
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 subdirs.el ->
/gnu/store/75z28mg9fd0v3mjcg3jmrah9ihnziqcb-emacs-subdirs/share/emacs/site-lisp/subdirs.el
#+end_example

** /gnu/store/8k18bghzcijbps8kix3wqp34x4smfc5l-pkgex-1

This very file (~pkgex.el.org~) is updated with this content then the
package is
built again.

#+begin_example
$ make build # equivalent to: guix build -f guix.scm
…
/gnu/store/8k18bghzcijbps8kix3wqp34x4smfc5l-pkgex-1
#+end_example

** pkgex-1 -> /gnu/store/0yk3xz85…

Unexpectedly, the package linked from within the container using the same
command as above is not updated, we observe:

- ~pkgex-1 -> /gnu/store/0yk3xz85…~

instead of:

- ~pkgex-1 -> /gnu/store/8k18bghz…~

#+begin_example
$ guix shell -C -f guix.scm ripgrep fd coreutils emacs
[env]$ ls -al $EMACSLOADPATH/
total 32
dr-xr-xr-x 2 65534 overflow 4096 Jan 1 1970 .
dr-xr-xr-x 3 65534 overflow 4096 Jan 1 1970 ..
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 guix-emacs.el ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/guix-emacs.el
lrwxrwxrwx 1 65534 overflow 91 Jan 1 1970 guix-emacs.elc ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/guix-emacs.elc
lrwxrwxrwx 1 65534 overflow 81 Jan 1 1970 pkgex-1 ->
/gnu/store/0yk3xz85gamig58iska1py6rvn9924ss-pkgex-1/share/emacs/site-lisp/pkgex-1
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 site-start.el ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/site-start.el
lrwxrwxrwx 1 65534 overflow 91 Jan 1 1970 site-start.elc ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/site-start.elc
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 subdirs.el ->
/gnu/store/75z28mg9fd0v3mjcg3jmrah9ihnziqcb-emacs-subdirs/share/emacs/site-lisp/subdirs.el
#+end_example

** pkgex-1 -> /gnu/store/8k18bghz…

Nevertheless, if we build a new environment (because we added the
~tree~ package), then, the newly built package
(~/gnu/store/8k18bghz…~) is taken into account.

#+begin_example
$ guix shell -C -f guix.scm ripgrep fd coreutils emacs tree
…
[env]$ ls -al $EMACSLOADPATH/
total 32
dr-xr-xr-x 2 65534 overflow 4096 Jan 1 1970 .
dr-xr-xr-x 3 65534 overflow 4096 Jan 1 1970 ..
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 guix-emacs.el ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/guix-emacs.el
lrwxrwxrwx 1 65534 overflow 91 Jan 1 1970 guix-emacs.elc ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/guix-emacs.elc
lrwxrwxrwx 1 65534 overflow 81 Jan 1 1970 pkgex-1 ->
/gnu/store/8k18bghzcijbps8kix3wqp34x4smfc5l-pkgex-1/share/emacs/site-lisp/pkgex-1
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 site-start.el ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/site-start.el
lrwxrwxrwx 1 65534 overflow 91 Jan 1 1970 site-start.elc ->
/gnu/store/0ibk105zcyg27i9gifbf3lhgm6n78z66-emacs-28.2/share/emacs/site-lisp/site-start.elc
lrwxrwxrwx 1 65534 overflow 90 Jan 1 1970 subdirs.el ->
/gnu/store/n7yizf59v4gvjlr66swh3q3kkz3v1vag-emacs-subdirs/share/emacs/site-lisp/subdirs.el
#+end_example
Attachment: file
Attachment: 1-bug.tar.gz
L
L
Ludovic Courtès wrote on 12 Oct 2023 17:25
Re: bug#64858: ~guix shell -C -f guix.scm …~ should not always need ~--rebuild-cache~ option to build the expected environment.
(name . Pierre-Henry Fröhring)(address . phfrohring@deeplinks.com)(address . 64858@debbugs.gnu.org)
8734yflv5o.fsf@gnu.org
Hi Pierre-Henry,

Pierre-Henry Fröhring <phfrohring@deeplinks.com> skribis:

Toggle quote (2 lines)
> $ guix shell -C -f guix.scm ripgrep fd coreutils emacs

[...]

Toggle quote (16 lines)
> This very file (~pkgex.el.org~) is updated with this content then the
> package is
> built again.
>
> #+begin_example
> $ make build # equivalent to: guix build -f guix.scm
> …
> /gnu/store/8k18bghzcijbps8kix3wqp34x4smfc5l-pkgex-1
> #+end_example
>
>
> ** pkgex-1 -> /gnu/store/0yk3xz85…
>
> Unexpectedly, the package linked from within the container using the same
> command as above is not updated, we observe:

I don’t fully understand the setup, but I can at least explain what you
can expect.

When using ‘-f guix.scm’, ‘guix shell’ caches based on the mtime of
‘guix.scm’: if ‘guix.scm’ is modified, then the cache is invalidated,
otherwise the cache is considered up-to-date and used.

IIUC, you’re modifying a different file, ‘pkgex.el.org’. ‘guix shell’
does not know about it and thus goes ahead and reuses the previous
value.

I guess the current behavior is good when you’re doing:

guix shell -D -f guix.scm

which is the primary use case that comes to mind, but it’s inappropriate
when doing:

guix shell -f guix.scm

in cases where ‘guix.scm’ defines a package with $PWD as its source.

I guess we could maybe try to special-case that in
‘profile-cached-gc-root’.

Ludo’.
F
F
Felix Lechner wrote on 20 Nov 2023 06:12
[PATCH] In 'guix shell', discard ad-hoc profile when 'guix.scm' file is newer. (Closes: #64858)
(address . 64858@debbugs.gnu.org)
9b178dff2dec09ca8af96fae54d5519c6d1cf1a4.1700457155.git.felix.lechner@lease-up.com
Fixes a bug that prevents rebuilds for folks who use 'guix shell' repeatedly
to refine a package declaration located in a 'guix.scm' file.

The mtime of 'guix.scm' is never evaluated because 'file' is #f here and the
code path taken always returns 0 as the purported mtime.


In the consuming code, the 'timestamp' is always less than the mtime of the
profile in the multi-line AND-ed conditional here:


As a result, outdated ad-hoc profiles are always reused unless they are
expressly deleted via:

rm ~/.cache/guix/profiles/*

The bug was potentially introduced in commit c42b7baf when 'package was
transformed to 'ad-hoc-package without also changing the line in this commit.

A minimal reproducer can be found here:

---

Hi Ludo'

Toggle quote (2 lines)
> ‘guix shell’ caches based on the mtime of ‘guix.scm’

That's how is should work, but it doesn't as explained above. Here is
a patch.

Please adjust the commit message to your liking. Thanks!

Kind regards
Felix

guix/scripts/shell.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (17 lines)
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 10ea110fee..d29367c8ee 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -398,7 +398,7 @@ (define (key->file key)
(values #f #f)))
((('nesting? . #t) . rest)
(loop rest system file (append specs '("nested guix"))))
- ((('load . ('package candidate)) . rest)
+ ((('load . ('ad-hoc-package candidate)) . rest)
(if (and (not file) (null? specs))
(loop rest system candidate specs)
(values #f #f)))

base-commit: 71b92466430acb8c91841522dc0eb7d766af4388
--
2.41.0
F
F
Felix Lechner wrote on 20 Nov 2023 06:41
(no subject)
(address . control@debbugs.gnu.org)
87edglq8vs.fsf@lease-up.com
tags 64858 + patch
thanks
L
L
Ludovic Courtès wrote on 22 Nov 2023 23:59
Re: bug#64858: ~guix shell -C -f guix.scm …~ should not always need ~--rebuild-cache~ option to build the expected environment.
(name . Felix Lechner)(address . felix.lechner@lease-up.com)
87v89tgzsz.fsf_-_@gnu.org
Hi Felix,

Felix Lechner <felix.lechner@lease-up.com> skribis:

Toggle quote (12 lines)
> --- a/guix/scripts/shell.scm
> +++ b/guix/scripts/shell.scm
> @@ -398,7 +398,7 @@ (define (key->file key)
> (values #f #f)))
> ((('nesting? . #t) . rest)
> (loop rest system file (append specs '("nested guix"))))
> - ((('load . ('package candidate)) . rest)
> + ((('load . ('ad-hoc-package candidate)) . rest)
> (if (and (not file) (null? specs))
> (loop rest system candidate specs)
> (values #f #f)))

Oooh. So there were really two bugs:

1. The one you describe Felix, where ‘guix shell -f guix.scm’ would
cache things in a nonsensical way (as if you had just run ‘guix
shell’ with no arguments and no ‘guix.scm’ or ‘manifest.scm’ files
in $PWD).

2. The use case issue that I understood from Pierre-Henry’s report,
which is that ‘guix shell -f guix.scm’ shouldn’t have any caching
in the first place.

I fixed it with these two commits (the first one is almost what you
proposed, Felix):

5283d24062 shell: Disable caching for ‘guix shell -f guix.scm’.
762be40098 shell: Correct cache key for ‘guix shell -f guix.scm’.

It seems to do the right thing now. Let me know what you think!

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 22 Nov 2023 23:59
control message for bug #64858
(address . control@debbugs.gnu.org)
87ttpdgzsq.fsf@gnu.org
close 64858
quit
F
F
Felix Lechner wrote on 23 Nov 2023 01:38
Re: bug#64858: ~guix shell -C -f guix.scm …~ should not always need ~--rebuild-cache~ option to build the expected environment.
(name . Ludovic Courtès)(address . ludo@gnu.org)
87sf4xmhgz.fsf@lease-up.com
Hi Ludo'

On Wed, Nov 22 2023, Ludovic Courtès wrote:

Toggle quote (2 lines)
> Oooh. So there were really two bugs:

Sorry I commingled them! I read through many bug report in hope of being
economical.

My hijacking worked out fine, I hope, since Pierre-Henry's report was
resolved at the same time.

I understand Pierre-Henry's use case now.

Toggle quote (2 lines)
> It seems to do the right thing now. Let me know what you think!

Thank you! The fixes look good. I am sure they will help many people.

Personally, I won't be able to test for a while. Updating my system
takes thirty hours because it needs a fixed eudev. The patch is tiny [1]
but not universally accepted as the best solution.

Also, the HEAD in Git doesn't always build. I update every two months.

Thank you for the quick fix!

Kind regards
Felix

F
F
Felix Lechner wrote on 26 Nov 2023 05:17
(name . Ludovic Courtès)(address . ludo@gnu.org)
878r6lm9lk.fsf@lease-up.com
Hi Ludo'

On Wed, Nov 22 2023, Felix Lechner wrote:

Toggle quote (2 lines)
> Personally, I won't be able to test for a while.

I cherry-picked your commits. They work as intended.

My part of this bug was solved. Thank you!

Kind regards
Felix
?