Time travel doesn't resist profile format changes

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Ricardo Wurmus
  • zimoun
Owner
unassigned
Submitted by
zimoun
Severity
important
Z
Z
zimoun wrote on 7 Jul 2022 18:42
guix time-machine broken by profiles speed-up
(address . bug-guix@gnu.org)
867d4oq3gr.fsf@gmail.com
Hi,

Bug#55499 [1] is fixed by 4ff12d1de7cd617b791996ee7ca1240660b4c20e.
However, because the manifest version is going from 3 to 4 with new
fields, the new Guix cannot builds the old Guix.

Commit 4ff12d1de7cd617b791996ee7ca1240660b4c20e is not able to go to its
parent 9b8c442b254b82196fe2492142b3c3bbbd891a1b.

Toggle snippet (18 lines)
$ git rev-parse 4ff12d1de7^
9b8c442b254b82196fe2492142b3c3bbbd891a1b

$ guix time-machine --commit=4ff12d1de7 -- time-machine --commit=9b8c442b25 -- help
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Computing Guix derivation for 'x86_64-linux'... \
The following derivation will be built:
/gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv

building package cache...
|builder for `/gnu/store/19nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv' failed to produce output path `/gnu/store/axqgrls563slnp76x60dqlv7sdwcm2ly-guix-package-cache'
build of /gnu/store/19nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv failed
View build log at '/var/log/guix/drvs/19/nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv.gz'.
cannot build derivation `/gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv': 1 dependencies couldn't be built
guix time-machine: error: build of `/gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv' failed





Cheers,
simon
R
R
Ricardo Wurmus wrote on 7 Jul 2022 19:29
(name . zimoun)(address . zimon.toutoune@gmail.com)
87wncozun2.fsf@elephly.net
zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (23 lines)
> Bug#55499 [1] is fixed by 4ff12d1de7cd617b791996ee7ca1240660b4c20e.
> However, because the manifest version is going from 3 to 4 with new
> fields, the new Guix cannot builds the old Guix.
>
> Commit 4ff12d1de7cd617b791996ee7ca1240660b4c20e is not able to go to its
> parent 9b8c442b254b82196fe2492142b3c3bbbd891a1b.
>
> $ git rev-parse 4ff12d1de7^
> 9b8c442b254b82196fe2492142b3c3bbbd891a1b
>
> $ guix time-machine --commit=4ff12d1de7 -- time-machine --commit=9b8c442b25 -- help
> Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
> Computing Guix derivation for 'x86_64-linux'... \
> The following derivation will be built:
> /gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv
>
> building package cache...
> |builder for `/gnu/store/19nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv' failed to produce output path `/gnu/store/axqgrls563slnp76x60dqlv7sdwcm2ly-guix-package-cache'
> build of /gnu/store/19nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv failed
> View build log at '/var/log/guix/drvs/19/nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv.gz'.
> cannot build derivation `/gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv': 1 dependencies couldn't be built
> guix time-machine: error: build of `/gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv' failed

IIUC the problem here is that PACKAGE-CACHE-FILE, a profile hook running
inside an inferior (= with an older Guix), is operating on a manifest
that uses the new version 4 format. That manifest was built with the
*current* version of Guix that understands the version 4 format.

PACKAGE-CACHE-FILE uses GEXP->DERIVATION-IN-INFERIOR and the inferior is
a PROFILE that’s made from a given MANIFEST value. That PROFILE is
*not* built inside the inferior, so it doesn’t use the old manifest
format.

The cache generation is happening in
/gnu/store/17v5781w8kl1snp826jl6z40z5lbbw1y-inferior-script.scm.drv, which — as the name indicates — is
run inside the inferior, i.e. the older Guix.

--
Ricardo
R
R
Ricardo Wurmus wrote on 7 Jul 2022 19:52
(name . zimoun)(address . zimon.toutoune@gmail.com)
87sfncztuq.fsf@elephly.net
Ricardo Wurmus <rekado@elephly.net> writes:

Toggle quote (42 lines)
> zimoun <zimon.toutoune@gmail.com> writes:
>
>> Bug#55499 [1] is fixed by 4ff12d1de7cd617b791996ee7ca1240660b4c20e.
>> However, because the manifest version is going from 3 to 4 with new
>> fields, the new Guix cannot builds the old Guix.
>>
>> Commit 4ff12d1de7cd617b791996ee7ca1240660b4c20e is not able to go to its
>> parent 9b8c442b254b82196fe2492142b3c3bbbd891a1b.
>>
>> $ git rev-parse 4ff12d1de7^
>> 9b8c442b254b82196fe2492142b3c3bbbd891a1b
>>
>> $ guix time-machine --commit=4ff12d1de7 -- time-machine --commit=9b8c442b25 -- help
>> Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>> Computing Guix derivation for 'x86_64-linux'... \
>> The following derivation will be built:
>> /gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv
>>
>> building package cache...
>> |builder for
>> `/gnu/store/19nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv'
>> failed to produce output path
>> `/gnu/store/axqgrls563slnp76x60dqlv7sdwcm2ly-guix-package-cache'
>> build of /gnu/store/19nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv failed
>> View build log at '/var/log/guix/drvs/19/nk2x26s0dp68r7d36ifbg0ck0q3xps-guix-package-cache.drv.gz'.
>> cannot build derivation `/gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv': 1 dependencies couldn't be built
>> guix time-machine: error: build of `/gnu/store/r5qk23fibxn5ryd2k7b8qkbryqv4m3ds-profile.drv' failed
>
> IIUC the problem here is that PACKAGE-CACHE-FILE, a profile hook running
> inside an inferior (= with an older Guix), is operating on a manifest
> that uses the new version 4 format. That manifest was built with the
> *current* version of Guix that understands the version 4 format.
>
> PACKAGE-CACHE-FILE uses GEXP->DERIVATION-IN-INFERIOR and the inferior is
> a PROFILE that’s made from a given MANIFEST value. That PROFILE is
> *not* built inside the inferior, so it doesn’t use the old manifest
> format.
>
> The cache generation is happening in
> /gnu/store/17v5781w8kl1snp826jl6z40z5lbbw1y-inferior-script.scm.drv, which — as the name indicates — is
> run inside the inferior, i.e. the older Guix.

It’s worse than that: an older Guix living in a new profile (i.e a
profile with a version 4 manifest) cannot describe itself, because it
cannot read the manifest.

I took the same profile that was generated by the time machine and
contains the old Guix. Here’s the manifest:

Toggle snippet (13 lines)
$ head /gnu/store/mwbgfyl0zzipyac1lbgss2gcji67fp4s-profile/manifest
;; This file was automatically generated and is for internal use only.
;; It cannot be passed to the '--manifest' option.
;; Run 'guix package --export-manifest' if you want to export a file
;; suitable for '--manifest'.

(manifest
(version 4)
(packages
(("guix"
"9b8c442"

And then:

Toggle snippet (4 lines)
$ /gnu/store/mwbgfyl0zzipyac1lbgss2gcji67fp4s-profile/bin/guix describe
guix describe: error: unsupported manifest format

This old Guix lives in a new profile. We must prevent this. But how?

How about this:

Build Guix with only the 'guix channel (no any other custom channels
allowed), and then build the channel’s profile (with all requested
channels) inside an inferior of that plain Guix. Since the inferior
Guix can only build profiles that it supports the resulting *profile*
will be compatible with the same Guix (and any additional channels).

--
Ricardo
L
L
Ludovic Courtès wrote on 8 Jul 2022 10:52
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87fsjcx9yc.fsf@gnu.org
Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

Toggle quote (4 lines)
> It’s worse than that: an older Guix living in a new profile (i.e a
> profile with a version 4 manifest) cannot describe itself, because it
> cannot read the manifest.

Argh.

[...]

Toggle quote (10 lines)
> This old Guix lives in a new profile. We must prevent this. But how?
>
> How about this:
>
> Build Guix with only the 'guix channel (no any other custom channels
> allowed), and then build the channel’s profile (with all requested
> channels) inside an inferior of that plain Guix. Since the inferior
> Guix can only build profiles that it supports the resulting *profile*
> will be compatible with the same Guix (and any additional channels).

Another option (thinking out loud):

• in ‘package-cache-file’, unconditionally generate a v3 profile (we
could add a ‘version’ field to <profile> etc.);

• likewise in ‘channel-instances->derivation’.

Problem: it doesn’t address the case where you install the ‘guix’
package in a regular profile (I think?).

Other option: run the whole ‘profile-derivation’ call of
‘channel-instances->derivation’ in the inferior, instead of just
‘package-cache-file’.

Needs more thought…

Ludo’.
L
L
Ludovic Courtès wrote on 8 Jul 2022 10:53
control message for bug #56441
(address . control@debbugs.gnu.org)
87edywx9xg.fsf@gnu.org
severity 56441 important
quit
L
L
Ludovic Courtès wrote on 8 Jul 2022 10:53
(address . control@debbugs.gnu.org)
87czegx9wo.fsf@gnu.org
retitle 56441 Time travel doesn't resist profile format changes
quit
L
L
Ludovic Courtès wrote on 8 Jul 2022 12:36
Re: bug#56441: guix time-machine broken by profiles speed-up
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87zghjx557.fsf@gnu.org
Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (7 lines)
> Another option (thinking out loud):
>
> • in ‘package-cache-file’, unconditionally generate a v3 profile (we
> could add a ‘version’ field to <profile> etc.);
>
> • likewise in ‘channel-instances->derivation’.

The patches below do that. As discussed on IRC, it’s not pretty but
it’s pragmatic.

Tested with:

./pre-inst-env guix time-machine \
--commit=85a5110de79f4fe9fd822ede3915654ee699d6c5 -- describe

How does that sound?

Toggle quote (3 lines)
> Problem: it doesn’t address the case where you install the ‘guix’
> package in a regular profile (I think?).

I haven’t yet checked whether this is the case.

Ludo’.
Attachment: 0001-profiles-Support-the-creation-of-profiles-with-versi.patch
From d404ef913482c6a9c692bad2142cf7202ebc1cc4 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Fri, 8 Jul 2022 12:31:25 +0200
Subject: [PATCH 2/2] channels: Emit version 3 profiles.

Reported by zimoun <zimon.toutoune@gmail.com>.

* guix/channels.scm (package-cache-file): Add 'format-version' field to
PROFILE.
(channel-instances->derivation): Pass #:format-version to
'profile-derivation'.
---
guix/channels.scm | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)

Toggle diff (41 lines)
diff --git a/guix/channels.scm b/guix/channels.scm
index ce1a60436f..689b30e0eb 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2018-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
;;; Copyright © 2021 Brice Waegeneire <brice@waegenei.re>
@@ -896,7 +896,12 @@ (define (instance->entry instance drv)
(define (package-cache-file manifest)
"Build a package cache file for the instance in MANIFEST. This is meant to
be used as a profile hook."
- (let ((profile (profile (content manifest) (hooks '()))))
+ ;; Note: Emit a profile in format version 3, which was introduced in 2017
+ ;; and is readable by Guix since before version 1.0. This ensures that the
+ ;; Guix in MANIFEST is able to read the manifest file created for its own
+ ;; profile below. See <https://issues.guix.gnu.org/56441>.
+ (let ((profile (profile (content manifest) (hooks '())
+ (format-version 3))))
(define build
#~(begin
(use-modules (gnu packages))
@@ -937,8 +942,12 @@ (define (channel-instances->derivation instances)
"Return the derivation of the profile containing INSTANCES, a list of
channel instances."
(mlet %store-monad ((manifest (channel-instances->manifest instances)))
+ ;; Emit a profile in format version so that, if INSTANCES denotes an old
+ ;; Guix, it can still read that profile, for instance for the purposes of
+ ;; 'guix describe'.
(profile-derivation manifest
- #:hooks %channel-profile-hooks)))
+ #:hooks %channel-profile-hooks
+ #:format-version 3)))
(define latest-channel-instances*
(store-lift latest-channel-instances))
--
2.36.1
R
R
Ricardo Wurmus wrote on 8 Jul 2022 12:46
Time travel doesn't resist profile format changes
(address . 56441@debbugs.gnu.org)
87fsjbzxo5.fsf@elephly.net
Attached is a patch that attempts to build the manifest in an inferior.
This fails because manifest->gexp returns a gexp that we can’t get out
of the inferior to pass to build-profile.

So even more of the surrounding code would have to be evaluated in the
inferior.

@Ludo: your patch looks good to me. It’s a pragmatic, minimally
invasive fix, so thumbs up emoji from me! Thank you!

--
Ricardo
Attachment: 0001-channel-stuff.patch
Z
Z
zimoun wrote on 8 Jul 2022 16:34
(name . Ludovic Courtès)(address . ludo@gnu.org)
875yk7brmp.fsf_-_@gmail.com
Hi,

On ven., 08 juil. 2022 at 12:36, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (21 lines)
> Hi,
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Another option (thinking out loud):
>>
>> • in ‘package-cache-file’, unconditionally generate a v3 profile (we
>> could add a ‘version’ field to <profile> etc.);
>>
>> • likewise in ‘channel-instances->derivation’.
>
> The patches below do that. As discussed on IRC, it’s not pretty but
> it’s pragmatic.
>
> Tested with:
>
> ./pre-inst-env guix time-machine \
> --commit=85a5110de79f4fe9fd822ede3915654ee699d6c5 -- describe
>
> How does that sound?

It sounds good! Here a check from v1.3 to v0.16.

Toggle snippet (45 lines)
$ for ci in a0178d34f582b50e9bdbb0403943129ae5b560ff \
a099685659b4bfa6b3218f84953cbb7ff9e88063 \
d62c9b2671be55ae0305bebfda17b595f33797f2 \
d68de958b60426798ed62797ff7c96c327a672ac \
6298c3ffd9654d3231a6f25390b056483e8f407c \
4a0b87f0ec5b6c2dcf82b372dd20ca7ea6acdd9c;
do
./pre-inst-env guix time-machine --commit=$ci -- describe
done

guix a0178d3
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: a0178d34f582b50e9bdbb0403943129ae5b560ff
guix a099685
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: a099685659b4bfa6b3218f84953cbb7ff9e88063
guile: warning: failed to install locale
guix d62c9b2
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: d62c9b2671be55ae0305bebfda17b595f33797f2
guile: warning: failed to install locale
guix d68de95
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: d68de958b60426798ed62797ff7c96c327a672ac
guile: warning: failed to install locale
guix 6298c3f
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: 6298c3ffd9654d3231a6f25390b056483e8f407c
guile: warning: failed to install locale
hint: Consider installing the `glibc-utf8-locales' or `glibc-locales' package and defining `GUIX_LOCPATH', along these
lines:

guix package -i glibc-utf8-locales
export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale"

See the "Application Setup" section in the manual, for more info.


guix 4a0b87f
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: 4a0b87f0ec5b6c2dcf82b372dd20ca7ea6acdd9c



Cheers,
simon
L
L
Ludovic Courtès wrote on 8 Jul 2022 18:13
(name . zimoun)(address . zimon.toutoune@gmail.com)
877d4ntwey.fsf@gnu.org
Hello,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (12 lines)
> It sounds good! Here a check from v1.3 to v0.16.
>
> $ for ci in a0178d34f582b50e9bdbb0403943129ae5b560ff \
> a099685659b4bfa6b3218f84953cbb7ff9e88063 \
> d62c9b2671be55ae0305bebfda17b595f33797f2 \
> d68de958b60426798ed62797ff7c96c327a672ac \
> 6298c3ffd9654d3231a6f25390b056483e8f407c \
> 4a0b87f0ec5b6c2dcf82b372dd20ca7ea6acdd9c;
> do
> ./pre-inst-env guix time-machine --commit=$ci -- describe
> done

That’s a great test, thanks for checking!

I would really like us to run that kind of test automatically. It’s
expensive, requires network access, Git repo access, etc., so I’m not
sure it’s suitable for “make check”. But we need to think about it.

I’ll commit either tonight or on Monday.

Ludo’.
L
L
Ludovic Courtès wrote on 8 Jul 2022 22:42
Re: bug#56441: guix time-machine broken by profiles speed-up
(name . Ricardo Wurmus)(address . rekado@elephly.net)(address . 56441@debbugs.gnu.org)
871quvtjyy.fsf_-_@gnu.org
Ricardo Wurmus <rekado@elephly.net> skribis:

Toggle quote (4 lines)
> Attached is a patch that attempts to build the manifest in an inferior.
> This fails because manifest->gexp returns a gexp that we can’t get out
> of the inferior to pass to build-profile.

Thanks for sharing! This sounded like the “right” approach, but there’s
a part that made me feel uneasy about it:

Toggle quote (23 lines)
> + (define (manifest-gexp)
> + (pk 'man-gexp
> + (if inferior-guix
> + (let* ((inferior (open-inferior inferior-guix))
> + (result
> + ;; TODO: this doesn't work because we can't lift the
> + ;; gexp out of the inferior.
> + (inferior-eval `(begin
> + (use-modules (guix profiles)
> + (guix derivations))
> + ((@@ (guix profiles) manifest->gexp)
> + (manifest
> + (list
> + ,@(map (lambda (entry)
> + `(manifest-entry
> + (name ,(manifest-entry-name entry))
> + (version ,(manifest-entry-version entry))
> + (output ,(manifest-entry-output entry))
> + (item (read-derivation-from-file
> + ,(derivation-file-name
> + (manifest-entry-item entry))))))
> + (manifest-entries manifest))))))

Here we have to rely on a larger part of the API: a subset of (guix
profiles) and (guix derivations).

Conversely, in (guix channels), the only assumption made about the API
implemented by the other Guix (which might be older or might be newer)
is the ‘generate-package-cache’ procedure. It’s a small requirement, so
potentially easier to satisfy in future versions for a long time.


When dealing with these time travel issues, I feel we have difficult
choices to make. It’s a bit of an unusual requirement that we have.

Ludo’.
L
L
Ludovic Courtès wrote on 9 Jul 2022 00:01
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87zghjqn6d.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (3 lines)
> The patches below do that. As discussed on IRC, it’s not pretty but
> it’s pragmatic.

Pushed!

e80f0cda96 etc: Add 'time-travel-manifest.scm'.
c9fbd40785 channels: Emit version 3 profiles.
89e2288751 profiles: Support the creation of profiles with version 3 manifests.

The last commit was inspired by zimoun’s test. I plan to add a
‘time-travel’ jobset on ci.guix.

Thank you zimoun & Ricardo!

Ludo’.
Closed
L
L
Ludovic Courtès wrote on 11 Jul 2022 10:27
87zghgnjex.fsf_-_@gnu.org
Hi!

Toggle quote (4 lines)
> e80f0cda96 etc: Add 'time-travel-manifest.scm'.
> c9fbd40785 channels: Emit version 3 profiles.
> 89e2288751 profiles: Support the creation of profiles with version 3 manifests.

And here’s the CI job:


OK, it’s currently broken, but it should be unbroken by commit
5d0437ea8ce0147b47b9df866fa2413b48f71a1a (I’ve set it to run every 12
hours at most, so we’ll have to check.)

Ludo’.
?
Your comment

This issue is archived.

To comment on this conversation send an email to 56441@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 56441
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch