'guix shell --export-manifest' fails for some transformations

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Suhail Singh
  • Simon Tournier
Owner
unassigned
Submitted by
Simon Tournier
Severity
normal
S
S
Simon Tournier wrote on 25 Oct 19:08 +0200
(address . bug-guix@gnu.org)
87a5eshzzy.fsf@gmail.com
Hi,

I notice this:

Toggle snippet (58 lines)
$ cat manifest.scm
(use-modules (guix transformations) (guix packages)
(guix download) (guix build-system r))

(define r-knitr
(specification->package "r-knitr"))

(define r-knitr-other
(package
(inherit r-knitr)
(version "1.47")
(source (origin
(method url-fetch)
(uri (cran-uri "knitr" version))
(sha256
(base32
"1kx6g8ddbigwx3bmq771gwi3din3fxjn59l801904kjaz6dq9pgs"))))))

(define r-ggplot2
(package
(inherit
((package-input-rewriting `((,r-knitr . ,r-knitr-other)))
(specification->package "r-ggplot2")))
(name "my-ggplot")))

(define transform
(options->transformation
'((with-c-toolchain . "r-minimal=gcc-toolchain@12")
(with-c-toolchain . "r=gcc-toolchain@12"))))

(packages->manifest
(append
(map specification->package
(list "python" "python-numpy"))
(map transform
(list (specification->package "r")
r-ggplot2))))

$ guix shell -m manifest.scm --export-manifest
guix shell: warning: transformation 'with-c-toolchain' had no effect on my-ggplot@3.5.1.
;; What follows is a "manifest" equivalent to the command line you gave.
;; You can store it in a file that you may then pass to any 'guix' command
;; that accepts a '--manifest' (or '-m') option.

(use-modules (guix transformations))

(define transform1
(options->transformation
'((with-c-toolchain . "r-minimal=gcc-toolchain@12")
(with-c-toolchain . "r=gcc-toolchain@12"))))

(packages->manifest
(list (specification->package "python")
(specification->package "python-numpy")
(transform1 (specification->package "r"))
(transform1 (specification->package "my-ggplot"))))

Well, this is incorrect. While I understand the approximation when
dealing with profiles built over time in the imperative way, here I am
missing why the transformation of ’r-ggplot2’ is skipped. And even the
export manifest is broken.

It’s skipped because:

Toggle snippet (6 lines)
$ guix shell -m manifest.scm
$ grep ggplot $GUIX_ENVIRONMENT/manifest
("my-ggplot"
"/gnu/store/x0jbmvg0nbl7lyk8zd938rq2c7c9qkn4-my-ggplot-3.5.1"

Please note that if the symbol ’r-ggplot2’ is not named “my-ggplot” but
kept as ’r-ggplot2“:

(define r-ggplot2
((package-input-rewriting `((,r-knitr . ,r-knitr-other)))
(specification->package "r-ggplot2")))

then the bug is maybe worse because the exported manifest “works” but
without the transformation… which can be misleading and hard to detect.

Well, I understand it’s hard to capture this kind of transformation.
Still, it appears to me as an half-baked feature. :-)

That’s annoying in some context as “guix pack --save-provenance” [1].
Even, it defeats the idea of a self-contained reproducible binary
container. ;-)

Maybe, aside the profile ’manifest’ file, we could store all the
manifests provided by the command line. Something as:

/gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
/gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
/gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2

where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
and ’bar.scm’ from:

-m foo.scm -m bar.scm

This way, the option --export-manifest could be improved.

WDYT?

Cheers,
simon

L
L
Ludovic Courtès wrote on 10 Nov 12:38 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 74015@debbugs.gnu.org)
87v7wvwc6t.fsf@gnu.org
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (2 lines)
> $ guix shell -m manifest.scm --export-manifest

‘--export-manifest’ is meant to “translate” a command line, which it can
do faithfully; there’s no way it could possibly “recreate” what
‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
already has a ‘manifest.scm’ file probably doesn’t need
‘--export-manifest’).

It’s necessarily a lossy process. Some of the transformations
‘manifest.scm’ uses may be recorded as metadata in manifest entries, but
others, such as uses of ‘package-input-rewriting’ here, inevitably get
lost.

I would close it as not-a-bug; WDYT?

Ludo’.
S
S
Simon Tournier wrote on 10 Nov 14:06 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 74015@debbugs.gnu.org)
87serzxmpj.fsf@gmail.com
Hi,

On Sun, 10 Nov 2024 at 12:38, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (8 lines)
>> $ guix shell -m manifest.scm --export-manifest
>
> ‘--export-manifest’ is meant to “translate” a command line, which it can
> do faithfully; there’s no way it could possibly “recreate” what
> ‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
> already has a ‘manifest.scm’ file probably doesn’t need
> ‘--export-manifest’).

Yes, one might need. :-)

As I explained: « That’s annoying in some context as “guix pack
--save-provenance” [1]. Even, it defeats the idea of a self-contained
reproducible binary container. »

For sure, I understand it’s hard to capture this kind of transformation.
Still, an improvement appears to me possible (see below).



Toggle quote (2 lines)
> It’s necessarily a lossy process.

This is where I disagree. :-) I mean, yes I agree that building
profile/manifest is somehow a lossy process because some Scheme is
potentially evaluated on the road. However, I am proposing: Aside the
profile ’manifest’ file (lossy process), we could store all the
manifests provided by the command line. Something as:

/gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
/gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
/gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2

where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
and ’bar.scm’ from:

-m foo.scm -m bar.scm

This way, the option --export-manifest could be improved, eventually.

In other word, I think that saving as-is the manifests costs almost
nothing and it paves the way to more robust self-contained binary packs.

Well, maybe this extra work could be only done when there is an option
’--save-provenance’ applied.

Toggle quote (2 lines)
> I would close it as not-a-bug; WDYT?

I still think it’s a feature that it’s possible to improve.

Cheers,
simon
S
S
Suhail Singh wrote on 12 Nov 16:33 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87frnwmpp4.fsf@gmail.com
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (19 lines)
> However, I am proposing: Aside the profile ’manifest’ file (lossy
> process), we could store all the manifests provided by the command
> line. Something as:
>
> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
> …
>
> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
> and ’bar.scm’ from:
>
> -m foo.scm -m bar.scm
>
> ...
>
> Well, maybe this extra work could be only done when there is an option
> ’--save-provenance’ applied.

For --save-provenance, is your proposal that in addition to saving the
additional manifests, the commandline that was used would also be saved?

--
Suhail
S
S
Simon Tournier wrote on 16 Nov 08:20 +0100
(name . Suhail Singh)(address . suhailsingh247@gmail.com)
87jzd3bq5q.fsf@gmail.com
Hi,

On Tue, 12 Nov 2024 at 10:33, Suhail Singh <suhailsingh247@gmail.com> wrote:
Toggle quote (24 lines)
> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
>> However, I am proposing: Aside the profile ’manifest’ file (lossy
>> process), we could store all the manifests provided by the command
>> line. Something as:
>>
>> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
>> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
>> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
>> …
>>
>> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
>> and ’bar.scm’ from:
>>
>> -m foo.scm -m bar.scm
>>
>> ...
>>
>> Well, maybe this extra work could be only done when there is an option
>> ’--save-provenance’ applied.
>
> For --save-provenance, is your proposal that in addition to saving the
> additional manifests, the commandline that was used would also be saved?

We could save the command line too using another file. :-)

Cheers,
simon
S
S
Suhail Singh wrote on 17 Nov 20:05 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87frnpd6kd.fsf@gmail.com
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (5 lines)
>> For --save-provenance, is your proposal that in addition to saving the
>> additional manifests, the commandline that was used would also be saved?
>
> We could save the command line too using another file. :-)

That would be helpful :)

Given an image generated with --save-provenance enabled, making it
easier for it to be reconstructed by another would certainly be useful.

--
Suhail
L
L
Ludovic Courtès wrote 5 days ago
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 74015@debbugs.gnu.org)
87ttc2yz9k.fsf@gnu.org
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (16 lines)
> On Sun, 10 Nov 2024 at 12:38, Ludovic Courtès <ludo@gnu.org> wrote:
>
>>> $ guix shell -m manifest.scm --export-manifest
>>
>> ‘--export-manifest’ is meant to “translate” a command line, which it can
>> do faithfully; there’s no way it could possibly “recreate” what
>> ‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
>> already has a ‘manifest.scm’ file probably doesn’t need
>> ‘--export-manifest’).
>
> Yes, one might need. :-)
>
> As I explained: « That’s annoying in some context as “guix pack
> --save-provenance” [1]. Even, it defeats the idea of a self-contained
> reproducible binary container. »

I see, but if you have the source ‘manifest.scm’, best practice is to
publish it.

Toggle quote (4 lines)
>> It’s necessarily a lossy process.
>
> This is where I disagree. :-)

You are free to disagree but it’s a fact.

Toggle quote (16 lines)
> I mean, yes I agree that building profile/manifest is somehow a lossy
> process because some Scheme is potentially evaluated on the road.
> However, I am proposing: Aside the profile ’manifest’ file (lossy
> process), we could store all the manifests provided by the command
> line. Something as:
>
> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
> /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
> …
>
> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
> and ’bar.scm’ from:
>
> -m foo.scm -m bar.scm

Hmm yes, maybe ‘guix pack -m manifest.scm --save-provenance’ could
arrange to put ‘manifest.scm’ inside the pack, similar to how ‘guix
system reconfigure’ inserts ‘configuration.scm’ inside the system.

I’m all for it!

Thanks,
Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74015@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74015
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch