different fixed-output when using ’--with-source’ transformation

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Maxime Devos
  • zimoun
Owner
unassigned
Submitted by
zimoun
Severity
normal
Z
Z
zimoun wrote on 15 Sep 2021 07:44
different fixed-output when using ’--with-sourc e’ transformation
(name . Bug Guix)(address . bug-guix@gnu.org)
86zgsecrzv.fsf@gmail.com
Hi,

If the package source tarball is provided to ’--with-source’, then the
hash differs. I would expect a fixed point.

Let investigate:

Toggle snippet (7 lines)
$ guix build hello@2.10 --with-source=hello=https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz
/gnu/store/zfwhbbknkhxi3yqmp0qgh1l1crljgbm6-hello-2.10

$ guix build hello@2.10
/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10

official mirror. I expect the same store item.

Why are they different? It is because the derivations are different.
The first call contains the source tarball as an “input”; that’s fine.
The second call contains the source tarball as a derivation; again
that’s fine. The corresponding first and second derivations are:

Toggle snippet (15 lines)
Derive
([("out","/gnu/store/zfwhbbknkhxi3yqmp0qgh1l1crljgbm6-hello-2.10","","")]
,[("/gnu/store/101ny738l311p4fm3cas7jgnkzagjv6a-module-import-compiled.drv",["out"])

[...]

,("/gnu/store/z4s27gwacbw8f38andfsh21r8v330dag-xz-5.2.4.drv",["out"])]
,["/gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import",
"/gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz",
"/gnu/store/rndq9g8877l29ha41dvsl3aj1z0gw0ng-hello-2.10-guile-builder"]

[...]


and

Toggle snippet (15 lines)
Derive
([("out","/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10","","")]
,[("/gnu/store/101ny738l311p4fm3cas7jgnkzagjv6a-module-import-compiled.drv",["out"])
,("/gnu/store/1a7xfcqcxj0pqi4f81x1agcxa46v2bbm-hello-2.10.tar.gz.drv",["out"])

[...]

,("/gnu/store/z4s27gwacbw8f38andfsh21r8v330dag-xz-5.2.4.drv",["out"])]
,["/gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import",
"/gnu/store/kql8b2hbsabcmany4m3hfm3wzdiymliy-hello-2.10-guile-builder"]

[...]


Therefore, I understand why the two store items have different hashes.
Although, it should be better that the derivations would be the same;
another story. :-)


What puzzled me are the corresponding ’guile-builder’s:

Toggle snippet (5 lines)
(define %build-inputs
(quote
(("source" . "/gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz")

and

Toggle snippet (5 lines)
(define %build-inputs
(quote
(("source" . "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz")

but these 2 items have the same hash:

Toggle snippet (7 lines)
$ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

$ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

Why? It is fixed-output so I was expecting the same thing. And the
same as:

Toggle snippet (11 lines)
$ guix download https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz

Starting download of /tmp/guix-file.qkhxkp
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirrors.sarata.com/gnu/hello/hello-2.10.tar.gz'...
…10.tar.gz 709KiB 549KiB/s 00:01 [##################]
100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

Why the store item of source tarball is it different when applying the
transformation?

Maxime provided a clue [1].

All the best,
simon

L
L
Ludovic Courtès wrote on 18 Sep 2021 18:06
Re: bug#50597: different fixed-output when using ’--with-source’ transformation
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 50597@debbugs.gnu.org)
87fsu1uaup.fsf@gnu.org
Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (5 lines)
> Why the store item of source tarball is it different when applying the
> transformation?
>
> Maxime provided a clue [1].

Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
whereas it’s usually #f for a ‘url-fetch’ kind of URI.

What we could do is make it #t if and only if the URL denotes a local
file and that file is a directory. I think that would achieve what you
want without any observable regression in practice.

WDYT?

Thanks,
Ludo’.
M
M
Maxime Devos wrote on 18 Sep 2021 19:35
(address . 50597@debbugs.gnu.org)
bfbc4c2bd52137e55afb32c424e62d576d9cbf73.camel@telenet.be
Ludovic Courtès schreef op za 18-09-2021 om 18:06 [+0200]:
Toggle quote (18 lines)
> Hello!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > Why the store item of source tarball is it different when applying the
> > transformation?
> >
> > Maxime provided a clue [1].
>
> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
> whereas it’s usually #f for a ‘url-fetch’ kind of URI.
>
> What we could do is make it #t if and only if the URL denotes a local
> file and that file is a directory. I think that would achieve what you
> want without any observable regression in practice.
>
> WDYT?

This seems reasonable to me.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYUYjURccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7oHOAQCg5aZN+D+ze0P5pr7U5H1WZM52
m2AS+jL24v5H/nSX4QD/fIPYjX4kXQ/IcB3OzPjeNFKnfCrqIDS57zgVDlKdBAA=
=OtFz
-----END PGP SIGNATURE-----


Z
Z
zimoun wrote on 21 Sep 2021 09:40
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 50597@debbugs.gnu.org)
86czp2cr63.fsf@gmail.com
Hi Ludo,

On Sat, 18 Sep 2021 at 18:06, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (10 lines)
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> Why the store item of source tarball is it different when applying the
>> transformation?
>>
>> Maxime provided a clue [1].
>
> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
> whereas it’s usually #f for a ‘url-fetch’ kind of URI.

Thanks for confirming. Why is it different?


Toggle quote (4 lines)
> What we could do is make it #t if and only if the URL denotes a local
> file and that file is a directory. I think that would achieve what you
> want without any observable regression in practice.

What I expect is that 2 (fixed-output) files with the same hash
(0ssi1w…) lives at the same store location…

Toggle snippet (7 lines)
$ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

$ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

…whatever if these (fixed-output) files come from “guix download”, “guix
build -S” or “--with-source=https://” or etc.

The mapping from hash (0ssi1w…) to store location should be 1:1 for
fixed-output, IMHO.

Cheers,
simon
L
L
Ludovic Courtès wrote on 22 Sep 2021 12:25
(name . zimoun)(address . zimon.toutoune@gmail.com)
87bl4kj4af.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (13 lines)
> On Sat, 18 Sep 2021 at 18:06, Ludovic Courtès <ludo@gnu.org> wrote:
>> zimoun <zimon.toutoune@gmail.com> skribis:
>>
>>> Why the store item of source tarball is it different when applying the
>>> transformation?
>>>
>>> Maxime provided a clue [1].
>>
>> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
>> whereas it’s usually #f for a ‘url-fetch’ kind of URI.
>
> Thanks for confirming. Why is it different?

Because we want ‘--with-source’ to also work with directories (see
comment in ‘package-with-source’).

Toggle quote (13 lines)
>> What we could do is make it #t if and only if the URL denotes a local
>> file and that file is a directory. I think that would achieve what you
>> want without any observable regression in practice.
>
> What I expect is that 2 (fixed-output) files with the same hash
> (0ssi1w…) lives at the same store location…
>
> $ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
> 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
>
> $ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
> 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i

The difference here is that one is marked as “recursive” and the other
is not.

The patch below does one I suggested earlier: passing #:recursive? #t if
and only if we know the source is a directory.

However it’s not enough to get the same derivation: we’d also need
<downloaded-file> to return a fixed-output derivation rather than a
plain store file name. (See also https://issues.guix.gnu.org/49696.)

To be continued…

Ludo’.
Toggle diff (41 lines)
diff --git a/guix/transformations.scm b/guix/transformations.scm
index 5ae1977cb2..e0f5579c28 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -46,6 +46,7 @@
#:use-module (srfi srfi-37)
#:use-module (ice-9 match)
#:use-module (ice-9 vlist)
+ #:autoload (web uri) (string->uri uri-scheme uri-path)
#:export (options->transformation
manifest-entry-with-transformations
@@ -106,15 +107,25 @@ extensions."
(define* (package-with-source p uri #:optional version)
"Return a package based on P but with its source taken from URI. Extract
the new package's version number from URI."
- (let ((base (tarball-base-name (basename uri))))
+ (let ((base (tarball-base-name (basename uri)))
+ (file (match (string->uri uri)
+ (#f uri)
+ (uri (and (eq? 'file (uri-scheme uri))
+ (uri-path uri))))))
(let-values (((_ version*)
(hyphen-package-name->name+version base)))
(package (inherit p)
(version (or version version*
(package-version p)))
- ;; Use #:recursive? #t to allow for directories.
- (source (downloaded-file uri #t))))))
+ ;; Default to #:recursive? #f to match what 'url-fetch' does,
+ ;; but use #t when URI denotes a directory.
+ (source (let ((recursive?
+ (and file
+ (match (stat file #f)
+ (#f #t)
+ (st (eq? 'directory (stat:type st)))))))
+ (downloaded-file uri recursive?)))))))
;;;
Z
Z
zimoun wrote on 13 Oct 2021 11:31
(name . Ludovic Courtès)(address . ludo@gnu.org)
86wnmhi83q.fsf@gmail.com
Hi,

On Wed, 22 Sep 2021 at 12:25, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (5 lines)
> However it’s not enough to get the same derivation: we’d also need
> <downloaded-file> to return a fixed-output derivation rather than a
> plain store file name. (See also
> <https://issues.guix.gnu.org/49696>.)

Thanks for the explanations and the pointer. Indeed, to be continued…


Toggle quote (39 lines)
> diff --git a/guix/transformations.scm b/guix/transformations.scm
> index 5ae1977cb2..e0f5579c28 100644
> --- a/guix/transformations.scm
> +++ b/guix/transformations.scm
> @@ -46,6 +46,7 @@
> #:use-module (srfi srfi-37)
> #:use-module (ice-9 match)
> #:use-module (ice-9 vlist)
> + #:autoload (web uri) (string->uri uri-scheme uri-path)
> #:export (options->transformation
> manifest-entry-with-transformations
>
> @@ -106,15 +107,25 @@ extensions."
> (define* (package-with-source p uri #:optional version)
> "Return a package based on P but with its source taken from URI. Extract
> the new package's version number from URI."
> - (let ((base (tarball-base-name (basename uri))))
> + (let ((base (tarball-base-name (basename uri)))
> + (file (match (string->uri uri)
> + (#f uri)
> + (uri (and (eq? 'file (uri-scheme uri))
> + (uri-path uri))))))
> (let-values (((_ version*)
> (hyphen-package-name->name+version base)))
> (package (inherit p)
> (version (or version version*
> (package-version p)))
>
> - ;; Use #:recursive? #t to allow for directories.
> - (source (downloaded-file uri #t))))))
> + ;; Default to #:recursive? #f to match what 'url-fetch' does,
> + ;; but use #t when URI denotes a directory.
> + (source (let ((recursive?
> + (and file
> + (match (stat file #f)
> + (#f #t)
> + (st (eq? 'directory (stat:type st)))))))
> + (downloaded-file uri recursive?)))))))

LGTM!

Cheers,
simon
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 50597@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 50597
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch