Rust packages are not reproducible

Open

Details

3 participants

Efraim Flashner
Ludovic Courtès
Maxim Cournoyer

Owner: unassigned

Submitted by: Ludovic Courtès

Severity: normal

Ludovic Courtès wrote on 11 Aug 2021 23:15

Recipients:(address . bug-guix@gnu.org)

Message-ID:87czqjk7j6.fsf@inria.fr

Hello!

Rust packages, which are essentially empty, are not bit-reproducible:

Toggle snippet (28 lines)$ ./pre-inst-env guix challenge rust-rocket-codegen --substitute-urls='https://ci.guix.gnu.org https://bordeaux.guix.gnu.org' 
/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7 contents differ:
  no local build for '/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7'
  https://ci.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 0j6zf2afvc49jnp7i6z7yvbxm0bmw8yc65hz3lncgvw5lc6z1bc1
  https://bordeaux.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 015kb637b56mqcsg3f6x1qggm2bybiszji2069gb913wxbj6rs7w
  differing file:
    /share/cargo/registry/rocket_codegen-0.4.7.crate

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive
ludo@ribbon ~/src/guix$ ./pre-inst-env guix challenge rust-rocket-codegen
/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7 contents differ:
  no local build for '/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7'
  https://ci.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 0j6zf2afvc49jnp7i6z7yvbxm0bmw8yc65hz3lncgvw5lc6z1bc1
  https://bordeaux.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 015kb637b56mqcsg3f6x1qggm2bybiszji2069gb913wxbj6rs7w
  differing file:
    /share/cargo/registry/rocket_codegen-0.4.7.crate

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive
$ git log |head -1
commit 973842acbc2d0dc1ab41738a534d4abda6d9efa7

The diffoscope output suggests it’s about timestamps on one file in the

.crate archive:

Toggle snippet (10 lines)

? ? ? ? --- /tmp/guix-directory.ii5wmv/share/cargo/registry/rocket_codegen-0.4.7.crate

? ? ? ??? +++ /tmp/guix-directory.uTTKSw/share/cargo/registry/rocket_codegen-0.4.7.crate

? ? ? ? ??? rocket_codegen-0.4.7.crate-content

? ? ? ? ? ??? file list

? ? ? ? ? ? @@ -1,67 +1,67 @@

? ? ? ? ? ? --rw-r--r-- 0 0 0 1293 2021-07-27 15:22:18.000000 rocket_codegen-0.4.7/Cargo.toml

[…]

? ? ? ? ? ? +-rw-r--r-- 0 0 0 1293 2021-07-27 22:01:49.000000 rocket_codegen-0.4.7/Cargo.toml

Does that ring a bell?

Thanks,

Ludo’.

PS: I noticed this via

http://data.guix.gnu.org/revision/973842acbc2d0dc1ab41738a534d4abda6d9efa7/package-reproducibility

with help from Chris. Fixing this could noticeably improve our

stats. :-)

Efraim Flashner wrote on 12 Aug 2021 08:44

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 50015@debbugs.gnu.org)

Message-ID:YRTDOLDQK3bAV6gK@3900XT

On Wed, Aug 11, 2021 at 11:15:09PM +0200, Ludovic Courtès wrote:

Toggle quote (58 lines)> Hello!
> 
> Rust packages, which are essentially empty, are not bit-reproducible:
> 
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix challenge rust-rocket-codegen --substitute-urls='https://ci.guix.gnu.org https://bordeaux.guix.gnu.org' 
> /gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7 contents differ:
>   no local build for '/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7'
>   https://ci.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 0j6zf2afvc49jnp7i6z7yvbxm0bmw8yc65hz3lncgvw5lc6z1bc1
>   https://bordeaux.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 015kb637b56mqcsg3f6x1qggm2bybiszji2069gb913wxbj6rs7w
>   differing file:
>     /share/cargo/registry/rocket_codegen-0.4.7.crate
> 
> 1 store items were analyzed:
>   - 0 (0.0%) were identical
>   - 1 (100.0%) differed
>   - 0 (0.0%) were inconclusive
> ludo@ribbon ~/src/guix$ ./pre-inst-env guix challenge rust-rocket-codegen
> /gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7 contents differ:
>   no local build for '/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7'
>   https://ci.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 0j6zf2afvc49jnp7i6z7yvbxm0bmw8yc65hz3lncgvw5lc6z1bc1
>   https://bordeaux.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 015kb637b56mqcsg3f6x1qggm2bybiszji2069gb913wxbj6rs7w
>   differing file:
>     /share/cargo/registry/rocket_codegen-0.4.7.crate
> 
> 1 store items were analyzed:
>   - 0 (0.0%) were identical
>   - 1 (100.0%) differed
>   - 0 (0.0%) were inconclusive
> $ git log |head -1
> commit 973842acbc2d0dc1ab41738a534d4abda6d9efa7
> --8<---------------cut here---------------end--------------->8---
> 
> The diffoscope output suggests it’s about timestamps on one file in the
> .crate archive:
> 
> --8<---------------cut here---------------start------------->8---
> ? ? ? ?   --- /tmp/guix-directory.ii5wmv/share/cargo/registry/rocket_codegen-0.4.7.crate
> ? ? ? ??? +++ /tmp/guix-directory.uTTKSw/share/cargo/registry/rocket_codegen-0.4.7.crate
> ? ? ? ? ??? rocket_codegen-0.4.7.crate-content
> ? ? ? ? ? ??? file list
> ? ? ? ? ? ? @@ -1,67 +1,67 @@
> ? ? ? ? ? ? --rw-r--r--   0        0        0     1293 2021-07-27 15:22:18.000000 rocket_codegen-0.4.7/Cargo.toml
> […]
> ? ? ? ? ? ? +-rw-r--r--   0        0        0     1293 2021-07-27 22:01:49.000000 rocket_codegen-0.4.7/Cargo.toml
> --8<---------------cut here---------------end--------------->8---
> 
> Does that ring a bell?
> 
> Thanks,
> Ludo’.
> 
> PS: I noticed this via
>     <http://data.guix.gnu.org/revision/973842acbc2d0dc1ab41738a534d4abda6d9efa7/package-reproducibility>
>     with help from Chris.  Fixing this could noticeably improve our
>     stats.  :-)
> 

I tried patching this a couple of ways, but it looks like the best
option is going to be a 'patch-and-repack phase after 'install. the
.crate file is really a gzip tarball, and I suspect that each time we
run 'cargo <something>' the timestamp gets updated.

-- 
Efraim Flashner   <efraim@flashner.co.il>   ????? ?????
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

Ludovic Courtès wrote on 12 Aug 2021 10:06

Recipients:(name . Efraim Flashner)(address . efraim@flashner.co.il)(address . 50015@debbugs.gnu.org)

Message-ID:874kbvhysf.fsf@gnu.org

Hello!

Efraim Flashner <efraim@flashner.co.il> skribis:

Toggle quote (5 lines)

> I tried patching this a couple of ways, but it looks like the best

> option is going to be a 'patch-and-repack phase after 'install. the

> .crate file is really a gzip tarball, and I suspect that each time we

> run 'cargo <something>' the timestamp gets updated.

So that ‘Cargo.toml’ file is not something taken from the build tree?

In that case we could reset the timestamp before the tarball is

created. But otherwise yeah, patch’n’repack.

Ludo’.

Maxim Cournoyer wrote on 4 Oct 2023 05:30

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:87sf6r2hfc.fsf@gmail.com

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (13 lines)> Hello!
>
> Efraim Flashner <efraim@flashner.co.il> skribis:
>
>> I tried patching this a couple of ways, but it looks like the best
>> option is going to be a 'patch-and-repack phase after 'install. the
>> .crate file is really a gzip tarball, and I suspect that each time we
>> run 'cargo <something>' the timestamp gets updated.
>
> So that ‘Cargo.toml’ file is not something taken from the build tree?
> In that case we could reset the timestamp before the tarball is
> created.  But otherwise yeah, patch’n’repack.

A better solution would be to have cargo honor SOURCE_DATE_EPOCH,
perhaps?  They'd probably accept such an improvement upstream.

-- 
Thanks,
Maxim

Efraim Flashner wrote on 4 Oct 2023 14:06

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:ZR1VVQOwzVJly35j@3900XT

On Tue, Oct 03, 2023 at 11:30:15PM -0400, Maxim Cournoyer wrote:

Toggle quote (20 lines)> Hello,
> 
> Ludovic Courtès <ludo@gnu.org> writes:
> 
> > Hello!
> >
> > Efraim Flashner <efraim@flashner.co.il> skribis:
> >
> >> I tried patching this a couple of ways, but it looks like the best
> >> option is going to be a 'patch-and-repack phase after 'install. the
> >> .crate file is really a gzip tarball, and I suspect that each time we
> >> run 'cargo <something>' the timestamp gets updated.
> >
> > So that ‘Cargo.toml’ file is not something taken from the build tree?
> > In that case we could reset the timestamp before the tarball is
> > created.  But otherwise yeah, patch’n’repack.
> 
> A better solution would be to have cargo honor SOURCE_DATE_EPOCH,
> perhaps?  They'd probably accept such an improvement upstream.

That'd be an interesting idea, having 'cargo package' set the timestamp
of all the files to SOURCE_DATE_EPOCH.  I guess I can look into how
feasible that would be and if they'd be likely to accept a change like
that.

I have a local patch which unpacks, resets the timestamp and repacks the
crate. I'll definitely push it to the rust-team branch before the next
merge.

With it I introduced an issue where the 'package phase would repack all
the crates, not just the current one, and ran into our
underscore-to-dash naming convention causing issues with how I'm reusing
the filename to work on the crate. I'll fix that, probably by only
repacking the current crate instead of all crates in the environment.

-- 
Efraim Flashner   <efraim@flashner.co.il>   ????? ?????
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

Maxim Cournoyer wrote on 4 Oct 2023 17:05

Recipients:(name . Efraim Flashner)(address . efraim@flashner.co.il)

Message-ID:8734yq2ztu.fsf@gmail.com

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> writes:

Toggle quote (36 lines)> On Tue, Oct 03, 2023 at 11:30:15PM -0400, Maxim Cournoyer wrote:
>> Hello,
>> 
>> Ludovic Courtès <ludo@gnu.org> writes:
>> 
>> > Hello!
>> >
>> > Efraim Flashner <efraim@flashner.co.il> skribis:
>> >
>> >> I tried patching this a couple of ways, but it looks like the best
>> >> option is going to be a 'patch-and-repack phase after 'install. the
>> >> .crate file is really a gzip tarball, and I suspect that each time we
>> >> run 'cargo <something>' the timestamp gets updated.
>> >
>> > So that ‘Cargo.toml’ file is not something taken from the build tree?
>> > In that case we could reset the timestamp before the tarball is
>> > created.  But otherwise yeah, patch’n’repack.
>> 
>> A better solution would be to have cargo honor SOURCE_DATE_EPOCH,
>> perhaps?  They'd probably accept such an improvement upstream.
>
> That'd be an interesting idea, having 'cargo package' set the timestamp
> of all the files to SOURCE_DATE_EPOCH.  I guess I can look into how
> feasible that would be and if they'd be likely to accept a change like
> that.
>
> I have a local patch which unpacks, resets the timestamp and repacks the
> crate. I'll definitely push it to the rust-team branch before the next
> merge.
>
> With it I introduced an issue where the 'package phase would repack all
> the crates, not just the current one, and ran into our
> underscore-to-dash naming convention causing issues with how I'm reusing
> the filename to work on the crate. I'll fix that, probably by only
> repacking the current crate instead of all crates in the environment.

From past interactions with Rust developers (via their web-based chat
system), I could get some change merged rather easily.  If you are
motivated to fix it cleanly I encourage you to pursue that way!

-- 
Thanks,
Maxim

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 50015@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 50015

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date