[PATCH 0/2] Workaround rust reproducibility issues

  • Done
  • quality assurance status badge
Details
4 participants
  • Danny Milosavljevic
  • Joe Hillenbrand
  • Maxim Cournoyer
  • Nikolai Merinov
Owner
unassigned
Submitted by
Nikolai Merinov
Severity
normal
N
N
Nikolai Merinov wrote on 16 Oct 2018 20:00
(address . guix-patches@gnu.org)
877eihu5hx.fsf@member.fsf.org
Hi,

I prepared patches for Rust packages.

First patch move rust 1.25-1.27 packages back to llvm 3.9.1. I spend much time
tring to find root of non-reproducibility in this releases, but I did not
succeeded. So as easiest solution I suggest just keep old LLVM for this
packages. Drawback of this solution is impossibility to use
"#[target_feature]" will not work for this releases.

Second patch have new 1.28.0, 1.29.1 rust packages. They was reproducible in
my local tests with llvm 6.0.1, so this packages was switched to new LLVM.

Please, note: I tested packages with suggested changes only from
https://github.com/mnd/guix-mnd-pkgs/repository. Full build chain
reproducibility test with exactly submitted patches currently in progress on
my laptop, but it can take another one or two days.

Regards,
Nikolai

Nikolai Merinov (2):
gnu: rust: workaround rust 1.25-27 reproducibility issues
gnu: rust: add rust 1.28 and rust 1.29 packages

gnu/local.mk | 1 +
...ible-builds-by-forcing-window.search.patch | 28 +++++++
gnu/packages/rust.scm | 76 +++++++++++++------
3 files changed, 81 insertions(+), 24 deletions(-)
create mode 100644 gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch

--
2.19.0
N
N
Nikolai Merinov wrote on 15 Oct 2018 23:32
[PATCH 1/2] gnu: rust: workaround rust 1.25-27 reproducibility issues
(address . 33066@debbugs.gnu.org)
875zy1u5ec.fsf@member.fsf.org
* gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
patch that make "searchindex.js" reproducible in rust 1.27 and newer.
* gnu/local.mk (dist_patch_DATA): Add new patch file.
* gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
during cargo build. Note: libgit2 still builded as part of cargo build,
because cargo tests assume specific libgit2 minor release.
(rust-1.23): inherit native-inputs from previous package.
(rust-1.25): switch back to llvm 3.9.1 as workaround for
(rust-1.27): apply patch to make "searchindex.js" files reproducible.
---
gnu/local.mk | 1 +
...ible-builds-by-forcing-window.search.patch | 28 ++++++++++++++++
gnu/packages/rust.scm | 32 ++++++-------------
3 files changed, 38 insertions(+), 23 deletions(-)
create mode 100644 gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch

Toggle diff (122 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index 1fa2eaba3..2150cbf13 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1130,6 +1130,7 @@ dist_patch_DATA = \
%D%/packages/patches/rust-1.25-accept-more-detailed-gdb-lines.patch \
%D%/packages/patches/rust-bootstrap-stage0-test.patch \
%D%/packages/patches/rust-coresimd-doctest.patch \
+ %D%/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch \
%D%/packages/patches/rxvt-unicode-escape-sequences.patch \
%D%/packages/patches/scheme48-tests.patch \
%D%/packages/patches/scotch-test-threading.patch \
diff --git a/gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch b/gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch
new file mode 100644
index 000000000..916662cbc
--- /dev/null
+++ b/gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch
@@ -0,0 +1,28 @@
+From b9ca108fcae2b738ca3f0c88c84ae5dc5a6f843f Mon Sep 17 00:00:00 2001
+From: Tim Ryan <id@timryan.org>
+Date: Mon, 14 May 2018 06:22:21 -0400
+Subject: [PATCH] Support reproducible builds by forcing window.search to use
+ stable key ordering. (#692)
+
+---
+ src/vendor/mdbook/src/renderer/html_handlebars/search.rs | 4 ++++
+ 1 file changed, 4 insertions(+)
+
+diff --git a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
+index d49772f8b..1ee66a511 100644
+--- a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
++++ b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
+@@ -205,6 +205,10 @@ fn write_to_js(index: Index, search_config: &Search) -> Result<String> {
+ searchoptions,
+ index,
+ };
++
++ // By converting to serde_json::Value as an intermediary, we use a
++ // BTreeMap internally and can force a stable ordering of map keys.
++ let json_contents = serde_json::to_value(&json_contents)?;
+ let json_contents = serde_json::to_string(&json_contents)?;
+
+ Ok(format!("window.search = {};", json_contents))
+--
+2.19.0
+
diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index 40633854a..cbfbdccf1 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -480,7 +480,8 @@ test = { path = \"../libtest\" }
`(("jemalloc" ,jemalloc-4.5.0)
("llvm" ,llvm-3.9.1)
("openssl" ,openssl)
- ("libcurl" ,curl))) ; For "cargo"
+ ("libssh2" ,libssh2) ; For "cargo"
+ ("libcurl" ,curl))) ; For "cargo"
;; rustc invokes gcc, so we need to set its search paths accordingly.
;; Note: duplicate its value here to cope with circular dependencies among
@@ -649,18 +650,11 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(name "rust")
(version "1.23.0")
(source (rust-source version "14fb8vhjzsxlbi6yrn1r6fl5dlbdd1m92dn5zj5gmzfwf4w9ar3l"))
+ ;; Use rust-bootstrap@1.22 package to build rust 1.23
(native-inputs
- `(("bison" ,bison) ; For the tests
- ("cmake" ,cmake)
- ("flex" ,flex) ; For the tests
- ("gdb" ,gdb) ; For the tests
- ("git" ,git)
- ("procps" ,procps) ; For the tests
- ("python-2" ,python-2)
- ("rustc-bootstrap" ,rust-bootstrap)
- ("cargo-bootstrap" ,rust-bootstrap "cargo")
- ("pkg-config" ,pkg-config) ; For "cargo"
- ("which" ,which)))
+ (alist-replace "cargo-bootstrap" (list rust-bootstrap "cargo")
+ (alist-replace "rustc-bootstrap" (list rust-bootstrap)
+ (package-native-inputs rust-1.20))))
(arguments
(substitute-keyword-arguments (package-arguments rust-1.20)
((#:phases phases)
@@ -699,10 +693,6 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
#:patches '("rust-1.25-accept-more-detailed-gdb-lines.patch"))))
(package
(inherit base-rust)
- (inputs
- ;; Use LLVM 6.0
- (alist-replace "llvm" (list llvm)
- (package-inputs base-rust)))
(arguments
(substitute-keyword-arguments (package-arguments base-rust)
((#:phases phases)
@@ -713,14 +703,9 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
;; This test wants to update the crate index.
(("fn no_index_update") "#[ignore]\nfn no_index_update"))
#t))
- (add-after 'configure 'enable-codegen-tests
- (lambda _
- (substitute* "config.toml"
- (("codegen-tests = false") ""))
- #t))
;; FIXME: Re-enable this test if it's indeed supposed to work.
;; See <https://github.com/rust-lang/rust/issues/54178>.
- (add-after 'enable-codegen-tests 'disable-nil-enum-test
+ (add-after 'configure 'disable-nil-enum-test
(lambda _
(substitute* "src/test/debuginfo/nil-enum.rs"
(("ignore-lldb") "ignore-gdb"))
@@ -795,7 +780,8 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
#:patches
'("rust-coresimd-doctest.patch"
"rust-bootstrap-stage0-test.patch"
- "rust-1.25-accept-more-detailed-gdb-lines.patch"))))
+ "rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
(package
(inherit base-rust)
(arguments
--
2.19.0
N
N
Nikolai Merinov wrote on 15 Oct 2018 23:37
[PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages
(address . 33066@debbugs.gnu.org)
874ldlu5dm.fsf@member.fsf.org
* gnu/packages/rust.scm (rust-1.27): rename "rust" to "rust-1.27"
(rust-1.28): Use llvm-6.0 finally. Disable test that broken on x86_64 machines
without "avx". Disable test that required network support.
(rust): Package for rust 1.29.1.
---
gnu/packages/rust.scm | 44 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 43 insertions(+), 1 deletion(-)

Toggle diff (61 lines)
diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index cbfbdccf1..d61b431e8 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -773,7 +773,7 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(("fn no_index_update") "#[ignore]\nfn no_index_update"))
#t)))))))))
-(define-public rust
+(define-public rust-1.27
(let ((base-rust
(rust-bootstrapped-package rust-1.26 "1.27.2"
"0pg1s37bhx9zqbynxyydq5j6q7kij9vxkcv8maz0m25prm88r0cs"
@@ -795,3 +795,45 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(mkdir-p (assoc-ref outputs "out"))
(mkdir-p (assoc-ref outputs "cargo"))
#t)))))))))
+
+(define-public rust-1.28
+ (let ((base-rust
+ (rust-bootstrapped-package rust-1.27 "1.28.0"
+ "11k4rn77bca2rikykkk9fmprrgjswd4x4kaq7fia08vgkir82nhx"
+ #:patches
+ '("rust-coresimd-doctest.patch"
+ "rust-bootstrap-stage0-test.patch"
+ "rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
+ (package
+ (inherit base-rust)
+ (inputs
+ ;; Use LLVM 6.0
+ (alist-replace "llvm" (list llvm)
+ (package-inputs base-rust)))
+ (arguments
+ (substitute-keyword-arguments (package-arguments base-rust)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (add-after 'configure 'enable-codegen-tests
+ ;; codegen tests should work fine with llvm 6
+ (lambda* _
+ (substitute* "config.toml"
+ (("codegen-tests = false") ""))
+ #t))
+ (add-after 'patch-tests 'disable-amd64-avx-test
+ ;; This test will fail on x86_64 machines without avx
+ (lambda* _
+ (substitute* "src/test/run-pass/issue-44056.rs"
+ (("only-x86_64") "ignore-test")))))))))))
+
+
+(define-public rust
+ (let ((base-rust
+ (rust-bootstrapped-package rust-1.28 "1.29.1"
+ "0jd3c57x3yndizns4pb68nh25si47agfmrdvf9nwwsyfcs5p5c7i"
+ #:patches
+ '("rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
+ (package
+ (inherit base-rust))))
--
2.19.0
D
D
Danny Milosavljevic wrote on 17 Oct 2018 13:14
Re: [bug#33066] [PATCH 1/2] gnu: rust: workaround rust 1.25-27 reproducibility issues
(name . Nikolai Merinov)(address . nikolai.merinov@member.fsf.org)(address . 33066@debbugs.gnu.org)
20181017131406.01c3b011@scratchpost.org
Hi Nikolai,

On Tue, 16 Oct 2018 02:32:11 +0500
Nikolai Merinov <nikolai.merinov@member.fsf.org> wrote:

Toggle quote (2 lines)
> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:

Nitpick: No big "S" (file names are easier to find if they are all lower case).

Toggle quote (2 lines)
> patch that make "searchindex.js" reproducible in rust 1.27 and newer.

"New file".

Toggle quote (3 lines)
> * gnu/local.mk (dist_patch_DATA): Add new patch file.
> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library

Hmm, I'm not sure about doing this in the same commit.
Is it also related to reproducibility?

Toggle quote (3 lines)
> during cargo build. Note: libgit2 still builded as part of cargo build,
> because cargo tests assume specific libgit2 minor release.

What does this mean? Does it mean "bundled"?

Toggle quote (2 lines)
> (rust-1.23): inherit native-inputs from previous package.

Ok.

Toggle quote (3 lines)
> (rust-1.25): switch back to llvm 3.9.1 as workaround for
> https://github.com/rust-lang/rust/issues/50556 issue.

Please add the reasoning as a comment inside the source code instead.

Toggle quote (2 lines)
> (rust-1.27): apply patch to make "searchindex.js" files reproducible.

Maybe add "[source]".

Toggle quote (6 lines)
> - (add-after 'configure 'enable-codegen-tests
> - (lambda _
> - (substitute* "config.toml"
> - (("codegen-tests = false") ""))
> - #t))

I think I had reproducibility problems when enabling codegen tests and
parallel tests. Is that not the case anymore?

Toggle quote (3 lines)
> ;; FIXME: Re-enable this test if it's indeed supposed to work.
> ;; See <https://github.com/rust-lang/rust/issues/54178>.

Note to myself: I think the issue comments indicate that the newer gdb output
is better - so we should create a patch similar to
rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvHGX4ACgkQ5xo1VCww
uqUBhQf9HbQqdFlnetu0WXtD6gxK8iYsoeKzhb+kgK/TKHDS6/IIVz2oMVF25D9w
/D38a65UsEwR6Eh16twZDWw2Kz4ubVPxhLWUixHUJLp/SXqXocVP9KpSNkgRE3nu
1r52AkZKjrglC/OGMUFWrf1ZvitCLDD+Qd4wcIIGkkOeSSjpJFB27QadcbdabCIN
sDa5/kMty3xVS38/0JALCdOnslyRqcaFIoBMhP/1hFEXfLU8whAD12WWQ6ydnwvq
qTdkj0nch62ICVoMEQy8CCrZNl1JsBNCxNdHI33hiJfG9jLjiMYEQ9eVbux0KPfy
gLKHu/DlKcfQ43C2Yqf3RYviGI75EA==
=802C
-----END PGP SIGNATURE-----


D
D
Danny Milosavljevic wrote on 17 Oct 2018 13:15
Re: [bug#33066] [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages
(name . Nikolai Merinov)(address . nikolai.merinov@member.fsf.org)(address . 33066@debbugs.gnu.org)
20181017131536.3eb2f38c@scratchpost.org
Toggle quote (6 lines)
> + (add-after 'patch-tests 'disable-amd64-avx-test
> + ;; This test will fail on x86_64 machines without avx
> + (lambda* _
> + (substitute* "src/test/run-pass/issue-44056.rs"
> + (("only-x86_64") "ignore-test")))))))))))

Please return #t.

Otherwise ok!
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvHGdgACgkQ5xo1VCww
uqUaVAf/dezyd8WhOgW6r8sqgB2DLx+0UVH2ZIfYNF240Xqw9XAl4BM0uIscAiKq
x5Y2N7GUZ+jIqlMjDyLTKE+e24vBbm9vRxqYEbG4fQxdt6iys0ro1CqVrw63j6ro
uzw/WOC67NOgzifk9zQV9phawuNfwrBdJWk4vAq7uM93+rJR/I7QDtplq2m4OiYo
f9vyyshHO0nSXFTrPERA64qW/M4fxXNV54V/k84kyipm+cmLh6YrbxK+2JENWxzc
y5wKndxxy6/9//PSrTfXJMHM8moH3XQU5tkELItW1dYIUBDnOdQtDToubc4f7sak
aED5ExtFMuDepTu/SBVwjXl9JueVWg==
=TnV5
-----END PGP SIGNATURE-----


J
J
Joe Hillenbrand wrote on 17 Oct 2018 16:14
(address . nikolai.merinov@member.fsf.org)(address . 33066@debbugs.gnu.org)
CANJQ5C5MtvxS6aDwLFLCPhqwajJ=VEUyPjpoaY_0-mPPc-sE6w@mail.gmail.com
On Tue, Oct 16, 2018 at 11:50 AM Nikolai Merinov
<nikolai.merinov@member.fsf.org> wrote:
Toggle quote (5 lines)
>
> +(define-public rust
> + (let ((base-rust
> + (rust-bootstrapped-package rust-1.28 "1.29.1"

The latest version is now 1.29.2
N
N
Nikolai Merinov wrote on 19 Oct 2018 21:44
Re: [bug#33066] [PATCH 1/2] gnu: rust: workaround rust 1.25-27 reproducibility issues
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 33066@debbugs.gnu.org)
87y3atsouq.fsf@member.fsf.org
Hi Danny,

I fixed patches according to your comments. Also I updated rust 1.29.1
to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested
reproducibility of each package previously only in private my private
repo where I disabled `check` phase for 1.25 and 1.27. I have pretty
slow machine, but currently I already checked reproducibility with
suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress.

Both updated patches attached.

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (9 lines)
> Hi Nikolai,
>
> On Tue, 16 Oct 2018 02:32:11 +0500
> Nikolai Merinov <nikolai.merinov@member.fsf.org> wrote:
>
>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
>
> Nitpick: No big "S" (file names are easier to find if they are all lower case).
>
Fixed.

Toggle quote (3 lines)
>> patch that make "searchindex.js" reproducible in rust 1.27 and newer.
>
> "New file".
Fixed.

Toggle quote (7 lines)
>
>> * gnu/local.mk (dist_patch_DATA): Add new patch file.
>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
>
> Hmm, I'm not sure about doing this in the same commit.
> Is it also related to reproducibility?

Looks like it should not, but still I started investigation with
non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2
related changes at very beginning and never tested wihtout it. I not
sure that it's good idea to remove it now.

Toggle quote (5 lines)
>
>> during cargo build. Note: libgit2 still builded as part of cargo build,
>> because cargo tests assume specific libgit2 minor release.
>
> What does this mean? Does it mean "bundled"?
Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo).

Toggle quote (9 lines)
>
>> (rust-1.23): inherit native-inputs from previous package.
>
> Ok.
>
>> (rust-1.25): switch back to llvm 3.9.1 as workaround for
>> https://github.com/rust-lang/rust/issues/50556 issue.
>
> Please add the reasoning as a comment inside the source code instead.
Added comment for rust-1.25 package.

Toggle quote (4 lines)
>
>> (rust-1.27): apply patch to make "searchindex.js" files reproducible.
>
> Maybe add "[source]".
I repharase this comment and added source URL to patch file itself.

Toggle quote (9 lines)
>
>> - (add-after 'configure 'enable-codegen-tests
>> - (lambda _
>> - (substitute* "config.toml"
>> - (("codegen-tests = false") ""))
>> - #t))
>
> I think I had reproducibility problems when enabling codegen tests and
> parallel tests. Is that not the case anymore?
Neither me nor Joe Hillenbrand in
reproducibility issues with this tests.
Toggle quote (7 lines)
>
>> ;; FIXME: Re-enable this test if it's indeed supposed to work.
>> ;; See <https://github.com/rust-lang/rust/issues/54178>.
>
> Note to myself: I think the issue comments indicate that the newer gdb output
> is better - so we should create a patch similar to
> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.
From 9cf7f62414c5e714d7bb0cc2874d91f561e418df Mon Sep 17 00:00:00 2001
From: Nikolai Merinov <nikolai.merinov@member.fsf.org>
Date: Tue, 16 Oct 2018 02:37:41 +0500
Subject: [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages
To: guix-patches@gnu.org

* gnu/packages/rust.scm (rust-1.27): rename "rust" to "rust-1.27"
(rust-1.28): Use llvm-6.0 finally. Disable test that broken on x86_64 machines
without "avx". Enable codegen tests. Enable cargo test for thinlto.
(rust): Package for rust 1.29.2.
---
gnu/packages/rust.scm | 46 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 45 insertions(+), 1 deletion(-)

Toggle diff (63 lines)
diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index 190dbcc3a..a0e891249 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -776,7 +776,7 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(("fn no_index_update") "#[ignore]\nfn no_index_update"))
#t)))))))))
-(define-public rust
+(define-public rust-1.27
(let ((base-rust
(rust-bootstrapped-package rust-1.26 "1.27.2"
"0pg1s37bhx9zqbynxyydq5j6q7kij9vxkcv8maz0m25prm88r0cs"
@@ -804,3 +804,47 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(substitute* "src/tools/cargo/tests/testsuite/path.rs"
(("fn thin_lto_works") "#[ignore]\nfn thin_lto_works"))
#t)))))))))
+
+(define-public rust-1.28
+ (let ((base-rust
+ (rust-bootstrapped-package rust-1.27 "1.28.0"
+ "11k4rn77bca2rikykkk9fmprrgjswd4x4kaq7fia08vgkir82nhx"
+ #:patches
+ '("rust-coresimd-doctest.patch"
+ "rust-bootstrap-stage0-test.patch"
+ "rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
+ (package
+ (inherit base-rust)
+ (inputs
+ ;; Use LLVM 6.0
+ (alist-replace "llvm" (list llvm)
+ (package-inputs base-rust)))
+ (arguments
+ (substitute-keyword-arguments (package-arguments base-rust)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (add-after 'configure 'enable-codegen-tests
+ ;; codegen tests should pass with llvm 6
+ (lambda* _
+ (substitute* "config.toml"
+ (("codegen-tests = false") ""))
+ #t))
+ (add-after 'patch-tests 'disable-amd64-avx-test
+ ;; This test will fail on x86_64 machines without avx
+ (lambda* _
+ (substitute* "src/test/run-pass/issue-44056.rs"
+ (("only-x86_64") "ignore-test"))
+ #t))
+ ;; thinlto test should pass with llvm 6
+ (delete 'disable-thinlto-test))))))))
+
+(define-public rust
+ (let ((base-rust
+ (rust-bootstrapped-package rust-1.28 "1.29.2"
+ "1jb787080z754caa2w3w1amsygs4qlzj9rs1vy64firfmabfg22h"
+ #:patches
+ '("rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
+ (package
+ (inherit base-rust))))
--
2.19.0
N
N
Nikolai Merinov wrote on 19 Oct 2018 22:58
Re: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27 reproducibility issues
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 33066@debbugs.gnu.org)
87sh11slfg.fsf_-_@member.fsf.org
Hi,

Sorry, ignore my previous mail. There was wrong patch
export. Re-attached patches.
From 4eb82de9971c13323e7b0195f9f178282039ba35 Mon Sep 17 00:00:00 2001
From: Nikolai Merinov <nikolai.merinov@member.fsf.org>
Date: Tue, 16 Oct 2018 02:37:41 +0500
Subject: [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages
To: guix-patches@gnu.org

* gnu/packages/rust.scm (rust-1.27): rename "rust" to "rust-1.27"
(rust-1.28): Use llvm-6.0 finally. Disable test that broken on x86_64 machines
without "avx". Enable codegen tests. Enable cargo test for thinlto.
(rust): Package for rust 1.29.2.
---
gnu/packages/rust.scm | 46 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 45 insertions(+), 1 deletion(-)

Toggle diff (63 lines)
diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index cd2a8fb9a..aafa3704b 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -776,7 +776,7 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(("fn no_index_update") "#[ignore]\nfn no_index_update"))
#t)))))))))
-(define-public rust
+(define-public rust-1.27
(let ((base-rust
(rust-bootstrapped-package rust-1.26 "1.27.2"
"0pg1s37bhx9zqbynxyydq5j6q7kij9vxkcv8maz0m25prm88r0cs"
@@ -804,3 +804,47 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(substitute* "src/tools/cargo/tests/testsuite/path.rs"
(("fn thin_lto_works") "#[ignore]\nfn thin_lto_works"))
#t)))))))))
+
+(define-public rust-1.28
+ (let ((base-rust
+ (rust-bootstrapped-package rust-1.27 "1.28.0"
+ "11k4rn77bca2rikykkk9fmprrgjswd4x4kaq7fia08vgkir82nhx"
+ #:patches
+ '("rust-coresimd-doctest.patch"
+ "rust-bootstrap-stage0-test.patch"
+ "rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch"))))
+ (package
+ (inherit base-rust)
+ (inputs
+ ;; Use LLVM 6.0
+ (alist-replace "llvm" (list llvm)
+ (package-inputs base-rust)))
+ (arguments
+ (substitute-keyword-arguments (package-arguments base-rust)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (add-after 'configure 'enable-codegen-tests
+ ;; codegen tests should pass with llvm 6
+ (lambda* _
+ (substitute* "config.toml"
+ (("codegen-tests = false") ""))
+ #t))
+ (add-after 'patch-tests 'disable-amd64-avx-test
+ ;; This test will fail on x86_64 machines without avx
+ (lambda* _
+ (substitute* "src/test/run-pass/issue-44056.rs"
+ (("only-x86_64") "ignore-test"))
+ #t))
+ ;; thinlto test should pass with llvm 6
+ (delete 'disable-thinlto-test))))))))
+
+(define-public rust
+ (let ((base-rust
+ (rust-bootstrapped-package rust-1.28 "1.29.2"
+ "1jb787080z754caa2w3w1amsygs4qlzj9rs1vy64firfmabfg22h"
+ #:patches
+ '("rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch"))))
+ (package
+ (inherit base-rust))))
--
2.19.0
Regards,
Nikolai

Nikolai Merinov <nikolai.merinov@member.fsf.org> writes:

Toggle quote (84 lines)
> Hi Danny,
>
> I fixed patches according to your comments. Also I updated rust 1.29.1
> to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested
> reproducibility of each package previously only in private my private
> repo where I disabled `check` phase for 1.25 and 1.27. I have pretty
> slow machine, but currently I already checked reproducibility with
> suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress.
>
> Both updated patches attached.
>
> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
>> Hi Nikolai,
>>
>> On Tue, 16 Oct 2018 02:32:11 +0500
>> Nikolai Merinov <nikolai.merinov@member.fsf.org> wrote:
>>
>>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
>>
>> Nitpick: No big "S" (file names are easier to find if they are all lower case).
>>
> Fixed.
>
>>> patch that make "searchindex.js" reproducible in rust 1.27 and newer.
>>
>> "New file".
> Fixed.
>
>>
>>> * gnu/local.mk (dist_patch_DATA): Add new patch file.
>>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
>>
>> Hmm, I'm not sure about doing this in the same commit.
>> Is it also related to reproducibility?
>
> Looks like it should not, but still I started investigation with
> non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2
> related changes at very beginning and never tested wihtout it. I not
> sure that it's good idea to remove it now.
>
>>
>>> during cargo build. Note: libgit2 still builded as part of cargo build,
>>> because cargo tests assume specific libgit2 minor release.
>>
>> What does this mean? Does it mean "bundled"?
> Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo).
>
>>
>>> (rust-1.23): inherit native-inputs from previous package.
>>
>> Ok.
>>
>>> (rust-1.25): switch back to llvm 3.9.1 as workaround for
>>> https://github.com/rust-lang/rust/issues/50556 issue.
>>
>> Please add the reasoning as a comment inside the source code instead.
> Added comment for rust-1.25 package.
>
>>
>>> (rust-1.27): apply patch to make "searchindex.js" files reproducible.
>>
>> Maybe add "[source]".
> I repharase this comment and added source URL to patch file itself.
>
>>
>>> - (add-after 'configure 'enable-codegen-tests
>>> - (lambda _
>>> - (substitute* "config.toml"
>>> - (("codegen-tests = false") ""))
>>> - #t))
>>
>> I think I had reproducibility problems when enabling codegen tests and
>> parallel tests. Is that not the case anymore?
> Neither me nor Joe Hillenbrand in
> https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00292.html got
> reproducibility issues with this tests.
>>
>>> ;; FIXME: Re-enable this test if it's indeed supposed to work.
>>> ;; See <https://github.com/rust-lang/rust/issues/54178>.
>>
>> Note to myself: I think the issue comments indicate that the newer gdb output
>> is better - so we should create a patch similar to
>> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.
N
N
Nikolai Merinov wrote on 21 Oct 2018 09:28
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 33066@debbugs.gnu.org)
87mur7sqrf.fsf@member.fsf.org
Hi,

Finally whole rust build chain 1.23-1.29 was built on my machine with next commands:
Toggle quote (2 lines)
> guix pull --url=/path/to/guix-f9a8fce10/with/patches --commit=<commit-with-patches>
> for v in 1.2{3..9}; do echo guix build -K --rounds=2 rust@$v || break ; done
and all builds were reproducible.

Regards,
Nikolai

Nikolai Merinov <nikolai.merinov@member.fsf.org> writes:

Toggle quote (97 lines)
> Hi,
>
> Sorry, ignore my previous mail. There was wrong patch
> export. Re-attached patches.
>
>
>
>
> Regards,
> Nikolai
>
> Nikolai Merinov <nikolai.merinov@member.fsf.org> writes:
>
>> Hi Danny,
>>
>> I fixed patches according to your comments. Also I updated rust 1.29.1
>> to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested
>> reproducibility of each package previously only in private my private
>> repo where I disabled `check` phase for 1.25 and 1.27. I have pretty
>> slow machine, but currently I already checked reproducibility with
>> suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress.
>>
>> Both updated patches attached.
>>
>> Danny Milosavljevic <dannym@scratchpost.org> writes:
>>
>>> Hi Nikolai,
>>>
>>> On Tue, 16 Oct 2018 02:32:11 +0500
>>> Nikolai Merinov <nikolai.merinov@member.fsf.org> wrote:
>>>
>>>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
>>>
>>> Nitpick: No big "S" (file names are easier to find if they are all lower case).
>>>
>> Fixed.
>>
>>>> patch that make "searchindex.js" reproducible in rust 1.27 and newer.
>>>
>>> "New file".
>> Fixed.
>>
>>>
>>>> * gnu/local.mk (dist_patch_DATA): Add new patch file.
>>>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
>>>
>>> Hmm, I'm not sure about doing this in the same commit.
>>> Is it also related to reproducibility?
>>
>> Looks like it should not, but still I started investigation with
>> non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2
>> related changes at very beginning and never tested wihtout it. I not
>> sure that it's good idea to remove it now.
>>
>>>
>>>> during cargo build. Note: libgit2 still builded as part of cargo build,
>>>> because cargo tests assume specific libgit2 minor release.
>>>
>>> What does this mean? Does it mean "bundled"?
>> Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo).
>>
>>>
>>>> (rust-1.23): inherit native-inputs from previous package.
>>>
>>> Ok.
>>>
>>>> (rust-1.25): switch back to llvm 3.9.1 as workaround for
>>>> https://github.com/rust-lang/rust/issues/50556 issue.
>>>
>>> Please add the reasoning as a comment inside the source code instead.
>> Added comment for rust-1.25 package.
>>
>>>
>>>> (rust-1.27): apply patch to make "searchindex.js" files reproducible.
>>>
>>> Maybe add "[source]".
>> I repharase this comment and added source URL to patch file itself.
>>
>>>
>>>> - (add-after 'configure 'enable-codegen-tests
>>>> - (lambda _
>>>> - (substitute* "config.toml"
>>>> - (("codegen-tests = false") ""))
>>>> - #t))
>>>
>>> I think I had reproducibility problems when enabling codegen tests and
>>> parallel tests. Is that not the case anymore?
>> Neither me nor Joe Hillenbrand in
>> https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00292.html got
>> reproducibility issues with this tests.
>>>
>>>> ;; FIXME: Re-enable this test if it's indeed supposed to work.
>>>> ;; See <https://github.com/rust-lang/rust/issues/54178>.
>>>
>>> Note to myself: I think the issue comments indicate that the newer gdb output
>>> is better - so we should create a patch similar to
>>> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.
D
D
Danny Milosavljevic wrote on 24 Oct 2018 21:43
(name . Nikolai Merinov)(address . nikolai.merinov@member.fsf.org)(address . 33066@debbugs.gnu.org)
20181024214330.1f90b72e@scratchpost.org
Hi Nikolai,

I have reviewed the patchset and am currently building the entire rust chain
with your patches with several rounds and plan on merging rust after a
successful run (probably Friday).
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvQy2IACgkQ5xo1VCww
uqV7qwf9H5ncb1epek8KbEqrpJorONzXST7LM4PB/eieVIm2YiwTjj1rNBWgswjz
kMC2adZbdqsDBB9Qvsy/r9+owXKR3f2aaiXEOcksQqnsYOBB5UCuTlsvEfNXmvFB
+XOZMoFDp5HO/v9O+AvX6p4gpofOpX5neHiQiwIsAPnbmPi7ozfJBcWPo31u+O7a
Zv4MDdWKM330ZNtFBIFQva+5J+egotFMguki7kVjmSq5HI464CzTECtqQ3OA5nVM
K+RL1S2tZ083d7LgT/GT0AYyUYnJ5o7hkOwuhl+IWxLfY6OSxQ+7PluxvnvU0/FQ
9br6nfBqphzEOy1jr4Dq5aS8LnJl0Q==
=DOO6
-----END PGP SIGNATURE-----


D
D
Danny Milosavljevic wrote on 25 Oct 2018 07:44
(name . Nikolai Merinov)(address . nikolai.merinov@member.fsf.org)(address . 33066@debbugs.gnu.org)
20181025074422.08edec2a@scratchpost.org
Hi Nikolai,

I've got a cargo reproducibility failure in rust 1.29.2. Attached is diffoscope output :(

Seems a little similar to what we had before (https://github.com/rust-lang/rust/issues/50556).
Attachment: cargo-1.29.2.diff
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvRWDYACgkQ5xo1VCww
uqU5DQf+IzEqRfhgj1iWJMtMPSBzQAF8hou9z3OcQ8L9rJ7pyVLRurKFpHRC5wxY
uBpvSvqjOMWPJHjcHuZLDL02aBQdeRwLceiVxlr05ks/KmD19KbNpjBDfezR84dC
X0MZU41qhQLpD/q0do6xYLQkNMbzSzFvl489Vz2Z3rBCnWHO/T0ChAIwZGcuhXGw
LWvBmJ9t8cM4FL8Gizkdpoos1tsFSB/J+zETW1jzOHaLfgmFsmgjSu/CqRDZRrfu
kqf3UUq9OYVCVL4+x0fODTk4t/bbsc9FlkHV2avZVwmg9lIJ6KxmL+0ryVXJ7DJ1
YN13YYiAbUfbSNsfCX+aFcFO69vntg==
=0WVQ
-----END PGP SIGNATURE-----


N
N
Nikolai Merinov wrote on 26 Oct 2018 13:40
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)(address . 33066@debbugs.gnu.org)
877ei5lyvh.fsf@member.fsf.org
Hi Danny,

I'm sad to hear it. All this issues looks for me as LLVM reproducibility
issue and there was some reproducibility-related changes after LLVM
https://reviews.llvm.org/D48392and others. Note: I checked previously
mentioned changes individually and it did not solved reproducibility
issue for rust.

Unfortunately I'll be not able to return to this reproducibility
investigation for a while. So currently I can suggest to merge changes
without rust 1.29 to have reproducible chain from 1.23 to 1.28.

Regards,
Nikolai

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (5 lines)
> Hi Nikolai,
>
> I've got a cargo reproducibility failure in rust 1.29.2. Attached is diffoscope output :(
>
> Seems a little similar to what we had before (https://github.com/rust-lang/rust/issues/50556 ).
D
D
Danny Milosavljevic wrote on 26 Oct 2018 18:12
(name . Nikolai Merinov)(address . nikolai.merinov@member.fsf.org)(address . 33066@debbugs.gnu.org)
20181026181234.62729a14@scratchpost.org
Hi Nikolai,

On Fri, 26 Oct 2018 16:40:34 +0500
Nikolai Merinov <nikolai.merinov@member.fsf.org> wrote:

Toggle quote (4 lines)
> Unfortunately I'll be not able to return to this reproducibility
> investigation for a while. So currently I can suggest to merge changes
> without rust 1.29 to have reproducible chain from 1.23 to 1.28.

I did so now.

Thanks for the hard work!
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvTPPIACgkQ5xo1VCww
uqWaYgf9FQ0tUhv1OckOXsOUtNVdN3uA2Orakb/dBc9df6nCvonXwlb4VjU4PJxN
ah2lItJ33ZVO0AsOOSodqT7NZsmipnMOc+zX8eoxHjekZepDChHTKIZh4bKerQe0
WI3fGZ/5fpOhIcyGf8ebdvQEQamdRohAS/qnBuu8N2OJ+7oHq1VLUk3asjhHD09T
Ar/zBag7QgNjEaYd73vDRb0HP/0/OGXuePIU3DbOUwPo95JvS7EKVePIHA6LzZDj
oXNM7JnBQ1j3hshf54WlWlR8KekC+YBlH9bfDwrc1zus8uAppqTv32vk/BBOFiXy
2YY5CBDhB7IV31XUmTBIMPzzf6pCYQ==
=P4AY
-----END PGP SIGNATURE-----


M
M
Maxim Cournoyer wrote on 5 Feb 2021 16:00
Re: bug#33066: [PATCH 0/2] Workaround rust reproducibility issues
(name . Danny Milosavljevic)(address . dannym@scratchpost.org)
878s82o8zu.fsf_-_@gmail.com
Hello,

Danny Milosavljevic <dannym@scratchpost.org> writes:

Toggle quote (13 lines)
> Hi Nikolai,
>
> On Fri, 26 Oct 2018 16:40:34 +0500
> Nikolai Merinov <nikolai.merinov@member.fsf.org> wrote:
>
>> Unfortunately I'll be not able to return to this reproducibility
>> investigation for a while. So currently I can suggest to merge changes
>> without rust 1.29 to have reproducible chain from 1.23 to 1.28.
>
> I did so now.
>
> Thanks for the hard work!

I'm closing this; the work to bootstrap using mrustc 0.9 to start from
rust 1.29.2 has been merged to core-updates; it brought many changes.

Thank you,

Maxim
Closed
?