Racket 8.6 is not reproducible

  • Open
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Philip McGrath
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 5 Sep 2022 12:04
(address . bug-guix@gnu.org)
87pmga3zot.fsf@inria.fr
Hey,

Just noticed that Racket does not build in a reproducible fashion:

Toggle snippet (52 lines)
$ guix describe
Generation 227 Sep 04 2022 23:39:52 (current)
guix aae98c2
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: aae98c297214f87eb45302863adb021078c41a6f
$ guix challenge racket | head -40
/gnu/store/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6 contents differ:
local hash: 1w4dnkvpbbrgfasyq8x1cbqw58jzqsny17ms5l1fb1h6iid38bs1
https://ci.guix.gnu.org/nar/lzip/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6: 1lnxklizpnc599w7n2svb1jaw595wranm9aagd2928fcbiaavbr6
differing files:
/lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.dep
/lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.zo
/lib/racket/pkgs/algol60/compiled/algol60_rkt.dep
/lib/racket/pkgs/algol60/compiled/algol60_scrbl.dep
/lib/racket/pkgs/algol60/compiled/algol60_scrbl.zo
/lib/racket/pkgs/algol60/compiled/cfg-parser_rkt.dep
/lib/racket/pkgs/algol60/compiled/compile_rkt.dep
/lib/racket/pkgs/algol60/compiled/parse_rkt.dep
/lib/racket/pkgs/algol60/compiled/parse_rkt.zo
/lib/racket/pkgs/algol60/compiled/simplify_rkt.dep
/lib/racket/pkgs/algol60/compiled/tool_rkt.dep
/lib/racket/pkgs/algol60/lang/compiled/reader_rkt.dep
/lib/racket/pkgs/algol60/tests/compiled/export_rkt.dep
/lib/racket/pkgs/algol60/tests/compiled/export_rkt.zo
/lib/racket/pkgs/algol60/tests/compiled/syncheck-test_rkt.dep
/lib/racket/pkgs/algol60/tests/compiled/test_rkt.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/awk_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/awk_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/cmdline_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/cmdline_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/cml_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/cml_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/common_rkt.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/common_rkt.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/compat_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/compat_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/compile_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/compile_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/contract-label_rkt.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/contract_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/contract_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/etc_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/etc_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/file_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/file_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/for_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/for_scrbl.zo
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/include_scrbl.dep
/lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/include_scrbl.zo

Ludo’.
P
P
Philip McGrath wrote on 7 Sep 2022 19:36
5f6e754e-1a0e-4f23-866d-08dcd7a111a3@www.fastmail.com
Hi,

On Mon, Sep 5, 2022, at 6:04 AM, Ludovic Courtès wrote:
Toggle quote (21 lines)
> Hey,
>
> Just noticed that Racket does not build in a reproducible fashion:
>
> --8<---------------cut here---------------start------------->8---
> $ guix describe
> Generation 227 Sep 04 2022 23:39:52 (current)
> guix aae98c2
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: aae98c297214f87eb45302863adb021078c41a6f
> $ guix challenge racket | head -40
> /gnu/store/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6 contents differ:
> local hash: 1w4dnkvpbbrgfasyq8x1cbqw58jzqsny17ms5l1fb1h6iid38bs1
>
> https://ci.guix.gnu.org/nar/lzip/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6:
> 1lnxklizpnc599w7n2svb1jaw595wranm9aagd2928fcbiaavbr6
> differing files:
> /lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.dep
> /lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.zo

Unfortunately, this is a known issue upstream.

One part of the problem is the way that Chez Scheme uses "type 4" (random) UUIDs to implement 'gensym'. There's some upstream discussion at https://github.com/cisco/ChezScheme/issues/585, including an idea from me about "type 5" UUIDs as part of a possible solution.

That may well be the only issue with the core compiler for Racket CS. I know effort has been taken to make Racket's front-end deterministic, and Kent Dybvig says that:

Toggle quote (8 lines)
> Chez Scheme compiles are reproducible modulo the globally unique
> names of generated symbols (gensyms), assuming macros don't do
> things like introduce time stamps or random expressions into their output.
> Because of differences in gensym names, the files produced by multiple
> runs are not byte-by-byte identical but should compare equal via
> the system primitive $fasl-file-equal? See, for example, the use
> of $fasl-file-equal? in the s/Mf-base checkboot target.

In the output you sent, the ".zo" files contain Chez machine code with a Racket-specific wrapper, and the ".dep" files contain hashes of dependencies (so they differ when the ".zo" files they track differ).


Toggle quote (4 lines)
> Build path captured in html documentation. docindex.sqlite
> contains content in arbitrary order. strings in binaries in
> arbitrary order.

Those issues don't appear to affect Guix: some might be caused by Debian conflating "build" and "install" for Racket packages, and, in any case, they seem related to the "scribble" and/or "racket-index" packages, not the core compiler.

IIRC, there were some reproducibility issues with the Racket BC bytecode compiler. If we can build reproducibly with CS but not BC, it might work out to disable compilation for the bootstrapping BC compilers, or maybe to configure them to use the machine-independent format (basically, a serialized form of what the front-end would hand off to the back-end), at the cost of some performance.

I hope to work on these issues, but I plan to work on a 'racket-build-system' first. Among other things, I hope it will be easier to investigate issues when Racket packages can be built independently.

-Philip
?