SBCL is not reproducible

Open

Details

4 participants

Guillaume Le Vaillant
Hendursaga
Pierre Neidhardt
zimoun

Owner: unassigned

Submitted by: zimoun

Severity: normal

Debbugs page

zimoun wrote 4 years ago

Recipients:(address . bug-guix@gnu.org)

Message-ID:86o8kw80bl.fsf@gmail.com

Dear,

Using Guix 58af4c9, the package ’sbcl’ seems not-reproducible.

The output of “guix challenge” is:

Toggle snippet (32 lines)  differing files:
    /lib/sbcl/contrib/sb-rt.fasl
    /lib/sbcl/contrib/uiop.fasl
    /lib/sbcl/contrib/sb-cover.fasl
    /lib/sbcl/contrib/sb-capstone.fasl
    /lib/sbcl/contrib/sb-executable.fasl
    /lib/sbcl/contrib/sb-bsd-sockets.fasl
    /lib/sbcl/contrib/sb-simple-streams.fasl
    /lib/sbcl/contrib/sb-queue.fasl
    /lib/sbcl/contrib/sb-gmp.fasl
    /lib/sbcl/contrib/sb-aclrepl.fasl
    /lib/sbcl/contrib/sb-sprof.fasl
    /lib/sbcl/contrib/sb-grovel.fasl
    /lib/sbcl/contrib/sb-posix.fasl
    /lib/sbcl/contrib/asdf.fasl
    /lib/sbcl/contrib/sb-mpfr.fasl
    /lib/sbcl/contrib/sb-cltl2.fasl
    /lib/sbcl/contrib/sb-concurrency.fasl
    /lib/sbcl/contrib/sb-md5.fasl
    /lib/sbcl/contrib/sb-introspect.fasl
    /lib/sbcl/contrib/sb-rotate-byte.fasl
    /lib/sbcl/sbcl.core
    /bin/sbcl
    /share/sbcl/src/runtime/coreparse.o
    /share/sbcl/src/runtime/sbcl

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive

which is confirmed by the experimental Data Service webpage:

https://data.guix-patches.cbaines.net/revision/4e0498c4d3f5ef03ac5afa4a3e570205c10c28c5/package-derivation-outputs?search_query=sbcl&output_consistency=not-matching&system=x86_64-linux&target=none&after_path=&limit_results=&all_results=on

I do not know if the patches in ’staging’ will fix this.

Note that this issue does not imply that the build system

’asdf-build-system/sbcl’ is or will be not reproducible. However, this

issues cuts any Diverse Double Compiling attempt.

All the best,

simon

Guillaume Le Vaillant wrote 4 years ago

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87wnzjzr77.fsf@yamatai

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (10 lines)> Using Guix 58af4c9, the package ’sbcl’ seems not-reproducible.
>
> [...]
>
> I do not know if the patches in ’staging’ will fix this.
>
> Note that this issue does not imply that the build system
> ’asdf-build-system/sbcl’ is or will be not reproducible.  However, this
> issues cuts any Diverse Double Compiling attempt.

IIRC, when compiling, SBCL puts the timestamp of the source file in the

compiled file. It's not a problem when just doing basic compilation

because the source files' timestamps have been set to "1970-01-01".

However, some packages generate some source files at build time, usually

containing things like data type sizes fetched from system header in

order to use C libraries with FFI. The timestamp of a generated file

is the current time, therefore the build is not reproducible.

IIRC, SBCL itself is built in 2 stages. First its core is compiled

using another Common Lisp implementation (currently clisp in Guix), then

the complete SBCL is compiled using the core compiled in stage 1. There

is probably also an embedded timestamp issue here (coming for clisp,

from SBCL, or both) causing the reproducibility issue.

The patches currently in the staging branch don't have any effect on

the generation of source files or on the format of the compiled files,

so they will not help with this issue.

Removing this source file timestamp from compiled files would simplify

things. Maybe nothing really depends on it and it would be possible...

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX5AsjA8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j9+WwD+NUWl7DQqk48V2PntLTM52xYfpmiBplWDSMrC

LBPgUYoA/1/e0p1lr5TV7cG9BJSmZzTHBubCEHDSIaRRyproDyuo

=M569

-----END PGP SIGNATURE-----

zimoun wrote 4 years ago

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)

Message-ID:CAJ3okZ1i-OqjjypxYeOcqXpxzAjKVHb9NUq7Gm4FQ76K8RbCmg@mail.gmail.com

Hi,

Thank you for the explanations.

On Wed, 21 Oct 2020 at 14:42, Guillaume Le Vaillant <glv@posteo.net> wrote:

Toggle quote (5 lines)

> However, some packages generate some source files at build time, usually

> containing things like data type sizes fetched from system header in

> order to use C libraries with FFI. The timestamp of a generated file

> is the current time, therefore the build is not reproducible.

This is an issue. Do you think it is affordable to fix these

timestamps to 1970-01-01?

Toggle quote (6 lines)

> IIRC, SBCL itself is built in 2 stages. First its core is compiled

> using another Common Lisp implementation (currently clisp in Guix), then

> the complete SBCL is compiled using the core compiled in stage 1. There

> is probably also an embedded timestamp issue here (coming for clisp,

> from SBCL, or both) causing the reproducibility issue.

Yes. But I have replaced this "clisp" by ECL or by SBCL itself.

Still unreproducible.

Out-of-scope with this bug report, my aim is to have a fixed point:

- first compile SBCL with CLISP (using the 2 stages you describe):

produce SBCL-A

- second recompile SBCL with the previous SBCL-A (again using the 2

stages): produce SBCL-B

- third recompile SBCL with the previous SBCL-B (again using the 2

stages): produces SBCL-C

The binary SBCL-A is not deterministic probably because of CLISP (and

CLISP should be fixed but that's another story :-)).

However, SBCL-B and SBCL-C must be deterministic. And ideally

bit-to-bit identical but that's another story. :-)

And they are not; from my experiments at least.

Even, you could do the same procedure replacing CLISP by ECL,

producing SBCL-{A,B,C}-bis. Then SBCL-C and SBCL-C-bis should be

bit-to-bit identical.

Toggle quote (3 lines)

> Removing this source file timestamp from compiled files would simplify

> things. Maybe nothing really depends on it and it would be possible...

Thanks for the explanation.

All the best,

simon

zimoun wrote 4 years ago

Re: bug#44112: SBCL is not reproducible

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)

Message-ID:CAJ3okZ1z0=jp8b=g4bqq=_=2yJk-z+McJ8PVcdy00cY2nwcyjg@mail.gmail.com

On Wed, 21 Oct 2020 at 15:39, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (6 lines)

> > IIRC, SBCL itself is built in 2 stages. First its core is compiled

> > using another Common Lisp implementation (currently clisp in Guix), then

> > the complete SBCL is compiled using the core compiled in stage 1. There

> > is probably also an embedded timestamp issue here (coming for clisp,

> > from SBCL, or both) causing the reproducibility issue.

In addition, GCC is also involved in the party. And I have also

replaced it by Clang with the same effect.

Cheers,

simon

Pierre Neidhardt wrote 4 years ago

Recipients:(address . 44112@debbugs.gnu.org)

Message-ID:87y2jzzm3d.fsf@ambrevar.xyz

zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (3 lines)

> In addition, GCC is also involved in the party. And I have also

> replaced it by Clang with the same effect.

Where?

Pierre Neidhardt

https://ambrevar.xyz/

Pierre Neidhardt wrote 4 years ago

Re: SBCL is not reproducible

Recipients:(address . 44112@debbugs.gnu.org)

Message-ID:87v9f3zm2t.fsf@ambrevar.xyz

Guillaume Le Vaillant <glv@posteo.net> writes:

Toggle quote (4 lines)

> IIRC, SBCL itself is built in 2 stages. First its core is compiled

> using another Common Lisp implementation (currently clisp in Guix), then

> the complete SBCL is compiled using the core compiled in stage 1.

Yes, this is correct.

Pierre Neidhardt

https://ambrevar.xyz/

zimoun wrote 4 years ago

Re: bug#44112: SBCL is not reproducible

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)

Message-ID:CAJ3okZ1UC3fXedXOkU=cwRQ6Tv_iy7xK3xUKtWbND2HBpaUwBA@mail.gmail.com

On Wed, 21 Oct 2020 at 16:32, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

Toggle quote (5 lines)

> > In addition, GCC is also involved in the party. And I have also

> > replaced it by Clang with the same effect.

> Where?

Toggle snippet (66 lines)

(define-module (ddc-sbcl)

#:use-module (guix packages)

#:use-module (guix utils)

#:use-module ((guix build utils) #:select (modify-phases))

#:use-module (srfi srfi-1)

#:use-module (ice-9 match)

#:use-module (gnu packages lisp)

#:use-module (gnu packages commencement)

#:use-module (gnu packages llvm))

(define (make-sbcl name string-compil compil cc)

(package

(inherit sbcl)

(name name)

(arguments

(substitute-keyword-arguments (package-arguments sbcl)

((#:phases phases)

`(modify-phases ,phases

(replace 'build

(lambda* (#:key outputs #:allow-other-keys)

(setenv "CC" ,cc)

(invoke "sh" "make.sh" ,string-compil

(string-append "--prefix="

(assoc-ref outputs "out"))

"--dynamic-space-size=2Gb"

"--with-sb-core-compression"

"--with-sb-xref-for-internals")))))))

(native-inputs

`(,@(alist-delete "clisp" (package-native-inputs sbcl))

("compil" ,compil)))))

(define-public sbcl-A

(package-with-c-toolchain

(make-sbcl "sbcl-A" "clisp" clisp "gcc")

`(("gcc" ,gcc-toolchain)))) ;fail with 6

(define-public sbcl-B

(package-with-c-toolchain

(make-sbcl "sbcl-B" "ecl" ecl "clang")

`(("clang" ,clang-toolchain))))

(define-public sbcl-AA

(package-with-c-toolchain

(make-sbcl "sbcl-AA" "sbcl" sbcl-A "gcc")

`(("gcc" ,gcc-toolchain))))

(define-public sbcl-BB

(package-with-c-toolchain

(make-sbcl "sbcl-BB" "sbcl" sbcl-B "clang")

`(("clang" ,clang-toolchain))))

(define-public sbcl-CC

(package-with-c-toolchain

(make-sbcl "sbcl-CC" "sbcl" sbcl-A "clang")

`(("clang" ,clang-toolchain))))

(define-public sbcl-DD

(package-with-c-toolchain

(make-sbcl "sbcl-DD" "sbcl" sbcl-B "gcc")

`(("gcc" ,gcc-toolchain))))

Guillaume Le Vaillant wrote 4 years ago

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87r1przfwo.fsf@yamatai

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (5 lines)

>> Removing this source file timestamp from compiled files would simplify

>> things. Maybe nothing really depends on it and it would be possible...

>Thanks for the explanation.

A developer of SBCL agrees that the timestamp should be removed [1], but
currently Slime has a function depending on it. I asked if this could
be fixed [2]. We'll see...

[1] https://sourceforge.net/p/sbcl/mailman/message/37133640/
[2] https://github.com/slime/slime/issues/583

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX5Bltw8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j/HgAD/UNRHvLALDFFdm71pw9wUxnTaLnrAzy//ifOO

O+ZuRroA/1U01XRKLlVACRVGA+xqEj0LDgY9iED5rJx0+hIZTVA4

=Rmx0

-----END PGP SIGNATURE-----

Pierre Neidhardt wrote 4 years ago

Recipients:(address . 44112@debbugs.gnu.org)

Message-ID:87imb334dv.fsf@ambrevar.xyz

Guillaume Le Vaillant <glv@posteo.net> writes:

Toggle quote (7 lines)

> A developer of SBCL agrees that the timestamp should be removed [1], but

> currently Slime has a function depending on it. I asked if this could

> be fixed [2]. We'll see...

> [1] https://sourceforge.net/p/sbcl/mailman/message/37133640/

> [2] https://github.com/slime/slime/issues/583

Thank you so much for asking!

Pierre Neidhardt

https://ambrevar.xyz/

zimoun wrote 4 years ago

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)

Message-ID:CAJ3okZ3_oSeEBgrKxKKevch39KGHF0Orkyqqk9m5on2VwQNbTw@mail.gmail.com

Hi Guillaume,

Thank you for asking upstream.

On Wed, 21 Oct 2020 at 18:45, Guillaume Le Vaillant <glv@posteo.net> wrote:

Toggle quote (6 lines)

> A developer of SBCL agrees that the timestamp should be removed [1], but

> currently Slime has a function depending on it. I asked if this could

> be fixed [2]. We'll see...

> [1] https://sourceforge.net/p/sbcl/mailman/message/37133640/

Ouch! Hard to read when not subscribed. :-)

Toggle quote (2 lines)

> [2] https://github.com/slime/slime/issues/583

If upstream does not want to patch, do you think it is doable to patch

SLIME and maintain the patch with the Guix package?

Thanks,

simon

Pierre Neidhardt wrote 4 years ago

Recipients:(address . 44112@debbugs.gnu.org)

Message-ID:87k0vi4fwi.fsf@ambrevar.xyz

zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (3 lines)

> If upstream does not want to patch, do you think it is doable to patch

> SLIME and maintain the patch with the Guix package?

Accepted or not, we also need to ping the SLY maintainer to apply the
same change.  (SLY is a fork of SLIME.)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

zimoun wrote 4 years ago

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)

Message-ID:86r1mg98la.fsf@gmail.com

Hi,

On Wed, 21 Oct 2020 at 14:41, Guillaume Le Vaillant <glv@posteo.net> wrote:

Toggle quote (4 lines)

> zimoun <zimon.toutoune@gmail.com> skribis:

>> Using Guix 58af4c9, the package ’sbcl’ seems not-reproducible.

[...]

Toggle quote (3 lines)

> Removing this source file timestamp from compiled files would simplify

> things. Maybe nothing really depends on it and it would be possible...

Any news from upstream? Still unreproducible for me.

All the best,

simon

Guillaume Le Vaillant wrote 4 years ago

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:874kj8ko2e.fsf@yamatai

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (18 lines)> Hi,
>
> On Wed, 21 Oct 2020 at 14:41, Guillaume Le Vaillant <glv@posteo.net> wrote:
>> zimoun <zimon.toutoune@gmail.com> skribis:
>>
>>> Using Guix 58af4c9, the package ’sbcl’ seems not-reproducible.
>
> [...]
>
>> Removing this source file timestamp from compiled files would simplify
>> things. Maybe nothing really depends on it and it would be possible...
>
> Any news from upstream?  Still unreproducible for me.
>
>
> All the best,
> simon

So far Slime still uses the function getting the source file timestamp,
so these timestamps can't be removed from SBCL.
A patch for Slime would be accepted (see [2]), but as far as I know
nobody has worked on it yet.

[2] https://github.com/slime/slime/issues/583

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCYAvqKQ8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j+WYgD/U5Pvxf1b08a5766PhJYlnBeD9rT6rRXaMoGa

mbGi5pIA/iGkqIHaAr7lzuKm+PHRtmkyJNTh3VLtymrt5Jyx2aoY

=HZ/Z

-----END PGP SIGNATURE-----

Hendursaga wrote 2 years ago

Recipients:(address . 44112@debbugs.gnu.org)

Message-ID:87zgcflzjn.fsf@aol.com

The patch[1] has been closed for some time, is there any other pending patches blocking going forward?

[1] https://github.com/slime/slime/issues/583

zimoun wrote 2 years ago

Recipients:(address . glv@posteo.net)

Message-ID:87k030pg1u.fsf@gmail.com

Hi,

Thank you for the follow up.

On Fri, 25 Nov 2022 at 12:22, Hendursaga via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:

Toggle quote (4 lines)

> The patch[1] has been closed for some time, is there any other pending patches blocking going forward?

> [1] https://github.com/slime/slime/issues/583

Indeed, but the package sbcl is not yet reproducible. The remaining

issue is about PDF.

Using Guix 28d1e51, I get:

Toggle snippet (4 lines)

$ diff -r --no-dereference /gnu/store/0q9z6j5ml4rkfrg57axznasdjfm7x803-sbcl-2.2.11-doc{,-check}

Binary files /gnu/store/0q9z6j5ml4rkfrg57axznasdjfm7x803-sbcl-2.2.11-doc/share/doc/sbcl/sbcl.pdf and /gnu/store/0q9z6j5ml4rkfrg57axznasdjfm7x803-sbcl-2.2.11-doc-check/share/doc/sbcl/sbcl.pdf differ

Cheers,

simon

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 44112@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 44112

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags

mumi command -t +confirmed -t +easy

Or, remove the moreinfo tag and set the help tag

mumi command -t -moreinfo -t +help

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date