git-annex is not Reproducible

DoneSubmitted by zimoun.
Details
6 participants
  • Efraim Flashner
  • Kyle Meyer
  • Ludovic Courtès
  • Marius Bakke
  • Ricardo Wurmus
  • zimoun
Owner
unassigned
Severity
normal
Z
Z
zimoun wrote on 7 Oct 2020 12:35
(address . bug-guix@gnu.org)
87zh4y8gs8.fsf@gmail.com
Dear,

Using Guix 04a459a, the package ’git-annex’ is not reproducible:

guix build git-annex
guix build git-annex --no-grafts --check -K

return:

Toggle snippet (3 lines)
guix build: error: derivation `/gnu/store/70qkv5hi5yq20m1h9jfm17m5lsc235mk-git-annex-8.20200908.drv' may not be deterministic: output `/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908' differs from ‘/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check’

then:

Toggle snippet (4 lines)
$ diff -r --no-dereference /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908{,-check}
Binary files /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908/bin/git-annex and /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check/bin/git-annex differ

Is it expected? I have not tried ’diffoscope’ yet to investigate.


Note that:


speaks about GCC patch… I do not know if it is relevant.


All the best,
simon
K
K
Kyle Meyer wrote on 8 Oct 2020 05:04
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 43843@debbugs.gnu.org)
87blhdv2oi.fsf@kyleam.com
On 10/07/20 12:35:51 +0200, zimoun wrote:
Toggle quote (22 lines)
> Dear,
>
> Using Guix 04a459a, the package ’git-annex’ is not reproducible:
>
> guix build git-annex
> guix build git-annex --no-grafts --check -K
>
> return:
>
> --8<---------------cut here---------------start------------->8---
> guix build: error: derivation `/gnu/store/70qkv5hi5yq20m1h9jfm17m5lsc235mk-git-annex-8.20200908.drv' may not be deterministic: output `/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908' differs from ‘/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check’
> --8<---------------cut here---------------end--------------->8---
>
> then:
>
> --8<---------------cut here---------------start------------->8---
> $ diff -r --no-dereference /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908{,-check}
> Binary files /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908/bin/git-annex and /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check/bin/git-annex differ
> --8<---------------cut here---------------end--------------->8---
>
> Is it expected? I have not tried ’diffoscope’ yet to investigate.

Thanks for noticing. I see the same on my end (and also with the
git-annex update [1] that I submitted before I saw this message).

I suspect this is a relatively new issue, as I've run `build --check'
successfully on git-annex in the past. The git-annex definition prior
to the one above was introduced with e13fefbcff (gnu: git-annex: Create
fake static output, 2020-08-17). On that commit, the check also failed
for me:

$ guix time-machine --commit=e13fefbcffa50144d9301d67b61928ac3c964fb3 -- build --no-grafts --check git-annex
guix build: error: derivation `/gnu/store/5illriaaijqrhl17aymzgiw6wzaq666s-git-annex-8.20200810.drv' may not be deterministic: output `/gnu/store/32h0xbqdwkm22naqba10fk9124cdgw9q-git-annex-8.20200810' differs

However, going back one more to 35e24fab73 (gnu: git-annex: Update to
8.20200810, 2020-08-11), the check was successful:

$ guix time-machine --commit=35e24fab73bc450c6e0d3b9a5bbdfa88d2c425f4 -- build --no-grafts --check git-annex
[...]
/gnu/store/m79ri0g2s3d8pk52ysjkdvallk2zqbwb-git-annex-8.20200810


[1]: bug#43854: <20201007204946.22878-1-kyle@kyleam.com>
E
E
Efraim Flashner wrote on 8 Oct 2020 08:17
(name . Kyle Meyer)(address . kyle@kyleam.com)
20201008061713.GM21174@E5400
On Wed, Oct 07, 2020 at 11:04:13PM -0400, Kyle Meyer wrote:
Toggle quote (46 lines)
> On 10/07/20 12:35:51 +0200, zimoun wrote:
> > Dear,
> >
> > Using Guix 04a459a, the package ’git-annex’ is not reproducible:
> >
> > guix build git-annex
> > guix build git-annex --no-grafts --check -K
> >
> > return:
> >
> > --8<---------------cut here---------------start------------->8---
> > guix build: error: derivation `/gnu/store/70qkv5hi5yq20m1h9jfm17m5lsc235mk-git-annex-8.20200908.drv' may not be deterministic: output `/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908' differs from ‘/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check’
> > --8<---------------cut here---------------end--------------->8---
> >
> > then:
> >
> > --8<---------------cut here---------------start------------->8---
> > $ diff -r --no-dereference /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908{,-check}
> > Binary files /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908/bin/git-annex and /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check/bin/git-annex differ
> > --8<---------------cut here---------------end--------------->8---
> >
> > Is it expected? I have not tried ’diffoscope’ yet to investigate.
>
> Thanks for noticing. I see the same on my end (and also with the
> git-annex update [1] that I submitted before I saw this message).
>
> I suspect this is a relatively new issue, as I've run `build --check'
> successfully on git-annex in the past. The git-annex definition prior
> to the one above was introduced with e13fefbcff (gnu: git-annex: Create
> fake static output, 2020-08-17). On that commit, the check also failed
> for me:
>
> $ guix time-machine --commit=e13fefbcffa50144d9301d67b61928ac3c964fb3 -- build --no-grafts --check git-annex
> guix build: error: derivation `/gnu/store/5illriaaijqrhl17aymzgiw6wzaq666s-git-annex-8.20200810.drv' may not be deterministic: output `/gnu/store/32h0xbqdwkm22naqba10fk9124cdgw9q-git-annex-8.20200810' differs
>
> However, going back one more to 35e24fab73 (gnu: git-annex: Update to
> 8.20200810, 2020-08-11), the check was successful:
>
> $ guix time-machine --commit=35e24fab73bc450c6e0d3b9a5bbdfa88d2c425f4 -- build --no-grafts --check git-annex
> [...]
> /gnu/store/m79ri0g2s3d8pk52ysjkdvallk2zqbwb-git-annex-8.20200810
>
>
> [1]: bug#43854: <20201007204946.22878-1-kyle@kyleam.com>
>

In between those two commits is also the change to the haskell build
system update, where we use the dynamic libraries. The best bet is to
make sure the dependencies are built reproducibly.

--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAl9+ruYACgkQQarn3Mo9
g1Hesw/6Ar7G6iG7jjFxd27yp2pNl9s0uM81jTslECd0Q3EA1ZkE61ny4HqwYBIm
ZYq/3V9fAOjC9KCbEN6vevcWAEIY+XGv2tNt6KsoDLTCqJgLydLX/TaczvYLHdT5
h32tGuHF4Te0UrNKtsbJf3apzMOqr2RGoXsSh0JlL/rjBEEBMWAv8eB+9E3vWreY
NIKIblcLqh6zTCjYglcwsDmpNYB8+o7k4ok+4K+5laZsvhGevU/nguVFvM9y0v28
6lm+sibTyKVv3eM8o/Z9ZjT/XgRJfCq1h0QpFneBP12tC9MkEOQGtTd5RJAoY1U2
hML/x9ycNnQKNwF7K778Ln0Uan5C5kFgYj+U77TR0FZeReRJDvFreN3lyZpj0ffv
wxSiP5EaDzfKSqx2i7AETANY3YYUdX6Abg6H3ve6hueujEo7Lt8IPL1P4pP4Kbxk
Y3j69bCK/1EPN39qthBAVZW1uB/lbaC3edGz37UCc5dE4oUykcVSXfyakYC7Turu
IMmJjMQmmyVq63oHmd0bpQBIuFlbYAXJMxJ3ccvVz1RLQdJQoHx9SzwpMDJRhaj8
KPUDt9+avAXZD6Dqmax/6O0tlg7/WRxafAQ3XbBPUsVHDfdh2ZXaaurCGZ0Le6aD
x8Joc/Y0fIAt8B5WwbHogyskusqCditrHo4hiPNjqfAn5IOFfBM=
=01H2
-----END PGP SIGNATURE-----


Z
Z
zimoun wrote on 9 Oct 2020 22:33
(name . Efraim Flashner)(address . efraim@flashner.co.il)
86imbjqguw.fsf@gmail.com
Dear,

On Thu, 08 Oct 2020 at 09:17, Efraim Flashner <efraim@flashner.co.il> wrote:

Toggle quote (19 lines)
>> $ guix time-machine --commit=e13fefbcffa50144d9301d67b61928ac3c964fb3 --
>> build --no-grafts --check git-annex
>> guix build: error: derivation
>> `/gnu/store/5illriaaijqrhl17aymzgiw6wzaq666s-git-annex-8.20200810.drv' may
>> not be deterministic: output
>> `/gnu/store/32h0xbqdwkm22naqba10fk9124cdgw9q-git-annex-8.20200810' differs
>>
>> However, going back one more to 35e24fab73 (gnu: git-annex: Update to
>> 8.20200810, 2020-08-11), the check was successful:
>>
>> $ guix time-machine --commit=35e24fab73bc450c6e0d3b9a5bbdfa88d2c425f4 --
>> build --no-grafts --check git-annex
>> [...]
>> /gnu/store/m79ri0g2s3d8pk52ysjkdvallk2zqbwb-git-annex-8.20200810

> In between those two commits is also the change to the haskell build
> system update, where we use the dynamic libraries. The best bet is to
> make sure the dependencies are built reproducibly.

Yep, it is what “git bisect” says.

Toggle snippet (20 lines)
67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 is the first bad commit
commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2
Author: Ricardo Wurmus <rekado@elephly.net>
Date: Tue Jun 16 22:25:48 2020 +0200

build-system/haskell: Support parallel builds.

* guix/build-system/haskell.scm (haskell-build): Add keyword
PARALLEL-BUILD?
and pass it on to the builder.
* guix/build/haskell-build-system.scm (build): Accept keyword
PARALLEL-BUILD?
and pass the number of parallel jobs to GHC.

guix/build-system/haskell.scm | 2 ++
guix/build/haskell-build-system.scm | 7 +++++--
2 files changed, 7 insertions(+), 2 deletions(-)
bisect run success

I did:

git bisect start e13fefbc 35e24fab
git bisest run ./check.sh

where check.sh reads:

Toggle snippet (25 lines)
#!/bin/sh

PKG=git-annex

git log -1 --format='%h %s [%an]'
COMMIT=$(git log -1 --format='%H')
echo "Start: $COMMIT"

guix time-machine \
--commit=$COMMIT \
-- build $PKG \
--no-grafts

guix time-machine \
--commit=$COMMIT \
-- build $PKG \
--no-grafts --check -K
status=$?

echo "End: [$status] $COMMIT"

exit $status


And I do not know if it is related, but these packages using the
’haskell-build-system’ are not reproducible neither (304/~640):

agda
cabal-install
cpphs
darcs
dhall
elm-compiler
ghc-abstract-deque
ghc-abstract-par
ghc-adjunctions
ghc-aeson
ghc-alex
ghc-asn1-encoding
ghc-asn1-types
ghc-assoc
ghc-atomic-primops
ghc-attoparsec
ghc-auto-update
ghc-aws
ghc-base-compat
ghc-basement
ghc-base-unicode-symbols
ghc-bifunctors
ghc-bitarray
ghc-blaze-html
ghc-blaze-markup
ghc-bloomfilter
ghc-bytes
ghc-bytestring-handle
ghc-c2hs
ghc-cairo
ghc-cborg
ghc-cereal
ghc-cgi
ghc-charset
ghc-chart
ghc-chasingbottoms
ghc-cheapskate
ghc-chunked-data
ghc-clientsession
ghc-cmdargs
ghc-colour
ghc-comonad
ghc-concurrent-extra
ghc-conduit
ghc-conduit-algorithms
ghc-conduit-extra
ghc-configurator
ghc-constraints
ghc-convertible
ghc-crypto-api
ghc-crypto-api-tests
ghc-crypto-cipher-tests
ghc-crypto-cipher-types
ghc-cryptohash
ghc-cryptonite
ghc-crypto-random
ghc-curl
ghc-data-accessor
ghc-dbus
ghc-descriptive
ghc-diagrams-core
ghc-diagrams-lib
ghc-diagrams-solve
ghc-diff
ghc-digest
ghc-doctest
ghc-edisonapi
ghc-edisoncore
ghc-edit-distance
ghc-either
ghc-errors
ghc-esqueleto
ghc-exactprint
ghc-extra
ghc-fast-logger
ghc-feed
ghc-fgl
ghc-filemanip
ghc-filepath-bytestring
ghc-fingertree
ghc-foldl
ghc-foundation
ghc-free
ghc-fsnotify
ghc-generic-deriving
ghc-generics-sop
ghc-genvalidity-property
ghc-getopt-generics
ghc-gluraw
ghc-glut
ghc-gnuplot
ghc-graphviz
ghc-gtk2hs-buildtools
ghc-hackage-security
ghc-haddock-api
ghc-haddock-library
ghc-happstack-server
ghc-happy
ghc-hashtables
ghc-haskeline
ghc-haskell-lexer
ghc-haskell-src
ghc-haskell-src-exts
ghc-haskell-src-exts-util
ghc-hedgehog
ghc-highlighting-kate
ghc-hindent
ghc-hmatrix
ghc-hmatrix-gsl
ghc-hmatrix-gsl-stats
ghc-hmatrix-special
ghc-hourglass
ghc-hpack
ghc-hslogger
ghc-hslua
ghc-hsopenssl
ghc-hspec
ghc-hspec-contrib
ghc-hspec-core
ghc-hspec-meta
ghc-hsyaml
ghc-http
ghc-http2
ghc-http-client
ghc-http-date
ghc-http-types
ghc-hunit
ghc-hxt
ghc-hxt-regex-xmlschema
ghc-hxt-unicode
ghcid
ghc-inline-c
ghc-inspection-testing
ghc-integer-logarithms
ghc-interpolate
ghc-intervalmap
ghc-intervals
ghc-io-streams
ghc-jira-wiki-markup
ghc-json
ghc-juicypixels
ghc-kan-extensions
ghc-language-c
ghc-language-glsl
ghc-language-javascript
ghc-lens
ghc-lens-family-core
ghc-libmpd
ghc-lib-parser
ghc-libxml
ghc-libyaml
ghc-lifted-base
ghc-linear
ghc-listlike
ghc-llvm-hs
ghc-llvm-hs-pure
ghc-logging-facade
ghc-math-functions
ghc-megaparsec
ghc-memory
ghc-missingh
ghc-monad-par
ghc-monad-par-extras
ghc-monads-tf
ghc-monoid-extras
ghc-mono-traversable
ghc-mwc-random
ghc-ncurses
ghc-network
ghc-non-negative
ghc-opengl
ghc-openglraw
ghc-optparse-applicative
ghc-pandoc
ghc-pandoc-citeproc
ghc-pandoc-types
ghc-parallel
ghc-parser-combinators
ghc-parsers
ghc-patience
ghc-pem
ghc-persistent
ghc-persistent-test
ghc-pgp-wordlist
ghc-polyparse
ghc-pqueue
ghc-prettyprinter
ghc-prettyprinter
ghc-pretty-show
ghc-pretty-simple
ghc-primitive
ghc-profunctors
ghc-protolude
ghc-psqueues
ghc-quickcheck
ghc-quickcheck-instances
ghc-rebase
ghc-reducers
ghc-regex
ghc-regex-applicative
ghc-regex-pcre
ghc-regex-pcre-builtin
ghc-regex-posix
ghc-regex-tdfa
ghc-regex-tdfa-text
ghc-rerebase
ghc-rio
ghc-safe
ghc-safesemaphore
ghc-sandi
ghc-scientific
ghc-sdl
ghc-sdl2
ghc-sdl2-image
ghc-sdl-mixer
ghc-semialign
ghc-semigroupoids
ghc-sendfile
ghc-serialise
ghc-shakespeare
ghc-shelly
ghc-size-based
ghc-skylighting-core
ghc-smallcheck
ghc-snap-core
ghc-snap-server
ghc-socks
ghc-sop-core
ghc-sourcemap
ghc-splitmix
ghc-statistics
ghc-stm-chans
ghc-stm-conduit
ghc-storable-record
ghc-storablevector
ghc-streaming-commons
ghc-strict
ghc-stringsearch
ghc-stylish-haskell
ghc-syb
ghc-tagged
ghc-tagsoup
ghc-tar
ghc-tasty
ghc-test-framework
ghc-testing-feat
ghc-texmath
ghc-tf-random
ghc-these
ghc-time-compat
ghc-timezone-olson
ghc-tls
ghc-transformers
ghc-transformers-compat
ghc-tree-diff
ghc-trifecta
ghc-turtle
ghc-unagi-chan
ghc-unicode-transforms
ghc-union-find
ghc-uniplate
ghc-unix-time
ghc-unliftio
ghc-unliftio-core
ghc-unordered-containers
ghc-uri-bytestring
ghc-utf8-string
ghc-utility-ht
ghc-uuid
ghc-vault
ghc-vector
ghc-vector-algorithms
ghc-wai-extra
ghc-warp
ghc-websockets
ghc-x11
ghc-x509
ghc-x509-store
ghc-x509-system
ghc-x509-validation
ghc-xml
ghc-xml-conduit
ghc-xmonad-contrib
ghc-yaml
ghc-yesod
ghc-yesod-core
ghc-yesod-form
ghc-zstd
git-annex
hlint
hoogle
hscolour
hspec-discover
idris
kmonad
ngless
pandoc
pandoc-citeproc
purescript
raincat
scroll
shellcheck
stylish-haskell
xmobar


All the best,
simon
R
R
Ricardo Wurmus wrote on 10 Oct 2020 11:35
(name . zimoun)(address . zimon.toutoune@gmail.com)
87a6wuzamh.fsf@elephly.net
zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (46 lines)
> Dear,
>
> On Thu, 08 Oct 2020 at 09:17, Efraim Flashner <efraim@flashner.co.il> wrote:
>
>>> $ guix time-machine --commit=e13fefbcffa50144d9301d67b61928ac3c964fb3 --
>>> build --no-grafts --check git-annex
>>> guix build: error: derivation
>>> `/gnu/store/5illriaaijqrhl17aymzgiw6wzaq666s-git-annex-8.20200810.drv' may
>>> not be deterministic: output
>>> `/gnu/store/32h0xbqdwkm22naqba10fk9124cdgw9q-git-annex-8.20200810' differs
>>>
>>> However, going back one more to 35e24fab73 (gnu: git-annex: Update to
>>> 8.20200810, 2020-08-11), the check was successful:
>>>
>>> $ guix time-machine --commit=35e24fab73bc450c6e0d3b9a5bbdfa88d2c425f4 --
>>> build --no-grafts --check git-annex
>>> [...]
>>> /gnu/store/m79ri0g2s3d8pk52ysjkdvallk2zqbwb-git-annex-8.20200810
>
>> In between those two commits is also the change to the haskell build
>> system update, where we use the dynamic libraries. The best bet is to
>> make sure the dependencies are built reproducibly.
>
> Yep, it is what “git bisect” says.
>
> --8<---------------cut here---------------start------------->8---
> 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 is the first bad commit
> commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2
> Author: Ricardo Wurmus <rekado@elephly.net>
> Date: Tue Jun 16 22:25:48 2020 +0200
>
> build-system/haskell: Support parallel builds.
>
> * guix/build-system/haskell.scm (haskell-build): Add keyword
> PARALLEL-BUILD?
> and pass it on to the builder.
> * guix/build/haskell-build-system.scm (build): Accept keyword
> PARALLEL-BUILD?
> and pass the number of parallel jobs to GHC.
>
> guix/build-system/haskell.scm | 2 ++
> guix/build/haskell-build-system.scm | 7 +++++--
> 2 files changed, 7 insertions(+), 2 deletions(-)
> bisect run success
> --8<---------------cut here---------------end--------------->8---

Shall we revert it? Previously the build system also supported parallel
builds, but didn’t seem to pass the right arguments to actually make it
happen.

--
Ricardo
Z
Z
zimoun wrote on 12 Oct 2020 11:31
(name . Ricardo Wurmus)(address . rekado@elephly.net)
CAJ3okZ2ND1sfC5yjhtmySb9WvrM9JDdonQnn56gTXNirLDKZgw@mail.gmail.com
On Sat, 10 Oct 2020 at 11:34, Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (25 lines)
> > --8<---------------cut here---------------start------------->8---
> > 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 is the first bad commit
> > commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2
> > Author: Ricardo Wurmus <rekado@elephly.net>
> > Date: Tue Jun 16 22:25:48 2020 +0200
> >
> > build-system/haskell: Support parallel builds.
> >
> > * guix/build-system/haskell.scm (haskell-build): Add keyword
> > PARALLEL-BUILD?
> > and pass it on to the builder.
> > * guix/build/haskell-build-system.scm (build): Accept keyword
> > PARALLEL-BUILD?
> > and pass the number of parallel jobs to GHC.
> >
> > guix/build-system/haskell.scm | 2 ++
> > guix/build/haskell-build-system.scm | 7 +++++--
> > 2 files changed, 7 insertions(+), 2 deletions(-)
> > bisect run success
> > --8<---------------cut here---------------end--------------->8---
>
> Shall we revert it? Previously the build system also supported parallel
> builds, but didn’t seem to pass the right arguments to actually make it
> happen.

What do the Haskellers think about that?
Does the parallel build save a lot of time and CPU? If yes, maybe we
could to provide a transformation for the expert, something like
"haskell-build-system-with-parellel-build" which tweaks
"PARALLEL-BUILD?", similarly to the recent "no tests". WDYT?


All the best,
simon
R
R
Ricardo Wurmus wrote on 12 Oct 2020 12:36
(name . zimoun)(address . zimon.toutoune@gmail.com)
87v9ffybn3.fsf@elephly.net
zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (29 lines)
> On Sat, 10 Oct 2020 at 11:34, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>> > --8<---------------cut here---------------start------------->8---
>> > 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 is the first bad commit
>> > commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2
>> > Author: Ricardo Wurmus <rekado@elephly.net>
>> > Date: Tue Jun 16 22:25:48 2020 +0200
>> >
>> > build-system/haskell: Support parallel builds.
>> >
>> > * guix/build-system/haskell.scm (haskell-build): Add keyword
>> > PARALLEL-BUILD?
>> > and pass it on to the builder.
>> > * guix/build/haskell-build-system.scm (build): Accept keyword
>> > PARALLEL-BUILD?
>> > and pass the number of parallel jobs to GHC.
>> >
>> > guix/build-system/haskell.scm | 2 ++
>> > guix/build/haskell-build-system.scm | 7 +++++--
>> > 2 files changed, 7 insertions(+), 2 deletions(-)
>> > bisect run success
>> > --8<---------------cut here---------------end--------------->8---
>>
>> Shall we revert it? Previously the build system also supported parallel
>> builds, but didn’t seem to pass the right arguments to actually make it
>> happen.
>
> What do the Haskellers think about that?

Let me answer as an ex-Haskeller :)

Toggle quote (5 lines)
> Does the parallel build save a lot of time and CPU? If yes, maybe we
> could to provide a transformation for the expert, something like
> "haskell-build-system-with-parellel-build" which tweaks
> "PARALLEL-BUILD?", similarly to the recent "no tests". WDYT?

We shouldn’t compromise reproducibility for parallel builds. Ideally we
would figure out what exactly causes the differences and fix that
instead of disabling parallel builds, but if that turns out to be too
difficult I think we should just revert this until we have a good fix.

Perhaps something can be done by fixing the order of files somewhere.

--
Ricardo
Z
Z
zimoun wrote on 12 Oct 2020 13:55
(name . Ricardo Wurmus)(address . rekado@elephly.net)
CAJ3okZ2KwAFTKGpWMO+rophDg1bAsa_dYNaEYcmMQa_ATcNmUw@mail.gmail.com
On Mon, 12 Oct 2020 at 12:34, Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (12 lines)
> > Does the parallel build save a lot of time and CPU? If yes, maybe we
> > could to provide a transformation for the expert, something like
> > "haskell-build-system-with-parellel-build" which tweaks
> > "PARALLEL-BUILD?", similarly to the recent "no tests". WDYT?
>
> We shouldn’t compromise reproducibility for parallel builds. Ideally we
> would figure out what exactly causes the differences and fix that
> instead of disabling parallel builds, but if that turns out to be too
> difficult I think we should just revert this until we have a good fix.
>
> Perhaps something can be done by fixing the order of files somewhere.

My proposal is for the expert and not at the CLI level. Something
like "(funky-name ghc-foo)" which returns a new package with
PARALLEL-BUILD? turned to #t. And the default should be #f. WDYT?


Well, I am going to try to rebuild the packages with #f and see if it
fixes the ~300 unreproducible packages.


Cheers,
simon
L
L
Ludovic Courtès wrote on 13 Oct 2020 15:05
(name . zimoun)(address . zimon.toutoune@gmail.com)
87tuuyjmx4.fsf@gnu.org
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (18 lines)
> On Mon, 12 Oct 2020 at 12:34, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>> > Does the parallel build save a lot of time and CPU? If yes, maybe we
>> > could to provide a transformation for the expert, something like
>> > "haskell-build-system-with-parellel-build" which tweaks
>> > "PARALLEL-BUILD?", similarly to the recent "no tests". WDYT?
>>
>> We shouldn’t compromise reproducibility for parallel builds. Ideally we
>> would figure out what exactly causes the differences and fix that
>> instead of disabling parallel builds, but if that turns out to be too
>> difficult I think we should just revert this until we have a good fix.
>>
>> Perhaps something can be done by fixing the order of files somewhere.
>
> My proposal is for the expert and not at the CLI level. Something
> like "(funky-name ghc-foo)" which returns a new package with
> PARALLEL-BUILD? turned to #t. And the default should be #f. WDYT?

You can always build with ‘--cores=1’, which will ensure both the build
and test will run sequentially.

Ludo’.
Z
Z
zimoun wrote on 13 Oct 2020 15:28
(name . Ludovic Courtès)(address . ludo@gnu.org)
CAJ3okZ2tWRY-8LTMQBOrNe7DZ61TN8W1du_Y4-pAN5s8-ZgKSw@mail.gmail.com
On Tue, 13 Oct 2020 at 15:06, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (3 lines)
> You can always build with ‘--cores=1’, which will ensure both the build
> and test will run sequentially.

Yes but in this case, "guix challenge" would still complain, isn't it?
M
M
Marius Bakke wrote on 19 Oct 2020 00:11
Re: [PATCH] build-system/haskell: Disable parallel builds.
87sgabjib1.fsf@gnu.org
zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (9 lines)
>
> PARALLEL-BUILD? introduced by commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2
> leads to unreproducibility. Instead of reverting, default is set to #false
> which allows user to locally build with parallelism.
>
> * guix/build-system/haskell.scm (haskell-build): Turn off PARALLEL-BUILD? by
> default.

Thanks! I shortened the comment a little and pushed to staging in
19d42e0e23a7f90ac2dcc1c279bd23a967ff0314.
-----BEGIN PGP SIGNATURE-----

iQFDBAEBCgAtFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAl+MvZIPHG1hcml1c0Bn
bnUub3JnAAoJEKKgbfKjOlT6YesH/0wf7s/gxMXl8ZpGljFeECdLFOFlwGm7g44+
oMzgdM2/NMcgLlZ0FEhng7/3ViyKZ3GmCaLlP7iAEFRCkfA+7e0aic1VZgFqdojk
z4hXpfoKK0CnpI+cXT5mVvzP8iBZDjvgX2liHCPFeNICao3akkOclmdSN1/jPfDn
5UtpOOm2tYKn0McCoyhaKcBBdm/XlWIjyB5CyQSITQddi3VtAA8Lx7ZJa5cjz5C4
TnHCq7MAeTHcbRAH8Wfwi9XTQoL73MptAN8GOvHuTGMLCLFMUXAV/qwUHS3zl5r9
beu27yPjp0CjSmk9I6j5axVYMTlA9V+TejsN//Ej01ZPZwowSqM=
=7cRX
-----END PGP SIGNATURE-----

Closed
?
Your comment

This issue is archived.

To comment on this conversation send email to 43843@debbugs.gnu.org