[PATCH python-team] gnu: python: Make the build reproducible.

  • Done
  • quality assurance status badge
Details
3 participants
  • Lars-Dominik Braun
  • Maxim Cournoyer
  • Tomas Volf
Owner
unassigned
Submitted by
Tomas Volf
Severity
normal

Debbugs page

Tomas Volf wrote 1 years ago
(address . guix-patches@gnu.org)(name . Tomas Volf)(address . ~@wolfsden.cz)
f0dcb5eadb333cf840f515ddda025734d924c25a.1705019596.git.~@wolfsden.cz
While python build was reproducible on a single machine, once multiple
file systems entered the picture, it was no longer true. My local builds on
BTRFS differed from build on ext4 done in a virtual machine.

The distutils library present in current python is sensitive to file system
ordering. The solution is the same opensuse used, sorting the list of files.

With this patch, build on my machine (BTRFS) and in a guix system vm (ext4)
produce the same store item.


* gnu/packages/python.scm (python-3.10)[arguments]<#:phases>: Add
'patch-distutils phase.

Change-Id: I0273dc0f8511a7acdcc2b462a26cc29a9756c801
---
gnu/packages/python.scm | 13 +++++++++++++
1 file changed, 13 insertions(+)

Toggle diff (28 lines)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 51d5f598d7..319a917b4b 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -515,6 +515,19 @@ (define-public python-3.10
(substitute* "Makefile.pre.in"
(("-j0") "-j1")))))
'())
+ (add-after 'unpack 'patch-distutils
+ (lambda _
+ ;; Ensure byte_compile produces the same output regardless
+ ;; filesystem ordering. For more information see:
+ ;; https://bugzilla.opensuse.org/show_bug.cgi?id=1049186
+ (let* ((file "Lib/distutils/util.py")
+ (old-content (call-with-input-file file get-string-all)))
+ (substitute* file
+ (("^ for file in py_files:\n$")
+ " for file in sorted(py_files):\n"))
+ (if (string=? old-content
+ (call-with-input-file file get-string-all))
+ (error "substitute did nothing, phase requires an update")))))
(add-after 'unpack 'remove-windows-binaries
(lambda _
;; Delete .exe from embedded .whl (zip) files

base-commit: 5c0f77f4241c9beac0c82deae946bfdc70b49ff0
--
2.41.0
Lars-Dominik Braun wrote 1 years ago
(name . Tomas Volf)(address . ~@wolfsden.cz)
ZaDsqL1b9VfIFJI9@noor.fritz.box
Hi,

Toggle quote (4 lines)
> + (substitute* file
> + (("^ for file in py_files:\n$")
> + " for file in sorted(py_files):\n"))

as far as I understand this change was rejected from upstream Python,
the accepted (and merged) solution:
That patch is also used by Debian.

Cheers,
Lars
Tomas Volf wrote 1 years ago
[PATCH python-team v2] gnu: python: Make the build reproducible.
(address . 68394@debbugs.gnu.org)(name . Tomas Volf)(address . ~@wolfsden.cz)
3171d7e17ba481e977e45fdf35fb0ef42250abb4.1705075889.git.~@wolfsden.cz
While python build was reproducible on a single machine, once multiple
file systems entered the picture, it was no longer true. My local builds on
BTRFS differed from build on ext4 done in a virtual machine.

The solution adopted by the upstream (and debian) was cherry-picked. With
this patch, build on my machine (BTRFS) and in a guix system vm (ext4) produce
the same store item.


* gnu/packages/python.scm (python-3.10)[source]: Apply reproducibility patch.

Change-Id: I0273dc0f8511a7acdcc2b462a26cc29a9756c801
---
Use patch directly from the upstream.

gnu/packages/python.scm | 21 +++++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)

Toggle diff (36 lines)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 51d5f598d7..c92ac720a5 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -428,12 +428,21 @@ (define-public python-3.10
(method url-fetch)
(uri (string-append "https://www.python.org/ftp/python/"
version "/Python-" version ".tar.xz"))
- (patches (search-patches
- "python-3-arm-alignment.patch"
- "python-3-deterministic-build-info.patch"
- "python-3-fix-tests.patch"
- "python-3-hurd-configure.patch"
- "python-3-search-paths.patch"))
+ (patches
+ (cons*
+ ;; https://github.com/python/cpython/pull/8226
+ (origin
+ (method url-fetch)
+ (uri "https://github.com/python/cpython/commit/6c8ea7c1dacd42f3ba00440231ec0e6b1a38300d.patch")
+ (sha256
+ (base32
+ "13llngsyskp4c9j8lwqqpwp7h07mxai734zk1i387z8g261jk46v")))
+ (search-patches
+ "python-3-arm-alignment.patch"
+ "python-3-deterministic-build-info.patch"
+ "python-3-fix-tests.patch"
+ "python-3-hurd-configure.patch"
+ "python-3-search-paths.patch")))
(sha256
(base32
"0j6wvh2ad5jjq5n7sjmj1k66mh6lipabavchc3rb4vsinwaq9vbf"))

base-commit: 5c0f77f4241c9beac0c82deae946bfdc70b49ff0
--
2.41.0
Maxim Cournoyer wrote 1 years ago
Re: bug#68394: [PATCH python-team] gnu: python: Make the build reproducible.
(name . Tomas Volf)(address . ~@wolfsden.cz)
87o7dc16m9.fsf_-_@gmail.com
Hi Tomas,

Tomas Volf <~@wolfsden.cz> writes:

Toggle quote (14 lines)
> While python build was reproducible on a single machine, once multiple
> file systems entered the picture, it was no longer true. My local builds on
> BTRFS differed from build on ext4 done in a virtual machine.
>
> The solution adopted by the upstream (and debian) was cherry-picked. With
> this patch, build on my machine (BTRFS) and in a guix system vm (ext4) produce
> the same store item.
>
> More info: https://github.com/python/cpython/pull/8226
>
> * gnu/packages/python.scm (python-3.10)[source]: Apply reproducibility patch.
>
> Change-Id: I0273dc0f8511a7acdcc2b462a26cc29a9756c801

I've pushed a variant of this on to core-updates in commit e84519a949
("gnu: python: Make the build reproducible."), thank you!

Toggle quote (30 lines)
> ---
> Use patch directly from the upstream.
>
> gnu/packages/python.scm | 21 +++++++++++++++------
> 1 file changed, 15 insertions(+), 6 deletions(-)
>
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index 51d5f598d7..c92ac720a5 100644
> --- a/gnu/packages/python.scm
> +++ b/gnu/packages/python.scm
> @@ -428,12 +428,21 @@ (define-public python-3.10
> (method url-fetch)
> (uri (string-append "https://www.python.org/ftp/python/"
> version "/Python-" version ".tar.xz"))
> - (patches (search-patches
> - "python-3-arm-alignment.patch"
> - "python-3-deterministic-build-info.patch"
> - "python-3-fix-tests.patch"
> - "python-3-hurd-configure.patch"
> - "python-3-search-paths.patch"))
> + (patches
> + (cons*
> + ;; https://github.com/python/cpython/pull/8226
> + (origin
> + (method url-fetch)
> + (uri "https://github.com/python/cpython/commit/6c8ea7c1dacd42f3ba00440231ec0e6b1a38300d.patch")
> + (sha256
> + (base32
> + "13llngsyskp4c9j8lwqqpwp7h07mxai734zk1i387z8g261jk46v")))

I've opted to keep the patch local, which has been discussed as
preferred in the past (I think for reliability -- they don't disappear).

--
Thanks,
Maxim
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 68394@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 68394
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch
You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags
mumi command -t +confirmed -t +easy
Or, remove the moreinfo tag and set the help tag
mumi command -t -moreinfo -t +help