Fixing timestamps in archives.

OpenSubmitted by Tim Gesthuizen.
Details
3 participants
  • Julien Lepiller
  • Ludovic Courtès
  • Tim Gesthuizen
Owner
unassigned
Severity
normal
T
T
Tim Gesthuizen wrote on 27 Jan 2019 18:58
e13090c1-dabb-da63-cc62-3975f2697527@yahoo.de
Hi Ludo,
as discussed before I have looked into the problems of timestamps in thezip files.I looked at the way this is solved in ant-build-system with jar filesand thought that this could be done in a more elegant way.Because of this I wrote a simple frontend for LibArchive in C thatrepacks archives and sets their timestamps to zero and disablescompression as it is done in the ant-build-system.Creative as I am the program is called repack.You find a git repository attached with the history of the repack program.The attached patches add repack to Guix and use it for pwsafe and theant-build-system.
This is a work in progress version: If you like these changes I willwork on missing details so that we can add it, otherwise now would be agood point to stop development on these changes.I can still fall back to the variant that ant-build-system uses now forpwsafe.
It would be nice if you could find the time to review everything andtell me what you think about the patches.The changes trigger a lot of rebuilds so it will take some time to builddependencies of the required programs.
The repack tar contains a bare git repository with the program.Because I could not yet find a place to host the repository you need tounpack it somewhere, create an archive with the source code and changethe guix definition of repack to use that source.
Tim.
From fe500f0ba03aebc3e0020d974f1266c00001d64e Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Thu, 17 Jan 2019 20:37:39 +0100Subject: [PATCH 01/11] gnu: Add repack
* gnu/packages/compression.scm (repack): New variable.--- gnu/packages/compression.scm | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
Toggle diff (53 lines)diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scmindex 327951fde..837fca477 100644--- a/gnu/packages/compression.scm+++ b/gnu/packages/compression.scm@@ -23,6 +23,7 @@ ;;; Copyright © 2018 Rutger Helling <rhelling@mykolab.com> ;;; Copyright © 2018 Joshua Sierles, Nextjournal <joshua@nextjournal.com> ;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -44,6 +45,7 @@ #:use-module (guix utils) #:use-module (guix packages) #:use-module (guix download)+ #:use-module (guix gexp) #:use-module (guix git-download) #:use-module (guix build-system cmake) #:use-module (guix build-system gnu)@@ -60,6 +62,7 @@ #:use-module (gnu packages perl) #:use-module (gnu packages pkg-config) #:use-module (gnu packages python)+ #:use-module (gnu packages texinfo) #:use-module (gnu packages tls) #:use-module (gnu packages valgrind) #:use-module (ice-9 match)@@ -1422,6 +1425,22 @@ Compression ratios of 2:1 to 3:1 are common for text files.") (license (license:non-copyleft "file://LICENSE" "See LICENSE in the distribution.")))) +(define-public repack+ (package+ (name "repack")+ (version "0.1")+ (source (local-file "/home/tibbe/src/repack.tar.gz"))+ (build-system cmake-build-system)+ (arguments `(#:tests? #f))+ (native-inputs+ `(("texinfo" ,texinfo)))+ (inputs+ `(("libarchive" ,libarchive)))+ (home-page #f)+ (synopsis "Repack archives to not include time stamps or compression.")+ (description #f)+ (license license:gpl3+)))+ (define-public unzip (package (inherit zip) (name "unzip")-- 2.20.1
From 394b939ed84154c203c614c23e4f6bc2df17613c Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Sat, 19 Jan 2019 16:58:34 +0100Subject: [PATCH 02/11] guix: utils: Add repack-archives
Add a function that repacks all archives in a certain directory to bereproduceably buildable.
* guix/build/utils.scm (repack-archives): New function.--- guix/build/utils.scm | 11 +++++++++++ 1 file changed, 11 insertions(+)
Toggle diff (38 lines)diff --git a/guix/build/utils.scm b/guix/build/utils.scmindex 5fe328684..3ca72299a 100644--- a/guix/build/utils.scm+++ b/guix/build/utils.scm@@ -3,6 +3,7 @@ ;;; Copyright © 2013 Andreas Enge <andreas@enge.fr> ;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org> ;;; Copyright © 2015, 2018 Mark H Weaver <mhw@netris.org>+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -88,6 +89,7 @@ fold-port-matches remove-store-references wrap-program+ repack-archives invoke invoke-error?@@ -1100,6 +1102,15 @@ with definitions for VARS." (chmod prog-tmp #o755) (rename-file prog-tmp prog)))) +(define (repack-archives root regex)+ "Repack all archives under ROOT that match REGEX using the repack program."+ (define (repack-archive jar)+ (format #t "repacking ~a\n" jar)+ (let* ((tmp (tmpnam)))+ (copy-file jar tmp)+ (invoke "repack" "-o" jar tmp)))+ (for-each repack-archive (find-files root regex)))+ ;;; ;;; Locales.-- 2.20.1
From bd552a577fb7dbe1233b366c95d1741754fbea33 Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Thu, 17 Jan 2019 20:38:10 +0100Subject: [PATCH 03/11] gnu: pwsafe: Remove timestamps from zip files
* gnu/packages/password-utils.scm (pwsafe):[native-inputs]: Add repack.[arguments]: Add modules and a phase to strip timestamps from the zip files.--- gnu/packages/password-utils.scm | 38 +++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-)
Toggle diff (72 lines)diff --git a/gnu/packages/password-utils.scm b/gnu/packages/password-utils.scmindex 5b60ad9cc..47fb9dccb 100644--- a/gnu/packages/password-utils.scm+++ b/gnu/packages/password-utils.scm@@ -21,7 +21,7 @@ ;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net> ;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz> ;;; Copyright © 2018 Amirouche Boubekki <amirouche@hypermove.net>-;;; Copyright © 2018 Tim Gesthuizen <tim.gesthuizen@yahoo.de>+;;; Copyright © 2018, 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -190,7 +190,8 @@ algorithms AES or Twofish.") (build-system cmake-build-system) (native-inputs `(("gettext" ,gettext-minimal) ("perl" ,perl)- ("zip" ,zip)))+ ("zip" ,zip)+ ("repack" ,repack))) (inputs `(("curl" ,curl) ("file" ,file) ("gtest" ,googletest)@@ -201,7 +202,11 @@ algorithms AES or Twofish.") ("qrencode" ,qrencode) ("wxwidgets" ,wxwidgets) ("xerces-c" ,xerces-c)))- (arguments '(#:configure-flags (list "-DNO_GTEST=YES")+ (arguments `(#:configure-flags (list "-DNO_GTEST=YES")+ #:modules ((guix build cmake-build-system)+ (guix build utils)+ (ice-9 regex)+ (ice-9 ftw)) #:phases (modify-phases %standard-phases (add-after 'unpack 'add-gtest (lambda* (#:key inputs #:allow-other-keys)@@ -219,7 +224,32 @@ add_subdirectory(src/test)\n" cmake-port) (("/usr/bin/file") (string-append (assoc-ref inputs "file") "/bin/file")))- #t)))))+ #t))+ (add-after 'install 'fix-zip-timestamps+ (lambda* (#:key outputs inputs+ #:allow-other-keys)+ (let ((out (assoc-ref outputs "out"))+ (repack (string-append+ (assoc-ref inputs "repack")+ "/bin/repack")))+ (with-directory-excursion out+ (file-system-fold+ (const #t)+ (lambda (path stat result)+ (when (string-match "^.*\\.zip$"+ (basename path))+ (let ((tmp (tmpnam)))+ (format #t "Repacking ~s\n" path)+ (copy-file path tmp)+ (invoke repack+ "-o" path+ tmp))))+ (const #t)+ (const #t)+ (const #t)+ (lambda _ (throw 'file-system-error))+ #t "."))+ #t)))))) (synopsis "Password safe with automatic input and key generation") (description "pwsafe is a password manager originally designed by Bruce Schneier. It offers a simple UI to manage passwords for different services.-- 2.20.1
From ed9db022d4e2ea9b6f1151fbc2785014b2a1bd8e Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Thu, 17 Jan 2019 22:49:46 +0100Subject: [PATCH 04/11] guix: ant: Use repack
* guix/build-system/ant.scm (default-zip): Deleted.* guix/build-system/ant.scm (lower): Change zip to repack.* guix/build/ant-build-system.scm (strip-jar-timestamps): Use repack for repack-archive.--- guix/build-system/ant.scm | 16 ++++++++-------- guix/build/ant-build-system.scm | 32 ++++---------------------------- 2 files changed, 12 insertions(+), 36 deletions(-)
Toggle diff (100 lines)diff --git a/guix/build-system/ant.scm b/guix/build-system/ant.scmindex b5626bd42..a4aa90eb8 100644--- a/guix/build-system/ant.scm+++ b/guix/build-system/ant.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -55,22 +56,21 @@ (let ((jdk-mod (resolve-interface '(gnu packages java)))) (module-ref jdk-mod 'ant))) -(define (default-zip)- "Return the default ZIP package."- ;; Lazily resolve the binding to avoid a circular dependency.- (let ((zip-mod (resolve-interface '(gnu packages compression))))- (module-ref zip-mod 'zip)))+(define (default-repack)+ "Return the default REPACK package"+ (let ((repack-mod (resolve-interface '(gnu packages compression))))+ (module-ref repack-mod 'repack))) (define* (lower name #:key source inputs native-inputs outputs system target (jdk (default-jdk)) (ant (default-ant))- (zip (default-zip))+ (repack (default-repack)) #:allow-other-keys #:rest arguments) "Return a bag for NAME." (define private-keywords- '(#:source #:target #:jdk #:ant #:zip #:inputs #:native-inputs))+ '(#:source #:target #:jdk #:ant #:repack #:inputs #:native-inputs)) (and (not target) ;XXX: no cross-compilation (bag@@ -85,7 +85,7 @@ ,@(standard-packages))) (build-inputs `(("jdk" ,jdk "jdk") ("ant" ,ant)- ("zip" ,zip)+ ("repack" ,repack) ,@native-inputs)) (outputs outputs) (build ant-build)diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scmindex d79a2d55e..540c985b7 100644--- a/guix/build/ant-build-system.scm+++ b/guix/build/ant-build-system.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016, 2018 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -191,34 +192,9 @@ dependencies of this jar file." repack them. This is necessary to ensure that archives are reproducible." (define (repack-archive jar) (format #t "repacking ~a\n" jar)- (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))- (manifest (string-append dir "/META-INF/MANIFEST.MF")))- (with-directory-excursion dir- (invoke "jar" "xf" jar))- (delete-file jar)- ;; XXX: copied from (gnu build install)- (for-each (lambda (file)- (let ((s (lstat file)))- (unless (eq? (stat:type s) 'symlink)- (utime file 0 0 0 0))))- (find-files dir #:directories? #t))-- ;; The jar tool will always set the timestamp on the manifest file- ;; and the containing directory to the current time, even when we- ;; reuse an existing manifest file. To avoid this we use "zip"- ;; instead of "jar". It is important that the manifest appears- ;; first.- (with-directory-excursion dir- (let* ((files (find-files "." ".*" #:directories? #t))- ;; To ensure that the reference scanner can detect all- ;; store references in the jars we disable compression- ;; with the "-0" option.- (command (if (file-exists? manifest)- `("zip" "-0" "-X" ,jar ,manifest ,@files)- `("zip" "-0" "-X" ,jar ,@files))))- (apply invoke command)))- (utime jar 0 0)- #t))+ (let* ((tmp (tmpnam)))+ (copy-file jar tmp)+ (invoke "repack" "-o" jar tmp))) (for-each (match-lambda ((output . directory)-- 2.20.1
From 9d5ac721b396c5642825deda48e2ef9da1d55597 Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Sat, 19 Jan 2019 17:02:31 +0100Subject: [PATCH 05/11] guix: ant-build-system: Use repack-archives
* guix/build/ant-build-system.scm (strip-jar-timestamps): Use repack-archives instead of own implementation.--- guix/build/ant-build-system.scm | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
Toggle diff (23 lines)diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scmindex 540c985b7..7d639f492 100644--- a/guix/build/ant-build-system.scm+++ b/guix/build/ant-build-system.scm@@ -190,15 +190,9 @@ dependencies of this jar file." #:allow-other-keys) "Unpack all jar archives, reset the timestamp of all contained files, and repack them. This is necessary to ensure that archives are reproducible."- (define (repack-archive jar)- (format #t "repacking ~a\n" jar)- (let* ((tmp (tmpnam)))- (copy-file jar tmp)- (invoke "repack" "-o" jar tmp)))- (for-each (match-lambda ((output . directory)- (for-each repack-archive (find-files directory "\\.jar$"))))+ (repack-archives directory "\\.jar$"))) outputs) #t) -- 2.20.1
From 77d6222ceb26838da63dd0e73013245157e25d40 Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Thu, 17 Jan 2019 22:51:10 +0100Subject: [PATCH 06/11] gnu: ant-bootstrap: Use strip-jar-timestamps
* gnu/packages/java.scm (ant-bootstrap):[arguments]: Use build phase from ant-build-system.--- gnu/packages/java.scm | 37 ++++++------------------------------- 1 file changed, 6 insertions(+), 31 deletions(-)
Toggle diff (72 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scmindex 163c29627..dd7abfaf3 100644--- a/gnu/packages/java.scm+++ b/gnu/packages/java.scm@@ -10,6 +10,7 @@ ;;; Copyright © 2018 Gábor Boskovits <boskovits@gmail.com> ;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com> ;;; Copyright © 2018 Efraim Flashner <efraim@flashner.co.il>+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -217,9 +218,11 @@ JNI.") (build-system gnu-build-system) (arguments `(#:imported-modules ((guix build syscalls)+ (guix build ant-build-system) ,@%gnu-build-system-modules) #:modules ((srfi srfi-1) (guix build gnu-build-system)+ ((guix build ant-build-system) #:prefix ant:) (guix build utils) (guix build syscalls)) #:tests? #f ; no "check" target@@ -267,42 +270,14 @@ JNI.") (invoke "bash" "bootstrap.sh" (string-append "-Ddist.dir=" (assoc-ref %outputs "out")))))- (add-after 'build 'strip-jar-timestamps ;based on ant-build-system- (lambda* (#:key outputs #:allow-other-keys)- (define (repack-archive jar)- (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))- (manifest (string-append dir "/META-INF/MANIFESTS.MF")))- (with-directory-excursion dir- (invoke "unzip" jar))- (delete-file jar)- ;; XXX: copied from (gnu build install)- (for-each (lambda (file)- (let ((s (lstat file)))- (unless (eq? (stat:type s) 'symlink)- (utime file 0 0 0 0))))- (find-files dir #:directories? #t))- ;; It is important that the manifest appears first.- (with-directory-excursion dir- (let* ((files (find-files "." ".*" #:directories? #t))- ;; To ensure that the reference scanner can- ;; detect all store references in the jars- ;; we disable compression with the "-0" option.- (command (if (file-exists? manifest)- `("zip" "-0" "-X" ,jar ,manifest- ,@files)- `("zip" "-0" "-X" ,jar ,@files))))- (apply invoke command)))))- (for-each repack-archive- (find-files- (string-append (assoc-ref %outputs "out") "/lib")- "\\.jar$"))- #t))+ (add-after 'build 'strip-jar-timestamps+ (assoc-ref ant:%standard-phases 'strip-jar-timestamps)) (delete 'install)))) (native-inputs `(("jikes" ,jikes) ("jamvm" ,jamvm-1-bootstrap) ("unzip" ,unzip)- ("zip" ,zip)))+ ("repack" ,repack))) (home-page "http://ant.apache.org") (synopsis "Build tool for Java") (description-- 2.20.1
From 05b4bc707afc976eef9d04e93ae38f700971e3d6 Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Sat, 19 Jan 2019 16:19:42 +0100Subject: [PATCH 07/11] gnu: icedtea-8: Use repack
* gnu/packages/java.scm (icedtea-8):[native-inputs]: Use repack--- gnu/packages/java.scm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Toggle diff (16 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scmindex dd7abfaf3..4211e9a1e 100644--- a/gnu/packages/java.scm+++ b/gnu/packages/java.scm@@ -1606,7 +1606,8 @@ new Date();")) (add-after 'install 'strip-jar-timestamps (assoc-ref ant:%standard-phases 'strip-jar-timestamps))))))) (native-inputs- `(("jdk" ,icedtea-7 "jdk")+ `(("repack" ,repack)+ ("jdk" ,icedtea-7 "jdk") ("openjdk-src" ,(drop "openjdk" "1mj6xgmw31i6qd30qi9dmv7160fbcfq5ikz1jwjihdg2793il19p"))-- 2.20.1
From 8dfda0f40e3419fcbd5702f54acec1f4121a6e6b Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Sat, 19 Jan 2019 17:08:52 +0100Subject: [PATCH 08/11] gnu: openjdk9: Use repack for repacking zips
* gnu/packages/java.scm (openjdk9):[native-inputs]: Remove zip and unzip in favour of repack.[phases]: Rewrite strip-zip-timestamps to use repack.--- gnu/packages/java.scm | 23 +++++------------------ 1 file changed, 5 insertions(+), 18 deletions(-)
Toggle diff (44 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scmindex 4211e9a1e..a6095f8e7 100644--- a/gnu/packages/java.scm+++ b/gnu/packages/java.scm@@ -1738,22 +1738,8 @@ new Date();")) #t)) (add-after 'install 'strip-zip-timestamps (lambda* (#:key outputs #:allow-other-keys)- (use-modules (guix build syscalls))- (for-each (lambda (zip)- (let ((dir (mkdtemp! "zip-contents.XXXXXX")))- (with-directory-excursion dir- (invoke "unzip" zip))- (delete-file zip)- (for-each (lambda (file)- (let ((s (lstat file)))- (unless (eq? (stat:type s) 'symlink)- (format #t "reset ~a~%" file)- (utime file 0 0 0 0))))- (find-files dir #:directories? #t))- (with-directory-excursion dir- (let ((files (find-files "." ".*" #:directories? #t)))- (apply invoke "zip" "-0" "-X" zip files)))))- (find-files (assoc-ref outputs "doc") ".*.zip$"))+ (repack-archives (assoc-ref outputs "doc")+ ".*.zip$") #t))))) (inputs `(("alsa-lib" ,alsa-lib)@@ -1776,9 +1762,10 @@ new Date();")) (native-inputs `(("icedtea-8" ,icedtea-8) ("icedtea-8:jdk" ,icedtea-8 "jdk")+ ("zip" ,zip) ("unzip" ,unzip)- ("which" ,which)- ("zip" ,zip)))+ ("repack" ,repack)+ ("which" ,which))) (home-page "https://openjdk.java.net/projects/jdk9/") (synopsis "Java development kit") (description-- 2.20.1
From e38f60d43402c13958b73558a75bc1b7538bb46e Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Sat, 19 Jan 2019 17:10:45 +0100Subject: [PATCH 09/11] gnu: openjdk: Use repack for repacking zips
* gnu/packages/java.scm (openjdk-10):[native-inputs]: Use repack in favour of zip and unzip.--- gnu/packages/java.scm | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Toggle diff (19 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scmindex a6095f8e7..775c69e0f 100644--- a/gnu/packages/java.scm+++ b/gnu/packages/java.scm@@ -1819,9 +1819,10 @@ new Date();")) (native-inputs `(("openjdk9" ,openjdk9) ("openjdk9:jdk" ,openjdk9 "jdk")+ ("zip" ,zip) ("unzip" ,unzip)- ("which" ,which)- ("zip" ,zip)))))+ ("repack" ,repack)+ ("which" ,which))))) (define-public icedtea icedtea-8) -- 2.20.1
From 389dd6ab4cba29df667179c7c943836e2d149537 Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Sat, 19 Jan 2019 17:12:24 +0100Subject: [PATCH 10/11] gnu: ant/java8: Use repack for repacking archives
* gnu/packages/java.scm (ant/java8):[native-inputs]: Use repack in favour of zip and unzip.--- gnu/packages/java.scm | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
Toggle diff (16 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scmindex 775c69e0f..ad61bf294 100644--- a/gnu/packages/java.scm+++ b/gnu/packages/java.scm@@ -1871,8 +1871,7 @@ new Date();")) (assoc-ref outputs "out"))))))))) (native-inputs `(("jdk" ,icedtea-8 "jdk")- ("zip" ,zip)- ("unzip" ,unzip)))))+ ("repack" ,repack))))) ;; The 1.9.x series is the last that can be built with GCJ. The 1.10.x series ;; requires Java 8.-- 2.20.1
From 3df6e33f52ac2906ec98cc9b74ef93d9cbb22108 Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Sat, 19 Jan 2019 17:13:45 +0100Subject: [PATCH 11/11] gnu: ant: Use repack for repacking archives
* gnu/packages/java.scm (ant):[native-inputs]: Use repack in favour of zip and unzip.--- gnu/packages/java.scm | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
Toggle diff (16 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scmindex ad61bf294..0002c1f83 100644--- a/gnu/packages/java.scm+++ b/gnu/packages/java.scm@@ -1887,8 +1887,7 @@ new Date();")) "1k28mka0m3isy9yr8gz84kz1f3f879rwaxrd44vdn9xbfwvwk86n")))) (native-inputs `(("jdk" ,icedtea-7 "jdk")- ("zip" ,zip)- ("unzip" ,unzip)))))+ ("repack" ,repack))))) (define-public ant-apache-bcel (package-- 2.20.1
Attachment: repack.tar.xz
Attachment: signature.asc
L
L
Ludovic Courtès wrote on 16 Feb 2019 23:35
(name . Tim Gesthuizen)(address . tim.gesthuizen@yahoo.de)(address . 34223@debbugs.gnu.org)
87imxjfjjt.fsf@gnu.org
Hi Tim,
Sorry for the delay!
Tim Gesthuizen <tim.gesthuizen@yahoo.de> skribis:
Toggle quote (12 lines)> as discussed before I have looked into the problems of timestamps in the> zip files.> I looked at the way this is solved in ant-build-system with jar files> and thought that this could be done in a more elegant way.> Because of this I wrote a simple frontend for LibArchive in C that> repacks archives and sets their timestamps to zero and disables> compression as it is done in the ant-build-system.> Creative as I am the program is called repack.> You find a git repository attached with the history of the repack program.> The attached patches add repack to Guix and use it for pwsafe and the> ant-build-system.
Nice work! It’s great that libarchive doesn’t need to actually extractthe zip file to operate on it.
Overall I think the approach of factorizing archive-timestamp-resettingin one place and using it everywhere (‘ant-build-system’ and all) is theright thing to do.
However, I’m not sure whether we should introduce a new program for thispurpose. I believe ‘strip-nondeterminism’¹ (in Perl) by fellowReproducible Builds hackers also addresses this problem, so it may bewiser to use it.
But really, since (guix build utils) already implements a significantsubset of ‘strip-nondeterminism’, it would be even better if could avoidto shell out to a C or Perl program.
I played a bit with this idea and, as an example, the attached fileallows you to traverse the list of entries in a zip file (it uses‘guile-bytestructures’). Specifically, you can get the list of filenames in a zip file by running:
(call-with-input-file "something.zip" (lambda (port) (fold-entries cons '() port)))
Resetting timestamps should be just as simple.
How about taking this route?
Thanks,Ludo’.
¹ https://salsa.debian.org/reproducible-builds/strip-nondeterminism
(define-module (guix zip) #:use-module (rnrs bytevectors) #:use-module (rnrs io ports) #:use-module (bytestructures guile) #:use-module (ice-9 match) #:export (fold-entries)) (define <file-header> ;; File header, see ;; <https://en.wikipedia.org/wiki/Zip_(file_format)#File_headers>. (bs:struct #t ;packed `((signature ,uint32le) (version-needed ,uint16le) (flags ,uint16le) (compression ,uint16le) (modification-time ,uint16le) (modification-date ,uint16le) (crc32 ,uint32le) (compressed-size ,uint32le) (uncompressed-size ,uint32le) (file-name-length ,uint16le) (extra-field-length ,uint16le)))) (define-bytestructure-accessors <file-header> file-header-unwrap file-header-ref set-file-header!) (define (fold-entries proc seed port) "Fold PROC over all the entries in the zip file at PORT." (let loop ((result seed)) (match (get-bytevector-n port (bytestructure-descriptor-size <file-header>)) ((? bytevector? bv) (match (file-header-ref bv signature) (#x04034b50 ;local file header (let* ((len (file-header-ref bv file-name-length)) (name (utf8->string (get-bytevector-n port len)))) (set-port-position! port (+ (file-header-ref bv extra-field-length) (file-header-ref bv compressed-size) (port-position port))) (loop (proc name result)))) (#x02014b50 ;central directory record result) (#x06054b50 ;end of central directory record result))) ((? eof-object?) result))))
J
J
Julien Lepiller wrote on 17 Feb 2019 08:42
(address . 34223@debbugs.gnu.org)
2223B2BF-B691-45B1-82FE-A781CBFF0CDB@lepiller.eu
Le 16 février 2019 23:35:50 GMT+01:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
Toggle quote (57 lines)>Hi Tim,>>Sorry for the delay!>>Tim Gesthuizen <tim.gesthuizen@yahoo.de> skribis:>>> as discussed before I have looked into the problems of timestamps in>the>> zip files.>> I looked at the way this is solved in ant-build-system with jar files>> and thought that this could be done in a more elegant way.>> Because of this I wrote a simple frontend for LibArchive in C that>> repacks archives and sets their timestamps to zero and disables>> compression as it is done in the ant-build-system.>> Creative as I am the program is called repack.>> You find a git repository attached with the history of the repack>program.>> The attached patches add repack to Guix and use it for pwsafe and the>> ant-build-system.>>Nice work! It’s great that libarchive doesn’t need to actually extract>the zip file to operate on it.>>Overall I think the approach of factorizing archive-timestamp-resetting>in one place and using it everywhere (‘ant-build-system’ and all) is>the>right thing to do.>>However, I’m not sure whether we should introduce a new program for>this>purpose. I believe ‘strip-nondeterminism’¹ (in Perl) by fellow>Reproducible Builds hackers also addresses this problem, so it may be>wiser to use it.>>But really, since (guix build utils) already implements a significant>subset of ‘strip-nondeterminism’, it would be even better if could>avoid>to shell out to a C or Perl program.>>I played a bit with this idea and, as an example, the attached file>allows you to traverse the list of entries in a zip file (it uses>‘guile-bytestructures’). Specifically, you can get the list of file>names in a zip file by running:>> (call-with-input-file "something.zip"> (lambda (port)> (fold-entries cons '() port)))>>Resetting timestamps should be just as simple.>>How about taking this route?>>Thanks,>Ludo’.>>¹ https://salsa.debian.org/reproducible-builds/strip-nondeterminism
One of the reasons why we extract jar files is to remove compression, because the content might have store references that would be hidden, so grafting for instance wouldn't work.
T
T
Tim Gesthuizen wrote on 18 Feb 2019 21:07
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 34223@debbugs.gnu.org)
87mumshndk.fsf@yahoo.de
Hi Ludo,
Toggle quote (2 lines)> Sorry for the delay!
No problem! I have very little time anyway.
Toggle quote (12 lines)> Nice work! It’s great that libarchive doesn’t need to actually extract> the zip file to operate on it.>> Overall I think the approach of factorizing archive-timestamp-resetting> in one place and using it everywhere (‘ant-build-system’ and all) is the> right thing to do.>> However, I’m not sure whether we should introduce a new program for this> purpose. I believe ‘strip-nondeterminism’¹ (in Perl) by fellow> Reproducible Builds hackers also addresses this problem, so it may be> wiser to use it.
I also think so. If there is already another program that does the jobwe should probably use it.
Toggle quote (17 lines)> But really, since (guix build utils) already implements a significant> subset of ‘strip-nondeterminism’, it would be even better if could avoid> to shell out to a C or Perl program.>> I played a bit with this idea and, as an example, the attached file> allows you to traverse the list of entries in a zip file (it uses> ‘guile-bytestructures’). Specifically, you can get the list of file> names in a zip file by running:>> (call-with-input-file "something.zip"> (lambda (port)> (fold-entries cons '() port)))>> Resetting timestamps should be just as simple.>> How about taking this route?
I also thought about taking this route.There are some problems with it though:
- As Julien pointed out, the archive contents need to be uncompressed. This makes the problem much more complex and keeps us from writing a partial ZIP parser that replaces the timestamps in place.- While it would be quite elegant to just implement the parser in Scheme it would be redundant. After all we are developing a package manager so we should use it. This approach would be more attractive if there would be a Guile library for this. The best solution would be creating a proper library for handling archives when going with Scheme.- Maintaining a ZIP parser in Guix is a burden we should not take.- We need to care about a lot of details (ZIP64, probably more exotic extensions).
I would be fine with writing an own parser in Scheme but I would like topoint out that in every other place in Guix we are using external toolsfor handling archives (AFAIK).
I am not quite sure which version would be the best, so I am open forother opinions on this.Maybe you could rephrase your position taking the compression probleminto consideration.
Tim.
L
L
Ludovic Courtès wrote on 18 Feb 2019 23:24
(name . Tim Gesthuizen)(address . tim.gesthuizen@yahoo.de)(address . 34223@debbugs.gnu.org)
87pnro91mc.fsf@gnu.org
Hello,
Tim Gesthuizen <tim.gesthuizen@yahoo.de> skribis:
Toggle quote (20 lines)>> I played a bit with this idea and, as an example, the attached file>> allows you to traverse the list of entries in a zip file (it uses>> ‘guile-bytestructures’). Specifically, you can get the list of file>> names in a zip file by running:>>>> (call-with-input-file "something.zip">> (lambda (port)>> (fold-entries cons '() port)))>>>> Resetting timestamps should be just as simple.>>>> How about taking this route?>> I also thought about taking this route.> There are some problems with it though:>> - As Julien pointed out, the archive contents need to be uncompressed.> This makes the problem much more complex and keeps us from writing> a partial ZIP parser that replaces the timestamps in place.
True, I had overlooked that. In that case, we should definitely unpackand repack using the ‘zip’ package (I wasn’t suggesting to write acomplete ‘zip’ implementation; I do think it would be valuable in thelong term, but it’s a project for another time, no question here.)
In that case though, it probably doesn’t buy us much to use libarchivein a separate C program, WDYT? Should we just stick to the currentapproach that invokes ‘unzip’ and ‘zip’?
Thanks,Ludo’.
T
T
Tim Gesthuizen wrote on 2 Mar 2019 00:23
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 34223@debbugs.gnu.org)
87lg1yw559.fsf@yahoo.de
Hi Ludo,
Sorry for the delay!
Ludovic Courtès writes:
Toggle quote (4 lines)> In that case though, it probably doesn’t buy us much to use libarchive> in a separate C program, WDYT? Should we just stick to the current> approach that invokes ‘unzip’ and ‘zip’?
This seems to be the best choice.Maybe we want to reevaluate when there is a proper ZIP-library forguile.
I have attached patches that isolate repack-archive from theant-build-system and use it for pwsafe.I only builded some java packages so I don't know if somethingelse might be broken because of the changes.
Tim.
From fe8e4da34a0806d6f444d2fce572a2a39533f0fc Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Fri, 1 Mar 2019 23:39:14 +0100Subject: [PATCH 1/2] guix: Generalize zip repacking
Move repack-archive into an own module and change ant-build-systemand icedtea accordingly.
* Makefile.am: Add archives.scm.* gnu/packages/java.scm (icedtea-8): Add unzip to the native-inputs.* guix/build-system/ant.scm (default-unzip): New function. (lower): Add unzip parameter and add it to the build-inputs.* guix/build/ant-build-system.scm (strip-jar-timestamps): Remove repack-archive declaration.* guix/build/archives.scm: New file. (repack-archive): New function.--- Makefile.am | 2 ++ gnu/packages/java.scm | 5 ++- guix/build-system/ant.scm | 10 ++++++ guix/build/ant-build-system.scm | 38 +++------------------- guix/build/archives.scm | 56 +++++++++++++++++++++++++++++++++ 5 files changed, 77 insertions(+), 34 deletions(-) create mode 100644 guix/build/archives.scm
Toggle diff (227 lines)diff --git a/Makefile.am b/Makefile.amindex fec9800ce..c6dad9bb5 100644--- a/Makefile.am+++ b/Makefile.am@@ -13,6 +13,7 @@ # Copyright © 2018 Julien Lepiller <julien@lepiller.eu> # Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com> # Copyright © 2018 Alex Vong <alexvong1995@gmail.com>+# Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> # # This file is part of GNU Guix. #@@ -153,6 +154,7 @@ MODULES = \ guix/build/font-build-system.scm \ guix/build/go-build-system.scm \ guix/build/asdf-build-system.scm \+ guix/build/archives.scm \ guix/build/git.scm \ guix/build/hg.scm \ guix/build/glib-or-gtk-build-system.scm \diff --git a/gnu/packages/java.scm b/gnu/packages/java.scmindex 974756900..fea7d20b7 100644--- a/gnu/packages/java.scm+++ b/gnu/packages/java.scm@@ -10,6 +10,7 @@ ;;; Copyright © 2018 Gábor Boskovits <boskovits@gmail.com> ;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com> ;;; Copyright © 2018 Efraim Flashner <efraim@flashner.co.il>+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de ;;; ;;; This file is part of GNU Guix. ;;;@@ -1549,6 +1550,7 @@ IcedTea build harness.") `(#:imported-modules ((guix build ant-build-system) (guix build syscalls)+ (guix build archives) ,@%gnu-build-system-modules) ,@(substitute-keyword-arguments (package-arguments icedtea-7) ((#:modules modules)@@ -1672,7 +1674,8 @@ new Date();")) "0k33anxdzw1icn072wynfmmdjhsv50hay0j1sfkfxny12rb3vgdy")) ,@(fold alist-delete (package-native-inputs icedtea-7) '("jdk" "openjdk-src" "corba-drop" "jaxp-drop" "jaxws-drop"- "jdk-drop" "langtools-drop" "hotspot-drop")))))))+ "jdk-drop" "langtools-drop" "hotspot-drop"))+ ("unzip" ,unzip)))))) (define-public openjdk9 (packagediff --git a/guix/build-system/ant.scm b/guix/build-system/ant.scmindex b5626bd42..e4cd4c1f7 100644--- a/guix/build-system/ant.scm+++ b/guix/build-system/ant.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -41,6 +42,7 @@ `((guix build ant-build-system) (guix build java-utils) (guix build syscalls)+ (guix build archives) ,@%gnu-build-system-modules)) (define (default-jdk)@@ -61,11 +63,18 @@ (let ((zip-mod (resolve-interface '(gnu packages compression)))) (module-ref zip-mod 'zip))) +(define (default-unzip)+ "Return the default UNZIP package."+ ;; Lazily resolve the binding to avoid a circular dependency.+ (let ((zip-mod (resolve-interface '(gnu packages compression))))+ (module-ref zip-mod 'unzip)))+ (define* (lower name #:key source inputs native-inputs outputs system target (jdk (default-jdk)) (ant (default-ant)) (zip (default-zip))+ (unzip (default-unzip)) #:allow-other-keys #:rest arguments) "Return a bag for NAME."@@ -86,6 +95,7 @@ (build-inputs `(("jdk" ,jdk "jdk") ("ant" ,ant) ("zip" ,zip)+ ("unzip" ,unzip) ,@native-inputs)) (outputs outputs) (build ant-build)diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scmindex d79a2d55e..fbde6fae7 100644--- a/guix/build/ant-build-system.scm+++ b/guix/build/ant-build-system.scm@@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016, 2018 Ricardo Wurmus <rekado@elephly.net>+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -20,6 +21,7 @@ #:use-module ((guix build gnu-build-system) #:prefix gnu:) #:use-module (guix build syscalls) #:use-module (guix build utils)+ #:use-module (guix build archives) #:use-module (sxml simple) #:use-module (ice-9 match) #:use-module (ice-9 ftw)@@ -185,44 +187,14 @@ dependencies of this jar file." outputs) #t) -(define* (strip-jar-timestamps #:key outputs+(define* (strip-jar-timestamps #:key outputs (unzip "unzip") (zip "zip") #:allow-other-keys) "Unpack all jar archives, reset the timestamp of all contained files, and repack them. This is necessary to ensure that archives are reproducible."- (define (repack-archive jar)- (format #t "repacking ~a\n" jar)- (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))- (manifest (string-append dir "/META-INF/MANIFEST.MF")))- (with-directory-excursion dir- (invoke "jar" "xf" jar))- (delete-file jar)- ;; XXX: copied from (gnu build install)- (for-each (lambda (file)- (let ((s (lstat file)))- (unless (eq? (stat:type s) 'symlink)- (utime file 0 0 0 0))))- (find-files dir #:directories? #t))-- ;; The jar tool will always set the timestamp on the manifest file- ;; and the containing directory to the current time, even when we- ;; reuse an existing manifest file. To avoid this we use "zip"- ;; instead of "jar". It is important that the manifest appears- ;; first.- (with-directory-excursion dir- (let* ((files (find-files "." ".*" #:directories? #t))- ;; To ensure that the reference scanner can detect all- ;; store references in the jars we disable compression- ;; with the "-0" option.- (command (if (file-exists? manifest)- `("zip" "-0" "-X" ,jar ,manifest ,@files)- `("zip" "-0" "-X" ,jar ,@files))))- (apply invoke command)))- (utime jar 0 0)- #t))- (for-each (match-lambda ((output . directory)- (for-each repack-archive (find-files directory "\\.jar$"))))+ (for-each repack-archive+ (find-files directory "\\.jar$")))) outputs) #t) diff --git a/guix/build/archives.scm b/guix/build/archives.scmnew file mode 100644index 000000000..d2c4815bd--- /dev/null+++ b/guix/build/archives.scm@@ -0,0 +1,56 @@+;;; GNU Guix --- Functional package management for GNU+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>+;;;+;;; This file is part of GNU Guix.+;;;+;;; GNU Guix is free software; you can redistribute it and/or modify it+;;; under the terms of the GNU General Public License as published by+;;; the Free Software Foundation; either version 3 of the License, or (at+;;; your option) any later version.+;;;+;;; GNU Guix is distributed in the hope that it will be useful, but+;;; WITHOUT ANY WARRANTY; without even the implied warranty of+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+;;; GNU General Public License for more details.+;;;+;;; You should have received a copy of the GNU General Public License+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.+++(define-module (guix build archives)+ #:use-module (ice-9 format)+ #:use-module (guix build utils)+ #:use-module (guix build syscalls)+ #:export (repack-archive))++(define* (repack-archive archive+ #:key (unzip "unzip") (zip "zip"))+ (format #t "repacking ~a\n" archive)+ (let* ((dir (mkdtemp! "archive-contents.XXXXXX"))+ (manifest (string-append dir "/META-INF/MANIFEST.MF")))+ (with-directory-excursion dir+ (invoke unzip archive))+ (delete-file archive)+ ;; XXX: copied from (gnu build install)+ (for-each (lambda (file)+ (let ((s (lstat file)))+ (unless (eq? (stat:type s) 'symlink)+ (utime file 0 0 0 0))))+ (find-files dir #:directories? #t))++ ;; The archive tool will always set the timestamp on the manifest file+ ;; and the containing directory to the current time, even when we+ ;; reuse an existing manifest file. To avoid this we use "zip"+ ;; instead of "archive". It is important that the manifest appears+ ;; first.+ (with-directory-excursion dir+ (let* ((files (find-files "." ".*" #:directories? #t))+ ;; To ensure that the reference scanner can detect all+ ;; store references in the archives we disable compression+ ;; with the "-0" option.+ (command (if (file-exists? manifest)+ `(,zip "-0" "-X" ,archive ,manifest ,@files)+ `(,zip "-0" "-X" ,archive ,@files))))+ (apply invoke command)))+ (utime archive 0 0)+ #t))-- 2.20.1
From 0bb0420dfdeb992b3ceafc815d42e6f403520b8d Mon Sep 17 00:00:00 2001From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>Date: Sat, 2 Mar 2019 00:10:19 +0100Subject: [PATCH 2/2] gnu: pwsafe: Make zip archives deterministic
* gnu/packages/password-utils.scm (pwsafe): [native-inputs]: Add unzip. [arguments]: Add a phase for resetting zip timestamps.--- gnu/packages/password-utils.scm | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
Toggle diff (51 lines)diff --git a/gnu/packages/password-utils.scm b/gnu/packages/password-utils.scmindex 9fd5a6ff0..52870050e 100644--- a/gnu/packages/password-utils.scm+++ b/gnu/packages/password-utils.scm@@ -21,7 +21,7 @@ ;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net> ;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz> ;;; Copyright © 2018 Amirouche Boubekki <amirouche@hypermove.net>-;;; Copyright © 2018 Tim Gesthuizen <tim.gesthuizen@yahoo.de>+;;; Copyright © 2018, 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de> ;;; ;;; This file is part of GNU Guix. ;;;@@ -190,7 +190,8 @@ algorithms AES or Twofish.") (build-system cmake-build-system) (native-inputs `(("gettext" ,gettext-minimal) ("perl" ,perl)- ("zip" ,zip)))+ ("zip" ,zip)+ ("unzip" ,unzip))) (inputs `(("curl" ,curl) ("file" ,file) ("gtest" ,googletest)@@ -201,7 +202,10 @@ algorithms AES or Twofish.") ("qrencode" ,qrencode) ("wxwidgets" ,wxwidgets) ("xerces-c" ,xerces-c)))- (arguments '(#:configure-flags (list "-DNO_GTEST=YES")+ (arguments `(#:configure-flags (list "-DNO_GTEST=YES")+ #:imported-modules ((guix build syscalls)+ (guix build archives)+ ,@%cmake-build-system-modules) #:phases (modify-phases %standard-phases (add-after 'unpack 'add-gtest (lambda* (#:key inputs #:allow-other-keys)@@ -219,6 +223,13 @@ add_subdirectory(src/test)\n" cmake-port) (("/usr/bin/file") (string-append (assoc-ref inputs "file") "/bin/file")))+ #t))+ (add-after 'install 'repack-archives+ (lambda* (#:key outputs #:allow-other-keys)+ (use-modules ((guix build archives)))+ (for-each repack-archive+ (find-files (assoc-ref outputs "out")+ "\\.zip$")) #t))))) (synopsis "Password safe with automatic input and key generation") (description "pwsafe is a password manager originally designed by Bruce-- 2.20.1
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 34223@debbugs.gnu.org