Fixing timestamps in archives.

Open

Details

3 participants

Julien Lepiller
Ludovic Courtès
Tim Gesthuizen

Owner: unassigned

Submitted by: Tim Gesthuizen

Severity: normal

Tim Gesthuizen wrote on 27 Jan 2019 18:58

Recipients:

Message-ID:e13090c1-dabb-da63-cc62-3975f2697527@yahoo.de

Hi Ludo,

as discussed before I have looked into the problems of timestamps in the
zip files.
I looked at the way this is solved in ant-build-system with jar files
and thought that this could be done in a more elegant way.
Because of this I wrote a simple frontend for LibArchive in C that
repacks archives and sets their timestamps to zero and disables
compression as it is done in the ant-build-system.
Creative as I am the program is called repack.
You find a git repository attached with the history of the repack program.
The attached patches add repack to Guix and use it for pwsafe and the
ant-build-system.

This is a work in progress version: If you like these changes I will
work on missing details so that we can add it, otherwise now would be a
good point to stop development on these changes.
I can still fall back to the variant that ant-build-system uses now for
pwsafe.

It would be nice if you could find the time to review everything and
tell me what you think about the patches.
The changes trigger a lot of rebuilds so it will take some time to build
dependencies of the required programs.

The repack tar contains a bare git repository with the program.
Because I could not yet find a place to host the repository you need to
unpack it somewhere, create an archive with the source code and change
the guix definition of repack to use that source.

Tim.

From fe500f0ba03aebc3e0020d974f1266c00001d64e Mon Sep 17 00:00:00 2001

* gnu/packages/compression.scm (repack): New variable.

---

gnu/packages/compression.scm | 19 +++++++++++++++++++

1 file changed, 19 insertions(+)

Toggle diff (53 lines)diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index 327951fde..837fca477 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -23,6 +23,7 @@
 ;;; Copyright © 2018 Rutger Helling <rhelling@mykolab.com>
 ;;; Copyright © 2018 Joshua Sierles, Nextjournal <joshua@nextjournal.com>
 ;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>
+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -44,6 +45,7 @@
   #:use-module (guix utils)
   #:use-module (guix packages)
   #:use-module (guix download)
+  #:use-module (guix gexp)
   #:use-module (guix git-download)
   #:use-module (guix build-system cmake)
   #:use-module (guix build-system gnu)
@@ -60,6 +62,7 @@
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
+  #:use-module (gnu packages texinfo)
   #:use-module (gnu packages tls)
   #:use-module (gnu packages valgrind)
   #:use-module (ice-9 match)
@@ -1422,6 +1425,22 @@ Compression ratios of 2:1 to 3:1 are common for text files.")
     (license (license:non-copyleft "file://LICENSE"
                                    "See LICENSE in the distribution."))))
 
+(define-public repack
+  (package
+    (name "repack")
+    (version "0.1")
+    (source (local-file "/home/tibbe/src/repack.tar.gz"))
+    (build-system cmake-build-system)
+    (arguments `(#:tests? #f))
+    (native-inputs
+     `(("texinfo" ,texinfo)))
+    (inputs
+     `(("libarchive" ,libarchive)))
+    (home-page #f)
+    (synopsis "Repack archives to not include time stamps or compression.")
+    (description #f)
+    (license license:gpl3+)))
+
 (define-public unzip
   (package (inherit zip)
     (name "unzip")
-- 
2.20.1

From 394b939ed84154c203c614c23e4f6bc2df17613c Mon Sep 17 00:00:00 2001

Add a function that repacks all archives in a certain directory to be

reproduceably buildable.

* guix/build/utils.scm (repack-archives): New function.

---

guix/build/utils.scm | 11 +++++++++++

1 file changed, 11 insertions(+)

Toggle diff (38 lines)diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index 5fe328684..3ca72299a 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2013 Andreas Enge <andreas@enge.fr>
 ;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
 ;;; Copyright © 2015, 2018 Mark H Weaver <mhw@netris.org>
+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -88,6 +89,7 @@
             fold-port-matches
             remove-store-references
             wrap-program
+            repack-archives
 
             invoke
             invoke-error?
@@ -1100,6 +1102,15 @@ with definitions for VARS."
         (chmod prog-tmp #o755)
         (rename-file prog-tmp prog))))
 
+(define (repack-archives root regex)
+  "Repack all archives under ROOT that match REGEX using the repack program."
+  (define (repack-archive jar)
+    (format #t "repacking ~a\n" jar)
+    (let* ((tmp (tmpnam)))
+      (copy-file jar tmp)
+      (invoke "repack" "-o" jar tmp)))
+  (for-each repack-archive (find-files root regex)))
+
 
 ;;;
 ;;; Locales.
-- 
2.20.1

From bd552a577fb7dbe1233b366c95d1741754fbea33 Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Thu, 17 Jan 2019 20:38:10 +0100
Subject: [PATCH 03/11] gnu: pwsafe: Remove timestamps from zip files

* gnu/packages/password-utils.scm (pwsafe):
[native-inputs]: Add repack.
[arguments]: Add modules and a phase to strip timestamps from the zip files.
---
 gnu/packages/password-utils.scm | 38 +++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

Toggle diff (72 lines)diff --git a/gnu/packages/password-utils.scm b/gnu/packages/password-utils.scm
index 5b60ad9cc..47fb9dccb 100644
--- a/gnu/packages/password-utils.scm
+++ b/gnu/packages/password-utils.scm
@@ -21,7 +21,7 @@
 ;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net>
 ;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>
 ;;; Copyright © 2018 Amirouche Boubekki <amirouche@hypermove.net>
-;;; Copyright © 2018 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
+;;; Copyright © 2018, 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -190,7 +190,8 @@ algorithms AES or Twofish.")
     (build-system cmake-build-system)
     (native-inputs `(("gettext" ,gettext-minimal)
                      ("perl" ,perl)
-                     ("zip" ,zip)))
+                     ("zip" ,zip)
+                     ("repack" ,repack)))
     (inputs `(("curl" ,curl)
               ("file" ,file)
               ("gtest" ,googletest)
@@ -201,7 +202,11 @@ algorithms AES or Twofish.")
               ("qrencode" ,qrencode)
               ("wxwidgets" ,wxwidgets)
               ("xerces-c" ,xerces-c)))
-    (arguments '(#:configure-flags (list "-DNO_GTEST=YES")
+    (arguments `(#:configure-flags (list "-DNO_GTEST=YES")
+                 #:modules ((guix build cmake-build-system)
+                            (guix build utils)
+                            (ice-9 regex)
+                            (ice-9 ftw))
                  #:phases (modify-phases %standard-phases
                             (add-after 'unpack 'add-gtest
                               (lambda* (#:key inputs #:allow-other-keys)
@@ -219,7 +224,32 @@ add_subdirectory(src/test)\n" cmake-port)
                                   (("/usr/bin/file")
                                    (string-append (assoc-ref inputs "file")
                                                   "/bin/file")))
-                                #t)))))
+                                #t))
+                            (add-after 'install 'fix-zip-timestamps
+                              (lambda* (#:key outputs inputs
+                                        #:allow-other-keys)
+                                (let ((out (assoc-ref outputs "out"))
+                                      (repack (string-append
+                                               (assoc-ref inputs "repack")
+                                               "/bin/repack")))
+                                  (with-directory-excursion out
+                                    (file-system-fold
+                                     (const #t)
+                                     (lambda (path stat result)
+                                       (when (string-match "^.*\\.zip$"
+                                                           (basename path))
+                                        (let ((tmp (tmpnam)))
+                                          (format #t "Repacking ~s\n" path)
+                                          (copy-file path tmp)
+                                          (invoke repack
+                                                  "-o" path
+                                                  tmp))))
+                                     (const #t)
+                                     (const #t)
+                                     (const #t)
+                                     (lambda _ (throw 'file-system-error))
+                                     #t "."))
+                                  #t))))))
     (synopsis "Password safe with automatic input and key generation")
     (description "pwsafe is a password manager originally designed by Bruce
 Schneier.  It offers a simple UI to manage passwords for different services.
-- 
2.20.1

From ed9db022d4e2ea9b6f1151fbc2785014b2a1bd8e Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Thu, 17 Jan 2019 22:49:46 +0100
Subject: [PATCH 04/11] guix: ant: Use repack

* guix/build-system/ant.scm (default-zip): Deleted.
* guix/build-system/ant.scm (lower): Change zip to repack.
* guix/build/ant-build-system.scm (strip-jar-timestamps): Use repack for
  repack-archive.
---
 guix/build-system/ant.scm       | 16 ++++++++--------
 guix/build/ant-build-system.scm | 32 ++++----------------------------
 2 files changed, 12 insertions(+), 36 deletions(-)

Toggle diff (100 lines)diff --git a/guix/build-system/ant.scm b/guix/build-system/ant.scm
index b5626bd42..a4aa90eb8 100644
--- a/guix/build-system/ant.scm
+++ b/guix/build-system/ant.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -55,22 +56,21 @@
   (let ((jdk-mod (resolve-interface '(gnu packages java))))
     (module-ref jdk-mod 'ant)))
 
-(define (default-zip)
-  "Return the default ZIP package."
-  ;; Lazily resolve the binding to avoid a circular dependency.
-  (let ((zip-mod (resolve-interface '(gnu packages compression))))
-    (module-ref zip-mod 'zip)))
+(define (default-repack)
+  "Return the default REPACK package"
+  (let ((repack-mod (resolve-interface '(gnu packages compression))))
+    (module-ref repack-mod 'repack)))
 
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (jdk (default-jdk))
                 (ant (default-ant))
-                (zip (default-zip))
+                (repack (default-repack))
                 #:allow-other-keys
                 #:rest arguments)
   "Return a bag for NAME."
   (define private-keywords
-    '(#:source #:target #:jdk #:ant #:zip #:inputs #:native-inputs))
+    '(#:source #:target #:jdk #:ant #:repack #:inputs #:native-inputs))
 
   (and (not target)                               ;XXX: no cross-compilation
        (bag
@@ -85,7 +85,7 @@
                         ,@(standard-packages)))
          (build-inputs `(("jdk" ,jdk "jdk")
                          ("ant" ,ant)
-                         ("zip" ,zip)
+                         ("repack" ,repack)
                          ,@native-inputs))
          (outputs outputs)
          (build ant-build)
diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm
index d79a2d55e..540c985b7 100644
--- a/guix/build/ant-build-system.scm
+++ b/guix/build/ant-build-system.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016, 2018 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -191,34 +192,9 @@ dependencies of this jar file."
 repack them.  This is necessary to ensure that archives are reproducible."
   (define (repack-archive jar)
     (format #t "repacking ~a\n" jar)
-    (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
-           (manifest (string-append dir "/META-INF/MANIFEST.MF")))
-      (with-directory-excursion dir
-        (invoke "jar" "xf" jar))
-      (delete-file jar)
-      ;; XXX: copied from (gnu build install)
-      (for-each (lambda (file)
-                  (let ((s (lstat file)))
-                    (unless (eq? (stat:type s) 'symlink)
-                      (utime file 0 0 0 0))))
-                (find-files dir #:directories? #t))
-
-      ;; The jar tool will always set the timestamp on the manifest file
-      ;; and the containing directory to the current time, even when we
-      ;; reuse an existing manifest file.  To avoid this we use "zip"
-      ;; instead of "jar".  It is important that the manifest appears
-      ;; first.
-      (with-directory-excursion dir
-        (let* ((files (find-files "." ".*" #:directories? #t))
-               ;; To ensure that the reference scanner can detect all
-               ;; store references in the jars we disable compression
-               ;; with the "-0" option.
-               (command (if (file-exists? manifest)
-                            `("zip" "-0" "-X" ,jar ,manifest ,@files)
-                            `("zip" "-0" "-X" ,jar ,@files))))
-          (apply invoke command)))
-      (utime jar 0 0)
-      #t))
+    (let* ((tmp (tmpnam)))
+      (copy-file jar tmp)
+      (invoke "repack" "-o" jar tmp)))
 
   (for-each (match-lambda
               ((output . directory)
-- 
2.20.1

From 9d5ac721b396c5642825deda48e2ef9da1d55597 Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Sat, 19 Jan 2019 17:02:31 +0100
Subject: [PATCH 05/11] guix: ant-build-system: Use repack-archives

* guix/build/ant-build-system.scm (strip-jar-timestamps): Use repack-archives
  instead of own implementation.
---
 guix/build/ant-build-system.scm | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

Toggle diff (23 lines)diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm
index 540c985b7..7d639f492 100644
--- a/guix/build/ant-build-system.scm
+++ b/guix/build/ant-build-system.scm
@@ -190,15 +190,9 @@ dependencies of this jar file."
                                #:allow-other-keys)
   "Unpack all jar archives, reset the timestamp of all contained files, and
 repack them.  This is necessary to ensure that archives are reproducible."
-  (define (repack-archive jar)
-    (format #t "repacking ~a\n" jar)
-    (let* ((tmp (tmpnam)))
-      (copy-file jar tmp)
-      (invoke "repack" "-o" jar tmp)))
-
   (for-each (match-lambda
               ((output . directory)
-               (for-each repack-archive (find-files directory "\\.jar$"))))
+               (repack-archives directory "\\.jar$")))
             outputs)
   #t)
 
-- 
2.20.1

From 77d6222ceb26838da63dd0e73013245157e25d40 Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Thu, 17 Jan 2019 22:51:10 +0100
Subject: [PATCH 06/11] gnu: ant-bootstrap: Use strip-jar-timestamps

* gnu/packages/java.scm (ant-bootstrap):
[arguments]: Use build phase from ant-build-system.
---
 gnu/packages/java.scm | 37 ++++++-------------------------------
 1 file changed, 6 insertions(+), 31 deletions(-)

Toggle diff (72 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 163c29627..dd7abfaf3 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -10,6 +10,7 @@
 ;;; Copyright © 2018 Gábor Boskovits <boskovits@gmail.com>
 ;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
 ;;; Copyright © 2018 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -217,9 +218,11 @@ JNI.")
     (build-system gnu-build-system)
     (arguments
      `(#:imported-modules ((guix build syscalls)
+                           (guix build ant-build-system)
                            ,@%gnu-build-system-modules)
        #:modules ((srfi srfi-1)
                   (guix build gnu-build-system)
+                  ((guix build ant-build-system) #:prefix ant:)
                   (guix build utils)
                   (guix build syscalls))
        #:tests? #f ; no "check" target
@@ -267,42 +270,14 @@ JNI.")
              (invoke "bash" "bootstrap.sh"
                      (string-append "-Ddist.dir="
                                     (assoc-ref %outputs "out")))))
-         (add-after 'build 'strip-jar-timestamps ;based on ant-build-system
-           (lambda* (#:key outputs #:allow-other-keys)
-             (define (repack-archive jar)
-               (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
-                      (manifest (string-append dir "/META-INF/MANIFESTS.MF")))
-                 (with-directory-excursion dir
-                   (invoke "unzip" jar))
-                 (delete-file jar)
-                 ;; XXX: copied from (gnu build install)
-                 (for-each (lambda (file)
-                             (let ((s (lstat file)))
-                               (unless (eq? (stat:type s) 'symlink)
-                                 (utime file  0 0 0 0))))
-                           (find-files dir #:directories? #t))
-                 ;; It is important that the manifest appears first.
-                 (with-directory-excursion dir
-                   (let* ((files (find-files "." ".*" #:directories? #t))
-                          ;; To ensure that the reference scanner can
-                          ;; detect all store references in the jars
-                          ;; we disable compression with the "-0" option.
-                          (command (if (file-exists? manifest)
-                                       `("zip" "-0" "-X" ,jar ,manifest
-                                         ,@files)
-                                       `("zip" "-0" "-X" ,jar ,@files))))
-                     (apply invoke command)))))
-             (for-each repack-archive
-                    (find-files
-                     (string-append (assoc-ref %outputs "out") "/lib")
-                     "\\.jar$"))
-             #t))
+         (add-after 'build 'strip-jar-timestamps
+                   (assoc-ref ant:%standard-phases 'strip-jar-timestamps))
          (delete 'install))))
     (native-inputs
      `(("jikes" ,jikes)
        ("jamvm" ,jamvm-1-bootstrap)
        ("unzip" ,unzip)
-       ("zip" ,zip)))
+       ("repack" ,repack)))
     (home-page "http://ant.apache.org")
     (synopsis "Build tool for Java")
     (description
-- 
2.20.1

From 05b4bc707afc976eef9d04e93ae38f700971e3d6 Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Sat, 19 Jan 2019 16:19:42 +0100
Subject: [PATCH 07/11] gnu: icedtea-8: Use repack

* gnu/packages/java.scm (icedtea-8):
[native-inputs]: Use repack
---
 gnu/packages/java.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Toggle diff (16 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index dd7abfaf3..4211e9a1e 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -1606,7 +1606,8 @@ new Date();"))
                  (add-after 'install 'strip-jar-timestamps
                    (assoc-ref ant:%standard-phases 'strip-jar-timestamps)))))))
       (native-inputs
-       `(("jdk" ,icedtea-7 "jdk")
+       `(("repack" ,repack)
+	 ("jdk" ,icedtea-7 "jdk")
          ("openjdk-src"
           ,(drop "openjdk"
                  "1mj6xgmw31i6qd30qi9dmv7160fbcfq5ikz1jwjihdg2793il19p"))
-- 
2.20.1

From 8dfda0f40e3419fcbd5702f54acec1f4121a6e6b Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Sat, 19 Jan 2019 17:08:52 +0100
Subject: [PATCH 08/11] gnu: openjdk9: Use repack for repacking zips

* gnu/packages/java.scm (openjdk9):
[native-inputs]: Remove zip and unzip in favour of repack.
[phases]: Rewrite strip-zip-timestamps to use repack.
---
 gnu/packages/java.scm | 23 +++++------------------
 1 file changed, 5 insertions(+), 18 deletions(-)

Toggle diff (44 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 4211e9a1e..a6095f8e7 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -1738,22 +1738,8 @@ new Date();"))
              #t))
          (add-after 'install 'strip-zip-timestamps
            (lambda* (#:key outputs #:allow-other-keys)
-             (use-modules (guix build syscalls))
-             (for-each (lambda (zip)
-                         (let ((dir (mkdtemp! "zip-contents.XXXXXX")))
-                           (with-directory-excursion dir
-                             (invoke "unzip" zip))
-                           (delete-file zip)
-                           (for-each (lambda (file)
-                                       (let ((s (lstat file)))
-                                         (unless (eq? (stat:type s) 'symlink)
-                                           (format #t "reset ~a~%" file)
-                                           (utime file 0 0 0 0))))
-                             (find-files dir #:directories? #t))
-                           (with-directory-excursion dir
-                             (let ((files (find-files "." ".*" #:directories? #t)))
-                               (apply invoke "zip" "-0" "-X" zip files)))))
-               (find-files (assoc-ref outputs "doc") ".*.zip$"))
+	     (repack-archives (assoc-ref outputs "doc")
+			      ".*.zip$")
              #t)))))
     (inputs
      `(("alsa-lib" ,alsa-lib)
@@ -1776,9 +1762,10 @@ new Date();"))
     (native-inputs
      `(("icedtea-8" ,icedtea-8)
        ("icedtea-8:jdk" ,icedtea-8 "jdk")
+       ("zip" ,zip)
        ("unzip" ,unzip)
-       ("which" ,which)
-       ("zip" ,zip)))
+       ("repack" ,repack)
+       ("which" ,which)))
     (home-page "https://openjdk.java.net/projects/jdk9/")
     (synopsis "Java development kit")
     (description
-- 
2.20.1

From e38f60d43402c13958b73558a75bc1b7538bb46e Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Sat, 19 Jan 2019 17:10:45 +0100
Subject: [PATCH 09/11] gnu: openjdk: Use repack for repacking zips

* gnu/packages/java.scm (openjdk-10):
[native-inputs]: Use repack in favour of zip and unzip.
---
 gnu/packages/java.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (19 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index a6095f8e7..775c69e0f 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -1819,9 +1819,10 @@ new Date();"))
     (native-inputs
      `(("openjdk9" ,openjdk9)
        ("openjdk9:jdk" ,openjdk9 "jdk")
+       ("zip" ,zip)
        ("unzip" ,unzip)
-       ("which" ,which)
-       ("zip" ,zip)))))
+       ("repack" ,repack)
+       ("which" ,which)))))
 
 (define-public icedtea icedtea-8)
 
-- 
2.20.1

From 389dd6ab4cba29df667179c7c943836e2d149537 Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Sat, 19 Jan 2019 17:12:24 +0100
Subject: [PATCH 10/11] gnu: ant/java8: Use repack for repacking archives

* gnu/packages/java.scm (ant/java8):
[native-inputs]: Use repack in favour of zip and unzip.
---
 gnu/packages/java.scm | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Toggle diff (16 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 775c69e0f..ad61bf294 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -1871,8 +1871,7 @@ new Date();"))
                                       (assoc-ref outputs "out")))))))))
     (native-inputs
      `(("jdk" ,icedtea-8 "jdk")
-       ("zip" ,zip)
-       ("unzip" ,unzip)))))
+       ("repack" ,repack)))))
 
 ;; The 1.9.x series is the last that can be built with GCJ.  The 1.10.x series
 ;; requires Java 8.
-- 
2.20.1

From 3df6e33f52ac2906ec98cc9b74ef93d9cbb22108 Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Sat, 19 Jan 2019 17:13:45 +0100
Subject: [PATCH 11/11] gnu: ant: Use repack for repacking archives

* gnu/packages/java.scm (ant):
[native-inputs]: Use repack in favour of zip and unzip.
---
 gnu/packages/java.scm | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Toggle diff (16 lines)diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index ad61bf294..0002c1f83 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -1887,8 +1887,7 @@ new Date();"))
                 "1k28mka0m3isy9yr8gz84kz1f3f879rwaxrd44vdn9xbfwvwk86n"))))
     (native-inputs
      `(("jdk" ,icedtea-7 "jdk")
-       ("zip" ,zip)
-       ("unzip" ,unzip)))))
+       ("repack" ,repack)))))
 
 (define-public ant-apache-bcel
   (package
-- 
2.20.1

Attachment: repack.tar.xz

Attachment: signature.asc

Ludovic Courtès wrote on 16 Feb 2019 23:35

Recipients:(name . Tim Gesthuizen)(address . tim.gesthuizen@yahoo.de)(address . 34223@debbugs.gnu.org)

Message-ID:87imxjfjjt.fsf@gnu.org

Hi Tim,

Sorry for the delay!

Tim Gesthuizen <tim.gesthuizen@yahoo.de> skribis:

Toggle quote (12 lines)> as discussed before I have looked into the problems of timestamps in the
> zip files.
> I looked at the way this is solved in ant-build-system with jar files
> and thought that this could be done in a more elegant way.
> Because of this I wrote a simple frontend for LibArchive in C that
> repacks archives and sets their timestamps to zero and disables
> compression as it is done in the ant-build-system.
> Creative as I am the program is called repack.
> You find a git repository attached with the history of the repack program.
> The attached patches add repack to Guix and use it for pwsafe and the
> ant-build-system.

Nice work! It’s great that libarchive doesn’t need to actually extract

the zip file to operate on it.

Overall I think the approach of factorizing archive-timestamp-resetting

in one place and using it everywhere (‘ant-build-system’ and all) is the

right thing to do.

However, I’m not sure whether we should introduce a new program for this

purpose. I believe ‘strip-nondeterminism’¹ (in Perl) by fellow

Reproducible Builds hackers also addresses this problem, so it may be

wiser to use it.

But really, since (guix build utils) already implements a significant

subset of ‘strip-nondeterminism’, it would be even better if could avoid

to shell out to a C or Perl program.

I played a bit with this idea and, as an example, the attached file

allows you to traverse the list of entries in a zip file (it uses

‘guile-bytestructures’). Specifically, you can get the list of file

names in a zip file by running:

(call-with-input-file "something.zip"

(lambda (port)

(fold-entries cons '() port)))

Resetting timestamps should be just as simple.

How about taking this route?

Thanks,

Ludo’.

¹ https://salsa.debian.org/reproducible-builds/strip-nondeterminism

(define-module (guix zip) #:use-module (rnrs bytevectors) #:use-module (rnrs io ports) #:use-module (bytestructures guile) #:use-module (ice-9 match) #:export (fold-entries)) (define <file-header> ;; File header, see ;; <https://en.wikipedia.org/wiki/Zip_(file_format)#File_headers>. (bs:struct #t ;packed `((signature ,uint32le) (version-needed ,uint16le) (flags ,uint16le) (compression ,uint16le) (modification-time ,uint16le) (modification-date ,uint16le) (crc32 ,uint32le) (compressed-size ,uint32le) (uncompressed-size ,uint32le) (file-name-length ,uint16le) (extra-field-length ,uint16le)))) (define-bytestructure-accessors <file-header> file-header-unwrap file-header-ref set-file-header!) (define (fold-entries proc seed port) "Fold PROC over all the entries in the zip file at PORT." (let loop ((result seed)) (match (get-bytevector-n port (bytestructure-descriptor-size <file-header>)) ((? bytevector? bv) (match (file-header-ref bv signature) (#x04034b50 ;local file header (let* ((len (file-header-ref bv file-name-length)) (name (utf8->string (get-bytevector-n port len)))) (set-port-position! port (+ (file-header-ref bv extra-field-length) (file-header-ref bv compressed-size) (port-position port))) (loop (proc name result)))) (#x02014b50 ;central directory record result) (#x06054b50 ;end of central directory record result))) ((? eof-object?) result))))

Julien Lepiller wrote on 17 Feb 2019 08:42

Recipients:(address . 34223@debbugs.gnu.org)

Message-ID:2223B2BF-B691-45B1-82FE-A781CBFF0CDB@lepiller.eu

Le 16 février 2019 23:35:50 GMT+01:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :

Toggle quote (57 lines)>Hi Tim,
>
>Sorry for the delay!
>
>Tim Gesthuizen <tim.gesthuizen@yahoo.de> skribis:
>
>> as discussed before I have looked into the problems of timestamps in
>the
>> zip files.
>> I looked at the way this is solved in ant-build-system with jar files
>> and thought that this could be done in a more elegant way.
>> Because of this I wrote a simple frontend for LibArchive in C that
>> repacks archives and sets their timestamps to zero and disables
>> compression as it is done in the ant-build-system.
>> Creative as I am the program is called repack.
>> You find a git repository attached with the history of the repack
>program.
>> The attached patches add repack to Guix and use it for pwsafe and the
>> ant-build-system.
>
>Nice work!  It’s great that libarchive doesn’t need to actually extract
>the zip file to operate on it.
>
>Overall I think the approach of factorizing archive-timestamp-resetting
>in one place and using it everywhere (‘ant-build-system’ and all) is
>the
>right thing to do.
>
>However, I’m not sure whether we should introduce a new program for
>this
>purpose.  I believe ‘strip-nondeterminism’¹ (in Perl) by fellow
>Reproducible Builds hackers also addresses this problem, so it may be
>wiser to use it.
>
>But really, since (guix build utils) already implements a significant
>subset of ‘strip-nondeterminism’, it would be even better if could
>avoid
>to shell out to a C or Perl program.
>
>I played a bit with this idea and, as an example, the attached file
>allows you to traverse the list of entries in a zip file (it uses
>‘guile-bytestructures’).  Specifically, you can get the list of file
>names in a zip file by running:
>
>  (call-with-input-file "something.zip"
>    (lambda (port)
>      (fold-entries cons '() port)))
>
>Resetting timestamps should be just as simple.
>
>How about taking this route?
>
>Thanks,
>Ludo’.
>
>¹ https://salsa.debian.org/reproducible-builds/strip-nondeterminism

One of the reasons why we extract jar files is to remove compression, because the content might have store references that would be hidden, so grafting for instance wouldn't work.

Tim Gesthuizen wrote on 18 Feb 2019 21:07

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 34223@debbugs.gnu.org)

Message-ID:87mumshndk.fsf@yahoo.de

Hi Ludo,

Toggle quote (2 lines)

> Sorry for the delay!

No problem! I have very little time anyway.

Toggle quote (12 lines)> Nice work!  It’s great that libarchive doesn’t need to actually extract
> the zip file to operate on it.
>
> Overall I think the approach of factorizing archive-timestamp-resetting
> in one place and using it everywhere (‘ant-build-system’ and all) is the
> right thing to do.
>
> However, I’m not sure whether we should introduce a new program for this
> purpose.  I believe ‘strip-nondeterminism’¹ (in Perl) by fellow
> Reproducible Builds hackers also addresses this problem, so it may be
> wiser to use it.

I also think so. If there is already another program that does the job

we should probably use it.

Toggle quote (17 lines)> But really, since (guix build utils) already implements a significant
> subset of ‘strip-nondeterminism’, it would be even better if could avoid
> to shell out to a C or Perl program.
>
> I played a bit with this idea and, as an example, the attached file
> allows you to traverse the list of entries in a zip file (it uses
> ‘guile-bytestructures’).  Specifically, you can get the list of file
> names in a zip file by running:
>
>   (call-with-input-file "something.zip"
>     (lambda (port)
>       (fold-entries cons '() port)))
>
> Resetting timestamps should be just as simple.
>
> How about taking this route?

I also thought about taking this route.
There are some problems with it though:

- As Julien pointed out, the archive contents need to be uncompressed.
  This makes the problem much more complex and keeps us from writing
  a partial ZIP parser that replaces the timestamps in place.
- While it would be quite elegant to just implement the parser in
  Scheme it would be redundant. After all we are developing a package
  manager so we should use it.
  This approach would be more attractive if there would be a Guile
  library for this.
  The best solution would be creating a proper library for handling
  archives when going with Scheme.
- Maintaining a ZIP parser in Guix is a burden we should not take.
- We need to care about a lot of details (ZIP64, probably more exotic
  extensions).

I would be fine with writing an own parser in Scheme but I would like to
point out that in every other place in Guix we are using external tools
for handling archives (AFAIK).

I am not quite sure which version would be the best, so I am open for
other opinions on this.
Maybe you could rephrase your position taking the compression problem
into consideration.

Tim.

Ludovic Courtès wrote on 18 Feb 2019 23:24

Recipients:(name . Tim Gesthuizen)(address . tim.gesthuizen@yahoo.de)(address . 34223@debbugs.gnu.org)

Message-ID:87pnro91mc.fsf@gnu.org

Hello,

Tim Gesthuizen <tim.gesthuizen@yahoo.de> skribis:

Toggle quote (20 lines)>> I played a bit with this idea and, as an example, the attached file
>> allows you to traverse the list of entries in a zip file (it uses
>> ‘guile-bytestructures’).  Specifically, you can get the list of file
>> names in a zip file by running:
>>
>>   (call-with-input-file "something.zip"
>>     (lambda (port)
>>       (fold-entries cons '() port)))
>>
>> Resetting timestamps should be just as simple.
>>
>> How about taking this route?
>
> I also thought about taking this route.
> There are some problems with it though:
>
> - As Julien pointed out, the archive contents need to be uncompressed.
>   This makes the problem much more complex and keeps us from writing
>   a partial ZIP parser that replaces the timestamps in place.

True, I had overlooked that.  In that case, we should definitely unpack
and repack using the ‘zip’ package (I wasn’t suggesting to write a
complete ‘zip’ implementation; I do think it would be valuable in the
long term, but it’s a project for another time, no question here.)

In that case though, it probably doesn’t buy us much to use libarchive
in a separate C program, WDYT?  Should we just stick to the current
approach that invokes ‘unzip’ and ‘zip’?

Thanks,
Ludo’.

Tim Gesthuizen wrote on 2 Mar 2019 00:23

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 34223@debbugs.gnu.org)

Message-ID:87lg1yw559.fsf@yahoo.de

Hi Ludo,

Sorry for the delay!

Ludovic Courtès writes:

Toggle quote (4 lines)

> In that case though, it probably doesn’t buy us much to use libarchive

> in a separate C program, WDYT? Should we just stick to the current

> approach that invokes ‘unzip’ and ‘zip’?

This seems to be the best choice.

Maybe we want to reevaluate when there is a proper ZIP-library for

guile.

I have attached patches that isolate repack-archive from the

ant-build-system and use it for pwsafe.

I only builded some java packages so I don't know if something

else might be broken because of the changes.

Tim.

Attachment: 0001-guix-Generalize-zip-repacking.patch

From 0bb0420dfdeb992b3ceafc815d42e6f403520b8d Mon Sep 17 00:00:00 2001
From: Tim Gesthuizen <tim.gesthuizen@yahoo.de>
Date: Sat, 2 Mar 2019 00:10:19 +0100
Subject: [PATCH 2/2] gnu: pwsafe: Make zip archives deterministic

* gnu/packages/password-utils.scm (pwsafe):
  [native-inputs]: Add unzip.
  [arguments]: Add a phase for resetting zip timestamps.
---
 gnu/packages/password-utils.scm | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

Toggle diff (51 lines)diff --git a/gnu/packages/password-utils.scm b/gnu/packages/password-utils.scm
index 9fd5a6ff0..52870050e 100644
--- a/gnu/packages/password-utils.scm
+++ b/gnu/packages/password-utils.scm
@@ -21,7 +21,7 @@
 ;;; Copyright © 2018 Arun Isaac <arunisaac@systemreboot.net>
 ;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>
 ;;; Copyright © 2018 Amirouche Boubekki <amirouche@hypermove.net>
-;;; Copyright © 2018 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
+;;; Copyright © 2018, 2019 Tim Gesthuizen <tim.gesthuizen@yahoo.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -190,7 +190,8 @@ algorithms AES or Twofish.")
     (build-system cmake-build-system)
     (native-inputs `(("gettext" ,gettext-minimal)
                      ("perl" ,perl)
-                     ("zip" ,zip)))
+                     ("zip" ,zip)
+                     ("unzip" ,unzip)))
     (inputs `(("curl" ,curl)
               ("file" ,file)
               ("gtest" ,googletest)
@@ -201,7 +202,10 @@ algorithms AES or Twofish.")
               ("qrencode" ,qrencode)
               ("wxwidgets" ,wxwidgets)
               ("xerces-c" ,xerces-c)))
-    (arguments '(#:configure-flags (list "-DNO_GTEST=YES")
+    (arguments `(#:configure-flags (list "-DNO_GTEST=YES")
+                 #:imported-modules ((guix build syscalls)
+                                     (guix build archives)
+                                     ,@%cmake-build-system-modules)
                  #:phases (modify-phases %standard-phases
                             (add-after 'unpack 'add-gtest
                               (lambda* (#:key inputs #:allow-other-keys)
@@ -219,6 +223,13 @@ add_subdirectory(src/test)\n" cmake-port)
                                   (("/usr/bin/file")
                                    (string-append (assoc-ref inputs "file")
                                                   "/bin/file")))
+                                #t))
+                            (add-after 'install 'repack-archives
+                              (lambda* (#:key outputs #:allow-other-keys)
+                                (use-modules ((guix build archives)))
+                                (for-each repack-archive
+                                          (find-files (assoc-ref outputs "out")
+                                                      "\\.zip$"))
                                 #t)))))
     (synopsis "Password safe with automatic input and key generation")
     (description "pwsafe is a password manager originally designed by Bruce
-- 
2.20.1

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 34223@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 34223

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date