Make 'guix pack -f docker' tarballs reproducible?

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Simon Josefsson
Owner
unassigned
Submitted by
Simon Josefsson
Severity
normal
S
S
Simon Josefsson wrote on 25 Dec 2024 18:10
(address . bug-guix@gnu.org)
87msgjofih.fsf@kaka.sjd.se
Hi

I am creating docker archives using:

guix pack guix bash-minimal coreutils-minimal net-base --save-provenance -S /bin=bin -S /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2

To my surprise the output was not reproducible between re-runs.

The reason is because of the timestamp and ownership information in the
outer tarball. The internals are identical and reproducible. See
diffoscope output below.

I tried to work around it by wrapping either the 'guix pack' or
'guix-daemon' commands with this environment variable, which I suggest
for inspiration as additional parameters to tar:

TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --sort=name --mode=go+u,go-w --mtime=@0"

I would prefer 'guix pack' produced reproducible archives by default.

Alternatively, provide a way to allow me as user to specify some
parameters for 'guix pack' to make that happen.

/Simon

jas@kaka:~/src/guix-container$ diffoscope stage1-docker-pack.tar.gz-1 stage1-docker-pack.tar.gz-2
--- stage1-docker-pack.tar.gz-1
+++ stage1-docker-pack.tar.gz-2
? --- stage1-docker-pack.tar.gz-1-content
??? +++ stage1-docker-pack.tar.gz-2-content
? ??? file list
? ? @@ -1,10 +1,10 @@
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 421457920 2024-12-25 16:31:15.000000 sha256:e69812bf459ea0fba42d1d6fd518410a4e588ddd4e4c007ddb4dd48c9c04293a/layer.tar
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 56330240 2024-12-25 16:31:16.000000 sha256:45e67bf9fcad2f255f20dc614224b9e4260da1b63f2a361c2479e1ed64a9210a/layer.tar
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 37632000 2024-12-25 16:31:16.000000 sha256:a8d1b46be57ba5a41051dedcf2d8d7bb2f13a9d58078729a962d04f5178274ba/layer.tar
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 41523200 2024-12-25 16:31:16.000000 sha256:0756f500c123ba4f34cda21e5232932799fd36c15243f7fcb1ef38ff6ec7533d/layer.tar
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 37806080 2024-12-25 16:31:17.000000 sha256:bf18d11d88b81af3f6fb49b7d4b092d479b7967ac8dc4980cc381170997c6ccf/layer.tar
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 17582080 2024-12-25 16:31:17.000000 sha256:9263a9904763737f9e8bdf08ca52cede34c2fa9e99abe7f9ef273111752cb2ca/layer.tar
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 147763200 2024-12-25 16:31:20.000000 sha256:3d9a70bc298db46d9fdd95badacd3ec5586f3965110bb85b748be6bcfc57b171/layer.tar
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 10240 2024-12-25 16:31:14.000000 sha256:3fb6718bc797283e8283fe1b843596ace2e62db47d5b38d228a64a6bbb7c3564/layer.tar
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 736 2024-12-25 16:31:21.000000 manifest.json
? ? --rw-r--r-- 0 nixbld (997) nixbld (999) 842 2024-12-25 16:31:21.000000 config.json
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 421457920 2024-12-25 16:41:20.000000 sha256:e69812bf459ea0fba42d1d6fd518410a4e588ddd4e4c007ddb4dd48c9c04293a/layer.tar
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 56330240 2024-12-25 16:41:21.000000 sha256:45e67bf9fcad2f255f20dc614224b9e4260da1b63f2a361c2479e1ed64a9210a/layer.tar
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 37632000 2024-12-25 16:41:22.000000 sha256:a8d1b46be57ba5a41051dedcf2d8d7bb2f13a9d58078729a962d04f5178274ba/layer.tar
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 41523200 2024-12-25 16:41:22.000000 sha256:0756f500c123ba4f34cda21e5232932799fd36c15243f7fcb1ef38ff6ec7533d/layer.tar
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 37806080 2024-12-25 16:41:22.000000 sha256:bf18d11d88b81af3f6fb49b7d4b092d479b7967ac8dc4980cc381170997c6ccf/layer.tar
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 17582080 2024-12-25 16:41:23.000000 sha256:9263a9904763737f9e8bdf08ca52cede34c2fa9e99abe7f9ef273111752cb2ca/layer.tar
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 147763200 2024-12-25 16:41:25.000000 sha256:3d9a70bc298db46d9fdd95badacd3ec5586f3965110bb85b748be6bcfc57b171/layer.tar
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 10240 2024-12-25 16:41:19.000000 sha256:3fb6718bc797283e8283fe1b843596ace2e62db47d5b38d228a64a6bbb7c3564/layer.tar
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 736 2024-12-25 16:41:26.000000 manifest.json
? ? +-rw-r--r-- 0 nixbld (997) nixbld (999) 842 2024-12-25 16:41:26.000000 config.json
jas@kaka:~/src/guix-container$
-----BEGIN PGP SIGNATURE-----

iIoEARYIADIWIQSjzJyHC50xCrrUzy9RcisI/kdFogUCZ2w8dhQcc2ltb25Aam9z
ZWZzc29uLm9yZwAKCRBRcisI/kdFolS9AP9SiacOwwv/Ljjy5xRtSr7oLC/qTI4N
bU55c/QBqV2EcAD/V4CFBFN63O7OFRDIQ100CYhotoRUnF7IQ+Pme7XPnQU=
=mjf4
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 7 Jan 23:57 +0100
(name . Simon Josefsson)(address . simon@josefsson.org)(address . 75090@debbugs.gnu.org)
87jzb6xme2.fsf@gnu.org
Hi Simon,

Simon Josefsson <simon@josefsson.org> skribis:

Toggle quote (18 lines)
> I am creating docker archives using:
>
> guix pack guix bash-minimal coreutils-minimal net-base --save-provenance -S /bin=bin -S /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
>
> To my surprise the output was not reproducible between re-runs.
>
> The reason is because of the timestamp and ownership information in the
> outer tarball. The internals are identical and reproducible. See
> diffoscope output below.
>
> I tried to work around it by wrapping either the 'guix pack' or
> 'guix-daemon' commands with this environment variable, which I suggest
> for inspiration as additional parameters to tar:
>
> TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --sort=name --mode=go+u,go-w --mtime=@0"
>
> I would prefer 'guix pack' produced reproducible archives by default.

Indeed. I sent a fix based on your suggestion:

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote 7 days ago
Re: [bug#75426] [PATCH] docker: Build tarballs reproducibly.
87wmejbyla.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (9 lines)
>
> * guix/docker.scm (tar): New procedure.
> (create-empty-tar, build-docker-image): Use it instead of calling
> ‘invoke’ directly.
>
> Reported-by: Simon Josefsson <simon@josefsson.org>
> Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609

Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.

Ludo’.
Closed
S
S
Simon Josefsson wrote 43 hours ago
7d999f7dfcfe6c2321e2456f60d43d30715b25f1.camel@josefsson.org
Hi! I suspect something went wrong with this patch, now 'guix pack'
fail and give the error below. Maybe the 'cf' has to come first?


/Simon

tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
label' options
Try 'tar --help' or 'tar --usage' for more information.
Backtrace:
7 (primitive-load "/gnu/store/hyx3flr5r251fc3x0z0l6r36159?")
In guix/docker.scm:
387:6 6 (build-docker-image "/gnu/store/vwia06dwxrsmf152spa6n2?"
?)
In ice-9/ports.scm:
433:17 5 (call-with-output-file _ _ #:binary _ #:encoding _)
476:4 4 (_ _)
In guix/docker.scm:
277:15 3 (_)
In srfi/srfi-1.scm:
586:17 2 (map1 ("/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-?" ?))
In guix/docker.scm:
279:18 1 (_ "/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-guix-1.?")
In guix/build/utils.scm:
822:6 0 (invoke "tar" "--mtime=@1" "--owner=0" "--group=0" "--?"
?)

guix/build/utils.scm:822:6: In procedure invoke:
ERROR:
1. &invoke-error:
program: "tar"
arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
"/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-guix-1.4.0-31.121e96d")
exit-status: 2
term-signal: #f
stop-signal: #f

lör 2025-01-25 klockan 00:07 +0100 skrev Ludovic Courtès:
Toggle quote (14 lines)
> Ludovic Courtès <ludo@gnu.org> skribis:
>
> > Fixes <https://issues.guix.gnu.org/75090>.
> >
> > * guix/docker.scm (tar): New procedure.
> > (create-empty-tar, build-docker-image): Use it instead of calling
> > ‘invoke’ directly.
> >
> > Reported-by: Simon Josefsson <simon@josefsson.org>
> > Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
>
> Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.
>
> Ludo’.
-----BEGIN PGP SIGNATURE-----

iQNTBAAWCAL7FiEEo8ychwudMQq61M8vUXIrCP5HRaIFAmeaezDCHCYAmDMEXJLO
tBYJKwYBBAHaRw8BAQdACIcrZIvhrxDBkK9fV+QlTmXxo2naObDuGtw58YaxlOu0
JVNpbW9uIEpvc2Vmc3NvbiA8c2ltb25Aam9zZWZzc29uLm9yZz6IlgQTFggAPgIb
AwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgBYhBLHSvRN1vst4TPT4xNc89jjFPAa+
BQJl/YgIBQkLehFUAAoJENc89jjFPAa+CboA+wUa06RD5e5VTCxvSWtPS75Wq2qB
eYGZnf0jvUMxa2n4AP4xkUeAPPnNuMsTm2fsFCDIGaEM2Yn6Vb2huzzT1Fw/BLgz
BFySz4EWCSsGAQQB2kcPAQEHQOxTCIOaeXAxI2hIX4HK9bQTpNVei708oNr1Klm8
qCGKiPUEGBYIACYCGwIWIQSx0r0Tdb7LeEz0+MTXPPY4xTwGvgUCZf2IKwUJC3oQ
qgCBdiAEGRYIAB0WIQSjzJyHC50xCrrUzy9RcisI/kdFogUCXJLPgQAKCRBRcisI
/kdFoqdMAQCgH45aseZgIrwKOvUOA9QfsmeE8GZHYNuFHmM9FEQS6AD6A4x5aYvo
Y6lo98pgtw2HPDhmcCXFItjXCrV4A0GmJA4JENc89jjFPAa+GcYA/26YQY05bLtn
XiIjTiAzrGQrRXxTHPA8Av7TDFHvIetWAP9sHSoU8OfTwmTiEnGwLlsV7QJclZg3
YNz/Ypcp9TqQBrg4BFySz2oSCisGAQQBl1UBBQEBB0AxlRumDW6nZY7A+VCfek9V
pEx6PJmdJyYPt3lNHMd6HAMBCAeIfgQYFggAJgIbDBYhBLHSvRN1vst4TPT4xNc8
9jjFPAa+BQJl/YgwBQkLehDGAAoJENc89jjFPAa+phoA/jrDqIrl/55vUMBhIQv+
TP635d2iCTEnyFmbUcP9+gh6APoDsXalVd2cOGxQtSC+TF8PkZMn1TLkJKAjVxr+
xx40AgAKCRBRcisI/kdFon8rAQDjyuNawr7l9rVVvvJF1/v9Ys8YEAguZBykLjXr
AlsSSQEAtr4WUwktnqhpoYgQ5/7RE9jq/sIOWEo1C42H9HzUtAE=
=Bo2J
-----END PGP SIGNATURE-----


?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 75090@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 75090
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch