Make 'guix pack -f docker' tarballs reproducible?

Done

Details

2 participants

Ludovic Courtès
Simon Josefsson

Owner: unassigned

Submitted by: Simon Josefsson

Severity: normal

Debbugs page

Simon Josefsson wrote 3 months ago

Recipients:(address . bug-guix@gnu.org)

Message-ID:87msgjofih.fsf@kaka.sjd.se

Hi

I am creating docker archives using:

guix pack guix bash-minimal coreutils-minimal net-base --save-provenance -S /bin=bin -S /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2

To my surprise the output was not reproducible between re-runs.

The reason is because of the timestamp and ownership information in the
outer tarball.  The internals are identical and reproducible.  See
diffoscope output below.

I tried to work around it by wrapping either the 'guix pack' or
'guix-daemon' commands with this environment variable, which I suggest
for inspiration as additional parameters to tar:

TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --sort=name --mode=go+u,go-w --mtime=@0"

I would prefer 'guix pack' produced reproducible archives by default.

Alternatively, provide a way to allow me as user to specify some
parameters for 'guix pack' to make that happen.

/Simon

jas@kaka:~/src/guix-container$ diffoscope stage1-docker-pack.tar.gz-1 stage1-docker-pack.tar.gz-2 
--- stage1-docker-pack.tar.gz-1
+++ stage1-docker-pack.tar.gz-2
│   --- stage1-docker-pack.tar.gz-1-content
├── +++ stage1-docker-pack.tar.gz-2-content
│ ├── file list
│ │ @@ -1,10 +1,10 @@
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999) 421457920 2024-12-25 16:31:15.000000 sha256:e69812bf459ea0fba42d1d6fd518410a4e588ddd4e4c007ddb4dd48c9c04293a/layer.tar
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999) 56330240 2024-12-25 16:31:16.000000 sha256:45e67bf9fcad2f255f20dc614224b9e4260da1b63f2a361c2479e1ed64a9210a/layer.tar
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999) 37632000 2024-12-25 16:31:16.000000 sha256:a8d1b46be57ba5a41051dedcf2d8d7bb2f13a9d58078729a962d04f5178274ba/layer.tar
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999) 41523200 2024-12-25 16:31:16.000000 sha256:0756f500c123ba4f34cda21e5232932799fd36c15243f7fcb1ef38ff6ec7533d/layer.tar
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999) 37806080 2024-12-25 16:31:17.000000 sha256:bf18d11d88b81af3f6fb49b7d4b092d479b7967ac8dc4980cc381170997c6ccf/layer.tar
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999) 17582080 2024-12-25 16:31:17.000000 sha256:9263a9904763737f9e8bdf08ca52cede34c2fa9e99abe7f9ef273111752cb2ca/layer.tar
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999) 147763200 2024-12-25 16:31:20.000000 sha256:3d9a70bc298db46d9fdd95badacd3ec5586f3965110bb85b748be6bcfc57b171/layer.tar
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999)    10240 2024-12-25 16:31:14.000000 sha256:3fb6718bc797283e8283fe1b843596ace2e62db47d5b38d228a64a6bbb7c3564/layer.tar
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999)      736 2024-12-25 16:31:21.000000 manifest.json
│ │ --rw-r--r--   0 nixbld     (997) nixbld     (999)      842 2024-12-25 16:31:21.000000 config.json
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999) 421457920 2024-12-25 16:41:20.000000 sha256:e69812bf459ea0fba42d1d6fd518410a4e588ddd4e4c007ddb4dd48c9c04293a/layer.tar
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999) 56330240 2024-12-25 16:41:21.000000 sha256:45e67bf9fcad2f255f20dc614224b9e4260da1b63f2a361c2479e1ed64a9210a/layer.tar
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999) 37632000 2024-12-25 16:41:22.000000 sha256:a8d1b46be57ba5a41051dedcf2d8d7bb2f13a9d58078729a962d04f5178274ba/layer.tar
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999) 41523200 2024-12-25 16:41:22.000000 sha256:0756f500c123ba4f34cda21e5232932799fd36c15243f7fcb1ef38ff6ec7533d/layer.tar
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999) 37806080 2024-12-25 16:41:22.000000 sha256:bf18d11d88b81af3f6fb49b7d4b092d479b7967ac8dc4980cc381170997c6ccf/layer.tar
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999) 17582080 2024-12-25 16:41:23.000000 sha256:9263a9904763737f9e8bdf08ca52cede34c2fa9e99abe7f9ef273111752cb2ca/layer.tar
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999) 147763200 2024-12-25 16:41:25.000000 sha256:3d9a70bc298db46d9fdd95badacd3ec5586f3965110bb85b748be6bcfc57b171/layer.tar
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999)    10240 2024-12-25 16:41:19.000000 sha256:3fb6718bc797283e8283fe1b843596ace2e62db47d5b38d228a64a6bbb7c3564/layer.tar
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999)      736 2024-12-25 16:41:26.000000 manifest.json
│ │ +-rw-r--r--   0 nixbld     (997) nixbld     (999)      842 2024-12-25 16:41:26.000000 config.json
jas@kaka:~/src/guix-container$

-----BEGIN PGP SIGNATURE-----

iIoEARYIADIWIQSjzJyHC50xCrrUzy9RcisI/kdFogUCZ2w8dhQcc2ltb25Aam9z

ZWZzc29uLm9yZwAKCRBRcisI/kdFolS9AP9SiacOwwv/Ljjy5xRtSr7oLC/qTI4N

bU55c/QBqV2EcAD/V4CFBFN63O7OFRDIQ100CYhotoRUnF7IQ+Pme7XPnQU=

=mjf4

-----END PGP SIGNATURE-----

Ludovic Courtès wrote 2 months ago

Recipients:(name . Simon Josefsson)(address . simon@josefsson.org)(address . 75090@debbugs.gnu.org)

Message-ID:87jzb6xme2.fsf@gnu.org

Hi Simon,

Simon Josefsson <simon@josefsson.org> skribis:

Toggle quote (18 lines)> I am creating docker archives using:
>
> guix pack guix bash-minimal coreutils-minimal net-base --save-provenance -S /bin=bin -S /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
>
> To my surprise the output was not reproducible between re-runs.
>
> The reason is because of the timestamp and ownership information in the
> outer tarball.  The internals are identical and reproducible.  See
> diffoscope output below.
>
> I tried to work around it by wrapping either the 'guix pack' or
> 'guix-daemon' commands with this environment variable, which I suggest
> for inspiration as additional parameters to tar:
>
> TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --sort=name --mode=go+u,go-w --mtime=@0"
>
> I would prefer 'guix pack' produced reproducible archives by default.

Indeed. I sent a fix based on your suggestion:

https://issues.guix.gnu.org/75426.

Thanks,

Ludo’.

Ludovic Courtès wrote 2 months ago

Re: [bug#75426] [PATCH] docker: Build tarballs reproducibly.

Recipients:

Message-ID:87wmejbyla.fsf@gnu.org

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (9 lines)

> Fixes https://issues.guix.gnu.org/75090.

> * guix/docker.scm (tar): New procedure.

> (create-empty-tar, build-docker-image): Use it instead of calling

> ‘invoke’ directly.

> Reported-by: Simon Josefsson <simon@josefsson.org>

> Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609

Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.

Ludo’.

Closed

Simon Josefsson wrote 1 months ago

Recipients:

Message-ID:7d999f7dfcfe6c2321e2456f60d43d30715b25f1.camel@josefsson.org

Hi!  I suspect something went wrong with this patch, now 'guix pack'
fail and give the error below.  Maybe the 'cf' has to come first?

https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317

/Simon

tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
label' options
Try 'tar --help' or 'tar --usage' for more information.
Backtrace:
           7 (primitive-load "/gnu/store/hyx3flr5r251fc3x0z0l6r36159?")
In guix/docker.scm:
    387:6  6 (build-docker-image "/gnu/store/vwia06dwxrsmf152spa6n2?"
?)
In ice-9/ports.scm:
   433:17  5 (call-with-output-file _ _ #:binary _ #:encoding _)
    476:4  4 (_ _)
In guix/docker.scm:
   277:15  3 (_)
In srfi/srfi-1.scm:
   586:17  2 (map1 ("/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-?" ?))
In guix/docker.scm:
   279:18  1 (_ "/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-guix-1.?")
In guix/build/utils.scm:
    822:6  0 (invoke "tar" "--mtime=@1" "--owner=0" "--group=0" "--?"
?)

guix/build/utils.scm:822:6: In procedure invoke:
ERROR:
  1. &invoke-error:
      program: "tar"
      arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
"/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-guix-1.4.0-31.121e96d")
      exit-status: 2
      term-signal: #f
      stop-signal: #f

lör 2025-01-25 klockan 00:07 +0100 skrev Ludovic Courtès:

Toggle quote (14 lines)> Ludovic Courtès <ludo@gnu.org> skribis:
> 
> > Fixes <https://issues.guix.gnu.org/75090>.
> > 
> > * guix/docker.scm (tar): New procedure.
> > (create-empty-tar, build-docker-image): Use it instead of calling
> > ‘invoke’ directly.
> > 
> > Reported-by: Simon Josefsson <simon@josefsson.org>
> > Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
> 
> Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.
> 
> Ludo’.

Ludovic Courtès wrote 1 months ago

Recipients:(name . Simon Josefsson)(address . simon@josefsson.org)

Message-ID:87ed0jc680.fsf@gnu.org

Hi,

Simon Josefsson <simon@josefsson.org> skribis:

Toggle quote (8 lines)

> Hi! I suspect something went wrong with this patch, now 'guix pack'

> fail and give the error below. Maybe the 'cf' has to come first?

> https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317

> tar: You must specify one of the '-Acdtrux', '--delete' or '--test-label' options

> Try 'tar --help' or 'tar --usage' for more information.

Hmm I don’t see this message in the GitLab log above, and I cannot

reproduce the problem:

Toggle snippet (8 lines)

$ ./pre-inst-env guix pack -f docker sed

/gnu/store/h2hdp469v3014b82qsvz5fkw00sfxdgh-sed-docker-pack.tar.gz

$ git log |head -3

commit 97fb1887ad10000c067168176c504274e29e4430

Author: Ashish SHUKLA <ashish.is@lostca.se>

Date: Mon Jan 20 21:46:10 2025 +0000

Could you try to come up with a command and commit that reproduces it?

Thanks,

Ludo’.

Simon Josefsson wrote 1 months ago

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:8c6aa44b639b28a35ecb6c9d1bc393b734e17f55.camel@josefsson.org

fre 2025-01-31 klockan 17:12 +0100 skrev Ludovic Courtès:

Toggle quote (29 lines)> Hi,
> 
> Simon Josefsson <simon@josefsson.org> skribis:
> 
> > Hi!  I suspect something went wrong with this patch, now 'guix
> > pack'
> > fail and give the error below.  Maybe the 'cf' has to come first?
> > 
> > https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317
> 
> > tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
> > label' options
> > Try 'tar --help' or 'tar --usage' for more information.
> 
> Hmm I don’t see this message in the GitLab log above, and I cannot
> reproduce the problem:
> 
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix pack -f docker sed
> /gnu/store/h2hdp469v3014b82qsvz5fkw00sfxdgh-sed-docker-pack.tar.gz
> $ git log |head -3
> commit 97fb1887ad10000c067168176c504274e29e4430
> Author: Ashish SHUKLA <ashish.is@lostca.se>
> Date:   Mon Jan 20 21:46:10 2025 +0000
> --8<---------------cut here---------------end--------------->8---
> 
> Could you try to come up with a command and commit that reproduces
> it?

Running the commands in the log like below on my local trisquel machine
with guix triggers the same problem.  Any ideas?

(FWIW, the message is hidden inside the log, but the error and filename
of the log is printed in the gitlab log above, same as below; it is
possible to click on 'Job artifacts' to find the log files.)

/Simon

jas@kaka:~$ guix describe
Generation 26	31 jan 2025 23:37:27	(aktuell)
  guix d48da2d
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: d48da2d21610f9cf5f76cd846703b12beedb1fd5
jas@kaka:~$ guix pack coreutils --save-provenance -S /bin=bin -S
/share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
net-base
The following derivation will be built:
  /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-base-
docker-pack.tar.gz.drv
bygger /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-base-
docker-pack.tar.gz.drv…
|builder for `/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-
net-base-docker-pack.tar.gz.drv' failed with exit code 1
bygge av /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-
base-docker-pack.tar.gz.drv misslyckades
Granska bygglogg vid
”/var/log/guix/drvs/qk/z6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-
base-docker-pack.tar.gz.drv.bz2”.
guix pack: fel: build of `/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-
coreutils-net-base-docker-pack.tar.gz.drv' failed
jas@kaka:~$ bzcat /var/log/guix/drvs/qk/z6wc1qq23ah1xk387givjvk9qlgwcn-
coreutils-net-base-docker-pack.tar.gz.drv.bz2 | tail -30
tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
label' options
Try 'tar --help' or 'tar --usage' for more information.
Backtrace:
           7 (primitive-load "/gnu/store/5z5322v39y5mwninj36m877zgsx?")
In guix/docker.scm:
    387:6  6 (build-docker-image "/gnu/store/8ac4lljjiqp3a7zydh6l9v?"
?)
In ice-9/ports.scm:
   433:17  5 (call-with-output-file _ _ #:binary _ #:encoding _)
    476:4  4 (_ _)
In guix/docker.scm:
   277:15  3 (_)
In srfi/srfi-1.scm:
   586:17  2 (map1 ("/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-?" ?))
In guix/docker.scm:
   279:18  1 (_ "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2?")
In guix/build/utils.scm:
    822:6  0 (invoke "tar" "--mtime=@1" "--owner=0" "--group=0" "--?"
?)

guix/build/utils.scm:822:6: In procedure invoke:
ERROR:
  1. &invoke-error:
      program: "tar"
      arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
"/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39")
      exit-status: 2
      term-signal: #f
      stop-signal: #f
jas@kaka:~$

Ludovic Courtès wrote 1 months ago

Recipients:(name . Simon Josefsson)(address . simon@josefsson.org)

Message-ID:8734gty908.fsf@gnu.org

Hello Simon,

Simon Josefsson <simon@josefsson.org> skribis:

Toggle quote (4 lines)

> jas@kaka:~$ guix pack coreutils --save-provenance -S /bin=bin -S

> /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2

> net-base

[...]

Toggle quote (8 lines)

> guix/build/utils.scm:822:6: In procedure invoke:

> ERROR:

> 1. &invoke-error:

> program: "tar"

> arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-

> owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"

> "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39")

Fixed in 285a1cb449f60798dc83f7f1016700b4ab2374a8.

Ludo’.

Simon Josefsson wrote 1 months ago

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:87v7tpnzli.fsf@josefsson.org

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (2 lines)

> Fixed in 285a1cb449f60798dc83f7f1016700b4ab2374a8.

Hi! Wonderful, confirmed working. These two pipelines produce

bit-by-bit identical docker pack images for stage1 of my guix gitlab

container images now:

https://gitlab.com/debdistutils/guix/container/-/jobs/9042454862

https://gitlab.com/debdistutils/guix/container/-/jobs/9042230873

58b98623ac2b75de521c8db6a904e60f4faad17dc08c2ccc6eab445a9f77cfdd

/Simon

Your comment

This issue is archived.

To comment on this conversation send an email to 75090@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 75090

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags

mumi command -t +confirmed -t +easy

Or, remove the moreinfo tag and set the help tag

mumi command -t -moreinfo -t +help

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date