[PATCH 0/2] Add Docker layered image for pack and system

  • Open
  • quality assurance status badge
Details
7 participants
  • Greg Hogan
  • Oleg Pykhalov
  • Ludovic Courtès
  • Christopher Baines
  • pelzflorian (Florian Pelz)
  • Ricardo Wurmus
  • Simon Tournier
Owner
unassigned
Submitted by
Oleg Pykhalov
Severity
normal
O
O
Oleg Pykhalov wrote on 13 Mar 01:30 +0100
(address . guix-patches@gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003012.14325-1-go.wigust@gmail.com
Hi Guix,

This patch series add to 'guix pack' and 'guix system image' formats with a
layered Docker image, which dicreases images size by sharing same layers on a
host.

The folling commands show an example for new image formats:

./pre-inst-env guix system image --image-type=docker-layered config.scm
docker load -i result

./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello
docker load -i result

The folloing tests passed:

make check-channel-news
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system docker-layered-system"

The gnu/packages/aux-files/python/stream-layered-image.py Python script is a
copy of github.com/NixOS/nixpkgs/pkgs/build-support/docker/stream_layered_image.py
with only a simple replacement "/nix" to "/gnu" string.

Oleg Pykhalov (2):
guix: docker: Build layered image.
news: Add entry for the new 'docker-layered' distribution format.

Makefile.am | 3 +-
doc/guix.texi | 16 +-
etc/news.scm | 38 ++
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 103 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
11 files changed, 813 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py


base-commit: 60a211ec705ac98483d76da7f2523f2b8966343a
--
2.38.0
O
O
Oleg Pykhalov wrote on 13 Mar 01:33 +0100
[PATCH 1/2] guix: docker: Build layered image.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003310.17129-1-go.wigust@gmail.com
* gnu/packages/aux-files/python/stream-layered-image.py: New file.
* Makefile.am (AUX_FILES): Add this.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* tests/pack.scm: Add docker-layered-image + localstatedir test.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New
variables.
(system-docker-image)[layered-image?]: New argument.
(stream-layered-image.py): New variable.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* gnu/image.scm (validate-image-format)[docker-layered]: New image format.
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image-type Reference): Document docker-layered-image-type.
---
Makefile.am | 3 +-
doc/guix.texi | 16 +-
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 103 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
10 files changed, 775 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py

Toggle diff (518 lines)
diff --git a/Makefile.am b/Makefile.am
index 23b939b674..9aca84f8f8 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -11,7 +11,7 @@
# Copyright © 2017 Arun Isaac <arunisaac@systemreboot.net>
# Copyright © 2018 Nikita <nikita@n0.is>
# Copyright © 2018 Julien Lepiller <julien@lepiller.eu>
-# Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
+# Copyright © 2018, 2023 Oleg Pykhalov <go.wigust@gmail.com>
# Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
@@ -435,6 +435,7 @@ AUX_FILES = \
gnu/packages/aux-files/python/sanity-check.py \
gnu/packages/aux-files/python/sanity-check-next.py \
gnu/packages/aux-files/python/sitecustomize.py \
+ gnu/packages/aux-files/python/stream-layered-image.py \
gnu/packages/aux-files/renpy/renpy.in \
gnu/packages/aux-files/run-in-namespace.c
diff --git a/doc/guix.texi b/doc/guix.texi
index b545751e1b..bd0ee126ee 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@ Copyright @copyright{} 2017 Andy Wingo@*
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6840,9 +6840,15 @@ the following command:
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -43631,6 +43637,10 @@ Build an image based on the @code{iso9660-image} image but with the
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-with-syntax-properties (name (value properties))
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
new file mode 100644
index 0000000000..9ad2168c2d
--- /dev/null
+++ b/gnu/packages/aux-files/python/stream-layered-image.py
@@ -0,0 +1,391 @@
+"""
+This script generates a Docker image from a set of store paths. Uses
+Docker Image Specification v1.2 as reference [1].
+
+It expects a JSON file with the following properties and writes the
+image as an uncompressed tarball to stdout:
+
+* "architecture", "config", "os", "created", "repo_tag" correspond to
+ the fields with the same name on the image spec [2].
+* "created" can be "now".
+* "created" is also used as mtime for files added to the image.
+* "store_layers" is a list of layers in ascending order, where each
+ layer is the list of store paths to include in that layer.
+
+The main challenge for this script to create the final image in a
+streaming fashion, without dumping any intermediate data to disk
+for performance.
+
+A docker image has each layer contents archived as separate tarballs,
+and they later all get enveloped into a single big tarball in a
+content addressed fashion. However, because how "tar" format works,
+we have to know about the name (which includes the checksum in our
+case) and the size of the tarball before we can start adding it to the
+outer tarball. We achieve that by creating the layer tarballs twice;
+on the first iteration we calculate the file size and the checksum,
+and on the second one we actually stream the contents. 'add_layer_dir'
+function does all this.
+
+[1]: https://github.com/moby/moby/blob/master/image/spec/v1.2.md
+[2]: https://github.com/moby/moby/blob/4fb59c20a4fb54f944fe170d0ff1d00eb4a24d6f/image/spec/v1.2.md#image-json-field-descriptions
+""" # noqa: E501
+
+
+import io
+import os
+import re
+import sys
+import json
+import hashlib
+import pathlib
+import tarfile
+import itertools
+import threading
+from datetime import datetime, timezone
+from collections import namedtuple
+
+
+def archive_paths_to(obj, paths, mtime):
+ """
+ Writes the given store paths as a tar file to the given stream.
+
+ obj: Stream to write to. Should have a 'write' method.
+ paths: List of store paths.
+ """
+
+ # gettarinfo makes the paths relative, this makes them
+ # absolute again
+ def append_root(ti):
+ ti.name = "/" + ti.name
+ return ti
+
+ def apply_filters(ti):
+ ti.mtime = mtime
+ ti.uid = 0
+ ti.gid = 0
+ ti.uname = "root"
+ ti.gname = "root"
+ return ti
+
+ def nix_root(ti):
+ ti.mode = 0o0555 # r-xr-xr-x
+ return ti
+
+ def dir(path):
+ ti = tarfile.TarInfo(path)
+ ti.type = tarfile.DIRTYPE
+ return ti
+
+ with tarfile.open(fileobj=obj, mode="w|") as tar:
+ # To be consistent with the docker utilities, we need to have
+ # these directories first when building layer tarballs.
+ tar.addfile(apply_filters(nix_root(dir("/gnu"))))
+ tar.addfile(apply_filters(nix_root(dir("/gnu/store"))))
+
+ for path in paths:
+ path = pathlib.Path(path)
+ if path.is_symlink():
+ files = [path]
+ else:
+ files = itertools.chain([path], path.rglob("*"))
+
+ for filename in sorted(files):
+ ti = append_root(tar.gettarinfo(filename))
+
+ # copy hardlinks as regular files
+ if ti.islnk():
+ ti.type = tarfile.REGTYPE
+ ti.linkname = ""
+ ti.size = filename.stat().st_size
+
+ ti = apply_filters(ti)
+ if ti.isfile():
+ with open(filename, "rb") as f:
+ tar.addfile(ti, f)
+ else:
+ tar.addfile(ti)
+
+
+class ExtractChecksum:
+ """
+ A writable stream which only calculates the final file size and
+ sha256sum, while discarding the actual contents.
+ """
+
+ def __init__(self):
+ self._digest = hashlib.sha256()
+ self._size = 0
+
+ def write(self, data):
+ self._digest.update(data)
+ self._size += len(data)
+
+ def extract(self):
+ """
+ Returns: Hex-encoded sha256sum and size as a tuple.
+ """
+ return (self._digest.hexdigest(), self._size)
+
+
+FromImage = namedtuple("FromImage", ["tar", "manifest_json", "image_json"])
+# Some metadata for a layer
+LayerInfo = namedtuple("LayerInfo", ["size", "checksum", "path", "paths"])
+
+
+def load_from_image(from_image_str):
+ """
+ Loads the given base image, if any.
+
+ from_image_str: Path to the base image archive.
+
+ Returns: A 'FromImage' object with references to the loaded base image,
+ or 'None' if no base image was provided.
+ """
+ if from_image_str is None:
+ return None
+
+ base_tar = tarfile.open(from_image_str)
+
+ manifest_json_tarinfo = base_tar.getmember("manifest.json")
+ with base_tar.extractfile(manifest_json_tarinfo) as f:
+ manifest_json = json.load(f)
+
+ image_json_tarinfo = base_tar.getmember(manifest_json[0]["Config"])
+ with base_tar.extractfile(image_json_tarinfo) as f:
+ image_json = json.load(f)
+
+ return FromImage(base_tar, manifest_json, image_json)
+
+
+def add_base_layers(tar, from_image):
+ """
+ Adds the layers from the given base image to the final image.
+
+ tar: 'tarfile.TarFile' object for new layers to be added to.
+ from_image: 'FromImage' object with references to the loaded base image.
+ """
+ if from_image is None:
+ print("No 'fromImage' provided", file=sys.stderr)
+ return []
+
+ layers = from_image.manifest_json[0]["Layers"]
+ checksums = from_image.image_json["rootfs"]["diff_ids"]
+ layers_checksums = zip(layers, checksums)
+
+ for num, (layer, checksum) in enumerate(layers_checksums, start=1):
+ layer_tarinfo = from_image.tar.getmember(layer)
+ checksum = re.sub(r"^sha256:", "", checksum)
+
+ tar.addfile(layer_tarinfo, from_image.tar.extractfile(layer_tarinfo))
+ path = layer_tarinfo.path
+ size = layer_tarinfo.size
+
+ print("Adding base layer", num, "from", path, file=sys.stderr)
+ yield LayerInfo(size=size, checksum=checksum, path=path, paths=[path])
+
+ from_image.tar.close()
+
+
+def overlay_base_config(from_image, final_config):
+ """
+ Overlays the final image 'config' JSON on top of selected defaults from the
+ base image 'config' JSON.
+
+ from_image: 'FromImage' object with references to the loaded base image.
+ final_config: 'dict' object of the final image 'config' JSON.
+ """
+ if from_image is None:
+ return final_config
+
+ base_config = from_image.image_json["config"]
+
+ # Preserve environment from base image
+ final_env = base_config.get("Env", []) + final_config.get("Env", [])
+ if final_env:
+ # Resolve duplicates (last one wins) and format back as list
+ resolved_env = {entry.split("=", 1)[0]: entry for entry in final_env}
+ final_config["Env"] = list(resolved_env.values())
+ return final_config
+
+
+def add_layer_dir(tar, paths, store_dir, mtime):
+ """
+ Appends given store paths to a TarFile object as a new layer.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ paths: List of store paths.
+ store_dir: the root directory of the nix store
+ mtime: 'mtime' of the added files and the layer tarball.
+ Should be an integer representing a POSIX time.
+
+ Returns: A 'LayerInfo' object containing some metadata of
+ the layer added.
+ """
+
+ invalid_paths = [i for i in paths if not i.startswith(store_dir)]
+ assert len(invalid_paths) == 0, \
+ f"Expecting absolute paths from {store_dir}, but got: {invalid_paths}"
+
+ # First, calculate the tarball checksum and the size.
+ extract_checksum = ExtractChecksum()
+ archive_paths_to(
+ extract_checksum,
+ paths,
+ mtime=mtime,
+ )
+ (checksum, size) = extract_checksum.extract()
+
+ path = f"{checksum}/layer.tar"
+ layer_tarinfo = tarfile.TarInfo(path)
+ layer_tarinfo.size = size
+ layer_tarinfo.mtime = mtime
+
+ # Then actually stream the contents to the outer tarball.
+ read_fd, write_fd = os.pipe()
+ with open(read_fd, "rb") as read, open(write_fd, "wb") as write:
+ def producer():
+ archive_paths_to(
+ write,
+ paths,
+ mtime=mtime,
+ )
+ write.close()
+
+ # Closing the write end of the fifo also closes the read end,
+ # so we don't need to wait until this thread is finished.
+ #
+ # Any exception from the thread will get printed by the default
+ # exception handler, and the 'addfile' call will fail since it
+ # won't be able to read required amount of bytes.
+ threading.Thread(target=producer).start()
+ tar.addfile(layer_tarinfo, read)
+
+ return LayerInfo(size=size, checksum=checksum, path=path, paths=paths)
+
+
+def add_customisation_layer(target_tar, customisation_layer, mtime):
+ """
+ Adds the customisation layer as a new layer. This is layer is structured
+ differently; given store path has the 'layer.tar' and corresponding
+ sha256sum ready.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ customisation_layer: Path containing the layer archive.
+ mtime: 'mtime' of the added layer tarball.
+ """
+
+ checksum_path = os.path.join(customisation_layer, "checksum")
+ with open(checksum_path) as f:
+ checksum = f.read().strip()
+ assert len(checksum) == 64, f"Invalid sha256 at ${checksum_path}."
+
+ layer_path = os.path.join(customisation_layer, "layer.tar")
+
+ path = f"{checksum}/layer.tar"
+ tarinfo = target_tar.gettarinfo(layer_path)
+ tarinfo.name = path
+ tarinfo.mtime = mtime
+
+ with open(layer_path, "rb") as f:
+ target_tar.addfile(tarinfo, f)
+
+ return LayerInfo(
+ size=None,
+ checksum=checksum,
+ path=path,
+ paths=[customisation_layer]
+ )
+
+
+def add_bytes(tar, path, content, mtime):
+ """
+ Adds a file to the tarball with given path and contents.
+
+ tar: 'tarfile.TarFile' object.
+ path: Path of the file as a string.
+ content: Contents of the file.
+ mtime: 'mtime' of the file. Should be an integer representing a POSIX time.
+ """
+ assert type(content) is bytes
+
+ ti = tarfile.TarInfo(path)
+ ti.size = len(content)
+ ti.mtime = mtime
+ tar.addfile(ti, io.BytesIO(content))
+
+
+def main():
+ with open(sys.argv[1], "r") as f:
+ conf = json.load(f)
+
+ created = (
+ datetime.now(tz=timezone.utc)
+ if conf["created"] == "now"
+ else datetime.fromisoformat(conf["created"])
+ )
+ mtime = int(created.timestamp())
+ store_dir = conf["store_dir"]
+
+ from_image = load_from_image(conf["from_image"])
+
+ with tarfile.open(mode="w|", fileobj=sys.stdout.buffer) as tar:
+ layers = []
+ layers.extend(add_base_layers(tar, from_image))
+
+ start = len(layers) + 1
+ for num, store_layer in enumerate(conf["store_layers"], start=start):
+ print("Creating layer", num, "from paths:", store_layer,
+ file=sys.stderr)
+ info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
+ layers.append(info)
+
+ print("Creating layer", len(layers) + 1, "with customisation...",
+ file=sys.stderr)
+ layers.append(
+ add_customisation_layer(
+ tar,
+ conf["customisation_layer"],
+ mtime=mtime
+ )
+ )
+
+ print("Adding manifests...", file=sys.stderr)
+
+ image_json = {
+ "created": datetime.isoformat(created),
+ "architecture": conf["architecture"],
+ "os": "linux",
+ "config": overlay_base_config(from_image, conf["config"]),
+ "rootfs": {
+ "diff_ids": [f"sha256:{layer.checksum}" for layer in layers],
+ "type": "layers",
+ },
+ "history": [
+ {
+ "created": datetime.isoformat(created),
+ "comment": f"store paths: {layer.paths}"
+ }
+ for layer in layers
+ ],
+ }
+
+ image_json = json.dumps(image_json, indent=4).encode("utf-8")
+ image_json_checksum = hashlib.sha256(image_json).hexdigest()
+ image_json_path = f"{image_json_checksum}.json"
+ add_bytes(tar, image_json_path, image_json, mtime=mtime)
+
+ manifest_json = [
+ {
+ "Config": image_json_path,
+ "RepoTags": [conf["repo_tag"]],
+ "Layers": [layer.path for layer in layers],
+ }
+ ]
+ manifest_json = json.dumps(manifest_json, indent=4).encode("utf-8")
+ add_bytes(tar, "manifest.json", manifest_json, mtime=mtime)
+
+ print("Done.", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..0bfd011ad4 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -45,6 +46,7 @@ (define-module (gnu system image)
#:use-module (gnu system uuid)
#:use-module (gnu system vm)
#:use-module (guix packages)
+ #:use-module ((gnu packages) #:select (search-auxiliary-file))
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages bootloaders)
@@ -58,6 +60,7 @@ (define-module (gnu system image)
#:use-module (gnu packages hurd)
#:use-module (gnu packages linux)
#:use-module (gnu packages mtools)
+ #:use-module (gnu packages python)
#:use-module (gnu packages virtualization)
#:use-module ((srfi srfi-1) #:prefix srfi-1:)
#:use-module (srfi srfi-11)
@@ -78,6 +81,7 @@ (define-module (gnu system image)
efi-disk-image
iso9660-image
docker-image
+ docker-layered-image
tarball-image
wsl2-image
raw-with-offset-disk-image
@@ -89,6 +93,7 @@ (define-module (gnu system image)

This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 13 Mar 01:33 +0100
[PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003310.17129-2-go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)

Toggle diff (58 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 924c2b35b4..98fb7f536c 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,43 @@
(channel-news
(version 0)
+ (entry (commit "45777c5b753ce330ad007d4e71189cf3fc627ccc")
+ (title
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "????? @samp{docker-layered} ?????? ??? @command{guix pack} ???????"))
+ (body
+ (en "Docker layered image can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+See @command{info \"(guix) Invoking guix pack\"} for more information.
+
+@command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+")
+ (ru "????????? ??????? ???????? ???????????? Docker ??????? ? ???????
+@command{guix pack --format=docker-layered}, ??????? ??????? Docker ????? ?
+?????? ? store ?????????????? ?? ????????? ?????, ??????? ????? ???????
+???????? ???????. ????? ????? ?????? ? GNU store ? ???????? gzipped tarball.
+
+?????? ???????? Docker layered image ? @code{hello} ???????:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+???????? @command{info \"(guix) Invoking guix pack\"} ??? ????????? ?????
+????????? ????????.
+
+@command{guix system image} ?????? ????? ????????? layered Docker image ?????
+???????? ? ????? @option{--image-type} ????????? @code{docker-layered}.")))
+
(entry (commit "598f4c509bbfec2b983a8ee246cce0a0fe45ec7f")
(title
(de "Neues Format @samp{rpm} für den Befehl @command{guix pack}")
--
2.38.0
O
O
Oleg Pykhalov wrote on 13 Mar 01:43 +0100
Cover lever typo in guix pack format example
(address . 62153@debbugs.gnu.org)
87wn3l32hs.fsf@gmail.com
The cover lever guix pack example should be:

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello

instead of

./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello

Apologies,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQOcZ8UHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzRkg/9H8+149MGcejnPteSR7C1XKWckEWi
xoADRq5qcwFf9JTGGNEDqEoq/z+BFCa5ainGpLKO6jzVGe0gCfzsmmxRA7V5fZY3
udwfPwVmhb3oV4CV8t8Gi3tqTnJ3meUG7CdzNttocYRlwLys2TUoy5t8fJ9xwg/v
6ZSVeKfcXJdM9BP5a+KkLMzfxGOGlfu8HFVIrVlPAMgRmliZLRKFdk3mmZ7jJEQv
OepVWlQ5j6BRHfbETEYz6nB7Hw2BYzfL5dwzkrsMBwO2KgtG9sYSgdqIbRVJXcP9
L3R5Z1Z51p4CcByYG35iq0ahwGy9daUL1UiCx8gIXJPEy7RGMFqjOAGekg+lN2PI
qOHLMpthqJBKQxKnYBxp7ZdgnvwX4iYs7YrUDDX6I507L1zYABWNM2exLl1N3PUF
vlHKUmI2la4ozWDaHNZsTa7qVtFtRwXr5Vs8hfkbqfEhAG3YHQKd0zMBr33Ht6bn
WYdT6zKq2Dnp3BjK1dkOKYhIJelmWlwptUYv88h7KiozgEOI84l8RpSOU7IXnEBY
CUhhGhPp2xu4w7NCkOA3TAYHGYFT2ebFnlZ+l3fTq6i8Y3p9LVD65Jvz9Pqgt0MQ
9v7+ArClkWYmKxc0XNrqpv2J8mAr5PlSvfXtHM0wTsZk+cpuaC/xIkCf4upHyLyh
1a4qZJxNTV9FjLA=
=z9Ac
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 13 Mar 16:01 +0100
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
87r0tsk85r.fsf@gmail.com
Hi,

Oh cool! Awesome! Thanks for pushing forward.

On lun., 13 mars 2023 at 03:33, Oleg Pykhalov <go.wigust@gmail.com> wrote:

Toggle quote (10 lines)
> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
> new file mode 100644
> index 0000000000..9ad2168c2d
> --- /dev/null
> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
> @@ -0,0 +1,391 @@
> +"""
> +This script generates a Docker image from a set of store paths. Uses
> +Docker Image Specification v1.2 as reference [1].

Instead of Python, would it possible to implement in Guile? I mean,
does Python have something that is missing in Guile?

The facility for manipulating Tar? Something else?


Because then, if I understand correctly…

Toggle quote (5 lines)
> diff --git a/guix/docker.scm b/guix/docker.scm
> index 5e6460f43f..f1adad26dc 100644
> --- a/guix/docker.scm
> +++ b/guix/docker.scm

[...]

Toggle quote (5 lines)
> + (if stream-layered-image
> + (let ((input (open-pipe* OPEN_READ "python3"
> + stream-layered-image
> + "config.json")))

…it requires to drag Python for building/packing layered Docker.


Well, I have not really look yet to the Python script which does most of
the job. Do you use a similar strategy as [1]?

And I remember something in that direction by Chris but I am unable to
find back the patch. )-:


Cheers,
simon
P
P
pelzflorian (Florian Pelz) wrote on 13 Mar 22:09 +0100
Re: [bug#62153] [PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)(address . 62153@debbugs.gnu.org)
87cz5ccqa5.fsf@pelzflorian.de
Thank you Oleg for this feature.

Could you change the following three things in the news:

Change the beginning of the English translation from
"Docker layered image can now be produced" to
"Docker layered images can now be produced".

And at the end, also in Russian, switch around these two paragraphs
and add a “the” and reference the System Images chapter:
Toggle quote (7 lines)
> @command{guix system image} can now produce layered Docker image by passing
> @code{docker-layered} to the @option{--image-type} option.
>
> See @command{info \"(guix) Invoking guix pack\"} and
> @command{info \"(guix) System Images\"} for more information.


Lastly, could you then also add a German translation:

(title
(de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")

(body
(de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
angelegt wird:

@example
guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
@end example

@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
übergeben.

Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")

Regards,
Florian
O
O
Oleg Pykhalov wrote on 13 Mar 22:10 +0100
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 62153@debbugs.gnu.org)
87bkkw2w7z.fsf@gmail.com
Hi Simon,

Thank you for the review.

Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (17 lines)
> On lun., 13 mars 2023 at 03:33, Oleg Pykhalov <go.wigust@gmail.com> wrote:
>
>> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
>> new file mode 100644
>> index 0000000000..9ad2168c2d
>> --- /dev/null
>> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
>> @@ -0,0 +1,391 @@
>> +"""
>> +This script generates a Docker image from a set of store paths. Uses
>> +Docker Image Specification v1.2 as reference [1].
>
> Instead of Python, would it possible to implement in Guile? I mean,
> does Python have something that is missing in Guile?
>
> The facility for manipulating Tar? Something else?

I think nothing else. As I understand Python implemented Tar inside the
language itself in 2500 lines of code by manipulating binary data.

/gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py

Technically it's probably possible to use tar utility with --append flag
instead of opening a new file and streaming to it as the Python script
does. To be honest I would like not to write it in this way if the
Python script does not block current patch for merge.

Also I don't see myself writing Tar implementation in Guile, yet. ;-)

The Nix project uses this script heavily to build layered images, so it
should be robust in terms of up to date to current Tar and Python
implementations.

Toggle quote (16 lines)
> Because then, if I understand correctly…
>
>> diff --git a/guix/docker.scm b/guix/docker.scm
>> index 5e6460f43f..f1adad26dc 100644
>> --- a/guix/docker.scm
>> +++ b/guix/docker.scm
>
> [...]
>
>> + (if stream-layered-image
>> + (let ((input (open-pipe* OPEN_READ "python3"
>> + stream-layered-image
>> + "config.json")))
>
> …it requires to drag Python for building/packing layered Docker.

Correct.

Toggle quote (8 lines)
> Well, I have not really look yet to the Python script which does most of
> the job. Do you use a similar strategy as [1]?
>
> And I remember something in that direction by Chris but I am unable to
> find back the patch. )-:
>
> 1: https://grahamc.com/blog/nix-and-layered-docker-images/

Not similar. My patch implements a very simple sorting by size, no
complex sorting by reference popularity as in [1], which is probably
implemented in the following file

github.com/NixOS/nixpkgs/pkgs/build-support/references-by-popularity/closure-graph.py


Toggle quote (27 lines)
> How Docker really represents an Image
>
> Docker’s layers are content addressable and aren’t required to
> explicitly reference a parent layer. This means a layer for
> readline-7.0p5 doesn’t have to mention that it has any relationship to
> ncurses-6.1 or glibc-2.27 at all.
>
> Instead each image has a manifest which defines the order:
>
> {
> "Layers": [
> "bash-interactive-4.4-p23",
> "bash-4.4p23",
> "readline-7.0p5",
> ...
> ]
> }
>
> If you have only built Docker images using a Dockerfile, then you
> would expect the way we flatten our graph to be critically
> important. If we sometimes picked readline-7.0p5 to come first and
> other times picked bash-4.4p23 then we may never make cache hits.
>
> However since the Image defines the order, we don’t have to solve this
> impossible problem: we can order the layers in any way we want and the
> layer cache will always hit.

In case of sorting by size, bigest layers will be on top of a container
image, which will produce a cache hit for bigest directories in the GNU
store during images transfer with same layers.

I would like to say this sorting could binifit more than sorting by
popularity during transfer but let's assume I didn't write it. ;-)

The following example shows common layers between images, which will be
not tranfered if you load image inside Docker as well as pull and push:

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello

and

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello emacs

share 6 layers in total

Toggle snippet (11 lines)
$ f() { docker image inspect "$1" | jq --raw-output '.[0].RootFS.Layers[] | .' | sort ; }
$ comm -1 -2 --total <(f sha256:fb43b32380a5e6a867410721f4ce2917db14d4ae943c433983afbaf84416c421) <(f sha256:0ce4a11973d1071aeec5441db228d6148dfd09fea3ae77b731c750ebfcc2fe1d)
sha256:3b3daa2a00f1acd12eeb16698bf1caeb6ba6c436e3dbca6259c3a9c622664e00
sha256:5c2be7469293854257221cb6aa8aa4af1e10e2c550935390dbcfeede3d3fbacd
sha256:60317981d94928659389f299e4b86703e5ded420a53537d67627952187fbd3f9
sha256:6d7c8ce5441d4c4c74e0ecff6c203a7b265b37137cca3b0a0ccf10526cfaa6e2
sha256:c2ded2ffe3f46fa7a64a62e0fc6b9d28cb7d4f8d9c64d5a52d137a508cba11fc
sha256:fbcad85d7d3c25bd2aa6d95bb3bf3d02c499ee3b3e443ddd3e5b679c2b33c139
5 94 6 total

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPkWAUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzFZBAApU6Atd2czoZgSRir9SG0/V/l7yva
zKst6JvXL77QxVU+e0QqVcY/o8rJg9hIGH+cQaOmlxmbYXJExEI1Go+tngk61OgQ
dsMrdBBPkaXvqzd7PKQRUtCn221CGfoMMfZGEzCUyQFTYgo6+K7s76Ep6++lODEW
nV/nPa/gqKhg6IZ2G+NeuNKeHsFN8YH/U3Si8myvyLSt0B/ZZEb/8eMumhl3lt61
4GXPPhlFXvp/8VebhMbvzN7TUGWl4z8uMpVExYdDjG3BxkcJBqlEfGfJPQlWc18E
+8G93/JCFYhtE/ae0d/qCpVs8k7CLRrJEkllQFARau+e0GxNggdA08mACmuxZqTF
u1t52FVvg+cgIO5XKN0x0HyR0t+Rv6gkZPA1b4EXF6t8XRCkdh5VXZr/v3Wvwea2
2wbTeX2vDeca3coyCxDNRjka+FgcLrcxxVBZNc33/76sgglXjovQI4g4LDsklkV9
pSa8StEt4/lCSPXRUQYy0g/nUO4Lk2QaIVx7FSefqCaS+EwCz6NkMGi8esKrtHuw
l1gFvYhMFtHvQjzbO50edR1QyLgbMTFbyJdFhHJOEI0rAeh4AIde+R/FfXbPFZR4
RCPwCij2+Ng0O8lrd9uQcdo92oms6FGHLN+ndgYm38hzMQdI+ecMpAQtlR09efsW
WBhi5/HWNf3372o=
=r2xS
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 14 Mar 01:24 +0100
[PATCH 0/2] Add Docker layered image for pack and system (v2)
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-1-go.wigust@gmail.com
Hi, Florian.

Thank you for the review.

This patch series applies your suggestions. Also it's rebased on
origin/master and added a missing documentation for ‘docker-layered’ format in
‘guix system image’ command in doc/guix.texi file (following diff).

<#part type="text/x-patch" buffer=m1.txt disposition=inline description="Add missing docker-layered format documentation for guix system image command">
<#/part>

The folloing tests passed:

make check-channel-news
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system docker-layered-system"

Oleg Pykhalov (2):
guix: docker: Build layered image.
news: Add entry for the new 'docker-layered' distribution format.

Makefile.am | 3 +-
doc/guix.texi | 18 +-
etc/news.scm | 58 +++
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 105 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
11 files changed, 837 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py


base-commit: 5312d798ac36a72d8a977325a7c6ff7647be670a
--
2.38.0
O
O
Oleg Pykhalov wrote on 14 Mar 01:24 +0100
[PATCH 1/2] guix: docker: Build layered image.
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-2-go.wigust@gmail.com
* gnu/packages/aux-files/python/stream-layered-image.py: New file.
* Makefile.am (AUX_FILES): Add this.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* tests/pack.scm: Add docker-layered-image + localstatedir test.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New
variables.
(system-docker-image)[layered-image?]: New argument.
(stream-layered-image.py): New variable.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* gnu/image.scm (validate-image-format)[docker-layered]: New image format.
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image Reference): Same.
(image-type Reference): Document docker-layered-image-type.
---
Makefile.am | 3 +-
doc/guix.texi | 18 +-
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 105 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
10 files changed, 779 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py

Toggle diff (518 lines)
diff --git a/Makefile.am b/Makefile.am
index 23b939b674..9aca84f8f8 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -11,7 +11,7 @@
# Copyright © 2017 Arun Isaac <arunisaac@systemreboot.net>
# Copyright © 2018 Nikita <nikita@n0.is>
# Copyright © 2018 Julien Lepiller <julien@lepiller.eu>
-# Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
+# Copyright © 2018, 2023 Oleg Pykhalov <go.wigust@gmail.com>
# Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
@@ -435,6 +435,7 @@ AUX_FILES = \
gnu/packages/aux-files/python/sanity-check.py \
gnu/packages/aux-files/python/sanity-check-next.py \
gnu/packages/aux-files/python/sitecustomize.py \
+ gnu/packages/aux-files/python/stream-layered-image.py \
gnu/packages/aux-files/renpy/renpy.in \
gnu/packages/aux-files/run-in-namespace.c
diff --git a/doc/guix.texi b/doc/guix.texi
index 39932d5aad..fa4b7586c9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@ Copyright @copyright{} 2017 Andy Wingo@*
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6837,9 +6837,15 @@ the following command:
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -43274,6 +43280,8 @@ one or multiple partitions.
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
@@ -43605,6 +43613,10 @@ Build an image based on the @code{iso9660-image} image but with the
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-with-syntax-properties (name (value properties))
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
new file mode 100644
index 0000000000..9ad2168c2d
--- /dev/null
+++ b/gnu/packages/aux-files/python/stream-layered-image.py
@@ -0,0 +1,391 @@
+"""
+This script generates a Docker image from a set of store paths. Uses
+Docker Image Specification v1.2 as reference [1].
+
+It expects a JSON file with the following properties and writes the
+image as an uncompressed tarball to stdout:
+
+* "architecture", "config", "os", "created", "repo_tag" correspond to
+ the fields with the same name on the image spec [2].
+* "created" can be "now".
+* "created" is also used as mtime for files added to the image.
+* "store_layers" is a list of layers in ascending order, where each
+ layer is the list of store paths to include in that layer.
+
+The main challenge for this script to create the final image in a
+streaming fashion, without dumping any intermediate data to disk
+for performance.
+
+A docker image has each layer contents archived as separate tarballs,
+and they later all get enveloped into a single big tarball in a
+content addressed fashion. However, because how "tar" format works,
+we have to know about the name (which includes the checksum in our
+case) and the size of the tarball before we can start adding it to the
+outer tarball. We achieve that by creating the layer tarballs twice;
+on the first iteration we calculate the file size and the checksum,
+and on the second one we actually stream the contents. 'add_layer_dir'
+function does all this.
+
+[1]: https://github.com/moby/moby/blob/master/image/spec/v1.2.md
+[2]: https://github.com/moby/moby/blob/4fb59c20a4fb54f944fe170d0ff1d00eb4a24d6f/image/spec/v1.2.md#image-json-field-descriptions
+""" # noqa: E501
+
+
+import io
+import os
+import re
+import sys
+import json
+import hashlib
+import pathlib
+import tarfile
+import itertools
+import threading
+from datetime import datetime, timezone
+from collections import namedtuple
+
+
+def archive_paths_to(obj, paths, mtime):
+ """
+ Writes the given store paths as a tar file to the given stream.
+
+ obj: Stream to write to. Should have a 'write' method.
+ paths: List of store paths.
+ """
+
+ # gettarinfo makes the paths relative, this makes them
+ # absolute again
+ def append_root(ti):
+ ti.name = "/" + ti.name
+ return ti
+
+ def apply_filters(ti):
+ ti.mtime = mtime
+ ti.uid = 0
+ ti.gid = 0
+ ti.uname = "root"
+ ti.gname = "root"
+ return ti
+
+ def nix_root(ti):
+ ti.mode = 0o0555 # r-xr-xr-x
+ return ti
+
+ def dir(path):
+ ti = tarfile.TarInfo(path)
+ ti.type = tarfile.DIRTYPE
+ return ti
+
+ with tarfile.open(fileobj=obj, mode="w|") as tar:
+ # To be consistent with the docker utilities, we need to have
+ # these directories first when building layer tarballs.
+ tar.addfile(apply_filters(nix_root(dir("/gnu"))))
+ tar.addfile(apply_filters(nix_root(dir("/gnu/store"))))
+
+ for path in paths:
+ path = pathlib.Path(path)
+ if path.is_symlink():
+ files = [path]
+ else:
+ files = itertools.chain([path], path.rglob("*"))
+
+ for filename in sorted(files):
+ ti = append_root(tar.gettarinfo(filename))
+
+ # copy hardlinks as regular files
+ if ti.islnk():
+ ti.type = tarfile.REGTYPE
+ ti.linkname = ""
+ ti.size = filename.stat().st_size
+
+ ti = apply_filters(ti)
+ if ti.isfile():
+ with open(filename, "rb") as f:
+ tar.addfile(ti, f)
+ else:
+ tar.addfile(ti)
+
+
+class ExtractChecksum:
+ """
+ A writable stream which only calculates the final file size and
+ sha256sum, while discarding the actual contents.
+ """
+
+ def __init__(self):
+ self._digest = hashlib.sha256()
+ self._size = 0
+
+ def write(self, data):
+ self._digest.update(data)
+ self._size += len(data)
+
+ def extract(self):
+ """
+ Returns: Hex-encoded sha256sum and size as a tuple.
+ """
+ return (self._digest.hexdigest(), self._size)
+
+
+FromImage = namedtuple("FromImage", ["tar", "manifest_json", "image_json"])
+# Some metadata for a layer
+LayerInfo = namedtuple("LayerInfo", ["size", "checksum", "path", "paths"])
+
+
+def load_from_image(from_image_str):
+ """
+ Loads the given base image, if any.
+
+ from_image_str: Path to the base image archive.
+
+ Returns: A 'FromImage' object with references to the loaded base image,
+ or 'None' if no base image was provided.
+ """
+ if from_image_str is None:
+ return None
+
+ base_tar = tarfile.open(from_image_str)
+
+ manifest_json_tarinfo = base_tar.getmember("manifest.json")
+ with base_tar.extractfile(manifest_json_tarinfo) as f:
+ manifest_json = json.load(f)
+
+ image_json_tarinfo = base_tar.getmember(manifest_json[0]["Config"])
+ with base_tar.extractfile(image_json_tarinfo) as f:
+ image_json = json.load(f)
+
+ return FromImage(base_tar, manifest_json, image_json)
+
+
+def add_base_layers(tar, from_image):
+ """
+ Adds the layers from the given base image to the final image.
+
+ tar: 'tarfile.TarFile' object for new layers to be added to.
+ from_image: 'FromImage' object with references to the loaded base image.
+ """
+ if from_image is None:
+ print("No 'fromImage' provided", file=sys.stderr)
+ return []
+
+ layers = from_image.manifest_json[0]["Layers"]
+ checksums = from_image.image_json["rootfs"]["diff_ids"]
+ layers_checksums = zip(layers, checksums)
+
+ for num, (layer, checksum) in enumerate(layers_checksums, start=1):
+ layer_tarinfo = from_image.tar.getmember(layer)
+ checksum = re.sub(r"^sha256:", "", checksum)
+
+ tar.addfile(layer_tarinfo, from_image.tar.extractfile(layer_tarinfo))
+ path = layer_tarinfo.path
+ size = layer_tarinfo.size
+
+ print("Adding base layer", num, "from", path, file=sys.stderr)
+ yield LayerInfo(size=size, checksum=checksum, path=path, paths=[path])
+
+ from_image.tar.close()
+
+
+def overlay_base_config(from_image, final_config):
+ """
+ Overlays the final image 'config' JSON on top of selected defaults from the
+ base image 'config' JSON.
+
+ from_image: 'FromImage' object with references to the loaded base image.
+ final_config: 'dict' object of the final image 'config' JSON.
+ """
+ if from_image is None:
+ return final_config
+
+ base_config = from_image.image_json["config"]
+
+ # Preserve environment from base image
+ final_env = base_config.get("Env", []) + final_config.get("Env", [])
+ if final_env:
+ # Resolve duplicates (last one wins) and format back as list
+ resolved_env = {entry.split("=", 1)[0]: entry for entry in final_env}
+ final_config["Env"] = list(resolved_env.values())
+ return final_config
+
+
+def add_layer_dir(tar, paths, store_dir, mtime):
+ """
+ Appends given store paths to a TarFile object as a new layer.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ paths: List of store paths.
+ store_dir: the root directory of the nix store
+ mtime: 'mtime' of the added files and the layer tarball.
+ Should be an integer representing a POSIX time.
+
+ Returns: A 'LayerInfo' object containing some metadata of
+ the layer added.
+ """
+
+ invalid_paths = [i for i in paths if not i.startswith(store_dir)]
+ assert len(invalid_paths) == 0, \
+ f"Expecting absolute paths from {store_dir}, but got: {invalid_paths}"
+
+ # First, calculate the tarball checksum and the size.
+ extract_checksum = ExtractChecksum()
+ archive_paths_to(
+ extract_checksum,
+ paths,
+ mtime=mtime,
+ )
+ (checksum, size) = extract_checksum.extract()
+
+ path = f"{checksum}/layer.tar"
+ layer_tarinfo = tarfile.TarInfo(path)
+ layer_tarinfo.size = size
+ layer_tarinfo.mtime = mtime
+
+ # Then actually stream the contents to the outer tarball.
+ read_fd, write_fd = os.pipe()
+ with open(read_fd, "rb") as read, open(write_fd, "wb") as write:
+ def producer():
+ archive_paths_to(
+ write,
+ paths,
+ mtime=mtime,
+ )
+ write.close()
+
+ # Closing the write end of the fifo also closes the read end,
+ # so we don't need to wait until this thread is finished.
+ #
+ # Any exception from the thread will get printed by the default
+ # exception handler, and the 'addfile' call will fail since it
+ # won't be able to read required amount of bytes.
+ threading.Thread(target=producer).start()
+ tar.addfile(layer_tarinfo, read)
+
+ return LayerInfo(size=size, checksum=checksum, path=path, paths=paths)
+
+
+def add_customisation_layer(target_tar, customisation_layer, mtime):
+ """
+ Adds the customisation layer as a new layer. This is layer is structured
+ differently; given store path has the 'layer.tar' and corresponding
+ sha256sum ready.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ customisation_layer: Path containing the layer archive.
+ mtime: 'mtime' of the added layer tarball.
+ """
+
+ checksum_path = os.path.join(customisation_layer, "checksum")
+ with open(checksum_path) as f:
+ checksum = f.read().strip()
+ assert len(checksum) == 64, f"Invalid sha256 at ${checksum_path}."
+
+ layer_path = os.path.join(customisation_layer, "layer.tar")
+
+ path = f"{checksum}/layer.tar"
+ tarinfo = target_tar.gettarinfo(layer_path)
+ tarinfo.name = path
+ tarinfo.mtime = mtime
+
+ with open(layer_path, "rb") as f:
+ target_tar.addfile(tarinfo, f)
+
+ return LayerInfo(
+ size=None,
+ checksum=checksum,
+ path=path,
+ paths=[customisation_layer]
+ )
+
+
+def add_bytes(tar, path, content, mtime):
+ """
+ Adds a file to the tarball with given path and contents.
+
+ tar: 'tarfile.TarFile' object.
+ path: Path of the file as a string.
+ content: Contents of the file.
+ mtime: 'mtime' of the file. Should be an integer representing a POSIX time.
+ """
+ assert type(content) is bytes
+
+ ti = tarfile.TarInfo(path)
+ ti.size = len(content)
+ ti.mtime = mtime
+ tar.addfile(ti, io.BytesIO(content))
+
+
+def main():
+ with open(sys.argv[1], "r") as f:
+ conf = json.load(f)
+
+ created = (
+ datetime.now(tz=timezone.utc)
+ if conf["created"] == "now"
+ else datetime.fromisoformat(conf["created"])
+ )
+ mtime = int(created.timestamp())
+ store_dir = conf["store_dir"]
+
+ from_image = load_from_image(conf["from_image"])
+
+ with tarfile.open(mode="w|", fileobj=sys.stdout.buffer) as tar:
+ layers = []
+ layers.extend(add_base_layers(tar, from_image))
+
+ start = len(layers) + 1
+ for num, store_layer in enumerate(conf["store_layers"], start=start):
+ print("Creating layer", num, "from paths:", store_layer,
+ file=sys.stderr)
+ info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
+ layers.append(info)
+
+ print("Creating layer", len(layers) + 1, "with customisation...",
+ file=sys.stderr)
+ layers.append(
+ add_customisation_layer(
+ tar,
+ conf["customisation_layer"],
+ mtime=mtime
+ )
+ )
+
+ print("Adding manifests...", file=sys.stderr)
+
+ image_json = {
+ "created": datetime.isoformat(created),
+ "architecture": conf["architecture"],
+ "os": "linux",
+ "config": overlay_base_config(from_image, conf["config"]),
+ "rootfs": {
+ "diff_ids": [f"sha256:{layer.checksum}" for layer in layers],
+ "type": "layers",
+ },
+ "history": [
+ {
+ "created": datetime.isoformat(created),
+ "comment": f"store paths: {layer.paths}"
+ }
+ for layer in layers
+ ],
+ }
+
+ image_json = json.dumps(image_json, indent=4).encode("utf-8")
+ image_json_checksum = hashlib.sha256(image_json).hexdigest()
+ image_json_path = f"{image_json_checksum}.json"
+ add_bytes(tar, image_json_path, image_json, mtime=mtime)
+
+ manifest_json = [
+ {
+ "Config": image_json_path,
+ "RepoTags": [conf["repo_tag"]],
+ "Layers": [layer.path for layer in layers],
+ }
+ ]
+ manifest_json = json.dumps(manifest_json, indent=4).encode("utf-8")
+ add_bytes(tar, "manifest.json", manifest_json, mtime=mtime)
+
+ print("Done.", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..0bfd011ad4 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -45,6 +46,7 @@ (define-module (gnu system image)
#:use-module (gnu system uuid)
#:use-module (gnu system vm)
#:use-module (guix packages)
+ #:use-module ((gnu packages) #:select (search-auxiliary-file))
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages bootloaders)
@@ -58,6 +60,7 @@ (define-module (gnu system image)
#:use-module (gnu packages hurd)
#:use-module (gnu packages linux)
#:use-module (gnu packages mtools)
+ #:use-module (gnu packages python)
#:use-module (gnu packages virtualization)
#:use-module ((srfi srfi-1) #:prefix srfi-1:)
#:use-module (srfi srfi-11)
@@ -78,6 +81,7 @@ (define-module (gnu system
This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 14 Mar 01:24 +0100
[PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-3-go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

Toggle diff (78 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 55d1218df5..4bbdfd2a59 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,63 @@
(channel-news
(version 0)
+ (entry (commit "a5c3baf510adab1f5b3bb855b1aa9cafe3cb66b9")
+ (title
+ (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "????? @samp{docker-layered} ?????? ??? @command{guix pack} ???????"))
+ (body
+ (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
+@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
+Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
+untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
+im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
+Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
+angelegt wird:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
+indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
+übergeben.
+
+Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
+@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")
+ (en "Docker layered images can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+The @command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+
+See @command{info \"(guix) Invoking guix pack\"} and
+@command{info \"(guix) System Images\"} for more information.")
+ (ru "????????? ??????? ???????? ???????????? Docker ??????? ? ???????
+@command{guix pack --format=docker-layered}, ??????? ??????? Docker ????? ?
+?????? ? store ?????????????? ?? ????????? ?????, ??????? ????? ???????
+???????? ???????. ????? ????? ?????? ? GNU store ? ???????? gzipped tarball.
+
+?????? ???????? Docker layered ????? ? @code{hello} ???????:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} ?????? ????? ????????? layered Docker ????? ?????
+???????? ? ????? @option{--image-type} ????????? @code{docker-layered}.
+
+???????? @command{info \"(guix) Invoking guix pack\"} ?
+@command{info \"(guix) System Images\"} ??? ????????? ????? ?????????
+????????.")))
+
(entry (commit "0e18c5e5bcb9204c278cfc75493d3b02b746d5c3")
(title
(en "Linux-libre kernel updated to 6.2")
--
2.38.0
O
O
Oleg Pykhalov wrote on 14 Mar 01:40 +0100
Missing diff in cover lever for v2 patch
(address . 62153@debbugs.gnu.org)
87wn3k17yf.fsf@gmail.com
Missing diff in cover leter for v2 patch attached below.

Toggle quote (4 lines)
> This patch series applies your suggestions. Also it's rebased on
> origin/master and added a missing documentation for ‘docker-layered’ format in
> ‘guix system image’ command in doc/guix.texi file (following diff).

Toggle snippet (14 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index bd0ee126ee..6938743154 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -43306,6 +43306,8 @@ one or multiple partitions.
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPwngUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pw5aw/+JeyQ3p6e4cfIX6+RHnDjUgP9HA2Q
ESTfjzWxsJcCxleL0LKO3AbOUVby/qq4TrUgNSm1cqL/S4vWDAIo34XEyT5Xerdg
oHSNtoaHqD467NsGhSvtdS8TYGe7JLTTr76nkNzwjQAGjDYXzJXvrcr758H4VMw0
6k143Q1uqDYj3+E2uqQWT9wme5oZ4IkxzZW/+XCME3s00yBF19Uti9VqROLAEIhV
OnQAuurDYScfZ7in17DMuyHxOfrJm4M2G7UxRSiVnKbBeGjbaQoL4+QSRCRa+/+O
0VABMVnYFYUqQPFKm+Y7csQiFbjE6mojc3nNwqBEd0i3rUwu4vPjad6mirKUC9FQ
C3gpBDAIVNJ6I0UWmmBdRH/p/hNRvtiAw0FAhBaV0arTsWoLMeybmZjoxDF533zo
wg7ZPbaZtJd8nHowzKsdbXYqYEf1bhio1goBcnraGJdgBFscSsT5eJhk/Tgp9n3a
yQB9NyF4E/Uc7WLzSiscwaxU2ksg+W1s9xZVrz5xcbu1yX8+yI9u+dSp6e6fa7qf
VLyYOGaD7Qju5alCPc6p4z12/a+DD5ZAEJeBReYoJCfEEPjfOClrwHtnJksQyPY/
PlCL+blYcC4yrMQgQN7UeWbDWTEGzwJPHIWgOMzR/meYOQO0TZ1yFj865E9VmyfF
TjXACct5ZaD5Y64=
=otdV
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 14 Mar 09:19 +0100
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
86cz5b923c.fsf@gmail.com
Hi Oleg,

CC: core teams

On Tue, 14 Mar 2023 at 00:10, Oleg Pykhalov <go.wigust@gmail.com> wrote:

Toggle quote (25 lines)
>>> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
>>> new file mode 100644
>>> index 0000000000..9ad2168c2d
>>> --- /dev/null
>>> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
>>> @@ -0,0 +1,391 @@
>>> +"""
>>> +This script generates a Docker image from a set of store paths. Uses
>>> +Docker Image Specification v1.2 as reference [1].
>>
>> Instead of Python, would it possible to implement in Guile? I mean,
>> does Python have something that is missing in Guile?
>>
>> The facility for manipulating Tar? Something else?
>
> I think nothing else. As I understand Python implemented Tar inside the
> language itself in 2500 lines of code by manipulating binary data.
>
> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>
> Technically it's probably possible to use tar utility with --append flag
> instead of opening a new file and streaming to it as the Python script
> does. To be honest I would like not to write it in this way if the
> Python script does not block current patch for merge.

Ok, thanks for explaining.

Toggle quote (2 lines)
> Also I don't see myself writing Tar implementation in Guile, yet. ;-)

Maybe not reimplementing Tar in Guile, maybe just enough for working.
Or maybe some Guile bindings. Or maybe something is already around for
the bootstrap story.

The use of external tools as Python for producing built-in Guix feature
will be the first time, no?

For what it is worth, I would prefer to consider the options before
emitting an opinion about dragging Python building/packing layered
Docker. :-)

Toggle quote (4 lines)
> The Nix project uses this script heavily to build layered images, so it
> should be robust in terms of up to date to current Tar and Python
> implementations.

Do you mean this script is coming from Nix. Well, in all cases, this
script is not trivial and so it requires Copyright for authorship.



[...]


Toggle quote (3 lines)
> The following example shows common layers between images, which will be
> not tranfered if you load image inside Docker as well as pull and push:

Thanks for explaining.


Cheers,
simon

PS: I will be off-line these 2-3 next weeks. So a lack of an answer
from me will not be a lack of interest. ;-)
C
C
Christopher Baines wrote on 14 Mar 10:11 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
8735671yu4.fsf@cbaines.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (3 lines)
> And I remember something in that direction by Chris but I am unable to
> find back the patch. )-:

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmQQOmNfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XeUkRAApwCv6Zh8MrAuDE5TRebOf0IJflIq98wc
iB1i643fUvGZzVsr/pjAXsrkKDqyBdPH9Oee/JHvPPtEcCVGJwwAtodiGC6Tnf1t
8eYzWSXdzYVK7oFReQZ/oHLzmA0zQYVTmcnIoqP4wy+Cgv/zE3fGWlLrpiIau2hV
NgPUo4wPdZu8ZwM0B2+gK2H4I5pNXUxhIuWFI8rORU6q0nxL/OqrDMh5G/+B1UPh
cuGHBOxoBx3JERUnaHkk0BkgS1TySYpaVDA/9RwtCeqV8cv0bSjznkOqdtusr0vK
prM5WJwzOpdlHkAqwkmEEuE93KjdAoMl5gmYWWlUobCdAtwLNLTXCGa+wAlx7VmW
kFl0XygIJCeXHICtDhiBkv/cfIVSpKYVbqJExRAhzeH6Ykllww6UUaEHC32Fy0ae
WsWUUpgpRGZSgbT70i05D8TaPBjNgIBUV5HfczWTsRgvR1b4fVaqrKaxCooy1gV1
YjIJO7ddNHGCSNp0uxGyMzZMmpEy46gQvSiUNTW46AYgl2c5TEVMaQ/gLNvgmAvt
K+PRQhOaHyoZNLLCIZaRhmuPwbwKEh3MLOGcR/AUF7nW5sp2wk9LBm889lDmarpz
bHk5ZfhqiZryeo36k9O3L3DE70h+fibm1uKr7zn6jRt74LmvGdWA4lCWaz4A9USE
VjZkN4ayEaQ=
=9k/e
-----END PGP SIGNATURE-----

R
R
Ricardo Wurmus wrote on 14 Mar 10:15 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
874jqnzo8a.fsf@elephly.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (23 lines)
>>> Instead of Python, would it possible to implement in Guile? I mean,
>>> does Python have something that is missing in Guile?
>>>
>>> The facility for manipulating Tar? Something else?
>>
>> I think nothing else. As I understand Python implemented Tar inside the
>> language itself in 2500 lines of code by manipulating binary data.
>>
>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>
>> Technically it's probably possible to use tar utility with --append flag
>> instead of opening a new file and streaming to it as the Python script
>> does. To be honest I would like not to write it in this way if the
>> Python script does not block current patch for merge.
>
> Ok, thanks for explaining.
>
>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>
> Maybe not reimplementing Tar in Guile, maybe just enough for working.
> Or maybe some Guile bindings. Or maybe something is already around for
> the bootstrap story.

gash-utils has (gash ustar); it’s about 620 lines of code.

--
Ricardo
L
L
Ludovic Courtès wrote on 16 Mar 11:37 +0100
(name . Ricardo Wurmus)(address . rekado@elephly.net)
877cvhhth5.fsf@gnu.org
Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

Toggle quote (27 lines)
> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
>>>> Instead of Python, would it possible to implement in Guile? I mean,
>>>> does Python have something that is missing in Guile?
>>>>
>>>> The facility for manipulating Tar? Something else?
>>>
>>> I think nothing else. As I understand Python implemented Tar inside the
>>> language itself in 2500 lines of code by manipulating binary data.
>>>
>>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>>
>>> Technically it's probably possible to use tar utility with --append flag
>>> instead of opening a new file and streaming to it as the Python script
>>> does. To be honest I would like not to write it in this way if the
>>> Python script does not block current patch for merge.
>>
>> Ok, thanks for explaining.
>>
>>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>>
>> Maybe not reimplementing Tar in Guile, maybe just enough for working.
>> Or maybe some Guile bindings. Or maybe something is already around for
>> the bootstrap story.
>
> gash-utils has (gash ustar); it’s about 620 lines of code.

Disarchive also has a tar implementation. No excuse! :-)

Oleg, could you check which of these would satisfy your needs? I had a
plan to improve the tar implementation in Gash-Utils, perhaps it’s a
good time to get my act together.

Thanks,
Ludo’.
O
O
Oleg Pykhalov wrote on 20 Mar 07:38 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)
87h6ugapwh.fsf@gmail.com
Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (35 lines)
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Simon Tournier <zimon.toutoune@gmail.com> writes:
>>
>>>>> Instead of Python, would it possible to implement in Guile? I mean,
>>>>> does Python have something that is missing in Guile?
>>>>>
>>>>> The facility for manipulating Tar? Something else?
>>>>
>>>> I think nothing else. As I understand Python implemented Tar inside the
>>>> language itself in 2500 lines of code by manipulating binary data.
>>>>
>>>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>>>
>>>> Technically it's probably possible to use tar utility with --append flag
>>>> instead of opening a new file and streaming to it as the Python script
>>>> does. To be honest I would like not to write it in this way if the
>>>> Python script does not block current patch for merge.
>>>
>>> Ok, thanks for explaining.
>>>
>>>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>>>
>>> Maybe not reimplementing Tar in Guile, maybe just enough for working.
>>> Or maybe some Guile bindings. Or maybe something is already around for
>>> the bootstrap story.
>>
>> gash-utils has (gash ustar); it’s about 620 lines of code.
>
> Disarchive also has a tar implementation. No excuse! :-)
>
> Oleg, could you check which of these would satisfy your needs? I had a
> plan to improve the tar implementation in Gash-Utils, perhaps it’s a
> good time to get my act together.

Gash-Utils should work, e.g. [1] script. It's already possible to
rewrite with Gash-Utils, but at least write-ustar-file and
write-ustar-footer should be exported in ustar.scm.

Disarchive requires to write a file specification in case of using
disarchive-assemble. And disarchive-assemble does not work, if I don't
miss anything [2]. Also gzip compression does not work in a Guile REPL.

[1]:

Toggle snippet (37 lines)
#!/usr/bin/env -S guile --no-auto-compile -e main -s
!#

(set! %load-path
(append '("/gnu/store/...-gash-utils-0.1.0/share/guile/site/3.0"
"/gnu/store/...-gash-0.2.0/share/guile/site/3.0")
%load-path))

(set! %load-compiled-path
(append '("/gnu/store/...-gash-utils-0.1.0/lib/guile/3.0/site-ccache"
"/gnu/store/...-gash-0.2.0/lib/guile/3.0/site-ccache")
%load-compiled-path))

(use-modules (gash ustar)
(srfi srfi-26)
(guix build utils))

(define write-ustar-file
(@@ (gash ustar) write-ustar-file))

(define write-ustar-footer
(@@ (gash ustar) write-ustar-footer))

(define (main . args)
(call-with-port (open-file "out.tar.gz" "wb")
(lambda (port)
(with-directory-excursion "."
(call-with-compressed-output-port 'gzip port
(cut write-ustar-file <> "Makefile.am"
#:verbosity 0)))
(with-directory-excursion "doc"
(call-with-compressed-output-port 'gzip port
(cut write-ustar-file <> "."
#:verbosity 0)))
(write-ustar-footer port))))

[2]:

Toggle snippet (263 lines)
$ guile
,m(disarchive assemblers tarball)
(assemble-tarball (disassemble-tarball "out.tar") "result/out.tar")

# Generated tarball:
$ tar tf result/out.tar/sha256/1e7100029373723df900712d1191be0bad5beadf752f367bb264fab891c36356
./
./images/
./images/bootstrap-graph.dot
./images/service-graph.pdf
./guix-cookbook.zh_Hans.texi
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

# Original tarball:
$ tar tf out.tar
./
./images/
./images/bootstrap-graph.dot
./images/bootstrap-graph.eps
./images/bootstrap-graph.pdf
./images/bootstrap-graph.png
./images/bootstrap-packages.dot
./images/bootstrap-packages.eps
./images/bootstrap-packages.pdf
./images/bootstrap-packages.png
./images/coreutils-bag-graph.dot
./images/coreutils-bag-graph.eps
./images/coreutils-bag-graph.pdf
./images/coreutils-bag-graph.png
./images/coreutils-graph.dot
./images/coreutils-graph.eps
./images/coreutils-graph.pdf
./images/coreutils-graph.png
./images/coreutils-size-map.eps
./images/coreutils-size-map.png
./images/gcc-core-mesboot0-graph.dot
./images/gcc-core-mesboot0-graph.eps
./images/gcc-core-mesboot0-graph.pdf
./images/gcc-core-mesboot0-graph.png
./images/installer-network.png
./images/installer-partitions.png
./images/installer-resume.png
./images/service-graph.dot
./images/service-graph.eps
./images/service-graph.pdf
./images/service-graph.png
./images/shepherd-graph.dot
./images/shepherd-graph.eps
./images/shepherd-graph.pdf
./images/shepherd-graph.png
./.dirstamp
./environment-gdb.scm
./fdl-1.3.texi
./os-config-bare-bones.texi
./os-config-desktop.texi
./os-config-lightweight-desktop.texi
./package-hello.json
./package-hello.scm
./stamp-1
./stamp-10
./stamp-11
./stamp-2
./stamp-3
./stamp-4
./stamp-5
./stamp-6
./stamp-7
./stamp-8
./stamp-9
./stamp-vti
./contributing.fa.texi
./guix.fa.texi
./contributing.fi.texi
./guix.fi.texi
./contributing.it.texi
./guix.it.texi
./contributing.ko.texi
./guix.ko.texi
./contributing.sk.texi
./guix.sk.texi
./guix-cookbook.es.texi
./guix-cookbook.fa.texi
./guix-cookbook.fi.texi
./guix-cookbook.pt_BR.texi
./guix-cookbook.ru.texi
./guix-cookbook.zh_Hans.texi
./version-fa.texi
./guix.fa.info-1
./guix.fa.info-2
./guix.fa.info-3
./guix.fa.info-4
./guix.fa.info-5
./guix.fa.info-6
./guix.fa.info
./version-fi.texi
./guix.fi.info-1
./guix.fi.info-2
./guix.fi.info-3
./guix.fi.info-4
./guix.fi.info-5
./guix.fi.info-6
./guix.fi.info
./version-it.texi
./guix.it.info-1
./guix.it.info-2
./guix.it.info-3
./guix.it.info-4
./guix.it.info-5
./guix.it.info-6
./guix.it.info
./version-ko.texi
./guix.ko.info-1
./guix.ko.info-2
./guix.ko.info-3
./guix.ko.info-4
./guix.ko.info-5
./guix.ko.info-6
./guix.ko.info
./version-sk.texi
./guix.sk.info-1
./guix.sk.info-2
./guix.sk.info-3
./guix.sk.info-4
./guix.sk.info-5
./guix.sk.info-6
./guix.sk.info
./guix-cookbook.es.info
./guix-cookbook.fa.info
./guix-cookbook.fi.info
./guix-cookbook.pt_BR.info
./guix-cookbook.ru.info
./guix-cookbook.zh_Hans.info
./he-config-bare-bones.scm
./guix-lint.1
./build.scm
./contributing.texi
./guix-cookbook.texi
./htmlxref.cnf
./local.mk
./guix-cookbook.info
./guix-gc.1
./guix-git.1
./guix-container.1
./guix-copy.1
./guix-describe.1
./guix-processes.1
./guix-size.1
./guix-weather.1
./guix-shell.1
./guix-cookbook.ko.texi
./guix-cookbook.fr.texi
./guix-cookbook.de.texi
./guix-cookbook.sk.texi
./contributing.zh_CN.texi
./contributing.ru.texi
./contributing.pt_BR.texi
./contributing.fr.texi
./contributing.es.texi
./contributing.de.texi
./guix.zh_CN.texi
./guix.ru.texi
./guix.pt_BR.texi
./guix.de.texi
./guix.es.texi
./guix.fr.texi
./guix-cookbook.sk.info
./guix-cookbook.de.info
./guix-cookbook.fr.info
./guix-cookbook.ko.info
./version-de.texi
./version-ru.texi
./version-fr.texi
./version-es.texi
./version-pt_BR.texi
./version-zh_CN.texi
./guix-daemon.1
./guix.pt_BR.info-1
./guix.pt_BR.info-2
./guix.fr.info-1
./guix.fr.info-2
./guix.de.info-1
./guix.de.info-2
./guix.pt_BR.info-3
./guix.ru.info-1
./guix.ru.info-2
./guix.fr.info-3
./guix.ru.info-3
./guix.zh_CN.info-1
./guix.zh_CN.info-2
./guix.es.info-1
./guix.es.info-2
./guix.de.info-3
./guix.pt_BR.info-4
./guix.fr.info-4
./guix.ru.info-4
./guix.zh_CN.info-3
./guix.de.info-4
./guix.es.info-3
./guix.ru.info-5
./guix.pt_BR.info-5
./guix.zh_CN.info-4
./guix.fr.info-5
./guix.es.info-4
./guix.de.info-5
./guix.pt_BR.info-6
./guix.ru.info-6
./guix.fr.info-6
./guix.pt_BR.info-7
./guix.es.info-5
./guix.zh_CN.info-5
./guix.de.info-6
./guix.pt_BR.info
./guix.ru.info-7
./guix.fr.info-7
./guix.ru.info-8
./guix.de.info-7
./guix.es.info-6
./guix.fr.info-8
./guix.zh_CN.info-6
./guix.ru.info
./guix.fr.info
./guix.de.info-8
./guix.zh_CN.info-7
./guix.es.info-7
./guix.de.info
./guix.zh_CN.info
./guix.es.info-8
./guix.es.info
./guix.texi
./version.texi
./guix.info-1
./guix.info-2
./guix.info-3
./guix.info-4
./guix.info-5
./guix.info-6
./guix.info-7
./guix.info
./guix-build.1
./guix-archive.1
./guix-import.1
./guix-offload.1
./guix-graph.1
./guix-hash.1
./guix-challenge.1
./guix-download.1
./guix-edit.1
./guix-repl.1
./guix-publish.1
./guix-package.1
./guix-style.1
./guix-pull.1
./guix-time-machine.1
./guix-environment.1
./guix-pack.1
./guix-deploy.1
./guix-refresh.1
./guix-home.1
./guix-system.1
./guix.1

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQX/18UHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pyHCg//Y+CwbxC5OzVdZFBdSDKYtxOV2w6Z
w6X/ZXRt5sorx2Wfjsn0QJIzO9+6CTEBn1RSeOCQxuVvlQqjdeKqxQ0C90YQQG5t
/R7PZ5wT2LHhztT1R52O8njqj93bppxkJDMsZVg4lg38WzNt2jVvL94mczS1V3zH
4VpccvDHYjIwK/ECkhTEKHRvRSGFqH7VcGJYbKYwGjmhF1Q6+lWq4XBiBhF9C6G0
+TwsS/GMXi207dFEtXdhit6DcxbEi70EseNj55aLDoM9xUumlQJuVTz5f9FRbL/C
QOQmI2xFMPqN2EwZhBX3YL7FjwooA+nyyrj7H0AjWe3IX4RuVyvF9XZNIm1HM4P3
VlcR1foBdKxo1JRCTCU7V/hlD44Yl3e/FXQbKIS6aQyhRahrudYjeeL1gf7jBRya
FAzkv2kP/ORoxP/bKj/BoNV372fOz8Sm5sSr5YzAZJz4DJQmctstIw+6WFcgP8Ca
az682R4sMLDWteAxhZp/NNSg21irM6S7U4Tu4em3nP69ky2Fs5wUy7m1JzMWx9Vg
IrODYJdDN0P5wgSM5hVKAW30J8uLXs3XxwRAl2W+NKCu8l4eonnpKU3KufaSclG+
Qz7cg6xNqTLzvDrYoDaaMhGd2WGJOdQ3wzH3TcWgR9Kh1STNwNnYfXXsZPvYQ5wi
QWmUkZU8RZRQ3BI=
=rP1f
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 20 Mar 17:51 +0100
Re: bug#62153: [PATCH 0/2] Disarchive vs Gash-Utils for docker-layered
(name . Ludovic Courtès)(address . ludo@gnu.org)
87cz53bc2f.fsf_-_@gmail.com
Oleg Pykhalov <go.wigust@gmail.com> writes:

Toggle quote (4 lines)
> […]
>
> And disarchive-assemble does not work, if I don't miss anything.

I forgot about that input tarball was generated with changing current
working directory, that's probably the reason of the broken archive.

According to "Assembles from stdin to file" test from
git.ngyro.com/disarchive/tests/cli.scm it is possible to generate a
tarball for docker-layered with Disarchive, but a file describing spec
like git.ngyro.com/disarchive/tests/data/test-archive.da is required.
Generation of this file without having a Tar archive beforehand is
complicated and because of that IMHO ustar.scm from Gash-Utils is
preferred for the task.

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQYjygUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzbEhAAy76WdREdgRRwg4j9h83H8dUa30x7
0+6pjM24kKBrDJYLGmv6aW2pumAr+YVre7n4/n1HdlXLStcdJ86/G0zraPj1wD7+
JvR04l0qGb7LY87lPdU/bGGkyt+YubnGbhZ6XoU0Q4CmHHo2aJzvq1S6+Z98m6YN
f/RWl2mkxnrXDau+rLwE2RXQ4wvjZIMg77SSI64lgWoo3IaiwOuhWxcs2Q+ImebS
ymirYoVzhFPCBgznkD2otEDNOvkUMxJ/e2caVGZ21F3guHhq0Df2gmYhRvaIqj/S
JH8xfguqOTntBSvzhi0zjj50A+pyn976IV2C2Mu5mAqtQB0HpSayF/79Y3kwYXYH
bteVg+tsGRrhC4VTPO3Tf5w2PE+SWr7j6ssSNNyRC9wcoYQjeE1y/btYEUeR85vc
L3TS5uD/KM9WJmmpzxyAnmAGR/WkXAvFDtMZ7ZNcq6qTzQVQYltNJD/K69UZ55c4
vPU4KrnEx49lvtnMDzxl8BhzZsgVGjG6k26iX3B5ZyGWpGGayellDVwPWP21MLPr
1R4wanmcyykJvLl1iAVkDDNCRY8cJn4QKQDmUd0Y3gkSJi4Ic7zqYHzIFaEHbLda
/jJ1XB1vTsadXQTJAIdOIQw0fXbK9NF7Tm68nm+czF6AyjUYU5Nz6NmQ85zAEAH4
1cpPWOSj0xOsoYQ=
=uXt9
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 31 May 10:45 +0200
[PATCH] Add Docker layered image for pack and system (v3)
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
cover.1685522135.git.go.wigust@gmail.com
Hi, Guix.

These patches series is rebased on origin/master. Also, the Python script is
replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed tests:
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system"
make check-system TESTS="docker-layered-system"

Oleg Pykhalov (2):
guix: docker: Build layered image.
news: Add entry for the new 'docker-layered' distribution format.

doc/guix.texi | 18 +++-
etc/news.scm | 58 ++++++++++++
gnu/image.scm | 3 +-
gnu/system/image.scm | 76 +++++++++++----
gnu/tests/docker.scm | 20 +++-
guix/docker.scm | 205 +++++++++++++++++++++++++++++++---------
guix/scripts/pack.scm | 62 ++++++++++--
guix/scripts/system.scm | 11 ++-
tests/pack.scm | 48 ++++++++++
9 files changed, 424 insertions(+), 77 deletions(-)


base-commit: 77f52db416a13e195d090cad4e9e7658feb2e86b
--
2.38.0
O
O
Oleg Pykhalov wrote on 31 May 10:47 +0200
[PATCH] guix: docker: Build layered image.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
dd6c7c816bcb414682e1006d7e83b45e8ac6c575.1685522135.git.go.wigust@gmail.com
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image Reference): Same.
(image-type Reference): Document docker-layered-image-type.
* gnu/image.scm
(validate-image-format)[docker-layered]: New image format.
* gnu/system/image.scm
(docker-layered-image, docker-layered-image-type): New variables.
(system-docker-image)[layered-image?]: New argument.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* tests/pack.scm: Add "docker-layered-image + localstatedir" test.
---
doc/guix.texi | 18 +++-
gnu/image.scm | 3 +-
gnu/system/image.scm | 76 +++++++++++----
gnu/tests/docker.scm | 20 +++-
guix/docker.scm | 205 +++++++++++++++++++++++++++++++---------
guix/scripts/pack.scm | 62 ++++++++++--
guix/scripts/system.scm | 11 ++-
tests/pack.scm | 48 ++++++++++
8 files changed, 366 insertions(+), 77 deletions(-)

Toggle diff (436 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 5fd2449ed5..1c95ec4320 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6984,9 +6984,15 @@ Invoking guix pack
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -44309,6 +44315,8 @@ image Reference
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
@@ -44644,6 +44652,10 @@ image-type Reference
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-syntax-rule (define-set-sanitizer name field set)
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..3a502f19ec 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -78,6 +79,7 @@ (define-module (gnu system image)
efi-disk-image
iso9660-image
docker-image
+ docker-layered-image
tarball-image
wsl2-image
raw-with-offset-disk-image
@@ -89,6 +91,7 @@ (define-module (gnu system image)
iso-image-type
uncompressed-iso-image-type
docker-image-type
+ docker-layered-image-type
tarball-image-type
wsl2-image-type
raw-with-offset-image-type
@@ -167,6 +170,10 @@ (define docker-image
(image-without-os
(format 'docker)))
+(define docker-layered-image
+ (image-without-os
+ (format 'docker-layered)))
+
(define tarball-image
(image-without-os
(format 'tarball)))
@@ -237,6 +244,11 @@ (define docker-image-type
(name 'docker)
(constructor (cut image-with-os docker-image <>))))
+(define docker-layered-image-type
+ (image-type
+ (name 'docker-layered)
+ (constructor (cut image-with-os docker-layered-image <>))))
+
(define tarball-image-type
(image-type
(name 'tarball)
@@ -633,9 +645,12 @@ (define (image-with-label base-image label)
(define* (system-docker-image image
#:key
- (name "docker-image"))
+ (name "docker-image")
+ (archiver tar)
+ layered-image?)
"Build a docker image for IMAGE. NAME is the base name to use for the
-output file."
+output file. If LAYERED-IMAGE? is true, the image will with many of the store
+paths being on their own layer to improve sharing between images."
(define boot-program
;; Program that runs the boot script of OS, which in turn starts shepherd.
(program-file "boot-program"
@@ -678,9 +693,11 @@ (define* (system-docker-image image
(use-modules (guix docker)
(guix build utils)
(gnu build image)
+ (srfi srfi-1)
(srfi srfi-19)
(guix build store-copy)
- (guix store database))
+ (guix store database)
+ (ice-9 receive))
;; Set the SQL schema location.
(sql-schema #$schema)
@@ -700,18 +717,31 @@ (define* (system-docker-image image
#:register-closures? #$register-closures?
#:deduplicate? #f
#:system-directory #$os)
- (build-docker-image
- #$output
- (cons* image-root
- (map store-info-item
- (call-with-input-file #$graph
- read-reference-graph)))
- #$os
- #:entry-point '(#$boot-program #$os)
- #:compressor '(#+(file-append gzip "/bin/gzip") "-9n")
- #:creation-time (make-time time-utc 0 1)
- #:system #$image-target
- #:transformations `((,image-root -> ""))))))))
+ (when #$layered-image?
+ (setenv "PATH"
+ (string-join (list #+(file-append archiver "/bin")
+ #+(file-append coreutils "/bin")
+ #+(file-append gzip "/bin"))
+ ":")))
+ (apply build-docker-image
+ (append (list #$output
+ (append (if #$layered-image?
+ '()
+ (list image-root))
+ (map store-info-item
+ (call-with-input-file #$graph
+ read-reference-graph)))
+ #$os
+ #:entry-point '(#$boot-program #$os)
+ #:compressor
+ '(#+(file-append gzip "/bin/gzip") "-9n")
+ #:creation-time (make-time time-utc 0 1)
+ #:system #$image-target
+ #:transformations `((,image-root -> "")))
+ (if #$layered-image?
+ (list #:root-system image-root
+ #:layered-image? #$layered-image?)
+ '()))))))))
(computed-file name builder
;; Allow offloading so that this I/O-intensive process
@@ -720,6 +750,18 @@ (define* (system-docker-image image
#:options `(#:references-graphs ((,graph ,os))
#:substitutable? ,substitutable?))))
+(define* (system-docker-layered-image image
+ #:key
+ (name "docker-image")
+ (archiver tar)
+ (layered-image? #t))
+ "Build a docker image for IMAGE. NAME is the base name to use for the
+output file."
+ (system-docker-image image
+ #:name name
+ #:archiver archiver
+ #:layered-image? layered-image?))
+
;;;
;;; Tarball image.
@@ -811,7 +853,7 @@ (define (image->root-file-system image)
"Return the IMAGE root partition file-system type."
(case (image-format image)
((iso9660) "iso9660")
- ((docker tarball wsl2) "dummy")
+ ((docker docker-layered tarball wsl2) "dummy")
(else
(partition-file-system (find-root-partition image)))))
@@ -948,6 +990,8 @@ (define* (system-image image)
("bootcfg" ,bootcfg))))
((memq image-format '(docker))
(system-docker-image image*))
+ ((memq image-format '(docker-layered))
+ (system-docker-layered-image image*))
((memq image-format '(tarball))
(system-tarball-image image*))
((memq image-format '(wsl2))
diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm
index edc9804414..0cccc02ad2 100644
--- a/gnu/tests/docker.scm
+++ b/gnu/tests/docker.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2019 Danny Milosavljevic <dannym@scratchpost.org>
;;; Copyright © 2019-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -43,7 +44,8 @@ (define-module (gnu tests docker)
#:use-module (guix build-system trivial)
#:use-module ((guix licenses) #:prefix license:)
#:export (%test-docker
- %test-docker-system))
+ %test-docker-system
+ %test-docker-layered-system))
(define %docker-os
(simple-operating-system
@@ -316,3 +318,19 @@ (define %test-docker-system
(locale-libcs (list glibc)))
#:type docker-image-type)))
run-docker-system-test)))))
+
+(define %test-docker-layered-system
+ (system-test
+ (name "docker-layered-system")
+ (description "Run a system image as produced by @command{guix system
+docker-layered-image} inside Docker.")
+ (value (with-monad %store-monad
+ (>>= (lower-object
+ (system-image (os->image
+ (operating-system
+ (inherit (simple-operating-system))
+ ;; Use locales for a single libc to
+ ;; reduce space requirements.
+ (locale-libcs (list glibc)))
+ #:type docker-layered-image-type)))
+ run-docker-system-test)))))
diff --git a/guix/docker.scm b/guix/docker.scm
index 5e6460f43f..e10b940aa4 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -28,6 +29,8 @@ (define-module (guix docker)
delete-file-recursively
with-directory-excursion
invoke))
+ #:use-module (guix diagnostics)
+ #:use-module (guix i18n)
#:use-module (gnu build install)
#:use-module (json) ;guile-json
#:use-module (srfi srfi-1)
@@ -38,6 +41,9 @@ (define-module (guix docker)
#:use-module (rnrs bytevectors)
#:use-module (ice-9 ftw)
#:use-module (ice-9 match)
+ #:use-module (ice-9 popen)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 receive)
#:export (build-docker-image))
;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image.
@@ -92,12 +98,12 @@ (define (canonicalize-repository-name name)
(make-string (- min-length l) padding-character)))
(_ normalized-name))))
-(define* (manifest path id #:optional (tag "guix"))
+(define* (manifest path layers #:optional (tag "guix"))
"Generate a simple image manifest."
(let ((tag (canonicalize-repository-name tag)))
`#(((Config . "config.json")
(RepoTags . #(,(string-append tag ":latest")))
- (Layers . #(,(string-append id "/layer.tar")))))))
+ (Layers . ,(list->vector layers))))))
;; According to the specifications this is required for backwards
;; compatibility. It duplicates information provided by the manifest.
@@ -106,8 +112,8 @@ (define* (repositories path id #:optional (tag "guix"))
`((,(canonicalize-repository-name tag) . ((latest . ,id)))))
;; See https://github.com/opencontainers/image-spec/blob/master/config.md
-(define* (config layer time arch #:key entry-point (environment '()))
- "Generate a minimal image configuration for the given LAYER file."
+(define* (config layers-diff-ids time arch #:key entry-point (environment '()))
+ "Generate a minimal image configuration for the given LAYERS files."
;; "architecture" must be values matching "platform.arch" in the
;; runtime-spec at
;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform
@@ -125,7 +131,7 @@ (define* (config layer time arch #:key entry-point (environment '()))
(container_config . #nil)
(os . "linux")
(rootfs . ((type . "layers")
- (diff_ids . #(,(layer-diff-id layer)))))))
+ (diff_ids . ,(list->vector layers-diff-ids))))))
(define directive-file
;; Return the file or directory created by a 'evaluate-populate-directive'
@@ -136,6 +142,37 @@ (define directive-file
(('directory name _ ...)
(string-trim name #\/))))
+(define %docker-image-max-layers
+ 100)
+
+(define (paths-split-sort paths)
+ "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage."
+ (let* ((paths-length (length paths))
+ (port (apply open-pipe* OPEN_READ
+ (append '("du" "--summarize") paths)))
+ (output (read-string port)))
+ (close-port port)
+ (receive (head tail)
+ (split-at
+ (map (match-lambda ((size . path) path))
+ (sort (map (lambda (line)
+ (match (string-split line #\tab)
+ ((size path)
+ (cons (string->number size) path))))
+ (string-split
+ (string-trim-right output #\newline)
+ #\newline))
+ (lambda (path1 path2)
+ (< (match path2 ((size . _) size))
+ (match path1 ((size . _) size))))))
+ (if (>= paths-length %docker-image-max-layers)
+ (- %docker-image-max-layers 2)
+ (1- paths-length)))
+ (list head tail))))
+
+(define (create-empty-tar file)
+ (invoke "tar" "-cf" file "--files-from" "/dev/null"))
+
(define* (build-docker-image image paths prefix
#:key
(repository "guix")
@@ -146,11 +183,13 @@ (define* (build-docker-image image paths prefix
entry-point
(environment '())
compressor
- (creation-time (current-time time-utc)))
- "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX
-must be a store path that is a prefix of any store paths in PATHS. REPOSITORY
-is a descriptive name that will show up in \"REPOSITORY\" column of the output
-of \"docker images\".
+ (creation-time (current-time time-utc))
+ layered-image?
+ root-system)
+ "Write to IMAGE a layerer Docker image archive containing the given PATHS.
+PREFIX must be a store path that is a prefix of any store paths in PATHS.
+REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of
+the output of \"docker images\".
When DATABASE is true, copy it to /var/guix/db in the image and create
/var/guix/gcroots and friends.
@@ -172,7 +211,14 @@ (define* (build-docker-image image paths prefix
SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in
PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a
command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a
-SRFI-19 time-utc object, as the creation time in metadata."
+SRFI-19 time-utc object, as the creation time in metadata.
+
+When LAYERED-IMAGE? is true build layered image, providing a Docker
+image with many of the store paths being on their own layer to improve sharing
+between images.
+
+ROOT-SYSTEM is a directory with a provisioned root file system, which will be
+added to image as a layer."
(define (sanitize path-fragment)
(escape-special-chars
;; GNU tar strips the leading slash off of absolute paths before applying
@@ -203,6 +249,53 @@ (define* (build-docker-image image paths prefix
(if (eq? '() transformations)
'()
`("--transform" ,(transformations->expression transformations))))
+ (define layers-hashes
+ (match-lambda
+ (((head ...) (tail ...) id)
+ (create-empty-tar "image.tar")
+ (let* ((head-layers
+ (map
+ (lambda (file)
+ (invoke "tar" "cf" "layer.tar" file)
+ (let* ((file-hash (layer-diff-id "layer.tar"))
+ (file-name (string-append file-hash "/layer.tar")))
+ (mkdir file-hash)
+ (rename-file "layer.tar" file-name)
+ (invoke "tar" "-rf" "image.tar" file-name)
+ (delete-file file-name)
+ file-hash))
+ head))
+ (tail-layer
+ (begin
+ (create-empty-tar "layer.tar")
+ (for-each
This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 31 May 10:47 +0200
[PATCH] news: Add entry for the new 'docker-layered' distribution format.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
e4326a843b52be3147cd9d495cb7bf92d0fab8ce.1685522135.git.go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

Toggle diff (78 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 314f0ab352..cb2dc34876 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021, 2023 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,63 @@
(channel-news
(version 0)
+ (entry (commit "dd6c7c816bcb414682e1006d7e83b45e8ac6c575")
+ (title
+ (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "????? @samp{docker-layered} ?????? ??? @command{guix pack} ???????"))
+ (body
+ (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
+@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
+Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
+untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
+im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
+Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
+angelegt wird:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
+indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
+übergeben.
+
+Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
+@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")
+ (en "Docker layered images can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+The @command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+
+See @command{info \"(guix) Invoking guix pack\"} and
+@command{info \"(guix) System Images\"} for more information.")
+ (ru "????????? ??????? ???????? ???????????? Docker ??????? ? ???????
+@command{guix pack --format=docker-layered}, ??????? ??????? Docker ????? ?
+?????? ? store ?????????????? ?? ????????? ?????, ??????? ????? ???????
+???????? ???????. ????? ????? ?????? ? GNU store ? ???????? gzipped tarball.
+
+?????? ???????? Docker layered ????? ? @code{hello} ???????:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} ?????? ????? ????????? layered Docker ????? ?????
+???????? ? ????? @option{--image-type} ????????? @code{docker-layered}.
+
+???????? @command{info \"(guix) Invoking guix pack\"} ?
+@command{info \"(guix) System Images\"} ??? ????????? ????? ?????????
+????????.")))
+
(entry (commit "ba5da5125a81307500982517e2f458d57b024668")
(title
(en "New @code{arguments} rule for @command{guix style}")
--
2.38.0
G
G
Greg Hogan wrote on 31 May 14:53 +0200
Re: [bug#62153] [PATCH] Add Docker layered image for pack and system (v3)
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)(address . 62153@debbugs.gnu.org)
CA+3U0Z=wZ6NaNX=rRz7m3CGwy8VA_Koe1Cv6Y-h3z7a7qMLz6Q@mail.gmail.com
On Wed, May 31, 2023 at 4:46?AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
Toggle quote (14 lines)
>
> Hi, Guix.
>
> These patches series is rebased on origin/master. Also, the Python script is
> replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed tests:
> make check TESTS="tests/pack.scm"
> make check-system TESTS="docker-system"
> make check-system TESTS="docker-layered-system"
>
>
> Oleg Pykhalov (2):
> guix: docker: Build layered image.
> news: Add entry for the new 'docker-layered' distribution format.

Why not use layered images for all docker packs?
O
O
Oleg Pykhalov wrote on 31 May 15:14 +0200
(name . Greg Hogan)(address . code@greghogan.com)(address . 62153@debbugs.gnu.org)
87cz2gd4mm.fsf@gmail.com
Greg Hogan <code@greghogan.com> writes:

Toggle quote (17 lines)
> On Wed, May 31, 2023 at 4:46?AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
>>
>> Hi, Guix.
>>
>> These patches series is rebased on origin/master. Also, the Python script is
>> replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed tests:
>> make check TESTS="tests/pack.scm"
>> make check-system TESTS="docker-system"
>> make check-system TESTS="docker-layered-system"
>>
>>
>> Oleg Pykhalov (2):
>> guix: docker: Build layered image.
>> news: Add entry for the new 'docker-layered' distribution format.
>
> Why not use layered images for all docker packs?

Do you mean use layered images by default without ability to build all
in a single layer? Current layered implementation is slow to build
because it needs to calculate a size of each layer, pack, and compress.
So if user wants a faster build, a non-layered image is still an option.

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmR3SFEUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzUmA/9Hqd7U8yB00GA/wF1F7TU/3AcpaFP
uY6LHfUB0ZePuu5r3cQcx2LlCISJ6KLrXOk+d6vTDc+WK4TpTQrH5V5Gt8WAU3L2
Nt58WtDiZnZwX6uuO1Dxk1efF4eJRIIzLqtXluDzJ+jpCj2Q9M43Js8EODRk21rC
w+j+e5Xq2xF3R7aBaGRUN8euFI6nyfZgZC2kiuOAc9xBCt6hMmOPEm4mwaqIUABG
BSa4Nr/Qv0W/GbUEhWgJOWI4NGljbanWLuRlC5IXd4tHEAU2p7lEGg4uWm1Brm+r
2mvHfu0y8tLwnb8F9SjscjBG3OXieM3YjmYeTeOSiNRY5tqnOTo9o4Kq7wmeezQU
02FYA3+U7vvIJ7WCWbEowZgv4ZEW3UCsDiNdt0+4GR72J10jR/MTdFVOhxV0Heo6
4alOxBm0qoLJh9+3dYwY6wEYlWfAwIPXaLeGX4F4GQ0YdPsMFMCCeSdkz1vN4A6l
1q8A29IAuRQcLxv9G3T9+bO4tsYpN7jtB6c8M/4+na03Tc/ugAOWIY+Ne0wBBAAA
k1sqv69JljPC9rE1gaxwUU7FTkY8/pbhGzH0MFZG/C7qUYbz4L5Z7ANzlAnhZGWl
isVFrq2VkTC0XZjxISaQEUsRLcAP82HrTVDfV7KIsK3iuczQVWVPJXgu/rgJ9rgi
rMJcHluPJHJ5IUc=
=mfr+
-----END PGP SIGNATURE-----

G
G
Greg Hogan wrote on 2 Jun 19:02 +0200
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)(address . 62153@debbugs.gnu.org)
CA+3U0Z=VktVBTjEZ5dYmLUJFoFPXjhDTw9aO-MrmLx1BV2cjcg@mail.gmail.com
On Wed, May 31, 2023 at 9:14?AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
[...]
Toggle quote (8 lines)
> Do you mean use layered images by default without ability to build all
> in a single layer? Current layered implementation is slow to build
> because it needs to calculate a size of each layer, pack, and compress.
> So if user wants a faster build, a non-layered image is still an option.
>
> Regards,
> Oleg.

I am trying out your patch, and wanted to benchmark the runtime
difference between docker and docker-layered packs, but the latter
looks to be failing with any compression other than the default gzip.
In particular, I was looking to disable compression with
'--compression=none'.
O
O
Oleg Pykhalov wrote on 3 Jun 21:10 +0200
Re: bug#62153: [PATCH 0/2] Add Docker layered image for pack and system
(name . Greg Hogan)(address . code@greghogan.com)(address . 62153@debbugs.gnu.org)
87y1l0icq0.fsf_-_@gmail.com
Greg Hogan <code@greghogan.com> writes:

Toggle quote (16 lines)
> On Wed, May 31, 2023 at 9:14?AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
> [...]
>> Do you mean use layered images by default without ability to build all
>> in a single layer? Current layered implementation is slow to build
>> because it needs to calculate a size of each layer, pack, and compress.
>> So if user wants a faster build, a non-layered image is still an option.
>>
>> Regards,
>> Oleg.
>
> I am trying out your patch, and wanted to benchmark the runtime
> difference between docker and docker-layered packs, but the latter
> looks to be failing with any compression other than the default gzip.
> In particular, I was looking to disable compression with
> '--compression=none'.

I'll send a fixed v4 revision for '--compression=none'. Unfortunately,
because we cannot append to an existing compressed tarball:

tar: Cannot update compressed archives
Try 'tar --help' or 'tar --usage' for more information.

adding more compression types requires to write a handler for every
compressor separately in guix/docker.scm file:
Toggle snippet (10 lines)
(if layered-image?
(begin
(invoke "tar" "-rf" "image.tar" "config.json")
(if compressor
(begin
(apply invoke `(,@compressor "image.tar"))
(copy-file "image.tar.gz" image))
(copy-file "image.tar" image)))

I would like to vote that addional compressors could be added later if
needed.

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmR7kBgUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pxoKA//fUia6BxwXHIQNdFmQueeIVN7RUvr
gx19n9Qpe3rmkqRwdXqqk9V81daqSYtEwm9/vpvo2HO85yliUxkMUOiHRQLJxOPQ
X0Vx1v871DyXkdg4D9Jh1/GE6sjqcT8Jka5u8pFB5LAPVto12tqy+0+lMxP4AcqX
X6GxWuEClj6yB+nqtE/00xul6qDF1YaHAeOrMJjpYf4y1dwBYw9WFQwJJvVuXxdE
WukW6F9qTqj9qXrG0+EkamWQjWcmKrSkv7+aTskIK/xdEuXor2OSOC2o6Tff7K7p
GeePvEdDIGetiV5bNVU4EwewvJH/oF/FFZcPdvBx0DCUJ8AeVMEzwqnWExTW/UXe
pTqQH3dwTNhnh1tDhAAEKXhcBuOntmiCp5knArmCCirpzyW+c7rZ/F8JVC5vZJTf
PS3x9O+R3ZkueoimoQzSd/JGPSH1eh2O8SMNy42QB1w4j3SLEkv0ShsswCbkdMhk
OXy/wjqo9GJRRMRNTYAxeu7OLDOwM5RrhBX3s2g0Nu+6YEa3ISDU7s4qEBbQHNFW
lD0dplzXNZudhLab/xTP/ptgqsh7SoaYszsRitz+noT35DuaJt7wwm7Qp+rn55ve
PqMd+P94YCrzq8RslsUfpEE+VBJKxJbFnNTZdkaDqAI/HCwWSSe9gzag3zsYpQDx
8blQqUJbYRIXBiE=
=xQTj
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 3 Jun 21:14 +0200
[PATCH v4 1/2] guix: docker: Build layered image.
(address . 62153@debbugs.gnu.org)
457c813653a44117e296deaa49e79fc701b90791.1685819700.git.go.wigust@gmail.com
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image Reference): Same.
(image-type Reference): Document docker-layered-image-type.
* gnu/image.scm
(validate-image-format)[docker-layered]: New image format.
* gnu/system/image.scm
(docker-layered-image, docker-layered-image-type): New variables.
(system-docker-image)[layered-image?]: New argument.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* tests/pack.scm: Add "docker-layered-image + localstatedir" test.
---
doc/guix.texi | 18 +++-
gnu/image.scm | 3 +-
gnu/system/image.scm | 76 +++++++++++----
gnu/tests/docker.scm | 20 +++-
guix/docker.scm | 208 +++++++++++++++++++++++++++++++---------
guix/scripts/pack.scm | 62 ++++++++++--
guix/scripts/system.scm | 11 ++-
tests/pack.scm | 48 ++++++++++
8 files changed, 369 insertions(+), 77 deletions(-)

Toggle diff (436 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 7f8d8d66e9..483be6ef16 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6984,9 +6984,15 @@ Invoking guix pack
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -44347,6 +44353,8 @@ image Reference
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
@@ -44682,6 +44690,10 @@ image-type Reference
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-syntax-rule (define-set-sanitizer name field set)
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..3a502f19ec 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -78,6 +79,7 @@ (define-module (gnu system image)
efi-disk-image
iso9660-image
docker-image
+ docker-layered-image
tarball-image
wsl2-image
raw-with-offset-disk-image
@@ -89,6 +91,7 @@ (define-module (gnu system image)
iso-image-type
uncompressed-iso-image-type
docker-image-type
+ docker-layered-image-type
tarball-image-type
wsl2-image-type
raw-with-offset-image-type
@@ -167,6 +170,10 @@ (define docker-image
(image-without-os
(format 'docker)))
+(define docker-layered-image
+ (image-without-os
+ (format 'docker-layered)))
+
(define tarball-image
(image-without-os
(format 'tarball)))
@@ -237,6 +244,11 @@ (define docker-image-type
(name 'docker)
(constructor (cut image-with-os docker-image <>))))
+(define docker-layered-image-type
+ (image-type
+ (name 'docker-layered)
+ (constructor (cut image-with-os docker-layered-image <>))))
+
(define tarball-image-type
(image-type
(name 'tarball)
@@ -633,9 +645,12 @@ (define (image-with-label base-image label)
(define* (system-docker-image image
#:key
- (name "docker-image"))
+ (name "docker-image")
+ (archiver tar)
+ layered-image?)
"Build a docker image for IMAGE. NAME is the base name to use for the
-output file."
+output file. If LAYERED-IMAGE? is true, the image will with many of the store
+paths being on their own layer to improve sharing between images."
(define boot-program
;; Program that runs the boot script of OS, which in turn starts shepherd.
(program-file "boot-program"
@@ -678,9 +693,11 @@ (define* (system-docker-image image
(use-modules (guix docker)
(guix build utils)
(gnu build image)
+ (srfi srfi-1)
(srfi srfi-19)
(guix build store-copy)
- (guix store database))
+ (guix store database)
+ (ice-9 receive))
;; Set the SQL schema location.
(sql-schema #$schema)
@@ -700,18 +717,31 @@ (define* (system-docker-image image
#:register-closures? #$register-closures?
#:deduplicate? #f
#:system-directory #$os)
- (build-docker-image
- #$output
- (cons* image-root
- (map store-info-item
- (call-with-input-file #$graph
- read-reference-graph)))
- #$os
- #:entry-point '(#$boot-program #$os)
- #:compressor '(#+(file-append gzip "/bin/gzip") "-9n")
- #:creation-time (make-time time-utc 0 1)
- #:system #$image-target
- #:transformations `((,image-root -> ""))))))))
+ (when #$layered-image?
+ (setenv "PATH"
+ (string-join (list #+(file-append archiver "/bin")
+ #+(file-append coreutils "/bin")
+ #+(file-append gzip "/bin"))
+ ":")))
+ (apply build-docker-image
+ (append (list #$output
+ (append (if #$layered-image?
+ '()
+ (list image-root))
+ (map store-info-item
+ (call-with-input-file #$graph
+ read-reference-graph)))
+ #$os
+ #:entry-point '(#$boot-program #$os)
+ #:compressor
+ '(#+(file-append gzip "/bin/gzip") "-9n")
+ #:creation-time (make-time time-utc 0 1)
+ #:system #$image-target
+ #:transformations `((,image-root -> "")))
+ (if #$layered-image?
+ (list #:root-system image-root
+ #:layered-image? #$layered-image?)
+ '()))))))))
(computed-file name builder
;; Allow offloading so that this I/O-intensive process
@@ -720,6 +750,18 @@ (define* (system-docker-image image
#:options `(#:references-graphs ((,graph ,os))
#:substitutable? ,substitutable?))))
+(define* (system-docker-layered-image image
+ #:key
+ (name "docker-image")
+ (archiver tar)
+ (layered-image? #t))
+ "Build a docker image for IMAGE. NAME is the base name to use for the
+output file."
+ (system-docker-image image
+ #:name name
+ #:archiver archiver
+ #:layered-image? layered-image?))
+
;;;
;;; Tarball image.
@@ -811,7 +853,7 @@ (define (image->root-file-system image)
"Return the IMAGE root partition file-system type."
(case (image-format image)
((iso9660) "iso9660")
- ((docker tarball wsl2) "dummy")
+ ((docker docker-layered tarball wsl2) "dummy")
(else
(partition-file-system (find-root-partition image)))))
@@ -948,6 +990,8 @@ (define* (system-image image)
("bootcfg" ,bootcfg))))
((memq image-format '(docker))
(system-docker-image image*))
+ ((memq image-format '(docker-layered))
+ (system-docker-layered-image image*))
((memq image-format '(tarball))
(system-tarball-image image*))
((memq image-format '(wsl2))
diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm
index edc9804414..0cccc02ad2 100644
--- a/gnu/tests/docker.scm
+++ b/gnu/tests/docker.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2019 Danny Milosavljevic <dannym@scratchpost.org>
;;; Copyright © 2019-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -43,7 +44,8 @@ (define-module (gnu tests docker)
#:use-module (guix build-system trivial)
#:use-module ((guix licenses) #:prefix license:)
#:export (%test-docker
- %test-docker-system))
+ %test-docker-system
+ %test-docker-layered-system))
(define %docker-os
(simple-operating-system
@@ -316,3 +318,19 @@ (define %test-docker-system
(locale-libcs (list glibc)))
#:type docker-image-type)))
run-docker-system-test)))))
+
+(define %test-docker-layered-system
+ (system-test
+ (name "docker-layered-system")
+ (description "Run a system image as produced by @command{guix system
+docker-layered-image} inside Docker.")
+ (value (with-monad %store-monad
+ (>>= (lower-object
+ (system-image (os->image
+ (operating-system
+ (inherit (simple-operating-system))
+ ;; Use locales for a single libc to
+ ;; reduce space requirements.
+ (locale-libcs (list glibc)))
+ #:type docker-layered-image-type)))
+ run-docker-system-test)))))
diff --git a/guix/docker.scm b/guix/docker.scm
index 5e6460f43f..b40cfb2374 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -28,6 +29,8 @@ (define-module (guix docker)
delete-file-recursively
with-directory-excursion
invoke))
+ #:use-module (guix diagnostics)
+ #:use-module (guix i18n)
#:use-module (gnu build install)
#:use-module (json) ;guile-json
#:use-module (srfi srfi-1)
@@ -38,6 +41,9 @@ (define-module (guix docker)
#:use-module (rnrs bytevectors)
#:use-module (ice-9 ftw)
#:use-module (ice-9 match)
+ #:use-module (ice-9 popen)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 receive)
#:export (build-docker-image))
;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image.
@@ -92,12 +98,12 @@ (define (canonicalize-repository-name name)
(make-string (- min-length l) padding-character)))
(_ normalized-name))))
-(define* (manifest path id #:optional (tag "guix"))
+(define* (manifest path layers #:optional (tag "guix"))
"Generate a simple image manifest."
(let ((tag (canonicalize-repository-name tag)))
`#(((Config . "config.json")
(RepoTags . #(,(string-append tag ":latest")))
- (Layers . #(,(string-append id "/layer.tar")))))))
+ (Layers . ,(list->vector layers))))))
;; According to the specifications this is required for backwards
;; compatibility. It duplicates information provided by the manifest.
@@ -106,8 +112,8 @@ (define* (repositories path id #:optional (tag "guix"))
`((,(canonicalize-repository-name tag) . ((latest . ,id)))))
;; See https://github.com/opencontainers/image-spec/blob/master/config.md
-(define* (config layer time arch #:key entry-point (environment '()))
- "Generate a minimal image configuration for the given LAYER file."
+(define* (config layers-diff-ids time arch #:key entry-point (environment '()))
+ "Generate a minimal image configuration for the given LAYERS files."
;; "architecture" must be values matching "platform.arch" in the
;; runtime-spec at
;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform
@@ -125,7 +131,7 @@ (define* (config layer time arch #:key entry-point (environment '()))
(container_config . #nil)
(os . "linux")
(rootfs . ((type . "layers")
- (diff_ids . #(,(layer-diff-id layer)))))))
+ (diff_ids . ,(list->vector layers-diff-ids))))))
(define directive-file
;; Return the file or directory created by a 'evaluate-populate-directive'
@@ -136,6 +142,37 @@ (define directive-file
(('directory name _ ...)
(string-trim name #\/))))
+(define %docker-image-max-layers
+ 100)
+
+(define (paths-split-sort paths)
+ "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage."
+ (let* ((paths-length (length paths))
+ (port (apply open-pipe* OPEN_READ
+ (append '("du" "--summarize") paths)))
+ (output (read-string port)))
+ (close-port port)
+ (receive (head tail)
+ (split-at
+ (map (match-lambda ((size . path) path))
+ (sort (map (lambda (line)
+ (match (string-split line #\tab)
+ ((size path)
+ (cons (string->number size) path))))
+ (string-split
+ (string-trim-right output #\newline)
+ #\newline))
+ (lambda (path1 path2)
+ (< (match path2 ((size . _) size))
+ (match path1 ((size . _) size))))))
+ (if (>= paths-length %docker-image-max-layers)
+ (- %docker-image-max-layers 2)
+ (1- paths-length)))
+ (list head tail))))
+
+(define (create-empty-tar file)
+ (invoke "tar" "-cf" file "--files-from" "/dev/null"))
+
(define* (build-docker-image image paths prefix
#:key
(repository "guix")
@@ -146,11 +183,13 @@ (define* (build-docker-image image paths prefix
entry-point
(environment '())
compressor
- (creation-time (current-time time-utc)))
- "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX
-must be a store path that is a prefix of any store paths in PATHS. REPOSITORY
-is a descriptive name that will show up in \"REPOSITORY\" column of the output
-of \"docker images\".
+ (creation-time (current-time time-utc))
+ layered-image?
+ root-system)
+ "Write to IMAGE a layerer Docker image archive containing the given PATHS.
+PREFIX must be a store path that is a prefix of any store paths in PATHS.
+REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of
+the output of \"docker images\".
When DATABASE is true, copy it to /var/guix/db in the image and create
/var/guix/gcroots and friends.
@@ -172,7 +211,14 @@ (define* (build-docker-image image paths prefix
SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in
PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a
command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a
-SRFI-19 time-utc object, as the creation time in metadata."
+SRFI-19 time-utc object, as the creation time in metadata.
+
+When LAYERED-IMAGE? is true build layered image, providing a Docker
+image with many of the store paths being on their own layer to improve sharing
+between images.
+
+ROOT-SYSTEM is a directory with a provisioned root file system, which will be
+added to image as a layer."
(define (sanitize path-fragment)
(escape-special-chars
;; GNU tar strips the leading slash off of absolute paths before applying
@@ -203,6 +249,53 @@ (define* (build-docker-image image paths prefix
(if (eq? '() transformations)
'()
`("--transform" ,(transformations->expression transformations))))
+ (define layers-hashes
+ (match-lambda
+ (((head ...) (tail ...) id)
+ (create-empty-tar "image.tar")
+ (let* ((head-layers
+ (map
+ (lambda (file)
+ (invoke "tar" "cf" "layer.tar" file)
+ (let* ((file-hash (layer-diff-id "layer.tar"))
+ (file-name (string-append file-hash "/layer.tar")))
+ (mkdir file-hash)
+ (rename-file "layer.tar" file-name)
+ (invoke "tar" "-rf" "image.tar" file-name)
+ (delete-file file-name)
+ file-hash))
+ head))
+ (tail-layer
+ (begin
+ (create-empty-tar "layer.tar")
+ (for-each
This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 3 Jun 21:16 +0200
[PATCH v4] news: Add entry for the new 'docker-layered' distribution format.
(address . 62153@debbugs.gnu.org)
270e247ee913f6ede40883919d3f6971fe1b01aa.1685819787.git.go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

Toggle diff (81 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 314f0ab352..158a9284b0 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021, 2023 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,63 @@
(channel-news
(version 0)
+ (entry (commit "457c813653a44117e296deaa49e79fc701b90791")
+ (title
+ (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "????? @samp{docker-layered} ?????? ??? @command{guix pack} ???????"))
+ (body
+ (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
+@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
+Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
+untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
+im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
+Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
+angelegt wird:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
+indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
+übergeben.
+
+Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
+@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")
+ (en "Docker layered images can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+The @command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+
+See @command{info \"(guix) Invoking guix pack\"} and
+@command{info \"(guix) System Images\"} for more information.")
+ (ru "????????? ??????? ???????? ???????????? Docker ??????? ? ???????
+@command{guix pack --format=docker-layered}, ??????? ??????? Docker ????? ?
+?????? ? store ?????????????? ?? ????????? ?????, ??????? ????? ???????
+???????? ???????. ????? ????? ?????? ? GNU store ? ???????? gzipped tarball.
+
+?????? ???????? Docker layered ????? ? @code{hello} ???????:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} ?????? ????? ????????? layered Docker ????? ?????
+???????? ? ????? @option{--image-type} ????????? @code{docker-layered}.
+
+???????? @command{info \"(guix) Invoking guix pack\"} ?
+@command{info \"(guix) System Images\"} ??? ????????? ????? ?????????
+????????.")))
+
(entry (commit "ba5da5125a81307500982517e2f458d57b024668")
(title
(en "New @code{arguments} rule for @command{guix style}")

base-commit: 66c9b82fed3c59ee07187898592c688c82fed273
prerequisite-patch-id: 9c90b67b3c2bb18d7fd17d083b0ab0d1cd5333cd
--
2.38.0
O
O
Oleg Pykhalov wrote on 27 Aug 05:16 +0200
Merging guix pack changes for Docker containers packaging
(name . guix-devel)(address . guix-devel@gnu.org)
878r9xb2e6.fsf@gmail.com
Hi Guix,

I would like to merge 62153. After 64173 will be merge, merging 62153
is not possible without conflict resolving with Git.

64173 introduces ‘%docker-format-options’ variable. With this variable
it's possible in 62153 to replace ‘--image-type=docker-layered’ with
‘--docker-layers=N’ option, where:

if ‘N’ is zero, then use current non layered format
if ‘N’ is bigger than zero, then use layered format

Number of layers specification is nice to have, because Docker layers
are limited. So if user would like to modify a Docker image by adding
more layers on top, then hacks like squashing layers are not required.
Also, it will be possible to delete code which builds non layered Docker
image without deprecating command line options.

Is it possible to partially merge 64173, specifically
‘%docker-format-options’ variable and it requirements, so it can be used
in 62153 for ‘--docker-layers=N’ option?



Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmTqwCEUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pwLNg//XqUbjKOGD1st7X1i7XAZKAwkPN/j
yOzVa1agcRi2bIwLqoh+/WWLf4afG0tE7b52VQKUbgSMdSfJi2RJoLs7PmTjhhhJ
/QKl35ffdDr4C6MgutK/nir/Dh2wYuZ4KtKdmj6VmXkC5uSd+GA4mNriwZGUf5IX
Oo2YqvEv9e9wkilRRX7BN/iVboKPlPWHxNT82x/gB2UTdgXxcICqUHa9L/h+4obl
7xrsBiGtIJMWmC0A6bZXscDrWTxlKdqAXZTkGRAh2XJ1EWdZeh6ED3s1n4lZySsE
gapcNf8112Th0+TRhSV3FP7OUc5LHK8nvosNCnr3WtLMEOG/TGeTx5POc6YwUVwA
oJQQu9Z9k8GCQKD58Uo4Bq7f8uhbq+iVfChESn0+ezNeJagir63CAD9eqegaQI1f
e2EVxEC3sjEe264ue4fW3L7zx5SkHTHehv8mVfDKAJJTmNgCXx1t+jjP51T1vmxg
n3wzP9hU4Ugg10r+whWC2BuH6k2B+wS/bJ1mkg/qSqHAB9/3wvhVikSSFFBmpI+s
FmRkjTDguLW9rfZQW8Q9EJtAqOF74FRt4JXkGZffOClTdLrYNuf958wN6r4uuW00
MTBRKy753fJrU7PlwvPi7SQEe6ziaCY7w+tvYOKSV8DPPShMZBhuYnrzRM4IvuDB
5Xgx6marsPHb7AA=
=VfnU
-----END PGP SIGNATURE-----

?