[PATCH 0/2] Add Docker layered image for pack and system

  • Done
  • quality assurance status badge
Details
8 participants
  • Greg Hogan
  • Oleg Pykhalov
  • Ludovic Courtès
  • Christopher Baines
  • Mathieu Othacehe
  • pelzflorian (Florian Pelz)
  • Ricardo Wurmus
  • Simon Tournier
Owner
unassigned
Submitted by
Oleg Pykhalov
Severity
normal
O
O
Oleg Pykhalov wrote on 13 Mar 2023 01:30
(address . guix-patches@gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003012.14325-1-go.wigust@gmail.com
Hi Guix,

This patch series add to 'guix pack' and 'guix system image' formats with a
layered Docker image, which dicreases images size by sharing same layers on a
host.

The folling commands show an example for new image formats:

./pre-inst-env guix system image --image-type=docker-layered config.scm
docker load -i result

./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello
docker load -i result

The folloing tests passed:

make check-channel-news
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system docker-layered-system"

The gnu/packages/aux-files/python/stream-layered-image.py Python script is a
copy of github.com/NixOS/nixpkgs/pkgs/build-support/docker/stream_layered_image.py
with only a simple replacement "/nix" to "/gnu" string.

Oleg Pykhalov (2):
guix: docker: Build layered image.
news: Add entry for the new 'docker-layered' distribution format.

Makefile.am | 3 +-
doc/guix.texi | 16 +-
etc/news.scm | 38 ++
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 103 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
11 files changed, 813 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py


base-commit: 60a211ec705ac98483d76da7f2523f2b8966343a
--
2.38.0
O
O
Oleg Pykhalov wrote on 13 Mar 2023 01:33
[PATCH 1/2] guix: docker: Build layered image.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003310.17129-1-go.wigust@gmail.com
* gnu/packages/aux-files/python/stream-layered-image.py: New file.
* Makefile.am (AUX_FILES): Add this.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* tests/pack.scm: Add docker-layered-image + localstatedir test.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New
variables.
(system-docker-image)[layered-image?]: New argument.
(stream-layered-image.py): New variable.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* gnu/image.scm (validate-image-format)[docker-layered]: New image format.
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image-type Reference): Document docker-layered-image-type.
---
Makefile.am | 3 +-
doc/guix.texi | 16 +-
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 103 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
10 files changed, 775 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py

Toggle diff (518 lines)
diff --git a/Makefile.am b/Makefile.am
index 23b939b674..9aca84f8f8 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -11,7 +11,7 @@
# Copyright © 2017 Arun Isaac <arunisaac@systemreboot.net>
# Copyright © 2018 Nikita <nikita@n0.is>
# Copyright © 2018 Julien Lepiller <julien@lepiller.eu>
-# Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
+# Copyright © 2018, 2023 Oleg Pykhalov <go.wigust@gmail.com>
# Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
@@ -435,6 +435,7 @@ AUX_FILES = \
gnu/packages/aux-files/python/sanity-check.py \
gnu/packages/aux-files/python/sanity-check-next.py \
gnu/packages/aux-files/python/sitecustomize.py \
+ gnu/packages/aux-files/python/stream-layered-image.py \
gnu/packages/aux-files/renpy/renpy.in \
gnu/packages/aux-files/run-in-namespace.c
diff --git a/doc/guix.texi b/doc/guix.texi
index b545751e1b..bd0ee126ee 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@ Copyright @copyright{} 2017 Andy Wingo@*
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6840,9 +6840,15 @@ the following command:
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -43631,6 +43637,10 @@ Build an image based on the @code{iso9660-image} image but with the
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-with-syntax-properties (name (value properties))
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
new file mode 100644
index 0000000000..9ad2168c2d
--- /dev/null
+++ b/gnu/packages/aux-files/python/stream-layered-image.py
@@ -0,0 +1,391 @@
+"""
+This script generates a Docker image from a set of store paths. Uses
+Docker Image Specification v1.2 as reference [1].
+
+It expects a JSON file with the following properties and writes the
+image as an uncompressed tarball to stdout:
+
+* "architecture", "config", "os", "created", "repo_tag" correspond to
+ the fields with the same name on the image spec [2].
+* "created" can be "now".
+* "created" is also used as mtime for files added to the image.
+* "store_layers" is a list of layers in ascending order, where each
+ layer is the list of store paths to include in that layer.
+
+The main challenge for this script to create the final image in a
+streaming fashion, without dumping any intermediate data to disk
+for performance.
+
+A docker image has each layer contents archived as separate tarballs,
+and they later all get enveloped into a single big tarball in a
+content addressed fashion. However, because how "tar" format works,
+we have to know about the name (which includes the checksum in our
+case) and the size of the tarball before we can start adding it to the
+outer tarball. We achieve that by creating the layer tarballs twice;
+on the first iteration we calculate the file size and the checksum,
+and on the second one we actually stream the contents. 'add_layer_dir'
+function does all this.
+
+[1]: https://github.com/moby/moby/blob/master/image/spec/v1.2.md
+[2]: https://github.com/moby/moby/blob/4fb59c20a4fb54f944fe170d0ff1d00eb4a24d6f/image/spec/v1.2.md#image-json-field-descriptions
+""" # noqa: E501
+
+
+import io
+import os
+import re
+import sys
+import json
+import hashlib
+import pathlib
+import tarfile
+import itertools
+import threading
+from datetime import datetime, timezone
+from collections import namedtuple
+
+
+def archive_paths_to(obj, paths, mtime):
+ """
+ Writes the given store paths as a tar file to the given stream.
+
+ obj: Stream to write to. Should have a 'write' method.
+ paths: List of store paths.
+ """
+
+ # gettarinfo makes the paths relative, this makes them
+ # absolute again
+ def append_root(ti):
+ ti.name = "/" + ti.name
+ return ti
+
+ def apply_filters(ti):
+ ti.mtime = mtime
+ ti.uid = 0
+ ti.gid = 0
+ ti.uname = "root"
+ ti.gname = "root"
+ return ti
+
+ def nix_root(ti):
+ ti.mode = 0o0555 # r-xr-xr-x
+ return ti
+
+ def dir(path):
+ ti = tarfile.TarInfo(path)
+ ti.type = tarfile.DIRTYPE
+ return ti
+
+ with tarfile.open(fileobj=obj, mode="w|") as tar:
+ # To be consistent with the docker utilities, we need to have
+ # these directories first when building layer tarballs.
+ tar.addfile(apply_filters(nix_root(dir("/gnu"))))
+ tar.addfile(apply_filters(nix_root(dir("/gnu/store"))))
+
+ for path in paths:
+ path = pathlib.Path(path)
+ if path.is_symlink():
+ files = [path]
+ else:
+ files = itertools.chain([path], path.rglob("*"))
+
+ for filename in sorted(files):
+ ti = append_root(tar.gettarinfo(filename))
+
+ # copy hardlinks as regular files
+ if ti.islnk():
+ ti.type = tarfile.REGTYPE
+ ti.linkname = ""
+ ti.size = filename.stat().st_size
+
+ ti = apply_filters(ti)
+ if ti.isfile():
+ with open(filename, "rb") as f:
+ tar.addfile(ti, f)
+ else:
+ tar.addfile(ti)
+
+
+class ExtractChecksum:
+ """
+ A writable stream which only calculates the final file size and
+ sha256sum, while discarding the actual contents.
+ """
+
+ def __init__(self):
+ self._digest = hashlib.sha256()
+ self._size = 0
+
+ def write(self, data):
+ self._digest.update(data)
+ self._size += len(data)
+
+ def extract(self):
+ """
+ Returns: Hex-encoded sha256sum and size as a tuple.
+ """
+ return (self._digest.hexdigest(), self._size)
+
+
+FromImage = namedtuple("FromImage", ["tar", "manifest_json", "image_json"])
+# Some metadata for a layer
+LayerInfo = namedtuple("LayerInfo", ["size", "checksum", "path", "paths"])
+
+
+def load_from_image(from_image_str):
+ """
+ Loads the given base image, if any.
+
+ from_image_str: Path to the base image archive.
+
+ Returns: A 'FromImage' object with references to the loaded base image,
+ or 'None' if no base image was provided.
+ """
+ if from_image_str is None:
+ return None
+
+ base_tar = tarfile.open(from_image_str)
+
+ manifest_json_tarinfo = base_tar.getmember("manifest.json")
+ with base_tar.extractfile(manifest_json_tarinfo) as f:
+ manifest_json = json.load(f)
+
+ image_json_tarinfo = base_tar.getmember(manifest_json[0]["Config"])
+ with base_tar.extractfile(image_json_tarinfo) as f:
+ image_json = json.load(f)
+
+ return FromImage(base_tar, manifest_json, image_json)
+
+
+def add_base_layers(tar, from_image):
+ """
+ Adds the layers from the given base image to the final image.
+
+ tar: 'tarfile.TarFile' object for new layers to be added to.
+ from_image: 'FromImage' object with references to the loaded base image.
+ """
+ if from_image is None:
+ print("No 'fromImage' provided", file=sys.stderr)
+ return []
+
+ layers = from_image.manifest_json[0]["Layers"]
+ checksums = from_image.image_json["rootfs"]["diff_ids"]
+ layers_checksums = zip(layers, checksums)
+
+ for num, (layer, checksum) in enumerate(layers_checksums, start=1):
+ layer_tarinfo = from_image.tar.getmember(layer)
+ checksum = re.sub(r"^sha256:", "", checksum)
+
+ tar.addfile(layer_tarinfo, from_image.tar.extractfile(layer_tarinfo))
+ path = layer_tarinfo.path
+ size = layer_tarinfo.size
+
+ print("Adding base layer", num, "from", path, file=sys.stderr)
+ yield LayerInfo(size=size, checksum=checksum, path=path, paths=[path])
+
+ from_image.tar.close()
+
+
+def overlay_base_config(from_image, final_config):
+ """
+ Overlays the final image 'config' JSON on top of selected defaults from the
+ base image 'config' JSON.
+
+ from_image: 'FromImage' object with references to the loaded base image.
+ final_config: 'dict' object of the final image 'config' JSON.
+ """
+ if from_image is None:
+ return final_config
+
+ base_config = from_image.image_json["config"]
+
+ # Preserve environment from base image
+ final_env = base_config.get("Env", []) + final_config.get("Env", [])
+ if final_env:
+ # Resolve duplicates (last one wins) and format back as list
+ resolved_env = {entry.split("=", 1)[0]: entry for entry in final_env}
+ final_config["Env"] = list(resolved_env.values())
+ return final_config
+
+
+def add_layer_dir(tar, paths, store_dir, mtime):
+ """
+ Appends given store paths to a TarFile object as a new layer.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ paths: List of store paths.
+ store_dir: the root directory of the nix store
+ mtime: 'mtime' of the added files and the layer tarball.
+ Should be an integer representing a POSIX time.
+
+ Returns: A 'LayerInfo' object containing some metadata of
+ the layer added.
+ """
+
+ invalid_paths = [i for i in paths if not i.startswith(store_dir)]
+ assert len(invalid_paths) == 0, \
+ f"Expecting absolute paths from {store_dir}, but got: {invalid_paths}"
+
+ # First, calculate the tarball checksum and the size.
+ extract_checksum = ExtractChecksum()
+ archive_paths_to(
+ extract_checksum,
+ paths,
+ mtime=mtime,
+ )
+ (checksum, size) = extract_checksum.extract()
+
+ path = f"{checksum}/layer.tar"
+ layer_tarinfo = tarfile.TarInfo(path)
+ layer_tarinfo.size = size
+ layer_tarinfo.mtime = mtime
+
+ # Then actually stream the contents to the outer tarball.
+ read_fd, write_fd = os.pipe()
+ with open(read_fd, "rb") as read, open(write_fd, "wb") as write:
+ def producer():
+ archive_paths_to(
+ write,
+ paths,
+ mtime=mtime,
+ )
+ write.close()
+
+ # Closing the write end of the fifo also closes the read end,
+ # so we don't need to wait until this thread is finished.
+ #
+ # Any exception from the thread will get printed by the default
+ # exception handler, and the 'addfile' call will fail since it
+ # won't be able to read required amount of bytes.
+ threading.Thread(target=producer).start()
+ tar.addfile(layer_tarinfo, read)
+
+ return LayerInfo(size=size, checksum=checksum, path=path, paths=paths)
+
+
+def add_customisation_layer(target_tar, customisation_layer, mtime):
+ """
+ Adds the customisation layer as a new layer. This is layer is structured
+ differently; given store path has the 'layer.tar' and corresponding
+ sha256sum ready.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ customisation_layer: Path containing the layer archive.
+ mtime: 'mtime' of the added layer tarball.
+ """
+
+ checksum_path = os.path.join(customisation_layer, "checksum")
+ with open(checksum_path) as f:
+ checksum = f.read().strip()
+ assert len(checksum) == 64, f"Invalid sha256 at ${checksum_path}."
+
+ layer_path = os.path.join(customisation_layer, "layer.tar")
+
+ path = f"{checksum}/layer.tar"
+ tarinfo = target_tar.gettarinfo(layer_path)
+ tarinfo.name = path
+ tarinfo.mtime = mtime
+
+ with open(layer_path, "rb") as f:
+ target_tar.addfile(tarinfo, f)
+
+ return LayerInfo(
+ size=None,
+ checksum=checksum,
+ path=path,
+ paths=[customisation_layer]
+ )
+
+
+def add_bytes(tar, path, content, mtime):
+ """
+ Adds a file to the tarball with given path and contents.
+
+ tar: 'tarfile.TarFile' object.
+ path: Path of the file as a string.
+ content: Contents of the file.
+ mtime: 'mtime' of the file. Should be an integer representing a POSIX time.
+ """
+ assert type(content) is bytes
+
+ ti = tarfile.TarInfo(path)
+ ti.size = len(content)
+ ti.mtime = mtime
+ tar.addfile(ti, io.BytesIO(content))
+
+
+def main():
+ with open(sys.argv[1], "r") as f:
+ conf = json.load(f)
+
+ created = (
+ datetime.now(tz=timezone.utc)
+ if conf["created"] == "now"
+ else datetime.fromisoformat(conf["created"])
+ )
+ mtime = int(created.timestamp())
+ store_dir = conf["store_dir"]
+
+ from_image = load_from_image(conf["from_image"])
+
+ with tarfile.open(mode="w|", fileobj=sys.stdout.buffer) as tar:
+ layers = []
+ layers.extend(add_base_layers(tar, from_image))
+
+ start = len(layers) + 1
+ for num, store_layer in enumerate(conf["store_layers"], start=start):
+ print("Creating layer", num, "from paths:", store_layer,
+ file=sys.stderr)
+ info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
+ layers.append(info)
+
+ print("Creating layer", len(layers) + 1, "with customisation...",
+ file=sys.stderr)
+ layers.append(
+ add_customisation_layer(
+ tar,
+ conf["customisation_layer"],
+ mtime=mtime
+ )
+ )
+
+ print("Adding manifests...", file=sys.stderr)
+
+ image_json = {
+ "created": datetime.isoformat(created),
+ "architecture": conf["architecture"],
+ "os": "linux",
+ "config": overlay_base_config(from_image, conf["config"]),
+ "rootfs": {
+ "diff_ids": [f"sha256:{layer.checksum}" for layer in layers],
+ "type": "layers",
+ },
+ "history": [
+ {
+ "created": datetime.isoformat(created),
+ "comment": f"store paths: {layer.paths}"
+ }
+ for layer in layers
+ ],
+ }
+
+ image_json = json.dumps(image_json, indent=4).encode("utf-8")
+ image_json_checksum = hashlib.sha256(image_json).hexdigest()
+ image_json_path = f"{image_json_checksum}.json"
+ add_bytes(tar, image_json_path, image_json, mtime=mtime)
+
+ manifest_json = [
+ {
+ "Config": image_json_path,
+ "RepoTags": [conf["repo_tag"]],
+ "Layers": [layer.path for layer in layers],
+ }
+ ]
+ manifest_json = json.dumps(manifest_json, indent=4).encode("utf-8")
+ add_bytes(tar, "manifest.json", manifest_json, mtime=mtime)
+
+ print("Done.", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..0bfd011ad4 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -45,6 +46,7 @@ (define-module (gnu system image)
#:use-module (gnu system uuid)
#:use-module (gnu system vm)
#:use-module (guix packages)
+ #:use-module ((gnu packages) #:select (search-auxiliary-file))
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages bootloaders)
@@ -58,6 +60,7 @@ (define-module (gnu system image)
#:use-module (gnu packages hurd)
#:use-module (gnu packages linux)
#:use-module (gnu packages mtools)
+ #:use-module (gnu packages python)
#:use-module (gnu packages virtualization)
#:use-module ((srfi srfi-1) #:prefix srfi-1:)
#:use-module (srfi srfi-11)
@@ -78,6 +81,7 @@ (define-module (gnu system image)
efi-disk-image
iso9660-image
docker-image
+ docker-layered-image
tarball-image
wsl2-image
raw-with-offset-disk-image
@@ -89,6 +93,7 @@ (define-module (gnu system image)

This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 13 Mar 2023 01:33
[PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003310.17129-2-go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)

Toggle diff (58 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 924c2b35b4..98fb7f536c 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,43 @@
(channel-news
(version 0)
+ (entry (commit "45777c5b753ce330ad007d4e71189cf3fc627ccc")
+ (title
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "????? @samp{docker-layered} ?????? ??? @command{guix pack} ???????"))
+ (body
+ (en "Docker layered image can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+See @command{info \"(guix) Invoking guix pack\"} for more information.
+
+@command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+")
+ (ru "????????? ??????? ???????? ???????????? Docker ??????? ? ???????
+@command{guix pack --format=docker-layered}, ??????? ??????? Docker ????? ?
+?????? ? store ?????????????? ?? ????????? ?????, ??????? ????? ???????
+???????? ???????. ????? ????? ?????? ? GNU store ? ???????? gzipped tarball.
+
+?????? ???????? Docker layered image ? @code{hello} ???????:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+???????? @command{info \"(guix) Invoking guix pack\"} ??? ????????? ?????
+????????? ????????.
+
+@command{guix system image} ?????? ????? ????????? layered Docker image ?????
+???????? ? ????? @option{--image-type} ????????? @code{docker-layered}.")))
+
(entry (commit "598f4c509bbfec2b983a8ee246cce0a0fe45ec7f")
(title
(de "Neues Format @samp{rpm} für den Befehl @command{guix pack}")
--
2.38.0
O
O
Oleg Pykhalov wrote on 13 Mar 2023 01:43
Cover lever typo in guix pack format example
(address . 62153@debbugs.gnu.org)
87wn3l32hs.fsf@gmail.com
The cover lever guix pack example should be:

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello

instead of

./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello

Apologies,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQOcZ8UHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzRkg/9H8+149MGcejnPteSR7C1XKWckEWi
xoADRq5qcwFf9JTGGNEDqEoq/z+BFCa5ainGpLKO6jzVGe0gCfzsmmxRA7V5fZY3
udwfPwVmhb3oV4CV8t8Gi3tqTnJ3meUG7CdzNttocYRlwLys2TUoy5t8fJ9xwg/v
6ZSVeKfcXJdM9BP5a+KkLMzfxGOGlfu8HFVIrVlPAMgRmliZLRKFdk3mmZ7jJEQv
OepVWlQ5j6BRHfbETEYz6nB7Hw2BYzfL5dwzkrsMBwO2KgtG9sYSgdqIbRVJXcP9
L3R5Z1Z51p4CcByYG35iq0ahwGy9daUL1UiCx8gIXJPEy7RGMFqjOAGekg+lN2PI
qOHLMpthqJBKQxKnYBxp7ZdgnvwX4iYs7YrUDDX6I507L1zYABWNM2exLl1N3PUF
vlHKUmI2la4ozWDaHNZsTa7qVtFtRwXr5Vs8hfkbqfEhAG3YHQKd0zMBr33Ht6bn
WYdT6zKq2Dnp3BjK1dkOKYhIJelmWlwptUYv88h7KiozgEOI84l8RpSOU7IXnEBY
CUhhGhPp2xu4w7NCkOA3TAYHGYFT2ebFnlZ+l3fTq6i8Y3p9LVD65Jvz9Pqgt0MQ
9v7+ArClkWYmKxc0XNrqpv2J8mAr5PlSvfXtHM0wTsZk+cpuaC/xIkCf4upHyLyh
1a4qZJxNTV9FjLA=
=z9Ac
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 13 Mar 2023 16:01
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
87r0tsk85r.fsf@gmail.com
Hi,

Oh cool! Awesome! Thanks for pushing forward.

On lun., 13 mars 2023 at 03:33, Oleg Pykhalov <go.wigust@gmail.com> wrote:

Toggle quote (10 lines)
> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
> new file mode 100644
> index 0000000000..9ad2168c2d
> --- /dev/null
> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
> @@ -0,0 +1,391 @@
> +"""
> +This script generates a Docker image from a set of store paths. Uses
> +Docker Image Specification v1.2 as reference [1].

Instead of Python, would it possible to implement in Guile? I mean,
does Python have something that is missing in Guile?

The facility for manipulating Tar? Something else?


Because then, if I understand correctly…

Toggle quote (5 lines)
> diff --git a/guix/docker.scm b/guix/docker.scm
> index 5e6460f43f..f1adad26dc 100644
> --- a/guix/docker.scm
> +++ b/guix/docker.scm

[...]

Toggle quote (5 lines)
> + (if stream-layered-image
> + (let ((input (open-pipe* OPEN_READ "python3"
> + stream-layered-image
> + "config.json")))

…it requires to drag Python for building/packing layered Docker.


Well, I have not really look yet to the Python script which does most of
the job. Do you use a similar strategy as [1]?

And I remember something in that direction by Chris but I am unable to
find back the patch. )-:


Cheers,
simon
P
P
pelzflorian (Florian Pelz) wrote on 13 Mar 2023 22:09
Re: [bug#62153] [PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)(address . 62153@debbugs.gnu.org)
87cz5ccqa5.fsf@pelzflorian.de
Thank you Oleg for this feature.

Could you change the following three things in the news:

Change the beginning of the English translation from
"Docker layered image can now be produced" to
"Docker layered images can now be produced".

And at the end, also in Russian, switch around these two paragraphs
and add a “the” and reference the System Images chapter:
Toggle quote (7 lines)
> @command{guix system image} can now produce layered Docker image by passing
> @code{docker-layered} to the @option{--image-type} option.
>
> See @command{info \"(guix) Invoking guix pack\"} and
> @command{info \"(guix) System Images\"} for more information.


Lastly, could you then also add a German translation:

(title
(de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")

(body
(de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
angelegt wird:

@example
guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
@end example

@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
übergeben.

Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")

Regards,
Florian
O
O
Oleg Pykhalov wrote on 13 Mar 2023 22:10
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 62153@debbugs.gnu.org)
87bkkw2w7z.fsf@gmail.com
Hi Simon,

Thank you for the review.

Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (17 lines)
> On lun., 13 mars 2023 at 03:33, Oleg Pykhalov <go.wigust@gmail.com> wrote:
>
>> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
>> new file mode 100644
>> index 0000000000..9ad2168c2d
>> --- /dev/null
>> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
>> @@ -0,0 +1,391 @@
>> +"""
>> +This script generates a Docker image from a set of store paths. Uses
>> +Docker Image Specification v1.2 as reference [1].
>
> Instead of Python, would it possible to implement in Guile? I mean,
> does Python have something that is missing in Guile?
>
> The facility for manipulating Tar? Something else?

I think nothing else. As I understand Python implemented Tar inside the
language itself in 2500 lines of code by manipulating binary data.

/gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py

Technically it's probably possible to use tar utility with --append flag
instead of opening a new file and streaming to it as the Python script
does. To be honest I would like not to write it in this way if the
Python script does not block current patch for merge.

Also I don't see myself writing Tar implementation in Guile, yet. ;-)

The Nix project uses this script heavily to build layered images, so it
should be robust in terms of up to date to current Tar and Python
implementations.

Toggle quote (16 lines)
> Because then, if I understand correctly…
>
>> diff --git a/guix/docker.scm b/guix/docker.scm
>> index 5e6460f43f..f1adad26dc 100644
>> --- a/guix/docker.scm
>> +++ b/guix/docker.scm
>
> [...]
>
>> + (if stream-layered-image
>> + (let ((input (open-pipe* OPEN_READ "python3"
>> + stream-layered-image
>> + "config.json")))
>
> …it requires to drag Python for building/packing layered Docker.

Correct.

Toggle quote (8 lines)
> Well, I have not really look yet to the Python script which does most of
> the job. Do you use a similar strategy as [1]?
>
> And I remember something in that direction by Chris but I am unable to
> find back the patch. )-:
>
> 1: https://grahamc.com/blog/nix-and-layered-docker-images/

Not similar. My patch implements a very simple sorting by size, no
complex sorting by reference popularity as in [1], which is probably
implemented in the following file

github.com/NixOS/nixpkgs/pkgs/build-support/references-by-popularity/closure-graph.py


Toggle quote (27 lines)
> How Docker really represents an Image
>
> Docker’s layers are content addressable and aren’t required to
> explicitly reference a parent layer. This means a layer for
> readline-7.0p5 doesn’t have to mention that it has any relationship to
> ncurses-6.1 or glibc-2.27 at all.
>
> Instead each image has a manifest which defines the order:
>
> {
> "Layers": [
> "bash-interactive-4.4-p23",
> "bash-4.4p23",
> "readline-7.0p5",
> ...
> ]
> }
>
> If you have only built Docker images using a Dockerfile, then you
> would expect the way we flatten our graph to be critically
> important. If we sometimes picked readline-7.0p5 to come first and
> other times picked bash-4.4p23 then we may never make cache hits.
>
> However since the Image defines the order, we don’t have to solve this
> impossible problem: we can order the layers in any way we want and the
> layer cache will always hit.

In case of sorting by size, bigest layers will be on top of a container
image, which will produce a cache hit for bigest directories in the GNU
store during images transfer with same layers.

I would like to say this sorting could binifit more than sorting by
popularity during transfer but let's assume I didn't write it. ;-)

The following example shows common layers between images, which will be
not tranfered if you load image inside Docker as well as pull and push:

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello

and

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello emacs

share 6 layers in total

Toggle snippet (11 lines)
$ f() { docker image inspect "$1" | jq --raw-output '.[0].RootFS.Layers[] | .' | sort ; }
$ comm -1 -2 --total <(f sha256:fb43b32380a5e6a867410721f4ce2917db14d4ae943c433983afbaf84416c421) <(f sha256:0ce4a11973d1071aeec5441db228d6148dfd09fea3ae77b731c750ebfcc2fe1d)
sha256:3b3daa2a00f1acd12eeb16698bf1caeb6ba6c436e3dbca6259c3a9c622664e00
sha256:5c2be7469293854257221cb6aa8aa4af1e10e2c550935390dbcfeede3d3fbacd
sha256:60317981d94928659389f299e4b86703e5ded420a53537d67627952187fbd3f9
sha256:6d7c8ce5441d4c4c74e0ecff6c203a7b265b37137cca3b0a0ccf10526cfaa6e2
sha256:c2ded2ffe3f46fa7a64a62e0fc6b9d28cb7d4f8d9c64d5a52d137a508cba11fc
sha256:fbcad85d7d3c25bd2aa6d95bb3bf3d02c499ee3b3e443ddd3e5b679c2b33c139
5 94 6 total

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPkWAUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzFZBAApU6Atd2czoZgSRir9SG0/V/l7yva
zKst6JvXL77QxVU+e0QqVcY/o8rJg9hIGH+cQaOmlxmbYXJExEI1Go+tngk61OgQ
dsMrdBBPkaXvqzd7PKQRUtCn221CGfoMMfZGEzCUyQFTYgo6+K7s76Ep6++lODEW
nV/nPa/gqKhg6IZ2G+NeuNKeHsFN8YH/U3Si8myvyLSt0B/ZZEb/8eMumhl3lt61
4GXPPhlFXvp/8VebhMbvzN7TUGWl4z8uMpVExYdDjG3BxkcJBqlEfGfJPQlWc18E
+8G93/JCFYhtE/ae0d/qCpVs8k7CLRrJEkllQFARau+e0GxNggdA08mACmuxZqTF
u1t52FVvg+cgIO5XKN0x0HyR0t+Rv6gkZPA1b4EXF6t8XRCkdh5VXZr/v3Wvwea2
2wbTeX2vDeca3coyCxDNRjka+FgcLrcxxVBZNc33/76sgglXjovQI4g4LDsklkV9
pSa8StEt4/lCSPXRUQYy0g/nUO4Lk2QaIVx7FSefqCaS+EwCz6NkMGi8esKrtHuw
l1gFvYhMFtHvQjzbO50edR1QyLgbMTFbyJdFhHJOEI0rAeh4AIde+R/FfXbPFZR4
RCPwCij2+Ng0O8lrd9uQcdo92oms6FGHLN+ndgYm38hzMQdI+ecMpAQtlR09efsW
WBhi5/HWNf3372o=
=r2xS
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 14 Mar 2023 01:24
[PATCH 0/2] Add Docker layered image for pack and system (v2)
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-1-go.wigust@gmail.com
Hi, Florian.

Thank you for the review.

This patch series applies your suggestions. Also it's rebased on
origin/master and added a missing documentation for ‘docker-layered’ format in
‘guix system image’ command in doc/guix.texi file (following diff).

<#part type="text/x-patch" buffer=m1.txt disposition=inline description="Add missing docker-layered format documentation for guix system image command">
<#/part>

The folloing tests passed:

make check-channel-news
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system docker-layered-system"

Oleg Pykhalov (2):
guix: docker: Build layered image.
news: Add entry for the new 'docker-layered' distribution format.

Makefile.am | 3 +-
doc/guix.texi | 18 +-
etc/news.scm | 58 +++
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 105 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
11 files changed, 837 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py


base-commit: 5312d798ac36a72d8a977325a7c6ff7647be670a
--
2.38.0
O
O
Oleg Pykhalov wrote on 14 Mar 2023 01:24
[PATCH 1/2] guix: docker: Build layered image.
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-2-go.wigust@gmail.com
* gnu/packages/aux-files/python/stream-layered-image.py: New file.
* Makefile.am (AUX_FILES): Add this.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* tests/pack.scm: Add docker-layered-image + localstatedir test.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New
variables.
(system-docker-image)[layered-image?]: New argument.
(stream-layered-image.py): New variable.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* gnu/image.scm (validate-image-format)[docker-layered]: New image format.
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image Reference): Same.
(image-type Reference): Document docker-layered-image-type.
---
Makefile.am | 3 +-
doc/guix.texi | 18 +-
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 105 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
10 files changed, 779 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py

Toggle diff (518 lines)
diff --git a/Makefile.am b/Makefile.am
index 23b939b674..9aca84f8f8 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -11,7 +11,7 @@
# Copyright © 2017 Arun Isaac <arunisaac@systemreboot.net>
# Copyright © 2018 Nikita <nikita@n0.is>
# Copyright © 2018 Julien Lepiller <julien@lepiller.eu>
-# Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
+# Copyright © 2018, 2023 Oleg Pykhalov <go.wigust@gmail.com>
# Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
@@ -435,6 +435,7 @@ AUX_FILES = \
gnu/packages/aux-files/python/sanity-check.py \
gnu/packages/aux-files/python/sanity-check-next.py \
gnu/packages/aux-files/python/sitecustomize.py \
+ gnu/packages/aux-files/python/stream-layered-image.py \
gnu/packages/aux-files/renpy/renpy.in \
gnu/packages/aux-files/run-in-namespace.c
diff --git a/doc/guix.texi b/doc/guix.texi
index 39932d5aad..fa4b7586c9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@ Copyright @copyright{} 2017 Andy Wingo@*
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6837,9 +6837,15 @@ the following command:
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -43274,6 +43280,8 @@ one or multiple partitions.
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
@@ -43605,6 +43613,10 @@ Build an image based on the @code{iso9660-image} image but with the
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-with-syntax-properties (name (value properties))
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
new file mode 100644
index 0000000000..9ad2168c2d
--- /dev/null
+++ b/gnu/packages/aux-files/python/stream-layered-image.py
@@ -0,0 +1,391 @@
+"""
+This script generates a Docker image from a set of store paths. Uses
+Docker Image Specification v1.2 as reference [1].
+
+It expects a JSON file with the following properties and writes the
+image as an uncompressed tarball to stdout:
+
+* "architecture", "config", "os", "created", "repo_tag" correspond to
+ the fields with the same name on the image spec [2].
+* "created" can be "now".
+* "created" is also used as mtime for files added to the image.
+* "store_layers" is a list of layers in ascending order, where each
+ layer is the list of store paths to include in that layer.
+
+The main challenge for this script to create the final image in a
+streaming fashion, without dumping any intermediate data to disk
+for performance.
+
+A docker image has each layer contents archived as separate tarballs,
+and they later all get enveloped into a single big tarball in a
+content addressed fashion. However, because how "tar" format works,
+we have to know about the name (which includes the checksum in our
+case) and the size of the tarball before we can start adding it to the
+outer tarball. We achieve that by creating the layer tarballs twice;
+on the first iteration we calculate the file size and the checksum,
+and on the second one we actually stream the contents. 'add_layer_dir'
+function does all this.
+
+[1]: https://github.com/moby/moby/blob/master/image/spec/v1.2.md
+[2]: https://github.com/moby/moby/blob/4fb59c20a4fb54f944fe170d0ff1d00eb4a24d6f/image/spec/v1.2.md#image-json-field-descriptions
+""" # noqa: E501
+
+
+import io
+import os
+import re
+import sys
+import json
+import hashlib
+import pathlib
+import tarfile
+import itertools
+import threading
+from datetime import datetime, timezone
+from collections import namedtuple
+
+
+def archive_paths_to(obj, paths, mtime):
+ """
+ Writes the given store paths as a tar file to the given stream.
+
+ obj: Stream to write to. Should have a 'write' method.
+ paths: List of store paths.
+ """
+
+ # gettarinfo makes the paths relative, this makes them
+ # absolute again
+ def append_root(ti):
+ ti.name = "/" + ti.name
+ return ti
+
+ def apply_filters(ti):
+ ti.mtime = mtime
+ ti.uid = 0
+ ti.gid = 0
+ ti.uname = "root"
+ ti.gname = "root"
+ return ti
+
+ def nix_root(ti):
+ ti.mode = 0o0555 # r-xr-xr-x
+ return ti
+
+ def dir(path):
+ ti = tarfile.TarInfo(path)
+ ti.type = tarfile.DIRTYPE
+ return ti
+
+ with tarfile.open(fileobj=obj, mode="w|") as tar:
+ # To be consistent with the docker utilities, we need to have
+ # these directories first when building layer tarballs.
+ tar.addfile(apply_filters(nix_root(dir("/gnu"))))
+ tar.addfile(apply_filters(nix_root(dir("/gnu/store"))))
+
+ for path in paths:
+ path = pathlib.Path(path)
+ if path.is_symlink():
+ files = [path]
+ else:
+ files = itertools.chain([path], path.rglob("*"))
+
+ for filename in sorted(files):
+ ti = append_root(tar.gettarinfo(filename))
+
+ # copy hardlinks as regular files
+ if ti.islnk():
+ ti.type = tarfile.REGTYPE
+ ti.linkname = ""
+ ti.size = filename.stat().st_size
+
+ ti = apply_filters(ti)
+ if ti.isfile():
+ with open(filename, "rb") as f:
+ tar.addfile(ti, f)
+ else:
+ tar.addfile(ti)
+
+
+class ExtractChecksum:
+ """
+ A writable stream which only calculates the final file size and
+ sha256sum, while discarding the actual contents.
+ """
+
+ def __init__(self):
+ self._digest = hashlib.sha256()
+ self._size = 0
+
+ def write(self, data):
+ self._digest.update(data)
+ self._size += len(data)
+
+ def extract(self):
+ """
+ Returns: Hex-encoded sha256sum and size as a tuple.
+ """
+ return (self._digest.hexdigest(), self._size)
+
+
+FromImage = namedtuple("FromImage", ["tar", "manifest_json", "image_json"])
+# Some metadata for a layer
+LayerInfo = namedtuple("LayerInfo", ["size", "checksum", "path", "paths"])
+
+
+def load_from_image(from_image_str):
+ """
+ Loads the given base image, if any.
+
+ from_image_str: Path to the base image archive.
+
+ Returns: A 'FromImage' object with references to the loaded base image,
+ or 'None' if no base image was provided.
+ """
+ if from_image_str is None:
+ return None
+
+ base_tar = tarfile.open(from_image_str)
+
+ manifest_json_tarinfo = base_tar.getmember("manifest.json")
+ with base_tar.extractfile(manifest_json_tarinfo) as f:
+ manifest_json = json.load(f)
+
+ image_json_tarinfo = base_tar.getmember(manifest_json[0]["Config"])
+ with base_tar.extractfile(image_json_tarinfo) as f:
+ image_json = json.load(f)
+
+ return FromImage(base_tar, manifest_json, image_json)
+
+
+def add_base_layers(tar, from_image):
+ """
+ Adds the layers from the given base image to the final image.
+
+ tar: 'tarfile.TarFile' object for new layers to be added to.
+ from_image: 'FromImage' object with references to the loaded base image.
+ """
+ if from_image is None:
+ print("No 'fromImage' provided", file=sys.stderr)
+ return []
+
+ layers = from_image.manifest_json[0]["Layers"]
+ checksums = from_image.image_json["rootfs"]["diff_ids"]
+ layers_checksums = zip(layers, checksums)
+
+ for num, (layer, checksum) in enumerate(layers_checksums, start=1):
+ layer_tarinfo = from_image.tar.getmember(layer)
+ checksum = re.sub(r"^sha256:", "", checksum)
+
+ tar.addfile(layer_tarinfo, from_image.tar.extractfile(layer_tarinfo))
+ path = layer_tarinfo.path
+ size = layer_tarinfo.size
+
+ print("Adding base layer", num, "from", path, file=sys.stderr)
+ yield LayerInfo(size=size, checksum=checksum, path=path, paths=[path])
+
+ from_image.tar.close()
+
+
+def overlay_base_config(from_image, final_config):
+ """
+ Overlays the final image 'config' JSON on top of selected defaults from the
+ base image 'config' JSON.
+
+ from_image: 'FromImage' object with references to the loaded base image.
+ final_config: 'dict' object of the final image 'config' JSON.
+ """
+ if from_image is None:
+ return final_config
+
+ base_config = from_image.image_json["config"]
+
+ # Preserve environment from base image
+ final_env = base_config.get("Env", []) + final_config.get("Env", [])
+ if final_env:
+ # Resolve duplicates (last one wins) and format back as list
+ resolved_env = {entry.split("=", 1)[0]: entry for entry in final_env}
+ final_config["Env"] = list(resolved_env.values())
+ return final_config
+
+
+def add_layer_dir(tar, paths, store_dir, mtime):
+ """
+ Appends given store paths to a TarFile object as a new layer.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ paths: List of store paths.
+ store_dir: the root directory of the nix store
+ mtime: 'mtime' of the added files and the layer tarball.
+ Should be an integer representing a POSIX time.
+
+ Returns: A 'LayerInfo' object containing some metadata of
+ the layer added.
+ """
+
+ invalid_paths = [i for i in paths if not i.startswith(store_dir)]
+ assert len(invalid_paths) == 0, \
+ f"Expecting absolute paths from {store_dir}, but got: {invalid_paths}"
+
+ # First, calculate the tarball checksum and the size.
+ extract_checksum = ExtractChecksum()
+ archive_paths_to(
+ extract_checksum,
+ paths,
+ mtime=mtime,
+ )
+ (checksum, size) = extract_checksum.extract()
+
+ path = f"{checksum}/layer.tar"
+ layer_tarinfo = tarfile.TarInfo(path)
+ layer_tarinfo.size = size
+ layer_tarinfo.mtime = mtime
+
+ # Then actually stream the contents to the outer tarball.
+ read_fd, write_fd = os.pipe()
+ with open(read_fd, "rb") as read, open(write_fd, "wb") as write:
+ def producer():
+ archive_paths_to(
+ write,
+ paths,
+ mtime=mtime,
+ )
+ write.close()
+
+ # Closing the write end of the fifo also closes the read end,
+ # so we don't need to wait until this thread is finished.
+ #
+ # Any exception from the thread will get printed by the default
+ # exception handler, and the 'addfile' call will fail since it
+ # won't be able to read required amount of bytes.
+ threading.Thread(target=producer).start()
+ tar.addfile(layer_tarinfo, read)
+
+ return LayerInfo(size=size, checksum=checksum, path=path, paths=paths)
+
+
+def add_customisation_layer(target_tar, customisation_layer, mtime):
+ """
+ Adds the customisation layer as a new layer. This is layer is structured
+ differently; given store path has the 'layer.tar' and corresponding
+ sha256sum ready.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ customisation_layer: Path containing the layer archive.
+ mtime: 'mtime' of the added layer tarball.
+ """
+
+ checksum_path = os.path.join(customisation_layer, "checksum")
+ with open(checksum_path) as f:
+ checksum = f.read().strip()
+ assert len(checksum) == 64, f"Invalid sha256 at ${checksum_path}."
+
+ layer_path = os.path.join(customisation_layer, "layer.tar")
+
+ path = f"{checksum}/layer.tar"
+ tarinfo = target_tar.gettarinfo(layer_path)
+ tarinfo.name = path
+ tarinfo.mtime = mtime
+
+ with open(layer_path, "rb") as f:
+ target_tar.addfile(tarinfo, f)
+
+ return LayerInfo(
+ size=None,
+ checksum=checksum,
+ path=path,
+ paths=[customisation_layer]
+ )
+
+
+def add_bytes(tar, path, content, mtime):
+ """
+ Adds a file to the tarball with given path and contents.
+
+ tar: 'tarfile.TarFile' object.
+ path: Path of the file as a string.
+ content: Contents of the file.
+ mtime: 'mtime' of the file. Should be an integer representing a POSIX time.
+ """
+ assert type(content) is bytes
+
+ ti = tarfile.TarInfo(path)
+ ti.size = len(content)
+ ti.mtime = mtime
+ tar.addfile(ti, io.BytesIO(content))
+
+
+def main():
+ with open(sys.argv[1], "r") as f:
+ conf = json.load(f)
+
+ created = (
+ datetime.now(tz=timezone.utc)
+ if conf["created"] == "now"
+ else datetime.fromisoformat(conf["created"])
+ )
+ mtime = int(created.timestamp())
+ store_dir = conf["store_dir"]
+
+ from_image = load_from_image(conf["from_image"])
+
+ with tarfile.open(mode="w|", fileobj=sys.stdout.buffer) as tar:
+ layers = []
+ layers.extend(add_base_layers(tar, from_image))
+
+ start = len(layers) + 1
+ for num, store_layer in enumerate(conf["store_layers"], start=start):
+ print("Creating layer", num, "from paths:", store_layer,
+ file=sys.stderr)
+ info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
+ layers.append(info)
+
+ print("Creating layer", len(layers) + 1, "with customisation...",
+ file=sys.stderr)
+ layers.append(
+ add_customisation_layer(
+ tar,
+ conf["customisation_layer"],
+ mtime=mtime
+ )
+ )
+
+ print("Adding manifests...", file=sys.stderr)
+
+ image_json = {
+ "created": datetime.isoformat(created),
+ "architecture": conf["architecture"],
+ "os": "linux",
+ "config": overlay_base_config(from_image, conf["config"]),
+ "rootfs": {
+ "diff_ids": [f"sha256:{layer.checksum}" for layer in layers],
+ "type": "layers",
+ },
+ "history": [
+ {
+ "created": datetime.isoformat(created),
+ "comment": f"store paths: {layer.paths}"
+ }
+ for layer in layers
+ ],
+ }
+
+ image_json = json.dumps(image_json, indent=4).encode("utf-8")
+ image_json_checksum = hashlib.sha256(image_json).hexdigest()
+ image_json_path = f"{image_json_checksum}.json"
+ add_bytes(tar, image_json_path, image_json, mtime=mtime)
+
+ manifest_json = [
+ {
+ "Config": image_json_path,
+ "RepoTags": [conf["repo_tag"]],
+ "Layers": [layer.path for layer in layers],
+ }
+ ]
+ manifest_json = json.dumps(manifest_json, indent=4).encode("utf-8")
+ add_bytes(tar, "manifest.json", manifest_json, mtime=mtime)
+
+ print("Done.", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..0bfd011ad4 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -45,6 +46,7 @@ (define-module (gnu system image)
#:use-module (gnu system uuid)
#:use-module (gnu system vm)
#:use-module (guix packages)
+ #:use-module ((gnu packages) #:select (search-auxiliary-file))
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages bootloaders)
@@ -58,6 +60,7 @@ (define-module (gnu system image)
#:use-module (gnu packages hurd)
#:use-module (gnu packages linux)
#:use-module (gnu packages mtools)
+ #:use-module (gnu packages python)
#:use-module (gnu packages virtualization)
#:use-module ((srfi srfi-1) #:prefix srfi-1:)
#:use-module (srfi srfi-11)
@@ -78,6 +81,7 @@ (define-module (gnu system
This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 14 Mar 2023 01:24
[PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-3-go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

Toggle diff (78 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 55d1218df5..4bbdfd2a59 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,63 @@
(channel-news
(version 0)
+ (entry (commit "a5c3baf510adab1f5b3bb855b1aa9cafe3cb66b9")
+ (title
+ (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "????? @samp{docker-layered} ?????? ??? @command{guix pack} ???????"))
+ (body
+ (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
+@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
+Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
+untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
+im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
+Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
+angelegt wird:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
+indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
+übergeben.
+
+Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
+@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")
+ (en "Docker layered images can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+The @command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+
+See @command{info \"(guix) Invoking guix pack\"} and
+@command{info \"(guix) System Images\"} for more information.")
+ (ru "????????? ??????? ???????? ???????????? Docker ??????? ? ???????
+@command{guix pack --format=docker-layered}, ??????? ??????? Docker ????? ?
+?????? ? store ?????????????? ?? ????????? ?????, ??????? ????? ???????
+???????? ???????. ????? ????? ?????? ? GNU store ? ???????? gzipped tarball.
+
+?????? ???????? Docker layered ????? ? @code{hello} ???????:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} ?????? ????? ????????? layered Docker ????? ?????
+???????? ? ????? @option{--image-type} ????????? @code{docker-layered}.
+
+???????? @command{info \"(guix) Invoking guix pack\"} ?
+@command{info \"(guix) System Images\"} ??? ????????? ????? ?????????
+????????.")))
+
(entry (commit "0e18c5e5bcb9204c278cfc75493d3b02b746d5c3")
(title
(en "Linux-libre kernel updated to 6.2")
--
2.38.0
O
O
Oleg Pykhalov wrote on 14 Mar 2023 01:40
Missing diff in cover lever for v2 patch
(address . 62153@debbugs.gnu.org)
87wn3k17yf.fsf@gmail.com
Missing diff in cover leter for v2 patch attached below.

Toggle quote (4 lines)
> This patch series applies your suggestions. Also it's rebased on
> origin/master and added a missing documentation for ‘docker-layered’ format in
> ‘guix system image’ command in doc/guix.texi file (following diff).

Toggle snippet (14 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index bd0ee126ee..6938743154 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -43306,6 +43306,8 @@ one or multiple partitions.
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPwngUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pw5aw/+JeyQ3p6e4cfIX6+RHnDjUgP9HA2Q
ESTfjzWxsJcCxleL0LKO3AbOUVby/qq4TrUgNSm1cqL/S4vWDAIo34XEyT5Xerdg
oHSNtoaHqD467NsGhSvtdS8TYGe7JLTTr76nkNzwjQAGjDYXzJXvrcr758H4VMw0
6k143Q1uqDYj3+E2uqQWT9wme5oZ4IkxzZW/+XCME3s00yBF19Uti9VqROLAEIhV
OnQAuurDYScfZ7in17DMuyHxOfrJm4M2G7UxRSiVnKbBeGjbaQoL4+QSRCRa+/+O
0VABMVnYFYUqQPFKm+Y7csQiFbjE6mojc3nNwqBEd0i3rUwu4vPjad6mirKUC9FQ
C3gpBDAIVNJ6I0UWmmBdRH/p/hNRvtiAw0FAhBaV0arTsWoLMeybmZjoxDF533zo
wg7ZPbaZtJd8nHowzKsdbXYqYEf1bhio1goBcnraGJdgBFscSsT5eJhk/Tgp9n3a
yQB9NyF4E/Uc7WLzSiscwaxU2ksg+W1s9xZVrz5xcbu1yX8+yI9u+dSp6e6fa7qf
VLyYOGaD7Qju5alCPc6p4z12/a+DD5ZAEJeBReYoJCfEEPjfOClrwHtnJksQyPY/
PlCL+blYcC4yrMQgQN7UeWbDWTEGzwJPHIWgOMzR/meYOQO0TZ1yFj865E9VmyfF
TjXACct5ZaD5Y64=
=otdV
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 14 Mar 2023 09:19
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
86cz5b923c.fsf@gmail.com
Hi Oleg,

CC: core teams

On Tue, 14 Mar 2023 at 00:10, Oleg Pykhalov <go.wigust@gmail.com> wrote:

Toggle quote (25 lines)
>>> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
>>> new file mode 100644
>>> index 0000000000..9ad2168c2d
>>> --- /dev/null
>>> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
>>> @@ -0,0 +1,391 @@
>>> +"""
>>> +This script generates a Docker image from a set of store paths. Uses
>>> +Docker Image Specification v1.2 as reference [1].
>>
>> Instead of Python, would it possible to implement in Guile? I mean,
>> does Python have something that is missing in Guile?
>>
>> The facility for manipulating Tar? Something else?
>
> I think nothing else. As I understand Python implemented Tar inside the
> language itself in 2500 lines of code by manipulating binary data.
>
> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>
> Technically it's probably possible to use tar utility with --append flag
> instead of opening a new file and streaming to it as the Python script
> does. To be honest I would like not to write it in this way if the
> Python script does not block current patch for merge.

Ok, thanks for explaining.

Toggle quote (2 lines)
> Also I don't see myself writing Tar implementation in Guile, yet. ;-)

Maybe not reimplementing Tar in Guile, maybe just enough for working.
Or maybe some Guile bindings. Or maybe something is already around for
the bootstrap story.

The use of external tools as Python for producing built-in Guix feature
will be the first time, no?

For what it is worth, I would prefer to consider the options before
emitting an opinion about dragging Python building/packing layered
Docker. :-)

Toggle quote (4 lines)
> The Nix project uses this script heavily to build layered images, so it
> should be robust in terms of up to date to current Tar and Python
> implementations.

Do you mean this script is coming from Nix. Well, in all cases, this
script is not trivial and so it requires Copyright for authorship.



[...]


Toggle quote (3 lines)
> The following example shows common layers between images, which will be
> not tranfered if you load image inside Docker as well as pull and push:

Thanks for explaining.


Cheers,
simon

PS: I will be off-line these 2-3 next weeks. So a lack of an answer
from me will not be a lack of interest. ;-)
C
C
Christopher Baines wrote on 14 Mar 2023 10:11
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
8735671yu4.fsf@cbaines.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (3 lines)
> And I remember something in that direction by Chris but I am unable to
> find back the patch. )-:

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmQQOmNfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XeUkRAApwCv6Zh8MrAuDE5TRebOf0IJflIq98wc
iB1i643fUvGZzVsr/pjAXsrkKDqyBdPH9Oee/JHvPPtEcCVGJwwAtodiGC6Tnf1t
8eYzWSXdzYVK7oFReQZ/oHLzmA0zQYVTmcnIoqP4wy+Cgv/zE3fGWlLrpiIau2hV
NgPUo4wPdZu8ZwM0B2+gK2H4I5pNXUxhIuWFI8rORU6q0nxL/OqrDMh5G/+B1UPh
cuGHBOxoBx3JERUnaHkk0BkgS1TySYpaVDA/9RwtCeqV8cv0bSjznkOqdtusr0vK
prM5WJwzOpdlHkAqwkmEEuE93KjdAoMl5gmYWWlUobCdAtwLNLTXCGa+wAlx7VmW
kFl0XygIJCeXHICtDhiBkv/cfIVSpKYVbqJExRAhzeH6Ykllww6UUaEHC32Fy0ae
WsWUUpgpRGZSgbT70i05D8TaPBjNgIBUV5HfczWTsRgvR1b4fVaqrKaxCooy1gV1
YjIJO7ddNHGCSNp0uxGyMzZMmpEy46gQvSiUNTW46AYgl2c5TEVMaQ/gLNvgmAvt
K+PRQhOaHyoZNLLCIZaRhmuPwbwKEh3MLOGcR/AUF7nW5sp2wk9LBm889lDmarpz
bHk5ZfhqiZryeo36k9O3L3DE70h+fibm1uKr7zn6jRt74LmvGdWA4lCWaz4A9USE
VjZkN4ayEaQ=
=9k/e
-----END PGP SIGNATURE-----

R
R
Ricardo Wurmus wrote on 14 Mar 2023 10:15
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
874jqnzo8a.fsf@elephly.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (23 lines)
>>> Instead of Python, would it possible to implement in Guile? I mean,
>>> does Python have something that is missing in Guile?
>>>
>>> The facility for manipulating Tar? Something else?
>>
>> I think nothing else. As I understand Python implemented Tar inside the
>> language itself in 2500 lines of code by manipulating binary data.
>>
>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>
>> Technically it's probably possible to use tar utility with --append flag
>> instead of opening a new file and streaming to it as the Python script
>> does. To be honest I would like not to write it in this way if the
>> Python script does not block current patch for merge.
>
> Ok, thanks for explaining.
>
>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>
> Maybe not reimplementing Tar in Guile, maybe just enough for working.
> Or maybe some Guile bindings. Or maybe something is already around for
> the bootstrap story.

gash-utils has (gash ustar); it’s about 620 lines of code.

--
Ricardo
L
L
Ludovic Courtès wrote on 16 Mar 2023 11:37
(name . Ricardo Wurmus)(address . rekado@elephly.net)
877cvhhth5.fsf@gnu.org
Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

Toggle quote (27 lines)
> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
>>>> Instead of Python, would it possible to implement in Guile? I mean,
>>>> does Python have something that is missing in Guile?
>>>>
>>>> The facility for manipulating Tar? Something else?
>>>
>>> I think nothing else. As I understand Python implemented Tar inside the
>>> language itself in 2500 lines of code by manipulating binary data.
>>>
>>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>>
>>> Technically it's probably possible to use tar utility with --append flag
>>> instead of opening a new file and streaming to it as the Python script
>>> does. To be honest I would like not to write it in this way if the
>>> Python script does not block current patch for merge.
>>
>> Ok, thanks for explaining.
>>
>>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>>
>> Maybe not reimplementing Tar in Guile, maybe just enough for working.
>> Or maybe some Guile bindings. Or maybe something is already around for
>> the bootstrap story.
>
> gash-utils has (gash ustar); it’s about 620 lines of code.

Disarchive also has a tar implementation. No excuse! :-)

Oleg, could you check which of these would satisfy your needs? I had a
plan to improve the tar implementation in Gash-Utils, perhaps it’s a
good time to get my act together.

Thanks,
Ludo’.
O
O
Oleg Pykhalov wrote on 20 Mar 2023 07:38
(name . Ludovic Courtès)(address . ludo@gnu.org)
87h6ugapwh.fsf@gmail.com
Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (35 lines)
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Simon Tournier <zimon.toutoune@gmail.com> writes:
>>
>>>>> Instead of Python, would it possible to implement in Guile? I mean,
>>>>> does Python have something that is missing in Guile?
>>>>>
>>>>> The facility for manipulating Tar? Something else?
>>>>
>>>> I think nothing else. As I understand Python implemented Tar inside the
>>>> language itself in 2500 lines of code by manipulating binary data.
>>>>
>>>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>>>
>>>> Technically it's probably possible to use tar utility with --append flag
>>>> instead of opening a new file and streaming to it as the Python script
>>>> does. To be honest I would like not to write it in this way if the
>>>> Python script does not block current patch for merge.
>>>
>>> Ok, thanks for explaining.
>>>
>>>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>>>
>>> Maybe not reimplementing Tar in Guile, maybe just enough for working.
>>> Or maybe some Guile bindings. Or maybe something is already around for
>>> the bootstrap story.
>>
>> gash-utils has (gash ustar); it’s about 620 lines of code.
>
> Disarchive also has a tar implementation. No excuse! :-)
>
> Oleg, could you check which of these would satisfy your needs? I had a
> plan to improve the tar implementation in Gash-Utils, perhaps it’s a
> good time to get my act together.

Gash-Utils should work, e.g. [1] script. It's already possible to
rewrite with Gash-Utils, but at least write-ustar-file and
write-ustar-footer should be exported in ustar.scm.

Disarchive requires to write a file specification in case of using
disarchive-assemble. And disarchive-assemble does not work, if I don't
miss anything [2]. Also gzip compression does not work in a Guile REPL.

[1]:

Toggle snippet (37 lines)
#!/usr/bin/env -S guile --no-auto-compile -e main -s
!#

(set! %load-path
(append '("/gnu/store/...-gash-utils-0.1.0/share/guile/site/3.0"
"/gnu/store/...-gash-0.2.0/share/guile/site/3.0")
%load-path))

(set! %load-compiled-path
(append '("/gnu/store/...-gash-utils-0.1.0/lib/guile/3.0/site-ccache"
"/gnu/store/...-gash-0.2.0/lib/guile/3.0/site-ccache")
%load-compiled-path))

(use-modules (gash ustar)
(srfi srfi-26)
(guix build utils))

(define write-ustar-file
(@@ (gash ustar) write-ustar-file))

(define write-ustar-footer
(@@ (gash ustar) write-ustar-footer))

(define (main . args)
(call-with-port (open-file "out.tar.gz" "wb")
(lambda (port)
(with-directory-excursion "."
(call-with-compressed-output-port 'gzip port
(cut write-ustar-file <> "Makefile.am"
#:verbosity 0)))
(with-directory-excursion "doc"
(call-with-compressed-output-port 'gzip port
(cut write-ustar-file <> "."
#:verbosity 0)))
(write-ustar-footer port))))

[2]:

Toggle snippet (263 lines)
$ guile
,m(disarchive assemblers tarball)
(assemble-tarball (disassemble-tarball "out.tar") "result/out.tar")

# Generated tarball:
$ tar tf result/out.tar/sha256/1e7100029373723df900712d1191be0bad5beadf752f367bb264fab891c36356
./
./images/
./images/bootstrap-graph.dot
./images/service-graph.pdf
./guix-cookbook.zh_Hans.texi
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

# Original tarball:
$ tar tf out.tar
./
./images/
./images/bootstrap-graph.dot
./images/bootstrap-graph.eps
./images/bootstrap-graph.pdf
./images/bootstrap-graph.png
./images/bootstrap-packages.dot
./images/bootstrap-packages.eps
./images/bootstrap-packages.pdf
./images/bootstrap-packages.png
./images/coreutils-bag-graph.dot
./images/coreutils-bag-graph.eps
./images/coreutils-bag-graph.pdf
./images/coreutils-bag-graph.png
./images/coreutils-graph.dot
./images/coreutils-graph.eps
./images/coreutils-graph.pdf
./images/coreutils-graph.png
./images/coreutils-size-map.eps
./images/coreutils-size-map.png
./images/gcc-core-mesboot0-graph.dot
./images/gcc-core-mesboot0-graph.eps
./images/gcc-core-mesboot0-graph.pdf
./images/gcc-core-mesboot0-graph.png
./images/installer-network.png
./images/installer-partitions.png
./images/installer-resume.png
./images/service-graph.dot
./images/service-graph.eps
./images/service-graph.pdf
./images/service-graph.png
./images/shepherd-graph.dot
./images/shepherd-graph.eps
./images/shepherd-graph.pdf
./images/shepherd-graph.png
./.dirstamp
./environment-gdb.scm
./fdl-1.3.texi
./os-config-bare-bones.texi
./os-config-desktop.texi
./os-config-lightweight-desktop.texi
./package-hello.json
./package-hello.scm
./stamp-1
./stamp-10
./stamp-11
./stamp-2
./stamp-3
./stamp-4
./stamp-5
./stamp-6
./stamp-7
./stamp-8
./stamp-9
./stamp-vti
./contributing.fa.texi
./guix.fa.texi
./contributing.fi.texi
./guix.fi.texi
./contributing.it.texi
./guix.it.texi
./contributing.ko.texi
./guix.ko.texi
./contributing.sk.texi
./guix.sk.texi
./guix-cookbook.es.texi
./guix-cookbook.fa.texi
./guix-cookbook.fi.texi
./guix-cookbook.pt_BR.texi
./guix-cookbook.ru.texi
./guix-cookbook.zh_Hans.texi
./version-fa.texi
./guix.fa.info-1
./guix.fa.info-2
./guix.fa.info-3
./guix.fa.info-4
./guix.fa.info-5
./guix.fa.info-6
./guix.fa.info
./version-fi.texi
./guix.fi.info-1
./guix.fi.info-2
./guix.fi.info-3
./guix.fi.info-4
./guix.fi.info-5
./guix.fi.info-6
./guix.fi.info
./version-it.texi
./guix.it.info-1
./guix.it.info-2
./guix.it.info-3
./guix.it.info-4
./guix.it.info-5
./guix.it.info-6
./guix.it.info
./version-ko.texi
./guix.ko.info-1
./guix.ko.info-2
./guix.ko.info-3
./guix.ko.info-4
./guix.ko.info-5
./guix.ko.info-6
./guix.ko.info
./version-sk.texi
./guix.sk.info-1
./guix.sk.info-2
./guix.sk.info-3
./guix.sk.info-4
./guix.sk.info-5
./guix.sk.info-6
./guix.sk.info
./guix-cookbook.es.info
./guix-cookbook.fa.info
./guix-cookbook.fi.info
./guix-cookbook.pt_BR.info
./guix-cookbook.ru.info
./guix-cookbook.zh_Hans.info
./he-config-bare-bones.scm
./guix-lint.1
./build.scm
./contributing.texi
./guix-cookbook.texi
./htmlxref.cnf
./local.mk
./guix-cookbook.info
./guix-gc.1
./guix-git.1
./guix-container.1
./guix-copy.1
./guix-describe.1
./guix-processes.1
./guix-size.1
./guix-weather.1
./guix-shell.1
./guix-cookbook.ko.texi
./guix-cookbook.fr.texi
./guix-cookbook.de.texi
./guix-cookbook.sk.texi
./contributing.zh_CN.texi
./contributing.ru.texi
./contributing.pt_BR.texi
./contributing.fr.texi
./contributing.es.texi
./contributing.de.texi
./guix.zh_CN.texi
./guix.ru.texi
./guix.pt_BR.texi
./guix.de.texi
./guix.es.texi
./guix.fr.texi
./guix-cookbook.sk.info
./guix-cookbook.de.info
./guix-cookbook.fr.info
./guix-cookbook.ko.info
./version-de.texi
./version-ru.texi
./version-fr.texi
./version-es.texi
./version-pt_BR.texi
./version-zh_CN.texi
./guix-daemon.1
./guix.pt_BR.info-1
./guix.pt_BR.info-2
./guix.fr.info-1
./guix.fr.info-2
./guix.de.info-1
./guix.de.info-2
./guix.pt_BR.info-3
./guix.ru.info-1
./guix.ru.info-2
./guix.fr.info-3
./guix.ru.info-3
./guix.zh_CN.info-1
./guix.zh_CN.info-2
./guix.es.info-1
./guix.es.info-2
./guix.de.info-3
./guix.pt_BR.info-4
./guix.fr.info-4
./guix.ru.info-4
./guix.zh_CN.info-3
./guix.de.info-4
./guix.es.info-3
./guix.ru.info-5
./guix.pt_BR.info-5
./guix.zh_CN.info-4
./guix.fr.info-5
./guix.es.info-4
./guix.de.info-5
./guix.pt_BR.info-6
./guix.ru.info-6
./guix.fr.info-6
./guix.pt_BR.info-7
./guix.es.info-5
./guix.zh_CN.info-5
./guix.de.info-6
./guix.pt_BR.info
./guix.ru.info-7
./guix.fr.info-7
./guix.ru.info-8
./guix.de.info-7
./guix.es.info-6
./guix.fr.info-8
./guix.zh_CN.info-6
./guix.ru.info
./guix.fr.info
./guix.de.info-8
./guix.zh_CN.info-7
./guix.es.info-7
./guix.de.info
./guix.zh_CN.info
./guix.es.info-8
./guix.es.info
./guix.texi
./version.texi
./guix.info-1
./guix.info-2
./guix.info-3
./guix.info-4
./guix.info-5
./guix.info-6
./guix.info-7
./guix.info
./guix-build.1
./guix-archive.1
./guix-import.1
./guix-offload.1
./guix-graph.1
./guix-hash.1
./guix-challenge.1
./guix-download.1
./guix-edit.1
./guix-repl.1
./guix-publish.1
./guix-package.1
./guix-style.1
./guix-pull.1
./guix-time-machine.1
./guix-environment.1
./guix-pack.1
./guix-deploy.1
./guix-refresh.1
./guix-home.1
./guix-system.1
./guix.1

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQX/18UHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pyHCg//Y+CwbxC5OzVdZFBdSDKYtxOV2w6Z
w6X/ZXRt5sorx2Wfjsn0QJIzO9+6CTEBn1RSeOCQxuVvlQqjdeKqxQ0C90YQQG5t
/R7PZ5wT2LHhztT1R52O8njqj93bppxkJDMsZVg4lg38WzNt2jVvL94mczS1V3zH
4VpccvDHYjIwK/ECkhTEKHRvRSGFqH7VcGJYbKYwGjmhF1Q6+lWq4XBiBhF9C6G0
+TwsS/GMXi207dFEtXdhit6DcxbEi70EseNj55aLDoM9xUumlQJuVTz5f9FRbL/C
QOQmI2xFMPqN2EwZhBX3YL7FjwooA+nyyrj7H0AjWe3IX4RuVyvF9XZNIm1HM4P3
VlcR1foBdKxo1JRCTCU7V/hlD44Yl3e/FXQbKIS6aQyhRahrudYjeeL1gf7jBRya
FAzkv2kP/ORoxP/bKj/BoNV372fOz8Sm5sSr5YzAZJz4DJQmctstIw+6WFcgP8Ca
az682R4sMLDWteAxhZp/NNSg21irM6S7U4Tu4em3nP69ky2Fs5wUy7m1JzMWx9Vg
IrODYJdDN0P5wgSM5hVKAW30J8uLXs3XxwRAl2W+NKCu8l4eonnpKU3KufaSclG+
Qz7cg6xNqTLzvDrYoDaaMhGd2WGJOdQ3wzH3TcWgR9Kh1STNwNnYfXXsZPvYQ5wi
QWmUkZU8RZRQ3BI=
=rP1f
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 20 Mar 2023 17:51
Re: bug#62153: [PATCH 0/2] Disarchive vs Gash-Utils for docker-layered
(name . Ludovic Courtès)(address . ludo@gnu.org)
87cz53bc2f.fsf_-_@gmail.com
Oleg Pykhalov <go.wigust@gmail.com> writes:

Toggle quote (4 lines)
> […]
>
> And disarchive-assemble does not work, if I don't miss anything.

I forgot about that input tarball was generated with changing current
working directory, that's probably the reason of the broken archive.

According to "Assembles from stdin to file" test from
git.ngyro.com/disarchive/tests/cli.scm it is possible to generate a
tarball for docker-layered with Disarchive, but a file describing spec
like git.ngyro.com/disarchive/tests/data/test-archive.da is required.
Generation of this file without having a Tar archive beforehand is
complicated and because of that IMHO ustar.scm from Gash-Utils is
preferred for the task.

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQYjygUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzbEhAAy76WdREdgRRwg4j9h83H8dUa30x7
0+6pjM24kKBrDJYLGmv6aW2pumAr+YVre7n4/n1HdlXLStcdJ86/G0zraPj1wD7+
JvR04l0qGb7LY87lPdU/bGGkyt+YubnGbhZ6XoU0Q4CmHHo2aJzvq1S6+Z98m6YN
f/RWl2mkxnrXDau+rLwE2RXQ4wvjZIMg77SSI64lgWoo3IaiwOuhWxcs2Q+ImebS
ymirYoVzhFPCBgznkD2otEDNOvkUMxJ/e2caVGZ21F3guHhq0Df2gmYhRvaIqj/S
JH8xfguqOTntBSvzhi0zjj50A+pyn976IV2C2Mu5mAqtQB0HpSayF/79Y3kwYXYH
bteVg+tsGRrhC4VTPO3Tf5w2PE+SWr7j6ssSNNyRC9wcoYQjeE1y/btYEUeR85vc
L3TS5uD/KM9WJmmpzxyAnmAGR/WkXAvFDtMZ7ZNcq6qTzQVQYltNJD/K69UZ55c4
vPU4KrnEx49lvtnMDzxl8BhzZsgVGjG6k26iX3B5ZyGWpGGayellDVwPWP21MLPr
1R4wanmcyykJvLl1iAVkDDNCRY8cJn4QKQDmUd0Y3gkSJi4Ic7zqYHzIFaEHbLda
/jJ1XB1vTsadXQTJAIdOIQw0fXbK9NF7Tm68nm+czF6AyjUYU5Nz6NmQ85zAEAH4
1cpPWOSj0xOsoYQ=
=uXt9
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 31 May 2023 10:45
[PATCH] Add Docker layered image for pack and system (v3)
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
cover.1685522135.git.go.wigust@gmail.com
Hi, Guix.

These patches series is rebased on origin/master. Also, the Python script is
replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed tests:
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system"
make check-system TESTS="docker-layered-system"

Oleg Pykhalov (2):
guix: docker: Build layered image.
news: Add entry for the new 'docker-layered' distribution format.

doc/guix.texi | 18 +++-
etc/news.scm | 58 ++++++++++++
gnu/image.scm | 3 +-
gnu/system/image.scm | 76 +++++++++++----
gnu/tests/docker.scm | 20 +++-
guix/docker.scm | 205 +++++++++++++++++++++++++++++++---------
guix/scripts/pack.scm | 62 ++++++++++--
guix/scripts/system.scm | 11 ++-
tests/pack.scm | 48 ++++++++++
9 files changed, 424 insertions(+), 77 deletions(-)


base-commit: 77f52db416a13e195d090cad4e9e7658feb2e86b
--
2.38.0
O
O
Oleg Pykhalov wrote on 31 May 2023 10:47
[PATCH] guix: docker: Build layered image.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
dd6c7c816bcb414682e1006d7e83b45e8ac6c575.1685522135.git.go.wigust@gmail.com
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image Reference): Same.
(image-type Reference): Document docker-layered-image-type.
* gnu/image.scm
(validate-image-format)[docker-layered]: New image format.
* gnu/system/image.scm
(docker-layered-image, docker-layered-image-type): New variables.
(system-docker-image)[layered-image?]: New argument.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* tests/pack.scm: Add "docker-layered-image + localstatedir" test.
---
doc/guix.texi | 18 +++-
gnu/image.scm | 3 +-
gnu/system/image.scm | 76 +++++++++++----
gnu/tests/docker.scm | 20 +++-
guix/docker.scm | 205 +++++++++++++++++++++++++++++++---------
guix/scripts/pack.scm | 62 ++++++++++--
guix/scripts/system.scm | 11 ++-
tests/pack.scm | 48 ++++++++++
8 files changed, 366 insertions(+), 77 deletions(-)

Toggle diff (436 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 5fd2449ed5..1c95ec4320 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6984,9 +6984,15 @@ Invoking guix pack
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -44309,6 +44315,8 @@ image Reference
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
@@ -44644,6 +44652,10 @@ image-type Reference
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-syntax-rule (define-set-sanitizer name field set)
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..3a502f19ec 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -78,6 +79,7 @@ (define-module (gnu system image)
efi-disk-image
iso9660-image
docker-image
+ docker-layered-image
tarball-image
wsl2-image
raw-with-offset-disk-image
@@ -89,6 +91,7 @@ (define-module (gnu system image)
iso-image-type
uncompressed-iso-image-type
docker-image-type
+ docker-layered-image-type
tarball-image-type
wsl2-image-type
raw-with-offset-image-type
@@ -167,6 +170,10 @@ (define docker-image
(image-without-os
(format 'docker)))
+(define docker-layered-image
+ (image-without-os
+ (format 'docker-layered)))
+
(define tarball-image
(image-without-os
(format 'tarball)))
@@ -237,6 +244,11 @@ (define docker-image-type
(name 'docker)
(constructor (cut image-with-os docker-image <>))))
+(define docker-layered-image-type
+ (image-type
+ (name 'docker-layered)
+ (constructor (cut image-with-os docker-layered-image <>))))
+
(define tarball-image-type
(image-type
(name 'tarball)
@@ -633,9 +645,12 @@ (define (image-with-label base-image label)
(define* (system-docker-image image
#:key
- (name "docker-image"))
+ (name "docker-image")
+ (archiver tar)
+ layered-image?)
"Build a docker image for IMAGE. NAME is the base name to use for the
-output file."
+output file. If LAYERED-IMAGE? is true, the image will with many of the store
+paths being on their own layer to improve sharing between images."
(define boot-program
;; Program that runs the boot script of OS, which in turn starts shepherd.
(program-file "boot-program"
@@ -678,9 +693,11 @@ (define* (system-docker-image image
(use-modules (guix docker)
(guix build utils)
(gnu build image)
+ (srfi srfi-1)
(srfi srfi-19)
(guix build store-copy)
- (guix store database))
+ (guix store database)
+ (ice-9 receive))
;; Set the SQL schema location.
(sql-schema #$schema)
@@ -700,18 +717,31 @@ (define* (system-docker-image image
#:register-closures? #$register-closures?
#:deduplicate? #f
#:system-directory #$os)
- (build-docker-image
- #$output
- (cons* image-root
- (map store-info-item
- (call-with-input-file #$graph
- read-reference-graph)))
- #$os
- #:entry-point '(#$boot-program #$os)
- #:compressor '(#+(file-append gzip "/bin/gzip") "-9n")
- #:creation-time (make-time time-utc 0 1)
- #:system #$image-target
- #:transformations `((,image-root -> ""))))))))
+ (when #$layered-image?
+ (setenv "PATH"
+ (string-join (list #+(file-append archiver "/bin")
+ #+(file-append coreutils "/bin")
+ #+(file-append gzip "/bin"))
+ ":")))
+ (apply build-docker-image
+ (append (list #$output
+ (append (if #$layered-image?
+ '()
+ (list image-root))
+ (map store-info-item
+ (call-with-input-file #$graph
+ read-reference-graph)))
+ #$os
+ #:entry-point '(#$boot-program #$os)
+ #:compressor
+ '(#+(file-append gzip "/bin/gzip") "-9n")
+ #:creation-time (make-time time-utc 0 1)
+ #:system #$image-target
+ #:transformations `((,image-root -> "")))
+ (if #$layered-image?
+ (list #:root-system image-root
+ #:layered-image? #$layered-image?)
+ '()))))))))
(computed-file name builder
;; Allow offloading so that this I/O-intensive process
@@ -720,6 +750,18 @@ (define* (system-docker-image image
#:options `(#:references-graphs ((,graph ,os))
#:substitutable? ,substitutable?))))
+(define* (system-docker-layered-image image
+ #:key
+ (name "docker-image")
+ (archiver tar)
+ (layered-image? #t))
+ "Build a docker image for IMAGE. NAME is the base name to use for the
+output file."
+ (system-docker-image image
+ #:name name
+ #:archiver archiver
+ #:layered-image? layered-image?))
+
;;;
;;; Tarball image.
@@ -811,7 +853,7 @@ (define (image->root-file-system image)
"Return the IMAGE root partition file-system type."
(case (image-format image)
((iso9660) "iso9660")
- ((docker tarball wsl2) "dummy")
+ ((docker docker-layered tarball wsl2) "dummy")
(else
(partition-file-system (find-root-partition image)))))
@@ -948,6 +990,8 @@ (define* (system-image image)
("bootcfg" ,bootcfg))))
((memq image-format '(docker))
(system-docker-image image*))
+ ((memq image-format '(docker-layered))
+ (system-docker-layered-image image*))
((memq image-format '(tarball))
(system-tarball-image image*))
((memq image-format '(wsl2))
diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm
index edc9804414..0cccc02ad2 100644
--- a/gnu/tests/docker.scm
+++ b/gnu/tests/docker.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2019 Danny Milosavljevic <dannym@scratchpost.org>
;;; Copyright © 2019-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -43,7 +44,8 @@ (define-module (gnu tests docker)
#:use-module (guix build-system trivial)
#:use-module ((guix licenses) #:prefix license:)
#:export (%test-docker
- %test-docker-system))
+ %test-docker-system
+ %test-docker-layered-system))
(define %docker-os
(simple-operating-system
@@ -316,3 +318,19 @@ (define %test-docker-system
(locale-libcs (list glibc)))
#:type docker-image-type)))
run-docker-system-test)))))
+
+(define %test-docker-layered-system
+ (system-test
+ (name "docker-layered-system")
+ (description "Run a system image as produced by @command{guix system
+docker-layered-image} inside Docker.")
+ (value (with-monad %store-monad
+ (>>= (lower-object
+ (system-image (os->image
+ (operating-system
+ (inherit (simple-operating-system))
+ ;; Use locales for a single libc to
+ ;; reduce space requirements.
+ (locale-libcs (list glibc)))
+ #:type docker-layered-image-type)))
+ run-docker-system-test)))))
diff --git a/guix/docker.scm b/guix/docker.scm
index 5e6460f43f..e10b940aa4 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -28,6 +29,8 @@ (define-module (guix docker)
delete-file-recursively
with-directory-excursion
invoke))
+ #:use-module (guix diagnostics)
+ #:use-module (guix i18n)
#:use-module (gnu build install)
#:use-module (json) ;guile-json
#:use-module (srfi srfi-1)
@@ -38,6 +41,9 @@ (define-module (guix docker)
#:use-module (rnrs bytevectors)
#:use-module (ice-9 ftw)
#:use-module (ice-9 match)
+ #:use-module (ice-9 popen)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 receive)
#:export (build-docker-image))
;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image.
@@ -92,12 +98,12 @@ (define (canonicalize-repository-name name)
(make-string (- min-length l) padding-character)))
(_ normalized-name))))
-(define* (manifest path id #:optional (tag "guix"))
+(define* (manifest path layers #:optional (tag "guix"))
"Generate a simple image manifest."
(let ((tag (canonicalize-repository-name tag)))
`#(((Config . "config.json")
(RepoTags . #(,(string-append tag ":latest")))
- (Layers . #(,(string-append id "/layer.tar")))))))
+ (Layers . ,(list->vector layers))))))
;; According to the specifications this is required for backwards
;; compatibility. It duplicates information provided by the manifest.
@@ -106,8 +112,8 @@ (define* (repositories path id #:optional (tag "guix"))
`((,(canonicalize-repository-name tag) . ((latest . ,id)))))
;; See https://github.com/opencontainers/image-spec/blob/master/config.md
-(define* (config layer time arch #:key entry-point (environment '()))
- "Generate a minimal image configuration for the given LAYER file."
+(define* (config layers-diff-ids time arch #:key entry-point (environment '()))
+ "Generate a minimal image configuration for the given LAYERS files."
;; "architecture" must be values matching "platform.arch" in the
;; runtime-spec at
;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform
@@ -125,7 +131,7 @@ (define* (config layer time arch #:key entry-point (environment '()))
(container_config . #nil)
(os . "linux")
(rootfs . ((type . "layers")
- (diff_ids . #(,(layer-diff-id layer)))))))
+ (diff_ids . ,(list->vector layers-diff-ids))))))
(define directive-file
;; Return the file or directory created by a 'evaluate-populate-directive'
@@ -136,6 +142,37 @@ (define directive-file
(('directory name _ ...)
(string-trim name #\/))))
+(define %docker-image-max-layers
+ 100)
+
+(define (paths-split-sort paths)
+ "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage."
+ (let* ((paths-length (length paths))
+ (port (apply open-pipe* OPEN_READ
+ (append '("du" "--summarize") paths)))
+ (output (read-string port)))
+ (close-port port)
+ (receive (head tail)
+ (split-at
+ (map (match-lambda ((size . path) path))
+ (sort (map (lambda (line)
+ (match (string-split line #\tab)
+ ((size path)
+ (cons (string->number size) path))))
+ (string-split
+ (string-trim-right output #\newline)
+ #\newline))
+ (lambda (path1 path2)
+ (< (match path2 ((size . _) size))
+ (match path1 ((size . _) size))))))
+ (if (>= paths-length %docker-image-max-layers)
+ (- %docker-image-max-layers 2)
+ (1- paths-length)))
+ (list head tail))))
+
+(define (create-empty-tar file)
+ (invoke "tar" "-cf" file "--files-from" "/dev/null"))
+
(define* (build-docker-image image paths prefix
#:key
(repository "guix")
@@ -146,11 +183,13 @@ (define* (build-docker-image image paths prefix
entry-point
(environment '())
compressor
- (creation-time (current-time time-utc)))
- "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX
-must be a store path that is a prefix of any store paths in PATHS. REPOSITORY
-is a descriptive name that will show up in \"REPOSITORY\" column of the output
-of \"docker images\".
+ (creation-time (current-time time-utc))
+ layered-image?
+ root-system)
+ "Write to IMAGE a layerer Docker image archive containing the given PATHS.
+PREFIX must be a store path that is a prefix of any store paths in PATHS.
+REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of
+the output of \"docker images\".
When DATABASE is true, copy it to /var/guix/db in the image and create
/var/guix/gcroots and friends.
@@ -172,7 +211,14 @@ (define* (build-docker-image image paths prefix
SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in
PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a
command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a
-SRFI-19 time-utc object, as the creation time in metadata."
+SRFI-19 time-utc object, as the creation time in metadata.
+
+When LAYERED-IMAGE? is true build layered image, providing a Docker
+image with many of the store paths being on their own layer to improve sharing
+between images.
+
+ROOT-SYSTEM is a directory with a provisioned root file system, which will be
+added to image as a layer."
(define (sanitize path-fragment)
(escape-special-chars
;; GNU tar strips the leading slash off of absolute paths before applying
@@ -203,6 +249,53 @@ (define* (build-docker-image image paths prefix
(if (eq? '() transformations)
'()
`("--transform" ,(transformations->expression transformations))))
+ (define layers-hashes
+ (match-lambda
+ (((head ...) (tail ...) id)
+ (create-empty-tar "image.tar")
+ (let* ((head-layers
+ (map
+ (lambda (file)
+ (invoke "tar" "cf" "layer.tar" file)
+ (let* ((file-hash (layer-diff-id "layer.tar"))
+ (file-name (string-append file-hash "/layer.tar")))
+ (mkdir file-hash)
+ (rename-file "layer.tar" file-name)
+ (invoke "tar" "-rf" "image.tar" file-name)
+ (delete-file file-name)
+ file-hash))
+ head))
+ (tail-layer
+ (begin
+ (create-empty-tar "layer.tar")
+ (for-each
This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 31 May 2023 10:47
[PATCH] news: Add entry for the new 'docker-layered' distribution format.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
e4326a843b52be3147cd9d495cb7bf92d0fab8ce.1685522135.git.go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

Toggle diff (78 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 314f0ab352..cb2dc34876 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021, 2023 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,63 @@
(channel-news
(version 0)
+ (entry (commit "dd6c7c816bcb414682e1006d7e83b45e8ac6c575")
+ (title
+ (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "????? @samp{docker-layered} ?????? ??? @command{guix pack} ???????"))
+ (body
+ (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
+@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
+Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
+untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
+im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
+Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
+angelegt wird:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
+indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
+übergeben.
+
+Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
+@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")
+ (en "Docker layered images can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+The @command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+
+See @command{info \"(guix) Invoking guix pack\"} and
+@command{info \"(guix) System Images\"} for more information.")
+ (ru "????????? ??????? ???????? ???????????? Docker ??????? ? ???????
+@command{guix pack --format=docker-layered}, ??????? ??????? Docker ????? ?
+?????? ? store ?????????????? ?? ????????? ?????, ??????? ????? ???????
+???????? ???????. ????? ????? ?????? ? GNU store ? ???????? gzipped tarball.
+
+?????? ???????? Docker layered ????? ? @code{hello} ???????:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} ?????? ????? ????????? layered Docker ????? ?????
+???????? ? ????? @option{--image-type} ????????? @code{docker-layered}.
+
+???????? @command{info \"(guix) Invoking guix pack\"} ?
+@command{info \"(guix) System Images\"} ??? ????????? ????? ?????????
+????????.")))
+
(entry (commit "ba5da5125a81307500982517e2f458d57b024668")
(title
(en "New @code{arguments} rule for @command{guix style}")
--
2.38.0
G
G
Greg Hogan wrote on 31 May 2023 14:53
Re: [bug#62153] [PATCH] Add Docker layered image for pack and system (v3)
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)(address . 62153@debbugs.gnu.org)
CA+3U0Z=wZ6NaNX=rRz7m3CGwy8VA_Koe1Cv6Y-h3z7a7qMLz6Q@mail.gmail.com
On Wed, May 31, 2023 at 4:46?AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
Toggle quote (14 lines)
>
> Hi, Guix.
>
> These patches series is rebased on origin/master. Also, the Python script is
> replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed tests:
> make check TESTS="tests/pack.scm"
> make check-system TESTS="docker-system"
> make check-system TESTS="docker-layered-system"
>
>
> Oleg Pykhalov (2):
> guix: docker: Build layered image.
> news: Add entry for the new 'docker-layered' distribution format.

Why not use layered images for all docker packs?
O
O
Oleg Pykhalov wrote on 31 May 2023 15:14
(name . Greg Hogan)(address . code@greghogan.com)(address . 62153@debbugs.gnu.org)
87cz2gd4mm.fsf@gmail.com
Greg Hogan <code@greghogan.com> writes:

Toggle quote (17 lines)
> On Wed, May 31, 2023 at 4:46?AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
>>
>> Hi, Guix.
>>
>> These patches series is rebased on origin/master. Also, the Python script is
>> replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed tests:
>> make check TESTS="tests/pack.scm"
>> make check-system TESTS="docker-system"
>> make check-system TESTS="docker-layered-system"
>>
>>
>> Oleg Pykhalov (2):
>> guix: docker: Build layered image.
>> news: Add entry for the new 'docker-layered' distribution format.
>
> Why not use layered images for all docker packs?

Do you mean use layered images by default without ability to build all
in a single layer? Current layered implementation is slow to build
because it needs to calculate a size of each layer, pack, and compress.
So if user wants a faster build, a non-layered image is still an option.

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmR3SFEUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzUmA/9Hqd7U8yB00GA/wF1F7TU/3AcpaFP
uY6LHfUB0ZePuu5r3cQcx2LlCISJ6KLrXOk+d6vTDc+WK4TpTQrH5V5Gt8WAU3L2
Nt58WtDiZnZwX6uuO1Dxk1efF4eJRIIzLqtXluDzJ+jpCj2Q9M43Js8EODRk21rC
w+j+e5Xq2xF3R7aBaGRUN8euFI6nyfZgZC2kiuOAc9xBCt6hMmOPEm4mwaqIUABG
BSa4Nr/Qv0W/GbUEhWgJOWI4NGljbanWLuRlC5IXd4tHEAU2p7lEGg4uWm1Brm+r
2mvHfu0y8tLwnb8F9SjscjBG3OXieM3YjmYeTeOSiNRY5tqnOTo9o4Kq7wmeezQU
02FYA3+U7vvIJ7WCWbEowZgv4ZEW3UCsDiNdt0+4GR72J10jR/MTdFVOhxV0Heo6
4alOxBm0qoLJh9+3dYwY6wEYlWfAwIPXaLeGX4F4GQ0YdPsMFMCCeSdkz1vN4A6l
1q8A29IAuRQcLxv9G3T9+bO4tsYpN7jtB6c8M/4+na03Tc/ugAOWIY+Ne0wBBAAA
k1sqv69JljPC9rE1gaxwUU7FTkY8/pbhGzH0MFZG/C7qUYbz4L5Z7ANzlAnhZGWl
isVFrq2VkTC0XZjxISaQEUsRLcAP82HrTVDfV7KIsK3iuczQVWVPJXgu/rgJ9rgi
rMJcHluPJHJ5IUc=
=mfr+
-----END PGP SIGNATURE-----

G
G
Greg Hogan wrote on 2 Jun 2023 19:02
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)(address . 62153@debbugs.gnu.org)
CA+3U0Z=VktVBTjEZ5dYmLUJFoFPXjhDTw9aO-MrmLx1BV2cjcg@mail.gmail.com
On Wed, May 31, 2023 at 9:14?AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
[...]
Toggle quote (8 lines)
> Do you mean use layered images by default without ability to build all
> in a single layer? Current layered implementation is slow to build
> because it needs to calculate a size of each layer, pack, and compress.
> So if user wants a faster build, a non-layered image is still an option.
>
> Regards,
> Oleg.

I am trying out your patch, and wanted to benchmark the runtime
difference between docker and docker-layered packs, but the latter
looks to be failing with any compression other than the default gzip.
In particular, I was looking to disable compression with
'--compression=none'.
O
O
Oleg Pykhalov wrote on 3 Jun 2023 21:10
Re: bug#62153: [PATCH 0/2] Add Docker layered image for pack and system
(name . Greg Hogan)(address . code@greghogan.com)(address . 62153@debbugs.gnu.org)
87y1l0icq0.fsf_-_@gmail.com
Greg Hogan <code@greghogan.com> writes:

Toggle quote (16 lines)
> On Wed, May 31, 2023 at 9:14?AM Oleg Pykhalov <go.wigust@gmail.com> wrote:
> [...]
>> Do you mean use layered images by default without ability to build all
>> in a single layer? Current layered implementation is slow to build
>> because it needs to calculate a size of each layer, pack, and compress.
>> So if user wants a faster build, a non-layered image is still an option.
>>
>> Regards,
>> Oleg.
>
> I am trying out your patch, and wanted to benchmark the runtime
> difference between docker and docker-layered packs, but the latter
> looks to be failing with any compression other than the default gzip.
> In particular, I was looking to disable compression with
> '--compression=none'.

I'll send a fixed v4 revision for '--compression=none'. Unfortunately,
because we cannot append to an existing compressed tarball:

tar: Cannot update compressed archives
Try 'tar --help' or 'tar --usage' for more information.

adding more compression types requires to write a handler for every
compressor separately in guix/docker.scm file:
Toggle snippet (10 lines)
(if layered-image?
(begin
(invoke "tar" "-rf" "image.tar" "config.json")
(if compressor
(begin
(apply invoke `(,@compressor "image.tar"))
(copy-file "image.tar.gz" image))
(copy-file "image.tar" image)))

I would like to vote that addional compressors could be added later if
needed.

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmR7kBgUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pxoKA//fUia6BxwXHIQNdFmQueeIVN7RUvr
gx19n9Qpe3rmkqRwdXqqk9V81daqSYtEwm9/vpvo2HO85yliUxkMUOiHRQLJxOPQ
X0Vx1v871DyXkdg4D9Jh1/GE6sjqcT8Jka5u8pFB5LAPVto12tqy+0+lMxP4AcqX
X6GxWuEClj6yB+nqtE/00xul6qDF1YaHAeOrMJjpYf4y1dwBYw9WFQwJJvVuXxdE
WukW6F9qTqj9qXrG0+EkamWQjWcmKrSkv7+aTskIK/xdEuXor2OSOC2o6Tff7K7p
GeePvEdDIGetiV5bNVU4EwewvJH/oF/FFZcPdvBx0DCUJ8AeVMEzwqnWExTW/UXe
pTqQH3dwTNhnh1tDhAAEKXhcBuOntmiCp5knArmCCirpzyW+c7rZ/F8JVC5vZJTf
PS3x9O+R3ZkueoimoQzSd/JGPSH1eh2O8SMNy42QB1w4j3SLEkv0ShsswCbkdMhk
OXy/wjqo9GJRRMRNTYAxeu7OLDOwM5RrhBX3s2g0Nu+6YEa3ISDU7s4qEBbQHNFW
lD0dplzXNZudhLab/xTP/ptgqsh7SoaYszsRitz+noT35DuaJt7wwm7Qp+rn55ve
PqMd+P94YCrzq8RslsUfpEE+VBJKxJbFnNTZdkaDqAI/HCwWSSe9gzag3zsYpQDx
8blQqUJbYRIXBiE=
=xQTj
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 3 Jun 2023 21:14
[PATCH v4 1/2] guix: docker: Build layered image.
(address . 62153@debbugs.gnu.org)
457c813653a44117e296deaa49e79fc701b90791.1685819700.git.go.wigust@gmail.com
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image Reference): Same.
(image-type Reference): Document docker-layered-image-type.
* gnu/image.scm
(validate-image-format)[docker-layered]: New image format.
* gnu/system/image.scm
(docker-layered-image, docker-layered-image-type): New variables.
(system-docker-image)[layered-image?]: New argument.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* tests/pack.scm: Add "docker-layered-image + localstatedir" test.
---
doc/guix.texi | 18 +++-
gnu/image.scm | 3 +-
gnu/system/image.scm | 76 +++++++++++----
gnu/tests/docker.scm | 20 +++-
guix/docker.scm | 208 +++++++++++++++++++++++++++++++---------
guix/scripts/pack.scm | 62 ++++++++++--
guix/scripts/system.scm | 11 ++-
tests/pack.scm | 48 ++++++++++
8 files changed, 369 insertions(+), 77 deletions(-)

Toggle diff (436 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 7f8d8d66e9..483be6ef16 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6984,9 +6984,15 @@ Invoking guix pack
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -44347,6 +44353,8 @@ image Reference
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
@@ -44682,6 +44690,10 @@ image-type Reference
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-syntax-rule (define-set-sanitizer name field set)
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..3a502f19ec 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -78,6 +79,7 @@ (define-module (gnu system image)
efi-disk-image
iso9660-image
docker-image
+ docker-layered-image
tarball-image
wsl2-image
raw-with-offset-disk-image
@@ -89,6 +91,7 @@ (define-module (gnu system image)
iso-image-type
uncompressed-iso-image-type
docker-image-type
+ docker-layered-image-type
tarball-image-type
wsl2-image-type
raw-with-offset-image-type
@@ -167,6 +170,10 @@ (define docker-image
(image-without-os
(format 'docker)))
+(define docker-layered-image
+ (image-without-os
+ (format 'docker-layered)))
+
(define tarball-image
(image-without-os
(format 'tarball)))
@@ -237,6 +244,11 @@ (define docker-image-type
(name 'docker)
(constructor (cut image-with-os docker-image <>))))
+(define docker-layered-image-type
+ (image-type
+ (name 'docker-layered)
+ (constructor (cut image-with-os docker-layered-image <>))))
+
(define tarball-image-type
(image-type
(name 'tarball)
@@ -633,9 +645,12 @@ (define (image-with-label base-image label)
(define* (system-docker-image image
#:key
- (name "docker-image"))
+ (name "docker-image")
+ (archiver tar)
+ layered-image?)
"Build a docker image for IMAGE. NAME is the base name to use for the
-output file."
+output file. If LAYERED-IMAGE? is true, the image will with many of the store
+paths being on their own layer to improve sharing between images."
(define boot-program
;; Program that runs the boot script of OS, which in turn starts shepherd.
(program-file "boot-program"
@@ -678,9 +693,11 @@ (define* (system-docker-image image
(use-modules (guix docker)
(guix build utils)
(gnu build image)
+ (srfi srfi-1)
(srfi srfi-19)
(guix build store-copy)
- (guix store database))
+ (guix store database)
+ (ice-9 receive))
;; Set the SQL schema location.
(sql-schema #$schema)
@@ -700,18 +717,31 @@ (define* (system-docker-image image
#:register-closures? #$register-closures?
#:deduplicate? #f
#:system-directory #$os)
- (build-docker-image
- #$output
- (cons* image-root
- (map store-info-item
- (call-with-input-file #$graph
- read-reference-graph)))
- #$os
- #:entry-point '(#$boot-program #$os)
- #:compressor '(#+(file-append gzip "/bin/gzip") "-9n")
- #:creation-time (make-time time-utc 0 1)
- #:system #$image-target
- #:transformations `((,image-root -> ""))))))))
+ (when #$layered-image?
+ (setenv "PATH"
+ (string-join (list #+(file-append archiver "/bin")
+ #+(file-append coreutils "/bin")
+ #+(file-append gzip "/bin"))
+ ":")))
+ (apply build-docker-image
+ (append (list #$output
+ (append (if #$layered-image?
+ '()
+ (list image-root))
+ (map store-info-item
+ (call-with-input-file #$graph
+ read-reference-graph)))
+ #$os
+ #:entry-point '(#$boot-program #$os)
+ #:compressor
+ '(#+(file-append gzip "/bin/gzip") "-9n")
+ #:creation-time (make-time time-utc 0 1)
+ #:system #$image-target
+ #:transformations `((,image-root -> "")))
+ (if #$layered-image?
+ (list #:root-system image-root
+ #:layered-image? #$layered-image?)
+ '()))))))))
(computed-file name builder
;; Allow offloading so that this I/O-intensive process
@@ -720,6 +750,18 @@ (define* (system-docker-image image
#:options `(#:references-graphs ((,graph ,os))
#:substitutable? ,substitutable?))))
+(define* (system-docker-layered-image image
+ #:key
+ (name "docker-image")
+ (archiver tar)
+ (layered-image? #t))
+ "Build a docker image for IMAGE. NAME is the base name to use for the
+output file."
+ (system-docker-image image
+ #:name name
+ #:archiver archiver
+ #:layered-image? layered-image?))
+
;;;
;;; Tarball image.
@@ -811,7 +853,7 @@ (define (image->root-file-system image)
"Return the IMAGE root partition file-system type."
(case (image-format image)
((iso9660) "iso9660")
- ((docker tarball wsl2) "dummy")
+ ((docker docker-layered tarball wsl2) "dummy")
(else
(partition-file-system (find-root-partition image)))))
@@ -948,6 +990,8 @@ (define* (system-image image)
("bootcfg" ,bootcfg))))
((memq image-format '(docker))
(system-docker-image image*))
+ ((memq image-format '(docker-layered))
+ (system-docker-layered-image image*))
((memq image-format '(tarball))
(system-tarball-image image*))
((memq image-format '(wsl2))
diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm
index edc9804414..0cccc02ad2 100644
--- a/gnu/tests/docker.scm
+++ b/gnu/tests/docker.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2019 Danny Milosavljevic <dannym@scratchpost.org>
;;; Copyright © 2019-2023 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -43,7 +44,8 @@ (define-module (gnu tests docker)
#:use-module (guix build-system trivial)
#:use-module ((guix licenses) #:prefix license:)
#:export (%test-docker
- %test-docker-system))
+ %test-docker-system
+ %test-docker-layered-system))
(define %docker-os
(simple-operating-system
@@ -316,3 +318,19 @@ (define %test-docker-system
(locale-libcs (list glibc)))
#:type docker-image-type)))
run-docker-system-test)))))
+
+(define %test-docker-layered-system
+ (system-test
+ (name "docker-layered-system")
+ (description "Run a system image as produced by @command{guix system
+docker-layered-image} inside Docker.")
+ (value (with-monad %store-monad
+ (>>= (lower-object
+ (system-image (os->image
+ (operating-system
+ (inherit (simple-operating-system))
+ ;; Use locales for a single libc to
+ ;; reduce space requirements.
+ (locale-libcs (list glibc)))
+ #:type docker-layered-image-type)))
+ run-docker-system-test)))))
diff --git a/guix/docker.scm b/guix/docker.scm
index 5e6460f43f..b40cfb2374 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -28,6 +29,8 @@ (define-module (guix docker)
delete-file-recursively
with-directory-excursion
invoke))
+ #:use-module (guix diagnostics)
+ #:use-module (guix i18n)
#:use-module (gnu build install)
#:use-module (json) ;guile-json
#:use-module (srfi srfi-1)
@@ -38,6 +41,9 @@ (define-module (guix docker)
#:use-module (rnrs bytevectors)
#:use-module (ice-9 ftw)
#:use-module (ice-9 match)
+ #:use-module (ice-9 popen)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 receive)
#:export (build-docker-image))
;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image.
@@ -92,12 +98,12 @@ (define (canonicalize-repository-name name)
(make-string (- min-length l) padding-character)))
(_ normalized-name))))
-(define* (manifest path id #:optional (tag "guix"))
+(define* (manifest path layers #:optional (tag "guix"))
"Generate a simple image manifest."
(let ((tag (canonicalize-repository-name tag)))
`#(((Config . "config.json")
(RepoTags . #(,(string-append tag ":latest")))
- (Layers . #(,(string-append id "/layer.tar")))))))
+ (Layers . ,(list->vector layers))))))
;; According to the specifications this is required for backwards
;; compatibility. It duplicates information provided by the manifest.
@@ -106,8 +112,8 @@ (define* (repositories path id #:optional (tag "guix"))
`((,(canonicalize-repository-name tag) . ((latest . ,id)))))
;; See https://github.com/opencontainers/image-spec/blob/master/config.md
-(define* (config layer time arch #:key entry-point (environment '()))
- "Generate a minimal image configuration for the given LAYER file."
+(define* (config layers-diff-ids time arch #:key entry-point (environment '()))
+ "Generate a minimal image configuration for the given LAYERS files."
;; "architecture" must be values matching "platform.arch" in the
;; runtime-spec at
;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform
@@ -125,7 +131,7 @@ (define* (config layer time arch #:key entry-point (environment '()))
(container_config . #nil)
(os . "linux")
(rootfs . ((type . "layers")
- (diff_ids . #(,(layer-diff-id layer)))))))
+ (diff_ids . ,(list->vector layers-diff-ids))))))
(define directive-file
;; Return the file or directory created by a 'evaluate-populate-directive'
@@ -136,6 +142,37 @@ (define directive-file
(('directory name _ ...)
(string-trim name #\/))))
+(define %docker-image-max-layers
+ 100)
+
+(define (paths-split-sort paths)
+ "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage."
+ (let* ((paths-length (length paths))
+ (port (apply open-pipe* OPEN_READ
+ (append '("du" "--summarize") paths)))
+ (output (read-string port)))
+ (close-port port)
+ (receive (head tail)
+ (split-at
+ (map (match-lambda ((size . path) path))
+ (sort (map (lambda (line)
+ (match (string-split line #\tab)
+ ((size path)
+ (cons (string->number size) path))))
+ (string-split
+ (string-trim-right output #\newline)
+ #\newline))
+ (lambda (path1 path2)
+ (< (match path2 ((size . _) size))
+ (match path1 ((size . _) size))))))
+ (if (>= paths-length %docker-image-max-layers)
+ (- %docker-image-max-layers 2)
+ (1- paths-length)))
+ (list head tail))))
+
+(define (create-empty-tar file)
+ (invoke "tar" "-cf" file "--files-from" "/dev/null"))
+
(define* (build-docker-image image paths prefix
#:key
(repository "guix")
@@ -146,11 +183,13 @@ (define* (build-docker-image image paths prefix
entry-point
(environment '())
compressor
- (creation-time (current-time time-utc)))
- "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX
-must be a store path that is a prefix of any store paths in PATHS. REPOSITORY
-is a descriptive name that will show up in \"REPOSITORY\" column of the output
-of \"docker images\".
+ (creation-time (current-time time-utc))
+ layered-image?
+ root-system)
+ "Write to IMAGE a layerer Docker image archive containing the given PATHS.
+PREFIX must be a store path that is a prefix of any store paths in PATHS.
+REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of
+the output of \"docker images\".
When DATABASE is true, copy it to /var/guix/db in the image and create
/var/guix/gcroots and friends.
@@ -172,7 +211,14 @@ (define* (build-docker-image image paths prefix
SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in
PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a
command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a
-SRFI-19 time-utc object, as the creation time in metadata."
+SRFI-19 time-utc object, as the creation time in metadata.
+
+When LAYERED-IMAGE? is true build layered image, providing a Docker
+image with many of the store paths being on their own layer to improve sharing
+between images.
+
+ROOT-SYSTEM is a directory with a provisioned root file system, which will be
+added to image as a layer."
(define (sanitize path-fragment)
(escape-special-chars
;; GNU tar strips the leading slash off of absolute paths before applying
@@ -203,6 +249,53 @@ (define* (build-docker-image image paths prefix
(if (eq? '() transformations)
'()
`("--transform" ,(transformations->expression transformations))))
+ (define layers-hashes
+ (match-lambda
+ (((head ...) (tail ...) id)
+ (create-empty-tar "image.tar")
+ (let* ((head-layers
+ (map
+ (lambda (file)
+ (invoke "tar" "cf" "layer.tar" file)
+ (let* ((file-hash (layer-diff-id "layer.tar"))
+ (file-name (string-append file-hash "/layer.tar")))
+ (mkdir file-hash)
+ (rename-file "layer.tar" file-name)
+ (invoke "tar" "-rf" "image.tar" file-name)
+ (delete-file file-name)
+ file-hash))
+ head))
+ (tail-layer
+ (begin
+ (create-empty-tar "layer.tar")
+ (for-each
This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 3 Jun 2023 21:16
[PATCH v4] news: Add entry for the new 'docker-layered' distribution format.
(address . 62153@debbugs.gnu.org)
270e247ee913f6ede40883919d3f6971fe1b01aa.1685819787.git.go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

Toggle diff (81 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 314f0ab352..158a9284b0 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021, 2023 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,63 @@
(channel-news
(version 0)
+ (entry (commit "457c813653a44117e296deaa49e79fc701b90791")
+ (title
+ (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "????? @samp{docker-layered} ?????? ??? @command{guix pack} ???????"))
+ (body
+ (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
+@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
+Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
+untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
+im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
+Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
+angelegt wird:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
+indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
+übergeben.
+
+Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
+@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")
+ (en "Docker layered images can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+The @command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+
+See @command{info \"(guix) Invoking guix pack\"} and
+@command{info \"(guix) System Images\"} for more information.")
+ (ru "????????? ??????? ???????? ???????????? Docker ??????? ? ???????
+@command{guix pack --format=docker-layered}, ??????? ??????? Docker ????? ?
+?????? ? store ?????????????? ?? ????????? ?????, ??????? ????? ???????
+???????? ???????. ????? ????? ?????? ? GNU store ? ???????? gzipped tarball.
+
+?????? ???????? Docker layered ????? ? @code{hello} ???????:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} ?????? ????? ????????? layered Docker ????? ?????
+???????? ? ????? @option{--image-type} ????????? @code{docker-layered}.
+
+???????? @command{info \"(guix) Invoking guix pack\"} ?
+@command{info \"(guix) System Images\"} ??? ????????? ????? ?????????
+????????.")))
+
(entry (commit "ba5da5125a81307500982517e2f458d57b024668")
(title
(en "New @code{arguments} rule for @command{guix style}")

base-commit: 66c9b82fed3c59ee07187898592c688c82fed273
prerequisite-patch-id: 9c90b67b3c2bb18d7fd17d083b0ab0d1cd5333cd
--
2.38.0
O
O
Oleg Pykhalov wrote on 27 Aug 2023 05:16
Merging guix pack changes for Docker containers packaging
(name . guix-devel)(address . guix-devel@gnu.org)
878r9xb2e6.fsf@gmail.com
Hi Guix,

I would like to merge 62153. After 64173 will be merge, merging 62153
is not possible without conflict resolving with Git.

64173 introduces ‘%docker-format-options’ variable. With this variable
it's possible in 62153 to replace ‘--image-type=docker-layered’ with
‘--docker-layers=N’ option, where:

if ‘N’ is zero, then use current non layered format
if ‘N’ is bigger than zero, then use layered format

Number of layers specification is nice to have, because Docker layers
are limited. So if user would like to modify a Docker image by adding
more layers on top, then hacks like squashing layers are not required.
Also, it will be possible to delete code which builds non layered Docker
image without deprecating command line options.

Is it possible to partially merge 64173, specifically
‘%docker-format-options’ variable and it requirements, so it can be used
in 62153 for ‘--docker-layers=N’ option?



Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmTqwCEUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pwLNg//XqUbjKOGD1st7X1i7XAZKAwkPN/j
yOzVa1agcRi2bIwLqoh+/WWLf4afG0tE7b52VQKUbgSMdSfJi2RJoLs7PmTjhhhJ
/QKl35ffdDr4C6MgutK/nir/Dh2wYuZ4KtKdmj6VmXkC5uSd+GA4mNriwZGUf5IX
Oo2YqvEv9e9wkilRRX7BN/iVboKPlPWHxNT82x/gB2UTdgXxcICqUHa9L/h+4obl
7xrsBiGtIJMWmC0A6bZXscDrWTxlKdqAXZTkGRAh2XJ1EWdZeh6ED3s1n4lZySsE
gapcNf8112Th0+TRhSV3FP7OUc5LHK8nvosNCnr3WtLMEOG/TGeTx5POc6YwUVwA
oJQQu9Z9k8GCQKD58Uo4Bq7f8uhbq+iVfChESn0+ezNeJagir63CAD9eqegaQI1f
e2EVxEC3sjEe264ue4fW3L7zx5SkHTHehv8mVfDKAJJTmNgCXx1t+jjP51T1vmxg
n3wzP9hU4Ugg10r+whWC2BuH6k2B+wS/bJ1mkg/qSqHAB9/3wvhVikSSFFBmpI+s
FmRkjTDguLW9rfZQW8Q9EJtAqOF74FRt4JXkGZffOClTdLrYNuf958wN6r4uuW00
MTBRKy753fJrU7PlwvPi7SQEe6ziaCY7w+tvYOKSV8DPPShMZBhuYnrzRM4IvuDB
5Xgx6marsPHb7AA=
=VfnU
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 22 Dec 2023 23:10
Re: [bug#62153] [PATCH v4 1/2] guix: docker: Build layered image.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
878r5l99df.fsf@gnu.org
Oleg Pykhalov <go.wigust@gmail.com> skribis:

Toggle quote (25 lines)
> * doc/guix.texi (Invoking guix pack): Document docker-layered format.
> (image Reference): Same.
> (image-type Reference): Document docker-layered-image-type.
> * gnu/image.scm
> (validate-image-format)[docker-layered]: New image format.
> * gnu/system/image.scm
> (docker-layered-image, docker-layered-image-type): New variables.
> (system-docker-image)[layered-image?]: New argument.
> (system-docker-layered-image): New procedure.
> (image->root-file-system)[docker-layered]: New image format.
> * gnu/tests/docker.scm (%test-docker-layered-system): New test.
> * guix/docker.scm (%docker-image-max-layers): New variable.
> (build-docker-image)[stream-layered-image, root-system]: New arguments.
> * guix/scripts/pack.scm (stream-layered-image.py): New variable.
> (docker-image)[layered-image?]: New argument.
> (docker-layered-image): New procedure.
> (%formats)[docker-layered]: New format.
> (show-formats): Document this.
> * guix/scripts/system.scm
> (system-derivation-for-action)[docker-layered-image]: New action.
> (show-help): Document this.
> (actions)[docker-layered-image]: New action.
> (process-action): Add this.
> * tests/pack.scm: Add "docker-layered-image + localstatedir" test.

[...]

Toggle quote (3 lines)
> + #:use-module (guix diagnostics)
> + #:use-module (guix i18n)

(guix docker) shouldn’t need these.

Toggle quote (4 lines)
> + #:use-module (ice-9 popen)
> + #:use-module (ice-9 rdelim)
> + #:use-module (ice-9 receive)

For consistency, I’d recommend (srfi srfi-71) instead of (ice-9
receive).

Toggle quote (3 lines)
> +(define %docker-image-max-layers
> + 100)

I’d add a comment on the second line, like “;; Maximum number of layers
allowed in a Docker image according to …”.

Toggle quote (3 lines)
> +(define (paths-split-sort paths)
> + "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage."

Nitpick: maybe (define (size-sorted-store-items items) …)?

Toggle quote (6 lines)
> + (let* ((paths-length (length paths))
> + (port (apply open-pipe* OPEN_READ
> + (append '("du" "--summarize") paths)))
> + (output (read-string port)))
> + (close-port port)

How about:

(map (lambda (item)
(cons item (file-size item)))
items)

?

See (guix build store-copy) for the definition of ‘file-size’.

That way we avoid the dependency on Coreutils and code that “parses” the
output of ‘du’.

Toggle quote (2 lines)
> + (define layers-hashes

A short comment explaining what the inputs and outputs of this procedure
are would be great.

Toggle quote (39 lines)
> + (match-lambda
> + (((head ...) (tail ...) id)
> + (create-empty-tar "image.tar")
> + (let* ((head-layers
> + (map
> + (lambda (file)
> + (invoke "tar" "cf" "layer.tar" file)
> + (let* ((file-hash (layer-diff-id "layer.tar"))
> + (file-name (string-append file-hash "/layer.tar")))
> + (mkdir file-hash)
> + (rename-file "layer.tar" file-name)
> + (invoke "tar" "-rf" "image.tar" file-name)
> + (delete-file file-name)
> + file-hash))
> + head))
> + (tail-layer
> + (begin
> + (create-empty-tar "layer.tar")
> + (for-each (lambda (file)
> + (invoke "tar" "-rf" "layer.tar" file))
> + tail)
> + (let* ((file-hash (layer-diff-id "layer.tar"))
> + (file-name (string-append file-hash "/layer.tar")))
> + (mkdir file-hash)
> + (rename-file "layer.tar" file-name)
> + (invoke "tar" "-rf" "image.tar" file-name)
> + (delete-file file-name)
> + file-hash)))
> + (customization-layer
> + (let* ((file-id (string-append id "/layer.tar"))
> + (file-hash (layer-diff-id file-id))
> + (file-name (string-append file-hash "/layer.tar")))
> + (mkdir file-hash)
> + (rename-file file-id file-name)
> + (invoke "tar" "-rf" "image.tar" file-name)
> + file-hash))
> + (all-layers
> + (append head-layers (list tail-layer customization-layer))))

Maybe this can be factorized a bit with:

(define (seal-layer)
;; Add 'layer.tar' to 'image.tar' under the right name. Return its hash.
(let* ((file-hash (layer-diff-id "layer.tar"))
(file-name (string-append file-hash "/layer.tar")))
(mkdir file-hash)
(rename-file "layer.tar" file-name)
(invoke "tar" "-rf" "image.tar" file-name)
(delete-file file-name)
file-hash)))

?

Apart from this stylistic issues, it looks great to me.

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 22 Dec 2023 23:11
Re: [bug#62153] Merging guix pack changes for Docker containers packaging
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
875y0p99c4.fsf@gnu.org
Hi Oleg,

Apologies for not replying earlier. I occasionally get reminded of the
fact that building single-layer images is a problem, but only now did I
take the time to look more closely at the latest version of these
patches.

Oleg Pykhalov <go.wigust@gmail.com> skribis:

Toggle quote (10 lines)
> I would like to merge 62153. After 64173 will be merge, merging 62153
> is not possible without conflict resolving with Git.
>
> 64173 introduces ‘%docker-format-options’ variable. With this variable
> it's possible in 62153 to replace ‘--image-type=docker-layered’ with
> ‘--docker-layers=N’ option, where:
>
> if ‘N’ is zero, then use current non layered format
> if ‘N’ is bigger than zero, then use layered format

OK we should do that. However, the original submitter of #64173
apparently dropped the ball as we were approaching the final version.

Would you like to adopt it and submit/push a version that incorporates
the latest comments?

Alternatively, we could do the opposite: merge the Docker layer patches
first, and then rebase the ‘%docker-format-options’ patch, after which
we could add the ‘--docker-layers’ option.

What’s your preference?

Toggle quote (6 lines)
> Number of layers specification is nice to have, because Docker layers
> are limited. So if user would like to modify a Docker image by adding
> more layers on top, then hacks like squashing layers are not required.
> Also, it will be possible to delete code which builds non layered Docker
> image without deprecating command line options.

Agreed.

Anyway, apart from the stylistic issues I reported, v4 of this patch set
looks great to me. (For clarity I’d have preferred three patches, one
for (guix docker), one for ‘guix pack’, and one for ‘guix system’; but
it’s really a detail, let’s not block this patch series any longer.)

Thanks,
Ludo’.
O
O
Oleg Pykhalov wrote on 26 Dec 2023 03:15
[PATCH v5 0/5] Add Docker layered image for pack and system
(address . 62153@debbugs.gnu.org)
cover.1703556298.git.go.wigust@gmail.com
This patch series applies 64173 and for 62153 replaces 'docker-layered-image'
format with '--max-layers=N' option for 'guix pack' and 'guix system image'.

Graham James Addis (1):
guix: pack: Add '--entry-point-argument' option.

Oleg Pykhalov (4):
tests: docker-system: Increase image size.
guix: docker: Build layered images.
guix: pack: Build layered images.
scripts: system: Build layered images.

doc/guix.texi | 40 +++++++-
gnu/image.scm | 4 +
gnu/system/image.scm | 41 +++++---
gnu/tests/docker.scm | 2 +-
guix/docker.scm | 212 +++++++++++++++++++++++++++++++---------
guix/scripts/pack.scm | 80 ++++++++++++---
guix/scripts/system.scm | 28 +++++-
tests/pack.scm | 49 ++++++++++
8 files changed, 379 insertions(+), 77 deletions(-)


base-commit: a4a14ab6d79f6f1f926a82dd50db4655232042b7
--
2.41.0
O
O
Oleg Pykhalov wrote on 26 Dec 2023 03:18
[PATCH 1/5] guix: pack: Add '--entry-point-argument' option.
(address . 62153@debbugs.gnu.org)
9e2ecba897127f31c8c6fedd9ce20b6cdc0be4a2.1703556298.git.go.wigust@gmail.com
From: Graham James Addis <grahamjamesaddis@gmail.com>

* guix/scripts/pack.scm:
(entry-point-argument-spec-option-parser): New procedure.
(docker-image, %default-options, %docker-format-options,
show-docker-format-options/detailed, %options, show-docker-format-options,
guix-pack): Handle '--entry-point-argument' option.
* doc/guix.texi: (Invoking guix pack): Document this

Signed-off-by: Oleg Pykhalov <go.wigust@gmail.com>
Change-Id: I1124feff6af39dcc63c85fd6cc7ad50f398489dc
---
doc/guix.texi | 14 +++++++++++-
guix/scripts/pack.scm | 50 +++++++++++++++++++++++++++++++++++--------
2 files changed, 54 insertions(+), 10 deletions(-)

Toggle diff (154 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index 76b4eae67f..cca250dc31 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -122,6 +122,7 @@
Copyright @copyright{} 2023 Foundation Devices, Inc.@*
Copyright @copyright{} 2023 Thomas Ieong@*
Copyright @copyright{} 2023 Saku Laesvuori@*
+Copyright @copyright{} 2023 Graham James Addis@*
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -7406,7 +7407,7 @@ Invoking guix pack
@env{GUIX_EXECUTION_ENGINE} environment variable accordingly.
@end quotation
-@cindex entry point, for Docker images
+@cindex entry point, for Docker and Singularity images
@item --entry-point=@var{command}
Use @var{command} as the @dfn{entry point} of the resulting pack, if the pack
format supports it---currently @code{docker} and @code{squashfs} (Singularity)
@@ -7429,6 +7430,17 @@ Invoking guix pack
docker run @var{image-id}
@end example
+@cindex entry point arguments, for docker images
+@item --entry-point-argument=@var{command}
+@itemx -A @var{command}
+Use @var{command} as an argument to @dfn{entry point} of the resulting pack.
+This option is only valid in conjunction with @code{--entry-point} and can
+appear multiple times on the command line.
+
+@example
+guix pack -f docker --entry-point=bin/guile --entry-point-argument="--help" guile
+@end example
+
@item --expression=@var{expr}
@itemx -e @var{expr}
Consider the package @var{expr} evaluates to.
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 8071840de1..4c0a602eb1 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2020, 2021, 2022, 2023 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;; Copyright © 2020 Eric Bavier <bavier@posteo.net>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Graham James Addis <graham@addis.org.uk>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -202,6 +203,16 @@ (define (symlink-spec-option-parser opt name arg result)
(leave (G_ "~a: invalid symlink specification~%")
arg))))
+(define (entry-point-argument-spec-option-parser opt name arg result)
+ "A SRFI-37 opion parser for the --entry-point-argument option. The spec
+takes multiple occurances. The entries are used in the exec form for the
+docker entry-point. The values are used as parameters in conjunction with
+the --entry-point option which is used as the first value in the exec form."
+ (let ((entry-point-argument (assoc-ref result 'entry-point-argument)))
+ (alist-cons 'entry-point-argument
+ (append entry-point-argument (list arg))
+ (alist-delete 'entry-point-argument result eq?))))
+
(define (set-utf8-locale profile)
"Configure the environment to use the \"en_US.utf8\" locale provided by the
GLIBC-UT8-LOCALES package."
@@ -562,10 +573,22 @@ (define* (docker-image name profile
`((directory "/tmp" ,(getuid) ,(getgid) #o1777)
,@(append-map symlink->directives '#$symlinks)))
+ (define (form-entry-point prefix entry-point entry-point-argument)
+ ;; Construct entry-point parameter for build-docker-image. The
+ ;; first entry is constructed by prefixing the entry-point with
+ ;; the supplied index subsequent entries are taken from the
+ ;; --entry-point-argument options.
+ (and=> entry-point
+ (lambda (entry-point)
+ (cons* (string-append prefix "/" entry-point)
+ entry-point-argument))))
+
(setenv "PATH" #+(file-append archiver "/bin"))
(let-keywords '#$extra-options #f
- ((image-tag #f))
+ ((image-tag #f)
+ (entry-point-argument #f))
+
(build-docker-image #$output
(map store-info-item
(call-with-input-file "profile"
@@ -578,11 +601,10 @@ (define* (docker-image name profile
#:database #+database
#:system (or #$target %host-type)
#:environment environment
- #:entry-point
- #$(and entry-point
- #~(list
- (string-append #$profile "/"
- #$entry-point)))
+ #:entry-point (form-entry-point
+ #$profile
+ #$entry-point
+ entry-point-argument)
#:extra-files directives
#:compressor
#+(compressor-command compressor)
@@ -1264,6 +1286,7 @@ (define %default-options
(debug . 0)
(verbosity . 1)
(symlinks . ())
+ (entry-point-argument . ())
(compressor . ,(first %compressors))))
(define %formats
@@ -1299,7 +1322,9 @@ (define (required-option symbol)
rest))))
(define %docker-format-options
- (list (required-option 'image-tag)))
+ (list (required-option 'image-tag)
+ (option '(#\A "entry-point-argument") #t #f
+ entry-point-argument-spec-option-parser)))
(define (show-docker-format-options)
(display (G_ "
@@ -1308,7 +1333,12 @@ (define (show-docker-format-options)
(define (show-docker-format-options/detailed)
(display (G_ "
--image-tag=NAME
- Use the given NAME for the Docker image repository"))
+ Use the given NAME for the Docker image repository
+
+ -A, --entry-point-argument=COMMAND/PARAMETER
+ Value(s) to use for the Docker EntryPoint arguments.
+ Multiple instances are accepted. This is only valid
+ in conjunction with the --entry-point option"))
(newline)
(exit 0))
@@ -1619,7 +1649,9 @@ (define-command (guix-pack . args)
(extra-options (match pack-format
('docker
(list #:image-tag
- (assoc-ref opts 'image-tag)))
+ (assoc-ref opts 'image-tag)
+ #:entry-point-argument
+ (assoc-ref opts 'entry-point-argument)))
('deb
(list #:control-file
(process-file-arg opts 'control-file)
--
2.41.0
O
O
Oleg Pykhalov wrote on 26 Dec 2023 03:18
[PATCH 2/5] tests: docker-system: Increase image size.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
6baac8dbf5f43a2e8458b592723463b205ba57f3.1703556298.git.go.wigust@gmail.com
* gnu/tests/docker.scm (run-docker-system-test)[vm]: Increase
'disk-image-size'.

Change-Id: If88588d8981efdfdc539460900f1cbb9a663f9cb
---
gnu/tests/docker.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (15 lines)
diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm
index edc9804414..9e9d2e2d07 100644
--- a/gnu/tests/docker.scm
+++ b/gnu/tests/docker.scm
@@ -212,7 +212,7 @@ (define (run-docker-system-test tarball)
(virtual-machine
(operating-system os)
(volatile? #f)
- (disk-image-size (* 5500 (expt 2 20)))
+ (disk-image-size (* 6000 (expt 2 20)))
(memory-size 2048)
(port-forwardings '())))
--
2.41.0
O
O
Oleg Pykhalov wrote on 26 Dec 2023 03:18
[PATCH 3/5] guix: docker: Build layered images.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
49f8906ba06af461e17d9badcbbf3967f1a8be3b.1703556298.git.go.wigust@gmail.com
* guix/docker.scm (%docker-image-max-layers): New variable.
(size-sorted-store-items, create-empty-tar): New procedures.
(config, manifest, build-docker-image): Build layered images.

Change-Id: I4c8846bff0a3ceccb77e6bdf95d4942e5c3efe41
---
guix/docker.scm | 212 +++++++++++++++++++++++++++++++++++++-----------
1 file changed, 166 insertions(+), 46 deletions(-)

Toggle diff (312 lines)
diff --git a/guix/docker.scm b/guix/docker.scm
index 5e6460f43f..5deca2afdb 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -3,6 +3,7 @@
;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -29,16 +30,27 @@ (define-module (guix docker)
with-directory-excursion
invoke))
#:use-module (gnu build install)
+ #:use-module ((guix build store-copy)
+ #:select (file-size))
#:use-module (json) ;guile-json
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-19)
#:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-71)
#:use-module ((texinfo string-utils)
#:select (escape-special-chars))
#:use-module (rnrs bytevectors)
#:use-module (ice-9 ftw)
#:use-module (ice-9 match)
- #:export (build-docker-image))
+ #:export (%docker-image-max-layers
+ build-docker-image))
+
+;; The maximum number of layers allowed in a Docker image is typically around
+;; 128, although it may vary depending on the Docker daemon. However, we
+;; recommend setting the limit to 100 to ensure sufficient room for future
+;; extensions.
+(define %docker-image-max-layers
+ #f)
;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image.
(define docker-id
@@ -92,12 +104,12 @@ (define (canonicalize-repository-name name)
(make-string (- min-length l) padding-character)))
(_ normalized-name))))
-(define* (manifest path id #:optional (tag "guix"))
+(define* (manifest path layers #:optional (tag "guix"))
"Generate a simple image manifest."
(let ((tag (canonicalize-repository-name tag)))
`#(((Config . "config.json")
(RepoTags . #(,(string-append tag ":latest")))
- (Layers . #(,(string-append id "/layer.tar")))))))
+ (Layers . ,(list->vector layers))))))
;; According to the specifications this is required for backwards
;; compatibility. It duplicates information provided by the manifest.
@@ -106,8 +118,8 @@ (define* (repositories path id #:optional (tag "guix"))
`((,(canonicalize-repository-name tag) . ((latest . ,id)))))
;; See https://github.com/opencontainers/image-spec/blob/master/config.md
-(define* (config layer time arch #:key entry-point (environment '()))
- "Generate a minimal image configuration for the given LAYER file."
+(define* (config layers-diff-ids time arch #:key entry-point (environment '()))
+ "Generate a minimal image configuration for the given LAYERS files."
;; "architecture" must be values matching "platform.arch" in the
;; runtime-spec at
;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform
@@ -125,7 +137,7 @@ (define* (config layer time arch #:key entry-point (environment '()))
(container_config . #nil)
(os . "linux")
(rootfs . ((type . "layers")
- (diff_ids . #(,(layer-diff-id layer)))))))
+ (diff_ids . ,(list->vector layers-diff-ids))))))
(define directive-file
;; Return the file or directory created by a 'evaluate-populate-directive'
@@ -136,6 +148,26 @@ (define directive-file
(('directory name _ ...)
(string-trim name #\/))))
+(define (size-sorted-store-items items max-layers)
+ "Split list of ITEMS at %MAX-LAYERS and sort by disk usage."
+ (let* ((items-length (length items))
+ (head tail
+ (split-at
+ (map (match-lambda ((size . item) item))
+ (sort (map (lambda (item)
+ (cons (file-size item) item))
+ items)
+ (lambda (item1 item2)
+ (< (match item2 ((size . _) size))
+ (match item1 ((size . _) size))))))
+ (if (>= items-length max-layers)
+ (- max-layers 2)
+ (1- items-length)))))
+ (list head tail)))
+
+(define (create-empty-tar file)
+ (invoke "tar" "-cf" file "--files-from" "/dev/null"))
+
(define* (build-docker-image image paths prefix
#:key
(repository "guix")
@@ -146,11 +178,13 @@ (define* (build-docker-image image paths prefix
entry-point
(environment '())
compressor
- (creation-time (current-time time-utc)))
- "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX
-must be a store path that is a prefix of any store paths in PATHS. REPOSITORY
-is a descriptive name that will show up in \"REPOSITORY\" column of the output
-of \"docker images\".
+ (creation-time (current-time time-utc))
+ max-layers
+ root-system)
+ "Write to IMAGE a layerer Docker image archive containing the given PATHS.
+PREFIX must be a store path that is a prefix of any store paths in PATHS.
+REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of
+the output of \"docker images\".
When DATABASE is true, copy it to /var/guix/db in the image and create
/var/guix/gcroots and friends.
@@ -172,7 +206,14 @@ (define* (build-docker-image image paths prefix
SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in
PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a
command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a
-SRFI-19 time-utc object, as the creation time in metadata."
+SRFI-19 time-utc object, as the creation time in metadata.
+
+When MAX-LAYERS is not false build layered image, providing a Docker
+image with many of the store paths being on their own layer to improve sharing
+between images.
+
+ROOT-SYSTEM is a directory with a provisioned root file system, which will be
+added to image as a layer."
(define (sanitize path-fragment)
(escape-special-chars
;; GNU tar strips the leading slash off of absolute paths before applying
@@ -203,6 +244,59 @@ (define* (build-docker-image image paths prefix
(if (eq? '() transformations)
'()
`("--transform" ,(transformations->expression transformations))))
+ (define (seal-layer)
+ ;; Add 'layer.tar' to 'image.tar' under the right name. Return its hash.
+ (let* ((file-hash (layer-diff-id "layer.tar"))
+ (file-name (string-append file-hash "/layer.tar")))
+ (mkdir file-hash)
+ (rename-file "layer.tar" file-name)
+ (invoke "tar" "-rf" "image.tar" file-name)
+ (delete-file file-name)
+ file-hash))
+ (define layers-hashes
+ ;; Generate a tarball that includes container image layers as tarballs,
+ ;; along with a manifest.json file describing the layer and config file
+ ;; locations.
+ (match-lambda
+ (((head ...) (tail ...) id)
+ (create-empty-tar "image.tar")
+ (let* ((head-layers
+ (map
+ (lambda (file)
+ (invoke "tar" "cf" "layer.tar" file)
+ (seal-layer))
+ head))
+ (tail-layer
+ (begin
+ (create-empty-tar "layer.tar")
+ (for-each (lambda (file)
+ (invoke "tar" "-rf" "layer.tar" file))
+ tail)
+ (let* ((file-hash (layer-diff-id "layer.tar"))
+ (file-name (string-append file-hash "/layer.tar")))
+ (mkdir file-hash)
+ (rename-file "layer.tar" file-name)
+ (invoke "tar" "-rf" "image.tar" file-name)
+ (delete-file file-name)
+ file-hash)))
+ (customization-layer
+ (let* ((file-id (string-append id "/layer.tar"))
+ (file-hash (layer-diff-id file-id))
+ (file-name (string-append file-hash "/layer.tar")))
+ (mkdir file-hash)
+ (rename-file file-id file-name)
+ (invoke "tar" "-rf" "image.tar" file-name)
+ file-hash))
+ (all-layers
+ (append head-layers (list tail-layer customization-layer))))
+ (with-output-to-file "manifest.json"
+ (lambda ()
+ (scm->json (manifest prefix
+ (map (cut string-append <> "/layer.tar")
+ all-layers)
+ repository))))
+ (invoke "tar" "-rf" "image.tar" "manifest.json")
+ all-layers))))
(let* ((directory "/tmp/docker-image") ;temporary working directory
(id (docker-id prefix))
(time (date->string (time-utc->date creation-time) "~4"))
@@ -229,26 +323,39 @@ (define* (build-docker-image image paths prefix
(with-output-to-file "json"
(lambda () (scm->json (image-description id time))))
- ;; Create a directory for the non-store files that need to go into the
- ;; archive.
- (mkdir "extra")
+ (if root-system
+ (let ((directory (getcwd)))
+ (with-directory-excursion root-system
+ (apply invoke "tar"
+ "-cf" (string-append directory "/layer.tar")
+ `(,@transformation-options
+ ,@(tar-base-options)
+ ,@(scandir "."
+ (lambda (file)
+ (not (member file '("." "..")))))))))
+ (begin
+ ;; Create a directory for the non-store files that need to go
+ ;; into the archive.
+ (mkdir "extra")
- (with-directory-excursion "extra"
- ;; Create non-store files.
- (for-each (cut evaluate-populate-directive <> "./")
- extra-files)
+ (with-directory-excursion "extra"
+ ;; Create non-store files.
+ (for-each (cut evaluate-populate-directive <> "./")
+ extra-files)
- (when database
- ;; Initialize /var/guix, assuming PREFIX points to a profile.
- (install-database-and-gc-roots "." database prefix))
+ (when database
+ ;; Initialize /var/guix, assuming PREFIX points to a
+ ;; profile.
+ (install-database-and-gc-roots "." database prefix))
- (apply invoke "tar" "-cf" "../layer.tar"
- `(,@transformation-options
- ,@(tar-base-options)
- ,@paths
- ,@(scandir "."
- (lambda (file)
- (not (member file '("." ".."))))))))
+ (apply invoke "tar" "-cf" "../layer.tar"
+ `(,@transformation-options
+ ,@(tar-base-options)
+ ,@(if max-layers '() paths)
+ ,@(scandir "."
+ (lambda (file)
+ (not (member file '("." ".."))))))))
+ (delete-file-recursively "extra")))
;; It is possible for "/" to show up in the archive, especially when
;; applying transformations. For example, the transformation
@@ -261,24 +368,37 @@ (define* (build-docker-image image paths prefix
;; error messages.
(with-error-to-port (%make-void-port "w")
(lambda ()
- (system* "tar" "--delete" "/" "-f" "layer.tar")))
-
- (delete-file-recursively "extra"))
+ (system* "tar" "--delete" "/" "-f" "layer.tar"))))
(with-output-to-file "config.json"
(lambda ()
- (scm->json (config (string-append id "/layer.tar")
- time arch
- #:environment environment
- #:entry-point entry-point))))
- (with-output-to-file "manifest.json"
- (lambda ()
- (scm->json (manifest prefix id repository))))
- (with-output-to-file "repositories"
- (lambda ()
- (scm->json (repositories prefix id repository)))))
-
- (apply invoke "tar" "-cf" image "-C" directory
- `(,@(tar-base-options #:compressor compressor)
- "."))
+ (scm->json
+ (config (if max-layers
+ (layers-hashes
+ (append (size-sorted-store-items paths max-layers)
+ (list id)))
+ (list (layer-diff-id (string-append id "/layer.tar"))))
+ time arch
+ #:environment environment
+ #:entry-point entry-point))))
+ (if max-layers
+ (begin
+ (invoke "tar" "-rf" "image.tar" "config.json")
+ (if compressor
+ (begin
+ (apply invoke `(,@compressor "image.tar"))
+ (copy-file "image.tar.gz" image))
+ (copy-file "image.tar" image)))
+ (begin
+ (with-output-to-file "manifest.json"
+ (lambda ()
+ (scm->json (manifest prefix
+ (list (string-append id "/layer.tar"))
+ repository))))
+ (with-output-to-file "repositories"
+ (lambda ()
+ (scm->json (repositories prefix id repository))))
+ (apply invoke "tar" "-cf" image
+ `(,@(tar-base-options #:compressor compressor)
+ ".")))))
(delete-file-recursively directory)))
--
2.41.0
O
O
Oleg Pykhalov wrote on 26 Dec 2023 03:18
[PATCH 4/5] guix: pack: Build layered images.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
77bae2565cb5506342028896e1a45c757f3bad51.1703556298.git.go.wigust@gmail.com
* guix/scripts/pack.scm (docker-image, guix-pack, %default-options,
%docker-format-options, show-docker-format-options/detailed): Handle
'--max-layers' option.
* doc/guix.texi (Invoking guix pack): Document this.

Change-Id: I90660b2421fcdde891f003469fe2e2edaac7da41
---
doc/guix.texi | 26 ++++++++++++++++++++++-
guix/scripts/pack.scm | 38 ++++++++++++++++++++++++++-------
tests/pack.scm | 49 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 104 insertions(+), 9 deletions(-)

Toggle diff (232 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index cca250dc31..d21048405a 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@
Copyright @copyright{} 2017, 2018, 2019, 2020, 2023 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -7441,6 +7441,30 @@ Invoking guix pack
guix pack -f docker --entry-point=bin/guile --entry-point-argument="--help" guile
@end example
+@cindex maximum layers argument, for docker images
+@item --max-layers=@code{n}
+Specifies the maximum number of Docker image layers allowed when
+building an image.
+
+@example
+guix pack -f docker --max-layers=100 guile
+@end example
+
+This option allows you to limit the number of layers in a Docker image.
+Docker images are comprised of multiple layers, and each layer adds to
+the overall size and complexity of the image. By setting a maximum
+number of layers, you can control the following effects:
+
+@itemize
+@item Disk Usage:
+Increasing the number of layers can help optimize the disk space
+required to store multiple images built with a similar package graph.
+
+@item Pulling:
+When transferring images between different nodes or systems, having more
+layers can reduce the time required to pull the image.
+@end itemize
+
@item --expression=@var{expr}
@itemx -e @var{expr}
Consider the package @var{expr} evaluates to.
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 4c0a602eb1..22f0dd6061 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -9,6 +9,7 @@
;;; Copyright © 2020 Eric Bavier <bavier@posteo.net>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
;;; Copyright © 2023 Graham James Addis <graham@addis.org.uk>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -48,6 +49,7 @@ (define-module (guix scripts pack)
#:use-module (guix scripts build)
#:use-module (guix transformations)
#:use-module ((guix self) #:select (make-config.scm))
+ #:use-module ((guix docker) #:select (%docker-image-max-layers))
#:use-module (gnu compression)
#:use-module (gnu packages)
#:use-module (gnu packages bootstrap)
@@ -517,12 +519,15 @@ (define* (docker-image name profile
localstatedir?
(symlinks '())
(archiver tar)
- (extra-options '()))
+ (extra-options '())
+ max-layers)
"Return a derivation to construct a Docker image of PROFILE. The
image is a tarball conforming to the Docker Image Specification, compressed
with COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it
must a be a GNU triplet and it is used to derive the architecture metadata in
-the image. EXTRA-OPTIONS may contain the IMAGE-TAG keyword argument."
+the image. EXTRA-OPTIONS may contain the IMAGE-TAG keyword argument.
+If MAX-LAYERS is not false, the image will with many of the store paths being
+on their own layer to improve sharing between images."
(define database
(and localstatedir?
(file-append (store-database (list profile))
@@ -583,11 +588,17 @@ (define* (docker-image name profile
(cons* (string-append prefix "/" entry-point)
entry-point-argument))))
- (setenv "PATH" #+(file-append archiver "/bin"))
+ (setenv "PATH"
+ (string-join `(#+(file-append archiver "/bin")
+ #+@(if max-layers
+ (list (file-append gzip "/bin"))
+ '()))
+ ":"))
(let-keywords '#$extra-options #f
((image-tag #f)
- (entry-point-argument #f))
+ (entry-point-argument #f)
+ (max-layers #f))
(build-docker-image #$output
(map store-info-item
@@ -609,7 +620,8 @@ (define* (docker-image name profile
#:compressor
#+(compressor-command compressor)
#:creation-time
- (make-time time-utc 0 1)))))))
+ (make-time time-utc 0 1)
+ #:max-layers max-layers))))))
(gexp->derivation (string-append name ".tar"
(compressor-extension compressor))
@@ -1287,6 +1299,7 @@ (define %default-options
(verbosity . 1)
(symlinks . ())
(entry-point-argument . ())
+ (max-layers . ,%docker-image-max-layers)
(compressor . ,(first %compressors))))
(define %formats
@@ -1324,7 +1337,11 @@ (define (required-option symbol)
(define %docker-format-options
(list (required-option 'image-tag)
(option '(#\A "entry-point-argument") #t #f
- entry-point-argument-spec-option-parser)))
+ entry-point-argument-spec-option-parser)
+ (option '("max-layers") #t #f
+ (lambda (opt name arg result)
+ (alist-cons 'max-layers (string->number* arg)
+ result)))))
(define (show-docker-format-options)
(display (G_ "
@@ -1338,7 +1355,10 @@ (define (show-docker-format-options/detailed)
-A, --entry-point-argument=COMMAND/PARAMETER
Value(s) to use for the Docker EntryPoint arguments.
Multiple instances are accepted. This is only valid
- in conjunction with the --entry-point option"))
+ in conjunction with the --entry-point option
+
+ --max-layers=N
+ Number of image layers"))
(newline)
(exit 0))
@@ -1651,7 +1671,9 @@ (define-command (guix-pack . args)
(list #:image-tag
(assoc-ref opts 'image-tag)
#:entry-point-argument
- (assoc-ref opts 'entry-point-argument)))
+ (assoc-ref opts 'entry-point-argument)
+ #:max-layers
+ (assoc-ref opts 'max-layers)))
('deb
(list #:control-file
(process-file-arg opts 'control-file)
diff --git a/tests/pack.scm b/tests/pack.scm
index ac78817a70..fda4dc04c6 100644
--- a/tests/pack.scm
+++ b/tests/pack.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2017-2021, 2023 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2021, 2023 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -29,6 +30,7 @@ (define-module (test-pack)
#:use-module (guix gexp)
#:use-module (guix modules)
#:use-module (guix utils)
+ #:use-module ((guix build utils) #:select (%store-directory))
#:use-module (gnu packages)
#:use-module ((gnu packages base) #:select (libc-utf8-locales-for-target))
#:use-module (gnu packages bootstrap)
@@ -250,6 +252,53 @@ (define rpm-for-tests
(mkdir #$output)))))))
(built-derivations (list check))))
+ (unless store (test-skip 1))
+ (test-assertm "docker-layered-image + localstatedir"
+ (mlet* %store-monad
+ ((guile (set-guile-for-build (default-guile)))
+ (profile -> (profile
+ (content (packages->manifest (list %bootstrap-guile)))
+ (hooks '())
+ (locales? #f)))
+ (tarball (docker-image "docker-pack" profile
+ #:symlinks '(("/bin/Guile" -> "bin/guile"))
+ #:localstatedir? #t
+ #:max-layers 100))
+ (check (gexp->derivation "check-tarball"
+ (with-imported-modules '((guix build utils))
+ #~(begin
+ (use-modules (guix build utils)
+ (ice-9 match))
+
+ (define bin
+ (string-append "." #$profile "/bin"))
+
+ (define store
+ (string-append "." #$(%store-directory)))
+
+ (setenv "PATH" (string-append #$%tar-bootstrap "/bin"))
+ (mkdir "base")
+ (with-directory-excursion "base"
+ (invoke "tar" "xvf" #$tarball))
+
+ (match (find-files "base" "layer.tar")
+ ((layers ...)
+ (for-each (lambda (layer)
+ (invoke "tar" "xvf" layer)
+ (invoke "chmod" "--recursive" "u+w" store))
+ layers)))
+
+ (when
+ (and (file-exists? (string-append bin "/guile"))
+ (file-exists? "var/guix/db/db.sqlite")
+ (file-is-directory? "tmp")
+ (string=? (string-append #$%bootstrap-guile "/bin")
+ (pk 'binlink (readlink bin)))
+ (string=? (string-append #$profile "/bin/guile")
+ (pk 'guilelink (readlink "bin/Guile"))))
+ (mkdir #$output)))))))
+ (built-derivations (list check))))
+
(unless store (test-skip 1))
(test-assertm "squashfs-image + localstatedir"
(mlet* %store-monad
--
2.41.0
O
O
Oleg Pykhalov wrote on 26 Dec 2023 03:18
[PATCH 5/5] scripts: system: Build layered images.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
99155dabc366c37acb71f6624aa6e6025b3e571b.1703556298.git.go.wigust@gmail.com
* guix/scripts/system.scm (show-help, %docker-format-options, %options,
%default-options, show-docker-format-options,
show-docker-format-options/detailed, process-action): Handle '--max-layers'
option.
* gnu/system/image.scm (system-docker-image): Same.
* gnu/image.scm (<image>)[max-layers]: New record field.

Change-Id: I2726655aefd6688b976057fd5a38e9972ebfc292
---
gnu/image.scm | 4 ++++
gnu/system/image.scm | 41 ++++++++++++++++++++++++++++-------------
guix/scripts/system.scm | 28 ++++++++++++++++++++++++++--
3 files changed, 58 insertions(+), 15 deletions(-)

Toggle diff (191 lines)
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..7fb06dec10 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -42,6 +43,7 @@ (define-module (gnu image)
image-format
image-platform
image-size
+ image-max-layers
image-operating-system
image-partition-table-type
image-partitions
@@ -170,6 +172,8 @@ (define-record-type* <image>
(size image-size ;size in bytes as integer
(default 'guess)
(sanitize validate-size))
+ (max-layers image-max-layers ;number of layers as integer
+ (default #false))
(operating-system image-operating-system) ;<operating-system>
(partition-table-type image-partition-table-type ; 'mbr or 'gpt
(default 'mbr)
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index b825892232..2cc1012893 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -5,6 +5,7 @@
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
;;; Copyright © 2023 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -686,7 +687,8 @@ (define (image-with-label base-image label)
(define* (system-docker-image image
#:key
- (name "docker-image"))
+ (name "docker-image")
+ (archiver tar))
"Build a docker image for IMAGE. NAME is the base name to use for the
output file."
(define boot-program
@@ -731,6 +733,7 @@ (define* (system-docker-image image
(use-modules (guix docker)
(guix build utils)
(gnu build image)
+ (srfi srfi-1)
(srfi srfi-19)
(guix build store-copy)
(guix store database))
@@ -754,18 +757,30 @@ (define* (system-docker-image image
#:register-closures? #$register-closures?
#:deduplicate? #f
#:system-directory #$os)
- (build-docker-image
- #$output
- (cons* image-root
- (map store-info-item
- (call-with-input-file #$graph
- read-reference-graph)))
- #$os
- #:entry-point '(#$boot-program #$os)
- #:compressor '(#+(file-append gzip "/bin/gzip") "-9n")
- #:creation-time (make-time time-utc 0 1)
- #:system #$image-target
- #:transformations `((,image-root -> ""))))))))
+ (when #$(image-max-layers image)
+ (setenv "PATH"
+ (string-join (list #+(file-append archiver "/bin")
+ #+(file-append gzip "/bin"))
+ ":")))
+ (apply build-docker-image
+ (append (list #$output
+ (append (if #$(image-max-layers image)
+ '()
+ (list image-root))
+ (map store-info-item
+ (call-with-input-file #$graph
+ read-reference-graph)))
+ #$os
+ #:entry-point '(#$boot-program #$os)
+ #:compressor
+ '(#+(file-append gzip "/bin/gzip") "-9n")
+ #:creation-time (make-time time-utc 0 1)
+ #:system #$image-target
+ #:transformations `((,image-root -> "")))
+ (if #$(image-max-layers image)
+ (list #:root-system image-root
+ #:max-layers #$(image-max-layers image))
+ '()))))))))
(computed-file name builder
;; Allow offloading so that this I/O-intensive process
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index f85b663d64..a21ecd4d1e 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts system)
#:use-module (guix scripts system reconfigure)
#:use-module (guix build utils)
#:use-module (guix progress)
+ #:use-module ((guix docker) #:select (%docker-image-max-layers))
#:use-module (gnu build image)
#:use-module (gnu build install)
#:autoload (gnu build file-systems)
@@ -1053,6 +1054,8 @@ (define (show-help)
(newline)
(show-native-build-options-help)
(newline)
+ (show-docker-format-options)
+ (newline)
(display (G_ "
-h, --help display this help and exit"))
(display (G_ "
@@ -1060,6 +1063,12 @@ (define (show-help)
(newline)
(show-bug-report-information))
+(define %docker-format-options
+ (list (option '("max-layers") #t #f
+ (lambda (opt name arg result)
+ (alist-cons 'max-layers (string->number* arg)
+ result)))))
+
(define %options
;; Specifications of the command-line options.
(cons* (option '(#\h "help") #f #f
@@ -1154,7 +1163,8 @@ (define %options
(alist-cons 'list-installed (or arg "") result)))
(append %standard-build-options
%standard-cross-build-options
- %standard-native-build-options)))
+ %standard-native-build-options
+ %docker-format-options)))
(define %default-options
;; Alist of default option values.
@@ -1175,7 +1185,8 @@ (define %default-options
(label . #f)
(volatile-image-root? . #f)
(volatile-vm-root? . #t)
- (graph-backend . "graphviz")))
+ (graph-backend . "graphviz")
+ (max-layers . ,%docker-image-max-layers)))
(define (verbosity-level opts)
"Return the verbosity level based on OPTS, the alist of parsed options."
@@ -1183,6 +1194,17 @@ (define (verbosity-level opts)
(if (eq? (assoc-ref opts 'action) 'build)
3 1)))
+(define (show-docker-format-options)
+ (display (G_ "
+ --help-docker-format list options specific to the docker image type.")))
+
+(define (show-docker-format-options/detailed)
+ (display (G_ "
+ --max-layers=N
+ Number of image layers"))
+ (newline)
+ (exit 0))
+
;;;
;;; Entry point.
@@ -1245,6 +1267,7 @@ (define (process-action action args opts)
((docker-image) docker-image-type)
(else image-type)))
(image-size (assoc-ref opts 'image-size))
+ (image-max-layers (assoc-ref opts 'max-layers))
(volatile?
(assoc-ref opts 'volatile-image-root?))
(shared-network?
@@ -1258,6 +1281,7 @@ (define (process-action action args opts)
(image-with-label base-image label)
base-image))
(size image-size)
+ (max-layers image-max-layers)
(volatile-root? volatile?)
(shared-network? shared-network?))))
(os (or (image-operating-system image)
--
2.41.0
O
O
Oleg Pykhalov wrote on 26 Dec 2023 03:40
Re: [bug#62153] Merging guix pack changes for Docker containers packaging
(name . Ludovic Courtès)(address . ludo@gnu.org)
87il4l4rez.fsf@gmail.com
Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (29 lines)
> Apologies for not replying earlier. I occasionally get reminded of the
> fact that building single-layer images is a problem, but only now did I
> take the time to look more closely at the latest version of these
> patches.
>
> Oleg Pykhalov <go.wigust@gmail.com> skribis:
>
>> I would like to merge 62153. After 64173 will be merge, merging 62153
>> is not possible without conflict resolving with Git.
>>
>> 64173 introduces ‘%docker-format-options’ variable. With this variable
>> it's possible in 62153 to replace ‘--image-type=docker-layered’ with
>> ‘--docker-layers=N’ option, where:
>>
>> if ‘N’ is zero, then use current non layered format
>> if ‘N’ is bigger than zero, then use layered format
>
> OK we should do that. However, the original submitter of #64173
> apparently dropped the ball as we were approaching the final version.
>
> Would you like to adopt it and submit/push a version that incorporates
> the latest comments?
>
> Alternatively, we could do the opposite: merge the Docker layer patches
> first, and then rebase the ‘%docker-format-options’ patch, after which
> we could add the ‘--docker-layers’ option.
>
> What’s your preference?

[…]

Patches 64173 and 62153 (v5) have been sent to 62153.

If you don't mind, I have changed the option naming to '--max-layers=N'
instead of '--docker-layers=N' to align with the format of
'--entry-point-argument' (without specifying Docker as the only image
format that utilizes layers).

I did not include code to check if 'N' is zero and use the current
non-layered format. Instead, I opted for the default value of '#false'
as it was easier to implement.


Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmWKPTQUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzYpBAAuIqdacRLFs0TgY6PKw6yiUARQSRb
AKT1b/9dryTHfxL9STKiHdgkSXzCK7SvfXX/SkQrACybakckunGf0BD8ZTCZ47i8
d3vLKOA8iRlewHStIVeuJUzzJgwb+CefTkkXX4WwEVrrfvdFbCqs6iDZPsGZzI90
KmYfRkbNwfwEGXV+efZMKZ982gs+56dCMbJk6rK9TYAXxx4zxQJdQ7zGHyG9J2I2
zAMK/YsfvmaVVzcgtDkiFn0FBZ1hrXtL2EipzEGs3VtUO0l1XdHmei4auTyaxpD5
xlMYEhSXrTlW0JxXzEEQigJMeoKECesj9CzO1C18ByVTnJNT5ABJAcwXKMO7JT7i
iJlrlO1vce+u2XdeAlIaNU0225Nn21NZWoAce2O424wrIUvmtp0w835R+Zgp9OTR
uan2o5XTYXTRp6F9IRdTiDORBF5C7T7vay+f7TVPBafJDMpz2it87FXp9FlICFPl
HqNW56R0780AOl0N+Qg6UOlr8rQ2wlmjeIRCeLiSuQBvKE/eJzaXL1i72b/koHn6
c47tZbWbjTJqJgoDXC7QnJiUlvhlLvbR7c9cH+QKgj4Oi7IejG96prSZXrDli12z
IVPERJM9UsPmFSNoqOlgaVChbda3i8gPJjzZro6TxFPlk7JorMlmsUURvMWYhi5z
HAhQvJxZLviRjEY=
=Dj7B
-----END PGP SIGNATURE-----

M
M
Mathieu Othacehe wrote on 27 Dec 2023 19:14
Re: [bug#62153] [PATCH 1/5] guix: pack: Add '--entry-point-argument' option.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
87edf75x7i.fsf@gnu.org
Hello Oleg,

Toggle quote (2 lines)
> + "A SRFI-37 opion parser for the --entry-point-argument option. The spec

option -> option

Toggle quote (2 lines)
> +takes multiple occurances. The entries are used in the exec form for the

orrurances -> occurrences, also use two spaces after a dot.

Toggle quote (3 lines)
> + ;; first entry is constructed by prefixing the entry-point with
> + ;; the supplied index subsequent entries are taken from the

subsequent -> , subsequent

Toggle quote (3 lines)
> + -A, --entry-point-argument=COMMAND/PARAMETER
> + Value(s) to use for the Docker EntryPoint arguments.

EntryPoint -> ENTRYPOINT

Thanks,

Mathieu
M
M
Mathieu Othacehe wrote on 27 Dec 2023 19:16
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
877ckz5x48.fsf@gnu.org
+ (and=> entry-point
+ (lambda (entry-point)
+ (cons* (string-append prefix "/" entry-point)
+ entry-point-argument))))
^ ^

There are also two tabulations here that need to be removed.

Thanks,

Mathieu
M
M
Mathieu Othacehe wrote on 27 Dec 2023 21:15
Re: [bug#62153] [PATCH 3/5] guix: docker: Build layered images.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
8734vn5rm0.fsf@gnu.org
Toggle quote (4 lines)
> +When MAX-LAYERS is not false build layered image, providing a Docker
> +image with many of the store paths being on their own layer to improve sharing
> +between images.

"many of the store paths being on their own layer" is a big vague.

It could be rephrased to "store paths splitted in their own layers" or so.

Thanks,

Mathieu
M
M
Mathieu Othacehe wrote on 27 Dec 2023 21:25
Re: [bug#62153] [PATCH 4/5] guix: pack: Build layered images.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
87y1df4cm9.fsf@gnu.org
Toggle quote (3 lines)
> +If MAX-LAYERS is not false, the image will with many of the store paths being
> +on their own layer to improve sharing between images."

Same comment as in previous patch, "many" is a bit vague. "the image
will be splitted in up to MAX-LAYERS layers" or so could work.

Toggle quote (3 lines)
> + (check (gexp->derivation "check-tarball"
> + (with-imported-modules '((guix build utils))

You could rearrange as:

Toggle snippet (5 lines)
(check (gexp->derivation
"check-tarball"
(with-imported-modules '((guix build utils))

to reduce the indentation of this block:

Toggle quote (4 lines)
> + #~(begin
> + (use-modules (guix build utils)
> + (ice-9 match))

...

Toggle quote (5 lines)
> + (string=? (string-append #$%bootstrap-guile "/bin")
> + (pk 'binlink (readlink bin)))
> + (string=? (string-append #$profile "/bin/guile")
> + (pk 'guilelink (readlink "bin/Guile"))))

Left over pk's?

Thanks,

Mathieu
M
M
Mathieu Othacehe wrote on 27 Dec 2023 21:29
Re: [bug#62153] [PATCH 5/5] scripts: system: Build layered images.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
87tto34cfi.fsf@gnu.org
Other than the few cosmetic remarks, the series looks great. I tested
producing multi-layers pack and images with success.

You may want to wait for Ludo's opinion as a reviewer of the v4, but as
far as I am concerned, I think that you can directly proceed with the
cosmetic issues fixed.

Thanks,

Mathieu
L
L
Ludovic Courtès wrote on 8 Jan 17:49 +0100
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87jzoj4vpc.fsf@gnu.org
Hello Oleg and all,

Mathieu Othacehe <othacehe@gnu.org> skribis:

Toggle quote (7 lines)
> Other than the few cosmetic remarks, the series looks great. I tested
> producing multi-layers pack and images with success.
>
> You may want to wait for Ludo's opinion as a reviewer of the v4, but as
> far as I am concerned, I think that you can directly proceed with the
> cosmetic issues fixed.

I haven’t been able to test it yet, but I agree with Mathieu’s
suggestions and I agree that you can proceed with this small fixes.

As a bonus, consider adding an entry in ‘etc/news.scm’: it’s definitely
news-worthy!

Ludo’.
O
O
Oleg Pykhalov wrote on 9 Jan 13:58 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)
877cki1x4d.fsf@gmail.com
Hello Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (13 lines)
>> Other than the few cosmetic remarks, the series looks great. I tested
>> producing multi-layers pack and images with success.
>>
>> You may want to wait for Ludo's opinion as a reviewer of the v4, but as
>> far as I am concerned, I think that you can directly proceed with the
>> cosmetic issues fixed.
>
> I haven’t been able to test it yet, but I agree with Mathieu’s
> suggestions and I agree that you can proceed with this small fixes.
>
> As a bonus, consider adding an entry in ‘etc/news.scm’: it’s definitely
> news-worthy!

All Mathieu's suggestions applied, etc/news.scm from v4 slightly
modified accordingly. Everything pushed to master.

Thanks everyone for helping adding this feature.

Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmWdQxIUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pxzsg/+NCIxAqM1ZRgHUmq4il9DhWwT7dxr
ibo6xCyqQtJe+9KVU5r2HU9ARZeA0MJWUKTC7HbGsztrSu5FidzJIDDFhhblGx95
JhlGPGKMKZpBaeGFnQl9yJTo0MjHnFlPlCiOIxXNSrsRe2W4qwLyrFBdUsnkkZ8U
O//avuJUNojiAipSIo/rDGYTTQQeoJe3zpwMlGA/YedQVL216QSUDIcUVs8+X2iJ
28CxqfNKGnW/XycPH81i/uZ3z4z6nv2y2FUk+u9mDho2QbAj6y+02siAwLMAMg2N
JMfW5s0YtIhXLgSzIBrz+T3objHh/NAqdN05KqtLhcVziYv6HzM/5mh5y8Fw8bwX
chpUKV7JqWyHUYbQJRZ1G6A0Xkku1sn/EN0HANa/teG9wXMN/DU/Our4z6db9IHb
T5NeegE4LO9VAg8+K8QmQq4toXyAicU5Cked2ar3zu2qDq37DYmk9XgY3yc97SqS
AW15wb//sJJki3nER+rQXmEFrkFlWGaNxlLK81oJP3YKYxxiraFOTugVF6NFOMLR
uwkK9h0jvKt7+8z+5JjReMggqNOn79sYx2vQTBiwA/iSv2rq8/6s/FWUTkRJ1ua4
nVYK6k67Wyc+V8soGxJSAu13bwih56Ji0geb2l91S4SXehfnTpNP2iV39q+mgtDZ
2w+tcjQp7WxGBxg=
=/NHv
-----END PGP SIGNATURE-----

Closed
L
L
Ludovic Courtès wrote on 18 Jan 15:55 +0100
Re: [bug#62153] [PATCH 3/5] guix: docker: Build layered images.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
871qaeeln8.fsf@gnu.org
Hi Oleg!

Oleg Pykhalov <go.wigust@gmail.com> skribis:

Toggle quote (7 lines)
> +;; The maximum number of layers allowed in a Docker image is typically around
> +;; 128, although it may vary depending on the Docker daemon. However, we
> +;; recommend setting the limit to 100 to ensure sufficient room for future
> +;; extensions.
> +(define %docker-image-max-layers
> + #f)

It just occurred to me that the meaning of #f is unclear here; also, the
manual does not specify the default value of ‘--max-layers’.

Should we:

1. Set ‘%docker-image-max-layers’ to an integer, maybe 100, in
accordance with the comment above?

2. Clarify in the manual what the default is, and explain that users
can pass ‘--max-layers=1’ if they want, well, a single layer.

Thoughts?

Ludo’.
?