[PATCH 0/2] Add Docker layered image for pack and system

  • Open
  • quality assurance status badge
Details
6 participants
  • Oleg Pykhalov
  • Ludovic Courtès
  • Christopher Baines
  • pelzflorian (Florian Pelz)
  • Ricardo Wurmus
  • Simon Tournier
Owner
unassigned
Submitted by
Oleg Pykhalov
Severity
normal
O
O
Oleg Pykhalov wrote on 13 Mar 01:30 +0100
(address . guix-patches@gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003012.14325-1-go.wigust@gmail.com
Hi Guix,

This patch series add to 'guix pack' and 'guix system image' formats with a
layered Docker image, which dicreases images size by sharing same layers on a
host.

The folling commands show an example for new image formats:

./pre-inst-env guix system image --image-type=docker-layered config.scm
docker load -i result

./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello
docker load -i result

The folloing tests passed:

make check-channel-news
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system docker-layered-system"

The gnu/packages/aux-files/python/stream-layered-image.py Python script is a
copy of github.com/NixOS/nixpkgs/pkgs/build-support/docker/stream_layered_image.py
with only a simple replacement "/nix" to "/gnu" string.

Oleg Pykhalov (2):
guix: docker: Build layered image.
news: Add entry for the new 'docker-layered' distribution format.

Makefile.am | 3 +-
doc/guix.texi | 16 +-
etc/news.scm | 38 ++
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 103 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
11 files changed, 813 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py


base-commit: 60a211ec705ac98483d76da7f2523f2b8966343a
--
2.38.0
O
O
Oleg Pykhalov wrote on 13 Mar 01:33 +0100
[PATCH 1/2] guix: docker: Build layered image.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003310.17129-1-go.wigust@gmail.com
* gnu/packages/aux-files/python/stream-layered-image.py: New file.
* Makefile.am (AUX_FILES): Add this.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* tests/pack.scm: Add docker-layered-image + localstatedir test.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New
variables.
(system-docker-image)[layered-image?]: New argument.
(stream-layered-image.py): New variable.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* gnu/image.scm (validate-image-format)[docker-layered]: New image format.
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image-type Reference): Document docker-layered-image-type.
---
Makefile.am | 3 +-
doc/guix.texi | 16 +-
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 103 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
10 files changed, 775 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py

Toggle diff (518 lines)
diff --git a/Makefile.am b/Makefile.am
index 23b939b674..9aca84f8f8 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -11,7 +11,7 @@
# Copyright © 2017 Arun Isaac <arunisaac@systemreboot.net>
# Copyright © 2018 Nikita <nikita@n0.is>
# Copyright © 2018 Julien Lepiller <julien@lepiller.eu>
-# Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
+# Copyright © 2018, 2023 Oleg Pykhalov <go.wigust@gmail.com>
# Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
@@ -435,6 +435,7 @@ AUX_FILES = \
gnu/packages/aux-files/python/sanity-check.py \
gnu/packages/aux-files/python/sanity-check-next.py \
gnu/packages/aux-files/python/sitecustomize.py \
+ gnu/packages/aux-files/python/stream-layered-image.py \
gnu/packages/aux-files/renpy/renpy.in \
gnu/packages/aux-files/run-in-namespace.c
diff --git a/doc/guix.texi b/doc/guix.texi
index b545751e1b..bd0ee126ee 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@ Copyright @copyright{} 2017 Andy Wingo@*
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6840,9 +6840,15 @@ the following command:
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -43631,6 +43637,10 @@ Build an image based on the @code{iso9660-image} image but with the
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-with-syntax-properties (name (value properties))
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
new file mode 100644
index 0000000000..9ad2168c2d
--- /dev/null
+++ b/gnu/packages/aux-files/python/stream-layered-image.py
@@ -0,0 +1,391 @@
+"""
+This script generates a Docker image from a set of store paths. Uses
+Docker Image Specification v1.2 as reference [1].
+
+It expects a JSON file with the following properties and writes the
+image as an uncompressed tarball to stdout:
+
+* "architecture", "config", "os", "created", "repo_tag" correspond to
+ the fields with the same name on the image spec [2].
+* "created" can be "now".
+* "created" is also used as mtime for files added to the image.
+* "store_layers" is a list of layers in ascending order, where each
+ layer is the list of store paths to include in that layer.
+
+The main challenge for this script to create the final image in a
+streaming fashion, without dumping any intermediate data to disk
+for performance.
+
+A docker image has each layer contents archived as separate tarballs,
+and they later all get enveloped into a single big tarball in a
+content addressed fashion. However, because how "tar" format works,
+we have to know about the name (which includes the checksum in our
+case) and the size of the tarball before we can start adding it to the
+outer tarball. We achieve that by creating the layer tarballs twice;
+on the first iteration we calculate the file size and the checksum,
+and on the second one we actually stream the contents. 'add_layer_dir'
+function does all this.
+
+[1]: https://github.com/moby/moby/blob/master/image/spec/v1.2.md
+[2]: https://github.com/moby/moby/blob/4fb59c20a4fb54f944fe170d0ff1d00eb4a24d6f/image/spec/v1.2.md#image-json-field-descriptions
+""" # noqa: E501
+
+
+import io
+import os
+import re
+import sys
+import json
+import hashlib
+import pathlib
+import tarfile
+import itertools
+import threading
+from datetime import datetime, timezone
+from collections import namedtuple
+
+
+def archive_paths_to(obj, paths, mtime):
+ """
+ Writes the given store paths as a tar file to the given stream.
+
+ obj: Stream to write to. Should have a 'write' method.
+ paths: List of store paths.
+ """
+
+ # gettarinfo makes the paths relative, this makes them
+ # absolute again
+ def append_root(ti):
+ ti.name = "/" + ti.name
+ return ti
+
+ def apply_filters(ti):
+ ti.mtime = mtime
+ ti.uid = 0
+ ti.gid = 0
+ ti.uname = "root"
+ ti.gname = "root"
+ return ti
+
+ def nix_root(ti):
+ ti.mode = 0o0555 # r-xr-xr-x
+ return ti
+
+ def dir(path):
+ ti = tarfile.TarInfo(path)
+ ti.type = tarfile.DIRTYPE
+ return ti
+
+ with tarfile.open(fileobj=obj, mode="w|") as tar:
+ # To be consistent with the docker utilities, we need to have
+ # these directories first when building layer tarballs.
+ tar.addfile(apply_filters(nix_root(dir("/gnu"))))
+ tar.addfile(apply_filters(nix_root(dir("/gnu/store"))))
+
+ for path in paths:
+ path = pathlib.Path(path)
+ if path.is_symlink():
+ files = [path]
+ else:
+ files = itertools.chain([path], path.rglob("*"))
+
+ for filename in sorted(files):
+ ti = append_root(tar.gettarinfo(filename))
+
+ # copy hardlinks as regular files
+ if ti.islnk():
+ ti.type = tarfile.REGTYPE
+ ti.linkname = ""
+ ti.size = filename.stat().st_size
+
+ ti = apply_filters(ti)
+ if ti.isfile():
+ with open(filename, "rb") as f:
+ tar.addfile(ti, f)
+ else:
+ tar.addfile(ti)
+
+
+class ExtractChecksum:
+ """
+ A writable stream which only calculates the final file size and
+ sha256sum, while discarding the actual contents.
+ """
+
+ def __init__(self):
+ self._digest = hashlib.sha256()
+ self._size = 0
+
+ def write(self, data):
+ self._digest.update(data)
+ self._size += len(data)
+
+ def extract(self):
+ """
+ Returns: Hex-encoded sha256sum and size as a tuple.
+ """
+ return (self._digest.hexdigest(), self._size)
+
+
+FromImage = namedtuple("FromImage", ["tar", "manifest_json", "image_json"])
+# Some metadata for a layer
+LayerInfo = namedtuple("LayerInfo", ["size", "checksum", "path", "paths"])
+
+
+def load_from_image(from_image_str):
+ """
+ Loads the given base image, if any.
+
+ from_image_str: Path to the base image archive.
+
+ Returns: A 'FromImage' object with references to the loaded base image,
+ or 'None' if no base image was provided.
+ """
+ if from_image_str is None:
+ return None
+
+ base_tar = tarfile.open(from_image_str)
+
+ manifest_json_tarinfo = base_tar.getmember("manifest.json")
+ with base_tar.extractfile(manifest_json_tarinfo) as f:
+ manifest_json = json.load(f)
+
+ image_json_tarinfo = base_tar.getmember(manifest_json[0]["Config"])
+ with base_tar.extractfile(image_json_tarinfo) as f:
+ image_json = json.load(f)
+
+ return FromImage(base_tar, manifest_json, image_json)
+
+
+def add_base_layers(tar, from_image):
+ """
+ Adds the layers from the given base image to the final image.
+
+ tar: 'tarfile.TarFile' object for new layers to be added to.
+ from_image: 'FromImage' object with references to the loaded base image.
+ """
+ if from_image is None:
+ print("No 'fromImage' provided", file=sys.stderr)
+ return []
+
+ layers = from_image.manifest_json[0]["Layers"]
+ checksums = from_image.image_json["rootfs"]["diff_ids"]
+ layers_checksums = zip(layers, checksums)
+
+ for num, (layer, checksum) in enumerate(layers_checksums, start=1):
+ layer_tarinfo = from_image.tar.getmember(layer)
+ checksum = re.sub(r"^sha256:", "", checksum)
+
+ tar.addfile(layer_tarinfo, from_image.tar.extractfile(layer_tarinfo))
+ path = layer_tarinfo.path
+ size = layer_tarinfo.size
+
+ print("Adding base layer", num, "from", path, file=sys.stderr)
+ yield LayerInfo(size=size, checksum=checksum, path=path, paths=[path])
+
+ from_image.tar.close()
+
+
+def overlay_base_config(from_image, final_config):
+ """
+ Overlays the final image 'config' JSON on top of selected defaults from the
+ base image 'config' JSON.
+
+ from_image: 'FromImage' object with references to the loaded base image.
+ final_config: 'dict' object of the final image 'config' JSON.
+ """
+ if from_image is None:
+ return final_config
+
+ base_config = from_image.image_json["config"]
+
+ # Preserve environment from base image
+ final_env = base_config.get("Env", []) + final_config.get("Env", [])
+ if final_env:
+ # Resolve duplicates (last one wins) and format back as list
+ resolved_env = {entry.split("=", 1)[0]: entry for entry in final_env}
+ final_config["Env"] = list(resolved_env.values())
+ return final_config
+
+
+def add_layer_dir(tar, paths, store_dir, mtime):
+ """
+ Appends given store paths to a TarFile object as a new layer.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ paths: List of store paths.
+ store_dir: the root directory of the nix store
+ mtime: 'mtime' of the added files and the layer tarball.
+ Should be an integer representing a POSIX time.
+
+ Returns: A 'LayerInfo' object containing some metadata of
+ the layer added.
+ """
+
+ invalid_paths = [i for i in paths if not i.startswith(store_dir)]
+ assert len(invalid_paths) == 0, \
+ f"Expecting absolute paths from {store_dir}, but got: {invalid_paths}"
+
+ # First, calculate the tarball checksum and the size.
+ extract_checksum = ExtractChecksum()
+ archive_paths_to(
+ extract_checksum,
+ paths,
+ mtime=mtime,
+ )
+ (checksum, size) = extract_checksum.extract()
+
+ path = f"{checksum}/layer.tar"
+ layer_tarinfo = tarfile.TarInfo(path)
+ layer_tarinfo.size = size
+ layer_tarinfo.mtime = mtime
+
+ # Then actually stream the contents to the outer tarball.
+ read_fd, write_fd = os.pipe()
+ with open(read_fd, "rb") as read, open(write_fd, "wb") as write:
+ def producer():
+ archive_paths_to(
+ write,
+ paths,
+ mtime=mtime,
+ )
+ write.close()
+
+ # Closing the write end of the fifo also closes the read end,
+ # so we don't need to wait until this thread is finished.
+ #
+ # Any exception from the thread will get printed by the default
+ # exception handler, and the 'addfile' call will fail since it
+ # won't be able to read required amount of bytes.
+ threading.Thread(target=producer).start()
+ tar.addfile(layer_tarinfo, read)
+
+ return LayerInfo(size=size, checksum=checksum, path=path, paths=paths)
+
+
+def add_customisation_layer(target_tar, customisation_layer, mtime):
+ """
+ Adds the customisation layer as a new layer. This is layer is structured
+ differently; given store path has the 'layer.tar' and corresponding
+ sha256sum ready.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ customisation_layer: Path containing the layer archive.
+ mtime: 'mtime' of the added layer tarball.
+ """
+
+ checksum_path = os.path.join(customisation_layer, "checksum")
+ with open(checksum_path) as f:
+ checksum = f.read().strip()
+ assert len(checksum) == 64, f"Invalid sha256 at ${checksum_path}."
+
+ layer_path = os.path.join(customisation_layer, "layer.tar")
+
+ path = f"{checksum}/layer.tar"
+ tarinfo = target_tar.gettarinfo(layer_path)
+ tarinfo.name = path
+ tarinfo.mtime = mtime
+
+ with open(layer_path, "rb") as f:
+ target_tar.addfile(tarinfo, f)
+
+ return LayerInfo(
+ size=None,
+ checksum=checksum,
+ path=path,
+ paths=[customisation_layer]
+ )
+
+
+def add_bytes(tar, path, content, mtime):
+ """
+ Adds a file to the tarball with given path and contents.
+
+ tar: 'tarfile.TarFile' object.
+ path: Path of the file as a string.
+ content: Contents of the file.
+ mtime: 'mtime' of the file. Should be an integer representing a POSIX time.
+ """
+ assert type(content) is bytes
+
+ ti = tarfile.TarInfo(path)
+ ti.size = len(content)
+ ti.mtime = mtime
+ tar.addfile(ti, io.BytesIO(content))
+
+
+def main():
+ with open(sys.argv[1], "r") as f:
+ conf = json.load(f)
+
+ created = (
+ datetime.now(tz=timezone.utc)
+ if conf["created"] == "now"
+ else datetime.fromisoformat(conf["created"])
+ )
+ mtime = int(created.timestamp())
+ store_dir = conf["store_dir"]
+
+ from_image = load_from_image(conf["from_image"])
+
+ with tarfile.open(mode="w|", fileobj=sys.stdout.buffer) as tar:
+ layers = []
+ layers.extend(add_base_layers(tar, from_image))
+
+ start = len(layers) + 1
+ for num, store_layer in enumerate(conf["store_layers"], start=start):
+ print("Creating layer", num, "from paths:", store_layer,
+ file=sys.stderr)
+ info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
+ layers.append(info)
+
+ print("Creating layer", len(layers) + 1, "with customisation...",
+ file=sys.stderr)
+ layers.append(
+ add_customisation_layer(
+ tar,
+ conf["customisation_layer"],
+ mtime=mtime
+ )
+ )
+
+ print("Adding manifests...", file=sys.stderr)
+
+ image_json = {
+ "created": datetime.isoformat(created),
+ "architecture": conf["architecture"],
+ "os": "linux",
+ "config": overlay_base_config(from_image, conf["config"]),
+ "rootfs": {
+ "diff_ids": [f"sha256:{layer.checksum}" for layer in layers],
+ "type": "layers",
+ },
+ "history": [
+ {
+ "created": datetime.isoformat(created),
+ "comment": f"store paths: {layer.paths}"
+ }
+ for layer in layers
+ ],
+ }
+
+ image_json = json.dumps(image_json, indent=4).encode("utf-8")
+ image_json_checksum = hashlib.sha256(image_json).hexdigest()
+ image_json_path = f"{image_json_checksum}.json"
+ add_bytes(tar, image_json_path, image_json, mtime=mtime)
+
+ manifest_json = [
+ {
+ "Config": image_json_path,
+ "RepoTags": [conf["repo_tag"]],
+ "Layers": [layer.path for layer in layers],
+ }
+ ]
+ manifest_json = json.dumps(manifest_json, indent=4).encode("utf-8")
+ add_bytes(tar, "manifest.json", manifest_json, mtime=mtime)
+
+ print("Done.", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..0bfd011ad4 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -45,6 +46,7 @@ (define-module (gnu system image)
#:use-module (gnu system uuid)
#:use-module (gnu system vm)
#:use-module (guix packages)
+ #:use-module ((gnu packages) #:select (search-auxiliary-file))
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages bootloaders)
@@ -58,6 +60,7 @@ (define-module (gnu system image)
#:use-module (gnu packages hurd)
#:use-module (gnu packages linux)
#:use-module (gnu packages mtools)
+ #:use-module (gnu packages python)
#:use-module (gnu packages virtualization)
#:use-module ((srfi srfi-1) #:prefix srfi-1:)
#:use-module (srfi srfi-11)
@@ -78,6 +81,7 @@ (define-module (gnu system image)
efi-disk-image
iso9660-image
docker-image
+ docker-layered-image
tarball-image
wsl2-image
raw-with-offset-disk-image
@@ -89,6 +93,7 @@ (define-module (gnu system image)

This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 13 Mar 01:33 +0100
[PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(address . 62153@debbugs.gnu.org)(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
20230313003310.17129-2-go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)

Toggle diff (58 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 924c2b35b4..98fb7f536c 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,43 @@
(channel-news
(version 0)
+ (entry (commit "45777c5b753ce330ad007d4e71189cf3fc627ccc")
+ (title
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "Новый @samp{docker-layered} формат для @command{guix pack} команды"))
+ (body
+ (en "Docker layered image can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+See @command{info \"(guix) Invoking guix pack\"} for more information.
+
+@command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+")
+ (ru "Появилась команда создания многослойных Docker образов с помощью
+@command{guix pack --format=docker-layered}, которая соберет Docker образ с
+путями в store расположенными на отдельных слоях, ускоряя таким образом
+передачу образов. Образ будет создан в GNU store в качестве gzipped tarball.
+
+Пример создания Docker layered image с @code{hello} пакетом:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+Смотрите @command{info \"(guix) Invoking guix pack\"} для получения более
+детальных сведений.
+
+@command{guix system image} теперь может создавать layered Docker image путем
+указания в опции @option{--image-type} параметра @code{docker-layered}.")))
+
(entry (commit "598f4c509bbfec2b983a8ee246cce0a0fe45ec7f")
(title
(de "Neues Format @samp{rpm} für den Befehl @command{guix pack}")
--
2.38.0
O
O
Oleg Pykhalov wrote on 13 Mar 01:43 +0100
Cover lever typo in guix pack format example
(address . 62153@debbugs.gnu.org)
87wn3l32hs.fsf@gmail.com
The cover lever guix pack example should be:

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello

instead of

./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello

Apologies,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQOcZ8UHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzRkg/9H8+149MGcejnPteSR7C1XKWckEWi
xoADRq5qcwFf9JTGGNEDqEoq/z+BFCa5ainGpLKO6jzVGe0gCfzsmmxRA7V5fZY3
udwfPwVmhb3oV4CV8t8Gi3tqTnJ3meUG7CdzNttocYRlwLys2TUoy5t8fJ9xwg/v
6ZSVeKfcXJdM9BP5a+KkLMzfxGOGlfu8HFVIrVlPAMgRmliZLRKFdk3mmZ7jJEQv
OepVWlQ5j6BRHfbETEYz6nB7Hw2BYzfL5dwzkrsMBwO2KgtG9sYSgdqIbRVJXcP9
L3R5Z1Z51p4CcByYG35iq0ahwGy9daUL1UiCx8gIXJPEy7RGMFqjOAGekg+lN2PI
qOHLMpthqJBKQxKnYBxp7ZdgnvwX4iYs7YrUDDX6I507L1zYABWNM2exLl1N3PUF
vlHKUmI2la4ozWDaHNZsTa7qVtFtRwXr5Vs8hfkbqfEhAG3YHQKd0zMBr33Ht6bn
WYdT6zKq2Dnp3BjK1dkOKYhIJelmWlwptUYv88h7KiozgEOI84l8RpSOU7IXnEBY
CUhhGhPp2xu4w7NCkOA3TAYHGYFT2ebFnlZ+l3fTq6i8Y3p9LVD65Jvz9Pqgt0MQ
9v7+ArClkWYmKxc0XNrqpv2J8mAr5PlSvfXtHM0wTsZk+cpuaC/xIkCf4upHyLyh
1a4qZJxNTV9FjLA=
=z9Ac
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 13 Mar 16:01 +0100
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
87r0tsk85r.fsf@gmail.com
Hi,

Oh cool! Awesome! Thanks for pushing forward.

On lun., 13 mars 2023 at 03:33, Oleg Pykhalov <go.wigust@gmail.com> wrote:

Toggle quote (10 lines)
> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
> new file mode 100644
> index 0000000000..9ad2168c2d
> --- /dev/null
> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
> @@ -0,0 +1,391 @@
> +"""
> +This script generates a Docker image from a set of store paths. Uses
> +Docker Image Specification v1.2 as reference [1].

Instead of Python, would it possible to implement in Guile? I mean,
does Python have something that is missing in Guile?

The facility for manipulating Tar? Something else?


Because then, if I understand correctly…

Toggle quote (5 lines)
> diff --git a/guix/docker.scm b/guix/docker.scm
> index 5e6460f43f..f1adad26dc 100644
> --- a/guix/docker.scm
> +++ b/guix/docker.scm

[...]

Toggle quote (5 lines)
> + (if stream-layered-image
> + (let ((input (open-pipe* OPEN_READ "python3"
> + stream-layered-image
> + "config.json")))

…it requires to drag Python for building/packing layered Docker.


Well, I have not really look yet to the Python script which does most of
the job. Do you use a similar strategy as [1]?

And I remember something in that direction by Chris but I am unable to
find back the patch. )-:


Cheers,
simon
P
P
pelzflorian (Florian Pelz) wrote on 13 Mar 22:09 +0100
Re: [bug#62153] [PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)(address . 62153@debbugs.gnu.org)
87cz5ccqa5.fsf@pelzflorian.de
Thank you Oleg for this feature.

Could you change the following three things in the news:

Change the beginning of the English translation from
"Docker layered image can now be produced" to
"Docker layered images can now be produced".

And at the end, also in Russian, switch around these two paragraphs
and add a “the” and reference the System Images chapter:
Toggle quote (7 lines)
> @command{guix system image} can now produce layered Docker image by passing
> @code{docker-layered} to the @option{--image-type} option.
>
> See @command{info \"(guix) Invoking guix pack\"} and
> @command{info \"(guix) System Images\"} for more information.


Lastly, could you then also add a German translation:

(title
(de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")

(body
(de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
angelegt wird:

@example
guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
@end example

@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
übergeben.

Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")

Regards,
Florian
O
O
Oleg Pykhalov wrote on 13 Mar 22:10 +0100
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 62153@debbugs.gnu.org)
87bkkw2w7z.fsf@gmail.com
Hi Simon,

Thank you for the review.

Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (17 lines)
> On lun., 13 mars 2023 at 03:33, Oleg Pykhalov <go.wigust@gmail.com> wrote:
>
>> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
>> new file mode 100644
>> index 0000000000..9ad2168c2d
>> --- /dev/null
>> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
>> @@ -0,0 +1,391 @@
>> +"""
>> +This script generates a Docker image from a set of store paths. Uses
>> +Docker Image Specification v1.2 as reference [1].
>
> Instead of Python, would it possible to implement in Guile? I mean,
> does Python have something that is missing in Guile?
>
> The facility for manipulating Tar? Something else?

I think nothing else. As I understand Python implemented Tar inside the
language itself in 2500 lines of code by manipulating binary data.

/gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py

Technically it's probably possible to use tar utility with --append flag
instead of opening a new file and streaming to it as the Python script
does. To be honest I would like not to write it in this way if the
Python script does not block current patch for merge.

Also I don't see myself writing Tar implementation in Guile, yet. ;-)

The Nix project uses this script heavily to build layered images, so it
should be robust in terms of up to date to current Tar and Python
implementations.

Toggle quote (16 lines)
> Because then, if I understand correctly…
>
>> diff --git a/guix/docker.scm b/guix/docker.scm
>> index 5e6460f43f..f1adad26dc 100644
>> --- a/guix/docker.scm
>> +++ b/guix/docker.scm
>
> [...]
>
>> + (if stream-layered-image
>> + (let ((input (open-pipe* OPEN_READ "python3"
>> + stream-layered-image
>> + "config.json")))
>
> …it requires to drag Python for building/packing layered Docker.

Correct.

Toggle quote (8 lines)
> Well, I have not really look yet to the Python script which does most of
> the job. Do you use a similar strategy as [1]?
>
> And I remember something in that direction by Chris but I am unable to
> find back the patch. )-:
>
> 1: https://grahamc.com/blog/nix-and-layered-docker-images/

Not similar. My patch implements a very simple sorting by size, no
complex sorting by reference popularity as in [1], which is probably
implemented in the following file

github.com/NixOS/nixpkgs/pkgs/build-support/references-by-popularity/closure-graph.py


Toggle quote (27 lines)
> How Docker really represents an Image
>
> Docker’s layers are content addressable and aren’t required to
> explicitly reference a parent layer. This means a layer for
> readline-7.0p5 doesn’t have to mention that it has any relationship to
> ncurses-6.1 or glibc-2.27 at all.
>
> Instead each image has a manifest which defines the order:
>
> {
> "Layers": [
> "bash-interactive-4.4-p23",
> "bash-4.4p23",
> "readline-7.0p5",
> ...
> ]
> }
>
> If you have only built Docker images using a Dockerfile, then you
> would expect the way we flatten our graph to be critically
> important. If we sometimes picked readline-7.0p5 to come first and
> other times picked bash-4.4p23 then we may never make cache hits.
>
> However since the Image defines the order, we don’t have to solve this
> impossible problem: we can order the layers in any way we want and the
> layer cache will always hit.

In case of sorting by size, bigest layers will be on top of a container
image, which will produce a cache hit for bigest directories in the GNU
store during images transfer with same layers.

I would like to say this sorting could binifit more than sorting by
popularity during transfer but let's assume I didn't write it. ;-)

The following example shows common layers between images, which will be
not tranfered if you load image inside Docker as well as pull and push:

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello

and

./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello emacs

share 6 layers in total

Toggle snippet (11 lines)
$ f() { docker image inspect "$1" | jq --raw-output '.[0].RootFS.Layers[] | .' | sort ; }
$ comm -1 -2 --total <(f sha256:fb43b32380a5e6a867410721f4ce2917db14d4ae943c433983afbaf84416c421) <(f sha256:0ce4a11973d1071aeec5441db228d6148dfd09fea3ae77b731c750ebfcc2fe1d)
sha256:3b3daa2a00f1acd12eeb16698bf1caeb6ba6c436e3dbca6259c3a9c622664e00
sha256:5c2be7469293854257221cb6aa8aa4af1e10e2c550935390dbcfeede3d3fbacd
sha256:60317981d94928659389f299e4b86703e5ded420a53537d67627952187fbd3f9
sha256:6d7c8ce5441d4c4c74e0ecff6c203a7b265b37137cca3b0a0ccf10526cfaa6e2
sha256:c2ded2ffe3f46fa7a64a62e0fc6b9d28cb7d4f8d9c64d5a52d137a508cba11fc
sha256:fbcad85d7d3c25bd2aa6d95bb3bf3d02c499ee3b3e443ddd3e5b679c2b33c139
5 94 6 total

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPkWAUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzFZBAApU6Atd2czoZgSRir9SG0/V/l7yva
zKst6JvXL77QxVU+e0QqVcY/o8rJg9hIGH+cQaOmlxmbYXJExEI1Go+tngk61OgQ
dsMrdBBPkaXvqzd7PKQRUtCn221CGfoMMfZGEzCUyQFTYgo6+K7s76Ep6++lODEW
nV/nPa/gqKhg6IZ2G+NeuNKeHsFN8YH/U3Si8myvyLSt0B/ZZEb/8eMumhl3lt61
4GXPPhlFXvp/8VebhMbvzN7TUGWl4z8uMpVExYdDjG3BxkcJBqlEfGfJPQlWc18E
+8G93/JCFYhtE/ae0d/qCpVs8k7CLRrJEkllQFARau+e0GxNggdA08mACmuxZqTF
u1t52FVvg+cgIO5XKN0x0HyR0t+Rv6gkZPA1b4EXF6t8XRCkdh5VXZr/v3Wvwea2
2wbTeX2vDeca3coyCxDNRjka+FgcLrcxxVBZNc33/76sgglXjovQI4g4LDsklkV9
pSa8StEt4/lCSPXRUQYy0g/nUO4Lk2QaIVx7FSefqCaS+EwCz6NkMGi8esKrtHuw
l1gFvYhMFtHvQjzbO50edR1QyLgbMTFbyJdFhHJOEI0rAeh4AIde+R/FfXbPFZR4
RCPwCij2+Ng0O8lrd9uQcdo92oms6FGHLN+ndgYm38hzMQdI+ecMpAQtlR09efsW
WBhi5/HWNf3372o=
=r2xS
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 14 Mar 01:24 +0100
[PATCH 0/2] Add Docker layered image for pack and system (v2)
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-1-go.wigust@gmail.com
Hi, Florian.

Thank you for the review.

This patch series applies your suggestions. Also it's rebased on
origin/master and added a missing documentation for ‘docker-layered’ format in
‘guix system image’ command in doc/guix.texi file (following diff).

<#part type="text/x-patch" buffer=m1.txt disposition=inline description="Add missing docker-layered format documentation for guix system image command">
<#/part>

The folloing tests passed:

make check-channel-news
make check TESTS="tests/pack.scm"
make check-system TESTS="docker-system docker-layered-system"

Oleg Pykhalov (2):
guix: docker: Build layered image.
news: Add entry for the new 'docker-layered' distribution format.

Makefile.am | 3 +-
doc/guix.texi | 18 +-
etc/news.scm | 58 +++
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 105 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
11 files changed, 837 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py


base-commit: 5312d798ac36a72d8a977325a7c6ff7647be670a
--
2.38.0
O
O
Oleg Pykhalov wrote on 14 Mar 01:24 +0100
[PATCH 1/2] guix: docker: Build layered image.
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-2-go.wigust@gmail.com
* gnu/packages/aux-files/python/stream-layered-image.py: New file.
* Makefile.am (AUX_FILES): Add this.
* guix/docker.scm (%docker-image-max-layers): New variable.
(build-docker-image)[stream-layered-image, root-system]: New arguments.
* guix/scripts/pack.scm (stream-layered-image.py): New variable.
(docker-image)[layered-image?]: New argument.
(docker-layered-image): New procedure.
(%formats)[docker-layered]: New format.
(show-formats): Document this.
* tests/pack.scm: Add docker-layered-image + localstatedir test.
* guix/scripts/system.scm
(system-derivation-for-action)[docker-layered-image]: New action.
(show-help): Document this.
(actions)[docker-layered-image]: New action.
(process-action): Add this.
* gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New
variables.
(system-docker-image)[layered-image?]: New argument.
(stream-layered-image.py): New variable.
(system-docker-layered-image): New procedure.
(image->root-file-system)[docker-layered]: New image format.
* gnu/tests/docker.scm (%test-docker-layered-system): New test.
* gnu/image.scm (validate-image-format)[docker-layered]: New image format.
* doc/guix.texi (Invoking guix pack): Document docker-layered format.
(image Reference): Same.
(image-type Reference): Document docker-layered-image-type.
---
Makefile.am | 3 +-
doc/guix.texi | 18 +-
gnu/image.scm | 3 +-
.../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++
gnu/system/image.scm | 84 +++-
gnu/tests/docker.scm | 20 +-
guix/docker.scm | 182 ++++++--
guix/scripts/pack.scm | 105 +++--
guix/scripts/system.scm | 11 +-
tests/pack.scm | 48 +++
10 files changed, 779 insertions(+), 86 deletions(-)
create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py

Toggle diff (518 lines)
diff --git a/Makefile.am b/Makefile.am
index 23b939b674..9aca84f8f8 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -11,7 +11,7 @@
# Copyright © 2017 Arun Isaac <arunisaac@systemreboot.net>
# Copyright © 2018 Nikita <nikita@n0.is>
# Copyright © 2018 Julien Lepiller <julien@lepiller.eu>
-# Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
+# Copyright © 2018, 2023 Oleg Pykhalov <go.wigust@gmail.com>
# Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
# Copyright © 2019 Efraim Flashner <efraim@flashner.co.il>
# Copyright © 2021 Chris Marusich <cmmarusich@gmail.com>
@@ -435,6 +435,7 @@ AUX_FILES = \
gnu/packages/aux-files/python/sanity-check.py \
gnu/packages/aux-files/python/sanity-check-next.py \
gnu/packages/aux-files/python/sitecustomize.py \
+ gnu/packages/aux-files/python/stream-layered-image.py \
gnu/packages/aux-files/renpy/renpy.in \
gnu/packages/aux-files/run-in-namespace.c
diff --git a/doc/guix.texi b/doc/guix.texi
index 39932d5aad..fa4b7586c9 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -56,7 +56,7 @@ Copyright @copyright{} 2017 Andy Wingo@*
Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@*
Copyright @copyright{} 2017 nee@*
Copyright @copyright{} 2018 Rutger Helling@*
-Copyright @copyright{} 2018, 2021 Oleg Pykhalov@*
+Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@*
Copyright @copyright{} 2018 Mike Gerwitz@*
Copyright @copyright{} 2018 Pierre-Antoine Rouby@*
Copyright @copyright{} 2018, 2019 Gábor Boskovits@*
@@ -6837,9 +6837,15 @@ the following command:
guix pack -f docker -S /bin=bin guile guile-readline
@end example
+or
+
+@example
+guix pack -f docker-layered -S /bin=bin guile guile-readline
+@end example
+
@noindent
-The result is a tarball that can be passed to the @command{docker load}
-command, followed by @code{docker run}:
+The result is a tarball with image or layered image that can be passed
+to the @command{docker load} command, followed by @code{docker run}:
@example
docker load < @var{file}
@@ -43274,6 +43280,8 @@ one or multiple partitions.
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
@@ -43605,6 +43613,10 @@ Build an image based on the @code{iso9660-image} image but with the
Build an image based on the @code{docker-image} image.
@end defvar
+@defvar docker-layered-image-type
+Build a layered image based on the @code{docker-layered-image} image.
+@end defvar
+
@defvar raw-with-offset-image-type
Build an MBR image with a single partition starting at a @code{1024KiB}
offset. This is useful to leave some room to install a bootloader in
diff --git a/gnu/image.scm b/gnu/image.scm
index 523653dd77..8a6a0d8479 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020, 2022 Mathieu Othacehe <othacehe@gnu.org>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -152,7 +153,7 @@ (define-with-syntax-properties (name (value properties))
;; The supported image formats.
(define-set-sanitizer validate-image-format format
- (disk-image compressed-qcow2 docker iso9660 tarball wsl2))
+ (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2))
;; The supported partition table types.
(define-set-sanitizer validate-partition-table-type partition-table-type
diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
new file mode 100644
index 0000000000..9ad2168c2d
--- /dev/null
+++ b/gnu/packages/aux-files/python/stream-layered-image.py
@@ -0,0 +1,391 @@
+"""
+This script generates a Docker image from a set of store paths. Uses
+Docker Image Specification v1.2 as reference [1].
+
+It expects a JSON file with the following properties and writes the
+image as an uncompressed tarball to stdout:
+
+* "architecture", "config", "os", "created", "repo_tag" correspond to
+ the fields with the same name on the image spec [2].
+* "created" can be "now".
+* "created" is also used as mtime for files added to the image.
+* "store_layers" is a list of layers in ascending order, where each
+ layer is the list of store paths to include in that layer.
+
+The main challenge for this script to create the final image in a
+streaming fashion, without dumping any intermediate data to disk
+for performance.
+
+A docker image has each layer contents archived as separate tarballs,
+and they later all get enveloped into a single big tarball in a
+content addressed fashion. However, because how "tar" format works,
+we have to know about the name (which includes the checksum in our
+case) and the size of the tarball before we can start adding it to the
+outer tarball. We achieve that by creating the layer tarballs twice;
+on the first iteration we calculate the file size and the checksum,
+and on the second one we actually stream the contents. 'add_layer_dir'
+function does all this.
+
+[1]: https://github.com/moby/moby/blob/master/image/spec/v1.2.md
+[2]: https://github.com/moby/moby/blob/4fb59c20a4fb54f944fe170d0ff1d00eb4a24d6f/image/spec/v1.2.md#image-json-field-descriptions
+""" # noqa: E501
+
+
+import io
+import os
+import re
+import sys
+import json
+import hashlib
+import pathlib
+import tarfile
+import itertools
+import threading
+from datetime import datetime, timezone
+from collections import namedtuple
+
+
+def archive_paths_to(obj, paths, mtime):
+ """
+ Writes the given store paths as a tar file to the given stream.
+
+ obj: Stream to write to. Should have a 'write' method.
+ paths: List of store paths.
+ """
+
+ # gettarinfo makes the paths relative, this makes them
+ # absolute again
+ def append_root(ti):
+ ti.name = "/" + ti.name
+ return ti
+
+ def apply_filters(ti):
+ ti.mtime = mtime
+ ti.uid = 0
+ ti.gid = 0
+ ti.uname = "root"
+ ti.gname = "root"
+ return ti
+
+ def nix_root(ti):
+ ti.mode = 0o0555 # r-xr-xr-x
+ return ti
+
+ def dir(path):
+ ti = tarfile.TarInfo(path)
+ ti.type = tarfile.DIRTYPE
+ return ti
+
+ with tarfile.open(fileobj=obj, mode="w|") as tar:
+ # To be consistent with the docker utilities, we need to have
+ # these directories first when building layer tarballs.
+ tar.addfile(apply_filters(nix_root(dir("/gnu"))))
+ tar.addfile(apply_filters(nix_root(dir("/gnu/store"))))
+
+ for path in paths:
+ path = pathlib.Path(path)
+ if path.is_symlink():
+ files = [path]
+ else:
+ files = itertools.chain([path], path.rglob("*"))
+
+ for filename in sorted(files):
+ ti = append_root(tar.gettarinfo(filename))
+
+ # copy hardlinks as regular files
+ if ti.islnk():
+ ti.type = tarfile.REGTYPE
+ ti.linkname = ""
+ ti.size = filename.stat().st_size
+
+ ti = apply_filters(ti)
+ if ti.isfile():
+ with open(filename, "rb") as f:
+ tar.addfile(ti, f)
+ else:
+ tar.addfile(ti)
+
+
+class ExtractChecksum:
+ """
+ A writable stream which only calculates the final file size and
+ sha256sum, while discarding the actual contents.
+ """
+
+ def __init__(self):
+ self._digest = hashlib.sha256()
+ self._size = 0
+
+ def write(self, data):
+ self._digest.update(data)
+ self._size += len(data)
+
+ def extract(self):
+ """
+ Returns: Hex-encoded sha256sum and size as a tuple.
+ """
+ return (self._digest.hexdigest(), self._size)
+
+
+FromImage = namedtuple("FromImage", ["tar", "manifest_json", "image_json"])
+# Some metadata for a layer
+LayerInfo = namedtuple("LayerInfo", ["size", "checksum", "path", "paths"])
+
+
+def load_from_image(from_image_str):
+ """
+ Loads the given base image, if any.
+
+ from_image_str: Path to the base image archive.
+
+ Returns: A 'FromImage' object with references to the loaded base image,
+ or 'None' if no base image was provided.
+ """
+ if from_image_str is None:
+ return None
+
+ base_tar = tarfile.open(from_image_str)
+
+ manifest_json_tarinfo = base_tar.getmember("manifest.json")
+ with base_tar.extractfile(manifest_json_tarinfo) as f:
+ manifest_json = json.load(f)
+
+ image_json_tarinfo = base_tar.getmember(manifest_json[0]["Config"])
+ with base_tar.extractfile(image_json_tarinfo) as f:
+ image_json = json.load(f)
+
+ return FromImage(base_tar, manifest_json, image_json)
+
+
+def add_base_layers(tar, from_image):
+ """
+ Adds the layers from the given base image to the final image.
+
+ tar: 'tarfile.TarFile' object for new layers to be added to.
+ from_image: 'FromImage' object with references to the loaded base image.
+ """
+ if from_image is None:
+ print("No 'fromImage' provided", file=sys.stderr)
+ return []
+
+ layers = from_image.manifest_json[0]["Layers"]
+ checksums = from_image.image_json["rootfs"]["diff_ids"]
+ layers_checksums = zip(layers, checksums)
+
+ for num, (layer, checksum) in enumerate(layers_checksums, start=1):
+ layer_tarinfo = from_image.tar.getmember(layer)
+ checksum = re.sub(r"^sha256:", "", checksum)
+
+ tar.addfile(layer_tarinfo, from_image.tar.extractfile(layer_tarinfo))
+ path = layer_tarinfo.path
+ size = layer_tarinfo.size
+
+ print("Adding base layer", num, "from", path, file=sys.stderr)
+ yield LayerInfo(size=size, checksum=checksum, path=path, paths=[path])
+
+ from_image.tar.close()
+
+
+def overlay_base_config(from_image, final_config):
+ """
+ Overlays the final image 'config' JSON on top of selected defaults from the
+ base image 'config' JSON.
+
+ from_image: 'FromImage' object with references to the loaded base image.
+ final_config: 'dict' object of the final image 'config' JSON.
+ """
+ if from_image is None:
+ return final_config
+
+ base_config = from_image.image_json["config"]
+
+ # Preserve environment from base image
+ final_env = base_config.get("Env", []) + final_config.get("Env", [])
+ if final_env:
+ # Resolve duplicates (last one wins) and format back as list
+ resolved_env = {entry.split("=", 1)[0]: entry for entry in final_env}
+ final_config["Env"] = list(resolved_env.values())
+ return final_config
+
+
+def add_layer_dir(tar, paths, store_dir, mtime):
+ """
+ Appends given store paths to a TarFile object as a new layer.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ paths: List of store paths.
+ store_dir: the root directory of the nix store
+ mtime: 'mtime' of the added files and the layer tarball.
+ Should be an integer representing a POSIX time.
+
+ Returns: A 'LayerInfo' object containing some metadata of
+ the layer added.
+ """
+
+ invalid_paths = [i for i in paths if not i.startswith(store_dir)]
+ assert len(invalid_paths) == 0, \
+ f"Expecting absolute paths from {store_dir}, but got: {invalid_paths}"
+
+ # First, calculate the tarball checksum and the size.
+ extract_checksum = ExtractChecksum()
+ archive_paths_to(
+ extract_checksum,
+ paths,
+ mtime=mtime,
+ )
+ (checksum, size) = extract_checksum.extract()
+
+ path = f"{checksum}/layer.tar"
+ layer_tarinfo = tarfile.TarInfo(path)
+ layer_tarinfo.size = size
+ layer_tarinfo.mtime = mtime
+
+ # Then actually stream the contents to the outer tarball.
+ read_fd, write_fd = os.pipe()
+ with open(read_fd, "rb") as read, open(write_fd, "wb") as write:
+ def producer():
+ archive_paths_to(
+ write,
+ paths,
+ mtime=mtime,
+ )
+ write.close()
+
+ # Closing the write end of the fifo also closes the read end,
+ # so we don't need to wait until this thread is finished.
+ #
+ # Any exception from the thread will get printed by the default
+ # exception handler, and the 'addfile' call will fail since it
+ # won't be able to read required amount of bytes.
+ threading.Thread(target=producer).start()
+ tar.addfile(layer_tarinfo, read)
+
+ return LayerInfo(size=size, checksum=checksum, path=path, paths=paths)
+
+
+def add_customisation_layer(target_tar, customisation_layer, mtime):
+ """
+ Adds the customisation layer as a new layer. This is layer is structured
+ differently; given store path has the 'layer.tar' and corresponding
+ sha256sum ready.
+
+ tar: 'tarfile.TarFile' object for the new layer to be added to.
+ customisation_layer: Path containing the layer archive.
+ mtime: 'mtime' of the added layer tarball.
+ """
+
+ checksum_path = os.path.join(customisation_layer, "checksum")
+ with open(checksum_path) as f:
+ checksum = f.read().strip()
+ assert len(checksum) == 64, f"Invalid sha256 at ${checksum_path}."
+
+ layer_path = os.path.join(customisation_layer, "layer.tar")
+
+ path = f"{checksum}/layer.tar"
+ tarinfo = target_tar.gettarinfo(layer_path)
+ tarinfo.name = path
+ tarinfo.mtime = mtime
+
+ with open(layer_path, "rb") as f:
+ target_tar.addfile(tarinfo, f)
+
+ return LayerInfo(
+ size=None,
+ checksum=checksum,
+ path=path,
+ paths=[customisation_layer]
+ )
+
+
+def add_bytes(tar, path, content, mtime):
+ """
+ Adds a file to the tarball with given path and contents.
+
+ tar: 'tarfile.TarFile' object.
+ path: Path of the file as a string.
+ content: Contents of the file.
+ mtime: 'mtime' of the file. Should be an integer representing a POSIX time.
+ """
+ assert type(content) is bytes
+
+ ti = tarfile.TarInfo(path)
+ ti.size = len(content)
+ ti.mtime = mtime
+ tar.addfile(ti, io.BytesIO(content))
+
+
+def main():
+ with open(sys.argv[1], "r") as f:
+ conf = json.load(f)
+
+ created = (
+ datetime.now(tz=timezone.utc)
+ if conf["created"] == "now"
+ else datetime.fromisoformat(conf["created"])
+ )
+ mtime = int(created.timestamp())
+ store_dir = conf["store_dir"]
+
+ from_image = load_from_image(conf["from_image"])
+
+ with tarfile.open(mode="w|", fileobj=sys.stdout.buffer) as tar:
+ layers = []
+ layers.extend(add_base_layers(tar, from_image))
+
+ start = len(layers) + 1
+ for num, store_layer in enumerate(conf["store_layers"], start=start):
+ print("Creating layer", num, "from paths:", store_layer,
+ file=sys.stderr)
+ info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime)
+ layers.append(info)
+
+ print("Creating layer", len(layers) + 1, "with customisation...",
+ file=sys.stderr)
+ layers.append(
+ add_customisation_layer(
+ tar,
+ conf["customisation_layer"],
+ mtime=mtime
+ )
+ )
+
+ print("Adding manifests...", file=sys.stderr)
+
+ image_json = {
+ "created": datetime.isoformat(created),
+ "architecture": conf["architecture"],
+ "os": "linux",
+ "config": overlay_base_config(from_image, conf["config"]),
+ "rootfs": {
+ "diff_ids": [f"sha256:{layer.checksum}" for layer in layers],
+ "type": "layers",
+ },
+ "history": [
+ {
+ "created": datetime.isoformat(created),
+ "comment": f"store paths: {layer.paths}"
+ }
+ for layer in layers
+ ],
+ }
+
+ image_json = json.dumps(image_json, indent=4).encode("utf-8")
+ image_json_checksum = hashlib.sha256(image_json).hexdigest()
+ image_json_path = f"{image_json_checksum}.json"
+ add_bytes(tar, image_json_path, image_json, mtime=mtime)
+
+ manifest_json = [
+ {
+ "Config": image_json_path,
+ "RepoTags": [conf["repo_tag"]],
+ "Layers": [layer.path for layer in layers],
+ }
+ ]
+ manifest_json = json.dumps(manifest_json, indent=4).encode("utf-8")
+ add_bytes(tar, "manifest.json", manifest_json, mtime=mtime)
+
+ print("Done.", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index afef79185f..0bfd011ad4 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -4,6 +4,7 @@
;;; Copyright © 2022 Pavel Shlyak <p.shlyak@pantherx.org>
;;; Copyright © 2022 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
;;; Copyright © 2022 Alex Griffin <a@ajgrf.com>
+;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -45,6 +46,7 @@ (define-module (gnu system image)
#:use-module (gnu system uuid)
#:use-module (gnu system vm)
#:use-module (guix packages)
+ #:use-module ((gnu packages) #:select (search-auxiliary-file))
#:use-module (gnu packages base)
#:use-module (gnu packages bash)
#:use-module (gnu packages bootloaders)
@@ -58,6 +60,7 @@ (define-module (gnu system image)
#:use-module (gnu packages hurd)
#:use-module (gnu packages linux)
#:use-module (gnu packages mtools)
+ #:use-module (gnu packages python)
#:use-module (gnu packages virtualization)
#:use-module ((srfi srfi-1) #:prefix srfi-1:)
#:use-module (srfi srfi-11)
@@ -78,6 +81,7 @@ (define-module (gnu system
This message was truncated. Download the full message here.
O
O
Oleg Pykhalov wrote on 14 Mar 01:24 +0100
[PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format.
(address . pelzflorian@pelzflorian.de)
20230314002453.24668-3-go.wigust@gmail.com
* etc/news.scm: Add entry.
---
etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

Toggle diff (78 lines)
diff --git a/etc/news.scm b/etc/news.scm
index 55d1218df5..4bbdfd2a59 100644
--- a/etc/news.scm
+++ b/etc/news.scm
@@ -18,6 +18,7 @@
;; Copyright © 2021 Andrew Tropin <andrew@trop.in>
;; Copyright © 2021 Jonathan Brielmaier <jonathan.brielmaier@web.de>
;; Copyright © 2022 Thiago Jung Bauermann <bauermann@kolabnow.com>
+;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
;;
;; Copying and distribution of this file, with or without modification, are
;; permitted in any medium without royalty provided the copyright notice and
@@ -26,6 +27,63 @@
(channel-news
(version 0)
+ (entry (commit "a5c3baf510adab1f5b3bb855b1aa9cafe3cb66b9")
+ (title
+ (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}")
+ (en "New @samp{docker-layered} format for the @command{guix pack} command")
+ (ru "Новый @samp{docker-layered} формат для @command{guix pack} команды"))
+ (body
+ (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl
+@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein
+Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“)
+untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird
+im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches
+Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello}
+angelegt wird:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen,
+indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type}
+übergeben.
+
+Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und
+@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.")
+ (en "Docker layered images can now be produced via the @command{guix
+pack --format=docker-layered} command, providing a Docker image with many of
+the store paths being on their own layer to improve sharing between images.
+The image is realized into the GNU store as a gzipped tarball. Here is a
+simple example that generates a layered Docker image for the @code{hello}
+package:
+
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+The @command{guix system image} can now produce layered Docker image by passing
+@code{docker-layered} to @option{--image-type} option.
+
+See @command{info \"(guix) Invoking guix pack\"} and
+@command{info \"(guix) System Images\"} for more information.")
+ (ru "Появилась команда создания многослойных Docker образов с помощью
+@command{guix pack --format=docker-layered}, которая соберет Docker образ с
+путями в store расположенными на отдельных слоях, ускоряя таким образом
+передачу образов. Образ будет создан в GNU store в качестве gzipped tarball.
+
+Пример создания Docker layered образ с @code{hello} пакетом:
+@example
+guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello
+@end example
+
+@command{guix system image} теперь может создавать layered Docker образ путем
+указания в опции @option{--image-type} параметра @code{docker-layered}.
+
+Смотрите @command{info \"(guix) Invoking guix pack\"} и
+@command{info \"(guix) System Images\"} для получения более детальных
+сведений.")))
+
(entry (commit "0e18c5e5bcb9204c278cfc75493d3b02b746d5c3")
(title
(en "Linux-libre kernel updated to 6.2")
--
2.38.0
O
O
Oleg Pykhalov wrote on 14 Mar 01:40 +0100
Missing diff in cover lever for v2 patch
(address . 62153@debbugs.gnu.org)
87wn3k17yf.fsf@gmail.com
Missing diff in cover leter for v2 patch attached below.

Toggle quote (4 lines)
> This patch series applies your suggestions. Also it's rebased on
> origin/master and added a missing documentation for ‘docker-layered’ format in
> ‘guix system image’ command in doc/guix.texi file (following diff).

Toggle snippet (14 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index bd0ee126ee..6938743154 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -43306,6 +43306,8 @@ one or multiple partitions.
@item @code{docker}, a Docker image.
+@item @code{docker-layered}, a layered Docker image.
+
@item @code{iso9660}, an ISO-9660 image.
@item @code{tarball}, a tar.gz image archive.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPwngUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pw5aw/+JeyQ3p6e4cfIX6+RHnDjUgP9HA2Q
ESTfjzWxsJcCxleL0LKO3AbOUVby/qq4TrUgNSm1cqL/S4vWDAIo34XEyT5Xerdg
oHSNtoaHqD467NsGhSvtdS8TYGe7JLTTr76nkNzwjQAGjDYXzJXvrcr758H4VMw0
6k143Q1uqDYj3+E2uqQWT9wme5oZ4IkxzZW/+XCME3s00yBF19Uti9VqROLAEIhV
OnQAuurDYScfZ7in17DMuyHxOfrJm4M2G7UxRSiVnKbBeGjbaQoL4+QSRCRa+/+O
0VABMVnYFYUqQPFKm+Y7csQiFbjE6mojc3nNwqBEd0i3rUwu4vPjad6mirKUC9FQ
C3gpBDAIVNJ6I0UWmmBdRH/p/hNRvtiAw0FAhBaV0arTsWoLMeybmZjoxDF533zo
wg7ZPbaZtJd8nHowzKsdbXYqYEf1bhio1goBcnraGJdgBFscSsT5eJhk/Tgp9n3a
yQB9NyF4E/Uc7WLzSiscwaxU2ksg+W1s9xZVrz5xcbu1yX8+yI9u+dSp6e6fa7qf
VLyYOGaD7Qju5alCPc6p4z12/a+DD5ZAEJeBReYoJCfEEPjfOClrwHtnJksQyPY/
PlCL+blYcC4yrMQgQN7UeWbDWTEGzwJPHIWgOMzR/meYOQO0TZ1yFj865E9VmyfF
TjXACct5ZaD5Y64=
=otdV
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 14 Mar 09:19 +0100
Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image.
(name . Oleg Pykhalov)(address . go.wigust@gmail.com)
86cz5b923c.fsf@gmail.com
Hi Oleg,

CC: core teams

On Tue, 14 Mar 2023 at 00:10, Oleg Pykhalov <go.wigust@gmail.com> wrote:

Toggle quote (25 lines)
>>> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py
>>> new file mode 100644
>>> index 0000000000..9ad2168c2d
>>> --- /dev/null
>>> +++ b/gnu/packages/aux-files/python/stream-layered-image.py
>>> @@ -0,0 +1,391 @@
>>> +"""
>>> +This script generates a Docker image from a set of store paths. Uses
>>> +Docker Image Specification v1.2 as reference [1].
>>
>> Instead of Python, would it possible to implement in Guile? I mean,
>> does Python have something that is missing in Guile?
>>
>> The facility for manipulating Tar? Something else?
>
> I think nothing else. As I understand Python implemented Tar inside the
> language itself in 2500 lines of code by manipulating binary data.
>
> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>
> Technically it's probably possible to use tar utility with --append flag
> instead of opening a new file and streaming to it as the Python script
> does. To be honest I would like not to write it in this way if the
> Python script does not block current patch for merge.

Ok, thanks for explaining.

Toggle quote (2 lines)
> Also I don't see myself writing Tar implementation in Guile, yet. ;-)

Maybe not reimplementing Tar in Guile, maybe just enough for working.
Or maybe some Guile bindings. Or maybe something is already around for
the bootstrap story.

The use of external tools as Python for producing built-in Guix feature
will be the first time, no?

For what it is worth, I would prefer to consider the options before
emitting an opinion about dragging Python building/packing layered
Docker. :-)

Toggle quote (4 lines)
> The Nix project uses this script heavily to build layered images, so it
> should be robust in terms of up to date to current Tar and Python
> implementations.

Do you mean this script is coming from Nix. Well, in all cases, this
script is not trivial and so it requires Copyright for authorship.



[...]


Toggle quote (3 lines)
> The following example shows common layers between images, which will be
> not tranfered if you load image inside Docker as well as pull and push:

Thanks for explaining.


Cheers,
simon

PS: I will be off-line these 2-3 next weeks. So a lack of an answer
from me will not be a lack of interest. ;-)
C
C
Christopher Baines wrote on 14 Mar 10:11 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
8735671yu4.fsf@cbaines.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (3 lines)
> And I remember something in that direction by Chris but I am unable to
> find back the patch. )-:

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmQQOmNfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XeUkRAApwCv6Zh8MrAuDE5TRebOf0IJflIq98wc
iB1i643fUvGZzVsr/pjAXsrkKDqyBdPH9Oee/JHvPPtEcCVGJwwAtodiGC6Tnf1t
8eYzWSXdzYVK7oFReQZ/oHLzmA0zQYVTmcnIoqP4wy+Cgv/zE3fGWlLrpiIau2hV
NgPUo4wPdZu8ZwM0B2+gK2H4I5pNXUxhIuWFI8rORU6q0nxL/OqrDMh5G/+B1UPh
cuGHBOxoBx3JERUnaHkk0BkgS1TySYpaVDA/9RwtCeqV8cv0bSjznkOqdtusr0vK
prM5WJwzOpdlHkAqwkmEEuE93KjdAoMl5gmYWWlUobCdAtwLNLTXCGa+wAlx7VmW
kFl0XygIJCeXHICtDhiBkv/cfIVSpKYVbqJExRAhzeH6Ykllww6UUaEHC32Fy0ae
WsWUUpgpRGZSgbT70i05D8TaPBjNgIBUV5HfczWTsRgvR1b4fVaqrKaxCooy1gV1
YjIJO7ddNHGCSNp0uxGyMzZMmpEy46gQvSiUNTW46AYgl2c5TEVMaQ/gLNvgmAvt
K+PRQhOaHyoZNLLCIZaRhmuPwbwKEh3MLOGcR/AUF7nW5sp2wk9LBm889lDmarpz
bHk5ZfhqiZryeo36k9O3L3DE70h+fibm1uKr7zn6jRt74LmvGdWA4lCWaz4A9USE
VjZkN4ayEaQ=
=9k/e
-----END PGP SIGNATURE-----

R
R
Ricardo Wurmus wrote on 14 Mar 10:15 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
874jqnzo8a.fsf@elephly.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (23 lines)
>>> Instead of Python, would it possible to implement in Guile? I mean,
>>> does Python have something that is missing in Guile?
>>>
>>> The facility for manipulating Tar? Something else?
>>
>> I think nothing else. As I understand Python implemented Tar inside the
>> language itself in 2500 lines of code by manipulating binary data.
>>
>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>
>> Technically it's probably possible to use tar utility with --append flag
>> instead of opening a new file and streaming to it as the Python script
>> does. To be honest I would like not to write it in this way if the
>> Python script does not block current patch for merge.
>
> Ok, thanks for explaining.
>
>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>
> Maybe not reimplementing Tar in Guile, maybe just enough for working.
> Or maybe some Guile bindings. Or maybe something is already around for
> the bootstrap story.

gash-utils has (gash ustar); it’s about 620 lines of code.

--
Ricardo
L
L
Ludovic Courtès wrote on 16 Mar 11:37 +0100
(name . Ricardo Wurmus)(address . rekado@elephly.net)
877cvhhth5.fsf@gnu.org
Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

Toggle quote (27 lines)
> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
>>>> Instead of Python, would it possible to implement in Guile? I mean,
>>>> does Python have something that is missing in Guile?
>>>>
>>>> The facility for manipulating Tar? Something else?
>>>
>>> I think nothing else. As I understand Python implemented Tar inside the
>>> language itself in 2500 lines of code by manipulating binary data.
>>>
>>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>>
>>> Technically it's probably possible to use tar utility with --append flag
>>> instead of opening a new file and streaming to it as the Python script
>>> does. To be honest I would like not to write it in this way if the
>>> Python script does not block current patch for merge.
>>
>> Ok, thanks for explaining.
>>
>>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>>
>> Maybe not reimplementing Tar in Guile, maybe just enough for working.
>> Or maybe some Guile bindings. Or maybe something is already around for
>> the bootstrap story.
>
> gash-utils has (gash ustar); it’s about 620 lines of code.

Disarchive also has a tar implementation. No excuse! :-)

Oleg, could you check which of these would satisfy your needs? I had a
plan to improve the tar implementation in Gash-Utils, perhaps it’s a
good time to get my act together.

Thanks,
Ludo’.
O
O
Oleg Pykhalov wrote on 20 Mar 07:38 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)
87h6ugapwh.fsf@gmail.com
Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (35 lines)
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Simon Tournier <zimon.toutoune@gmail.com> writes:
>>
>>>>> Instead of Python, would it possible to implement in Guile? I mean,
>>>>> does Python have something that is missing in Guile?
>>>>>
>>>>> The facility for manipulating Tar? Something else?
>>>>
>>>> I think nothing else. As I understand Python implemented Tar inside the
>>>> language itself in 2500 lines of code by manipulating binary data.
>>>>
>>>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py
>>>>
>>>> Technically it's probably possible to use tar utility with --append flag
>>>> instead of opening a new file and streaming to it as the Python script
>>>> does. To be honest I would like not to write it in this way if the
>>>> Python script does not block current patch for merge.
>>>
>>> Ok, thanks for explaining.
>>>
>>>> Also I don't see myself writing Tar implementation in Guile, yet. ;-)
>>>
>>> Maybe not reimplementing Tar in Guile, maybe just enough for working.
>>> Or maybe some Guile bindings. Or maybe something is already around for
>>> the bootstrap story.
>>
>> gash-utils has (gash ustar); it’s about 620 lines of code.
>
> Disarchive also has a tar implementation. No excuse! :-)
>
> Oleg, could you check which of these would satisfy your needs? I had a
> plan to improve the tar implementation in Gash-Utils, perhaps it’s a
> good time to get my act together.

Gash-Utils should work, e.g. [1] script. It's already possible to
rewrite with Gash-Utils, but at least write-ustar-file and
write-ustar-footer should be exported in ustar.scm.

Disarchive requires to write a file specification in case of using
disarchive-assemble. And disarchive-assemble does not work, if I don't
miss anything [2]. Also gzip compression does not work in a Guile REPL.

[1]:

Toggle snippet (37 lines)
#!/usr/bin/env -S guile --no-auto-compile -e main -s
!#

(set! %load-path
(append '("/gnu/store/...-gash-utils-0.1.0/share/guile/site/3.0"
"/gnu/store/...-gash-0.2.0/share/guile/site/3.0")
%load-path))

(set! %load-compiled-path
(append '("/gnu/store/...-gash-utils-0.1.0/lib/guile/3.0/site-ccache"
"/gnu/store/...-gash-0.2.0/lib/guile/3.0/site-ccache")
%load-compiled-path))

(use-modules (gash ustar)
(srfi srfi-26)
(guix build utils))

(define write-ustar-file
(@@ (gash ustar) write-ustar-file))

(define write-ustar-footer
(@@ (gash ustar) write-ustar-footer))

(define (main . args)
(call-with-port (open-file "out.tar.gz" "wb")
(lambda (port)
(with-directory-excursion "."
(call-with-compressed-output-port 'gzip port
(cut write-ustar-file <> "Makefile.am"
#:verbosity 0)))
(with-directory-excursion "doc"
(call-with-compressed-output-port 'gzip port
(cut write-ustar-file <> "."
#:verbosity 0)))
(write-ustar-footer port))))

[2]:

Toggle snippet (263 lines)
$ guile
,m(disarchive assemblers tarball)
(assemble-tarball (disassemble-tarball "out.tar") "result/out.tar")

# Generated tarball:
$ tar tf result/out.tar/sha256/1e7100029373723df900712d1191be0bad5beadf752f367bb264fab891c36356
./
./images/
./images/bootstrap-graph.dot
./images/service-graph.pdf
./guix-cookbook.zh_Hans.texi
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

# Original tarball:
$ tar tf out.tar
./
./images/
./images/bootstrap-graph.dot
./images/bootstrap-graph.eps
./images/bootstrap-graph.pdf
./images/bootstrap-graph.png
./images/bootstrap-packages.dot
./images/bootstrap-packages.eps
./images/bootstrap-packages.pdf
./images/bootstrap-packages.png
./images/coreutils-bag-graph.dot
./images/coreutils-bag-graph.eps
./images/coreutils-bag-graph.pdf
./images/coreutils-bag-graph.png
./images/coreutils-graph.dot
./images/coreutils-graph.eps
./images/coreutils-graph.pdf
./images/coreutils-graph.png
./images/coreutils-size-map.eps
./images/coreutils-size-map.png
./images/gcc-core-mesboot0-graph.dot
./images/gcc-core-mesboot0-graph.eps
./images/gcc-core-mesboot0-graph.pdf
./images/gcc-core-mesboot0-graph.png
./images/installer-network.png
./images/installer-partitions.png
./images/installer-resume.png
./images/service-graph.dot
./images/service-graph.eps
./images/service-graph.pdf
./images/service-graph.png
./images/shepherd-graph.dot
./images/shepherd-graph.eps
./images/shepherd-graph.pdf
./images/shepherd-graph.png
./.dirstamp
./environment-gdb.scm
./fdl-1.3.texi
./os-config-bare-bones.texi
./os-config-desktop.texi
./os-config-lightweight-desktop.texi
./package-hello.json
./package-hello.scm
./stamp-1
./stamp-10
./stamp-11
./stamp-2
./stamp-3
./stamp-4
./stamp-5
./stamp-6
./stamp-7
./stamp-8
./stamp-9
./stamp-vti
./contributing.fa.texi
./guix.fa.texi
./contributing.fi.texi
./guix.fi.texi
./contributing.it.texi
./guix.it.texi
./contributing.ko.texi
./guix.ko.texi
./contributing.sk.texi
./guix.sk.texi
./guix-cookbook.es.texi
./guix-cookbook.fa.texi
./guix-cookbook.fi.texi
./guix-cookbook.pt_BR.texi
./guix-cookbook.ru.texi
./guix-cookbook.zh_Hans.texi
./version-fa.texi
./guix.fa.info-1
./guix.fa.info-2
./guix.fa.info-3
./guix.fa.info-4
./guix.fa.info-5
./guix.fa.info-6
./guix.fa.info
./version-fi.texi
./guix.fi.info-1
./guix.fi.info-2
./guix.fi.info-3
./guix.fi.info-4
./guix.fi.info-5
./guix.fi.info-6
./guix.fi.info
./version-it.texi
./guix.it.info-1
./guix.it.info-2
./guix.it.info-3
./guix.it.info-4
./guix.it.info-5
./guix.it.info-6
./guix.it.info
./version-ko.texi
./guix.ko.info-1
./guix.ko.info-2
./guix.ko.info-3
./guix.ko.info-4
./guix.ko.info-5
./guix.ko.info-6
./guix.ko.info
./version-sk.texi
./guix.sk.info-1
./guix.sk.info-2
./guix.sk.info-3
./guix.sk.info-4
./guix.sk.info-5
./guix.sk.info-6
./guix.sk.info
./guix-cookbook.es.info
./guix-cookbook.fa.info
./guix-cookbook.fi.info
./guix-cookbook.pt_BR.info
./guix-cookbook.ru.info
./guix-cookbook.zh_Hans.info
./he-config-bare-bones.scm
./guix-lint.1
./build.scm
./contributing.texi
./guix-cookbook.texi
./htmlxref.cnf
./local.mk
./guix-cookbook.info
./guix-gc.1
./guix-git.1
./guix-container.1
./guix-copy.1
./guix-describe.1
./guix-processes.1
./guix-size.1
./guix-weather.1
./guix-shell.1
./guix-cookbook.ko.texi
./guix-cookbook.fr.texi
./guix-cookbook.de.texi
./guix-cookbook.sk.texi
./contributing.zh_CN.texi
./contributing.ru.texi
./contributing.pt_BR.texi
./contributing.fr.texi
./contributing.es.texi
./contributing.de.texi
./guix.zh_CN.texi
./guix.ru.texi
./guix.pt_BR.texi
./guix.de.texi
./guix.es.texi
./guix.fr.texi
./guix-cookbook.sk.info
./guix-cookbook.de.info
./guix-cookbook.fr.info
./guix-cookbook.ko.info
./version-de.texi
./version-ru.texi
./version-fr.texi
./version-es.texi
./version-pt_BR.texi
./version-zh_CN.texi
./guix-daemon.1
./guix.pt_BR.info-1
./guix.pt_BR.info-2
./guix.fr.info-1
./guix.fr.info-2
./guix.de.info-1
./guix.de.info-2
./guix.pt_BR.info-3
./guix.ru.info-1
./guix.ru.info-2
./guix.fr.info-3
./guix.ru.info-3
./guix.zh_CN.info-1
./guix.zh_CN.info-2
./guix.es.info-1
./guix.es.info-2
./guix.de.info-3
./guix.pt_BR.info-4
./guix.fr.info-4
./guix.ru.info-4
./guix.zh_CN.info-3
./guix.de.info-4
./guix.es.info-3
./guix.ru.info-5
./guix.pt_BR.info-5
./guix.zh_CN.info-4
./guix.fr.info-5
./guix.es.info-4
./guix.de.info-5
./guix.pt_BR.info-6
./guix.ru.info-6
./guix.fr.info-6
./guix.pt_BR.info-7
./guix.es.info-5
./guix.zh_CN.info-5
./guix.de.info-6
./guix.pt_BR.info
./guix.ru.info-7
./guix.fr.info-7
./guix.ru.info-8
./guix.de.info-7
./guix.es.info-6
./guix.fr.info-8
./guix.zh_CN.info-6
./guix.ru.info
./guix.fr.info
./guix.de.info-8
./guix.zh_CN.info-7
./guix.es.info-7
./guix.de.info
./guix.zh_CN.info
./guix.es.info-8
./guix.es.info
./guix.texi
./version.texi
./guix.info-1
./guix.info-2
./guix.info-3
./guix.info-4
./guix.info-5
./guix.info-6
./guix.info-7
./guix.info
./guix-build.1
./guix-archive.1
./guix-import.1
./guix-offload.1
./guix-graph.1
./guix-hash.1
./guix-challenge.1
./guix-download.1
./guix-edit.1
./guix-repl.1
./guix-publish.1
./guix-package.1
./guix-style.1
./guix-pull.1
./guix-time-machine.1
./guix-environment.1
./guix-pack.1
./guix-deploy.1
./guix-refresh.1
./guix-home.1
./guix-system.1
./guix.1

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQX/18UHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pyHCg//Y+CwbxC5OzVdZFBdSDKYtxOV2w6Z
w6X/ZXRt5sorx2Wfjsn0QJIzO9+6CTEBn1RSeOCQxuVvlQqjdeKqxQ0C90YQQG5t
/R7PZ5wT2LHhztT1R52O8njqj93bppxkJDMsZVg4lg38WzNt2jVvL94mczS1V3zH
4VpccvDHYjIwK/ECkhTEKHRvRSGFqH7VcGJYbKYwGjmhF1Q6+lWq4XBiBhF9C6G0
+TwsS/GMXi207dFEtXdhit6DcxbEi70EseNj55aLDoM9xUumlQJuVTz5f9FRbL/C
QOQmI2xFMPqN2EwZhBX3YL7FjwooA+nyyrj7H0AjWe3IX4RuVyvF9XZNIm1HM4P3
VlcR1foBdKxo1JRCTCU7V/hlD44Yl3e/FXQbKIS6aQyhRahrudYjeeL1gf7jBRya
FAzkv2kP/ORoxP/bKj/BoNV372fOz8Sm5sSr5YzAZJz4DJQmctstIw+6WFcgP8Ca
az682R4sMLDWteAxhZp/NNSg21irM6S7U4Tu4em3nP69ky2Fs5wUy7m1JzMWx9Vg
IrODYJdDN0P5wgSM5hVKAW30J8uLXs3XxwRAl2W+NKCu8l4eonnpKU3KufaSclG+
Qz7cg6xNqTLzvDrYoDaaMhGd2WGJOdQ3wzH3TcWgR9Kh1STNwNnYfXXsZPvYQ5wi
QWmUkZU8RZRQ3BI=
=rP1f
-----END PGP SIGNATURE-----

O
O
Oleg Pykhalov wrote on 20 Mar 17:51 +0100
Re: bug#62153: [PATCH 0/2] Disarchive vs Gash-Utils for docker-layered
(name . Ludovic Courtès)(address . ludo@gnu.org)
87cz53bc2f.fsf_-_@gmail.com
Oleg Pykhalov <go.wigust@gmail.com> writes:

Toggle quote (4 lines)
> […]
>
> And disarchive-assemble does not work, if I don't miss anything.

I forgot about that input tarball was generated with changing current
working directory, that's probably the reason of the broken archive.

According to "Assembles from stdin to file" test from
git.ngyro.com/disarchive/tests/cli.scm it is possible to generate a
tarball for docker-layered with Disarchive, but a file describing spec
like git.ngyro.com/disarchive/tests/data/test-archive.da is required.
Generation of this file without having a Tar archive beforehand is
complicated and because of that IMHO ustar.scm from Gash-Utils is
preferred for the task.

Regards,
Oleg.
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQYjygUHGdvLndpZ3Vz
dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzbEhAAy76WdREdgRRwg4j9h83H8dUa30x7
0+6pjM24kKBrDJYLGmv6aW2pumAr+YVre7n4/n1HdlXLStcdJ86/G0zraPj1wD7+
JvR04l0qGb7LY87lPdU/bGGkyt+YubnGbhZ6XoU0Q4CmHHo2aJzvq1S6+Z98m6YN
f/RWl2mkxnrXDau+rLwE2RXQ4wvjZIMg77SSI64lgWoo3IaiwOuhWxcs2Q+ImebS
ymirYoVzhFPCBgznkD2otEDNOvkUMxJ/e2caVGZ21F3guHhq0Df2gmYhRvaIqj/S
JH8xfguqOTntBSvzhi0zjj50A+pyn976IV2C2Mu5mAqtQB0HpSayF/79Y3kwYXYH
bteVg+tsGRrhC4VTPO3Tf5w2PE+SWr7j6ssSNNyRC9wcoYQjeE1y/btYEUeR85vc
L3TS5uD/KM9WJmmpzxyAnmAGR/WkXAvFDtMZ7ZNcq6qTzQVQYltNJD/K69UZ55c4
vPU4KrnEx49lvtnMDzxl8BhzZsgVGjG6k26iX3B5ZyGWpGGayellDVwPWP21MLPr
1R4wanmcyykJvLl1iAVkDDNCRY8cJn4QKQDmUd0Y3gkSJi4Ic7zqYHzIFaEHbLda
/jJ1XB1vTsadXQTJAIdOIQw0fXbK9NF7Tm68nm+czF6AyjUYU5Nz6NmQ85zAEAH4
1cpPWOSj0xOsoYQ=
=uXt9
-----END PGP SIGNATURE-----

?