[PATCH 0/3] Add the '--with-branch' package transformation option

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Ludovic Courtès
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
normal
L
L
Ludovic Courtès wrote on 27 Nov 2018 16:31
(address . guix-patches@gnu.org)(name . Ludovic Courtès)(address . ludo@gnu.org)
20181127153147.21247-1-ludo@gnu.org
Hello,

This patch series adds a ‘git-checkout’ file-like construct along with
a ‘--with-branch’ package transformation option.

The ‘git-checkout’ construct may be used like this:

(define-public guile-git
(package
(inherit guile-next)
(version "git")
(source (git-checkout
(branch "master")))))

Then you can do this:

Toggle snippet (7 lines)
$ guix build -L ~/.config/guix/personal/ guile-next@git
updating checkout of 'https://git.savannah.gnu.org/git/guile.git'...
retrieved commit 1f6786db06fee5f0bbdbd637acc6aa46aecd4169
building /gnu/store/7l168l4lgpjjg9jq7lh2416cc884r3s9-guile-next-git.drv...
[…]

IOW, every time you build the package, it gets built off the latest
commit of the specified branch.

The next patch add ‘--with-branch’ which makes that functionality
available from the command line. For example:

guix build --with-branch=guile-gcrypt=master cuirass

builds the latest guile-gcrypt, guix, and cuirass, using the latest
guile-gcrypt commit on ‘master’.

The ‘--with-branch’ option only works with packages that have a
‘git-fetch’ origin or a ‘git-checkout’ as their source.

I find it rather neat. WDYT? :-)

Ludo’.

Ludovic Courtès (3):
git: 'latest-repository-commit' logs its progress.
git: Add <git-checkout> record type.
guix build: Add '--with-branch' transformation option.

Makefile.am | 1 +
doc/guix.texi | 27 ++++++++++++++++++
guix/git.scm | 41 ++++++++++++++++++++++++++--
guix/scripts/build.scm | 56 ++++++++++++++++++++++++++++++++++++--
tests/guix-build-branch.sh | 48 ++++++++++++++++++++++++++++++++
5 files changed, 168 insertions(+), 5 deletions(-)
create mode 100644 tests/guix-build-branch.sh

--
2.19.2
L
L
Ludovic Courtès wrote on 27 Nov 2018 16:34
[PATCH 1/3] git: 'latest-repository-commit' logs its progress.
(address . 33522@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
20181127153402.21357-1-ludo@gnu.org
From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/git.scm (latest-repository-commit): Add #:log-port and honor it.
---
guix/git.scm | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)

Toggle diff (38 lines)
diff --git a/guix/git.scm b/guix/git.scm
index d007916662..78645024c9 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -154,6 +154,7 @@ data, respectively [<branch name> | <sha1> | <tag name>]."
(define* (latest-repository-commit store url
#:key
+ (log-port (%make-void-port "w"))
(cache-directory
(%repository-cache-directory))
(ref '(branch . "master")))
@@ -164,11 +165,14 @@ REF is pair whose key is [branch | commit | tag] and value the associated
data, respectively [<branch name> | <sha1> | <tag name>].
Git repositories are kept in the cache directory specified by
-%repository-cache-directory parameter."
+%repository-cache-directory parameter.
+
+Log progress and checkout info to LOG-PORT."
(define (dot-git? file stat)
(and (string=? (basename file) ".git")
(eq? 'directory (stat:type stat))))
+ (format log-port "updating checkout of '~a'...~%" url)
(let*-values
(((checkout commit)
(update-cached-checkout url
@@ -177,6 +181,7 @@ Git repositories are kept in the cache directory specified by
(url-cache-directory url cache-directory)))
((name)
(url+commit->name url commit)))
+ (format log-port "retrieved commit ~a~%" commit)
(values (add-to-store store name #t "sha256" checkout
#:select? (negate dot-git?))
commit)))
--
2.19.2
L
L
Ludovic Courtès wrote on 27 Nov 2018 16:34
[PATCH 2/3] git: Add <git-checkout> record type.
(address . 33522@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
20181127153402.21357-2-ludo@gnu.org
From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/git.scm (<git-checkout>): New record type.
(latest-repository-commit*): New procedure.
(git-checkout-compiler): New gexp compiler.
---
guix/git.scm | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)

Toggle diff (58 lines)
diff --git a/guix/git.scm b/guix/git.scm
index 78645024c9..56cebb06ed 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -25,6 +25,8 @@
#:use-module ((guix build utils) #:select (mkdir-p))
#:use-module (guix store)
#:use-module (guix utils)
+ #:use-module (guix records)
+ #:use-module (guix gexp)
#:use-module (rnrs bytevectors)
#:use-module (ice-9 match)
#:use-module (srfi srfi-1)
@@ -33,7 +35,12 @@
#:use-module (srfi srfi-35)
#:export (%repository-cache-directory
update-cached-checkout
- latest-repository-commit))
+ latest-repository-commit
+
+ git-checkout
+ git-checkout?
+ git-checkout-url
+ git-checkout-branch))
(define %repository-cache-directory
(make-parameter (string-append (cache-directory #:ensure? #f)
@@ -185,3 +192,28 @@ Log progress and checkout info to LOG-PORT."
(values (add-to-store store name #t "sha256" checkout
#:select? (negate dot-git?))
commit)))
+
+
+;;;
+;;; Checkouts.
+;;;
+
+;; Representation of the "latest" checkout of a branch.
+(define-record-type* <git-checkout>
+ git-checkout make-git-checkout
+ git-checkout?
+ (url git-checkout-url)
+ (branch git-checkout-branch (default "master")))
+
+(define latest-repository-commit*
+ (store-lift latest-repository-commit))
+
+(define-gexp-compiler (git-checkout-compiler (checkout <git-checkout>)
+ system target)
+ ;; "Compile" CHECKOUT by updating the local checkout and adding it to the
+ ;; store.
+ (match checkout
+ (($ <git-checkout> url branch)
+ (latest-repository-commit* url
+ #:ref `(branch . ,branch)
+ #:log-port (current-error-port)))))
--
2.19.2
L
L
Ludovic Courtès wrote on 27 Nov 2018 16:34
[PATCH 3/3] guix build: Add '--with-branch' transformation option.
(address . 33522@debbugs.gnu.org)(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
20181127153402.21357-3-ludo@gnu.org
From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/scripts/build.scm (evaluate-git-replacement-specs)
(transform-package-source-branch): New procedures.
(%transformations, %transformation-options): Add 'with-branch'.
(show-transformation-options-help): Likewise.
* tests/guix-build-branch.sh: New file.
* Makefile.am (SH_TESTS): Add it.
* doc/guix.texi (Package Transformation Options): Document it.
---
Makefile.am | 1 +
doc/guix.texi | 27 ++++++++++++++++++
guix/scripts/build.scm | 56 ++++++++++++++++++++++++++++++++++++--
tests/guix-build-branch.sh | 48 ++++++++++++++++++++++++++++++++
4 files changed, 129 insertions(+), 3 deletions(-)
create mode 100644 tests/guix-build-branch.sh

Toggle diff (201 lines)
diff --git a/Makefile.am b/Makefile.am
index c5676b0b94..4f134ab174 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -407,6 +407,7 @@ endif
SH_TESTS = \
tests/guix-build.sh \
+ tests/guix-build-branch.sh \
tests/guix-download.sh \
tests/guix-gc.sh \
tests/guix-hash.sh \
diff --git a/doc/guix.texi b/doc/guix.texi
index 917a3e9d57..18bc3508c1 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6447,6 +6447,33 @@ must be compatible. If @var{replacement} is somehow incompatible with
@var{package}, then the resulting package may be unusable. Use with
care!
+@item --with-branch=@var{package}=@var{branch}
+@cindex Git, using the latest commit
+@cindex latest commit, building
+Build @var{package} from the latest commit of @var{branch}. The @code{source}
+field of @var{package} must be an origin with the @code{git-fetch} method
+(@pxref{origin Reference}) or a @code{git-checkout} object; the repository URL
+is taken from that @code{source}.
+
+For instance, the following command builds @code{guile-sqlite3} from the
+latest commit of its @code{master} branch, and then builds @code{guix} (which
+depends on it) and @code{cuirass} (which depends on @code{guix}) against this
+specific @code{guile-sqlite3} build:
+
+@example
+guix build --with-branch=guile-sqlite3=master cuirass
+@end example
+
+@cindex continuous integration
+Obviously, since it uses the latest commit of the given branch, the result of
+such a command varies over time. Nevertheless it is a convenient way to
+rebuild entire software stacks against the latest commit of one or more
+packages. This is particularly useful in the context of continuous
+integration (CI).
+
+Checkouts are kept in a cache under @file{~/.cache/guix/checkouts} to speed up
+consecutive accesses to the same repository. You may want to clean it up once
+in a while to save disk space.
@end table
@node Additional Build Options
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 13978abb77..e8f2fe973d 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -45,6 +45,8 @@
#:use-module (srfi srfi-37)
#:autoload (gnu packages) (specification->package %package-module-path)
#:autoload (guix download) (download-to-store)
+ #:autoload (guix git-download) (git-reference?)
+ #:autoload (guix git) (git-checkout?)
#:use-module (guix status)
#:use-module ((guix progress) #:select (current-terminal-columns))
#:use-module ((guix build syscalls) #:select (terminal-columns))
@@ -270,6 +272,48 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them."
(rewrite obj)
obj))))
+(define (evaluate-git-replacement-specs specs)
+ "Parse SPECS, a list of strings like \"guile=stable-2.2\", and return a list
+of package pairs. Raise an error if an element of SPECS uses invalid syntax,
+or if a package it refers to could not be found."
+ (define not-equal
+ (char-set-complement (char-set #\=)))
+
+ (map (lambda (spec)
+ (match (string-tokenize spec not-equal)
+ ((name branch)
+ (let* ((old (specification->package name))
+ (source (package-source old))
+ (url (cond ((and (origin? source)
+ (git-reference? (origin-uri source)))
+ (git-reference-url (origin-uri source)))
+ ((git-checkout? source)
+ (git-checkout-url source))
+ (else
+ (leave (G_ "the source of ~a is not a Git \
+reference~%")
+ (package-full-name old))))))
+ (cons old
+ (package
+ (inherit old)
+ (version (string-append "git." branch))
+ (source (git-checkout (url url) (branch branch)))))))
+ (x
+ (leave (G_ "invalid replacement specification: ~s~%") spec))))
+ specs))
+
+(define (transform-package-source-branch replacement-specs)
+ "Return a procedure that, when passed a package, replaces its direct
+dependencies according to REPLACEMENT-SPECS. REPLACEMENT-SPECS is a list of
+strings like \"guile-next=stable-3.0\" meaning that packages are built using
+'guile-next' from the latest commit on its 'stable-3.0' branch."
+ (let* ((replacements (evaluate-git-replacement-specs replacement-specs))
+ (rewrite (package-input-rewriting replacements)))
+ (lambda (store obj)
+ (if (package? obj)
+ (rewrite obj)
+ obj))))
+
(define %transformations
;; Transformations that can be applied to things to build. The car is the
;; key used in the option alist, and the cdr is the transformation
@@ -277,7 +321,8 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them."
;; things to build.
`((with-source . ,transform-package-source)
(with-input . ,transform-package-inputs)
- (with-graft . ,transform-package-inputs/graft)))
+ (with-graft . ,transform-package-inputs/graft)
+ (with-branch . ,transform-package-source-branch)))
(define %transformation-options
;; The command-line interface to the above transformations.
@@ -291,7 +336,9 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them."
(option '("with-input") #t #f
(parser 'with-input))
(option '("with-graft") #t #f
- (parser 'with-graft)))))
+ (parser 'with-graft))
+ (option '("with-branch") #t #f
+ (parser 'with-branch)))))
(define (show-transformation-options-help)
(display (G_ "
@@ -302,7 +349,10 @@ current 'gnutls' package, after which version 3.5.4 is grafted onto them."
replace dependency PACKAGE by REPLACEMENT"))
(display (G_ "
--with-graft=PACKAGE=REPLACEMENT
- graft REPLACEMENT on packages that refer to PACKAGE")))
+ graft REPLACEMENT on packages that refer to PACKAGE"))
+ (display (G_ "
+ --with-branch=PACKAGE=BRANCH
+ build PACKAGE from the latest commit of BRANCH")))
(define (options->transformation opts)
diff --git a/tests/guix-build-branch.sh b/tests/guix-build-branch.sh
new file mode 100644
index 0000000000..bc50d9c0ef
--- /dev/null
+++ b/tests/guix-build-branch.sh
@@ -0,0 +1,48 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test 'guix build --with-branch'.
+#
+
+guix build --version
+
+# 'guix build --with-branch' requires access to the network to clone the
+# Git repository below.
+
+if ! guile -c '(getaddrinfo "www.gnu.org" "80" AI_NUMERICSERV)' 2> /dev/null
+then
+ # Skipping.
+ exit 77
+fi
+
+orig_drv="`guix build guile-gcrypt -d`"
+latest_drv="`guix build guile-gcrypt --with-branch=guile-gcrypt=master -d`"
+test -n "$latest_drv"
+test "$orig_drv" != "$latest_drv"
+
+# FIXME: '-S' currently doesn't work with non-derivation source.
+# checkout="`guix build guile-gcrypt --with-branch=guile-gcrypt=master -S`"
+checkout="`guix gc --references "$latest_drv" | grep guile-gcrypt | grep -v -E '(-builder|\.drv)'`"
+test -d "$checkout"
+test -f "$checkout/COPYING"
+
+orig_drv="`guix build guix -d`"
+latest_drv="`guix build guix --with-branch=guile-gcrypt=master -d`"
+guix gc -R "$latest_drv" | grep guile-gcrypt-git.master
+test "$orig_drv" != "$latest_drv"
--
2.19.2
L
L
Ludovic Courtès wrote on 30 Nov 2018 17:04
Re: [bug#33522] [PATCH 0/3] Add the '--with-branch' package transformation option
(address . 33522-done@debbugs.gnu.org)
871s72wnz3.fsf@gnu.org
Hi there!

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (5 lines)
> Ludovic Courtès (3):
> git: 'latest-repository-commit' logs its progress.
> git: Add <git-checkout> record type.
> guix build: Add '--with-branch' transformation option.

I pushed these 3 commits and added a 4th one that implements
‘--with-commit’ along the same lines.

Enjoy!

Ludo’.
Closed
?