[PATCH 0/2] self: Apply grafts to the outputs of the guix derivation.

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Christopher Baines
  • Simon Tournier
Owner
unassigned
Submitted by
Christopher Baines
Severity
normal
C
C
Christopher Baines wrote on 8 Feb 2023 08:46
(address . guix-patches@gnu.org)
871qn08uo7.fsf@cbaines.net
These patches mean that grafts apply to the outputs of the guix
derivation, rather than having grafts apply to the derivation
itself. This moves grafting here to work like grafting for packages,
where you can think of the grafted outputs as a transformed variant of
the ungrafted outputs.

I'm looking at this as it'll allow the Guix Data Service to compute the
derivations without grafts, and for these to be useful for substitutes
regardless of whether users are using grafts.


Christopher Baines (2):
packages: Add explicit-grafting record type to assist with grafts.
self: Apply grafts to the outputs of the guix derivation.

build-aux/build-self.scm | 4 ++-
guix/packages.scm | 45 +++++++++++++++++++++++++++-
guix/self.scm | 65 ++++++++++++++++++++++++++--------------
3 files changed, 89 insertions(+), 25 deletions(-)

--
2.38.1
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmPjU7hfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XcgJw//bYzgsMxFE0DJYm1QniQG6g+NI11p6Yes
JYdFHEAzYUWGXXyjosqG9o83956yfrNLIpXykHOejthOd8u64pv4KpetZq60aak4
QVu40WXHgeXeYySflnPzQoc7L27fQFE7uaDPXTei3Re/B73A+f6DYyCuiEfkc62f
iqxfk9ImY/5sXQWoIU+ebk+ZyVByGBr4HhaSb9IGM79yWKjht+FS+Vl7+wmysgLg
GvAz9MmRvGubwv3I6uFks7jHR0x+z6hcOiZvMneQSTq0uVhmq3zYcLNGW/oc/e/6
VfqDkNF0NZs/HV9iPgQgYIX0FO/9PAWeXsroHrv8bBqP0n/RCrJUbUjv569RF94n
e4tPEbWluGtGYxq8fc8KGkgQvJ29WqF+mCweYdPlGSOtVQvP+Cuq38KDnQ+Z1IwO
kDgTryb/OmC6+y78k0TUEizEA3IBQFdvxC/yVT4Sv5tjRVtOJreDhIp0LtV1Ajbk
9gv/N1m8RviWRfh6neNvrLAk9SytXlJn7JpbUTbOnkbKlOFDFFlOUGmPVf2wyWF2
7bl616qm/jatiBXjChYkRCxuTeM5HcF35onHh92tn93LygdkIGWsHb2E6Z/zYJcs
/j2BsG61oNEKBntptu7+/zQhyjIfhuzwaucFpTu/kEIhrrVEDlIYbrXvp7a1DZ7I
eRetV7hjFpY=
=EhSi
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 8 Feb 2023 08:54
[PATCH 1/2] packages: Add explicit-grafting record type to assist with grafts.
(address . 61363@debbugs.gnu.org)
20230208075403.11788-1-mail@cbaines.net
Normally the grafting takes place when lowering packages, but this record
assists with applying the same transformation to arbitrary objects/store
items.

I'm adding this to allow grafting the channel instance derivation outputs.

* guix/packages.scm (explicit-grafting, explicit-grafting?,
explicit-grafting-obj, explicit-grafting-grafts): New procedures.
---
guix/packages.scm | 45 ++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 44 insertions(+), 1 deletion(-)

Toggle diff (62 lines)
diff --git a/guix/packages.scm b/guix/packages.scm
index 041a872f9d..877bf89522 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -188,7 +188,12 @@ (define-module (guix packages)
package-file
package->derivation
package->cross-derivation
- origin->derivation))
+ origin->derivation
+
+ explicit-grafting
+ explicit-grafting?
+ explicit-grafting-obj
+ explicit-grafting-grafts))
;; The 'source-module-closure' procedure ca. 1.2.0 did not recognize
;; #:re-export-and-replace: <https://issues.guix.gnu.org/52694>.
@@ -2093,3 +2098,41 @@ (define package-source-derivation ;somewhat deprecated
(add-to-store store (basename file) #t "sha256" file))
(_
(lower store source system))))))
+
+;; Apply grafts explicitly
+(define-immutable-record-type <explicit-grafting>
+ (%explicit-grafting obj packages)
+ explicit-grafting?
+ (obj explicit-grafting-obj) ;obj
+ (packages explicit-grafting-packages)) ;list of <package>s
+
+(define (write-explicit-grafting rec port)
+ (match rec
+ (($ <explicit-grafting> obj packages)
+ (format port "#<explicit-grafting ~s ~s>" obj packages))))
+
+(define (explicit-grafting obj packages)
+ (%explicit-grafting obj packages))
+
+(define-gexp-compiler (explicit-grafting-compiler (explicit-grafting <explicit-grafting>)
+ system target)
+ (match explicit-grafting
+ (($ <explicit-grafting> obj packages)
+ (mlet* %store-monad ((drv (without-grafting
+ (lower-object obj system #:target target)))
+ (grafts
+ (mapm %store-monad
+ (lambda (pkg)
+ (package-grafts* pkg system #:target target))
+ packages)))
+ (match (delete-duplicates
+ (concatenate grafts))
+ (()
+ (return drv))
+ (grafts
+ (mlet %store-monad ((guile (package->derivation
+ (guile-for-grafts)
+ system #:graft? #f)))
+ (graft-derivation* drv grafts
+ #:system system
+ #:guile guile))))))))
--
2.38.1
C
C
Christopher Baines wrote on 8 Feb 2023 08:54
[PATCH 2/2] self: Apply grafts to the outputs of the guix derivation.
(address . 61363@debbugs.gnu.org)
20230208075403.11788-2-mail@cbaines.net
Rather than having grafts apply to the derivation itself. This moves grafting
here to work like grafting for packages, where you can think of the grafted
outputs as a transformed variant of the ungrafted outputs.

I'm looking at this as it'll allow the Guix Data Service to compute the
derivations without grafts, and for these to be useful for substitutes
regardless of whether users are using grafts.

* guix/self.scm (compiled-guix, guix-derivation): Add a #:graft? keyword
argument, to control grafting when computing the guix derivation.
* build-aux/build-self.scm (build-program): Call guix-derivation with
#:graft? (%graft?) to make the compute-guix-derivation script use or not use
grafts as desired.
---
build-aux/build-self.scm | 4 ++-
guix/self.scm | 65 ++++++++++++++++++++++++++--------------
2 files changed, 45 insertions(+), 24 deletions(-)

Toggle diff (123 lines)
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 02822a2ee8..6d0037f20c 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -353,7 +353,9 @@ (define fake-git
#:channel-metadata
'#$channel-metadata
#:pull-version
- #$pull-version)
+ #$pull-version
+ #:graft?
+ #$(%graft?))
#:system system))
derivation-file-name))))))
#:module-path (list source))))
diff --git a/guix/self.scm b/guix/self.scm
index 93019e1c64..c944dbe9ce 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -752,7 +752,8 @@ (define* (compiled-guix source #:key
(gzip (specification->package "gzip"))
(bzip2 (specification->package "bzip2"))
(xz (specification->package "xz"))
- (guix (specification->package "guix")))
+ (guix (specification->package "guix"))
+ (graft? #t))
"Return a file-like object that contains a compiled Guix."
(define guile-avahi
(specification->package "guile-avahi"))
@@ -802,6 +803,12 @@ (define dependencies
guile-json guile-semver guile-ssh guile-sqlite3
guile-lib guile-zlib guile-lzlib guile-zstd)))
+ (define packages
+ (cons* gzip
+ bzip2
+ xz
+ dependencies))
+
(define *core-modules*
(scheme-node "guix-core"
'((guix)
@@ -1022,28 +1029,35 @@ (define (built-modules node-subset)
guile-lzma
dependencies)
#:guile guile-for-build
- #:guile-version guile-version)))
- (whole-package name modules dependencies
- #:command command
- #:guile guile-for-build
-
- ;; Include 'guix-daemon'. XXX: Here we inject an
- ;; older snapshot of guix-daemon, but that's a good
- ;; enough approximation for now.
- #:daemon (module-ref (resolve-interface
- '(gnu packages
- package-management))
- 'guix-daemon)
-
- #:info (info-manual source)
- #:miscellany (miscellaneous-files source)
- #:guile-version guile-version)))
+ #:guile-version guile-version))
+ (obj
+ (whole-package name modules dependencies
+ #:command command
+ #:guile guile-for-build
+
+ ;; Include 'guix-daemon'. XXX: Here we inject
+ ;; an older snapshot of guix-daemon, but
+ ;; that's a good enough approximation for now.
+ #:daemon (module-ref (resolve-interface
+ '(gnu packages
+ package-management))
+ 'guix-daemon)
+
+ #:info (info-manual source)
+ #:miscellany (miscellaneous-files source)
+ #:guile-version guile-version)))
+ (if graft?
+ (explicit-grafting obj packages)
+ obj)))
((= 0 pull-version)
;; Legacy 'guix pull': return the .scm and .go files as one
;; directory.
- (built-modules (lambda (node)
- (list (node-source node)
- (node-compiled node)))))
+ (let ((obj (built-modules (lambda (node)
+ (list (node-source node)
+ (node-compiled node))))))
+ (if graft?
+ (explicit-grafting obj packages)
+ obj)))
(else
;; Unsupported 'guix pull' version.
#f)))
@@ -1273,7 +1287,8 @@ (define (process-directory directory files output)
(define* (guix-derivation source version
#:optional (guile-version (effective-version))
#:key (pull-version 0)
- channel-metadata)
+ channel-metadata
+ (graft? #t))
"Return, as a monadic value, the derivation to build the Guix from SOURCE
for GUILE-VERSION. Use VERSION as the version string. Use CHANNEL-METADATA
as the channel metadata sexp to include in (guix config).
@@ -1310,7 +1325,11 @@ (define guile
#:pull-version pull-version
#:guile-version (if (>= pull-version 1)
"3.0" guile-version)
- #:guile-for-build guile)))
+ #:guile-for-build guile
+ #:graft? graft?)))
(if guix
- (lower-object guix)
+ (if graft?
+ (lower-object guix)
+ (without-grafting
+ (lower-object guix)))
(return #f)))))
--
2.38.1
C
C
Christopher Baines wrote on 10 Feb 2023 10:16
Re: [bug#61363] [PATCH 0/2] self: Apply grafts to the outputs of the guix derivation.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 61363@debbugs.gnu.org)
87h6vt6foi.fsf@cbaines.net
The data service comparison is now available for this, and while there
are no differences in the packages, you can see some information.

This is the channel instances before:


and this is after:

-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmPmDB1fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XfyxA//WSnQZUWm2/qLSUknAYqL/Mr+YiaqtIw+
oCTLm96qrlZSl1VU7XSPsSCn7TbS6n6f7gSUBh7MSmaWHBbVRr/HsoLK2owTRqRv
w34wkn5m0fZUKLqtt8fTum5Zkc8tQoz+ZgABRYLQQwRymssVistnG25A7vgrq6bD
sfuEwQjk02Aejc9Mmshh6b14Nd1bq/NulgNbncqNkkMiHoiaGGHPvbBkfoCGAVrM
TwhO6VY4cfge9c5cGrtrAKZo5YsB7Ergy2Ixx9KkTOUN6O1FWVtrXMijP6Nf+Gky
asCHWfDR+YXl6EVOKcL8JZ/4JNKKkPP9qHT69JTvL+kgNdBaE18ucWvlBJMagQyk
FqwotXQVqTON6ZHYR5IE6FV/RK+KgqXvJcBzIUdgVUKfQeSqRr/J2atXhJZ80TDr
rHlS/RUnDG/Qcp//vTxdPX/ZkUulumVDmtKSuoWZ/xtaEeIy0EOsJCK0oZ0xSJZB
P6ZwmYTJUVDSpGlstgaPpTKo/5zwOdKmnZlcAv3zT5G7v2pediK49g9FVqMBQA0h
edsbnMCjVGHXmJgvvyBJdAyRxU28MiYM4gm0wbAe9AqnAnStKxV2KMMK03joLK08
S6UeAfr4HSeUkT6UNUH/0ET9P7+Th8kFGlr78yn0HomTgiXaj9NozbHZ3V4m5GqE
KeIOjfiahbg=
=4/2T
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 22 Feb 2023 10:16
Re: [bug#61363] [PATCH 2/2] self: Apply grafts to the outputs of the guix derivation.
(name . Christopher Baines)(address . mail@cbaines.net)(address . 61363@debbugs.gnu.org)
87sfey9i1t.fsf@gnu.org
Hi,

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (4 lines)
> Rather than having grafts apply to the derivation itself. This moves grafting
> here to work like grafting for packages, where you can think of the grafted
> outputs as a transformed variant of the ungrafted outputs.

Hmm.

Toggle quote (4 lines)
> I'm looking at this as it'll allow the Guix Data Service to compute the
> derivations without grafts, and for these to be useful for substitutes
> regardless of whether users are using grafts.

How does it help exactly? By disabling grafts in that context?

Toggle quote (22 lines)
> +++ b/guix/self.scm
> @@ -752,7 +752,8 @@ (define* (compiled-guix source #:key
> (gzip (specification->package "gzip"))
> (bzip2 (specification->package "bzip2"))
> (xz (specification->package "xz"))
> - (guix (specification->package "guix")))
> + (guix (specification->package "guix"))
> + (graft? #t))
> "Return a file-like object that contains a compiled Guix."
> (define guile-avahi
> (specification->package "guile-avahi"))
> @@ -802,6 +803,12 @@ (define dependencies
> guile-json guile-semver guile-ssh guile-sqlite3
> guile-lib guile-zlib guile-lzlib guile-zstd)))
>
> + (define packages
> + (cons* gzip
> + bzip2
> + xz
> + dependencies))
> +

[...]

Toggle quote (7 lines)
> + (let ((obj (built-modules (lambda (node)
> + (list (node-source node)
> + (node-compiled node))))))
> + (if graft?
> + (explicit-grafting obj packages)
> + obj)))

There are two things I’m not comfortable with:

1. Having <explicit-grafting> in (guix packages); it looks misplaced.

2. More importantly, manually listing packages that might require
grafting looks like a slippery slope (“oops! we’re not getting the
GnuTLS graft for that CVE, too bad”).

I designed and implemented several variants to try and delay grafting.
One of them consisted in carrying graft information in gexps:


It’s kinda similar to what you’re proposing in that graft information is
carried as far as possible. The main difference is that it’s automated.

Hmm needs more thought.

Ludo’.
C
C
Christopher Baines wrote on 22 Feb 2023 12:17
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 61363@debbugs.gnu.org)
878rgpeo28.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (6 lines)
>> I'm looking at this as it'll allow the Guix Data Service to compute the
>> derivations without grafts, and for these to be useful for substitutes
>> regardless of whether users are using grafts.
>
> How does it help exactly? By disabling grafts in that context?

So the Guix Data Service is somewhat built on the assumption that it's
cheap to compute derivations, at least with grafts disabled. That's
always been the case for packages, but for channel instance derivations
it's not reliably the case, since currently disabling grafts doesn't
apply to the whole process, and even if it did, the derivations you'd
get out wouldn't be that useful (since you can't transform the outputs
from those derivations to the outputs you'd get if using grafts).

With these changes, it's always relatively cheap to compute the channel
instance derivations, and it's always possible to compute the
derivations for any system without needing to be able to perform builds
for that system.

You can see this in how the data service has processed Guix before and
after these patches.

This is the channel instances before:


and this is after:


Given data.qa.guix.gnu.org is running on an x86_64-linux system, that
and i686-linux isn't generally a problem, but I'm guessing it only
managed to compute the powerpc64le-linux and aarch64-linux derivations
because it was able to substitute the necessary store items. For other
system computing the derivations would have failed.

I believe this change will also mean that the build farms will go from
performing the grafting for these builds, to being able to not do so, in
line with how builds for packages are handled. This isn't a big thing,
but I think it makes sense.

Toggle quote (35 lines)
>> +++ b/guix/self.scm
>> @@ -752,7 +752,8 @@ (define* (compiled-guix source #:key
>> (gzip (specification->package "gzip"))
>> (bzip2 (specification->package "bzip2"))
>> (xz (specification->package "xz"))
>> - (guix (specification->package "guix")))
>> + (guix (specification->package "guix"))
>> + (graft? #t))
>> "Return a file-like object that contains a compiled Guix."
>> (define guile-avahi
>> (specification->package "guile-avahi"))
>> @@ -802,6 +803,12 @@ (define dependencies
>> guile-json guile-semver guile-ssh guile-sqlite3
>> guile-lib guile-zlib guile-lzlib guile-zstd)))
>>
>> + (define packages
>> + (cons* gzip
>> + bzip2
>> + xz
>> + dependencies))
>> +
>
> [...]
>
>> + (let ((obj (built-modules (lambda (node)
>> + (list (node-source node)
>> + (node-compiled node))))))
>> + (if graft?
>> + (explicit-grafting obj packages)
>> + obj)))
>
> There are two things I’m not comfortable with:
>
> 1. Having <explicit-grafting> in (guix packages); it looks misplaced.

I didn't put it there at first, but I think it makes sense since
grafting is currently specific to packages, as is this additional code.

Toggle quote (12 lines)
> 2. More importantly, manually listing packages that might require
> grafting looks like a slippery slope (“oops! we’re not getting the
> GnuTLS graft for that CVE, too bad”).
>
> I designed and implemented several variants to try and delay grafting.
> One of them consisted in carrying graft information in gexps:
>
> https://git.savannah.gnu.org/cgit/guix.git/log?h=wip-gexp-grafts
>
> It’s kinda similar to what you’re proposing in that graft information is
> carried as far as possible. The main difference is that it’s automated.

That's interesting, I think that making grafting not specific to
packages, and something where the replacement is handled at a lower
level (e.g. gexps) would be an alternative way to handle this.

Given that this approach works though, maybe the explicit-grafting
functionality could just sit and be used inside of (guix self). Given
that module is very explicit about what packages are used, it should be
possible to arrange the code so it's very hard to miss a package out,
which should address your concern about manually listing packages (maybe
specification->package can be tweaked so that it's possible to get all
the packages, and that can be the list considered for grafting).

I don't know of any other places where this approach would be useful, so
while it would be nice to have a more general grafting mechanism
eventually, I'd also like to be able to make these changes to channel
instance grafts sooner rather than later.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmP2L69fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xf3pw/9EtEo0UOool4+8PfEFZ/Df9MYigLmm/y9
MWN0gQrfhosBdRVVGGZlCDlz9Hgh1ix07pJKSUELHxaoLGwhyKgPvxghz7A9BbBi
DN4D+AJL/8Xe8aPblQvoXWwMnGViHdK1HW9uxey8UJwCl2Np/0jrYq3H30WzXPCU
XWCSEz4hcFV4DTb7VDaLEINtkthv0lEPkKnaMP0ojGNoSztPxaUtCzNVc78zMxCV
6wMxa7HK7ew5131RYnGpn13p1hHFhhfnnY8EbmhkxfnZ9huZOCCNez4ifkO5jjjP
CpztgnQPBe+/EgApp71Vwmx5s2ZrQMIhNykLg6e1wtRhDygtEYmI7RlzXMWVrYTn
XY0XZ9d5xlc9K4hvFbvl3BA8yzjRXNSeSJMDKlvEGqy8qpl3BxH1MxXdGHkN0yGc
9M7cv9hMqwFcE7rVUKrEKfHwASDGwMwe9yTnE7eSlLhmO/xv2h+aHsmQQfrGe3y2
g5KnoOlBO26bbfXIf5rGOOSOCAmQUHMb25KFMMhdfTgP0Z6BLqNtLMbIKptEr6cM
dGhaxv47kEXEEP8TcuCezdAd3N3oTwPneUM/BKmUVEKmktPlRBkR2beKxp35tFb8
ngxgSLlUj8xhf7wTf22OzaUeuXmGNmQkaycpxwQhtuEfCtZXF6G6XUU5xouXO7RE
tUzp7aAE1oU=
=1/IL
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 28 Feb 2023 16:47
[PATCH v2 1/3] packages: Export guile-for-grafts.
(address . 61363@debbugs.gnu.org)
20230228154703.3952-1-mail@cbaines.net
So this can be used in (guix self).

* guix/packages.scm (guile-for-grafts): Export.
---
guix/packages.scm | 1 +
1 file changed, 1 insertion(+)

Toggle diff (14 lines)
diff --git a/guix/packages.scm b/guix/packages.scm
index 041a872f9d..2f81ad0284 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -183,6 +183,7 @@ (define-module (guix packages)
package-closure
default-guile
+ guile-for-grafts
default-guile-derivation
set-guile-for-build
package-file
--
2.39.1
C
C
Christopher Baines wrote on 28 Feb 2023 16:47
[PATCH v2 2/3] self: Restructure accessing packages.
(address . 61363@debbugs.gnu.org)
20230228154703.3952-2-mail@cbaines.net
Both for consistency (always use specification->package as defined in this
module) and so that all the packages that are used can be accessed (which
comes in useful when applying grafts).

* guix/self.scm (%packages): New variable.
(specification->package): Use %packages.
(locale-data, translate-texi-manuals, info-manual, guix-command,
compiled-guix): Use specification->package.
---
guix/self.scm | 97 +++++++++++++++++++++++++--------------------------
1 file changed, 48 insertions(+), 49 deletions(-)

Toggle diff (173 lines)
diff --git a/guix/self.scm b/guix/self.scm
index 93019e1c64..c5de3ab8fc 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -44,34 +44,42 @@ (define-module (guix self)
;;; Dependency handling.
;;;
-(define specification->package
+(define %packages
+ (let ((ref (lambda (module variable)
+ (delay
+ (module-ref (resolve-interface
+ `(gnu packages ,module))
+ variable)))))
+ `(("guile" . ,(ref 'guile 'guile-3.0-latest))
+ ("guile-avahi" . ,(ref 'guile-xyz 'guile-avahi))
+ ("guile-json" . ,(ref 'guile 'guile-json-4))
+ ("guile-ssh" . ,(ref 'ssh 'guile-ssh))
+ ("guile-git" . ,(ref 'guile 'guile-git))
+ ("guile-semver" . ,(ref 'guile-xyz 'guile-semver))
+ ("guile-lib" . ,(ref 'guile-xyz 'guile-lib))
+ ("guile-sqlite3" . ,(ref 'guile 'guile-sqlite3))
+ ("guile-zlib" . ,(ref 'guile 'guile-zlib))
+ ("guile-lzlib" . ,(ref 'guile 'guile-lzlib))
+ ("guile-zstd" . ,(ref 'guile 'guile-zstd))
+ ("guile-gcrypt" . ,(ref 'gnupg 'guile-gcrypt))
+ ("guile-gnutls" . ,(ref 'tls 'guile-gnutls))
+ ("guix-daemon" . ,(ref 'package-management 'guix-daemon))
+ ("disarchive" . ,(ref 'backup 'disarchive))
+ ("guile-lzma" . ,(ref 'guile 'guile-lzma))
+ ("gzip" . ,(ref 'compression 'gzip))
+ ("bzip2" . ,(ref 'compression 'bzip2))
+ ("xz" . ,(ref 'compression 'xz))
+ ("po4a" . ,(ref 'gettext 'po4a))
+ ("gettext-minimal" . ,(ref 'gettext 'gettext-minimal))
+ ("gcc-toolchain" . ,(ref 'commencement 'gcc-toolchain))
+ ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales))
+ ("graphviz" . ,(ref 'graphviz 'graphviz))
+ ("texinfo" . ,(ref 'texinfo 'texinfo)))))
+
+(define (specification->package name)
;; Use our own variant of that procedure because that of (gnu packages)
;; would traverse all the .scm files, which is wasteful.
- (let ((ref (lambda (module variable)
- (module-ref (resolve-interface module) variable))))
- (match-lambda
- ("guile" (ref '(gnu packages guile) 'guile-3.0-latest))
- ("guile-avahi" (ref '(gnu packages guile-xyz) 'guile-avahi))
- ("guile-json" (ref '(gnu packages guile) 'guile-json-4))
- ("guile-ssh" (ref '(gnu packages ssh) 'guile-ssh))
- ("guile-git" (ref '(gnu packages guile) 'guile-git))
- ("guile-semver" (ref '(gnu packages guile-xyz) 'guile-semver))
- ("guile-lib" (ref '(gnu packages guile-xyz) 'guile-lib))
- ("guile-sqlite3" (ref '(gnu packages guile) 'guile-sqlite3))
- ("guile-zlib" (ref '(gnu packages guile) 'guile-zlib))
- ("guile-lzlib" (ref '(gnu packages guile) 'guile-lzlib))
- ("guile-zstd" (ref '(gnu packages guile) 'guile-zstd))
- ("guile-gcrypt" (ref '(gnu packages gnupg) 'guile-gcrypt))
- ("guile-gnutls" (ref '(gnu packages tls) 'guile-gnutls))
- ("disarchive" (ref '(gnu packages backup) 'disarchive))
- ("guile-lzma" (ref '(gnu packages guile) 'guile-lzma))
- ("gzip" (ref '(gnu packages compression) 'gzip))
- ("bzip2" (ref '(gnu packages compression) 'bzip2))
- ("xz" (ref '(gnu packages compression) 'xz))
- ("po4a" (ref '(gnu packages gettext) 'po4a))
- ("gettext" (ref '(gnu packages gettext) 'gettext-minimal))
- ("gcc-toolchain" (ref '(gnu packages commencement) 'gcc-toolchain))
- (_ #f)))) ;no such package
+ (and=> (assoc-ref %packages name) force))
;;;
@@ -240,9 +248,8 @@ (define* (locale-data source domain
#:optional (directory domain))
"Return the locale data from 'po/DIRECTORY' in SOURCE, corresponding to
DOMAIN, a gettext domain."
- (define gettext
- (module-ref (resolve-interface '(gnu packages gettext))
- 'gettext-minimal))
+ (define gettext-minimal
+ (specification->package "gettext-minimal"))
(define build
(with-imported-modules '((guix build utils))
@@ -258,7 +265,7 @@ (define (compile language)
(let ((gmo (string-append #$output "/" language "/LC_MESSAGES/"
#$domain ".mo")))
(mkdir-p (dirname gmo))
- (invoke #+(file-append gettext "/bin/msgfmt")
+ (invoke #+(file-append gettext-minimal "/bin/msgfmt")
"-c" "--statistics" "--verbose"
"-o" gmo
(string-append po-directory "/" language ".po"))))
@@ -280,20 +287,19 @@ (define (translate-texi-manuals source)
"Return the translated texinfo manuals built from SOURCE."
(define po4a
(specification->package "po4a"))
-
- (define gettext
- (specification->package "gettext"))
+
+ (define gettext-minimal
+ (specification->package "gettext-minimal"))
(define glibc-utf8-locales
- (module-ref (resolve-interface '(gnu packages base))
- 'glibc-utf8-locales))
+ (specification->package "glibc-utf8-locales"))
(define documentation
(file-append* source "doc"))
(define documentation-po
(file-append* source "po/doc"))
-
+
(define build
(with-imported-modules '((guix build utils) (guix build po))
#~(begin
@@ -365,7 +371,7 @@ (define parallel-jobs
(setenv "GUIX_LOCPATH"
#+(file-append glibc-utf8-locales "/lib/locale"))
- (setenv "PATH" #+(file-append gettext "/bin"))
+ (setenv "PATH" #+(file-append gettext-minimal "/bin"))
(setenv "LC_ALL" "en_US.UTF-8")
(setlocale LC_ALL "en_US.UTF-8")
@@ -394,16 +400,13 @@ (define parallel-jobs
(define (info-manual source)
"Return the Info manual built from SOURCE."
(define texinfo
- (module-ref (resolve-interface '(gnu packages texinfo))
- 'texinfo))
+ (specification->package "texinfo"))
(define graphviz
- (module-ref (resolve-interface '(gnu packages graphviz))
- 'graphviz))
+ (specification->package "graphviz"))
(define glibc-utf8-locales
- (module-ref (resolve-interface '(gnu packages base))
- 'glibc-utf8-locales))
+ (specification->package "glibc-utf8-locales"))
(define documentation
(file-append* source "doc"))
@@ -586,8 +589,7 @@ (define* (guix-command modules
"Return the 'guix' command such that it adds MODULES and DEPENDENCIES in its
load path."
(define glibc-utf8-locales
- (module-ref (resolve-interface '(gnu packages base))
- 'glibc-utf8-locales))
+ (specification->package "glibc-utf8-locales"))
(define module-directory
;; To minimize the number of 'stat' calls needed to locate a module,
@@ -1030,10 +1032,7 @@ (define (built-modules node-subset)
;; Include 'guix-daemon'. XXX: Here we inject an
;; older snapshot of guix-daemon, but that's a good
;; enough approximation for now.
- #:daemon (module-ref (resolve-interface
- '(gnu packages
- package-management))
- 'guix-daemon)
+ #:daemon (specification->package "guix-daemon")
#:info (info-manual source)
#:miscellany (miscellaneous-files source)
--
2.39.1
C
C
Christopher Baines wrote on 28 Feb 2023 16:47
[PATCH v2 3/3] self: Apply grafts to the outputs of the guix derivation.
(address . 61363@debbugs.gnu.org)
20230228154703.3952-3-mail@cbaines.net
Rather than having grafts apply to the derivation itself. This moves grafting
here to work like grafting for packages, where you can think of the grafted
outputs as a transformed variant of the ungrafted outputs.

I'm looking at this as it'll allow the Guix Data Service to compute the
derivations without grafts, and for these to be useful for substitutes
regardless of whether users are using grafts.

* guix/self.scm (compiled-guix, guix-derivation): Add a #:graft? keyword
argument, to control grafting when computing the guix derivation.
* build-aux/build-self.scm (build-program): Call guix-derivation with
#:graft? (%graft?) to make the compute-guix-derivation script use or not use
grafts as desired.
---
build-aux/build-self.scm | 4 +-
guix/self.scm | 101 +++++++++++++++++++++++++++++++--------
2 files changed, 84 insertions(+), 21 deletions(-)

Toggle diff (173 lines)
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 02822a2ee8..6d0037f20c 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -353,7 +353,9 @@ (define fake-git
#:channel-metadata
'#$channel-metadata
#:pull-version
- #$pull-version)
+ #$pull-version
+ #:graft?
+ #$(%graft?))
#:system system))
derivation-file-name))))))
#:module-path (list source))))
diff --git a/guix/self.scm b/guix/self.scm
index c5de3ab8fc..8842275ff8 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -22,6 +22,7 @@ (define-module (guix self)
#:use-module (guix i18n)
#:use-module (guix modules)
#:use-module (guix gexp)
+ #:use-module (guix grafts)
#:use-module (guix store)
#:use-module (guix monads)
#:use-module (guix discovery)
@@ -32,6 +33,7 @@ (define-module (guix self)
#:use-module ((guix build utils) #:select (find-files))
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-9 gnu)
#:use-module (srfi srfi-35)
#:use-module (ice-9 match)
#:export (make-config.scm
@@ -244,6 +246,50 @@ (define* (file-append* item file #:key (recursive? #t))
;; which isn't great.
(file-append item "/" file))))
+(define graft-derivation*
+ (store-lift graft-derivation))
+
+(define package-grafts*
+ (store-lift package-grafts))
+
+;; Apply grafts explicitly
+(define-immutable-record-type <explicit-grafting>
+ (%explicit-grafting obj packages)
+ explicit-grafting?
+ (obj explicit-grafting-obj) ;obj
+ (packages explicit-grafting-packages)) ;list of <package>s
+
+(define (write-explicit-grafting rec port)
+ (match rec
+ (($ <explicit-grafting> obj packages)
+ (format port "#<explicit-grafting ~s ~s>" obj packages))))
+
+(define (explicit-grafting obj packages)
+ (%explicit-grafting obj packages))
+
+(define-gexp-compiler (explicit-grafting-compiler (explicit-grafting <explicit-grafting>)
+ system target)
+ (match explicit-grafting
+ (($ <explicit-grafting> obj packages)
+ (mlet* %store-monad ((drv (without-grafting
+ (lower-object obj system #:target target)))
+ (grafts
+ (mapm %store-monad
+ (lambda (pkg)
+ (package-grafts* pkg system #:target target))
+ packages)))
+ (match (delete-duplicates
+ (concatenate grafts))
+ (()
+ (return drv))
+ (grafts
+ (mlet %store-monad ((guile (package->derivation
+ (guile-for-grafts)
+ system #:graft? #f)))
+ (graft-derivation* drv grafts
+ #:system system
+ #:guile guile))))))))
+
(define* (locale-data source domain
#:optional (directory domain))
"Return the locale data from 'po/DIRECTORY' in SOURCE, corresponding to
@@ -754,7 +800,8 @@ (define* (compiled-guix source #:key
(gzip (specification->package "gzip"))
(bzip2 (specification->package "bzip2"))
(xz (specification->package "xz"))
- (guix (specification->package "guix")))
+ (guix (specification->package "guix"))
+ (graft? #t))
"Return a file-like object that contains a compiled Guix."
(define guile-avahi
(specification->package "guile-avahi"))
@@ -1024,25 +1071,34 @@ (define (built-modules node-subset)
guile-lzma
dependencies)
#:guile guile-for-build
- #:guile-version guile-version)))
- (whole-package name modules dependencies
- #:command command
- #:guile guile-for-build
-
- ;; Include 'guix-daemon'. XXX: Here we inject an
- ;; older snapshot of guix-daemon, but that's a good
- ;; enough approximation for now.
- #:daemon (specification->package "guix-daemon")
-
- #:info (info-manual source)
- #:miscellany (miscellaneous-files source)
- #:guile-version guile-version)))
+ #:guile-version guile-version))
+ (obj
+ (whole-package name modules dependencies
+ #:command command
+ #:guile guile-for-build
+
+ ;; Include 'guix-daemon'. XXX: Here we inject
+ ;; an older snapshot of guix-daemon, but
+ ;; that's a good enough approximation for now.
+ #:daemon (specification->package "guix-daemon")
+
+ #:info (info-manual source)
+ #:miscellany (miscellaneous-files source)
+ #:guile-version guile-version)))
+ (if graft?
+ (explicit-grafting obj
+ (map (compose force cdr) %packages))
+ obj)))
((= 0 pull-version)
;; Legacy 'guix pull': return the .scm and .go files as one
;; directory.
- (built-modules (lambda (node)
- (list (node-source node)
- (node-compiled node)))))
+ (let ((obj (built-modules (lambda (node)
+ (list (node-source node)
+ (node-compiled node))))))
+ (if graft?
+ (explicit-grafting obj
+ (map (compose force cdr) %packages))
+ obj)))
(else
;; Unsupported 'guix pull' version.
#f)))
@@ -1272,7 +1328,8 @@ (define (process-directory directory files output)
(define* (guix-derivation source version
#:optional (guile-version (effective-version))
#:key (pull-version 0)
- channel-metadata)
+ channel-metadata
+ (graft? #t))
"Return, as a monadic value, the derivation to build the Guix from SOURCE
for GUILE-VERSION. Use VERSION as the version string. Use CHANNEL-METADATA
as the channel metadata sexp to include in (guix config).
@@ -1309,7 +1366,11 @@ (define guile
#:pull-version pull-version
#:guile-version (if (>= pull-version 1)
"3.0" guile-version)
- #:guile-for-build guile)))
+ #:guile-for-build guile
+ #:graft? graft?)))
(if guix
- (lower-object guix)
+ (if graft?
+ (lower-object guix)
+ (without-grafting
+ (lower-object guix)))
(return #f)))))
--
2.39.1
C
C
Christopher Baines wrote on 28 Feb 2023 16:47
Re: [bug#61363] [PATCH 2/2] self: Apply grafts to the outputs of the guix derivation.
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 61363@debbugs.gnu.org)
871qm9aiw7.fsf@cbaines.net
Christopher Baines <mail@cbaines.net> writes:

Toggle quote (31 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> 2. More importantly, manually listing packages that might require
>> grafting looks like a slippery slope (“oops! we’re not getting the
>> GnuTLS graft for that CVE, too bad”).
>>
>> I designed and implemented several variants to try and delay grafting.
>> One of them consisted in carrying graft information in gexps:
>>
>> https://git.savannah.gnu.org/cgit/guix.git/log?h=wip-gexp-grafts
>>
>> It’s kinda similar to what you’re proposing in that graft information is
>> carried as far as possible. The main difference is that it’s automated.
>
> That's interesting, I think that making grafting not specific to
> packages, and something where the replacement is handled at a lower
> level (e.g. gexps) would be an alternative way to handle this.
>
> Given that this approach works though, maybe the explicit-grafting
> functionality could just sit and be used inside of (guix self). Given
> that module is very explicit about what packages are used, it should be
> possible to arrange the code so it's very hard to miss a package out,
> which should address your concern about manually listing packages (maybe
> specification->package can be tweaked so that it's possible to get all
> the packages, and that can be the list considered for grafting).
>
> I don't know of any other places where this approach would be useful, so
> while it would be nice to have a more general grafting mechanism
> eventually, I'd also like to be able to make these changes to channel
> instance grafts sooner rather than later.

I've sent a v2 series which changes along the above lines. The explicit
grafting stuff just sits in (guix self), and (guix self) more
rigeriously uses it's own definition of specification->package, which
should provide some protection against missing packages out. Obviously
it's not quite as rigerous as moving the grafting functionality in to
gexps, but hopefully it's rigerous enough for now.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmP+IthfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xf2iw/8DZikf+lixwrP883zXiQT2XSlzldmnxnT
Hv0mZyZl1pPIo78jVTcgumLasEsPa9CL7pVKjwvMYSmGLIhtdsAnV0heII2G15ch
GVigOj2u74rexLjjbP1FabA244qDLFN+3BZAtwl9beqC+1B5FwLZzoHWPE/CW2/Z
WluT3F2gBr/GX+61ksRHETzm+OHnwMCawHEOkhcmTtF9yQiSW2K0NkOvI8k1n5v7
rsQIbJ5BQdzZvbwKl5Q2oQd6y7nyliZctOl+rFjZ9a9rkrTNT4LdWC+5bPhsai8m
uV8d+G0fOPgfaUvomvsxRqo9Xhi92KSFaoBwdgxMzoByuBF0U6k2QUQEyS1CwZVO
j8jTGb+qnDfRlGbitdqjERGf4Yx4XAFfpuXYqBUJJLqG9dnES5JyOT7WLhrsVdFx
wAI35UkCoopEirexXJ8gdmKFWKo3YEeRFXKme8qP+8wvFZ0jv4p7quEwBibE+8Tw
JHh7my12WxJ/YURo4DoM6h2U25kq5QUtzHVyTSdrYFIeM1kCcy0c2DllWG7nUYzW
23Kz4veoXfY7hyfuU4Bi+VWFdVxlDcZS06wItO4VUbibJ3PPcOZ7yt1cA2DLLMDs
zJgxebMvTusv/l2QLXvRJTv9/Y1h66Brs8sE4N4UQOb7Fmtom07N+u5C+MtXzcq7
re6XFrk5shU=
=afSK
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 17 Apr 2023 16:59
[PATCH v3] self: Apply grafts to the outputs of the guix derivation.
(address . 61363@debbugs.gnu.org)(name . Christopher Baines)(address . mail@cbaines.net)
20230417145928.16693-1-mail@cbaines.net
Rather than having grafts apply to the derivation itself. This moves grafting
here to work like grafting for packages, where you can think of the grafted
outputs as a transformed variant of the ungrafted outputs.

I'm looking at this as it'll allow the Guix Data Service to compute the
derivations without grafts, and for these to be useful for substitutes
regardless of whether users are using grafts.

* guix/self.scm (compiled-guix, guix-derivation): Add a #:graft? keyword
argument, to control grafting when computing the guix derivation.
* build-aux/build-self.scm (build-program): Call guix-derivation with
#:graft? (%graft?) to make the compute-guix-derivation script use or not use
grafts as desired.

Signed-off-by: Christopher Baines <mail@cbaines.net>
---
build-aux/build-self.scm | 4 +-
guix/self.scm | 101 +++++++++++++++++++++++++++++++--------
2 files changed, 84 insertions(+), 21 deletions(-)

Toggle diff (173 lines)
diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 02822a2ee8..6d0037f20c 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -353,7 +353,9 @@ (define fake-git
#:channel-metadata
'#$channel-metadata
#:pull-version
- #$pull-version)
+ #$pull-version
+ #:graft?
+ #$(%graft?))
#:system system))
derivation-file-name))))))
#:module-path (list source))))
diff --git a/guix/self.scm b/guix/self.scm
index 74c953bd50..bbc0beaca8 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -21,6 +21,7 @@ (define-module (guix self)
#:use-module (guix config)
#:use-module (guix modules)
#:use-module (guix gexp)
+ #:use-module (guix grafts)
#:use-module (guix store)
#:use-module (guix monads)
#:use-module (guix discovery)
@@ -31,6 +32,7 @@ (define-module (guix self)
#:use-module ((guix build utils) #:select (find-files))
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-9 gnu)
#:use-module (srfi srfi-35)
#:use-module (ice-9 match)
#:export (make-config.scm
@@ -243,6 +245,50 @@ (define* (file-append* item file #:key (recursive? #t))
;; which isn't great.
(file-append item "/" file))))
+(define graft-derivation*
+ (store-lift graft-derivation))
+
+(define package-grafts*
+ (store-lift package-grafts))
+
+;; Apply grafts explicitly
+(define-immutable-record-type <explicit-grafting>
+ (%explicit-grafting obj packages)
+ explicit-grafting?
+ (obj explicit-grafting-obj) ;obj
+ (packages explicit-grafting-packages)) ;list of <package>s
+
+(define (write-explicit-grafting rec port)
+ (match rec
+ (($ <explicit-grafting> obj packages)
+ (format port "#<explicit-grafting ~s ~s>" obj packages))))
+
+(define (explicit-grafting obj packages)
+ (%explicit-grafting obj packages))
+
+(define-gexp-compiler (explicit-grafting-compiler (explicit-grafting <explicit-grafting>)
+ system target)
+ (match explicit-grafting
+ (($ <explicit-grafting> obj packages)
+ (mlet* %store-monad ((drv (without-grafting
+ (lower-object obj system #:target target)))
+ (grafts
+ (mapm %store-monad
+ (lambda (pkg)
+ (package-grafts* pkg system #:target target))
+ packages)))
+ (match (delete-duplicates
+ (concatenate grafts))
+ (()
+ (return drv))
+ (grafts
+ (mlet %store-monad ((guile (package->derivation
+ (guile-for-grafts)
+ system #:graft? #f)))
+ (graft-derivation* drv grafts
+ #:system system
+ #:guile guile))))))))
+
(define* (locale-data source domain
#:optional (directory domain))
"Return the locale data from 'po/DIRECTORY' in SOURCE, corresponding to
@@ -753,7 +799,8 @@ (define* (compiled-guix source #:key
(gzip (specification->package "gzip"))
(bzip2 (specification->package "bzip2"))
(xz (specification->package "xz"))
- (guix (specification->package "guix")))
+ (guix (specification->package "guix"))
+ (graft? #t))
"Return a file-like object that contains a compiled Guix."
(define guile-avahi
(specification->package "guile-avahi"))
@@ -1023,25 +1070,34 @@ (define (built-modules node-subset)
guile-lzma
dependencies)
#:guile guile-for-build
- #:guile-version guile-version)))
- (whole-package name modules dependencies
- #:command command
- #:guile guile-for-build
-
- ;; Include 'guix-daemon'. XXX: Here we inject an
- ;; older snapshot of guix-daemon, but that's a good
- ;; enough approximation for now.
- #:daemon (specification->package "guix-daemon")
-
- #:info (info-manual source)
- #:miscellany (miscellaneous-files source)
- #:guile-version guile-version)))
+ #:guile-version guile-version))
+ (obj
+ (whole-package name modules dependencies
+ #:command command
+ #:guile guile-for-build
+
+ ;; Include 'guix-daemon'. XXX: Here we inject
+ ;; an older snapshot of guix-daemon, but
+ ;; that's a good enough approximation for now.
+ #:daemon (specification->package "guix-daemon")
+
+ #:info (info-manual source)
+ #:miscellany (miscellaneous-files source)
+ #:guile-version guile-version)))
+ (if graft?
+ (explicit-grafting obj
+ (map (compose force cdr) %packages))
+ obj)))
((= 0 pull-version)
;; Legacy 'guix pull': return the .scm and .go files as one
;; directory.
- (built-modules (lambda (node)
- (list (node-source node)
- (node-compiled node)))))
+ (let ((obj (built-modules (lambda (node)
+ (list (node-source node)
+ (node-compiled node))))))
+ (if graft?
+ (explicit-grafting obj
+ (map (compose force cdr) %packages))
+ obj)))
(else
;; Unsupported 'guix pull' version.
#f)))
@@ -1271,7 +1327,8 @@ (define (process-directory directory files output)
(define* (guix-derivation source version
#:optional (guile-version (effective-version))
#:key (pull-version 0)
- channel-metadata)
+ channel-metadata
+ (graft? #t))
"Return, as a monadic value, the derivation to build the Guix from SOURCE
for GUILE-VERSION. Use VERSION as the version string. Use CHANNEL-METADATA
as the channel metadata sexp to include in (guix config).
@@ -1308,7 +1365,11 @@ (define guile
#:pull-version pull-version
#:guile-version (if (>= pull-version 1)
"3.0" guile-version)
- #:guile-for-build guile)))
+ #:guile-for-build guile
+ #:graft? graft?)))
(if guix
- (lower-object guix)
+ (if graft?
+ (lower-object guix)
+ (without-grafting
+ (lower-object guix)))
(return #f)))))
--
2.39.1
C
C
Christopher Baines wrote on 17 Apr 2023 17:06
Re: [bug#61363] [PATCH 2/2] self: Apply grafts to the outputs of the guix derivation.
(address . 61363@debbugs.gnu.org)
87bkjmfsy5.fsf@cbaines.net
Christopher Baines <mail@cbaines.net> writes:

Toggle quote (42 lines)
> [[PGP Signed Part:Undecided]]
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> 2. More importantly, manually listing packages that might require
>>> grafting looks like a slippery slope (“oops! we’re not getting the
>>> GnuTLS graft for that CVE, too bad”).
>>>
>>> I designed and implemented several variants to try and delay grafting.
>>> One of them consisted in carrying graft information in gexps:
>>>
>>> https://git.savannah.gnu.org/cgit/guix.git/log?h=wip-gexp-grafts
>>>
>>> It’s kinda similar to what you’re proposing in that graft information is
>>> carried as far as possible. The main difference is that it’s automated.
>>
>> That's interesting, I think that making grafting not specific to
>> packages, and something where the replacement is handled at a lower
>> level (e.g. gexps) would be an alternative way to handle this.
>>
>> Given that this approach works though, maybe the explicit-grafting
>> functionality could just sit and be used inside of (guix self). Given
>> that module is very explicit about what packages are used, it should be
>> possible to arrange the code so it's very hard to miss a package out,
>> which should address your concern about manually listing packages (maybe
>> specification->package can be tweaked so that it's possible to get all
>> the packages, and that can be the list considered for grafting).
>>
>> I don't know of any other places where this approach would be useful, so
>> while it would be nice to have a more general grafting mechanism
>> eventually, I'd also like to be able to make these changes to channel
>> instance grafts sooner rather than later.
>
> I've sent a v2 series which changes along the above lines. The explicit
> grafting stuff just sits in (guix self), and (guix self) more
> rigeriously uses it's own definition of specification->package, which
> should provide some protection against missing packages out. Obviously
> it's not quite as rigerous as moving the grafting functionality in to
> gexps, but hopefully it's rigerous enough for now.

This has stalled a bit, but it would be good to try and get things
merged. I've gone ahead and pushed the first two patches in the series I
last sent, these just make minor changes to prepare for the functional
change here. I've also resent that patch as as v3.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmQ9YLJfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xcl8BAAm4rgvPrHIUH95Ip/HYbEafd0LmxShmR2
Wo7T1NXRkriwnp7S4PZ7xMALF8ztK5ggzZBvWrHdRo52GiEWen9nxyj7a6/PMZ5y
IdWgjqJGLn4KNXo5E4FPEJ2ahgLtXwWcfF97sI8n4JSrrRB3B7cGO/0++P2I6ZsB
kOd1/cKQTKvn+I35umZijhGUNbgewv0T5m5Cj0CW9wQPUyYSj1LKsp0SmMbz36Uk
kVWtRE1rHbjqqMDqckbsqt8T2C4hagzkZXoRlKLvT2PJoQg8pjdoRMiEdzmz7WOU
BNUVN8ATuLUTQaYqu349YeYrcmJWBgLNnw6FKaFhbqrhy3TfUcuZ6a8KavyF6riN
4LXpfFZOz05J9l+VExd6d+jWzfn8YX5Hnvj1WNOOuNbZngLstpM2prMXbJr3Lfv7
pdrpOULYcE8sBIAQfUMH7Gs9Ca7ktyW/GSsaq2wdYdNKkT3uWqIkpnuae+u3tAZL
sOlRQ5qAsgc78AAcjscWxuhbvDjQ/ghtbLT4coIJ3S9Yk48bWkNzrZ6pTXZd8Fen
cHlY2pzQgObFbBamZdsV+xLFlMOHbt87IXP/WqwsGLKKCjQ8P4I7FPvuegZy2l5j
LNNmHsJKck4++Mr5HNH626jUKdgo0H9d5JjaF2mBahFlFUHj266SBuzVwnSsgtWI
YzNQZs7oCI0=
=1IL7
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 16 May 2023 15:25
Re: [bug#61363] [PATCH v3] self: Apply grafts to the outputs of the guix derivation.
(name . Christopher Baines)(address . mail@cbaines.net)
86353w76i6.fsf@gmail.com
Hi Chris,

I am late to the party and probably do not well understand all that
part. Just a quick comment in the same direction as Ludo.

On Mon, 17 Apr 2023 at 15:59, Christopher Baines <mail@cbaines.net> wrote:

Toggle quote (5 lines)
> diff --git a/guix/self.scm b/guix/self.scm
> index 74c953bd50..bbc0beaca8 100644
> --- a/guix/self.scm
> +++ b/guix/self.scm

[...]

Toggle quote (5 lines)
> + (if graft?
> + (explicit-grafting obj
> + (map (compose force cdr) %packages))
> + obj)))

[...]

Toggle quote (5 lines)
> + (if graft?
> + (explicit-grafting obj
> + (map (compose force cdr) %packages))
> + obj)))

It means that the grafts are only applied to %packages, right?

Other said, defined by:

Toggle snippet (33 lines)
(define %packages
(let ((ref (lambda (module variable)
(delay
(module-ref (resolve-interface
`(gnu packages ,module))
variable)))))
`(("guile" . ,(ref 'guile 'guile-3.0-latest))
("guile-avahi" . ,(ref 'guile-xyz 'guile-avahi))
("guile-json" . ,(ref 'guile 'guile-json-4))
("guile-ssh" . ,(ref 'ssh 'guile-ssh))
("guile-git" . ,(ref 'guile 'guile-git))
("guile-semver" . ,(ref 'guile-xyz 'guile-semver))
("guile-lib" . ,(ref 'guile-xyz 'guile-lib))
("guile-sqlite3" . ,(ref 'guile 'guile-sqlite3))
("guile-zlib" . ,(ref 'guile 'guile-zlib))
("guile-lzlib" . ,(ref 'guile 'guile-lzlib))
("guile-zstd" . ,(ref 'guile 'guile-zstd))
("guile-gcrypt" . ,(ref 'gnupg 'guile-gcrypt))
("guile-gnutls" . ,(ref 'tls 'guile-gnutls))
("guix-daemon" . ,(ref 'package-management 'guix-daemon))
("disarchive" . ,(ref 'backup 'disarchive))
("guile-lzma" . ,(ref 'guile 'guile-lzma))
("gzip" . ,(ref 'compression 'gzip))
("bzip2" . ,(ref 'compression 'bzip2))
("xz" . ,(ref 'compression 'xz))
("po4a" . ,(ref 'gettext 'po4a))
("gettext-minimal" . ,(ref 'gettext 'gettext-minimal))
("gcc-toolchain" . ,(ref 'commencement 'gcc-toolchain))
("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales))
("graphviz" . ,(ref 'graphviz 'graphviz))
("texinfo" . ,(ref 'texinfo 'texinfo)))))

tweaked by e5c33837cbee98d460d9ae09b463501de6f15d97. And there is a
slippery slope: the manual addition. These had been added with
e5c33837cbee98d460d9ae09b463501de6f15d97:

+ ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales))
+ ("graphviz" . ,(ref 'graphviz 'graphviz))
+ ("guix-daemon" . ,(ref 'package-management 'guix-daemon))
+ ("texinfo" . ,(ref 'texinfo 'texinfo)))))

Other said, what does it happen if we forget to manually update this
list?


Cheers,
simon
C
C
Christopher Baines wrote on 3 Jun 2023 13:41
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 61363@debbugs.gnu.org)
87bkhw6a9k.fsf@cbaines.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (74 lines)
> Hi Chris,
>
> I am late to the party and probably do not well understand all that
> part. Just a quick comment in the same direction as Ludo.
>
> On Mon, 17 Apr 2023 at 15:59, Christopher Baines <mail@cbaines.net> wrote:
>
>> diff --git a/guix/self.scm b/guix/self.scm
>> index 74c953bd50..bbc0beaca8 100644
>> --- a/guix/self.scm
>> +++ b/guix/self.scm
>
> [...]
>
>> + (if graft?
>> + (explicit-grafting obj
>> + (map (compose force cdr) %packages))
>> + obj)))
>
> [...]
>
>> + (if graft?
>> + (explicit-grafting obj
>> + (map (compose force cdr) %packages))
>> + obj)))
>
> It means that the grafts are only applied to %packages, right?
>
> Other said, defined by:
>
> (define %packages
> (let ((ref (lambda (module variable)
> (delay
> (module-ref (resolve-interface
> `(gnu packages ,module))
> variable)))))
> `(("guile" . ,(ref 'guile 'guile-3.0-latest))
> ("guile-avahi" . ,(ref 'guile-xyz 'guile-avahi))
> ("guile-json" . ,(ref 'guile 'guile-json-4))
> ("guile-ssh" . ,(ref 'ssh 'guile-ssh))
> ("guile-git" . ,(ref 'guile 'guile-git))
> ("guile-semver" . ,(ref 'guile-xyz 'guile-semver))
> ("guile-lib" . ,(ref 'guile-xyz 'guile-lib))
> ("guile-sqlite3" . ,(ref 'guile 'guile-sqlite3))
> ("guile-zlib" . ,(ref 'guile 'guile-zlib))
> ("guile-lzlib" . ,(ref 'guile 'guile-lzlib))
> ("guile-zstd" . ,(ref 'guile 'guile-zstd))
> ("guile-gcrypt" . ,(ref 'gnupg 'guile-gcrypt))
> ("guile-gnutls" . ,(ref 'tls 'guile-gnutls))
> ("guix-daemon" . ,(ref 'package-management 'guix-daemon))
> ("disarchive" . ,(ref 'backup 'disarchive))
> ("guile-lzma" . ,(ref 'guile 'guile-lzma))
> ("gzip" . ,(ref 'compression 'gzip))
> ("bzip2" . ,(ref 'compression 'bzip2))
> ("xz" . ,(ref 'compression 'xz))
> ("po4a" . ,(ref 'gettext 'po4a))
> ("gettext-minimal" . ,(ref 'gettext 'gettext-minimal))
> ("gcc-toolchain" . ,(ref 'commencement 'gcc-toolchain))
> ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales))
> ("graphviz" . ,(ref 'graphviz 'graphviz))
> ("texinfo" . ,(ref 'texinfo 'texinfo)))))
>
> tweaked by e5c33837cbee98d460d9ae09b463501de6f15d97. And there is a
> slippery slope: the manual addition. These had been added with
> e5c33837cbee98d460d9ae09b463501de6f15d97:
>
> + ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales))
> + ("graphviz" . ,(ref 'graphviz 'graphviz))
> + ("guix-daemon" . ,(ref 'package-management 'guix-daemon))
> + ("texinfo" . ,(ref 'texinfo 'texinfo)))))
>
> Other said, what does it happen if we forget to manually update this
> list?

Well, specification->package in (guix self) won't work for the missing
packages.

It's possible to use packages outside of this list, but that doesn't
happen currently.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmR7J4dfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XdA9Q//dkxQkEFKMrfuSmaL04m90ysfEDI1pa8e
ZPAXcD4UaEwHOPq/uKjmyAmUdsFnzRdsMIZjWJZkGh1UVJiGmA+g55h46F6Zf5Pb
jTHHvZeonmMSoi/YP5cLif4W0mvRpRoGRgtjdhBbHMEbfR3ww3AYoU8CrYs0AhHD
8C+pgnw6p+muWXQ4qVtaoXvN6EsyMzh++C4LrhMH7uTJp+C4Ki87LmAESTa8f7vl
4ti2k/Geawu/ENIouA8jG9SFOBOAGnGMacxd36XgpHBJs70e/oug5tkA/u9MwA20
pEXBXCDaZJiu1WkX3iLdiaQuRvE78pG8TJ2afl1iGIrG7HZ1E33kTXJqTYyYMcaH
8mBtar22hmRvvmQWLHUDchgA+wBeWWcuJZMLGQHrwV0zRyBPqD5HfQlKWRU4NFg+
PD+svkV9O1VkCVCqJ4Azm8CNE6FIyp+AuY+gdETZtWk+srkIC3yUD/30n3rssrKI
PqhfftkVQQNi4Ehdzt5ck7h+lIYBXXHv+nqitD1QtRVy1DCc+U9wAHRdMhbjBxYN
5Bp3xXOH4v/0+kLzx3GqfupU/8E7ik4xYAly6QlBUrIS7yiuOl7ktM/6uZkrJ3iM
UWIQCexXpQTXkrT7y1FIHPtTcPExgdLsotKj7LwCZlhPP6BlqvMeFIIoQDV8cKU2
xlwMQWIfyqw=
=0LsZ
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 3 Jun 2023 13:44
tag 61363 moreinfo
(address . control@debbugs.gnu.org)
875y84bwit.fsf@cbaines.net
tags 61363 + moreinfo
quit
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 61363@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 61363
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch