Incorrect propagation chain reporting on profile collision

  • Open
  • quality assurance status badge
Details
One participant
  • Ulf Herrman
Owner
unassigned
Submitted by
Ulf Herrman
Severity
normal
U
U
Ulf Herrman wrote on 5 May 2023 22:32
(address . bug-guix@gnu.org)
87v8h6r024.fsf@tilde.club
A <manifest-entry> has a `parent' field that contains a promise that
returns either #f or another <manifest-entry>. This is somewhat
incomplete information - a given entry can be a propagated-input to
multiple entries, so there can actually be multiple parents. Given that
this is only used for tracing a propagation chain to display to the
user, though, that shouldn't be a problem. What is a problem, though,
is that this field is determined when the manifest is read in and
remains static thereafter. To demonstrate this, consider the following
example:

User installs package A1, which propagates B1. User also installs
package C, which also propagates B1. Later the package definition for A
is updated with a newer version, A2, and this propagates a newer version
of B, B2. User runs 'guix pull', and subsequently runs 'guix package
-u'. Note that C is unchanged and still propagates B1. Guix sees that
the proposed new manifest contains A2, C, B1, and B2, and properly
detects that B1 and B2 represent a collision. It traces B2 back to A2
correctly, and then goes to hunt down what propagated B1. What happens
next depends somewhat on the arbitrary ordering of entries in the
manifest, but it is entirely possible that the parent recorded for B1 is
not C, but A1, even though the proposed new manifest doesn't even
contain A1.

So you end up with bizarre error messages like this:

-----------------------------------------
$ guix package -u certbot
The following package will be upgraded:
certbot 1.28.0 → 2.3.0

guix package: error: profile contains conflicting entries for python-cryptography
guix package: error: first entry: python-cryptography@40.0.1 /gnu/store/bcixgv8vg9bz6gyvijavqiha77h17icv-python-cryptography-40.0.1
guix package: error: ... propagated from python-josepy@1.13.0
guix package: error: ... propagated from python-acme@2.3.0
guix package: error: ... propagated from certbot@2.3.0
guix package: error: second entry: python-cryptography@3.4.8 /gnu/store/2kz701xhwvcswdp4dj3fd7v86v2ra70x-python-cryptography-3.4.8
guix package: error: ... propagated from python-josepy@1.13.0
guix package: error: ... propagated from python-acme@1.28.0
guix package: error: ... propagated from certbot@1.28.0
hint: You cannot have two different versions or variants of `certbot' in the same
profile.
-----------------------------------------

This happens because the collision check is performed on the new
manifest, and the parent fields are leftover from the old manifest.

Ultimately, the parent field is just an optimization - it can always be
re-derived from a manifest, and indeed, when a <manifest-entry> is
shared across multiple manifests, *must* be re-derived with respect to a
given manifest in order for it to make sense. I propose that we remove
the 'parent' field entirely and modify &profile-collision-error so that
it also reports the manifest the collision occurred in. (guix ui) can
then use this to provide more complete (and also accurate) error
messages. Note that at present check-for-collisions actually can report
manifest entries that aren't "part of" *any* manifest, since they're
freshly created by lower-manifest-entry.



Tangentially related: it's worth noting that only showing two
propagation chains tends to lead to frustrating "whack-a-mole"-style
package transactions, since a decision about which package to keep,
remove, or upgrade has to be made before the next collision source can
be found. We should probably add a profile mode to 'guix graph' that
shows the propagation graph for a proposed new profile manifest, or at
least let 'guix package' show all collisions when invoked with higher
verbosity.
-----BEGIN PGP SIGNATURE-----

iQHIBAEBCAAyFiEEn6BUn0yca1D9JsMa1lV76sJM9mgFAmRVaAkUHHN0cmluZXNz
QHRpbGRlLmNsdWIACgkQ1lV76sJM9mi53wv/bGzoi5Ro1ifiG0xtSn2Ej24FVZyv
XAQMHnLp7N15wb2e8Gbf+JpWp+bO0Iwhi17FaNCyaCcwrhcNl5uWkGczyvYsk0DV
OX4BrisqAc/5OFOo5KmrRqCZycAXr0uzd9o9o/XafSoaWjFymIkebLJGVsaEZcxG
MZbQ0EyORSsTlLO+m1BqzDWznDEN04zaoyIBA3tNO3hkFf5M0Smzp5vS+DolCuVE
YPqSH8JpS4Zl6hMKluuN65mkAM8s2QmvhWs3lujnNdLu61LdvXPZHTFBXJ5qcWOP
0w12K0CM4arj5AXX1S3PaRqnmfJWDW9PukKolVhtUOo/N9Apc/Z2msx9LrtHBGQG
RqH+5dPtxUpqqbagh9cz9vA3kbRapAqC/ba/dvcrNm1zwqu+CqjMZdmvKtaY4Bax
bfZF1/m5ubL9GiGHb54dsVejsaM7bpFC5JllJlpA6YVc6i2KkQzE39JMIDFoIlo1
sxNDKXSur1ef+C8t0TdCgddYmWBKUM9wjvaG
=E6NE
-----END PGP SIGNATURE-----

U
U
Ulf Herrman wrote on 8 May 2023 22:33
[PATCH 0/3]
(address . 63319@debbugs.gnu.org)
20230508203335.30659-1-striness@tilde.club
This patch series fixes issue #63319, making propagation chains leading to
profile collisions be reported properly. It does this by having
'check-for-collisions' fill in the &profile-collision-error with the manifest
and the lowered versions of the entries in question, and (guix ui) derive
parent information from the manifest. It then removes the now-unused 'parent'
field of <manifest-entry>.

Ulf Herrman (3):
profiles: include non-lowered entries and manifest in collision error.
ui: derive parents of profile collision entries from manifest.
profiles: remove `parent' field.

guix/inferior.scm | 36 ++++-----
guix/profiles.scm | 183 +++++++++++++++++++++++++---------------------
guix/ui.scm | 32 ++++++--
3 files changed, 142 insertions(+), 109 deletions(-)

--
2.39.1
U
U
Ulf Herrman wrote on 8 May 2023 22:33
[PATCH 1/3] profiles: include non-lowered entries and manifest in collision error.
(address . 63319@debbugs.gnu.org)
20230508203335.30659-2-striness@tilde.club
This provides the necessary information for (guix ui) to accurately determine
the actual entries causing the collision. The entries alone aren't enough,
since they inherit their parent (singular!) field from whatever it happened to
be before any manifest transaction was applied. The lowered variants are
included because (guix ui) needs them for reporting store paths, and the
non-lowered variants are included so that the proper parents can be derived
from the included manifest, which must contain them.

We also add and export a convenience procedure for finding the parents of menu
entries in a particular manifest.

* guix/profiles.scm (profile-collision-error-entry-lowered,
profile-collision-error-conflict-lowered, profile-collision-error-manifest):
new fields.
(check-for-collisions): populate them.
(manifest-entry->parents): new procedure.
* guix/ui.scm (call-with-error-handling): use lowered entries.
---
guix/profiles.scm | 60 ++++++++++++++++++++++++++++++++++++++---------
guix/ui.scm | 4 ++--
2 files changed, 51 insertions(+), 13 deletions(-)

Toggle diff (126 lines)
diff --git a/guix/profiles.scm b/guix/profiles.scm
index 03333785f9..b812a6f7d9 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -64,7 +64,10 @@ (define-module (guix profiles)
&profile-collision-error
profile-collision-error?
profile-collision-error-entry
+ profile-collision-error-entry-lowered
profile-collision-error-conflict
+ profile-collision-error-conflict-lowered
+ profile-collision-error-manifest
&missing-generation-error
missing-generation-error?
missing-generation-error-generation
@@ -107,6 +110,7 @@ (define-module (guix profiles)
manifest-installed?
manifest-matching-entries
manifest-search-paths
+ manifest-entry->parents
check-for-collisions
manifest->code
@@ -186,7 +190,10 @@ (define-condition-type &profile-not-found-error &profile-error
(define-condition-type &profile-collision-error &error
profile-collision-error?
(entry profile-collision-error-entry) ;<manifest-entry>
- (conflict profile-collision-error-conflict)) ;<manifest-entry>
+ (conflict profile-collision-error-conflict) ;<manifest-entry>
+ (entry-lowered profile-collision-error-entry-lowered) ;<manifest-entry>
+ (conflict-lowered profile-collision-error-conflict-lowered) ;<manifest-entry>
+ (manifest profile-collision-error-manifest)) ;<manifest>
(define-condition-type &unmatched-pattern-error &error
unmatched-pattern-error?
@@ -329,6 +336,34 @@ (define (recurse entry)
(item (derivation->output-path drv output))
(dependencies dependencies)))))))
+(define (manifest-entry->parents manifest)
+ "Return a procedure that maps each <manifest-entry> in MANIFEST to the list
+of <manifest-entry>s in MANIFEST or their dependencies, recursively, that
+have the entry in question as a direct dependency."
+ (define (visit-entries entries mapping visited?)
+ (match entries
+ (((and entry ($ <manifest-entry> _ _ _ _ dependencies)) . rest)
+ (if (vhash-assq entry visited?)
+ (visit-entries rest mapping visited?)
+ (call-with-values
+ (lambda ()
+ (visit-entries dependencies
+ (fold (lambda (dependency mapping)
+ (vhash-consq dependency entry mapping))
+ mapping
+ dependencies)
+ (vhash-consq entry #t visited?)))
+ (lambda (mapping visited?)
+ (visit-entries rest mapping visited?)))))
+ (()
+ (values mapping visited?))))
+
+ (define mapping
+ (visit-entries (manifest-entries manifest) vlist-null vlist-null))
+
+ (lambda (entry)
+ (vhash-foldq* cons '() entry mapping)))
+
(define* (check-for-collisions manifest system #:key target)
"Check whether the entries of MANIFEST conflict with one another; raise a
'&profile-collision-error' when a conflict is encountered."
@@ -348,25 +383,28 @@ (define candidates
(define lower-pair
(match-lambda
((first second)
- (mlet %store-monad ((first (lower-manifest-entry first system
- #:target target))
- (second (lower-manifest-entry second system
- #:target target)))
- (return (list first second))))))
+ (mlet %store-monad ((first-low (lower-manifest-entry first system
+ #:target target))
+ (second-low (lower-manifest-entry second system
+ #:target target)))
+ (return (list first first-low second second-low))))))
;; Start by lowering CANDIDATES "in parallel".
- (mlet %store-monad ((lst (mapm/accumulate-builds lower-pair candidates)))
+ (mlet* %store-monad ((lst (mapm/accumulate-builds lower-pair candidates)))
(foldm %store-monad
(lambda (entries result)
(match entries
- ((first second)
- (if (string=? (manifest-entry-item first)
- (manifest-entry-item second))
+ ((first first-low second second-low)
+ (if (string=? (manifest-entry-item first-low)
+ (manifest-entry-item second-low))
(return result)
(raise (condition
(&profile-collision-error
(entry first)
- (conflict second))))))))
+ (entry-lowered first-low)
+ (conflict second)
+ (conflict-lowered second-low)
+ (manifest manifest))))))))
#t
lst)))
diff --git a/guix/ui.scm b/guix/ui.scm
index d75243458d..5d2ae23c25 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -750,8 +750,8 @@ (define (port-filename* port)
("out" #f)
(output output)))))
((profile-collision-error? c)
- (let ((entry (profile-collision-error-entry c))
- (conflict (profile-collision-error-conflict c)))
+ (let ((entry (profile-collision-error-entry-lowered c))
+ (conflict (profile-collision-error-conflict-lowered c)))
(define (report-parent-entries entry)
(let ((parent (force (manifest-entry-parent entry))))
(when (manifest-entry? parent)
--
2.39.1
U
U
Ulf Herrman wrote on 8 May 2023 22:33
[PATCH 2/3] ui: derive parents of profile collision entries from manifest.
(address . 63319@debbugs.gnu.org)
20230508203335.30659-3-striness@tilde.club

* guix/ui.scm (display-collision-resolution-hint, call-with-error-handling):
use manifest-entry->parents and the manifest included with the collision to
get the parents.
---
guix/ui.scm | 30 +++++++++++++++++++++++++-----
1 file changed, 25 insertions(+), 5 deletions(-)

Toggle diff (65 lines)
diff --git a/guix/ui.scm b/guix/ui.scm
index 5d2ae23c25..71fe5caa72 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -674,9 +674,19 @@ (define unit
(define (display-collision-resolution-hint collision)
"Display hints on how to resolve COLLISION, a &profile-collistion-error."
+ (define manifest (profile-collision-error-manifest collision))
+
+ (define entry->parents
+ (manifest-entry->parents manifest))
+
+ (define (manifest-entry-parent entry)
+ (match (entry->parents entry)
+ (() #f)
+ ((x . _) x)))
+
(define (top-most-entry entry)
(let loop ((entry entry))
- (match (force (manifest-entry-parent entry))
+ (match (manifest-entry-parent entry)
(#f entry)
(parent (loop parent)))))
@@ -751,9 +761,19 @@ (define (port-filename* port)
(output output)))))
((profile-collision-error? c)
(let ((entry (profile-collision-error-entry-lowered c))
- (conflict (profile-collision-error-conflict-lowered c)))
+ (conflict (profile-collision-error-conflict-lowered c))
+ (manifest (profile-collision-error-manifest c)))
+
+ (define entry->parents
+ (manifest-entry->parents manifest))
+
+ (define (manifest-entry-parent entry)
+ (match (entry->parents entry)
+ (() #f)
+ ((x . rest) x)))
+
(define (report-parent-entries entry)
- (let ((parent (force (manifest-entry-parent entry))))
+ (let ((parent (manifest-entry-parent entry)))
(when (manifest-entry? parent)
(report-error (G_ " ... propagated from ~a@~a~%")
(manifest-entry-name parent)
@@ -773,13 +793,13 @@ (define (manifest-entry-output* entry)
(manifest-entry-version entry)
(manifest-entry-output* entry)
(manifest-entry-item entry))
- (report-parent-entries entry)
+ (report-parent-entries (profile-collision-error-entry c))
(report-error (G_ " second entry: ~a@~a~a ~a~%")
(manifest-entry-name conflict)
(manifest-entry-version conflict)
(manifest-entry-output* conflict)
(manifest-entry-item conflict))
- (report-parent-entries conflict)
+ (report-parent-entries (profile-collision-error-conflict c))
(display-collision-resolution-hint c)
(exit 1)))
((nar-error? c)
--
2.39.1
U
U
Ulf Herrman wrote on 8 May 2023 22:33
[PATCH 3/3] profiles: remove `parent' field.
(address . 63319@debbugs.gnu.org)
20230508203335.30659-4-striness@tilde.club
This field was only present for consumption by (guix ui) when reporting
propagation chains that lead to profile collision errors, but it is only valid
in general with respect to a single manifest. (guix ui) now derives parent
information by itself with respect to an explicit manifest, so this field is
no longer needed.

* guix/profiles.scm (manifest-entry-parent): remove field.
(package->manifest-entry, sexp->manifest): do not populate it.
(manifest->gexp): adjust match specifications to account for its absence.
* guix/inferior.scm (inferior-package->manifest-entry): do not populate
nonexistent parent field.
---
guix/inferior.scm | 36 ++++++--------
guix/profiles.scm | 123 +++++++++++++++++++---------------------------
2 files changed, 67 insertions(+), 92 deletions(-)

Toggle diff (259 lines)
diff --git a/guix/inferior.scm b/guix/inferior.scm
index 5dfd30a6c8..4030640f6d 100644
--- a/guix/inferior.scm
+++ b/guix/inferior.scm
@@ -819,27 +819,23 @@ (define-syntax-rule (memoized package output exp)
result))))
(let loop ((package package)
- (output output)
- (parent (delay #f)))
+ (output output))
(memoized package output
- ;; For each dependency, keep a promise pointing to its "parent" entry.
- (letrec* ((deps (map (match-lambda
- ((label package)
- (loop package "out" (delay entry)))
- ((label package output)
- (loop package output (delay entry))))
- (inferior-package-propagated-inputs package)))
- (entry (manifest-entry
- (name (inferior-package-name package))
- (version (inferior-package-version package))
- (output output)
- (item package)
- (dependencies (delete-duplicates deps))
- (search-paths
- (inferior-package-transitive-native-search-paths package))
- (parent parent)
- (properties properties))))
- entry))))
+ (let ((deps (map (match-lambda
+ ((label package)
+ (loop package "out"))
+ ((label package output)
+ (loop package output)))
+ (inferior-package-propagated-inputs package))))
+ (manifest-entry
+ (name (inferior-package-name package))
+ (version (inferior-package-version package))
+ (output output)
+ (item package)
+ (dependencies (delete-duplicates deps))
+ (search-paths
+ (inferior-package-transitive-native-search-paths package))
+ (properties properties))))))
;;;
diff --git a/guix/profiles.scm b/guix/profiles.scm
index b812a6f7d9..0d22667362 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -90,7 +90,6 @@ (define-module (guix profiles)
manifest-entry-item
manifest-entry-dependencies
manifest-entry-search-paths
- manifest-entry-parent
manifest-entry-properties
lower-manifest-entry
@@ -229,8 +228,6 @@ (define-record-type* <manifest-entry> manifest-entry
(default '()))
(search-paths manifest-entry-search-paths ; search-path-specification*
(default '()))
- (parent manifest-entry-parent ; promise (#f | <manifest-entry>)
- (default (delay #f)))
(properties manifest-entry-properties ; list of symbol/value pairs
(default '())))
@@ -416,29 +413,23 @@ (define (default-properties package)
(transformations `((transformations . ,transformations)))))
(define* (package->manifest-entry package #:optional (output "out")
- #:key (parent (delay #f))
(properties (default-properties package)))
"Return a manifest entry for the OUTPUT of package PACKAGE."
- ;; For each dependency, keep a promise pointing to its "parent" entry.
- (letrec* ((deps (map (match-lambda
- ((label package)
- (package->manifest-entry package
- #:parent (delay entry)))
- ((label package output)
- (package->manifest-entry package output
- #:parent (delay entry))))
- (package-propagated-inputs package)))
- (entry (manifest-entry
- (name (package-name package))
- (version (package-version package))
- (output output)
- (item package)
- (dependencies (delete-duplicates deps))
- (search-paths
- (package-transitive-native-search-paths package))
- (parent parent)
- (properties properties))))
- entry))
+ (let ((deps (map (match-lambda
+ ((label package)
+ (package->manifest-entry package))
+ ((label package output)
+ (package->manifest-entry package output)))
+ (package-propagated-inputs package))))
+ (manifest-entry
+ (name (package-name package))
+ (version (package-version package))
+ (output output)
+ (item package)
+ (dependencies (delete-duplicates deps))
+ (search-paths
+ (package-transitive-native-search-paths package))
+ (properties properties))))
(define* (package->development-manifest package
#:optional
@@ -534,7 +525,7 @@ (define (entry->gexp entry)
(return
(match entry
(($ <manifest-entry> name version output (? string? path)
- (_ ...) (search-paths ...) _ (properties ...))
+ (_ ...) (search-paths ...) (properties ...))
#~(#$name #$version #$output #$path
#$@(optional 'propagated-inputs deps)
#$@(optional 'search-paths
@@ -542,7 +533,7 @@ (define (entry->gexp entry)
search-paths))
#$@(optional 'properties properties)))
(($ <manifest-entry> name version output package
- (_deps ...) (search-paths ...) _ (properties ...))
+ (_deps ...) (search-paths ...) (properties ...))
#~(#$name #$version #$output
(ungexp package (or output "out"))
#$@(optional 'propagated-inputs deps)
@@ -565,7 +556,7 @@ (define (entry->gexp entry)
(define (sexp->manifest sexp)
"Parse SEXP as a manifest."
- (define (infer-dependency item parent)
+ (define (infer-dependency item)
;; Return a <manifest-entry> for ITEM.
(let-values (((name version)
(package-name->name+version
@@ -573,31 +564,25 @@ (define (infer-dependency item parent)
(manifest-entry
(name name)
(version version)
- (item item)
- (parent parent))))
+ (item item))))
- (define* (sexp->manifest-entry/v3 sexp #:optional (parent (delay #f)))
+ (define* (sexp->manifest-entry/v3 sexp)
;; Read SEXP as a version 3 manifest entry.
(match sexp
((name version output path
('propagated-inputs deps)
('search-paths search-paths)
extra-stuff ...)
- ;; For each of DEPS, keep a promise pointing to ENTRY.
- (letrec* ((deps* (map (cut sexp->manifest-entry/v3 <> (delay entry))
- deps))
- (entry (manifest-entry
- (name name)
- (version version)
- (output output)
- (item path)
- (dependencies deps*)
- (search-paths (map sexp->search-path-specification
- search-paths))
- (parent parent)
- (properties (or (assoc-ref extra-stuff 'properties)
- '())))))
- entry))))
+ (manifest-entry
+ (name name)
+ (version version)
+ (output output)
+ (item path)
+ (dependencies (map sexp->manifest-entry/v3 deps))
+ (search-paths (map sexp->search-path-specification
+ search-paths))
+ (properties (or (assoc-ref extra-stuff 'properties)
+ '()))))))
(define-syntax let-fields
(syntax-rules ()
@@ -611,7 +596,7 @@ (define-syntax let-fields
((_ lst () body ...)
(begin body ...))))
- (define* (sexp->manifest-entry sexp #:optional (parent (delay #f)))
+ (define* (sexp->manifest-entry sexp)
(match sexp
(('repeated name version path)
;; This entry is the same as another one encountered earlier; look it
@@ -628,23 +613,20 @@ (define* (sexp->manifest-entry sexp #:optional (parent (delay #f)))
((name version output path fields ...)
(let-fields fields (propagated-inputs search-paths properties)
(mlet* %state-monad
- ((entry -> #f)
- (deps (mapm %state-monad
- (cut sexp->manifest-entry <> (delay entry))
+ ((deps (mapm %state-monad
+ sexp->manifest-entry
propagated-inputs))
+ (entry -> (manifest-entry
+ (name name)
+ (version version)
+ (output output)
+ (item path)
+ (dependencies deps)
+ (search-paths (map sexp->search-path-specification
+ search-paths))
+ (properties properties)))
(visited (current-state))
(key -> (list name version path)))
- (set! entry ;XXX: emulate 'letrec*'
- (manifest-entry
- (name name)
- (version version)
- (output output)
- (item path)
- (dependencies deps)
- (search-paths (map sexp->search-path-specification
- search-paths))
- (parent parent)
- (properties properties)))
(mbegin %state-monad
(set-current-state (vhash-cons key entry visited))
(return entry)))))))
@@ -661,18 +643,15 @@ (define* (sexp->manifest-entry sexp #:optional (parent (delay #f)))
...)))
(manifest
(map (lambda (name version output path deps search-paths)
- (letrec* ((deps* (map (cute infer-dependency <> (delay entry))
- deps))
- (entry (manifest-entry
- (name name)
- (version version)
- (output output)
- (item path)
- (dependencies deps*)
- (search-paths
- (map sexp->search-path-specification
- search-paths)))))
- entry))
+ (manifest-entry
+ (name name)
+ (version version)
+ (output output)
+ (item path)
+ (dependencies (map infer-dependency deps))
+ (search-paths
+ (map sexp->search-path-specification
+ search-paths))))
name version output path deps search-paths)))
;; Version 3 represents DEPS as full-blown manifest entries.
--
2.39.1
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 63319@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 63319
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch