Cookbook recipe about profile collisions

Recently, Matt pointed out that profile collisions can be confusing and

difficult to resolve:

Specifically see this blog post:

I think we should add a Cookbook chapter on the subject.

This particular error state seems unavoidable given the size of the

distro and the number of packages that use propagation.

So, let's help users understand it and give them the knowledge to even

help resolve such collisions as contributors.

Speaking as somebody who understands intimately what a profile is, what

propagated inputs are, and how they can interact to cause "profile

contains conflicting entries", I am sympathetic.

Although I know exactly how to resolve such errors, I have also

explained the situation to new Guix users several times on IRC.

Many people come to Guix, install some packages, use them, then want to

upgrade them, and they have never learned what a profile is or how it is

the fundamental mechanism by which Guix implements package management.

It also seems to me that plenty of people would prefer not to ask for

help, but rather go searching online and in the documentation, which

does not include the string "contains conflicting entries".

I envision a Cookbook chapter that explains what a profile is, what

propagation does, what profile generations are, and then how they all

combine to cause this error state. And an example of a resolution,

whether by adjusting a package definition, uninstalling a package, etc.

There are some older Guix presentation videos that I think do a great

job of explaining profiles and generations, including slides with good

illustrations of the subject. It would be nice to try using that style

of illustration, as I found it clear and illuminating when I first

learned about Guix.

Hi,

Leo Famulari <leo@famulari.name> skribis:

I don’t see the words “profile” and “collision” here; maybe that was

upthread?

Yeah, we need to get a good understanding of when that happens, and

what’s confusing.

Currently, on profile collisions, the error message shows where the

collision originates and a hint on how to work around it. Perhaps the

hint is sometimes wrong (in which cases?), or perhaps it’s too terse?

Can it be improved?

The definition of what a profile is is another topic. Currently the

term “profile” is defined in “Getting Started”:

https://guix.gnu.org/manual/en/html_node/Getting-Started.html#index-profile

It’s very much defined in passing though. How can we improve on that?

In some early talks we had illustrations of the symlink forest of a

profile borrowed from Eelco Dolstra’s own talks on the matter; see for

instance p. 17 of https://guix.gnu.org/guix-fosdem-20140201.pdf. I

stopped using it because I think those symlinks are an implementation

detail and it doesn’t help to focus on symlinks (and hashes and all

that) when giving an overview of the tool. Now, perhaps that

illustration could be useful in the manual.

WDYT?

Thanks,

Ludo’.

Hi,

On Fri, 14 Jan 2022 at 10:02, Ludovic Courtès <ludo@gnu.org> wrote:

I do not know how it could be improved since it is often case per

case. For instance, recently we had a collision for python-numpy and

the fix was to introduce python-numpy-next. That's said, to me the

origin of the collision is often enough for resolving. However,

indeed the user must have a clear understanding of Guix profile and

propagation. Therefore, an entry to the Cookbook explaining what does

it mean and where to look appears to me a good idea.

Well, I do not want to drift the initial message. I am just keeping

the rolling ball. :-)

In the tutorial I am preparing for JRES, I am doing this way, as a

dialogue (below). Basically, it is how I am somehow explaining to the

few users I have here, with various background. This cannot go to the

manual but maybe it could be useful or worth for the Cookbook.

To me, what is an implementation detail is all the dance with /var/,

the generations, the garbage collection, etc. But the fact that a

profile is symlink pointing a directory with the structure of /usr/

and containing all the environment variables is not an implementation

detail. It appears to me a core concept to switch from traditional

distro with only one /usr/ to Guix and how to compose many /usr/.

Alice: Where are my packages?

Bob: Installing a package ends with this hint ...snippet...

Alice: Ah, I am confused

Bob: Well, $HOME/.guix-profile is thus the default profile

Alice: What does it mean?

Bob: Look at guix package --search-paths -p $HOME/.guix-profile

Alice: I see, but what is this $HOME/.guix-profile

Bob: Look at: file $HOME/.guix-profile

Alice: Ah, right. My conclusion is:

- a profile contains the environment variables

- a profile is a symlink

Bob: Correct.

Alice: But a symlink pointing to where?

Bob: Look at: readlink -f $HOME/.guix-profile

Alice: It points to a store item, but what does it mean?

Bob: Look at: file $(readlink -f $HOME/.guix-profile)

Alice: Ah right. My conclusion is:

- a profile points to a directory item in the store

Bob: Correct.

Alice: What is inside this directory

Bob: Look at: ls $(readlink -f $HOME/.guix-profile)

Alice: Oh, it looks the same hierarchy as the usual and well-know /usr/

Bob: Correct.

Alice: What does it happen when I type python3 I just installed

Bob: Look at: which python3

Alice: Ah right. Is the binary really in the directory item above?

Bob: Look at: readlink $(which python3)

Alice: Oh right. A profile points to store item which points to other

store items.

Bob: Correct.

Alice: I guess these other store items have also an /usr/ hierarchy, right?

Bob: Correct. Do you know how to check that?

Alice: ls $(readlink $(which python3))

Bob: Correct.

Alice: I summarize. A profile is:

- a symlink

- which points to a store item

- which points to many other store items

- and all have the structure hierarchy of /usr/

Woo, it is a forest of symlinks!

... image ...

Cheers,

simon

On Fri, Jan 14, 2022 at 09:47:52AM +0100, Ludovic Courtès wrote:

I pointed to this blog post, which was linked from that email:

It's the story of an attempt to learn what the "conflicting entries"

message means and how to resolve it.

The hint is perfectly clear, when you understand "profiles" and

"propagation".

It's like I said later in my message: People use Guix without knowing

very much about it.

That's surprising to me, because I started using Guix *because* I

learned about profiles and how they are used: it's one of the core

innovations of Guix / Nix, to implement unprivileged package management.

But nevertheless, people are using Guix without understanding how it

works. And it's hard to learn about Guix when you are dealing with a

profile collision that you don't understand at all. For many people,

that is not a good moment to start learning. On top of that, profile

collisions are an error state that we can't fix as a bug: we have to

give users the knowledge to resolve the collisions themselves.

For example, the blog post that I linked to ends with "Lessons Learned",

which includes this:

------

What is a profile?

A profile is a directory of symlinks located at ~/.guix-profile.

Propagated inputs can cause conflicts

Propagated inputs are treated somehow differently. It's still not 100%

clear how or why, but it's good to know they're a potential source of

errors.

------

This person spent a lot of time trying to understand the situation and

writing the blog post, but their understanding is still rather weak.

That's why I propose a Cookbook chapter that specifically addresses

profile collisions, to help new users go from "oh no, an error message"

to "aha!"

I think this part of the manual is fine and can stay as it is.

The illustrations of the symlink forest were *extremely* helpful to me

when learning how Guix implements unprivileged package management. I

think that later talks from the 2015-2017 era refined the illustrations

to be even more clear.

I'm sorry if the use case for my proposal is still unclear. I can work

on the Cookbook chapter myself.

Hi Leo,

Leo Famulari <leo@famulari.name> skribis:

Oh sorry, I had overlooked this. It makes perfect sense now, and this

blog post is useful in understand what it feels like to encounter that

error message for the first time.

Yeah. So, given that “profile” is already defined in “Getting Started”,

I think we can’t do a lot more in the manual.

However, as a way of helping users incrementally discover concepts they

need to understand, we could use one-time hints (sorta like the “Did you

know?” boxes in GUIs.) ‘guix shell’ already uses a one-time hint

inviting users to run ‘--check’.

We could imagine that the first run of ‘guix install’ & co. would print

a message explaining that your packages’ files are now in

~/.guix-profile, and that it’s what we call a “profile”.

Would that make sense?

To be fair, the person didn’t look for “profile” in the manual. I’m not

blaming—mentioning it to clarify what it is we’re trying to fix.

Makes sense!

[...]

OK, let’s see if we can include an illustration like that under

“Managing Software the Guix Way” or something like that.

Great, thanks.

Ludo’.

---- On Mon, 17 Jan 2022 09:16:28 -0500 Ludovic Courtès <ludo@gnu.org> wrote ----

> > This person spent a lot of time trying to understand the situation and

> > writing the blog post, but their understanding is still rather weak.

>

> To be fair, the person didn’t look for “profile” in the manual. I’m not

> blaming—mentioning it to clarify what it is we’re trying to fix.

With all respect, I *did* look at "profile" in the manual. I spent a lot of time looking and trying to understand things. I hear you say you're not trying to blame me and I trust that's not your intent. However, what was said *does* blame me. It says what I did and did not do, independent of reality: you are not me and you were not present. Unfortunately, what was said carries all sorts of judgments and implications (ouch!) which, opposite to your intent, is not fair.

I see you want clarification on what we're trying to fix. May I suggest instead asking, "What problem are we trying to solve?"

I see several problems beyond what I've already said. However, I'll try to stick to just one that I encountered with the documentation. Leo is certainly working toward something specific which I suspect is different from what I see. I'll let them speak for themselves.

I'm going to assume that you probably wanted to ask something like, "It looks to me like the manual adequately explains profiles. But, since I'm the main architect of this system, maybe I have knowledge that someone new doesn't have. Person who is confused, are you able to say where you're confused?"

I would reply:

There are several places I've been confused. Let me give you a specific example.

The manual has at least four places that "profile" is defined:

1. [[https://guix.gnu.org/en/manual/en/html_node/Getting-Started.html#Getting-Started][(guix)Getting Started]] says,

#+begin_quote

"A profile is a directory containing installed packages"

#+end_quote

2. [[https://guix.gnu.org/en/manual/en/html_node/Invoking-guix-package.html#Invoking-guix-package][(guix)Invoking guix package]] says,

#+begin_quote

"a directory of installed packages"

#+end_quote

and 3, yet in the guix package options:

#+begin_quote

'-p PROFILE'

Use PROFILE instead of the user's default profile.

PROFILE must be the name of a file that will be created upon

completion. Concretely, PROFILE will be a mere symbolic link

("symlink") pointing to the actual profile where packages are

installed:

#+begin_example

$ guix install hello -p ~/code/my-profile

...

$ ~/code/my-profile/bin/hello

Hello, world!

#+end_example

#+end_quote

Elsewhere in (guix) Features is a 4th which says,

#+begin_quote

users have their own “profile”, which points to the packages that they actually want to use

#+end_quote

So, is the profile a directory or a pointer file (e.g. symlink)? I tend to think of a directory as a container, like a manilla folder that contains papers. To me, something that points is a file that contains a path to another location. I see that I used the word "contains" to describe both file and directory, so maybe that's a sign to me I'm missing something there.

Regardless, I hope you can see that it's not always clear whether a profile is a directory or a file. Yes, on Unix-like systems, directories are files. But Guix throws an error if you call =guix package -p= with a directory:

: guix package: error: rename-file: Is a directory

If you follow the symlinks, the profile is indeed a directory; it is a directory in the store. But the way users interact with profiles is

GUIX_PROFILE="$HOME/.guix-profile"

. "$GUIX_PROFILE/etc/profile"

which is a file. And there are a bunch of symlinks in general. Those appear to be implementation details. But I think it's reasonable to say the abstraction isn't airtight and that, as a user, I have to interact with the implementation details at some level. Certainly at the documentation level.

Leo is 100% correct that my understanding is still rather weak. I'll do my best despite that to help make the documentation better.

On Mon, Jan 17, 2022 at 12:56:20PM -0500, Matt wrote:

I hope you will not feel too bad about that. Remember, everyone begins

by not knowing anything. Your situation is not unique at all. Rather,

your energy for improving the documentation for yourself and others is

exemplary, and will improve Guix in the long run.

Overall, this highlights a case where there is tension between when an

implementation detail doesn't matter, and when it does. That is, the

ideal situation is that the implementation details of profiles do not

matter. However, when there are "profile collisions", it does matter.

Hi,

Leo Famulari <leo@famulari.name> skribis:

I agree, and I apologize for throwing my own guess of what you did and

did not do, Matt.

Ludo’.