'guix shell' skips profile collision checks

  • Open
  • quality assurance status badge
Details
6 participants
  • Greg Hogan
  • Konrad Hinsen
  • Liliana Marie Prikler
  • Ludovic Courtès
  • Maxime Devos
  • Suhail
Owner
unassigned
Submitted by
Maxime Devos
Severity
important
M
M
Maxime Devos wrote on 12 Mar 2022 10:45
Profile collisions are ignored, installing multiple versions of the same package is silently broken
(address . bug-guix@gnu.org)
afde4231155f7fe32e709dc0d3a4c12d71ea8011.camel@telenet.be
Hi guix,

Try

$ guix shell --pure bash pango@1.42 pango@1.48
$ ~/.guix-profile/bin/ls -l $GUIX_ENVIRONMENT/lib/libpango*.so

Result:
[...]/lib/libpango-1.0.so -> [...]-pango-1.42.4/lib/libpango-1.0.so
[...]/lib/libpangocairo-1.0.so -> [...]-pango-1.42.4/lib/libpangocairo-1.0.so
[...]/lib/libpangoft2-1.0.so -> [...]-1.42.4/lib/libpangoft2-1.0.so
[...]/lib/libpangoxft-1.0.so -> [...]-pango-1.42.4/lib/libpangoxft-1.0.so

Question: how does this even work? Aren't error messages about profile
collisions a thing? Another question: if this use case was supported,
shouldn't it point to the newer pango instead of the older pango?

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYixryhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7scMAP4m3Lc0IrXnN283dLcFjqCQLEKj
kOFxW92uoNb34Izj0wEA5SlU+JT7Dmb2aklTtFWmYyjRmErfv3nSeiS5J6Adsgc=
=9KfV
-----END PGP SIGNATURE-----


L
L
Liliana Marie Prikler wrote on 12 Mar 2022 15:34
80793191d3eea8c53cec9a61515905327844c73e.camel@gmail.com
Am Samstag, dem 12.03.2022 um 10:45 +0100 schrieb Maxime Devos:
Toggle quote (17 lines)
> Hi guix,
>
> Try
>
> $ guix shell --pure bash pango@1.42 pango@1.48
> $ ~/.guix-profile/bin/ls -l $GUIX_ENVIRONMENT/lib/libpango*.so
>
> Result:
> [...]/lib/libpango-1.0.so -> [...]-pango-1.42.4/lib/libpango-1.0.so
> [...]/lib/libpangocairo-1.0.so -> [...]-pango-
> 1.42.4/lib/libpangocairo-1.0.so
> [...]/lib/libpangoft2-1.0.so -> [...]-1.42.4/lib/libpangoft2-1.0.so
> [...]/lib/libpangoxft-1.0.so -> [...]-pango-1.42.4/lib/libpangoxft-
> 1.0.so
>
> Question: how does this even work?  Aren't error messages about
> profile collisions a thing? 
They ought to be, but perhaps these are only checked for propagated
packages, not for explicitly specified ones. Would still be weird, if
you e.g. put pango@1.48 and something propagating pango@1.42 together,
it ought to error. Perhaps an oversight?

Toggle quote (2 lines)
> Another question: if this use case was supported, shouldn't it point
> to the newer pango instead of the older pango?
union-build implicitly takes the first file it finds -- this is
actually sane, but sadly undocumented behaviour. In other words, if
you specify pango@1.48 first, it should be the one being picked.

Cheers
L
L
Ludovic Courtès wrote on 15 Mar 2022 14:50
Re: bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 54350@debbugs.gnu.org)
87zglrjpef.fsf@gnu.org
Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (12 lines)
> $ guix shell --pure bash pango@1.42 pango@1.48
> $ ~/.guix-profile/bin/ls -l $GUIX_ENVIRONMENT/lib/libpango*.so
>
> Result:
> [...]/lib/libpango-1.0.so -> [...]-pango-1.42.4/lib/libpango-1.0.so
> [...]/lib/libpangocairo-1.0.so -> [...]-pango-1.42.4/lib/libpangocairo-1.0.so
> [...]/lib/libpangoft2-1.0.so -> [...]-1.42.4/lib/libpangoft2-1.0.so
> [...]/lib/libpangoxft-1.0.so -> [...]-pango-1.42.4/lib/libpangoxft-1.0.so
>
> Question: how does this even work? Aren't error messages about profile
> collisions a thing?

Unlike ‘guix package’, ‘guix environment/shell’ explicitly disables
collision checks—see #:allow-collisions? in environment.scm.

The rationale back then was that if you do:

guix shell -D guile

then you likely end up with profile collisions, like so (I modified the
code to do #:allow-collisions? #f):

Toggle snippet (7 lines)
$ ./pre-inst-env guix shell -D guile -n
guix shell: error: profile contains conflicting entries for bash-minimal
guix shell: error: first entry: bash-minimal@5.1.8 /gnu/store/chfwin3a4qp1znnpsjbmydr2jbzk0d6y-bash-minimal-5.1.8
guix shell: error: second entry: bash-minimal@5.1.8 /gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8
hint: You cannot have two different versions or variants of `bash-minimal' in the same profile.

These collisions are harmless but frequent in package development
environments, which is why ‘guix environment’ had to turn off those
checks (commit afd06f605bf88a796acefc7ed598b43879346a6b).

That’s a bit of a hack though.

What we could do, as a mitigation, is to enable collision checks unless
there’s at least one ‘-D’ flag. We’d also need to introduce
‘--allow-collisions’ for ‘guix shell’.

WDYT?

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 15 Mar 2022 14:51
control message for bug #54350
(address . control@debbugs.gnu.org)
87y21bjpdj.fsf@gnu.org
retitle 54350 'guix shell' skips profile collision checks
quit
M
M
Maxime Devos wrote on 15 Mar 2022 16:58
Re: bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 54350@debbugs.gnu.org)
b1c608f460465b7c4421aed44677f3a2f94302ec.camel@telenet.be
Ludovic Courtès schreef op di 15-03-2022 om 14:50 [+0100]:
Toggle quote (13 lines)
> $ ./pre-inst-env guix shell -D guile -n
> guix shell: error: profile contains conflicting entries for bash-minimal
> guix shell: error:   first entry: bash-minimal@5.1.8 /gnu/store/chfwin3a4qp1znnpsjbmydr2jbzk0d6y-bash-minimal-5.1.8
> guix shell: error:   second entry: bash-minimal@5.1.8 /gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8
> hint: You cannot have two different versions or variants of `bash-minimal' in the same profile.
> --8<---------------cut here---------------end--------------->8---
>
> These collisions are harmless but frequent in package development
> environments, which is why ‘guix environment’ had to turn off those
> checks (commit afd06f605bf88a796acefc7ed598b43879346a6b).
>
> That’s a bit of a hack though.

These collisions happen because the bag of a package contains of multiple
variants of bash: the default canonical bash-minimal, from (gnu packages
commencement), not directly visible (i.e., with "guix show" and
'specification->package'), which is overidden by the bash-minimal from
native-inputs, which is visible from specification->package and as a public
variable, and as I understand it, the bash-minimal supposed to end up
in the references of non-bootstrap packages.

For these kind of collisions, I think the non-canonical package (that's
actually visible with "guix show FOO") is preferred here, so perhaps
the canonical variants can be filtered out whenever that would resolve
a collision?

That seems also a bit of a hack though.

Toggle quote (4 lines)
> What we could do, as a mitigation, is to enable collision checks
> unless there’s at least one ‘-D’ flag. We’d also need to introduce
> ‘--allow-collisions’ for ‘guix shell’.

Special-casing -D like this also seems like a hack to me. I'm not sure
which hack is better, though I prefer this one and the ‘filter out
canonical packages’ above the original ‘just disable collision
checking’.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYjC3sBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7oDxAP9eipXUqNPc36a8Dy1OE/61ePW0
3vstSYCI6hH+RrCgogD+KNGMRFaS7UfPk/yFkCeLhPgYijpJiexQnaec0xxELgU=
=MzrT
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 16 Mar 2022 11:18
(name . Maxime Devos)(address . maximedevos@telenet.be)(address . 54350@debbugs.gnu.org)
87czimgpz9.fsf@gnu.org
Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

Toggle quote (8 lines)
> Ludovic Courtès schreef op di 15-03-2022 om 14:50 [+0100]:
>> $ ./pre-inst-env guix shell -D guile -n
>> guix shell: error: profile contains conflicting entries for bash-minimal
>> guix shell: error:   first entry: bash-minimal@5.1.8 /gnu/store/chfwin3a4qp1znnpsjbmydr2jbzk0d6y-bash-minimal-5.1.8
>> guix shell: error:   second entry: bash-minimal@5.1.8 /gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8
>> hint: You cannot have two different versions or variants of `bash-minimal' in the same profile.
>> --8<---------------cut here---------------end--------------->8---

I wonder whether this is the exception rather than the rule. Here’s an
unscientific not-quite-random sample:

Toggle snippet (49 lines)
$ ./pre-inst-env guix shell -D guix
guix shell: error: profile contains conflicting entries for bzip2
guix shell: error: first entry: bzip2@1.0.8 /gnu/store/gzmwrl6cb8nxmc68hpgqzblng2kamkgc-bzip2-1.0.8
guix shell: error: second entry: bzip2@1.0.8 /gnu/store/s3hl12jxz9ybs7nsy7kq7ybzz7qnzmsg-bzip2-1.0.8
hint: You cannot have two different versions or variants of `bzip2' in the same profile.

$ ./pre-inst-env guix shell -D inkscape --no-grafts -n
guix shell: error: profile contains conflicting entries for libsigc++
guix shell: error: first entry: libsigc++@3.0.6 /gnu/store/ng4k2yl94d758p5vnashd4nvyb1aw8s1-libsigc++-3.0.6
guix shell: error: ... propagated from cairomm@1.14.2
guix shell: error: ... propagated from gtkmm@3.24.5
guix shell: error: second entry: libsigc++@2.9.3 /gnu/store/iad8jg1fm7jsq0pqj547f3n5s2jn9rp0-libsigc++-2.9.3
hint: Try upgrading both `gtkmm' and `libsigc++', or remove one of them from the profile.

$ ./pre-inst-env guix shell -D gimp --no-grafts -n
guix shell: error: profile contains conflicting entries for librsvg
guix shell: error: first entry: librsvg@2.50.7 /gnu/store/wj3sigq6pwjc2z68qgbpv8pxcv8haklf-librsvg-2.50.7
guix shell: error: second entry: librsvg@2.50.7 /gnu/store/q3vymxinnp6cxqq1cpz1rdqmnwq5x1ni-librsvg-2.50.7
hint: You cannot have two different versions or variants of `librsvg' in the same profile.

$ ./pre-inst-env guix shell -D openmpi --no-grafts -n
The following derivation would be built:
/gnu/store/fpj74z159bf6v7cxsxs5rxm3xm4w2amz-profile.drv

$ ./pre-inst-env guix shell -D hwloc --no-grafts -n
The following derivation would be built:
/gnu/store/ikf40sfmk3y7r3frxm9fbkzmma4ym8ay-profile.drv

$ ./pre-inst-env guix shell -D guile-gcrypt --no-grafts -n
The following derivation would be built:
/gnu/store/3ip5kq50b97bsm6zhygz0fjx54w0fqa0-profile.drv

$ ./pre-inst-env guix shell -D libgit2 --no-grafts -n
The following derivation would be built:
/gnu/store/8f83l9p88g40mrnl5l4wrzqg5zwgn6cm-profile.drv

$ ./pre-inst-env guix shell -D python-pytorch --no-grafts -n
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivation would be built:
/gnu/store/aly6xd3vi5j0qimjzj2hj0wry5zma376-profile.drv

37.2 MB would be downloaded
$ ./pre-inst-env guix shell -D python-itsdangerous --no-grafts -n
The following derivation would be built:
/gnu/store/cgxgvj73wgx263lzd3f82vvjl39vn4pn-profile.drv


We should check automatically the fraction of packages for which
‘package->development-manifest’ contains colliding entries.

If it’s rather rare, we might as well have ‘guix shell’ default to
#:allow-collisions? #f.

(The issue for GIMP comes from librsvg vs. librsvg-bootstrap. The one
for inkscape is a genuine problem.)

If we take that route, we could also add a linter that checks for
collisions in package dependencies.

Toggle quote (19 lines)
>> These collisions are harmless but frequent in package development
>> environments, which is why ‘guix environment’ had to turn off those
>> checks (commit afd06f605bf88a796acefc7ed598b43879346a6b).
>>
>> That’s a bit of a hack though.
>
> These collisions happen because the bag of a package contains of multiple
> variants of bash: the default canonical bash-minimal, from (gnu packages
> commencement), not directly visible (i.e., with "guix show" and
> 'specification->package'), which is overidden by the bash-minimal from
> native-inputs, which is visible from specification->package and as a public
> variable, and as I understand it, the bash-minimal supposed to end up
> in the references of non-bootstrap packages.
>
> For these kind of collisions, I think the non-canonical package (that's
> actually visible with "guix show FOO") is preferred here, so perhaps
> the canonical variants can be filtered out whenever that would resolve
> a collision?

I’m not fond of filtering out, because that can make it hard to
understand what the inputs of a package really are.

However, we could tweak the collision detection code to ignore
collisions with hidden packages or something along these lines.

Thoughts?

Ludo’.
L
L
Ludovic Courtès wrote on 16 Mar 2022 14:49
control message for bug #54350
(address . control@debbugs.gnu.org)
877d8uf1ns.fsf@gnu.org
severity 54350 important
quit
G
G
Greg Hogan wrote on 16 Mar 2022 21:31
Re: bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken
(name . Ludovic Courtès)(address . ludo@gnu.org)
CA+3U0Z=JbVMq1PLqZrFh8-ZBW=emtdRa0XvVEEJZPpG_r0HmEw@mail.gmail.com
On Tue, Mar 15, 2022 at 9:51 AM Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (5 lines)
> What we could do, as a mitigation, is to enable collision checks unless
> there’s at least one ‘-D’ flag. We’d also need to introduce
> ‘--allow-collisions’ for ‘guix shell’.
>

Also 'guix pack'.
Attachment: file
K
K
Konrad Hinsen wrote on 12 Dec 2023 14:10
Profile collisions in "guix shell"
(address . 54350@debbugs.gnu.org)
m1jzpjzhux.fsf@fastmail.net
Hi everyone,

I ran into this issue when I was trying to turn a much-used shell
environment into a profile for persisting the binaries in the store.
I had been using it for several months, believing it to be OK.
Fortunately it was easy to fix.

Background: as part of a reproducible computation workflow (see my talk
at the ten-year meeting,
I run everything that is part of a research project in containers,
meaning "guix shell -C". Since I rely on Python 2 software, I end up
using package transformations frequently to get old software to work. As
it turns out now, package transformations are also an excellent way to
create profile collisions: three out of five of my commonly used
containers have collisions, which I wasn't aware of until a few days
ago.

From that perspective, the current behavior of "guix shell" is... let's
say "suboptimal".

I do understand the motivation, as explained in this thread, but I don't
quite understand (1) why profile collisions are so frequent in
development environments and (2) why ignoring them in that use case
doesn't cause any trouble.

When I run "guix shell -D inkscape", I expect to have, as much as
possible, the same environment as the one the daemon uses when build
inkscape. Does the daemon also switch off collision detection?

Cheers,
Konrad.
K
K
Konrad Hinsen wrote on 15 Dec 2023 07:51
(address . 54350@debbugs.gnu.org)
m1v890x8ju.fsf@fastmail.net
Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

Toggle quote (3 lines)
> I run everything that is part of a research project in containers,
> meaning "guix shell -C".

Related: HPC users on clusters using NFS are also advised to use
"guix shell" rather than profiles:


Cheers,
Konrad.
S
S
Suhail wrote on 16 Jan 19:36 +0100
bug#54350: Profile collisions in "guix shell"
(address . 54350@debbugs.gnu.org)
<87wms9nn1p.fsf@>
Hello Guix,

I came across this issue the other day when attempting to create a
profile that could act as a development environment for Guix. In
addition to noting my observations below, I'd like to understand why
there are collisions in a *new* profile tasked with installing the
dependencies of guix? Is this a bug?

The [documentation] notes the below command to start "a new shell where
all the dependencies and appropriate environment variables are set up to
hack on Guix":

#+begin_src sh
guix shell -D guix -CPW
#+end_src

Using guix-shell to export the manifest corresponding to the above one
gets:

#+header: :tangle ./manifest.scm
#+begin_src scheme
;; What follows is a "manifest" equivalent to the command line you gave.
;; You can store it in a file that you may then pass to any 'guix' command
;; that accepts a '--manifest' (or '-m') option.
(package->development-manifest
(specification->package "guix"))
#+end_src

Saving the above to manifest.scm and evaluating the below:

#+begin_src sh
mkdir -p ${GUIX_EXTRA_PROFILES}/guix-build-2
guix package -p ${GUIX_EXTRA_PROFILES}/guix-build-2/guix-build-2 -m ./manifest.scm
#+end_src

Results in the following error:

#+begin_example
...
guix package: error: profile contains conflicting entries for bzip2
guix package: error: first entry: bzip2@1.0.8 /gnu/store/4jkvfh9hp9zf9plrcw490ci6vpvc033d-bzip2-1.0.8
guix package: error: second entry: bzip2@1.0.8 /gnu/store/0ngy7fmrp9g3q35wwccnjxzaaqq9f8wi-bzip2-1.0.8
hint: You cannot have two different versions or variants of `bzip2' in the same profile.
#+end_example

Adding --allow-collisions to match guix-shell behaviour makes the
installation succeed, as expected.


--
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.
?