conda 4.8.3 on guix cannot activate environments

Dear Ricardo et al.,

The conda 4.8.3 package on guix does not seem to work as expected.

Conda info:

hugo@alex ~$ which conda

/home/hugo/.guix-profile/bin/conda

hugo@alex ~$ realpath $(which conda)

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/conda

hugo@alex ~$ conda -V

conda 4.8.3

Creating an environment works fine:

hugo@alex ~$ conda create -n testenv -y

Collecting package metadata (current_repodata.json): done

Solving environment: done

## Package Plan ##

environment location: /home/hugo/.conda/envs/testenv

Preparing transaction: done

Verifying transaction: done

Executing transaction: done

#

# To activate this environment, use

#

# $ conda activate testenv

#

# To deactivate an active environment, use

#

# $ conda deactivate

But activating it does not work properly:

hugo@alex ~$ conda activate testenv

CommandNotFoundError: Your shell has not been properly configured to use

'conda activate'.

To initialize your shell, run

$ conda init <SHELL_NAME>

Currently supported shells are:

- bash

- fish

- tcsh

- xonsh

- zsh

- powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running

'conda init'.

Unfortunately, conda init does not work well, because it needs root access:

hugo@alex ~$ conda init bash --dry-run

no change

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/condabin/conda

modified

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/conda

modified

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/conda-env

modified

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/activate

modified

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/deactivate

no change

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/etc/profile.d/conda.sh

no change

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/etc/fish/conf.d/conda.fish

no change

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/shell/condabin/Conda.psm1

no change

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/shell/condabin/conda-hook.ps1

no change

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/lib/python3.8/site-packages/xontrib/conda.xsh

no change

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/etc/profile.d/conda.csh

modified /home/hugo/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

The last change of the conda package seems to be 638ef1e81d8 from August 13.

This is my first week with guix, so I'm not even sure where to begin to

debug this.

FWIW, maybe it would be simpler to create a micromamba package than a conda

package:

Cheers,

Hugo

Hi Hugo,

Thanks for letting me know. The upgrade took a long time and required

quite a bit of patching.

This is very unfortunate, but perhaps we can work around this.

Initially I noticed that almost none of the Conda features worked

without “conda init”, so I decided to run “conda init” as part of the

build. This installs a whole bunch of shell initialization files into

the prefix directory, which — I assume — are meant to be evaluated when

the shell starts.

So I suspect that you can get around this by manually sourcing the

appropriate shell init files. Which of these need to be sourced

probably depends on your current shell.

Yeah, it’s a terrible design decision on the part of the Conda

developers. Luckily you won’t need any of that because we already

installed all these files at build time.

Why not both?

--

Ricardo

Hi Ricardo,

On Wed, 19 Aug 2020 at 12:07, Ricardo Wurmus <rekado@elephly.net> wrote:

Thanks for doing this hard work! It is much appreciated, I don't think I

would have started experimenting with guix without your head start.

That indeed kinda works. `conda init bash` adds this to `.bashrc`

(apparently before asking for the sudo password):

# >>> conda initialize >>>

# !! Contents within this block are managed by 'conda init' !!

__conda_setup="$('/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/conda'

'shell.bash' 'hook' 2> /dev/null)"

if [ $? -eq 0 ]; then

eval "$__conda_setup"

else

if [ -f

"/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/etc/profile.d/conda.sh"

]; then

.

"/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/etc/profile.d/conda.sh"

else

export

PATH="/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin:$PATH"

fi

fi

unset __conda_setup

# <<< conda initialize <<<

And running this will indeed allow an environment to be activated. The

`testenv3` environment contains only Python:

hugo@alex ~$ which python

which: no python in

(/run/setuid-programs:/home/hugo/.config/guix/current/bin:/home/hugo/.guix-profile/bin:/run/current-system/profile/bin:/run/current-system/profile/sbin)

hugo@alex ~$ source .bashrc

(base) conda activate testenv3

(testenv3)

(testenv3) which python

/home/hugo/.conda/envs/testenv3/bin/python

(testenv3) python

-bash: /home/hugo/.conda/envs/testenv3/bin/python: No such file or directory

(testenv3) readelf --all $(which python) | grep interpreter

[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

So there are three problems:

1. The PATHS in .bashrc should be to ~/.guix-profile/etc/profile.d/, not to

/gnu/store directly. This is trivial to fix manually.

2. The prompt is not set correctly, as in, what should happen is that the

current conda environment is added to the prompt. Instead, the prompt is

replaced entirely by the environment. This shouldn't be too hard to fix

manually though.

3. The installed software does not have the proper interpreter set. This

can probably be fixed with patchelf, but now I'm wondering whether this is

the right approach, because that would be necessary for all packages

installed through conda. (Or is there a way to do that automatically?

Apparently just symlinking ld-linux-x86.so.2 into (from?) /lib64 also

works, but that feels like a very un-guixy hack.)

Sidenote, FWIW, there are also some problems with some of the scripts

installed in /gnu/store:

hugo@alex ~$

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/condabin/conda

-bash:

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/condabin/conda:

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/python: bad

interpreter: No such file or directory

hugo@alex ~$

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/activate

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/activate: line

3:

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/.activate-real:

Permission denied

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/activate: line

3: exec:

/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/.activate-real:

cannot execute: Permission denied

It seems not necessary to actually use these scripts though.

Ultimately it would be nice if all conda packages would be available in

guix directly (as guix packages, removing the need for conda), but that is

going to take a while. The above workarounds are probably sufficient to get

something going until then.

Once we settle on a good way to approach this, where should we document

this? As in, if we want users to ignore the complaints from conda and

instead put something in their .bashrc manually, then that information

should be discoverable somehow.

Maybe I'll give it a shot. (micro)mamba is still a bit unstable it seems,

but it might be a good fit for guix.

Cheers,

Hugo

Hugo Buddelmeijer <hugo@buddelmeijer.nl> writes:

Perhaps we can replace these locations with

${GUIX_PROFILE:-/gnu/store/…} just as we do in our generated etc/profile

file.

Weird! Is this something Conda does wrong because it assumes too much

about the existing prompt?

There’s nothing we can do about this in a general fashion. Creating a

link from /lib64/ld-linux-x86-64.so.2 to the loader in the glibc package

is indeed what is necessary on Guix System. If you don’t like to do

this manually, you can use the extra-special-file service.

Would be good to fix them, though. The activate script may just need a

chmod. Obviously, the bin/python thing is dead wrong — I must have

missed that instance of prefix confusion that litters the Conda code

base.

We shouldn’t need to document this at all; we should patch Conda to do

mostly the right thing. This involves limiting “conda init” to

user-level changes.

--

Ricardo

Just sharing a bit of more info about why the conda prompt breaks.

On Fri, 21 Aug 2020 at 05:52, Ricardo Wurmus <rekado@elephly.net> wrote:

This is what happens, reducing the files to the essence:

1) .bashrc runs

__conda_setup="$('/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/conda'

'shell.bash' 'hook' 2> /dev/null)"

eval "$__conda_setup"

2) __conda_setup runs

__conda_activate() {

ask_conda="$(PS1="$PS1" "$CONDA_EXE" $_CE_M $_CE_CONDA shell.posix

"$cmd" "$@")" || \return $?

\eval "$ask_conda"

}

conda activate base

3) ask_conda will contain the new PS1 definition. The command ran to create

ask_conda is

PS1="\u@\h \w\$"

"/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/conda"

shell.posix "activate" "base"

4) conda runs

#!/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash

export PYTHONPATH="/gnu/store....."

exec -a "$0"

"/gnu/store/ihn8dbs84rmc3ai7r1vkvh4cya518wmx-conda-4.8.3/bin/.conda-real"

"$@"

5) .conda.real generates the prompt in Python:

from conda.cli import main

sys.exit(main())

6) .conda.real outputs

PS1='(base) '

export PATH='...

The problem with the prompt arises from step 4), because 'conda' has bash

as interpreter, and "non-interactive bashes go out of their way to unset

PS1":

So .conda.real never gets the PS1 variable and it disappears.

I don't really understand why there is a bash function, a conda bash script

and a .conda-real python script.

My local miniconda 4.8.3 installation on an Ubuntu machine (in my home

directory) does not have a ".conda-real": "conda" directly is the Python

script, so everything works.

Maybe more later.

Hugo

Hi Hugo,

Good sleuthing!

That’s because we wrap application to ensure that they run with expected

environment variables. By default this wrapper is implemented as a Bash

script, but we also have “wrap-script”, which turns scripts using “#” as

a comment character into polyglottal Guile scripts, setting variables

with Guile and then reexecuting themselves with the actual interpreter.

Not sure if that would help here.

(There also have been reports about “wrap-script” being buggy, but I

don’t recall any details.)

--

Ricardo

Hi Ricardo,

On Tue, 25 Aug 2020 at 14:35, Ricardo Wurmus <rekado@elephly.net> wrote:

I did some more experiments, and forcing the wrapper script to start in

interactive mode solves the problem.

`conda init` adds some things to `.bashrc`, in particular it `eval`s the

output from `conda shell.bash hook`, which creates the `conda` bash

function (and associated functions).

The output from `conda shell.bash hook` seems to be based on

`/etc/profile.d/conda.sh`:

As in, it prefixes some environment variables and appends the activation of

the base environment.

Now if we replace all 5 occurances of

"$CONDA_EXE" $_CE_M $_CE_CONDA

in that file with

bash --norc -i "$CONDA_EXE" $_CE_M $_CE_CONDA

then everything works as expected.

The -i is to enable interactive mode, preventing PS1 from being eaten.

--norc is necessary to not source .bashrc.

There are two occurences of "$CONDA_EXE" that should not be replaced, hence

adding the CE variables.

Would editing files like that be possible in a guix package script?

It seems that it is only a simple sed replace on a single file. It is a bit

hackish, because it adds knowledge to the package that a) there is a

wrapper script and b) it is in bash.

I have only experimented with it by copying the output off `conda

shell.bash hook` to a file in my home directory and editing that. Maybe I

can try to update the package too.

Other options:

- Use some other sh than bash, one that does not eat PS1, (e.g. dash) as

wrapper script. I just verified that *all* wrappers on my machine are bash.

However, some are using bash-minimal, so that means that there is a choice,

so maybe switching shells is possible?

- Somehow let conda work with another variable than PS1. Seems more

complicated.

Then what is left is updating `conda init` such that

- it doesn't complain when it cannot edit files

- it only uses paths in ~/.guix-profile

but this is not essential to get conda to work.

Cheers,

Hugo

Hi Ricardo et al.,

I've not yet looked into updating the conda package, but want to add a

small thing to the bug report, so it can easily be found.

Lines in the sourced conda.sh script like the one below won't work anymore

once the conda executable is wrapped in the guix shell script, because PS1

is eaten by the wrapper:

"$CONDA_EXE" $_CE_M $_CE_CONDA

It is necessary to explicitly call bash in interactive mode to have PS1

available:

bash --norc -i "$CONDA_EXE" $_CE_M $_CE_CONDA

However, that will add every executed line to the .bash_history file,

including very long "export PYTHONPATH" lines. This can be avoided by

explicitly setting the history file to nothing:

HISTFILE= bash --norc -i "$CONDA_EXE" $_CE_M $_CE_CONDA

Cheers,

Hugo

On Tue, 25 Aug 2020 at 17:03, Hugo Buddelmeijer <hugo@buddelmeijer.nl>

wrote:

We have since upgraded to 4.10.3, but the problems still persist.

Conda still asks users to run “conda init”, which attempts to run

“sudo” to modify root-owned files. It’s a big mess.

Any ideas?

--

Ricardo

Hi Ricardo,

On Wed, 18 Aug 2021 at 12:07, Ricardo Wurmus <rekado@elephly.net> wrote:

What is the point to have Conda installed by Guix?

Cheers,

simon

zimoun <zimon.toutoune@gmail.com> writes:

Convenience.

--

Ricardo

I don't think conda init will ever work on Guix, and it seems out

of scope to try and prevent Conda from prompting users to do

that. I was able to get conda working with the following steps:

1. Expose the glibc interpreter so that conda python can function

#+begin_src scheme

(extra-special-file "/lib64/ld-linux-x86-64.so.2"

(file-append glibc

"/lib/ld-linux-x86-64.so.2"))

#+end_src

2. Source the conda environment variables. I do this both in my

profile (through home-shell-profile-service-type) and through my

rc file for interactive shells, as I think I remember having

problems not sourcing it in both.

#+begin_src scheme

(define-public (home-profile-service script)

"Creates a service that sources a home profile.d script provided

by a package."

(simple-service 'profile.d-service

home-shell-profile-service-type

`(,(plain-file

(string-append script "-profile")

(string-append "source

~/.guix-home/profile/etc/profile.d/" script

".sh")))))

(home-profile-service "conda")

#+end_src

#+begin_src shell

source ~/.guix-home/profile/etc/profile.d/conda.sh

#+end_src

I agree that this setup is not ideal, but I think that guix home

has the promise of fixing a lot of these issues with home

services! Hopefully in the near future we'll be able to turn these

fixed into services for users.

Dominic Martinez <dom@dominicm.dev> writes:

Why? It used to work just fine IIRC. The fact that it tries to write

to root-owned files is arguably a bug in Conda.

--

Ricardo