4

I'm running some python scripts on some Linux clusters using SGE or SLURM. I already have my conda environment set up properly using the login node. I have been writing something like

source ~/.bashrc
module purge #Kill all active modules
conda init bash
conda deactivate
conda deactivate
conda activate my_env

python my_script.py

to activate the environment properly. (I have done a lot of work to figure this out) However, I just found some example codes like

/anaconda3/envs/my_env/bin/python my_script.py

seems to do the same thing without the need for tedious deactivation and activation. Are they actually doing the same thing? If so, which would be the better practice?

Improve this question
2
  • Most of that in the script is unnecessary - conda init needs only get run once ever per user. Commented May 18, 2022 at 2:47
  • @merv They are necessary for the cluster I used. Otherwise, the environment can't be activated properly. Commented May 18, 2022 at 2:58

2 Answers 2

Reset to default
6

Programmatic execution with an environment is usually better done through the conda run subcommand. E.g.,

my_slurm_script.sh

#!/bin/bash -l
conda run -n my_env python my_script.py

Read the conda run --help for details.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

I tried to use this for an environment installed outside of the default envs directory and found that the -n flag can't take a path as argument. But I found the solution here
However, when I tried to test my scripts interactively using conda run, I found no console output, which is a bit annoying to me. Then I found this issue. Does this mean that conda run is somewhat problematic?
@WillYang no, that’s an old issue. Please see conda run --help. You’ll need either (or both) of the --live-stream or --no-capture-output flags, depending on your requirements.
0

When you activate an env, it just changes the python executable to /anaconda3/envs/my_env/bin/python instead of the system's python executable /usr/bin/python in layman terms.

A bertter approach will be to use conda's built in method conda run -n env_name script.py.

Improve this answer

1 Comment

I would caution against the use of "just" here. It also runs activation scripts, some of which may be important for correct functionality of some modules. For example, some packages rely on compilers and Conda's compilers set things like CC and CFLAGS environment variables in their activation scripts.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.