Issue running SystemTests due to "conda activate" error

[samsrabin]

The FSURDATMODIFYCTSM test fails for me on Cheyenne, with cs.status saying SHAREDLIB_BUILD failed to initialize. Looking at the log, there's a problem using conda activate (full error message below). This occurs on my branch for #1863, but also with tag ctsm5.1.dev128.

I am able to use conda activate in login shells, and my .bashrc file seems to be set up correctly, with the following at the end:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/glade/u/apps/opt/conda/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/glade/u/apps/opt/conda/etc/profile.d/conda.sh" ]; then
        . "/glade/u/apps/opt/conda/etc/profile.d/conda.sh"
    else
        export PATH="/glade/u/apps/opt/conda/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

I had the same issue in the custom SystemTest I wrote for #1863, but I worked around it by doing this:

conda run -n ctsm_pylib python3 script.py

instead of how it's done in FSURDATMODIFYCTSM and other SystemTests:

conda activate ctsm_pylib && python3 script.py

I'm sure there's something about my setup that's causing this to break, but I'm wondering: Might it be worth just changing all the SystemTests to use this apparently more robust method? I doubt I'll be the last person to run into this issue.

---

Full error message from FSURDATMODIFYCTSM:

Building test for FSURDATMODIFYCTSM in directory /glade/scratch/samrabin/tests_0622-141618ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.C.0622-141618ch_int

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.


Traceback (most recent call last):
  File "./case.build", line 256, in <module>
    _main_func(__doc__)
  File "./case.build", line 216, in _main_func
    test = find_system_test(testname, case)(case)
  File "/glade/u/home/samrabin/ctsm_rx_crop_calendars4/cime_config/SystemTests/fsurdatmodifyctsm.py", line 46, in __init__
    self._run_modify_fsurdat()
  File "/glade/u/home/samrabin/ctsm_rx_crop_calendars4/cime_config/SystemTests/fsurdatmodifyctsm.py", line 78, in _run_modify_fsurdat
    subprocess.run( conda_env+"python3 "+tool_path+" "+self._cfg_file_path, shell=True, check=True)
  File "/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '. /glade/scratch/samrabin/tests_0622-141618ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_i
ntel.C.0622-141618ch_int/.env_mach_specific.sh;   conda activate ctsm_pylib && python3 /glade/u/home/samrabin/ctsm_rx_crop_calendars4/tools/modify_input_files
/fsurdat_modifier /glade/scratch/samrabin/tests_0622-141618ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.C.0622-141618ch_int
/modify_fsurdat.cfg' returned non-zero exit status 1.

[ekluzek]

I Googled "conda run" vs. "conda activate" and found this article. Which points out that you should use "conda run" for for non-interactive environments -- which is what we are doing. So it sounds like you are correct the way you suggest is a more robust way, and certainly how we should change it.

https://www.scivision.dev/conda-run-non-interactive-script/

We talk about using "conda activate" in our documentation, should that also change to say use "conda run"? You could still use a different environment, so I'm not quite sure here...

[samsrabin]

Ah okay! In that case, is it okay if I change FSURDATMODIFYCTSM (which is actually the only SystemTest that uses conda activate) to use conda run -n instead? It'd be easiest to do this as a commit in my branch for #1863, but of course I can make a separate PR if you prefer.

I might actually add a new module in cime_config/SystemTests/ that contains some code for this to share between FSURDATMODIFYCTSM and my test.

I'm not sure whats best to do about the documentation; I'd guess it should be a case-by-case basis based on whether we expect the user to run the script interactively or not.

[ekluzek]

Yes, I'm good with either solution (current or separate PR). You can just change it in your current PR. Adding a new module sounds useful as well.

We might need to look through the documentation to assess what might need to change and what wouldn't. You are probably right maybe the "conda run" syntax is only needed for testing scripts that run the scripts, from within other scripts.