conda fails to load for SystemTests

[samsrabin]

Brief summary of bug

For some users, FSURDATMODIFYCTSM is failing because conda (called in a subprocess from Python) doesn't load. Note that this is a different issue (and gives a different sort of error) from #2109.

General bug information

CTSM version you are using: ctsm5.1.dev133-47-g6925f8cc9

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: FSURDATMODIFYCTSM and RXCROPMATURITY tests.

Details of bug

@rgknox and @ekluzek (#1959), as well as @slevis-lmwg (#2106), are all affected. This doesn't happen for me or, I think, @adrifoster. I think it's thus something to do with our shell environments.

We suspect this stems from changes introduced in cime_config/SystemTests/ in ctsm5.1.dev131 (as was the case for #2109). The module unload python; module load conda; step fails to load conda, so then the call of conda run -n fails and ends the test. However, that module un/loading step was present before, so there's something subtle going on.

Important details of your setup / configuration so we can reproduce the bug

Unknown.

Important output or errors that show the problem

From TestStatus.log:

which: no conda in (/glade/u/apps/ch/opt/netcdf/4.9.0/intel/19.1.1/bin:/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/19.1.1:/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/19.1.1/mpi:/glade/p/cesmdata/cseg/PROGS/esmf/8.4.1b02/mpiuni/2.3.0/intel/19.1.1/bin/bing/Linux.intel.64.mpiuni.default:/glade/u/apps/opt/intel/2020u1/vtune_profiler/bin64:/glade/u/apps/opt/intel/2020u1/inspector/bin64:/glade/u/apps/opt/intel/2020u1/advisor/bin64:/glade/u/apps/opt/intel/2020u1/compilers_and_libraries/linux/bin/intel64:/glade/u/apps/ch/opt/cmake/3.22.0/bin:/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/bin:/glade/u/apps/opt/vncmgr:/glade/u/apps/opt/globus-utils:/glade/u/apps/ch/opt/usr/bin:/glade/u/apps/ch/opt/lmod/8.7.13/bin:/glade/u/apps/opt/intel/2020u1/vtune_amplifier/bin64:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin/X11:/usr/games:/opt/ibutils/bin:/usr/lib/mit/bin:/opt/pbs/bin:/glade/u/apps/opt/hpss/hpss-client.5.0.2.p5/bin:/glade/u/apps/ch/os/usr/bin:/glade/u/apps/ch/opt/hpe-cpe/22.02/bin)
Traceback (most recent call last):
  File "./case.build", line 256, in <module>
    _main_func(__doc__)
  File "./case.build", line 216, in _main_func
    test = find_system_test(testname, case)(case)
  File "/glade/u/home/rgknox/ctsm/cime_config/SystemTests/fsurdatmodifyctsm.py", line 46, in __init__
    self._run_modify_fsurdat()
  File "/glade/u/home/rgknox/ctsm/cime_config/SystemTests/fsurdatmodifyctsm.py", line 77, in _run_modify_fsurdat
    tool_path,
  File "/glade/u/home/rgknox/ctsm/cime_config/SystemTests/systemtest_utils.py", line 39, in run_python_script
    command, shell=True, check=True, text=True, stdout=f, stderr=subprocess.STDOUT
  File "/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '. /glade/scratch/rgknox/tests_0814-134713ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.GC.0814-134713ch_int/.env_mach_specific.sh;  module unload python; module load conda; conda run -n ctsm_pylib python3 /glade/u/home/rgknox/ctsm/tools/modify_input_files/fsurdat_modifier /glade/scratch/rgknox/tests_0814-134713ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.GC.0814-134713ch_int/modify_fsurdat.cfg' returned non-zero exit status 127.

And fsurdat_modifier.log:

Lmod has detected the following error: Unable to load module because of error
when evaluating modulefile:
     /glade/u/apps/ch/modulefiles/default/idep/conda/latest.lua: [string
"require("posix")..."]:54: attempt to call global 'source_sh' (a nil value)
     Please check the modulefile and especially if there is a the line number
specified in the above message
While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    conda/latest     /glade/u/apps/ch/modulefiles/default/idep/conda/latest.lua

/bin/sh: conda: command not found

[ekluzek]

Note this is something that @rgknox and I see, but @samsrabin and @adrifoster don't? So works for some and not for others. Still a good thing to figure out.

Also there error checking for these situations could be better so that it's more obvious what is going on.

[billsacks]

I am seeing this error in my testing for ctsm5.1.dev136. The test FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel fails in the SHAREDLIB_BUILD phase.

[adrifoster]

This was the same error I got because I had a conda environment loaded Can you try doing "conda deactivate" and then rebuilding?

…

On Tue, Aug 22, 2023 at 11:34 AM Bill Sacks ***@***.***> wrote: I am seeing this error in my testing for ctsm5.1.dev136. The test FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel fails in the SHAREDLIB_BUILD phase. — Reply to this email directly, view it on GitHub <#2111 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADE42IWRH7XJG3QUJOIZW4TXWTUUBANCNFSM6AAAAAA3SUUV7Y> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[samsrabin]

@adrifoster I think your error was actually the one given in #2109, no?

[samsrabin]

@ekluzek and I worked on this a few days ago, and the conclusion (among other notes) was that there's probably something up with his environment. This may be biting all users with old Cheyenne accounts, as Adrianna and I don't experience the issue.

This may end up taking a long time to debug. In the interim, I think I'll add a fallback to the original behavior if the new behavior fails.

[billsacks]

This was the same error I got because I had a conda environment loaded

I don't think this was the case for me.

I agree with the idea of doing a quick-ish workaround for now, not spending a lot of time on this. Time would probably better be spent figuring out what needs to be done to get this working on derecho, if it doesn't work out-of-the-box.

[samsrabin]

A perplexing new development from some troubleshooting just now with @slevis-lmwg… I had him use the cime_config/SystemTests/fsurdatmodifyctsm.py from tag ctsm5.1.dev129—i.e., bypassing all my changes to how the test is run—and it still failed with the same "conda not found" error! It seems like something else might be causing the issue.

@billsacks @ekluzek @rgknox: Would any of y'all be willing to try this to confirm?

[billsacks]

I tried 4 things: From both out-of-the-box ctsm5.1.dev129, and from master but with fsurdatmodifyctsm.py backed out to the version in ctsm5.1.dev129, I tried both runningFSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel both via run_sys_tests (so building on compute node) and directly via create_test (so building on the login node). All four failed with a message like this:

    Errors were:
        Building test for FSURDATMODIFYCTSM in directory /glade/scratch/sacks/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.20230829_085910_cihtdp
        which: no conda in (/glade/u/apps/ch/opt/netcdf/4.9.0/intel/19.1.1/bin:/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/19.1.1:/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/19.1.1/mpi:/glade/p/cesmdata/cseg/PROGS/esmf/8.4.1b02/mpiuni/2.3.0/intel/19.1.1/bin/bing/Linux.intel.64.mpiuni.default:/glade/u/apps/opt/intel/2020u1/vtune_profiler/bin64:/glade/u/apps/opt/intel/2020u1/inspector/bin64:/glade/u/apps/opt/intel/2020u1/advisor/bin64:/glade/u/apps/opt/intel/2020u1/compilers_and_libraries/linux/bin/intel64:/glade/u/apps/ch/opt/cmake/3.22.0/bin:/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/bin:/glade/u/apps/opt/vncmgr:/glade/u/apps/opt/globus-utils:/glade/u/apps/ch/opt/usr/bin:/glade/u/apps/ch/opt/lmod/8.7.13/bin:/glade/u/home/sacks/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/opt/ibutils/bin:/usr/lib/mit/bin:/opt/pbs/bin:/sbin:/glade/u/home/sacks/bin/svn_scripts:/glade/p/cesmdata/cseg/tools/cime/tools/cprnc:/glade/u/home/sacks/bin_external/bin:/glade/u/apps/opt/hpss/hpss-client.5.0.2.p5/bin:/glade/u/apps/ch/os/usr/bin:/glade/u/apps/ch/opt/hpe-cpe/22.02/bin)
        Lmod has detected the following error: Unable to load module because of error when evaluating modulefile:
             /glade/u/apps/ch/modulefiles/default/idep/conda/latest.lua: [string "require("posix")..."]:56: attempt to call global 'source_sh' (a nil value)
             Please check the modulefile and especially if there is a the line number specified in the above message
        While processing the following module(s):
            Module fullname  Module Filename
            ---------------  ---------------
            conda/latest     /glade/u/apps/ch/modulefiles/default/idep/conda/latest.lua

        /bin/sh: conda: command not found
        Traceback (most recent call last):
          File "./case.build", line 256, in <module>
            _main_func(__doc__)
          File "./case.build", line 216, in _main_func
            test = find_system_test(testname, case)(case)
          File "/glade/work/sacks/ctsm_code/current_branch1/cime_config/SystemTests/fsurdatmodifyctsm.py", line 46, in __init__
            self._run_modify_fsurdat()
          File "/glade/work/sacks/ctsm_code/current_branch1/cime_config/SystemTests/fsurdatmodifyctsm.py", line 80, in _run_modify_fsurdat
            check=True,
          File "/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/lib/python3.7/subprocess.py", line 512, in run
            output=stdout, stderr=stderr)
        subprocess.CalledProcessError: Command '. /glade/scratch/sacks/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.20230829_085910_cihtdp/.env_mach_specific.sh; module unload python; module load conda; conda activate ctsm_pylib && python3 /glade/work/sacks/ctsm_code/current_branch1/tools/modify_input_files/fsurdat_modifier /glade/scratch/sacks/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.20230829_085910_cihtdp/modify_fsurdat.cfg' returned non-zero exit status 127.

[samsrabin]

Could it be that something changed in the Cheyenne software stack? I wonder if it'd work on Izumi or Derecho.

For now, it seems clear that this isn't actually an issue that I introduced with ctsm5.1.dev131. Would it be okay for me to remove this issue from #2108 and thus allow that PR to move forward?

[slevis-lmwg]

@samsrabin I'm not answering your question. I'm confirming that the test first failed in dev134, so after your changes had already passed.

[samsrabin]

Or alternatively… perhaps I could make it so that run_python_script() calls the main() function of fsurdat_modifier directly. That might just sidestep all these seemingly environment-related issues, in addition to making it easier to debug.

[ekluzek]

@samsrabin yes, this is a point where we are stuck on a problem and not sure how to proceed. But, now I saw your new suggestion that is probably worth trying to see if it fixes it.

Anyway, when we are stuck on an issue and don't know what to do, I think it's reasonable (and probably best practice) to remove that issue from a PR and leave it open until later -- but bring that PR in. We still want to fix this issue, but we can sideline it until we have a new idea on something to try.

[billsacks]

I'm a bit hesitant to wade more deeply into this since I've only been half-following, but I'd be interested in the group exploring whether we can use a different – and in my mind more robust and easier to maintain – solution as we transition to derecho: Rather than having individual system tests try to do something with your environment, instead require that the user has set up their python environment appropriately before running the given test. This could be done manually by the user, or maybe could be built into run_sys_tests – so moving any conda setup to run_sys_tests as part of the subprocess where we kick off the create_test job.

See also ESMCI/cime#4059

[glemieux]

Adding another data point: I tested out-of-the box dev129 and dev136 and saw similar build failures and the same stacktrace as what @billsacks reported above. The only difference between the two tags were the preambles leading up to the stacktrace; the dev136 failure message didn't have the CommandNotFoundError output:

ctsm5.1.dev129:

ERROR while getting the conda environment and/or 
running the fsurdat_modifier tool: 
(1) If your ctsm_pylib environment is out of date or you 
have not created the ctsm_pylib environment, yet, you may 
get past this error by running ./py_env_create 
in your ctsm directory and trying this test again. 
(2) If conda is not available, install and load conda, 
run ./py_env_create, and then try this test again. 
(3) If (1) and (2) are not the issue, then you may be 
getting an error within the fsurdat_modifier tool itself. 
Default error message: 
None
Building test for FSURDATMODIFYCTSM in directory /glade/scratch/glemieux/ctsm-tests/tests_fsurdat_test-dev129/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.fsurdat_test-dev129

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.


Traceback (most recent call last):  File "./case.build", line 256, in <module>
    _main_func(__doc__)
  File "./case.build", line 216, in _main_func
    test = find_system_test(testname, case)(case)
  File "/glade/u/home/glemieux/ctsm-test/cime_config/SystemTests/fsurdatmodifyctsm.py", line 46, in __init__
    self._run_modify_fsurdat()
  File "/glade/u/home/glemieux/ctsm-test/cime_config/SystemTests/fsurdatmodifyctsm.py", line 80, in _run_modify_fsurdat
    check=True,
  File "/glade/work/glemieux/conda-envs/ctsm_pylib/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '. /glade/scratch/glemieux/ctsm-tests/tests_fsurdat_test-dev129/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.fsurdat_test-dev129/.env_mach_specific.sh;   conda activate ctsm_pylib && python3 /glade/u/home/glemieux/ctsm-test/tools/modify_input_files/fsurdat_modifier /glade/scratch/glemieux/ctsm-tests/tests_fsurdat_test-dev129/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.fsurdat_test-dev129/modify_fsurdat.cfg' returned non-zero exit status 1.

UPDATE: adding full ctsm5.1.dev136 TestStatus.log error message:

ERROR while getting the conda environment and/or 
running the fsurdat_modifier tool: 
(1) If your ctsm_pylib environment is out of date or you 
have not created the ctsm_pylib environment, yet, you may 
get past this error by running ./py_env_create 
in your ctsm directory and trying this test again. 
(2) If conda is not available, install and load conda, 
run ./py_env_create, and then try this test again. 
(3) If (1) and (2) are not the issue, then you may be 
getting an error within fsurdat_modifier itself. 
Default error message: 
None
Building test for FSURDATMODIFYCTSM in directory /glade/scratch/glemieux/ctsm-tests/tests_fsurdat_test-dev136/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.fsurdat_test-dev136
Traceback (most recent call last):
  File "./case.build", line 256, in <module>
    _main_func(__doc__)
  File "./case.build", line 216, in _main_func
    test = find_system_test(testname, case)(case)
  File "/glade/u/home/glemieux/ctsm-test/cime_config/SystemTests/fsurdatmodifyctsm.py", line 46, in __init__
    self._run_modify_fsurdat()
  File "/glade/u/home/glemieux/ctsm-test/cime_config/SystemTests/fsurdatmodifyctsm.py", line 77, in _run_modify_fsurdat
    tool_path,
  File "/glade/u/home/glemieux/ctsm-test/cime_config/SystemTests/systemtest_utils.py", line 39, in run_python_script
    command, shell=True, check=True, text=True, stdout=f, stderr=subprocess.STDOUT
  File "/glade/work/glemieux/conda-envs/ctsm_pylib/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '. /glade/scratch/glemieux/ctsm-tests/tests_fsurdat_test-dev136/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.fsurdat_test-dev136/.env_mach_specific.sh;  conda run -n ctsm_pylib python3 /glade/u/home/glemieux/ctsm-test/tools/modify_input_files/fsurdat_modifier /glade/scratch/glemieux/ctsm-tests/tests_fsurdat_test-dev136/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.fsurdat_test-dev136/modify_fsurdat.cfg' returned non-zero exit status 1.

Per @samsrabin request, the contents of the dev136 fsurdat_modifier.log:

Traceback (most recent call last):
  File "/glade/u/home/glemieux/ctsm-test/tools/modify_input_files/fsurdat_modifier", line 15, in <module>
    from ctsm.modify_input_files.fsurdat_modifier import main
  File "/glade/u/home/glemieux/ctsm-test/tools/modify_input_files/../../python/ctsm/modify_input_files/fsurdat_modifier.py", line 20, in <module>
    from ctsm.modify_input_files.modify_fsurdat import ModifyFsurdat
  File "/glade/u/home/glemieux/ctsm-test/tools/modify_input_files/../../python/ctsm/modify_input_files/modify_fsurdat.py", line 12, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'

ERROR conda.cli.main_run:execute(47): `conda run python3 /glade/u/home/glemieux/ctsm-test/tools/modify_input_files/fsurdat_modifier /glade/scratch/glemieux/ctsm-tests/tests_fsurdat_test-dev136/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.fsurdat_test-dev136/modify_fsurdat.cfg` failed. (See above for error)

Location of test run folders:

• /glade/u/home/glemieux/scratch/ctsm-tests/tests_fsurdat_test-dev136
• /glade/u/home/glemieux/scratch/ctsm-tests/tests_fsurdat_test-dev129

[samsrabin]

@glemieux Good news! It looks like your ctsm5.1.dev129 error was the same one I encountered, which I fixed in ctsm5.1.dev131. In ctsm5.1.dev136, you're experiencing the same error that Adrianna did in #2109, which should be fixed by #2108. So it seems that you're not bit by this particular issue (2111).

[ekluzek]

Note, that @johnpaulalex ran into this as well. But, I suspect that deactivating his conda environment before running run_sys_tests might have got it to work for him? He tried a bunch of things, that I'll put below. And hopefully the fix @samsrabin has will also work for him.

These problems that "work for me, but not thee", are a real pain to figure out. And everyone's environment seems to be different enough that we run into different issues.

Here's John's experience. I don't think there's anything new here, but putting it in, in case it's helpful...

I looked at the log of that test and it says that when trying to run 'conda activate ctsm_pylib' it is still complaining about conda: "CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'." It tells me to run 'conda init' but 1) I already did that and 2) when I rerun it, it says "no action taken".

If I run that conda command by itself, it runs fine. But in the test failure it's part of this large command:

. /glade/scratch/jpalex/tests_0825-142614ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.C.0825-142614ch_int/.env_mach_specific.sh; conda activate ctsm_pylib && python3 /glade/scratch/jpalex/ctsm_bug29/tools/modify_input_files/fsurdat_modifier /glade/scratch/jpalex/tests_0825-142614ch/FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.C.0825-142614ch_int/[modify_fsurdat.cf](http://modify_fsurdat.cf/)

TIL that '.' at the front means 'source'. So the prob is apparently that env_mach_specific.sh causes the trouble with conda activate - and I've reproduced that. That sh file says:

# This file is for user convenience only and is not used by the model
# Changes to this file will be ignored and overwritten
# Changes to the environment should be made in env_mach_specific.xml
# Run ./case.setup --reset to regenerate this file
. /glade/u/apps/ch/opt/lmod/7.5.3/lmod/lmod/init/sh
module purge 
module load ncarenv/1.3 python/3.7.9 cmake/3.22.0 intel/19.1.1 esmf_libs mkl mpi-serial/2.3.0
module use /glade/p/cesmdata/cseg/PROGS/modulefiles/esmfpkgs/intel/19.1.1/
module load esmf-8.4.1b02-ncdfio-mpiuni-g ncarcompilers/0.5.0 netcdf/4.9.0 pio/2.5.10d
export OMP_STACKSIZE=1024M
export TMPDIR=/glade/scratch/jpalex
export MPI_TYPE_DEPTH=16
export MPI_USE_ARRAY=None
export ESMF_RUNTIME_PROFILE=ON
export ESMF_RUNTIME_PROFILE_OUTPUT=SUMMARY
export UGCSINPUTPATH=/glade/work/turuncu/FV3GFS/benchmark-inputs/2012010100/gfs/fcst
export UGCSFIXEDFILEPATH=/glade/work/turuncu/FV3GFS/fix_am
export UGCSADDONPATH=/glade/work/turuncu/FV3GFS/addon
export OMP_WAIT_POLICY=PASSIVE
export MPI_DSM_VERBOSE=true

My guess is that 'module purge' is the problem. If I remove that, it gives me another error that "Lmod has detected the following error: Cannot load module "python/3.7.9" because these module(s) are
loaded:
conda"
If I do a 'module unload conda', then the 'conda activate' command complains that it can't find 'conda'. Blerg

[samsrabin]

@johnpaulalex, my fix is at #2125. Would you be able to give that a try?

[johnpaulalex]

Hey Sam, I'm not confident in my choice of git/manage_externals commands, but in this case I did a git merge of your fix onto my branch, then ran that one test:
./run_sys_tests -t FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel -c ctsm5.1.dev130 --skip-generate

and its TestStatus.log says:

Building test for FSURDATMODIFYCTSM in directory /glade/scratch/jpalex/tests_0830-124645ch/FSURDATMODIFYCTSM_D_Mm
pi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.C.0830-124645ch
/glade/scratch/jpalex/ctsm_bug29/src/main/histFileMod.F90(25): error #6580: Name in only-list does not exist or i
s not accessible.   [NCLMAX]

/glade/scratch/jpalex/ctsm_bug29/src/main/histFileMod.F90(26): error #6580: Name in only-list does not exist or i
s not accessible.   [NLEVLEAF]

/glade/scratch/jpalex/ctsm_bug29/src/main/histFileMod.F90(30): error #6580: Name in only-list does not exist or i
s not accessible.   [NFSC]

/glade/scratch/jpalex/ctsm_bug29/src/main/histFileMod.F90(2496): error #6404: This name does not have a type, and
 must have an explicit type.   [NFSC]

/glade/scratch/jpalex/ctsm_bug29/src/main/histFileMod.F90(2500): error #6404: This name does not have a type, and
 must have an explicit type.   [NCLMAX]

/glade/scratch/jpalex/ctsm_bug29/src/main/histFileMod.F90(2501): error #6404: This name does not have a type, and
 must have an explicit type.   [NLEVLEAF]

ERROR: BUILD FAIL: clm.buildlib failed, cat /glade/scratch/jpalex/tests_0830-124645ch/FSURDATMODIFYCTSM_D_Mmpi-se
rial_Ld1.5x5_amazon.I2000Clm50SpRs.cheyenne_intel.C.0830-124645ch/bld/lnd.bldlog.230830-124732

...which is at least a different error :) but I'm not sure if it means it got past the 'conda activate' command or not.

[samsrabin]

Looks like it did, hurray! I can tell because of the presence of fsurdat.nc and done_FSURDATMODIFYCTSM_setup.txt in that directory.