PVT_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesLUPFT fails in RUN phase

[slevis-lmwg]

Error message

Error shown here appears in TestStatus.log:

• The first timestamp is from submitting the fates test-suite on derecho.
• The second timestamp from trying ./case_submit manually in case the first failure was a fluke.
• I made a third attempt by submitting the test with ./create_test and got the same error as in the first timestamp.

2024-12-30 13:49:44: ERROR: Cannot modify case, read_only. Case must be opened with read_only=False and can only be modified within a context manager
 ---------------------------------------------------
2024-12-30 16:10:46: ERROR: Command: 'git -C /glade/derecho/scratch/slevis/tests_1230-131139de/PVT_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesLUPFT.GC.1230-131139de_int commit -m "
ERROR: RUN FAIL: Command 'mpibind  --label  --line-buffer  --  /glade/derecho/scratch/slevis/tests_1230-131139de/PVT_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesLUPFT.GC.1230-131139de_int/bld/cesm.exe   >> cesm.log.$LID 2>&1 ' failed
See log file for details: /glade/derecho/scratch/slevis/tests_1230-131139de/PVT_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesLUPFT.GC.1230-131139de_int.potveg/run/cesm.log.7350030.desched1.241230-155114"' failed with error 'On branch PVT_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesLUPFT.GC.1230-131139de_int
nothing to commit, working tree clean' from dir '/glade/derecho/scratch/slevis/tests_1230-131139de/PVT_Lm3.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesLUPFT.GC.1230-131139de_int.potveg'

General bug information

CTSM version you are using:
branch_tags/tmp-241219.n01.ctsm5.3.016
...would have been ctsm5.3.017 if master were not "locked" while we wait for a cesm tag.

Does this bug cause significantly incorrect results in the model's science? [Yes / No]
??

Configurations affected: [Fill this in if known.]
??

**Other info
I was running this test to generate this new fates baseline after merging b4b-dev to master (in this case tmp-241219 because master is "locked" while we wait for a new cesm tag):
fates-sci.1.79.3_api.37.0.0-tmp-241219.n01.ctsm5.3.016

The same test appears to have not completed also in the previous baseline that I was comparing to:
fates-sci.1.79.3_api.37.0.0-ctsm5.3.016

[glemieux]

Looking at the code where this error comes up (relevant CIME line in case.py), the trigger appears to be one of the many self._case.set_value calls in the pvt.py system test definiton:

CTSM/cime_config/SystemTests/pvt.py

Lines 73 to 77 in f437651

	with clone:
	clone.set_value("CLM_ACCELERATED_SPINUP", "off")
	clone.set_value("STOP_N", stop_n_pveg)
	clone.set_value("STOP_OPTION", "nyears")
	clone.set_value("RUN_STARTDATE", "{}-01-01".format(startyear_pveg))

CTSM/cime_config/SystemTests/pvt.py

Lines 123 to 130 in f437651

	self._case.set_value("CLM_ACCELERATED_SPINUP", "off")
	self._case.set_value("RUN_TYPE", "hybrid")
	self._case.set_value("GET_REFCASE", False)
	self._case.set_value("RUN_REFCASE", "{}.potveg".format(orig_casevar))
	self._case.set_value("RUN_REFDATE", "{}-01-01".format(refcase_year))
	self._case.set_value("RUN_STARTDATE", "{}-01-01".format(refcase_year))
	self._case.set_value("DOUT_S", False)
	self._case.flush()

[wwieder]

@glemieux asked if this had something to do with submodules.

@ekluzek suggested that something may have changed in cime related to clones that could be worth looking into, which came in with the rpointer tag on 016