Interrupted when run 12years in CLM-FATES

[liujia96]

I want run a 50 years with FATES-CLM. I then took the cesm log when run about 12 years. There is "ERROR in FatesAllometryMod.F90 at line 767" in cesm.log.

[glemieux]

Hi @liujia96, can you provide some more detail about the case that you are running, specifically the ctsm and fates tags that you are using? Pasting some of the cesm.log output prior to the failure you reported would be helpful too.

[liujia96]

Hi @liujia96, can you provide some more detail about the case that you are running, specifically the ctsm and fates tags that you are using? Pasting some of the cesm.log output prior to the failure you reported would be helpful too.
Hi @glemienx, the details of the case that I'm running as follows
COMP= I2000Clm50Fates
./xmlchange RUN_TYPE=startup
./xmlchange RUN_STARTDATE=2015-01-01
./xmlchange DATM_CLMNCEP_YR_START=2015
./xmlchange DATM_CLMNCEP_YR_END=2050
./xmlchange STOP_N=41,STOP_OPTION=nyears
./xmlchange REST_N=10,STOP_OPTION=nyears
./xmlchange RESUBMIT=20,STOP_OPTION=nyears
And the cesm.log is too large to upload. The error details ad follows,
ENDRUN:
ERROR in FatesAllometryMod.F90 at line 767
ERROR: Unknown error submitted to shr_abort_abort.
Image PC Routine Line Source
cesm.exe 00000000016C32A6 Unknown Unknown Unknown
cesm.exe 000000000118E070 shr_abort_mod_mp_ 114 shr_abort_mod.F90
cesm.exe 00000000007AE830 fatesglobals_mp_f 65 FatesGlobals.F90
cesm.exe 00000000007A836A fatesallometrymod 767 FatesAllometryMod.F90
cesm.exe 000000000074FC0C edcohortdynamicsm 1162 EDCohortDynamicsMod.F90
cesm.exe 0000000000746833 edcanopystructure 211 EDCanopyStructureMod.F90
cesm.exe 0000000000758AB1 edmainmod_mp_ed_u 525 EDMainMod.F90
cesm.exe 0000000000537A74 clmfatesinterface 684 clmfates_interfaceMod.F90
cesm.exe 0000000000515FB3 clm_driver_mp_clm 904 clm_driver.F90
cesm.exe 000000000050275D lnd_comp_mct_mp_l 456 lnd_comp_mct.F90
cesm.exe 00000000004368F0 component_mod_mp_ 728 component_mod.F90
cesm.exe 000000000041ACFF cime_comp_mod_mp_ 2720 cime_comp_mod.F90
cesm.exe 0000000000436537 MAIN__ 125 cime_driver.F90
cesm.exe 00000000004189E2 Unknown Unknown Unknown
libc-2.17.so 00002B882D2B9445 __libc_start_main Unknown Unknown
cesm.exe 00000000004188E9 Unknown Unknown Unknown
Abort(1001) on node 121 (rank 121 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 121

[glemieux]

@liujia96 can you provide the ctsm model version, please? You should be able to access this from case directory by running ./xmlquery MODEL_VERSION.

[liujia96]

@liujia96 can you provide the ctsm model version, please? You should be able to access this from case directory by running ../xmlquery MODEL_VERSION
When I running ./xmlquery MODEL_VERSION, it is shown MODEL_VERSION: unknown.

[glemieux]

Huh, that's interesting. Would you try this instead, please? The following assumes you didn't change your ctsm directory name when you cloned it:

$ cd CTSM
$ git describe

this should give you the relevant ctsm tag. You can get the fates tag by changing into the fates directory and using the same command:

$ cd src/fates
$ git describe

[liujia96]

Huh, that's interesting. Would you try this instead, please? The following assumes you didn't change your ctsm directory name when you cloned it:

$ cd CTSM
$ git describe

this should give you the relevant ctsm tag. You can get the fates tag by changing into the fates directory and using the same command:

$ cd src/fates
$ git describe

I get the following results:
cd CTSM
$ git describe
release-clm5.0.30
cd src/fates
$ git describe
sci.1.30.0_api.8.0.0

[glemieux]

Thanks for that information. @rgknox suggests that you should try and increase the nlevleaf value to see if you can get the run to pass. That said, he noted that this may be an indication of odd allometry: could you also provide some background on what your are broadly attempting?

[rgknox]

Yeah, @liujia96 , if I were you, I would try increasing nlevleaf (see main/EDTYpesMod.F90) to something like 50. It will slow down the model a little bit, but if the simulation completes, or you need faster runs, you can then try decreasing that number a little bit.
The other thing that you might want to consider is using a more recent version of the model. That FATES tag is quite old. If you use a more recent version of CLM, it will also automatically pull in a more recent version of FATES. FATES and CLM have changed and improved quite a bit since sci.1.30_api8.0.

Do you have other constraints guiding the tag you are using (like is this the version that you and your team started with a while back and have been modifying it for your specific research needs over time)?

If you need guidance on a best recent version of CLM to use, we can ping @ekluzek @billsacks and others.

[liujia96]

Thanks rgknox and glemieux. This infomation gives me plenty to plenty to explore. The exciting thing is that this case has been successfully run for 50 years when I set nlevleaf=40.