FATES API 35 compatibilty

[rgknox]

The FATE API has been updated which includes

1. control over the dimensionality of FATES history output and
2. an update to the parameter file format. This set of changes makes
   E3SM compatible with FATES tag sci.1.76.3_api.35.1.0.

[BFB] except for FATES

[rgknox]

This PR is a work in progress. At the least, an updated FATES parameter file has to be loaded and pointed to as the default, before tests can complete.

[glemieux]

Running fates test list on perlmutter. If the fates tests pass, this will be ready for review.

[glemieux]

All tests ran successfullly except for fates_cold_allvars test and ERS_Ld60.f45_g37.IELMFATES.pm-cpu_intel.elm-fates_cold_pphys.

The former is due to needing to update the history output list for the testmod, which I've fixed with bd00c67. I'm retesting that now.

The later is a little weirder as it hit the wall clock limit, even though the gnu version of this test ran successfully well under the limit. I'm resubmitting this to see if its a glitch of some sort.

[glemieux]

Resubmitting ERS_Ld60.f45_g37.IELMFATES.pm-cpu_intel.elm-fates_cold_pphys resulted in a succesful run with a much faster run time.

The fates_cold_allvars test is passing now as well after including an additional commit (9be6081) to change how the fates_history_dimlevel is being set. For some reason that I could quite figure out, we can't use fates_history_dimlevel = 2,2 as this would cause an ELMBuildNamelist error during setup. Splitting the setting and being explicit about the index that was being set works, however.
perlmutter location for the allvars test: /global/homes/g/glemieux/scratch/e3sm-tests/pr6419-allvarsrerun-splitarray.fates.pm-cpu.gnu.Cbd00c6724c-Ff0185f7c

@bishtgautam this is ready to review.

[rgknox]

Thanks for applying that fix @glemieux

[bishtgautam]

Looks good and I have one minor comment for you to consider.

[glemieux]

FYI @bishtgautam there's a recent fix (NGEET/fates#1208) to a fates-side issue related to this API update that I'd like to point this PR at once it's done testing. ETA for it is sometime later today.

[glemieux]

FYI @bishtgautam there's a recent fix (NGEET/fates#1208) to a fates-side issue related to this API update that I'd like to point this PR at once it's done testing. ETA for it is sometime later today.

@bishtgautam The testing on the fates side for the new PR is running into snags, so I'm going stick with the current fates tag that this PR is pointing to now.

Regression testing on pm-cpu against master baseline using the e3sm_land_developer list comes back with B4B results on all non-fates test except the 1x1_smallvilleIA tests. I think this is consistent with results looking back on the Cdash tests from when the perlmutter master was created. All fates tests diffs are as expected.

[glemieux]

@bishtgautam I've updated this PR with the latest fates tag having completed testing and integrated NGEET/fates#1208. I've retested this PR and the results are still the same as prior. This is good to integrate.