SASU crashes when altering SOM C:N ratio parameters

[djk2120]

Brief summary of bug

The matrix code (SASU) crashes when I edit the C:N ratio parameters for soil organic matter.
i.e. cn_s1_bgc, cn_s2_bgc, cn_s3_bgc

General bug information

CTSM version I am using: branch_tags/PPE.n10_ctsm5.1.dev030

Does this bug cause significantly incorrect results in the model's science? [No]

Configurations affected: [SASU]

Details of bug

Within the parameter perturbation experiment, we are running a four-step simulation:
(1) AD mode
(2) SASU (i.e. matrix)
(3) postSASU (regular BGC, but with matrix turned on)
(4) followed by a ten-year production run.

AD mode runs fine with the new parameter values, but SASU crashes in the third timestep. If I use the AD restart, (skipping SASU), postSASU and production also run without issue.

Important details of your setup / configuration so we can reproduce the bug

I have changed the value of cn_s1_bgc from 8 to 30.

Important output or errors that show the problem

column nbalance error = -1.343758647711035E-003 11
Latdeg,Londeg = 23.6842105263157 80.0000000000000
begnb = 1231.56523014687
endnb = 1231.56666712747
delta store = 1.436980597418369E-003
input mass = 1.416105268681299E-004
output mass = 4.838857716079532E-005
net flux = 9.322194970733462E-005
inputs,ffix,nfix,ndep = 4.189005887432090E-005 0.000000000000000E+000
9.972046799380904E-005
outputs,ffix,nfix,ndep = 3.555868345013498E-007 7.480727370490524E-023
3.907686301959916E-008
ENDRUN:
ERROR: ERROR in CNBalanceCheckMod.F90 at line 555

[ekluzek]

I'm able to replicate this problem with the branch version. Doing the following test:

SMS_D_Ld1.f10_f10_mg37.I1850Clm51Bgc.cheyenne_intel.clm-monthly_matrixcn_fast_spinup

Where I point to a paramfile that has cn_s1_bgc changed to 30.

I also noticed that it works if I increase it to 9., but fails on an increase to 10.75 or more.

[ekluzek]

I tried to turn off various things to see if one thing would show up for getting it to work. And I did find one, if I turn use_vertsoilc to FALSE it will run a short test. I'm not sure if that helps, but that might help looking into the guts of the code...

[ekluzek]

I also tried running with soil matrix off and works . And I tried with lower values of CN below 8 and it works for the default model but fails for soil-matrix until cn_s1_bgc is at least 7.2. So there's only a narrow range of the CN ratio that works for this parameter when soil matrix is on.

[chrislxj]

I tried to turn off various things to see if one thing would show up for getting it to work. And I did find one, if I turn use_vertsoilc to FALSE it will run a short test. I'm not sure if that helps, but that might help looking into the guts of the code...

It seems that soil CN ratio for SASU works only in a narrow range. I am not sure if this is related to the nonlinearity in FUN. Did you try run SASU by turning off FUN or Flexible_CN? I remember Cuijuan also had some issues to run SASU without FUN or Flexible_CN. But she didn't crash the model but got a ridiculous results instead. I am happy to test it when you have a specific case and corresponding code for me.

[ekluzek]

@chrislxj the only two things I saw that caused the error to go away was if use_soil_matrixcn was false, or use_vertsoilc was false. Turning off flexible-CN or FUN or Luna or anything else I tried didn't seem to matter.

I've pushed a test case to your cn-matrix_v3 branch. To run it on cheyenne do the following under cime/scripts. Be sure
to pull the latest changes to the branch, and also run manage_externals on it also.

qcmd -- ./create_test SMS_D_Lm1.f10_f10_mg37.I1850Clm45Bgc.cheyenne_intel.clm-clm45_monthly_matrixcn_soilCN30 -r . --queue premium --walltime 00:40:00

[ekluzek]

@chrislxj has a fix for this here: 624cce7

[ekluzek]

I tried @chrislxj solution for some different values of cn_s?_bgc from 2 to 45 as well as the default value of 8.0 and they all worked. Setting it to 1. failed, but only after 21 days. So the fix seems to be robust. I'll also run it through standard testing and make sure it works as expected.

[ekluzek]

The update passes tests. But, it does look like there is a roundoff change to it.

The cheyenne tests are all identical to branch_tags/PPE.n09_ctsm5.1.dev030 other than the following two which show differences in some Nitrogen terms to roundoff level...

ERS_D_Lm3.f09_g17.I1850Clm50BgcCrop.cheyenne_intel.clm-ciso_monthly_matrixcn
SMS_D.f10_f10_mg37.I1850Clm51BgcCrop.cheyenne_intel.clm-ciso_soil_matrixcn_only

on izumi more tests show differences and the differences are larger, and over all fields. Here 24 of the 50 tests show differences and those differences are larger. But, this is compared to the "n09" tag rather than the "n10" tag, because I didn't run baselines for the "n10" tag. So I reran the "n10" baseline and compared and as I suspected I see that fewer tests are actually different. On izumi in contrast to cheyenne the "n10" tag made a difference in answers for quite a few simulations. Then the only test that is different (and again only to roundoff level for a few Nitrogen fields) are:

SMS_D.f10_f10_mg37.I1850Clm51BgcCrop.izumi_nag.clm-ciso_soil_matrixcn_only

[ekluzek]

Closing as this is now fixed in branch_tags/PPE.n11_ctsm5.1.dev030