soil temperature, soil moisture BandDiagonal solver blowing up.

[timhoar]

Brief summary of bug

I am running a multi-instance case with data assimilation. I am (gently) modifying soil moisture in one cycle that causes the BandDiagonal solver to return an error code which halts the execution of the model.

General bug information

CTSM version you are using: [output of git describe]
release-clm5.0.06

Does this bug cause significantly incorrect results in the model's science? [Yes / No]
model fails to run

Configurations affected: [Fill this in if known.]

Details of bug

The traceback indicates the SoilTemperatureMod is throwing the error - again - I am not adjusting the soil temperature, but I am screwing with the soil moisture. The range of soil moisture values at the column I believe is in question is from 10 to several hundred, and the adjusted values differ from these less than 1.0,
usually much less. Andy Fox has run single-point experiments successfully, but both he and I cannot run a global simulation where we can adjust the soil moisture.

414:Image PC Routine Line Source
414:cesm.exe 00000000015C0D5D Unknown Unknown Unknown
414:cesm.exe 0000000000D46B92 shr_abort_mod_mp_ 114 shr_abort_mod.F90
414:cesm.exe 00000000004F9FBB abortutils_mp_end 50 abortutils.F90
414:cesm.exe 0000000000B2ECD1 banddiagonalmod_m 213 BandDiagonalMod.F90
414:cesm.exe 000000000098394B soiltemperaturemo 408 SoilTemperatureMod.F90
414:cesm.exe 000000000050047F clm_driver_mp_clm 649 clm_driver.F90
414:cesm.exe 00000000004EEC0B lnd_comp_mct_mp_l 456 lnd_comp_mct.F90
414:cesm.exe 0000000000424974 component_mod_mp_ 728 component_mod.F90
414:cesm.exe 000000000040A0BA cime_comp_mod_mp_ 2662 cime_comp_mod.F90
414:cesm.exe 00000000004246A2 MAIN__ 103 cime_driver.F90
414:cesm.exe 00000000004080DE Unknown Unknown Unknown
414:libc-2.19.so 00002AAAB04DCB25 __libc_start_main Unknown Unknown
414:cesm.exe 0000000000407FE9 Unknown Unknown Unknown

I have looked at other Github issues: #202, #223 which may be related. It is entirely possible the code is doing the right thing and I am simply making the soil unstable. If this is the case, I'd like some guidance on how to modify soil moisture (and soil temperature, ultimately) such that the new state is stable.

Important details of your setup / configuration so we can reproduce the bug

I extracted the messages from a single task (for clarity) and include them here.
clear.txt

Important output or errors that show the problem

/glade/work/thoar/cases/ctsm/clm5_f09_SIF_e40
/glade/scratch/thoar/ctsm/clm5_f09_SIF_e40/run/cesm.log.3197825.chadmin1.181031-133515

[billsacks]

@timhoar I'm going through some old issues. Sorry you never received a reply here. Do you still need help with this, or should we close this issue?

[timhoar]

Good timing ... I am getting back into this and will surely need some help. When I get to the point where it is clear I have a grasp of the problem and can articulate it, I'll get back to you. Thanks for following up. Tim Hoar Data Assimilation Research Section National Center for Atmospheric Research [email protected] 303.497.1708

…

On Fri, May 24, 2019 at 5:22 AM Bill Sacks ***@***.***> wrote: @timhoar <https://github.com/timhoar> I'm going through some old issues. Sorry you never received a reply here. Do you still need help with this, or should we close this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#556>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADTRYOU77ZLPW37YLGQ52MTPW7FXTANCNFSM4GBDAVIQ> .

[ekluzek]

Unfortunately Tim has retired, and we didn't have an answer for this. So I'm closing this as a wontfix.