Plant Growth Integrator error

[ckoven]

I'm currently running a parameter ensemble and have the following crash occurring about 150 years into the simulation. It shows up as an error in the plant growth integrator. I note that the trees causing the crash are really very big -- probably unrealistically so, but not so unrealistic that it should be out of bounds and taking down the whole ensemble for this kind of perturbed parameter case. Perhaps we should loosen the tolerance for gigantic trees a bit if there is a safe way to do so?

64: ERROR: Unknown error submitted to shr_abort_abort.
64:Image              PC                Routine            Line        Source             
64:cesm.exe           000000000128F3AD  Unknown               Unknown  Unknown
64:cesm.exe           0000000000D72A32  shr_abort_mod_mp_         114  shr_abort_mod.F90
64:cesm.exe           0000000000788520  fatesglobals_mp_f          62  FatesGlobals.F90
64:cesm.exe           000000000075C275  edphysiologymod_m        1292  EDPhysiologyMod.F90
64:cesm.exe           00000000007425AF  edmainmod_mp_ed_e         273  EDMainMod.F90
64:cesm.exe           000000000052AEFF  clmfatesinterface         654  clmfates_interfaceMod.F90
64:cesm.exe           000000000050B142  clm_driver_mp_clm         879  clm_driver.F90
64:cesm.exe           00000000004F9756  lnd_comp_mct_mp_l         451  lnd_comp_mct.F90
64:cesm.exe           0000000000430E24  component_mod_mp_         688  component_mod.F90
64:cesm.exe           0000000000417D98  cime_comp_mod_mp_        2652  cime_comp_mod.F90
64:cesm.exe           0000000000430B4D  MAIN__                     68  cime_driver.F90
64:cesm.exe           0000000000415C5E  Unknown               Unknown  Unknown
64:libc-2.19.so       00002AAAB190AB25  __libc_start_main     Unknown  Unknown
64:cesm.exe           0000000000415B69  Unknown               Unknown  Unknown
64:MPT ERROR: Rank 64(g:64) is aborting with error code 1001.
64:	Process ID: 9744, Host: r1i7n30, Program: /glade2/scratch2/charlie/fates_clm5_fullmodel_bci_parameter_ensemble_1pft_v003_6341c18_819609f/bld/cesm.exe
64:	MPT Version: SGI MPT 2.15  12/18/16 02:58:06
64:

and the error log for the corresponding land process shows:

 clm: calling FATES model      2568145
FATES Dynamics:  147-12-31
 Plant Growth Integrator could not find
 a solution in less than          300  tries
 Aborting
 carbon_balance   7.76424553045382     
 deltaC  3.811540699651795E-090
 totalC   7.76424553045382     
 leaf: F   357.914575887660       -75.8192683239706     
 froot: F   180.318847691102       -39.0183027932908     
 sap: F   25.3445500428634       -1.41657728097951     
 store: F   715.829151775321       -33.0119530905745     
 dead:   161061.451321624       0.000000000000000E+000
 ----------------------------------------
  Dumping Cohort Information             
 ----------------------------------------
 co%pft                    =            1
 co%n                      =   0.257690742056602     
 co%dbh                    =    481.627999928781     
 co%hite                   =    47.0786569111734     
 co%bdead                  =    161061.451321624     
 co%bstore                 =    748.841104865895     
 co%laimemory              =   0.000000000000000E+000
 co%bsw                    =    26.7611273238429     
 co%bl                     =    433.733844211631     
 co%br                     =    219.337150484392     
 co%lai                    =    1.00043141597921     
 co%sai                    =   2.080905428110708E-002
 co%g_sb_laweight          =    1.29030773571305     
 co%leaf_cost              =   6.451141059669688E-002
 co%canopy_layer           =            1
 co%canopy_layer_yesterday =    1.00000000000000     
 co%nv                     =            6
 co%status_coh             =            2
 co%canopy_trim            =   0.737827523911549     
 co%excl_weight            =   0.000000000000000E+000
 co%prom_weight            =   0.000000000000000E+000
 co%size_class             =           26
 co%size_by_pft_class      =           26
 co%gpp_acc_hold           =    10482.9782596925     
 co%gpp_acc                =    28.7204883827191     
 co%gpp_tstep              =   0.000000000000000E+000
 co%npp_acc_hold           =    6879.99164164544     
 co%npp_tstep              =  -0.145551316647942     
 co%npp_acc                =    18.8492921688916     
 co%resp_tstep             =   0.145551316647942     
 co%resp_acc               =    9.87119621382747     
 co%resp_acc_hold          =    3602.98661804703     
 co%npp_leaf               =    981.560303882605     
 co%npp_fnrt              =    219.337150484392     
 co%npp_sapw               =   0.000000000000000E+000
 co%npp_dead              =    2845.14456866280     
 co%npp_seed              =   0.000000000000000E+000
 co%npp_stor              =   0.000000000000000E+000
 co%ode_opt_step          =   1.905770349825897E-090
 co%rdark                  =   6.100419266429463E-005
 co%resp_m                 =   0.145551316647942     
 co%resp_g                 =   0.000000000000000E+000
 co%livestem_mr            =   1.193984983147526E-006
 co%livecroot_mr           =   8.685155689630378E-007
 co%froot_mr               =   1.779514936578481E-005
 co%md                     =   0.000000000000000E+000
 co%leaf_md                =    981.560303882605     
 co%root_md                =    219.337150484392     
 co%bstore_md              =    14.9776427900735     
 co%bdead_md               =    3221.24963492925     
 co%bsw_md                 =   0.535251875346741     
 co%dmort                  =   3.092458657920361E-002
 co%seed_prod              =   0.000000000000000E+000
 co%treelai                =    5.80310526942006     
 co%treesai                =   0.120705058459350     
 co%leaf_litter            =   0.000000000000000E+000
 co%c_area                 =    916.925387002005     
 co%cmort                  =   0.000000000000000E+000
 co%bmort                  =   3.092458657920361E-002
 co%fmort                  =   0.000000000000000E+000
 co%hmort                  =   0.000000000000000E+000
 co%frmort                 =   0.000000000000000E+000
 co%isnew                  =  F
 co%dndt                   =  -3.984489831694304E-003
 co%dhdt                   =   0.000000000000000E+000
 co%ddbhdt                 =   0.000000000000000E+000
 co%dbdeaddt               =   0.000000000000000E+000
 co%dbstoredt              =   0.000000000000000E+000
 co%cfa                    =   0.000000000000000E+000
 co%fire_mort              =   0.000000000000000E+000
 co%crownfire_mort         =   0.000000000000000E+000
 co%cambial_mort           =   0.000000000000000E+000
 co%size_class             =           26
 co%size_by_pft_class      =           26
 ----------------------------------------
 ENDRUN:
 ERROR in EDPhysiologyMod.F90 at line 1292                                      
                                                      

[rgknox]

A 5 meter diameter tree could exist, so its big but still feasible right?

Lets increase max_trunc_error to something suitably high, like 100, and make an issue?

[ckoven]

closing, as it seems to have been fixed by #391.