Implement "equilibrium state" test for SA part of calibration process.

[tobeycarman]

Need to expand upon the concept that is drafted in "mads_calibration/equlibrium_check.py".

[tobeycarman]

Thinking that this might get built as part of drivers/Sensitivity.py such that the "collate results" function will run the various equilibrium state checks (stability, slope, etc) and output a file equilibrium_state.csv in each sample directory. Each column of the csv file may have a boolean value for whether each of the respective tests pass. Then this csv file can be read at a later date to assess whether the sample run reached equlibrium via various metrics.

[tobeycarman]

So far checks we have thought of are:

• slope of a linear regression over N years
• "stability" which is the coefficient of variation over N years
• long term stability where you look at a long time series and see if there are periods of equilibrium separated by distinct steps from one equilibrium state to another

[tobeycarman]

Propose that the "equlibrium check" logic is added to the util/metrics.py module.

[Elchin]

In general, it would be nice to build the equilibrium check into the C++ code of the dvm-dos-them. This will eliminate the extra steps that need to be taken to make sure that the model is equilibrated. In addition, this could help with the overall model -eq case when the model will check and decide when the equilibrium run needs to be terminated without the user specifying the number of years because some cases might take longer or shorter to equilibrate. When we provide a hard number, usually 2000 years, if the model is not at equilibrium, then it will continue to lose carbon during the -tr and -sc stages, which would make it harder to match observations.

[tobeycarman]

@Elchin yeah, we would like to put this in the C++ code but decided that we should implement in Python first as a "post hoc" check because it will be faster for development and will allow us to experiment with different methods of testing for equilibrium state.

[valeriabriones]

@tobeycarman not sure if this structurally makes sense, but in thinking about the workflow post SA, right now the equilibrium check is for all samples run from SA, but am curious if there's another way to have it work from the SA_post_hoc_analysis.n_top_runs(), which selects the samples which meet these criteria based on r2lim and have an equilibrium check work from these top samples as a way to complement or supplement the SA results? Just a thought! Have been trying to work through it but its messy

[tobeycarman]

Yes! Have been thinking along those lines. Current thought is that there will be a step in the end of each run that makes a bunch of measurements about the equilibrium “quality” of the equilibrium and then summarizes this info in a csv file (similar to the results.csv). Then in post hoc analysis, user can filter list of runs based on the quality measurements, similar to the n top runs.

…

On Mon, Feb 5, 2024 at 10:42 AM valeriabriones ***@***.***> wrote: @tobeycarman <https://github.com/tobeycarman> not sure if this structurally makes sense, but in thinking about the workflow post SA, right now the equilibrium check is for all samples run from SA, but am curious if there's another way to have it work from the SA_post_hoc_analysis.n_top_runs(), which selects the samples which meet these criteria based on r2lim and have an equilibrium check work from these top samples as a way to complement or supplement the SA results? Just a thought! Have been trying to work through it but its messy — Reply to this email directly, view it on GitHub <#666 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGMYT6HBZZ3PIDEFQ5GD63YSEYZTAVCNFSM6AAAAABCJQJOIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRXHEZDGNRQGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hgenet]

And just a reminder here - the parameters/tolerance to assess the stability of an eq run might vary depending on the variable and associated turnover.

…

On Feb 5, 2024, at 10:59 AM, tobey ***@***.***> wrote: Yes! Have been thinking along those lines. Current thought is that there will be a step in the end of each run that makes a bunch of measurements about the equilibrium “quality” of the equilibrium and then summarizes this info in a csv file (similar to the results.csv). Then in post hoc analysis, user can filter list of runs based on the quality measurements, similar to the n top runs. On Mon, Feb 5, 2024 at 10:42 AM valeriabriones ***@***.***> wrote: > @tobeycarman <https://github.com/tobeycarman> not sure if this > structurally makes sense, but in thinking about the workflow post SA, right > now the equilibrium check is for all samples run from SA, but am curious if > there's another way to have it work from the > SA_post_hoc_analysis.n_top_runs(), which selects the samples which meet > these criteria based on r2lim and have an equilibrium check work from these > top samples as a way to complement or supplement the SA results? Just a > thought! Have been trying to work through it but its messy > > — > Reply to this email directly, view it on GitHub > <#666 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAGMYT6HBZZ3PIDEFQ5GD63YSEYZTAVCNFSM6AAAAABCJQJOIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRXHEZDGNRQGY> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#666 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANVJF6ADDY26H7U6ELFEV3YSE23JAVCNFSM6AAAAABCJQJOIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRXHE4TSMBVGM>. You are receiving this because you are subscribed to this thread.

[tobeycarman]

@valeriabriones, @Benjamin-Maglio, @jsclein-uaf, just pushed changes (commit: ffd82c0) implementing a draft roughly as discussed above. Details in the commit message.🤞 that it might run for you. Note that I did not push anything that actually "checks" the equilibrium - the code I pushed makes various "measurements" and summarizes the data. The "check" still needs to be written...i.e. "only use samples with slope < .1" or "only use samples with cv<.5".