Updates to gcm-driven single column calibrations

[costachris]

Improve gcm-driven single column calibrations:

1. modify priors
2. add entrainment pi group calibration
3. improve plotting scripts
4. add rmse metrics and plotting
5. add eki metrics and plotting
6. parallelize CA single column runs (cases) over cpu cores
7. easier specification of normalization for LES/EDMF profiles (mean, std to transform vars -> N(0, 1)); option for taking log

A summary of the

The observation vector for a single configuration is formed by concatenating profiles across calibration variables, where each geophysical variable is normalized to have approximately unit variance and zero mean. These variable-by-variable normalization factors are precomputed (norm_factors_dict) and applied to all observations. Following this operation, the spatiotemporal calibration window is applied and temporal means are computed to form the observation vector y. Because variables are normalized to have 0 mean and unit variance, a constant diagonal noise matrix is used (configurable as const_noise).

Observation Map

1. Time-mean: Time-mean of profiles taken between [y_t_start_sec, y_t_end_sec] for y and [g_t_start_sec, g_t_end_sec] for G.
2. Interpolation: Case-specific (i.e., "shallow", "deep") interpolation to the calibration grid, defined with stretch-grid parameters in z_cal_grid.
3. Normalization: Variable-specific normalization using the mean and standard deviation defined in norm_factors_by_var. Optionally, take log of variables using log_vars before normalization.
4. Concatenation: Stack across cases in a batch, forming y, G.

Prognostic EDMF results after 11 iterations with default calibration configurations (defined in experiment_config.yml)

cfSite 23

cfSite 17

RMSE Plot

[odunbar]

I've taken a look, and LGTM!

• Could you add into the PR message about the additional normalization / data processing added for LES
• Could you check the minibatching? The setting of FixedSizeMinibatcher[1:k] will only (i think) create one minibatch. i.e. this is the same as calibrating over only cases 1:k - was this the intent? or did you want something like collect(1:k),collect(k+1:2k),... If so this needs to be changed.

Otherwise my comments are really just small questions.

[odunbar]

Do you think this input format is sustainable? is there something you would have preferred to put in as input in an ideal world. We could add something into the EKP toml parsing if this is the case

[costachris]

I think it can get unwieldy with i.e. hundreds of NN parameters. As is, I believe it allows for "repeat" logic to define the prior vector, which I could use to make it more concise.

Otherwise I have a use case where I need to load pretrained NN weights as prior means, so I will probably need to load that from a file and set a constant spread. This may be a niche use case though.