Allow cfg files to be more machine-independent

[forsyth2]

https://github.com/E3SM-Project/zppy/blob/main/tests/integration/utils.py#L125 -- can we reduce the number of parameters that are machine-specific?

For example, the diags paths could be consolidated into a single parameter.

[xylar]

Shouldn't the diags_path come from mache anyway, at least for supported machines?

[forsyth2]

yes, but there are a variety of derived paths. E.g.,

        "diags_obs_climo": f"{diags_base_path}/observations/Atm/climatology/",
        "diags_obs_tc": f"{diags_base_path}/observations/Atm/tc-analysis/",
        "diags_obs_ts": f"{diags_base_path}/observations/Atm/time-series/",

[xylar]

I see. I think those should just be hard-coded at this point, and the diags_base_path should come from mache when possible but be a config option for machines that mache doesn't support (unless zppy only supports machines that mache also supports).

[forsyth2]

I think those should just be hard-coded at this point

I think some users may want to use completely different dataset paths though.

unless zppy only supports machines that mache also supports

I think we're officially supporting the same machines -- the ones on Unified.

[xylar]

I see, in that case carry on. It doesn't sound like you need any input from me, but ping me if so.

[golaz]

@forsyth2 : in most cases (99%+), a user will want to use the standard observational datasets for diags_obs_climo, diags_obs_tc, diags_obs_ts and should not have to know or worry about where these datasets reside.

The paths should be set transparently, either in zppy or E3SM Diags with the diags_base_path coming from mache.

[golaz]

@forsyth2 : here is an example of an e3sm_diags section from a zppy configuration file on chrysalis:

[e3sm_diags]
active = True
years = "1985:2014:30",
ts_num_years = 5
ref_start_yr = 1979
ref_final_yr = 2016

  [[ atm_monthly_180x360_aave ]]
  short_name = 'v2_1.LR.historical_0101'
  grid = '180x360_aave'
  sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere'
  reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
  obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
  dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
  climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
  climo_diurnal_frequency = "diurnal_8xdaily"
  output_format_subplot = "pdf",

We want this to work properly without having to specify reference_data_path, obs_ts, dc_obs_climo. Those are the parameters that make the file difficult to port.

For other tasks, the main offending parameter is

  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc

which should support specifying directly a filename

  mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc

with zppy relying on mache to find the absolute path where these files reside. The user should still have the option of specifying and absolute path in they choose to bypass the official mapping files.

[golaz]

Another parameter would be

environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"

which a user should have the option of omitting completely with zppy relying on mache to determine the proper way to load the latest E3SM Unified environment.

There are other machine dependent parameters such as input, output, www, partition which a user will still have to modify manually when going from one machine to another.

[forsyth2]

Re: diags observations paths -- I'm working on automating those parameters.

Re: mapping files -- this was completed in #381.

Re: environment_commands -- this was completed in #330:

    # Determine environment_commands                                                                                                                                                                                      
    if config["default"]["environment_commands"] == "":
        machine = config["default"]["machine"]
        config["default"][
            "environment_commands"
        ] = f"source {unified_base}/load_latest_e3sm_unified_{machine}.sh"

[forsyth2]

Re: diags observations paths -- I'm working on automating those parameters.

Observation paths are automated in #426.