ILAMB for Compy

[forsyth2]

Support ILAMB on Compy. The Compy component of #229.

[forsyth2]

Testing with:

[default]
case = 20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis
environment_commands = "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh"
#input = /compyfs/fors729/e3sm_unified_test_zppy/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis                     
input = /compyfs/fors729/zstash_extraction_20220526/v2.LR.historical_0201
input_subdir = archive/atm/hist
mapping_file = /compyfs/zender/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
output = /compyfs/fors729/zppy_ilamb_compy_output/v2.LR.historical_0201
partition = short
walltime = "02:00:00"
www = /compyfs/www/fors729/zppy_ilamb_compy_output

[ts]
active = True
frequency = "monthly"
years = "1851:1855:2",

  [[ atm_monthly_180x360_aave ]]
  input_files = "eam.h0"

  [[ land_monthly ]]
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  vars = "FSH,LAISHA,LAISUN,RH2M"

[ilamb_run]
active = True
grid = '180x360_aave'
short_name = 'v2.LR.historical_0201'
ts_num_years = 2
years = "1851:1853:2",

[forsyth2]

Testing with the following:

[default]
case = v2.LR.historical_0201
environment_commands = "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh"
#input = /compyfs/fors729/e3sm_unified_test_zppy/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis                                                                                                        
input = /compyfs/fors729/zstash_extraction_20220526/v2.LR.historical_0201
input_subdir = archive/atm/hist
mapping_file = /compyfs/zender/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
output = /compyfs/fors729/zppy_ilamb_compy_output/v2.LR.historical_0201
partition = short
walltime = "02:00:00"
www = /compyfs/www/fors729/zppy_ilamb_compy_output

[ts]
active = True
frequency = "monthly"
years = "1851:1855:2",
ts_fmt = "cmip"

  [[ atm_monthly_180x360_aave ]]
  input_files = "eam.h0"

  [[ land_monthly ]]
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  vars = "FSH,LAISHA,LAISUN,RH2M"

[ilamb_run]
active = True
grid = '180x360_aave'
short_name = 'v2.LR.historical_0201'
ts_num_years = 2
years = "1851:1853:2",

Results: https://compy-dtn.pnl.gov/fors729/zppy_ilamb_compy_output/v2.LR.historical_0201/ilamb/180x360_aave_1851-1852/

[forsyth2]

@chengzhuzhang This is ready for review. I was able to generate https://compy-dtn.pnl.gov/fors729/zppy_ilamb_compy_output/v2.LR.historical_0201/ilamb/180x360_aave_1851-1852/ using the cfg file in the previous comment.

I'll also test on Cori.

[forsyth2]

I can generate https://compy-dtn.pnl.gov/fors729/zppy_ilamb_compy_output/v2.LR.historical_0201/ilamb/180x360_aave_1851-1852/ without this TODO item. Is it necessary? (Or is it just that Compy and LCRC use the same mpi run format?)

[chengzhuzhang]

I think using ilamb_run means to run in serial, with more variables it could be more helpful using mpi. It might be good to check with @minxu74 . Hey Min, we are working on support ilamb in zppy on multiple platforms, and do you think it is necessary to running with MPI? Thanks!

[chengzhuzhang]

I think the PR itself is okay to merge.

[forsyth2]

I'm getting the following running with these changes on Cori. So that looks like an MPI issue. I think I can still merge this PR and get a release candidate started though, debugging this issue as part of the larger release testing.

Fri Jun  3 09:07:46 2022: [unset]:_pmi_alps_init:alps_get_placement_info returned with error -1
Fri Jun  3 09:07:46 2022: [unset]:_pmi_init:_pmi_alps_init returned -1
[Fri Jun  3 09:07:46 2022] [c6-5c0s14n1] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(537):
MPID_Init(246).......: channel initialization failed
MPID_Init(647).......:  PMI2 init failed: 1
/var/spool/slurmd/job59814450/slurm_script: line 234: 61937 Aborted                 e3sm_to_cmip --output-path ${dest_cmip}/${tmp_dir} --\
var-list 'pr, tas, rsds, rlds, rsus' --realm atm --input-path ${input_dir} --user-metadata /global/cscratch1/sd/forsyth/zppy_ilamb_cori_o\
utput/v2.LR.historical_0201/post/scripts/${workdir}/default_metadata.json --num-proc 12 --tables-path ${cmortables_dir}

[forsyth2]

Not going to commit the example cfg, but copying here for reference:

[default]
case = v2.LR.historical_0201
environment_commands = "source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh"
input = /compyfs/fors729/zstash_extraction_20220526/v2.LR.historical_0201
input_subdir = archive/atm/hist
mapping_file = /compyfs/zender/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
output = /compyfs/fors729/zppy_ilamb_compy_output/v2.LR.historical_0201
partition = short
walltime = "02:00:00"
www = /compyfs/www/fors729/zppy_ilamb_compy_output

[ts]
active = True
frequency = "monthly"
years = "1851:1855:2",
ts_fmt = "cmip"

  [[ atm_monthly_180x360_aave ]]
  input_files = "eam.h0"

  [[ land_monthly ]]
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  vars = "FSH,LAISHA,LAISUN,RH2M"

[ilamb_run]
active = True
grid = '180x360_aave'
short_name = 'v2.LR.historical_0201'
ts_num_years = 2
years = "1851:1853:2",