Skip to content

Commit

Permalink
Merge pull request #8 from ESMCI/maint-5.6
Browse files Browse the repository at this point in the history
Updating  maint-5.6 before pushing cime_reanalysis_2019 from cheyenne:...cesm2_1_maint-5.6
  • Loading branch information
kdraeder authored Dec 17, 2018
2 parents 2a7a98a + 5e2af84 commit e91b287
Showing 34 changed files with 607 additions and 339 deletions.
141 changes: 140 additions & 1 deletion ChangeLog
Original file line number Diff line number Diff line change
@@ -1,7 +1,146 @@
======================================================================

Originator: Chris Fischer
Date: 11-12-2018
Date: 11-29-2018
Tag: cime_cesm2_1_rel_06
Answer Changes: None
Tests: scripts_regression_tests
Dependencies:

Brief Summary:
- Fix new pylint errors.
- Update addmetadata for POP-ECT tests.

User interface changes:

PR summary: git log --oneline --first-parent [previous_tag]..master
5a86646 Merge pull request #2921 from jedwards4b/fixpylint3errors
98821ff Merge pull request #2919 from ESMCI/fischer/addmetadata

Modified files: git diff --name-status [previous_tag]
M scripts/lib/CIME/BuildTools/macrowriterbase.py
M scripts/lib/CIME/SystemTests/erio.py
M scripts/lib/CIME/SystemTests/ers.py
M scripts/lib/CIME/SystemTests/ers2.py
M scripts/lib/CIME/SystemTests/homme.py
M scripts/lib/CIME/SystemTests/system_tests_common.py
M scripts/lib/CIME/SystemTests/system_tests_compare_two.py
M scripts/lib/CIME/case/case_run.py
M scripts/lib/CIME/case/case_setup.py
M scripts/lib/CIME/case/case_st_archive.py
M scripts/lib/CIME/provenance.py
M scripts/lib/e3sm_cime_mgmt.py
M src/build_scripts/buildlib.gptl
M src/build_scripts/buildlib.pio
M tools/statistical_ensemble_test/addmetadata.sh

======================================================================

======================================================================

Originator: Chris Fischer
Date: 11-17-2018
Tag: cime_cesm2_1_rel_05
Answer Changes: None
Tests: ./create_test cime_developer on cheyenne
Dependencies:

Brief Summary:
- Do not include time of day in cpl auxiliary hist file names.

User interface changes:
- Changes names of cpl auxiliary history files.

PR summary: git log --oneline --first-parent [previous_tag]..master
b3fc9d6 Merge pull request #2914 from billsacks/auxhist_no_time

Modified files: git diff --name-status [previous_tag]
M src/drivers/mct/main/seq_hist_mod.F90

======================================================================
======================================================================

Originator: Chris Fischer
Date: 11-13-2018
Tag: cime_cesm2_1_rel_04
Answer Changes: None
Tests:
Dependencies:

Brief Summary:
- Fix 2010 period for datm GSWP3v1.
- Update pyCECT to v3.0.8

User interface changes:

PR summary: git log --oneline --first-parent [previous_tag]..master
2e72d20 Merge pull request #2908 from ekluzek/fix2010datm
5f4a911 Merge pull request #2905 from ESMCI/fischer/pyCECT_v3.0.8

Modified files: git diff --name-status [previous_tag]
M src/components/data_comps/datm/cime_config/config_component.xml
M tools/statistical_ensemble_test/pyCECT/README_pyCECT.rst
M tools/statistical_ensemble_test/pyCECT/README_pyEnsSum.rst
M tools/statistical_ensemble_test/pyCECT/README_pyEnsSumPop.rst
M tools/statistical_ensemble_test/pyCECT/pyCECT.py
M tools/statistical_ensemble_test/pyCECT/pyEnsLib.py
M tools/statistical_ensemble_test/pyCECT/pyEnsSum.py
M tools/statistical_ensemble_test/pyCECT/pyEnsSumPop.py
D tools/statistical_ensemble_test/pyCECT/pyEnsSumPop_test.sh
M tools/statistical_ensemble_test/pyCECT/test_pop_CECT.sh
D tools/statistical_ensemble_test/pyCECT/test_pop_ensemble.sh
A tools/statistical_ensemble_test/pyCECT/test_pyEnsSumPop.sh

======================================================================

======================================================================

Originator: Chris Fischer
Date: 11-5-2018
Tag: cime_cesm2_1_rel_03
Answer Changes: None
Tests: code_checker, stand-alone tests, scripts_regression_tests
Dependencies:

Brief Summary:
- Add Total Time parsing for timeseries and xconform to archive_metadata
- Fine tuning for CMIP6 cases in archive_metadata.
- CESM-CECT update.
- Python3 compatibility update for archive_metadata.
- Remove reference to deleted testmods directories.
- Compute tasks_per_node via floor division.
- Response to review comments of pr 2844 and remove user-grid option.

User interface changes:

PR summary: git log --oneline --first-parent [previous_tag]..master
c515c37 Merge pull request #2900 from bertinia/maint-5.6_archive_metadata
47f3970 Merge pull request #2884 from bertinia/maint-5.6_archive_metadata
3ad6a6e Merge pull request #2866 from jedwards4b/set_update
a2b6885 Merge pull request #2865 from bertinia/maint-5.6_archive_metadata
b56664c Merge pull request #2862 from billsacks/doc_testmods
ad361d2 Merge pull request #2860 from jedwards4b/py3fix
7e82e9b Merge pull request #2853 from jedwards4b/cleanuppr


Modified files: git diff --name-status [previous_tag]
M doc/source/users_guide/testing.rst
M scripts/Tools/archive_metadata
M scripts/create_newcase
M scripts/fortran_unit_testing/run_tests.py
M scripts/lib/CIME/XML/env_mach_pes.py
M tools/statistical_ensemble_test/README
M tools/statistical_ensemble_test/ensemble.py
D tools/statistical_ensemble_test/ensemble.sh
M tools/statistical_ensemble_test/single_run.py
D tools/statistical_ensemble_test/single_run.sh

======================================================================

======================================================================

Originator: Chris Fischer
Date: 10-12-2018
Tag: cime_cesm2_1_rel_02
Answer Changes: None
Tests: Z_FullSystemTest, scripts_regression_tests, hand tests
14 changes: 7 additions & 7 deletions config/cesm/config_grids.xml
Original file line number Diff line number Diff line change
@@ -1071,8 +1071,8 @@
<nx>144</nx> <ny>96</ny>
<file grid="atm|lnd" mask="gx1v6">domain.lnd.fv1.9x2.5_gx1v6.090206.nc</file>
<file grid="ocnice" mask="gx1v6">domain.ocn.1.9x2.5_gx1v6_090403.nc</file>
<file grid="atm|lnd" mask="gx1v7">domain.lnd.fv1.9x2.5_gx1v7.170518.nc</file>
<file grid="ocnice" mask="gx1v7">domain.ocn.fv1.9x2.5_gx1v7.170518.nc</file>
<file grid="atm|lnd" mask="gx1v7">domain.lnd.fv1.9x2.5_gx1v7.181205.nc</file>
<file grid="ocnice" mask="gx1v7">domain.ocn.fv1.9x2.5_gx1v7.181205.nc</file>
<file grid="ocnice" mask="null">domain.aqua.fv1.9x2.5.nc</file>
<desc>1.9x2.5 is FV 2-deg grid:</desc>
</domain>
@@ -1521,11 +1521,11 @@
<map name="OCN2ATM_SMAPNAME">cpl/gridmaps/gx1v6/map_gx1v6_TO_fv1.9x2.5_aave.130322.nc</map>
</gridmap>
<gridmap atm_grid="1.9x2.5" ocn_grid="gx1v7">
<map name="ATM2OCN_FMAPNAME">cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v7_aave.170518.nc</map>
<map name="ATM2OCN_SMAPNAME">cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v7_blin.170518.nc</map>
<map name="ATM2OCN_VMAPNAME">cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v7_patc.170518.nc</map>
<map name="OCN2ATM_FMAPNAME">cpl/gridmaps/gx1v7/map_gx1v7_TO_fv1.9x2.5_aave.170518.nc</map>
<map name="OCN2ATM_SMAPNAME">cpl/gridmaps/gx1v7/map_gx1v7_TO_fv1.9x2.5_aave.170518.nc</map>
<map name="ATM2OCN_FMAPNAME">cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v7_aave.181205.nc</map>
<map name="ATM2OCN_SMAPNAME">cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v7_blin.181205.nc</map>
<map name="ATM2OCN_VMAPNAME">cpl/gridmaps/fv1.9x2.5/map_fv1.9x2.5_TO_gx1v7_patc.181205.nc</map>
<map name="OCN2ATM_FMAPNAME">cpl/gridmaps/gx1v7/map_gx1v7_TO_fv1.9x2.5_aave.181205.nc</map>
<map name="OCN2ATM_SMAPNAME">cpl/gridmaps/gx1v7/map_gx1v7_TO_fv1.9x2.5_aave.181205.nc</map>
</gridmap>
<gridmap atm_grid="1.9x2.5" ocn_grid="tx1v1">
<map name="ATM2OCN_FMAPNAME">cpl/cpl6/map_fv1.9x2.5_to_tx1v1_aave_da_090710.nc</map>
15 changes: 10 additions & 5 deletions scripts/Tools/Makefile
Original file line number Diff line number Diff line change
@@ -308,6 +308,11 @@ else
EXTRA_PIO_CPPDEFS = -D_NO_MPI_RSEND
endif

ifdef LIB_PNETCDF
CPPDEFS += -D_PNETCDF
SLIBS += -L$(LIB_PNETCDF) -lpnetcdf
endif

ifdef CPRE
FPPDEFS := $(subst $(comma),\\$(comma),$(CPPDEFS))
FPPDEFS := $(patsubst -D%,$(CPRE)%,$(FPPDEFS))
@@ -478,8 +483,11 @@ FFLAGS_NOOPT += $(FPPDEFS)


ifeq ($(findstring -cosp,$(CAM_CONFIG_OPTS)),-cosp)
# The following is for the COSP simulator code:
COSP_LIBDIR:=$(abspath $(EXEROOT)/atm/obj/cosp)
# The following is for the COSP simulator code:
COSP_LIBDIR:=$(abspath $(EXEROOT)/atm/obj/cosp)
ifeq ($(MODEL),driver)
INCLDIR+=-I$(COSP_LIBDIR)
endif
endif

ifeq ($(MODEL),cam)
@@ -524,9 +532,6 @@ ifndef SLIBS
endif
endif

ifdef LIB_PNETCDF
SLIBS += -L$(LIB_PNETCDF) -lpnetcdf
endif
ifdef LAPACK_LIBDIR
SLIBS += -L$(LAPACK_LIBDIR) -llapack -lblas
endif
74 changes: 54 additions & 20 deletions scripts/Tools/archive_metadata
Original file line number Diff line number Diff line change
@@ -37,8 +37,7 @@ _xml_vars = ['CASE', 'COMPILER', 'COMPSET', 'CONTINUE_RUN', 'DOUT_S', 'DOUT_S_RO
'STOP_N', 'STOP_OPTION', 'USER']
_run_vars = ['JOB_QUEUE', 'JOB_WALLCLOCK_TIME', 'PROJECT']
_archive_list = ['Buildconf', 'CaseDocs', 'CaseStatus', 'LockedFiles',
'Macros.make', 'README.case', 'SourceMods', 'software_environment.txt',
'timing', 'logs', 'postprocess/logs']
'Macros.make', 'README.case', 'SourceMods', 'software_environment.txt']
_call_template = Template('in "$function" - Ignoring SVN repo update\n'
'SVN error executing command "$cmd". \n'
'$error: $strerror')
@@ -412,16 +411,20 @@ def get_pp_status(case_dict):
case_dict['timeseries_size'] = 0
case_dict['timeseries_dates'] = '{0}-{1}'.format(case_dict['RUN_STARTDATE'].replace("-", ""),
case_dict['RUN_STARTDATE'].replace("-", ""))
case_dict['timeseries_total_time'] = 0
tseries_logs = list()
tseries_file_pattern = ("{0}/timeseries.log.*".format(pp_log_dir))
tseries_logs = glob.glob(tseries_file_pattern)
case_dict['timeseries_size'] = 0
if tseries_logs:
log_file = max(tseries_logs, key=os.path.getctime)
if is_last_process_complete(filepath=log_file,
expect_text='Successfully completed',
fail_text='opening'):
case_dict['timeseries_status'] = 'Succeeded'
with open(log_file, 'r') as fname:
log_content = fname.readlines()
total_time = [line for line in log_content if 'Total Time:' in line]
case_dict['timeseries_total_time'] = ' '.join(total_time[0].split())
else:
case_dict['timeseries_status'] = 'Started'
sta_dates = case_dict['sta_last_date'].split("-")
@@ -435,7 +438,6 @@ def get_pp_status(case_dict):
case_dict['iconform_status'] = 'Unknown'
case_dict['iconform_path'] = ''
case_dict['iconform_size'] = 0
# TODO - check the date ranges once we have conformed data
case_dict['iconform_dates'] = case_dict['timeseries_dates']

iconform_logs = list()
@@ -453,12 +455,11 @@ def get_pp_status(case_dict):
case_dict['xconform_path'] = ''
case_dict['xconform_path'] = get_pp_path(pp_dir, 'xconform')
if (len(case_dict['xconform_path']) > 2):
case_dict['xconform_path'] = os.path.join(case_dict['xconform_path'], case_dict['case_id'])
case_dict['xconform_path'] = os.path.join(case_dict['xconform_path'], str(case_dict['case_id']))
case_dict['xconform_status'] = 'Unknown'
case_dict['xconform_size'] = get_disk_usage(case_dict['xconform_path'])

# TODO - check the date ranges once we have conformed data
case_dict['xconform_dates'] = case_dict['timeseries_dates']
case_dict['xconform_total_time'] = 0

xconform_logs = list()
xconform_file_pattern = ("{0}/xconform.log.*".format(pp_log_dir))
@@ -470,6 +471,10 @@ def get_pp_status(case_dict):
'cesm_conform_generator INFO')):
case_dict['xconform_status'] = 'Succeeded'
case_dict['xconform_size'] = get_disk_usage(case_dict['xconform_path'])
with open(log_file, 'r') as fname:
log_content = fname.readlines()
total_time = [line for line in log_content if 'Total Time:' in line]
case_dict['xconform_total_time'] = ' '.join(total_time[0].split())
else:
case_dict['xconform_status'] = 'Started'

@@ -839,15 +844,16 @@ def update_repo_add_file(filename, dir1, dir2):
src = os.path.join(dir1, filename)
dest = os.path.join(dir2, filename)
logger.debug('left_only: '+src+' -> '+dest)
shutil.copy2(src, dest)
cmd = ['svn', 'add', dest]
try:
subprocess.check_call(cmd)
except subprocess.CalledProcessError as error:
msg = _call_template.substitute(function='update_lcoal_repo', cmd=cmd,
error=error.returncode, strerror=error.output)
logger.warning(msg)
raise SVNException(msg)
if not os.path.exists(dest):
shutil.copy2(src, dest)
cmd = ['svn', 'add', dest]
try:
subprocess.check_call(cmd)
except subprocess.CalledProcessError as error:
msg = _call_template.substitute(function='update_lcoal_repo', cmd=cmd,
error=error.returncode, strerror=error.output)
logger.warning(msg)
raise SVNException(msg)

# ---------------------------------------------------------------------
def update_repo_rm_file(filename, dir1, dir2):
@@ -908,7 +914,7 @@ def compare_dir_trees(dir1, dir2, archive_list):
new_dir1 = os.path.join(dir1, filename)
new_dir2 = os.path.join(dir2, filename)
os.makedirs(new_dir2)
cmd = ['svn', 'add', '--depth=empty', new_dir2]
cmd = ['svn', 'add', new_dir2]
try:
subprocess.check_call(cmd)
except subprocess.CalledProcessError as error:
@@ -918,8 +924,8 @@ def compare_dir_trees(dir1, dir2, archive_list):
raise SVNException(msg)

# recurse through this new subdir
archive_list = archive_list + filename
compare_dir_trees(new_dir1, new_dir2, archive_list)
new_archive_list = [filename]
compare_dir_trees(new_dir1, new_dir2, new_archive_list)

# files need to be removed from svn repo that are no longer in the caseroot
if right_only:
@@ -973,7 +979,7 @@ def update_local_repo(case_dict, ignore_logs, ignore_timing):

# check if ignore_logs is specified
if ignore_logs:
os.chdir(case_dict['archive_temp_dir'])
os.chdir(to_dir)
if os.path.isdir('./logs'):
try:
shutil.rmtree('./logs')
@@ -1005,6 +1011,24 @@ def update_local_repo(case_dict, ignore_logs, ignore_timing):
error=error.returncode, strerror=error.output)
logger.warning(msg)
raise SVNException(msg)
else:
# add log files
if os.path.exists('{0}/logs'.format(from_dir)):
if not os.path.exists('{0}/logs'.format(to_dir)):
os.makedirs('{0}/logs'.format(to_dir))
os.chdir(os.path.join(from_dir, 'logs'))
for filename in glob.glob('*.*'):
update_repo_add_file(filename, os.path.join(from_dir, 'logs'),
os.path.join(to_dir, 'logs'))

if os.path.exists('{0}/postprocess/logs'.format(from_dir)):
if not os.path.exists('{0}/postprocess/logs'.format(to_dir)):
os.makedirs('{0}/postprocess/logs'.format(to_dir))
os.chdir(os.path.join(from_dir, 'postprocess/logs'))
for filename in glob.glob('*.*'):
update_repo_add_file(filename, os.path.join(from_dir, 'postprocess', 'logs'),
os.path.join(to_dir, 'postprocess', 'logs'))


# check if ignore_timing is specified
if ignore_timing:
@@ -1023,6 +1047,16 @@ def update_local_repo(case_dict, ignore_logs, ignore_timing):
error=error.returncode, strerror=error.output)
logger.warning(msg)
raise SVNException(msg)
else:
# add timing files
if os.path.exists('{0}/timing'.format(from_dir)):
if not os.path.exists('{0}/timing'.format(to_dir)):
os.makedirs('{0}/timing'.format(to_dir))
os.chdir(os.path.join(from_dir, 'timing'))
for filename in glob.glob('*.*'):
update_repo_add_file(filename, os.path.join(from_dir, 'timing'),
os.path.join(to_dir, 'timing'))


# ---------------------------------------------------------------------
def populate_local_repo(case_dict, ignore_logs, ignore_timing):
Loading

0 comments on commit e91b287

Please sign in to comment.