Merge branch 'jgfouca/hist_tools_conv_to_python' (PR #413)

Upgrade history tools to python

Full change list:
1) Remove old shell-based tools and update calls to use python versions
2) Move most functionality in old shell tools into hist_utils.py
3) Make thin python wrapper programs to access hist_utils from the command line
4) Do st_archive as LAST step in run_indv so that coupler_log_path is not needed
5) Fix ERR test
6) Update fake tests to create a fake hist file
7) Large refactor of bless_test_results
8) Add new compare_test_results, counterpart to bless_test_results.

Test suite: scripts_regression_tests
Test baseline:
Test namelist changes:
Test status: bit for bit

Fixes #332

User interface changes?: Significant changes to compare_* scripts

Code review: @jedwards4b @mvertens @billsacks @gold2718

* jgfouca/hist_tools_conv_to_python:
  Make comparison matchups more robust
  Fix user docs for compare_test_results
  improved reporting of baseline file count mismatch
  correct location of debug log in help message, store baselines with original filename
  Add usage example for typical CESM workflow
  Get rid of pdb trace that I believe was mistakenly left in
  Make a very obvious simplification to code
  Remove unneeded global
  Update hist infra to better-support user-chosen baseline_root
  minor help string fix
  More fixes from review
  fix issue in component_generate_baseline, get only most recent files
  Remove last cwd default args
  Remove dangerous cwd defaults, add documentation to hist_utils public API
  Add new compare_test_results, counterpart to bless_test_results
  bless_test_results: Need sane error code
  remove check for None
  fixes in hist_utils
  Fix mistake caught by code review
  Upgrade history tools to python

Conflicts:
	utils/python/CIME/check_lockedfiles.py
	utils/python/CIME/test_status.py

scripts/Testing/Testcases/NOC_script

@@ -75,10 +75,10 @@ else
 endif
 
 echo "moving relevant history files to suffix with command " >>& $TESTSTATUS_LOG
-echo "$SCRIPTSROOT/Tools/component_compare_move.sh -rundir $RUNDIR -testcase $CASE -suffix base" >>& $TESTSTATUS_LOG
+echo "$SCRIPTSROOT/Tools/component_compare_move base" >>& $TESTSTATUS_LOG
 echo "" >>& $TESTSTATUS_LOG
 
-$SCRIPTSROOT/Tools/component_compare_move.sh -rundir $RUNDIR -testcase $CASE -suffix "base"
+$SCRIPTSROOT/Tools/component_compare_move base
 
 #======================================================================
 # do an initial run test with NINST 2, with the
@@ -117,10 +117,10 @@ else
 endif
 
 echo "moving relevant history files to suffix with commands " >>& $TESTSTATUS_LOG
-echo "$SCRIPTSROOT/Tools/component_compare_move.sh -rundir $RUNDIR -testcase $CASE -suffix inst2mod"  >>& $TESTSTATUS_LOG
+echo "$SCRIPTSROOT/Tools/component_compare_move inst2mod"  >>& $TESTSTATUS_LOG
 echo "" >>& $TESTSTATUS_LOG
 
-$SCRIPTSROOT/Tools/component_compare_move.sh -rundir $RUNDIR -testcase $CASE -suffix "inst2mod"
+$SCRIPTSROOT/Tools/component_compare_move inst2mod
 
 #======================================================================
 # Check test status for all relevant component history files
@@ -130,10 +130,7 @@ echo "DONE ${CASEBASEID} : (test finished, successful coupler log) " >>& $TESTST
 echo "" >>& $TESTSTATUS_LOG
 
 echo "obtaining test functionality from history files with commands " >>& $TESTSTATUS_LOG
-echo "$SCRIPTSROOT/Tools/component_compare_test.sh -rundir $RUNDIR -testcase $CASE -testcase_base $CASEBASEID -suffix1 base -suffix2 multiinst -msg compare .base with .inst2mod" >>& $TESTSTATUS_LOG
+echo "$SCRIPTSROOT/Tools/component_compare_test base multiinst" >>& $TESTSTATUS_LOG
 echo "" >>& $TESTSTATUS_LOG
 
-$SCRIPTSROOT/Tools/component_compare_test.sh -rundir $RUNDIR -testcase $CASE -testcase_base $CASEBASEID -suffix1 "base" -suffix2 "multiinst" -msg "compare .base with .inst2mod" >>& $TESTSTATUS_OUT
-
-
-
+$SCRIPTSROOT/Tools/component_compare_test base multiinst >>& $TESTSTATUS_OUT

scripts/Tools/bless_test_results

@@ -6,8 +6,7 @@ changes, and updating baselines. Purpose is, instead of re-running tests
 in generate mode, which is very slow, allow for very fast analsis and
 blessing of diffs.
 
-Be aware that restart test will overwrite the original namelist files
-with versions of the files that you should not bless.
+You may need to load modules for cprnc to work.
 """
 
 from standard_script_setup import *
@@ -49,6 +48,7 @@ formatter_class=argparse.ArgumentDefaultsHelpFormatter
 )
 
     default_baseline_name = CIME.utils.get_current_branch(repo=CIME.utils.get_cime_root())
+    default_baseline_root = _MACHINE.get_value("CCSM_BASELINE")
     default_compiler      = _MACHINE.get_default_compiler()
     scratch_root          = _MACHINE.get_value("CESMSCRATCHROOT")
     default_testroot      = os.path.join(scratch_root)
@@ -64,6 +64,9 @@ formatter_class=argparse.ArgumentDefaultsHelpFormatter
     parser.add_argument("-b", "--baseline-name", default=default_baseline_name,
                         help="Name of baselines to use, corresponds to branch used.")
 
+    parser.add_argument("--baseline-root", default=default_baseline_root,
+                        help="Root of baselines.")
+
     parser.add_argument("-c", "--compiler", default=default_compiler,
                         help="Compiler of run you want to bless")
 
@@ -91,7 +94,7 @@ formatter_class=argparse.ArgumentDefaultsHelpFormatter
     expect(not (args.namelists_only and args.hist_only),
            "Makes no sense to use --namelists-only and --hist-only simultaneously")
 
-    return args.baseline_name, args.test_root, args.compiler, args.test_id, args.namelists_only, args.hist_only, args.report_only, args.force, args.bless_tests
+    return args.baseline_name, args.baseline_root, args.test_root, args.compiler, args.test_id, args.namelists_only, args.hist_only, args.report_only, args.force, args.bless_tests
 
 ###############################################################################
 def _main_func(description):
@@ -100,10 +103,11 @@ def _main_func(description):
         test_results = doctest.testmod(verbose=True)
         sys.exit(1 if test_results.failed > 0 else 0)
 
-    baseline_name, test_root, compiler, test_id, namelists_only, hist_only, report_only, force, bless_tests = \
+    baseline_name, baseline_root, test_root, compiler, test_id, namelists_only, hist_only, report_only, force, bless_tests = \
         parse_command_line(sys.argv, description)
 
-    CIME.bless_test_results.bless_test_results(baseline_name, test_root, compiler, test_id, namelists_only, hist_only, report_only, force, bless_tests)
+    success = CIME.bless_test_results.bless_test_results(baseline_name, baseline_root, test_root, compiler, test_id, namelists_only, hist_only, report_only, force, bless_tests)
+    sys.exit(0 if success else 1)
 
 ###############################################################################