Now allow specification of the number of cores for benchmark plotting jobs

[yantosca]

Description

This PR does the following:

1. Adds the options.n_cores YAML tag to benchmark YAML configuration files, in the benchmark/config and benchmark/cloud folders, e.g.:

#
# options: Customizes the benchmark plot output.
#
options:
  #
  # bmk_type: Specifies the type of benchmark.
  #
  bmk_type: FullChemBenchmark
  #
  # comparisons: Specifies the comparisons to perform.
  #
  comparisons:
  ... etc not shown ...
  #
  # outputs: Specifies the plots and tables to generate.
  #
  outputs:
  ... etc not shown ...
  #
  # n_cores: Specify the number of cores to use:
  # -1: Use as many cores as possible
  #  N: Use N cores
  #  1: Disable parallelization (use a single core)
  #
  n_cores: -1

2. Reads options.n_cores into the config["options"]["n_cores"] dict entry.

3. Passes n_job=config["options"]["n_cores"] to the relevant benchmark plotting routines in the following modules:

• benchmark/run_benchmark.py
• benchmark/modules/run_1yr_fullchem_benchmark,py
• benchmark/modules/run_1yr_tt_benchmark.py

Related issues:

• Closes [FEATURE REQUEST] Add switch to quickly disable parallelization in gcpy #179

[yantosca]

@laestrada and I have discovered that the joblib Parallel function can potentially mask errors in other areas of code. For this reason, I propose to rewrite the code so that we use an if block to execute without parallelization if n_job=1, or to call the joblib Parallel function otherwise. Stay tuned.

[yantosca]

We have now added if blocks around the calls to joblib.Parallel() so that separate, non-parallelized code will execute when n_job==1, such as shown in the example below:

        # ==================================================================
        # GCC vs GCC operations budgets tables
        # ==================================================================
        if config["options"]["outputs"]["ops_budget_table"]:
            print("\n%%% Creating GCC vs. GCC operations budget tables %%%")

            def gcc_vs_gcc_ops_budg(mon):
                """
                Create budget table for each benchmark month m in parallel
                """

                # Filepaths
                refpath = get_filepath(
                    gcc_vs_gcc_refdir,
                    "Budget",
                    bmk_mons_ref[mon]
                )
                devpath = get_filepath(
                    gcc_vs_gcc_devdir,
                    "Budget",
                    bmk_mons_dev[mon]
                )

                # Create tables
                bmk.make_benchmark_operations_budget(
                    config["data"]["ref"]["gcc"]["version"],
                    refpath,
                    config["data"]["dev"]["gcc"]["version"],
                    devpath,
                    sec_per_month_ref[mon],
                    sec_per_month_dev[mon],
                    benchmark_type=bmk_type,
                    label=f"at 01{bmk_mon_yr_strs_dev[mon]}",
                    dst=gcc_vs_gcc_tablesdir,
                )

            # Create tables in parallel
            # Turn off parallelization if n_jobs==1
            if n_jobs != 1:
                results = Parallel(n_jobs=config["options"]["n_cores"])(
                    delayed(gcc_vs_gcc_ops_budg)(mon) \
                    for mon in range(bmk_n_months)
                )
            else:
                for mon in range(bmk_n_months):
                    results = gcc_vs_gcc_ops_budg(mon)

@laestrada and I have found that joblib.Parallel() often obfuscates error messages from other Python packages. For the sake of easy debugging, we no longer rely on joblib.Parallel(n_jobs=1) to disable parallelization.

[yantosca]

This PR should be merged before PR #250.