Workflow Example: evaluate and finish calibration work.

Intro

Calibration of dvmdostem involves adjusting a set of parameters so that certain model outputs are as close as possible to our defined target values. Calibration is generally done for each CMT (community type). In addition to the many adjustable parameters for dvmdostem, there are many ways to setup the calibration process, for example choosing a location, choosing settings for the run such as the stages to run, the modules to enable and the number of years to run. This document is an attempt to outline a standard process so that we can perform calibration and calibration evaluations the same way each time. This document does not attempt to explain the details of using the calibration-viewer and adjusting the parameters while the model is running.

Process Overview

1. Select a CMT you wish to calibrate.
2. Select a site (physical location) you wish to calibrate.
3. Select a dvmdostem model version that you wish to calibrate.
4. Setup your working directory and make sure all the appropriate input data files are available, including your target values.
5. Perform the calibration process until you are satisfied with the results.
6. Produce the final evaluation graphs.
7. Bundle the results and submit to the group for review.
8. Pending the review of the group, the new calibration numbers will be comitted to the repository and the new numbers merged to the master branch. This will trigger a change to the middle version number.

1. Select CMT.

This part is pretty easy, especially if the CMT exists. Just pick one!

2. Select a site (physical location) for calibration.

Also pretty easy. Browse the input data catalog and see if you can find a site that matches your needs. If nothing exists, then you may need to create a new set of inputs or ask someone to create them for you.

3. Select a dvmdostem model version to work with.

Also pretty easy, just choose one! And then keep the version of the code you are working on up-to-date. The one thing to be aware of here is that if the model version increments a major or minor number (version numbers follow the pattern: v<major>.<minor>.<patch>) while you are working on your calibration, you will need to make a decision about whether the update will interfere with your calibration work. This will entirely depend on the circumstances of your calibration work, and and the nature of the updates that have been made to dvmdostem.

4. Setup your working directory.

Use the scripts/setup_working_directory.py script that comes with dvmdostem, make sure to pass the path to the input data files you selected above.

If you are working with a new CMT that is not yet incorporated into the main dmvdostem code, then you will need to add the appropriate blocks of data to all the parameter files as well as the calibration_targets.py file. Other items that you might end up changing:

• Using outspec_utils.py to setup netcdf outputs.
• Using runmask_util.py to set the pixel used for the calibration.
• Modifying config/config.js for some reason?

5. Perform the calibration process.

Details not included in this document. See <page that we have not written yet>.

6. Produce the final evaluation graphs.

When you are finally done with the calibration process and are happy with the numbers you have achieved, then run the model one last time, from start to finish, with your new numbers in place in the parameter files.

Note the following examples assume the following: you have the dvm-dos-tem repo in your home directory; the working directory you setup above is also in your home directory, and named "example-cal-work"; you have the input catalog on your machine under /data/. Adjust your own paths accordingly.

(/home/user)$ cd ~/example-cal-work
(/home/user/example-cal-work)$ ~/dvmdostem \
--log-level info \
--cal-mode --force-cmt <YOUR CMTNUM!> \
--tar-caljson \
--pr-yrs 100 \
--eq-yrs 1000 \
--sp-yrs 250 \
--tr-yrs 85 \
--sc-yrs 115

When the run is done you should have in your output/ folder one *-data.tar.gz file for each run-stage containing the calibration json files along with whatever netcdf outputs were enabled.

Now create the plots that summarize the calibration result:

(/home/user/example-cal-work)$ ~/dvm-dos-tem/calibration/calibration-viewer.py \
--from-archive /home/user/example-cal-work/output/eq-data.tar.gz \
--archive-series sp-data.tar.gz tr-data.tar.gz \
--help # WHAT OPTIONS ARE BEST HERE???
       # Are the above options how folks have been running this?
       # Maybe we can define a custom calibration suite that has only
       # the plots we are interested in? User could always run
       # the dynamic viewer over the archive later if they want to see more
       # detail?
       # Do we need to call this in a loop for all the PFTS?

Proposed steps once qcal branch is merged

Make a place to work:

(/home/user) $ ./dvm-dos-tem/scripts/setup_working_directory.py \
   --input-data-path /data/mirrors/atlas/dvmdostem-input-catalog/cru-ts40_ar5_rcp85_ncar ccsm4_kougarok_hillslope_10x10/ \
   kg-shrub
(/home/user/kg-shrub) $ cd kg-shrub/

Enable all the calibration specific output variables:

(/home/user/kg-shrub) $ ~/dvm-dos-tem/scripts/outspec_utils.py --enable-cal-vars config/output_spec.csv 

Turn on the eq stage netcdf output

(/home/user/kg-shrub) $ vim config/config.js 
# Edit so as to produce eq stage output

Turn on pixel of interest in the run-mask:

(/home/user/kg-shrub) $ ~/dvm-dos-tem/scripts/runmask-util.py --reset --yx 0 0 run-mask.nc 

Run the model

(/home/user/kg-shrub) $ ~/dvm-dos-tem/dvmdostem -c --tar-caljson -p 100 -e 1000 -s 250 -t 85 -n 115 --force-cmt 4 -l warn

Run the Qcal to get a metric:

(/home/user/kg-shrub) $ ../dvm-dos-tem/scripts/qcal.py \
  --ref-params parameters/ \
  --ref-targets calibration/ \
  output/

Loading calibration_targets from : ['/tmp/dvmdostem-user-1003-tmp-cal']
Cleaning up temporary targets and __init__.py file used for import...
Resetting path...
        modeled data: output/eq-data.tar.gz
        targets file: calibration/
     parameter files: parameters/
                 CMT: set([u'CMT04'])
                 QCR: 1003.33614482
        Weighted QCR: 4.63824022755

Run the viewer

(/home/user/kg-shrub) $ ../dvm-dos-tem/calibration/calibration-viewer.py \
  --no-blit \
  --from-archive output/eq-data.tar.gz \
  --archive-series output/sp-data.tar.gz output/tr-data.tar.gz \
  --suite Calibration --targets \
  --ref-targets calibration/

7. Bundle the results and submit to the group for review.

NEED TO DEVISE THIS STEP.

• Tar the entire working directory?
• Is there more metadata that needs to be included?
• Do we want a way to bundle just the plots so that we don't have to send around the heaps of output data?

8. Pending review, commit to repo and merge to master.