Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate @bekozi's ocgis tool in mkmapdata.sh #815

Closed
wants to merge 14 commits into from
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 75 additions & 39 deletions bld/namelist_files/namelist_defaults_ctsm_tools.xml
Original file line number Diff line number Diff line change
Expand Up @@ -55,48 +55,19 @@ attributes from the config_cache.xml file (with keys converted to upper-case).
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.1x0.1_nomask_c110712.nc</scripgriddata>
<scripgriddata hgrid="0.5x0.5" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.5x0.5_nomask_c110308.nc</scripgriddata>
<scripgriddata hgrid="0.5x0.5" lmask="AVHRR"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.5x0.5_AVHRR_c110228.nc</scripgriddata>
<scripgriddata hgrid="0.5x0.5" lmask="MODIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.5x0.5_MODIS_c110228.nc</scripgriddata>
<scripgriddata hgrid="0.25x0.25" lmask="MODIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.25x0.25_MODIS_c170321.nc</scripgriddata>
<scripgriddata hgrid="0.25x0.25" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.25x0.25_nomask_c191014.nc</scripgriddata>
<scripgriddata hgrid="5x5min" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_nomask_c110530.nc</scripgriddata>
<scripgriddata hgrid="5x5min" lmask="IGBP-GSDP"
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_IGBP-GSDP_c110228.nc</scripgriddata>
<scripgriddata hgrid="5x5min" lmask="ISRIC-WISE"
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_ISRIC-WISE_c111114.nc</scripgriddata>
<scripgriddata hgrid="5x5min" lmask="ORNL-Soil"
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_ORNL-Soil_c170630.nc</scripgriddata>
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_nomask_c191107.nc</scripgriddata>
<scripgriddata hgrid="10x10min" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_10x10min_nomask_c110228.nc</scripgriddata>
<scripgriddata hgrid="10x10min" lmask="IGBPmergeICESatGIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_10x10min_IGBPmergeICESatGIS_c110818.nc</scripgriddata>
<scripgriddata hgrid="3x3min" lmask="MODIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_MODIS_c110915.nc</scripgriddata>
<scripgriddata hgrid="3x3min" lmask="MODIS-wCsp"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_MODISwcspsea_c151020.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="MODIS" >64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="MODIS-wCsp">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="3x3min" lmask="USGS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3x3_USGS_c120912.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="USGS">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="3x3min" lmask="LandScan2004"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_LandScan2004_c120517.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="LandScan2004">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="3x3min" lmask="GLOBE-Gardner"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_GLOBE-Gardner_c120922.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="GLOBE-Gardner">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="3x3min" lmask="GLOBE-Gardner-mergeGIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_GLOBE-Gardner-mergeGIS_c120922.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="GLOBE-Gardner-mergeGIS">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="360x720cru" lmask="cruncep"
>lnd/clm2/mappingdata/grids/SCRIPgrid_360x720_cruncep_c120830.nc</scripgriddata>
<scripgriddata hgrid="3x3min" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3x3min_nomask_c191102.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="nomask">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="360x720cru" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_360x720_nomask_c120830.nc</scripgriddata>
<scripgriddata hgrid="0.9x1.25" lmask="GRDC"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.9x1.25_GRDC_c130307.nc</scripgriddata>
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.5x0.5_nomask_c110308.nc</scripgriddata>
<!--scripgriddata hgrid="0.9x1.25" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.9x1.25_nomask_c191014.nc</scripgriddata-->

<!-- Other raw data grids -->
<!-- Note that the following is a UGRID file rather than a SCRIP grid file -->
Expand All @@ -118,6 +89,45 @@ attributes from the config_cache.xml file (with keys converted to upper-case).
<scripgriddata hgrid="1x1_vancouverCAN" lmask="nomask" >lnd/clm2/mappingdata/grids/SCRIPgrid_1x1pt_vancouverCAN_nomask_c110308.nc</scripgriddata>
<scripgriddata hgrid="5x5_amazon" lmask="nomask" >lnd/clm2/mappingdata/grids/SCRIPgrid_5x5pt_amazon_nomask_c110308.nc</scripgriddata>

<max_res hgrid="1km-merge-10min" lmask="HYDRO1K-merge-nomask">0.01</max_res>
<max_res hgrid="0.23x0.31" lmask="nomask">0.23</max_res>
<max_res hgrid="0.47x0.63" lmask="nomask">0.47</max_res>
<max_res hgrid="0.9x1.25" lmask="nomask">0.9</max_res>
<max_res hgrid="1.9x2.5" lmask="nomask">1.9</max_res>
<max_res hgrid="2.5x3.33" lmask="nomask">2.5</max_res>
<max_res hgrid="4x5" lmask="nomask">4</max_res>
<max_res hgrid="10x15" lmask="nomask">10</max_res>
<max_res hgrid="512x1024" lmask="nomask">0.35</max_res>
<max_res hgrid="128x256" lmask="nomask">1.4</max_res>
<max_res hgrid="94x192" lmask="nomask">1.9</max_res>
<max_res hgrid="64x128" lmask="nomask">2.8</max_res>
<max_res hgrid="48x96" lmask="nomask">3.75</max_res>
<max_res hgrid="ne240np4" lmask="nomask">0.1</max_res>
<max_res hgrid="ne120np4" lmask="nomask">0.1</max_res>
<max_res hgrid="ne60np4" lmask="nomask">0.1</max_res>
<max_res hgrid="ne30np4" lmask="nomask">0.1</max_res>
<max_res hgrid="ne16np4" lmask="nomask">0.1</max_res>
<max_res hgrid="ne4np4" lmask="nomask">0.1</max_res>
<max_res hgrid="0.33x0.33" lmask="nomask">0.33</max_res>
<max_res hgrid="0.1x0.1" lmask="nomask">0.1</max_res>
<max_res hgrid="0.5x0.5" lmask="nomask">0.5</max_res>
<max_res hgrid="0.25x0.25" lmask="nomask">0.25</max_res>
<max_res hgrid="0.125x0.125" lmask="nomask">0.125</max_res>
<max_res hgrid="5x5min" lmask="nomask">0.08</max_res>
<max_res hgrid="10x10min" lmask="nomask">0.16</max_res>
<max_res hgrid="3x3min" lmask="nomask">0.05</max_res>
<max_res hgrid="360x720cru" lmask="nomask">0.5</max_res>
<max_res hgrid="0.125nldas2" lmask="nomask">0.125</max_res>
<max_res hgrid="1x1_asphaltjungleNJ" lmask="nomask">2</max_res>
<max_res hgrid="1x1_brazil" lmask="nomask">2</max_res>
<max_res hgrid="1x1_camdenNJ" lmask="nomask">2</max_res>
<max_res hgrid="1x1_mexicocityMEX" lmask="nomask">1</max_res>
<max_res hgrid="1x1_numaIA" lmask="nomask">0.5</max_res>
<max_res hgrid="1x1_smallvilleIA" lmask="nomask">0.5</max_res>
<max_res hgrid="1x1_urbanc_alpha" lmask="nomask">1</max_res>
<max_res hgrid="1x1_vancouverCAN" lmask="nomask">1</max_res>
<max_res hgrid="5x5_amazon" lmask="nomask">2.2</max_res>

<!-- Ocean grids -->
<scripgriddata hgrid="0.33x0.33" lmask="0.33x0.33" >lnd/clm2/mappingdata/grids/SCRIPgrid_0.33x0.33_navy_c111207.nc</scripgriddata>
<scripgriddata hgrid="5x5_amazon" lmask="navy" >lnd/clm2/mappingdata/grids/SCRIPgrid_5x5_amazon_navy_c111207.nc</scripgriddata>
Expand All @@ -126,6 +136,32 @@ attributes from the config_cache.xml file (with keys converted to upper-case).
<scripgriddata hgrid="tx1v1" lmask="tx1v1" >/glade/proj3/cseg/mapping/grids/tx1v1_090122.nc</scripgriddata>
<scripgriddata hgrid="tx0.1v2" lmask="tx0.1v2" >/glade/proj3/cseg/mapping/grids/tx0.1v2_090127.nc</scripgriddata>

<!-- UNSTRUCT grid data files -->
<unstructdata hgrid="0.5x0.5" lmask="nomask"
>lnd/clm2/mappingdata/grids/UNSTRUCTgrid_0.5x0.5_nomask_c191114.nc</unstructdata>
<unstructdata_type hgrid="0.5x0.5" lmask="nomask">ESMFMESH</unstructdata_type>
<unstructdata hgrid="0.25x0.25" lmask="nomask"
>lnd/clm2/mappingdata/grids/UNSTRUCTgrid_0.25x0.25_nomask_c191114.nc</unstructdata>
<unstructdata_type hgrid="0.25x0.25" lmask="nomask">ESMFMESH</unstructdata_type>
<unstructdata hgrid="0.125x0.125" lmask="nomask"
>lnd/clm2/mappingdata/grids/UNSTRUCTgrid_0.125x0.125_nomask_c191114.nc</unstructdata>
<unstructdata_type hgrid="0.125x0.125" lmask="nomask">ESMFMESH</unstructdata_type>
<unstructdata hgrid="10x10min" lmask="nomask"
>lnd/clm2/mappingdata/grids/UNSTRUCTgrid_10x10min_nomask_c191114.nc</unstructdata>
<unstructdata_type hgrid="10x10min" lmask="nomask">ESMFMESH</unstructdata_type>
<unstructdata hgrid="5x5min" lmask="nomask"
>lnd/clm2/mappingdata/grids/UNSTRUCTgrid_5x5min_nomask_c191114.nc</unstructdata>
<unstructdata_type hgrid="5x5min" lmask="nomask">ESMFMESH</unstructdata_type>
<unstructdata hgrid="3x3min" lmask="nomask"
>lnd/clm2/mappingdata/grids/UNSTRUCTgrid_3x3min_nomask_c191114.nc</unstructdata>
<unstructdata_type hgrid="3x3min" lmask="nomask">ESMFMESH</unstructdata_type>
<unstructdata hgrid="0.9x1.25" lmask="nomask"
>lnd/clm2/mappingdata/grids/UNSTRUCTgrid_0.9x1.25_nomask_c191114.nc</unstructdata>
<unstructdata_type hgrid="0.9x1.25" lmask="nomask">ESMFMESH</unstructdata_type>
<unstructdata hgrid="1km-merge-10min" lmask="HYDRO1K-merge-nomask"
>lnd/clm2/mappingdata/grids/UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc</unstructdata>
<unstructdata_type hgrid="1km-merge-10min" lmask="HYDRO1K-merge-nomask">UGRID</unstructdata_type>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept the above as placeholders. I'm ok with removing if that's preferred.

<!-- land masks for different types of mksurfdata_map input files -->
<lmask type="lak" >MODIS-wCsp</lmask>
<lmask type="wet" >AVHRR</lmask>
Expand Down Expand Up @@ -286,7 +322,7 @@ attributes from the config_cache.xml file (with keys converted to upper-case).
>lnd/clm2/rawdata/mksrf_vic_0.9x1.25_GRDC_simyr2000.c130307.nc</mksrf_fvic>

<mksrf_fch4 hgrid="360x720cru" lmask="cruncep"
>lnd/clm2/rawdata/mksrf_ch4inversion_360x720_cruncep_simyr2000.c130322.nc</mksrf_fch4>
>lnd/clm2/rawdata/mksrf_ch4inversion_0.5x0.5_cruncep_simyr2000.c191112.nc</mksrf_fch4>

<!-- mksrf_fvegtyp -->

Expand Down
19 changes: 18 additions & 1 deletion bld/namelist_files/namelist_definition_ctsm.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1159,6 +1159,11 @@ Toggle to turn on on diagnostic Snow Radiative Effect
SCRIP format grid data file
</entry>

<entry id="unstructdata" type="char*256" category="mkmapdata"
input_pathname="abs" group="clmexp" valid_values="" >
UNSTRUCT format grid data file
</entry>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Placeholder. I'm ok with removing if that's preferred.

<entry id="scripgriddata_lrgfile_needed" type="char*256" category="mkmapdata"
group="clmexp" valid_values="none,64bit_offset,netcdf4" >
Flag to pass to the ESMF mapping utility, telling it what kind of large
Expand All @@ -1172,13 +1177,25 @@ Flag to pass to the ESMF mapping utility, telling it what kind of grid
file this is (SCRIP or UGRID).
</entry>

<entry id="unstructdata_type" type="char*256" category="mkmapdata"
group="clmexp" valid_values="ESMFMESH" >
Flag to pass to the ESMF mapping utility, telling it what kind of grid
file this is.
</entry>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Placeholder. I'm ok with removing if that's preferred.

<entry id="scripgriddata_meshname" type="char*256" category="mkmapdata"
group="clmexp" valid_values="" >
For UGRID files, flag to pass to the ESMF mapping utility, telling it the
name of the dummy variable that has all of the topology information stored
in its attributes. (Only used if scripgriddata_src_type = UGRID.)
</entry>

<entry id="max_res" type="real" category="mkmapdata"
group="clmexp" valid_values="" >
Maximum resolution of source and destination files required by the ocgis
tool.
</entry>

<!-- -->
<!-- mksurfdata namelist -->
<!-- -->
Expand Down Expand Up @@ -1849,7 +1866,7 @@ CLM run type.
<entry id="res" type="char*30" category="default_settings"
group="default_settings"
valid_values=
"512x1024,360x720cru,128x256,64x128,48x96,32x64,8x16,94x192,0.23x0.31,0.47x0.63,0.9x1.25,1.9x2.5,2.5x3.33,4x5,10x15,0.125nldas2,5x5_amazon,1x1_camdenNJ,1x1_vancouverCAN,1x1_mexicocityMEX,1x1_asphaltjungleNJ,1x1_brazil,1x1_urbanc_alpha,1x1_numaIA,1x1_smallvilleIA,0.1x0.1,0.25x0.25,0.5x0.5,3x3min,5x5min,10x10min,0.33x0.33,0.125x0.125,ne4np4,ne16np4,ne30np4,ne60np4,ne120np4,ne240np4,1km-merge-10min">
"512x1024,360x720cru,128x256,64x128,48x96,94x192,0.23x0.31,0.47x0.63,0.9x1.25,1.9x2.5,2.5x3.33,4x5,10x15,0.125nldas2,5x5_amazon,1x1_camdenNJ,1x1_vancouverCAN,1x1_mexicocityMEX,1x1_asphaltjungleNJ,1x1_brazil,1x1_urbanc_alpha,1x1_numaIA,1x1_smallvilleIA,0.1x0.1,0.25x0.25,0.5x0.5,3x3min,5x5min,10x10min,0.33x0.33,0.125x0.125,ne4np4,ne16np4,ne30np4,ne60np4,ne120np4,ne240np4,1km-merge-10min">
Horizontal resolutions
Note: 0.1x0.1, 0.25x0.25, 0.5x0.5, 5x5min, 10x10min, 3x3min and 0.33x0.33 are only used for CLM tools
</entry>
Expand Down
137 changes: 137 additions & 0 deletions doc/ChangeLog
Original file line number Diff line number Diff line change
@@ -1,4 +1,141 @@
===============================================================
Tag name: ctsm1.0.dev081
Originator(s): slevis (Samuel Levis,SLevis Consulting LLC,303-665-1310)
Date: Mon Nov 18 17:41:44 MST 2019
One-line Summary: Integrate ocgis tool in mkmapdata.sh

Purpose of changes
------------------

Modify mkmapdata.sh to run with @bekozi's ocgis tool because it allows
for:
- chunking of the data while regridding to reduce memory requirements
- subsetting of the source files when regridding to regional
destination files


Bugs fixed or introduced
------------------------

Issues fixed (include CTSM Issue #): #643 #648 #806 others?
CIME Issues fixed (include issue #): [If none, remove this line]

Known bugs introduced in this tag (include github issue ID): [If none, remove this line]

Known bugs found since the previous tag (include github issue ID): [If none, remove this line]


Significant changes to scientifically-supported configurations
--------------------------------------------------------------

Does this tag change answers significantly for any of the following physics configurations?
(Details of any changes will be given in the "Answer changes" section below.)

[Put an [X] in the box for any configuration with significant answer changes.]

[ ] clm5_0

[ ] ctsm5_0-nwp

[ ] clm4_5

Notes of particular relevance for users
---------------------------------------

Caveats for users (e.g., need to interpolate initial conditions):
None once #645 is resolved.

Changes to CTSM's user interface (e.g., new/renamed XML or namelist variables):
None.

Changes made to namelist defaults (e.g., changed parameter values):
Changes needed to accommodate UNSTRUCT grid source files.

Changes to the datasets (e.g., parameter, surface or initial files):
Switch from using SCRIP to UNSTRUCT grid source files when running
mkmapdata.sh

Substantial timing or memory changes:
More efficient and faster mkmapdata.sh:

Notes of particular relevance for developers: (including Code reviews and testing)
---------------------------------------------
NOTE: Be sure to review the steps in README.CHECKLIST.master_tags as well as the coding style in the Developers Guide

Caveats for developers (e.g., code that is duplicated that requires double maintenance):

Changes to tests or testing:

Code reviewed by:
@ekluzek @billsacks @negin513 @bekozi

CTSM testing:

[... Remove before making master tag. Available test levels:

a) regular (must be run before handing off a tag to SEs and must be run
before committing a tag)
b) build_namelist (if namelists and/or build_system changed))
c) tools (only if tools are modified and no CTSM source is modified)
d) short (for use during development and in rare cases where only a small
change with known behavior is added ... eg. a minor bug fix)
e) doc (no source testing required)

... ]

[PASS means all tests PASS and OK means tests PASS other than expected fails.]

build-namelist tests:

cheyenne -

tools-tests (test/tools):

cheyenne -

PTCLM testing (tools/shared/PTCLM/test):

cheyenne -

python testing (see instructions in python/README.md; document testing done):

(any machine) -

regular tests (aux_clm):

cheyenne ----
izumi -------

If the tag used for baseline comparisons was NOT the previous tag, note that here:


Answer changes
--------------

Changes answers relative to baseline:

If a tag changes answers relative to baseline comparison the
following should be filled in (otherwise remove this section):

Summarize any changes to answers, i.e.,
- what code configurations: ALL
- what platforms/compilers: ALL
- nature of change: roundoff; larger than roundoff/same climate

If bitwise differences were observed, how did you show they were no worse
than roundoff?


Detailed list of changes
------------------------

List any externals directories updated (cime, rtm, mosart, cism, fates, etc.):

Pull Requests that document the changes (include PR ids):
https://github.com/ESCOMP/CTSM/pull/815

===============================================================
===============================================================
Tag name: ctsm1.0.dev079
Originator(s): sacks (Bill Sacks)
Date: Mon Nov 4 14:41:41 MST 2019
Expand Down
1 change: 1 addition & 0 deletions doc/ChangeSum
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
Tag Who Date Summary
============================================================================================================================
ctsm1.0.dev081 slevis 11/18/2019 Integrate ocgis tool in mkmapdata.sh
ctsm1.0.dev079 sacks 11/04/2019 Change a few uses of shr_kind
ctsm1.0.dev078 oleson 10/31/2019 Fix rootr calculation with use_hydrstress true
ctsm1.0.dev077 sacks 10/27/2019 Consistently use frac_sno_eff rather than frac_sno in qg calculations
Expand Down
Loading