Integrate @bekozi's ocgis tool in mkmapdata.sh #815

slevis-lmwg · 2019-10-05T01:30:12Z

Description of changes

Modify mkmapdata.sh to run with @bekozi's ocgis tool because it allows for

chunking of the data while regridding to reduce memory requirements and
subsetting of the source files when regridding to regional destination files

Specific notes

Contributors other than yourself, if any:
@bekozi @negin513 @ekluzek @billsacks

CTSM Issues Fixed (include github issue #):
Begins to address #643 #644
Fixes #806
I could see it possibly fixing #648

Are answers expected to change (and if so in what way)?
I expect roundoff differences in output weight (aka map) files from testing this far.
I use roundoff differences as a minimum requirement for my ocgis tests to pass.

Any User Interface Changes (namelist or namelist defaults changes)?
None once #645 is resolved

Testing performed, if any:

Manual testing, i.e. run default mkmapdata.sh and new mkmapdata.sh and compare the S variable in the weight files. For regional destination files, must also compare the i and j index vectors.
Testing will become simpler (fewer weight files to compare) once For mksurfdata_map: use maps with no source masking, applying mask separately #286 is resolved

Add config_archive.xml to cime_config for CTSM. Update the CODE_OF_CONDUCT to the August/2019 UCAR version. No testing done.

@ekluzek

With initial help from @ekluzek on 9/27/2019 and additional modifications made today, I am able to run @bekozi's ocgis tool by running a new version of mkmapdata.sh that I'm committing here.

billsacks · 2019-10-06T15:14:34Z

@slevisconsulting looking at your PR comment, it looks like GitHub is trying to be too clever, and thinks that this PR resolves some issues that aren't really resolved. It looks like this is truly supposed to close #806, but the way you reference #645 and #286 makes GitHub think that this PR closes those two issues, too. Can you please reword your PR comment to avoid this? For example, maybe use passive voice (e.g., "once #645 is resolved").

slevis-lmwg · 2019-11-13T02:45:03Z

@bekozi I've reached the point where I need the weight files from ocgis to contain the same metadata as NCAR's weight files in order to test running them through our script mksurfdata_map

If you look in one of the NCAR weight files that I've shared with you in the past, you'll see what I mean. Pls let me know if you need anything from me. Thanks!

bekozi · 2019-11-13T14:37:09Z

@slevisconsulting Would you mind pointing me towards one (Cheyenne is fine)? I just want to make sure we are on the same page.

slevis-lmwg · 2019-11-13T15:51:15Z

@bekozi some of the data in the NCAR weight files is redundant now, so I will first confirm what's really needed and then point you to a file.

slevis-lmwg · 2019-11-13T16:49:09Z

@bekozi I take it back. Removing things that I thought might be redundant caused me problems, so let's keep everything. Here's a weight file with everything in it:
/glade/p/cesm/cseg/inputdata/lnd/clm2/mappingdata/maps/0.9x1.25/map_0.25x0.25_nomask_to_0.9x1.25_nomask_aave_da_c191016.nc

bekozi · 2019-11-13T18:32:10Z

Got it. You only need the global attributes added, correct?

slevis-lmwg · 2019-11-13T20:53:01Z

As far as I can tell, the code that makes the surface datasets needs all the variables and dimensions that you see in there, where

yc and xc are the lats and lons at grid cell centers
yv and xv are the lats and lons at grid cell vertices
suffixes a and b refer to the SRC and DST resolutions

In terms of the global attributes, some now already make it in my ocgis-made weight files. See here:
/glade/work/slevis/git/mksurfdata_maps_wo_src_masks/tools/mkmapdata/map_files_ocgis/0.25x0.25/map_0.25x0.25_nomask_to_0.9x1.25_nomask_aave_da_c191111.nc
I do not know if we need the other global attributes for the code to run...

Change a few uses of shr_kind Change a few uses of shr_kind to work with latest cime. These changes are from Mariana Vertenstein.

bekozi · 2019-11-14T14:29:16Z

ESMF is moving away from including center coordinates and other auxiliary variables in the weight files. The concurrent weight file write routine (used by ESMPy) also does not support them. The reason for the removal is the increasing size of grids and associated coordinate arrays. If this is a deal breaker, I can definitely investigate getting them output in the concurrent routine.

I'll add the additional global attributes no problem!

slevis-lmwg · 2019-11-14T15:23:56Z

ESMF is moving away from including center coordinates and other auxiliary variables in the weight files. The concurrent weight file write routine (used by ESMPy) also does not support them. The reason for the removal is the increasing size of grids and associated coordinate arrays. If this is a deal breaker, I can definitely investigate getting them output in the concurrent routine.

@ekluzek I will let you decide how to approach this.

...so as to compare map_ files generated by ocgis (this PR) vs. generated by the default mkmapdata.sh tool in ESCOMP#823.

...also first (incomplete) draft of ChangeLog

Only one src/dst does not and will not work because the region in the dst file does not overlap the src file at all: SRC=UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc DST=SCRIPgrid_1x1pt_urbanc_alpha_nomask_c110308.nc

ekluzek · 2019-12-16T17:28:57Z

ESMF is moving away from including center coordinates and other auxiliary variables in the weight files. The concurrent weight file write routine (used by ESMPy) also does not support them. The reason for the removal is the increasing size of grids and associated coordinate arrays. If this is a deal breaker, I can definitely investigate getting them output in the concurrent routine.

@ekluzek I will let you decide how to approach this.

Thanks for pointing this out @slevisconsulting. I'm checking with Jim Edwards and Mariana V. about this

ekluzek

I have some questions, comments, and change requests. Thanks.

ekluzek · 2019-12-16T17:31:14Z

tools/mkmapdata/mkmapdata.sh

@@ -446,17 +429,9 @@ if [ "$interactive" = "NO" ]; then
   mpirun=$MPIEXEC
   echo "Running in batch mode"
 else
-   mpirun=""
+   mpirun="mpirun -np 1"


It would be better for the serial case to NOT be required to run with MPI. Is always running with MPI a requirement of OCGIS?

It is not a requirement for ocgis. ESMF must be built with MPI support for IO operations...

Thanks for that @bekozi. On cheyenne there are versions of ESMF that are built without MPI (the "uni" versions). So we used them for the serial cases. Can ocgis choose to point to a version of ESMF that is built without MPI?

I think we will encounter a problem in the weight file write routine since it uses PIO v1 which ESMF does not wrap with mpiuni. This could be another reason to fall back on the other weight file write routine which, while not writing concurrently, uses netcdf directly so supports mpiuni (I'm 99% sure this is the case). I'll confirm this with the ESMF group.

OK, thanks for pointing that out. If we need to require MPI, I think that's OK. But, because it's so much simpler to run without MPI, I like to have it as an option. It's often easier to setup the non-MPI version on a machine first, rather than having to also bring in the extra complexity for MPI. Practically, I'm not sure you can run the regridding for a non-MPI case at least for any sizable grid. It would be hopeful (and helpful) that you could run a single point grid without MPI though for example though.

I'll talk with @rokuingh about adding another option for writing weight files in ESMPy. This should allow us to handle the mpiuni case. We will also be able to maintain the center coordinates in the weight file for the interim. An option to switch between the write routines will be available. I created an issue on the ocgis side (NCPP/ocgis#511) to track this capability.

I did want to amend my comment on ESMF weight file center coordinates from earlier. The center coordinates in the ESMF weight file are not repeated for each source/destination index combination. The center coordinates are copied over from the source/destination grids. This can be inefficient for larger grids, but not as inefficient as my earlier comment would make it appear.

Ping @oehmke

@rokuingh is planning to work on these two features (verbose weight file and arbitrary sequence indices for factor lists) hopefully next week.

ekluzek · 2019-12-16T17:33:52Z

tools/mkmapdata/mkmapdata.sh

@@ -288,7 +280,7 @@ else
 fi

 # Set timestamp for names below 
-CDATE="c"`date +%y%m%d`
+CDATE="c"`date -d "0 days" +%y%m%d`


See comment about this in the other PR. Is this for setting of daylight savings time? If so I especially think we shouldn't mess with it.

Adding comment explaining the use of the date -d flag, as discussed here.

ekluzek · 2019-12-16T17:34:57Z

tools/mkmapdata/mkmapdata.sh

  if [ -z "$MPIEXEC" ]; then
     MPIEXEC="mpiexec_mpt -np $REGRID_PROC"
  fi
+
+  module load ncarenv/1.3


Is it critical to use specific versions of software packages? This can be something difficult to stay up to date with.

This is the combination of module loads that @bekozi determined we needed for ocgis to run. I can test by trial-and-error whether removing the version numbers will continue to work, unless @bekozi advises against it.

@bekozi, before attempting to remove specific version numbers by trial-and-error, I returned to my previously working script on cheyenne:
/glade/work/slevis/ocgis_work/subset_1x1_20191213.sh
and found that it now fails with this error in the PET0.ESMF_LogFile:
20191229 114147.072 ERROR PET0 ESMF_Array.F90:3962 ESMF_ArrayWrite Unable to write to file - Internal subroutine call returned Error
20191229 114147.073 ERROR PET0 ESMF_IOScrip.F90:367 ESMF_OutputWeight File Unable to write to file - Internal subroutine call returned Error
20191229 114147.073 ERROR PET0 ESMF_IOScrip.F90:136 ESMF_SparseMatrix Write Unable to write to file - Internal subroutine call returned Error
20191229 114147.073 ERROR PET0 ESMF_Field_C.F90:1269 f_esmf_regridstorefile Unable to write to file - Internal subroutine call returned Error
It still writes out the same subset file as before:
/glade/work/slevis/ocgis_work/subsets/spatial_subset_191209.nc
but fails to generate the weight file that it had previously:
/glade/work/slevis/ocgis_work/weights/weights_191209_after_subsetg_1x1.nc

I tried two things that didn't help.
In /glade/work/slevis/git_ocgis/ocgis and
in /glade/work/slevis/git_esmf/esmf I typed
git fetch
git pull
to get caught up with the latest codes.

What else am I missing?

Turns out the job fails when the weight file already exists. I'm surprised that I had not run into this before :-) Thanks for troubleshooting with me, @bekozi. Reminder for you to put a more explanatory error message in the code about this.

Replaced specific module loads with generic module loads now.

ekluzek · 2019-12-16T17:36:37Z

tools/mkmapdata/mkmapdata.sh

-      cmd="$cmd -d $GRIDFILE -m conserve -w ${OUTFILE[nfile]}"
-      if [ $type = "regional" ]; then
-        cmd="$cmd --dst_regional"
+      PATH_PREFIX=/glade/work/slevis


We should have this outside of a particular user's space. On cheyenne it could be under: /glade/tss/

For now I changed these path assignments to be based off of $dir because this is already assigned at the top of the script. This way I avoid using slevis in the path.

I am open to changing to something like /glade/tss/ later.

ekluzek · 2019-12-16T17:38:38Z

tools/mkmapdata/mkmapdata.sh

+      CHUNKDIR=${WD}/chunking
+      rm -rf ${CHUNKDIR}
+      OCLI_FILE=${PATH_PREFIX}/git_ocgis/ocgis/src/ocgis/ocli.py
+      CRWG="python ${OCLI_FILE} chunked-rwg"


Some of these names are a bit short and cryptic. I get WD for Working Directory. And I suppose from context SS_PATH is Spatial Subset path. But, I'm not sure what CRWG means? What does the "chunked -rwg" option mean?

I added this variable for the development script. It allows switching the CLI executable easily. For production code, I expect you could just use ocli chunked-rwg (chunked-rwg is the CLI subcommand).

I'm mainly complaining about the names being too short to be meaningful and obvious (at least to me). So if it could be a little longer and more descriptive that would be useful. Or adding a comment that describes what it's doing and what its purpose is. But, typically it's better to just have a longer more descriptive name.

👍 Names are indeed a bit cryptic.

Renamed CRWG to CLI_EXECUTABLE.
Renamed SS_PATH to SUBSETS_PATH.
Removed PATH_PREFIX and WD.

These are minor, but I have not tested them, yet, because my simple ocgis script that last worked on 12/13/2019 is now failing. I will test when that is resolved.

slevis-lmwg · 2020-01-02T19:56:58Z

This error came up while running my ocgis script on izumi. I confirmed that the same SRC/DST combination leads to the same error on cheyenne:
ValueError: wrap action combination not supported

@bekozi this is the script that gives the error on cheyenne:
/glade/work/slevis/ocgis_work/do-this-20200101_subsetting.sh

Thanks, again, for setting up the call today!

slevis-lmwg · 2020-01-02T20:15:18Z

@bekozi sorry... I renamed the file to no_subset_20200102.sh to make it more descriptive.

bekozi · 2020-01-03T15:21:41Z

@slevisconsulting The two issues we identified yesterday are fixed in ocgis/master:

Wrap action no supported (Fix wrap action check when preparing subset geometries NCPP/ocgis#509) -- The wrapped state was using an incorrect variable when identifying the appropriate action. The test script runs to completion. Two variables needed swapping and only one was switched the previous time we encountered this issue. 😱
Check for weight file presence before overwriting (Add check for weight file presence before overwriting in chunked-rwg NCPP/ocgis#510).

slevis-lmwg · 2020-01-06T21:45:14Z

Thank you, @bekozi , I pulled your mods and the test script runs.

bekozi · 2020-07-24T15:05:16Z

@slevisconsulting I'm doing pretty well thanks for asking. I hope it's the same for you!

My last update for the issue is here: NCPP/ocgis#494 (comment). I should have pinged you on it. @rokuingh and I got the arbitrary sequence indices for grids working in both ESMF and ocgis. The mesh is still outstanding. We are aiming to do it the right way, and once the MOAB mesh in ESMF is implemented fully, we can change the sequence indices in-memory without rebuilding the mesh. I know this has taken awhile. It ended up being more complex than I had expected. Given the success with the grid, I expect once the new mesh capabilities are available, we can move on it pretty quickly. Is this holding you up in anyway? The mesh changes are getting close, but it's difficult to give you a reliable time estimate...

slevis-lmwg · 2020-07-24T17:27:09Z

My last update for the issue is here: NCPP/ocgis#494 (comment). I should have pinged you on it. @rokuingh and I got the arbitrary sequence indices for grids working in both ESMF and ocgis. The mesh is still outstanding. We are aiming to do it the right way, and once the MOAB mesh in ESMF is implemented fully, we can change the sequence indices in-memory without rebuilding the mesh. I know this has taken awhile. It ended up being more complex than I had expected. Given the success with the grid, I expect once the new mesh capabilities are available, we can move on it pretty quickly. Is this holding you up in anyway? The mesh changes are getting close, but it's difficult to give you a reliable time estimate...

Thank you @bekozi for this update. One more piece of work discussed above was the "weight file with auxiliary variables." Is that one done?

I will let somebody from the NCAR group respond to your question about timing.

bekozi · 2020-07-24T19:18:51Z

One more piece of work discussed above was the "weight file with auxiliary variables." Is that one done?

It is!: NCPP/ocgis#511

The argument --weightfilemode WITHAUX will write a verbose weight file. @ekluzek and I went back and forth on the history string and arrived at something he liked. The argument gets passed down to ESMPy which also required a feature improvement. Hence, a newer beta snapshot of ESMF/ESMPy is required.

slevis-lmwg · 2020-07-24T19:27:29Z

Great, thank you!

dlawrenncar · 2020-07-24T20:22:32Z

I'm not able to follow all the technical details enough to even know what part of our project is or is not being held up. I suggest that we discuss at a forthcoming WRF-CTSM meeting.

…

On Fri, Jul 24, 2020 at 1:27 PM Samuel Levis ***@***.***> wrote: Great, thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#815 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVCNWF7RJVVY4474ERTR5HODDANCNFSM4I5VZVVQ> .

These mods came in with ESCOMP@ef5f3d5 mkmapdata.sh with ocgis still runs for certain SRC/DST combinations and not for others. Will keep working on it.

NCHUNKS_DST=10 works for all DST files. NCHUNKS_DST=20 failed for 10x15 DST files.

slevis-lmwg · 2020-08-04T19:56:44Z

bld/namelist_files/namelist_defaults_ctsm_tools.xml

+<unstructdata hgrid="1km-merge-10min" lmask="HYDRO1K-merge-nomask"
+   >lnd/clm2/mappingdata/grids/UGRID_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc</unstructdata>
+<unstructdata_type hgrid="1km-merge-10min" lmask="HYDRO1K-merge-nomask">UGRID</unstructdata_type>
+


I kept the above as placeholders. I'm ok with removing if that's preferred.

slevis-lmwg · 2020-08-04T19:57:12Z

bld/namelist_files/namelist_definition_ctsm.xml

+       input_pathname="abs" group="clmexp" valid_values="" >
+UNSTRUCT format grid data file
+</entry>
+


Placeholder. I'm ok with removing if that's preferred.

slevis-lmwg · 2020-08-04T19:57:25Z

bld/namelist_files/namelist_definition_ctsm.xml

+Flag to pass to the ESMF mapping utility, telling it what kind of grid
+file this is.
+</entry>
+


Placeholder. I'm ok with removing if that's preferred.

slevis-lmwg · 2020-08-04T20:05:01Z

tools/mkmapdata/mkmapdata.sh

           "5x5min_IGBP-GSDP"  \
           "5x5min_ISRIC-WISE" \
           "5x5min_ORNL-Soil" \
-           "10x10min_nomask"   \


Oops, I missed two lines. I will put them back in the next commit.

slevis-lmwg · 2020-08-05T21:29:15Z

Update:
I have run qsub regridbatch and regenerated all the map_files except for the ones pointing to our highest resolution SRC files, i.e. 3x3min and 1km-merge-10min_HYDRO1K-merge-nomask.

The latter SRC files return this error:
NetCDF Error: ./chunking/esmf_weights_1.nc : NetCDF: One or more variable sizes violate format constraints

I request 10 chunks and end up with 1 for these SRC files. As a test I requested 1 chunk but the error remains the same.

@bekozi suggested pointing to older esmf builds, so I tried changing my ESMFMKFILE to
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b23/mpt/2.21/intel/19.0.5/lib/libO/Linux.intel.64.mpt.default/esmf.mk
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b23/mpt/2.21/intel/19.0.5/lib/libg/Linux.intel.64.mpt.default/esmf.mk
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b21/mpt/2.21/intel/19.0.5/lib/libO/Linux.intel.64.mpt.default/esmf.mk
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b21/mpt/2.21/intel/19.0.5/lib/libg/Linux.intel.64.mpt.default/esmf.mk
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b17/mpt/2.21/intel/19.0.5/lib/libO/Linux.intel.64.mpt.default/esmf.mk
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b17/mpt/2.21/intel/19.0.5/lib/libg/Linux.intel.64.mpt.default/esmf.mk
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b17/mpt/2.19/intel/19.0.5/lib/libO/Linux.intel.64.mpt.default/esmf.mk
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b16/mpt/2.21/intel/19.0.5/lib/libO/Linux.intel.64.mpt.default/esmf.mk
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b16/mpt/2.21/intel/19.0.5/lib/libg/Linux.intel.64.mpt.default/esmf.mk
All cases gave the same error.

@bekozi will be taking time off from today until 8/17.

slevis-lmwg · 2020-08-06T23:28:51Z

Latest:
Turns out that all along the problem with the 3x3min and 1km SRC files has been the new option (--weightfilemode WITHAUX). Somehow I had not tested this correctly before.

So, back to using the newer esmf build
/glade/p/cesmdata/cseg/PROGS/esmf/8.1.0b24/mpt/2.21/intel/19.0.5/lib/libO/Linux.intel.64.mpt.default/esmf.mk
the 3x3min and 1km SRC files generate weight files, except when I include the new WITHAUX option.

The WITHAUX option is what will allow us to generate fsurdat files from the map_files.

slevis-lmwg · 2020-08-18T14:10:19Z

Update:
@bekozi has pointed me to a new esmf.mk file. The 3x3min SRC files work now and I am generating corresponding map_files.

The 1km SRC files still give errors and @bekozi is investigating.

slevis-lmwg · 2020-08-25T23:35:37Z

Good news:
@bekozi has a combination of ocgis/esmf that appears to work with the 1km SRC files now. Ben and I need to clarify how to get this working more reliably. Then I will proceed with the generation of map_files for 1km SRC files.

slevis-lmwg · 2020-08-28T15:33:17Z

Update
I have been running two separate regridbatch jobs:

one to generate the map_3x3min_* files
the other to generate the map_1km-* files

There are 34 DST files per SRC file:

the 3x3min SRC files are almost done
the 1km SRC file is a third of the way
other SRC files are done

I have found that the 1km SRC file works with 8 ncpus/mpiprocs but not with 12, 16, 20, 36, or 72.
@bekozi did some of his own testing and gave me this assessment:
"From what I can tell, this is memory-related on the ESMF side. [...] what is the overall goal regarding the flexibility of ocli for the high resolution grid combinations? My sense is that the approach works well for most grid combos, it's just these really high resolution ones that take more tuning. I don't think we have the time (at least I don't right now) to make this completely general. Is there a proc count / chunk count that you are targeting for all regridding operations? If there is one, I can look into those settings in more depth."

I will let @ekluzek respond to Ben's questions.

Reduce duplication between caps Eliminate duplication of the "derived quantities for required fields" and corresponding error checking codes that repeat across the mct, nuopc, and lilac caps. This consolidates the code and reduces maintenance requirements.

slevis-lmwg · 2020-10-01T21:47:32Z

At Sep 24 meeting we decided to move fwd with the same testing that @ekluzek and @billsacks had proposed for #823 .

The first test-suite is /glade/work/slevis/git/mkmapdata_ocgis/test/tools/test_driver.sh, which I compared to the ctsm5.1.dev002 baseline. As far as I can tell this test-suite passed. I attach the std out file to share the results that I see:

run.txt

ekluzek · 2020-10-01T22:04:27Z

OK, note the relevant tests are passing. If you look at /glade/work/slevis/git/mkmapdata_ocgis/test/tools/td.2615.status you'll see the summary of the tests. They all pass, except the PTCLM tests, but they don't matter for this work. Likely you would see the same for running the baseline version. But, the problem with those tests don't matter for this work.

slevis-lmwg · 2020-10-02T15:11:28Z

[...] They all pass, except the PTCLM tests, but they don't matter for this work. Likely you would see the same for running the baseline version.

Thanks @ekluzek and you are right that tests 31-35 fail in the baseline, as well.

slevis-lmwg · 2020-10-12T20:19:01Z

Ben removed the cause of blank row, col, S variables in the map_files when running ocgis through mkmapdata.sh

I have now generated the first surface dataset using map_files that I generated with ocgis in mkmapdata.sh.

Although the surface dataset seems good, the corresponding log file shows an error-checking issue where all area variables (input and output) are zero. I have posted this here.

ekluzek · 2022-02-28T23:25:26Z

The work on parallel mksurfdata probably means this work will be closed without coming in.

ekluzek · 2022-04-14T04:52:11Z

With PE #1663 this is something that we just aren't going to do.

slevis-lmwg added 2 commits September 27, 2019 11:07

Merge tag 'ctsm1.0.dev067' into mkmapdata_ocgis

431aa58

Add config_archive.xml to cime_config for CTSM. Update the CODE_OF_CONDUCT to the August/2019 UCAR version. No testing done.

Integrating @bekozi's ocgis tool within mkmapdata.sh

1cd3689

With initial help from @ekluzek on 9/27/2019 and additional modifications made today, I am able to run @bekozi's ocgis tool by running a new version of mkmapdata.sh that I'm committing here.

slevis-lmwg mentioned this pull request Oct 17, 2019

For mksurfdata_map: use maps with no source masking, applying mask separately #286

Closed

ekluzek added PR status: work in progress enhancement new capability or improved behavior of existing capability labels Oct 21, 2019

ekluzek self-assigned this Oct 21, 2019

slevis-lmwg added 2 commits November 13, 2019 18:03

Correcting ocgis working directory in mkmapdata.sh

36d2041

Merge tag 'ctsm1.0.dev079' into mkmapdata_ocgis

b230b9c

Change a few uses of shr_kind Change a few uses of shr_kind to work with latest cime. These changes are from Mariana Vertenstein.

Switch to working with UNSTUCTgrid nomask SRC files

ef5f3d5

...so as to compare map_ files generated by ocgis (this PR) vs. generated by the default mkmapdata.sh tool in ESCOMP#823.

slevis-lmwg mentioned this pull request Nov 18, 2019

mksurfdata_map: replace source (SRC) files of various masks with SRC files with no mask #823

Merged

slevis-lmwg added 2 commits November 21, 2019 15:16

mkmapdata.sh with ocgis: updates for regional DST grids

e6d69b4

...also first (incomplete) draft of ChangeLog

ekluzek requested changes Dec 16, 2019

View reviewed changes

Revisions in response to @ekluzek's review

e7d6e97

These are minor, but I have not tested them, yet, because my simple ocgis script that last worked on 12/13/2019 is now failing. I will test when that is resolved.

Changing from specific to generic module loads

504a8aa

bekozi mentioned this pull request Jan 3, 2020

Add option to write verbose ESMF weight file NCPP/ocgis#511

Closed

slevis-lmwg added 2 commits August 3, 2020 18:03

Mainly reversing mods that were for using "nomask" SRC files

3199196

These mods came in with ESCOMP@ef5f3d5 mkmapdata.sh with ocgis still runs for certain SRC/DST combinations and not for others. Will keep working on it.

Replace NCHUNKS_DST with a value that works for all DST files

e424fc5

NCHUNKS_DST=10 works for all DST files. NCHUNKS_DST=20 failed for 10x15 DST files.

slevis-lmwg commented Aug 4, 2020

View reviewed changes

Small correction to fully revert to the modifications without nomask

bde5781

slevis-lmwg mentioned this pull request Aug 17, 2020

Add more metadata to the output file... NCPP/ocgis#506

Open

slevis-lmwg added 2 commits September 29, 2020 08:53

Changes needed for 1km SRC files to work with ocgis

5af5fa3

ekluzek marked this pull request as draft September 2, 2021 19:38

ekluzek added closed: wontfix We won't fix this issue, because it would be too difficult and/or isn't important enough to fix and removed PR status: work in progress labels Apr 14, 2022

ekluzek closed this Apr 14, 2022

Integrate @bekozi's ocgis tool in mkmapdata.sh #815

Integrate @bekozi's ocgis tool in mkmapdata.sh #815

Conversation

slevis-lmwg commented Oct 5, 2019 • edited Loading

Description of changes

Specific notes

billsacks commented Oct 6, 2019

slevis-lmwg commented Nov 13, 2019

bekozi commented Nov 13, 2019

slevis-lmwg commented Nov 13, 2019

slevis-lmwg commented Nov 13, 2019

bekozi commented Nov 13, 2019

slevis-lmwg commented Nov 13, 2019

bekozi commented Nov 14, 2019

slevis-lmwg commented Nov 14, 2019

ekluzek commented Dec 16, 2019

ekluzek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slevis-lmwg commented Jan 2, 2020

slevis-lmwg commented Jan 2, 2020

bekozi commented Jan 3, 2020

slevis-lmwg commented Jan 6, 2020

bekozi commented Jul 24, 2020

slevis-lmwg commented Jul 24, 2020

bekozi commented Jul 24, 2020

slevis-lmwg commented Jul 24, 2020

dlawrenncar commented Jul 24, 2020 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slevis-lmwg commented Aug 5, 2020

slevis-lmwg commented Aug 6, 2020

slevis-lmwg commented Aug 18, 2020

slevis-lmwg commented Aug 25, 2020

slevis-lmwg commented Aug 28, 2020 • edited Loading

slevis-lmwg commented Oct 1, 2020 • edited Loading

ekluzek commented Oct 1, 2020

slevis-lmwg commented Oct 2, 2020

slevis-lmwg commented Oct 12, 2020 • edited Loading

ekluzek commented Feb 28, 2022

ekluzek commented Apr 14, 2022

slevis-lmwg commented Oct 5, 2019 •

edited

Loading

slevis-lmwg commented Aug 28, 2020 •

edited

Loading

slevis-lmwg commented Oct 1, 2020 •

edited

Loading

slevis-lmwg commented Oct 12, 2020 •

edited

Loading