Large memory footprint when appending to an existing netCDF file

[DavidHuber-NOAA]

Testing has revealed that the GSI is using far too much memory when compiled with netCDF 4.9.2/gsi-ncdiag 1.1.1 and attempting to append to existing netCDF diagnostic files. For the global_4denvar test, this first shows up when the temperature operator setup function is called the second time. This tracks down to the ncdiag subroutine nc_diag_init with append=.true.. Running the test with just 12 PEs, the memory allocations suddenly jump from ~1.5GB/PE to ~6GB/PE. The exact function/subroutine called within nc_diag_init that causes this is still under investigation.

[DavidHuber-NOAA]

The issue is the way that the existing netCDF file is open when appending. The optional parameter specifies an override HDF5 cache size of 16GB. I suspect this was intended to be 16MB (the default is 1 MB). This occurs at nc_diag_write_mode.F90:326-327:

call nclayer_check( nf90_open(filename, NF90_WRITE, ncid, &
         bsize, cache_nelems = 16777216) ) ! Optimization settings

Excluding the cache_nelems parameter (i.e. using the default 1MB) results in a much smaller memory allocation as does using a 16MB value (cache_nelems = 16384), the latter being what I will use to fix this issue.

FYI @aerorahul