[Bug]: Support regridding for curvilinear grid #339

pochedls · 2022-09-08T20:31:01Z

Is your feature request related to a problem?

CDAT supported regridding for curvilinear grids, but this does not appear to work in xCDAT.

import cdms2
import numpy as np

# open file
fn = '/p/css03/cmip5_css01/data/cmip5/output1/CNRM-CERFACS/CNRM-CM5-2/historical/mon/ocean/Omon/r1i1p1/v20130401/thetao/thetao_Omon_CNRM-CM5-2_historical_r1i1p1_185001-185912.nc'
f = cdms2.open(fn)
thetao = f('thetao')
f.close()
# define new grid
gridOut = createGeoGrid(np.arange(-88.5, 90, 2.5), np.arange(1.25, 360, 2.5))
# do regridding
diag = {}
thetaor = field.thetao(gridOut, regridTool='esmf', regridMethod='linear', missing=np.nan, coordSys='deg', diag=diag, periodicity=1)
thetaor.getAxisIds()
thetaor.shape

['time', 'lev', 'latitude', 'longitude']
(120, 42, 72, 144)

import xcdat as xc
from xcdat.regridder import grid
import numpy as np

nlat = np.arange(-88.75, 90, 2.5)
nlon = np.arange(1.25, 360, 2.5)
ngrid = grid.create_grid(nlat, nlon)
fn = '/p/css03/cmip5_css01/data/cmip5/output1/CNRM-CERFACS/CNRM-CM5-2/historical/mon/ocean/Omon/r1i1p1/v20130401/thetao/thetao_Omon_CNRM-CM5-2_historical_r1i1p1_185001-185912.nc'
ds = xc.open_dataset(fn)
print(ds)
ds = ds.regridder.horizontal('thetao', ngrid, tool='xesmf', method='bilinear', periodic=True)

<xarray.Dataset>
Dimensions: (time: 120, bnds: 2, lev: 42, j: 292, i: 362, vertices: 4)
Coordinates:

time (time) datetime64[ns] 1850-01-16T12:00:00 ... 1859-12-16T12...

lev (lev) float64 5.022 15.08 25.16 ... 5.051e+03 5.35e+03

j (j) int32 1 2 3 4 5 6 7 8 ... 285 286 287 288 289 290 291 292

i (i) int32 1 2 3 4 5 6 7 8 ... 355 356 357 358 359 360 361 362
lat (j, i) float32 -78.39 -78.39 -78.39 ... 50.23 50.01 50.01
lon (j, i) float32 72.5 73.5 74.5 75.5 ... 73.05 73.04 73.0 72.99
Dimensions without coordinates: bnds, vertices
Data variables:
time_bnds (time, bnds) datetime64[ns] ...
lev_bnds (lev, bnds) float64 ...
lat_vertices (j, i, vertices) float32 ...
lon_vertices (j, i, vertices) float32 ...
thetao (time, lev, j, i) float32 ...
Attributes: (12/28)
institution: CNRM (Centre National de Recherches Meteorologiqu...
institute_id: CNRM-CERFACS
experiment_id: historical
source: CNRM-CM5-2 2010 Atmosphere: ARPEGE-Climat (V5.2.3...
model_id: CNRM-CM5-2
forcing: GHG, SA, Sl, Vl, BC, OC
... ...
table_id: Table Omon (26 July 2011) e26ed05e72b60ab734809dc...
title: CNRM-CM5-2 model output prepared for CMIP5 histor...
parent_experiment: pre-industrial control
modeling_realm: ocean
realization: 1
cmor_version: 2.7.1

ValueError: ESMC_FieldRegridStore failed with rc = 506. Please check the log files (named "*ESMF_LogFile").

Describe the solution you'd like

Ideally, xcdat could recognize this grid and have the capability to regrid it.

Describe alternatives you've considered

No response

Additional context

Actually – this was tested (see this thread). I'm having issues in this issue here and the dirst sea ice dataset I pulled up (/p/css03/cmip5_css01/data/cmip5/output1/CNRM-CERFACS/CNRM-CM5-2/historical/mon/seaIce/OImon/r1i1p1/v20130401/sic/).

durack1 · 2022-09-09T02:28:34Z

@pochedls not to derail your aim of this issue, as global regridding of any CMIP grid would be ideal, but it does appear other options may exist, particularly if a sun-global focus is the goal, see https://notebooksharing.space/view/c6c1f3a7d0c260724115eaa2bf78f3738b275f7f633c1558639e7bbd75b31456#displayOptions=

Thomas-Moore-Creative · 2022-09-09T03:41:15Z

@pochedls not to derail your aim of this issue, as global regridding of any CMIP grid would be ideal, but it does appear other options may exist, particularly if a sun-global focus is the goal, see https://notebooksharing.space/view/c6c1f3a7d0c260724115eaa2bf78f3738b275f7f633c1558639e7bbd75b31456#displayOptions=

@durack1 assume the issue here is the xCDAT API that wraps xESMF which does support [curvilinear grids] (https://xesmf.readthedocs.io/en/latest/notebooks/Curvilinear_grid.html)

But, as you point out, it is useful to note ongoing discussion about using other methods - particularly when you want to go beyond bilinear interp e.g. this related Pangeo discourse

pochedls · 2022-09-09T04:25:06Z

@durack1 – I don't know what you mean by "global regridding" and "sun-global focus," but I was hoping that the xcdat implementation of xESMF would work for transient grids to rectilinear grids. This worked in early testing so this issue is aimed to determine what the problem is in this case and make it work. If there is another package that does this more seamlessly – please let us know.

pochedls · 2022-09-28T04:38:00Z

Just noting some datasets that fail with a common error message:

ValueError: ESMC_FieldRegridStore failed with rc = 506. Please check the log files (named "*ESMF_LogFile").

20220913 131259.970 ERROR PET0 ESMCI_Mesh_Regrid_Glue.C:192 ESMCI_regrid_create() Invalid argument - - Src contains a cell that has corners close enough that the cell collapses to a line or point
20220913 131259.970 ERROR PET0 ESMCI_Mesh_Regrid_Glue.C:558 ESMCI_regrid_create() Invalid argument - Internal subroutine call returned Error
20220913 131259.970 ERROR PET0 ESMCI_MeshCap.C:1599 MeshCap::regrid_create() Invalid argument - Internal subroutine call returned Error
20220913 131259.970 ERROR PET0 ESMF_Regrid.F90:372 ESMF_RegridStore Invalid argument - Internal subroutine call returned Error
20220913 131259.970 ERROR PET0 ESMF_FieldRegrid.F90:1172 ESMF_FieldRegridStoreNX Invalid argument - Internal subroutine call returned Error
20220913 131259.970 ERROR PET0 ESMF_Field_C.F90:1137 f_esmf_regridstore Invalid argument - Internal subroutine call returned Error
20220913 131259.970 ERROR PET0 ESMCI_Field.C:1468 ESMCI::Field::regridstore() Invalid argument - Internal subroutine call returned Error
20220913 131259.970 ERROR PET0 ESMC_Field.C:459 ESMC_FieldRegridStore() Invalid argument - Internal subroutine call returned Error

/p/css03/esgf_publish/CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/SImon/siconc/gn/v20190313/
/p/css03/esgf_publish/CMIP6/CMIP/NCAR/CESM2-FV2/historical/r1i1p1f1/SImon/siconc/gn/v20191120/
/p/css03/esgf_publish/CMIP6/CMIP/NCAR/CESM2-WACCM/historical/r1i1p1f1/SImon/siconc/gn/v20190227/
/p/css03/esgf_publish/CMIP6/CMIP/NCAR/CESM2-WACCM-FV2/historical/r1i1p1f1/SImon/siconc/gn/v20191120/
/p/css03/esgf_publish/CMIP6/CMIP/CMCC/CMCC-CM2-HR4/historical/r1i1p1f1/SImon/siconc/gn/v20200904/
/p/css03/esgf_publish/CMIP6/CMIP/CMCC/CMCC-CM2-SR5/historical/r10i1p2f1/SImon/siconc/gn/v20220401/
/p/css03/esgf_publish/CMIP6/CMIP/CMCC/CMCC-ESM2/historical/r1i1p1f1/SImon/siconc/gn/v20210114/
/p/css03/esgf_publish/CMIP6/CMIP/CNRM-CERFACS/CNRM-CM6-1/historical/r10i1p1f2/SImon/siconc/gn/v20190125/
/p/css03/scratch/cmip6/CMIP/CNRM-CERFACS/CNRM-CM6-1-HR/historical/r1i1p1f2/SImon/siconc/gn/v20191021/
/p/css03/esgf_publish/CMIP6/CMIP/CNRM-CERFACS/CNRM-ESM2-1/historical/r11i1p1f2/SImon/siconc/gn/v20200408/
/p/css03/esgf_publish/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r101i1p1f1/SImon/siconc/gn/v20200919/
/p/css03/esgf_publish/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-AerChem/historical/r1i1p1f1/SImon/siconc/gn/v20200624/
/p/css03/esgf_publish/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-CC/historical/r10i1p1f1/SImon/siconc/gn/v20210630/
/p/css03/esgf_publish/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-Veg/historical/r10i1p1f1/SImon/siconc/gn/v20210523/
/p/css03/esgf_publish/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-Veg-LR/historical/r1i1p1f1/SImon/siconc/gn/v20200919/
/p/css03/esgf_publish/CMIP6/CMIP/FIO-QLNM/FIO-ESM-2-0/historical/r1i1p1f1/SImon/siconc/gn/v20191127/
/p/css03/esgf_publish/CMIP6/CMIP/MOHC/HadGEM3-GC31-MM/historical/r1i1p1f3/SImon/siconc/gn/v20191207/
/p/css03/esgf_publish/CMIP6/CMIP/IPSL/IPSL-CM5A2-INCA/historical/r1i1p1f1/SImon/siconc/gn/v20200729/
/p/css03/esgf_publish/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r10i1p1f1/SImon/siconc/gn/v20180803/
/p/css03/esgf_publish/CMIP6/CMIP/IPSL/IPSL-CM6A-LR-INCA/historical/r1i1p1f1/SImon/siconc/gn/v20210216/
/p/css03/esgf_publish/CMIP6/CMIP/HAMMOZ-Consortium/MPI-ESM-1-2-HAM/historical/r1i1p1f1/SImon/siconc/gn/v20190627/
/p/css03/esgf_publish/CMIP6/CMIP/MPI-M/MPI-ESM1-2-HR/historical/r10i1p1f1/SImon/siconc/gn/v20190710/
/p/css03/esgf_publish/CMIP6/CMIP/MPI-M/MPI-ESM1-2-LR/historical/r10i1p1f1/SImon/siconc/gn/v20190710/
/p/css03/esgf_publish/CMIP6/CMIP/MRI/MRI-ESM2-0/historical/r10i1p1f1/SImon/siconc/gn/v20210909/
/p/css03/esgf_publish/CMIP6/CMIP/NUIST/NESM3/historical/r1i1p1f1/SImon/siconc/gn/v20190704/
/p/css03/esgf_publish/CMIP6/CMIP/NCC/NorESM2-LM/historical/r2i1p1f1/SImon/siconc/gn/v20190920/
/p/css03/esgf_publish/CMIP6/CMIP/AS-RCEC/TaiESM1/historical/r1i1p1f1/SImon/siconc/gn/v20200630/

pochedls · 2022-10-24T21:30:45Z

Just confirming that PR #374 fixes the original issue/example documented here.

jasonb5 · 2022-10-25T16:48:31Z

From @pochedls concerning existing issues with curvilinear grids. I have ideas about some of these issues but others I may need to see the sources to investigate.

Error	Comment	Instances
`TypeError: buffer is too small for requested array`	This file needs to be opened chunked, probably just too large for memory.	1
`KeyError: "Receive multiple variables for key 'longitude': {'lon', 'longitude'}. Expected only one. Please pass a list ['longitude'] instead to get all variables matching 'longitude'."`	I'd need to see the source for this one.	2
`KeyError: 'lon_b'`	I'd need to see the source, it's possible the file is missing cell corner information required for conservative regridding.	3
`ValueError: Bounds format not understood. Got ('nj', 'ni', 'vertices') with shape (384, 320, 2).`	I would need to see the source.	1
`KeyError: "No results found for 'longitude'."`	Might be dimension is miss named or missing.	1
`xarray.core.merge.MergeError: conflicting values for variable 'vertices_latitude' on objects to be combined. You can skip this check by specifying compat='override'.`	I remember seeing this and I think it's just solved by changing the compat mode as suggested.	1

pochedls · 2022-10-25T16:53:43Z

From @pochedls concerning existing issues with curvilinear grids. I have ideas about some of these issues but others I may need to see the sources to investigate.

Yeah – my main comment was that you solved the majority of the issues (27 / 58 models had the issue solved in #374). I think some of these other are I/O or metadata issues that will need to be investigated individually.

pochedls added type: enhancement New enhancement request type: bug Inconsistencies or issues which will cause an issue or problem for users or implementors. labels Sep 8, 2022

pochedls changed the title ~~[Feature]: Support regridding for curvilinear grid~~ [Bug]: Support regridding for curvilinear grid Sep 8, 2022

jasonb5 mentioned this issue Oct 21, 2022

Fixes xesmf flag #374

Merged

9 tasks

jasonb5 closed this as completed in #374 Oct 25, 2022

pochedls mentioned this issue Jan 8, 2025

[Bug]: Getting data CF-compliant for regridding (with xesmf) #718

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Support regridding for curvilinear grid #339

[Bug]: Support regridding for curvilinear grid #339

pochedls commented Sep 8, 2022 •

edited

Loading

durack1 commented Sep 9, 2022

Thomas-Moore-Creative commented Sep 9, 2022

pochedls commented Sep 9, 2022

pochedls commented Sep 28, 2022

pochedls commented Oct 24, 2022

jasonb5 commented Oct 25, 2022

pochedls commented Oct 25, 2022

[Bug]: Support regridding for curvilinear grid #339

[Bug]: Support regridding for curvilinear grid #339

Comments

pochedls commented Sep 8, 2022 • edited Loading

Is your feature request related to a problem?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

durack1 commented Sep 9, 2022

Thomas-Moore-Creative commented Sep 9, 2022

pochedls commented Sep 9, 2022

pochedls commented Sep 28, 2022

pochedls commented Oct 24, 2022

jasonb5 commented Oct 25, 2022

pochedls commented Oct 25, 2022

pochedls commented Sep 8, 2022 •

edited

Loading