Regridding functionalities (powered by xESMF) #243

sol1105 · 2022-08-17T09:36:08Z

Pull Request Checklist:

This PR addresses an already opened issue (for bug fixes / features)
- This PR closes Regridding Prototype #68, closes Check outputs from regridding #180, closes Add pytest marker(s) for regridding (to include/exclude) tests #182, and closes Review the regrid-main branch ready for merger #215
- Follows up on New functionalities related to regridding #196, Grid halo #141, Move target grids location to configuration #168. These will have to be addressed again at a later point
Tests for the changes have been added (for bug fixes / features)
Documentation has been added / updated (for bug fixes / features)
HISTORY.rst has been updated (with summary of main changes) -> Will be added in a release preparing PR
I have added my relevant user information to AUTHORS.md

What kind of change does this PR introduce?:
- Extending Fix 224 inconsistent bounds #225 for all data_vars and coords of the xarray object (requires cf-xarray >= 0.7.5)
- Adding jupyter notebook to showcase remapping functionalities
- Adding regridding functionalities (powered by xESMF):
  - clisops.ops.regrid.regrid one line remapping function orchestrating below functions and classes
  - clisops.core.Grid class:
    - create xarray.Dataset holding description / coordinates of a regular lat-lon grid in CF compliant format:
      - from xarray.Dataset/DataArray (optionally of adaptive resolution, if source is not a regular lat-lon grid)
      - via grid_instructor (creating regional or global grid)
      - selecting a pre-defined grid (https://github.com/roocs/roocs-grids)
    - reformat grids (SCRIP, CF, xESMF formats)
    - detect extent, shape, format, type of the grid
    - detect collapsing or duplicated cells
    - create hash, compare grid objects
    - save to disk
    - exchange attributes and non-horizontal coordinates between datasets
    - calculate bounds (for regular lat-lon and curvilinear grids)
    - re-define data_vars and coords of xarray.Dataset for xESMF application
  - clisops.core.Weights:
    - orchestrate creation of weights with xESMF
    - holds xesmf.Regridder object
    - read from or store to local remapping weights cache (lock file mechanic for contemporary weights creation)
    - generate hash to identify similar weights
  - clisops.core.regrid:
    - application of remapping weights on xarray.Dataset/DataArray
    - optionally transfer attributes and non-horizontal coordinate variables from source to target dataset
    - set new attributes related to the remapping operation
- Adding lock file mechanic (from source)
Does this PR introduce a breaking change?:
- adding regrid operator in ops
- adding regrid function, Grid and Weights classes to core
- the remapping makes use of xesmf, which is already a dependency
- adding dependency for roocs-grids (might be removed with resolving Move target grids location to configuration #168)
- potentially TBA
Other information:

Future planned PR(s) specific for remapping:

Support for manually provided masks
Support for out-of-domain masking for nearest neighbour (likely better put to xesmf)
Deal with the periodic attribute of xesmf (xesmf resets periodic to False for conservative remapping, but probably should not)
Support datasets with shifted longitude frames (eg. ranging from (-60, 300) degrees_east) - base work already done in previous PR
Support reformatting from/to further formats
Support grid type detection for other formats
Support reading / reformatting / using weight files from other tools like nco, cdo, ...
Calculate nominal_resolution of the target grid, if not present
Set up central / web based remapping weights cache and synchronizing cron job
Find solution for vector variables / variables defined on cell edges
Support vertical interpolation (eg. with xgcm)
(Support other remapping backends than xesmf in the far future)

Future planned PR(s) generally for clisops:

Unify the detect_coordinate functions
Attribute a new tracking_id / PID (general requirement for clisops)
Support datasets with missing missing_value / _FillValue attribute that feature missing values (add fix in dachar?)

- Fix typo importing cf_xarray - Fix missing brackets in if statement in Grid.__repr__() - Fix reformatting the bounds in Grid.grid_reformat() when reformatting xESMF to CF. Now using cfxr.vertices_to_bounds instead of _reravel - Now updating the format attribute when executing Grid.grid_reformat() - Added compute_bounds for format CF and grid_type regular_lat_lon

- Added basic adaptive_masking function that will be obsolete as soon as adaptive masking is implemented in xesmf. - Added basic regrid function. - Added basic Weights class. The method add_matrix_NaNs to mask out-of-source-domain grid cells in the remapping weights will likely be obsolete with future xesmf versions. Yet, Weights can only be generated by xesmf by specifying input and output grid as well as the remapping method. - Minor changes to the Grid class: - Default attributes mostly set to None. - Now raising exception when none of the input parameters are specified (Dataset, grid_id or grid_instructor). - Added temporary fix for a small cf_xarray bug that identifies lat_bnds/lon_bnds as latitude and longitude coordinates if the bounds are specified as xarray.Dataset.coords and they have a units attribute. - detect_coords now uses roocs_utils.xarray_utils.get_coord_by_attr - Cleared open questions regarding tests/core/test_regrid.py leading to minor changes.

… core/regrid.py - Added the necessary methods to the Regrid class in ops/regrid.py Yet adaptive masking can result in an error when providing an xarray.Dataset as input. Also, the xesmf regridding seems to drop a few variables and keep others that are not necessary to be kept. - Added Grid.__str__() method to core/regrid.py - Added Grid.remove_halo() method to core/regrid.py. For partially duplicated rows/columns an exception will be raised. Fully duplicated rows/columns will be cut off.

- Added unmapped_to_nan parameter for init calls to xesmf.Regridder class, replaces Weights.add_matrix_NaNs - Added parameters skip_na and na_thres for calls to xesmf.Regridder instances, replacing custom adaptive_masking function calls - core/regrid.py - calls to roocs-utils.xarray-utils.get_coord_by_type adjusted to include ignore_aux_coords=False to allow 2D coordinate variables to be identified as coordinates - Requirement for xesmf set to xesmf>=0.6.0 - test_grid_init_ds_tos_curvilinear adjusted for removed halo - Regridding tests only run for xesmf>=0.6

sol1105 · 2023-11-10T14:25:32Z

@Zeitsperre @cehbrecht I removed python 3.8 from the CI checks because of the cf_xarray problem you adressed in roocs_utils using the following lines in the requirements.txt:

cf-xarray>=0.3.1,<=0.8.4; python_version == '3.8'
cf-xarray>=0.3.1; python_version >= '3.9'

I am not sure how such a setting can be added in the dependencies entry of the pyproject.toml, in case python 3.8 is still required. Would it simply be:

dependencies = [
  "bottleneck>=1.3.1",
  # cf-xarray is differently named on conda-forge
  "cf-xarray>=0.8.6;python_version>='3.9'",
  "cf-xarray>=0.7.5,<=0.8.0;python_version=='3.8'",
...

Zeitsperre · 2023-11-10T15:36:12Z

@sol1105

I am not sure how such a setting can be added in the dependencies entry of the pyproject.toml, in case python 3.8 is still required. Would it simply be:
dependencies = [
  "bottleneck>=1.3.1",
  # cf-xarray is differently named on conda-forge
  "cf-xarray>=0.8.6; python_version>='3.9'",
  "cf-xarray>=0.7.5,<=0.8.0; python_version=='3.8'",
...

That's exactly what you would need to do to maintain Python3.8 support. Everything is using pip under-the-hood. The only thing I'm not certain about is that I believe cf_xarray up until v0.8.4 is compatible with Python3.8 (at least, that's what I indicated in other places).

cehbrecht · 2023-11-20T15:10:38Z

@Zeitsperre I leave the last words to you :) We would like to merge this PR and make the new clisops release with the regrid operator. I have prepared daops and rook already for the new operator and it works.

pyproject.toml

Zeitsperre · 2023-11-20T19:15:53Z

Hey all,

Working on this now. I'm going to be adding a few changes to docstrings and updating some deprecated calls. I also might split the failing upstream build into its own workflow so that it isn't constantly failing. Will give everything a final overview after that's all done.

Should have something today.

Zeitsperre

There's a ton going on here, but we've collectively been examining this PR for a very long time. Things look good beyond a handful of small issues I noticed when cleaning up the typing/docstrings. Congrats!

clisops/core/regrid.py

environment.yml

docs/environment.yml

- Addressed review comments regarding environment.yml - Added test: regridding of data array - core.regrid: Fixed problems in case of DataArray instead of Dataset

sol1105 · 2023-11-21T11:56:41Z

@Zeitsperre Thanks a lot for the review and your contribution. I addressed the outstanding issues you found

Zeitsperre · 2023-11-21T21:49:55Z

Well done. Feel free to merge whenever you're ready!

ellesmith88 and others added 30 commits March 4, 2021 10:39

put in basic structure and notes for regrid

ff698e5

change params to strings

4869291

import xarray

963ad64

add halo comment

d8d250c

A first version of the Grid class for the regrid-prototype.

1ae4120

Merge branch 'master' into regrid-main-elle

ccccdc7

Start adding tests for Grid class

7476750

merge most recent update

7bac759

update tests

7b49c24

update tests to include bounds and grid

efaf5e0

update dependencies and add tests for Weights

b4ae320

Added a comment to Grid.grid_remove_halo() in core/regrid.py

5e944fb

remove bottleneck

a1136e7

update tests

3d12c73

undo rename

23a75fb

merge master

56654fe

Merged master

8925d0b

Updated to get regrid tests working to regular grid

c0bd806

Removed unused variable in test

054537c

Tidied.

46947e8

Fixed attribute bug in tests.

8d9e55e

run pre-commit

f4f3bbc

remove scipy from requirements

4f2845e

Add scipy into requirements and environment.yml

b6fc19e

require scipy>=1.5.4 to work with python 3.6

2a57286

Added newlines

4ce0342

Update requirements

dcadea5

sol1105 and others added 5 commits November 13, 2023 07:42

Update requirements

de4d0fe

Merge branch 'master' into regrid-main

7035a17

Update dependencies (roocs_grids) and HISTORY.rst

c560e48

Update dependencies (roocs_grids)

da34513

Update pyproject.toml

16ca1d3

cehbrecht approved these changes Nov 20, 2023

View reviewed changes

Zeitsperre reviewed Nov 20, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

Zeitsperre added 7 commits November 20, 2023 14:21

docstring cleanup and static typing

7a0d988

move upstream builds to push on main, cronjob, or workflow_dispatch

21f7260

refactoring fixes

192ee2e

refactor black to lint, naming fixes

80a681d

remove self from staticmethods

5b2ea42

remove deprecated pkg_resources calls, update pre-commit

8281f8a

simplify flake8 ignores

51620a0

Zeitsperre approved these changes Nov 20, 2023

View reviewed changes

clisops/core/regrid.py Outdated Show resolved Hide resolved

clisops/core/regrid.py Outdated Show resolved Hide resolved

environment.yml Outdated Show resolved Hide resolved

docs/environment.yml Outdated Show resolved Hide resolved

Zeitsperre and others added 3 commits November 20, 2023 16:37

update HISTORY.rst

de831fe

Updates to environment.yml and core.regrid

98fef4d

- Addressed review comments regarding environment.yml - Added test: regridding of data array - core.regrid: Fixed problems in case of DataArray instead of Dataset

Update docs/environment.yml to be consistent with environment.yml

fc7b31c

sol1105 merged commit a999bcb into master Nov 23, 2023

sol1105 deleted the regrid-main branch November 23, 2023 10:42

This was referenced Nov 23, 2023

Grid halo #141

Closed

Error: The defined _FillValue and missing_value are not the same #308

Closed

sol1105 mentioned this pull request Aug 14, 2024

Suggested approach for default grid when regridding (C3S approach) #181

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regridding functionalities (powered by xESMF) #243

Regridding functionalities (powered by xESMF) #243

sol1105 commented Aug 17, 2022 •

edited by Zeitsperre

Loading

sol1105 commented Nov 10, 2023

Zeitsperre commented Nov 10, 2023

cehbrecht commented Nov 20, 2023

Zeitsperre commented Nov 20, 2023

Zeitsperre left a comment

sol1105 commented Nov 21, 2023

Zeitsperre commented Nov 21, 2023

Regridding functionalities (powered by xESMF) #243

Regridding functionalities (powered by xESMF) #243

Conversation

sol1105 commented Aug 17, 2022 • edited by Zeitsperre Loading

Pull Request Checklist:

sol1105 commented Nov 10, 2023

Zeitsperre commented Nov 10, 2023

cehbrecht commented Nov 20, 2023

Zeitsperre commented Nov 20, 2023

Zeitsperre left a comment

Choose a reason for hiding this comment

sol1105 commented Nov 21, 2023

Zeitsperre commented Nov 21, 2023

sol1105 commented Aug 17, 2022 •

edited by Zeitsperre

Loading