Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release fixed scipy version by solving interpolation issue introduced since scipy 1.10.0 #557

Closed
Tracked by #332
veenstrajelmer opened this issue Sep 30, 2023 · 0 comments · Fixed by #561
Closed
Tracked by #332

Comments

@veenstrajelmer
Copy link
Collaborator

veenstrajelmer commented Sep 30, 2023

There are xarray interpolation issues with scipy>=1.10.0:

Time is running out since scipy<1.10.0 has memory leak

Also, Python 3.11 environment now gives a deprecationwarning which disappears when using scipy 1.10.0 and later versions:

C:\Users\veenstra\Anaconda3\envs\dfm_tools_py311\Lib\site-packages\scipy\sparse\_sputils.py:44: DeprecationWarning: np.find_common_type is deprecated.  Please use `np.result_type` or `np.promote_types`.
See https://numpy.org/devdocs/release/1.25.0-notes.html and the docs for more information.  (Deprecated NumPy 1.25)
  upcast = np.find_common_type(args, [])

MWE (dfmt testcase test_xugrid_opendataset_ugridplot_contourf_scipy_numpy_deprecation() implemented in #560):

import dfm_tools as dfmt
import xugrid as xu

file_nc = dfmt.data.fm_curvedbend_map(return_filepath=True)

uds = xu.open_dataset(file_nc)
uds['mesh2d_flowelem_bl'].ugrid.plot.contourf()

Also rerun postprocessing notebook

The unexpected nans in the interpolation to a new xarray dimension, occurs due to new behaviour of scipy's RegularGridInterpolator. This is not unexpected or undesired, but just inconvenient in this case. It seems to have to do with the ordering of the arrays, but flipping the input array does not solve it since there is some sorting going on somewhere in the code. interp1d was tested by looping over all points and concatenating the arrays, and also by interp1d on a grid of lat*lon. Both were significantly slower than interpn/RegularGridInterpolator. @Huite suggested to stick to the linear interpn, but follow-up with a nearest interpn ([NearestNDInterpolator](https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.NearestNDInterpolator.html)) and combine with combine_first. This does not seem to affect performance too much and it does solve the problem of too much nans.

@veenstrajelmer veenstrajelmer changed the title release fixed scipy version release fixed scipy version by solving interpolation issue introduced since scipy 1.10.0 Oct 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant