-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Supporting xarray.apply_ufunc #67
Comments
Another question I had is about the fact that |
Yes, I think it would be possible to write a
To some extent it does, but there are things we could do. We could ask users to provide an (estimate) of the extra memory required, like I've created some notebooks that help judge if the memory estimates are working in practice, see https://github.com/tomwhite/cubed/blob/main/examples/lithops-add-random-local.ipynb for example. So there's some tooling like this we could provide to make it easier to provide and check memory estimates. |
Sounds good. I would be interested in trying to implement |
That would be great! |
A couple of thoughts on implementation. It should be possible to follow Dask's implementation, at least for the case where a single output array is returned, which can use blockwise. That's probably the thing to get working first. For the multiple output array case, we could use Zarr structured arrays, like we do already for the implementation of mean. Related: #69 This should live in the |
A very large proportion of xarray's code goes through
xarray.core.computation.apply_ufunc
, which is also exposed publicly for advanced users to wrap their own ufuncs.Internally if a dask array is present then we call
dask.array.apply_gufunc
- perhaps cubed should expose a similar function in the same way that it exposesmap_blocks
etc.?xref pydata/xarray#6807
The text was updated successfully, but these errors were encountered: