Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compute multiple dask backed arrays at once #804

Closed
jcrist opened this issue Mar 25, 2016 · 5 comments
Closed

Compute multiple dask backed arrays at once #804

jcrist opened this issue Mar 25, 2016 · 5 comments

Comments

@jcrist
Copy link

jcrist commented Mar 25, 2016

In dask, a user can compute multiple arrays in a single scheduler run using the dask.compute function:

>>> a_computed, b_computed = dask.compute(a, b)

This is nice for when a and b might share intermediates. The same can be done currently in xarray if a and b are first put into a dataset:

>>> both = xr.Dataset(dict(a=a, b=b))
>>> both.load()    # Compute all the arrays in a single pass

This is fine, but it might also be nice to be able to do this without first putting everything into a dataset. I'm not sure what a good api is here, as xarray objects mutate when computed. Perhaps just adding an xr.compute(*args) function that fully realizes all dask backed variables.

>>> xr.compute(a, b)    # a and b now contain numpy arrays, not dask arrays
@shoyer
Copy link
Member

shoyer commented Mar 25, 2016

It would be nice to be able to use dask.compute() for this, by defining a duck type system for dask objects such that xarray objects can participate. (xref dask/dask#700)

I'm not sure what a good api is here, as xarray objects mutate when computed.

Yeah, this is somewhat unfortunate, but convenient (though somewhat obsolete with dask's cachey). Perhaps adding a dedicated .compute() method that does not mutate would be a good idea.

@jcrist
Copy link
Author

jcrist commented Mar 25, 2016

I'm fine with the duck typing, and could make that work fairly easily. The mutation is a bit trickier though. I see two good options here for what b = dask.compute(a) would do with a being an xarray object:

  1. b is a new object with the computation performed, a still contains a dask array (uncomputed).
  2. b is the same object as a, and the internals of a are mutated to contain evaluated data.

I'm slightly partial to option 2, as it seems to mesh more with how xarray currently uses dask. Thoughts?

@shoyer
Copy link
Member

shoyer commented Mar 25, 2016

I would lean towards keeping compute a pure function. That seems more straightforward to implement with duck typing in dask, anyways. I guess we could go either way, though.

@jhamman
Copy link
Member

jhamman commented Mar 7, 2018

I believe this should have been close by #1674. Anyone object or am I missing something?

@shoyer shoyer closed this as completed Mar 7, 2018
@shoyer
Copy link
Member

shoyer commented Mar 7, 2018

agreed, this is fixed now by dask.compute()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants