Fixed dask.optimize on datasets #4438
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Another attempt to fix pydata#3698. The issue with my fix in is that we hit `Variable._dask_finalize` in both `dask.optimize` and `dask.persist`. We want to do the culling of unnecessary tasks (`test_persist_Dataset`) but only in the persist case, not optimize (`test_optimize`).
|
The RTD build failure is unrelated and may result from these warnings? |
|
I think it's OK to merge #4435 first and then this. I'll do another PR if we need any more touch-ups. That way we can see if RTD is fixed there... |
max-sixty
reviewed
Sep 20, 2020
dcherian
added a commit
to dcherian/xarray
that referenced
this pull request
Oct 9, 2020
…pagate-attrs * 'propagate-attrs' of github.com:dcherian/xarray: (22 commits) silence sphinx warnings about broken rst (pydata#4448) Xarray open_mfdataset with engine Zarr (pydata#4187) Fix release notes formatting (pydata#4443) fix typo in io.rst (pydata#4250) Fix typo (pydata#4181) Fix release notes typo New whatsnew section Add notes re doctests (pydata#4440) Fixed dask.optimize on datasets (pydata#4438) Release notes for 0.16.1 (pydata#4435) Small updates to How-to-release + lint (pydata#4436) Fix doctests (pydata#4439) add a ci for doctests (pydata#4437) preserve original dimension, coordinate and variable order in ``concat`` (pydata#4419) Fix for h5py deepcopy issues (pydata#4426) Keep the original ordering of the coordinates (pydata#4409) Clearer Vectorized Indexing example (pydata#4433) Revert "Fix optimize for chunked DataArray (pydata#4432)" (pydata#4434) Fix optimize for chunked DataArray (pydata#4432) fix doc dataarray to netcdf (pydata#4424) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Another attempt to fix #3698. The issue with my fix in is that we hit
Variable._dask_finalizein bothdask.optimizeanddask.persist. Wewant to do the culling of unnecessary tasks (
test_persist_Dataset) butonly in the persist case, not optimize (
test_optimize).isort . && black . && mypy . && flake8whats-new.rst