Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add template xarray object kwarg to map_blocks #3816

Merged
merged 29 commits into from
May 6, 2020
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
064b583
Allow providing template dataset to map_blocks.
dcherian Feb 25, 2020
1f14b11
Update dimension shape check.
dcherian Feb 26, 2020
045ae2b
Allow user function to add new unindexed dimension.
dcherian Feb 26, 2020
0e37b63
Add docstring for template.
dcherian Feb 26, 2020
5704e84
renaming
dcherian Feb 26, 2020
2c458e1
Raise nice error if adding a new chunked dimension,
dcherian Feb 26, 2020
dced076
Raise nice error message when expected dimension is missing on return…
dcherian Feb 26, 2020
717d900
Revert "Allow user function to add new unindexed dimension."
dcherian Mar 2, 2020
42a9070
Add test + fix output_chunks for dataarray template
dcherian Mar 3, 2020
64ba31f
typing
dcherian Mar 3, 2020
a68cb41
fix test
dcherian Mar 3, 2020
0bc3754
Add nice error messages when result doesn't match template.
dcherian Mar 6, 2020
ee66c88
blacken
dcherian Mar 6, 2020
d52dfd6
Add template kwarg to DataArray.map_blocks & Dataset.map_blocks
dcherian Mar 19, 2020
8ef47f6
minor error message fixes.
dcherian Mar 19, 2020
376f242
docstring updates.
dcherian Mar 19, 2020
d9029eb
bugfix for expected shapes when template is not specified
dcherian Mar 19, 2020
6f69955
Add map_blocks docs.
dcherian Mar 19, 2020
da62535
Update doc/dask.rst
dcherian Mar 19, 2020
4a355e6
Merge remote-tracking branch 'upstream/master' into map-blocks-schema
dcherian Mar 28, 2020
66fe4c4
Merge branch 'map-blocks-schema' of github.com:dcherian/xarray into m…
dcherian Mar 28, 2020
085ce9a
Merge remote-tracking branch 'upstream/master' into map-blocks-schema
dcherian Apr 30, 2020
0f741e2
refactor out slicer for chunks
dcherian May 1, 2020
869e62d
Check expected index values.
dcherian May 1, 2020
334e8d3
Raise nice error when template object does not have required number o…
dcherian May 1, 2020
4dd699e
doc updates.
dcherian May 1, 2020
27eb873
more review comments.
dcherian May 1, 2020
04a2c3c
Mention that attrs are taken from template.
dcherian May 1, 2020
4b92168
Add test and explicit point out that attrs is copied from template
dcherian May 5, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Mention that attrs are taken from template.
dcherian committed May 1, 2020
commit 04a2c3c1c16d332233602e3718248902c767b6ea
2 changes: 1 addition & 1 deletion doc/dask.rst
Original file line number Diff line number Diff line change
@@ -469,7 +469,7 @@ Here is a common example where automated inference will not work.
``func`` cannot be run on 0-shaped inputs because it is not possible to extract element 1 along a
dimension of size 0. In this case we need to tell :py:func:`map_blocks` what the returned result looks
like using the ``template`` kwarg. ``template`` must be an xarray Dataset or DataArray (depending on
what the function returns) with dimensions, shapes, chunk sizes, coordinate variables *and* data
what the function returns) with dimensions, shapes, chunk sizes, attributes, coordinate variables *and* data
variables that look exactly like the expected result. The variables should be dask-backed and hence not
incur much memory cost.

2 changes: 1 addition & 1 deletion xarray/core/parallel.py
Original file line number Diff line number Diff line change
@@ -170,7 +170,7 @@ def map_blocks(
xarray object representing the final result after compute is called. If not provided,
the function will be first run on mocked-up data, that looks like 'obj' but
has sizes 0, to determine properties of the returned object such as dtype,
variable names, new dimensions and new indexes (if any).
variable names, attributes, new dimensions and new indexes (if any).
'template' must be provided if the function changes the size of existing dimensions.

Returns