Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rolling.construct: Add sliding_window_view_kwargs to pipe arguments down to sliding_window_view #9720

Merged
merged 12 commits into from
Nov 18, 2024
Merged

rolling.construct: Add sliding_window_view_kwargs to pipe arguments down to sliding_window_view #9720

merged 12 commits into from
Nov 18, 2024

Conversation

dcherian
Copy link
Contributor

@dcherian dcherian commented Nov 5, 2024 •
edited
Loading

Closes #9550
xref #4325

  • Tests added
  • User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @phofl

Blocked till the next dask release, so we can verify tests

@dcherian dcherian force-pushed the rolling-auto-rechunk branch 4 times, most recently from 223390c to ae9e8c0 Compare November 5, 2024 21:12
@dcherian dcherian force-pushed the rolling-auto-rechunk branch from ae9e8c0 to 1bdd60b Compare November 5, 2024 21:16
phofl
phofl reviewed Nov 5, 2024
View reviewed changes
Copy link
Contributor

@phofl phofl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!

@max-sixty
Copy link
Collaborator

I don't have much context here, but should we always do this if possible, rather than add an option? Or we want to allow an escape hatch for instances where copies are made?

@dcherian
Copy link
Contributor Author

dcherian commented Nov 5, 2024

It is on by default in the public APIs (construct and reduce). Advanced users can opt-out. For internal uses, where copies are not made, I set it to False by default (in _reduce_method) and opt-in to rechunking when necessary for each individual method.

I think we should expose it in construct since that is a thin wrapper around sliding_window_view. Are you concerned about exposing it in reduce?

@max-sixty
Copy link
Collaborator

No! Generally keen on keeping the API small but totally reasonable here!

@dcherian
Copy link
Contributor Author

dcherian commented Nov 6, 2024

@max-sixty one option would be to add sliding_window_view_kwargs to forward arbitrary kwargs down to the implementing library. For example, numpy has writeable which someone might want to set (we do not allow this currently)

@dcherian dcherian changed the title sliding_window_view: add new automatic_rechunk kwarg rolling.construct: Add sliding_window_kwargs to pipe arguments down to sliding_window_view Nov 9, 2024
@dcherian dcherian changed the title rolling.construct: Add sliding_window_kwargs to pipe arguments down to sliding_window_view rolling.construct: Add sliding_window_kwargs to pipe arguments down to sliding_window_view Nov 9, 2024
@dcherian
Merge branch 'main' into rolling-auto-rechunk
f6dc60b 
* main:
  Compress PNG files (pydata#9747)
  Dispatch to Dask if nanquantile is available (pydata#9719)
@dcherian
Add one more
236ae38
@dcherian
Copy link
Contributor Author

dcherian commented Nov 9, 2024

I went with sliding_window_kwargs to save some typing...

@dcherian dcherian marked this pull request as ready for review November 9, 2024 19:21
@dcherian dcherian added needs review topic-rolling topic-chunked-arrays Managing different chunked backends, e.g. dask labels Nov 9, 2024
@dcherian dcherian mentioned this pull request Nov 12, 2024
Merged
4 tasks
@dcherian dcherian requested a review from max-sixty November 13, 2024 13:42
@dcherian
Copy link
Contributor Author

@max-sixty any more thoughts here?

@dcherian dcherian added the plan to merge Final call for comments label Nov 15, 2024
@dcherian dcherian enabled auto-merge (squash) November 18, 2024 16:03
@dcherian dcherian changed the title rolling.construct: Add sliding_window_kwargs to pipe arguments down to sliding_window_view rolling.construct: Add sliding_window_view_kwargs to pipe arguments down to sliding_window_view Nov 18, 2024
@dcherian dcherian mentioned this pull request Nov 18, 2024
Merged
6 tasks
@dcherian dcherian merged commit 993300b into pydata:main Nov 18, 2024
29 checks passed
@dcherian dcherian deleted the rolling-auto-rechunk branch November 18, 2024 18:14
dcherian added a commit that referenced this pull request Nov 19, 2024
@dcherian
Merge branch 'main' into backend-indexing
ab20531 
* main: (24 commits)
  Bump minimum versions (#9796)
  Namespace-aware `xarray.ufuncs` (#9776)
  Add prettier and pygrep hooks to pre-commit hooks (#9644)
  `rolling.construct`: Add `sliding_window_kwargs` to pipe arguments down to `sliding_window_view` (#9720)
  Bump codecov/codecov-action from 4.6.0 to 5.0.2 in the actions group (#9793)
  Buffer types (#9787)
  Add download stats badges (#9786)
  Fix open_mfdataset for list of fsspec files (#9785)
  add 'User-Agent'-header to pooch.retrieve (#9782)
  Optimize `ffill`, `bfill` with dask when `limit` is specified (#9771)
  fix cf decoding of grid_mapping (#9765)
  Allow wrapping `np.ndarray` subclasses (#9760)
  Optimize polyfit (#9766)
  Use `map_overlap` for rolling reductions with Dask (#9770)
  fix html repr indexes section (#9768)
  Bump pypa/gh-action-pypi-publish from 1.11.0 to 1.12.2 in the actions group (#9763)
  unpin array-api-strict, as issues are resolved upstream (#9762)
  rewrite the `min_deps_check` script (#9754)
  CI runs ruff instead of pep8speaks (#9759)
  Specify copyright holders in main license file (#9756)
  ...
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 19, 2024
@dcherian
Merge branch 'main' into groupby-shuffle
d533638 
* main:
  Bump minimum versions (pydata#9796)
  Namespace-aware `xarray.ufuncs` (pydata#9776)
  Add prettier and pygrep hooks to pre-commit hooks (pydata#9644)
  `rolling.construct`: Add `sliding_window_kwargs` to pipe arguments down to `sliding_window_view` (pydata#9720)
  Bump codecov/codecov-action from 4.6.0 to 5.0.2 in the actions group (pydata#9793)
  Buffer types (pydata#9787)
  Add download stats badges (pydata#9786)
  Fix open_mfdataset for list of fsspec files (pydata#9785)
  add 'User-Agent'-header to pooch.retrieve (pydata#9782)
  Optimize `ffill`, `bfill` with dask when `limit` is specified (pydata#9771)
  fix cf decoding of grid_mapping (pydata#9765)
  Allow wrapping `np.ndarray` subclasses (pydata#9760)
  Optimize polyfit (pydata#9766)
  Use `map_overlap` for rolling reductions with Dask (pydata#9770)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@phofl phofl phofl left review comments

@max-sixty max-sixty Awaiting requested review from max-sixty

Assignees
No one assigned
Labels
plan to merge Final call for comments topic-chunked-arrays Managing different chunked backends, e.g. dask topic-rolling
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

rolling(...).construct(...) blows up chunk size
3 participants
@dcherian @max-sixty @phofl