Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement skipna in xr.quantile for speedup #3843

Closed
aaronspring opened this issue Mar 6, 2020 · 1 comment · Fixed by #3844
Closed

Implement skipna in xr.quantile for speedup #3843

aaronspring opened this issue Mar 6, 2020 · 1 comment · Fixed by #3844

Comments

@aaronspring
Copy link
Contributor

aaronspring commented Mar 6, 2020

xr.quantile uses np.nanquantile which is slower than np.quantile but only needed when ignoring nans is needed. Adding skipna as kwarg would lead to a speedup for many use-cases.

MCVE Code Sample

np.quantile is much faster than np.nanquantile

control = xr.DataArray(np.random.random((50,256,192)),dims=['time','x','y'])
%time _ = control.quantile(dim='time',q=q)
CPU times: user 4.14 s, sys: 61.4 ms, total: 4.2 s
Wall time: 4.3 s


%time _ = np.quantile(control,q,axis=0)
CPU times: user 47.1 ms, sys: 4.27 ms, total: 51.4 ms
Wall time: 52.6 ms

%time _ = np.nanquantile(control,q,axis=0)
CPU times: user 3.18 s, sys: 21.4 ms, total: 3.2 s
Wall time: 3.22 s

Expected Output

faster xr.quantile:

%time _ = control.quantile(dim='time',q=q)
CPU times: user 4.95 s, sys: 34.3 ms, total: 4.98 s
Wall time: 5.88 s

%time _ = control.quantile(dim='time',q=q, skipna=False)
CPU times: user 85.3 ms, sys: 16.7 ms, total: 102 ms
Wall time: 127 ms

Problem Description

np.nanquantile not always needed

Versions

Output of `xr.show_versions()` xr=0.15.1
@max-sixty
Copy link
Collaborator

Great -- we'd take a PR for this. Cheers @aaronspring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants