-
-
Notifications
You must be signed in to change notification settings - Fork 18.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PERF: perform reductions block-wise #29847
Changes from 7 commits
d1d07ff
9469400
038697a
94a2ee1
9e21d77
237253a
8757c8a
a1b653d
f8c3d24
ebb33c1
c0eb05c
4a16663
e4c0466
3e7da1e
9370b1a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -345,6 +345,37 @@ def _verify_integrity(self): | |
"tot_items: {1}".format(len(self.items), tot_items) | ||
) | ||
|
||
def reduce(self, func, *args, **kwargs): | ||
# If 2D, we assume that we're operating column-wise | ||
if self.ndim == 1: | ||
# we'll be returning a scalar | ||
blk = self.blocks[0] | ||
return func(blk.values, *args, **kwargs) | ||
|
||
res = {} | ||
for blk in self.blocks: | ||
bres = func(blk.values, *args, **kwargs) | ||
if np.ndim(bres) == 0 and blk.shape[0] != 1: | ||
# i.e. we reduced over all axes and not just one; re-do column-wise | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't quite understand this case. How do we get here? re-calling There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IIRC it was when we have axis=None There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. after some digging, this appears to be coming from nanops funcs that are either not getting axis passed or are not handling it correctly. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. updated to make this check unnecessary |
||
new_res = { | ||
blk.mgr_locs.as_array[i]: func(blk.values[i], *args, **kwargs) | ||
for i in range(len(blk.values)) | ||
} | ||
elif np.ndim(bres) == 0: | ||
# EA | ||
assert blk.shape[0] == 1 | ||
new_res = zip(blk.mgr_locs.as_array, [bres]) | ||
else: | ||
assert bres.ndim == 1, bres.shape | ||
assert blk.shape[0] == len(bres) | ||
new_res = zip(blk.mgr_locs.as_array, bres) | ||
|
||
nr = dict(new_res) | ||
assert not any(key in res for key in nr) | ||
res.update(nr) | ||
|
||
return res | ||
|
||
def apply( | ||
self, | ||
f, | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Planning to keep these asserts in?