Merge Index and Series binops #8166
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…aFrame." This reverts commit a49f3748c7d19fa39181df05549c7a7dcced4a9a.
…ently still a pass-through to Series).
vyasr
added
Python
|
Here's a snapshot of the performance improvements for doing binary operations. Each line should be compared to the line directly below it so that the only difference is whether it's |
kkraus14
requested changes
May 5, 2021
Codecov Report
@@ Coverage Diff @@
## branch-0.20 #8166 +/- ##
===============================================
- Coverage 82.88% 82.86% -0.03%
===============================================
Files 103 104 +1
Lines 17668 17884 +216
===============================================
+ Hits 14645 14820 +175
- Misses 3023 3064 +41
Continue to review full report at Codecov.
|
kkraus14
approved these changes
May 6, 2021
kkraus14
added
5 - Ready to Merge
|
@gpucibot merge |
rapids-bot bot
pushed a commit
that referenced
this pull request
May 18, 2021
Continuation of #8115 and #8166. Moves more logic out of the Index/Series classes into the new common parent class to reduce code duplication and ensure feature parity. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - https://github.com/brandon-b-miller - Michael Wang (https://github.com/isVoid) URL: #8253
rapids-bot bot
pushed a commit
that referenced
this pull request
Jul 15, 2021
…e support (#8598) This PR builds on #8166 to introduce a common code path for binary operations between all Frame types, not just Index and Series. The central component is a new internal API for operations between a sequence of pairs of columns (or column-like objects) that subclasses of Frame can call after performing any preprocessing necessary on the operands. The result is (in addition to the benefits of reducing total code and addressing some tech debt) substantial performance improvements for DataFrame binops, new features in the form of supporting binary operations between a much wider class of data types, and some minor bug fixes that were previously quite challenging to address. Resolves #4536. ***Performance*** By avoiding calling through to Series binary operations to perform binops between DataFrames (which requires lots of ColumnAccessor operations, construction of Series objects, and lots of indexing), we gain substantial improvements in DataFrame binops. On average I see at minimum a 20% improvement in binops between DataFrames, with significantly larger improvements (up to 60%) depending on data size, nullability, and other related factors. The performance impact of the new code paths for Series and Index binops is negligible, since the changes effectively amount to a loop over a single element dictionary instead of just operating on that element directly. ***New features*** The centralized binop code made it easy to insert an `as_column` at an appropriate point for the right operand to a binop, which enables support for a wide range of other objects. For instance, it is now possible to add numpy arrays or pandas Series objects to a cudf Series, which was not previously possible. This code also appropriately handles the return of `NotImplemented` for all data types, enabling reflected operations for other libraries that build on `cudf`. ***Bug fixes*** This PR improves the handling of NULL and NaN columns in a number of scenarios, for instance where a column is missing from one input or the other. Additionally, this PR fixes the behavior of RangeIndexes when they are multiplied by integers (pandas returns RangeIndex objects in that case). This fix is largely independent of the new code since it essentially intercepts scalar ints up front and reconstructs a new object. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - https://github.com/brandon-b-miller URL: #8598
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on #8115, moving all binary operations from
IndexandSeriesinto theSingleColumnFrameclass. It really should be a negative LOC change, but it doesn't look like it for two reasons: 1)Indexobjects require some special handling due to the awkwardness of needing to return the right type ofIndex, which is frequently not the type that is being operated on (e.g.RangeIndex + RangeIndexresults in anInt64Index), and that's something we'll want to refactor in a future PR, and 2) I've added a significant number of comments both in the form of docstrings and to give context for the issues arising from (1). This PR also significantly speeds up all binary operations forIndexobjects because it removes the round-tripping of data fromIndex->Series->Indexthat was previously being done to implement binary operations. The percent speedup depends on how expensive the operation itself is, but having tested for a number of data sizes it is >=15%, ranging up to 40% for simpler operations like__ne__. Benchmarks to follow.