[NSE-581] Improve GetArrayItem(Split()) performance #933
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/oap-project/native-sql-engine/issues Then could you also rename commit message and pull request title in the following format? See also: |
Signed-off-by: Yuan Zhou <[email protected]>
zhouyuan
force-pushed
the
wip_splitpart
branch
2 times, most recently
from
8810e41 to
5a41c36
Compare
| copy(str = newFirst, regex = newSecond, limit = newThird) | ||
| } | ||
|
|
||
| case class StringSplitPart(str: Expression, regex: Expression, limit: Expression, index: Expression) |
There was a problem hiding this comment.
nit: I prefer to register StringSplitPart through UDFRegistration, rather than adding whole expression code to gazelle, which will affect the difficulty of upgrading spark versions later.
There was a problem hiding this comment.
yes, these codes will be moved to Spark321 shim layers or dynamically register this on session starts - will also try to upstream SplitPart to upstream Spark
| StringSplit(str.expr, Literal(pattern), Literal(limit)) | ||
| } | ||
| def splitpart(str: Column, pattern: String, limit: Int, index: Int): Column = withExpr { | ||
| StringSplitPart(str.expr, Literal(pattern), Literal(limit), Literal(index)) |
Signed-off-by: Yuan Zhou <[email protected]>
|
To align with spark's behavior, please apply this patch to arrow: oap-project/arrow#107. |
Signed-off-by: Yuan Zhou <[email protected]>
zhouyuan
changed the title
Signed-off-by: Yuan Zhou <[email protected]>
Signed-off-by: Yuan Zhou <[email protected]>
This reverts commit 4cda7d5.
What changes were proposed in this pull request?
The implementation converts
GetArrayItem(Split())intoSplitPart()as Gazelle does not support Array based functions yet.How was this patch tested?
pass jenkins