[feature](jni) support complex types in jni framework #24810
Conversation
AshinGau
changed the title
|
clang-tidy review says "All clean, LGTM! 👍" |
AshinGau
force-pushed
the
jni-complex-type
branch
from
7f85045 to
8994001
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
TeamCity be ut coverage result: |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
PR approved by anyone and no changes requested. |
| offsets_data[origin_size + i] = offsets[i] + start_offset; | ||
| } | ||
|
|
||
| // offsets[num_rows - 1] == offsets_data[origin_size + num_rows - 1] - start_offset |
There was a problem hiding this comment.
Can this line of code be deleted?
AshinGau
force-pushed
the
jni-complex-type
branch
from
1c09dd3 to
651ee06
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
Support complex types in jni framework, and successfully run end-to-end on hudi. ### How to Use Other scanners only need to implement three interfaces in `ColumnValue`: ``` // Get array elements and append into values void unpackArray(List<ColumnValue> values); // Get map key array&value array, and append into keys&values void unpackMap(List<ColumnValue> keys, List<ColumnValue> values); // Get the struct fields specified by `structFieldIndex`, and append into values void unpackStruct(List<Integer> structFieldIndex, List<ColumnValue> values); ``` Developers can take `HudiColumnValue` as an example.
Support complex types in jni framework, and successfully run end-to-end on hudi. ### How to Use Other scanners only need to implement three interfaces in `ColumnValue`: ``` // Get array elements and append into values void unpackArray(List<ColumnValue> values); // Get map key array&value array, and append into keys&values void unpackMap(List<ColumnValue> keys, List<ColumnValue> values); // Get the struct fields specified by `structFieldIndex`, and append into values void unpackStruct(List<Integer> structFieldIndex, List<ColumnValue> values); ``` Developers can take `HudiColumnValue` as an example.
Support complex types in jni framework, and successfully run end-to-end on hudi. ### How to Use Other scanners only need to implement three interfaces in `ColumnValue`: ``` // Get array elements and append into values void unpackArray(List<ColumnValue> values); // Get map key array&value array, and append into keys&values void unpackMap(List<ColumnValue> keys, List<ColumnValue> values); // Get the struct fields specified by `structFieldIndex`, and append into values void unpackStruct(List<Integer> structFieldIndex, List<ColumnValue> values); ``` Developers can take `HudiColumnValue` as an example.
The offset in map type column is int64, but #24810 has put as int32, causing error like:
The offset in map type column is int64, but #24810 has put as int32, causing error like:
The offset in map type column is int64, but apache#24810 has put as int32, causing error like:
Proposed changes
Support complex types in jni framework, and successfully run end-to-end on hudi.
How to Use
Other scanners only need to implement three interfaces in
ColumnValue:Developers can take
HudiColumnValueas an example.Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...