Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add JNI for ORC/Parquet writer compression statistics #13376

Merged
merged 12 commits into from
May 18, 2023
Merged

Add JNI for ORC/Parquet writer compression statistics #13376

merged 12 commits into from
May 18, 2023

Conversation

ttnghia
Copy link
Contributor

@ttnghia ttnghia commented May 18, 2023 •
edited
Loading

This adds JNI to expose compression statistics for ORC and Parquet writer, which was implemented in #13294.

For doing so, TableWriter has been changed from an interface into an abstract class. Related derived writer classes are also updated accordingly.

Closes #13252.

@github-actions github-actions bot added the Java Affects Java cuDF API. label May 18, 2023
@ttnghia ttnghia changed the title Jni compression statistics Add JNI for ORC/Parquet writer compression statistics May 18, 2023
@ttnghia ttnghia self-assigned this May 18, 2023
@ttnghia ttnghia added feature request New feature or request 3 - Ready for Review Ready for review by team Spark Functionality that helps Spark RAPIDS non-breaking Non-breaking change labels May 18, 2023
@ttnghia ttnghia marked this pull request as ready for review May 18, 2023 04:25
@ttnghia ttnghia requested a review from a team as a code owner May 18, 2023 04:25
@ttnghia ttnghia force-pushed the jni_compression_statistics branch from 49efaf7 to 68b31ea Compare May 18, 2023 04:32
revans2
revans2 approved these changes May 18, 2023
View reviewed changes
Copy link
Contributor

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few nits, but nothing important enough to hold up this PR.

Also the JNI code to return the stats could be a lot more efficient, but this is really minor because it is going to be called once per output file. So it should be totally fine.

java/src/main/native/src/CudfJni.cpp Outdated Show resolved Hide resolved
java/src/main/native/src/TableJni.cpp Show resolved Hide resolved
@ttnghia
Copy link
Contributor Author

ttnghia commented May 18, 2023

/merge

@rapids-bot rapids-bot bot merged commit 0c142f1 into rapidsai:branch-23.06 May 18, 2023
@ttnghia ttnghia deleted the jni_compression_statistics branch May 18, 2023 22:22
@ttnghia ttnghia mentioned this pull request May 22, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revans2 revans2 revans2 approved these changes

Assignees

@ttnghia ttnghia

Labels
3 - Ready for Review Ready for review by team feature request New feature or request Java Affects Java cuDF API. non-breaking Non-breaking change Spark Functionality that helps Spark RAPIDS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Return compression statistics from ORC and Parquet writers
2 participants
@ttnghia @revans2