-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build: Exclude zstd-jni from iceberg-spark3-runtime #3058
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1. We have tested this internally and it is needed as this can't be shaded (it's a dependency that comes from parquet 1.12 by means of spark).
cc @RussellSpitzer, please take a look if you have time. |
Confirming what @kbendick said we made this change internally a while back. If everyone else is ok with it I think we should remove the library but this is the kind of change I think we need more folks to check out before we merge just incase we are missing someone's prod usage. @rdblue + @jackye1995 + @openinx |
Sounds good to me. |
@jackye1995 would u pls take a look as well? |
Thanks, @pan3793! I went ahead and merged this since it's been a few days without comment. |
I added this to 0.13.0 milestone as it appears it didn't ship with 0.12.1. I'm not sure if this merits a patch release to 0.12.2... I'd say since we're in progress towards 0.13.0 that it would be easier to hold off. |
spark-core
already has the dependencyzstd-jni
, it should be exclude fromiceberg-spark3-runtime
.