-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent Hive from returning table handles for Hudi tables #16605
Prevent Hive from returning table handles for Hudi tables #16605
Conversation
/test-with-secrets sha=3fbb7a66c17124359eb5bb392ac84066372a445f |
@findepi I guess we can merge, cant we ? |
Does this need release notes? cc @findepi |
@colebow nope |
i don't think trino-hive connector can be used to read Hudi tables. For instance, I believe it lacks necessary libraries on its classpath |
@findepi Hudi tables are queryable using the Hive connector provided users have the hudi presto/trino bundle under the hive plugin directory. This is done through the custom input format integration. This change has been there even before Trino was forked off Presto. |
Landing this without us having the necessary migration path to hudi-connector, will cause pain & break production queries for users out there. @bvaradar this is going to break production at Robinhood. We are still waiting on/working through PR reviews there to get hudi-connector on parity. |
cc @electrum Can we please be notified about such breaking changes in the future, so we can be part of the conversation. |
@homar Trino-hudi connector is not efficient enough and can't trim the search scope using metadata or metastore efficiently resulting in basically a full table scan. We have been using hive-connector for a long time to read hudi tables and it just works for us. I had to undo this change on our side to get things back to normal. |
Description
Additional context and related issues
Release notes
( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: