Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement] validate spark connector on spark 3.1.2 #3007

Closed
Tracked by #1574
FANNG1 opened this issue Apr 18, 2024 · 1 comment
Closed
Tracked by #1574

[Improvement] validate spark connector on spark 3.1.2 #3007

FANNG1 opened this issue Apr 18, 2024 · 1 comment
Assignees
Labels
improvement Improvements on everything

Comments

@FANNG1
Copy link
Contributor

FANNG1 commented Apr 18, 2024

What would you like to be improved?

validate spark connector on spark 3.1.2

How should we improve?

No response

@FANNG1 FANNG1 added the improvement Improvements on everything label Apr 18, 2024
@FANNG1 FANNG1 self-assigned this Apr 18, 2024
@jerryshao jerryshao added this to the Gravitino June Release milestone Apr 24, 2024
@FANNG1
Copy link
Contributor Author

FANNG1 commented May 8, 2024

spark connector couldn't compile or run on spark 3.1

  1. the minimum spark version of the Kyuubi spark hive connector is 3.3, it couldn't compile or start up in spark 3.1 envriment. this's the biggest challenge.
  2. spark connector compile failed for API change, this could be resolved by some methods.
    a. create table API change
    b. not using SortedBucketTransform to represent the bucket table.
    c. not support UpdateColumnDefaultValue
    d. dropNamespace API change
    e. deleteColumn API change
    f. doesn't support timestampNTZ data type.
    g. doesn't support FunctionCatalog, this mainly affect Iceberg catalog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvements on everything
Projects
None yet
Development

No branches or pull requests

2 participants