Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE] Support Spark 4.0 and Deprecate JDK 8 #8574

Open
weiting-chen opened this issue Jan 20, 2025 · 6 comments
Open

[CORE] Support Spark 4.0 and Deprecate JDK 8 #8574

weiting-chen opened this issue Jan 20, 2025 · 6 comments
Labels
enhancement New feature or request

Comments

@weiting-chen
Copy link
Contributor

weiting-chen commented Jan 20, 2025

Description

Current Spark and JDK support status are in below
Spark 3.4.4 support JDK8(8u362) or above version
Spark 3.5.4 support JDK8(8u371) or above version
Spark 4.0 can only support JDK 11 and 17
For better CI resource usage, we can consider to deprecate JDK 8 support when adding Spark 4.0 support.
The plan to support Spark 4.0 will be initialized in Gluten 1.4 and done the work in Gluten 1.5.

One other consideration is in Iceberg cannot support JDK 8 with Spark 3.4.4, we have Iceberg UT may consider to separate the UT testing for Spark 3.4.4 and Iceberg or just support JDK 11/17 for better alignment.

@weiting-chen weiting-chen added the enhancement New feature or request label Jan 20, 2025
@Yohahaha
Copy link
Contributor

would you send a email to gluten dev mail list to notify all developers?

@weiting-chen
Copy link
Contributor Author

Yes, will send a mail to dev mail list.

@FelixYBW
Copy link
Contributor

@weiting-chen we may move our CI to JDK17 now but still leave the legacy jdk8 support code there, we still accept PRs for JDK8. Customer who are using jdk8 needs to run the CI from their internal env.

Once we release Gluten1.4, we may remove the JDK8 support completely.

@FelixYBW FelixYBW changed the title Support Spark 4.0 and Deprecate JDK 8 [CORE] Support Spark 4.0 and Deprecate JDK 8 Jan 21, 2025
@jackylee-ch
Copy link
Contributor

jackylee-ch commented Jan 22, 2025

Once we release Gluten1.4, we may remove the JDK8 support completely.

This means we would drop Spark 3.2 and Spark 3.3 support after Gluten 1.4?

@zhztheplayer
Copy link
Member

Is it possible to build Gluten with higher JDK version only for -P spark-4.0?

@xumingming
Copy link
Contributor

Can we keep the support for Spark 3.3?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

6 participants