Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

Add Hadoop 3.3 profile to pom.xml #913

Closed
HongW2019 opened this issue May 10, 2022 · 1 comment
Closed

Add Hadoop 3.3 profile to pom.xml #913

HongW2019 opened this issue May 10, 2022 · 1 comment
Labels
enhancement New feature or request feature

Comments

@HongW2019
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We notice that official Spark-3.2.1 distribution is built with Hadoop-3.3.1 as below. Currently, our package for Spark-3.2.1 is built with Hadoop-3.2.x.

image

Describe the solution you'd like

We'd better add Hadoop 3.3 profile to pom.xml and ship a package built with the same Hadoop version as official Spark distribution in the next release.
@HongW2019 HongW2019 added the enhancement New feature or request label May 10, 2022
@HongW2019
Copy link
Contributor Author

@yao531441 has recompiled Gazelle with <hadoop.version>3.3.1</hadoop.version> for Gazelle v1.3.1 and enabled Gazelle on Spark-3.2.1 + Hadoop-3.3.1 AWS clusters with S3. It works well with this package.

@weiting-chen Please help track this issue, thanks.

@HongW2019 HongW2019 mentioned this issue Jun 10, 2022
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
enhancement New feature or request feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@HongW2019 @haojinIntel