Upgrade repository-hdfs to Hadoop 3 #76897
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elasticmachine
added
the
Team:Data Management
|
Pinging @elastic/es-core-features (Team:Core/Features) |
masseyke
changed the title
|
@elasticmachine update branch |
|
I think we should probably split out the Hadoop 3 upgrade from the testing additions for over-the-wire hdfs encryption, just to keep things tidy. |
|
@elasticmachine update branch |
masseyke
changed the title
|
@elasticmachine update branch |
|
@elasticmachine run elasticsearch-ci/part-2 |
jbaiera
approved these changes
Sep 23, 2021
plugins/repository-hdfs/build.gradle
Outdated
Comment on lines
33
to
34
| final int minTestedHadopoVersion = 2; | ||
| final int maxTestedHadopoVersion = 3; |
| @@ -95,6 +97,7 @@ tasks.named("dependencyLicenses").configure { | |||
|
|
|||
| tasks.named("integTest").configure { | |||
| dependsOn(project.tasks.named("bundlePlugin")) | |||
| enabled = false | |||
There was a problem hiding this comment.
oh I see, do we discard this test and just create new ones down below for hadoop v2 + v3?
There was a problem hiding this comment.
Yeah, I wanted to be very explicit about hadoop 2 vs hadoop 3. Not sure if this was the best way to go about it, but it works.
Yeah I had the same thought. I changed it locally just to make sure there were no problems (there weren't) but left it at 2 since that's what's currently tested. |
|
@elasticmachine update branch |
arteam
added a commit
to arteam/elasticsearch
that referenced
this pull request
Sep 24, 2021
In elastic#76897 the `hadoop-common` module was renamed to `hadoop-client-ide`, but the change wasn't reflected in `elasticsearch.ide.gradle` script. Because of that an IDE import started failing with the `Task with path ':plugins:repository-hdfs:hadoop-common:shadowJar" not found` error. Fix the import by setting the correct module name in the `buildDependencyArtifacts` task.
arteam
added a commit
that referenced
this pull request
Sep 24, 2021
In #76897 the `hadoop-common` module was renamed to `hadoop-client-ide`, but the change wasn't reflected in `elasticsearch.ide.gradle` script. Because of that an IDE import started failing with the `Task with path ':plugins:repository-hdfs:hadoop-common:shadowJar" not found` error. Fix the import by setting the correct module name in the `buildDependencyArtifacts` task.
arteam
added a commit
to arteam/elasticsearch
that referenced
this pull request
Sep 27, 2021
In elastic#76897 the `hadoop-common` module was renamed to `hadoop-client-ide`, but the change wasn't reflected in `elasticsearch.ide.gradle` script. Because of that an IDE import started failing with the `Task with path ':plugins:repository-hdfs:hadoop-common:shadowJar" not found` error. Fix the import by setting the correct module name in the `buildDependencyArtifacts` task.
masseyke
added a commit
that referenced
this pull request
Oct 6, 2021
…on fails (#78409) Until recently, if a user configured over-the-wire encryption for repository-hdfs they would get an exception. That was fixed in an upgraded ticket in two ways: (1) jvm permissions were opened up for haddop2, and (2) support for the hadoop 3 hdfs client was added. This commit adds configuration to a couple of integration tests so that they fail if over-the-wire encryption is not working. Relates #76897 #76734
masseyke
added a commit
that referenced
this pull request
Oct 6, 2021
This upgrades the repository-hdfs plugin to hadoop 3. Tests are performed against both hadoop 2 and hadoop 3 HDFS. The advantages of using the hadoop 3 client are: Over-the-wire encryption works (tests coming in an upcoming PR). We don't have to add (or ask customers to add) additional jvm permissions to the elasticsearch jvm It's compatible with java versions higher than java 8 Relates #76897
arteam
added a commit
to arteam/elasticsearch
that referenced
this pull request
Oct 11, 2021
In elastic#76897 the `hadoop-common` module was renamed to `hadoop-client-ide`, but the change wasn't reflected in `elasticsearch.ide.gradle` script. Because of that an IDE import started failing with the `Task with path ':plugins:repository-hdfs:hadoop-common:shadowJar" not found` error.
arteam
added a commit
that referenced
this pull request
Oct 11, 2021
In #76897 the `hadoop-common` module was renamed to `hadoop-client-ide`, but the change wasn't reflected in `elasticsearch.ide.gradle` script. Because of that an IDE import started failing with the `Task with path ':plugins:repository-hdfs:hadoop-common:shadowJar" not found` error.
jakelandis
added
v8.0.0-beta1
:Distributed Coordination/Snapshot/Restore
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This upgrades the repository-hdfs plugin to hadoop 3. Tests are performed against both hadoop 2 and hadoop 3 HDFS. The advantages of using the hadoop 3 client are:
Over-the-wire encryption works (tests coming in an upcoming PR).
We don't have to add (or ask customers to add) additional jvm permissions to the elasticsearch jvm
It's compatible with java versions higher than java 8