Updates to latest Zipkin supporting Elasticsearch 8.x #215
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This drops support of Elasticsearch 6.x, including writing to old index templates, as it is no longer possible with the driver that supports 7.x and 8.x. See elastic/elasticsearch-hadoop#2188 This also updates to the latest Spark 3.x version possible, currently 3.3.x. Note: we can't update to Scala 2.13 due to a conflict between connectors: (DataStax) Cassandra requires Spark 3.4 on Scala 2.13, but Elastic requires Spark 3.3. See elastic/elasticsearch-hadoop#2187 Signed-off-by: Adrian Cole <[email protected]>
codefromthecrypt
force-pushed
the
ES8
branch
from
b14b786 to
ad20993
Compare
Signed-off-by: Adrian Cole <[email protected]>
|
cc @xeera on notes about ES conflicts, but good news is that 7.x and 8.x pass tests! The main issue is moving up to a more recent scala+spark as that allows JDKs beyond 11 to work. |
added 2 commits
December 30, 2023 20:04
Signed-off-by: Adrian Cole <[email protected]>
Signed-off-by: Adrian Cole <[email protected]>
| <!-- TODO: can't update to 2.13 without updating ES driver --> | ||
| <!-- Use a common scala binary version, and the highest spark patch version. | ||
| Scala 2.13 is held back due to spark version conflicts until Elastic | ||
| updates their version. Once 2.13 works, we can also use JDK 17 per |
There was a problem hiding this comment.
Theoretically, you can get scala 2.12 to work on later JREs, but SPARK-33772 says it is not advised and to update to 2.13 first
|
going to release this as those trying ES 8 will immediately run into dependencies not working, yet. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This drops support of Elasticsearch 6.x, including writing to old index
templates, as it is no longer possible with the driver that supports 7.x
and 8.x.
See elastic/elasticsearch-hadoop#2188
This also updates to the latest Spark 3.x version possible, currently
3.3.x. Note: we can't update to Scala 2.13 due to a conflict between
connectors: (DataStax) Cassandra requires Spark 3.4 on Scala 2.13, but
Elastic requires Spark 3.3.
See elastic/elasticsearch-hadoop#2187