-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Elasticsearch 8.x support #3571
Comments
Would you like to submit a PR? Besides the logic fix it would need to add v8 to the CI steps. |
Sure, I'll give it a try. |
I spent time investigating the effort in achieving it. I think this is going to be difficult now that we use same elasticsearch client olivere/elastic to support both of our elasticsearch and opensearch implementations. Major challenge: Not sure how we should proceed. cc @yurishkuro |
I have tried latest helm operator installation and setting this parameter didn't result in anything better. |
Would it be possible for Jaeger to add a separate docker image/branch which maintains opensearch support and the main branch getting support for Elasticsearch 8.x ? |
hope to support elasticsearch 8.1.x |
We are working on bootstrapping Jaeger v2 (#3500) which will be based on OTEL collector. This will allow us to have ES exporter per ES version. |
Is there any workaround currently available, e.g., creating the index manually? I tried the OP's suggested workaround (with elastic 8.1.3) and I get the following error:
|
I would be thankful for any workaround too. We need to upgrade our elasticsearch cluster but we are stuck because of this problem |
we also need this asap!!!!!!1 |
we also need this asap! +1 |
Hi, Workaround Force the ES Version to 7 via --es.version=7 and es.create-index-templates: false Add add this two indexes. If some of them fails related to "index priority" just change it
|
It Works ! thanks 👍 |
As part of adding support for the newer index template API, it might be nice to have the option for Jaeger to create component templates instead of index templates. With the current API we have a couple of other index templates targetting jaeger indices that all get merged together to perform the following modifications:
With the new index template API this won't be possible as only one index template wll be chosen (highest priority) and even if Jaeger supporting passing I'm not sure if this is a common use case or not, otherwise we may choose to disable Jaeger index template creation and just managed them entirely ourselves, however pulling in any new field mappings etc. automatically when upgrading Jaeger is handy |
Thanks @sergiomcalzada that worked:
|
Fyi, https://github.com/jaegertracing/spark-dependencies is not working with es8.x also. That means it cannot really be used in production IMHO |
We are experiencing problems connecting Jaeger to our existing ES setup due to this issue. Ex:
But no matter I still see this error relating to the template index patterns:
I have also tried removing the leading * from the templates, but the error message stayed the same which leads me to believe this is still on the Jaeger end of things. Anyone have further workaround info/updates on ES8 support? |
Looks like this was some yaml mangling - No matter what was specified, checking the config state on the ingester showed
Solution: quote the false
Which becomes:
|
@Ashmita152 I came across olivere/elastic#1533, where the author of the library unequivocally states:
So there seems to be little point in upgrading to v7 of the library as it won't support v8 of ES. Unlike our Cassandra implementation that uses an internal abstraction over gocql driver, the ES implementation has a large exposure to many different types from ESv7 will be EOL-ed by 2023-08-01. |
@yurishkuro Could you please clarify whether Jaeger is going to support ElasticSearch v8.+? At least, if there are any plans to do so in near times? This issue blocks our team and forces either to use outdated infrastructure or switch to another distributed tracing system. |
@yurishkuro @pavolloffay Any thoughts about my previous comment? |
Facing issues after upgrading to ES8. Jaeger Collector startup errors can be fixed with workaround by adding --es.version=7 and --es.create-index-templates=false command parameters. But then Jaeger Query is erroring with Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]. Thanks to sergiomcalzada, the suggested manual index template creation is helping to restore Jaeger Query functionality. I am applying suggested templates using Kibana->Dev Tools->Console. |
|
Have you solved it? How did you solve it? |
I have the same problem with you |
I think the |
for the most part our system is working, however the init phase of rollover still returns: the actual rollover returns: is there a known workaround for this? Also, is there a timeline on when actual support will be ready for elasticsearch 8? thank you. |
Thanks. It works.感谢方法有效。 |
@sergiomcalzada, on 8.7 I had to delete from both templates, should I replace it with something else? |
Unfortunately, none of the others have operators as robust as ECK. |
@jpuskar what are your requirements in terms of features/dataset size? |
Hi. This topic is still relevant. Is there any plan to support Elasticsearch 8? The problem is exacerbated by the fact that there are currently no supported helm charts Elasticsearch 7. |
The First, execute the following command:
Next, copy the response content. Change Finally, execute the following command in ES:
This will update the template accordingly, notice the response warning. |
@henrygas2018 or you can just don't set prefix and everything will be work (if you created ILM policy before jaeger start)
|
Elastic v7 has ben EOL as of 1st of August. If someone wants to use jaeger properly, they would have to install an EOL version. Is there at least a timeline on when can we expect support for Elasticsearch 8? |
@fatihdogmus This has been proposed as a project for LFX mentorship: #4600 So someone's going to start working on it in Sep. |
Just in case someone bumps into this:
Any suggestions? |
Is there a way to create all the configurations in a fresh installation on |
You can apply all the workarounds in a fresh v8 (tested). You will need to create the index by your self, you can find the code for the indexes in a previous comment from me |
I did this, but there are missing aliases and ILM related configurations. |
## Which problem is this PR solving? - Resolves #3571 ## Description of the changes - Added index templates for esv8 - Added a esv8 client to handle the put index template request that is not currently possible by olivere/elastic - After these changes are merged , users no more need to use create.index-templates= false unless they want to add their custom index templates and force the es.version = 7 --------- Signed-off-by: bugslayer-332 <[email protected]> Signed-off-by: Yashwanth Reddy <[email protected]> Signed-off-by: Yuri Shkuro <[email protected]> Co-authored-by: bugslayer-332 <[email protected]> Co-authored-by: Yuri Shkuro <[email protected]>
Requirement - what kind of business use case are you trying to solve?
Elastic recently released Elasticsearch 8.0, by now they've released 8.1.
Problem - what in Jaeger blocks you from solving the requirement?
The ES index can't be created anymore, I think the legacy template api part is now completely deprecated:
The https://www.elastic.co/guide/en/elasticsearch/reference/8.1/migrating-8.0.html lists now (somewhere in the middle)
The [create or update index template API](https://www.elastic.co/guide/en/elasticsearch/reference/8.1/indices-templates-v1.html)’s template parameter has been removed. Details In 6.0, we deprecated the template parameter in create or update index template requests in favor of using index_patterns. Support for the template parameter is now removed in 8.0. Impact Use the create or update index template API's index_patterns parameter. Requests that include the template parameter will return an error.
I think the code falls back to the unversioned mappings if it does not find a specific mapping
So, in my opinion the code should not fall back to the jaeger-span.json mapping but instead use the
jaeger-span-7.json for newer versions, e.g. use the -7 jsons for 8 etc.
e.g. change line 42 from == 7 to >=7
jaeger/plugin/storage/es/mappings/mapping.go
Lines 40 to 46 in 7872d1b
Workaround
Force the ES Version to 7 via --es.version=7
The text was updated successfully, but these errors were encountered: