ElasticSearch 6 support

[pschulten]

We're logging to ES 6 and the heapster export fails with:

W1211 13:30:19.368537 1 driver.go:75] Failed to export data to ElasticSearch sink: elastic: Error 400 (Bad Request): Failed to parse mapping [general]: No handler for type [string] declared on field [cluster_name] [type=mapper_parsing_exception]

[tstoermer]

ES 6 does no longer support multiple types in mappings per index. The metrics must be merged to one type or written to different indices.
https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking-changes-6.0.html

There is also a new version of the used ES client
https://github.com/olivere/elastic

[DirectXMan12]

cc @AlmogBaku @huangyuqi @andyxning

[AlmogBaku]

I'll try to find the time to make it supports ES6. If someone can pick it up please comment

[AlmogBaku]

Guys we need to decide what the new schema should be for ES6 support. You're welcome to raise ideas here

[Adriien-M]

We are looking for this update, what is the status of this issue?

[f0]

any progress to support elasticsearch 6?

[Freyert]

@AlmogBaku I'm postulating that these tests represent the original schema?

@tstoermer is this the documentation you were referencing?

Indices created here.
Sink Code is Here.

[tstoermer]

@Freyert
Right, just wanted to point the official breaking changes - linking to the details you mentioned.

[ihardzeenka]

Hi guys. About schema for ES 6.
Now you have single index with mapping types like cpu, memory, events and etc.
According to ES documentation there is 2 options here (https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html):

1. Simply split single index into multiple indexes like: heapster-cpu , heapster-memory and etc.
2. Merge types into single type (I don't really like that option)
   I don't know go language but looking into your code I believe it is not a big change to simply add existing type name into index name.

[MnrGreg]

Would be great to get something going here!

[jimk-osu]

+1

Would having a temporary countermeasure of single typing be OK with the goal of having a better solution as a permanent countermeasure?

I agree with @ihardzeenka that single typing may not be the right way, but it in the mean time a working solution would be best.

[DirectXMan12]

Heapster is now deprecated, so we will no longer be adding new features or sinks. Please see https://github.com/kubernetes/heapster/blob/master/docs/deprecation.md#initial-deprecation-kubernetes-111 for more information. I apologize for any in progress sinks or features that will now no longer be merged.

[jhorwit2]

@DirectXMan12 what's the deprecation plan for eventer since that's not included functionality in the metrics-server?

[smelchior]

we do currently have the same issue, maybe https://www.elastic.co/guide/en/beats/metricbeat/6.x/metricbeat-metricset-kubernetes-event.html can be used as an alternative?

[DirectXMan12]

yeah, there's a few different solutions, depending on what your backend is. Someone could also try to adapt https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/event-exporter, or fork eventer.