[elastic plugin] cluster health index-level stats + index node stats are merged into one measurement

[msoltyspl]

This was somewhat surprising. I noticed it by seeing clusterhealth-like fields in elasticsearch_indices and then after a bit of digging (essentially checking what is tagged with 'index') it turned out (and plugin code confirmed it) that elasticsearch_indices is essentially a "merge" of 2 queries:

• /_cluster/health?level=indices (tagged per index)
• /_nodes/stats/indices (tagged per node's host/id/name)

But the former is on cluster level, the latter is on node level. So mixing them is somewhat unusual choice. Both parts are also completely disjoint (no common fields)

Weren't they supposed to go (tagged per index) to elasticsearch_cluster_health ?

P.S.
A bit of a side not - but detailed per index stats can be obtained (and tagged with index name) by doing e.g. /_stats query on a node. It's also possible (though level option is documented only for index api) to do e.g. /_nodes/_local/stats/indices?level=indices . This would likely be more fitting for elasticsearch_indices - giving per node + per index stats.

[danielnelson]

Can you include an example of the expected metric in line protocol as well as the actual metric in line protocol?

[danielnelson]

Looked into this a little closer and it does feel like a bug. This would give some additional incentive to make the metric rename change which was also suggested in #2518. I encourage you to join in on the planning and review of that issue.

[danielnelson]

Prior to 3be111a the measurements were named elasticsearch_indices (/_nodes/stats) and indices (/_cluster/health?level=indices)

[danielnelson]

elasticsearch_cluster_health has some field overlap with the cluster health indices. We will want to make sure we can sum the fields easily.

[danielnelson]

I suggest we rename the indices level measurement to elasticsearch_cluster_health with the tags name containing the cluster name and index containing the name of the index. For reference, the current measurement containing cluster health stats has a name tag containing the name of the cluster.

This would be a breaking change.

@lpic10 @mhohara Does this sound okay?

[MatthewOHaraTR]

@danielnelson #2518 has the cluster information from that endpoint going under elasticsearch_cluster_health (tagged by cluster name), and the cluster stats indices metrics going under elasticsearch_cluster_health_indices (tagged with the index name). This is working well to allow differentiation between these two different types of data via Grafana. I suspect combining them under the same elasticsearch_cluster_health measurement would likewise work ok, although I don't understand why this would be necessary?

[danielnelson]

I have them as one measurement because fields are similar and they all represent cluster_health, is there a reason they should be separate?

[voiprodrigo]

Hi.

I have just started to use this plugin. I'm trying to use it to monitor a cluster with 4 master nodes, 4 hot nodes and 8 warm nodes. I was hoping to see things like disk occupation and documents count per-index. However, I can't find any measurement that includes the index name as a tag. I'm not sure if this is a symptom related to this issue in particular, can someone clarify that please?

Thanks to everyone working on this!

[danielnelson]

@voiprodrigo This issue doesn't cause any metrics to be lost, they are just named confusingly. Make sure you have:

  cluster_health = true
  cluster_health_level = "indices"
  cluster_stats = true
  # node_stats = ["jvm", "http"]

If that doesn't help can you open a new issue.

[voiprodrigo]

@danielnelson thanks. I do now see series tagged by index, however most of the fields have null value, including docs_count, which is may main interest. I'm using ES6(.2.2), maybe that's a factor. Is there a related issue for this, or should I open a new one?

[danielnelson]

ES6 is indeed broken: #3630

[smmortazavi]

@voiprodrigo , is a new issues open for the null values in the "index" tag for docs_count ?

[dupondje]

Fixed in 1775e1c ?
Or do we still want to rename elasticsearch_indices to something like elasticsearch_cluster_health_indices?

[danielnelson]

Been so long, I think though we should move the metrics from _cluster/health?level=indices as elasticsearch_cluster_health_indices.