Use Elasticsearch histogram type to store Prometheus histograms

[MichaelKatsoulis]

Allow to store Prometheus histograms types as Elasticsearch histograms using metricbeat.

For example prometheus histograms metrics like

# TYPE apiserver_request_duration_seconds histogram
apiserver_request_duration_seconds_bucket{component="",dry_run="",group="",resource="",scope="",subresource="/healthz",verb="GET",version="",le="0.05"}

are currently stored in elasticsearch as

"request": {
  "component": "apiserver",
  "duration": {
    "us": {
     "bucket": {
        "+Inf": 2,
        "100000": 0,
	"1000000": 0,
	"10000000": 0,
	"1250000": 0,
	"150000": 0,
	"1500000": 0,
	"15000000": 0,
	"9000000": 0
   },
   "count": 2,
   "sum": 1081003139.2180002
  }
},			

Instead according to elastic/kibana#59387 they could be stored as Histogram Data Type like

"request.duration.us": {
   "histogram": {
       "counts": [
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        0,
                        2
        ],
        "values": [
                        100000,
                        1000000,
                        10000000,
                        1250000,
                        150000,
                        1500000,
                        15000000,
                        9000000,
                        +Inf
        ]}},

This can then help in better visualisations of those data in Kibana using the percentiles aggregation.
An implementation of this data transformation has already been done in #17061 for prometheus collector.

We should consider doing the same in prometheus helper.

We should proceed with caution as hard updating this can lead to breaking changes.

[MichaelKatsoulis]

@ChrsMark, @gizas

[MichaelKatsoulis]

An alternative solution to consider is instead of adding the histogram types to prometheus helper, to leverage lightweight modules. That way in integrations like kubernetes apiserver or/and scheduler, we could use prometheus collector metricset to collect the prometheus metrics. In that case we get rid of not needed metricset implementations and take advantage of already implemented histogram types.

We should be very careful with breaking changes to the integration.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[qqiu-rbx]

Hey @MichaelKatsoulis , just want to check my understanding.
This request has not been implemented, it is just closed by the bot, does this sound correct to you?