Support "cluster" scope in Metricbeat elasticsearch module

[ycombinator]

What does this PR do?

This PR introduces a new scope setting for the elasticsearch Metricbeat module. This setting can take one of two values:

• node (default): indicates that each item in the hosts list points to a distinct Elasticsearch node in a cluster, or
• cluster: indicates that each item in the hosts lists points to a single endpoint for a distinct Elasticsearch cluster (e.g. a load-balancing proxy fronting the cluster).

Why is it important?

Sometimes it may not be possible for Metricbeat to reach individual Elasticsearch nodes. It might only have access to a single endpoint that fronts the entire Elasticsearch cluster.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Manual testing

Testing the new scope: cluster functionality introduced in this PR requires an Elasticsearch cluster with multiple nodes. The easiest way is probably to spin up an Elastic Cloud cluster.

1. Enable the elasticsearch-xpack module.

   ./metricbeat modules enable elasticsearch-xpack
   

2. Configure the module (./modules.d/elasticsearch-xpack.yml) like so:

   - module: elasticsearch
     xpack.enabled: true
     period: 10s
     hosts:
       - XXXXX
     scope: cluster
     username: XXXXX
     password: XXXXX
   

   Obviously, replace the XXXXX placeholders with your own.

   The key is that there should be exactly one item under hosts, pointing to single endpoint for your cluster. This could be the Elasticsearch endpoint obtained from Elastic Cloud or the address of a single node in your on-prem/local Elasticsearch cluster.

3. Configure Metricbeat to send the collected stats to an Elasticsearch cluster. This will act as your Monitoring Cluster. It could be the same cluster you're using to collect stats from (as configured in your elasticsearch-xpack module configuration above) or it could be an entirely separate cluster.

   metricbeat.config.modules:
   path: ${path.config}/modules.d/*.yml
   reload.enabled: false
   
   output.elasticsearch:
     enabled: true
     hosts:
       - XXXXXX
     username: XXXXXX
     password: XXXXXX
   

4. Start Metricbeat.

   ./metricbeat -e
   

5. Let Metricbeat run for ~30 seconds. Make sure there are no errors in the Metricbeat logs.

6. Perform the following query against your Monitoring Cluster.

   POST .monitoring-es-7-mb-*/_search
   {
     "size": 0,
     "aggs": {
       "by_cluster_uuid": {
         "terms": {
           "field": "cluster_uuid"
         }
       },
       "cluster_stats": {
         "filter": {
           "term": {
             "type": "cluster_stats"
           }
         },
         "aggs": {
           "by_period": {
             "date_histogram": {
               "field": "timestamp",
               "interval": "10s"
             }
           }
         }
       },
       "node_stats": {
         "filter": {
           "term": {
             "type": "node_stats"
           }
         },
         "aggs": {
           "by_period": {
             "date_histogram": {
               "field": "timestamp",
               "interval": "10s"
             }
           }
         }
       }
     }
   }
   

   This query checks 3 things:

   1. The aggs.by_cluster_uuid aggregation checks that we are only seeing data for a single cluster and that all documents contain that single cluster UUID.
      • Verify that aggregations.by_cluster_uuid.buckets only contains a single bucket, for the single cluster UUID of the Elasticsearch cluster you are monitoring.
      • Verify that aggregations.by_cluster_uuid.buckets[0].doc_count is the same as the hits.total.value.
   2. The aggs.cluster_stats aggregation checks that type: cluster_stats documents are only indexed once every collection period (10 seconds) and that there is at most one document per collection period. We are using type: cluster_stats here as an example; the same should be true for any types other than type: node_stats.
      • Verify that aggregations.cluster_stats.by_period.buckets have several buckets, each corresponding to a time period. Each buckets should be 10 seconds "wide". Within each bucket, verify that doc_count is <= 1.
   3. The aggs.node_stats aggregation checks that type: node_stats documents are only indexed once every collection period (10 seconds) and that there are at most N documents per collection period, where N is the number of nodes in the cluster you are monitoring.
      • Verify that aggregations.node_stats.by_period.buckets have several buckets, each corresponding to a time period. Each buckets should be 10 seconds "wide". Within each bucket, verify that doc_count is <= N, where N is the number of nodes in the cluster you are monitoring.

Related issues

• Resolves Metricbeat elasticsearch module should be able to collect from a single cluster endpoint #18539.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request Support "cluster" scope in Metricbeat elasticsearch module #18547 updated]

• Start Time: 2020-05-18T09:15:01.442+0000

• Duration: 73 min 14 sec (4394428)

Test stats 🧪

Test	Results
Failed	0
Passed	3975
Skipped	935
Total	4910

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[exekias]

this logic repeats across the board, I wonder if it should go into a helper function

[ycombinator]

Refactored in 348c9b9.

[exekias]

I guess this TODO is done by GetMasterNodeID

[ycombinator]

Removed in 34cd849.

[exekias]

I wonder if mode is enough, in other places we have used scope too. Anyway I don't have a strong opinion here

[ycombinator]

I was reserving mode for this change: #9424 (comment).

Using scope here instead of hosts_mode sounds good to me.

[ycombinator]

Changed in f2a5618.

[exekias]

This is looking great Shaunak! thank you for working on this

[ycombinator]

I tested the changes in this PR for heap usage w.r.t # of nodes in the ES cluster being monitored. There was no significant difference in heap usage, no matter how many nodes were in the the ES cluster being monitored. Details below.

Setup

1. Spun up an Elastic Cloud deployment, initially with just 1 node in 1 zone. Pre-allocated 64GB RAM since that would be necessary later to scale up the number of nodes.

2. Modified the elasticsearch module code to only load the index metricset.

3. Enabled the elasticsearch-xpack module and configured it like so:

- module: elasticsearch
  xpack.enabled: true
  period: 1s
  hosts:
  - https://XXXXXX.us-central1.gcp.cloud.es.io:9243
  username: elastic
  password: XXXXXXXXXX
  scope: cluster

3. Configured Metricbeat like so:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.console.enabled: true

4. Ran Metricbeat with the HTTP profiling endpoint enabled.

./metricbeat -c metricbeat.test.yml --httpprof localhost:9000 -e

5. Collected Metricbeat heap usage every 200ms.

while true; do curl -s 'http://localhost:9000/debug/vars' | jq '.["beat.memstats.memory_alloc"]'; sleep 0.2; done

6. Stopped collecting heap usage and Metricbeat. Recorded usage in Google sheet (see below).

7. Incremented number of nodes in ES cluster by 1, repeated above steps.

Results

https://docs.google.com/spreadsheets/d/1Boi0uw846OSY3vnGqC604Dj8Lh28lD3G9OsKFgPGz8I/edit?usp=sharing

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)