Collect bulk indexing stats for Elasticsearch metricsets

[ycombinator]

What does this PR do?

Collects new bulk indexing metrics being reported by Elasticsearch in the index, index_summary, and node_stats metricsets for the Stack Monitoring code path (i.e. when xpack.enabled: true is set in these metricsets' configurations).

These metricsets correspond to type:index_stats, type:indices_stats, and type:node_stats documents, respectively, in .monitoring-es-* indices used by the Stack Monitoring UI.

Why is it important?

To restore parity between legacy internal collection and Metricbeat collection for Elasticsearch monitoring. The changes corresponding to this PR for legacy internal collection were made in elastic/elasticsearch#52208.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works.
  • In addition to the unit test added in this PR, parity tests already exist that are currently failing and should start passing once the changes in this PR are released.
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

1. Run the latest 8.0.0-SNAPSHOT build of Elasticsearch so it exposes the new bulk indexing metrics being collected by the code in this PR.

   docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT
   docker run --name elasticsearch -p 9200:9200 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT
   

2. Build Metricbeat with the changes in this PR.

   cd $GOPATH/src/github.com/elastic/beats/metricbeat
   mage build
   

3. Enable the elasticsearch-xpack module.

   metricbeat modules enable elasticsearch-xpack
   

4. Create a minimal Metricbeat configuration for testing.

   cat <<EOF > ./metricbeat.test.yml
   metricbeat.config.modules:
     path: \${path.config}/modules.d/*.yml
     reload.enabled: false
   
   output.console.enabled: true
   
   monitoring.elasticsearch:
     hosts: [ "localhost:9200" ]
   EOF
   

   Here, we only enable monitoring.elasticsearch to generate some bulk indexing requests to Elasticsearch so we can see non-zero bulk metrics.

5. Run Metricbeat with the minimal configuration and look for the new bulk indexing metrics' fields. You may need to run these for ~30 seconds for some bulk indexing to happen and corresponding metrics to be generated by Elasticsearch.

   1. For the index metricset, i.e.type:index_stats documents:

      ./metricbeat -c metricbeat.test.yml | jq 'select(.type == "index_stats") | .index_stats.total.bulk'
      

      ./metricbeat -c metricbeat.test.yml | jq 'select(.type == "index_stats") | .index_stats.primaries.bulk'
      

   2. For the index_summary metricset, i.e.type:indices_stats documents:

      ./metricbeat -c metricbeat.test.yml | jq 'select(.type == "indices_stats") | .indices_stats._all.total.bulk'
      

      ./metricbeat -c metricbeat.test.yml | jq 'select(.type == "indices_stats") | .indices_stats._all.primaries.bulk'
      

   3. For the node_stats metricset, i.e.type:node_stats documents:

      ./metricbeat -c metricbeat.test.yml | jq 'select(.type == "node_stats") | .node_stats.indices.bulk'
      

Related issues

• Resolves [Metricbeat][parity-tests] Missing some newly added ES fields #17977

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[ycombinator]

@igoristic I've requested your review on this PR as a functional reviewer. Please see the "How to test this PR locally" section in the PR description for instructions on how to functionally test this PR.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request Collect bulk indexing stats for Elasticsearch metricsets #17992 updated]

• Start Time: 2020-05-01T02:05:06.360+0000

• Duration: 82 min 29 sec (4948853)

• Commit: a87c509

Test stats 🧪

Test	Results
Failed	0
Passed	3008
Skipped	744
Total	3752

[jsoriano]

Failure in statsd tests should be fixed by #18130, the other failures may be related.

[ycombinator]

Looking at the CI errors from the elasticsearch module system tests, they are legitimate and need to be addressed in this PR.

This PR adds bulk to the list of metrics requested from the Elasticsearch Stats API. This metric is only available starting Elasticsearch 7.8.0. However, our tests use Elasticsearch 7.5.2. Which is why the tests are failing.

To address this correctly, we need to add bulk to the list of metrics requested from the Elasticsearch Stats API if the Elasticsearch version is >= 7.8.0. This will be similar to the work done recently in the logstash module in #17497.

Accordingly, I'm moving this PR back into draft state. Once I've implemented the necessary changes, I'll put it back into review.

[igoristic]

Looking at the CI errors from the elasticsearch module system tests, they are legitimate and need to be addressed in this PR ...

Yeah I was wondering if I should wait for the CI to turn green before I can test it

[ycombinator]

Yeah I was wondering if I should wait for the CI to turn green before I can test it

That would be ideal but there might be unrelated test failures. I'll ping you here where it's ready to test.

[ycombinator]

@igoristic CI is looking good now. There are failures but they are unrelated to this PR. Please go ahead and test it. Thanks!

[igoristic]

Tested all case scenarios mentioned in step 5, and those fields are no longer null (like they are in master):

Thank you for fixing this @ycombinator 🙇

[chrisronline]

@ycombinator Is this a typo? Should it be json:"avg_time_in_millis"?

[ycombinator]

Yes, it should 😞. Good eye, nice catch! I assume parity tests failed? I'll put up a PR to fix it. Thanks!

[ycombinator]

PR is up: #18420