Add elasticsearch/ingest_pipeline metricset

[joshdover]

What does this PR do?

Depends on:

• Add support for optional metricsets in xpack mode #34273

Ports elastic/integrations#4597 to metricbeat. This adds a new ingest_pipeline metricset to the elasticsearch module of Metricbeat. This module does the following:

• Fetches the Nodes Stats API's ingest metrics
• Ingest two types of documents:
  • Top-level pipeline metrics are ingested on every interval, including the counters for total time, documents processed, and failure counts. We also calculate a "self time" metric which will subtract the time spent on processor calls to other pipelines.
  • Processor-level metrics are ingested on a sampling rate (25% - every 4th interval by default, configurable). For these metrics, a separate document is created for every processor in every pipeline, on every ES node. For this reason, the sampling strategy is used to minimize the amount of data that is produced.
• Supports local and cluster scopes for better performance

The UI for visualizing this data will be included as a dashboard. Right now this is only being shipped in an Agent integration with support for Stack Monitoring and Metricbeat index patterns. In a follow up PR to Kibana, a link will be added from the Stack Monitoring UI to this dashboard, or direct the user to install the package to get the dashboard.

Here's a summary of the new fields that are added:

- name: ingest_pipeline
  type: group
  release: beta
  description: Runtime metrics on ingest pipeline execution
  fields:
    - name: name
      type: wildcard
      description: Name / id of the ingest pipeline
    - name: total
      type: group
      description: Metrics on the total ingest pipeline execution, including all processors.
      fields:
        - name: count
          type: long
          description: Number of documents processed by this pipeline
        - name: failed
          type: long
          description: Number of documented failed to process by this pipeline
        - name: time.total.ms
          type: long
          description: Total time spent processing documents through this pipeline, inclusive of other pipelines called
        - name: time.self.ms
          type: long
          description: Time spent processing documents through this pipeline, exclusive of other pipelines called
    - name: processor
      type: group
      fields:
        - name: type
          type: keyword
          description: The type of ingest processor
        - name: type_tag
          type: keyword
          description: The type and the tag for this processor in the format "<type>:<tag>"
        - name: order_index
          type: long
          description: The order this processor appears in the pipeline definition
        - name: count
          type: long
          description: Number of documents processed by this processor
        - name: failed
          type: long
          description: Number of documented failed to process by this processor
        - name: time.total.ms
          type: long
          description: Total time spent processing documents through this processor

Why is it important?

With the adoption of Agent integrations, ingest pipeline performance is very important to overall ingest performance. Users have very little insight into this today and giving them dashboards and metrics from the existing ES APIs is a great first start in improving the situation.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Add unit tests
• Add monitoring mappings for es ingest metricset elasticsearch#92950
• Determine how we want to handle XPack mode. By default, xpack mode enables all metricset but we want to ship this as beta, so we may need to modify this behavior
  • Solved by Add support for optional metricsets in xpack mode #34273

How to test this PR locally

Related issues

• Relates to [Stack Monitoring] Ingest pipeline monitoring kibana#41936

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @joshdover? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-27T13:38:35.125+0000

• Duration: 58 min 33 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4067
Skipped	887
Total	4954

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
No Duplication information

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
No Duplication information

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[ruflin]

Overall LGTM. Left some minor comment. +1 on moving forward with it.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b es-ingest upstream/es-ingest
git merge upstream/main
git push upstream es-ingest

[klacabane]

ShouldSkipFetch will skip when we have a ScopeNode and we're not talking to master. This is implemented in most metricsets because we hit an API that returns the global state from master and we don't need to collect individual nodes data, but this does not seem to apply here.
iiuc we want to fetch individual node data when the ScopeNode, and hit the global API when ScopeCluster so we should remove that call and implement our own logic (similar to the node_stats metricset)

[joshdover]

Good catch, thank you. I think I can just remove this entirely as I'm already setting the URI correctly on the lines below this. Do you agree?

[klacabane]

Sounds good

[amitkanfer]

🚢