Metricbeat enterprise search module: add xpack.enabled support

[carlosdelest]

What does this PR do?

Add xpack.enabled support for Enterprise Search Metricbeat module.

This allows metrics retrieved to be available in Stack Monitoring Kibana plugin, as this plugin changes the indices retrieved from .metricbeat-* to .monitoring-*. xpack.enabled: true allows changing the index to be used to .monitoring-*, as already implemented for Elasticsearch, Kibana and Logstash modules.

Why is it important?

This change is needed to allow Enterprise Search Metricbeat module to produce metrics that can be consumed by Stack Monitoring Kibana plugin. Stack Monitoring won't display Enterprise Search metrics otherwise.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates Stack monitoring.entsearch - 8.0 compatibility kibana#121975
• Relates https://github.com/elastic/enterprise-search-team/issues/987

[mergify]

This pull request does not have a backport label. Could you fix it @carlosdelest? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-01-18T11:09:49.356+0000

• Duration: 101 min 14 sec

• Commit: 1d70a33

Test stats 🧪

Test	Results
Failed	0
Passed	9722
Skipped	2530
Total	12252

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[ruflin]

@carlosdelest Is enterprise search already today with a different mechanism shipping metrics to the .monitoring index? If no, I wonder why we need this? @jasonrhodes I would assume stack monitoring can pick up data from any index assuming the user permissions match?

[carlosdelest]

Hi @ruflin :

Enterprise Search can't use Metricbeat 8.0 to ship monitoring information, as Metricbeat no longer supports ILM settings that allows shipping to .monitoring-* indices using Elasticsearch output configuration.

Enterprise Search 8.0 will be using a bundled 7.16 Metricbeat version so it can ship monitoring information using an ILM-backed .monitoring-* index.

This is temporary, as the final picture would be for Enterprise Search to use Metricbeat 8.x with xpack.enabled support (this PR), plus Elasticsearch index templates for creating the data streams once they are sent (elastic/elasticsearch#82743).

Stack Monitoring will be able to gather Enterprise Search monitoring as long as it is sent to .monitoring-ent-search-* indices, which both approaches provide. We're just trying to align on the monitoring approach used by other solutions (Elasticsearch, Kibana, Logstash).

Happy to discuss and consider alternatives!

[jasonrhodes]

@ruflin this surprises me as well. I assumed that since this is a new metricbeat module, there's no reason to maintain any xpack.enabled fork. I'm re-reading the conversation over here: elastic/kibana#121975 (comment)

My question for @carlosdelest (and possibly for @sayden / @klacabane) is: does the enterprise search MB module need to be able to write to metricbeat-* for any reason going forward? If not, can we at the very least hardcode xpack.enabled: true as the only way for this module to run, even if that flag technically needs to be set for MB internals to write the data to the correct indices?

[carlosdelest]

does the enterprise search MB module need to be able to write to metricbeat-* for any reason going forward?

Not that I can think of, unless we wanted to use Metricbeat 8.x with a Kibana + Enterprise Search 7.16.

I agree on hardcoding xpack.enabled. Would you like this to be the default behaviour, and be disabled explicitly in configuration if it is ever needed? Or always use the new behaviour?

Is this the way to go as well for the rest of the solutions, so we are aligned with them?

[matschaffer]

I tested this today along with the ES change and this metricbeat config:

  - module: enterprisesearch
    xpack.enabled: true
    period: 10s
    hosts: ["http://localhost:3002"]
    username: elastic
    password: changeme

Looks good!

[matschaffer]

Maybe this should be 8.1? I'm still not totally clear on where these compose files factor into the release pipeline, so feel free to ignore if I'm wrong.

[matschaffer]

Do we need the metricsets? The elasticsearch module specifies a locked metricset when running in xpack mode

beats/metricbeat/module/elasticsearch/elasticsearch.go

Line 47 in a78436b

xpackEnabledMetricSets := []string{

Seems to work find w/o the list in my testing. Maybe because it just enables all of them.

[ruflin]

The part I'm still missing, is there anything else already today writing to .monitoring-* for enterprise search? If not, couldn't we jump directly to the "end" story in this scenario which is the logs-*-* and metrics-*-* data streams? https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme

[carlosdelest]

The part I'm still missing, is there anything else already today writing to .monitoring-* for enterprise search?

I believe not.

If not, couldn't we jump directly to the "end" story in this scenario which is the logs-- and metrics-- data streams?

@ruflin , I agree on hardcoding xpack.enabled. Would you like this to be the default behaviour, and be disabled explicitly in configuration if it is ever needed? Or always use the new behaviour and remove xpack.enabled?

Is this the way to go as well for the rest of the solutions, so we are aligned with them?

[ruflin]

It might be a bit unexpected from a user perspective that metricbeat has a module which cannot ship to metricbeat-* indices but at the same time if there are no dashboards and the data is only for internal use why support it in the first place.

For all the other stack monitoring modules, a discussion is ongoing on how we get to metrics-*-* indices but there is a catch because there is also internal data shipping in ES and Kibana. But for enterprise search it sounds like we don't have this limitation. So instead of shipping to .monitoring-* we ship directly to metrics-*-*. This would allow in theory users to also build dashboards and have the setup at one stage rely on packages But for now also an internal template in Elasticsearch can be used how we would do it for .monitoring indices.

My understanding of hardcoding xpack.enabled is more that it doesn't exist in the first place as there is only one way to ship the data. But I think we talk about the same thing.

@jasonrhodes Does this align with your long term plan for stack monitoring?

[carlosdelest]

pinging @ravikesarwani for comments . Thanks!

[ravikesarwani]

I will let @jasonrhodes weigh in on this as well but as I understand the current UI code (8.0) doesn't read from metrics-*.
We are working on it but it's targeted for 8.1. Making all the metricbeat modules work the same way is a big win from the customer understanding perspective and avoids a lot of confusion (like retention management and so on considering both self-managed and cloud environments) as metricbeat method will exist for a little while.

The Agent package will write to metrics-* and provide backward compatibility from UI perspective and that's where we are heading for all stack packages including Enterprise search. This is where we will focus on adding default dashboards etc. as well. There's already quite a bit of complexities and IMO making one metricbeat module work completely different will just to the confusion from users perspective.

[matschaffer]

Basic support for metrics-*-* just landed in elastic/kibana#119112 for 8.1 (cc @neptunian), but I understand we haven't done comprehensive testing of using the new index patterns.

They're present and we've validated that the changes don't break the .monitoring-* pattern used by other metricbeat stack monitoring modules.

It might just work, but I think we should still merge this change to minimize the number of ent search releases that might land documents in the no-longer-queried metricbeat-* pattern.

Then we can work on a comprehensive change from .monitoring-* to metrics-* for all stack components.

[carlosdelest]

Thanks for all your feedback on this issue!

Re xpack.enabled behaviour, I think it's better from a user / compatibility perspective not to break the 7.x behaviour, shipping to .metricbeat-* indices. It also aligns with other current Stack Monitoring behaviour and configuration for other modules.

I'll let the PR as is, and I'll be merging this tomorrow unless some objection is raised.

[ruflin]

.metricbeat-* indices? I assume you mean .monitoring-*? Just double checking.

[carlosdelest]

.metricbeat-* indices? I assume you mean .monitoring-*? Just double checking.

• With xpack.enabled: false (the default behaviour in case it is not configured), it will ship to metricbeat-* indices (same behaviour as 7.x for Enterprise Search module)
• With xpack.enabled: true (change introduced in this PR), it will ship to .monitoring-* indices. Needs explicit configuration.

[ruflin]

I'm stumbling over the . in front of metricbeat. This is not something use anywhere AFAIK.

[carlosdelest]

I'm stumbling over the . in front of metricbeat. This is not something use anywhere AFAIK.

My bad 😅 . It's metricbeat-* and not .metricbeat-*. Edited the previous comment, thanks for double checking!