[Stack Monitoring] create enterprise search package

[klacabane]

Summary

Blocked by elastic/elastic-agent#2121

To get feature parity with metricbeat one should be able to monitor enterprise search with the agent.
We should create an enterprise search package that spawns the metricbeat module under the hood.

A mechanism exists to create a package from a beats module - see https://github.com/elastic/integrations/blob/main/docs/import_from_beats.md

AC

• Entreprise search package exists
• Enterprise search metrics dataset has the .stack_monitoring. infix like the other SM packages (elasticsearch example)
• Enterprise search's Stack Monitoring section is populated when monitored with the agent

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[klacabane]

@elastic/ent-search-application-backend we'll have the opportunity to bundle this dashboard in the entsearch package. Can you confirm this is desired ?

We'll have to update the Stack Monitoring code to support the data collected by the package and that change should land in 8.7.0. Now if the package includes what looks like a thorough dashboard, we could already make it available to earlier versions (thinking of >= 8.5.0 to align with other packages).

[richkuz]

Thanks @klacabane for driving this! ❤️

Can you please help me understand a little more about the change, so I have full context:

1. Will anything need to change in ent-search server code itself (i.e. the Rails JVM process that powers Enterprise Search) to support this change?
2. Will we need to ask self-managed customers to install the agent if they want Kibana -> Stack Monitoring to continue to show Enterprise Search metrics?
3. Will Cloud need to deploy any extra processes inside the Enterprise Search container in order for this to work?
4. What exactly does it mean to "bundle this dashboard in the entsearch package" -- what is the entsearch package? (sorry, I don't tread in this area often, just getting my bearings)
5. Does the ent-search server itself need to change how it reports metric data to satisfy the acceptance criteria bullet point "Enterprise search metrics dataset has the .stack_monitoring. infix"? Will this name change break existing customer dashboards?
6. Are there any other impacts to customers we should document with this change?

To be sure, this issue sounds like it's helping move the product forward, I don't want to block momentum, I mainly want to learn more to make an informed decision.

If folks reading this issue are curious:

Here is what the enterprise-search-overview-8.0.ndjson dashboard imported into Kibana -> Saved Objects looks like. This is the dashboard @klacabane proposes to bundle in the entsearch package:

---

For comparison, here is what the pre-built Enterprise Search dashboard looks like in Kibana -> Stack Monitoring today:

---

My own opinion: I'm fine with either dashboard. I've never needed to look at JVM Thread Creation Rate or Background Queue counts myself. I think the Kibana -> Stack Monitoring dashboard is more pleasing on the eyes and better labeled, but that's a minor subjective opinion and trivial to change later on. So... tentatively .. yes? I support the change (pending answers to my questions above).

cc @carlosdelest

[klacabane]

Hi @richkuz!

This effort is building another way to collect monitoring metrics, and the changes are self-contained in the ingestion side. For context, the stack products can currently be monitored with either an internal collection mechanism, or a metricbeat sidecar. We're working on enabling a third approach with the elastic-agent. The agent is similar to the metricbeat approach, and has a platform that allows devs to define the monitoring logic and assets (mappings, dashboards..) for a given service. For metricbeat this logic is defined in a module and in agent-world it is a package. An important difference is that a package update can be shipped out of the stack releases, when a metricbeat module cannot.

1. No changes are required in ent-search code, this only impacts the collection and storage of metrics. The existing APIs (/stats and /health) will be used, but the output will be stored in different data streams. When metricbeat stored the data in .monitoring-ent-search-mb data stream, agent will store it in metrics-enterprisesearch.*
2. Self-managed customers can keep using metricbeat to power Stack Monitoring, but the package will allow customers that want to migrate their fleet of metricbeat to elastic-agents while maintaining existing functionalities
3. There is no onboarding planned for Cloud yet, but if/when that eventually happens the most likely change will be to replace the metricbeat process with an elastic-agent running the same configuration.
4. Dashboards and other assets can be bundled to be automatically installed when setting up the package.
5. The package itself will define the targeted data stream and no changes are required in ent-search. Packages follow the data stream naming scheme and customer dashboards will need to include the new pattern. The stack_monitoring label is used as versioning.
6. The location of the monitoring data is the breaking part for users that built custom dashboard with the .monitoring-ent-search-mb data, and we should cover that in a documentation section similar to the metricbeat module.

Note that Dashboard and Kibana's Stack Monitoring are not mutually exclusive, and both views can be told to read from the new package data streams without much burden. Now Stack Monitoring is not expected to evolve a lot in the future and we're aiming to replace the application with more flexible Dashboards, so that could be an opportunity to already move in that direction.

[klacabane]

Currently blocked by elastic/elastic-agent#2121

As an update, the team discussed whether to include the dashboard in the packages. For the initial release the package will only power the Stack Monitoring UI. We'll then work on aligning the Kibana dashboard with the SM one to get feature parity and look into leveraging portable dashboards to replace the SM UI with the embedded Kibana dashboard.