Stack monitoring.entsearch - 8.0 compatibility

[matschaffer]

Enterprise search stack monitoring seems to work in 8.0

But the data is being ingested into metricbeat-*

We have plans to remove these indices from search paths due to a large number fo support cases, so we probably shouldn't release 8.0 with stack monitoring data there.

[matschaffer]

At first pass it seems like we might be good here.

I set up the demo engine, but "engines" still says 0. I'll take a look to see if maybe there's some mapping/beats issue causing trouble with that.

[matschaffer]

Interesting that ent search bundles its own metricbeat:

❯ docker run --rm --entrypoint /usr/share/enterprise-search/bin/vendor/metricbeat/metricbeat-linux-x86_64 docker.elastic.co/enterprise-search/enterprise-search:8.0.0-SNAPSHOT version
metricbeat version 8.0.0 (amd64), libbeat 8.0.0 [2789c948cf501021550268a2b5dd37c247abeeb0 built 2021-12-20 01:49:40 +0000 UTC]

Ah, end my engine went to "1" after a bit.

Looks like ent search monitoring happens via it's own module without any sort of "xpack" setting.

[matschaffer]

So I think for the scope of #120825 ent search is basically okay, but there's a risk of breakage from #119112 due to the ent search documents landing in metricbeat-* indices.

kibana/x-pack/plugins/monitoring/common/constants.ts

Line 134 in 3dcfdef

export const INDEX_PATTERN_ENTERPRISE_SEARCH = '.monitoring-ent-search-*';

is set to .monitoring-ent-search-* but in my test setup there are no indices there.

Additionally if I set monitoring.ui.metricbeat.index: '.monitoring-metricbeat-*' to stop querying metricbeat-* indices. The ent search stack monitoring data isn't shown.

[matschaffer]

I'll leave this open to address the issue of the data going to metricbeat-*, if we release 8.0.0 with ent search available we'll have to keep that pattern around or offer a re-index solution to move the data into .monitoring-ent-search-*.

[matschaffer]

Looks like 7.16.2 also lands the enterprise search data into metricbeat-*, I suspect the enterprise search beats module has always worked this way and we just never noticed since SM UI also queries those indices.

[matschaffer]

Alternatively we might consider reverting #120630 and not supporting entsearch stack monitoring until 8.1.

[klacabane]

Looks like we'll need to introduce a conditional on xpack.enabled like we did for other Stack Monitoring modules in this PR in order to write to the .monitoring-ent-search-* index, or maybe we should hardcode the index ? Since this is a new module and Stack monitoring does not officially support data stored in metricbeat index, I'd rather avoid that path completely.

I'm wondering where the index template for entsearch is created though, did the module always wrote to metricbeat-8 and used the default mappings ? cc @kovyrin

[neptunian]

@richkuz Can you clarify what the the expected behavior is to where docs should be landing? As @klacabane said, we don't officially support data in the metricbeat-* index and are removing this index pattern.

[richkuz]

Can you clarify what the the expected behavior is to where docs should be landing

@carlosdelest returns on Thursday and may know the answer to this question. @ioanatia would you happen to know in the interim?

[carlosdelest]

I believe that the original intention was to use the .monitoring-ent-search-* index. However, as the stack monitoring code was not ready to be merged for 7.16, the tactical solution was to create a metricbeat dashboard to display the information.

In order for the metricbeat dashboard to work, indices had to be renamed to metricbeat-* to be available for metricbeat dashboards (see https://github.com/elastic/ent-search/pull/5086 for the related change).

@kovyrin had the intention to change the index pattern for 8.0:

• I’ll change our 7.x metricbeat integration to not override the default metricbeat index pattern
• I’ll focus on delivering a metricbeat dashboard that works with default metricbeat configs and with vendored metricbeat out of the box (there is a limitation on what index names metricbeat dashboards can use)

Then, in 8.0, we’ll change the default index pattern used by our vendored metricbeat and will ship stack monitoring changes that would rely on that new index name (stack monitoring components can only read from .monitoring-*, which is different from the default index metricbeat uses).

So, I think the changes needed on Enterprise Search are:

• Remove the current metricbeat dashboard used by Enterprise Search (included in the Enterprise Search Metricbeat module).
• Change the vendored metricbeat index configuration to use .monitoring-ent-search-* index pattern in Enterprise Search.

Changes on Kibana side would be to use the .monitoring-ent-search-* index pattern for Enterprise Search in the Stack Monitoring plugin.

Does it make sense to you @richkuz , @neptunian , @matschaffer , @klacabane ?

What would be the feasible release version target for making these changes?

[klacabane]

Stack monitoring already uses .monitoring-ent-search-* index pattern which is covered by functional tests (see the loaded mappings) so updating entsearch vendored metricbeat to write data there will be working out of the box as long as the mappings are installed. @carlosdelest how are these mappings installed at the moment ?

Now we've planned to drop support for metricbeat-* indices in Stack monitoring but given entsearch might have documents stored there we'll lose historical data once kibana is upgraded to only read off the new pattern. If this is the accepted path forward we'll ideally have to ship these breaking changes for 8.0, otherwise we'll have to keep the metricbeat-* pattern around. The re-index solution mentioned by Mat would be a good compromise that would allow removal of metricbeat-* and keeping the historical data, @matschaffer can this be automated as a migration step ?

[carlosdelest]

how are these mappings installed at the moment ?

I believe Metricbeat does this automatically, according to the fields.yml files included in both health and stats metricsets in the Enterprise Search module.

Now we've planned to drop support for metricbeat-* indices in Stack monitoring but given entsearch might have documents stored there we'll lose historical data once kibana is upgraded to only read off the new pattern. If this is the accepted path forward we'll ideally have to ship these breaking changes for 8.0, otherwise we'll have to keep the metricbeat-* pattern around.

@richkuz , I believe this could be done for 8.0 release and seize the opportunity to make the breaking changes now. I can work on that and try to get it done before the next BC to ensure we have time to double check.

[richkuz]

Thanks @carlosdelest , this plan makes sense to me. I think we should proceed with the proposed changes for 8.0. cc @serenachou

Can/should we also change these 8.0 monitoring dashboards to work with the new index names, too?

We'll also need to update this section of our documentation.

[matschaffer]

The re-index solution mentioned by Mat would be a good compromise that would allow removal of metricbeat-* and keeping the historical data, @matschaffer can this be automated as a migration step ?

I haven't written one, but I've used reindexing steps as part of the kibana upgrade assistant in the past.

Not sure timing aligns well though. We'd need to get the assistant code merged and released as part of 7.(final) ahead of 8.0 release.

Not sure if @jasonrhodes, @smith, or maybe another participant in this issue can recommend a person/approach to get a reindex step built into the upgrade assistance on such short notice.

[carlosdelest]

TLDR; We need to make work in Metricbeat, Elasticsearch and Enterprise Search to do this change using Metricbeat 8.0. I suggest we use a Metricbeat 7.16 vendored version in Enterprise Search 8.0, and then transition to Metricbeat 8.0 once we can get the needed work done.

Metricbeat 8.0 needed work

Some things have changed for Metrictbeat 8.0 that makes difficult to change the indices used to .monitoring-ent-search-*. Metricbeat uses data streams now instead of ILM backed indices (see elastic/beats#28450).

Specific monitoring products (Logstash, Kibana, Elasticsearch) have associated index templates (included into Elasitcsearch) that create these data streams when events are received for the matching index patterns.

Metricbeat modules use xpack.enabled: true configuration to send events to the corresponding indices that will match the index patterns, and thus create the data streams.

In case we want to use Metricbeat 8.0 version, the following changes need to be made:

• Metricbeat Enterprise Search module needs to be modified to add the xpack.enabled: true option that enables selecting a product specific monitoring index (like Logstash, Elasticsearch and Kibana are doing) instead of using the default .metricbeat-* indices.
• Elasticsearch must add Enterprise Search as a monitoring product. Index templates for the new Enterprise Search monitoring product need to be included into the Elasitcsearch templates so sending events with the appropriate index creates the Enterprise Search monitoring data stream.

That way, when Enterprise Search Metricbeat module sends events, the appropriate data stream according to the template will be created with the expected name pattern.

I think this might be risky to land in 8.0, as it implies changes both in Metricbeat and Elasticsearch and we're already in RC1.

Use vendored Metricbeat 7.16 version

We could use a vendored Metricbeat version 7.16 in Enterprise Search, and keep using ILM to generate .monitoring-ent-search-7.16.0-* indices.

Once we complete the work needed to use Metricbeat 8.0 version in Enterprise Search, we can switch the vendored version and create the corresponding data stream. As the index pattern for stack monitoring would hit both the ILM indices created by 7.16 and the data stream created by 8.0, there would be no migration needed. Please confirm this is possible and I'm not getting this wrong!

Thoughts?

[carlosdelest]

Added @sayden into the loop for review of the above approach

[klacabane]

Metricbeat Enterprise Search module needs to be modified to add the xpack.enabled: true option that enables selecting a product specific monitoring index

I'm wondering if we can omit that branch completely and write directly to .monitoring-ent-search-*. New major version is a good opportunity to make that breaking change but I don't know much about user expectations and metricbeat's contract regarding the indice destination (ie should it always be possible for a metricbeat module to write to metricbeat-* ?) cc @sayden @kvch

I think this might be risky to land in 8.0, as it implies changes both in Metricbeat and Elasticsearch and we're already in RC1.

Agree with this although we recently did all the required work for the supported stack monitoring modules, so we have a clear footprint of the work needed to be done and I'm happy to help on this, but again we're pretty late in the release schedule

We could use a vendored Metricbeat version 7.16 in Enterprise Search, and keep using ILM to generate .monitoring-ent-search-7.16.0-* indices.
As the index pattern for stack monitoring would hit both the ILM indices created by 7.16 and the data stream created by 8.0, there would be no migration needed.

Correct, stack monitoring will pull any data matching .monitoring-ent-search-* so that will include both 7.x and 8.x data , however we'll still need the changes to write to .monitoring-ent-search-* since SM will drop support for metricbeat-* pattern in 8.1 which means the SM modules that rely on that pattern to display stack monitoring data should stop writing there as of 8.0 cc @jasonrhodes

[matschaffer]

Looks like some PRs are open here (thanks @carlosdelest! 🧡 ), we may want to perform our overview testing using code from those PRs.

[carlosdelest]

As commented here, there are two different tests that can be performed:

• Test our current Enterprise Search implementation (8.0 RC2 out today will contain the change). This uses a bundled Metricbeat 7.16 and creates .monitoring-ent-search-* ILM-based indices.
• Test the new implementation:
  • Metricbeat Enterprise Search module support for xpack.enabled, PR here.
  • Elasticsearch monitoring index templates for Enterprise Search, PR here.

Both work out of the box with the current Stack Monitoring Kibana plugin, and should be compatible in the future (we can switch to the Metricbeat implementation in the future and not lose the data ingested by the current approach).

The new implementation (Metricbeat module + Elasticsearch templates) will be targeted for 8.1.

[matschaffer]

Test our current Enterprise Search implementation (8.0 RC2 out today will contain the change).

Ah great. I tested docker.elastic.co/enterprise-search/enterprise-search:8.1.0-SNAPSHOT yesterday and got metricbeat-* indices but looks like that image was from https://github.com/elastic/ent-search/commit/a51fc709d289e6da8056d6e7c0ad0252566d1f85 - I'll watch for RC2 and try that next.

[matschaffer]

ah, I re-pulled 8.1.0-SNAPSHOT (2759822424661508084344c42dbf41fe7b1dd903) and its looking good for internal collection.

I can give your PRs a try too, but nice that they're no longer required for 8.0 and we can just stick to "internal" collection via the embedded 7.16 metricbeat until 8.1

[carlosdelest]

We can wait for the Metricbeat + Enterprisesearch definitive solution for 8.1. We could create a separate issue for it, and close this one when your testing finishes 👍

[klacabane]

Tested with 8.0.0 snapshot and verified SM UI was powered by .monitoring-ent-search-7.16.2-mb-2022.01.20-000001 indice and no metricbeat indice was created. Let's close this issue with Mat's feedback and track 8.1 related changes here :)

[matschaffer]

Though I think in 8.0 if someone tries to use an external metricbeat to monitor ent search, the docs will land in metricbeat-* and the UI will miss the docs.

We should probably have something in documentation so customers don't try to do that. Lots of other docs push the user toward metricbeat for monitoring, so seems reasonable someone would try it with ent search.

[matschaffer]

Basically thinking we should probably remove this page before we launch 8.0 https://www.elastic.co/guide/en/beats/metricbeat/8.0/metricbeat-module-enterprisesearch.html - as well as the bullet point on https://www.elastic.co/guide/en/enterprise-search/8.0/monitoring.html for "External Metricbeat Monitoring"

Alternatively, backport @carlosdelest 's changes to 8.0 and try to get them in the next beats 8.0 rc.

[carlosdelest]

Basically thinking we should probably remove this page before we launch 8.0 https://www.elastic.co/guide/en/beats/metricbeat/8.0/metricbeat-module-enterprisesearch.html -

The module will work correctly, but it won't be compatible with Stack Monitoring. Would that be a good enough reason for removing it from documentation? Users can still use custom dashboards to display information.

as well as the bullet point on https://www.elastic.co/guide/en/enterprise-search/8.0/monitoring.html for "External Metricbeat Monitoring"

We got this in https://github.com/elastic/enterprise-search-pubs/pull/1954/files for Enterprise Search:

• Removed instructions for Dashboard setup using metricbeat setup
• Set Metricbeat specific version to monitor Enterprise Search (it is a valid use case in Cloud)

Alternatively, backport @carlosdelest 's changes to 8.0 and try to get them in the next beats 8.0 rc.

We'd need to land elastic/elasticsearch#82743 as well in the next Elasticsearch 8.0 rc. I believe coordinating the two changes together at RC2 is risky, happy to discuss though.

[serenachou]

Based on Carlos statement: that the user can still use the custom dashboards, it seems like it would be still fine to keep that line in. But I thought y'all were trying to get rid of metricbeat-* entirely? Maybe I saw that out of context somewhere but wanted to check if there's any issue if customers continue to utilize it until we settle al the compatibility tickets?

[matschaffer]

We got this in https://github.com/elastic/enterprise-search-pubs/pull/1954/files for Enterprise Search

Ah, sweet. I have some mild concern that users will click the link in https://www.elastic.co/guide/en/enterprise-search/8.0/monitoring.html#monitoring-overview and never see the warning further down the page.

But at least it's there for reference in any possible support cases that might come as a result of users attempting to monitor ent search with metricbeat 8.0

I thought y'all were trying to get rid of metricbeat-* entirely?

It's been removed in https://github.com/elastic/kibana/pull/119112/files#diff-e73bd280b80c4cce6c0f3c372573b8c6394d189bb9611f202501734744a55837 for 8.1 and onward.

So if users do set up their ent search metrics to ingest into metricbeat-* they'll need to reindex that data to get continuous graphs in stack monitoring UI.

[carlosdelest]

Thanks for your feedback @matschaffer !

Is there anything else we need to wrap this up? Do you think we can close this issue, as it has already been tested for 8.0? We can keep the work in the one opened for 8.1.

Thanks!

[matschaffer]

Moving to close a part of the packages sync we're having atm.