Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Stack monitoring] adapt ES queries to updated document structure #121641

Merged
merged 23 commits into from
Dec 23, 2021
Merged

[Stack monitoring] adapt ES queries to updated document structure #121641

merged 23 commits into from
Dec 23, 2021

Conversation

klacabane
Copy link
Contributor

@klacabane klacabane commented Dec 20, 2021 •
edited
Loading

Summary

Part of #121282

Updates stack monitoring to read off .monitoring-{product}-8-* datastreams created by ES at startup. This change only covers the elasticsearch product.
The _mb archived datasets used by functional and api tests were updated to target the new indices (with this script) and the elasticsearch tests updated to support datastreams. also manually tested and compared the views against the 7.16 xpack.enabled version with a single-node snapshot setup

small fixes were required in the metricbeat module to get some data to load

Testing

  • build metricbeat with this change
  • start metricbeat with elasticsearch-xpack module enabled
  • load SM UI, elasticsearch section should appear and every screen should display correct informations

@klacabane klacabane added v8.0.0 Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services auto-backport Deprecated - use backport:version if exact versions are needed v8.1.0 labels Dec 20, 2021
@klacabane klacabane self-assigned this Dec 20, 2021
@klacabane klacabane requested a review from a team as a code owner December 20, 2021 14:09
@elasticmachine
Copy link
Contributor

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

@klacabane klacabane added the release_note:skip Skip the PR/issue when compiling release notes label Dec 20, 2021
@klacabane klacabane mentioned this pull request Dec 20, 2021
Closed
3 tasks
@klacabane
Copy link
Contributor Author

@elasticmachine merge upstream

@klacabane
Copy link
Contributor Author

@elasticmachine merge upstream

@klacabane klacabane marked this pull request as draft December 21, 2021 14:58
@klacabane
Copy link
Contributor Author

@elasticmachine merge upstream

@klacabane klacabane marked this pull request as ready for review December 21, 2021 21:02
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
monitoring 445.6KB 445.6KB +2.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
monitoring 23.7KB 23.6KB -97.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @klacabane

@neptunian
Copy link
Contributor

Looks good! Sounds like we are ok removing the versioning also changed and discussed here https://github.com/elastic/kibana/pull/121609/files#r773630577

@matschaffer matschaffer mentioned this pull request Dec 23, 2021
Closed
9 tasks
matschaffer
matschaffer approved these changes Dec 23, 2021
View reviewed changes
Copy link
Contributor

@matschaffer matschaffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. I noticed a few empty graphs but maybe just no data, we can also fix those in followup.

Stack_Monitoring_-_Elasticsearch_-_Elastic

Stack_Monitoring_-_Elasticsearch_-_Nodes_-_runTask-0_-_Overview_-_Elastic

One thing that was a little curious is that the monitoring indices are getting created as dated data streams. But this is more a factor of metricbeat than kibana so shouldn't block this PR.

Screen Shot 2021-12-23 at 12 23 04

@matschaffer
Copy link
Contributor

I also didn't test CCR (or ML). I'll keep exploring the elastic-stack-testing today to see if that might offer a good place to exercise all these different stack permutations.

@matschaffer
Copy link
Contributor

Looks like I/O Operations Rate (/s) is blank on internal collections as well. I do get indexing latency though.

@matschaffer
Copy link
Contributor

Screen Shot 2021-12-23 at 15 11 45

I worked out how to get CCR going from a localDistro build of ES. Looks good via metricbeat monitoring.

@klacabane klacabane merged commit 3dcfdef into elastic:main Dec 23, 2021
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Dec 23, 2021
@klacabane @kibanamachine
[Stack monitoring] adapt ES queries to updated document structure (el…
cd148a2 
…astic#121641)

* adapt ES queries to updated document structure

* lint

* Revert "lint"

This reverts commit 2ad61e4.

* Revert "adapt ES queries to updated document structure"

This reverts commit 25d4357.

* fix cluster state property accesses

* only query index_recovery with ecs compliant indices

* lint

* setup/teardown to support datastreams

* lint

* update _mb archived data to target .monitoring index

* load archive with useCreate for datastream compatibility

* Revert "update _mb archived data to target .monitoring index"

This reverts commit 7bad361.

* ignore non-elasticsearch archived documents

* Revert "ignore non-elasticsearch archived documents"

This reverts commit 26daeb7.

* use correct index for archived documents

* add datastream lifecycle helper for api-integration tests

* use datastream lifecycle methods

* transform ccr_mb archive

* clean up datastreams

* lint

* load _mb data

Co-authored-by: Kibana Machine <[email protected]>
@kibanamachine kibanamachine mentioned this pull request Dec 23, 2021
Merged
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
✅ 8.0

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Dec 23, 2021
@kibanamachine @klacabane
[Stack monitoring] adapt ES queries to updated document structure (#1…
cf37e5a 
…21641) (#121949)

* adapt ES queries to updated document structure

* lint

* Revert "lint"

This reverts commit 2ad61e4.

* Revert "adapt ES queries to updated document structure"

This reverts commit 25d4357.

* fix cluster state property accesses

* only query index_recovery with ecs compliant indices

* lint

* setup/teardown to support datastreams

* lint

* update _mb archived data to target .monitoring index

* load archive with useCreate for datastream compatibility

* Revert "update _mb archived data to target .monitoring index"

This reverts commit 7bad361.

* ignore non-elasticsearch archived documents

* Revert "ignore non-elasticsearch archived documents"

This reverts commit 26daeb7.

* use correct index for archived documents

* add datastream lifecycle helper for api-integration tests

* use datastream lifecycle methods

* transform ccr_mb archive

* clean up datastreams

* lint

* load _mb data

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kevin Lacabane <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@neptunian neptunian neptunian approved these changes

@matschaffer matschaffer matschaffer approved these changes

Assignees

@klacabane klacabane

Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v8.0.0 v8.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@klacabane @elasticmachine @kibana-ci @neptunian @matschaffer @kibanamachine