Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stack monitoring - logstash 8.0 metricbeat standalone compatibility #122177

Merged
merged 18 commits into from
Jan 5, 2022

Conversation

klacabane
Copy link
Contributor

@klacabane klacabane commented Dec 31, 2021

Summary

This change introduces the updates required to make the SM logstash section compatible with metricbeat 8.x.

  • Updated query filters to target correct metricset name (node_stats instead of stats, see sample event)
  • Logstash stats metricsets include a nested property defining the pipelines that we aggregate over in multiple places. A nested property can't be aliased so we need to define two aggregations for each legacy and metricbeat property path (see pipelines_nested and pipelines_nested_mb). The implementation of the aggregation for the metricbeat path was incomplete and fixed by this change
  • Updated logstash metricbeat apis and functional tests to run against the new .monitoring-{product}-8-* datastreams. updated the es-archives with this script

Testing

You can find relevant informations to build each service in this draft doc

  • start an ES service that includes this mapping change
  • start a logstash service
  • start a metricbeat instance that monitors logstash and the cluster (required to load the view, logstash-only documents will load the no-data screen)
  • verify data is loaded in .monitoring-{product}-8-*
  • verify each logstash screen is loading and showing data

@klacabane
Copy link
Contributor Author

@elasticmachine merge upstream

@klacabane klacabane self-assigned this Jan 4, 2022
@klacabane klacabane added auto-backport Deprecated - use backport:version if exact versions are needed backport-v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v8.0.0 v8.1.0 labels Jan 4, 2022
@klacabane klacabane marked this pull request as ready for review January 4, 2022 15:57
@klacabane klacabane requested a review from a team as a code owner January 4, 2022 15:57
@elasticmachine
Copy link
Contributor

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

@klacabane klacabane changed the title 121346 logstash 8 compatibility Stack monitoring - logstash 8.0 metricbeat standalone compatibility Jan 4, 2022
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
monitoring 445.6KB 445.7KB +20.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @klacabane

Copy link
Contributor

@matschaffer matschaffer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks solid to me code wise. Some nice cleanups.

Also testing per the instructions you provided on my observability-dev PR looked good.

@klacabane klacabane merged commit 72297e5 into elastic:main Jan 5, 2022
@kibanamachine
Copy link
Contributor

💔 Backport failed

Status Branch Result
8.0 Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 122177

klacabane added a commit to klacabane/kibana that referenced this pull request Jan 5, 2022
…lastic#122177)

* use node_stats metricset filter

* fix pipeline dropdown redirection

* add get_pipeline metricbeat aggregation

* add get_pipeline_vertex metricbeat aggregation

* update logstash metricbeat archived data

* merge_pipeline_versions

* type parameter

* use datastream for mb functional tests

* fix metricbeat apis tests

* lint

* lint

* lint

* fix multicluster api test

* lint

* fix singlecluster_green_gold_mb tests

* lint

* add test case

Co-authored-by: Kibana Machine <[email protected]>
# Conflicts:
#	x-pack/plugins/monitoring/server/lib/logstash/get_logstash_for_clusters.ts
klacabane added a commit that referenced this pull request Jan 5, 2022
…lity (#122177) (#122345)

* Stack monitoring - logstash 8.0 metricbeat standalone compatibility  (#122177)

* use node_stats metricset filter

* fix pipeline dropdown redirection

* add get_pipeline metricbeat aggregation

* add get_pipeline_vertex metricbeat aggregation

* update logstash metricbeat archived data

* merge_pipeline_versions

* type parameter

* use datastream for mb functional tests

* fix metricbeat apis tests

* lint

* lint

* lint

* fix multicluster api test

* lint

* fix singlecluster_green_gold_mb tests

* lint

* add test case

Co-authored-by: Kibana Machine <[email protected]>
# Conflicts:
#	x-pack/plugins/monitoring/server/lib/logstash/get_logstash_for_clusters.ts

* remove bluebird import
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v8.0.0 v8.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants