Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[elasticsearch module] serialize shards properties #30408

Merged
merged 4 commits into from
Feb 16, 2022
Merged

[elasticsearch module] serialize shards properties #30408

merged 4 commits into from
Feb 16, 2022

Conversation

klacabane
Copy link
Contributor

@klacabane klacabane commented Feb 15, 2022 •
edited
Loading

Summary

Rel elastic/kibana#125315

In 8.0 most of the elasticsearch.index.shards properties are omitted when serializing the objects. Stack Monitoring relies on the shards.primaries property (I didn't find any references of the other omitted fields) to compute an alert state. This change re-enables the property.

Testing

  • started metricbeat with elasticsearch-xpack enabled
  • inspected metricset.name: index documents and ensured it contained properties

Screenshot 2022-02-15 at 19 34 08

@klacabane klacabane added Team:Integrations Label for the Integrations team Feature:Stack Monitoring v8.0.0 backport-v8.0.0 Automated backport with mergify v8.1.0 v8.2.0 backport-v8.1.0 Automated backport with mergify labels Feb 15, 2022
@klacabane klacabane self-assigned this Feb 15, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/stack-monitoring (Stack monitoring)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 15, 2022
@botelastic
Copy link

botelastic bot commented Feb 15, 2022

This pull request doesn't have a Team:<team> label.

@klacabane
Copy link
Contributor Author

@sayden any idea why we omit the shards properties in 8.0 ?

@klacabane klacabane mentioned this pull request Feb 15, 2022
Closed
4 tasks
@elasticmachine
Copy link
Collaborator

elasticmachine commented Feb 15, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-02-16T11:05:45.282+0000

  • Duration: 95 min 23 sec

Test stats 🧪

Test Results
Failed 0
Passed 6688
Skipped 1733
Total 8421

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@jasonrhodes
Copy link
Member

@sayden any idea why we omit the shards properties in 8.0 ?

I imagine it is somehow related to this investigation: elastic/kibana#125258

i.e. we've known for a while that we collect too much shard information, so we have been trying to reduce that footprint piece by piece. If there is a way to only use a single field for what we need in alerts, that'd be great, but if that's not feasible / stable, then we should prioritize making the alert work for 8.0.1 for now and address this later.

@jasonrhodes jasonrhodes requested a review from a team February 16, 2022 02:16
@jasonrhodes jasonrhodes mentioned this pull request Feb 16, 2022
Merged
4 tasks
sayden
sayden approved these changes Feb 16, 2022
View reviewed changes
Copy link
Contributor

@sayden sayden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Regarding the question, at some point we tried to reduce the amount of data we were ingesting related to shards because it's one of those metricsets were we stored a lot of documents in each period. Imagine, a 1000 nodes cluster will generate 1000 docs every period. So we try to reduce the size of each of those documents.

@kvch kvch merged commit 2571b52 into elastic:main Feb 16, 2022
mergify bot pushed a commit that referenced this pull request Feb 16, 2022
@klacabane
[elasticsearch module] serialize shards properties (#30408)
9e7bd23 
(cherry picked from commit 2571b52)
@mergify mergify bot mentioned this pull request Feb 16, 2022
Merged
mergify bot pushed a commit that referenced this pull request Feb 16, 2022
@klacabane
[elasticsearch module] serialize shards properties (#30408)
598c932 
(cherry picked from commit 2571b52)
@mergify mergify bot mentioned this pull request Feb 16, 2022
Merged
kvch pushed a commit that referenced this pull request Feb 16, 2022
@mergify @klacabane
[elasticsearch module] serialize shards properties (#30408) (#30430)
303638d 
(cherry picked from commit 2571b52)

Co-authored-by: Kevin Lacabane <[email protected]>
kvch pushed a commit that referenced this pull request Feb 16, 2022
@mergify @klacabane
[elasticsearch module] serialize shards properties (#30408) (#30431)
11a544b 
(cherry picked from commit 2571b52)

Co-authored-by: Kevin Lacabane <[email protected]>
v1v added a commit to v1v/beats that referenced this pull request Feb 21, 2022
@v1v
Merge remote-tracking branch 'upstream/main' into feature/use-with-ki…
6c6cce5 
…nd-k8s-env

* upstream/main:
  fix typos and improve sentences (elastic#30432)
  Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs (elastic#30440)
  {,x-pack/}auditbeat: replace uses of github.com/pkg/errors with stdlib equivalents (elastic#30321)
  Spelling fix (elastic#30439)
  packetbeat/beater: make sure Npcap installation runs before interfaces are needed in all cases (elastic#30438)
  Add BC about Homebrew no longer being available in 8.0 (elastic#30419)
  Install gawk as a replacement for mawk in Docker containers. (elastic#30452)
  Clean up python-related system tests (elastic#30415)
  Fix TestNewModuleRegistry flakiness (elastic#30453)
  [Filebeat] [auditd]: Support EXECVE events with truncated argument list (elastic#30382)
  Set `log.offset` to the start of the reported line in filestream (elastic#30445)
  clarify SelectedPackageTypes meaning and improve its usage (elastic#30142)
  [elasticsearch module] serialize shards properties (elastic#30408)
  Add docs about hints and templates autodiscovery priority (elastic#30343)
v1v added a commit to v1v/beats that referenced this pull request Feb 22, 2022
@v1v
Merge remote-tracking branch 'upstream/main' into feature/refactor-pa…
29471a7 
…ckaging-docker

* upstream/main: (26 commits)
  Update docker/distribution to 2.8.0 (elastic#30462)
  Add `parsers` examples to `filestream` reference configuration (elastic#30529)
  extend documentation about setting orchestrator.cluster fields (elastic#30518)
  Forward-port 8.0.1 changelog to main (elastic#30522)
  Switch skip to use `CI` (elastic#30512)
  packetbeat/beater: don't attempt to install npcap when already installed (elastic#30509)
  Fix Docker module: rename fields on dashboards (elastic#30500)
  fix typos and improve sentences (elastic#30432)
  Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs (elastic#30440)
  {,x-pack/}auditbeat: replace uses of github.com/pkg/errors with stdlib equivalents (elastic#30321)
  Spelling fix (elastic#30439)
  packetbeat/beater: make sure Npcap installation runs before interfaces are needed in all cases (elastic#30438)
  Add BC about Homebrew no longer being available in 8.0 (elastic#30419)
  Install gawk as a replacement for mawk in Docker containers. (elastic#30452)
  Clean up python-related system tests (elastic#30415)
  Fix TestNewModuleRegistry flakiness (elastic#30453)
  [Filebeat] [auditd]: Support EXECVE events with truncated argument list (elastic#30382)
  Set `log.offset` to the start of the reported line in filestream (elastic#30445)
  clarify SelectedPackageTypes meaning and improve its usage (elastic#30142)
  [elasticsearch module] serialize shards properties (elastic#30408)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sayden sayden sayden approved these changes

@kvch kvch kvch approved these changes

Assignees

@klacabane klacabane

Labels
backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify Feature:Stack Monitoring Team:Integrations Label for the Integrations team v8.0.0 v8.1.0 v8.2.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@klacabane @elasticmachine @jasonrhodes @kvch @sayden