[Monitoring] JVM memory usage alert #79039

igoristic · 2020-09-30T21:29:59Z

Resolves #74824

This is part of the "Additional Alerting" effort for Stack Monitoring

The check calculates each data node to make sure the JVM memory usage is below the implied threshold.

Testing:

Create a Stack Monitoring environment
Through the regular Setup Mode > Alert Edit flow/ux, set the threshold to something low (like 10%)

*Note: that it might take a couple of minutes for the notification to show up in the UI

elasticmachine · 2020-09-30T21:30:02Z

Pinging @elastic/stack-monitoring (Team:Monitoring)

chrisronline

Overall, looking good! I found a few things

x-pack/plugins/monitoring/server/alerts/memory_usage_alert.ts

chrisronline · 2020-10-01T14:20:45Z

x-pack/plugins/monitoring/server/alerts/memory_usage_alert.ts

+      nextSteps: [
+        createLink(
+          i18n.translate('xpack.monitoring.alerts.memoryUsage.ui.nextSteps.tuneThreadPools', {
+            defaultMessage: '#start_linkTune thread pools#end_link',


I don't see this listed in Ravi's doc. Was it a last minute addition?

It wasn't on the list, but based on my research it is also a valuable resource to mitigate high memory usage. cc: @ravikesarwani wdyt?

x-pack/plugins/monitoring/server/alerts/memory_usage_alert.ts

…emory-usage

chrisronline · 2020-10-02T13:15:24Z

x-pack/plugins/monitoring/server/alerts/base_alert.ts

+    if (ccs) {
+      globalState.push(`ccs:${ccs}`);
+    }
+    globalState.push('refreshInterval:(pause:!f,value:10000),time:(from:now-1h,to:now)');


Why add this?

To keep it consistent with our defaults, otherwise it'll use timerange default of 15min, which will then persist to next state once they start navigating the app

But our defaults are defined here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/monitoring/public/plugin.ts#L134?

Yeah, but it won't transfer to url_state unless we explicitly provide it (like I'm doing above). You can test this for yourself

Sorry, but I'm still not following.

This is what I'm seeing which seems fine:

There isn't much to figure out. Like I said it falls back to defaults, and for me it's:

Anyways, we're digressing here. These are all trivial things that are not really related to the scope of this PR and can be addressed/debated as a separate discussion/issue

My defaults are the same:

It also relies on localStorage. Maybe try clearing it, or at least the following key:
kibana.timepicker.timeHistory: "[{"from":"now-1h","to":"now"},{"from":"now-15m","to":"now"}]"

I agree it isn't related to this PR and we should defer the conversation, but we should have it at some point. I'm curious if there is an actual bug around this code that we need to fix.

Hey catching up on this, I think introducing more than one place to define defaults is probably not a good idea and we should understand why it's needed before we introduce that kind of foot-gun.

For now, if the goal is to get this alert in before FF, I recommend removing that line and then during the test plan phase try to understand why you both are seeing these different behaviors, and then document that back on this PR or in a new ticket. Hopefully it's not necessary at all, but if it is, we should export the defaults as a constant and then use it in both places at the very least.

x-pack/plugins/monitoring/server/alerts/memory_usage_alert.ts

x-pack/plugins/monitoring/server/lib/alerts/fetch_memory_usage_node_stats.ts

…emory-usage

igoristic · 2020-10-05T11:00:54Z

@elasticmachine merge upstream

kibanamachine · 2020-10-05T12:26:56Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 15dc2da

Metrics [docs]

@kbn/optimizer bundle module count

id	before	after	diff
`monitoring`	635	638	+3

async chunks size

id	before	after	diff
`monitoring`	1.2MB	1.2MB	+60.0B

distributable file count

id	before	after	diff
`default`	47085	47087	+2

page load bundle size

id	before	after	diff
`monitoring`	222.7KB	247.7KB	+25.0KB

History

💔 Build #79342 failed 7405c18
💚 Build #79298 succeeded 5d28f00
💚 Build #79278 succeeded 635dfd0
💚 Build #79029 succeeded 7ddae02
💔 Build #79022 failed 509be7e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

chrisronline

LGTM!

* Memory usage first draft * Fixed tests * CR feedback * Feedback and tests * Added size to optimize query * Removed scheduled check * Removed globalstate date Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

igoristic · 2020-10-05T19:19:08Z

Backport:
7.x: 246f163

igoristic added 2 commits September 30, 2020 17:14

Memory usage first draft

f8cba32

Fixed tests

5320edc

igoristic added release_note:enhancement Team:Monitoring Stack Monitoring team v8.0.0 Feature:Stack Monitoring v7.10.0 labels Sep 30, 2020

igoristic requested a review from a team September 30, 2020 21:30

igoristic added the review label Sep 30, 2020

chrisronline suggested changes Oct 1, 2020

View reviewed changes

igoristic added 2 commits October 1, 2020 21:22

Merge branch 'master' of https://github.com/elastic/kibana into jvm-m…

f08e5f6

…emory-usage

CR feedback

509be7e

igoristic requested a review from chrisronline October 2, 2020 01:45

igoristic added 2 commits October 1, 2020 22:29

Merge branch 'master' of https://github.com/elastic/kibana into jvm-m…

14c456e

…emory-usage

Feedback and tests

7ddae02

chrisronline suggested changes Oct 2, 2020

View reviewed changes

igoristic added 2 commits October 2, 2020 13:51

Merge branch 'master' of https://github.com/elastic/kibana into jvm-m…

132ca13

…emory-usage

Added size to optimize query

635dfd0

igoristic requested a review from chrisronline October 2, 2020 18:07

igoristic added 3 commits October 2, 2020 15:04

Removed scheduled check

5d28f00

Removed globalstate date

72823a0

Merge branch 'master' of https://github.com/elastic/kibana into jvm-m…

7405c18

…emory-usage

igoristic added this to the Logs UI 7.10 milestone Oct 5, 2020

Merge branch 'master' into jvm-memory-usage

15dc2da

sgrodzicki modified the milestones: Logs UI 7.10, Stack Monitoring UI 7.10 Oct 5, 2020

chrisronline approved these changes Oct 5, 2020

View reviewed changes

igoristic merged commit f490268 into elastic:master Oct 5, 2020

igoristic deleted the jvm-memory-usage branch October 5, 2020 15:22

igoristic mentioned this pull request Oct 5, 2020

[7.x] [Monitoring] JVM memory usage alert (#79039) #79512

Merged

igoristic added backported and removed review labels Oct 5, 2020

chrisronline mentioned this pull request Dec 14, 2020

[Stack Monitoring] [Test Scenario] Out of the box alerting #85841

Closed

23 tasks

chrisronline mentioned this pull request Mar 1, 2021

[Stack Monitoring] [Test Scenario] Out of the box alerting #93072

Closed

24 tasks

simianhacker mentioned this pull request Apr 29, 2021

[Stack Monitoring] [Test Scenario] Out of the box alerting #98765

Closed

24 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Monitoring] JVM memory usage alert #79039

[Monitoring] JVM memory usage alert #79039

igoristic commented Sep 30, 2020

elasticmachine commented Sep 30, 2020

chrisronline left a comment

chrisronline Oct 1, 2020

igoristic Oct 2, 2020

chrisronline Oct 2, 2020

igoristic Oct 2, 2020

chrisronline Oct 2, 2020

igoristic Oct 2, 2020

chrisronline Oct 2, 2020

igoristic Oct 2, 2020

chrisronline Oct 2, 2020

igoristic Oct 2, 2020

chrisronline Oct 2, 2020

jasonrhodes Oct 2, 2020

igoristic commented Oct 5, 2020

kibanamachine commented Oct 5, 2020

chrisronline left a comment

igoristic commented Oct 5, 2020

[Monitoring] JVM memory usage alert #79039

[Monitoring] JVM memory usage alert #79039

Conversation

igoristic commented Sep 30, 2020

elasticmachine commented Sep 30, 2020

chrisronline left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

igoristic commented Oct 5, 2020

kibanamachine commented Oct 5, 2020

💚 Build Succeeded

Metrics [docs]

@kbn/optimizer bundle module count

async chunks size

distributable file count

page load bundle size

History

chrisronline left a comment

Choose a reason for hiding this comment

igoristic commented Oct 5, 2020