[ML] Transforms: uses health information for alerting rule #152561

darnautov · 2023-03-02T12:15:12Z

Summary

Replaces the "Errors in transform messages" check with "Unhealthy transform" that relies on the health info from the transform stats.

Alerting context has been extended with health_status and issues properties for both checks.

The "Errors in transform messages" check will be still working for existing rules if it's been set explicitly.

Example of the alert message:

[farequote transform health rule] farequote transform health check result:
Transforms fq_response_times_continuous, tr_fail_02 are unhealthy.
  Transform ID: fq_response_times_continuous
  Transform state: started
  Transform health status: yellow
  Issue: Transform indexer failed
  Issue count: 2
  Issue details: Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[farequote-2019][0]]. Consider using 'allow_partial_search_results' setting to bypass this error.
  First occurrence: Mar 6, 2023 @ 09:43:16.532
  Node name: node-0

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2023-03-02T12:15:17Z

Pinging @elastic/ml-ui (:ml)

x-pack/plugins/transform/common/constants.ts

szabosteve

UI text LGTM. (Left one note on Pete's comment.)

x-pack/plugins/transform/common/constants.ts

darnautov · 2023-03-06T16:30:47Z

@elasticmachine merge upstream

walterra · 2023-03-07T07:50:06Z

...nsform/server/lib/alerting/transform_health_rule_type/register_transform_health_rule_type.ts

 import { PLUGIN, TRANSFORM_RULE_TYPE } from '../../../../common/constants';
 import { transformHealthRuleParams, TransformHealthRuleParams } from './schema';
 import { transformHealthServiceProvider } from './transform_health_service';

-export interface BaseResponse {
+export interface TransformHealth {
+  status: 'green' | 'unknown' | 'yellow' | 'red';


In x-pack/plugins/transform/common/constants.ts we have export type TransformHealth which you could reuse here. To avoid a naming clash we could consolidate this a bit further. In x-pack/plugins/transform/common/types/transform_stats.ts the full transform stats are defined where health is a part of. You could your interface here move to that file and make it part of TransformStats where it's defined inline now.

Updated in d1b6e8b

walterra · 2023-03-07T07:53:35Z

...plugins/transform/server/lib/alerting/transform_health_rule_type/transform_health_service.ts

+/**
+ * TODO update types in the es client
+ */
+type TransformGetTransformStats = TransformGetTransformStatsTransformStats & {


Maybe this can also be combined/cleaned up with what we have in x-pack/plugins/transform/common/types/transform_stats.ts.

Updated in d1b6e8b

walterra · 2023-03-07T07:57:56Z

...plugins/transform/server/lib/alerting/transform_health_rule_type/transform_health_service.ts

+        await esClient.transform.getTransformStats({
+          transform_id: transformIds.join(','),
+        })
+      ).transforms as TransformGetTransformStats[];


getTransformStats itself has a try/catch block and might also return IHttpFetchError so I'm not sure if we can assume there will also be .transforms available.

Sorry, I misread the code, it's wrapped the other way around so the try/catch is on the outer level when the stats are requested via API call. It looks like in this case there's then no try/catch at all, should we add one?

If we are unable to fetch transform stats, the rule can't be executed. It's ok to throw an error because it'll end up in the Rules UI.

Sounds good, thanks for clarifying!

…api-update' into ml-144158-transform-health-rule-api-update

x-pack/plugins/transform/common/utils/alerts.ts

peteharverson

Tested and looks good, apart from a question I have around the behavior of mapping the enabled state of the previous error message check to the new unhealthy transform check.

walterra

Tested and latest changes LGTM (pending Pete's question).

peteharverson

Latest changes for the enabled state of rules created with the previous config LGTM.

peteharverson · 2023-03-07T14:13:10Z

x-pack/plugins/transform/common/constants.ts

+    description: i18n.translate(
+      'xpack.transform.alertTypes.transformHealth.healthCheckDescription',
+      {
+        defaultMessage: 'Get alerts if a transform health status is not green.',


I was wondering if this should be consistent with the messaging in the transforms list, using Healthy in place of green, but Get alerts if a transform health status is not healthy. reads a bit odd.

kibana-ci · 2023-03-07T14:49:56Z

💚 Build Succeeded

Buildkite Build
Commit: 798f236

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`transform`	371.0KB	371.1KB	+102.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`transform`	16.8KB	17.7KB	+892.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	428	430	+2
`transform`	31	32	+1
total			+3

Total ESLint disabled count

id	before	after	diff
`securitySolution`	506	508	+2
`transform`	34	35	+1
total			+3

History

💚 Build #112332 succeeded 5ff2c64
💚 Build #112124 succeeded 922e641
💚 Build #111678 succeeded f5fc494

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @darnautov

…2561)

darnautov added 4 commits March 1, 2023 14:48

getUnhealthyTransformsReport

93dd0e7

disable errorMessages check

18bb3d5

update context, update default template

e5ade23

use fields format

f5fc494

darnautov added release_note:enhancement :ml Feature:Transforms ML transforms Feature:Alerting/RuleTypes Issues related to specific Alerting Rules Types Team:ML Team label for ML (also use :ml) v8.8.0 labels Mar 2, 2023

darnautov requested review from walterra and peteharverson March 2, 2023 12:15

darnautov self-assigned this Mar 2, 2023

darnautov requested a review from a team as a code owner March 2, 2023 12:15

peteharverson reviewed Mar 6, 2023

View reviewed changes

x-pack/plugins/transform/common/constants.ts Outdated Show resolved Hide resolved

darnautov requested a review from szabosteve March 6, 2023 11:14

szabosteve approved these changes Mar 6, 2023

View reviewed changes

x-pack/plugins/transform/common/constants.ts Outdated Show resolved Hide resolved

fix message

e5afe62

darnautov requested a review from peteharverson March 6, 2023 16:01

Merge branch 'main' into ml-144158-transform-health-rule-api-update

922e641

walterra reviewed Mar 7, 2023

View reviewed changes

darnautov added 2 commits March 7, 2023 11:46

refactor types

d1b6e8b

Merge remote-tracking branch 'origin/ml-144158-transform-health-rule-…

5ff2c64

…api-update' into ml-144158-transform-health-rule-api-update

darnautov requested a review from walterra March 7, 2023 10:47

peteharverson reviewed Mar 7, 2023

View reviewed changes

x-pack/plugins/transform/common/utils/alerts.ts Outdated Show resolved Hide resolved

peteharverson reviewed Mar 7, 2023

View reviewed changes

walterra approved these changes Mar 7, 2023

View reviewed changes

sets healthCheck based on the errorMessages

798f236

darnautov requested a review from peteharverson March 7, 2023 13:31

peteharverson mentioned this pull request Mar 7, 2023

[ML] Increase Test Coverage 8.8.0 #152009

Closed

7 tasks

peteharverson approved these changes Mar 7, 2023

View reviewed changes

darnautov enabled auto-merge (squash) March 7, 2023 14:26

darnautov merged commit 56a7851 into elastic:main Mar 7, 2023

kibanamachine added the backport:skip This commit does not require backporting label Mar 7, 2023

bmorelli25 pushed a commit to bmorelli25/kibana that referenced this pull request Mar 10, 2023

[ML] Transforms: use health information for alerting rule (elastic#15…

d339b75

…2561)

szabosteve changed the title ~~[ML] Transforms: use health information for alerting rule~~ [ML] Transforms: uses health information for alerting rule Apr 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Transforms: uses health information for alerting rule #152561

[ML] Transforms: uses health information for alerting rule #152561

darnautov commented Mar 2, 2023 •

edited

Loading

elasticmachine commented Mar 2, 2023

szabosteve left a comment

darnautov commented Mar 6, 2023

walterra Mar 7, 2023

darnautov Mar 7, 2023

walterra Mar 7, 2023

darnautov Mar 7, 2023

walterra Mar 7, 2023

walterra Mar 7, 2023

darnautov Mar 7, 2023

walterra Mar 7, 2023

peteharverson left a comment

walterra left a comment

peteharverson left a comment

peteharverson Mar 7, 2023

kibana-ci commented Mar 7, 2023

ESLint disabled line counts

Total ESLint disabled count

[ML] Transforms: uses health information for alerting rule #152561

[ML] Transforms: uses health information for alerting rule #152561

Conversation

darnautov commented Mar 2, 2023 • edited Loading

Summary

Checklist

elasticmachine commented Mar 2, 2023

szabosteve left a comment

Choose a reason for hiding this comment

darnautov commented Mar 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

walterra left a comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Mar 7, 2023

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

ESLint disabled line counts

Total ESLint disabled count

History

darnautov commented Mar 2, 2023 •

edited

Loading