[ML] Prevent duplicate notifications about the same anomaly result #91485
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
darnautov
added
release_note:enhancement
:ml
Feature:Anomaly Detection
darnautov
requested review from
walterra,
alvarezmelissa87 and
peteharverson
|
Pinging @elastic/ml-ui (:ml) |
|
Code LGTM aside from Pete's comment. Will test when PR is updated 👌 |
| } else if (source.result_type === ANOMALY_RESULT_TYPE.RECORD) { | ||
| const fieldName = getEntityFieldName(source); | ||
| const fieldValue = getEntityFieldValue(source); | ||
| alertInstanceKey += `_${source.detector_index}_${source.function}_${fieldName}_${fieldValue}`; |
There was a problem hiding this comment.
source.detector_index is undefined in the key my test generated. Is this available in the source ?
There was a problem hiding this comment.
forgot to include it into the source, fixed in 61764c5
peteharverson
approved these changes
Feb 17, 2021
10 tasks
darnautov
added
the
auto-backport
💚 Build SucceededMetrics [docs]Page load bundle
History
To update your PR or re-run it, just comment with: |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Feb 17, 2021
…lastic#91485) * [ML] check kibana even logs for existing alert instance * [ML] create alert instance key, add check for alert id * [ML] use anomaly_utils, check interval gap * [ML] add detector index * [ML] fix unit test * [ML] include detector_index into source
kibanamachine
added a commit
that referenced
this pull request
Feb 17, 2021
…91485) (#91720) * [ML] check kibana even logs for existing alert instance * [ML] create alert instance key, add check for alert id * [ML] use anomaly_utils, check interval gap * [ML] add detector index * [ML] fix unit test * [ML] include detector_index into source Co-authored-by: Dima Arnautov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Related issue #88940.
The alerting framework allows configuring when to get notified. By default, it's "Only on status change", which means that in the case of the ML Anomaly detection alert schedules
Anomaly score matched the conditionaction on each execution multiple times in a row, the user will be notified only on the first status change, so receiving duplicates won't be an issue.But setting it to "Every time alert is active" might result in multiple notifications for the same anomaly, depending on the check interval and the result bucket span. This PR adds a check to the ML alert executor for the existing alert instance with
anomaly_score_matchaction group in.kibana-event-log-*index that helps to avoid it.The alert instance key generated to check for duplicates is of the form
Buckets alerts:
jobId_highestRecordTimestampInfluencer alerts:
jobId_highestRecordTimestamp_influencerFieldName_influencerFieldValueRecord alerts:
jobId_highestRecordTimestamp_detectorIndex_function_entityFieldName_entityFieldValuee.g. for a record alert instance:
ecommerce_high_sum_total_sales_1613584800000_0_high_sum_customer_full_name.keyword_Rabbia Al PowellAlso, another scenario is possible when the check interval is significantly bigger than the result bucket span we look back during the alert condition execution. In that case, we risk missing the anomaly, hence using
previousStartedAttime from the previous execution helps to detect the check gap and use this interval as a time range for querying anomalies.How to test
You should get notified based on the selected action only once for a particular anomaly result.
Checklist