Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor MonitorRunner #143

Merged
merged 1 commit into from
Aug 20, 2021

Conversation

qreshi
Copy link
Contributor

@qreshi qreshi commented Aug 16, 2021

Signed-off-by: Mohammad Qureshi [email protected]

Issue #, if available: #86

Description of changes:
Refactor MonitorRunner to organize code in preparation for Bucket-Level Monitor changes.

CheckList:
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Mohammad Qureshi <[email protected]>
@qreshi qreshi requested a review from lezzago August 16, 2021 14:29
@qreshi
Copy link
Contributor Author

qreshi commented Aug 16, 2021

Given the way the commits/files were re-organized to prepare merging Bucket-Level Alerting into main, the tests/build are expected to fail until we get to the PR that updates the MonitorRunner logic. So I'll be overriding the approval workflow failures as long as the PRs are approved by the reviewers until we've caught up to the end of the dev branch. If at that point, there are any build failures, I'll fix them to pass (although I confirmed the head of the dev branch was stable before beginning to push these PRs).

@adityaj1107
Copy link
Contributor

Given the way the commits/files were re-organized to prepare merging Bucket-Level Alerting into main, the tests/build are expected to fail until we get to the PR that updates the MonitorRunner logic. So I'll be overriding the approval workflow failures as long as the PRs are approved by the reviewers until we've caught up to the end of the dev branch. If at that point, there are any build failures, I'll fix them to pass (although I confirmed the head of the dev branch was stable before beginning to push these PRs).

Can we push to a single branch (bucket-level-alerting-main?) in this repository and not to main branch? That would ensure all the changes for the bucket level alerting goes in a single commit and devs working locally can get those changes once whenever it is merged and not in increments. I am okay either way just wanted to voice my opinion. :)

@qreshi
Copy link
Contributor Author

qreshi commented Aug 18, 2021

Given the way the commits/files were re-organized to prepare merging Bucket-Level Alerting into main, the tests/build are expected to fail until we get to the PR that updates the MonitorRunner logic. So I'll be overriding the approval workflow failures as long as the PRs are approved by the reviewers until we've caught up to the end of the dev branch. If at that point, there are any build failures, I'll fix them to pass (although I confirmed the head of the dev branch was stable before beginning to push these PRs).

Can we push to a single branch (bucket-level-alerting-main?) in this repository and not to main branch? That would ensure all the changes for the bucket level alerting goes in a single commit and devs working locally can get those changes once whenever it is merged and not in increments. I am okay either way just wanted to voice my opinion. :)

Hmm, we don't keep main level branches for features on our repo. So say that we did create that to review each piece currently in the bucket-level-alerting-dev branch. Once each one is merged, we would still need to squash that and make one huge PR into main. At that time, not only would this PR be so large it would be difficult for anyone to look through it reasonably in posterity but we would also need to relate every single PR it was composed of should anyone want to look back at it later. Eventually the bucket-level-alerting branches would be deleted from the repo to keep it clean and then we would lose the incremental history since main only captured it as a single push.

I'm curious what others think though, in case one method is more conventional than the other.

import org.opensearch.script.ScriptService

/** Service that handles executing Triggers */
class TriggerService(val client: Client, val scriptService: ScriptService) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we using Client anywhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will update this and the address the comment below in a subsequent PR which has the full implementation of the TriggerService since some things changed between this commit and the final state.


private val logger = LogManager.getLogger(TriggerService::class.java)

fun isTriggerActionable(ctx: TriggerExecutionContext, result: TriggerRunResult): Boolean {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be a static method.

@qreshi qreshi merged commit 352d272 into opensearch-project:main Aug 20, 2021
@qreshi qreshi deleted the refactor-monitor-runner branch August 20, 2021 19:14
@dblock
Copy link
Member

dblock commented Aug 27, 2021

Did this break the build and still got merged?

@qreshi
Copy link
Contributor Author

qreshi commented Aug 27, 2021

Did this break the build and still got merged?

As is noted above, the way that the files were split to breakup the feature of Bucket-Level Alerting, the tests will fail until we catch up to one of the later changes in which it will be stable again, we aim to be there by today.

@dblock
Copy link
Member

dblock commented Aug 27, 2021

@qreshi Bundle build depends on you each repo to at least build without errors, so we can't do that moving forward :( Use branches for similar situations please.

@qreshi
Copy link
Contributor Author

qreshi commented Aug 27, 2021

@dblock Got it, we did have a dev branch to organize the changes in preparation for committing to main but what you're saying is you would have wanted the PRs to happen on that branch (or some other one that is not main) and then once it's reviewed up to a stable point have a big PR to merge it all in?

@adityaj1107
Copy link
Contributor

adityaj1107 commented Aug 27, 2021

dB. correct me if I am wrong but I think what you are saying is have a main branch whose build is stable and do whatever you want to do on other dev branches. Have a big PR with links to PRs on the feature-level-main branches etc. etc. or merging each PRs one by one in a short span of time (~1 day) to preserve history is a call which each repo owner can take.

adityaj1107 added a commit that referenced this pull request Sep 2, 2021
* Added release notes for OpenSearch 1.0.0.0. (#123) (#124)

Co-authored-by: AWSHurneyt <[email protected]>

* Add Integtest.sh for OpenSearch integtest setups (#121)

* Add integtest script to the repo

Signed-off-by: Peter Zhu <[email protected]>

* Add Alerting specific security param for integTest

Signed-off-by: Peter Zhu <[email protected]>

* Remove default assignee (#127)

Signed-off-by: Ashish Agrawal <[email protected]>

* Removing All Usages of Action Get Method Calls and adding the listeners (#130)

Signed-off-by: Aditya Jindal <[email protected]>

* Fix snapshot build and increment to 1.1.0. (#142)

Signed-off-by: dblock <[email protected]>

* Refactor MonitorRunner (#143)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update Bucket-Level Alerting RFC (#145)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add BucketSelector pipeline aggregation extension (#144)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Add AggregationResultBucket (#148)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Add ActionExecutionPolicy (#149)

* Add ActionExecutionPolicy

Signed-off-by: Mohammad Qureshi <[email protected]>

* Throw exception if there is an invalid field in PER_ALERT config when parsing

Signed-off-by: Mohammad Qureshi <[email protected]>

* Don't allow throttle to be configured for PerExecutionActionScope at the data class level since it is not supported yet

Signed-off-by: Mohammad Qureshi <[email protected]>

* Refactor Monitor and Trigger to split into Query-Level and Bucket-Lev… (#150)

* Refactor Monitor and Trigger to split into Query-Level and Bucket-Level Monitors

Signed-off-by: Mohammad Qureshi <[email protected]>

* Require condition to not be null when parsing Bucket-Level Trigger

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update InputService for Bucket-Level Alerting (#152)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Update TriggerService for Bucket-Level Alerting (#153)

* Update TriggerService for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Remove client from TriggerService

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update AlertService for Bucket-Level Alerting (#154)

* Update AlertService for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Move Alert search size for Bucket-Level Monitors to a const

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add worksheets to help with testing (#151)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update MonitorRunner for Bucket-Level Alerting (#155)

* Update MonitorRunner for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update regressed comment in MonitorRunnerIT

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add TODO to break down runBucketLevelMonitor method in MonitorRunner

Signed-off-by: Mohammad Qureshi <[email protected]>

* Fix ktlint formatting issues (#156)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Execute Actions on runTrigger exceptions for Bucket-Level Monitor (#157)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Skip execution of Actions on ACKNOWLEDGED Alerts for Bucket-Level Monitors (#158)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Return first page of input results in MonitorRunResult for Bucket-Level Monitor (#159)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add setting to limit per alert action executions and don't save Alerts for test Bucket-Level Monitors (#161)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Fix bug in paginating multiple bucket paths for Bucket-Level Monitor (#163)

* Fix bug in paginating multiple bucket paths for Bucket-Level Monitor

Signed-off-by: Mohammad Qureshi <[email protected]>

* Change trigger after key conditionals to when statement

Signed-off-by: Mohammad Qureshi <[email protected]>

* Various bug fixes pertaining to throttling on PER_ALERT, saving COMPLETED Alerts and rewriting input query for Bucket-Level Monitors (#164)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Return only monitors for /monitors/_search. (#162)

* Return only monitors for /monitors/_search.

* Added missing imports

* Added additional check to the unit test

* Resolve default for ActionExecutionPolicy at runtime (#165)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: AWSHurneyt <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Co-authored-by: Ashish Agrawal <[email protected]>
Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]>
Co-authored-by: Mohammad Qureshi <[email protected]>
Co-authored-by: Rishabh Maurya <[email protected]>
Co-authored-by: Sriram <[email protected]>
@qreshi qreshi added feature A change that introduces a new unit of functionality refactor and removed feature A change that introduces a new unit of functionality labels Sep 8, 2021
AWSHurneyt added a commit that referenced this pull request Oct 15, 2021
…ledging more than 10 alerts at once. (#205)

* Added release notes for OpenSearch 1.0.0.0. (#123)

* Merge commits from the main branch to the 1.x branch.  (#133)

* Added release notes for OpenSearch 1.0.0.0. (#123) (#124)

Co-authored-by: AWSHurneyt <[email protected]>

* Add Integtest.sh for OpenSearch integtest setups (#121)

* Add integtest script to the repo

Signed-off-by: Peter Zhu <[email protected]>

* Add Alerting specific security param for integTest

Signed-off-by: Peter Zhu <[email protected]>

* Remove default assignee (#127)

Signed-off-by: Ashish Agrawal <[email protected]>

* Removing All Usages of Action Get Method Calls and adding the listeners (#130)

Signed-off-by: Aditya Jindal <[email protected]>

* Fix snapshot build and increment to 1.1.0. (#142)

Signed-off-by: dblock <[email protected]>

* Refactor MonitorRunner (#143)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update Bucket-Level Alerting RFC (#145)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add BucketSelector pipeline aggregation extension (#144)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Add AggregationResultBucket (#148)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Add ActionExecutionPolicy (#149)

* Add ActionExecutionPolicy

Signed-off-by: Mohammad Qureshi <[email protected]>

* Throw exception if there is an invalid field in PER_ALERT config when parsing

Signed-off-by: Mohammad Qureshi <[email protected]>

* Don't allow throttle to be configured for PerExecutionActionScope at the data class level since it is not supported yet

Signed-off-by: Mohammad Qureshi <[email protected]>

* Refactor Monitor and Trigger to split into Query-Level and Bucket-Lev… (#150)

* Refactor Monitor and Trigger to split into Query-Level and Bucket-Level Monitors

Signed-off-by: Mohammad Qureshi <[email protected]>

* Require condition to not be null when parsing Bucket-Level Trigger

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update InputService for Bucket-Level Alerting (#152)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Update TriggerService for Bucket-Level Alerting (#153)

* Update TriggerService for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Remove client from TriggerService

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update AlertService for Bucket-Level Alerting (#154)

* Update AlertService for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Move Alert search size for Bucket-Level Monitors to a const

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add worksheets to help with testing (#151)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update MonitorRunner for Bucket-Level Alerting (#155)

* Update MonitorRunner for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update regressed comment in MonitorRunnerIT

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add TODO to break down runBucketLevelMonitor method in MonitorRunner

Signed-off-by: Mohammad Qureshi <[email protected]>

* Fix ktlint formatting issues (#156)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Execute Actions on runTrigger exceptions for Bucket-Level Monitor (#157)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Skip execution of Actions on ACKNOWLEDGED Alerts for Bucket-Level Monitors (#158)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Return first page of input results in MonitorRunResult for Bucket-Level Monitor (#159)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add setting to limit per alert action executions and don't save Alerts for test Bucket-Level Monitors (#161)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Fix bug in paginating multiple bucket paths for Bucket-Level Monitor (#163)

* Fix bug in paginating multiple bucket paths for Bucket-Level Monitor

Signed-off-by: Mohammad Qureshi <[email protected]>

* Change trigger after key conditionals to when statement

Signed-off-by: Mohammad Qureshi <[email protected]>

* Various bug fixes pertaining to throttling on PER_ALERT, saving COMPLETED Alerts and rewriting input query for Bucket-Level Monitors (#164)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Return only monitors for /monitors/_search. (#162)

* Return only monitors for /monitors/_search.

* Added missing imports

* Added additional check to the unit test

* Resolve default for ActionExecutionPolicy at runtime (#165)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: AWSHurneyt <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Co-authored-by: Ashish Agrawal <[email protected]>
Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]>
Co-authored-by: Mohammad Qureshi <[email protected]>
Co-authored-by: Rishabh Maurya <[email protected]>
Co-authored-by: Sriram <[email protected]>

* Add release notes for 1.1.0.0 release (#166) (#167)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Remove default integtest.sh. (#181)

Signed-off-by: dblock <[email protected]>

* Add valid search filters. (#191)

* Add valid search filters.

* Added this fix to release notes

* Publish notification JARs checksums. (#197)

Signed-off-by: dblock <[email protected]>

* Also publish SHA 256 and 512 checksums. (#198)

* Also publish SHA 256 and 512 checksums.

Signed-off-by: dblock <[email protected]>

* Remove sonatype staging.

Signed-off-by: dblock <[email protected]>

* Fixed a bug that was preventing the AcknowledgeAlerts API from acknowledging more than 10 alerts at once. Signed-off-by: Thomas Hurney <[email protected]>

* Implemented integration tests to ensure fix for issue 203 is working as expected. Signed-off-by: Thomas Hurney <[email protected]>

* Refactored integ tests based on PR feedback, and listed the bug fix in the release notes. Signed-off-by: Thomas Hurney <[email protected]>

* Removing bug fixes from release notes. Currently discussing adding separate notes for this patch. Signed-off-by: Thomas Hurney <[email protected]>

Co-authored-by: Aditya Jindal <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Co-authored-by: Ashish Agrawal <[email protected]>
Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]>
Co-authored-by: Mohammad Qureshi <[email protected]>
Co-authored-by: Rishabh Maurya <[email protected]>
Co-authored-by: Sriram <[email protected]>
AWSHurneyt added a commit to AWSHurneyt/OpenSearch-Alerting that referenced this pull request Mar 30, 2022
…ject#133)

* Added release notes for OpenSearch 1.0.0.0. (opensearch-project#123) (opensearch-project#124)

Co-authored-by: AWSHurneyt <[email protected]>

* Add Integtest.sh for OpenSearch integtest setups (opensearch-project#121)

* Add integtest script to the repo

Signed-off-by: Peter Zhu <[email protected]>

* Add Alerting specific security param for integTest

Signed-off-by: Peter Zhu <[email protected]>

* Remove default assignee (opensearch-project#127)

Signed-off-by: Ashish Agrawal <[email protected]>

* Removing All Usages of Action Get Method Calls and adding the listeners (opensearch-project#130)

Signed-off-by: Aditya Jindal <[email protected]>

* Fix snapshot build and increment to 1.1.0. (opensearch-project#142)

Signed-off-by: dblock <[email protected]>

* Refactor MonitorRunner (opensearch-project#143)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update Bucket-Level Alerting RFC (opensearch-project#145)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add BucketSelector pipeline aggregation extension (opensearch-project#144)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Add AggregationResultBucket (opensearch-project#148)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Add ActionExecutionPolicy (opensearch-project#149)

* Add ActionExecutionPolicy

Signed-off-by: Mohammad Qureshi <[email protected]>

* Throw exception if there is an invalid field in PER_ALERT config when parsing

Signed-off-by: Mohammad Qureshi <[email protected]>

* Don't allow throttle to be configured for PerExecutionActionScope at the data class level since it is not supported yet

Signed-off-by: Mohammad Qureshi <[email protected]>

* Refactor Monitor and Trigger to split into Query-Level and Bucket-Lev… (opensearch-project#150)

* Refactor Monitor and Trigger to split into Query-Level and Bucket-Level Monitors

Signed-off-by: Mohammad Qureshi <[email protected]>

* Require condition to not be null when parsing Bucket-Level Trigger

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update InputService for Bucket-Level Alerting (opensearch-project#152)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: Rishabh Maurya <[email protected]>

* Update TriggerService for Bucket-Level Alerting (opensearch-project#153)

* Update TriggerService for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Remove client from TriggerService

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update AlertService for Bucket-Level Alerting (opensearch-project#154)

* Update AlertService for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Move Alert search size for Bucket-Level Monitors to a const

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add worksheets to help with testing (opensearch-project#151)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update MonitorRunner for Bucket-Level Alerting (opensearch-project#155)

* Update MonitorRunner for Bucket-Level Alerting

Signed-off-by: Mohammad Qureshi <[email protected]>

* Update regressed comment in MonitorRunnerIT

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add TODO to break down runBucketLevelMonitor method in MonitorRunner

Signed-off-by: Mohammad Qureshi <[email protected]>

* Fix ktlint formatting issues (opensearch-project#156)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Execute Actions on runTrigger exceptions for Bucket-Level Monitor (opensearch-project#157)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Skip execution of Actions on ACKNOWLEDGED Alerts for Bucket-Level Monitors (opensearch-project#158)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Return first page of input results in MonitorRunResult for Bucket-Level Monitor (opensearch-project#159)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Add setting to limit per alert action executions and don't save Alerts for test Bucket-Level Monitors (opensearch-project#161)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Fix bug in paginating multiple bucket paths for Bucket-Level Monitor (opensearch-project#163)

* Fix bug in paginating multiple bucket paths for Bucket-Level Monitor

Signed-off-by: Mohammad Qureshi <[email protected]>

* Change trigger after key conditionals to when statement

Signed-off-by: Mohammad Qureshi <[email protected]>

* Various bug fixes pertaining to throttling on PER_ALERT, saving COMPLETED Alerts and rewriting input query for Bucket-Level Monitors (opensearch-project#164)

Signed-off-by: Mohammad Qureshi <[email protected]>

* Return only monitors for /monitors/_search. (opensearch-project#162)

* Return only monitors for /monitors/_search.

* Added missing imports

* Added additional check to the unit test

* Resolve default for ActionExecutionPolicy at runtime (opensearch-project#165)

Signed-off-by: Mohammad Qureshi <[email protected]>

Co-authored-by: AWSHurneyt <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Co-authored-by: Ashish Agrawal <[email protected]>
Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]>
Co-authored-by: Mohammad Qureshi <[email protected]>
Co-authored-by: Rishabh Maurya <[email protected]>
Co-authored-by: Sriram <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants