Percolate Query implementation in doc-level alerting #399

sbcd90 · 2022-04-13T00:14:34Z

percolate query implementation in doc-level alerting

Issue #, if available:
Percolate Query implementation in Document Level Alerting

Description of changes:
This PR provides Percolate Query based implementation in Document Level Alerting. It implements the following design with its limitations.

CheckList:
[X] Commits are signed per the DCO using --signoff
[X] Integration tests work for doc-level alerting
[ ] Performance test results - pending
[ ] Additional tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-04-13T01:48:21Z

Codecov Report

❗ No coverage uploaded for pull request base (doc-level-2.0@5f4d9fd). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             doc-level-2.0     #399   +/-   ##
================================================
  Coverage                 ?   78.68%           
  Complexity               ?      285           
================================================
  Files                    ?      198           
  Lines                    ?     8612           
  Branches                 ?     1148           
================================================
  Hits                     ?     6776           
  Misses                   ?     1248           
  Partials                 ?      588

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f4d9fd...4e9a856. Read the comment docs.

getsaurabh02

Thanks for the changes. I have added few comments/clarification.
I am yet to look into the Transport Action part and will review it later today.

getsaurabh02 · 2022-04-14T17:00:28Z

alerting/build.gradle

@@ -71,7 +72,7 @@ dependencies {
 }

 javadoc.enabled = false // turn off javadoc as it barfs on Kotlin code
-licenseHeaders.enabled = true
+licenseHeaders.enabled = false


Shouldn't we add license headers in the new files being added?

added license to all files missing license.

getsaurabh02 · 2022-04-14T17:02:11Z

alerting/src/main/java/org/opensearch/percolator/PercolateQueryBuilderExt.java

+import static org.opensearch.common.xcontent.ConstructingObjectParser.optionalConstructorArg;
+import static org.opensearch.search.SearchService.ALLOW_EXPENSIVE_QUERIES;
+
+public class PercolateQueryBuilderExt extends AbstractQueryBuilder<PercolateQueryBuilderExt> {


These Ext classes seems to be the Stop Gap solutions unless we have figured out the right way to extend and use the Percolate client capabilities from the alerting plugin.
Can we add a documentation here and ToDo for cleanup?
Lets also add an issue in core repo to address the issue discovered and tag it here.

added comments to the java classes. will create an issue in core.

getsaurabh02 · 2022-04-14T17:04:44Z

alerting/src/main/java/org/opensearch/percolator/PercolatorPluginExt.java

+import static java.util.Collections.singletonList;
+import static java.util.Collections.singletonMap;
+
+public class PercolatorPluginExt extends Plugin implements MapperPlugin, SearchPlugin, ExtensiblePlugin {


Were we able to create the full distribution and verify the same in single and multinode setup with no issues?

yes. the integration test build passes for single node & multi node with 3 nodes.

getsaurabh02 · 2022-04-14T17:08:12Z

alerting/src/main/java/org/opensearch/percolator/PercolatorFieldMapperExt.java

+
+import static org.opensearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder;
+
+public class PercolatorFieldMapperExt extends ParametrizedFieldMapper {


Will a change in the PercolatorFieldMapper in core repo create the type cast exception issue with objects if this class is not kept up to date? This could be a huge maintenance overhead if not addressed shortly.
Also wondering if we also evaluated the object deserialization and serialization route for improved handling of such issues?

not PercolatorFieldMapper. But if api signatures from other classes in percolator-client changes, it will need change in the copied classes too.
object deserialization and serialization route needs to handled in core. spi approach to make percolate plugin extensible could also be an idea that could be pursued.

getsaurabh02 · 2022-04-14T17:36:01Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt

        val docsToQueries = mutableMapOf<String, MutableList<String>>()
        val docExecutionContext = DocumentExecutionContext(queries, lastRunContext, updatedLastRunContext)
        val idQueryMap = mutableMapOf<String, DocLevelQuery>()
-        queries.forEach { query ->
-            val matchingDocIds = runForEachQuery(monitorCtx, docExecutionContext, query, index)


Lets cleanup runForEachQuery impl.

removed it.

getsaurabh02 · 2022-04-14T17:42:57Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt

+                logger.info("Failed to run for shard $shard. Error: ${e.message}")
+                logger.debug("Failed to run for shard $shard", e)


not sure with the intent of double logging here? To get the exception trace?

removed it.

getsaurabh02 · 2022-04-14T17:44:06Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt

    private fun getAllDocIds(hits: SearchHits): List<String> {
        return hits.map { hit -> hit.id }
    }


replaced by getAllDocs now?

removed it.

getsaurabh02 · 2022-04-14T17:46:02Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt

+    ): SearchHits {
+        val percolateQueryBuilder = PercolateQueryBuilderExt("query", docs, XContentType.JSON)
+
+        val searchRequest = SearchRequest(ScheduledJob.DOC_LEVEL_QUERIES_INDEX)


Do we need a safe check if the query index is already created/exists? It could be in Runner itself. Just ensuring the exception is floated up the Runner with the right messaging

removed the safe check.

What happens if DOC_LEVEL_QUERIES_INDEX is not created or is deleted for some reason? I dont think we have solved this use-case yet. Not a blocker, but something for a quick follow up.

Yes. the current logic does not take care of high availability. This can be improved in future.
Currently, if index doesnt exist, there will be no exceptions. only no search results will be returned.

getsaurabh02 · 2022-04-14T17:51:06Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt

@@ -410,7 +491,47 @@ object DocumentReturningMonitorRunner : MonitorRunner {
        return response.hits
    }

+    private fun getMatchedQueries(


Is this not to get Matching Documents instead?

this method matches incoming docs with queries. named another method to getMatchingDocs

getsaurabh02 · 2022-04-14T17:57:19Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt

+        searchRequest.source(searchSourceBuilder)
+
+        return if (monitorCtx.clusterService!!.state().routingTable.hasIndex(ScheduledJob.DOC_LEVEL_QUERIES_INDEX)) {
+            val response: SearchResponse = monitorCtx.client!!.execute(SearchAction.INSTANCE, searchRequest).actionGet()


Critical - This will end up running percolate for all the incoming documents against all the queries without the filtering the monitor type against the query index. Should we have the monitor.id field added upfront?

passing monitor.id as a filter also so that it filters by monitor id & then matches only those queries. https://github.com/opensearch-project/alerting/blob/6328f06451ced797f66d481f47a9fd757944f116/alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt#L464-#L477

getsaurabh02 · 2022-04-15T02:50:57Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentReturningMonitorRunner.kt

+    ): SearchHits {
+        val percolateQueryBuilder = PercolateQueryBuilderExt("query", docs, XContentType.JSON)
+
+        val searchRequest = SearchRequest(ScheduledJob.DOC_LEVEL_QUERIES_INDEX)


What happens if DOC_LEVEL_QUERIES_INDEX is not created or is deleted for some reason? I dont think we have solved this use-case yet. Not a blocker, but something for a quick follow up.

getsaurabh02 · 2022-04-15T03:00:18Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportIndexMonitorAction.kt

@@ -409,6 +424,83 @@ class TransportIndexMonitorAction @Inject constructor(
            )
        }

+        @Suppress("UNCHECKED_CAST")
+        private fun indexDocLevelMonitorQueries(monitor: Monitor, monitorId: String, refreshPolicy: RefreshPolicy) {


This seems to be code duplication from the monitor execution action for mapping updates. I would rather define this as a util method in one of doc level monitor related class and use at both the places if required.

moved the duplicate code here.https://github.com/opensearch-project/alerting/blob/79a368caa73b64ab08b0cd14c4da0324ded8c1db/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt#L57-#L114

this cleaned up transport logic.https://github.com/opensearch-project/alerting/blob/79a368caa73b64ab08b0cd14c4da0324ded8c1db/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportIndexMonitorAction.kt#L425-#L474

getsaurabh02 · 2022-04-15T03:02:01Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt

@@ -74,6 +91,65 @@ class TransportExecuteMonitorAction @Inject constructor(
                    }
                }
            }
+            val indexDocLevelMonitorQueries = fun(monitor: Monitor, monitorId: String) {


Do we really need to update the mapping each time during the monitor run execution? Shouldn't the Index Monitor path take care of any update in the mapping?

there are 2 types of execute action. one which takes already created monitor_id as input. the other one directly takes entire monitor payload as input.
update mapping is needed in 2nd case.
https://github.com/opensearch-project/alerting/blob/79a368caa73b64ab08b0cd14c4da0324ded8c1db/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt#L89-#L180

lezzago · 2022-04-18T15:24:29Z

core/src/main/resources/mappings/doc-level-queries.json

@@ -0,0 +1,13 @@
+{
+  "_meta": {
+    "schema_version": 5


change this to 1

Signed-off-by: Subhobrata Dey <[email protected]>

* Rebase to push doc level changes on latest main changes (#391) * Document level alerting dev (#272) Signed-off-by: Ashish Agrawal <[email protected]> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <[email protected]> * add Update Monitor function Signed-off-by: Ashish Agrawal <[email protected]> * fix integ test Signed-off-by: Ashish Agrawal <[email protected]> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <[email protected]> * POC for doc-level-alerting (#277) Signed-off-by: Sriram <[email protected]> * Add connection to triggers for doc level alerting (#316) Signed-off-by: Ashish Agrawal <[email protected]> * CRUD APIs integration Tests and validation"conflict resolved" (#362) Signed-off-by: charliezhangaws <[email protected]> * Segregate monitor runner logic for separation of concerns (#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <[email protected]> * Add action and alert flow and findings schema and additional fixes (#381) Signed-off-by: Ashish Agrawal <[email protected]> * Finding Search API (#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <[email protected]> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (#405) Signed-off-by: Saurabh Singh <[email protected]> * percolate query implementation in doc-level alerting (#399) Signed-off-by: Subhobrata Dey <[email protected]> * Finding Index rollover (#408) * Finding Index rollover Signed-off-by: jiahe zhang <[email protected]> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: jiahe zhang <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]>

* Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <[email protected]> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <[email protected]> * add Update Monitor function Signed-off-by: Ashish Agrawal <[email protected]> * fix integ test Signed-off-by: Ashish Agrawal <[email protected]> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <[email protected]> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <[email protected]> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <[email protected]> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <[email protected]> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <[email protected]> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <[email protected]> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <[email protected]> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <[email protected]> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <[email protected]> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <[email protected]> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: jiahe zhang <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]>

* fix security test workflow (#407) Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Integrate Document Level Alerting changes (#410) * Rebase to push doc level changes on latest main changes (#391) * Document level alerting dev (#272) Signed-off-by: Ashish Agrawal <[email protected]> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <[email protected]> * add Update Monitor function Signed-off-by: Ashish Agrawal <[email protected]> * fix integ test Signed-off-by: Ashish Agrawal <[email protected]> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <[email protected]> * POC for doc-level-alerting (#277) Signed-off-by: Sriram <[email protected]> * Add connection to triggers for doc level alerting (#316) Signed-off-by: Ashish Agrawal <[email protected]> * CRUD APIs integration Tests and validation"conflict resolved" (#362) Signed-off-by: charliezhangaws <[email protected]> * Segregate monitor runner logic for separation of concerns (#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <[email protected]> * Add action and alert flow and findings schema and additional fixes (#381) Signed-off-by: Ashish Agrawal <[email protected]> * Finding Search API (#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <[email protected]> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (#405) Signed-off-by: Saurabh Singh <[email protected]> * percolate query implementation in doc-level alerting (#399) Signed-off-by: Subhobrata Dey <[email protected]> * Finding Index rollover (#408) * Finding Index rollover Signed-off-by: jiahe zhang <[email protected]> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: jiahe zhang <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Remove write Destination APIs (#412) * Remove write Destination API REST handlers Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination transport actions Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination action, request and response classes Signed-off-by: Mohammad Qureshi <[email protected]> * Create Alerting config index if it doesn't exist before legacy Destination indexing Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination related security tests Signed-off-by: Mohammad Qureshi <[email protected]> * Remove unused access roles and imports Signed-off-by: Mohammad Qureshi <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Deprecate the Master nomenclature in 2.0 (#415) Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Remove Alerting's notification subproject (#413) * Remove notification subproject Signed-off-by: Mohammad Qureshi <[email protected]> * Remove publishing to maven in build script Signed-off-by: Mohammad Qureshi <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Skipping destination migration if alerting index is not initialized (#417) Signed-off-by: Ravi [email protected] Signed-off-by: AWSHurneyt <[email protected]> * Alias support for Document Level Monitors (#416) * Implemented support for defining doc level monitors using aliases. Signed-off-by: AWSHurneyt <[email protected]> * Fix integ tests and cleaup alias logic Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Fixed a flaky test condition. (#375) Signed-off-by: AWSHurneyt <[email protected]> * Replace checked-in ZIP for bwc tests with a dynamic dependency (#411) * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <[email protected]> * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Update integTest gradle scripts to run via remote cluster independently (#418) Signed-off-by: AWSHurneyt <[email protected]> * Removed tests that were duplicated while resolving merge conflicts. Signed-off-by: AWSHurneyt <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Co-authored-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Mohammad Qureshi <[email protected]> Co-authored-by: Ravi <[email protected]>

* fix security test workflow (opensearch-project#407) Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Integrate Document Level Alerting changes (opensearch-project#410) * Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <[email protected]> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <[email protected]> * add Update Monitor function Signed-off-by: Ashish Agrawal <[email protected]> * fix integ test Signed-off-by: Ashish Agrawal <[email protected]> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <[email protected]> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <[email protected]> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <[email protected]> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <[email protected]> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <[email protected]> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <[email protected]> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <[email protected]> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <[email protected]> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <[email protected]> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <[email protected]> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: jiahe zhang <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Remove write Destination APIs (opensearch-project#412) * Remove write Destination API REST handlers Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination transport actions Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination action, request and response classes Signed-off-by: Mohammad Qureshi <[email protected]> * Create Alerting config index if it doesn't exist before legacy Destination indexing Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination related security tests Signed-off-by: Mohammad Qureshi <[email protected]> * Remove unused access roles and imports Signed-off-by: Mohammad Qureshi <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Deprecate the Master nomenclature in 2.0 (opensearch-project#415) Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Remove Alerting's notification subproject (opensearch-project#413) * Remove notification subproject Signed-off-by: Mohammad Qureshi <[email protected]> * Remove publishing to maven in build script Signed-off-by: Mohammad Qureshi <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Skipping destination migration if alerting index is not initialized (opensearch-project#417) Signed-off-by: Ravi [email protected] Signed-off-by: AWSHurneyt <[email protected]> * Alias support for Document Level Monitors (opensearch-project#416) * Implemented support for defining doc level monitors using aliases. Signed-off-by: AWSHurneyt <[email protected]> * Fix integ tests and cleaup alias logic Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Fixed a flaky test condition. (opensearch-project#375) Signed-off-by: AWSHurneyt <[email protected]> * Replace checked-in ZIP for bwc tests with a dynamic dependency (opensearch-project#411) * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <[email protected]> * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Update integTest gradle scripts to run via remote cluster independently (opensearch-project#418) Signed-off-by: AWSHurneyt <[email protected]> * Removed tests that were duplicated while resolving merge conflicts. Signed-off-by: AWSHurneyt <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Co-authored-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Mohammad Qureshi <[email protected]> Co-authored-by: Ravi <[email protected]> Signed-off-by: Angie Zhang <[email protected]>

* Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <[email protected]> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <[email protected]> * add Update Monitor function Signed-off-by: Ashish Agrawal <[email protected]> * fix integ test Signed-off-by: Ashish Agrawal <[email protected]> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <[email protected]> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <[email protected]> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <[email protected]> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <[email protected]> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <[email protected]> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <[email protected]> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <[email protected]> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <[email protected]> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <[email protected]> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <[email protected]> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: jiahe zhang <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Signed-off-by: Angie Zhang <[email protected]>

* fix security test workflow (opensearch-project#407) Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Integrate Document Level Alerting changes (opensearch-project#410) * Rebase to push doc level changes on latest main changes (opensearch-project#391) * Document level alerting dev (opensearch-project#272) Signed-off-by: Ashish Agrawal <[email protected]> * Add last run context to Monitor data model Signed-off-by: Ashish Agrawal <[email protected]> * add Update Monitor function Signed-off-by: Ashish Agrawal <[email protected]> * fix integ test Signed-off-by: Ashish Agrawal <[email protected]> * Implemented draft of Finding data model, a new Input type, and some basic unit tests. (opensearch-project#260) * Implemented draft of Finding data model, and some basic unit tests for it. Signed-off-by: AWSHurneyt <[email protected]> * POC for doc-level-alerting (opensearch-project#277) Signed-off-by: Sriram <[email protected]> * Add connection to triggers for doc level alerting (opensearch-project#316) Signed-off-by: Ashish Agrawal <[email protected]> * CRUD APIs integration Tests and validation"conflict resolved" (opensearch-project#362) Signed-off-by: charliezhangaws <[email protected]> * Segregate monitor runner logic for separation of concerns (opensearch-project#363) * Refactor monitor runner logic for separation of concerns and better testability. Signed-off-by: Saurabh Singh <[email protected]> * Add action and alert flow and findings schema and additional fixes (opensearch-project#381) Signed-off-by: Ashish Agrawal <[email protected]> * Finding Search API (opensearch-project#385) * Findings search API based on Annie's work Signed-off-by: Annie Lee <[email protected]> * Fix Search API and add IT tests Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Fix integ tests and minor issues from doc level changes Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> * Add Trigger condition resolver which parses and evaluates the Trigger expression. (opensearch-project#405) Signed-off-by: Saurabh Singh <[email protected]> * percolate query implementation in doc-level alerting (opensearch-project#399) Signed-off-by: Subhobrata Dey <[email protected]> * Finding Index rollover (opensearch-project#408) * Finding Index rollover Signed-off-by: jiahe zhang <[email protected]> * Apply fixes to make rollover work Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: jiahe zhang <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Remove write Destination APIs (opensearch-project#412) * Remove write Destination API REST handlers Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination transport actions Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination action, request and response classes Signed-off-by: Mohammad Qureshi <[email protected]> * Create Alerting config index if it doesn't exist before legacy Destination indexing Signed-off-by: Mohammad Qureshi <[email protected]> * Remove write Destination related security tests Signed-off-by: Mohammad Qureshi <[email protected]> * Remove unused access roles and imports Signed-off-by: Mohammad Qureshi <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Deprecate the Master nomenclature in 2.0 (opensearch-project#415) Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Remove Alerting's notification subproject (opensearch-project#413) * Remove notification subproject Signed-off-by: Mohammad Qureshi <[email protected]> * Remove publishing to maven in build script Signed-off-by: Mohammad Qureshi <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Skipping destination migration if alerting index is not initialized (opensearch-project#417) Signed-off-by: Ravi [email protected] Signed-off-by: AWSHurneyt <[email protected]> * Alias support for Document Level Monitors (opensearch-project#416) * Implemented support for defining doc level monitors using aliases. Signed-off-by: AWSHurneyt <[email protected]> * Fix integ tests and cleaup alias logic Signed-off-by: Ashish Agrawal <[email protected]> Co-authored-by: AWSHurneyt <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Fixed a flaky test condition. (opensearch-project#375) Signed-off-by: AWSHurneyt <[email protected]> * Replace checked-in ZIP for bwc tests with a dynamic dependency (opensearch-project#411) * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <[email protected]> * Replace checked-in ZIP with a dynamic dependency Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: AWSHurneyt <[email protected]> * Update integTest gradle scripts to run via remote cluster independently (opensearch-project#418) Signed-off-by: AWSHurneyt <[email protected]> * Removed tests that were duplicated while resolving merge conflicts. Signed-off-by: AWSHurneyt <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Co-authored-by: Ashish Agrawal <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]> Co-authored-by: Sriram <[email protected]> Co-authored-by: charliezhangaws <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Mohammad Qureshi <[email protected]> Co-authored-by: Ravi <[email protected]> Signed-off-by: Angie Zhang <[email protected]>

sbcd90 requested a review from a team April 13, 2022 00:14

sbcd90 force-pushed the perc_query_proposal branch 2 times, most recently from bcc41a4 to 4e9a856 Compare April 13, 2022 01:38

sbcd90 force-pushed the perc_query_proposal branch 10 times, most recently from fb8dee8 to ba89f5e Compare April 13, 2022 22:23

sbcd90 requested review from getsaurabh02 and lezzago April 13, 2022 23:09

sbcd90 force-pushed the perc_query_proposal branch 3 times, most recently from 9a3e538 to 826e827 Compare April 14, 2022 17:08

getsaurabh02 reviewed Apr 14, 2022

View reviewed changes

sbcd90 force-pushed the perc_query_proposal branch from 826e827 to 6328f06 Compare April 14, 2022 20:34

getsaurabh02 reviewed Apr 15, 2022

View reviewed changes

sbcd90 force-pushed the perc_query_proposal branch 2 times, most recently from 79a368c to a55fe04 Compare April 15, 2022 21:31

getsaurabh02 approved these changes Apr 16, 2022

View reviewed changes

lezzago reviewed Apr 18, 2022

View reviewed changes

sbcd90 force-pushed the perc_query_proposal branch from a55fe04 to 41d962b Compare April 18, 2022 15:31

percolate query implementation in doc-level alerting

0bab665

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 force-pushed the perc_query_proposal branch from 41d962b to 0bab665 Compare April 18, 2022 15:53

lezzago approved these changes Apr 18, 2022

View reviewed changes

sbcd90 merged commit 8e991f6 into opensearch-project:doc-level-2.0 Apr 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Percolate Query implementation in doc-level alerting #399

Percolate Query implementation in doc-level alerting #399

sbcd90 commented Apr 13, 2022

codecov-commenter commented Apr 13, 2022 •

edited

Loading

getsaurabh02 left a comment

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 15, 2022 •

edited

Loading

sbcd90 Apr 15, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 14, 2022

sbcd90 Apr 14, 2022

getsaurabh02 Apr 15, 2022 •

edited

Loading

getsaurabh02 Apr 15, 2022

sbcd90 Apr 15, 2022

getsaurabh02 Apr 15, 2022

sbcd90 Apr 15, 2022

lezzago Apr 18, 2022


		import static org.opensearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder;

		public class PercolatorFieldMapperExt extends ParametrizedFieldMapper {

		logger.info("Failed to run for shard $shard. Error: ${e.message}")
		logger.debug("Failed to run for shard $shard", e)

Percolate Query implementation in doc-level alerting #399

Percolate Query implementation in doc-level alerting #399

Conversation

sbcd90 commented Apr 13, 2022

codecov-commenter commented Apr 13, 2022 • edited Loading

Codecov Report

getsaurabh02 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

getsaurabh02 Apr 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

getsaurabh02 Apr 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Apr 13, 2022 •

edited

Loading

getsaurabh02 Apr 15, 2022 •

edited

Loading

getsaurabh02 Apr 15, 2022 •

edited

Loading