Categorize search queries by type and log query shape #10502

deshsidd · 2023-10-09T08:56:27Z

Description

Changes to categorize search queries based on the type of the query (eg: bool, match, multi-match, etc).
Changes also log the query shape of a query (queries can have a nested structure with a hierarchy of subqueries).

Use QueryShapeVisitor to debug log the shape of the queries. Example
Use QueryBuilderVisitor to visit all the nodes of the querybuilder tree for incoming search queries and increment counters to capture query type information
Integrate with Metric Framework for the counters

Related Issues

Resolves #10250

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-10-09T09:01:26Z

Compatibility status:

Checks if related components are compatible with change b2d7e67

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-10-09T09:06:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27427/
CommitID: 4e79d74
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-09T16:50:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27446/
CommitID: 3a46da1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-09T17:04:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27447/
CommitID: 2634bfe
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-09T17:35:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27450/
CommitID: f8631f3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…h-project#9255) * Fix dissect ingest processor parsing empty brackets failed Signed-off-by: Gao Binlong <[email protected]> * Modify change log Signed-off-by: Gao Binlong <[email protected]> * Modify change log Signed-off-by: Gao Binlong <[email protected]> * Add assertion Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

…oject#10395) * Add telemetry tracer/metric enable flag and integ test Signed-off-by: Gagan Juneja <[email protected]> * Add Changelog Signed-off-by: Gagan Juneja <[email protected]> * Fix compilation issue Signed-off-by: Gagan Juneja <[email protected]> * Empty-Commit Signed-off-by: Gagan Juneja <[email protected]> * Add component flag to traceable wrappers Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> * Address review comment Signed-off-by: Gagan Juneja <[email protected]> --------- Signed-off-by: Gagan Juneja <[email protected]> Signed-off-by: Gagan Juneja <[email protected]> Co-authored-by: Gagan Juneja <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

…search-project#10671) This test is flaky in two ways, one it may hit a case where a merge occurs and the hardcoded segment does not exist to delete. Two with certain mock filesystems a corruption exception won't be thrown here, only a NoSuchFile - This doesn't change our handling in the engine, but the test should conditionally handle if the store is marked corrupted on close. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

…0557) * Change filenames for IndexMetadata and Manifest Signed-off-by: Dhwanil Patel <[email protected]>

…ect#10503) * revive remote cluster state auto restore integ tests Signed-off-by: bansvaru <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

* Upload global cluster state to remote store Signed-off-by: Dhwanil Patel <[email protected]>

…s.testShardAlreadyReplicating (opensearch-project#10660) * Remove Awaits fix. Signed-off-by: Rishikesh1159 <[email protected]> * apply spotlesscheck. Signed-off-by: Rishikesh1159 <[email protected]> * add test logging. Signed-off-by: Rishikesh1159 <[email protected]> --------- Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

opensearch-project#10690) Signed-off-by: vikasvb90 <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Signed-off-by: Bukhtawar Khan <[email protected]>

OnBehalfOf claims take second duration Signed-off-by: Stephen Crawford <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

…ch-project#10535) * Download funcationality of global metadata from remote store Signed-off-by: Dhwanil Patel <[email protected]>

Signed-off-by: Siddhant Deshmukh <[email protected]>

… loss recovery (opensearch-project#10447) * Remote Restore Cluster Metadata if local disk state lost after quorum loss Signed-off-by: bansvaru <[email protected]>

…ct#10712) Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Signed-off-by: David Zane <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Signed-off-by: Siddhant Deshmukh <[email protected]>

github-actions · 2023-10-19T00:05:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28345/
CommitID: 3ab78de
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-19T00:10:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28346/
CommitID: cb4a6bf
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-19T00:16:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28347/
CommitID: b2d7e67
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-19T00:19:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28350/
CommitID: 8cfe2bf
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions bot added the enhancement Enhancement or improvement to existing feature or request label Oct 9, 2023

gaobinlong and others added 23 commits October 18, 2023 16:57

Change filenames for IndexMetadata and Manifest (opensearch-project#1…

7cb87c0

…0557) * Change filenames for IndexMetadata and Manifest Signed-off-by: Dhwanil Patel <[email protected]>

revive remote cluster state auto restore integ tests (opensearch-proj…

b921942

…ect#10503) * revive remote cluster state auto restore integ tests Signed-off-by: bansvaru <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

:pload global cluster state to remote store (opensearch-project#10404)

16e75f3

* Upload global cluster state to remote store Signed-off-by: Dhwanil Patel <[email protected]>

Added BufferedInputStream to allow mark and reset ops during IO errors (

d79a6cc

opensearch-project#10690) Signed-off-by: vikasvb90 <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Repository stats for remote store (opensearch-project#10567)

ffe24c3

Signed-off-by: Bukhtawar Khan <[email protected]>

OnBehalfOf claims take second duration (opensearch-project#10664)

4b33c59

OnBehalfOf claims take second duration Signed-off-by: Stephen Crawford <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Fix download reporting for segment replication (opensearch-project#10644

f3c6f8b

) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Download functionality of global metadata from remote store (opensear…

611983a

…ch-project#10535) * Download funcationality of global metadata from remote store Signed-off-by: Dhwanil Patel <[email protected]>

Build fix

d93f06b

Signed-off-by: Siddhant Deshmukh <[email protected]>

Fix build tests failure

f458ef2

Signed-off-by: Siddhant Deshmukh <[email protected]>

Empty commit

411164d

Signed-off-by: Siddhant Deshmukh <[email protected]>

Remote Restore Cluster Metadata if local disk state lost after quorum…

c56a532

… loss recovery (opensearch-project#10447) * Remote Restore Cluster Metadata if local disk state lost after quorum loss Signed-off-by: bansvaru <[email protected]>

Version fixes for Resource Usage Stats and FS Stats (opensearch-proje…

9ee6697

…ct#10712) Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Add missing entry for 2.11 release notes (opensearch-project#10679)

e0a5244

Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Per request phase latency (opensearch-project#10351)

f8f35ca

Signed-off-by: David Zane <[email protected]> Signed-off-by: Siddhant Deshmukh <[email protected]>

Empty commit

6b8e907

Signed-off-by: Siddhant Deshmukh <[email protected]>

Empty commit

35ae1d0

Signed-off-by: Siddhant Deshmukh <[email protected]>

Empty commit

5fcde3e

Signed-off-by: Siddhant Deshmukh <[email protected]>

Empty commit

8cfe2bf

Signed-off-by: Siddhant Deshmukh <[email protected]>

deshsidd force-pushed the sid/search-query-category branch from b2d7e67 to 8cfe2bf Compare October 19, 2023 00:01

deshsidd mentioned this pull request Oct 19, 2023

Categorize search queries by type and log query shape #10724

Merged

deshsidd closed this Oct 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Categorize search queries by type and log query shape #10502

Categorize search queries by type and log query shape #10502

deshsidd commented Oct 9, 2023 •

edited

Loading

github-actions bot commented Oct 9, 2023 •

edited

Loading

github-actions bot commented Oct 9, 2023

github-actions bot commented Oct 9, 2023

github-actions bot commented Oct 9, 2023

github-actions bot commented Oct 9, 2023

github-actions bot commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

Categorize search queries by type and log query shape #10502

Categorize search queries by type and log query shape #10502

Conversation

deshsidd commented Oct 9, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Oct 9, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Oct 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

deshsidd commented Oct 9, 2023 •

edited

Loading

github-actions bot commented Oct 9, 2023 •

edited

Loading