Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support common function in SQL/PPL query engine #190

Closed
6 tasks
chloe-zh opened this issue Aug 21, 2021 · 2 comments
Closed
6 tasks

Support common function in SQL/PPL query engine #190

chloe-zh opened this issue Aug 21, 2021 · 2 comments

Comments

@chloe-zh
Copy link
Contributor

Related design is presented in issue #182

Todo list

  • Support the common functionality by pushing it down to the search engine
  • Enable common function in SQL and PPL syntax and parser, including all the available parameters
  • Make sure the function in new engine is compatible with the old engine support
  • Add unit tests. (Unit tests are mandatory for all code changes.)
  • Add integration test cases for common
  • Update user manual

Function details

The common function maps to the common terms query used in the search engine. This query is a modern alternative to stopwords which improves the precision and recall of search results (by taking stopwords into account), without sacrificing performance.

Syntax:
common(field_expression, query_expression[, option=<option_value>]*)

Available options:

  • analyzer
  • cutoff_frequency
  • high_frequency_operator
  • low_frequency_operator
  • minimum_should_match
  • boost

Sample query:

# Search query 1
GET my_index/_search
{
  "query": {
    "common": {
      "message": {
        "query": "this is bonsai cool",
        "cutoff_frequency": 0.001
      }
    }
  }
}

# SQL
SELECT * FROM my_index WHERE common(message, "this is bonsai cool", cutoff_frequency=0.001)

# PPL
source=my_index | where common(message, "this is bonsai cool", cutoff_frequency=0.001)
@acarbonetto
Copy link
Collaborator

OpenSearch doesn't support common, the function is obsolete. Can we close this ticket?

@acarbonetto
Copy link
Collaborator

@anirudha common terms is marked as deprecated, see explanation here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/query/CommonTermsQueryBuilder.java#L64

TL;DR: MatchQuery does the same thing.

@dai-chen dai-chen closed this as completed Aug 9, 2022
penghuo pushed a commit that referenced this issue Jan 6, 2023
Added Implementation And Testing For Day_Of_Week Function

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
opensearch-trigger-bot bot pushed a commit that referenced this issue Jan 6, 2023
Added Implementation And Testing For Day_Of_Week Function

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
(cherry picked from commit bac9c37)
penghuo pushed a commit that referenced this issue Jan 9, 2023
Added Implementation And Testing For Day_Of_Week Function

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
(cherry picked from commit bac9c37)

Co-authored-by: GabeFernandez310 <[email protected]>
penghuo added a commit that referenced this issue Jan 10, 2023
* Add BWC tests for running against distribution bundle.  (#1209)

Signed-off-by: Zelin Hao <[email protected]>

* Add Alternate Syntax For Match_Query And Other Functions (#1166)

Added Tests And Implementation For Match_Query, Match_Phrase, and Multi_Match Functions

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
Signed-off-by: GabeFernandez310 <[email protected]>

* Merge pull request #1241 from Bit-Quill/Failing-CI-Hot-Fix

Hot Fix For CI Build

(cherry picked from commit aae57a0)
Signed-off-by: GabeFernandez310 <[email protected]>

* Fixed error with single timestamp query (#1244) (#1246)

Signed-off-by: vamsi-amazon <[email protected]>

Signed-off-by: vamsi-amazon <[email protected]>
(cherry picked from commit ee949cc)

Co-authored-by: vamsi-amazon <[email protected]>

* Add Second_Of_Minute Function As An Alias Of The Second Function (#1231) (#1237)

Added Testing And Implementation For Second_Of_Minute Function

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
(cherry picked from commit dce7d0e)

Co-authored-by: GabeFernandez310 <[email protected]>

* Add functions `ADDTIME` and `SUBTIME`. (#132) (#1194) (#1252)

* Add functions `ADDTIME` and `SUBTIME`. (#132)

Signed-off-by: Yury-Fridlyand <[email protected]>
(cherry picked from commit 7630f87)

Co-authored-by: Yury-Fridlyand <[email protected]>

* Add Day_Of_Week Function As An Alias Of DayOfWeek (#190) (#1228) (#1239)

Added Implementation And Testing For Day_Of_Week Function

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
(cherry picked from commit bac9c37)

Co-authored-by: GabeFernandez310 <[email protected]>

* [Backport 2.x] Add Minute_Of_Hour Function As An Alias Of Minute Function (#1253)

* Add Minute_Of_Hour Function As An Alias Of Minute Function (#196) (#1230)

Added Testing And Implementation For Minute_Of_Hour Function

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>
(cherry picked from commit 61e2374)

* Added Missing Imports

Signed-off-by: GabeFernandez310 <[email protected]>

Signed-off-by: GabeFernandez310 <[email protected]>

* Add support for long value return for CEIL, CEILING and FLOOR math functions (#1205) (#1255)

* Added long fix for CEIL, CEILING and FLOOR functions using LONG instead of INT for RETURN.

Signed-off-by: MitchellGale-BitQuill <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>

* Support JOIN query on object field with unexpanded name (#1229) (#1250)

* Resolve sub object field in search hit source

Signed-off-by: Chen Dai <[email protected]>

* Rename to unexpanded object

Signed-off-by: Chen Dai <[email protected]>

* Update IT with where condition

Signed-off-by: Chen Dai <[email protected]>

* Fix test index mapping

Signed-off-by: Chen Dai <[email protected]>

Signed-off-by: Chen Dai <[email protected]>
(cherry picked from commit 151f4cc)

Co-authored-by: Chen Dai <[email protected]>

* Remove unnecessary scripts after repo split (#1256)

Signed-off-by: Joshua Li <[email protected]>

* Add Support For `TIME` Type in "*_OF_YEAR" Functions (#199) (#1223) (#1258)

Added Support And Tests For Time Type in day_of_year, week_of_year, month_of_year Functions
Signed-off-by: GabeFernandez310 <[email protected]>
(cherry picked from commit 6e72f18)

Co-authored-by: GabeFernandez310 <[email protected]>

Signed-off-by: Zelin Hao <[email protected]>
Signed-off-by: GabeFernandez310 <[email protected]>
Signed-off-by: GabeFernandez310 <[email protected]>
Signed-off-by: MitchellGale-BitQuill <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Signed-off-by: Joshua Li <[email protected]>
Co-authored-by: Zelin Hao <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>
Co-authored-by: YANGDB <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: vamsi-amazon <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>
Co-authored-by: Chen Dai <[email protected]>
Co-authored-by: Joshua Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants