Backport/backport 1645 to 2.x sig test #284
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add table write operator and builder Signed-off-by: Chen Dai <[email protected]> * Add UT for all new classes Signed-off-by: Chen Dai <[email protected]> * Rename child field Signed-off-by: Chen Dai <[email protected]> * Add columns field Signed-off-by: Chen Dai <[email protected]> * Update javadoc to prepare PR Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>
…nsearch-project#1169) (opensearch-project#1173) Signed-off-by: Peng Huo <[email protected]> (cherry picked from commit 5073215) Co-authored-by: Peng Huo <[email protected]>
* Add between grammar and in-memory impl Signed-off-by: Chen Dai <[email protected]> * Add comparison test for between Signed-off-by: Chen Dai <[email protected]> * Add doctest for between Signed-off-by: Chen Dai <[email protected]> * Add not between support Signed-off-by: Chen Dai <[email protected]> * Fix doctest failure Signed-off-by: Chen Dai <[email protected]> * Refactor to rewrite to basic comparison expression Signed-off-by: Chen Dai <[email protected]> * Clean up unused code Signed-off-by: Chen Dai <[email protected]> * Prepare to publish PR Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>
…rch-project#1188) * Fix precedence by reordering grammar rule Signed-off-by: Chen Dai <[email protected]> * Fix precedence in PPL Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]> (cherry picked from commit ab46fdd) Co-authored-by: Chen Dai <[email protected]>
…pensearch-project#1198) * Updated UTC TIME Functions for like now FunctionProperties. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Finished updates to tests. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed unused import. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Addressed PR comments. Signed-off-by: MitchellGale-BitQuill <[email protected]> * Removed extra import. Signed-off-by: MitchellGale-BitQuill <[email protected]> * removed `DateTimeFunction` used within class. Signed-off-by: MitchellGale-BitQuill <[email protected]> Signed-off-by: MitchellGale-BitQuill <[email protected]> (cherry picked from commit 94b6bec) Co-authored-by: MitchellGale-BitQuill <[email protected]>
…nsearch-project#1208) * Unquote from subquery alias Signed-off-by: Chen Dai <[email protected]> * Add comparison test case Signed-off-by: Chen Dai <[email protected]> * Add more comparison test case Signed-off-by: Chen Dai <[email protected]> * Update doc to remove limitations Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]> (cherry picked from commit 91ef34d) Co-authored-by: Chen Dai <[email protected]>
…pensearch-project#1067) (opensearch-project#1199) Change relevance functions that query fields to throw a SemanticCheckException when a field is queried that does not exist. Signed-off-by: forestmvey <[email protected]> (cherry picked from commit d03c176) Co-authored-by: Forest Vey <[email protected]>
…#1213) * Fix truncate() function (#188) Signed-off-by: Margarit Hakobyan <[email protected]> Signed-off-by: Margarit Hakobyan <[email protected]> (cherry picked from commit 7714819) Co-authored-by: Margarit Hakobyan <[email protected]>
…pensearch-project#1214) Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 2c46797) Co-authored-by: GabeFernandez310 <[email protected]>
…ensearch-project#1191) (opensearch-project#1212) * Allow score, type and scalar function name as identifier Signed-off-by: Chen Dai <[email protected]> * Revert score and ignore failed IT Signed-off-by: Chen Dai <[email protected]> * Add comparison test to address PR comment Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]> (cherry picked from commit 2f4924a) Co-authored-by: Chen Dai <[email protected]>
…pensearch-project#1220) Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> (cherry picked from commit 683d92f) Co-authored-by: Yury-Fridlyand <[email protected]>
) (opensearch-project#1234) * Add `TIMEDIFF` and `DATEDIFF` functions. Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Max Ksyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Max Ksyunz <[email protected]> (cherry picked from commit 438c44d) Co-authored-by: Yury-Fridlyand <[email protected]>
opensearch-project#1224) (opensearch-project#1233) Signed-off-by: dblock <[email protected]> Signed-off-by: dblock <[email protected]> (cherry picked from commit f4ab469) Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>
…oject#1209) Signed-off-by: Zelin Hao <[email protected]>
…project#1166) Added Tests And Implementation For Match_Query, Match_Phrase, and Multi_Match Functions Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]>
…Hot-Fix Hot Fix For CI Build (cherry picked from commit aae57a0) Signed-off-by: GabeFernandez310 <[email protected]>
…ckport-1166-to-2.x [Backport 2.x] Add Alternate Syntax For Match_Query And Other Functions
…ensearch-project#1246) Signed-off-by: vamsi-amazon <[email protected]> Signed-off-by: vamsi-amazon <[email protected]> (cherry picked from commit ee949cc) Co-authored-by: vamsi-amazon <[email protected]>
…nsearch-project#1231) (opensearch-project#1237) Added Testing And Implementation For Second_Of_Minute Function Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit dce7d0e) Co-authored-by: GabeFernandez310 <[email protected]>
… (opensearch-project#1252) * Add functions `ADDTIME` and `SUBTIME`. (#132) Signed-off-by: Yury-Fridlyand <[email protected]> (cherry picked from commit 7630f87) Co-authored-by: Yury-Fridlyand <[email protected]>
…project#1228) (opensearch-project#1239) Added Implementation And Testing For Day_Of_Week Function Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit bac9c37) Co-authored-by: GabeFernandez310 <[email protected]>
…tion (opensearch-project#1253) * Add Minute_Of_Hour Function As An Alias Of Minute Function (#196) (opensearch-project#1230) Added Testing And Implementation For Minute_Of_Hour Function Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 61e2374) * Added Missing Imports Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]>
…nctions (opensearch-project#1205) (opensearch-project#1255) * Added long fix for CEIL, CEILING and FLOOR functions using LONG instead of INT for RETURN. Signed-off-by: MitchellGale-BitQuill <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]>
…roject#1229) (opensearch-project#1250) * Resolve sub object field in search hit source Signed-off-by: Chen Dai <[email protected]> * Rename to unexpanded object Signed-off-by: Chen Dai <[email protected]> * Update IT with where condition Signed-off-by: Chen Dai <[email protected]> * Fix test index mapping Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]> (cherry picked from commit 151f4cc) Co-authored-by: Chen Dai <[email protected]>
Signed-off-by: Joshua Li <[email protected]>
…ch-project#1223) (opensearch-project#1258) Added Support And Tests For Time Type in day_of_year, week_of_year, month_of_year Functions Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 6e72f18) Co-authored-by: GabeFernandez310 <[email protected]>
…ct#1227) * Add Day_Of_Month Function As An Alias Of DayOfMonth (#194) Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 1108379)
…ect#1264) update build add dependency in jdbc maven artifact add build phase for git clone of sql-cli Signed-off-by: YANGDB <[email protected]>
…ckport/backport-1227-to-2.x [Backport 2.x] Add Day_Of_Month Function As An Alias Of DayOfMonth
opensearch-project#1269) * Add Hour_Of_Day Function As An Alias Of Hour Added Tests And ImplementationFor Hour_Of_Day Function Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Checkstyle Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 9e67e5b) Co-authored-by: GabeFernandez310 <[email protected]>
…ct#1640) (opensearch-project#1670) Signed-off-by: forestmvey <[email protected]> Co-authored-by: forestmvey <[email protected]>
…on (opensearch-project#1657) (opensearch-project#1672) Signed-off-by: forestmvey <[email protected]> Co-authored-by: forestmvey <[email protected]>
…#1680) Signed-off-by: Andrew Carbonetto <[email protected]> (cherry picked from commit 567cabc) Co-authored-by: Andrew Carbonetto <[email protected]>
…earch-project#1677) * Create new anonymizer for new engine (#266) * Created anonymizer listener for anonymizing SQL queries through the new engine Signed-off-by: Matthew Wells <[email protected]> * Update for review comments Signed-off-by: Andrew Carbonetto <[email protected]> * added missing file header, change public variable to private Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: Matthew Wells <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> (cherry picked from commit 62120fd) Co-authored-by: Matthew Wells <[email protected]>
…rch-project#1682) * Support Alternate Datetime Formats (#268) * Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> (cherry picked from commit 986db39) Co-authored-by: Guian Gumpac <[email protected]>
…<table>` queries (opensearch-project#1684) * Support for pagination in v2 engine of `SELECT * FROM <table>` queries (opensearch-project#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Max Ksyunz <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> (cherry picked from commit 57ce303) * Fix test build failure. Somehow the import is required in 2.x but not 3.0 Signed-off-by: MaxKsyunz <[email protected]> --------- Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Max Ksyunz <[email protected]>
…pensearch-project#1689) * Allow backtick around fields in sort cmd Signed-off-by: Joshua Li <[email protected]> * Add test Signed-off-by: Joshua Li <[email protected]> * Update test setup Signed-off-by: Joshua Li <[email protected]> * revert ignoring sql-cli repo if exists Signed-off-by: Joshua Li <[email protected]> --------- Signed-off-by: Joshua Li <[email protected]> (cherry picked from commit 691012d) Co-authored-by: Joshua Li <[email protected]>
…pensearch-project#1729) (cherry picked from commit bd9b180)
Signed-off-by: opensearch-ci-bot <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]>
…pensearch-project#1742) Signed-off-by: Max Ksyunz <[email protected]> (cherry picked from commit 29f99aa) Co-authored-by: Max Ksyunz <[email protected]>
… (opensearch-project#1716) (opensearch-project#1734) * Fixed bug of byte/short values not handling divide/modulus arithmetic equations Signed-off-by: Matthew Wells <[email protected]> (cherry picked from commit 2c80631) Co-authored-by: Matthew Wells <[email protected]>
…lause and for functions and expressions in the query. (opensearch-project#1500) (opensearch-project#1741) * Add support for `WHERE` clause, column list in `SELECT` clause and for functions and expressions in the query. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix merge issue and address PR feedback by updating comments. Signed-off-by: Yury-Fridlyand <[email protected]> * More comments. Signed-off-by: Yury-Fridlyand <[email protected]> * Add extra check for unset `initialSearchRequest`. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> (cherry picked from commit da386e5) Co-authored-by: Yury-Fridlyand <[email protected]>
…ROM`. (opensearch-project#1599) (opensearch-project#1745) * Support `ORDER BY` clauses in pagination and queries without `FROM`. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix IT. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> (cherry picked from commit 94d5479) Co-authored-by: Yury-Fridlyand <[email protected]>
* Fix ML-commons missing dependency. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix `mockito` dependency. Signed-off-by: Yury-Fridlyand <[email protected]> * Revert changes in `:opensearch` since it is not needed anymore. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> (cherry picked from commit f6e2a97) Co-authored-by: Yury-Fridlyand <[email protected]>
…s upon project failure in startOpenSearch (opensearch-project#1763) * fixed buildgradle merge conflicts Signed-off-by: Mitchell Gale <[email protected]> * Fix CI (opensearch-project#1760) * Fix ML-commons missing dependency. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix `mockito` dependency. Signed-off-by: Yury-Fridlyand <[email protected]> * Revert changes in `:opensearch` since it is not needed anymore. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Mitchell Gale <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]>
…ct#1725) (opensearch-project#1770) * fixed round documentation Signed-off-by: Matthew Wells <[email protected]> (cherry picked from commit 34cad6e) Co-authored-by: Matthew Wells <[email protected]>
(cherry picked from commit c7dfdb3) Signed-off-by: forestmvey <[email protected]> Co-authored-by: Forest Vey <[email protected]>
…ensearch-project#1751) Signed-off-by: Joshua Li <[email protected]> (cherry picked from commit 7525bb1) Co-authored-by: Joshua Li <[email protected]>
* Update SQL plugin for core refactor (opensearch-project#1571) Signed-off-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> * Fix plugin compilation (opensearch-project#1580) * Changed gradle version and removed values iterator Signed-off-by: Guian Gumpac <[email protected]> * Update a test to match new indexResponse.aliases() type. Signed-off-by: MaxKsyunz <[email protected]> * Ran ./gradlew wrapper Signed-off-by: Guian Gumpac <[email protected]> --------- Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> * Update sqlite-jdbc to 3.41.2.2 to address CVE-2023-32697 (opensearch-project#1667) * Update sqlite-jdbc to 3.41.2.2 to address CVE-2023-32697 Signed-off-by: MaxKsyunz <[email protected]> * Don't check column names on H2 results for correctness tests as described in opensearch-project#1667 (comment). Signed-off-by: Yury-Fridlyand <[email protected]> * Address PR review comment. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Co-authored-by: MaxKsyunz <[email protected]>
…ct#1737) (opensearch-project#1791) * Add support for Array and ExprValue Parsing With Inner Hits * Adding schema validation for IT test, and another UT for nested arrays. * Making handleAggregationResponse a private function. --------- (cherry picked from commit 9fbcf11) Signed-off-by: forestmvey <[email protected]> Co-authored-by: Forest Vey <[email protected]>
…t#1789) (opensearch-project#1794) * Add Support for Nested Function in Order By Clause (#280) * Adding order by clause support for nested function. Signed-off-by: forestmvey <[email protected]> * Adding test coverage for nested in ORDER BY clause. Signed-off-by: forestmvey <[email protected]> * Added nested function validation to NestedAnalyzer. Signed-off-by: forestmvey <[email protected]> --------- Signed-off-by: forestmvey <[email protected]> * Adding semantic check for missing arguments in function and unit test. Signed-off-by: forestmvey <[email protected]> --------- Signed-off-by: forestmvey <[email protected]> (cherry picked from commit 3302ec8) Co-authored-by: Forest Vey <[email protected]>
* Fix CSV/RAW output header being application/json rather than plain/text (opensearch-project#1779) * Fix CI (opensearch-project#1760) * Fix ML-commons missing dependency. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix `mockito` dependency. Signed-off-by: Yury-Fridlyand <[email protected]> * Revert changes in `:opensearch` since it is not needed anymore. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Matthew Wells <[email protected]> * Fix CSV/RAW outputting wrong format (#279) * Fixed bug where CSV/RAW outputs as JSON rather than plain text Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Matthew Wells <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]> (cherry picked from commit 1ec696d) * updated tests Signed-off-by: Matthew Wells <[email protected]> * updated tests to return name Signed-off-by: Matthew Wells <[email protected]> * changed tests to return value Signed-off-by: Matthew Wells <[email protected]> * removed unneeded imports Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]>
…ensearch-project#1802) * Keep up with refactoring in OpenSearch. * Updating code formatting. --------- (cherry picked from commit 3fc11a4) Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Max Ksyunz <[email protected]>
…roject#1645) Signed-off-by: MaxKsyunz <[email protected]> (cherry picked from commit 8e5d766) Signed-off-by: Max Ksyunz <[email protected]>
andy-k-improving
pushed a commit
that referenced
this pull request
Nov 16, 2024
* Implement creation of ip2geo feature (#257) * Update gradle version to 7.6 (#265) Signed-off-by: Vijayan Balasubramanian <[email protected]> * Implement creation of ip2geo feature * Implementation of ip2geo datasource creation * Implementation of ip2geo processor creation Signed-off-by: Heemin Kim <[email protected]> --------- Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Heemin Kim <[email protected]> Co-authored-by: Vijayan Balasubramanian <[email protected]> * Added unit tests with some refactoring of codes (#271) * Add Unit tests * Set cache true for search query * Remove in memory cache implementation (Two way door decision) * Relying on search cache without custom cache * Renamed datasource state from FAILED to CREATE_FAILED * Renamed class name from *Helper to *Facade * Changed updateIntervalInDays to updateInterval * Changed value type of default update_interval from TimeValue to Long * Read setting value from cluster settings directly Signed-off-by: Heemin Kim <[email protected]> * Sync from main (#280) * Update gradle version to 7.6 (#265) Signed-off-by: Vijayan Balasubramanian <[email protected]> * Exclude lombok generated code from jacoco coverage report (#268) Signed-off-by: Heemin Kim <[email protected]> * Make jacoco report to be generated faster in local (#267) Signed-off-by: Heemin Kim <[email protected]> * Update dependency org.json:json to v20230227 (#273) Co-authored-by: mend-for-github.aaakk.us.kg[bot] <50673670+mend-for-github.aaakk.us.kg[bot]@users.noreply.github.com> * Baseline owners and maintainers (#275) Signed-off-by: Vijayan Balasubramanian <[email protected]> --------- Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Heemin Kim <[email protected]> Co-authored-by: Vijayan Balasubramanian <[email protected]> Co-authored-by: mend-for-github.aaakk.us.kg[bot] <50673670+mend-for-github.aaakk.us.kg[bot]@users.noreply.github.com> * Add datasource name validation (#281) Signed-off-by: Heemin Kim <[email protected]> * Refactoring of code (#282) 1. Change variable name from datasourceName to name 2. Change variable name from id to name 3. Added helper methods in test code Signed-off-by: Heemin Kim <[email protected]> * Change field name from md5 to sha256 (#285) Signed-off-by: Heemin Kim <[email protected]> * Implement get datasource api (#279) Signed-off-by: Heemin Kim <[email protected]> * Update index option (#284) 1. Make geodata index as hidden 2. Make geodata index as read only allow delete after creation is done 3. Refresh datasource index immediately after update Signed-off-by: Heemin Kim <[email protected]> * Make some fields in manifest file as mandatory (#289) Signed-off-by: Heemin Kim <[email protected]> * Create datasource index explicitly (#283) Signed-off-by: Heemin Kim <[email protected]> * Add wrapper class of job scheduler lock service (#290) Signed-off-by: Heemin Kim <[email protected]> * Remove all unused client attributes (#293) Signed-off-by: Heemin Kim <[email protected]> * Update copyright header (#298) Signed-off-by: Heemin Kim <[email protected]> * Run system index handling code with stashed thread context (#297) Signed-off-by: Heemin Kim <[email protected]> * Reduce lock duration and renew the lock during update (#299) Signed-off-by: Heemin Kim <[email protected]> * Implements delete datasource API (#291) Signed-off-by: Heemin Kim <[email protected]> * Set User-Agent in http request (#300) Signed-off-by: Heemin Kim <[email protected]> * Implement datasource update API (#292) Signed-off-by: Heemin Kim <[email protected]> * Refactoring test code (#302) Make buildGeoJSONFeatureProcessorConfig method to be more general Signed-off-by: Heemin Kim <[email protected]> * Add ip2geo processor integ test for failure case (#303) Signed-off-by: Heemin Kim <[email protected]> * Bug fix and refactoring of code (#305) 1. Bugfix: Ingest metadata can be null if there is no processor created 2. Refactoring: Moved private method to another class for better testing support 3. Refactoring: Set some private static final variable as public so that unit test can use it 4. Refactoring: Changed string value to static variable Signed-off-by: Heemin Kim <[email protected]> * Add integration test for Ip2GeoProcessor (#306) Signed-off-by: Heemin Kim <[email protected]> * Add ConcurrentModificationException (#308) Signed-off-by: Heemin Kim <[email protected]> * Add integration test for UpdateDatasource API (#307) Signed-off-by: Heemin Kim <[email protected]> * Bug fix on lock management and few performance improvements (#310) * Release lock before response back to caller for update/delete API * Release lock in background task for creation API * Change index settings to improve indexing performance Signed-off-by: Heemin Kim <[email protected]> * Change index setting from read_only_allow_delete to write (#311) read_only_allow_delete does not block write to an index. The disk-based shard allocator may add and remove this block automatically. Therefore, use index.blocks.write instead. Signed-off-by: Heemin Kim <[email protected]> * Fix bug in get datasource API and improve memory usage (#313) Signed-off-by: Heemin Kim <[email protected]> * Change package for Strings.hasText (#314) (#317) Signed-off-by: Heemin Kim <[email protected]> * Remove jitter and move index setting from DatasourceFacade to DatasourceExtension (#319) Signed-off-by: Heemin Kim <[email protected]> * Do not index blank value and do not enrich null property (#320) Signed-off-by: Heemin Kim <[email protected]> * Move index setting keys to constants (#321) Signed-off-by: Heemin Kim <[email protected]> * Return null index name for expired data (#322) Return null index name for expired data so that it can be deleted by clean up process. Clean up process exclude current index from deleting. Signed-off-by: Heemin Kim <[email protected]> * Add new fields in datasource (#325) Signed-off-by: Heemin Kim <[email protected]> * Delete index once it is expired (#326) Signed-off-by: Heemin Kim <[email protected]> * Add restoring event listener (#328) In the listener, we trigger a geoip data update Signed-off-by: Heemin Kim <[email protected]> * Reverse forcemerge and refresh order (#331) Otherwise, opensearch does not clear old segment files Signed-off-by: Heemin Kim <[email protected]> * Removed parameter and settings (#332) * Removed first_only parameter * Removed max_concurrency and batch_size setting first_only parameter was added as current geoip processor has it. However, the parameter have no benefit for ip2geo processor as we don't do a sequantial search for array data but use multi search. max_concurrency and batch_size setting is removed as these are only reveal internal implementation and could be a future blocker to improve performance later. Signed-off-by: Heemin Kim <[email protected]> * Add a field in datasource for current index name (#333) Signed-off-by: Heemin Kim <[email protected]> * Delete GeoIP data indices after restoring complete (#334) We don't want to use restored GeoIP data indices. Therefore we delete the indices once restoring process complete. When GeoIP metadata index is restored, we create a new GeoIP data index instead. Signed-off-by: Heemin Kim <[email protected]> * Use bool query for array form of IPs (#335) Signed-off-by: Heemin Kim <[email protected]> * Run update/delete request in a new thread (#337) This is not to block transport thread Signed-off-by: Heemin Kim <[email protected]> * Remove IP2Geo processor validation (#336) Cannot query index to get data to validate IP2Geo processor. Will add validation when we decide to store some of data in cluster state metadata. Signed-off-by: Heemin Kim <[email protected]> * Acquire lock sychronously (#339) By acquiring lock asychronously, the remaining part of the code is being run by transport thread which does not allow blocking code. We want only single update happen in a node using single thread. However, it cannot be acheived if I acquire lock asynchronously and pass the listener. Signed-off-by: Heemin Kim <[email protected]> * Added a cache to store datasource metadata (#338) Signed-off-by: Heemin Kim <[email protected]> * Changed class name and package (#341) Signed-off-by: Heemin Kim <[email protected]> * Refactoring of code (#342) 1. Changed class name from Ip2GeoCache to Ip2GeoCachedDao 2. Moved the Ip2GeoCachedDao from cache to dao package Signed-off-by: Heemin Kim <[email protected]> * Add geo data cache (#340) Signed-off-by: Heemin Kim <[email protected]> * Add cache layer to reduce GeoIp data retrieval latency (#343) Signed-off-by: Heemin Kim <[email protected]> * Use _primary in query preference and few changes (#347) 1. Use _primary preference to get datasource metadata so that it can read the latest data. RefreshPolicy.IMMEDIATE won't refresh replica shards immediately according to #346 2. Update datasource metadata index mapping 3. Move batch size from static value to setting Signed-off-by: Heemin Kim <[email protected]> * Wait until GeoIP data to be replicated to all data nodes (#348) Signed-off-by: Heemin Kim <[email protected]> * Update packages according to a change in OpenSearch core (opensearch-project#354) * Update packages according to a change in OpenSearch core Signed-off-by: Heemin Kim <[email protected]> * Update packages according to a change in OpenSearch core (opensearch-project#353) Signed-off-by: Heemin Kim <[email protected]> --------- Signed-off-by: Heemin Kim <[email protected]> --------- Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Heemin Kim <[email protected]> Co-authored-by: Vijayan Balasubramanian <[email protected]> Co-authored-by: mend-for-github.aaakk.us.kg[bot] <50673670+mend-for-github.aaakk.us.kg[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.