Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] QueryIndex rollover when field mapping limit is reached. #729

Merged
merged 2 commits into from
Jan 5, 2023

Conversation

petardz
Copy link
Contributor

@petardz petardz commented Jan 3, 2023

Signed-off-by: Petar Dzepina [email protected]

Issue #, if available:

Description of changes:

CheckList:
[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@petardz petardz requested a review from a team January 3, 2023 17:18
Signed-off-by: Petar Dzepina <[email protected]>
@codecov-commenter
Copy link

codecov-commenter commented Jan 4, 2023

Codecov Report

Merging #729 (4885c6a) into 2.x (6eb12f7) will decrease coverage by 0.84%.
The diff coverage is 66.36%.

@@             Coverage Diff              @@
##                2.x     #729      +/-   ##
============================================
- Coverage     76.15%   75.30%   -0.85%     
  Complexity      116      116              
============================================
  Files           124      125       +1     
  Lines          6617     6819     +202     
  Branches        982     1020      +38     
============================================
+ Hits           5039     5135      +96     
- Misses         1080     1153      +73     
- Partials        498      531      +33     
Impacted Files Coverage Δ
...in/kotlin/org/opensearch/alerting/MonitorRunner.kt 63.29% <ø> (-0.46%) ⬇️
...pensearch/alerting/model/AlertingConfigAccessor.kt 75.00% <ø> (-3.38%) ⬇️
...tlin/org/opensearch/alerting/util/AlertingUtils.kt 53.12% <ø> (-7.41%) ⬇️
...alerting/transport/TransportDeleteMonitorAction.kt 66.29% <55.88%> (-8.34%) ⬇️
.../opensearch/alerting/DocumentLevelMonitorRunner.kt 77.58% <63.15%> (-4.58%) ⬇️
...n/org/opensearch/alerting/model/MonitorMetadata.kt 44.31% <64.00%> (+5.51%) ⬆️
.../org/opensearch/alerting/MonitorMetadataService.kt 65.07% <65.07%> (ø)
...lerting/transport/TransportExecuteMonitorAction.kt 82.08% <66.66%> (+0.55%) ⬆️
...opensearch/alerting/util/DocLevelMonitorQueries.kt 72.87% <69.81%> (-7.49%) ⬇️
.../alerting/transport/TransportIndexMonitorAction.kt 54.93% <75.00%> (-5.01%) ⬇️
... and 9 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@lezzago
Copy link
Member

lezzago commented Jan 5, 2023

Is this a backport PR for #725? If so, can we clarify that in the PR title and description. Additionally can we rename the title to then be: [Backport 2.x] QueryIndex rollover when field mapping limit is reached.

@sbcd90 sbcd90 changed the title conflict resolve for #725 to 2.x [Backport 2.x] QueryIndex rollover when field mapping limit is reached. Jan 5, 2023
@sbcd90 sbcd90 merged commit c0f539b into opensearch-project:2.x Jan 5, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.3 2.3
# Navigate to the new working tree
cd .worktrees/backport-2.3
# Create a new branch
git switch --create backport/backport-729-to-2.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c0f539ba8089296037194ed46ae3c7d5a9fad940
# Push it to GitHub
git push --set-upstream origin backport/backport-729-to-2.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.3

Then, create a pull request where the base branch is 2.3 and the compare/head branch is backport/backport-729-to-2.3.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.5 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.5 2.5
# Navigate to the new working tree
cd .worktrees/backport-2.5
# Create a new branch
git switch --create backport/backport-729-to-2.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c0f539ba8089296037194ed46ae3c7d5a9fad940
# Push it to GitHub
git push --set-upstream origin backport/backport-729-to-2.5
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.5

Then, create a pull request where the base branch is 2.5 and the compare/head branch is backport/backport-729-to-2.5.

AWSHurneyt pushed a commit to AWSHurneyt/OpenSearch-Alerting that referenced this pull request Jun 6, 2023
lezzago pushed a commit to lezzago/alerting-opensearch that referenced this pull request Jun 7, 2023
lezzago added a commit that referenced this pull request Jun 8, 2023
* Added exception check once the .opendistro-alerting-config index is b… (#650)

* Added exception check once the .opendistro-alerting-config index is being created

During .opendistro-alerting-config index creation, if ResourceAlreadyExists exception is being raised, the flow will check first if the index is in yellow state and then it will re-try to index monitor

Signed-off-by: Stevan Buzejic <[email protected]>

* Formating of the file fixed

Signed-off-by: Stevan Buzejic <[email protected]>

Signed-off-by: Stevan Buzejic <[email protected]>

* refactored DeleteMonitor Action to be synchronious (#628) (#630)

* refactored DeleteMonitor Action to be synchronious

Signed-off-by: Petar Dzepina <[email protected]>

Signed-off-by: Petar Dzepina <[email protected]>
Co-authored-by: Petar Dzepina <[email protected]>

* [Backport 2.x] QueryIndex rollover when field mapping limit is reached (#729)

Signed-off-by: Petar Dzepina <[email protected]>

* Mappings fix backport 2.x (#730)

* Added support for "nested" mappings (#645)

* example

Signed-off-by: Petar Dzepina <[email protected]>

* fixed updating mappings for queryIndex

Signed-off-by: Petar Dzepina <[email protected]>

Signed-off-by: Petar Dzepina <[email protected]>

* mappings traversal bug fix (#669)

Signed-off-by: Petar Dzepina <[email protected]>

Signed-off-by: Petar Dzepina <[email protected]>

* Added unwrapping exception from core; added more debug logs (#728)

Signed-off-by: Petar Dzepina <[email protected]>

* Add DataSources test for future backports

Signed-off-by: Ashish Agrawal <[email protected]>

* fix  percolator mapping error when having field name 'type' (#726)

Signed-off-by: Raj Chakravarthi <[email protected]>

* [BUG] ExecuteMonitor inserting metadata doc during dry run (#758)

* execute monitor bugfix

Signed-off-by: Petar Dzepina <[email protected]>

* added IT

Signed-off-by: Petar Dzepina <[email protected]>

* fixed created retval when skipIndex=true

Signed-off-by: Petar Dzepina <[email protected]>

---------

Signed-off-by: Petar Dzepina <[email protected]>

* fix for ERROR alert state generation in doc-level monitors (#768)

Signed-off-by: Subhobrata Dey <[email protected]>

* Adjusting max field index setting dynamically for query index (#776)

* added adjusting max field index setting dynamicly for query index

Signed-off-by: Petar Dzepina <[email protected]>

* Multiple indices support in DocLevelMonitorInput (#784)

Signed-off-by: Petar Dzepina <[email protected]>

* Doc transform 2.x backport (#853)

* conflict resovle - backport from main to 2.x

Signed-off-by: Petar Dzepina <[email protected]>

* fixed module class names

Signed-off-by: Petar Dzepina <[email protected]>

---------

Signed-off-by: Petar Dzepina <[email protected]>

* Update config index schema if needed at the start of each monitor execution (#849)

* Update config index schema if needed at the start of each monitor execution

Signed-off-by: Ashish Agrawal <[email protected]>

* Mappings parsing fix (#851)

* fixed mappings parsing when field name named "properties" exists in mappings

Signed-off-by: Petar Dzepina <[email protected]>

* message typo fix

Signed-off-by: Petar Dzepina <[email protected]>

---------

Signed-off-by: Petar Dzepina <[email protected]>

* [Backport 2.x] Notification security fix (#861)

* Notification security fix (#852)

* added injecting whole user object in threadContext before calling notification APIs so that backend roles are available to notification plugin

Signed-off-by: Petar Dzepina <[email protected]>

* compile fix

Signed-off-by: Petar Dzepina <[email protected]>

* refactored user_info injection to use InjectSecurity

Signed-off-by: Petar Dzepina <[email protected]>

* ktlint fix

Signed-off-by: Petar Dzepina <[email protected]>

---------

Signed-off-by: Petar Dzepina <[email protected]>
(cherry picked from commit e0b7a5a)

* remove unneeded import

Signed-off-by: Ashish Agrawal <[email protected]>

---------

Signed-off-by: Ashish Agrawal <[email protected]>
Co-authored-by: Petar Dzepina <[email protected]>
Co-authored-by: Ashish Agrawal <[email protected]>

* Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names. (#894)

Signed-off-by: AWSHurneyt <[email protected]>

* fixed security tests (#484) (#794)

* fixed security tests

Signed-off-by: Raj Chakravarthi <[email protected]>
(cherry picked from commit c51940f)

---------

Signed-off-by: Stevan Buzejic <[email protected]>
Signed-off-by: Petar Dzepina <[email protected]>
Signed-off-by: Ashish Agrawal <[email protected]>
Signed-off-by: Raj Chakravarthi <[email protected]>
Signed-off-by: Subhobrata Dey <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Co-authored-by: Stevan Buzejic <[email protected]>
Co-authored-by: Surya Sashank Nistala <[email protected]>
Co-authored-by: Petar Dzepina <[email protected]>
Co-authored-by: RAJ CHAKRAVARTHI <[email protected]>
Co-authored-by: Subhobrata Dey <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: AWSHurneyt <[email protected]>
Co-authored-by: RAJ CHAKRAVARTHI <[email protected]>
AWSHurneyt added a commit that referenced this pull request Jun 8, 2023
* [Backport 2.x] QueryIndex rollover when field mapping limit is reached (#729)

Signed-off-by: Petar Dzepina <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>

* log error messages and clean up monitor when indexing doc level queries or metadata creation fails (#900) (#912)

* log errors and clean up monitor when indexing doc level queries or metadata creation fails
* refactor delete monitor action to re-use delete methods
Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>

* [Backport 2.x] Notification security fix (#861)

* Notification security fix (#852)

* added injecting whole user object in threadContext before calling notification APIs so that backend roles are available to notification plugin

Signed-off-by: Petar Dzepina <[email protected]>

* compile fix

Signed-off-by: Petar Dzepina <[email protected]>

* refactored user_info injection to use InjectSecurity

Signed-off-by: Petar Dzepina <[email protected]>

* ktlint fix

Signed-off-by: Petar Dzepina <[email protected]>

---------

Signed-off-by: Petar Dzepina <[email protected]>
(cherry picked from commit e0b7a5a)

* remove unneeded import

Signed-off-by: Ashish Agrawal <[email protected]>

---------

Signed-off-by: Ashish Agrawal <[email protected]>
Co-authored-by: Petar Dzepina <[email protected]>
Co-authored-by: Ashish Agrawal <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>

* Added missing imports.

Signed-off-by: AWSHurneyt <[email protected]>

* Multiple indices support in DocLevelMonitorInput (#784) (#808)

Signed-off-by: Petar Dzepina <[email protected]>

* Removed redundant calls to initDocLevelQueryIndex and indexDocLevelQueries.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names. (#894) (#902)

Signed-off-by: AWSHurneyt <[email protected]>
(cherry picked from commit 8c033b9)

Co-authored-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>

* Resolved backport issue for PR 729.

Signed-off-by: AWSHurneyt <[email protected]>

* Resolved backport issue for PR 758.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: Petar Dzepina <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: Ashish Agrawal <[email protected]>
Co-authored-by: Petar Dzepina <[email protected]>
Co-authored-by: Surya Sashank Nistala <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Ashish Agrawal <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants