Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: register mulitple extensions. #10256

Merged

Conversation

dblock
Copy link
Member

@dblock dblock commented Sep 27, 2023

Description

Fixes registering multiple extensions with OpenSearch.

To reproduce the issue install two extensions, using https://github.com/opensearch-project/opensearch-sdk-java and https://github.com/opensearch-project/opensearch-sdk-py samples.

# java extension
cd ~/source/opensearch-project/OpenSearch/dblock-OpenSearch/
git diff

cd ~/source/opensearch-project/opensearch-sdk-java/dblock-opensearch-sdk-java/
./gradlew helloWorld
cat bin/main/org/opensearch/sdk/sample/helloworld/hello.json | jq
curl -s -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data @bin/main/org/opensearch/sdk/sample/helloworld/hello.json | jq
curl http://localhost:9200/_extensions/_hello-world-java/hello

# python extension
cd ~/source/opensearch-project/opensearch-sdk-py/dblock-opensearch-sdk-py/
cat samples/hello/hello.json | jq
poetry run samples/hello/hello.py
curl -s -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data @samples/hello/hello.json | jq
curl http://localhost:9200/_extensions/_hello-world-py/hello

without the first fix Java extension receives

08:59:01.589 [opensearch[hello-world][generic][T#4]] INFO  org.opensearch.sdk.handlers.AcknowledgedResponseHandler - Extension Request failed
org.opensearch.transport.RemoteTransportException: [runTask-0][127.0.0.1:9300][internal:discovery/registercustomsettings]
Caused by: org.opensearch.common.settings.SettingsException: Could not register setting:custom.validate
        at org.opensearch.common.settings.SettingsModule.registerDynamicSetting(SettingsModule.java:224) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.settings.CustomSettingsRequestHandler.handleRegisterCustomSettingsRequest(CustomSettingsRequestHandler.java:50) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.ExtensionsManager.lambda$registerRequestHandler$1(ExtensionsManager.java:233) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:454) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
08:59:01.589 [opensearch[hello-world][generic][T#3]] INFO  org.opensearch.sdk.handlers.AcknowledgedResponseHandler - Extension Request failed
org.opensearch.transport.RemoteTransportException: [runTask-0][127.0.0.1:9300][internal:discovery/registerrestactions]
Caused by: java.lang.IllegalArgumentException: route [NamedRoute [method=GET, path=/_extensions/_hello-world-java/hello, name=greet, actionNames=[cluster:admin/opensearch/hw/greet]]] already registered
        at org.opensearch.action.ActionModule$DynamicActionRegistry.registerDynamicRoute(ActionModule.java:1122) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.rest.RestSendToExtensionAction.<init>(RestSendToExtensionAction.java:123) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.rest.RestActionsRequestHandler.handleRegisterRestActionsRequest(RestActionsRequestHandler.java:65) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.ExtensionsManager.lambda$registerRequestHandler$0(ExtensionsManager.java:224) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:454) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

without the second fix, java ext receives

09:03:21.793 [opensearch[hello-world][generic][T#2]] INFO  org.opensearch.sdk.handlers.AcknowledgedResponseHandler - Extension Request failed
org.opensearch.transport.RemoteTransportException: [runTask-0][127.0.0.1:9300][internal:discovery/registercustomsettings]
Caused by: org.opensearch.common.settings.SettingsException: Could not register setting:custom.validate
        at org.opensearch.common.settings.SettingsModule.registerDynamicSetting(SettingsModule.java:224) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.settings.CustomSettingsRequestHandler.handleRegisterCustomSettingsRequest(CustomSettingsRequestHandler.java:50) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.ExtensionsManager.lambda$registerRequestHandler$1(ExtensionsManager.java:233) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:454) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
09:03:21.793 [opensearch[hello-world][generic][T#1]] INFO  org.opensearch.sdk.handlers.AcknowledgedResponseHandler - Extension Request failed
org.opensearch.transport.RemoteTransportException: [runTask-0][127.0.0.1:9300][internal:discovery/registerrestactions]
Caused by: java.lang.IllegalArgumentException: route [NamedRoute [method=GET, path=/_extensions/_hello-world-java/hello, name=greet, actionNames=[cluster:admin/opensearch/hw/greet]]] already registered
        at org.opensearch.action.ActionModule$DynamicActionRegistry.registerDynamicRoute(ActionModule.java:1122) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.rest.RestSendToExtensionAction.<init>(RestSendToExtensionAction.java:123) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.rest.RestActionsRequestHandler.handleRegisterRestActionsRequest(RestActionsRequestHandler.java:65) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.extensions.ExtensionsManager.lambda$registerRequestHandler$0(ExtensionsManager.java:224) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:454) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

python ext receives

INFO:root:b'ES\x00\x00\x06g\x00\x00\x00\x00\x00\x00\x00\x06\x03\x08 2\x93\x00\x00\x00%\x01\x1c_system_index_access_allowed\x05false\x00\x01\x00g\x01D[runTask-0][127.0.0.1:50617][internal:discovery/registerrestactions]\x01\x06\x01\xdf\x01handler of type [org.opensearch.extensions.rest.RestSendToExtensionAction] conflicts with handler of type [org.opensearch.extensions.rest.RestSendToExtensionAction] as they both have the same name [send_to_extension_action]\x00\x0f!org.opensearch.usage.UsageService\x01\x11UsageService.java\x0eaddRestHandlerc"org.opensearch.rest.RestController\x01\x13RestController.java\x0fregisterHandler\xd8\x01"org.opensearch.rest.RestController\x01\x13RestController.java\x18lambda$registerHandler$3\xea\x01\x13java.util.ArrayList\x01\x0eArrayList.java\x07forEach\xe7\x0b,java.util.Collections$UnmodifiableCollection\x01\x10Collections.java\x07forEach\xc4\x08"org.opensearch.rest.RestController\x01\x13RestController.java\x0fregisterHandler\xea\x018org.opensearch.extensions.rest.RestActionsRequestHandler\x01\x1eRestActionsRequestHandler.java handleRegisterRestActionsRequestH+org.opensearch.extensions.ExtensionsManager\x01\x16ExtensionsManager.java\x1flambda$registerRequestHandler$0\xe0\x01/org.opensearch.transport.RequestHandlerRegistry\x01\x1bRequestHandlerRegistry.java\x16processMessageReceivedj6org.opensearch.transport.InboundHandler$RequestHandler\x01\x13InboundHandler.java\x05doRun\xc6\x03Uorg.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable\x01\x12ThreadContext.java\x05doRun\x8c\x076org.opensearch.common.util.concurrent.AbstractRunnable\x01\x15AbstractRunnable.java\x03run4\'java.util.concurrent.ThreadPoolExecutor\x01\x17ThreadPoolExecutor.java\trunWorker\xf0\x08.java.util.concurrent.ThreadPoolExecutor$Worker\x01\x17ThreadPoolExecutor.java\x03run\xfb\x04\x10java.lang.Thread\x01\x0bThread.java\x03run\xc1\x06\x00\x00\x00\x00\x00\x01\x04\x7f\x00\x00\x01\t127.0.0.1\x00\x00\xc5\xb9\x01&internal:discovery/registerrestactions'

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dblock
Copy link
Member Author

dblock commented Sep 27, 2023

cc: @dbwiddis

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Sep 27, 2023

Compatibility status:

Checks if related components are compatible with change cf741e1

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

@dbwiddis
Copy link
Member

For context of reviewers:

  • bugs were discovered preparing a demo in OpenSearchCon. A full bug issue has not yet been filed
  • this PR fixes the bugs but does not include needed tests etc. which is why it is in draft
  • this PR was committed to preserve the "real time" fix to follow up with properly later
  • there will be later followup when time permits with a full bug description, and this PR will be improved with proper tests, etc.

Quick summary of 2 bugs:

  • initialization sequence tries to re-initialize all previously initialized extensions
  • the "handler" object forwarding REST requests to an extension needs a unique name but the name was not unqiue

@dblock dblock force-pushed the fix-two-extensions-registration branch 2 times, most recently from 35c6b61 to 3db7b0e Compare September 29, 2023 15:20
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dblock dblock force-pushed the fix-two-extensions-registration branch from b14d35c to c16dc56 Compare September 29, 2023 22:31
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dblock dblock force-pushed the fix-two-extensions-registration branch from c16dc56 to 39f209e Compare September 30, 2023 00:57
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT:
  • URL:
  • CommitID: 39f209e
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@dblock dblock force-pushed the fix-two-extensions-registration branch from 39f209e to 5ecf203 Compare September 30, 2023 01:05
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL: null
  • CommitID: 5ecf203
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Oct 2, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Oct 2, 2023

Gradle Check (Jenkins) Run Completed with:

@dblock dblock force-pushed the fix-two-extensions-registration branch from 6386913 to cf741e1 Compare October 2, 2023 20:59
@github-actions
Copy link
Contributor

github-actions bot commented Oct 2, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Oct 2, 2023

Gradle Check (Jenkins) Run Completed with:

@codecov
Copy link

codecov bot commented Oct 2, 2023

Codecov Report

Merging #10256 (cf741e1) into main (d3bf230) will decrease coverage by 0.01%.
Report is 1 commits behind head on main.
The diff coverage is 75.00%.

@@             Coverage Diff              @@
##               main   #10256      +/-   ##
============================================
- Coverage     71.15%   71.14%   -0.01%     
- Complexity    58247    58258      +11     
============================================
  Files          4831     4831              
  Lines        274684   274689       +5     
  Branches      40026    40027       +1     
============================================
- Hits         195443   195428      -15     
+ Misses        62884    62777     -107     
- Partials      16357    16484     +127     
Files Coverage Δ
...rch/extensions/rest/RestActionsRequestHandler.java 100.00% <100.00%> (ø)
...extensions/rest/RestInitializeExtensionAction.java 65.00% <100.00%> (-0.44%) ⬇️
...rch/extensions/rest/RestSendToExtensionAction.java 42.14% <100.00%> (ø)
...a/org/opensearch/extensions/ExtensionsManager.java 52.14% <62.50%> (+8.12%) ⬆️

... and 461 files with indirect coverage changes

@dblock dblock merged commit 1d66af3 into opensearch-project:main Oct 2, 2023
13 checks passed
@dblock dblock added the backport 2.x Backport to 2.x branch label Oct 2, 2023
@dblock dblock deleted the fix-two-extensions-registration branch October 2, 2023 21:52
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-10256-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1d66af346f7863bbca44a3102a1af2fed5fd3804
# Push it to GitHub
git push --set-upstream origin backport/backport-10256-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-10256-to-2.x.

dblock added a commit to dblock/OpenSearch that referenced this pull request Oct 2, 2023
* Fix: register mulitple extensions.

Signed-off-by: dblock <[email protected]>

* Updated CHANGELOG.

Signed-off-by: dblock <[email protected]>

* Added tests.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
(cherry picked from commit 1d66af3)
dblock added a commit to dblock/OpenSearch that referenced this pull request Oct 2, 2023
* Fix: register mulitple extensions.

Signed-off-by: dblock <[email protected]>

* Updated CHANGELOG.

Signed-off-by: dblock <[email protected]>

* Added tests.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
(cherry picked from commit 1d66af3)
Signed-off-by: dblock <[email protected]>
dblock added a commit to dblock/OpenSearch that referenced this pull request Oct 2, 2023
* Fix: register mulitple extensions.

Signed-off-by: dblock <[email protected]>

* Updated CHANGELOG.

Signed-off-by: dblock <[email protected]>

* Added tests.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
(cherry picked from commit 1d66af3)
Signed-off-by: dblock <[email protected]>
dblock added a commit to dblock/OpenSearch that referenced this pull request Oct 3, 2023
* Fix: register mulitple extensions.

Signed-off-by: dblock <[email protected]>

* Updated CHANGELOG.

Signed-off-by: dblock <[email protected]>

* Added tests.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
(cherry picked from commit 1d66af3)
Signed-off-by: dblock <[email protected]>
dblock added a commit that referenced this pull request Oct 3, 2023
* Fix: register mulitple extensions.



* Updated CHANGELOG.



* Added tests.



---------


(cherry picked from commit 1d66af3)

Signed-off-by: dblock <[email protected]>
rayshrey pushed a commit to rayshrey/OpenSearch that referenced this pull request Oct 3, 2023
* Fix: register mulitple extensions.

Signed-off-by: dblock <[email protected]>

* Updated CHANGELOG.

Signed-off-by: dblock <[email protected]>

* Added tests.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
deshsidd pushed a commit to deshsidd/OpenSearch that referenced this pull request Oct 9, 2023
* Fix: register mulitple extensions.

Signed-off-by: dblock <[email protected]>

* Updated CHANGELOG.

Signed-off-by: dblock <[email protected]>

* Added tests.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
vikasvb90 pushed a commit to vikasvb90/OpenSearch that referenced this pull request Oct 10, 2023
* Fix: register mulitple extensions.

Signed-off-by: dblock <[email protected]>

* Updated CHANGELOG.

Signed-off-by: dblock <[email protected]>

* Added tests.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
* Fix: register mulitple extensions.

Signed-off-by: dblock <[email protected]>

* Updated CHANGELOG.

Signed-off-by: dblock <[email protected]>

* Added tests.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport-failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants