-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Communication Mechanism Work Item 6 Release lock api #312
Communication Mechanism Work Item 6 Release lock api #312
Conversation
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## main #312 +/- ##
============================================
- Coverage 31.17% 30.23% -0.95%
- Complexity 66 97 +31
============================================
Files 12 22 +10
Lines 773 1128 +355
Branches 80 107 +27
============================================
+ Hits 241 341 +100
- Misses 513 766 +253
- Partials 19 21 +2
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doing an initial pass
src/test/java/org/opensearch/jobscheduler/multinode/ReleaseLockActionMultiNodeRestIT.java
Show resolved
Hide resolved
Signed-off-by: Varun Jain <[email protected]>
Just a premise, This PR has below changes
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM but a few questions for my own awareness, and strong suggestion of improvement. Feel free to push back if I missed something.
src/main/java/org/opensearch/jobscheduler/rest/action/RestGetJobDetailsAction.java
Show resolved
Hide resolved
src/main/java/org/opensearch/jobscheduler/rest/action/RestGetJobDetailsAction.java
Show resolved
Hide resolved
src/main/java/org/opensearch/jobscheduler/rest/action/RestGetLockAction.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/jobscheduler/rest/action/RestReleaseLockAction.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving based on the intent to refactor the temporary fields to completable future .get()
in a future PR.
Signed-off-by: Varun Jain <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Second pass. The modifications made to the Rest actions to use completableFuture.get() looks good to me, thanks @dbwiddis for the suggestion. Just had a few minor nits
src/main/java/org/opensearch/jobscheduler/rest/action/RestReleaseLockAction.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/jobscheduler/rest/action/RestReleaseLockAction.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/jobscheduler/rest/action/RestReleaseLockAction.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/jobscheduler/rest/action/RestReleaseLockAction.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Varun Jain <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes @vibrantvarun
@@ -70,7 +69,7 @@ public List<Route> routes() { | |||
} | |||
|
|||
@Override | |||
protected RestChannelConsumer prepareRequest(RestRequest restRequest, NodeClient client) throws IOException { | |||
public RestChannelConsumer prepareRequest(RestRequest restRequest, NodeClient client) throws IOException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious why public
? Does it matter?
This interface is anyway called internally by BaseRestHandler[1].
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, because if it is not public then I cannot call it from Test class
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you could make it package private and available by tests
public class RestReleaseLockAction extends BaseRestHandler { | ||
|
||
public static final String RELEASE_LOCK_ACTION = "release_lock_action"; | ||
public static final String LOCK_ID = "lock_id"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets import this from org.opensearch.jobscheduler.spi.LockModel.LOCK_ID
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK
if (e.getCause() instanceof TimeoutException) { | ||
logger.info(" Request timed out with an exception ", e); | ||
} else { | ||
throw e; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm.. do we need the else ? And throw the exception?
The catch all at 87 will anyway take care of it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dbwiddis suggested to catch time out exception seperately
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok if so we need need handle this cleanly, would it drop the rest request ? (I see we are throwing the error, I wonder how this is handled)
Is there a test which tests this case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No as of now there is no test for this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets add one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I think I may have been confused before and reviewing a PR on opensearch cleared it up. We do need to clean up this exception handling. Take a look at ExternsionsManager
and try to follow this pattern:
} catch (CompletionException e) {
if (e.getCause() instanceof TimeoutException) {
logger.info("No response from extension to request.");
}
if (e.getCause() instanceof RuntimeException) {
throw (RuntimeException) e.getCause();
} else if (e.getCause() instanceof Error) {
throw (Error) e.getCause();
} else {
throw new RuntimeException(e.getCause());
}
}
throw e; | ||
} | ||
} catch (Exception e) { | ||
logger.info(" Could not find lock model with lockId due to exception ", e); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have this log at debug level?[1] Info seems like a lot.
[1] https://www.tutorialspoint.com/log4j/log4j_logging_levels.htm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACk
BytesRestResponse bytesRestResponse; | ||
try { | ||
builder.startObject(); | ||
builder.field("response", restResponseString); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would re-name the field to release-lock
to be more explicit when the client reads it. response
is generic and it brings ambiguity.
builder.field("response", restResponseString); | |
builder.field("release-lock", restResponseString); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall the changes look good to me.
Few minor comments..
Signed-off-by: Varun Jain <[email protected]>
5fdfcd9
Signed-off-by: Varun Jain <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM but you're duplicating that annotation from elsewhere
*/ | ||
package org.opensearch.jobscheduler.rest.action; | ||
|
||
public @interface VisibleForTesting { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't this already exist?
import com.google.common.annotations.VisibleForTesting;
if you have that dependencyimport org.opensearch.jobscheduler.utils.VisibleForTesting;
otherwise
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it does have but it only works when when it is in the same package
Signed-off-by: Varun Jain <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doing another pass, some minor nits and a request to add guardrails to the releaseLock API
src/main/java/org/opensearch/jobscheduler/rest/action/RestGetLockAction.java
Show resolved
Hide resolved
src/main/java/org/opensearch/jobscheduler/rest/action/RestGetLockAction.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/jobscheduler/rest/action/RestReleaseLockAction.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
Signed-off-by: Varun Jain <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes @vibrantvarun
…#382) * Communication mechanism for js (#289) * Job Details from Extension for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> * Communication Mechanism for JS Signed-off-by: Varun Jain <[email protected]> Signed-off-by: Varun Jain <[email protected]> * [Extensions] Synchronize opensearch-plugin-job-details with JobDetailsService map (#299) * Added JobDetailsService as an indexOperationListener to synchronize metadata index with internal job details m and indicesToListen set. Signed-off-by: Joshua Palis <[email protected]> * Changes indexToJobDetails to a ConcurrentMap, adds getter method for indexToJobDetails Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments, fixing log message Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments Signed-off-by: Joshua Palis <[email protected]> * Added test to updateIndexToJobDetails Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments, changing extensionId to extensionUniqueId Signed-off-by: Joshua Palis <[email protected]> * Addressing PR Comments : Updating Job Details index Name and mapping file to opensearch-job-scheduler-job-details Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments, enabling extensions to submit more than 1 job details entry to support extensions registering multiple types of jobs, updated all integration, multinode tests now that the rest response value is the document Id Signed-off-by: Joshua Palis <[email protected]> * Renaming TestHelper base URI name Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments, made multinode test request strings constant Signed-off-by: Joshua Palis <[email protected]> Signed-off-by: Joshua Palis <[email protected]> * [Extensions] Create a proxy ScheduledJobRunner, ScheduledJobParser to invoke corresponding registered Job Detail actions (#306) * Draft : Added JobParameterRequest to translate ScheduledJobParser parameters nto compatible inputs for the ExtensionActionRequest. Completed initial proxy scheduled job parser implementation. Added ExtensionJobParameter class to deserialize ExtensionActionResponsebyte array into an object of type ScheduledJobParameter Signed-off-by: Joshua Palis <[email protected]> * Added JobRunnerRequest, modified JobExecutionContext to implement writeable, created initial proxy ScheduledJobRunner, fixed failing tests Signed-off-by: Joshua Palis <[email protected]> * Added generic ExtensionJobActionRequest that extends ExtensionActionRequest, modified JobParameter/Runner request to implement writeable, refactored proxy object creation so that the requests are added to the ExtensionJobActionRequest, which in turn upcasts the request into an ExtensionActionRequest Signed-off-by: Joshua Palis <[email protected]> * Added byte array constructors for the JobRunner/Parameter requests, added javadocs Signed-off-by: Joshua Palis <[email protected]> * Fixing javadocs Signed-off-by: Joshua Palis <[email protected]> * Fixing javadocs Signed-off-by: Joshua Palis <[email protected]> * Added placeholder string for an eventual access token to be sent to extensions to validate prior to invoking an action Signed-off-by: Joshua Palis <[email protected]> * Added ExtensionJobActionResponse, JobParameterResponse, JobRunnerResponse to facilitate the response of extension actions Signed-off-by: Joshua Palis <[email protected]> * Fixing javadocs Signed-off-by: Joshua Palis <[email protected]> * Added JobRunnerResponse handling Signed-off-by: Joshua Palis <[email protected]> * Separating updateIndexToJobProviders into separate methods Signed-off-by: Joshua Palis <[email protected]> * Fixing javadocs Signed-off-by: Joshua Palis <[email protected]> * SpotlessApply Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments Signed-off-by: Joshua Palis <[email protected]> * Added tests for serialization/deserialization of JobExecutionContext, JobDocVersion, ExtensionJobParameter, ExtensionJobActionRequest, ExtensionJobActionResponse, JobRunner/JobParameter/Request/Response Signed-off-by: Joshua Palis <[email protected]> * Writing ExtensionActionRequest/Response to bytestream output to test deserialization Signed-off-by: Joshua Palis <[email protected]> * Fixing imports Signed-off-by: Joshua Palis <[email protected]> * Changing to extensionActionResponse constructor Signed-off-by: Joshua Palis <[email protected]> * Adding tests for updateIndexToJobProviders Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments, added getters for lock duration seconds and jitter values, added javadocs to ScheduleType enum and made this public Signed-off-by: Joshua Palis <[email protected]> * Removing Strings dependency from lock model to fix BWC tests Signed-off-by: Joshua Palis <[email protected]> * Fixes BWC fullRestartClusterTask, rollingUpgradeClusterTask, oldVersionCluster task. mixedClusterTask is still failing Signed-off-by: Joshua Palis <[email protected]> * Modified createProxyScheduledJobRunner to return the document Id of the extension job parameter entry within the registered job index, modifed the ExtensionJobParameter to null check the jitter value and setting this to 0.0 if null Signed-off-by: Joshua Palis <[email protected]> --------- Signed-off-by: Joshua Palis <[email protected]> * [Extensions] Exposes a GetLock REST API to enable extensions to acquire a lock model for their job execution (#311) * Added RestGetLockAction API, added AcquireLockRequest, enables extensions to retrieve a lock model object to run their Job Signed-off-by: Joshua Palis <[email protected]> * Adding Lock ID field to RestGetLockAction response Signed-off-by: Joshua Palis <[email protected]> * Added multi node integration tests for GetLockApi Signed-off-by: Joshua Palis <[email protected]> * Making lock service in RestGetLockAction private Signed-off-by: Joshua Palis <[email protected]> * Added Rest integration tests for RestGetLockAction Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments Signed-off-by: Joshua Palis <[email protected]> * Updating get lock rest path in tests Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments Signed-off-by: Joshua Palis <[email protected]> --------- Signed-off-by: Joshua Palis <[email protected]> * Communication Mechanism Work Item 6 Release lock api (#312) * Release lock API Signed-off-by: Varun Jain <[email protected]> * Release Lock API Signed-off-by: Varun Jain <[email protected]> * Release Lock API Signed-off-by: Varun Jain <[email protected]> * Release Lock API Signed-off-by: Varun Jain <[email protected]> * Reformatting Signed-off-by: Varun Jain <[email protected]> * Reformatting Signed-off-by: Varun Jain <[email protected]> * Reformatting Signed-off-by: Varun Jain <[email protected]> * Addressing Dan's Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Dan's Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Sarat's Comments Signed-off-by: Varun Jain <[email protected]> * Addressing Sarat's Comments Signed-off-by: Varun Jain <[email protected]> * Fixing test cases Signed-off-by: Varun Jain <[email protected]> * Fixing test cases Signed-off-by: Varun Jain <[email protected]> * Fixing ReleaseLocktestcase Signed-off-by: Varun Jain <[email protected]> * Fixing ReleaseLocktestcase Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> * Bumping BWC version to 2.7 and Fixing xcontent imports (#322) * [Extensions] Add all fields of LockModel to RestGetLockAction response (#342) * Modifies the RestGetLockAction response to include all fields of the lock model object, rather than the lock model object itself Signed-off-by: Joshua Palis <[email protected]> * Removing unused fields Signed-off-by: Joshua Palis <[email protected]> * reverting field removal Signed-off-by: Joshua Palis <[email protected]> * Adding lockID to back to response Signed-off-by: Joshua Palis <[email protected]> * Adding checks to multinode get lock rest integration tests to ensure that all response fields are populated Signed-off-by: Joshua Palis <[email protected]> * fixing lock time parsing Signed-off-by: Joshua Palis <[email protected]> * Adds a new Response class to house serde logic for RestGetLockAction response Signed-off-by: Joshua Palis <[email protected]> * Moving parser.nextToken() within AcquireLockResponse.parse() Signed-off-by: Joshua Palis <[email protected]> * Implementing toXContentObject for AcquireLockRequest Signed-off-by: Joshua Palis <[email protected]> * Moving AcquireLockResponse to transpor package Signed-off-by: Joshua Palis <[email protected]> --------- Signed-off-by: Joshua Palis <[email protected]> * Increasing JobDetailsService request timeoufrom 10 to 15, since the initial job detail registration request will initialize the job details metadata index, which takes some time (#346) Signed-off-by: Joshua Palis <[email protected]> * Fixes serde logic for proxy scheduled jobrunner/parser requests/responses (#349) Signed-off-by: Joshua Palis <[email protected]> * Fixes serde logic for JobParameterRequest/JobRunnerRequest. Removes extra trimming of request bytes for the streaminput constructor for the JobParameter/RunnerRequest, as this is already trimmed by the SDK prior to forwarding the actionrequest to the transport action (#351) Signed-off-by: Joshua Palis <[email protected]> * [Extensions] Makes JobRunner/Parameter/Request/Response classes extend from ActionRequest/Response (#352) * Replaces ExtensionActionRequest class name with the JobParameter/RunnerRequest fully qualified class names, modifes JobParameter/Runner/Request/Response classes to extend from ActionRequest/Response Signed-off-by: Joshua Palis <[email protected]> * Removes ExtensionJobActionResponse and fixes affected test classes Signed-off-by: Joshua Palis <[email protected]> * Fixing javadocs Signed-off-by: Joshua Palis <[email protected]> --------- Signed-off-by: Joshua Palis <[email protected]> * Consuming breaking changes from moving ExtensionActionRequest (#381) Signed-off-by: Sarat Vemulapalli <[email protected]> * spotless Signed-off-by: Joshua Palis <[email protected]> * Fixing apache imports for TestHelpers class Signed-off-by: Joshua Palis <[email protected]> * Modofies ODFERestTestCase to work with 2.x Signed-off-by: Joshua Palis <[email protected]> * Fixing wipeAllODFEIndices Signed-off-by: Joshua Palis <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> Signed-off-by: Joshua Palis <[email protected]> Signed-off-by: Sarat Vemulapalli <[email protected]> Co-authored-by: Varun Jain <[email protected]> Co-authored-by: Sarat Vemulapalli <[email protected]>
Description
Release Lock API
Issues Resolved
opensearch-project/opensearch-sdk-java#370
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.