From 36f02f663410adeaf205661cb2a3c1faab65bcf8 Mon Sep 17 00:00:00 2001 From: "Daniel Doubrovkine (dB.)" Date: Wed, 6 Apr 2022 18:37:49 -0400 Subject: [PATCH 1/9] Updated issue templates from .github. (#165) Signed-off-by: dblock changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi changed assemble Signed-off-by: pgodithi Enables locking on an arbitrary lockID (#164) * Enables locking on an arbitrary lockID Signed-off-by: Clay Downs Test removing find cli's Signed-off-by: pgodithi Enables locking on an arbitrary lockID (#164) * Enables locking on an arbitrary lockID Signed-off-by: Clay Downs Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Adding back find cli's Signed-off-by: pgodithi Updated issue templates from .github. (#165) Signed-off-by: dblock Enables locking on an arbitrary lockID (#164) * Enables locking on an arbitrary lockID Signed-off-by: Clay Downs zip maven publish Signed-off-by: pgodithi zip maven publish Signed-off-by: pgodithi zip maven publish Signed-off-by: pgodithi Adding code block for maven plugin zips Signed-off-by: pgodithi zip maven publish Signed-off-by: pgodithi zip maven publish Signed-off-by: pgodithi zip maven publish Signed-off-by: pgodithi zip maven publish Signed-off-by: pgodithi --- .github/ISSUE_TEMPLATE/bug_report.md | 31 ++++++ .github/ISSUE_TEMPLATE/bug_template.md | 33 ------ .github/ISSUE_TEMPLATE/config.yml | 7 ++ .github/ISSUE_TEMPLATE/feature_request.md | 17 ++- build.gradle | 2 +- scripts/build.sh | 10 +- settings.gradle | 3 +- .../jobscheduler/spi/LockModel.java | 2 + .../jobscheduler/spi/utils/LockService.java | 37 +++++-- .../jobscheduler/spi/utils/LockServiceIT.java | 78 ++++++++++---- zipsmaven/build.gradle | 102 ++++++++++++++++++ 11 files changed, 249 insertions(+), 73 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md delete mode 100644 .github/ISSUE_TEMPLATE/bug_template.md create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 zipsmaven/build.gradle diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..29eddb95 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,31 @@ +--- +name: 🐛 Bug report +about: Create a report to help us improve +title: '[BUG]' +labels: 'bug, untriaged' +assignees: '' +--- + +**What is the bug?** +A clear and concise description of the bug. + +**How can one reproduce the bug?** +Steps to reproduce the behavior: +1. Go to '...' +2. Click on '....' +3. Scroll down to '....' +4. See error + +**What is the expected behavior?** +A clear and concise description of what you expected to happen. + +**What is your host/environment?** + - OS: [e.g. iOS] + - Version [e.g. 22] + - Plugins + +**Do you have any screenshots?** +If applicable, add screenshots to help explain your problem. + +**Do you have any additional context?** +Add any other context about the problem. diff --git a/.github/ISSUE_TEMPLATE/bug_template.md b/.github/ISSUE_TEMPLATE/bug_template.md deleted file mode 100644 index 8af6ebb5..00000000 --- a/.github/ISSUE_TEMPLATE/bug_template.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -name: 🐛 Bug report -about: Create a report to help us improve -title: "[BUG]" -labels: 'bug, untriaged, Beta' -assignees: '' ---- - -**Describe the bug** -A clear and concise description of what the bug is. - -**To Reproduce** -Steps to reproduce the behavior: -1. Go to '...' -2. Click on '....' -3. Scroll down to '....' -4. See error - -**Expected behavior** -A clear and concise description of what you expected to happen. - -**Plugins** -Please list all plugins currently enabled. - -**Screenshots** -If applicable, add screenshots to help explain your problem. - -**Host/Environment (please complete the following information):** - - OS: [e.g. iOS] - - Version [e.g. 22] - -**Additional context** -Add any other context about the problem here. \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..a8199a10 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,7 @@ +contact_links: + - name: OpenSearch Community Support + url: https://discuss.opendistrocommunity.dev/ + about: Please ask and answer questions here. + - name: AWS/Amazon Security + url: https://aws.amazon.com/security/vulnerability-reporting/ + about: Please report security vulnerabilities here. \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index 2791b808..6198f338 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -1,19 +1,18 @@ --- name: 🎆 Feature request -about: Suggest an idea for this project -title: '' -labels: enhancement +about: Request a feature in this project +title: '[FEATURE]' +labels: 'enhancement, untriaged' assignees: '' --- +**Is your feature request related to a problem?** +A clear and concise description of what the problem is, e.g. _I'm always frustrated when [...]_ -**Is your feature request related to a problem? Please describe.** -A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] - -**Describe the solution you'd like** +**What solution would you like?** A clear and concise description of what you want to happen. -**Describe alternatives you've considered** +**What alternatives have you considered?** A clear and concise description of any alternative solutions or features you've considered. -**Additional context** +**Do you have any additional context?** Add any other context or screenshots about the feature request here. \ No newline at end of file diff --git a/build.gradle b/build.gradle index ac588e81..aa084ccb 100644 --- a/build.gradle +++ b/build.gradle @@ -184,4 +184,4 @@ afterEvaluate { doLast { delete file("$buildDir/distributions/$archiveName") } } } -} +} \ No newline at end of file diff --git a/scripts/build.sh b/scripts/build.sh index 40a856a2..f7ffb095 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -67,12 +67,12 @@ fi [[ "$SNAPSHOT" == "true" ]] && VERSION=$VERSION-SNAPSHOT [ -z "$OUTPUT" ] && OUTPUT=artifacts -./gradlew publishToMavenLocal -Dopensearch.version=$VERSION -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER -./gradlew publishAllPublicationsToStagingRepository -Dopensearch.version=$VERSION -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER -mkdir -p $OUTPUT/maven/org/opensearch -cp -r ./build/local-staging-repo/org/opensearch/. $OUTPUT/maven/org/opensearch - ./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true -Dopensearch.version=$VERSION -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER [ -z "$OUTPUT" ] && OUTPUT=artifacts mkdir -p $OUTPUT/plugins cp ./build/distributions/*.zip $OUTPUT/plugins + +./gradlew publishToMavenLocal -Dopensearch.version=$VERSION -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER +./gradlew publishAllPublicationsToStagingRepository -Dopensearch.version=$VERSION -Dbuild.snapshot=$SNAPSHOT -Dbuild.version_qualifier=$QUALIFIER +mkdir -p $OUTPUT/maven/org/opensearch +cp -r ./build/local-staging-repo/org/opensearch/. $OUTPUT/maven/org/opensearch \ No newline at end of file diff --git a/settings.gradle b/settings.gradle index e70743c2..6bfbc2c5 100644 --- a/settings.gradle +++ b/settings.gradle @@ -9,4 +9,5 @@ include "spi" project(":spi").name = rootProject.name + "-spi" include "sample-extension-plugin" -project(":sample-extension-plugin").name = rootProject.name + "-sample-extension" \ No newline at end of file +project(":sample-extension-plugin").name = rootProject.name + "-sample-extension" +include "zipsmaven" diff --git a/spi/src/main/java/org/opensearch/jobscheduler/spi/LockModel.java b/spi/src/main/java/org/opensearch/jobscheduler/spi/LockModel.java index 2ff8876a..767d6054 100644 --- a/spi/src/main/java/org/opensearch/jobscheduler/spi/LockModel.java +++ b/spi/src/main/java/org/opensearch/jobscheduler/spi/LockModel.java @@ -80,6 +80,8 @@ public LockModel(String jobIndexName, String jobId, Instant lockTime, long lockDurationSeconds, boolean released, long seqNo, long primaryTerm) { this.lockId = jobIndexName + LOCK_ID_DELIMITR + jobId; this.jobIndexName = jobIndexName; + // The jobId parameter does not necessarily need to represent the id of a job scheduler job, as it is being used + // to scope the lock, and could represent any resource. this.jobId = jobId; this.lockTime = lockTime; this.lockDurationSeconds = lockDurationSeconds; diff --git a/spi/src/main/java/org/opensearch/jobscheduler/spi/utils/LockService.java b/spi/src/main/java/org/opensearch/jobscheduler/spi/utils/LockService.java index 435135c2..9b135bb4 100644 --- a/spi/src/main/java/org/opensearch/jobscheduler/spi/utils/LockService.java +++ b/spi/src/main/java/org/opensearch/jobscheduler/spi/utils/LockService.java @@ -106,22 +106,46 @@ public void acquireLock(final ScheduledJobParameter jobParameter, final JobExecutionContext context, ActionListener listener) { final String jobIndexName = context.getJobIndexName(); final String jobId = context.getJobId(); - if (jobParameter.getLockDurationSeconds() == null) { + final long lockDurationSeconds = jobParameter.getLockDurationSeconds(); + acquireLockWithId(jobIndexName, lockDurationSeconds, jobId, listener); + } + + /** + * Attempts to acquire a lock with a specific lock Id. If the lock does not exist it attempts to create the lock document. + * If the Lock document exists, it will try to update and acquire the lock. + * + * @param jobIndexName a non-null job index name. + * @param lockDurationSeconds the amount of time in seconds that the lock should exist + * @param lockId the unique Id for the lock. This should represent the resource that the lock is on, whether it be + * a job, or some other arbitrary resource. If the lockID matches a jobID, then the lock will be deleted + * when the job is deleted. + * @param listener an {@code ActionListener} that has onResponse and onFailure that is used to return the lock if it was acquired + * or else null. Passes {@code IllegalArgumentException} to onFailure if the {@code ScheduledJobParameter} does not + * have {@code LockDurationSeconds}. + */ + public void acquireLockWithId(final String jobIndexName, + final Long lockDurationSeconds, + final String lockId, + ActionListener listener) { + if (lockDurationSeconds == null) { listener.onFailure(new IllegalArgumentException("Job LockDuration should not be null")); + } else if (jobIndexName == null) { + listener.onFailure(new IllegalArgumentException("Job index name should not be null")); + } else if (lockId == null) { + listener.onFailure(new IllegalArgumentException("Lock ID should not be null")); } else { - final long lockDurationSecond = jobParameter.getLockDurationSeconds(); createLockIndex(ActionListener.wrap( created -> { if (created) { try { - findLock(LockModel.generateLockId(jobIndexName, jobId), ActionListener.wrap( + findLock(LockModel.generateLockId(jobIndexName, lockId), ActionListener.wrap( existingLock -> { if (existingLock != null) { if (isLockReleasedOrExpired(existingLock)) { // Lock is expired. Attempt to acquire lock. logger.debug("lock is released or expired: " + existingLock); LockModel updateLock = new LockModel(existingLock, getNow(), - lockDurationSecond, false); + lockDurationSeconds, false); updateLock(updateLock, listener); } else { logger.debug("Lock is NOT released or expired. " + existingLock); @@ -130,8 +154,9 @@ public void acquireLock(final ScheduledJobParameter jobParameter, } } else { // There is no lock object and it is first time. Create new lock. - LockModel tempLock = new LockModel(jobIndexName, jobId, getNow(), - lockDurationSecond, false); + // Note that the lockID will be set to {jobIndexName}-{lockId} + LockModel tempLock = new LockModel(jobIndexName, lockId, getNow(), + lockDurationSeconds, false); logger.debug("Lock does not exist. Creating new lock" + tempLock); createLock(tempLock, listener); } diff --git a/spi/src/test/java/org/opensearch/jobscheduler/spi/utils/LockServiceIT.java b/spi/src/test/java/org/opensearch/jobscheduler/spi/utils/LockServiceIT.java index 48699aa3..20c0c293 100644 --- a/spi/src/test/java/org/opensearch/jobscheduler/spi/utils/LockServiceIT.java +++ b/spi/src/test/java/org/opensearch/jobscheduler/spi/utils/LockServiceIT.java @@ -5,18 +5,18 @@ package org.opensearch.jobscheduler.spi.utils; +import org.junit.Before; +import org.junit.Ignore; +import org.mockito.Mockito; +import org.opensearch.action.ActionListener; +import org.opensearch.cluster.service.ClusterService; +import org.opensearch.common.xcontent.XContentBuilder; import org.opensearch.jobscheduler.spi.JobDocVersion; import org.opensearch.jobscheduler.spi.JobExecutionContext; import org.opensearch.jobscheduler.spi.LockModel; import org.opensearch.jobscheduler.spi.ScheduledJobParameter; import org.opensearch.jobscheduler.spi.schedule.Schedule; -import org.opensearch.action.ActionListener; -import org.opensearch.cluster.service.ClusterService; -import org.opensearch.common.xcontent.XContentBuilder; import org.opensearch.test.OpenSearchIntegTestCase; -import org.junit.Before; -import org.junit.Ignore; -import org.mockito.Mockito; import java.io.IOException; import java.time.Duration; @@ -117,17 +117,56 @@ public void testSanity() throws Exception { latch.await(5L, TimeUnit.SECONDS); } + public void testSanityWithCustomLockID() throws Exception { + String lockID = "sanity_test_lock"; + String uniqSuffix = "_sanity"; + CountDownLatch latch = new CountDownLatch(1); + LockService lockService = new LockService(client(), this.clusterService); + final JobExecutionContext context = new JobExecutionContext(Instant.now(), new JobDocVersion(0, 0, 0), + lockService, JOB_INDEX_NAME + uniqSuffix, JOB_ID + uniqSuffix); + Instant testTime = Instant.now(); + lockService.setTime(testTime); + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( + lock -> { + assertNotNull("Expected to successfully grab lock.", lock); + assertEquals("job_id does not match.", lockID, lock.getJobId()); + assertEquals("job_index_name does not match.", JOB_INDEX_NAME + uniqSuffix, lock.getJobIndexName()); + assertEquals("lock_id does not match.", lock.getJobIndexName() + "-" + lockID, lock.getLockId()); + assertEquals("lock_duration_seconds does not match.", LOCK_DURATION_SECONDS, lock.getLockDurationSeconds()); + assertEquals("lock_time does not match.", testTime.getEpochSecond(), lock.getLockTime().getEpochSecond()); + assertFalse("Lock should not be released.", lock.isReleased()); + assertFalse("Lock should not expire.", lock.isExpired()); + lockService.release(lock, ActionListener.wrap( + released -> { + assertTrue("Failed to release lock.", released); + lockService.deleteLock(lock.getLockId(), ActionListener.wrap( + deleted -> { + assertTrue("Failed to delete lock.", deleted); + latch.countDown(); + }, + exception -> fail(exception.getMessage()) + )); + }, + exception -> fail(exception.getMessage()) + )); + }, + exception -> fail(exception.getMessage()) + )); + latch.await(5L, TimeUnit.SECONDS); + } + public void testSecondAcquireLockFail() throws Exception { String uniqSuffix = "_second_acquire"; + String lockID = randomAlphaOfLengthBetween(6, 15); CountDownLatch latch = new CountDownLatch(1); LockService lockService = new LockService(client(), this.clusterService); final JobExecutionContext context = new JobExecutionContext(Instant.now(), new JobDocVersion(0, 0, 0), lockService, JOB_INDEX_NAME + uniqSuffix, JOB_ID + uniqSuffix); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock -> { assertNotNull("Expected to successfully grab lock", lock); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock2 -> { assertNull("Expected to failed to get lock.", lock2); lockService.release(lock, ActionListener.wrap( @@ -154,18 +193,19 @@ public void testSecondAcquireLockFail() throws Exception { public void testLockReleasedAndAcquired() throws Exception { String uniqSuffix = "_lock_release+acquire"; + String lockID = randomAlphaOfLengthBetween(6, 15); CountDownLatch latch = new CountDownLatch(1); LockService lockService = new LockService(client(), this.clusterService); final JobExecutionContext context = new JobExecutionContext(Instant.now(), new JobDocVersion(0, 0, 0), lockService, JOB_INDEX_NAME + uniqSuffix, JOB_ID + uniqSuffix); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock -> { assertNotNull("Expected to successfully grab lock", lock); lockService.release(lock, ActionListener.wrap( released -> { assertTrue("Failed to release lock.", released); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock2 -> { assertNotNull("Expected to successfully grab lock2", lock2); lockService.release(lock2, ActionListener.wrap( @@ -195,6 +235,7 @@ public void testLockReleasedAndAcquired() throws Exception { public void testLockExpired() throws Exception { String uniqSuffix = "_lock_expire"; + String lockID = randomAlphaOfLengthBetween(6, 15); CountDownLatch latch = new CountDownLatch(1); LockService lockService = new LockService(client(), this.clusterService); // Set lock time in the past. @@ -202,13 +243,12 @@ public void testLockExpired() throws Exception { final JobExecutionContext context = new JobExecutionContext(Instant.now(), new JobDocVersion(0, 0, 0), lockService, JOB_INDEX_NAME + uniqSuffix, JOB_ID + uniqSuffix); - - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock -> { assertNotNull("Expected to successfully grab lock", lock); // Set lock back to current time to make the lock expire. lockService.setTime(null); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock2 -> { assertNotNull("Expected to successfully grab lock", lock2); lockService.release(lock, ActionListener.wrap( @@ -280,12 +320,12 @@ public void testDeleteNonExistingLock() throws Exception { @Ignore public void testMultiThreadCreateLock() throws Exception { String uniqSuffix = "_multi_thread_create"; + String lockID = randomAlphaOfLengthBetween(6, 15); CountDownLatch latch = new CountDownLatch(1); final LockService lockService = new LockService(client(), this.clusterService); final JobExecutionContext context = new JobExecutionContext(Instant.now(), new JobDocVersion(0, 0, 0), lockService, JOB_INDEX_NAME + uniqSuffix, JOB_ID + uniqSuffix); - lockService.createLockIndex(ActionListener.wrap( created -> { if (created) { @@ -293,7 +333,7 @@ public void testMultiThreadCreateLock() throws Exception { final AtomicReference lockModelAtomicReference = new AtomicReference<>(null); Callable callable = () -> { CountDownLatch callableLatch = new CountDownLatch(1); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock -> { if (lock != null) { lockModelAtomicReference.set(lock); @@ -355,6 +395,7 @@ public void testMultiThreadCreateLock() throws Exception { @Ignore public void testMultiThreadAcquireLock() throws Exception { String uniqSuffix = "_multi_thread_acquire"; + String lockID = randomAlphaOfLengthBetween(6, 15); CountDownLatch latch = new CountDownLatch(1); final LockService lockService = new LockService(client(), this.clusterService); final JobExecutionContext context = new JobExecutionContext(Instant.now(), new JobDocVersion(0, 0, 0), @@ -365,7 +406,7 @@ public void testMultiThreadAcquireLock() throws Exception { if (created) { // Set lock time in the past. lockService.setTime(Instant.now().minus(Duration.ofSeconds(LOCK_DURATION_SECONDS + LOCK_DURATION_SECONDS))); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( createdLock -> { assertNotNull(createdLock); // Set lock back to current time to make the lock expire. @@ -375,7 +416,7 @@ public void testMultiThreadAcquireLock() throws Exception { final AtomicReference lockModelAtomicReference = new AtomicReference<>(null); Callable callable = () -> { CountDownLatch callableLatch = new CountDownLatch(1); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock -> { if (lock != null) { lockModelAtomicReference.set(lock); @@ -430,12 +471,13 @@ public void testMultiThreadAcquireLock() throws Exception { public void testRenewLock() throws Exception { String uniqSuffix = "_lock_renew"; + String lockID = randomAlphaOfLengthBetween(6, 15); CountDownLatch latch = new CountDownLatch(1); LockService lockService = new LockService(client(), this.clusterService); final JobExecutionContext context = new JobExecutionContext(Instant.now(), new JobDocVersion(0, 0, 0), lockService, JOB_INDEX_NAME + uniqSuffix, JOB_ID + uniqSuffix); - lockService.acquireLock(TEST_SCHEDULED_JOB_PARAM, context, ActionListener.wrap( + lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap( lock -> { assertNotNull("Expected to successfully grab lock", lock); // Set the time of LockService (the 'lockTime' of acquired locks) to a fixed time. diff --git a/zipsmaven/build.gradle b/zipsmaven/build.gradle new file mode 100644 index 00000000..12386b1e --- /dev/null +++ b/zipsmaven/build.gradle @@ -0,0 +1,102 @@ +plugins { + id 'maven-publish' +} + +ext { + isSnapshot = "true" == System.getProperty("build.snapshot", "true") + opensearch_version = System.getProperty("opensearch.version", "2.0.0-alpha1-SNAPSHOT") + buildVersionQualifier = System.getProperty("build.version_qualifier", "alpha1") + // 2.0.0-alpha1-SNAPSHOT -> 2.0.0.0-alpha1-SNAPSHOT + version_tokens = opensearch_version.tokenize('-') + opensearch_build = version_tokens[0] + '.0' + if (buildVersionQualifier) { + opensearch_build += "-${buildVersionQualifier}" + opensearch_build_nosnapshot = opensearch_build + } + if (isSnapshot) { + opensearch_build += "-SNAPSHOT" + } +} + + +allprojects { + // Default to the apache license + project.ext.licenseName = 'The Apache Software License, Version 2.0' + project.ext.licenseUrl = 'http://www.apache.org/licenses/LICENSE-2.0.txt' + publishing { + repositories { + maven { + name = 'staging' + url = "${rootProject.buildDir}/local-staging-repo" + } + } + publications { + // add license information to generated poms + all { + pom.withXml { XmlProvider xml -> + Node node = xml.asNode() + node.appendNode('inceptionYear', '2021') + + Node license = node.appendNode('licenses').appendNode('license') + license.appendNode('name', project.licenseName) + license.appendNode('url', project.licenseUrl) + + Node developer = node.appendNode('developers').appendNode('developer') + developer.appendNode('name', 'OpenSearch') + developer.appendNode('url', 'https://github.com/opensearch-project/job-scheduler') + } + } + } + } +} + +def group = "org.opensearch.plugin" +def pluginversion = opensearch_build +def component = "opensearch-job-scheduler" +def description = "OpenSearch Job Scheduler Zip" +def zipFile = "${component}-${pluginversion}.zip" +def zipArtifact = "${rootProject.buildDir}/distributions/${zipFile}" + + +publishing { + publications { + maven(MavenPublication) { + //Add ext.publish = 'skip' to skip publishing skip to maven staging repo + groupId = "${group}" + artifactId = "${component}" + version = "${pluginversion}" + // Add extension, to get zip as in pom file + artifact(zipArtifact) { + extension 'zip' + } + pom { + name = "${component}" + description = "${description}" + } + } + } +} + +task printZipsPublishProperty { + doLast { + tasks.withType(PublishToMavenRepository).all { publishTask -> + if (publication.hasProperty('publish')) { + println(publication.publish) + }else { + println("null") + } + } + } +} + +afterEvaluate { + tasks.withType(PublishToMavenRepository).all { publishTask -> + publishTask.onlyIf { task -> + if (publication.hasProperty('publish') && publication.publish == 'skip') { + return false + } + return true + } + } +} + From 55ffa6731660fd14501be8d0f336b9bebcd51166 Mon Sep 17 00:00:00 2001 From: pgodithi Date: Mon, 11 Apr 2022 08:57:21 -0700 Subject: [PATCH 2/9] Adding better gradle project dir Signed-off-by: pgodithi --- {zipsmaven => maven/plugin-zips}/build.gradle | 0 settings.gradle | 5 ++++- 2 files changed, 4 insertions(+), 1 deletion(-) rename {zipsmaven => maven/plugin-zips}/build.gradle (100%) diff --git a/zipsmaven/build.gradle b/maven/plugin-zips/build.gradle similarity index 100% rename from zipsmaven/build.gradle rename to maven/plugin-zips/build.gradle diff --git a/settings.gradle b/settings.gradle index 6bfbc2c5..f4ba8b93 100644 --- a/settings.gradle +++ b/settings.gradle @@ -10,4 +10,7 @@ project(":spi").name = rootProject.name + "-spi" include "sample-extension-plugin" project(":sample-extension-plugin").name = rootProject.name + "-sample-extension" -include "zipsmaven" + +//Adding pluginZips project under maven folder, this is to publish generated plugin zips to maven repo +include 'pluginZips' +project(':pluginZips').projectDir = file('maven/plugin-zips') From 1b03814a1239202e231c7a53008634b6fb548de5 Mon Sep 17 00:00:00 2001 From: pgodithi Date: Mon, 11 Apr 2022 09:17:02 -0700 Subject: [PATCH 3/9] Added info in readme file Signed-off-by: pgodithi --- DEVELOPER_GUIDE.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index b19599e5..5d83985a 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -53,6 +53,16 @@ opensearch-plugin install file:///path/to/target/releases/opensearch-job-schedul ``` to install the JobScheduler plugin to your OpenSearch. +## To Publish generated plugin zips to maven repo +The generated plugin zip `opensearch-job-scheduler-.zip` after the build, inside the folder `build/distributions` can be published to maven repo. This can be done by including the project `pluginZips` inside `settings.gradle` file. Once added the generated plugin zip will be published to maven repo with following maven coordinates +``` + org.opensearch.plugin + opensearch-security + + zip +``` +To skip the publish, while this sub-project is still included, pass the flag as ` ext.publish = 'skip'` to file `build.gradle` inside `maven/plugin-zips/` folder. + ## Develop a plugin that extends JobScheduler JobScheduler plugin provides a SPI for other plugins to implement. Essentially, you need to 1. Define your *JobParameter* type by implementing `ScheduledJobParameter` interface From f4a9b64ebfcf3f8cc210063c9f27bde0b9c55e0a Mon Sep 17 00:00:00 2001 From: pgodithi Date: Mon, 11 Apr 2022 09:18:25 -0700 Subject: [PATCH 4/9] Added info in readme file Signed-off-by: pgodithi --- DEVELOPER_GUIDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 5d83985a..0e4b7a60 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -6,6 +6,7 @@ - [Build](#build) - [Building from the command line](#building-from-the-command-line) - [Debugging](#debugging) + - [Publish zips to maven](#to-publish-generated-plugin-zips-to-maven-repo) - [Using IntelliJ IDEA](#using-intellij-idea) - [Submitting Changes](#submitting-changes) From 44564642f1d5aab766c972f6e87080da0d54aa60 Mon Sep 17 00:00:00 2001 From: pgodithi Date: Mon, 11 Apr 2022 09:25:15 -0700 Subject: [PATCH 5/9] Added info in readme file Signed-off-by: pgodithi --- maven/plugin-zips/build.gradle | 1 + 1 file changed, 1 insertion(+) diff --git a/maven/plugin-zips/build.gradle b/maven/plugin-zips/build.gradle index 12386b1e..35caa4fb 100644 --- a/maven/plugin-zips/build.gradle +++ b/maven/plugin-zips/build.gradle @@ -63,6 +63,7 @@ publishing { maven(MavenPublication) { //Add ext.publish = 'skip' to skip publishing skip to maven staging repo groupId = "${group}" + ext.publish = 'skip' artifactId = "${component}" version = "${pluginversion}" // Add extension, to get zip as in pom file From 87270d413ec7bbebb1eb1cbf5e375e15db94e2c9 Mon Sep 17 00:00:00 2001 From: pgodithi Date: Mon, 11 Apr 2022 09:26:14 -0700 Subject: [PATCH 6/9] Added info in readme file Signed-off-by: pgodithi --- maven/plugin-zips/build.gradle | 1 - 1 file changed, 1 deletion(-) diff --git a/maven/plugin-zips/build.gradle b/maven/plugin-zips/build.gradle index 35caa4fb..12386b1e 100644 --- a/maven/plugin-zips/build.gradle +++ b/maven/plugin-zips/build.gradle @@ -63,7 +63,6 @@ publishing { maven(MavenPublication) { //Add ext.publish = 'skip' to skip publishing skip to maven staging repo groupId = "${group}" - ext.publish = 'skip' artifactId = "${component}" version = "${pluginversion}" // Add extension, to get zip as in pom file From d2acc48df7add48a494a99a9555c76e710037da1 Mon Sep 17 00:00:00 2001 From: pgodithi Date: Mon, 11 Apr 2022 10:18:26 -0700 Subject: [PATCH 7/9] fixed typo in readme file Signed-off-by: pgodithi --- DEVELOPER_GUIDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 0e4b7a60..a979ae63 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -58,7 +58,7 @@ to install the JobScheduler plugin to your OpenSearch. The generated plugin zip `opensearch-job-scheduler-.zip` after the build, inside the folder `build/distributions` can be published to maven repo. This can be done by including the project `pluginZips` inside `settings.gradle` file. Once added the generated plugin zip will be published to maven repo with following maven coordinates ``` org.opensearch.plugin - opensearch-security + opensearch-job-scheduler zip ``` From d34c67c9e58a6a695157448ad98d6721e6122d42 Mon Sep 17 00:00:00 2001 From: pgodithi Date: Mon, 11 Apr 2022 10:20:13 -0700 Subject: [PATCH 8/9] test skip Signed-off-by: pgodithi --- maven/plugin-zips/build.gradle | 1 + 1 file changed, 1 insertion(+) diff --git a/maven/plugin-zips/build.gradle b/maven/plugin-zips/build.gradle index 12386b1e..35caa4fb 100644 --- a/maven/plugin-zips/build.gradle +++ b/maven/plugin-zips/build.gradle @@ -63,6 +63,7 @@ publishing { maven(MavenPublication) { //Add ext.publish = 'skip' to skip publishing skip to maven staging repo groupId = "${group}" + ext.publish = 'skip' artifactId = "${component}" version = "${pluginversion}" // Add extension, to get zip as in pom file From 2093e2415eee02cddc957d06c8c261144a8b3148 Mon Sep 17 00:00:00 2001 From: pgodithi Date: Mon, 11 Apr 2022 10:23:18 -0700 Subject: [PATCH 9/9] test skip Signed-off-by: pgodithi --- maven/plugin-zips/build.gradle | 1 - 1 file changed, 1 deletion(-) diff --git a/maven/plugin-zips/build.gradle b/maven/plugin-zips/build.gradle index 35caa4fb..12386b1e 100644 --- a/maven/plugin-zips/build.gradle +++ b/maven/plugin-zips/build.gradle @@ -63,7 +63,6 @@ publishing { maven(MavenPublication) { //Add ext.publish = 'skip' to skip publishing skip to maven staging repo groupId = "${group}" - ext.publish = 'skip' artifactId = "${component}" version = "${pluginversion}" // Add extension, to get zip as in pom file