Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds documentation for Job Scheduler plugin #2242

Merged
merged 25 commits into from
Jan 5, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
32cf9d8
Adds documentation for Job Scheduler plugin
ariamarble Dec 15, 2022
47a3c3a
Merge branch 'main' of https://github.com/opensearch-project/document…
ariamarble Dec 16, 2022
cb72861
made tech changes
ariamarble Dec 27, 2022
a7d9fa9
add to dropdown
ariamarble Dec 27, 2022
66509a5
made suggested changes
ariamarble Dec 27, 2022
e29fc37
Update _job-scheduler-plugin/build-plugin.md
ariamarble Dec 28, 2022
5bc4f7c
made suggested changes
ariamarble Dec 28, 2022
9e09e3a
index page re-write
ariamarble Dec 28, 2022
c85cb51
wording changes. moving to single page
ariamarble Dec 28, 2022
19a4ee2
rephrasing and suggested changes
ariamarble Dec 30, 2022
2d1c02a
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
cd2ca22
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
fcb7051
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
0517522
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
9eb7e27
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
ff4d155
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
f30c3a2
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
0059ffd
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
cdf12ff
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
164f7d1
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
c073dc3
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
0a51d69
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
5c55ffd
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
efad68b
Update _job-scheduler-plugin/index.md
ariamarble Jan 5, 2023
37dc92c
editorial
ariamarble Jan 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ collections:
notifications-plugin:
permalink: /:collection/:path/
output: true
job-scheduler-plugin:
permalink: /:collection/:path/
output: true
clients:
permalink: /:collection/:path/
output: true
Expand Down Expand Up @@ -136,6 +139,9 @@ just_the_docs:
notifications-plugin:
name: Notifications plugin
nav_fold: true
job-scheduler-plugin:
name: Job Scheduler plugin
nav_fold: true
clients:
name: Clients
nav_fold: true
Expand Down
130 changes: 130 additions & 0 deletions _job-scheduler-plugin/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
layout: default
title: Job Scheduler
nav_order: 1
has_children: false
has_toc: false
---

# Job Scheduler

The OpenSearch Job Scheduler plugin provides a framework that can be used to build schedules for common tasks performed on your cluster. You use the job scheduler’s Service Provider Interface (SPI) to define schedules for cluster management tasks such as taking snapshots, managing your data’s lifecycle, and running periodic jobs. Job Scheduler has a sweeper that listens for updated events on the OpenSearch cluster, and a scheduler that manages when jobs run.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

You can install the Job Scheduler plugin by following the standard [OpenSearch plugin installation]({{site.url}}{{site.baseurl}}/install-and-configure/install-opensearch/plugins/) process. The sample-extension-plugin example provided in the [Job Scheduler Github repo](https://github.com/opensearch-project/job-scheduler) provides a complete example of utilizing Job Scheduler when building a plugin. To define schedules, you build a plugin that implements the interfaces provided in the job scheduler library. You can schedule jobs by specifying an interval, or you can use a Unix cron expression such as `0 12 * * ?` which runs at noon every day, to define a more flexible schedule for which to execute a job.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

## Building a plugin for Job Scheduler

OpenSearch plugin developers can extend the Job Scheduler plugin to schedule jobs to perform on the cluster. Jobs you can schedule include running aggregation queries against raw data and saving the aggregated data into a new index every hour, or continuing to monitor the shard allocation by calling the OpenSearch API and then posting the output to a webhook.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

For examples of building a plugin that utilizes the Job Scheduler plugin, see the Job Scheduler [README](https://github.com/opensearch-project/job-scheduler/blob/main/README.md).
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

## Defining an endpoint

You can configure your plugin's API endpoint by referencing the [example](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleExtensionRestHandler.java) `SampleExtensionRestHandler.java` file. Set the endpoint URL that your plugin will expose with `WATCH_INDEX_URI`:

```java
public class SampleExtensionRestHandler extends BaseRestHandler {
public static final String WATCH_INDEX_URI = "/_plugins/scheduler_sample/watch";
```

You can define the job configuration by [extending](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobParameter.java) `ScheduledJobParameter`. You can also define the fields used by your plugin like `indexToWatch` as shown in the [example](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobParameter.java) `SampleJobParameter` file. This job configuration will be saved as a document in an index defined by you as shown in this [example](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleExtensionPlugin.java#L54).
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

## Parameters configuration
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

You can configure your plugin's parameters by referencing and modifying the [example](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobParameter.java) `SampleJobParameter.java` file to fit your needs:
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

```java
/**
* A sample job parameter.
* <p>
* It adds an additional "indexToWatch" field to {@link ScheduledJobParameter}, which stores the index
* the job runner will watch.
*/
public class SampleJobParameter implements ScheduledJobParameter {
public static final String NAME_FIELD = "name";
public static final String ENABLED_FILED = "enabled";
public static final String LAST_UPDATE_TIME_FIELD = "last_update_time";
public static final String LAST_UPDATE_TIME_FIELD_READABLE = "last_update_time_field";
public static final String SCHEDULE_FIELD = "schedule";
public static final String ENABLED_TIME_FILED = "enabled_time";
public static final String ENABLED_TIME_FILED_READABLE = "enabled_time_field";
public static final String INDEX_NAME_FIELD = "index_name_to_watch";
public static final String LOCK_DURATION_SECONDS = "lock_duration_seconds";
public static final String JITTER = "jitter";

private String jobName;
private Instant lastUpdateTime;
private Instant enabledTime;
private boolean isEnabled;
private Schedule schedule;
private String indexToWatch;
private Long lockDurationSeconds;
private Double jitter;
```

Next, configure the request parameters you would like your plugin to use with Job Scheduler. These will be based on the variables that are declared when configuring your plugin. The sample below shows the request parameters you set when building your plugin:
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

```java
public SampleJobParameter(String id, String name, String indexToWatch, Schedule schedule, Long lockDurationSeconds, Double jitter) {
this.jobName = name;
this.indexToWatch = indexToWatch;
this.schedule = schedule;

Instant now = Instant.now();
this.isEnabled = true;
this.enabledTime = now;
this.lastUpdateTime = now;
this.lockDurationSeconds = lockDurationSeconds;
this.jitter = jitter;
}

@Override
public String getName() {
return this.jobName;
}

@Override
public Instant getLastUpdateTime() {
return this.lastUpdateTime;
}

@Override
public Instant getEnabledTime() {
return this.enabledTime;
}

@Override
public Schedule getSchedule() {
return this.schedule;
}

@Override
public boolean isEnabled() {
return this.isEnabled;
}

@Override
public Long getLockDurationSeconds() {
return this.lockDurationSeconds;
}

@Override public Double getJitter() {
return jitter;
}
```

The following table describes the request parameters configured in the previous sample. All the request parameters shown are required.
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

| Field | Data type | Description |
:--- | :--- | :---
| getName | String | Returns the name of the job. |
| getLastUpdateTime | Time unit | Returns the last time the job ran. |
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
| getEnabledTime | Time unit | Returns the time that the job was enabled. |
| getSchedule | Unix cron | Returns the schedule that the job runs on formatted in Unix cron syntax. |
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
| isEnabled | Boolean | Whether or not the job is enabled. |
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
| getLockDurationSeconds | Integer | Returns the duration of time that the job is locked. |
ariamarble marked this conversation as resolved.
Show resolved Hide resolved
| getJitter | Integer | Returns the defined jitter value. |

The logic used by your job should be defined by a class extended from `ScheduledJobRunner` in the `SampleJobParameter.java` sample file, such as `SampleJobRunner`. While the job is running, there is a locking mechanism you can use to prevent other nodes from running the same job. [Acquire](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobRunner.java#L96) the lock. Then, make sure to release the lock before the [job finishes](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobRunner.java#L116).
ariamarble marked this conversation as resolved.
Show resolved Hide resolved

For more information, see the Job Scheduler [sample extention](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobParameter.java) directory in the [Job Scheduler Github repository](https://github.com/opensearch-project/job-scheduler).
ariamarble marked this conversation as resolved.
Show resolved Hide resolved