[Feature/extensions] Determining whether Job Scheduler should be in core or an extension

[ryanbogan]

Since the extensibility team is taking over ownership for job-scheduler, we need to determine how it will be integrated with extensions moving forward.

Research tasks:

• Consolidate pros/cons for pulling Job Scheduler into core
• Research : How does Job Scheduler communicate and interact with OpenSearch
• Research : What features / APIs does Job Scheduler provide for plugins and for future extension developers
• Research : What plugins currently have a dependency on Job Scheduler and how do these plugins currenltly use Job Scheduler features
• Research : Should Job Scheduler maintain the current Service Provider Interface design as an extension?
• Determine how much work would be required to make Job Scheduler an extension.
• Determine if Job Scheduler should be pulled into core as a module, relocated to a native plugin, or refactored into an independent extension

Duplicate of opensearch-project/job-scheduler#147

[ryanbogan]

Possible Routes :

• Job Scheduler as a Module

  • Multiple plugins already have a dependency on job-scheduler (one to many relationship)
  • Plugins / extensions can directly extend Job Scheduler from core without taking any additional dependencies (simpler dependency management)
  • Release / version upgrades would be automated as part of OpenSearch core
  • Current plugin architecture does not have a demand for pulling Job Scheduler features into core, however, for extensions, this would provide a generalized solution (distributed job scheduling) for future extension developers, which would ultimately help them create new features, which aligns with the end-goal of extensibility: making it easier for people to extend OpenSearch’s functionality

• Job Scheduler as an Extension

  • Modifying Job Scheduler as an extension is possible, however it would require removing the Service Provider Interface through which Job Scheduler currently communicates with plugins. This would directly impact all the dependent plugins, such as Anomaly Detection, Index Service Management, and Reporting. Another issue that would arise from this decision would be extension-to-extension communication. The transformation of Job Scheduler to an extension would require adding additional support for the extension points that are not currently complete. There are six total extension points present in the Job Scheduler repository. However, after exploration, one of these extension points is not currently used for any of the dependent plugins. Additionally, one of the remaining extensions points, createComponents, has already been implemented for the anomaly detection extension. This leaves the functionality for four extension points to be repurposed for the extension framework: getSettings, onIndexModule, loadExtensions, and getNamedXContent. All dependent plugins would be required to modify their code base to match the changes made to the Job Scheduler repository.

• Job Scheduler as a Native Plugin

  • Implementing Job Scheduler as a native plugin will have most of the same benefits as making it a module. The main difference is that installation will not be mandated, but it will be much simpler for the user to install the plugin than the current method. However, this will still be an extra step for users of any dependent plugins.

• Job Scheduler as is:

  • No mandated installation
  • Release / version upgrades do not take much time, would remain the same
  • Job Scheduler is not directly related to searching and indexing, keeps core related to the essentials of OpenSearch

Summary / initial thoughts :

Our recommendation is to move job scheduler into core as a native plugin. This course of action would make it easier for contributors to extend the core functionality of OpenSearch. Previous discussions have pointed out that the functionality provided by Job Scheduler is not necessary for all deployments, thus mandating installation would only consume resources and not add any inherent value. Therefore, the native plugin design being superior to the module design for Job Scheduler. Since Job Scheduler already has a one-to-many relationship with other plugins, simplifying dependency management provides enough justification for pulling it into core. Additionally, communication between Job Scheduler and other extensions must go through core in order to utilize the REST API framework, so implementing Job Scheduler as an extension would not be as efficient.

[ryanbogan]

@dblock @kartg @nknize @Bukhtawar @praveensameneni @owaiskazi19 @anasalkouz @reta @bbarani @mch2 @andrross @peternied @saratvemulapalli @anirudha

Many of you were involved in previous discussions about the location of Job Scheduler, and we would appreciate your input on this decision.

[peternied]

Our recommendation is to move job scheduler into core as a native plugin.

💯

Additional there are many reoccurring processes inside of OpenSearch itself, having a single mechanism available for defining and scheduling those actions would be of great long term value.

[dblock]

I think the benefits of making job scheduler a native plugin are clear.

[kartg]

I agree with the recommendation and the comments above to model job scheduler as a native plugin.

That said, I did have a follow-up question around this:

Implementing Job Scheduler as a native plugin will have most of the same benefits as making it a module.... However, this will still be an extra step for users of any dependent plugins.

Can we improve this experience so that the onus is on the upstream plugin rather than the user to install job-scheduler? It seems like there's an opportunity here to improve our guardrails so that users aren't expected to know that job-scheduler is a pre-requisite for certain plugins. Such a mechanism would also prevent users from accidentally uninstalling job-scheduler if a plugin is using it.

[ryanbogan]

Can we improve this experience so that the onus is on the upstream plugin rather than the user to install job-scheduler? It seems like there's an opportunity here to improve our guardrails so that users aren't expected to know that job-scheduler is a pre-requisite for certain plugins. Such a mechanism would also prevent users from accidentally uninstalling job-scheduler if a plugin is using it.

We can definitely look into that.

[minalsha]

I agree with the recommendation and the comments above to model job scheduler as a native plugin.

That said, I did have a follow-up question around this:

Implementing Job Scheduler as a native plugin will have most of the same benefits as making it a module.... However, this will still be an extra step for users of any dependent plugins.

Can we improve this experience so that the onus is on the upstream plugin rather than the user to install job-scheduler? It seems like there's an opportunity here to improve our guardrails so that users aren't expected to know that job-scheduler is a pre-requisite for certain plugins. Such a mechanism would also prevent users from accidentally uninstalling job-scheduler if a plugin is using it.

@kartg this is exactly what we want to do in Extensions world as part of extensibility project.

[saratvemulapalli]

+1 I like the idea of putting getting it in as a native plugin, it really doesn't mandate the installation but if customers need it, its simple to install.
As a follow up, we should take the opportunity to make the interfaces generic enough which helps the community.

Can we improve this experience so that the onus is on the upstream plugin rather than the user to install job-scheduler? It seems like there's an opportunity here to improve our guardrails so that users aren't expected to know that job-scheduler is a pre-requisite for certain plugins. Such a mechanism would also prevent users from accidentally uninstalling job-scheduler if a plugin is using it.

@kartg Our plugin installation tool ./bin/opensearch-plugin already has these guardrails built in[1]. It makes sure a dependent plugin is installed and cannot be removed unless all extended plugins are removed.

A sample run on opensearch 2.2.1 distribution:

./bin/opensearch-plugin remove opensearch-job-scheduler
ERROR: plugin [opensearch-job-scheduler] cannot be removed because it is extended by other plugins: [opensearch-reports-scheduler, opensearch-anomaly-detection, opensearch-index-management]

[1]

OpenSearch/distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/RemovePluginCommand.java

Lines 110 to 116 in 54364a5

	if (usedBy.isEmpty() == false) {
	throw new UserException(
	PLUGIN_STILL_USED,
	"plugin [" + pluginName + "] cannot be removed" + " because it is extended by other plugins: " + usedBy
	);
	}

[sean-zheng-amazon]

I'd also vote native plugin for less dependency and the flexibility on installation, and there seems to be no real downside.

[praveensameneni]

Glad to see the issue re-surface from the previous issue/discussion
I think this is a good move.

[nknize]

I'm actually quite confused how this is a different issue than opensearch-project/job-scheduler#147 where there was strong opposition to moving it to the core.

Tangentially, it looks like there is an unresolved question on whether it should be a plugin or module. Should min distribution install the job scheduler by default (module) if no other plugin or module is using it? Separately, the opensearch distribution can install it as part of the bundle since Index Management, Anomaly Detection, Reporting, and Alerting take it as a dependency.

If extensions is picking this up, is refactoring to the core necessary as a stepping stone to "extension-ifying" job scheduler?

[ryanbogan]

I'm actually quite confused how this is a different issue than opensearch-project/job-scheduler#147 where there was strong opposition to moving it to the core.

Tangentially, it looks like there is an unresolved question on whether it should be a plugin or module. Should min distribution install the job scheduler by default (module) if no other plugin or module is using it? Separately, the opensearch distribution can install it as part of the bundle since Index Management, Anomaly Detection, Reporting, and Alerting take it as a dependency.

If extensions is picking this up, is refactoring to the core necessary as a stepping stone to "extension-ifying" job scheduler?

From what I understand, the main opposition to moving Job Scheduler in previous discussions was due to the mandated installation required by making Job Scheduler a module. Our recommendation is to make Job Scheduler a native plugin, which does not require installation, since not every user needs Job Scheduler. However, our proposal would make it easier for the user to install Job Scheduler. Another justification for moving to core is the 1-Many relationship between Job Scheduler and IM, AD, Reporting, and Alerting.

The refactor to core is not absolutely necessary in order to function with extensions, but it is a major improvement. The communication framework that we are building for extensions uses many of the API's found in core, which would simplify communication between Job Scheduler and the plugins/extensions that depend on it.

[nknize]

From what I understand, the main opposition to moving Job Scheduler in previous discussions was due to the mandated installation required by making Job Scheduler a module.

I see. That wasn't my take of the original issue so maybe I misinterpreted the original feedback. Moving job-scheduler to core (regardless of plugin vs module) has been a long running controversial issue because no min-distro functionality uses it, only bundled plugins that depend on it. Do not take this feedback as my veto to refactor job-scheduler to core. Having it as an optional plugin I think is a net positive, minimally-disruptive, change. There are a few things, though, I think we should sort out:

1. It's odd we opened a separate issue on the exact same topic and feedback is 180 (perhaps minds changed after sleeping a while). We should've continued the discussion on the original issue, clarifying intent, and then transferred to this repo once a consensus decision was made. Let's make a note of this for the future scenarios.
2. Are we setting a precedent that a future external plugin should move to the core if there is a one-to-many relationship with external plugins but a one-to-zero relationship with internal functionality? 🤔 Is that a precedent we want to set? Or should we think differently about how external plugin dependencies work? Perhaps instead of a lone job-scheduler plugin there's an optional extensions plugin that adds the core APIs needed (think abstract and interface contracts instead of concrete implementations) for external extensions but nothing else in core uses it?
3. The communication framework that we are building for extensions uses many of the API's found in core.... But I think most of those API's are also used by other features in core. This is a scenario where we're adding APIs only for external plugins. Maybe this is a stronger argument for a single extensions plugin where those add-on API contracts that are not used in the core can live instead of refactoring an API plugin to core on a case by case basis?

Update: if we do decide to rote refactor job scheduler to plugins, then we should do this in #2608 so @dbbaughe get's primary authorship credit since he largely took care of everything. We just need to move it to plugins instead of modules.

[minalsha]

@anirudha could you please share your input here on this proposal as well since Reporting plugin has a dependency on JS plugin?

[ryanbogan]

Based on the feedback provided, we have decided to proceed with moving Job Scheduler to a native plugin.

[kavilla]

@ryanbogan I am curious what the feedback provided that made the decision final. Would you be able to provide that insight or link to that convo? Seems like some of questions by @nknize were still open before proceeding.

I think it would be good to be have a definition what belongs in core and what doesn't. It would probably be important to have the same alignment with OpenSearch Dashboards.

[ryanbogan]

@kavilla @nknize said offline that he did not have any concerns about moving Job Scheduler to a native plugin and agreed with our decision. The questions posed above were for further discussion about what belongs in core going forward.