Onboard components that were part of a previous release

[peternied]

We can modify our existing manifest updating workflow to check for scan for component versions, update the manifest for components that are ready, and then the pull request part of workflow.

See manifest docs
See the github workflow, versions.yml
See update method input_manifest.yml

Concerns

Order of components required for building the distribution

Acceptance Criteria

Create a POC to see if adding all the "ready" components works fine.

[peternied]

A couple of callouts:

• This workflow is executed once every 24 hours, if we wanted to catch availability of new manifests we should do this more frequently
• The job takes ~36 minutes to run and its only building scanning opensearch & opensearch-dashboards today. There might be issues running this job in a timely manner when componets are also being built to check the version number, we might need an alternative to the build, or a way to parallelize it.
• If components don't work with gradle this method doesn't work

[prudhvigodithi]

Hey @peternied, from I understood, you concern was beyond just version increment, I have proposed a solution for version increment, this should take care of creating version increment PR's for the plugin. But your point was about to update the manifest file, please correct me if i'm wrong.
As today the order of plugins is manifest varies for each phase
for 1.x series,
there is a still dependency of zip files for testing example index-management, dashboards-reports depends on job-scheduler new version zip, for this we have to start the distribution with build components core + common-utils + job-scheduler.
sql: depends on ml-commons add ml-commons zip
For both 1.x and 2.x series ml-commons: depends on docker container with core + common-utils + job-scheduler. + performance-analyzer.
security-dashboards-plugin uses security
anomaly-detection uses job-scheduler https://github.com/opensearch-project/anomaly-detection/blob/main/DEVELOPER_GUIDE.md

Possibly we have to address this as Phase 2 (Phase 1 would be auto create the version increment PR).
manage the manifest flow and steps to build in an order.

Thoughts ? @dblock @peternied @bbarani @peterzhuamazon

[peternied]

I'm not sure what you mean by phases 1/2 but I am all onboard for more processes that make the distribution with less effort from OpenSearch-Project teams.

One element that might need to be considered is how a dashboard plugin says it is dependent on a back-end component to pick on Security/Security Dashboard Plugin. Maybe there is a little work that can be done in the manifests alongside the work you describe to codify this relationship

[dblock]

I think a simple manifest auto-add process could work like so:

for each manifest that doesn't have tags in it (product hasn't shipped) [this can be done with a GHA matrix]:
    previous_version = calculate previous version in some clever way
    for each component in a previous_version of the manifest that is not in the new manifest:
        add the component
        run manifest ci checks
        if build successful, raise a new/update an existing pr
        if build failed, stop

This way automation will keep updating the manifest similar to how humans do it, unattended.

When a new component is added to the manifest the above logic still works.
We very rarely to almost never remove components, but we can then just close the PR without merging it.

[peterzhuamazon]

We can extend the template yml file to include all the plugins in one take, then use the components parameter in the jenkins to build if we have to skip component.

This will also encourage plugin owners to start working on their plugins during a release.

Thanks.