Introducing incremental builds

[gaiksaya]

Is your feature request related to a problem? Please describe

Building OpenSearch and OpenSearch dashboards takes a significant amount of time. An end to end distribution roughly takes 45min-1hour to build. The current process builds all the components along with the core from source everytime a build is triggered. However, this can be made efficient to accommodate following features:

• Allow building only failed components on top of (given) previously built successful components.
• Allow rebuilding and re-installing one single component
• Allow rebuilding all dependent components if main dependent component changes. (Example: Job-scheduler needed a rebuild. All components dependent on JS needs and can be rebuilt)

Describe the solution you'd like

• Add dependent component logic in input manifest #4157
• [Incremental Build] Differentiate the commits changes and finalize rebuild components #4201
• [Incremental Build] Build incremental components through build workflow #4210
• [Incremental build] Onboard incremental build from Python workflow to jenkins CI. #4292

Describe alternatives you've considered

No response

Additional context

No response

[zelinh]

Proposed approach:

We will enhance the current build workflow with a new --incremental binary argument.
e.g. ./build.sh manifests/2.10.0/opensearch-2.10.0.yml --incremental

For a complete incremental build:

• Compare references between the build manifest from previous build and current input manifest.
• Identify components for incremental rebuilding based on the previous build manifest.
• Build these components, replacing previous artifacts with new ones in the builds folder, and generate a new build manifest.
• Run the assemble workflow based on the new build manifest to generate the new distribution bundle.

[zelinh]

The incremental build is onboarded to our python and Jenkins workflow. We now have it disabled as default. Once we validated it, we will enable it as default for distribution build Jenkins jobs.

[zelinh]

Closing this issue as incremental build is now enabled as default in our CI build.