Make using Docker images in CI reliable and repeatable

[dblock]

Is your feature request related to a problem? Please describe

Currently images used in https://github.com/opensearch-project/opensearch-build/tree/main/docker/ci/dockerfiles contain too many things and keep changing while development is ongoing. Old images get deleted, too. This makes CI non-repeatable and non-reliable.

Describe the solution you'd like

Propose to follow the GitHub actions' images' model in every way.

• Immutable images, so that one can check out source from this repo at any time and reproduce builds without any changes.
• Reduce images to not contain the kitchen sink of dependencies. Make it easier to install a custom version of java/ruby/whatever at build time so developers can move fast.
• Add ability to target the "latest" image available for ongoing CI where a specific image version doesn't matter.
• Add CI image locking to the manifest .lock mechanism from latest -> specific current image.
• Enable developers to contribute, aka commit changes to images, build and push them.
• Make a menu of images available for a developer to pick from for various platforms.
• Document images and working with images.
• Add CI to verify that the manifests contain images that are valid.
• Centralized constants file for docker Images in Jenkinsfile #1325

Describe alternatives you've considered

No response

Additional context

No response

[zelinh]

[Triage] Added to Product backlog and we will start look into it.

[gaiksaya]

We have added most of what is requested in the solution. We are lacking documentation of available CI images.
cc: @peterzhuamazon

[peterzhuamazon]

Sure.

Close this as we have already used docker for majority of the builds, except macos. The documentations has been tracked in another issue.

Thanks.