Ensure OpenSearch drop-in replacement for ES and ODFE

[dblock]

Is your feature request related to a problem? Please describe.

So far we are hearing from discuss#5788 that most users would like OpenSearch to be a drop in replacement for ES/ODFE. Working backwards from that assumption we need mechanisms to enable, then ensure backwards compatibility with such things as settings, indexes, APIs, namespaces and possibly more. This is the umbrella issue for technical discussions and PRs that ensure backwards compatibility with ES/ODFE, as well as tools that can be used by plugins to enable such backwards compatibility.

Describe the solution you'd like

• As an admin of a deployment, be able to replace an existing installation of ES/ODFE by OpenSearch and existing ES/ODFE plugins that have newer compatible OpenSearch versions, without making any configuration or data changes. Everything should just work.
• As an author of a plugin, be able to upgrade dependencies to OpenSearch and make minimal changes that enable me to build a new version that becomes a drop-in replacement to my previous plugin version.
• See what is being deprecated and have a clear deprecation path for anything that will not be forever backwards compatible.

Describe alternatives you've considered
We considered not being backwards compatible at all, but discuss#5788 is saying that it's a bad idea.

Additional context

[dblock]

For settings we could have a "magical" implementation that does s/opendistro/opensearch/g and s/elasticsearch/opensearch ala how different list formats were changed.

[rursprung]

note: i (not sure about others) would be ok with having to update the config files with the elasticsearch => opensearch change since we anyway need to touch the system (i.e. replace the installation). since we're running it in docker containers this isn't a big issue for us.

more interesting would be if opensearch 1.0.0 would be able to join an elasticsearch 7.10.x cluster - presuming you have the same plugins (with compatible versions) enabled in both - so that you could even do a rolling upgrade and not incur any downtime. this isn't a hard requirement for us, but based on the forum posts this might be of more interest to others.
if this isn't possible then a re-use of the datafiles would be welcome as this would avoid any backup/restore operations. which brings me to the next point: backups need to be compatible (at least for ES 7.10 & opensearch 1.x).

[saratvemulapalli]

Created a meta issue to list down all the things which we have broken.
#640

[saratvemulapalli]

These are all the items which we should consider while working on the drop in upgrades.

• Ordering of upgrading
• Naming conventions
• Upgrade time-lag (aka downtime)
• Replication
• Security
• Downstream Dependencies
• Upgrade alerting and notifications
• Limits
• Performance
• Availability
• Latency
• Compatibility Matrix
• Downgrade
• APIs

[dblock]

Superseded by #671