[META] [FEATURE] Design and build forward compatibility and backward compatibility for SDK

[ryanbogan]

The SDK must be able to function properly no matter what versions of the SDK or OpenSearch the user is running. We need to design and build a system that can accomplish this goal.

• Build a message which works with forward compatibility #415
• Design a generic message format for all extensions to ignore unintended bytes. #414

[ryanbogan]

To simulate backwards compatibility, I added a new String parameter to AcknowledgedResponse in the OpenSearch repo, and ran the HelloWorld extension with this change. Neither OpenSearch nor the SDK crashed, but the extension was not initialized due to the following exception:

java.lang.IllegalStateException: Message not fully read (response) for requestId [3], handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.sdk.handlers.AcknowledgedResponseHandler@3b87747c], error [false]; resetting

[dbwiddis]

Can't help debug without code, but the issue is that if you're adding a new parameter you need to make it optional, which generally means if you're doing a Writeable transport request you need to either use the existing writeOptionalString() or equivalent; or manually do the optional bit by writing a boolean and then only if the boolean is true, the writeable object.

See how ExtensionRequest was implemented in opensearch-project/OpenSearch#5658