Design a generic message format for all extensions to ignore unintended bytes.

[saratvemulapalli]

Design a generic message format for all extensions to ignore unintended bytes.

[saratvemulapalli]

Coming from #415 there are few ideas which solves the problem.
Mostly inspired from Protobuf[1]

[1] https://protobuf.dev/

[saratvemulapalli]

So following up on #415 I was able to get a protobuf serialization on top of OpenSearch SteamInput/StreamOutput which was able to cleanly support backward compatible and forward compatible message.

I've pushed the code changes to my fork and a different branch
OpenSearch: saratvemulapalli/OpenSearch@3c506cb
SDK: saratvemulapalli@e7def3a

Here is an example sample message in OpenSearch:

message Extension {
  string name = 1;
  int32 id = 2;
  string description = 3;
}

Generate the proto serializer and deserializer: protoc --java_out=server/src/main/java/ server/src/main/java/org/opensearch/extensions/proto/extension.proto

On the extension:

• On same version as OpenSearch:

11:50:23.941 [opensearch[hello-world][generic][T#1]] INFO  org.opensearch.sdk.handlers.ExtensionsInitRequestHandler - ExtensionProto message: name: "FirstExtension"
id: 10
description: "Extension working with Proto"

• OpenSearch in new version (added new field):

message Extension {
  string name = 1;
  int32 id = 2;
  string description = 3;
  int32 ephemeral_id = 4;
}

The extension could cleanly de-serialize the message though new extra bytes are sent, it knows how to ignore it.

11:50:23.941 [opensearch[hello-world][generic][T#1]] INFO  org.opensearch.sdk.handlers.ExtensionsInitRequestHandler - ExtensionProto message: name: "FirstExtension"
id: 10
description: "Extension working with Proto"
4: 1219407348

• OpenSearch is on older version(one field removed) and SDK is compiled with newer interfaces of OpenSearch:

message Extension {
  string name = 1;
}

Extension cleanly accepts partial information and can decide how to proceed.

11:50:23.941 [opensearch[hello-world][generic][T#1]] INFO  org.opensearch.sdk.handlers.ExtensionsInitRequestHandler - ExtensionProto message: name: "FirstExtension"

Sample test from OpenSearch Serialization vs Protobuf on top of OpenSearch Serialization:

[2023-03-14T21:53:24,623][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageTransport] before test
[2023-03-14T21:53:24,794][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageTransport] Test started
[2023-03-14T21:53:24,796][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageTransport] Init complete
[2023-03-14T21:53:24,814][INFO ][o.o.e.FirstMessageTransport] [testFirstMessageTransport] Write Start
[2023-03-14T21:53:24,815][INFO ][o.o.e.FirstMessageTransport] [testFirstMessageTransport] Write Complete
[2023-03-14T21:53:24,818][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageTransport] De-serialization started
[2023-03-14T21:53:24,819][INFO ][o.o.e.FirstMessageTransport] [testFirstMessageTransport] Read Start
[2023-03-14T21:53:24,822][INFO ][o.o.e.FirstMessageTransport] [testFirstMessageTransport] Read Complete
[2023-03-14T21:53:24,822][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageTransport] De-serialization complete
[2023-03-14T21:53:24,850][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageTransport] after test
[2023-03-14T21:53:24,858][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageProto] before test
[2023-03-14T21:53:24,859][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageProto] Test started
[2023-03-14T21:53:24,882][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageProto] Init complete
[2023-03-14T21:53:24,882][INFO ][o.o.e.FirstMessageProto  ] [testFirstMessageProto] Write Start
[2023-03-14T21:53:24,895][INFO ][o.o.e.FirstMessageProto  ] [testFirstMessageProto] Write Complete
[2023-03-14T21:53:24,895][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageProto] De-serialization started
[2023-03-14T21:53:24,895][INFO ][o.o.e.FirstMessageProto  ] [testFirstMessageProto] Read Start
[2023-03-14T21:53:24,900][INFO ][o.o.e.FirstMessageProto  ] [testFirstMessageProto] Read Complete
[2023-03-14T21:53:24,900][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageProto] De-serialization complete
[2023-03-14T21:53:24,901][INFO ][o.o.e.RegisterTransportActionsRequestTests] [testFirstMessageProto] after test

From the above example, Protobuf helps:

• Supports backward and forward compatibility.
• Simple human readable message contracts for OpenSearch/SDK.
• Generated code for serialization/de-serialization in any language which enables us to offer SDKs in different languages.
• Potentially performance, this example implementation is on top of existing OpenSearch serialization/de-serialization which incurs latency.