[BUG] Plugins integration tests are failing after upgrade to 1.1 w/TransportSerializationException

[dblock]

Describe the bug

Looks like multiple plugins have failing integration tests when taking a 1.1 dependency of OpenSearch and common utils.

org.opensearch.indexmanagement.indexstatemanagement.MetadataRegressionIT > test move metadata service FAILED
    TransportSerializationException[Failed to deserialize response from handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@3f6b757b/org.opensearch.client.transport.TransportClientNodesService$RetryListener@1a5133b8]]; nested: IOException[Invalid string; unexpected character: 179 hex: b3];
        at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:250)
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:146)
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102)

        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:713)
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:155)

at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:130)
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:95)
        at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:309)
        at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:241)
        at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:53)
        at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152)
        at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:164)
        at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:433)
        at org.opensearch.nio.NioSelector.processKey(NioSelector.java:259)
        at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:187)
        at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:144)
        at java.base/java.lang.Thread.run(Thread.java:832)

        Caused by:
        java.io.IOException: Invalid string; unexpected character: 179 hex: b3
            at org.opensearch.common.io.stream.StreamInput.throwOnBrokenChar(StreamInput.java:545)
            at org.opensearch.common.io.stream.StreamInput.readString(StreamInput.java:504)
            at org.opensearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1233)
            at org.opensearch.common.io.stream.StreamInput.readList(StreamInput.java:1183)
            at org.opensearch.common.io.stream.StreamInput.readStringList(StreamInput.java:1194)
            at org.opensearch.plugins.PluginInfo.<init>(PluginInfo.java:120)
            at org.opensearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1233)
            at org.opensearch.common.io.stream.StreamInput.readList(StreamInput.java:1183)
            at org.opensearch.action.admin.cluster.node.info.PluginsAndModules.<init>(PluginsAndModules.java:60)
            at org.opensearch.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:1016)
            at org.opensearch.action.admin.cluster.node.info.NodeInfo.<init>(NodeInfo.java:98)
            at org.opensearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1233)
            at org.opensearch.common.io.stream.StreamInput.readList(StreamInput.java:1183)
            at org.opensearch.action.admin.cluster.node.info.NodesInfoResponse.readNodesFrom(NodesInfoResponse.java:71)
            at org.opensearch.action.support.nodes.BaseNodesResponse.<init>(BaseNodesResponse.java:57)
            at org.opensearch.action.admin.cluster.node.info.NodesInfoResponse.<init>(NodesInfoResponse.java:62)
            at org.opensearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:82)
            at org.opensearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:49)
            at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1198)
            at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1185)
            at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:247)
            ... 16 more

opensearch-notebooks

opensearch-project/dashboards-notebooks#62
https://github.com/opensearch-project/dashboards-notebooks/pull/62/checks?check_run_id=3352176545

index-management

opensearch-project/index-management#119
https://github.com/opensearch-project/index-management/pull/119/checks?check_run_id=3344612616

[dblock]

First, we found out that main wasn't bumped to 2.0 (@nknize is working on this), so we've been dropping 1.1-SNAPSHOT to opesearch.org from main and from the 1.x branches, overwriting each-other. We disabled the main snapshot build for now to avoid this problem. Next time we do version increments we'll need to make sure we both increment 1.0 to 1.1 and bump main to 2.0 kind of thing.

[nknize]

I haven't dug into this yet, but this usually means there was a change to the transport layer that isn't matching between Version.V_M_m_p. Perhaps index management plugin has doRead / doWrite logic that isn't aligning w/ the core codebase?

[nknize]

/cc @opensearch-project/index-management

[shwetathareja]

@dblock : not aware of any transport change which made into 1.1 core codebase, will take a look if anything pops related to this exception.

[dblock]

I am pretty sure this was caused by the fact that we were building against OpenSearch and common-utils 1.1, but running tests against what was effectively OpenSearch 2.0 that seems to have had backwards incompatible transport changes. After fixing that in our CI, opensearch-project/dashboards-notebooks#62 and opensearch-project/index-management#119 are passing.