Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] BWC Rolling Upgrade Tests are failing on neural search due to changes in MLPredictionTaskRequest Object #1838

Closed
vibrantvarun opened this issue Jan 4, 2024 · 13 comments · Fixed by #2085
Closed

[Bug] BWC Rolling Upgrade Tests are failing on neural search due to changes in MLPredictionTaskRequest Object #1838

vibrantvarun opened this issue Jan 4, 2024 · 13 comments · Fixed by #2085
Assignees
@b4sjoo
Labels
bug Something isn't working

Comments

@vibrantvarun
Copy link
Member

vibrantvarun commented Jan 4, 2024 •
edited
Loading

What is the bug?
The BWC Rolling upgrade tests are failing in the neural search because of recent changes in ML-Commons.
So the scenario is when the bwc version is 2.10 and current version is 2.12.0-SNAPSHOT

./gradlew :qa:rolling-upgrade:testAgainstOneThirdUpgradedCluster -Dbwc.version=2.10.0

The tests fails with a reason

org.opensearch.transport.RemoteTransportException: [neuralSearchBwcCluster-rolling-1][127.0.0.1:35231][cluster:admin/opensearch/ml/predict]
Caused by: java.lang.IllegalStateException: unexpected byte [0x48]
	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:592) ~[opensearch-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:582) ~[opensearch-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.ml.common.transport.prediction.MLPredictionTaskRequest.<init>(MLPredictionTaskRequest.java:56) ~[?:?]
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) ~[opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:283) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:243) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:133) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:115) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:767) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.101.Final.jar:4.1.101.Final]

It is because there is one more constructor for MLPredictionTaskRequest has been added in the PR mentioned above.

In BWC tests when cluster has 3 nodes and 2 of them are running on 2.10 and one node gets upgraded to 2.12.0-SNAPSHOT. Now
when neural search calls predict api then while initializing the MLPredictionTaskRequest here it fails at this line with error mentioned above.

It is happening because it is either getting more parameters or no parameters for boolean while executing in.readBoolean().

The same is happening when the BWC tests are executed for 2.9 OS version.

The tests are running fine for 2.11 OS version.
But it makes this tests execution flaky.
Because if Out of 3 nodes if it uses the node which is running 2.11 version for calling predict api rather than 2.12.0-SNAPSHOT then it will face the same issue.

How can one reproduce the bug?
Steps to reproduce the behavior:

  1. Clone the neural search project
  2. checkout on 2.x branch
  3. Run ./gradlew :qa:rolling-upgrade:testAgainstOneThirdUpgradedCluster -Dbwc.version=2.10.0
  4. See error

What is the expected behavior?
The BWC rolling upgrade tests should run with all previous version with no error.

What is your host/environment?

  • OS: linux
  • Version
  • Plugins Neural Search
@vibrantvarun vibrantvarun added bug Something isn't working untriaged labels Jan 4, 2024
@vibrantvarun
Copy link
Member Author

I have discussed the issue with @reta .

@vibrantvarun vibrantvarun mentioned this issue Jan 4, 2024
Closed
@vibrantvarun vibrantvarun changed the title [Bug] BWC Rolling Upgrade Tests are failing on neural search due to recent changes in MLPredictionTaskRequest [Bug] BWC Rolling Upgrade Tests are failing on neural search due to changes in MLPredictionTaskRequest Object Jan 5, 2024
@b4sjoo b4sjoo removed the untriaged label Jan 5, 2024
@b4sjoo
Copy link
Collaborator

b4sjoo commented Jan 12, 2024

I am still writing the test cases now, since in previous we only used built in function to test and we need to test neural models this time it’s actually like writing from very beginning.

@b4sjoo b4sjoo assigned b4sjoo and unassigned ylwu-amzn Jan 12, 2024
@b4sjoo
Copy link
Collaborator

b4sjoo commented Jan 25, 2024

Can confirm bug reproduced in manually setup cluster when sending predict request to 2.11 cluster with 2.9 master node:

POST /_plugins/_ml/models/bsBlQI0Bqvn5_05_or5u/_predict
{
  "text_docs":[ "today is sunny"],
  "return_number": true,
  "target_response": ["sentence_embedding"]
}

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "unexpected byte [0x73]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "unexpected byte [0x73]"
  },
  "status": 500
}

@b4sjoo
Copy link
Collaborator

b4sjoo commented Jan 25, 2024 •
edited
Loading

In the meantime we noticed that pretrained model registration tasks also failed from 2.9 to 2.11

Caused by: java.lang.IllegalStateException: Message fully read (request) but more data is expected for requestId [1421], action [cluster:admin/opensearch/mlinternal/forward]; resetting
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:304) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.EOFException
	at org.opensearch.core.common.io.stream.InputStreamStreamInput.readByte(InputStreamStreamInput.java:74) ~[?:?]
	at org.opensearch.core.common.io.stream.FilterStreamInput.readByte(FilterStreamInput.java:55) ~[?:?]
	at org.opensearch.core.common.io.stream.StreamInput.readOptionalBoolean(StreamInput.java:598) ~[?:?]
	at org.opensearch.ml.common.transport.register.MLRegisterModelInput.<init>(MLRegisterModelInput.java:163) ~[?:?]
	at org.opensearch.ml.common.transport.forward.MLForwardInput.<init>(MLForwardInput.java:63) ~[?:?]
	at org.opensearch.ml.common.transport.forward.MLForwardRequest.<init>(MLForwardRequest.java:43) ~[?:?]
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:300) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

@b4sjoo
Copy link
Collaborator

b4sjoo commented Jan 26, 2024

Stack trace for the error:

[2024-01-25T12:21:09,697][ERROR][o.o.m.c.MLCommonsClassLoader] [data1] Failed to init instance for type TEXT_DOCS
java.lang.reflect.InvocationTargetException: null
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
"mlplugin-bwc.log" 971L, 138488B                                                    341,86        34%
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: unexpected byte [0x64]
        at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:592) ~[opensearch-core-2.11.0.jar:2.11.0]
        at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:582) ~[opensearch-core-2.11.0.jar:2.11.0]
        at org.opensearch.core.common.io.stream.StreamInput.readOptionalString(StreamInput.java:372) ~[opensearch-core-2.11.0.jar:2.11.0]
        at org.opensearch.ml.common.dataset.TextDocsInputDataSet.<init>(TextDocsInputDataSet.java:47) ~[?:?]
        ... 52 more

@b4sjoo
Copy link
Collaborator

b4sjoo commented Jan 27, 2024 •
edited
Loading

After fixing, the predict still return the exception:

org.opensearch.transport.TransportSerializationException: Failed to deserialize response from handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:375) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:160) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Message not fully read (response) for requestId [472], handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a], error [false]; resetting
	at org.opensearch.transport.InboundHandler.checkStreamIsFullyConsumed(InboundHandler.java:341) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:373) ~[opensearch-2.9.0.jar:2.9.0]
	... 26 more
[2024-01-27T00:03:25,979][WARN ][r.suppressed             ] [master1] path: /_plugins/_ml/models/2Ws3SI0Btppe6bLUosJi/_predict, params: {model_id=2Ws3SI0Btppe6bLUosJi}
org.opensearch.transport.RemoteTransportException: [Failed to deserialize response from handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a]]
Caused by: org.opensearch.transport.TransportSerializationException: Failed to deserialize response from handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:375) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:160) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-2.9.0.jar:2.9.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Message not fully read (response) for requestId [472], handler [org.opensearch.transport.TransportService$ContextRestoreResponseHandler/org.opensearch.action.ActionListenerResponseHandler@32734b9b/org.opensearch.action.ActionListener$5@129fd05a], error [false]; resetting
	at org.opensearch.transport.InboundHandler.checkStreamIsFullyConsumed(InboundHandler.java:341) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:373) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:160) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) [opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.9.0.jar:2.9.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) **~[?:?]**

@b4sjoo
Copy link
Collaborator

b4sjoo commented Jan 27, 2024

It seems this is a change from 2.9.0 to 2.9.1 (Not release) at this line:

This was referenced Jan 27, 2024
Merged
Merged
@vibrantvarun vibrantvarun mentioned this issue Jan 31, 2024
Closed
5 tasks
@b4sjoo
Copy link
Collaborator

b4sjoo commented Feb 1, 2024

PR is merged and I manually did the sanity test on local neural models on different version, looks good to me, but when will this update be applied to neural search team's CI workflow still seems uncleared to me.

@vibrantvarun vibrantvarun mentioned this issue Feb 1, 2024
Merged
5 tasks
@b4sjoo
Copy link
Collaborator

b4sjoo commented Feb 3, 2024

When testing remote model:

 [2024-02-03T11:01:09,767][ERROR][o.o.m.c.MLCommonsClassLoader] [data1] Failed to init instance for type TEXT_SIMILARITY
java.lang.reflect.InvocationTargetException: null
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.ml.common.MLCommonsClassLoader.init(MLCommonsClassLoader.java:240) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.common.MLCommonsClassLoader.initMLInstance(MLCommonsClassLoader.java:204) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.common.dataset.MLInputDataset.fromStream(MLInputDataset.java:33) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.common.input.MLInput.<init>(MLInput.java:104) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
        at org.opensearch.ml.common.transport.prediction.MLPredictionTaskRequest.<init>(MLPredictionTaskRequest.java:59) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
        at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:300) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.11.1.jar:2.11.1]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.11.1.jar:2.11.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.EOFException: tried to read: 109 bytes but this stream is limited to: 98
        at org.opensearch.core.common.io.stream.InputStreamStreamInput.ensureCanReadBytes(InputStreamStreamInput.java:135) ~[opensearch-core-2.11.1.jar:2.11.1]
        at org.opensearch.core.common.io.stream.FilterStreamInput.ensureCanReadBytes(FilterStreamInput.java:110) ~[opensearch-core-2.11.1.jar:2.11.1]
        at org.opensearch.core.common.io.stream.StreamInput.readArraySize(StreamInput.java:1284) ~[opensearch-core-2.11.1.jar:2.11.1]
        at org.opensearch.core.common.io.stream.StreamInput.readString(StreamInput.java:424) ~[opensearch-core-2.11.1.jar:2.11.1]
        at org.opensearch.ml.common.dataset.TextSimilarityInputDataSet.<init>(TextSimilarityInputDataSet.java:61) ~[?:?]
        ... 43 more

[2024-02-03T11:01:09,773][WARN ][r.suppressed             ] [master1] path: /_plugins/_ml/models/qx-gbo0BVjqxw2WNkMgc/_predict, params: {model_id=qx-gbo0BVjqxw2WNkMgc}
org.opensearch.transport.RemoteTransportException: [data1][127.0.0.1:9301][cluster:admin/opensearch/ml/predict]
Caused by: java.lang.IllegalStateException: unexpected byte [0x6f]
        at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:592) ~[opensearch-core-2.11.0.jar:2.11.0]
        at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:582) ~[opensearch-core-2.11.0.jar:2.11.0]
        at org.opensearch.ml.common.transport.prediction.MLPredictionTaskRequest.<init>(MLPredictionTaskRequest.java:60) ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:300) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.11.0.jar:2.11.0]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-02-03T11:01:31,356][INFO ][o.o.i.i.ManagedIndexCoordinator] [master1] Performing move cluster state metadata.
[2024-02-03T11:01:31,356][INFO ][o.o.i.i.MetadataService  ] [master1] Cluster still has nodes running old version of ISM plugin, skip ping execution on new nodes until all nodes upgraded

        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.11.0.jar:2.11.0]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]

@b4sjoo
Copy link
Collaborator

b4sjoo commented Feb 5, 2024 

[2024-02-05T08:24:30,271][ERROR][o.o.m.c.MLCommonsClassLoader] [data1] Failed to init instance for type LINEAR_REGRESSION
java.lang.reflect.InvocationTargetException: null
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.ml.common.MLCommonsClassLoader.init(MLCommonsClassLoader.java:240) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.MLCommonsClassLoader.initMLInstance(MLCommonsClassLoader.java:204) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.input.MLInput.<init>(MLInput.java:114) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.transport.prediction.MLPredictionTaskRequest.<init>(MLPredictionTaskRequest.java:59) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:300) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.11.1.jar:2.11.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: unexpected byte [0x03]
	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:592) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:582) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.ml.common.input.parameter.regression.LinearRegressionParams.<init>(LinearRegressionParams.java:85) ~[?:?]
	... 42 more
[2024-02-05T08:24:30,278][ERROR][o.o.m.c.MLCommonsClassLoader] [data1] Failed to init instance for type DATA_FRAME
java.lang.reflect.InvocationTargetException: null
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.ml.common.MLCommonsClassLoader.init(MLCommonsClassLoader.java:240) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.MLCommonsClassLoader.initMLInstance(MLCommonsClassLoader.java:204) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.dataset.MLInputDataset.fromStream(MLInputDataset.java:33) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.input.MLInput.<init>(MLInput.java:117) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.transport.prediction.MLPredictionTaskRequest.<init>(MLPredictionTaskRequest.java:59) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:300) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.11.1.jar:2.11.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.IOException: Unknown DataFrameType ordinal [6]
	at org.opensearch.core.common.io.stream.StreamInput.readEnum(StreamInput.java:1240) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamInput.readEnum(StreamInput.java:1234) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.ml.common.dataset.DataFrameInputDataset.<init>(DataFrameInputDataset.java:40) ~[?:?]
	... 43 more
[2024-02-05T08:24:32,384][ERROR][o.o.m.c.MLCommonsClassLoader] [data1] Failed to init instance for type LINEAR_REGRESSION
java.lang.reflect.InvocationTargetException: null
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.ml.common.MLCommonsClassLoader.init(MLCommonsClassLoader.java:240) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.MLCommonsClassLoader.initMLInstance(MLCommonsClassLoader.java:204) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.input.MLInput.<init>(MLInput.java:114) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.transport.prediction.MLPredictionTaskRequest.<init>(MLPredictionTaskRequest.java:59) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:300) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.11.1.jar:2.11.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: unexpected byte [0x03]
	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:592) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:582) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.ml.common.input.parameter.regression.LinearRegressionParams.<init>(LinearRegressionParams.java:85) ~[?:?]
	... 42 more
[2024-02-05T08:24:32,386][ERROR][o.o.m.c.MLCommonsClassLoader] [data1] Failed to init instance for type DATA_FRAME
java.lang.reflect.InvocationTargetException: null
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.ml.common.MLCommonsClassLoader.init(MLCommonsClassLoader.java:240) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.MLCommonsClassLoader.initMLInstance(MLCommonsClassLoader.java:204) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.dataset.MLInputDataset.fromStream(MLInputDataset.java:33) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.input.MLInput.<init>(MLInput.java:117) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.ml.common.transport.prediction.MLPredictionTaskRequest.<init>(MLPredictionTaskRequest.java:59) [opensearch-ml-common-2.11.1.0-SNAPSHOT.jar:?]
	at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:85) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.newRequest(InboundHandler.java:300) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:255) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.11.1.jar:2.11.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.IOException: Unknown DataFrameType ordinal [6]
	at org.opensearch.core.common.io.stream.StreamInput.readEnum(StreamInput.java:1240) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.core.common.io.stream.StreamInput.readEnum(StreamInput.java:1234) ~[opensearch-core-2.11.1.jar:2.11.1]
	at org.opensearch.ml.common.dataset.DataFrameInputDataset.<init>(DataFrameInputDataset.java:40) ~[?:?]
	... 43 more

This was referenced Feb 13, 2024
Merged
Merged
Merged
@b4sjoo b4sjoo moved this to In Progress in ml-commons projects Feb 13, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in ml-commons projects Feb 13, 2024
@b4sjoo b4sjoo reopened this Feb 16, 2024
@b4sjoo b4sjoo removed the untriaged label Feb 16, 2024
@b4sjoo b4sjoo moved this from Done to In Progress in ml-commons projects Feb 16, 2024
@vibrantvarun
Copy link
Member Author

@b4sjoo Can you share what is the current update on this bug?

@b4sjoo
Copy link
Collaborator

b4sjoo commented Feb 22, 2024

@vibrantvarun Right now with a new master node (>= 2.11) and an old data node (<= 2.10), it seems remote model cannot be deployed in this case, except for this issue our bwc looks good.

@b4sjoo
Copy link
Collaborator

b4sjoo commented Feb 28, 2024 •
edited
Loading

@vibrantvarun Quick notes: there are backward compatibility issue in ml plugin 2.9 and 2.10 towards 2.11 or later version. There was PR fixed this issue recently but since 2.9.0/2.10.0 release has already cutoff and no patch version has been released, they are still using old problematics source code to build. Accordingly, since neural search team's bwc test workflow use them as test artifacts, it will failed anyways.

@b4sjoo b4sjoo closed this as completed May 9, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in ml-commons projects May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@b4sjoo b4sjoo

Labels
bug Something isn't working
Projects
ml-commons projects
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix bwc issue in remote prediction b4sjoo/ml-commons
3 participants
@vibrantvarun @ylwu-amzn @b4sjoo