Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OptionalDataException in 1.3.13 #3531

Closed
amitmun opened this issue Oct 11, 2023 · 5 comments
Closed

OptionalDataException in 1.3.13 #3531

amitmun opened this issue Oct 11, 2023 · 5 comments
Assignees
Labels
bug Something isn't working triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. v1.3.14 Issues targeting release v1.3.14

Comments

@amitmun
Copy link

amitmun commented Oct 11, 2023

What is the bug?
After upgrading from 1.3.12 to 1.3.13 we occasionally see OptionalDataException errors. The errors come and go, even without restarting the nodes.
This issue has been discussed a lot in early v2 versions, until fixed on 2.2.0: #1961
But it seems it was introduced to 1.3 now.
I believe the commit that caused it is this one: 2d4a2ad
And the commit that solved the issue for 2.2.0 is this one: 0b2280f
Perhaps the same fix should be applied to 1.3, as I see it has not been applied: https://github.com/opensearch-project/security/blob/1.3.13.0/src/main/java/org/opensearch/security/user/User.java

How can one reproduce the bug?
See #1961 with all the details about the issue and reproduction.

What is the expected behavior?
Queries and indexing should not fail.

What is your host/environment?

  • OS: CentOS Linux release 7.9.2009 (Core)
  • Version 1.3.13
  • Plugins: opensearch-anomaly-detection, opensearch-asynchronous-search, opensearch-cross-cluster-replication, opensearch-job-scheduler, opensearch-knn, opensearch-observability], opensearch-reports-scheduler, opensearch-security

Do you have any screenshots?
Logs:

org.opensearch.transport.RemoteTransportException: [Node-4][127.0.0.1:9303][indices:data/read/get[s]]
Caused by: org.opensearch.OpenSearchException: java.io.OptionalDataException
        at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:274) ~[?:?]
        at org.opensearch.security.auditlog.impl.AbstractAuditLog.getUser(AbstractAuditLog.java:629) ~[?:?]
        at org.opensearch.security.auditlog.impl.AbstractAuditLog.logMissingPrivileges(AbstractAuditLog.java:234) ~[?:?]
        at org.opensearch.security.auditlog.impl.AuditLogImpl.logMissingPrivileges(AuditLogImpl.java:176) ~[?:?]
        at org.opensearch.security.auditlog.AuditLogSslExceptionHandler.logError(AuditLogSslExceptionHandler.java:77) ~[?:?]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:169) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:667) ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91) ~[opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:244) [opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:127) [opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:109) [opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:759) [opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:170) [opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:145) [opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:110) [opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) [transport-netty4-client-1.3.13.jar:1.3.13]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1466) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1329) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1378) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.97.Final.jar:4.1.97.Final]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.io.IOException
        at java.io.ObjectInputStream.readObject0(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:?]
        at java.util.HashSet.readObject(Unknown Source) ~[?:?]
        at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at java.io.ObjectStreamClass.invokeReadObject(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readSerialData(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readObject0(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readSerialData(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readObject0(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:?]
        at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:?]
        at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:272) ~[?:?]
        at org.opensearch.security.auditlog.impl.AbstractAuditLog.getUser(AbstractAuditLog.java:629) ~[?:?]
        at org.opensearch.security.auditlog.impl.AbstractAuditLog.logMissingPrivileges(AbstractAuditLog.java:234) ~[?:?]
        at org.opensearch.security.auditlog.impl.AuditLogImpl.logMissingPrivileges(AuditLogImpl.java:176) ~[?:?]
        at org.opensearch.security.auditlog.AuditLogSslExceptionHandler.logError(AuditLogSslExceptionHandler.java:77) ~[?:?]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:169) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:667) ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91) ~[opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:244) ~[opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:127) ~[opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:109) [opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:759) ~[opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:170) ~[opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:145) ~[opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:110) ~[opensearch-1.3.13.jar:1.3.13]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[transport-netty4-client-1.3.13.jar:1.3.13]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[netty-handler-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1466) ~[netty-handler-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1329) ~[netty-handler-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1378) ~[netty-handler-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
@amitmun amitmun added bug Something isn't working untriaged Require the attention of the repository maintainers and may need to be prioritized labels Oct 11, 2023
@amitmun
Copy link
Author

amitmun commented Oct 11, 2023

@cwperks Since you fixed the issue in 2.2.0, can you please have a look and see if you think it's the same?

@stephen-crawford
Copy link
Contributor

stephen-crawford commented Oct 16, 2023

[Triage] Hi @amitmun, thank you for filing this issue and leaving such great details. We will have someone follow-up shortly to get this addressed with likely backporting the fix.

Action item:

  • Confirm root cause
  • Backport fix assuming it is the same as the 2.x line

@stephen-crawford stephen-crawford added triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. and removed untriaged Require the attention of the repository maintainers and may need to be prioritized labels Oct 16, 2023
@cwperks cwperks added the v1.3.14 Issues targeting release v1.3.14 label Oct 20, 2023
@cwperks
Copy link
Member

cwperks commented Oct 20, 2023

Hi @amitmun, apologies for not replying sooner. I believe this is the same issue encountered in 2.x. I added a label for 1.3.14 to track this for the 1.3.14 release in December. The release window for 1.3.14 is Dev 5 - 12 so the bug fix will need to be backported before then. https://opensearch.org/releases.html

@stephen-crawford stephen-crawford self-assigned this Nov 16, 2023
willyborankin pushed a commit that referenced this issue Nov 17, 2023
#3725)

### Description
This change backports #1970 in order to fix the OptionalDataException
issue encountered on the 1.3.x line.
The original change did not have any tests but #3637 added tests, so I
backported those as well and the changes required to support them. This
resulted in changes to the TestSecurityConfig.User class, the addition
of a IndexStateIsEqualToMatcher and User.java.

### Issues Resolved
- Resolves #3531
- Backports #1970 
- Backports #3637

### Check List
- [X] New functionality includes testing
- [ ] ~New functionality has been documented~
- [x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Stephen Crawford <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Co-authored-by: Peter Nied <[email protected]>
@stephen-crawford
Copy link
Contributor

This was completed with #3725

@bkkoli
Copy link

bkkoli commented Feb 5, 2024

This issue is helpful for me!! thank you!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. v1.3.14 Issues targeting release v1.3.14
Projects
None yet
Development

No branches or pull requests

4 participants