Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitoring fails to serialize results when free swap is reported as negative #57312

Closed
danhermann opened this issue May 28, 2020 · 1 comment · Fixed by #57317
Closed

Monitoring fails to serialize results when free swap is reported as negative #57312

danhermann opened this issue May 28, 2020 · 1 comment · Fixed by #57317
Assignees
Labels
>bug :Data Management/Monitoring needs:triage Requires assignment of a team area label Team:Data Management Meta label for data/management team

Comments

@danhermann
Copy link
Contributor

Elasticsearch version (bin/elasticsearch --version): many versions through 7.7.0

Description of the problem including expected versus actual behavior:

On occasion, free swap memory in the monitoring stats is reported as negative and that negative value fails to serialize.

E.g.:

GET _nodes/instance-0000000314/stats?error_trace=true

returns:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Values less than -1 bytes are not supported: -1282646016b",
        "stack_trace": "[Values less than -1 bytes are not supported: -1282646016b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -1282646016b];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:49)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -1282646016b\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)\n\tat org.elasticsearch.monitor.os.OsStats$Swap.getFree(OsStats.java:204)\n\tat org.elasticsearch.monitor.os.OsStats$Swap.toXContent(OsStats.java:219)\n\tat org.elasticsearch.monitor.os.OsStats.toXContent(OsStats.java:114)\n\tat org.elasticsearch.action.admin.cluster.node.stats.NodeStats.toXContent(NodeStats.java:287)\n\tat org.elasticsearch.action.admin.cluster.node.stats.NodesStatsResponse.toXContent(NodesStatsResponse.java:61)\n\tat org.elasticsearch.rest.action.RestActions.nodesResponse(RestActions.java:183)\n\tat org.elasticsearch.rest.action.RestActions$NodesResponseRestListener.buildResponse(RestActions.java:232)\n\tat org.elasticsearch.rest.action.RestActions$NodesResponseRestListener.buildResponse(RestActions.java:223)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:38)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)\n\t... 50 more\n\tSuppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder\n\t\tat org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1011)\n\t\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:37)\n\t\t... 52 more\n\tCaused by: java.io.IOException: Unclosed object or array found\n\t\tat org.elasticsearch.common.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:469)\n\t\tat org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1009)\n\t\t... 53 more\n"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Values less than -1 bytes are not supported: -1282646016b",
    "stack_trace": "java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -1282646016b\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)\n\tat org.elasticsearch.monitor.os.OsStats$Swap.getFree(OsStats.java:204)\n\tat org.elasticsearch.monitor.os.OsStats$Swap.toXContent(OsStats.java:219)\n\tat org.elasticsearch.monitor.os.OsStats.toXContent(OsStats.java:114)\n\tat org.elasticsearch.action.admin.cluster.node.stats.NodeStats.toXContent(NodeStats.java:287)\n\tat org.elasticsearch.action.admin.cluster.node.stats.NodesStatsResponse.toXContent(NodesStatsResponse.java:61)\n\tat org.elasticsearch.rest.action.RestActions.nodesResponse(RestActions.java:183)\n\tat org.elasticsearch.rest.action.RestActions$NodesResponseRestListener.buildResponse(RestActions.java:232)\n\tat org.elasticsearch.rest.action.RestActions$NodesResponseRestListener.buildResponse(RestActions.java:223)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:38)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n\tSuppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder\n\t\tat org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1011)\n\t\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:37)\n\t\t... 52 more\n\tCaused by: java.io.IOException: Unclosed object or array found\n\t\tat org.elasticsearch.common.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:469)\n\t\tat org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1009)\n\t\t... 53 more\n",
    "suppressed": [
      {
        "type": "illegal_state_exception",
        "reason": "Failed to close the XContentBuilder",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unclosed object or array found",
          "stack_trace": "java.io.IOException: Unclosed object or array found\n\tat org.elasticsearch.common.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:469)\n\tat org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1009)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:37)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
        },
        "stack_trace": "java.lang.IllegalStateException: Failed to close the XContentBuilder\n\tat org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1011)\n\tat org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:37)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: java.io.IOException: Unclosed object or array found\n\tat org.elasticsearch.common.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:469)\n\tat org.elasticsearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1009)\n\t... 53 more\n"
      }
    ]
  },
  "status": 400
}

May be related to #42725 and #56435.

@danhermann danhermann added >bug :Data Management/Monitoring needs:triage Requires assignment of a team area label labels May 28, 2020
@danhermann danhermann self-assigned this May 28, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Monitoring)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Data Management/Monitoring needs:triage Requires assignment of a team area label Team:Data Management Meta label for data/management team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants