Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Netty4HttpServerPipeliningTests fails on 6.4 due to invalid ref count decrement #32900

Closed
colings86 opened this issue Aug 16, 2018 · 3 comments
Closed

Netty4HttpServerPipeliningTests fails on 6.4 due to invalid ref count decrement #32900

colings86 opened this issue Aug 16, 2018 · 3 comments
Assignees
@jasontedor
Labels
:Distributed Coordination/Network Http and internode communication implementations >test-failure Triaged test failures from CI v6.4.1

Comments

@colings86
Copy link
Contributor

Build URL: http://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+intake/60

Reproduce commands:

./gradlew :modules:transport-netty4:test \
  -Dtests.seed=FB3A869C6124570A \
  -Dtests.class=org.elasticsearch.http.netty4.Netty4HttpServerPipeliningTests \
  -Dtests.method="testThatHttpPipeliningCanBeDisabled" \
  -Dtests.security.manager=true \
  -Dtests.locale=sl \
  -Dtests.timezone=Asia/Magadan

./gradlew :modules:transport-netty4:test \
  -Dtests.seed=FB3A869C6124570A \
  -Dtests.class=org.elasticsearch.http.netty4.Netty4HttpServerPipeliningTests \
  -Dtests.method="testThatHttpPipeliningCanBeDisabled" \
  -Dtests.security.manager=true \
  -Dtests.locale=sl \
  -Dtests.timezone=Asia/Magadan

This doesn't reproduce for me locally but it seems that there is a bug where we are trying to decrement a reference counter when its already zero:

 1> [2018-08-16T03:19:43,021][INFO ][o.e.h.n.Netty4HttpServerPipeliningTests] Caught exception
16:20:19   1> io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
16:20:19   1> 	at io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:102) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:90) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.buffer.WrappedCompositeByteBuf.release(WrappedCompositeByteBuf.java:43) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.buffer.SimpleLeakAwareCompositeByteBuf.release(SimpleLeakAwareCompositeByteBuf.java:38) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.release(AdvancedLeakAwareCompositeByteBuf.java:1023) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.release(HttpObjectAggregator.java:386) ~[netty-codec-http-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84) ~[netty-common-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:112) ~[netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
16:20:19   1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
@colings86 colings86 added :Distributed Coordination/Network Http and internode communication implementations >test-failure Triaged test failures from CI labels Aug 16, 2018
colings86 added a commit that referenced this issue Aug 16, 2018
@colings86
Mutes
42a57b3 
Netty4HttpServerPipeliningTests.testThatHttpPipeliningCanBeDisabled

Due to #32900
@colings86
Copy link
Contributor Author

Muted on 6.4 branch in 42a57b3

@lcawl lcawl added v6.4.1 and removed v6.4.0 labels Aug 23, 2018
@jasontedor jasontedor self-assigned this Aug 30, 2018
@jasontedor
Copy link
Member

I will backport #32368 to address this.

@jasontedor
Copy link
Member

Closed by b28044c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jasontedor jasontedor

Labels
:Distributed Coordination/Network Http and internode communication implementations >test-failure Triaged test failures from CI v6.4.1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@colings86 @jasontedor @lcawl