Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throwing an Exception from an onStatus handler in WebClient leads to a ByteBuf leak #746

Closed
indiv1d opened this issue May 21, 2019 · 16 comments
Labels
for/springframework This belongs to the Spring Framework project

Comments

@indiv1d
Copy link

indiv1d commented May 21, 2019

Expected behavior

No leak

Actual behavior

LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: #1: io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base/java.lang.Thread.run(Thread.java:834) #2: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793) io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:592) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base/java.lang.Thread.run(Thread.java:834) #3: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793) io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:572) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base/java.lang.Thread.run(Thread.java:834) #4: io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160) io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:557) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:193) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base/java.lang.Thread.run(Thread.java:834) #5: Hint: 'reactor.left.httpCodec' will handle the message from this point. io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base/java.lang.Thread.run(Thread.java:834) #6: io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736) io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1488) io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166) io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base/java.lang.Thread.run(Thread.java:834) #7: io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706) io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1488) io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166) io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base/java.lang.Thread.run(Thread.java:834) Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115) io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2125) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1327) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base/java.lang.Thread.run(Thread.java:834) : 6 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

Steps to reproduce

Don't know how to reproduce. It's happening only in production environment.

Reactor Netty version

0.9.0.M1 / Spring Boot 2.2.0.M3
also
0.8.5.RELEASE / Spring Boot 2.1.3.RELEASE

JVM version (e.g. java -version)

openjdk 11.0.3 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+1-Debian-1bpo91)
OpenJDK 64-Bit Server VM (build 11.0.3+1-Debian-1bpo91, mixed mode, sharing)

OS version (e.g. uname -a)

Linux 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 GNU/Linux

@indiv1d indiv1d added the type/bug A general bug label May 21, 2019
@smaldini
Copy link
Contributor

Can you please elaborate or complete with a scenario/use case ? Have you tried the snapshot in case (0.9.0.BUILD-SNAPSHOT) ?

@indiv1d
Copy link
Author

indiv1d commented May 22, 2019

We haven't checked SNAPSHOT yet.
Scenario is simple. We have a lot of http clients (Spring WebClient with Netty implementation) to access our microservices some of them using https connection. As i see from log the problem with them.

From previous issues like this one i have checked .onStatus calls (but as i see it reads response always now), .exchange without body read. No problems with it. I have no ideas where i should search cause of failure.

@violetagg
Copy link
Member

violetagg commented Jun 10, 2019

@1nDivid Closing this. Try 0.8.9.RELEASE/0.9.0.M2. If the issue still persists we can reopen the issue.

@violetagg violetagg added for/user-attention This issue needs user attention (feedback, rework, etc...) and removed type/bug A general bug labels Jun 10, 2019
@indiv1d
Copy link
Author

indiv1d commented Jun 11, 2019

Just updated to 0.9.0.M2 and nothing new. Problem still there @violetagg :

LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: #1: io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #2: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793) io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:592) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #3: io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670) io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793) io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:842) io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:199) io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #4: Hint: 'reactor.left.httpCodec' will handle the message from this point. io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #5: io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736) io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1488) io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166) io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) #6: io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706) io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1488) io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166) io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178) io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115) io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2125) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1327) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) : 1 leak records were discarded because they were duplicates : 6 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

@violetagg
Copy link
Member

@1nDivid Please do one of the following options:

  • Enable reactor.netty=DEBUG log level, and run your application with -Dio.netty.leakDetection.level=paranoid and then provide us the new logs (there should be more information about the memory leak)
  • Or provide some sample that we can use to reproduce/or show us some code so that we know how you use WebClient.

@violetagg violetagg reopened this Jun 11, 2019
@indiv1d
Copy link
Author

indiv1d commented Jun 11, 2019

@violetagg Will try to catch details.
Here is simple client code in app (all clients have been written in the same way):

@Qualifier("deal") private val client: WebClient

fun createParticipant(participantRequest: ParticipantRequest) =
        client.post()
            .uri(CREATE_PARTICIPANTS_PATH)
            .body(BodyInserters.fromObject(participantRequest))
            .retrieve()
            .bodyToMono(object : ParameterizedTypeReference<DealResponse<Participant>>() {})
            .flatMap { it.data.first().toMono() }

and client configuration:

    @Bean("deal")
    fun dealClient(config: DealClientConfig) =
        webClientBuilder.clone()
            .baseUrl(config.url)
            .defaultHeader("X-Source-Token", config.token)
            .defaultHeader("X-Cas-Id", "1")
            .build()

@simondaudin
Copy link
Contributor

Hi,
We have the same issue in our environment.

Reactor-netty 0.8.8-RELEASE
Reactor-core 3.2.9-RELEASE

After setting the debug environment as you mentioned above @violetagg, I managed to get a stacktrace

Stacktrace

2019-06-27T13:38:04,157Z [ERROR] i.n.u.ResourceLeakDetector {reactor-http-epoll-3} - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getBytes(AdvancedLeakAwareByteBuf.java:250)
        io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:1029)
        io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:48)
        io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:889)
        io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:2047)
        io.netty.buffer.WrappedCompositeByteBuf.readBytes(WrappedCompositeByteBuf.java:1052)
        io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.readBytes(AdvancedLeakAwareCompositeByteBuf.java:483)
        io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.readBytes(AdvancedLeakAwareCompositeByteBuf.java:36)
        io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:180)
        com.fasterxml.jackson.dataformat.smile.SmileParser._loadMore(SmileParser.java:248)
        com.fasterxml.jackson.dataformat.smile.SmileParser.nextToken(SmileParser.java:376)
        com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4141)
        com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3971)
        com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2300)
        com.fasterxml.jackson.core.JsonParser.readValueAs(JsonParser.java:1752)
        com.liveaction.reactiff.codec.jackson.JacksonCodec.lambda$decodeMono$0(JacksonCodec.java:52)
        reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:100)
        reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:113)
        reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
        reactor.core.publisher.FluxUsing$UsingFuseableSubscriber.onNext(FluxUsing.java:350)
        reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:113)
        reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
        reactor.core.publisher.MonoReduceSeed$ReduceSeedSubscriber.onComplete(MonoReduceSeed.java:156)
        reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
        reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144)
        reactor.core.publisher.UnicastProcessor.checkTerminated(UnicastProcessor.java:336)
        reactor.core.publisher.UnicastProcessor.drainRegular(UnicastProcessor.java:232)
        reactor.core.publisher.UnicastProcessor.drain(UnicastProcessor.java:312)
        reactor.core.publisher.UnicastProcessor.onComplete(UnicastProcessor.java:414)
        com.livingobjects.longback.core.internal.reactive.client.HttpClientMethodImpl.lambda$null$11(HttpClientMethodImpl.java:217)
        reactor.core.publisher.LambdaSubscriber.onComplete(LambdaSubscriber.java:102)
        reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
        reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:390)
        reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:197)
        reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:338)
        reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:350)
        reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:399)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:555)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        java.lang.Thread.run(Thread.java:748)
#2:
        Hint: [id: 0xa20cf247, L:/10.0.0.131:34158 - R:bach/10.0.0.18:3000] Buffered ByteBufHolder in Inbound Flux Queue
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:24)
        reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:317)
        reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:335)
        reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:571)
        reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        java.lang.Thread.run(Thread.java:748)
#3:
        Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:88)
        io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:24)
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        java.lang.Thread.run(Thread.java:748)
#4:
        Hint: Caller of readInbound() will handle the message from this point
        io.netty.channel.embedded.EmbeddedChannel.readInbound(EmbeddedChannel.java:293)
        io.netty.handler.codec.http.HttpContentDecoder.fetchDecoderOutput(HttpContentDecoder.java:267)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:255)
        io.netty.handler.codec.http.HttpContentDecoder.decodeContent(HttpContentDecoder.java:161)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:150)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:47)
        io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        java.lang.Thread.run(Thread.java:748)
#5:
        Hint: 'DefaultChannelPipeline$TailContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:325)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:254)
        io.netty.handler.codec.http.HttpContentDecoder.decodeContent(HttpContentDecoder.java:161)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:150)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:47)
        io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        java.lang.Thread.run(Thread.java:748)
#6:
        io.netty.buffer.AdvancedLeakAwareByteBuf.ensureWritable(AdvancedLeakAwareByteBuf.java:136)
        io.netty.handler.codec.compression.JdkZlibDecoder.decode(JdkZlibDecoder.java:211)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:325)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:254)
        io.netty.handler.codec.http.HttpContentDecoder.decodeContent(HttpContentDecoder.java:161)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:150)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:47)
        io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        java.lang.Thread.run(Thread.java:748)
Created at:
        io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:332)
        io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:168)
        io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:159)
        io.netty.handler.codec.compression.JdkZlibDecoder.decode(JdkZlibDecoder.java:180)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:325)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:254)
        io.netty.handler.codec.http.HttpContentDecoder.decodeContent(HttpContentDecoder.java:161)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:150)
        io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:47)
        io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
        io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        java.lang.Thread.run(Thread.java:748)
: 3 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

It seems that it goes through a part of our code.

public <T> Mono<T> decodeMono(Publisher<ByteBuf> byteBufFlux, TypeToken<T> typeToken) {
        return ByteBufFlux.fromInbound(byteBufFlux)
                .aggregate()
                .asInputStream()
                .map(inputStream -> {
                    try {
                        return jsonFactory.createParser(inputStream).readValueAs(toTypeReference(typeToken));
                    } catch (IOException e) {
                        throw new RuntimeException(e);
                    }
                });
    }

I replaced it by

public <T> Mono<T> decodeMono(Publisher<ByteBuf> byteBufFlux, TypeToken<T> typeToken) {
        return ByteBufFlux.fromInbound(byteBufFlux)
                .aggregate()
                .asByteArray()
                .map(bytes -> {
                    try {
                        return jsonFactory.createParser(bytes).readValueAs(toTypeReference(typeToken));
                    } catch (IOException e) {
                        throw new RuntimeException(e);
                    }
                });
    }

Since then, it seemed to have gone away.

I Hope it can help

@icecreamhead
Copy link

icecreamhead commented Jun 28, 2019

We also have this issue.
spring-boot-starter-parent: 2.1.6.RELEASE
reactor-netty: 0.8.9.RELEASE
reactor-core: 3.2.10.RELEASE
spring-webflux: spring-webflux-5.1.8.RELEASE

LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:332)
	io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:168)
	io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:159)
	io.netty.handler.codec.compression.JdkZlibDecoder.decode(JdkZlibDecoder.java:180)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:325)
	io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:254)
	io.netty.handler.codec.http.HttpContentDecoder.decodeContent(HttpContentDecoder.java:161)
	io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:150)
	io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:47)
	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478)
	io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)
	io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	java.base/java.lang.Thread.run(Thread.java:834)

@violetagg
Copy link
Member

@simondaudin reactor.netty.ByteBufFlux#asInputStream keeps a reference to the byte buffer and releases it when the stream is closed.
https://github.com/reactor/reactor-netty/blob/0.8.x/src/main/java/reactor/netty/ByteBufMono.java#L160-L171
If an error/cancel happens your code does not close the stream. (you should use at least doOnDiscard)
The code with byte[] works because the byte buffer is copied to byte[] and then released by Reactor Netty.

@icecreamhead Follow the steps in the comment #746 (comment) and created a new issue so that we can track your use case there. As you can see (previous comment) the leak might be because of an application code and not Reactor Netty code.

@violetagg
Copy link
Member

@1nDivid Can you give us more info what this it.data.first().toMono() is doing?

@indiv1d
Copy link
Author

indiv1d commented Jun 28, 2019

@violetagg yeap of course.
DealResponse is a complex object ({ success: true/false, data: array of payload }) and has a field called data storing list of Participant objects. We just extracting first inner object (because in this case there is only one in array).

Also it is a good-old external service without json streams so we are always working with Mono objects as responses.

@indiv1d
Copy link
Author

indiv1d commented Jun 28, 2019

@violetagg and i have tried to enable DEBUG and it was a big mistake.
We are collecting logs into elastic cluster with pretty big rps it is impossible to find anything helpful in this unstoppable flow of logs :(

@violetagg
Copy link
Member

@1nDivid and in DealResponse you do not work with ByteBuf, correct?
about debugging instead of reactor.netty=DEBUG try only this reactor.netty.channel.FluxReceive=DEBUG

@indiv1d
Copy link
Author

indiv1d commented Jun 28, 2019

@violetagg we don't work directly with buffers at all. All calls look like .retrieve().bodyToMono...

@indiv1d
Copy link
Author

indiv1d commented Jun 29, 2019

@violetagg ok i think we found the problem.
We were throwing exception in onStatus handlers and this cause LEAK problem. We have changed throw to Mono.error(Throwable) and the problem have been fixed.

@rstoyanchev
Copy link
Contributor

rstoyanchev commented Jul 3, 2019

@1nDivid we have fixed some issue in the past for onStatus handlers that don't consume the body spring-projects/spring-framework#22005, but overlooked the fact the handler could throw the error vs returning it in an Mono as expected by the contract. Feel free to create a ticket in the Spring Framework. Thanks.

@rstoyanchev rstoyanchev changed the title LEAK: ByteBuf.release() was not called before it's garbage-collected Throwing an Exception from an onStatus handler in WebClient leads to a ByteBuf leak Jul 3, 2019
@violetagg violetagg added for/springframework This belongs to the Spring Framework project and removed for/user-attention This issue needs user attention (feedback, rework, etc...) labels Jul 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
for/springframework This belongs to the Spring Framework project
Projects
None yet
Development

No branches or pull requests

6 participants