Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure to recover from network interruption in event hub binder #537

Closed
chadshowalter opened this issue Jul 24, 2019 · 11 comments
Closed

Failure to recover from network interruption in event hub binder #537

chadshowalter opened this issue Jul 24, 2019 · 11 comments
Labels
azure-spring All azure-spring related issues Event Hub

Comments

@chadshowalter
Copy link

Expected Behavior

A spring cloud azure stream binder application is able to reconnect, and continue to receive events, after a network interruption occurs.

Current Behavior

When connected to an event hub, and the application loses connectivity to the hub, a ReceiverDisconnectedException may occur, and the application does not recover.

Steps to Reproduce

Create a Sink something like this:

@EnableBinding(Sink.class)
public class EventhubSink {

    private static final Logger LOGGER = LoggerFactory.getLogger(EventhubSink.class);


    @StreamListener(value = Sink.INPUT)
    public void handleEverything(String message, @Header("NotificationType") String notificationType) {
        LOGGER.info("New message received: {}: '{}'", notificationType, message);
    }
}

and configuration something like this:

spring.cloud.azure.eventhub.connection-string=<your connection string>
spring.cloud.azure.eventhub.checkpoint-storage-account=<your account>
spring.cloud.azure.eventhub.checkpoint-access-key=<your key>

spring.cloud.stream.bindings.input.destination=<input destination>
spring.cloud.stream.bindings.input.group=$Default
spring.cloud.stream.eventhub.bindings.input.consumer.checkpoint-mode=BATCH
spring.cloud.stream.eventhub.bindings.input.consumer.start-position=EARLIEST

using these gradle dependencies

dependencies {
   implementation('org.springframework.boot:spring-boot-starter-actuator')
   implementation('com.fasterxml.jackson.core:jackson-databind:2.9.7')
   implementation('com.microsoft.azure:spring-cloud-azure-eventhubs-stream-binder:1.1.0.RC5')
   
}

Cause a network interruption (e.g., I close the lid on my computer for a few minutes). When the application picks up where it left off, I get a stack trace something like this:

java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.ReceiverDisconnectedException: New receiver with higher epoch of '7' is created hence current receiver with epoch '7' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:b6d36b3a000057de002251345d38ad40_G3_B12, SystemTracker:cnrg-nonprod:eventhub:cnrg~16383|$default, Timestamp:2019-07-24T19:11:01, errorContext[NS: cnrg-nonprod.servicebus.windows.net, PATH: cnrg/ConsumerGroups/$Default/Partitions/0, REFERENCE_ID: cbe82b_ddb0_G3_1563995456749, PREFETCH_COUNT: 300, LINK_CREDIT: 600, PREFETCH_Q_LEN: 0]
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[na:na]
	at com.microsoft.azure.eventhubs.impl.ExceptionUtil.completeExceptionally(ExceptionUtil.java:98) ~[azure-eventhubs-2.2.0.jar:na]
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.drainPendingReceives(MessageReceiver.java:462) ~[azure-eventhubs-2.2.0.jar:na]
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onError(MessageReceiver.java:447) ~[azure-eventhubs-2.2.0.jar:na]
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:736) ~[azure-eventhubs-2.2.0.jar:na]
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:74) ~[azure-eventhubs-2.2.0.jar:na]
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:113) ~[azure-eventhubs-2.2.0.jar:na]
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48) ~[azure-eventhubs-2.2.0.jar:na]
	at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) ~[proton-j-0.31.0.jar:na]
	at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) ~[proton-j-0.31.0.jar:na]
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) ~[proton-j-0.31.0.jar:na]
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) ~[proton-j-0.31.0.jar:na]
	at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:507) ~[azure-eventhubs-2.2.0.jar:na]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: com.microsoft.azure.eventhubs.ReceiverDisconnectedException: New receiver with higher epoch of '7' is created hence current receiver with epoch '7' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:b6d36b3a000057de002251345d38ad40_G3_B12, SystemTracker:cnrg-nonprod:eventhub:cnrg~16383|$default, Timestamp:2019-07-24T19:11:01, errorContext[NS: cnrg-nonprod.servicebus.windows.net, PATH: cnrg/ConsumerGroups/$Default/Partitions/0, REFERENCE_ID: cbe82b_ddb0_G3_1563995456749, PREFETCH_COUNT: 300, LINK_CREDIT: 600, PREFETCH_Q_LEN: 0]
	at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:35) ~[azure-eventhubs-2.2.0.jar:na]
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:735) ~[azure-eventhubs-2.2.0.jar:na]
	... 14 common frames omitted

and then ultimately this:

2019-07-24 15:11:01.952  INFO 47779 --- [b58cc1fa6]-1-14] c.m.a.s.i.e.impl.EventHubProcessor       : Partition 1 is closing for reason Shutdown
2019-07-24 15:11:01.952  INFO 47779 --- [b58cc1fa6]-1-12] c.m.a.s.i.e.impl.EventHubProcessor       : Partition 0 is closing for reason Shutdown

The application doesn't exit, but event processing is shut down. I would like a way to recover from this error.

Your Environment

  • Version used: 1.1.0RC5
  • Operating System and version (desktop or mobile): OSX 10.14.5
@warrenzhu25 warrenzhu25 changed the title Failure to recover from network interruption Failure to recover from network interruption in event hub binder Jul 25, 2019
@warrenzhu25 warrenzhu25 mentioned this issue Jul 25, 2019
Closed
@pinnamaneni
Copy link

Hi,
Initially I was able to connect and process messages. After some time, I am continuously getting the error java.lang.IllegalStateException: There are no partitions in Event Hub xyz.
Version Used: 1.2.6

It might be related to issue Azure/azure-sdk-for-java#8439

@yiliuTo
Copy link
Member

yiliuTo commented Jun 20, 2020

Thanks for your reporting, according to your error log it seems to be the same as the issue you mentioned. Could you provide more infomation about how you use the event hub binder and the situation that the error occurs?

@pinnamaneni
Copy link

I do not see this error java.lang.IllegalStateException: There are no partitions in Event Hub xyz any more.

I am getting different error. This is my current environment.
No of Consumer Groups: 4
No of Partitions: 32
No of Instances running against a specific consumer group: 4
When the application starts the 32 partitions are assigned to 4 instances (8 Partitions each).
After 8-9 hours of consuming the messages getting following error. Similar error for all the partitions.

2020-06-21T12:11:01.12-0400 [APP/PROC/WEB/0] OUT 2020-06-21 16:11:01.121 [INFO ] 23 --- [or-http-epoll-2] c.m.a.s.i.e.impl.EventHubProcessor : Started receiving on partition: 30
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT 2020-06-21 16:11:06.364 [ERROR ] 15 --- [ elastic-320] c.m.a.s.i.e.impl.EventHubProcessor : Error occurred on partition: 30. Error: {}
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT com.azure.core.amqp.exception.AmqpException: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:8787b8d80004053e003632125eef864e_G5S3_B7S3, SystemTracker:test-event-hub-np:eventhub:test-event-hub-np-32~31143|eventhub-test-consumer, Timestamp:2020-06-21T16:11:06, errorContext[NAMESPACE: test-event-hub-np.servicebus.windows.net, PATH: test-event-hub-np-32/ConsumerGroups/eventhub-test-consumer/Partitions/30, REFERENCE_ID: 30_d07581_1592755790248, LINK_CREDIT: 1]
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.ExceptionUtil.toException(ExceptionUtil.java:77)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.handler.LinkHandler.processOnClose(LinkHandler.java:90)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.handler.LinkHandler.handleRemoteLinkClosed(LinkHandler.java:108)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.handler.LinkHandler.onLinkRemoteClose(LinkHandler.java:50)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.handler.ReceiveLinkHandler.onLinkRemoteClose(ReceiveLinkHandler.java:17)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:82)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:70)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-06-21T12:11:06.36-0400 [APP/PROC/WEB/2] OUT at java.lang.Thread.run(Thread.java:748)

@pinnamaneni
Copy link

Noticing following error from one of the instance, remaining instances runs without any errors.

2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT 2020-06-22 14:12:15.567 [ERROR ] 24 --- [ parallel-3] c.m.a.s.i.e.impl.EventHubProcessor : Error occurred on partition: NONE. Error: {}
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 60000ms in 'lift' (and no fallback has been configured)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:289)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:274)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:396)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:70)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-06-22T10:12:15.56-0400 [APP/PROC/WEB/2] OUT at java.lang.Thread.run(Thread.java:748)

@yiliuTo
Copy link
Member

yiliuTo commented Jun 23, 2020

Hi @pinnamaneni , according to the error log, connection to event hub has interrupted for some unknown reason and failed to recover. After refering the issue you mentioned, we update the version of azure-messaging-eventhubs to 5.1.1 and azure-core to 1.5.1, and we can see the reconnection succeed in our test case that interrupt network manually and recover it after a while. However, given that we would't release to next version right away, could you try the version updating as well to see whether your issue would be resolved?

@pinnamaneni
Copy link

Hi @yiliuTo , I tried with the updated libs. Getting following errors.

2020-06-23T13:19:09.16-0400 [APP/PROC/WEB/0] OUT 2020-06-23 17:19:09.163 [INFO ] 14 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T13:19:09.54-0400 [APP/PROC/WEB/1] OUT 2020-06-23 17:19:09.542 [INFO ] 22 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T13:19:11.11-0400 [APP/PROC/WEB/2] OUT 2020-06-23 17:19:11.113 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T13:19:11.72-0400 [APP/PROC/WEB/3] OUT 2020-06-23 17:19:11.726 [INFO ] 14 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT 2020-06-23 17:19:13.145 [ERROR ] 14 --- [or-http-epoll-1] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT Recent access records:
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT Created at:
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2134)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1319)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1231)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1268)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-23T13:19:13.14-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748) []

2020-06-23T16:01:27.47-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:01:27.476 [INFO ] 14 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Starting load balancer for b2b654e6-4251-4dbc-8023-53dc8c5cc92f []
2020-06-23T16:01:30.04-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:01:30.034 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:32.66-0400 [APP/PROC/WEB/1] OUT 2020-06-23 20:01:32.664 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:33.51-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:01:33.513 [INFO ] 15 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:37.47-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:01:37.477 [INFO ] 14 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:40.57-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:01:40.569 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:42.66-0400 [APP/PROC/WEB/1] OUT 2020-06-23 20:01:42.664 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:44.30-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:01:44.304 [INFO ] 15 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:47.47-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:01:47.478 [INFO ] 14 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:51.48-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:01:51.488 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:52.66-0400 [APP/PROC/WEB/1] OUT 2020-06-23 20:01:52.664 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:54.30-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:01:54.306 [INFO ] 15 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:57.34-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:01:57.344 [INFO ] 23 --- [or-http-epoll-1] c.a.m.e.PartitionBasedLoadBalancer : Successfully claimed ownership of partition 22
2020-06-23T16:01:57.37-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:01:57.374 [INFO ] 23 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Successfully claimed ownership of partition 11
2020-06-23T16:01:57.41-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:01:57.410 [INFO ] 23 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Successfully claimed ownership of partition 23
2020-06-23T16:01:57.42-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:01:57.415 [INFO ] 23 --- [or-http-epoll-1] c.a.m.e.PartitionBasedLoadBalancer : Successfully claimed ownership of partition 21
2020-06-23T16:01:57.61-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:01:57.614 [INFO ] 14 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:01:58.90-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:01:58.902 [INFO ] 23 --- [or-http-epoll-2] c.a.m.e.PartitionBasedLoadBalancer : Successfully claimed ownership of partition 10
2020-06-23T16:02:02.31-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:02:02.311 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:02:02.72-0400 [APP/PROC/WEB/1] OUT 2020-06-23 20:02:02.729 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:02:05.50-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:05.505 [INFO ] 15 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:02:05.92-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:05.926 [INFO ] 14 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : CheckpointStore returned 32 ownership records
2020-06-23T16:02:05.92-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:05.927 [INFO ] 14 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Event Hubs service returned 32 partitions
2020-06-23T16:02:05.92-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:05.928 [INFO ] 14 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Number of active ownership records 5
2020-06-23T16:02:05.92-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:05.929 [INFO ] 14 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Number of active event processors 2
2020-06-23T16:02:05.92-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:05.929 [INFO ] 14 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Expected min partitions per event processor = 16, expected number of event processors with additional partition = 0
2020-06-23T16:02:05.93-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:05.930 [INFO ] 14 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Load is unbalanced and this event processor owns 0 partitions and should own more partitions
2020-06-23T16:02:05.94-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:05.944 [INFO ] 14 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Attempting to claim ownership of partition 20
2020-06-23T16:02:05.94-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:05.945 [INFO ] 14 --- [or-http-epoll-3] c.a.m.e.PartitionBasedLoadBalancer : Load balancing completed successfully
2020-06-23T16:02:08.66-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:08.654 [INFO ] 14 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:02:12.79-0400 [APP/PROC/WEB/1] OUT 2020-06-23 20:02:12.789 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:02:12.84-0400 [APP/PROC/WEB/3] OUT 2020-06-23 20:02:12.844 [INFO ] 23 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:02:16.50-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:16.505 [INFO ] 15 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:02:19.96-0400 [APP/PROC/WEB/0] OUT 2020-06-23 20:02:19.959 [INFO ] 14 --- [pool-7-thread-1] c.a.m.e.PartitionBasedLoadBalancer : Load balancer already running []
2020-06-23T16:02:22.54-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.547 [INFO ] 15 --- [ single-1] c.a.c.a.i.handler.ReceiveLinkHandler : onLinkRemoteClose connectionId[MF_f86e4b_1592942203264], linkName[20_eaef9c_1592942203267], errorCondition[amqp:link:stolen], errorDescription[New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:c6bdcb8b0004053d00003_G5_B3S1, SystemTracker:XXXXXXXX|YYYYYYY, Timestamp:2020-06-23T20:01:30]
2020-06-23T16:02:22.54-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.548 [INFO ] 15 --- [ single-1] c.a.c.a.i.handler.ReceiveLinkHandler : processOnClose connectionId[MF_f86e4b_1592942203264], linkName[20_eaef9c_1592942203267], errorCondition[amqp:link:stolen], errorDescription[New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:c6bdcb8b0004053d00003_G5_B3S1, SystemTracker:XXXXXXXX|YYYYYYY, Timestamp:2020-06-23T20:01:30]
2020-06-23T16:02:22.55-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.551 [ERROR ] 15 --- [ single-1] c.a.c.a.implementation.ReactorReceiver : connectionId[MF_f86e4b_1592942203264] linkName[20_eaef9c_1592942203267] entityPath[xyz-32/ConsumerGroups/YYYYYYY/Partitions/20] Error occurred in link.
2020-06-23T16:02:22.55-0400 [APP/PROC/WEB/2] OUT New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:c6bdcb8b0004053d00003_G5_B3S1, SystemTracker:XXXXXXXX|YYYYYYY, Timestamp:2020-06-23T20:01:30, errorContext[NAMESPACE: abc.servicebus.windows.net, PATH: xyz-32/ConsumerGroups/YYYYYYY/Partitions/20, REFERENCE_ID: 20_eaef9c_1592942203267, LINK_CREDIT: 0]
2020-06-23T16:02:22.55-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.552 [INFO ] 15 --- [ single-1] c.a.c.a.implementation.ReactorSession : linkName[20_eaef9c_1592942203267] entityPath[xyz-32/ConsumerGroups/YYYYYYY/Partitions/20]: Error occurred. Removing receive link.
2020-06-23T16:02:22.55-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.554 [WARN ] 15 --- [ single-1] c.a.m.e.i.AmqpReceiveLinkProcessor : linkName[20_eaef9c_1592942203267] entityPath[xyz-32/ConsumerGroups/YYYYYYY/Partitions/20]. Error occurred in link.
2020-06-23T16:02:22.55-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.554 [INFO ] 15 --- [ single-1] c.a.m.e.i.AmqpReceiveLinkProcessor : Error on receive link null
2020-06-23T16:02:22.55-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.558 [INFO ] 15 --- [ single-1] c.a.c.a.i.handler.ReceiveLinkHandler : onLinkLocalClose connectionId[MF_f86e4b_1592942203264], linkName[20_eaef9c_1592942203267], errorCondition[amqp:link:stolen], errorDescription[New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:c6bdcb8b0004053d00003_G5_B3S1, SystemTracker:XXXXXXXX|YYYYYYY, Timestamp:2020-06-23T20:01:30]
2020-06-23T16:02:22.56-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.563 [INFO ] 15 --- [ elastic-2] c.a.m.e.EventHubConsumerAsyncClient : linkName[20_eaef9c_1592942203267], partitionId[20], signal[cancel]: Receiving completed.
2020-06-23T16:02:22.59-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.564 [INFO ] 15 --- [ elastic-2] c.a.m.e.EventHubPartitionAsyncConsumer : Closed consumer for partition 20
2020-06-23T16:02:22.59-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.564 [WARN ] 15 --- [ elastic-2] c.a.m.eventhubs.PartitionPumpManager : Error receiving events from partition 20
2020-06-23T16:02:22.59-0400 [APP/PROC/WEB/2] OUT New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:c6bdcb8b0004053d00003_G5_B3S1, SystemTracker:XXXXXXXX|YYYYYYY, Timestamp:2020-06-23T20:01:30, errorContext[NAMESPACE: abc.servicebus.windows.net, PATH: xyz-32/ConsumerGroups/YYYYYYY/Partitions/20, REFERENCE_ID: 20_eaef9c_1592942203267, LINK_CREDIT: 0]
2020-06-23T16:02:22.59-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.565 [INFO ] 15 --- [ single-1] c.a.c.a.i.handler.ReceiveLinkHandler : onLinkFinal connectionId[MF_f86e4b_1592942203264], linkName[20_eaef9c_1592942203267]
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT 2020-06-23 20:02:22.594 [ERROR ] 15 --- [ elastic-2] c.m.a.s.i.e.impl.EventHubProcessor : Error occurred on partition: 20. Error: {}
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT com.azure.core.amqp.exception.AmqpException: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:c6bdcb8b0004053d00003_G5_B3S1, SystemTracker:XXXXX|YYYYYYY, Timestamp:2020-06-23T20:01:30, errorContext[NAMESPACE: abc.servicebus.windows.net, PATH: xyz-32/ConsumerGroups/YYYYYYY/Partitions/20, REFERENCE_ID: 20_eaef9c_1592942203267, LINK_CREDIT: 0]
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.ExceptionUtil.toException(ExceptionUtil.java:78)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.handler.LinkHandler.processOnClose(LinkHandler.java:90)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.handler.LinkHandler.handleRemoteLinkClosed(LinkHandler.java:108)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.handler.LinkHandler.onLinkRemoteClose(LinkHandler.java:50)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.handler.ReceiveLinkHandler.onLinkRemoteClose(ReceiveLinkHandler.java:18)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:82)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:70)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-06-23T16:02:22.60-0400 [APP/PROC/WEB/2] OUT at java.lang.Thread.run(Thread.java:748)

@yiliuTo
Copy link
Member

yiliuTo commented Jun 24, 2020

Hi @pinnamaneni , first I am sorry that I forgot to remind you updating another library azure-messaging-eventhubs-checkpointstore-blob to 1.1.1 which is supposed to update with azure-messaging-eventhubs synchronously. However, your issue seems not just about reconnection but also related with this issue. We will investigate it since this bug has already been fixed in SDK repo.

@pinnamaneni
Copy link

pinnamaneni commented Jun 24, 2020 •
edited
Loading

Hi @yiliuTo

Spring Boot: 2.3.0.RELEASE
Azure Event Hub:
implementation 'com.microsoft.azure:spring-cloud-azure-eventhubs-stream-binder:1.2.6'
constraints {
implementation('com.azure:azure-messaging-eventhubs:5.1.1') {
because 'previous versions have a bug impacting this application'
}
implementation('com.azure:azure-messaging-eventhubs-checkpointstore-blob:1.1.1') {
because 'previous versions have a bug impacting this application'
}
}

Getting following error:

2020-06-24T14:33:25.71-0400 [APP/PROC/WEB/1] OUT 2020-06-24 18:33:25.712 [ERROR ] 15 --- [or-http-epoll-3] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
2020-06-24T14:33:25.71-0400 [APP/PROC/WEB/1] OUT Recent access records:
2020-06-24T14:33:25.71-0400 [APP/PROC/WEB/1] OUT Created at:
2020-06-24T14:33:25.71-0400 [APP/PROC/WEB/1] OUT io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
2020-06-24T14:33:25.71-0400 [APP/PROC/WEB/1] OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)

@pinnamaneni
Copy link

Hi @yiliuTo

Enabled advanced leak report which might be helpful for you.

2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT 2020-06-24 20:10:39.269 [ERROR ] 16 --- [or-http-epoll-4] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT Recent access records:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT #1:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT Hint: 'reactor.right.reactiveBridge' will handle the message from this point.
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.DefaultHttpContent.touch(DefaultHttpContent.java:86)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:88)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.DefaultLastHttpContent.touch(DefaultLastHttpContent.java:28)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1279)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1316)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT #2:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:285)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1279)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1316)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT #3:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:872)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:572)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:214)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1279)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1316)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT #4:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:872)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:552)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:214)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1279)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1316)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT #5:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT Hint: 'reactor.left.httpCodec' will handle the message from this point.
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1279)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1316)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT #6:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1528)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:167)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:207)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1279)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1316)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT #7:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1528)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:167)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:207)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1279)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1316)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT Created at:
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2188)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1367)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1279)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1316)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT java.lang.Thread.run(Thread.java:748)
2020-06-24T16:10:39.27-0400 [APP/PROC/WEB/0] OUT : 10 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit. []

@yiliuTo
Copy link
Member

yiliuTo commented Jun 28, 2020

Hi @pinnamaneni sorry for the late response due to a local festival. Considering your comment on issue 709, we will invesigate these two issues together.

@saragluna saragluna added the azure-spring All azure-spring related issues label Jul 22, 2020
@chenrujun
Copy link

Closing this issue.
Because it's not active for a long time.
If anyone have similar issue, please create issue in new repo: https://github.com/Azure/azure-sdk-for-java/issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
azure-spring All azure-spring related issues Event Hub
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@warrenzhu25 @chadshowalter @pinnamaneni @chenrujun @saragluna @yiliuTo