GeflUdpAppender throws ClosedChannelException

[pfeiler]

Hi, when logging to Graylog with logback-gelf-1.1.0.jar it stops logging after a minute.
By debugging I see that gelf throws a ClosedChannelException. Maybe gelf needs to check the state of channel and reopen if closed?:

[Ljava.lang.StackTraceElement;@39e25fb8java.nio.channels.ClosedChannelException
	at sun.nio.ch.DatagramChannelImpl.ensureOpen(DatagramChannelImpl.java:320)
	at sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:428)
	at de.siegmar.logbackgelf.GelfUdpAppender.appendMessage(GelfUdpAppender.java:81)
	at de.siegmar.logbackgelf.AbstractGelfAppender.append(AbstractGelfAppender.java:106)
	at de.siegmar.logbackgelf.AbstractGelfAppender.append(AbstractGelfAppender.java:28)
	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
	at ch.qos.logback.classic.Logger.filterAndLog_2(Logger.java:414)
	at ch.qos.logback.classic.Logger.info(Logger.java:587)

[osiegmar]

From my understanding this can only happen if the DatagramChannel has been closed by the method de.siegmar.logbackgelf.GelfUdpAppender#close as the channel itself is not connected to a remote system. This close method is called by logback when it is stopping (commonly because of shutting down the whole application). Could that be the case in your scenario?

Is the exception thrown all the times after this first minute or does it stop at some point?

Furthermore, the whole stack trace would be interesting. It might contain additional information...

[pfeiler]

Thanks for having a look into this.

• Exception is thrown with every log statement after channel has closed.
• The channel closes always at the same log statement, why I don't know.
• The next entries in the stacktrace are from classes from our application (can't disclose).

I googled a bit, would you please have a look at:
t0xa/gelfj#87 (comment) and t0xa/gelfj@15842b8

[osiegmar]

Ok, I need some more time to investigate. I tried to reproduce this by shutting down the graylog server and starting it again, but as expected it did not result in any exception. What operating system and Java version do you use with logback-gelf?

[osiegmar]

Cannot reproduce here. Can you try the branch https://github.com/osiegmar/logback-gelf/tree/udp-reopen (version 1.1.1-SNAPSHOT)? I'm testing for isOpen() here and reopen the channel if required.

[etluchs]

similar problem here. Only it appears after quite long time (weeks) and only sporadic. Will try the "robust" version ...

[osiegmar]

Did the udp-reopen branch solved the issue for someone?

[pfeiler]

I am sorry, I could not test it. We switched to TCP which works with no problems so far. Please close and reopen when needed. Oliver Siegmar <[email protected]> schrieb am Fr., 28. Sep. 2018, 20:48:

…

Did the udp-reopen branch solved the issue for someone? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#20 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOfj0Ad1nL_zlB2UsIggrSnEgMdGyVTRks5ufmzkgaJpZM4VOvje> .

[etluchs]

Similar here :-( sorry. For the installation in question, we switched to logging to stdout and let docker the forwarding to graylog.
We may still use it for one of the non-docker environments though ...

[osiegmar]

Please reopen this ticket if someone can test the branch

[Rustam-Kadyrov]

Had similar problem with version 1.x under docker container. After certain activities GELF appender stopped sending messages to Graylog with exception: java.nio.channels.ClosedChannelException.
Unfortunately I have no chance to check your branch on prod.

[hlzhang]

Had similar problem with version 1.x under docker container. After certain activities GELF appender stopped sending messages to Graylog with exception: java.nio.channels.ClosedChannelException.
Unfortunately I have no chance to check your branch on prod.

+1

[skreuzhuber]

We had the very same issue and have tested your branch successfully.

We had lost GELF messages after an uptime of about 1-2 days, e.g. no more GELF messages were sent by our applications. In our log files we have encountered the stated ClosedChannelException. We use the GELF Appender within various Spring Boot applications that are deployed as Docker containers and send logs via GELF to Graylog.

This issue was solved by building and including your fix (https://github.com/osiegmar/logback-gelf/tree/udp-reopen). Since about 2 weeks no GELF messages were lost on various environments.

Thanks a lot for your effort. We would kindly appreciate if you merge the fix into the master and include it within one of your next releases. Thanks!

[osiegmar]

@skreuzhuber Thanks for your test and for confirming, that the udp-reopen branch fixes the problem!