LogstashEncoder own pattern definition

[floruschbaschan]

While working with logback-core as well as logback-classic version 1.2.3 we tried to change the default log pattern. Our first approach was to use the <pattern> in the LogstashEncoder. Thereby we run into the following problem:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
	<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
		<encoder class="net.logstash.logback.encoder.LogstashEncoder">
			<pattern>${OUR_OWN_LOG_PATTERN}</pattern>
		</encoder>
	</appender>
</configuration>


Results in the following exception:
Exception in thread "main" java.lang.IllegalStateException: Logback configuration error detected: 
ERROR in ch.qos.logback.core.joran.spi.Interpreter@43:22 - no applicable action for [pattern], current ElementPath  is [[configuration][appender][encoder][pattern]]
ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - Could not find an appender named [CONSOLE]. Did you define it below instead of above in the configuration file?
ERROR in ch.qos.logback.core.joran.action.AppenderRefAction - See http://logback.qos.ch/codes.html#appender_order for more details.
...

Another approach we tried was to define the PatternLayout for the the LayoutWrappingEncoder:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
	<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
		<encoder class="net.logstash.logback.encoder.LogstashEncoder">
			<prefix class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
				<layout class="ch.qos.logback.classic.PatternLayout">
					<pattern>${OUR_OWN_LOG_PATTERN}</pattern>
				</layout>
			</prefix>
		</encoder>
	</appender>
</configuration>


Results in the following exception:
Exception in thread "main" java.lang.IllegalStateException: Logback configuration error detected: 
ERROR in ch.qos.logback.core.joran.util.PropertySetter@51399530 - A "net.logstash.logback.encoder.LogstashEncoder" object is not assignable to a "ch.qos.logback.core.Appender" variable.
ERROR in ch.qos.logback.core.joran.util.PropertySetter@51399530 - The class "ch.qos.logback.core.Appender" was loaded by 
ERROR in ch.qos.logback.core.joran.util.PropertySetter@51399530 - [sun.misc.Launcher$AppClassLoader@18b4aac2] whereas object of type 
ERROR in ch.qos.logback.core.joran.util.PropertySetter@51399530 - "net.logstash.logback.encoder.LogstashEncoder" was loaded by [sun.misc.Launcher$AppClassLoader@18b4aac2].
...

As some GitHub issues (#158, #232) already suggest, a version downgrade of logback-core and logback-classic to version 1.2.0 will fix it. However, a downgrade was not acceptable for us and we noticed that the order of elements defined in the appender seems to matter. We managed to define our own log pattern for the LogstashEncoder by one of the following configurations:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
	<!-- Option 1 - Multiple <encoder> -->
	<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
		<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
		<encoder>
			<pattern>${OUR_OWN_LOG_PATTERN}</pattern>
		</encoder>
	</appender>

	<!-- Option 2 - Using <layout> -->
	<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
		<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
		<layout class="ch.qos.logback.classic.PatternLayout">
			<pattern>${OUR_OWN_LOG_PATTERN}</pattern>
		</layout>
	</appender>

	<!-- Option 3 - Using LoggingEventCompositeJsonEncoder -->
	<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
		<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
			<providers>
				<timestamp/>
				<version/>
				<pattern>
					<pattern>
						{
						"logger": "%logger",
						"level": "%level",
						"thread": "%thread",
						"message": "%message",
						"traceId": "%mdc"
						}
					</pattern>
				</pattern>
			</providers>
		</encoder>
	</appender>
</configuration>

Defining the log pattern before the LogstashEncoder, results in the default log pattern. Therefore we would assume that the order is important here. These approaches would allow us to customize the log pattern definition without the need to downgrade any versions of the logback dependencies.

However, all examples I have seen are using only one encoder and put LogstashEncoder right at the end before </appender>. We are not sure if there are any disadvantages for one of the options above. Any help/advice on this would be appreciated.

[philsttr]

Can you provide an example of the output you are trying to achieve?

Are you trying to achieve a prefix before the JSON output? or are you trying to change the JSON output itself?

Logback appenders only accept one encoder or layout. So, your option 1 and 2 are invalid logback configurations.

[floruschbaschan]

Are you trying to achieve a prefix before the JSON output? or are you trying to change the JSON output itself?

Yes, our intention was to define a specific JSON output format.

Logback appenders only accept one encoder or layout. So, your option 1 and 2 are invalid logback configurations.

I see, thanks for pointing that out.

While working with Docker we defined a specific log driver and configured ch.qos.logback.classic.PatternLayout within ch.qos.logback.core.ConsoleAppender which makes LogstashEncoder in our case unnecessary.

Nevertheless, option 3 could be used if a pattern definition of LogstashEncoder is still required.

But once again, thanks for your fast reply and support :-)