config.rails_semantic_logger.filter doesn't apply to initialized Loggers

[fractaledmind]

Environment

Provide at least:

• Ruby Version: 3.0.1
• Rails Version: 6.1.3.2
• Semantic Logger Version: 4.8.0
• Rails Semantic Logger Version: 4.6.0
• Other Application/framework names and versions
  • puma: 5.3.2
• Rails configuration. Only need the settings related to Rails Semantic Logger and Semantic Logger.

config/application.rb

config.rails_semantic_logger.started    = true
config.rails_semantic_logger.processing = true
config.rails_semantic_logger.rendered   = true
config.log_tags = {
  request_id: :uuid,
  ip:         :remote_ip,
}
config.rails_semantic_logger.quiet_assets = true
config.rails_semantic_logger.filter = Proc.new { |log| log.message !~ /Webhooks::GithubController/ }

config/environments/production.rb

if ENV["RAILS_LOG_TO_STDOUT"].present?
  $stdout.sync = true
  config.rails_semantic_logger.add_file_appender = false
  config.semantic_logger.add_appender(
    io: $stdout,
    level: config.log_level,
    formatter: config.rails_semantic_logger.format,
    filter: Proc.new { |log| log.message !~ /Webhooks::GithubController/ }
  )
end

# The log level is usually set with the config setting config.log_level,
# but Heroku also allows the log level to be set via the LOG_LEVEL env variable.
if ENV["LOG_LEVEL"].present?
  config.log_level = ENV["LOG_LEVEL"].downcase.strip.to_sym
end

I have a Rails app running on Heroku using the Papertrail service for logs. My Rails app is integrated with a GitHub app, so I receive GitHub webhooks. I store the webhook payloads in the database for processing in a background job, and they are large and often, so they fill up my logging bucket fairly easily. I am trying to exclude the logs of the GitHub webhook requests. You can see the filter I am trying to use above, but I still see the webhook requests in my logs.

How can I exclude logs in Heroku production that match my regex?

[danielwellman]

Hello, I use Semantic Logger with Heroku and Papertrail - I don't know why the code you posted isn't working, but I can share what we do -- you may already be doing something like this.

We use the feature in Papertrail to filter out log messages matching certain regular expressions. It's in the Papertrail to menu > Settings > Account page and then the "Filter logs" button. From there you specify the regular expressions that should be omitted when matched, so you might be able to add "Webhooks::GithubController" as a rule.

Here are the docs:
https://documentation.solarwinds.com/en/success_center/papertrail/content/kb/how-it-works/log-filtering.htm

From what I can see, it looks like you have the syntax correct for the filter argument to add_appender; I'm looking at the documentation comment here:

https://github.com/reidmorrison/semantic_logger/blob/3f01cbb64fb8c1bbca14401703d81287399151ec/lib/semantic_logger/semantic_logger.rb#L132-L136

[reidmorrison]

Looking at the filter above, looks to me like you want to filter on the class name, not the message.

filter: Proc.new { |log| log.name != "Webhooks::GithubController" }

The log object has many elements that can be filtered on:
https://github.com/reidmorrison/semantic_logger/blob/master/lib/semantic_logger/log.rb#L13

[fractaledmind]

@reidmorrison: I have found the source of the problem. The filter that I set in my configuration is never passed to the initialised loggers when the Engine is mounted.

Consider

rails_semantic_logger/lib/rails_semantic_logger/engine.rb

Lines 45 to 71 in c94ae1f

	Rails.logger = config.logger =
	begin
	if config.rails_semantic_logger.add_file_appender
	path = config.paths["log"].first
	FileUtils.mkdir_p(File.dirname(path)) unless File.exist?(File.dirname(path))
	# Add the log file to the list of appenders
	# Use the colorized formatter if Rails colorized logs are enabled
	ap_options = config.rails_semantic_logger.ap_options
	formatter = config.rails_semantic_logger.format
	formatter = {color: {ap: ap_options}} if (formatter == :default) && (config.colorize_logging != false)
	# Set internal logger to log to file only, in case another appender experiences errors during writes
	appender = SemanticLogger::Appender::File.new(path, formatter: formatter)
	appender.name = "SemanticLogger"
	SemanticLogger::Processor.logger = appender
	# Check for previous file or stdout loggers
	SemanticLogger.appenders.each do |app|
	next unless app.is_a?(SemanticLogger::Appender::File) || app.is_a?(SemanticLogger::Appender::IO)
	app.formatter = formatter
	end
	SemanticLogger.add_appender(file_name: path, formatter: formatter, filter: config.rails_semantic_logger.filter)
	end
	SemanticLogger[Rails]

When SemanticLogger[Rails] is called, there is no way or attempt at connecting the filter created for config.rails_semantic_logger.filter. It is only used if the config.rails_semantic_logger.add_file_appender is passed and then the filter is passed to the appender.

I currently am just brute force hacking around this with an initializer:

ObjectSpace.each_object(SemanticLogger::Logger).each do |logger|
  logger.filter = Rails.application.config.rails_semantic_logger.filter
end

[fractaledmind]

To clarify, because Heroku requires logging to STDOUT (https://logger.rocketjob.io/rails#log-to-standard-out) and thus requires disabling the file appender, currently any of the options passed and used to configure the appender are ignored