Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use block to output logs due to reduce memory usage #17

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Watson1978
Copy link

The many long duplicated logs seem to use a lot of memory at record.to_json in log.debug.

log.debug "suppressed record: #{record.to_json}"

log.debug "log message" does not output the message with info log level.
However, the message have been evaluated.

If passes the block instead, log.debug will evaluate the block only when output the message.

Related to fluent/fluentd#4808

Reproduce

Fluentd configuration

<source>
  @type tail
  path "#{File.expand_path '~/tmp/access*.log'}"
  pos_file "#{File.expand_path '~/tmp/fluentd/access.log.pos'}"
  tag log
  <parse>
    @type none
  </parse>
</source>

<filter log.**>
    @type suppress
    log_suppress_interval 10
    num 1
    max_slot_num 10
    attr_keys short_message
</filter>

<match **>
  @type stdout
</match>

Log generate script

require "json"

path = File.expand_path("~/tmp/access.log")
File.open(path, "w") do |f|
  log = { "message": "a" * 500_000 }.to_json
  loop do
    f.puts log
    sleep 0.0001
  end
end

Result

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant