Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support nested flieds in CSV format #2630

Closed
Horkyze opened this issue Sep 26, 2019 · 2 comments · Fixed by #2643
Closed

Support nested flieds in CSV format #2630

Horkyze opened this issue Sep 26, 2019 · 2 comments · Fixed by #2643
Labels
enhancement Feature request or improve operations

Comments

@Horkyze
Copy link

Horkyze commented Sep 26, 2019

Consider the following record:

 {
    "hostname":"buster",
    "priority":"6",
    "selinux_context":"unconfined\n",
    "pid":"4866",
    "image_name":"nginx",
    "container_name":"fluentd-docker_nginx_1",
    "container_id":"109e52fb8de8",
    "message":{
       "remote":"172.20.0.1",
       "host":"-",
       "user":"-",
       "method":"GET",
       "path":"/",
       "code":"200",
       "size":"612",
       "referer":"-",
       "agent":"curl/7.64.0",
       "http_x_forwarded_for":"-"
    },
    "source_realtime_timestamp":"1569258331312517"
 }

I would be nice to access the flieds in message fields in CSV format

  <filter **>
    @type parser
    key_name message
    reserve_data true
    hash_value_field message
    <parse>
      @type regexp
      expression /^(?<method>[^ ]*) (?<path>[^ ]*) (?<code>[0-9]*) (?<ms>[^ ]*) ms - (?<bytes_out>[0-9-]*)$/
    </parse>
  </filter>

  <match **>
    #@type file
    @type stdout

    #path /tmp/access.log
    <format>
      @type csv
      force_quotes false
      delimiter " "
      fields container_tag,message['method']
    </format>
  </match>

I have tried message['method'] message.method and message[:method] none of them works

@repeatedly
Copy link
Member

I have tried message['method'] message.method and message[:method] none of them works

Because ruby's CSV module doesn't support nested field.
To support nested field, need to create target array before passing to CSV module.

@repeatedly repeatedly added the enhancement Feature request or improve operations label Oct 8, 2019
repeatedly added a commit that referenced this issue Oct 9, 2019
@repeatedly
Copy link
Member

Patch: #2643

repeatedly added a commit that referenced this issue Oct 11, 2019
formatter_csv: support nested fields. fix #2630
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature request or improve operations
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants