Every log line sent to ES create new index

[flodumi]

Hello,
I'm using fluentd to send pod logs to ES 8.5

My problem is that each log sent is creating a new index, so i end up having a very large number of indices.

What I'd like to have is one index per day, and keep them for only 7 days.

this is the config i have for the index name:
index_name odc-dev-new

And what i see in kibana are a lot of indexes created like this:

odc-dev-new-1970.02.10
odc-dev-new-1970.02.23
odc-dev-new-1970.03.12

this is the config i'm using to send the logs:

<source>
  @type tail
  #@log_level debug
  path /var/log/containers/*.log
  exclude_path /var/log/containers/*fluent*.log
  pos_file /tmp/es-containers.log.pos
  pos_file_compaction_interval 72h
  tag kubernetes.*
  read_from_head false
  format json
  refresh_interval 3
  #keep_time_key true
  time_format %Y-%m-%dT%H:%M:%S.%sZ
</source>

<filter kubernetes.**>
  @type stdout
  output_type json
  key_name log
  reserve_time true
  reserve_data true
  remove_key_name_field true
  time_key time
</filter>

For index name i have:
index_name odc-dev-new

What am I missing in my config ?

thank you,
Florin

[fujimotos]

And what i see in kibana are a lot of indexes created like this:
odc-dev-new-1970.02.10

This is a logstash-format index. The part YYYY-MM-DD is filled
by Fluentd using the timestamp of each message.

My guess is that you're teaching out_elasticsarch to look at a wrong time field.
Try removing time_key option from your config and see if it works:

<filter kubernetes.**>
  @type stdout
  output_type json
  key_name log
  reserve_time true
  reserve_data true
  remove_key_name_field true
  #time_key time
</filter>

[flodumi]

That was it.
Thanks a lot for the help!