Document change to LS index setting in Breaking Changes

[andrewkroh]

Users that send data from Beats to LS need to update their LS configuration to specify a new index name that includes the Beat version. Previously we recommended

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
}

And in 6.0 we updated to:

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
}

The index templates we supply all apply to [beat]-[version]-* so if you don't update LS the template won't be applied.

[kylegordon]

This is still an issue in the documentation

https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_literal_index_literal
https://www.elastic.co/guide/en/beats/metricbeat/current/elasticsearch-output.html#_literal_index_literal
https://www.elastic.co/guide/en/beats/winlogbeat/current/elasticsearch-output.html#_literal_index_literal
etc etc

[ruflin]

@dedemorton Not sure if this was backported yet?

[dedemorton]

@ruflin No, the PR has not been backported yet. However the topics that Kyle points to are about the Elasticsearch output config, not the Logstash configuration, which this issue focuses on.

I do think a small update of the ES output config docs is still required. We should show that the default index name is "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}" correct?

But I'm not sure if we want to recommend that users include the beat version when they define a custom index name. WDYT?

[ruflin]

We should recommend to include the beats version as otherwise in case we do bug fixes to the template and they update the version, they would not automatically get the newest template.

[dedemorton]

Note that #5841 fixes the remaining examples to include the version info in the index name.