Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load index templates v2 by default when talking to ES 7.16 or ES 8.x #28538

Merged
merged 11 commits into from
Oct 21, 2021
Merged

Load index templates v2 by default when talking to ES 7.16 or ES 8.x #28538

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
211f0ec
Load index templates by default && deprecation warning
kvch Oct 19, 2021
61e4395
add to reference config
kvch Oct 19, 2021
d5f555e
add changelog entry
kvch Oct 19, 2021
55418d9
Follow up changes in E2E tests
kvch Oct 20, 2021
faa2f28
add more details to changelog
kvch Oct 20, 2021
dc6ad85
fix significant amount of tests
kvch Oct 20, 2021
61cdd11
increase metricbeat timeout
kvch Oct 20, 2021
f4c33a4
rm legacy ref
kvch Oct 20, 2021
86847de
rm legacy ref
kvch Oct 20, 2021
a6e188d
Merge remote-tracking branch 'upstream/7.16' into forward-compatibili…
kvch Oct 20, 2021
178a273
more follow up
kvch Oct 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix panic with inline SSL when the certificate or key were small than 256 bytes. {pull}23820[23820]
- Skip add_kubernetes_metadata processor when kubernetes metadata are already present {pull}27689[27689]
- Enable IMDSv2 support for `add_cloud_metadata` processor on AWS. {issue}22101[22101] {pull}28285[28285]
- Load index templates v2 by default when talking to ES 7.16 or ES 8.x {pull}28538[28538]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a note here that it also means template can only be loaded for Elasticsearch versions newer 7.8 (I think that is when the index templates were introduced).


*Auditbeat*

Expand Down
4 changes: 2 additions & 2 deletions auditbeat/auditbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1206,8 +1206,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default auditbeat uses the legacy index templates.
#setup.template.type: legacy
# By default auditbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "auditbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2117,8 +2117,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default filebeat uses the legacy index templates.
#setup.template.type: legacy
# By default filebeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "filebeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions heartbeat/heartbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1398,8 +1398,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default heartbeat uses the legacy index templates.
#setup.template.type: legacy
# By default heartbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "heartbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions journalbeat/journalbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1148,8 +1148,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default journalbeat uses the legacy index templates.
#setup.template.type: legacy
# By default journalbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "journalbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions libbeat/_meta/config/setup.template.reference.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default {{.BeatName}} uses the legacy index templates.
#setup.template.type: legacy
# By default {{.BeatName}} uses index templates.
#setup.template.type: index

# Template name. By default the template name is "{{.BeatIndexPrefix}}-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
7 changes: 5 additions & 2 deletions libbeat/template/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ package template
import (
"fmt"

"github.com/elastic/beats/v7/libbeat/common/cfgwarn"
"github.com/elastic/beats/v7/libbeat/mapping"
)

Expand Down Expand Up @@ -69,18 +70,20 @@ func DefaultConfig() TemplateConfig {
return TemplateConfig{
Enabled: true,
Fields: "",
Type: IndexTemplateLegacy,
Type: IndexTemplateIndex,
Order: 1,
Priority: 150,
}
}

func (t *IndexTemplateType) Unpack(v string) error {
if v == "" {
*t = IndexTemplateLegacy
*t = IndexTemplateIndex
return nil
}

cfgwarn.Deprecate("8.0.0", "do not use setup.template.type, it is deprecated, data streams will be loaded automatically")

var tt IndexTemplateType
var ok bool
if tt, ok = templateTypes[v]; !ok {
Expand Down
33 changes: 19 additions & 14 deletions libbeat/template/load_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ import (
func TestFileLoader_Load(t *testing.T) {
ver := "7.0.0"
prefix := "mock"
order := 1
info := beat.Info{Version: ver, IndexPrefix: prefix}
tmplName := fmt.Sprintf("%s-%s", prefix, ver)

Expand All @@ -42,29 +41,35 @@ func TestFileLoader_Load(t *testing.T) {
"load minimal config info": {
body: common.MapStr{
"index_patterns": []string{"mock-7.0.0-*"},
"order": order,
"settings": common.MapStr{"index": nil}},
"priority": 150,
"template": common.MapStr{
"settings": common.MapStr{"index": nil}},
},
},
"load minimal config with index settings": {
settings: TemplateSettings{Index: common.MapStr{"code": "best_compression"}},
body: common.MapStr{
"index_patterns": []string{"mock-7.0.0-*"},
"order": order,
"settings": common.MapStr{"index": common.MapStr{"code": "best_compression"}}},
"priority": 150,
"template": common.MapStr{
"settings": common.MapStr{"index": common.MapStr{"code": "best_compression"}}},
},
},
"load minimal config with source settings": {
settings: TemplateSettings{Source: common.MapStr{"enabled": false}},
body: common.MapStr{
"index_patterns": []string{"mock-7.0.0-*"},
"order": order,
"settings": common.MapStr{"index": nil},
"mappings": common.MapStr{
"_source": common.MapStr{"enabled": false},
"_meta": common.MapStr{"beat": prefix, "version": ver},
"date_detection": false,
"dynamic_templates": nil,
"properties": nil,
}},
"priority": 150,
"template": common.MapStr{
"settings": common.MapStr{"index": nil},
"mappings": common.MapStr{
"_source": common.MapStr{"enabled": false},
"_meta": common.MapStr{"beat": prefix, "version": ver},
"date_detection": false,
"dynamic_templates": nil,
"properties": nil,
}},
},
},
} {
t.Run(name, func(t *testing.T) {
Expand Down
29 changes: 17 additions & 12 deletions libbeat/template/template.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,13 +200,6 @@ func (t *Template) LoadMinimal() (common.MapStr, error) {
return nil, fmt.Errorf("unknown template type %v", t.templateType)
}

if t.config.Settings.Source != nil {
m["mappings"] = buildMappings(
t.beatVersion, t.esVersion, t.beatName,
nil, nil,
common.MapStr(t.config.Settings.Source))
}

return m, nil
}

Expand All @@ -218,22 +211,34 @@ func (t *Template) loadMinimalLegacy() common.MapStr {
"settings": common.MapStr{
"index": t.config.Settings.Index,
},
"mappings": buildMappings(
t.beatVersion, t.esVersion, t.beatName,
nil, nil,
common.MapStr(t.config.Settings.Source)),
}
}

func (t *Template) loadMinimalComponent() common.MapStr {
templ := common.MapStr{}
if t.config.Settings.Source != nil {
templ["mappings"] = buildMappings(
t.beatVersion, t.esVersion, t.beatName,
nil, nil,
common.MapStr(t.config.Settings.Source))
}
templ["settings"] = common.MapStr{
"index": t.config.Settings.Index,
}
return common.MapStr{
"template": common.MapStr{
"settings": common.MapStr{
"index": t.config.Settings.Index,
},
},
"template": templ,
}
}

func (t *Template) loadMinimalIndex() common.MapStr {
keyPattern, patterns := buildPatternSettings(t.esVersion, t.GetPattern())
m := t.loadMinimalComponent()
m["priority"] = t.priority
m[keyPattern] = patterns
return m
}

Expand Down
14 changes: 7 additions & 7 deletions libbeat/template/template_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,9 @@ func TestTemplate(t *testing.T) {
currentVersion := getVersion("")

t.Run("for ES 6.x", func(t *testing.T) {
template := createTestTemplate(t, currentVersion, "6.4.0", DefaultConfig())
cfg := DefaultConfig()
cfg.Type = IndexTemplateLegacy
template := createTestTemplate(t, currentVersion, "6.4.0", cfg)
template.Assert("index_patterns", []string{"testbeat-" + currentVersion + "-*"})
template.Assert("order", 1)
template.Assert("mappings.doc._meta", common.MapStr{"beat": "testbeat", "version": currentVersion})
Expand All @@ -120,17 +122,15 @@ func TestTemplate(t *testing.T) {
t.Run("for ES 7.x", func(t *testing.T) {
template := createTestTemplate(t, currentVersion, "7.2.0", DefaultConfig())
template.Assert("index_patterns", []string{"testbeat-" + currentVersion + "-*"})
template.Assert("order", 1)
template.Assert("mappings._meta", common.MapStr{"beat": "testbeat", "version": currentVersion})
template.Assert("settings.index.max_docvalue_fields_search", 200)
template.Assert("template.mappings._meta", common.MapStr{"beat": "testbeat", "version": currentVersion})
template.Assert("template.settings.index.max_docvalue_fields_search", 200)
})

t.Run("for ES 8.x", func(t *testing.T) {
template := createTestTemplate(t, currentVersion, "8.0.0", DefaultConfig())
template.Assert("index_patterns", []string{"testbeat-" + currentVersion + "-*"})
template.Assert("order", 1)
template.Assert("mappings._meta", common.MapStr{"beat": "testbeat", "version": currentVersion})
template.Assert("settings.index.max_docvalue_fields_search", 200)
template.Assert("template.mappings._meta", common.MapStr{"beat": "testbeat", "version": currentVersion})
template.Assert("template.settings.index.max_docvalue_fields_search", 200)
})
}

Expand Down
4 changes: 2 additions & 2 deletions metricbeat/metricbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2017,8 +2017,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default metricbeat uses the legacy index templates.
#setup.template.type: legacy
# By default metricbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "metricbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions packetbeat/packetbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1700,8 +1700,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default packetbeat uses the legacy index templates.
#setup.template.type: legacy
# By default packetbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "packetbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions winlogbeat/winlogbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1128,8 +1128,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default winlogbeat uses the legacy index templates.
#setup.template.type: legacy
# By default winlogbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "winlogbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/auditbeat/auditbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1262,8 +1262,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default auditbeat uses the legacy index templates.
#setup.template.type: legacy
# By default auditbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "auditbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4355,8 +4355,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default filebeat uses the legacy index templates.
#setup.template.type: legacy
# By default filebeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "filebeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/functionbeat/functionbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1109,8 +1109,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default functionbeat uses the legacy index templates.
#setup.template.type: legacy
# By default functionbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "functionbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/heartbeat/heartbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1398,8 +1398,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default heartbeat uses the legacy index templates.
#setup.template.type: legacy
# By default heartbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "heartbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/metricbeat/metricbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2548,8 +2548,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default metricbeat uses the legacy index templates.
#setup.template.type: legacy
# By default metricbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "metricbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/osquerybeat/osquerybeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -720,8 +720,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default osquerybeat uses the legacy index templates.
#setup.template.type: legacy
# By default osquerybeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "osquerybeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/packetbeat/packetbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1700,8 +1700,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default packetbeat uses the legacy index templates.
#setup.template.type: legacy
# By default packetbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "packetbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/winlogbeat/winlogbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1171,8 +1171,8 @@ output.elasticsearch:

# Select the kind of index template. From Elasticsearch 7.8, it is possible to
# use component templates. Available options: legacy, component, index.
# By default winlogbeat uses the legacy index templates.
#setup.template.type: legacy
# By default winlogbeat uses index templates.
#setup.template.type: index

# Template name. By default the template name is "winlogbeat-%{[agent.version]}"
# The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
Expand Down