Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to override the ignore_above option for field of the type keyword. #7238

Merged
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Allow to override the ignore_above option for field of the type
keyword.

By default all fields of the type keyword were limitted to 1024
characters, this PR keep the default but allow developers to redefine that limit using the
`ignore_above` directive in the field.yml.
ph committed Jun 5, 2018

Verified

This commit was signed with the committer’s verified signature.
ph Pier-Hugues Pellerin
commit ce69b9bc364ea1bf3cd5d677a72a06b8b0cd0b75
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
@@ -67,6 +67,7 @@ https://github.com/elastic/beats/compare/v6.2.3...master[Check the HEAD diff]
- Fix delays on autodiscovery events handling caused by blocking runner stops. {pull}7170[7170]
- Error out on invalid Autodiscover template conditions settings. {pull}7200[7200]
- Do not emit Kubernetes autodiscover events for Pods without IP address. {pull}7235[7235]
- Allow to override the `ignore_above` option when defining new field with the type keyword. {pull}7238[7238]

*Auditbeat*

1 change: 1 addition & 0 deletions libbeat/common/field.go
Original file line number Diff line number Diff line change
@@ -34,6 +34,7 @@ type Field struct {
Index *bool `config:"index"`
DocValues *bool `config:"doc_values"`
CopyTo string `config:"copy_to"`
IgnoreAbove int `config:"ignore_above"`

// Kibana specific
Analyzed *bool `config:"analyzed"`
8 changes: 6 additions & 2 deletions libbeat/template/processor.go
Original file line number Diff line number Diff line change
@@ -14,6 +14,7 @@ type Processor struct {

var (
defaultScalingFactor = 1000
defaultIgnoreAbove = 1024
)

// Process recursively processes the given fields and writes the template in the given output
@@ -150,11 +151,14 @@ func (p *Processor) keyword(f *common.Field) common.MapStr {
defaultFields = append(defaultFields, fullName)

property["type"] = "keyword"
property["ignore_above"] = 1024
if f.IgnoreAbove == 0 {
property["ignore_above"] = defaultIgnoreAbove
} else {
property["ignore_above"] = f.IgnoreAbove
}

if p.EsVersion.IsMajor(2) {
property["type"] = "string"
property["ignore_above"] = 1024
property["index"] = "not_analyzed"
}

14 changes: 14 additions & 0 deletions libbeat/template/processor_test.go
Original file line number Diff line number Diff line change
@@ -130,6 +130,20 @@ func TestProcessor(t *testing.T) {
},
},
},
{
output: p.keyword(&common.Field{Type: "keyword", IgnoreAbove: 256}),
expected: common.MapStr{
"type": "keyword",
"ignore_above": 256,
},
},
{
output: p.keyword(&common.Field{Type: "keyword"}),
expected: common.MapStr{
"type": "keyword",
"ignore_above": 1024,
},
},
{
output: p.text(&common.Field{Type: "text", MultiFields: common.Fields{
common.Field{Name: "raw", Type: "keyword"},