Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libbeat]: add uppercase processor #41535

Merged
merged 2 commits into from
Nov 7, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Replace Ubuntu 20.04 with 24.04 for Docker base images {issue}40743[40743] {pull}40942[40942]
- Reduce memory consumption of k8s autodiscovery and the add_kubernetes_metadata processor when Deployment metadata is enabled
- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424]

- Add `uppercase` processor. {issue}22254[22254]
*Auditbeat*

- Added `add_session_metadata` processor, which enables session viewer on Auditbeat data. {pull}37640[37640]
Expand Down
119 changes: 119 additions & 0 deletions libbeat/processors/actions/docs/uppercase.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
[[uppercase]]
=== Uppercase fields in events

++++
<titleabbrev>uppercase</titleabbrev>
++++

The `uppercase` processor specifies a list of `fields` and `values` to be converted to uppercase. Keys listed in `fields` will be matched case-insensitively and converted to uppercase. For `values`, only exact, case-sensitive matches are transformed to uppercase. This way, keys and values can be selectively converted based on the specified criteria.


==== Examples:

1. Default scenario

[source,yaml]
----
processors:
- rename:
fields:
- "ab.cd"
values:
- "testKey"
ignore_missing: false
fail_on_error: true
alter_full_field: true
----
[source,json]
----
// Input
{
"ab": {"cd":"data"},
"CD": {"ef":"data"},
"testKey": {"testvalue"}
}


// output
{
"ab": {"cd":"data"}, // `ab.cd` -> `AB.CD`
"CD": {"ef":"data"},
"testKey": {"TESTVALUE"} // `testvalue` -> `TESTVALUE` is uppercased
}
----

[start=2]
2. When `alter_full_field` is false (applicable only for fields)

[source,yaml]
----
processors:
- rename:
fields:
- "ab.cd"
ignore_missing: false
fail_on_error: true
alter_full_field: false
----

[source,json]
----
// Input
{
"ab": {"cd":"data"},
"CD": {"ef":"data"},
}


// output
{
"ab": {"CD":"data"}, // `ab.cd` -> `ab.CD` (only `cd` is uppercased)
"CD": {"ef":"data"},
}
----

[start=2]
2. In case of non unique path to the key

[source,yaml]
----
processors:
- rename:
fields:
- "ab"
ignore_missing: false
fail_on_error: true
alter_full_field: true
----

[source,json]
----
// Input
{
"ab": "first",
"aB": "second"
}

// Output
{
"ab": "first",
"aB": "second",
"err": "... Error: key collision"
}
----

==== Configuration:

The `uppercase` processor has the following configuration settings:

`fields`:: The field names to uppercase. The match is case-insensitive, e.g. `a.b.c.d` would match `A.b.C.d` or `A.B.C.D`.
`values`:: (Optional) Specifies the exact values to be converted to uppercase. Each entry should include the full path to the value. Key matching is case-sensitive. If the target value is not a string, an error is triggered (`fail_on_error: true`) or the value is skipped (`fail_on_error: false`).
`ignore_missing`:: (Optional) Indicates whether to ignore events that lack the source field.
The default is `false`, which will fail processing of an event if a field is missing.
`fail_on_error`:: (Optional) If set to `true` and an error occurs, the changes are reverted and the original event is returned.
If set to `false`, processing continues if an error occurs. Default is `true`.
`alter_full_field`:: (Optional) If set to `true`, the entire key path is uppercased. If set to `false` only the final part of the key path is uppercased. Default is true



See <<conditions>> for a list of supported conditions.
26 changes: 0 additions & 26 deletions libbeat/processors/actions/lowercase_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,32 +59,6 @@ func TestLowerCaseProcessorRun(t *testing.T) {
Output mapstr.M
Error bool
}{
{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate test case

Name: "Lowercase Fields",
Fields: []string{"a.b.c", "Field1"},
IgnoreMissing: false,
FailOnError: true,
FullPath: true,
Input: mapstr.M{
"Field1": mapstr.M{"Field2": "Value"},
"Field3": "Value",
"a": mapstr.M{
"B": mapstr.M{
"C": "D",
},
},
},
Output: mapstr.M{
"field1": mapstr.M{"Field2": "Value"}, // field1 is lowercased
"Field3": "Value",
"a": mapstr.M{
"b": mapstr.M{
"c": "D",
},
},
},
Error: false,
},
{
Name: "Lowercase Fields",
Fields: []string{"a.b.c", "Field1"},
Expand Down
47 changes: 47 additions & 0 deletions libbeat/processors/actions/uppercase.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
// Licensed to Elasticsearch B.V. under one or more contributor
// license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright
// ownership. Elasticsearch B.V. licenses this file to you under
// the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

package actions

import (
"strings"

"github.com/elastic/beats/v7/libbeat/beat"
"github.com/elastic/beats/v7/libbeat/processors"
"github.com/elastic/beats/v7/libbeat/processors/checks"
conf "github.com/elastic/elastic-agent-libs/config"
)

func init() {
processors.RegisterPlugin(
"uppercase",
checks.ConfigChecked(
NewUpperCaseProcessor,
checks.RequireFields("fields"),
checks.AllowedFields("fields", "ignore_missing", "fail_on_error", "alter_full_field", "values"),
),
)
}

// NewUpperCaseProcessor converts event keys matching the provided fields to uppercase
func NewUpperCaseProcessor(c *conf.C) (beat.Processor, error) {
return NewAlterFieldProcessor(c, "uppercase", upperCase)
}

func upperCase(field string) (string, error) {
return strings.ToUpper(field), nil
}
Loading
Loading