-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add regex pattern matching to add_kubernetes_metadata processor #41903
Conversation
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very good!
@tetianakravchenko @constanca-m could you have a look? 🙏 |
Can you add documentation on this here as well? Especially how it is necessary to have |
Done. |
45302a6
to
8cebc68
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
I didn't manage to test, there was some issue building the Elastic-Agent image with this Beats PR :/
I didn't notice auto-merge was enabled 😅 |
That's fine, it was enabled on purpose. 😈 |
* Add regex pattern matching to add_kubernetes_metadata processor * Add changelog entry * Add documentation * Fix changelog (cherry picked from commit da7bbf6)
* Add regex pattern matching to add_kubernetes_metadata processor * Add changelog entry * Add documentation * Fix changelog (cherry picked from commit da7bbf6)
* Add regex pattern matching to add_kubernetes_metadata processor * Add changelog entry * Add documentation * Fix changelog (cherry picked from commit da7bbf6)
…) (#42087) * Add regex pattern matching to add_kubernetes_metadata processor * Add changelog entry * Add documentation * Fix changelog (cherry picked from commit da7bbf6) Co-authored-by: Mikołaj Świątek <[email protected]>
…) (#42085) * Add regex pattern matching to add_kubernetes_metadata processor * Add changelog entry * Add documentation * Fix changelog (cherry picked from commit da7bbf6) Co-authored-by: Mikołaj Świątek <[email protected]>
Proposed commit message
Add the ability to extract the matching key from fields using a regex pattern to the add_kubernetes_metadata processor. I've added a new field -
regex_pattern
to the existingfields
matcher. If the field isn't present, the matcher continues to work the same way it used to. The regex pattern has to contain a named capture group:key
, and the first leftmost match for this group is the extracted key value. Other than this, the semantics are identical to the existing ones for thefields
matcher.Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.How to test this PR locally
This was originally intended to fix https://github.com/elastic/ingest-dev/issues/3428, where we want to match on a part of
system.process.cgroup.id
. I've verified this by building an agent container image with this change enabled, and making the following changes to the default agent manifest:With the Helm Chart, you can use the following values:
Related issues
Use cases
It's now possible to extract parts of fields for matching with kubernetes metadata. A use case which this is intended to make possible is adding kubernetes metadata to system process metrics. These metrics will contain the cgroup id under the field
system.process.cgroup.id
if cgroups are enabled, but the container id as reported by Kubernetes is only part of this value. In order to match it against the latter, we need the ability to only use a part of it.Screenshots
Example document