Extending Attributes processor

[ccaraman]

Currently the design for the attributes processor is limited. It supports some basic functions(insert/upsert/update/delete/hash/...) however the configuration is cumbersome and limits any richer functionalities being added. ex. Should there be mathematical operations? What about the new attribute types (nested arrays and maps)? Should action represent the value source and add a concept of operator(insert/update/...)?

Please refer to #979 for an example

This issue tracks figuring out the plan for the attributes processor.

Several open questions:

• To avoid breaking changes, should there be versioning of this processor?

Related issues to be tagged

• Breaking config change process

[tigrannajaryan]

@ccaraman is this now resolved or there is more work to do?

[ccaraman]

@tigrannajaryan - there is work to be done with this issue - I'll be adding some of the examples of what the configuration could look like that would have a richer feature set.

[ccaraman]

Ex of a configuration entry that would better work for maps See #979 (comment) for full context

# Delete nested
del attr["foo"]["bar"]

# Add dictionary
attr["bar"] = {"one": "two"}

# Overwrite list
attr["list"] = [1,2,3]

# Append to list
attr["list"] += [4,5,6]

# regex
attr["http.url"]  = /^(?P<http_protocol>.*):\/\/(?P<http_domain>.*)\/(?P<http_path>.*)(\?|\&)(?P<http_query_params>.*)/

# from another key
attr["string key"] = attr["another key"]

[ccaraman]

Or separating action and operations into two different concepts from comment https://github.com/open-telemetry/opentelemetry-collector/pull/981/files#r426882973

SetValue -> {AttributeValue, Action}
SetValueFromAttribute -> {FromAttribute, Action}
SetValuesFromRegexp -> ....

[jrcamp]

If we didn't want to invent our own syntax another option would be to use an existing embeddable scripting language. Fluentbit, for instance, can use lua:

https://docs.fluentbit.io/manual/pipeline/filters/lua

Another option is Tengo: https://github.com/d5/tengo

[jrcamp]

See Bogdan's expression prototype #2408

[bhautikpip]

@jrcamp @ccaraman I can work with you to contribute on this issue. However, I am not super familiar with collector code and looking to contribute for my first issue. Any pointers or help would be appreciated.

[alolita]

@bhautikpip can you start working on a design proposal for this issue?

[alolita]

This issue is related to the design proposal in #3185

[carlosalberto]

I can take this item in case @bhautikpip is not available anymore ;)

[carlosalberto]

Ping @alolita @bhautikpip

[bhautikpip]

@carlosalberto @alolita I can start looking on this once I am back as I am currently not available due to traveling. @carlosalberto feel free to take this.

[carlosalberto]

Thanks!

[bogdandrutu]

This should be part of #3185. We need first to understand what components do we offer, etc.