[connector/routing] Support log record routing

[djaglowski]

Component(s)

connector/routing

Is your feature request related to a problem? Please describe.

Currently the routing connector only has the ability to route based on resource information. In other words, all logs associated with a given resource will be routed together. (The same is true of metrics and spans as well, but this issue directly addresses logs as a first step.)

Describe the solution you'd like

Add a context field to routing table options, with possible values resource (default) or log. If log context is specified, execute the OTTL condition or statement as a log statement. Based on the result, route the individual log record appropriately.

Importantly, there is an implementation challenge here because appropriate routing at a per-log level requires retention of the higher level scope and resource information, without pulling in other logs as well.

I believe I finally have a workable solution, demonstrated in #35939. This solution is based on the addition of helper functions that can "split" a plog.Logs according to criteria at either the resource or log level. When splitting at the log level, scope and resource information are retained.

I'd like to implement this functionality for logs first, then later for other signals once the strategy proves out. The initial sequence of PRs will look like this:

1. Add an internal package to the routing connector. This will contain the splitting functions and will be well tested in a generic way. (Later, these may be moved to pkg/pdatautil or eventually into pdata, but for now keep them unexported.)
2. Update the current implementation to use the split-by-resource function. This should have no noticeable change to users.
3. Add support for log routing, using the split-by-log function.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• connector/routing: @jpkrohling @mwear

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

@mwear, @jpkrohling, I'm excited to move my PoC forward with the plan proposed above, if it sounds good to you.

Also cc'ing @evan-bradley @TylerHelmuth as I think some version of this might eventually be able to solve cases like #32080.