New include/exclude transform

[bruceg]

Vector needs a transform that can selectively drop events based on flexible criteria. In particular, it needs a generic method of either dropping selected or non-selected events with a set of fields matching a set of patterns.

There already exists a field_filter that drops logs unless a field matches a particular value. However, it only works with log events, does not support patterns, allows only a single field/value pair, and cannot be inverted for an exclude mode. It could be extended to support these changes, but not without breaking existing users.

There is also a proposal for a "shape" transform (#750). That appears to be more intended for rewriting events rather than choosing them, though the activities are related.

Ref #327

[binarylogic]

Thanks. We are starting to think about a project that would focus on improving the transform experience as well as general data model management. We'll add this to the project.

[Jeffail]

Found a duplicate with more context: #479

I can imagine that users will eventually use this to build huge filters with complex logic. It might be worth considering how we can make it easier for those users to break their logic down and avoid duplicating it. A common use case will be a large filter to detect messages that shouldn't reach sink A, but should be directed to sink B. If we can avoid forcing users to write their filter logic out twice (once for a filter Foo, another for a NOT filter Foo).

This also ties into #232 because I'll need to implement ways of querying events, which will overlap. I was thinking maybe we could introduce a new core Vector type condition to encompass boolean expressions around an event type.

For example, it might look something like this:

[conditions.is_thing]
  type = "check_fields"
  "status.eq" = 500
  "bar.contains" = "giraffe"

[conditions.is_not_a_thing]
  type = "not"
  child = "is_thing"

[transforms.filter_things]
  type = "filter"
  condition = "is_thing"

[transforms.filter_not_things]
  type = "filter"
  condition = "is_not_a_thing"

[sinks.things]
  type = "foo"
  inputs = [ "filter_things" ]

[sinks.not_things]
  type = "bar"
  inputs = [ "filter_not_things" ]

With something like that as is_thing becomes larger over time it doesn't impact is_not_a_thing or any other component that reuses the same logic. It would also be useful for breaking conditions down and bringing them together with boolean expressions, which could be a condition type of its own:

[conditions.lots_of_logic]
  type = "expression"
  expression = "not is_thing and (some_other_condition or another_one)"

[Jeffail]

It would also be cool to allow either a condition type embedded within the filter transform itself, or a string pointing to a target global condition:

[conditions.is_thing]
  type = "check_fields"
  "status.eq" = 500
  "bar.contains" = "giraffe"

[transforms.filter_things]
  type = "filter"
  condition = "is_thing"

[transforms.filter_not_things]
  type = "filter"
  [transforms.filter_not_things.condition]
    type = "not"
    child = "is_thing"

[sinks.things]
  type = "foo"
  inputs = [ "filter_things" ]

[sinks.not_things]
  type = "bar"
  inputs = [ "filter_not_things" ]

[binarylogic]

This is superseded by #479