Don't override user-supplied data_stream fields

[felixbarny]

While adding support for data_stream.dataset and data_stream.namespace fields for ECS loggers (elastic/ecs-logging#38). I noticed that Filebeat overrides the fields with the static value from the integration policy rather than using the fields from the log events. That happens even though the input setting json.override_keys is set to true.

I do think we'd want users to define the dataset in their ECS logging configuration for several reasons:

• We don't want users having to create a dedicated log configuration per application. Ideally, they'd be able to set up the logging for all their applications with just one integration. For example, by providing several log file paths pointing to ECS log files of different applications.
• We already make event.dataset configurable which is used for the log anomaly ML job. Couldn't we use that to set the data_stream.dataset? No, we can't: The event.dataset and data_stream.dataset fields should always have the same values. However, the data_stream.dataset imposes more restrictions on the allowed characters (as it ends up as a part of the index name). Thus, we can't use the values of event.dataset for data_stream.dataset.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ph]

this is a good catch, I think we all the add_fields processor but we don't have any conditional to check if the values is present do nothing.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[felixbarny]

@ruflin do these PRs also address this issue?

• [Filebeat] Add support for dataset configs #35287
• [Filebeat] Add dataset support to output #35288

[ruflin]

It doesn't directly, but it can solve it. The problem today is that an add_field processor is used to set this field. Having it configured as a config option by Elastic Agent for Beats would mean the field would not be set directly and then in the output it can only be set, if it doesn't already exist.

We could take this to the next level that Filebeat already sends the data based on the field to the correct data stream? All the features for this already exist in Filebeat, see index settings here: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es

[felixbarny]

in the output it can only be set, if it doesn't already exist.
sends the data based on the field to the correct data stream

++ to both of these statements

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!