Meta fields are not handled consistently in different processors

[jsoriano]

TLDR; All processors should use event.PutValue when setting fields to have a consistent behaviour.

Some fields set in events are intended to provide information to the pipelines, for example @timestamp to set the timestamp of the event or @metadata._id to set its id. This is not being handled in a consistent way between processors.

For example when the id is set using the fingerprint processor, it works as expected:

  - fingerprint:
      fields: ["id"]
      target_field: "@metadata._id"

It doesn't set @metadata._id in the fields of the event, but it uses it as the id of the generated document.

On the other hand, if add_fields is used to set one of these special fields, the field is set in the event and not handled by the pipeline:

  - add_fields:
      target: "@metadata"
      fields:
        op_type: "index"

In this case the index bulk operation should be used, but it isn't, and @metadata.op_type is set as a field in the event.

The cause of this difference is that some processors use the event.PutValue method, that handles these cases, and other processors use lower level mechanisms as event.Fields.Put, or directly events.Fields[key] = value.

All processors should use event.PutValue when setting fields to have a consistent behaviour.

For confirmed bugs, please report:

• Version: All versions at least till 7.12.
• Steps to Reproduce: The following configuration should create index bulk requests so documents are updated when they have the same fingerprint, but instead create bulk requests are used and then the documents are not updated:

processors:
  - fingerprint:
      fields: ["id"]
      target_field: "@metadata._id"
  - add_fields:
      target: "@metadata"
      fields:
        op_type: "index"

As a workaround, expected result can be achieved setting @metadata.op_type with the convert processor, that uses PutValue:

processors:
  - fingerprint:
      fields: ["id"]
      target_field: "@metadata._id"
  - add_fields:
      target: "_tmp"
      fields:
        op_type: "index"
  - convert:
      fields:
      - from: "_tmp.op_type"
        to: "@metadata.op_type"
        type: "string"
  - drop_fields.fields:
    - "_tmp"

Related:

• Add docs about some of these @metadata fields Document @metadata fields #23091

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)