[Automatic Import] Do not remove message field for unstructured logs

[bhapas]

Summary

The pipeline template by default removes the message field. But in case of unstructured logs the grok processor adds the unstructured message part into a message field and it should be part of the final pipeline.

Hence, the remove processor is not executed in case the log_format is unstructured.

Before this PR

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "hostname": "mymachine",
          "process": "su",
          "ecs": {
            "version": "8.11.0"
          },
          "related": {
            "hosts": [
              "mymachine"
            ]
          },
          "priority": "34",
          "event": {
            "category": [
              "authentication",
              "process"
            ],
            "type": [
              "start"
            ]
          },
          "timestamp": "Oct 11 00:14:05"
        },
        "_ingest": {
          "timestamp": "2024-09-20T16:10:01.656597092Z"
        }
      }
    }
  ]
}

After this PR

[elasticmachine]

Pinging @elastic/security-scalability (Team:Security-Scalability)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 945f28e

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @bhapas

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.15
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation