Set ignore_malformed=true on .alerts-* indices

[kobelb]

Feature Description

Currently, the .alerts-* indices do not have ignore_malformed set. As a result, alerting rule implementors are responsible for ensuring that the alerts they generate are compatible with the mappings of the associated .alert-* indices. If the alerts aren't compatible, then the alert documents will NOT be created and our users will miss alerts.

The security solution team has had to implement rather complex logic to strip non-compliant fields from their alert documents. By setting ignored_malformed on the alert indices, this would potentially reduce the complexity of the logic stripping alert fields and act as a fallback to ensure that alert documents are created in as many situations as possible.

However, we should be aware of the caveat that ignore_malformed does not work on all data-types, so we should not be solely relying on this setting and alerting rule implementors should still do their absolute best to ensure alerts are compatible with the mappings.

Business Value

Reduce the likelihood of missed alerts.

Definition of Done

• .alerts-* indices have ignore_malformed set
• regression testing performed to ensure this setting does not detrimentally alter the behavior of alert creation/updates

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

Reverted due to possible bug with ignore_malformed and datastreams. #163610 . Elasticsearch issue: elastic/elasticsearch#98511

When that issue is fixed, we can revert this revert.

[mbudge]

We raised this 2 years ago after seeing indicator_match alerts not being raised because of the user_agent field.

Ignore_malformed sounds like the best way to fix this issue.

[pmuellr]

As a work-around for the problem with ignored_maformed: true on data streams, a solution is to remove @timestamp from the mappings, as data streams will create one automatically if it's not in the mappings. And let you use ignore_malformed: true at the index level.

See elastic/elasticsearch#98511 for more details. Sounds like we might change the way this works in the future to allow the original code to work (using the setting AND having a @timestamp mapping), but TBD.

[mikecote]

@pmuellr thanks for the update, does that mean we're unblocked and we can go forward with the approach you described?

a solution is to remove @timestamp from the mappings, as data streams will create one automatically if it's not in the mappings. And let you use ignore_malformed: true at the index level.

Seems reasonable to me.

[pmuellr]

does that mean we're unblocked and we can go forward with the approach you described?

We are unblocked; but we'll need to change the mappings for serverless to remove the @timestamp field. Suggest we treat this as blocked on #154266, where we'll have structure to make this easy.

[pmuellr]

My original testing for this was broken somehow. I thought it wasn't possible to use an index-level setting for ignore_malformed and then override the setting for just `@timestamp. But apparently you can - now anyway., suggested by felix: elastic/elasticsearch#98511 (comment) .

Not sure if we want to use the same template for data streams / indices, or have them differ just by the @timestamp setting of ignore_malformed. Having them the same is easier, but making them differ won't be hard either.

Here's a complete incantation that works:

PUT _index_template/pmuellr-index-template
{
  "index_patterns": ["pmuellr-data-stream"],
  "data_stream": { },
  "template": {
    "settings": {
      "index.mapping.ignore_malformed": true 
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "ignore_malformed": false,
          "type": "date"
        },
        "message": {
          "type": "text"
        }
      }      
    }
  }
}