Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution] Fixes sorting issues related to unmapped fields (e…
…lastic#132190) This PR fixes the following issues related to sorting unmapped fields in timelines and the events / alerts tables: - <elastic#129603> - <elastic#123912> - <elastic#131625> The `unmapped_type` property [addition](https://github.com/elastic/kibana/pull/87241/files#diff-52fd5870dcd5f783f9fc8ac3a18a8674d83ac6136e09fe0e0bcae30427d61c3fR55) to the `sort` parameter of requests was using the `type` field metadata from `BrowserFields`, but the `type` metadata (for some fields) contains the value `string`, which is not a [valid field data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). The fix for the issues above: - Populates the `sort` property of requests with values from the `esTypes` `BrowserFields` metadata (instead of `type`) - The `esTypes` metadata may specify more than one field value type. When `esTypes` contains more than one type, and `keyword` is one of the types, the `sort` property of the request will prefer `keyword` over other the other types - When the field metadata has an empty `esTypes` collection, the `sort` property of the request will default to using `"unmapped_type": "keyword"` - The field type displayed in tooltips when hovering over columns in a timeline now displays values from `esTypes` instead of `type` To reproduce issue <elastic#129603> and to verify the fix: 1) Open Kibana `Dev tools` 2) Execute the following query to delete any exiting `logs-ti_test` index: ``` DELETE logs-ti_test ``` 3) Execute the following query to create an index named `logs-ti_test`, which has the following properities: - Dynamic mappings are disabled via `"dynamic": false` - It does NOT contain a mapping for `event.action` (we will sort by this field in later steps) - It contains a mapping for the non-ECS `testing` field ``` PUT logs-ti_test { "mappings": { "dynamic": false, "properties": { "@timestamp": { "type": "date" }, "event": { "properties": { "category": { "type": "keyword" }, "dataset": { "type": "keyword" }, "kind": { "type": "keyword" }, "type": { "type": "keyword" } } }, "host": { "properties": { "name": { "type": "keyword" } } }, "testing": { "type": "keyword", "ignore_above": 1024 }, "threat": { "properties": { "indicator": { "properties": { "file": { "properties": { "hash": { "properties": { "md5": { "type": "keyword" } } } } } } } } } } } } ``` 4) Execute the following query to add a new document to the `logs-ti_test` index, and note that: - It does NOT contain a `event.action` field - It contains a value for the non-ECS `testing` field ``` POST logs-ti_test/_doc/ { "@timestamp": "2022-05-12T00:00:14.725Z", "host": { "name": "foozle" }, "threat": { "indicator": { "file": { "hash": { "md5": "a4f87cbcd2a4241da77b6bf0c5d9e8553fec991f" } } } }, "event": { "kind": "enrichment", "type": "indicator", "dataset": "ti_*", "category": "threat" }, "testing": "simulated threat intel data" } ``` 5) Navigate to the Security > Hosts page 6) Select `Last 1 year` from the date picker 7) Click the `Events` tab 8) Enter the following KQL query in the search bar at the top of the page: ``` host.name: foozle ``` 9) Hover over the `foozle` entry in the `host.name` column in the Events table, and click the `Add to timeline investigation` cell action 10) Open the timeline 11) Hover over the `event.action` field **Expected result** - The tooltip displays type `keyword` for the `event.action` field **Actual result** - The tooltip displays type `string` for the `event.action` field 12) Click the `event.action` column to add a secondary sort **Expected result** - The table is sorted by `@timestamp` and `event.action` - The table contents are (still) visible **Actual result** - The table is sorted by `@timestamp` and `event.action` - The contents of the table are now empty 13) Click the timeline's `Inspect` button 14) In the `Inspect Timeline` dialog, click the `Request` tab 15) Scroll down to the `sort` property of the request **Expected result** - The `event.action` field contains a `"unmapped_type": "keyword"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "date" } }, { "event.action": { "order": "desc", "unmapped_type": "keyword" } } ], ``` **Actual result** - The request's `event.action` field contains a `"unmapped_type": "string"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "number" } }, { "event.action": { "order": "desc", "unmapped_type": "string" } } ], ``` 16) In the `Inspect Timeline` dialog, click the `Response` tab **Expected result** - The response contains `0` `failed` shards / no failures **Actual result** - The response contains failures for the `logs-ti_test` index, with the following reason: ``` "reason": "No mapper found for type [string]" ``` per the example below: ```json { "took": 1, "timed_out": false, "_shards": { "total": 4, "successful": 3, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "logs-ti_test", "node": "NCRcGeDqSlKQiuPWVFvMEg", "reason": { "type": "illegal_argument_exception", "reason": "No mapper found for type [string]" } } ] }, ``` (cherry picked from commit f2c8b2c)
- Loading branch information