[Security Solution] Sorting does not work on non-ECS fields #131625

MadameSheema · 2022-05-05T12:17:02Z

Describe the bug:

Sorting does not work on non-ECS fields

Kibana/Elasticsearch Stack version:

8.3.0 (main 58bc0f7)

Initial Setup:

To have a couple of documents with a non-ECS field with a map i.e.
Mapping:

{
  "properties" : {
    "@timestamp" : {
    	"type" : "date"
  },
   "dummy_field_1" : { 
    "type" : "keyword", 
    "ignore_above" : 1024
   },
   "dummy_field_2" : { 
    "type" : "keyword", 
    "ignore_above" : 1024
   }

Documents

{
   "@timestamp":"2022-05-05T09:38:19.579Z",
   "dummy_field_1":"field_1",
   "dummy_field_2":"field_2"
}

{
   "@timestamp" : "2022-05-05T09:40:19.579Z",
   "dummy_field_1" : "another_field_1",
   "dummy_field_2" : "another_field_2"
}

Make sure that your index is part of the Security Solution data view
To generate an alert using the above data (make sure you are using a big look-back time for the rule in order to fit the @timestamp)

Steps to reproduce:

Navigate to the alerts page
Click on the fields browser
Search for dummy
Select dummy_field_1
Click on Close
Click on the proper Sort direction to see if the values can be sorted

Current behavior:

The values are not sorted
An error is displayed

Expected behavior:

The values are properly sorted
No error message is displayed

Additional information:

Once the error is displayed, if you navigate to a different page and you come back to the alerts page, the error is displayed again, but, this time, no alerts are displayed as well

In order to solve the issue and be able to see the alerts again, you need to manually clean the Local Storage

The text was updated successfully, but these errors were encountered:

elasticmachine · 2022-05-05T12:17:05Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2022-05-05T12:17:05Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

@timestamp

… in timelines and the events / alerts tables: - <elastic#129603> - <elastic#123912> - <elastic#131625> The `unmapped_type` property [addition](https://github.com/elastic/kibana/pull/87241/files#diff-52fd5870dcd5f783f9fc8ac3a18a8674d83ac6136e09fe0e0bcae30427d61c3fR55) to the `sort` parameter of requests was using the `type` field metadata from `BrowserFields`, but the `type` metadata (for some fields) contains the value `string`, which is not a [valid field data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). The fix for the issues above: - Populates the `sort` property of requests with values from the `esTypes` `BrowserFields` metadata (instead of `type`) - The `esTypes` metadata may specify more than one field value type. When `esTypes` contains more than one type, and `keyword` is one of the types, the `sort` property of the request will prefer `keyword` over other the other types - When the field metadata has an empty `esTypes` collection, the `sort` property of the request will default to using `"unmapped_type": "keyword"` - The field type displayed in tooltips when hovering over columns in a timeline now displays values from `esTypes` instead of `type` To reproduce issue <elastic#129603> and to verify the fix: 1) Open Kibana `Dev tools` 2) Execute the following query to delete any exiting `logs-ti_test` index: ``` DELETE logs-ti_test ``` 3) Execute the following query to create an index named `logs-ti_test`, which has the following properities: - Dynamic mappings are disabled via `"dynamic": false` - It does NOT contain a mapping for `event.action` (we will sort by this field in later steps) - It contains a mapping for the non-ECS `testing` field ``` PUT logs-ti_test { "mappings": { "dynamic": false, "properties": { "@timestamp": { "type": "date" }, "event": { "properties": { "category": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "dataset": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "kind": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "host": { "properties": { "name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "testing": { "type": "keyword", "ignore_above": 1024 }, "threat": { "properties": { "indicator": { "properties": { "file": { "properties": { "hash": { "properties": { "md5": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } } } } } } } } } } ``` 4) Execute the following query to add a new document to the `logs-ti_test` index, and note that: - It does NOT contain a `event.action` field - It contains a value for the non-ECS `testing` field ``` POST logs-ti_test/_doc/ { "@timestamp": "2022-05-12T00:00:14.725Z", "host": { "name": "foozle" }, "threat": { "indicator": { "file": { "hash": { "md5": "a4f87cbcd2a4241da77b6bf0c5d9e8553fec991f" } } } }, "event": { "kind": "enrichment", "type": "indicator", "dataset": "ti_*", "category": "threat" }, "testing": "simulated threat intel data" } ``` 5) Navigate to the Security > Hosts page 6) Select `Last 1 year` from the date picker 7) Click the `Events` tab 8) Enter the following KQL query in the search bar at the top of the page: ``` host.name: foozle ``` 9) Hover over the `foozle` entry in the `host.name` column in the Events table, and click the `Add to timeline investigation` cell action 10) Open the timeline 11) Hover over the `event.action` field **Expected result** - The tooltip displays type `keyword` for the `event.action` field **Actual result** - The tooltip displays type `string` for the `event.action` field 12) Click the `event.action` column to add a secondary sort **Expected result** - The table is sorted by `@timestamp` and `event.action` - The table contents are (still) visible **Actual result** - The table is sorted by `@timestamp` and `event.action` - The contents of the table are now empty 13) Click the timeline's `Inspect` button 14) In the `Inspect Timeline` dialog, click the `Request` tab 15) Scroll down to the `sort` property of the request **Expected result** - The `event.action` field contains a `"unmapped_type": "keyword"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "number" } }, { "event.action": { "order": "desc", "unmapped_type": "keyword" } } ], ``` **Actual result** - The request's `event.action` field contains a `"unmapped_type": "string"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "number" } }, { "event.action": { "order": "desc", "unmapped_type": "string" } } ], ``` 16) In the `Inspect Timeline` dialog, click the `Response` tab **Expected result** - The response contains `0` `failed` shards / no failures **Actual result** - The response contains failures for the `logs-ti_test` index, with the following reason: ``` "reason": "No mapper found for type [string]" ``` per the example below: ```json { "took": 1, "timed_out": false, "_shards": { "total": 4, "successful": 3, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "logs-ti_test", "node": "NCRcGeDqSlKQiuPWVFvMEg", "reason": { "type": "illegal_argument_exception", "reason": "No mapper found for type [string]" } } ] }, ```

@timestamp

…132190) ## [Security Solution] Fixes sorting issues related to unmapped fields This PR fixes the following issues related to sorting unmapped fields in timelines and the events / alerts tables: - <#129603> - <#123912> - <#131625> The `unmapped_type` property [addition](https://github.com/elastic/kibana/pull/87241/files#diff-52fd5870dcd5f783f9fc8ac3a18a8674d83ac6136e09fe0e0bcae30427d61c3fR55) to the `sort` parameter of requests was using the `type` field metadata from `BrowserFields`, but the `type` metadata (for some fields) contains the value `string`, which is not a [valid field data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). The fix for the issues above: - Populates the `sort` property of requests with values from the `esTypes` `BrowserFields` metadata (instead of `type`) - The `esTypes` metadata may specify more than one field value type. When `esTypes` contains more than one type, and `keyword` is one of the types, the `sort` property of the request will prefer `keyword` over other the other types - When the field metadata has an empty `esTypes` collection, the `sort` property of the request will default to using `"unmapped_type": "keyword"` - The field type displayed in tooltips when hovering over columns in a timeline now displays values from `esTypes` instead of `type` ### Desk testing To reproduce issue <#129603> and to verify the fix: 1) Open Kibana `Dev tools` 2) Execute the following query to delete any exiting `logs-ti_test` index: ``` DELETE logs-ti_test ``` 3) Execute the following query to create an index named `logs-ti_test`, which has the following properities: - Dynamic mappings are disabled via `"dynamic": false` - It does NOT contain a mapping for `event.action` (we will sort by this field in later steps) - It contains a mapping for the non-ECS `testing` field ``` PUT logs-ti_test { "mappings": { "dynamic": false, "properties": { "@timestamp": { "type": "date" }, "event": { "properties": { "category": { "type": "keyword" }, "dataset": { "type": "keyword" }, "kind": { "type": "keyword" }, "type": { "type": "keyword" } } }, "host": { "properties": { "name": { "type": "keyword" } } }, "testing": { "type": "keyword", "ignore_above": 1024 }, "threat": { "properties": { "indicator": { "properties": { "file": { "properties": { "hash": { "properties": { "md5": { "type": "keyword" } } } } } } } } } } } } ``` 4) Execute the following query to add a new document to the `logs-ti_test` index, and note that: - It does NOT contain a `event.action` field - It contains a value for the non-ECS `testing` field ``` POST logs-ti_test/_doc/ { "@timestamp": "2022-05-12T00:00:14.725Z", "host": { "name": "foozle" }, "threat": { "indicator": { "file": { "hash": { "md5": "a4f87cbcd2a4241da77b6bf0c5d9e8553fec991f" } } } }, "event": { "kind": "enrichment", "type": "indicator", "dataset": "ti_*", "category": "threat" }, "testing": "simulated threat intel data" } ``` 5) Navigate to the Security > Hosts page 6) Select `Last 1 year` from the date picker 7) Click the `Events` tab 8) Enter the following KQL query in the search bar at the top of the page: ``` host.name: foozle ``` 9) Hover over the `foozle` entry in the `host.name` column in the Events table, and click the `Add to timeline investigation` cell action 10) Open the timeline 11) Hover over the `event.action` field **Expected result** - The tooltip displays type `keyword` for the `event.action` field **Actual result** - The tooltip displays type `string` for the `event.action` field 12) Click the `event.action` column to add a secondary sort **Expected result** - The table is sorted by `@timestamp` and `event.action` - The table contents are (still) visible **Actual result** - The table is sorted by `@timestamp` and `event.action` - The contents of the table are now empty 13) Click the timeline's `Inspect` button 14) In the `Inspect Timeline` dialog, click the `Request` tab 15) Scroll down to the `sort` property of the request **Expected result** - The `event.action` field contains a `"unmapped_type": "keyword"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "date" } }, { "event.action": { "order": "desc", "unmapped_type": "keyword" } } ], ``` **Actual result** - The request's `event.action` field contains a `"unmapped_type": "string"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "number" } }, { "event.action": { "order": "desc", "unmapped_type": "string" } } ], ``` 16) In the `Inspect Timeline` dialog, click the `Response` tab **Expected result** - The response contains `0` `failed` shards / no failures **Actual result** - The response contains failures for the `logs-ti_test` index, with the following reason: ``` "reason": "No mapper found for type [string]" ``` per the example below: ```json { "took": 1, "timed_out": false, "_shards": { "total": 4, "successful": 3, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "logs-ti_test", "node": "NCRcGeDqSlKQiuPWVFvMEg", "reason": { "type": "illegal_argument_exception", "reason": "No mapper found for type [string]" } } ] }, ```

@timestamp

…lastic#132190) This PR fixes the following issues related to sorting unmapped fields in timelines and the events / alerts tables: - <elastic#129603> - <elastic#123912> - <elastic#131625> The `unmapped_type` property [addition](https://github.com/elastic/kibana/pull/87241/files#diff-52fd5870dcd5f783f9fc8ac3a18a8674d83ac6136e09fe0e0bcae30427d61c3fR55) to the `sort` parameter of requests was using the `type` field metadata from `BrowserFields`, but the `type` metadata (for some fields) contains the value `string`, which is not a [valid field data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). The fix for the issues above: - Populates the `sort` property of requests with values from the `esTypes` `BrowserFields` metadata (instead of `type`) - The `esTypes` metadata may specify more than one field value type. When `esTypes` contains more than one type, and `keyword` is one of the types, the `sort` property of the request will prefer `keyword` over other the other types - When the field metadata has an empty `esTypes` collection, the `sort` property of the request will default to using `"unmapped_type": "keyword"` - The field type displayed in tooltips when hovering over columns in a timeline now displays values from `esTypes` instead of `type` To reproduce issue <elastic#129603> and to verify the fix: 1) Open Kibana `Dev tools` 2) Execute the following query to delete any exiting `logs-ti_test` index: ``` DELETE logs-ti_test ``` 3) Execute the following query to create an index named `logs-ti_test`, which has the following properities: - Dynamic mappings are disabled via `"dynamic": false` - It does NOT contain a mapping for `event.action` (we will sort by this field in later steps) - It contains a mapping for the non-ECS `testing` field ``` PUT logs-ti_test { "mappings": { "dynamic": false, "properties": { "@timestamp": { "type": "date" }, "event": { "properties": { "category": { "type": "keyword" }, "dataset": { "type": "keyword" }, "kind": { "type": "keyword" }, "type": { "type": "keyword" } } }, "host": { "properties": { "name": { "type": "keyword" } } }, "testing": { "type": "keyword", "ignore_above": 1024 }, "threat": { "properties": { "indicator": { "properties": { "file": { "properties": { "hash": { "properties": { "md5": { "type": "keyword" } } } } } } } } } } } } ``` 4) Execute the following query to add a new document to the `logs-ti_test` index, and note that: - It does NOT contain a `event.action` field - It contains a value for the non-ECS `testing` field ``` POST logs-ti_test/_doc/ { "@timestamp": "2022-05-12T00:00:14.725Z", "host": { "name": "foozle" }, "threat": { "indicator": { "file": { "hash": { "md5": "a4f87cbcd2a4241da77b6bf0c5d9e8553fec991f" } } } }, "event": { "kind": "enrichment", "type": "indicator", "dataset": "ti_*", "category": "threat" }, "testing": "simulated threat intel data" } ``` 5) Navigate to the Security > Hosts page 6) Select `Last 1 year` from the date picker 7) Click the `Events` tab 8) Enter the following KQL query in the search bar at the top of the page: ``` host.name: foozle ``` 9) Hover over the `foozle` entry in the `host.name` column in the Events table, and click the `Add to timeline investigation` cell action 10) Open the timeline 11) Hover over the `event.action` field **Expected result** - The tooltip displays type `keyword` for the `event.action` field **Actual result** - The tooltip displays type `string` for the `event.action` field 12) Click the `event.action` column to add a secondary sort **Expected result** - The table is sorted by `@timestamp` and `event.action` - The table contents are (still) visible **Actual result** - The table is sorted by `@timestamp` and `event.action` - The contents of the table are now empty 13) Click the timeline's `Inspect` button 14) In the `Inspect Timeline` dialog, click the `Request` tab 15) Scroll down to the `sort` property of the request **Expected result** - The `event.action` field contains a `"unmapped_type": "keyword"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "date" } }, { "event.action": { "order": "desc", "unmapped_type": "keyword" } } ], ``` **Actual result** - The request's `event.action` field contains a `"unmapped_type": "string"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "number" } }, { "event.action": { "order": "desc", "unmapped_type": "string" } } ], ``` 16) In the `Inspect Timeline` dialog, click the `Response` tab **Expected result** - The response contains `0` `failed` shards / no failures **Actual result** - The response contains failures for the `logs-ti_test` index, with the following reason: ``` "reason": "No mapper found for type [string]" ``` per the example below: ```json { "took": 1, "timed_out": false, "_shards": { "total": 4, "successful": 3, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "logs-ti_test", "node": "NCRcGeDqSlKQiuPWVFvMEg", "reason": { "type": "illegal_argument_exception", "reason": "No mapper found for type [string]" } } ] }, ``` (cherry picked from commit f2c8b2c)

@timestamp

…elds (#132190) (#132357) ## [Security Solution] Fixes sorting issues related to unmapped fields This PR fixes the following issues related to sorting unmapped fields in timelines and the events / alerts tables: - <#129603> - <#123912> - <#131625> The `unmapped_type` property [addition](https://github.com/elastic/kibana/pull/87241/files#diff-52fd5870dcd5f783f9fc8ac3a18a8674d83ac6136e09fe0e0bcae30427d61c3fR55) to the `sort` parameter of requests was using the `type` field metadata from `BrowserFields`, but the `type` metadata (for some fields) contains the value `string`, which is not a [valid field data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). The fix for the issues above: - Populates the `sort` property of requests with values from the `esTypes` `BrowserFields` metadata (instead of `type`) - The `esTypes` metadata may specify more than one field value type. When `esTypes` contains more than one type, and `keyword` is one of the types, the `sort` property of the request will prefer `keyword` over other the other types - When the field metadata has an empty `esTypes` collection, the `sort` property of the request will default to using `"unmapped_type": "keyword"` - The field type displayed in tooltips when hovering over columns in a timeline now displays values from `esTypes` instead of `type` ### Desk testing To reproduce issue <#129603> and to verify the fix: 1) Open Kibana `Dev tools` 2) Execute the following query to delete any exiting `logs-ti_test` index: ``` DELETE logs-ti_test ``` 3) Execute the following query to create an index named `logs-ti_test`, which has the following properities: - Dynamic mappings are disabled via `"dynamic": false` - It does NOT contain a mapping for `event.action` (we will sort by this field in later steps) - It contains a mapping for the non-ECS `testing` field ``` PUT logs-ti_test { "mappings": { "dynamic": false, "properties": { "@timestamp": { "type": "date" }, "event": { "properties": { "category": { "type": "keyword" }, "dataset": { "type": "keyword" }, "kind": { "type": "keyword" }, "type": { "type": "keyword" } } }, "host": { "properties": { "name": { "type": "keyword" } } }, "testing": { "type": "keyword", "ignore_above": 1024 }, "threat": { "properties": { "indicator": { "properties": { "file": { "properties": { "hash": { "properties": { "md5": { "type": "keyword" } } } } } } } } } } } } ``` 4) Execute the following query to add a new document to the `logs-ti_test` index, and note that: - It does NOT contain a `event.action` field - It contains a value for the non-ECS `testing` field ``` POST logs-ti_test/_doc/ { "@timestamp": "2022-05-12T00:00:14.725Z", "host": { "name": "foozle" }, "threat": { "indicator": { "file": { "hash": { "md5": "a4f87cbcd2a4241da77b6bf0c5d9e8553fec991f" } } } }, "event": { "kind": "enrichment", "type": "indicator", "dataset": "ti_*", "category": "threat" }, "testing": "simulated threat intel data" } ``` 5) Navigate to the Security > Hosts page 6) Select `Last 1 year` from the date picker 7) Click the `Events` tab 8) Enter the following KQL query in the search bar at the top of the page: ``` host.name: foozle ``` 9) Hover over the `foozle` entry in the `host.name` column in the Events table, and click the `Add to timeline investigation` cell action 10) Open the timeline 11) Hover over the `event.action` field **Expected result** - The tooltip displays type `keyword` for the `event.action` field **Actual result** - The tooltip displays type `string` for the `event.action` field 12) Click the `event.action` column to add a secondary sort **Expected result** - The table is sorted by `@timestamp` and `event.action` - The table contents are (still) visible **Actual result** - The table is sorted by `@timestamp` and `event.action` - The contents of the table are now empty 13) Click the timeline's `Inspect` button 14) In the `Inspect Timeline` dialog, click the `Request` tab 15) Scroll down to the `sort` property of the request **Expected result** - The `event.action` field contains a `"unmapped_type": "keyword"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "date" } }, { "event.action": { "order": "desc", "unmapped_type": "keyword" } } ], ``` **Actual result** - The request's `event.action` field contains a `"unmapped_type": "string"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "number" } }, { "event.action": { "order": "desc", "unmapped_type": "string" } } ], ``` 16) In the `Inspect Timeline` dialog, click the `Response` tab **Expected result** - The response contains `0` `failed` shards / no failures **Actual result** - The response contains failures for the `logs-ti_test` index, with the following reason: ``` "reason": "No mapper found for type [string]" ``` per the example below: ```json { "took": 1, "timed_out": false, "_shards": { "total": 4, "successful": 3, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "logs-ti_test", "node": "NCRcGeDqSlKQiuPWVFvMEg", "reason": { "type": "illegal_argument_exception", "reason": "No mapper found for type [string]" } } ] }, ```

@timestamp

…ields (#132190) (#132369) ## [Security Solution] Fixes sorting issues related to unmapped fields This PR fixes the following issues related to sorting unmapped fields in timelines and the events / alerts tables: - <#129603> - <#123912> - <#131625> The `unmapped_type` property [addition](https://github.com/elastic/kibana/pull/87241/files#diff-52fd5870dcd5f783f9fc8ac3a18a8674d83ac6136e09fe0e0bcae30427d61c3fR55) to the `sort` parameter of requests was using the `type` field metadata from `BrowserFields`, but the `type` metadata (for some fields) contains the value `string`, which is not a [valid field data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). The fix for the issues above: - Populates the `sort` property of requests with values from the `esTypes` `BrowserFields` metadata (instead of `type`) - The `esTypes` metadata may specify more than one field value type. When `esTypes` contains more than one type, and `keyword` is one of the types, the `sort` property of the request will prefer `keyword` over other the other types - When the field metadata has an empty `esTypes` collection, the `sort` property of the request will default to using `"unmapped_type": "keyword"` - The field type displayed in tooltips when hovering over columns in a timeline now displays values from `esTypes` instead of `type` ### Desk testing To reproduce issue <#129603> and to verify the fix: 1) Open Kibana `Dev tools` 2) Execute the following query to delete any exiting `logs-ti_test` index: ``` DELETE logs-ti_test ``` 3) Execute the following query to create an index named `logs-ti_test`, which has the following properities: - Dynamic mappings are disabled via `"dynamic": false` - It does NOT contain a mapping for `event.action` (we will sort by this field in later steps) - It contains a mapping for the non-ECS `testing` field ``` PUT logs-ti_test { "mappings": { "dynamic": false, "properties": { "@timestamp": { "type": "date" }, "event": { "properties": { "category": { "type": "keyword" }, "dataset": { "type": "keyword" }, "kind": { "type": "keyword" }, "type": { "type": "keyword" } } }, "host": { "properties": { "name": { "type": "keyword" } } }, "testing": { "type": "keyword", "ignore_above": 1024 }, "threat": { "properties": { "indicator": { "properties": { "file": { "properties": { "hash": { "properties": { "md5": { "type": "keyword" } } } } } } } } } } } } ``` 4) Execute the following query to add a new document to the `logs-ti_test` index, and note that: - It does NOT contain a `event.action` field - It contains a value for the non-ECS `testing` field ``` POST logs-ti_test/_doc/ { "@timestamp": "2022-05-12T00:00:14.725Z", "host": { "name": "foozle" }, "threat": { "indicator": { "file": { "hash": { "md5": "a4f87cbcd2a4241da77b6bf0c5d9e8553fec991f" } } } }, "event": { "kind": "enrichment", "type": "indicator", "dataset": "ti_*", "category": "threat" }, "testing": "simulated threat intel data" } ``` 5) Navigate to the Security > Hosts page 6) Select `Last 1 year` from the date picker 7) Click the `Events` tab 8) Enter the following KQL query in the search bar at the top of the page: ``` host.name: foozle ``` 9) Hover over the `foozle` entry in the `host.name` column in the Events table, and click the `Add to timeline investigation` cell action 10) Open the timeline 11) Hover over the `event.action` field **Expected result** - The tooltip displays type `keyword` for the `event.action` field **Actual result** - The tooltip displays type `string` for the `event.action` field 12) Click the `event.action` column to add a secondary sort **Expected result** - The table is sorted by `@timestamp` and `event.action` - The table contents are (still) visible **Actual result** - The table is sorted by `@timestamp` and `event.action` - The contents of the table are now empty 13) Click the timeline's `Inspect` button 14) In the `Inspect Timeline` dialog, click the `Request` tab 15) Scroll down to the `sort` property of the request **Expected result** - The `event.action` field contains a `"unmapped_type": "keyword"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "date" } }, { "event.action": { "order": "desc", "unmapped_type": "keyword" } } ], ``` **Actual result** - The request's `event.action` field contains a `"unmapped_type": "string"` property, per the example below: ```json "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "number" } }, { "event.action": { "order": "desc", "unmapped_type": "string" } } ], ``` 16) In the `Inspect Timeline` dialog, click the `Response` tab **Expected result** - The response contains `0` `failed` shards / no failures **Actual result** - The response contains failures for the `logs-ti_test` index, with the following reason: ``` "reason": "No mapper found for type [string]" ``` per the example below: ```json { "took": 1, "timed_out": false, "_shards": { "total": 4, "successful": 3, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "logs-ti_test", "node": "NCRcGeDqSlKQiuPWVFvMEg", "reason": { "type": "illegal_argument_exception", "reason": "No mapper found for type [string]" } } ] }, ```

andrew-goldstein · 2022-05-18T02:45:40Z

Please test the fix for this issue in the following branches / releases:

main / 8.3.0 [Security Solution] Fixes sorting issues related to unmapped fields #132190
8.2 / 8.2.1 [8.2] [Security Solution] Fixes sorting issues related to unmapped fields (#132190) #132357
7.17 / 7.17.4 [7.17] [Security Solution] Fixes sorting issues related to unmapped fields (#132190) #132369

ghost · 2022-05-18T11:59:33Z

Hi @MadameSheema ,

We have Validated this issue on 8.2 Branch and observed that issue is Partially Fixed

Error is not occurring on when apply the sorting on Non-ECS fields. However, Sorting is not working for non-ECS fields.

Please find the below testing details:

Build Details:

Branch: 8.2 branch
Commit : f788645c2363933eefac58a8a1569777eaa9b75f

Screencast

non-ecs.mp4

We will validate this issue on 7.17 branch and will share our observation as well

Thanks!!

MadameSheema · 2022-05-18T12:09:23Z

@andrew-goldstein can you please take a look at the above? Thanks :)

andrew-goldstein · 2022-05-19T01:00:44Z

Hi @deepikakeshav-qasource, thanks for testing this issue! Regarding the test in the video above:

We wouldn't expect dummy_field_1 and dummy_field_2 to be sortable in the Alerts table, because the alerts index (e.g. .alerts-security.alerts-default) doesn't contain a mapping for dummy_field_1 and dummy_field_2. (Unmapped fields are not sortable.)
If the sample events that populate dummy_field_1 and dummy_field_2 in the video above also have a host.name, the events that triggered the alerts should:
- Be visible in the Host > Events table (if the events have a host.name)
- Both non-ECS fields should be sortable in the Host > Events table

Would you be willing to index additional events (that have a host.name) and verify they are sortable in the Host > Events table?

andrew-goldstein · 2022-05-19T05:26:54Z

@deepikakeshav-qasource while desk testing, I found two (unrelated) issues with non-ECS fields, and assigned them to the Explore and Investigations teams respectively:

Explore: [Security Solution] The useEffect that cleans removed runtime fields prevents unmapped fields from being added as columns in timelines / Host > Events and Alerts #132489
- CC: @stephmilovic
Investigations: [Security Solution] Non-ECS columns for fields that are only one level deep can't be sorted by clicking on the column header in a timeline, and display incomplete metadata in the tooltip #132490
- CC: @michaelolo24

CC: @MadameSheema

ghost · 2022-05-19T07:14:30Z

Hi @andrew-goldstein ,

Thank you for the details!!

We have validated this issue on 8.2.1 BC1. This issue is occurring on 8.2.1 BC1 build 🔴 . Looks like backport is not merged on 8.2.1 BC1

Please find the below testing Details:

Build info

Version: 8.2.1 BC1
Build: 52170
Commit: 88889128ccda6f7e13983b0087e9aaa183818473

Error is coming on sorting for non-ECS fields under alerts Page:

Screencast:

Alerts.Page.mp4

We wouldn't expect dummy_field_1 and dummy_field_2 to be sortable in the Alerts table, because the alerts index (e.g. .alerts-security.alerts-default) doesn't contain a mapping for dummy_field_1 and dummy_field_2. (Unmapped fields are not sortable.)

Thank you for the confirmation

Both non-ECS fields should be sortable in the Host > Events table

Sorting is working incorrectly.

Details of dummy_field_1 column under Host > Event Table

Total Event Count	Event having under dummy_field_1 column	Event having dash under dummy_field_1 column
11	2	9

On applying sorting under host event table events having Dash or events with no data under that field got hidden

Host_events.mp4

Expected result:

All events should be shown on applying sorting with no data under that field

Thanks!!

andrew-goldstein · 2022-05-19T20:19:05Z

Looks like backport is not merged on 8.2.1 BC1

None of the commits to the 8.2 Kibana branch made the day before BC1 was built, Tuesday, May 17th, were included in BC1. A new build candidate is in progress now to resolve this.

@deepikakeshav-qasource would you be willing to test it when the next BC is available?

ghost · 2022-05-20T05:13:26Z

Hi @andrew-goldstein ,

We have validated this issue on 8.2.1 BC2 and observe that issue is Fixed 🟢

Please find the below testing Details:

Build info

Version: 8.2.1 BC2
Build: 52192
Commit: d4427fadd63ee9face4438cb3a4e17aaec93afe4

Error is not coming on sorting for non-ECS fields under alerts Page:

Screencast:

Alerts.Page.mp4

Sorting is working correctly for non-ECS fields under Host > Events Tab.

Screencast:

Host.Page.mp4

We will validate this issue on 7.17.4 as well

cc : @MadameSheema

Thanks!!

ghost · 2022-05-23T12:09:25Z

Hi @andrew-goldstein ,

We have validated this issue on 7.17.4 BC1 and observe that issue is Fixed 🟢

Please find the below testing Details:

Build info

Version: 7.17.4 BC1
Build: 46909
Commit: a408358a8fc5671f5eb7985678a1733684441b37

Error is not coming and sorting is disable for non-ECS fields under alerts Page

Screencast:

Alerts.sorting.mp4

Sorting is working correctly for non-ECS fields under Host > Events Tab.

Screencast:

Events.sorting.mp4

cc : @MadameSheema

Thanks!!

michaelolo24 · 2022-11-14T16:10:16Z

Closing this as it was validated as fixed and validated by QA here: #131625 (comment) Thanks again @deepikakeshav-qasource !

MadameSheema assigned michaelolo24 May 5, 2022

michaelolo24 removed their assignment May 12, 2022

michaelolo24 added impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. and removed triage_needed labels May 12, 2022

andrew-goldstein mentioned this issue May 13, 2022

[Security Solution] Fixes sorting issues related to unmapped fields #132190

Merged

michaelolo24 added the fixed label Nov 14, 2022

michaelolo24 added the QA:Validated Issue has been validated by QA label Nov 14, 2022

michaelolo24 closed this as completed Nov 14, 2022

ghost mentioned this issue Aug 1, 2023

[Security Solution]Copy to Clipboard not working for the Non-ECS Field #162784

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Sorting does not work on non-ECS fields #131625

[Security Solution] Sorting does not work on non-ECS fields #131625

MadameSheema commented May 5, 2022

elasticmachine commented May 5, 2022

elasticmachine commented May 5, 2022

andrew-goldstein commented May 18, 2022

ghost commented May 18, 2022

MadameSheema commented May 18, 2022

andrew-goldstein commented May 19, 2022

andrew-goldstein commented May 19, 2022

ghost commented May 19, 2022

andrew-goldstein commented May 19, 2022

ghost commented May 20, 2022

ghost commented May 23, 2022

michaelolo24 commented Nov 14, 2022

[Security Solution] Sorting does not work on non-ECS fields #131625

[Security Solution] Sorting does not work on non-ECS fields #131625

Comments

MadameSheema commented May 5, 2022

elasticmachine commented May 5, 2022

elasticmachine commented May 5, 2022

andrew-goldstein commented May 18, 2022

ghost commented May 18, 2022

MadameSheema commented May 18, 2022

andrew-goldstein commented May 19, 2022

andrew-goldstein commented May 19, 2022

ghost commented May 19, 2022

andrew-goldstein commented May 19, 2022

ghost commented May 20, 2022

ghost commented May 23, 2022

michaelolo24 commented Nov 14, 2022