Skip to content

Commit

Permalink
This PR fixes the following issues related to sorting unmapped fields…
Browse files Browse the repository at this point in the history
… in timelines and the events / alerts tables:

- <elastic#129603>
- <elastic#123912>
- <elastic#131625>

The `unmapped_type` property [addition](https://github.com/elastic/kibana/pull/87241/files#diff-52fd5870dcd5f783f9fc8ac3a18a8674d83ac6136e09fe0e0bcae30427d61c3fR55) to the `sort` parameter of requests was using the `type` field metadata from `BrowserFields`, but the `type` metadata (for some fields) contains the value `string`, which is not a [valid field data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html).

The fix for the issues above:

- Populates the `sort` property of requests with values from the `esTypes` `BrowserFields` metadata (instead of `type`)
  - The `esTypes` metadata may specify more than one field value type. When `esTypes` contains more than one type, and `keyword` is one of the types, the `sort` property of the request will prefer `keyword` over other the other types
- When the field metadata has an empty `esTypes` collection, the `sort` property of the request will default to using `"unmapped_type": "keyword"`
- The field type displayed in tooltips when hovering over columns in a timeline now displays values from `esTypes` instead of `type`

To reproduce issue <elastic#129603> and to verify the fix:

1) Open Kibana `Dev tools`

2) Execute the following query to delete any exiting `logs-ti_test` index:

```
DELETE logs-ti_test
```

3) Execute the following query to create an index named `logs-ti_test`, which has the following properities:

- Dynamic mappings are disabled via `"dynamic": false`
- It does NOT contain a mapping for `event.action` (we will sort by this field in later steps)
- It contains a mapping for the non-ECS `testing` field

```
PUT logs-ti_test
{
  "mappings": {
    "dynamic": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "event": {
        "properties": {
          "category": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "dataset": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "kind": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "host": {
        "properties": {
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "testing": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "threat": {
        "properties": {
          "indicator": {
            "properties": {
              "file": {
                "properties": {
                  "hash": {
                    "properties": {
                      "md5": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

4) Execute the following query to add a new document to the `logs-ti_test` index, and note that:

- It does NOT contain a `event.action` field
- It contains a value for the non-ECS `testing` field

```
POST logs-ti_test/_doc/
{
  "@timestamp": "2022-05-12T00:00:14.725Z",
  "host": {
    "name": "foozle"
  },
  "threat": {
    "indicator": {
      "file": {
        "hash": {
          "md5": "a4f87cbcd2a4241da77b6bf0c5d9e8553fec991f"
        }
      }
    }
  },
  "event": {
    "kind": "enrichment",
    "type": "indicator",
    "dataset": "ti_*",
    "category": "threat"
  },
  "testing": "simulated threat intel data"
}
```

5) Navigate to the Security > Hosts page

6) Select `Last 1 year` from the date picker

7) Click the `Events` tab

8) Enter the following KQL query in the search bar at the top of the page:

```
host.name: foozle
```

9) Hover over the `foozle` entry in the `host.name` column in the Events table, and click the `Add to timeline investigation` cell action

10) Open the timeline

11) Hover over the `event.action` field

**Expected result**

- The tooltip displays  type `keyword` for the `event.action` field

**Actual result**

- The tooltip displays type `string` for the `event.action` field

12) Click the `event.action` column to add a secondary sort

**Expected result**

- The table is sorted by `@timestamp` and `event.action`
- The table contents are (still) visible

**Actual result**

- The table is sorted by `@timestamp` and `event.action`
- The contents of the table are now empty

13) Click the timeline's `Inspect` button

14) In the `Inspect Timeline` dialog, click the `Request` tab

15) Scroll down to the `sort` property of the request

**Expected result**

- The `event.action` field contains a `"unmapped_type": "keyword"` property, per the example below:

```json
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "number"
      }
    },
    {
      "event.action": {
        "order": "desc",
        "unmapped_type": "keyword"
      }
    }
  ],
  ```

**Actual result**

- The request's `event.action` field contains a `"unmapped_type": "string"` property, per the example below:

```json
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "number"
      }
    },
    {
      "event.action": {
        "order": "desc",
        "unmapped_type": "string"
      }
    }
  ],
  ```

16) In the `Inspect Timeline` dialog, click the `Response` tab

**Expected result**

- The response contains `0` `failed` shards / no failures

**Actual result**

- The response contains failures for the `logs-ti_test` index, with the following reason:

```
"reason": "No mapper found for type [string]"
```

per the example below:

```json
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 4,
    "successful": 3,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "logs-ti_test",
        "node": "NCRcGeDqSlKQiuPWVFvMEg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "No mapper found for type [string]"
        }
      }
    ]
  },
```
  • Loading branch information
andrew-goldstein committed May 16, 2022
1 parent e79ab07 commit 691c895
Show file tree
Hide file tree
Showing 60 changed files with 871 additions and 138 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ export interface TimelineRequestBasicOptions extends IEsSearchRequest {

export interface TimelineRequestSortField<Field = string> extends SortField<Field> {
type: string;
esTypes: string[];
}

export interface TimelineRequestOptionsPaginated<Field = string>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ export type SortDirection = 'none' | 'asc' | 'desc' | Direction;
export interface SortColumnTimeline {
columnId: string;
columnType: string;
esTypes?: string[];
sortDirection: SortDirection;
}

Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ export const defaultHeaders: ColumnHeaderOptions[] = [
columnHeaderType: defaultColumnHeaderType,
id: '@timestamp',
initialWidth: DEFAULT_DATE_COLUMN_MIN_WIDTH,
esTypes: ['date'],
type: 'date',
},
{
columnHeaderType: defaultColumnHeaderType,
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 691c895

Please sign in to comment.