[Security Solution] Non-ECS columns for fields that are only one level deep can't be sorted by clicking on the column header in a timeline, and display incomplete metadata in the tooltip

[andrew-goldstein]

[Security Solution] Non-ECS columns for fields that are only one level deep can't be sorted by clicking on the column header in a timeline, and display incomplete metadata in the tooltip

When non-ECS columns are added to a timeline for fields that are only one level deep, e.g. the mapping for the following non-ECS field:

      "test_field_1": {
        "type": "keyword",
        "ignore_above": 1024
      }

Above: a mapping for test_field_1 that's just one-level deep

• The column can't be sorted by clicking on the column header in a timeline
• Incomplete metadata for the field is displayed in the column header tooltip

Sorting and tooltips behave correctly when mappings for the non-ECS fields are more than one level deep, per the example below:

      "foo": {
        "properties": {
          "bar": {
            "type": "keyword"
          }
        }
      }

Above: a mapping for a field named foo.bar that's two levels deep

It appears the issue is related to this statement:

          ...get(
            [splitHeader.length > 1 ? splitHeader[0] : 'base', 'fields', header.id],
            browserFields
          ),

where the code in x-pack/plugins/security_solution/public/timelines/components/timeline/body/column_headers/helpers.ts above assumes that mappings for fields just one level deep will always be contained in the base BrowserFields category, (like message, and _id, which are included in the base category.)

It appears that this issue may be resolved by adding the following (new) function to x-pack/plugins/security_solution/public/timelines/components/timeline/body/column_headers/helpers.ts:

export const getRootCategory = ({
  browserFields,
  headerId,
}: {
  browserFields: BrowserFields;
  headerId: string;
}): string => (has(`base.fields.${headerId}`, browserFields) ? 'base' : headerId);

, which returns the correct category for fields just one level deep, and then calling it by replacing the current code:

          ...get(
            [splitHeader.length > 1 ? splitHeader[0] : 'base', 'fields', header.id],
            browserFields
          ),

with the following code that invokes the new getRootCategory function:

          ...get(
            [
              splitHeader.length > 1
                ? splitHeader[0]
                : getRootCategory({ headerId: header.id, browserFields }),
              'fields',
              header.id,
            ],
            browserFields
          ),

Kibana/Elasticsearch Stack version:

7.15.0, 8.0.0

Steps to reproduce:

1. Navigate to Kibana Dev Tools

2. Execute the following query to delete any existing logs-ti_non_ecs_test index

DELETE logs-ti_non_ecs_test

3. Create the logs-ti_non_ecs_test index via by executing the following query:

PUT logs-ti_non_ecs_test
{
  "mappings": {
    "dynamic": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "host": {
        "properties": {
          "name": {
            "type": "keyword"
          }
        }
      },
      "foo": {
        "properties": {
          "bar": {
            "type": "keyword"
          }
        }
      },
      "test_field_1": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "test_field_2": {
        "type": "keyword",
        "ignore_above": 1024
      }
    }
  }
}

Note that query above:

• Defines a mapping for a field named foo.bar that's two levels deep
• Also defines a mapping for test_field_1 that's just one level deep

4. Using the Chrome developer tools (NOT the Kibana Dev Tools), create a valid date string close to "now" by executing the following statement in the Console tab:

JSON.stringify(new Date())

as illustrated by the screenshot below:

Expected result

• A new datetime string close to "now", e.g.

"2022-05-18T23:54:27.943Z"

appears in the console, per the example in the screenshot (above).

5. In Kibana Dev Tools, edit the two POST queries below such that:

✅ the @timestamp in the first query below uses the new datetime string created in the previous step, e.g. "2022-05-18T23:54:27.943Z"
✅ the @timestamp in the second query below uses the new datetime string, one second later, to ensure the events have unique timestamps, e.g. "2022-05-18T23:54:28.943Z"

POST logs-ti_non_ecs_test/_doc/
{
  "@timestamp": "2022-05-18T23:54:27.943Z",
  "host": {
    "name": "foozle"
  },
  "foo": {
    "bar": "baz"
  },
  "test_field_1": "field_1",
  "test_field_2": "field_2",
  "test_unsortable_has_no_mapping": "unsortable_1"
}

POST logs-ti_non_ecs_test/_doc/
{
  "@timestamp": "2022-05-18T23:54:28.943Z",
  "host": {
    "name": "foozle"
  },
  "foo": {
    "bar": "apple"
  },
  "test_field_1": "another_field_1",
  "test_field_2": "another_field_2",
  "test_unsortable_has_no_mapping": "unsortable_2"
}

Note that the sample events in the queries above:

• Contain a host.name: "foozle" entry, to make it easy to find the sample events
• Contain values for a field named foo.bar that's two levels deep
• Contain values for test_field_1, which is just one level deep

6. In Kibana Dev Tools, execute the two queries above to index the documents

7. In the Security Solution, navigate to the Security > Overview page

8. Click the Untitled timeline button at the bottom of the page to open a timeline

9. Enter the KQL query:

host.name: "foozle"

in the search bar

Expected result

• Timeline displays the two sample events

10. Click the View details action for an event to display the Event details flyout

11. In the flyout, hover next to the test_field_1 field and click the ... button

12. Click the Toggle test_field_1 column in table action from the popover

Expected result

• The test_field_1 column, that's just one level deep, is added to the timeline

13. Once again In the flyout, hover next to the foo.bar column and click the ... button

14. Click the Toggle foo.bar column in table action from the popover

Expected result

• A column for foo.bar, that's two levels deep, is added to the timeline

15. Close the Event details panel

Expected results

• The timeline is sorted by @timestamp

• The foo.bar and test_field_1 columns appear in the table, per the screenshot below:

16. Hover over the foo.bar (two level) column

Expected result

• The tooltip for the foo.bar column displays type keyword, per the screenshot below:

17. Click on the foo.bar header to sort the column

Expected result

• Both the @timestamp and foo.bar columns are sorted, per the screenshot below:

18. Hover over the test_field_1 column

Expected result

• The tooltip for the test_field_1 column should display type keyword, per the sceenshot below:

Actual result

• The tooltip for the foo.bar column displays incomplete metadata, per the screenshot below:

19. Click on the test_field_1 header to sort the column

Expected result

• The timeline is now sorted by all three columns
  • @timestamp
  • foo.bar
  • test_field_1

Actual result

• The timeline is NOT sorted by test_field_1, per the screenshot below:

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[andrew-goldstein]

Fixes for this issue were merged into the following branches:

• main / v8.3.0 [Security Solution] Fixes sorting and tooltips on columns for non-ECS fields that are only one level deep #132570
• 8.2 / v8.2.2 [8.2] [Security Solution] Fixes sorting and tooltips on columns for non-ECS fields that are only one level deep (#132570) #132663
• 7.17 / v7.17.5 [7.17] [Security Solution] Fixes sorting and tooltips on columns for non-ECS fields that are only one level deep (#132570) #132665

[ghost]

Hi @andrew-goldstein,

We have validated above issue on 8.3.0 BC2 and it's working fine ✅.

Build info:

Version: 8.3.0 BC2
Build: 53231
Commit: 25476b531ba9f32292bde85508d342aa5e1c29eb

Below are the observations:

• A new datetime string close to "now", is shown on executing JSON.stringify(new Date())

• Timeline displays the two sample events on executing KQL Query host.name: "foozle"

• The test_field_1 column, that's just one level deep, is added to the timeline on clicking the ... button from the alerts flyout
• A column for foo.bar, that's two levels deep, is added to the timeline on clicking the ... button from the alerts flyout

• The timeline is sorted by @timestamp

• The tooltip for the foo.bar column displays type keyword on hovering over the foo.bar

• Both the @timestamp and foo.bar columns are sorted on clicking the foo.bar header to sort the column

• The tooltip for the test_field_1 column should display type keyword on hovering over the test_field_1

• The timeline is now sorted by all three columns on clicking the test_field_1 header to sort the column

1. @timestamp
2. foo.bar
3. test_field_1

Hence, marking this as QA validated.

Thanks!