[Security Solution] The useEffect that cleans removed runtime fields prevents unmapped fields from being added as columns in timelines / Host > Events and Alerts

[andrew-goldstein]

[Security Solution] The useEffect that cleans removed runtime fields prevents unmapped fields from being added as columns in timelines / Host > Events and Alerts

The useEffect that cleans removed runtime fields prevents unmapped fields from being added as columns to timelines.

There is a similar useEffect in x-pack/plugins/timelines/public/components/t_grid/body/index.tsx here that sometimes prevents unmapped fields from being added to the Host > Events and Alerts tables, but this behavior is not always reproducible. (The reproduction steps below will note this. The unexpected behavior is always reproducible in timeline, per the last reproduction step.)

The useEffects linked above immediately dispatch an uexpected REMOVE_COLUMN column action when an unmapped field is added as a column. This effectively prevents unmapped fields from being added to the tables, because columns are immediately (automatically) removed.

Kibana/Elasticsearch Stack version:

8.0.0

Steps to reproduce:

1. Navigate to Kibana Dev Tools

2. Execute the following query to delete any existing logs-ti_non_ecs_test index

DELETE logs-ti_non_ecs_test

3. Create the logs-ti_non_ecs_test index via by executing the following query:

PUT logs-ti_non_ecs_test
{
  "mappings": {
    "dynamic": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "host": {
        "properties": {
          "name": {
            "type": "keyword"
          }
        }
      },
      "foo": {
        "properties": {
          "bar": {
            "type": "keyword"
          }
        }
      },
      "test_field_1": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "test_field_2": {
        "type": "keyword",
        "ignore_above": 1024
      }
    }
  }
}

Note that query above:

• Defines a mapping for the field test_field_1
• Does NOT define a mapping for a field named test_unsortable_has_no_mapping.

4. Using the Chrome developer tools (NOT the Kibana Dev Tools), create a valid date string close to "now" by executing the following statement in the Console tab:

JSON.stringify(new Date())

as illustrated by the screenshot below:

Expected result

• A new datetime string close to "now", e.g.

"2022-05-18T23:54:27.943Z"

appears in the console, per the example in the screenshot (above).

5. In Kibana Dev Tools, edit the two POST queries below such that:

✅ the @timestamp in the first query below uses the new datetime string created in the previous step, e.g. "2022-05-18T23:54:27.943Z"
✅ the @timestamp in the second query below uses the new datetime string, one second later, to ensure the events have unique timestamps, e.g. "2022-05-18T23:54:28.943Z"

POST logs-ti_non_ecs_test/_doc/
{
  "@timestamp": "2022-05-18T23:54:27.943Z",
  "host": {
    "name": "foozle"
  },
  "foo": {
    "bar": "baz"
  },
  "test_field_1": "field_1",
  "test_field_2": "field_2",
  "test_unsortable_has_no_mapping": "unsortable_1"
}

POST logs-ti_non_ecs_test/_doc/
{
  "@timestamp": "2022-05-18T23:54:28.943Z",
  "host": {
    "name": "foozle"
  },
  "foo": {
    "bar": "apple"
  },
  "test_field_1": "another_field_1",
  "test_field_2": "another_field_2",
  "test_unsortable_has_no_mapping": "unsortable_2"
}

Note that the sample events in the queries above:

• Contain a host.name: "foozle" entry, to ensure the events show up in the Host > Events table (in later steps).
• Contain an entry for the mapped field test_field_1
• Contain an entry for the unmapped field test_unsortable_has_no_mapping

6. In Kibana Dev Tools, execute the two queries above to index the documents

7. In the Security Solution, navigate to the Hosts > Events table

8. Enter the KQL query:

host.name: "foozle"

in the search bar

Expected result

• The page is filtered to only display the two events inserted via Dev Tools

9. Click the View details action for an event to display the Event details flyout

10. In the flyout, hover next to the test_field_1 field and click the ... button

11. Click the Toggle test_field_1 column in table action from the popover

Expected result

• The mapped test_field_1 column is added to the Host > Events table

12. Once again In the flyout, hover next to the test_unsortable_has_no_mapping column and click the ... button

13. Click the Toggle test_unsortable_has_no_mapping column in table action from the popover, as shown in the screenshot below:

Expected result

• A column for the unmapped field test_unsortable_has_no_mapping is added to the Host > Events table

Actual result

• Sometimes, a column for the unmapped field test_unsortable_has_no_mapping is NOT added to the Host > Events table

14. Close the Event details panel

15. In the Host > Events table, click the Investigate in timeline action for an event

Expected result

• The event is opened in timeline

16. In Timeline, click the View details action to view the Event details (in Timeline)

17. In Timeline's Event details flyout, hover next to the test_field_1 field and click the ... button

18. Click the Toggle test_field_1 column in table action from the popover

Expected result

• The mapped test_field_1 column is added to the timeline

19. Once again In Timeline's Event details flyout, hover next to the test_unsortable_has_no_mapping column and click the ... button

20. Click the Toggle test_unsortable_has_no_mapping column in table action from the popover

Expected results

• A column for the unmapped field test_unsortable_has_no_mapping is added to the timeline
• (optional) The Redux dev tools only show an UPSERT_COLUMN action being dispatched, per the example below:

{
  type: 'x-pack/timelines/t-grid/UPSERT_COLUMN',
  payload: {
    column: {
      columnHeaderType: 'not-filtered',
      id: 'test_unsortable_has_no_mapping',
      initialWidth: 180
    },
    id: 'timeline-1',
    index: 1
  }
}

Actual results

• A column for the unmapped field test_unsortable_has_no_mapping is NOT added to the timeline
• This unexpected result is always reproducible. (Similar unexpected behavior in Host > Events was only sometimes reproducible.)
• (optional) In addition to the expected UPSERT_COLUMN action being dispatched, the Redux dev tools also show an unexpected REMOVE_COLUMN action being dispatched, per the example below:

{
  type: 'x-pack/timelines/t-grid/REMOVE_COLUMN',
  payload: {
    id: 'timeline-1',
    columnId: 'test_unsortable_has_no_mapping'
  }
}

Additional notes

When the useEffect that cleans removed runtime fields for timeline is commented-out, the unmapped field test_unsortable_has_no_mapping is added to the table as expected, and the unexpected REMOVE_COLUMN column action is NOT dispatched

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[ghost]

Hi @andrew-goldstein

we have validated this issue on 8.4.0-SNAPSHOT and found this issue to be fixed, ✔️ both column got added under event table and under timeline table on toggle action and redux have correct response of UPSERT_COLUMN and not REMOVE_COLUM.

Build info

Version:8.4.0 SNAPSHOT
Commit:976a669f92e0e516a87fefabf0642105e58065b4

Screen-Shoot:

we will going to retest this issue on BC4 so keeping the ticket open till then.

thanks !!

c.c @MadameSheema

[MadameSheema]

Fixed on BC4!!