[Security Solution] [Sourcerer] Update to use Kibana Data Views #111461

stephmilovic · 2021-09-07T20:27:17Z

Kibana Index Patterns Data Views & Runtime Fields in Security Solution Sourcerer

Issue: https://github.com/elastic/security-team/issues/772

Removes config index patterns and only use Kibana index patterns. Create a special security-solution KIP from the config index patterns. PR Reviewed: [Security Solution] [Sourcerer] KIP Feature Branch Kickoff, remove config index patterns #106460
Changes to the config index patterns to update the security-solution Kibana data view
Add signals index to default Kibana data view
Updates requestIndexFieldSearch in timelines to take a dataViewId
Switch to get fields directly from Kibana data view
Enable sourcerer to break up indices on each Kibana data view once selected
Update timeline sourcerer ui (pick_events.tsx)
Add dataViewId property to timeline, stored in references on the saved object

Quirks

the siem signals index needs to have a wildcard in the data view, but not in the security solution. The result is using strange checks like index.indexOf(signalIndexName) === 0 instead of index === signalIndexName
data view titles allow for duplicate indices, so we need to allow for duplicates to display accurate data view titles. For example: auditbeat-*,auditbeat-*,auditbeat-*,auditbeat-* and auditbeat-*,auditbeat-* are both valid. We're only going to allow them to select one instance of auditbeat from our combo box, but the select needs the full name so the user can distinguish between the data views and know where they put their runtime fields
- there is another spot where i know you'll be like, why check for duplicates here? its in case something wonky is in the url state
the data view gets updated when the Advanced Settings default index is updated securitySolution:defaultIndex. A page refresh is required for updates to take place
the siem signals index is appended to the end of the security-solution data view. on initial app load, or if the signals index gets deleted, signalIndexName is null and quickly set. We have a call to update the security-solution data view when the signalIndexName is updated
when the data view gets deleted and you navigate back to the security solution, a page refresh will be forced to regenerate the data view
to check if an index pattern is valid, we are now using esClient.fieldCaps instead of esClient.search or esClient.count as both of those methods require there to be data on the index pattern. When .siem-signals-default is initially created, it will not immediately have data. So using esClient.fieldCaps will tell us if the index exists even without data

UI Updates coming in a follow up PR. See Monina's excellent design direction in the original issue: https://github.com/elastic/security-team/issues/772

Sourcerer Updates to KIP/Data Views:

Runtime Fields in Security Solution:

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

…ourcerer_kip

x-pack/plugins/security_solution/public/common/components/top_n/selectors.tsx

x-pack/plugins/security_solution/public/common/components/url_state/initialize_redux_by_url.tsx

x-pack/plugins/security_solution/public/common/containers/source/index.tsx

XavierM · 2021-10-05T14:52:20Z

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx

+    const { id, ...rest } = defaultDataView;
+    if (id === null) {
+      // if id is null, rest is error
+      addError(rest, {


I am wondering if we can use internal kibana user to create this data view instead of a super user

I talked to @stephmilovic and we are using the saved object client from our route's context to create our data view and that's why we need to do it like that. I do think that we need a different splash screen to tell user that they do not have the right to create this dataview

XavierM · 2021-10-05T14:56:51Z

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx

-    path: [ALERTS_PATH, `${RULES_PATH}/id/:id`, `${UEBA_PATH}/:id`],
+): SourcererScopeName.default | SourcererScopeName.detections =>
+  matchPath(pathname, {
+    path: [ALERTS_PATH, RULES_PATH, `${RULES_PATH}/:anything`, `${UEBA_PATH}/:id`],


I think we just wanted to do that on the rule details page

x-pack/plugins/security_solution/public/common/store/sourcerer/helpers.ts

XavierM · 2021-10-05T16:45:00Z

x-pack/plugins/security_solution/public/common/store/sourcerer/model.ts

@@ -27,9 +28,14 @@ export interface ManageScope {
  docValueFields: DocValueFields[];
  errorMessage: string | null;
  id: SourcererScopeName;
+  // the index pattern value passed to the search to make the query
+  // includes fields and a title with active index names
  indexPattern: IIndexPattern;


I feel like we should no have anymore IIndexPattern since it is deprecated and use DataviewSpec.Maybe in another follow up PR

https://github.com/elastic/kibana/pull/111461/files#r721780757

XavierM · 2021-10-05T16:53:51Z

...k/plugins/security_solution/public/detections/pages/detection_engine/rules/details/index.tsx

-                          )}
-                        />
-                      </EuiFlexItem>
+      <StyledFullHeightContainer onKeyDown={onKeyDown} ref={containerElement}>


I do not understand what you mean here, I think it is still possible that user does not have any alert index created in this page. I think you are on something here but I will like to brainstorm.

x-pack/plugins/security_solution/public/plugin.tsx

stephmilovic · 2021-10-06T15:31:34Z

@elasticmachine merge upstream

stephmilovic · 2021-10-13T13:45:33Z

Closing in favor of: #114806

kibanamachine · 2021-10-13T16:58:04Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: f4d8061
Storybooks Preview
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures
Documentation Changes

Failed CI Steps

Test Failures

Kibana Pipeline / general / User sec_all_user with role(s) sec_all_role can create a case.Cases privileges User sec_all_user with role(s) sec_all_role can create a case

Link to Jenkins

Stack Trace

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

AssertionError: Timed out retrying after 60000ms: Expected to find element: `[data-test-subj="caseTitle"] [data-test-subj="input"]`, but never found it.
    at Object.fillCasesMandatoryfields (http://localhost:61171/__cypress/tests?p=cypress/integration/cases/privileges.spec.ts:33782:8)
    at Context.eval (http://localhost:61171/__cypress/tests?p=cypress/integration/cases/privileges.spec.ts:32794:31)

Kibana Pipeline / general / User sec_read_cases_all_user with role(s) sec_read_cases_all_role can create a case.Cases privileges User sec_read_cases_all_user with role(s) sec_read_cases_all_role can create a case

Link to Jenkins

Stack Trace

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

AssertionError: Timed out retrying after 60000ms: Expected to find element: `[data-test-subj="createNewCaseBtn"]`, but never found it.
    at Object.goToCreateNewCase (http://localhost:61171/__cypress/tests?p=cypress/integration/cases/privileges.spec.ts:33583:8)
    at Context.eval (http://localhost:61171/__cypress/tests?p=cypress/integration/cases/privileges.spec.ts:32793:25)

Kibana Pipeline / general / User sec_all_user with role(s) sec_all_role can create a case.Cases privileges User sec_all_user with role(s) sec_all_role can create a case

Link to Jenkins

Stack Trace

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://dryrun

AssertionError: Timed out retrying after 60000ms: Expected to find element: `[data-test-subj="createNewCaseBtn"]`, but never found it.
    at Object.goToCreateNewCase (http://localhost:61171/__cypress/tests?p=cypress/integration/cases/privileges.spec.ts:33583:8)
    at Context.eval (http://localhost:61171/__cypress/tests?p=cypress/integration/cases/privileges.spec.ts:32793:25)

and 1 more failures, only showing the first 3.

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`securitySolution`	1307	1319	+12
`timelines`	847	863	+16
total			+28

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`observability`	363.1KB	363.2KB	+19.0B
`securitySolution`	4.6MB	4.6MB	+7.9KB
`timelines`	240.0KB	240.2KB	+194.0B
total			+8.1KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	104.1KB	104.9KB	+787.0B
`timelines`	156.6KB	156.8KB	+166.0B
total			+953.0B

Unknown metric groups

API count

id	before	after	diff
`securitySolution`	1361	1373	+12
`timelines`	968	984	+16
total			+28

References to deprecated APIs

id	before	after	diff
`securitySolution`	1001	995	-6
`timelines`	84	91	+7
total			+1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

stephmilovic and others added 30 commits July 19, 2021 11:27

initialize default KIP

4deabab

rm configIndexPatterns

8c452c2

no more config patterns

fde9056

fix sourcerer jest

919a310

another test fix

3b6cf6a

Merge branch 'master' into sourcerer_kip

957bb97

more test fixes

8458b6f

rm some comments

c3df135

Merge branch 'master' into sourcerer_kip

0342da6

Merge branch 'master' into sourcerer_kip

af76d50

why

a386a11

fix cy

d10ec88

Merge branch 'master' into sourcerer_kip

33d44f1

merge

e79a9b1

k

3c13e86

move to server

6cc917c

rm cy com

343354d

ui for permissions and fix tests

86bc00d

pass as arg

255bc1f

fix mistakes

434f9bb

fix types

49098d4

merge master

de97889

fix ml test

ffc08eb

Merge branch 'master' into sourcerer_kip

23fba2c

start

385a91d

Merge branch 'master' into sourcerer_kip

11c741d

wip

4e81552

default sourcerer working me thinks

f4341d9

Merge branch 'sourcerer_kip' of github.com:stephmilovic/kibana into s…

32025be

…ourcerer_kip

Merge branch 'sourcerer_kip' into sourcerer_kip_as

e33abe3

XavierM reviewed Oct 4, 2021

View reviewed changes

x-pack/plugins/security_solution/public/common/components/top_n/selectors.tsx Outdated Show resolved Hide resolved

XavierM reviewed Oct 4, 2021

View reviewed changes

x-pack/plugins/security_solution/public/common/components/url_state/initialize_redux_by_url.tsx Outdated Show resolved Hide resolved

XavierM reviewed Oct 4, 2021

View reviewed changes

x-pack/plugins/security_solution/public/common/containers/source/index.tsx Show resolved Hide resolved

XavierM reviewed Oct 4, 2021

View reviewed changes

x-pack/plugins/security_solution/public/common/containers/source/index.tsx Outdated Show resolved Hide resolved

stephmilovic added 4 commits October 4, 2021 19:44

review ii

358cad8

fix cypress

d8cf285

revert

4df9c41

fix conflict

1d2e8ed

XavierM reviewed Oct 5, 2021

View reviewed changes

stephmilovic added 6 commits October 5, 2021 12:21

Merge branch 'master' into sourcerer_kip_as

f715950

revert some stuff

aebe8e0

add trick to cypress

1552c11

rm wildcard from pattern

36c4a7b

fix jest

87bd0ab

Merge branch 'master' into sourcerer_kip_as

732e3dc

kibanamachine and others added 6 commits October 6, 2021 11:31

Merge branch 'master' into sourcerer_kip_as

8e8d84c

fix timeline bug

b7a3caf

resolve conflicts

0f5c93b

Merge branch 'master' into sourcerer_kip_as

e04ffb0

fix routes

6e2a364

Resolve merge

7d804a2

stephmilovic added v8.0.0 and removed v7.16.0 labels Oct 11, 2021

stephmilovic added 2 commits October 11, 2021 08:29

Fix types

f6635c8

resolve conflicts

f4d8061

stephmilovic mentioned this pull request Oct 13, 2021

[Security Solution] [Sourcerer] [Feature Branch] Update to use Kibana Data Views #114806

Merged

3 tasks

stephmilovic closed this Oct 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] [Sourcerer] Update to use Kibana Data Views #111461

[Security Solution] [Sourcerer] Update to use Kibana Data Views #111461

stephmilovic commented Sep 7, 2021 •

edited

Loading

XavierM Oct 5, 2021

XavierM Oct 5, 2021

XavierM Oct 5, 2021

XavierM Oct 5, 2021

stephmilovic Oct 5, 2021

XavierM Oct 5, 2021

stephmilovic commented Oct 6, 2021

stephmilovic commented Oct 13, 2021

kibanamachine commented Oct 13, 2021 •

edited

Loading

Stack Trace

Stack Trace

Stack Trace

API count

References to deprecated APIs

[Security Solution] [Sourcerer] Update to use Kibana Data Views #111461

[Security Solution] [Sourcerer] Update to use Kibana Data Views #111461

Conversation

stephmilovic commented Sep 7, 2021 • edited Loading

Kibana Index Patterns Data Views & Runtime Fields in Security Solution Sourcerer

Quirks

Checklist

XavierM Oct 5, 2021

Choose a reason for hiding this comment

XavierM Oct 5, 2021

Choose a reason for hiding this comment

XavierM Oct 5, 2021

Choose a reason for hiding this comment

XavierM Oct 5, 2021

Choose a reason for hiding this comment

stephmilovic Oct 5, 2021

Choose a reason for hiding this comment

XavierM Oct 5, 2021

Choose a reason for hiding this comment

stephmilovic commented Oct 6, 2021

stephmilovic commented Oct 13, 2021

kibanamachine commented Oct 13, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Stack Trace

Stack Trace

Stack Trace

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

References to deprecated APIs

History

stephmilovic commented Sep 7, 2021 •

edited

Loading

kibanamachine commented Oct 13, 2021 •

edited

Loading