[CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations #115940

nkhristinin · 2021-10-21T13:40:40Z

Summary

Instruction on how to install fleet and elastic-agent locally: https://docs.google.com/document/d/17QQTVi3Rk_Sh0BctNKCujucV3PApZvpbTDtWUvE8AX8/edit?usp=sharing

Video demo

Screen.Recording.2021-11-22.at.18.36.43.mov

Changes

Naming - In the code I try to make the difference between integrations and data source
Integrations - it's something related to fleet packages, which you can install. For example AbuseCH.

Data source - it's related to event.dataset field. For example, one integration AbuseCH has 3 data sources - AbuseCH URL, AbuseCH Malware, AbuseCH Malware.

Added new searchStrategy, which looks for threat data sources with aggregation by threat.feed.name and threat.feed.dashboard_id
Initially we get all available data sources. We use this list, to be able later to show 0 when the data source doesn't have any events in the time range.
We fetch all TI integration packages from fleet API. We show a warning if there are some not installed integrations.
Removed some components and hooks which are not used.

How to test:

At that moment, the fleet integrations don't have threat.feed.name, but filebeat-8 has it.

Fleet integrations

Use instruction from the beginning to run Kibana with the fleet agent.
Install AbuseCH.
The overview card should display it's in the Others row. It's fine now, because the packages don't have threat.feed.name

Filebeat

Pull the latest changes of beats
add file beats as described here
In Advanced settings, change securitySolution:defaultThreatIndex to : logs-ti_*, filebeat-8*

You should see the filebeat data source on Overview card.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

nkhristinin · 2021-10-21T13:44:38Z

@elasticmachine merge upstream

nkhristinin · 2021-10-25T08:52:20Z

@elasticmachine merge upstream

elasticmachine · 2021-10-25T16:26:29Z

Pinging @elastic/security-solution (Team: SecuritySolution)

nkhristinin · 2021-10-26T13:26:37Z

@elasticmachine merge upstream

nkhristinin · 2021-10-26T15:49:31Z

@elasticmachine merge upstream

ecezalp

Code looks great. I had some issues with the setup and wasn't able to obtain indicators, I will review again once I can locally test the feature.

...ugins/security_solution/public/overview/components/overview_cti_links/cti_enabled_module.tsx

...ck/plugins/security_solution/public/overview/components/overview_cti_links/cti_no_events.tsx

ecezalp · 2021-10-26T21:59:31Z

...curity_solution/public/overview/components/overview_cti_links/use_integrations_page_link.tsx

+import { useBasePath } from '../../../common/lib/kibana';
+
+export const useIntegrationsPageLink = () =>


it looks like this is not a hook, just a helper function. In that case we should probably rename it not to start with use. We could call it getIntegrationsPageLink or something along those lines. thanks for the modularization

But it is just a combination with other hookuseBasePath from x-pack/plugins/security_solution/public/common/lib/kibana/hooks.ts

And inside it uses useKibana which relay on useContext.

ecezalp · 2021-10-26T22:07:49Z

...ugins/security_solution/public/overview/containers/overview_cti_links/use_ti_integrations.ts

+}
+
+export const useTIIntegrations = () => {
+  const [TIIntegrationsStatus, setTIIntegrationsStatus] = useState<TIIntegrationStatus | null>(


not sure about TIIntegrationsStatus, I believe we don't want to start with a capital letter in this context, @rylnd seemed to have some opinions on this, maybe we can get his opinion here

ecezalp

LGTM

ecezalp · 2021-10-27T15:43:58Z

@elasticmachine merge upstream

ecezalp · 2021-10-27T17:23:40Z

@elasticmachine merge upstream

nkhristinin · 2021-10-28T09:52:06Z

@elasticmachine merge upstream

nkhristinin · 2021-10-28T12:25:29Z

@elasticmachine merge upstream

rylnd

I think we can remove the 7.16.0 tag here and should target 8.0 instead, right?

rylnd · 2021-10-27T14:52:33Z

.../plugins/security_solution/public/overview/components/overview_cti_links/cti_with_events.tsx

@@ -33,6 +38,7 @@ export const CtiWithEventsComponent = ({
      isPluginDisabled={isPluginDisabled}
      listItems={listItems}
      totalCount={totalCount}
+      isSomeIntegrationsDisabled={isSomeIntegrationsDisabled}


Maybe at this level the prop should be closer to its semantic use:

Suggested change

isSomeIntegrationsDisabled={isSomeIntegrationsDisabled}

showIntegrationsCTA={isSomeIntegrationsDisabled}

nkhristinin · 2021-11-15T17:55:40Z

@elasticmachine merge upstream

ecezalp · 2021-11-15T19:50:31Z

@elasticmachine merge upstream

nkhristinin · 2021-11-30T18:16:19Z

@elasticmachine merge upstream

rylnd

Code review only. LGTM.

rylnd · 2021-12-02T03:43:54Z

...tegy/security_solution/factory/cti/threat_intel_source/query.threat_intel_source.dsl.test.ts

+ */
+
+import { buildTiDataSourceQuery } from './query.threat_intel_source.dsl';
+import { mockOptions, expectedDsl } from './__mocks__';


These __mocks__ files are a relic of jest auto-mocking that we aren't using, here. I would just name that file index.mock.ts instead (or e.g. query.threat_intel_source.dsl.mock.ts if that's more accurate)

Just move the mock value, exactly to the tests

x-pack/plugins/security_solution/public/overview/containers/overview_cti_links/api.ts

rylnd · 2021-12-02T03:53:01Z

...ugins/security_solution/public/overview/containers/overview_cti_links/use_ti_data_sources.ts

+      const datasets = result?.rawResponse?.aggregations?.dataset?.buckets ?? [];
+      const getChildAggregationValue = (aggregation?: Bucket) => aggregation?.buckets?.[0]?.key;
+
+      const integrationMap = datasets.reduce((acc: Record<string, TiDataSources>, dataset) => {


I assume this is going to be a relatively small number? If not, we might need to consider not creating new objects on each loop here.

Yes, it will be about how much data source clients have, not a big number

rylnd · 2021-12-02T03:55:45Z

...ugins/security_solution/public/overview/containers/overview_cti_links/use_ti_data_sources.ts

+  dashboardId?: string;
+}
+interface TiDataSourcesProps extends Partial<GlobalTimeArgs> {
+  allTiDataSources?: TiDataSources[];


I believe this prop can be made non-optional, no?

Suggested change

allTiDataSources?: TiDataSources[];

allTiDataSources: TiDataSources[];

Not really, we have a case when don't pass it, because we don't have it yet
x-pack/plugins/security_solution/public/overview/containers/overview_cti_links/use_all_ti_data_sources.ts

nkhristinin · 2021-12-02T18:25:17Z

@elasticmachine merge upstream

kibana-ci · 2021-12-02T20:07:05Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	2793	2791	-2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`securitySolution`	1325	1357	+32

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	4.6MB	4.6MB	-2.0KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`securitySolution`	33	30	-3

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	249.5KB	251.5KB	+2.1KB

Unknown metric groups

API count

id	before	after	diff
`securitySolution`	1374	1406	+32

References to deprecated APIs

id	before	after	diff
`securitySolution`	55	54	-1

History

💚 Build #10663 succeeded 3c16a96
💚 Build #10303 succeeded b61b368
💔 Build #10165 failed 2522dc8
💚 Build #10101 succeeded 8c8fbc5
💛 Build #9785 was flaky 53bb048
💚 Build #9223 succeeded 3b93684

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

MadameSheema

Checked just the Cypress updates, LGTM

…e Fleet TI integrations (#115940)" This reverts commit 6640357.

brianseeders · 2021-12-03T15:53:45Z

@nkhristinin I apologize, I had to revert this PR as it was causing CI to fail. It must have been incompatible with another change that was merged since the last time you merged upstream.

See failures here and here

kibanamachine · 2021-12-07T13:54:59Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 115940 or prevent reminders by adding the backport:skip label.

…TI integrations (elastic#115940) * Add support integrations * Fix types * fix unit tests * Fix tests and types * fix eslint * fix file case * add cy tests * Revert test * Add tests * Add support of installed integrations * Fix types * Add isntalled ingtegration case for cypress tests * Fix cypress tests * Fix comments * Fix capital naming * Fix again capital naming * Add dynamic dashboard for a new integrations packages * intermidiate changes, to keep it remote * Big refactoring * Tests and refactoring * Remove unused constanrs * Fix e2e tests * PR comments fix * fix ts * Fix translations * Remove stubs * Rename isSomeIntegrationsDisabled -> allIntegrationsInstalled * Add buildQuery tests * Fix type * Add tests for Enable Source button * Remove copied file * Move api call to api.ts * Rename fetchFleetIntegrations * Remove __mocks__ Co-authored-by: Kibana Machine <[email protected]>

…e Fleet TI integrations (elastic#115940)" This reverts commit 6640357.

nkhristinin requested review from ecezalp and rylnd October 21, 2021 13:40

nkhristinin added release_note:skip Skip the PR/issue when compiling release notes Team: CTI Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.16.0 v7.16.1 v8.0.0 and removed v7.16.1 labels Oct 21, 2021

nkhristinin marked this pull request as ready for review October 25, 2021 16:26

nkhristinin requested a review from a team as a code owner October 25, 2021 16:26

nkhristinin added the auto-backport Deprecated - use backport:version if exact versions are needed label Oct 26, 2021

ecezalp reviewed Oct 26, 2021

View reviewed changes

ecezalp approved these changes Oct 27, 2021

View reviewed changes

rylnd reviewed Nov 1, 2021

View reviewed changes

rylnd removed the v7.16.0 label Nov 2, 2021

nkhristinin force-pushed the cti-card-ecs branch from 09459b1 to e889d24 Compare November 15, 2021 17:54

elastic deleted a comment from kibanamachine Nov 15, 2021

nkhristinin force-pushed the cti-card-ecs branch from 838a107 to 554873a Compare November 19, 2021 15:25

Merge branch 'main' into cti-card-ecs

53bb048

kibanamachine and others added 8 commits November 30, 2021 13:16

Merge branch 'main' into cti-card-ecs

8c8fbc5

Remove stubs

a62193e

Rename isSomeIntegrationsDisabled -> allIntegrationsInstalled

1d2bb14

Add buildQuery tests

f563313

Fix type

b30e143

Add tests for Enable Source button

2522dc8

Remove copied file

b618acf

Move api call to api.ts

b61b368

rylnd approved these changes Dec 2, 2021

View reviewed changes

nkhristinin added 2 commits December 2, 2021 15:09

Rename fetchFleetIntegrations

a3a2e7f

Remove __mocks__

3c16a96

Merge branch 'main' into cti-card-ecs

17ec4ce

MadameSheema approved these changes Dec 3, 2021

View reviewed changes

nkhristinin merged commit 6640357 into elastic:main Dec 3, 2021

brianseeders added a commit that referenced this pull request Dec 3, 2021

Revert " [CTI] Threat Intel Card on Overview page needs to accommodat…

a0db43e

…e Fleet TI integrations (#115940)" This reverts commit 6640357.

nkhristinin mentioned this pull request Dec 6, 2021

[CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations - #120459

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 7, 2021

nkhristinin added backport:skip This commit does not require backporting and removed backport missing Added to PRs automatically when the are determined to be missing a backport. auto-backport Deprecated - use backport:version if exact versions are needed labels Dec 7, 2021

TinLe pushed a commit to TinLe/kibana that referenced this pull request Dec 22, 2021

Revert " [CTI] Threat Intel Card on Overview page needs to accommodat…

87e4bfc

…e Fleet TI integrations (elastic#115940)" This reverts commit 6640357.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations #115940

[CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations #115940

nkhristinin commented Oct 21, 2021 •

edited

Loading

nkhristinin commented Oct 21, 2021

nkhristinin commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

nkhristinin commented Oct 26, 2021

nkhristinin commented Oct 26, 2021

ecezalp left a comment

ecezalp Oct 26, 2021

nkhristinin Oct 27, 2021

ecezalp Oct 26, 2021

ecezalp left a comment

ecezalp commented Oct 27, 2021

ecezalp commented Oct 27, 2021

nkhristinin commented Oct 28, 2021

nkhristinin commented Oct 28, 2021

rylnd left a comment

rylnd Oct 27, 2021

nkhristinin commented Nov 15, 2021

ecezalp commented Nov 15, 2021

nkhristinin commented Nov 30, 2021

rylnd left a comment

rylnd Dec 2, 2021

nkhristinin Dec 2, 2021

rylnd Dec 2, 2021

nkhristinin Dec 2, 2021

rylnd Dec 2, 2021

nkhristinin Dec 2, 2021

nkhristinin commented Dec 2, 2021

kibana-ci commented Dec 2, 2021

API count

References to deprecated APIs

MadameSheema left a comment

brianseeders commented Dec 3, 2021

kibanamachine commented Dec 7, 2021

		import { useBasePath } from '../../../common/lib/kibana';

		export const useIntegrationsPageLink = () =>

	isSomeIntegrationsDisabled={isSomeIntegrationsDisabled}
	showIntegrationsCTA={isSomeIntegrationsDisabled}

	allTiDataSources?: TiDataSources[];
	allTiDataSources: TiDataSources[];

[CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations #115940

[CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations #115940

Conversation

nkhristinin commented Oct 21, 2021 • edited Loading

Summary

Video demo

Changes

How to test:

Fleet integrations

Filebeat

Checklist

nkhristinin commented Oct 21, 2021

nkhristinin commented Oct 25, 2021

elasticmachine commented Oct 25, 2021

nkhristinin commented Oct 26, 2021

nkhristinin commented Oct 26, 2021

ecezalp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecezalp left a comment

Choose a reason for hiding this comment

ecezalp commented Oct 27, 2021

ecezalp commented Oct 27, 2021

nkhristinin commented Oct 28, 2021

nkhristinin commented Oct 28, 2021

rylnd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkhristinin commented Nov 15, 2021

ecezalp commented Nov 15, 2021

nkhristinin commented Nov 30, 2021

rylnd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkhristinin commented Dec 2, 2021

kibana-ci commented Dec 2, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

References to deprecated APIs

History

MadameSheema left a comment

Choose a reason for hiding this comment

brianseeders commented Dec 3, 2021

kibanamachine commented Dec 7, 2021

nkhristinin commented Oct 21, 2021 •

edited

Loading