[ti_cif3] Add New TI integration for the Collective Intelligence Framework v3 #3839

mdavis332 · 2022-07-26T20:38:03Z

What does this PR do?

Added a new TI package integration for Collective Intelligence Framework v3 (CIFv3)
Added a data stream
Added data collection logic to the data stream
Added the ingest pipeline for the data stream
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files
Added dashboards and visualizations
Added test for pipeline for the data stream
Added system test cases for the data stream
Included test results were generated automatically

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo
Install elastic package locally
Start elastic stack using elastic-package stack up -v -d
Move to integrations/packages/ti_cif3 directory.
Run the following command to run tests:

elastic-package test

Screenshots

cla-checker-service · 2022-07-26T20:38:06Z

💚 CLA has been signed

elasticmachine · 2022-07-26T20:41:25Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-15T01:21:35.644+0000
Duration: 14 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	11
Skipped	0
Total	11

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

mdavis332 · 2022-07-26T20:51:34Z

/test

jsoriano · 2022-07-29T14:33:19Z

Hey @mdavis332, as mentioned in #3904 (comment), please check that you are using the same email address in your commits that you used to sign the CLA.

elasticmachine · 2022-07-29T14:34:22Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

peasead · 2022-08-02T14:26:56Z

What I really like about this is the inclusion of Threat Fox as a Provider. It's a high-quality and contextual feed.

peasead · 2022-08-02T14:31:06Z

packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml

+  - rename:
+      field: cif3.indicator
+      target_field: tls.client.ja3
+      ignore_missing: true
+      if: "ctx.cif3?.itype == 'md5' && ctx.cif3?.tags.contains('ja3')"


Looking at field reuse for the Threat ECS fieldset, I think this is a gap that needs to be addressed by adding tls. as a valid reuse field for threat.*.

Ideally, we'd want this to be something like threat.indicator.tls.client.ja3.

@ebeahan would we be able to make a parallel effort in adding TLS as a valid reuse field and adjusting this pipeline to be threat.indicator.tls.client.ja3 or do we need to adjust ECS first?

@peasead thanks so much for carefully looking at the integration. Although I'm not elastic staff, I agree with the suggestion of adding the TLS field for reuse under threat.indicator. I had the same thought about threat.indicator.network.cidr. Although it's not currently on the main ECS-threat reuse section, I went ahead and used that network.cidr field under threat.indicator due to having read through this related convo: elastic/beats#29949

Sure thing.

This all looks really good and we'll wait from @ebeahan to see what his thoughts are on the ECS side.

Apologies - missed the mention here until now.

No objection on the naming for threat.indicator.tls.client.ja3. Since the field doesn't exist in ECS, it will need defining in the integration itself.

If there's interest to add tls.* field reuse under threat.indicator.*, I think it's a small enough addition it can be suggested in a PR vs. through an ECS RFC.

Hi @ebeahan,

Since the field doesn't exist in ECS, it will need defining in the integration itself.

Would that involve tweaking data_stream/feed/fields/ecs.yml

- external: ecs name: tls.client.ja3

to

- external: ecs name: threat.indicator.tls.client.ja3

?

I think it's a small enough addition it can be suggested in a PR

Is that a suggestion for me or elastic staff? I'm not as familiar with https://github.com/elastic/ecs. I assume it would need to be added somewhere in https://github.com/elastic/ecs/blob/main/schemas/threat.yml, but I'm unsure.

It would need to be an explicit definition, analogous to this

integrations/packages/google_workspace/data_stream/groups/fields/ecs.yml

Lines 65 to 68 in ce52809

- name: source.geo.location

level: core

type: geo_point

description: Longitude and latitude.

This was done for geo.location before it was included in ECS formally as a geo_point type.

What needs to be added is just the type and the relevant details for the ja3 field from https://www.elastic.co/guide/en/ecs/current/ecs-tls.html#field-tls-client-ja3.

Thanks @efd6. I just commited b6e54db which hopefully resolves that suggestion.

mdavis332 · 2022-08-02T20:05:01Z

What I really like about this is the inclusion of Threat Fox as a Provider. It's a high-quality and contextual feed.

just so there's no unmet expectations, ThreatFox isn't inherently part of this integration or one of the default CIFv3 ingestion rules, but since it's public info and a worthwhile provider of info, I included it as our pipeline test example :) It's definitely possible to configure as a an ingestion rule in csirtg-smrt, the ingest module of CIFv3.

eriroley · 2022-08-09T12:16:56Z

Because I'm looking to use this - what is the current status of this integration? and what still needs to happen?
I see "integrations/pr-merge — The build of this commit was aborted" with "The PR is not allowed to run in the CI yet"

mdavis332 · 2022-08-16T02:10:31Z

Because I'm looking to use this - what is the current status of this integration? and what still needs to happen? I see "integrations/pr-merge — The build of this commit was aborted" with "The PR is not allowed to run in the CI yet"

hey @eriroley, I think we may be waiting on @ebeahan to give his thoughts on the ECS side of things. Not sure what else may be required.

mdavis332 · 2022-08-23T15:23:02Z

To anyone from the elastic team: just checking in to see if there's anything I can do to help move forward here.

peasead · 2022-08-23T19:18:59Z

Checking internally on the status.

mdavis332 · 2022-08-30T15:54:43Z

Checking internally on the status.

Hi @peasead, just my weekly checkin :)

There's an event coming up in the middle of September for which I'd love this to be ready to install by end users. If it's not possible to make it into the EPM by then, is there an easy way for end-users to download the integration and locally load it? Kinda like installing/side-loading a local plugin from zip or something?

Thanks for checking, and nice work on the recent blog post for the elastic container project.

jamiehynds · 2022-08-31T13:29:20Z

@mdavis332 apologies for the delay on getting your PR reviewed. We're working through a backlog of new integration PR's at the moment, so it's taking a little longer than expected. We'll certainly aim to have it published in time for your mid-September event.

@P1llus do you have the bandwidth to review this PR over the next week or so? No worries if not, can ask someone else on the team to review.

packages/ti_cif3/changelog.yml

packages/ti_cif3/manifest.yml

packages/ti_cif3/img/csg_logo_big.svg

packages/ti_cif3/_dev/build/docs/README.md

packages/ti_cif3/data_stream/feed/_dev/test/pipeline/test-common-config.yml

packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml

updated real PR link Co-authored-by: Dan Kortschak <[email protected]>

Add support for IPV6 addresses with ports by looking for surrounding brackets, e.g., `http://[2001:d34d:b33f:0000:0000:0000:d34d:b33f]:8080`

mdavis332 · 2022-09-06T15:04:47Z

Hi @efd6, thanks so much for your time doing a thorough review. I believe I've addressed all of your review feedback and left comments on each with referenced commits.

Please let me know if there's anything else to adjust at this time. Thank you again!

efd6 · 2022-09-06T22:32:47Z

/test

efd6 · 2022-09-06T23:20:32Z

The README.md will need to be rebuilt.

mdavis332 · 2022-09-07T00:21:57Z

The README.md will need to be rebuilt.

Done via commit 57a110a.

efd6 · 2022-09-07T01:28:55Z

/test

packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Dan Kortschak <[email protected]>

efd6 · 2022-09-07T02:28:10Z

/test

packages/ti_cif3/data_stream/feed/_dev/test/pipeline/test-cif3-sample.ndjson.log

efd6 · 2022-09-07T22:03:11Z

/test

elasticmachine · 2022-09-07T22:18:37Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 2.693
Classes	100.0% (`1/1`)	💚 2.693
Methods	85.714% (`12/14`)	👎 -3.984
Lines	39.615% (`103/260`)	👎 -51.789
Conditionals	100.0% (`0/0`)	💚

efd6

LGTM, but please wait for @ebeahan for comment on the ECS issue.

efd6 · 2022-09-07T22:43:20Z

Thanks

mdavis332 · 2022-09-07T22:53:46Z

LGTM, but please wait for @ebeahan for comment on the ECS issue.

Thanks again, Dan, for all your time reviewing and working through the CI stuff.

mdavis332 · 2022-09-13T13:24:57Z

Hi @jamiehynds, we're coming up on that mid-September timeframe, so just wanted to check back in this week. I appreciate everyone that's contributed to moving this PR forward.

packages/ti_cif3/data_stream/feed/fields/ecs.yml

jamiehynds · 2022-09-19T13:37:58Z

@efd6 is this PR ok to merge now? The only outstanding item was the TI ECS query, which Eric commented on and @mdavis332 has addressed. Thanks!

efd6 · 2022-09-20T04:33:36Z

🚀 Thanks @mdavis332

init commit

a67d1fa

mdavis332 mentioned this pull request Jul 29, 2022

[docs] Add contributor agreement prereq to dev docs #3904

Merged

jsoriano requested a review from a team July 29, 2022 14:34

jsoriano added the Team:Security-External Integrations label Jul 29, 2022

jamiehynds added the New Integration Issue or pull request for creating a new integration package. label Aug 2, 2022

peasead reviewed Aug 2, 2022

View reviewed changes

efd6 reviewed Sep 6, 2022

View reviewed changes

mdavis332 and others added 10 commits September 6, 2022 08:25

Update packages/ti_cif3/changelog.yml

7a6d916

updated real PR link Co-authored-by: Dan Kortschak <[email protected]>

update ecs reference

5c12a46

Update link used for REST API docs

d6c4fcb

Update to ecs 8.4.0

b0bf2eb

Update to ecs 8.4.0

70b27c8

Add security-external-integrations for ti_cif3 pkg

77099c5

Remove unnecessary yml

4e6fe71

Update to ECS 8.4.0

5cfc95b

Update URL grok pattern

478bb34

Add support for IPV6 addresses with ports by looking for surrounding brackets, e.g., `http://[2001:d34d:b33f:0000:0000:0000:d34d:b33f]:8080`

Document confidence conversion

696c2a0

Update logo to fix missing G

c66d43b

Rebuild README from new template

57a110a

efd6 reviewed Sep 7, 2022

View reviewed changes

packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Update grok pattern with additional escaping

7944a06

Co-authored-by: Dan Kortschak <[email protected]>

efd6 reviewed Sep 7, 2022

View reviewed changes

packages/ti_cif3/data_stream/feed/_dev/test/pipeline/test-cif3-sample.ndjson.log Outdated Show resolved Hide resolved

rename test files

d81c946

efd6 approved these changes Sep 7, 2022

View reviewed changes

Explicit definition for ja3 under threat.indicator

b6e54db

efd6 reviewed Sep 14, 2022

View reviewed changes

packages/ti_cif3/data_stream/feed/fields/ecs.yml Outdated Show resolved Hide resolved

Adjust ja3 field from core to extended

b557362

efd6 merged commit e44f73b into elastic:main Sep 20, 2022

	- name: source.geo.location
	level: core
	type: geo_point
	description: Longitude and latitude.

[ti_cif3] Add New TI integration for the Collective Intelligence Framework v3 #3839

[ti_cif3] Add New TI integration for the Collective Intelligence Framework v3 #3839

Conversation

mdavis332 commented Jul 26, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Screenshots

cla-checker-service bot commented Jul 26, 2022 • edited Loading

elasticmachine commented Jul 26, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

mdavis332 commented Jul 26, 2022

jsoriano commented Jul 29, 2022

elasticmachine commented Jul 29, 2022

peasead commented Aug 2, 2022

peasead Aug 2, 2022

Choose a reason for hiding this comment

mdavis332 Aug 2, 2022

Choose a reason for hiding this comment

peasead Aug 3, 2022

Choose a reason for hiding this comment

ebeahan Sep 14, 2022

Choose a reason for hiding this comment

mdavis332 Sep 14, 2022

Choose a reason for hiding this comment

efd6 Sep 14, 2022

Choose a reason for hiding this comment

mdavis332 Sep 14, 2022

Choose a reason for hiding this comment

mdavis332 commented Aug 2, 2022

eriroley commented Aug 9, 2022

mdavis332 commented Aug 16, 2022

mdavis332 commented Aug 23, 2022

peasead commented Aug 23, 2022

mdavis332 commented Aug 30, 2022

jamiehynds commented Aug 31, 2022

mdavis332 commented Sep 6, 2022

efd6 commented Sep 6, 2022

efd6 commented Sep 6, 2022

mdavis332 commented Sep 7, 2022

efd6 commented Sep 7, 2022

efd6 commented Sep 7, 2022

efd6 commented Sep 7, 2022

elasticmachine commented Sep 7, 2022 • edited Loading

🌐 Coverage report

efd6 left a comment

Choose a reason for hiding this comment

efd6 commented Sep 7, 2022

mdavis332 commented Sep 7, 2022

mdavis332 commented Sep 13, 2022

jamiehynds commented Sep 19, 2022

efd6 commented Sep 20, 2022

mdavis332 commented Jul 26, 2022 •

edited

Loading

cla-checker-service bot commented Jul 26, 2022 •

edited

Loading

elasticmachine commented Jul 26, 2022 •

edited

Loading

elasticmachine commented Sep 7, 2022 •

edited

Loading