Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Recorded Future support to threatintel module #26481

Merged
merged 3 commits into from
Jun 28, 2021
Merged

Add Recorded Future support to threatintel module #26481

merged 3 commits into from
Jun 28, 2021

Conversation

adriansr
Copy link
Contributor

This adds a new fileset, recordedfuture, to the treatintel module. It ingests indicators via the Recorded Future Connect API.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Screenshots

image

@adriansr
Add Recorded Future support to threatintel module
0625638 
This adds a new fileset, `recordedfuture`, to the treatintel module. It
ingests indicators via the Recorded Future Connect API.
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 24, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 24, 2021 •
edited by jenkins-beats-ci bot
Loading

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: Pull request #26481 updated

  • Start Time: 2021-06-27T16:26:58.662+0000

  • Duration: 183 min 34 sec

  • Commit: 3e2d0e0

Test stats 🧪

Test Results
Failed 0
Passed 12505
Skipped 1972
Total 14477

Trends 🧪

Image of Build Times

Image of Tests

Log output

Expand to view the last 100 lines of log output 

[2021-06-27T17:55:27.542Z] 3.50s call     filebeat/tests/system/test_registrar.py::Test::test_state_after_rotation
[2021-06-27T17:55:27.542Z] 3.47s call     filebeat/tests/system/test_reload_inputs.py::Test::test_start_stop
[2021-06-27T17:55:27.542Z] 3.46s call     filebeat/tests/system/test_reload_modules.py::Test::test_start_stop
[2021-06-27T17:55:27.542Z] 3.28s call     filebeat/tests/system/test_shutdown.py::Test::test_shutdown
[2021-06-27T17:55:27.542Z] 2.91s call     filebeat/tests/system/test_multiline.py::Test::test_timeout
[2021-06-27T17:55:27.542Z] 2.49s call     filebeat/tests/system/test_registrar.py::Test::test_registrar_files_with_input_level_processors
[2021-06-27T17:55:27.542Z] ================ 166 passed, 183 skipped in 206.02s (0:03:26) =================
[2021-06-27T17:55:27.542Z] >> python test: Unit Testing Complete
[2021-06-27T17:55:27.885Z] 
[2021-06-27T17:55:27.885Z] C:\Users\jenkins\workspace\PR-26481-3-654d075a-3954-4bc5-8333-fcdaed68f56e\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-06-27T17:55:30.732Z] 
[2021-06-27T17:55:30.732Z] C:\Users\jenkins\workspace\PR-26481-3-654d075a-3954-4bc5-8333-fcdaed68f56e\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-27T17:55:36.000Z] Copy .\filebeat\build into build\filebeat\build
[2021-06-27T17:55:36.000Z] Copy .\filebeat\null\build into build\filebeat\null\build
[2021-06-27T17:55:36.012Z] Running in C:\Users\jenkins\workspace\PR-26481-3-654d075a-3954-4bc5-8333-fcdaed68f56e\src\github.com\elastic\beats\build
[2021-06-27T17:55:36.027Z] Recording test results
[2021-06-27T17:55:37.193Z] [Checks API] No suitable checks publisher found.
[2021-06-27T17:55:37.537Z] 
[2021-06-27T17:55:37.537Z] C:\Users\jenkins\workspace\PR-26481-3-654d075a-3954-4bc5-8333-fcdaed68f56e\src\github.com\elastic\beats>go clean -modcache 
[2021-06-27T17:55:51.523Z] ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\jenkins\\AppData\\Local\\Temp\\pip-uninstall-rs3aawsg\\pip.exe'
[2021-06-27T17:55:51.523Z] Consider using the `--user` option or check the permissions.
[2021-06-27T17:55:51.523Z] 
[2021-06-27T17:56:11.847Z] + gsutil --version
[2021-06-27T17:56:13.810Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-27T17:56:14.122Z] + gcloud auth activate-service-account --key-file ****
[2021-06-27T17:56:15.075Z] Activated service account credentials for: [[email protected]]
[2021-06-27T17:56:15.401Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtNy13aW5kb3dzLTczZTJkMGUwZDM5ODE4N2E0YjczYTAwN2FmMTg1ZTRlOWU4MThkZWQy gs://beats-ci-temp/ci/cache/
[2021-06-27T17:56:46.637Z] + gsutil --version
[2021-06-27T17:56:48.609Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-27T17:56:48.928Z] + gcloud auth activate-service-account --key-file ****
[2021-06-27T17:56:49.881Z] Activated service account credentials for: [[email protected]]
[2021-06-27T17:56:50.202Z] + gsutil -m -q cp -a public-read ZmlsZWJlYXQtd2luZG93cy0xMC13aW5kb3dzLTEwM2UyZDBlMGQzOTgxODdhNGI3M2EwMDdhZjE4NWU0ZTllODE4ZGVkMg gs://beats-ci-temp/ci/cache/
[2021-06-27T17:57:13.299Z] warn: failed to upgrade pip (ignoring): running "null\build\ve\windows\Scripts\pip install -U pip" failed with exit code 1============================= test session starts =============================
[2021-06-27T17:57:13.299Z] platform win32 -- Python 3.8.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
[2021-06-27T17:57:13.299Z] rootdir: C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats, configfile: pytest.ini
[2021-06-27T17:57:13.299Z] plugins: rerunfailures-9.1.1, timeout-1.4.2
[2021-06-27T17:57:13.299Z] timeout: 90.0s
[2021-06-27T17:57:13.299Z] timeout method: thread
[2021-06-27T17:57:13.299Z] timeout func_only: True
[2021-06-27T17:57:13.299Z] collected 167 items
[2021-06-27T17:57:13.299Z] 
[2021-06-27T17:57:20.830Z] tests\system\test_filebeat_xpack.py .....                                [  2%]
[2021-06-27T17:57:56.204Z] tests\system\test_http_endpoint.py ...................                   [ 14%]
[2021-06-27T17:57:56.204Z] tests\system\test_xpack_modules.py sssssssssssssssssssssssssssssssssssss [ 36%]
[2021-06-27T17:57:56.480Z] ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 79%]
[2021-06-27T17:57:56.480Z] ssssssssssssssssssssssssssssssssss                                       [100%]
[2021-06-27T17:57:56.480Z] 
[2021-06-27T17:57:56.480Z] - generated xml file: C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats\x-pack\filebeat\build\TEST-python-unit.xml -
[2021-06-27T17:57:56.480Z] ============================ slowest 20 durations =============================
[2021-06-27T17:57:56.480Z] 4.08s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_index_pattern
[2021-06-27T17:57:56.480Z] 4.01s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_index_pattern_migration
[2021-06-27T17:57:56.480Z] 3.43s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_template
[2021-06-27T17:57:56.480Z] 2.41s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_config
[2021-06-27T17:57:56.480Z] 2.14s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_request
[2021-06-27T17:57:56.480Z] 2.11s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_request_ndjson
[2021-06-27T17:57:56.480Z] 2.06s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_request_multiple_documents
[2021-06-27T17:57:56.480Z] 2.01s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_valid_hmac
[2021-06-27T17:57:56.480Z] 2.01s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_get_request
[2021-06-27T17:57:56.480Z] 2.01s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_include_headers_without_header
[2021-06-27T17:57:56.480Z] 1.98s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_include_headers_not_canonical_config
[2021-06-27T17:57:56.480Z] 1.98s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_correct_auth_header
[2021-06-27T17:57:56.480Z] 1.96s call     x-pack/filebeat/tests/system/test_filebeat_xpack.py::FilebeatXPackTest::test_export_ilm_policy
[2021-06-27T17:57:56.480Z] 1.96s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_preserve_original_event
[2021-06-27T17:57:56.480Z] 1.95s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_auth_value
[2021-06-27T17:57:56.480Z] 1.95s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_empty_body
[2021-06-27T17:57:56.480Z] 1.95s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_wrong_auth_header
[2021-06-27T17:57:56.480Z] 1.94s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_include_headers_empty_value
[2021-06-27T17:57:56.480Z] 1.94s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_include_headers_single_value
[2021-06-27T17:57:56.480Z] 1.94s call     x-pack/filebeat/tests/system/test_http_endpoint.py::Test::test_http_endpoint_invalid_hmac
[2021-06-27T17:57:56.480Z] ====================== 24 passed, 143 skipped in 53.38s =======================
[2021-06-27T17:57:56.741Z] >> python test: Unit Testing Complete
[2021-06-27T17:57:57.084Z] 
[2021-06-27T17:57:57.085Z] C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-06-27T17:57:59.316Z] 
[2021-06-27T17:57:59.316Z] C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-27T17:58:00.270Z] Copy .\x-pack\filebeat\build into build\x-pack\filebeat\build
[2021-06-27T17:58:00.270Z] Copy .\x-pack\filebeat\null\build into build\x-pack\filebeat\null\build
[2021-06-27T17:58:00.283Z] Running in C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats\build
[2021-06-27T17:58:00.298Z] Recording test results
[2021-06-27T17:58:01.665Z] [Checks API] No suitable checks publisher found.
[2021-06-27T17:58:02.015Z] 
[2021-06-27T17:58:02.015Z] C:\Users\jenkins\workspace\PR-26481-3-8b885fb8-5e8f-4472-baa3-95aff92ca9bc\src\github.com\elastic\beats>go clean -modcache 
[2021-06-27T17:59:11.646Z] + gsutil --version
[2021-06-27T17:59:13.608Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-27T17:59:13.922Z] + gcloud auth activate-service-account --key-file ****
[2021-06-27T17:59:14.502Z] Activated service account credentials for: [[email protected]]
[2021-06-27T17:59:15.082Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LXdpbmRvd3MtMTAtd2luZG93cy0xMDNlMmQwZTBkMzk4MTg3YTRiNzNhMDA3YWYxODVlNGU5ZTgxOGRlZDI gs://beats-ci-temp/ci/cache/
[2021-06-27T19:29:31.747Z] Cancelling nested steps due to timeout
[2021-06-27T19:29:31.780Z] Failed in branch filebeat-windows-8-windows-8
[2021-06-27T19:29:31.803Z] Failed in branch x-pack/filebeat-windows-8-windows-8
[2021-06-27T19:29:31.867Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-27T19:29:31.895Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-27T19:29:31.944Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26481/src/github.com/elastic/beats
[2021-06-27T19:29:32.218Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26481
[2021-06-27T19:29:32.263Z] [INFO] getVaultSecret: Getting secrets
[2021-06-27T19:29:32.300Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-27T19:29:32.943Z] + chmod 755 generate-build-data.sh
[2021-06-27T19:29:32.943Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26481/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26481/runs/3 ABORTED 10954019
[2021-06-27T19:29:32.943Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26481/runs/3/steps/?limit=10000 -o steps-info.json
[2021-06-27T19:29:34.286Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26481/runs/3/tests/?status=FAILED -o tests-errors.json

@mergify
Copy link
Contributor

mergify bot commented Jun 24, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b recorded_future upstream/recorded_future
git merge upstream/master
git push upstream recorded_future

andrewkroh
andrewkroh approved these changes Jun 25, 2021
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adriansr
Copy link
Contributor Author

CI failure is due to Windows workers being offline. Merging.

@adriansr adriansr merged commit 6d89566 into elastic:master Jun 28, 2021
mergify bot pushed a commit that referenced this pull request Jun 28, 2021
@adriansr
Add Recorded Future support to threatintel module (#26481)
cb591a9 
This adds a new fileset, `recordedfuture`, to the treatintel module. It
ingests indicators via the Recorded Future Connect API.

(cherry picked from commit 6d89566)
@mergify mergify bot mentioned this pull request Jun 28, 2021
Merged
adriansr added a commit that referenced this pull request Jun 28, 2021
@mergify @adriansr
Add Recorded Future support to threatintel module (#26481) (#26516)
dbe17e4 
This adds a new fileset, `recordedfuture`, to the treatintel module. It
ingests indicators via the Recorded Future Connect API.

(cherry picked from commit 6d89566)

Co-authored-by: Adrian Serrano <[email protected]>
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 29, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
d4a87cb 
* master:
  Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545)
  [Heartbeat] add screenshots config to synthetics (elastic#26455)
  [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474)
  Remove all docs about  Beats central management (elastic#26399)
  update data.json for gcp billing (elastic#26506)
  Skip x-pack metricbeat tests (elastic#26537)
  [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529)
  Add changelog entry for  elastic#26224 (elastic#26531)
  Add inttests for the x-pack/metricbeat on a PR/branches basis (elastic#26526)
  Suppress too many errors (elastic#26224)
  Fix master's linting issue (elastic#26517)
  [libbeat] Fix encoding and file offset issues in the disk queue (elastic#26484)
  Add log_group_name_prefix config option for aws-cloudwatch input (elastic#26187)
  Update shared-deduplication.asciidoc (elastic#26492)
  Add Recorded Future support to threatintel module (elastic#26481)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees
No one assigned
Labels
backport-v7.14.0 Automated backport with mergify enhancement review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@adriansr @elasticmachine @andrewkroh