Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjusted IOCTypes usage. #1156

Merged

Conversation

AWSHurneyt
Copy link
Collaborator

@AWSHurneyt AWSHurneyt commented Jul 10, 2024

Description

Adjusted IOCTypes.

The jar that includes the changes in PR https://github.com/opensearch-project/security-analytics-commons/pull/18 is included in this PR.

Testing

The SATIFSourceConfigRestApiIT test suite requires a additional system parameter inputs to execute; bucketName, region, and roleArn. The github CI is not currently configured to supply those inputs. Those tests passed locally with those inputs provided.

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
…mons, instead of the SA commons jar being generated withhout those dependencies.

Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
@AWSHurneyt
Copy link
Collaborator Author

Security, and multi-node workflows are failing for unrelated reasons. They also failed in previous PRs.
https://github.com/opensearch-project/security-analytics/pull/1133/checks

Will investigate those failures separately.

// add ioc type filter
boolQueryBuilder.must(new TermsQueryBuilder(STIX2.TYPE_FIELD, iocType.toLowerCase()));
boolQueryBuilder.must(new TermsQueryBuilder(STIX2.TYPE_FIELD + ".keyword", iocType.toLowerCase(Locale.ROOT)));
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, after changing the IOCTypes from using underscores (e.g., domain_name) to using hyphens (e.g., domain-name), it was necessary to append the fields with .keyword in order for this search request to return results. This change is needed to ensure the potentially-hyphenated values of the fields are being queried as-is.

Without these changes, the query returns 0 results despite a match_all query confirming there are docs in the concrete index. This was the behavior regardless of whether the index mappings were specified during index creation.

Signed-off-by: AWSHurneyt <[email protected]>
@AWSHurneyt AWSHurneyt merged commit a6eb64d into opensearch-project:main Jul 12, 2024
11 of 16 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.x
# Create a new branch
git switch --create backport-1156-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a6eb64d30d5a6914b201b81491a406a297f13af8
# Push it to GitHub
git push --set-upstream origin backport-1156-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport-1156-to-2.x.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.15 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.15 2.15
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.15
# Create a new branch
git switch --create backport-1156-to-2.15
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a6eb64d30d5a6914b201b81491a406a297f13af8
# Push it to GitHub
git push --set-upstream origin backport-1156-to-2.15
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.15

Then, create a pull request where the base branch is 2.15 and the compare/head branch is backport-1156-to-2.15.

AWSHurneyt added a commit to AWSHurneyt/security-analytics that referenced this pull request Jul 12, 2024
* Removed TODOs.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored how STIX2IOCGenerator creates IOCs of specific types.

Signed-off-by: AWSHurneyt <[email protected]>

* Added additional integration tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Changed IOCType usage as it's no longer an enum in SA commons.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar file.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed unit tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored build.gradle to exclude redundant dependencies from SA commons, instead of the SA commons jar being generated withhout those dependencies.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Reverted changes to build.gradle.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed IOCType usage.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed log message.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Addressed PR feedback.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: AWSHurneyt <[email protected]>
@AWSHurneyt AWSHurneyt mentioned this pull request Jul 12, 2024
5 tasks
AWSHurneyt added a commit that referenced this pull request Jul 12, 2024
* adds new tif source config type - url download (#1142)

* adds new tif source config type - url download

Signed-off-by: Surya Sashank Nistala <[email protected]>

* set up create default tif configs

Signed-off-by: Surya Sashank Nistala <[email protected]>

* address review comments

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add check to block create and delete operation url download type tif source configs

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* Adjusted IOCTypes usage. (#1156)

* Removed TODOs.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored how STIX2IOCGenerator creates IOCs of specific types.

Signed-off-by: AWSHurneyt <[email protected]>

* Added additional integration tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Changed IOCType usage as it's no longer an enum in SA commons.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar file.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed unit tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored build.gradle to exclude redundant dependencies from SA commons, instead of the SA commons jar being generated withhout those dependencies.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Reverted changes to build.gradle.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed IOCType usage.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed log message.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Addressed PR feedback.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed jar.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: AWSHurneyt <[email protected]>
Co-authored-by: Surya Sashank Nistala <[email protected]>
AWSHurneyt added a commit to AWSHurneyt/security-analytics that referenced this pull request Jul 12, 2024
* Removed TODOs.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored how STIX2IOCGenerator creates IOCs of specific types.

Signed-off-by: AWSHurneyt <[email protected]>

* Added additional integration tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Changed IOCType usage as it's no longer an enum in SA commons.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar file.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed unit tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored build.gradle to exclude redundant dependencies from SA commons, instead of the SA commons jar being generated withhout those dependencies.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Reverted changes to build.gradle.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed IOCType usage.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed log message.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Addressed PR feedback.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: AWSHurneyt <[email protected]>

(cherry picked from commit a6eb64d)
Signed-off-by: AWSHurneyt <[email protected]>
AWSHurneyt added a commit that referenced this pull request Jul 12, 2024
* Adjusted IOCTypes usage. (#1156)

* Removed TODOs.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored how STIX2IOCGenerator creates IOCs of specific types.

Signed-off-by: AWSHurneyt <[email protected]>

* Added additional integration tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Changed IOCType usage as it's no longer an enum in SA commons.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar file.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed unit tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored build.gradle to exclude redundant dependencies from SA commons, instead of the SA commons jar being generated withhout those dependencies.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Reverted changes to build.gradle.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed IOCType usage.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed log message.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Addressed PR feedback.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: AWSHurneyt <[email protected]>

(cherry picked from commit a6eb64d)
Signed-off-by: AWSHurneyt <[email protected]>

* Fixed jar.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: AWSHurneyt <[email protected]>
jowg-amazon pushed a commit that referenced this pull request Jul 15, 2024
* Removed TODOs.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored how STIX2IOCGenerator creates IOCs of specific types.

Signed-off-by: AWSHurneyt <[email protected]>

* Added additional integration tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Changed IOCType usage as it's no longer an enum in SA commons.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar file.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed unit tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Refactored build.gradle to exclude redundant dependencies from SA commons, instead of the SA commons jar being generated withhout those dependencies.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Reverted changes to build.gradle.

Signed-off-by: AWSHurneyt <[email protected]>

* Updated jar.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed IOCType usage.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed log message.

Signed-off-by: AWSHurneyt <[email protected]>

* Fixed tests.

Signed-off-by: AWSHurneyt <[email protected]>

* Addressed PR feedback.

Signed-off-by: AWSHurneyt <[email protected]>

---------

Signed-off-by: AWSHurneyt <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants