adding a for loop with script processor to add values from cisco.umbrella.identities to related.user field #25776

hungnguyen-elastic · 2021-05-18T22:40:24Z

Enhancement Request

What does this PR do?

This PR added the foreach loop to append the value from cisco.umbrella.identities field to related.user field in cisco umbrella module pipeline

- script:
    if: ctx?.cisco?.umbrella?.identities != null && ctx?.related?.user!= null
    lang: painless
    description: "Extract user name values from ctx.cisco.umbrella.identities and append it to related.user"
    source: |
      for (def x=0; x < ctx?.cisco?.umbrella?.identities.size(); x++) {
        def cisco_identity = ctx?.cisco?.umbrella?.identities[x];
        if (cisco_identity.contains('@') && !ctx?.related?.user.contains(cisco_identity)) {
          ctx?.related?.user.add(cisco_identity)
        }
      }

Why is it important?

There are instances where cisco.umbrella.identities actually contains the username that is involved with the action and is not included in the related.user field. This addition will make sure the values are added.

for example: event below will be missing the actual username in the related.user field

"message" : ""2021-05-14 19:39:58","hostname34P787","hostname34P787,User Name([email protected])","1.1.1.1","2.2.2.2","Allowed","1 (A)","NOERROR","v-google.com.","Infrastructure","Roaming Computers","Roaming Computers,AD Users","""

Output:

          "related" : {
              "user" : [
                "hostname34P787"
              ]
           }

With the script processor

          "related" : {
            "user" : [
              "hostname34P787",
              "User Name([email protected])"
            ]

Checklist

[ x ] My code follows the style guidelines of this project
[ x ] I have commented my code, particularly in hard-to-understand areas
[ x ] I have made corresponding changes to the documentation
[ x ] I have made corresponding change to the default configuration files
[ x ] I have added tests that prove my fix is effective or that my feature works
[ x ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2021-05-19T00:30:22Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25776 updated
Start Time: 2021-05-25T15:09:22.262+0000
Duration: 99 min 54 sec
Commit: 9788abd

Test stats 🧪

Test	Results
Failed	0
Passed	7193
Skipped	1193
Total	8386

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	7193
Skipped	1193
Total	8386

leehinman

Thank You.

Do you have any example logs we can add to the test directory that will exercise this?

For the script, I made a few changes:

will add related.user if it doesn't exist
will only iterate on cisco.umbrella.identities if it is a list

Can you test to see if this works?

- script:
    if: ctx?.cisco?.umbrella?.identites != null && ctx.cisco.umbrella.identites instanceof List
    lang: painless
    description: "Extract user name values from ctx.cisco.umbrella.identities and append it to related.user"
    source: |-
      void addRelatedUser(def x) {
        if (ctx?.related == null) {
          Map map = new HashMap();
          ctx.put("related", map);
        }
        if (ctx?.related?.user == null) {
          ArrayList al = new ArrayList();
          ctx.related.put("user", al);
        }
        if (!ctx.related.user.contains(x)) {
          ctx.related.user.add(x);
        }
      }

      for (cisco_identity in ctx.cisco.umbrella.identities) {
        if (cisco_identity_contains('@')) {
          addRelatedUser(cisco_identity);
        }
      }

hungnguyen-elastic · 2021-05-20T13:36:31Z

Hi @leehinman , I like the changes. I tried your version and it complains with error below. Maybe you can spot what can be changed quick?

    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "compile error",
        "processor_type" : "script",
        "script_stack" : [
          "... ) {\n              if (ctx?.related == null) {\n     ...",
          "                             ^---- HERE"
        ],
        "script" : "\n            void addRelatedUser(def x) {\n              if (ctx?.related == null) {\n                Map map = new HashMap();\n                ctx.put(\"related\", map);\n              }\n              if (ctx?.related?.user == null) {\n                ArrayList al = new ArrayList();\n                ctx.related.put(\"user\", al);\n              }\n              if (!ctx.related.user.contains(x)) {\n                ctx.related.user.add(x);\n              }\n            }\n      \n            for (cisco_identity in ctx.cisco.umbrella.identities) {\n              if (cisco_identity_contains('@')) {\n                addRelatedUser(cisco_identity);\n              }\n            }\n            ",
        "lang" : "painless",
        "position" : {
          "offset" : 63,
          "start" : 38,
          "end" : 88
        }
      }
    ],
    "type" : "script_exception",
    "reason" : "compile error",
    "processor_type" : "script",
    "script_stack" : [
      "... ) {\n              if (ctx?.related == null) {\n     ...",
      "                             ^---- HERE"
    ]

leehinman · 2021-05-20T13:59:33Z

oops, need to pass in the context.

- script:
    if: ctx?.cisco?.umbrella?.identites != null && ctx.cisco.umbrella.identites instanceof List
    lang: painless
    description: "Extract user name values from ctx.cisco.umbrella.identities and append it to related.user"
    source: |-
      void addRelatedUser(def ctx, def x) {
        if (ctx?.related == null) {
          Map map = new HashMap();
          ctx.put("related", map);
        }
        if (ctx?.related?.user == null) {
          ArrayList al = new ArrayList();
          ctx.related.put("user", al);
        }
        if (!ctx.related.user.contains(x)) {
          ctx.related.user.add(x);
        }
      }

      for (cisco_identity in ctx.cisco.umbrella.identities) {
        if (cisco_identity.contains('@')) {
          addRelatedUser(ctx, cisco_identity);
        }
      }

hungnguyen-elastic · 2021-05-20T14:36:26Z

@leehinman There was a typo in the condition which I corrected and it works like a charm. identites --> identities. I will sanitize the sample logs and share it

- script:
    if: ctx?.cisco?.umbrella?.identities != null && ctx.cisco.umbrella.identities instanceof List
    lang: painless
    description: "Extract user name values from ctx.cisco.umbrella.identities and append it to related.user"
    source: |-
      void addRelatedUser(def x) {
        if (ctx?.related == null) {
          Map map = new HashMap();
          ctx.put("related", map);
        }
        if (ctx?.related?.user == null) {
          ArrayList al = new ArrayList();
          ctx.related.put("user", al);
        }
        if (!ctx.related.user.contains(x)) {
          ctx.related.user.add(x);
        }
      }

      for (cisco_identity in ctx.cisco.umbrella.identities) {
        if (cisco_identity_contains('@')) {
          addRelatedUser(cisco_identity);
        }
      }

P1llus

LGTM

elasticmachine · 2021-05-25T14:41:39Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-05-25T15:07:53Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hnguyen/append_related_username upstream/hnguyen/append_related_username
git merge upstream/master
git push upstream hnguyen/append_related_username

mergify · 2021-05-25T15:08:03Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hnguyen/append_related_username upstream/hnguyen/append_related_username
git merge upstream/master
git push upstream hnguyen/append_related_username

mergify · 2021-05-25T15:08:14Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hnguyen/append_related_username upstream/hnguyen/append_related_username
git merge upstream/master
git push upstream hnguyen/append_related_username

mergify · 2021-05-25T15:08:25Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hnguyen/append_related_username upstream/hnguyen/append_related_username
git merge upstream/master
git push upstream hnguyen/append_related_username

mergify · 2021-05-25T15:08:38Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hnguyen/append_related_username upstream/hnguyen/append_related_username
git merge upstream/master
git push upstream hnguyen/append_related_username

…elated.user field

…le duplicative values

…ella.identities to related.user field (#25776) * adding users from cisco.umbrella.identities to related.user field Co-authored-by: Lee E. Hinman <[email protected]> (cherry picked from commit cec5906)

…ella.identities to related.user field (#25776) (#25856) * adding users from cisco.umbrella.identities to related.user field Co-authored-by: Lee E. Hinman <[email protected]> (cherry picked from commit cec5906) Co-authored-by: hungnguyen-elastic <[email protected]>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 18, 2021

hungnguyen-elastic requested review from andrewkroh, leehinman and adriansr May 18, 2021 23:02

hungnguyen-elastic added the Filebeat Filebeat label May 18, 2021

leehinman requested changes May 20, 2021

View reviewed changes

leehinman self-assigned this May 20, 2021

leehinman approved these changes May 20, 2021

View reviewed changes

P1llus approved these changes May 25, 2021

View reviewed changes

leehinman added backport-v7.14.0 Automated backport with mergify Team:Security-External Integrations labels May 25, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 25, 2021

leehinman added the enhancement label May 25, 2021

hungnguyen-elastic and others added 7 commits May 25, 2021 10:08

adding foreach loop to add values from cisco.umbrella.identities to r…

4657508

…elated.user field

change the from foreach processor to painless script for loop to hand…

b9742b7

…le duplicative values

added the change log to CHANGELOG.next.asciidoc

8426208

adding sample logs and add the suggested changes by Lee Hinman

5748ea1

remove the unnecessary log sample

0a62f45

Updated expected logs

9d13513

update changelog

9788abd

leehinman force-pushed the hnguyen/append_related_username branch from 1060d21 to 9788abd Compare May 25, 2021 15:09

leehinman merged commit cec5906 into master May 25, 2021

mergify bot mentioned this pull request May 25, 2021

[7.x](backport #25776) adding a for loop with script processor to add values from cisco.umbrella.identities to related.user field #25856

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding a for loop with script processor to add values from cisco.umbrella.identities to related.user field #25776

adding a for loop with script processor to add values from cisco.umbrella.identities to related.user field #25776

hungnguyen-elastic commented May 18, 2021 •

edited

Loading

elasticmachine commented May 19, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

leehinman left a comment

hungnguyen-elastic commented May 20, 2021

leehinman commented May 20, 2021

hungnguyen-elastic commented May 20, 2021

P1llus left a comment

elasticmachine commented May 25, 2021

mergify bot commented May 25, 2021

mergify bot commented May 25, 2021

mergify bot commented May 25, 2021

mergify bot commented May 25, 2021

mergify bot commented May 25, 2021

adding a for loop with script processor to add values from cisco.umbrella.identities to related.user field #25776

adding a for loop with script processor to add values from cisco.umbrella.identities to related.user field #25776

Conversation

hungnguyen-elastic commented May 18, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented May 19, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

leehinman left a comment

Choose a reason for hiding this comment

hungnguyen-elastic commented May 20, 2021

leehinman commented May 20, 2021

hungnguyen-elastic commented May 20, 2021

P1llus left a comment

Choose a reason for hiding this comment

elasticmachine commented May 25, 2021

mergify bot commented May 25, 2021

mergify bot commented May 25, 2021

mergify bot commented May 25, 2021

mergify bot commented May 25, 2021

mergify bot commented May 25, 2021

hungnguyen-elastic commented May 18, 2021 •

edited

Loading

elasticmachine commented May 19, 2021 •

edited by jenkins-beats-ci bot

Loading