Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Okta module mapping issue for 'okta.target' #24354

Closed
Trogdor-007 opened this issue Mar 4, 2021 · 3 comments · Fixed by #24636
Closed

[Filebeat] Okta module mapping issue for 'okta.target' #24354

Trogdor-007 opened this issue Mar 4, 2021 · 3 comments · Fixed by #24636

Comments

@Trogdor-007
Copy link

Parse the field "okta.target" with filebeat module

Right now, all other fields are parsed correctly. But for "okta.target" field, that still shows up as JSON and there are 4 sub-fields that you can't parse normally with mappings. Working with support to figure out how I can do this as-is but would be nice if that field was parsed out with filebeat module.

You should be able to see this with ANY okta log but I have attached 2 screenshots with field values obfuscated (they are all strings).
okta_2
okta_1
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 4, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 4, 2021
@legoguy1000
Copy link
Contributor

This could be as simple as changing the field type from array to nested

@legoguy1000
Copy link
Contributor

Changing the mapping to nested seems to have done the trick. I add some data from the module test logs and was able to run the following query.

GET /filebeat-7.11.2-2021.03.18-000002/_search
{
  "query": {
    "nested": {
      "path": "okta.target",
      "score_mode": "avg",
      "query": {
        "bool": {
          "must": [
            {
              "match": {
                "okta.target.alternate_id": "unknown"
              }
            }
          ]
        }
      }
    }
  }
}

and returned

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0296195,
    "hits" : [
      {
        "_index" : "filebeat-7.11.2-2021.03.18-000002",
        "_type" : "_doc",
        "_id" : "a1WwRXgBxkqHutLYs8ga",
        "_score" : 1.0296195,
        "_source" : {
          "@timestamp" : "2020-02-14T20:18:57.762Z",
          "client.geo.city_name" : "Dublin",
          "client.geo.country_name" : "United States",
          "client.geo.location" : {
            "lat" : 37.7201,
            "lon" : -121.919
          },
          "client.geo.region_name" : "California",
          "client.ip" : "108.255.197.247",
          "client.user.full_name" : "xxxxxx",
          "client.user.id" : "00u1abvz4pYqdM8ms4x6",
          "event.action" : "policy.evaluate_sign_on",
          "event.category" : [
            "authentication"
          ],
          "event.dataset" : "okta.system",
          "event.id" : "3af594f9-4f67-11ea-abd3-1f5d113f2546",
          "event.kind" : "event",
          "event.module" : "okta",
          "event.original" : """{"actor":{"alternateId":"[email protected]","detailEntry":null,"displayName":"xxxxxx","id":"00u1abvz4pYqdM8ms4x6","type":"User"},"authenticationContext":{"authenticationProvider":null,"authenticationStep":0,"credentialProvider":null,"credentialType":null,"externalSessionId":"102bZDNFfWaQSyEZQuDgWt-uQ","interface":null,"issuer":null},"client":{"device":"Computer","geographicalContext":{"city":"Dublin","country":"United States","geolocation":{"lat":37.7201,"lon":-121.919},"postalCode":"94568","state":"California"},"id":null,"ipAddress":"108.255.197.247","userAgent":{"browser":"FIREFOX","os":"Mac OS X","rawUserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0"},"zone":"null"},"debugContext":{"debugData":{"deviceFingerprint":"541daf91d15bef64a7e08c946fd9a9d0","requestId":"XkcAsWb8WjwDP76xh@1v8wAABp0","requestUri":"/api/v1/authn","threatSuspected":"false","url":"/api/v1/authn?"}},"displayMessage":"Evaluation of sign-on policy","eventType":"policy.evaluate_sign_on","legacyEventType":null,"outcome":{"reason":"Sign-on policy evaluation resulted in ALLOW","result":"ALLOW"},"published":"2020-02-14T20:18:57.762Z","request":{"ipChain":[{"geographicalContext":{"city":"Dublin","country":"United States","geolocation":{"lat":37.7201,"lon":-121.919},"postalCode":"94568","state":"California"},"ip":"108.255.197.247","source":null,"version":"V4"}]},"securityContext":{"asNumber":null,"asOrg":null,"domain":null,"isProxy":null,"isp":null},"severity":"INFO","target":[{"alternateId":"unknown","detailEntry":{"policyType":"OktaSignOn"},"displayName":"Default Policy","id":"00p1abvweGGDW10Ur4x6","type":"PolicyEntity"},{"alternateId":"00p1abvweGGDW10Ur4x6","detailEntry":null,"displayName":"Default Rule","id":"0pr1abvwfqGFI4n064x6","type":"PolicyRule"}],"transaction":{"detail":{},"id":"XkcAsWb8WjwDP76xh@1v8wAABp0","type":"WEB"},"uuid":"3af594f9-4f67-11ea-abd3-1f5d113f2546","version":"0"}""",
          "event.outcome" : "success",
          "event.type" : [
            "info"
          ],
          "fileset.name" : "system",
          "input.type" : "log",
          "log.offset" : 3287,
          "okta.actor.alternate_id" : "[email protected]",
          "okta.actor.display_name" : "xxxxxx",
          "okta.actor.id" : "00u1abvz4pYqdM8ms4x6",
          "okta.actor.type" : "User",
          "okta.authentication_context.authentication_step" : 0,
          "okta.authentication_context.external_session_id" : "102bZDNFfWaQSyEZQuDgWt-uQ",
          "okta.client.device" : "Computer",
          "okta.client.ip" : "108.255.197.247",
          "okta.client.user_agent.browser" : "FIREFOX",
          "okta.client.user_agent.os" : "Mac OS X",
          "okta.client.user_agent.raw_user_agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0",
          "okta.client.zone" : "null",
          "okta.debug_context.debug_data.device_fingerprint" : "541daf91d15bef64a7e08c946fd9a9d0",
          "okta.debug_context.debug_data.request_id" : "XkcAsWb8WjwDP76xh@1v8wAABp0",
          "okta.debug_context.debug_data.request_uri" : "/api/v1/authn",
          "okta.debug_context.debug_data.threat_suspected" : "false",
          "okta.debug_context.debug_data.url" : "/api/v1/authn?",
          "okta.display_message" : "Evaluation of sign-on policy",
          "okta.event_type" : "policy.evaluate_sign_on",
          "okta.outcome.reason" : "Sign-on policy evaluation resulted in ALLOW",
          "okta.outcome.result" : "ALLOW",
          "okta.target" : [
            {
              "alternate_id" : "unknown",
              "display_name" : "Default Policy",
              "id" : "00p1abvweGGDW10Ur4x6",
              "type" : "PolicyEntity"
            },
            {
              "alternate_id" : "00p1abvweGGDW10Ur4x6",
              "display_name" : "Default Rule",
              "id" : "0pr1abvwfqGFI4n064x6",
              "type" : "PolicyRule"
            }
          ],
          "okta.transaction.id" : "XkcAsWb8WjwDP76xh@1v8wAABp0",
          "okta.transaction.type" : "WEB",
          "okta.uuid" : "3af594f9-4f67-11ea-abd3-1f5d113f2546",
          "related.ip" : [
            "108.255.197.247"
          ],
          "related.user" : [
            "xxxxxx"
          ],
          "service.type" : "okta",
          "source.as.number" : 7018,
          "source.as.organization.name" : "AT&T Services, Inc.",
          "source.geo.city_name" : "Dublin",
          "source.geo.continent_name" : "North America",
          "source.geo.country_iso_code" : "US",
          "source.geo.country_name" : "United States",
          "source.geo.location" : {
            "lat" : 37.7201,
            "lon" : -121.919
          },
          "source.geo.region_iso_code" : "US-CA",
          "source.geo.region_name" : "California",
          "source.ip" : "108.255.197.247",
          "source.user.full_name" : "xxxxxx",
          "source.user.id" : "00u1abvz4pYqdM8ms4x6",
          "tags" : [
            "forwarded"
          ],
          "user.full_name" : "xxxxxx",
          "user_agent.device.name" : "Mac",
          "user_agent.name" : "Firefox",
          "user_agent.original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0",
          "user_agent.os.full" : "Mac OS X 10.15",
          "user_agent.os.name" : "Mac OS X",
          "user_agent.os.version" : "10.15",
          "user_agent.version" : "72.0."
        }
      },
      {
        "_index" : "filebeat-7.11.2-2021.03.18-000002",
        "_type" : "_doc",
        "_id" : "c1W2RXgBxkqHutLY6MiI",
        "_score" : 1.0296195,
        "_source" : {
          "@timestamp" : "2020-02-14T20:18:57.762Z",
          "client.geo.city_name" : "Dublin",
          "client.geo.country_name" : "United States",
          "client.geo.location" : {
            "lat" : 37.7201,
            "lon" : -121.919
          },
          "client.geo.region_name" : "California",
          "client.ip" : "108.255.197.247",
          "client.user.full_name" : "xxxxxx",
          "client.user.id" : "00u1abvz4pYqdM8ms4x6",
          "event.action" : "policy.evaluate_sign_on",
          "event.category" : [
            "authentication"
          ],
          "event.dataset" : "okta.system",
          "event.id" : "3af594f9-4f67-11ea-abd3-1f5d113f2546",
          "event.kind" : "event",
          "event.module" : "okta",
          "event.original" : """{"actor":{"alternateId":"[email protected]","detailEntry":null,"displayName":"xxxxxx","id":"00u1abvz4pYqdM8ms4x6","type":"User"},"authenticationContext":{"authenticationProvider":null,"authenticationStep":0,"credentialProvider":null,"credentialType":null,"externalSessionId":"102bZDNFfWaQSyEZQuDgWt-uQ","interface":null,"issuer":null},"client":{"device":"Computer","geographicalContext":{"city":"Dublin","country":"United States","geolocation":{"lat":37.7201,"lon":-121.919},"postalCode":"94568","state":"California"},"id":null,"ipAddress":"108.255.197.247","userAgent":{"browser":"FIREFOX","os":"Mac OS X","rawUserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0"},"zone":"null"},"debugContext":{"debugData":{"deviceFingerprint":"541daf91d15bef64a7e08c946fd9a9d0","requestId":"XkcAsWb8WjwDP76xh@1v8wAABp0","requestUri":"/api/v1/authn","threatSuspected":"false","url":"/api/v1/authn?"}},"displayMessage":"Evaluation of sign-on policy","eventType":"policy.evaluate_sign_on","legacyEventType":null,"outcome":{"reason":"Sign-on policy evaluation resulted in ALLOW","result":"ALLOW"},"published":"2020-02-14T20:18:57.762Z","request":{"ipChain":[{"geographicalContext":{"city":"Dublin","country":"United States","geolocation":{"lat":37.7201,"lon":-121.919},"postalCode":"94568","state":"California"},"ip":"108.255.197.247","source":null,"version":"V4"}]},"securityContext":{"asNumber":null,"asOrg":null,"domain":null,"isProxy":null,"isp":null},"severity":"INFO","target":[{"alternateId":"unknown","detailEntry":{"policyType":"OktaSignOn"},"displayName":"Default Policy","id":"00p1abvweGGDW10Ur4x6","type":"PolicyEntity"},{"alternateId":"00p1abvweGGDW10Ur4x6","detailEntry":null,"displayName":"Default Rule","id":"0pr1abvwfqGFI4n064x6","type":"PolicyRule"}],"transaction":{"detail":{},"id":"XkcAsWb8WjwDP76xh@1v8wAABp0","type":"WEB"},"uuid":"3af594f9-4f67-11ea-abd3-1f5d113f2546","version":"0"}""",
          "event.outcome" : "success",
          "event.type" : [
            "info"
          ],
          "fileset.name" : "system",
          "input.type" : "log",
          "log.offset" : 3287,
          "okta.actor.alternate_id" : "[email protected]",
          "okta.actor.display_name" : "xxxxxx",
          "okta.actor.id" : "00u1abvz4pYqdM8ms4x6",
          "okta.actor.type" : "User",
          "okta.authentication_context.authentication_step" : 0,
          "okta.authentication_context.external_session_id" : "102bZDNFfWaQSyEZQuDgWt-uQ",
          "okta.client.device" : "Computer",
          "okta.client.ip" : "108.255.197.247",
          "okta.client.user_agent.browser" : "FIREFOX",
          "okta.client.user_agent.os" : "Mac OS X",
          "okta.client.user_agent.raw_user_agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0",
          "okta.client.zone" : "null",
          "okta.debug_context.debug_data.device_fingerprint" : "541daf91d15bef64a7e08c946fd9a9d0",
          "okta.debug_context.debug_data.request_id" : "XkcAsWb8WjwDP76xh@1v8wAABp0",
          "okta.debug_context.debug_data.request_uri" : "/api/v1/authn",
          "okta.debug_context.debug_data.threat_suspected" : "false",
          "okta.debug_context.debug_data.url" : "/api/v1/authn?",
          "okta.display_message" : "Evaluation of sign-on policy",
          "okta.event_type" : "policy.evaluate_sign_on",
          "okta.outcome.reason" : "Sign-on policy evaluation resulted in ALLOW",
          "okta.outcome.result" : "ALLOW",
          "okta.target" : [
            {
              "alternate_id" : "unknown",
              "display_name" : "Default Policy",
              "id" : "00p1abvweGGDW10Ur4x6",
              "type" : "PolicyEntity"
            },
            {
              "alternate_id" : "00p1abvweGGDW10Ur4x6",
              "display_name" : "Default Rule",
              "id" : "0pr1abvwfqGFI4n064x6",
              "type" : "PolicyRule"
            }
          ],
          "okta.transaction.id" : "XkcAsWb8WjwDP76xh@1v8wAABp0",
          "okta.transaction.type" : "WEB",
          "okta.uuid" : "3af594f9-4f67-11ea-abd3-1f5d113f2546",
          "related.ip" : [
            "108.255.197.247"
          ],
          "related.user" : [
            "xxxxxx"
          ],
          "service.type" : "okta",
          "source.as.number" : 7018,
          "source.as.organization.name" : "AT&T Services, Inc.",
          "source.geo.city_name" : "Dublin",
          "source.geo.continent_name" : "North America",
          "source.geo.country_iso_code" : "US",
          "source.geo.country_name" : "United States",
          "source.geo.location" : {
            "lat" : 37.7201,
            "lon" : -121.919
          },
          "source.geo.region_iso_code" : "US-CA",
          "source.geo.region_name" : "California",
          "source.ip" : "108.255.197.247",
          "source.user.full_name" : "xxxxxx",
          "source.user.id" : "00u1abvz4pYqdM8ms4x6",
          "tags" : [
            "forwarded"
          ],
          "user.full_name" : "xxxxxx",
          "user_agent.device.name" : "Mac",
          "user_agent.name" : "Firefox",
          "user_agent.original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0",
          "user_agent.os.full" : "Mac OS X 10.15",
          "user_agent.os.name" : "Mac OS X",
          "user_agent.os.version" : "10.15",
          "user_agent.version" : "72.0."
        }
      }
    ]
  }
}

which was successful as there are 3 okta docs and only 2 matched which whats was expected.

This was referenced Mar 18, 2021
Merged
Closed
@marc-gr marc-gr mentioned this issue Apr 7, 2021
Merged
6 tasks
@andrewkroh andrewkroh changed the title Field not parsing correctly [Filebeat] Okta module mapping issue for 'okta.target' Oct 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Filebeat] Change okta.target to nested field legoguy1000/beats
4 participants
@andresrc @legoguy1000 @elasticmachine @Trogdor-007