[Security Solution] Handle indices without geoip schema defined in mapping

[jonathan-buttner]

Describe the feature:

The securitySolution:defaultIndex now includes a broad logs-* index pattern to include the endpoint specific indices and in preparation for the beats indices. An issue could occur when building the layers for the map on the Network tab in Security Solution app.

If a user has some documents that have geoip schema defined in their mapping and match the logs-* and some documents that do not have the geoip schema defined in their mapping, the Network tab will display shard failures for any of the shards that did not have mappings defined.

ES error response

In this example logs-endpoint.events.network-default has geoip schema defined and many indices like logs-endpoint.events.registry-default do not have the fields defined.

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 7,
    "successful": 2,
    "skipped": 1,
    "failed": 5,
    "failures": [
      {
        "shard": 0,
        "index": ".ds-logs-endpoint.alerts-default-000001",
        "node": "xM2iw6pHTwSjhphvBCFpPQ",
        "reason": {
          "type": "query_shard_exception",
          "reason": "failed to find geo_point field [destination.geo.location]",
          "index_uuid": "mbDTvT6vTY6JnOkNKggR0g",
          "index": ".ds-logs-endpoint.alerts-default-000001"
        }
      },
      {
        "shard": 0,
        "index": ".ds-logs-endpoint.events.file-default-000001",
        "node": "xM2iw6pHTwSjhphvBCFpPQ",
        "reason": {
          "type": "query_shard_exception",
          "reason": "failed to find geo_point field [destination.geo.location]",
          "index_uuid": "2UqMoGduQsWXcsVfjNfwtw",
          "index": ".ds-logs-endpoint.events.file-default-000001"
        }
      },
      {
        "shard": 0,
        "index": ".ds-logs-endpoint.events.process-default-000001",
        "node": "xM2iw6pHTwSjhphvBCFpPQ",
        "reason": {
          "type": "query_shard_exception",
          "reason": "failed to find geo_point field [destination.geo.location]",
          "index_uuid": "GCuKhrv2RMe4WHV4qvl3Sg",
          "index": ".ds-logs-endpoint.events.process-default-000001"
        }
      },
      {
        "shard": 0,
        "index": ".ds-logs-endpoint.events.registry-default-000001",
        "node": "xM2iw6pHTwSjhphvBCFpPQ",
        "reason": {
          "type": "query_shard_exception",
          "reason": "failed to find geo_point field [destination.geo.location]",
          "index_uuid": "Xgdkl3ecS96b6vmc9hwv5Q",
          "index": ".ds-logs-endpoint.events.registry-default-000001"
        }
      },
      {
        "shard": 0,
        "index": ".ds-logs-endpoint.events.security-default-000001",
        "node": "xM2iw6pHTwSjhphvBCFpPQ",
        "reason": {
          "type": "query_shard_exception",
          "reason": "failed to find geo_point field [destination.geo.location]",
          "index_uuid": "NVH2l6LBSDSaxf_0WrwR7g",
          "index": ".ds-logs-endpoint.events.security-default-000001"
        }
      }
    ]
  },
  "hits": {
    "total": 2,
    "max_score": 0,
    "hits": [
      {
        "_index": ".ds-logs-endpoint.events.network-default-000001",
        "_id": "cc0beHMBqsIPdTvX9ewZ",
        "_score": 0,
        "fields": {
          "destination.geo.location": [
            "37.3387999786064, -121.89140005968511"
          ]
        }
      },
      {
        "_index": ".ds-logs-endpoint.events.network-default-000001",
        "_id": "o80aeHMBqsIPdTvXTetF",
        "_score": 0,
        "fields": {
          "destination.geo.location": [
            "37.3387999786064, -121.89140005968511"
          ]
        }
      }
    ]
  }
}

One possible solution would be to wrap the map layer creation in an extra field check to make sure the indices being queried have the geoip in their mapping.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[commandline-be]

I have this exact error "failed to find geo_point field [destination.geo.location]" on a fresh installation with only agents installed on the system itself feeding into elastic, it is currently a one node setup but will evolve into a multi-node setup.

What i fail to understand here if this is due to Elastic or due to poluted or incomplete data ?

[jonathan-buttner]

I have this exact error "failed to find geo_point field [destination.geo.location]" on a fresh installation with only agents installed on the system itself feeding into elastic, it is currently a one node setup but will evolve into a multi-node setup.

What i fail to understand here if this is due to Elastic or due to poluted or incomplete data ?

@spong might know more but I believe this is because the Network tab in the Security Solution app is expecting all the indices that match the securitySolution:defaultIndex patterns to have destination.geo.location defined in the mapping for those indices (and possible source.geo.location too).

@commandline-be if you remove the logs-* pattern from securitySolution:defaultIndex do the errors still occur?

[jeffvestal]

Chiming in here. I've been following this issue as I have the same issue on a clean cluster install, with agent, and endpoint module deployed on gcp VMs.

I took @jonathan-buttner 's suggestion and removed the logs-* from the securitySolution:defaultIndex and replaced it with just .ds-logs-endpoint.events.network*

After that, I do not get the geo_point error. The network map loads correctly with data.

[commandline-be]

thanks, interesting responses.

What i'm afraid of is i'll also purge the data.

[jeffvestal]

I should have added this to my above comment.
I tried first stepping in to this by using .ds-logs-endpoint.events* to get all endpoint data. But the process and file streams kept popping up with the same geo error.
Having only network events obviously leaves out the file and process endpoint data....

For example on the Hosts -> Events tab

.ds-logs-endpoint.events.network*

.ds-logs-endpoint.events.*

[spong]

@spong might know more but I believe this is because the Network tab in the Security Solution app is expecting all the indices that match the securitySolution:defaultIndex patterns to have destination.geo.location defined in the mapping for those indices (and possible source.geo.location too).

As @jeffvestal confirmed, that is correct @jonathan-buttner. With the upcoming Search Strategy changes for 7.10, we'll be able to set the index patterns specific to the Network page, but as @jeffvestal mentions, constraining such that you're only showing geo data will result in not having all desired events in the remaining Network page components.

So the two options I see here looks like either having finer grained index selection specific to the map, or looking into enriching the .ds-logs-endpoint.events* process and file stream events with the necessary geo data. I'm not sure of the feasibility of the latter @jonathan-buttner, but either way I think it would be worth while for the map to have it's own selection of index patterns for any other instances of this type of mis-match when users are using their own custom index patterns. Does this seem reasonable @XavierM?

[jonathan-buttner]

@jeffvestal @commandline-be the errors coming from the elastic endpoint's indices should be resolved in the 7.10 release. These errors could still happen if a custom index pattern is added to securitySolution:defaultIndex that has the same problem.

The fix is here: elastic/endpoint-package#87

[jeffvestal]

@jonathan-buttner Thanks for the update. Great that the default set of indices won't cause that problem.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security Solution)

[jonathan-buttner]

Hey @MadameSheema saw your message in email just wanted to double check that it was resolved.

[TheFausap]

Hello, I have the exact same problem using Elastic 8.0.0
Is it possible the reason is the same, or is there another issue related to that version?

query_shard_exception at shard 0index .ds-logs-elastic_agent.metricbeat-default-000001node ZydB65CpTRe6_GCPN7ZHVw

reason is failed to find geo field [destination.geo.location]

thanks

[jonathan-buttner]

@XavierM Any ideas who we could contact from the maps team to get some thoughts on this?

[spong]

@jonathan-buttner -- @thomasneirynck or @nreese from maps should be able to provide some input here.

@TheFausap Yeah, reason is the same, but this issue was specific to the .ds-logs-endpoint.events datastream. Ideally either the map should have it's own configurable Data sources UI since logs-* can easily contain data w/o the necessary geo fields, or we could do a field check before generating the map layers and not include any that are missing the required fields.

[elasticmachine]

Pinging @elastic/kibana-gis (Team:Geo)

[ManuelFFF]

Hi,

I'm experiencing the same issue on a fresh ELK 7.10 installation. I just deployed the Elastic-Agent with the only integration Endpoint, not even system. Is there a solution or workaround available or this is still under research?

Thank you

[willemdh]

Same issue here after enabling endpoint integration on 7.10.1.

THis is the value of securitySolution:defaultIndex:

apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, logs-*, packetbeat-*, winlogbeat-*

[MarcusCaepio]

Same here with a fresh install of 7.11.1 and elastic-agents only, with endpoint and system integrations:

[sphr2k]

Same here. Fresh install of 7.12.1

[Deathwing182]

Same here also, fresh install of the 7.12.1 Elastic Agent on about 10 endpoints this morning, immediately started having this issue after they showed up in Kibana.

[thomasneirynck]

thx all. Maybe this would best be handled on the maps-side, rather than in Security-app.

Some things jump out that the MapsEmbeddable might be able to improve on:

• improve the maps-query so sparse docs make ES tip over
• disallow shard-errors to bubble-up from Elasticsearch into the UX

We need to investigate, will ping on this thread when more info.

[iamhowardtheduck]

This issue continues with 7.13; this is an Elastic Cloud based deployment, with the Elastic-Agent running in AWS.

Would the mapping have anything to do it with it?