Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Google Workspace] Missing ECS fields in index.query.default_field when using import_mappings #7582

Open
SpencerLN opened this issue Aug 29, 2023 · 15 comments
Assignees
Labels
bug Something isn't working, use only for issues Integration:google_workspace Google Workspace Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]

Comments

@SpencerLN
Copy link
Contributor

The Google Workspace data streams are missing the ECS fields; this results in searches that a user would expect to work not returning the same results that they do for other integrations. Just a single example, the source.user.email field is missing from the index.query.default_field setting in the Login stream, so if you search for a user's email in quotes, i.e. "[email protected]" it will return no results, while source.user.email: "[email protected]" returns results.

"query": {
          "default_field": [
            "input.type",
            "tags",
            "google_workspace.login.affected_email_address",
            "google_workspace.login.challenge_method",
            "google_workspace.login.failure_type",
            "google_workspace.login.challenge_status",
            "google_workspace.login.type",
            "google_workspace.actor.type",
            "google_workspace.actor.key",
            "google_workspace.event.type",
            "google_workspace.kind",
            "google_workspace.organization.domain"
          ]
        }

I didn't review all of the data streams in the integration, but at least several of them seem impacted.

@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@andrewkroh
Copy link
Member

andrewkroh commented Aug 29, 2023

This package is using the dynamic ECS mappings (source). So the template has dynamic mappings for all ECS fields.

My assumption is that without static mappings in the template then Fleet is unaware of the ECS fields that it should add to the index.query.default_field. IIRC it was designed to take the first 1024 static fields in the template and add them to the default_field list.

So I think we need a way to populate a useful index.query.default_field list when using the dynamic ECS mappings.


For reference:

GET _component_template/logs-google_workspace.admin@package

{
  "component_templates": [
    {
      "name": "logs-google_workspace.admin@package",
      "component_template": {
        "template": {
          "settings": {
            "index": {
              "lifecycle": {
                "name": "logs"
              },
              "codec": "best_compression",
              "default_pipeline": "logs-google_workspace.admin-2.13.0",
              "mapping": {
                "total_fields": {
                  "limit": "10000"
                },
                "ignore_malformed": "true"
              },
              "query": {
                "default_field": [
                  "input.type",
                  "tags",
                  "google_workspace.admin.application.edition",
                  "google_workspace.admin.application.name",
                  "google_workspace.admin.application.enabled",
                  "google_workspace.admin.application.licences_order_number",
                  "google_workspace.admin.application.id",
                  "google_workspace.admin.application.asp_id",
                  "google_workspace.admin.application.package_id",
                  "google_workspace.admin.group.email",
                  "google_workspace.admin.group.priorities",
                  "google_workspace.admin.group.allowed_list",
                  "google_workspace.admin.new_value",
                  "google_workspace.admin.old_value",
                  "google_workspace.admin.org_unit.name",
                  "google_workspace.admin.org_unit.full",
                  "google_workspace.admin.setting.name",
                  "google_workspace.admin.setting.description",
                  "google_workspace.admin.user_defined_setting.name",
                  "google_workspace.admin.domain.alias",
                  "google_workspace.admin.domain.name",
                  "google_workspace.admin.domain.secondary_name",
                  "google_workspace.admin.managed_configuration",
                  "google_workspace.admin.non_featured_services_selection",
                  "google_workspace.admin.field",
                  "google_workspace.admin.resource.id",
                  "google_workspace.admin.user.email",
                  "google_workspace.admin.user.nickname",
                  "google_workspace.admin.gateway.name",
                  "google_workspace.admin.chrome_os.session_type",
                  "google_workspace.admin.device.serial_number",
                  "google_workspace.admin.device.id",
                  "google_workspace.admin.device.type",
                  "google_workspace.admin.device.command_details",
                  "google_workspace.admin.print_server.name",
                  "google_workspace.admin.printer.name",
                  "google_workspace.admin.role.id",
                  "google_workspace.admin.role.name",
                  "google_workspace.admin.privilege.name",
                  "google_workspace.admin.service.name",
                  "google_workspace.admin.url.name",
                  "google_workspace.admin.product.name",
                  "google_workspace.admin.product.sku",
                  "google_workspace.admin.email.quarantine_name",
                  "google_workspace.admin.email.log_search_filter.message_id",
                  "google_workspace.admin.email.log_search_filter.recipient.value",
                  "google_workspace.admin.email.log_search_filter.sender.value",
                  "google_workspace.admin.chrome_licenses.enabled",
                  "google_workspace.admin.chrome_licenses.allowed",
                  "google_workspace.admin.oauth2.service.name",
                  "google_workspace.admin.oauth2.application.id",
                  "google_workspace.admin.oauth2.application.name",
                  "google_workspace.admin.oauth2.application.type",
                  "google_workspace.admin.verification_method",
                  "google_workspace.admin.alert.name",
                  "google_workspace.admin.rule.name",
                  "google_workspace.admin.api.client.name",
                  "google_workspace.admin.api.scopes",
                  "google_workspace.admin.mdm.token",
                  "google_workspace.admin.mdm.vendor",
                  "google_workspace.admin.info_type",
                  "google_workspace.admin.email_monitor.dest_email",
                  "google_workspace.admin.email_monitor.level.chat",
                  "google_workspace.admin.email_monitor.level.draft",
                  "google_workspace.admin.email_monitor.level.incoming",
                  "google_workspace.admin.email_monitor.level.outgoing",
                  "google_workspace.admin.email_dump.package_content",
                  "google_workspace.admin.email_dump.query",
                  "google_workspace.admin.request.id",
                  "google_workspace.admin.mobile.action.id",
                  "google_workspace.admin.mobile.action.type",
                  "google_workspace.admin.mobile.certificate.name",
                  "google_workspace.admin.distribution.entity.name",
                  "google_workspace.admin.distribution.entity.type",
                  "google_workspace.actor.type",
                  "google_workspace.actor.key",
                  "google_workspace.event.type",
                  "google_workspace.kind",
                  "google_workspace.organization.domain"
                ]
              }
            }
          },
          "mappings": {
            "dynamic_templates": [
              {
                "_embedded_ecs-ecs_timestamp": {
                  "path_match": "@timestamp",
                  "mapping": {
                    "ignore_malformed": false,
                    "type": "date"
                  }
                }
              },
              {
                "_embedded_ecs-data_stream_to_constant": {
                  "path_match": "data_stream.*",
                  "mapping": {
                    "type": "constant_keyword"
                  }
                }
              },
              {
                "_embedded_ecs-resolved_ip_to_ip": {
                  "mapping": {
                    "type": "ip"
                  },
                  "match": "resolved_ip"
                }
              },
              {
                "_embedded_ecs-forwarded_ip_to_ip": {
                  "mapping": {
                    "type": "ip"
                  },
                  "match_mapping_type": "string",
                  "match": "forwarded_ip"
                }
              },
              {
                "_embedded_ecs-ip_to_ip": {
                  "mapping": {
                    "type": "ip"
                  },
                  "match_mapping_type": "string",
                  "match": "ip"
                }
              },
              {
                "_embedded_ecs-port_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "port"
                }
              },
              {
                "_embedded_ecs-thread_id_to_long": {
                  "path_match": "*.thread.id",
                  "mapping": {
                    "type": "long"
                  }
                }
              },
              {
                "_embedded_ecs-status_code_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "status_code"
                }
              },
              {
                "_embedded_ecs-line_to_long": {
                  "path_match": "*.file.line",
                  "mapping": {
                    "type": "long"
                  }
                }
              },
              {
                "_embedded_ecs-priority_to_long": {
                  "path_match": "log.syslog.priority",
                  "mapping": {
                    "type": "long"
                  }
                }
              },
              {
                "_embedded_ecs-code_to_long": {
                  "path_match": "*.facility.code",
                  "mapping": {
                    "type": "long"
                  }
                }
              },
              {
                "_embedded_ecs-bytes_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "path_unmatch": "*.data.bytes",
                  "match": "bytes"
                }
              },
              {
                "_embedded_ecs-packets_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "packets"
                }
              },
              {
                "_embedded_ecs-public_key_exponent_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "public_key_exponent"
                }
              },
              {
                "_embedded_ecs-severity_to_long": {
                  "path_match": "event.severity",
                  "mapping": {
                    "type": "long"
                  }
                }
              },
              {
                "_embedded_ecs-duration_to_long": {
                  "path_match": "event.duration",
                  "mapping": {
                    "type": "long"
                  }
                }
              },
              {
                "_embedded_ecs-pid_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "pid"
                }
              },
              {
                "_embedded_ecs-uptime_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "uptime"
                }
              },
              {
                "_embedded_ecs-sequence_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "sequence"
                }
              },
              {
                "_embedded_ecs-entropy_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "*entropy"
                }
              },
              {
                "_embedded_ecs-size_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "*size"
                }
              },
              {
                "_embedded_ecs-entrypoint_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "entrypoint"
                }
              },
              {
                "_embedded_ecs-ttl_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "ttl"
                }
              },
              {
                "_embedded_ecs-major_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "major"
                }
              },
              {
                "_embedded_ecs-minor_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "minor"
                }
              },
              {
                "_embedded_ecs-as_number_to_long": {
                  "path_match": "*.as.number",
                  "mapping": {
                    "type": "long"
                  }
                }
              },
              {
                "_embedded_ecs-pgid_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "pgid"
                }
              },
              {
                "_embedded_ecs-exit_code_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "exit_code"
                }
              },
              {
                "_embedded_ecs-chi_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "chi2"
                }
              },
              {
                "_embedded_ecs-args_count_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "args_count"
                }
              },
              {
                "_embedded_ecs-virtual_address_to_long": {
                  "mapping": {
                    "type": "long"
                  },
                  "match": "virtual_address"
                }
              },
              {
                "_embedded_ecs-io_text_to_wildcard": {
                  "path_match": "*.io.text",
                  "mapping": {
                    "type": "wildcard"
                  }
                }
              },
              {
                "_embedded_ecs-strings_to_wildcard": {
                  "path_match": "registry.data.strings",
                  "mapping": {
                    "type": "wildcard"
                  }
                }
              },
              {
                "_embedded_ecs-path_to_wildcard": {
                  "path_match": "*url.path",
                  "mapping": {
                    "type": "wildcard"
                  }
                }
              },
              {
                "_embedded_ecs-message_id_to_wildcard": {
                  "mapping": {
                    "type": "wildcard"
                  },
                  "match": "message_id"
                }
              },
              {
                "_embedded_ecs-command_line_to_multifield": {
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "wildcard"
                  },
                  "match": "command_line"
                }
              },
              {
                "_embedded_ecs-error_stack_trace_to_multifield": {
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "wildcard"
                  },
                  "match": "stack_trace"
                }
              },
              {
                "_embedded_ecs-http_content_to_multifield": {
                  "path_match": "*.body.content",
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "wildcard"
                  }
                }
              },
              {
                "_embedded_ecs-url_full_to_multifield": {
                  "path_match": "*.url.full",
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "wildcard"
                  }
                }
              },
              {
                "_embedded_ecs-url_original_to_multifield": {
                  "path_match": "*.url.original",
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "wildcard"
                  }
                }
              },
              {
                "_embedded_ecs-user_agent_original_to_multifield": {
                  "path_match": "user_agent.original",
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "wildcard"
                  }
                }
              },
              {
                "_embedded_ecs-error_message_to_match_only": {
                  "path_match": "error.message",
                  "mapping": {
                    "type": "match_only_text"
                  }
                }
              },
              {
                "_embedded_ecs-message_match_only_text": {
                  "path_match": "message",
                  "mapping": {
                    "type": "match_only_text"
                  }
                }
              },
              {
                "_embedded_ecs-agent_name_to_keyword": {
                  "path_match": "agent.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-service_name_to_keyword": {
                  "path_match": "*.service.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-sections_name_to_keyword": {
                  "path_match": "*.sections.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-resource_name_to_keyword": {
                  "path_match": "*.resource.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-observer_name_to_keyword": {
                  "path_match": "observer.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-question_name_to_keyword": {
                  "path_match": "*.question.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-group_name_to_keyword": {
                  "path_match": "*.group.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-geo_name_to_keyword": {
                  "path_match": "*.geo.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-host_name_to_keyword": {
                  "path_match": "host.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-severity_name_to_keyword": {
                  "path_match": "*.severity.name",
                  "mapping": {
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-title_to_multifield": {
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "keyword"
                  },
                  "match": "title"
                }
              },
              {
                "_embedded_ecs-executable_to_multifield": {
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "keyword"
                  },
                  "match": "executable"
                }
              },
              {
                "_embedded_ecs-file_path_to_multifield": {
                  "path_match": "*.file.path",
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-file_target_path_to_multifield": {
                  "path_match": "*.file.target_path",
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-name_to_multifield": {
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "keyword"
                  },
                  "match": "name"
                }
              },
              {
                "_embedded_ecs-full_name_to_multifield": {
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "keyword"
                  },
                  "match": "full_name"
                }
              },
              {
                "_embedded_ecs-os_full_to_multifield": {
                  "path_match": "*.os.full",
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "keyword"
                  }
                }
              },
              {
                "_embedded_ecs-working_directory_to_multifield": {
                  "mapping": {
                    "fields": {
                      "text": {
                        "type": "match_only_text"
                      }
                    },
                    "type": "keyword"
                  },
                  "match": "working_directory"
                }
              },
              {
                "_embedded_ecs-timestamp_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "timestamp"
                }
              },
              {
                "_embedded_ecs-delivery_timestamp_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "delivery_timestamp"
                }
              },
              {
                "_embedded_ecs-not_after_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "not_after"
                }
              },
              {
                "_embedded_ecs-not_before_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "not_before"
                }
              },
              {
                "_embedded_ecs-accessed_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "accessed"
                }
              },
              {
                "_embedded_ecs-origination_timestamp_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "origination_timestamp"
                }
              },
              {
                "_embedded_ecs-created_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "created"
                }
              },
              {
                "_embedded_ecs-installed_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "installed"
                }
              },
              {
                "_embedded_ecs-creation_date_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "creation_date"
                }
              },
              {
                "_embedded_ecs-ctime_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "ctime"
                }
              },
              {
                "_embedded_ecs-mtime_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "mtime"
                }
              },
              {
                "_embedded_ecs-ingested_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "ingested"
                }
              },
              {
                "_embedded_ecs-start_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "start"
                }
              },
              {
                "_embedded_ecs-end_to_date": {
                  "mapping": {
                    "type": "date"
                  },
                  "match": "end"
                }
              },
              {
                "_embedded_ecs-score_base_to_float": {
                  "path_match": "*.score.base",
                  "mapping": {
                    "type": "float"
                  }
                }
              },
              {
                "_embedded_ecs-score_temporal_to_float": {
                  "path_match": "*.score.temporal",
                  "mapping": {
                    "type": "float"
                  }
                }
              },
              {
                "_embedded_ecs-score_to_float": {
                  "mapping": {
                    "type": "float"
                  },
                  "match": "*_score"
                }
              },
              {
                "_embedded_ecs-score_norm_to_float": {
                  "mapping": {
                    "type": "float"
                  },
                  "match": "*_score_norm"
                }
              },
              {
                "_embedded_ecs-usage_to_float": {
                  "mapping": {
                    "scaling_factor": 1000,
                    "type": "scaled_float"
                  },
                  "match": "usage"
                }
              },
              {
                "_embedded_ecs-location_to_geo_point": {
                  "mapping": {
                    "type": "geo_point"
                  },
                  "match": "location"
                }
              },
              {
                "_embedded_ecs-same_as_process_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "same_as_process"
                }
              },
              {
                "_embedded_ecs-established_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "established"
                }
              },
              {
                "_embedded_ecs-resumed_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "resumed"
                }
              },
              {
                "_embedded_ecs-max_bytes_per_process_exceeded_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "max_bytes_per_process_exceeded"
                }
              },
              {
                "_embedded_ecs-interactive_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "interactive"
                }
              },
              {
                "_embedded_ecs-exists_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "exists"
                }
              },
              {
                "_embedded_ecs-trusted_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "trusted"
                }
              },
              {
                "_embedded_ecs-valid_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "valid"
                }
              },
              {
                "_embedded_ecs-go_stripped_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "go_stripped"
                }
              },
              {
                "_embedded_ecs-coldstart_to_boolean": {
                  "mapping": {
                    "type": "boolean"
                  },
                  "match": "coldstart"
                }
              },
              {
                "_embedded_ecs-exports_to_flattened": {
                  "mapping": {
                    "type": "flattened"
                  },
                  "match": "exports"
                }
              },
              {
                "_embedded_ecs-structured_data_to_flattened": {
                  "mapping": {
                    "type": "flattened"
                  },
                  "match": "structured_data"
                }
              },
              {
                "_embedded_ecs-imports_to_flattened": {
                  "mapping": {
                    "type": "flattened"
                  },
                  "match": "*imports"
                }
              },
              {
                "_embedded_ecs-attachments_to_nested": {
                  "mapping": {
                    "type": "nested"
                  },
                  "match": "attachments"
                }
              },
              {
                "_embedded_ecs-segments_to_nested": {
                  "mapping": {
                    "type": "nested"
                  },
                  "match": "segments"
                }
              },
              {
                "_embedded_ecs-elf_sections_to_nested": {
                  "path_match": "*.elf.sections",
                  "mapping": {
                    "type": "nested"
                  }
                }
              },
              {
                "_embedded_ecs-pe_sections_to_nested": {
                  "path_match": "*.pe.sections",
                  "mapping": {
                    "type": "nested"
                  }
                }
              },
              {
                "_embedded_ecs-macho_sections_to_nested": {
                  "path_match": "*.macho.sections",
                  "mapping": {
                    "type": "nested"
                  }
                }
              }
            ],
            "properties": {
              "input": {
                "properties": {
                  "type": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              },
              "@timestamp": {
                "ignore_malformed": false,
                "type": "date"
              },
              "log": {
                "properties": {
                  "offset": {
                    "type": "long"
                  }
                }
              },
              "google_workspace": {
                "properties": {
                  "actor": {
                    "properties": {
                      "type": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "key": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  },
                  "kind": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "organization": {
                    "properties": {
                      "domain": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  },
                  "admin": {
                    "properties": {
                      "request": {
                        "properties": {
                          "id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "info_type": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "role": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "bulk_upload": {
                        "properties": {
                          "total": {
                            "type": "long"
                          },
                          "failed": {
                            "type": "long"
                          }
                        }
                      },
                      "print_server": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "chrome_licenses": {
                        "properties": {
                          "allowed": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "enabled": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "non_featured_services_selection": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "rule": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "privilege": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "oauth2": {
                        "properties": {
                          "application": {
                            "properties": {
                              "name": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              },
                              "id": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              },
                              "type": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              }
                            }
                          },
                          "service": {
                            "properties": {
                              "name": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              }
                            }
                          }
                        }
                      },
                      "email_monitor": {
                        "properties": {
                          "level": {
                            "properties": {
                              "incoming": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              },
                              "outgoing": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              },
                              "chat": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              },
                              "draft": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              }
                            }
                          },
                          "dest_email": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "distribution": {
                        "properties": {
                          "entity": {
                            "properties": {
                              "name": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              },
                              "type": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              }
                            }
                          }
                        }
                      },
                      "setting": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "description": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "alert": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "verification_method": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "chrome_os": {
                        "properties": {
                          "session_type": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "api": {
                        "properties": {
                          "client": {
                            "properties": {
                              "name": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              }
                            }
                          },
                          "scopes": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "managed_configuration": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "new_value": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "email": {
                        "properties": {
                          "log_search_filter": {
                            "properties": {
                              "end_date": {
                                "type": "date"
                              },
                              "sender": {
                                "properties": {
                                  "ip": {
                                    "type": "ip"
                                  },
                                  "value": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                  }
                                }
                              },
                              "recipient": {
                                "properties": {
                                  "ip": {
                                    "type": "ip"
                                  },
                                  "value": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                  }
                                }
                              },
                              "message_id": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              },
                              "start_date": {
                                "type": "date"
                              }
                            }
                          },
                          "quarantine_name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "group": {
                        "properties": {
                          "priorities": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "allowed_list": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "email": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "org_unit": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "full": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "product": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "sku": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "user_defined_setting": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "resource": {
                        "properties": {
                          "id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "email_dump": {
                        "properties": {
                          "query": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "include_deleted": {
                            "type": "boolean"
                          },
                          "package_content": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "printer": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "mobile": {
                        "properties": {
                          "company_owned_devices": {
                            "type": "long"
                          },
                          "certificate": {
                            "properties": {
                              "name": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              }
                            }
                          },
                          "action": {
                            "properties": {
                              "id": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              },
                              "type": {
                                "ignore_above": 1024,
                                "type": "keyword"
                              }
                            }
                          }
                        }
                      },
                      "url": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "application": {
                        "properties": {
                          "licences_purchased": {
                            "type": "long"
                          },
                          "asp_id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "licences_order_number": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "edition": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "package_id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "enabled": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "field": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "service": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "domain": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "alias": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "secondary_name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "mdm": {
                        "properties": {
                          "vendor": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "token": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "old_value": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      },
                      "user": {
                        "properties": {
                          "birthdate": {
                            "type": "date"
                          },
                          "nickname": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "email": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "device": {
                        "properties": {
                          "command_details": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "serial_number": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          },
                          "type": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      },
                      "gateway": {
                        "properties": {
                          "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                          }
                        }
                      }
                    }
                  },
                  "event": {
                    "properties": {
                      "type": {
                        "ignore_above": 1024,
                        "type": "keyword"
                      }
                    }
                  }
                }
              },
              "data_stream": {
                "properties": {
                  "namespace": {
                    "type": "constant_keyword"
                  },
                  "type": {
                    "type": "constant_keyword"
                  },
                  "dataset": {
                    "type": "constant_keyword"
                  }
                }
              },
              "event": {
                "properties": {
                  "module": {
                    "type": "constant_keyword",
                    "value": "google_workspace"
                  },
                  "dataset": {
                    "type": "constant_keyword",
                    "value": "google_workspace.admin"
                  }
                }
              },
              "tags": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          }
        },
        "_meta": {
          "package": {
            "name": "google_workspace"
          },
          "managed_by": "fleet",
          "managed": true
        }
      }
    }
  ]
}

@andrewkroh andrewkroh changed the title [Google Workspace] ECS Fields Missing from Google Workspace Data Streams [Google Workspace] Missing ECS fields in index.query.default_field when using import_mappings=true Aug 29, 2023
@andrewkroh andrewkroh changed the title [Google Workspace] Missing ECS fields in index.query.default_field when using import_mappings=true [Google Workspace] Missing ECS fields in index.query.default_field when using import_mappings Aug 29, 2023
@andrewkroh andrewkroh added the bug Something isn't working, use only for issues label Aug 29, 2023
@P1llus
Copy link
Member

P1llus commented Sep 1, 2023

There is unfortunately no "quick fix" for this one. We are discussing the best approach here for the future, as in the end, all integrations will use the newer dynamic ECS template.

We can keep this issue open for now for tracking purposes, @felixbarny @ruflin, but the fix is not only for this integration, but rather for all that will be or are using the dynamic template currently.

@ruflin
Copy link
Contributor

ruflin commented Sep 1, 2023

If I remember correctly, the reason that Fleet does not just set * for the default field is because otherwise users see errors on queries because foo query on ip field returns error. The list of fields added to the default_fields can be found here: https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/default_settings.ts#L11

@felixbarny
Copy link
Member

Seems like we're working around limitations in Elasticsearch here. Do we already have an Elasticserach issue to track that enhancement?

@P1llus
Copy link
Member

P1llus commented Sep 1, 2023

I think that we should look into possibilities to make default_fields: * work properly @felixbarny, that would be an enhancement issue with elasticsearch.

@felixbarny
Copy link
Member

felixbarny commented Sep 1, 2023

There's quite a bit of history around default_field. This is what I've found out so far:

Overall, I think there are two options to move forward

  1. Set default_field to message for data streams where message is defined as a field.
  2. Fix Elasticsearch query issues related to setting default_field to *.

As 1) may be considered a breaking change, 2) seems more feasible.

@javanna could you chime in here? Would it be feasible to make default_field=* more lenient to avoid exceptions at search time? Or maybe we could add another way of expressing that the first 1024 fields should be searched by default if we're worried to change the semantics of *.

@ruflin
Copy link
Contributor

ruflin commented Sep 4, 2023

What's a bit surprising to me is that we didn't just remove default_field from metric data streams (why would we want to offer full text search on metrics?).

Lets assume a users as tags: ["foo"] or host.name: bar in their metric document, if they put in foo or bar in the query bar, these docs will show up.

++ on Fix Elasticsearch query issues related to setting default_field to *. Only being able to query on message field by default without being more specific seems to be too limiting.

@javanna
Copy link
Member

javanna commented Sep 7, 2023

I am not entirely up to speed on the issues with setting default field to *. I would expect that to make ES expand to fields that can be queried only and not cause errors. That may slow things down though as it ends up querying a lot of fields, and potentially doc_value only fields too that are slower to query than fields that have an inverted index. Are we absolutely sure that we want to go down that route? Have we considered using a catch_all multi_field that includes all the fields that need querying and set that as a default field?

@jpountz do you have more history / opinions here?

@ruflin
Copy link
Contributor

ruflin commented Sep 7, 2023

I did some testing and so far I have not found issues around IP addresses for setting default fields to *. This is good news, ideally we would have a test suite going through the different types to see if anything stands out. Being able to query all fields I consider to be one of the super powers of Elasticsearch. I think it is ok if things are slower in this scenario (if we should users progress). As usual, the challenge is that in advance, we don't know the fields that will come in, the shipper decides it and we most provide a good experience without requiring the user to do work up front.

@SpencerLN Could you try to use an @custom template to overwrite on your end the default_field setting with * to see if you get the expected result? You should also be able to overwrite it on query time. I'll play around with it on some of our clusters.

I tried the following for the IP address where I thought it had some issues in the past but now the results are as expected:

PUT _index_template/logs-foo
{
  "index_patterns": [
    "logs-foo-*"
  ],
  "data_stream": {},
  "priority": 500,
  "template": {
    "settings": {
      "index.query.default_field": "*"
    },
    "mappings": {
      "properties": {
        "host.name": {
          "type": "keyword"
        },
        "source.ip": {
          "type": "ip"
        }
      }
    }
  }
}

POST logs-foo-bar/_doc
{
  "@timestamp": "2023-09-07T15:04:05.000000001Z",
  "host.name": "elastic.co",
  "source.ip": "34.107.161.234"
}


GET logs-foo-bar/_search?q=elastic.co

GET logs-foo-bar/_search?q=34.107.161.234

@javanna
Copy link
Member

javanna commented Sep 7, 2023

Thanks for the tests, this helps a lot. I believe that the 1024 limit no longer applies after elastic/elasticsearch#81850 and there may have been other changes in the meantime that made support for default_field: * better. Please let us know what issues you encounter and we'll look deeper.

@SpencerLN
Copy link
Contributor Author

@ruflin, I updated our existing google_workspace indices to use the wildcard, and it seems to work alright. After the change, I could search for an email address and IP successfully directly in the query bar in Discover without specifying the field name. I didn't notice any error messages, but I didn't do extensive testing.

PUT .ds-logs-google_workspace.*/_settings
{
  "index": {
    "query": {
      "default_field": "*"
    }
  }
}

@ruflin
Copy link
Contributor

ruflin commented Sep 8, 2023

This is great news @SpencerLN , did also some more tests on my on some larger clusters and could not find issues so far. I'll follow up soon with a bit more specific proposal and what we could do as next step.

@ruflin
Copy link
Contributor

ruflin commented Sep 8, 2023

Here is a proposal on a potential path forward: https://docs.google.com/document/d/1EA6jeWM1VElGuQwEXzxDZ1hNmVYPCDIGLxU_zl0xeR4/edit @javanna @jpountz Would be great if you could a look at this from the Elasticsearch perspective. I think the combination of changes that happened over the past 12 months will allow us to have a simpler implementation that "just works". @andrewkroh @P1llus Please also have a look.

@ruflin
Copy link
Contributor

ruflin commented Sep 25, 2023

After same discussion in the doc I know opened the following issue in the Elasticsearch repo with the proposal of changing the default to *: elastic/elasticsearch#99872

@narph narph added Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] and removed Team:Security-External Integrations labels Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working, use only for issues Integration:google_workspace Google Workspace Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Development

No branches or pull requests

8 participants