[Fleet] Avoid subobject/scalar mapping conflicts by setting subobjects: false on custom integrations

[flash1293]

In case of a property in indexed documents is sometimes a scalar value (e.g. a number or string) and sometimes an object with sub-properties, mapping conflicts will prevent data from being indexed properly.

The subobjects mapping property makes sure no additional subobject mappings will be defined during runtime. You can learn more here: https://www.elastic.co/guide/en/elasticsearch/reference/current/subobjects.html

This PR sets subobjects: false on logs data streams for custom integrations to avoid. This only affects newly created integrations.

The setting is set via the @package component template.

This only requires a small change in the generated datastream entry, as index_template.mappings are already spread into the index config of the component template:

kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts

Line 380 in c98ee2f

...omit(indexTemplateMappings, 'properties', 'dynamic_templates', '_source', 'runtime'),

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[flash1293]

@felixbarny is it fair to call this a "feature" in the release notes? What would be a good doc page to link to? Simply https://www.elastic.co/guide/en/elasticsearch/reference/current/subobjects.html ?

[felixbarny]

@felixbarny is it fair to call this a "feature" in the release notes?

Yes, I think so. But I would not frame it as "setting subobjects to false" but talk more about the impact this has i.e. avoiding object/scalar conflicts.

What would be a good doc page to link to? Simply https://www.elastic.co/guide/en/elasticsearch/reference/current/subobjects.html ?

Yes, I think so. Note that on the master branch (https://www.elastic.co/guide/en/elasticsearch/reference/master/subobjects.html), this contains more info about auto flattening. This will be current once 8.14 is released.

[juliaElastic]

LGTM

[flash1293]

Updated the description

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: bd3754c

The CI Stats report is too large to be displayed here, check out the CI build annotation for this information.

History

• 💔 Build #197217 failed 4562529

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @flash1293