x-pack/plugin/apm-data: map some APM fields as flattened and fix error.grouping_name script

[axw]

Various fixes to the apm-data plugin:

• fix the error.grouping_name script to not throw an exception when an error.exception object lacks a message attribute
• statically map various APM-specific fields as flattened to prevent mapping conflicts

Also expand the YAML REST test coverage.

[axw]

Requires #103033 and #103035

[elasticsearchmachine]

Hi @axw, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

Hi @axw, I have a couple of contribution drive-by comments:

• We try not to force-push PRs once they've been opened publicly, as it makes tracking changes over a period of time much harder for a reviewer, since that history is lost. Usually we merge main (if we need to update a PR to be based on the latest changes in main) which retains the history of the PR. We even have a built-in command for this: commenting @elasticmachine update branch will automatically merge main into your PR.
• For these, can you also increment the REGISTRY_VERSION, since you're changing the mappings? Even though they may be in the same point release as other changes, on Serverless there are no versions, so we'll rely on the registry version to be able to install the latest templates/policies/etc on upgrade.
• In the future, can you use a more descriptive title or changelog summary for these PRs? The changelog gets automatically generated from the PR title, which in this case means it'll go into the public docs as "X-pack/plugin/apm-data: minor fixes & tests". We want these changelogs to be as useful as possible for users, so they really know what changed when they go to install 8.13.0. Perhaps something like "Map some APM fields as flattened and fix error.grouping_name script" would be useful for our users when they're evaluating the changelog.

Hope this makes sense, let me know if it doesn't, thanks!

[axw]

Thanks for taking a look @dakrone!

We try not to force-push PRs once they've been opened publicly, as it makes tracking changes over a period of time much harder for a reviewer, since that history is lost. Usually we merge main (if we need to update a PR to be based on the latest changes in main) which retains the history of the PR. We even have a built-in command for this: commenting @elasticmachine update branch will automatically merge main into your PR.

Got it. Just to be sure, this applies even to draft PRs? (It was draft when I did that.)

For these, can you also increment the REGISTRY_VERSION, since you're changing the mappings? Even though they may be in the same point release as other changes, on Serverless there are no versions, so we'll rely on the registry version to be able to install the latest templates/policies/etc on upgrade.

I don't think it's necessary yet -- we've not yet enabled this in serverless (or anywhere else). The templates/pipelines were a little bit broken, and this PR is getting it to a state where it can start to be enabled. WDYT?

In the future, can you use a more descriptive title or changelog summary for these PRs? The changelog gets automatically generated from the PR title, which in this case means it'll go into the public docs as "X-pack/plugin/apm-data: minor fixes & tests". We want these changelogs to be as useful as possible for users, so they really know what changed when they go to install 8.13.0. Perhaps something like "Map some APM fields as flattened and fix error.grouping_name script" would be useful for our users when they're evaluating the changelog.

Understood, sorry! Will endeavour to make them more meaningful.

[axw]

@elasticmachine update branch

[dakrone]

Got it. Just to be sure, this applies even to draft PRs? (It was draft when I did that.)

This is more personal taste (the no-force-merge rule is mostly for non-draft PRs), so I think this one in particular depends on the person.

I don't think it's necessary yet -- we've not yet enabled this in serverless (or anywhere else). The templates/pipelines were a little bit broken, and this PR is getting it to a state where it can start to be enabled. WDYT?

Ahh okay, I hadn't realized this wasn't enabled in Serverless yet, can you point me to where that flag is? Or an issue I can watch for where it'll be enabled eventually?

Thanks!

[axw]

Ahh okay, I hadn't realized this wasn't enabled in Serverless yet, can you point me to where that flag is? Or an issue I can watch for where it'll be enabled eventually?

The flag to enable the plugin is:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackSettings.java

Lines 93 to 98 in 79385d2

	/** Setting for enabling or disabling APM Data. Defaults to false. */
	public static final Setting<Boolean> APM_DATA_ENABLED = Setting.boolSetting(
	"xpack.apm_data.enabled",
	false,
	Setting.Property.NodeScope
	);

There's no specific issue yet for enabling this in serverless, but the general issue for tracking related changes is elastic/apm-server#11529. I've written down the rough steps that we'll take to roll out the changes, which I expect we'll make issues out of later.

[axw]

@elasticmachine update branch