[InfraUI] Allow users to filter by service.name

[roncohen]

We should make it possible to filter the InfraUI view to show hosts/pods/containers that are serving a specific APM service. You can imagine linking directly from the APM UI service list to the infra UI to see the an overview of the infrastructure that currently deals with this service.

I did a preliminary investigation and the good news is that it almost works. APM sends up data that includes service.name, host.name, container.id, and kubernetes.pod.id where available. There are 3 outstanding issues:

1. APM sends host.hostname while infra expects host.name. By setting up an ingest pipeline that just copies the value for APM data i got it to work for the "host" page. I'll talk to ECS folks and APM folks separately to see what we do about it.

2. When querying, we include host.name, container.name etc. in the composite grouping key. The APM documents do not have container.name while the metricbeat documents have container.name and so when we try to merge buckets it does not work because the keys don't match up. I wonder if there's a reason we use the names in additions to the IDs in the grouping or if it's a leftover from [InfraUI] Use container IDs instead of names for grouping, filtering and linking #26359

3. Users will need to alter their source settings. We should consider adding apm-* or similar to the default

[elasticmachine]

Pinging @elastic/infra-logs-ui

[skh]

I wonder if there's a reason we use the names in additions to the IDs in the grouping

IIRC we add them to the composite query to be able to use them for displaying the nodes. According to https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html we should be able to set missing_bucket: true on the name fields and fall back to the id fields when needed.

[roncohen]

will retest once #40804 is in

[roncohen]

having Infra UI automatically query APM indices relates to ensure index patterns are defined in the same place: #41360

[jasonrhodes]

This work relies on: elastic/apm-server#2502

[sgrodzicki]

As elastic/apm-server#2540 got merged, what are the next steps for this one?

[sgrodzicki]

@roncohen are there still any blockers?

[roncohen]

closing as the code has been massaged to allow for this use case. Will open separate discussion on what to do with the index patterns.