API Key support for APM Integration when using non-Elasticsearch output

[simitt]

Define if/how APM Integration should support API Keys for auth handling between APM Agents and APM Server (apm-server.auth.api_key.enabled: true) when a different output than Elasticsearch is configured.

As pointed out by @joshdover in #7025 (comment) the output might be changed after the apm integration was added and configured.
For now, the API Key setup could either not be supported or a yaml box for ES host configuration could be added.

See #7025 for more discussion for a similar issue.

[axw]

@ruflin and I are discussing API Key auth as part of an evolution of the Elastic Agent architecture. It's still early days, but the idea would be for APM Server to make API Key queries via Elastic Agent. In that kind of architecture, I believe the APM Server's output is irrelevant.

[simitt]

Interesting, thanks for the pointer @axw .

Since logstash output is currently planned to be supported in the near future, we could scope this issue to disable changing the output type to something else than Elasticsearch, if API Key auth is enabled on a package policy and not allow enabling API Keys on an APM Integration if the logstash output is already set.
The goal is to avoid breaking data ingestion from APM Agents to Server, when output type is changed or set to logstash. This is a non-obvious dependency, and it would be hard for users to find the root cause of why ingestion is broken, so we should avoid even reaching this state.

Support for API Key handling when output != ES can then be added once API Key handling via Elastic Agent is supported.

[axw]

Sounds good to me.

[ruflin]

I like the idea that a package can define what sorts of outputs it supports.

[simitt]

I like the idea that a package can define what sorts of outputs it supports.

It would not be on a per package base though. The apm package has some features that don't support other outputs yet, so it would be conditional.

[ruflin]

This makes it unfortunately more complex. Now on each change it has to be checked if a feature is around and if an integration still is working for a policy. And if the output on a policy is changed, also checks need to happen. From a Fleet perspective I guess it means an integration policy will contain some meta information on output compatibility. If we go down that path we need to define the exact error behaviours in the different scenarios.

[simitt]

If restrictions are not applied on a feature base, it would mean that APM Server could not support any other output than ES until Elastic Agent/Fleet Server support the features. Is it realistic that the API Key handling and the ES syncronisation required for TBS (#7025) is planned and implemented any time soon?

[alex-fedotyev]

Could you help understanding couple more details on this subject please?

1. I suspect that the answer is YES, but want to confirm: Kafka output would introduce same challenges as Logstash output, right? Main reason to bring that up is that Kafka is more common output for APM than Logstash in my experience.
2. Will other functionality like agent config management suffer from this same way as API keys?

Below is the configuration which comes to mind:

[axw]

I suspect that the answer is YES, but want to confirm: Kafka output would introduce same challenges as Logstash output, right? Main reason to bring that up is that Kafka is more common output for APM than Logstash in my experience.

Yes, any non-Elasticsearch output will have the same challenge.

Will other functionality like agent config management suffer from this same way as API keys?

The only other feature that currently needs a direct connection to Elasticsearch is TBS. Agent configuration is pushed down to APM Server via Fleet.

In the past, APM Server would search Elasticsearch directly for RUM source maps; it's now using a hybrid push/pull method that only requires connection to fleet-server, and does not require access to Elasticsearch.

[simitt]

Conclusion reached with @joshdover and @mostlyjason and @chrisdistasio:
for the initial release of logstash support in Fleet and Agent (planned for 8.2), the APM Integration will not be supported. If an agent policy contains the APM Integration, the logstash output will not be a valid choice. For existing agent policies with configured logstash output, adding the APM Integration will be prohibited.
For this initial phase, no changes are required from the APM side (neither apm-server nor apm-ui).

The issue remains open until we found a longer term solution how APM can support logstash output, while maintaining apm agent auth via API Keys.