[Jaeger] Support receiving Jaeger spans

[axw]

We have heard from users that they would like to use their existing Jaeger instrumentation with Elastic APM. It is already possible to store and search span data in Elasticsearch, but what is missing is integration with Elastic APM and related features, such as visualisation in the APM UI, central configuration, log correlation.

There are at least a couple of ways we can go about this:

• We already have an issue to implement an OpenTelemetry Exporter (Feature: OpenTelemetry exporter apm#98). We could wait for OpenTelemetry Collector to be ready, and require users to redirect their Jaeger agents to OpenTelemetry Collector.
• We could add support to the APM Server to ingest Jaeger data directly.

The exporter option has a couple of problems. First and foremost, it's more complex: an additional component is required, complicating users' lives. Second, we would not be able to support Jaeger's remote sampling in this way.

I suggest that we add support directly to the APM Server for receiving spans, and translating them into our data schema. The server would effectively be a drop-in for jaeger-collector. To do this, we'll need to introduce HTTP and gRPC endpoints, for receiving spans directly from clients, and from jaeger-agent, respectively. Jaeger also supports TChannel, but I don't think we need to support this.

It may be useful to reuse some of the machinery inside opentelemetry-collector, so that the server is effectively implementing an OpenTelemetry Exporter within the same process. This may enable us to add support for additional trace data sources in the future. I have prototyped this approach here: 7.x...axw:otel-7.x (sorry, it's a bit of a mess.)

One thing that is yet to be solved is auth. A limited subset of the Jaeger clients (Java and C#) support the JAEGER_AUTH_TOKEN config, which can be used to send a bearer token along with requests to the collector. This only works when using HTTP. The Jaeger agent only supports client certificate auth. We do support this, but it might also be worth a lighter-weight option, such as ferrying a secret/bearer token through tags, using the --jaeger.tags CLI flag.

[simitt]

TODOs:

• [impl]: Finish adding gRPC endpoint, running on port 14250, to ingest spans sent by jaeger-agent in model.proto format ([Jaeger] Add gRPC endpoint to APM Server to allow ingesting spans from jaeger agent #2962, @simitt )
• [impl]: Finish adding HTTP endpoint, running on port 14268, to ingest spans sent by jaeger-client via thrift over http ([Jaeger] Add HTTP endpoint accepting spans via thrift from Jaeger client #3008)
• [impl]: Finish implementation for transformation of jaeger events to Elastic APM events. Validate input and decide on a strategy for invalid input: figure out which attributes are required for UI and how to fill them if missing, or log event and drop it ([Jaeger] Implement event processor #3007, WIP @simitt )
• [impl]: Investigate error responses expected by jaeger
• [impl]: Investigate if healthcheck endpoint on port 14269 is needed and implement if so
  -> Investigated jaeger agent and the two jaeger client libs, supporting thrift/HTTP (java and c#) and didn't find any usage of the collector healthcheck endpoint that would be relevant.
• [ ] [testing]: auth handling: Add integration tests for client auth via TLS
  -> TLS setup including client auth has been reused from apm-server.ssl, which is tested in test_access.py
• [testing]: Figure out how to and to which extent to add system tests in the APM Server (maybe the hotrod example app can be leveraged) ([Jaeger] Add system tests for gRPC integration with Jaeger agent #3009, [Jaeger] Add system tests for http thrift integration with Jaeger agent #3010)

Docs:

• [documentation] [docs] Document experimental Jaeger support #3123

End-to-end Testing:

• [testing]: Add dockerized setup for automated tests in the integration repo: have a dedicated test app using the jaeger agent for instrumentation and expecting events in ES in a certain format ([Jaeger] Add setup for jaeger agent integration with APM Server via gRPC apm-integration-testing#679, WIP @graphaelli )
• [testing]: Add dockerized setup for automated tests in the integration repo: have a dedicated test app using the jaeger client via thrift over http for instrumentation and expecting events in ES in a certain format ([Jaeger] Add setup for jaeger client integration with APM Server via thrift over http apm-integration-testing#680)

Follow up topics/concepts:

• [concept+impl]: When using probabilistic sampling jaeger only sends sampled spans. Elastic APM requires a 100% of the transaction events and only samples spans. Investigate how to solve sampling with jaeger integration ([Jaeger] How to support sampling #3011)
• [concept]: auth handling: Investigate further options, e.g. using special tags

Out of scope:

• support for tchannel
• providing endpoint exposing metrics
• support on ESS

[axw]

[concept]: When using probabilistic sampling jaeger only sends sampled spans. Elastic APM requires a 100% of the transaction events and only samples spans. Investigate how to solve sampling with jaeger integration.

Outside of the context of Jaeger, there's been some discussion around relying on metrics so we don't need to send and store non-sampled transactions. This is elastic/apm#104.

If we were to do that, we could have the server aggregate metrics based on sampled Jaeger trace data. Jaeger sends up the sampling rate in effect at the time a trace is captured, so we can scale the count/total based on that.

[simitt]

Work for 7.6 release has been finished. Moving follow up topics to new meta issue #3340.