docs: OpenTelemetry Elastic exporter

[bmorelli25]

Motivation/summary

This PR adds initial documentation for the OpenTelemetery collector Elastic exporter. I ended up providing a lot more detail than I expected--mostly because the official OT docs are still quite light.

I think it'd be nice to add a quick start or docker demo as an example.

Any and all feedback is welcome.

Documentation preivew

Click here to see a doc preview 🚀

Notes

I've used two different setups to play around with this integration. The first only kind of worked:
Jaeger hot rod --> Jaeger Agent --> Jaeger oltp collector --> Otel Collector --> ESS. I was able to get traces into ES, but I don't think the Otel collector liked the formatting of http.url from Jaeger hot rod demo.

I then altered the OpenTelemetry demo app to use the zipkin sleuth webmvc example app --> Otel collector --> ESS. Super basic, but it worked nicely. You can see the dockerfile/compose/etc., here.

@cyrille-leclerc gave me access to his demo application, so I'll give that a shot tonight or tomorrow as well.

To do

• Real architecture diagram instead of my text-based version
  

• Define IDs for all headers

- [ ] Quickstart?

Related issues

Closes elastic/apm#269.

[apmmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #3845 updated]

• Start Time: 2020-06-08T17:13:17.126+0000

• Duration: 47 min 31 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3204
Skipped	146
Total	3350

Steps errors

Expand to view the steps failures

• Name: Run Window tests

  • Description:

  • Duration: 7 min 26 sec

  • Start Time: 2020-06-08T17:31:12.680+0000

  • log

[axw]

I've used two different setups to play around with this integration. The first only kind of worked:
Jaeger hot rod --> Jaeger Agent --> Jaeger oltp collector --> Otel Collector --> ESS. I was able to get traces into ES, but I don't think the Otel collector liked the formatting of http.url from Jaeger hot rod demo.

What do you mean it didn't like it? Were there errors? Dropped transaction/spans?

[bmorelli25]

I've used two different setups to play around with this integration. The first only kind of worked:
Jaeger hot rod --> Jaeger Agent --> Jaeger oltp collector --> Otel Collector --> ESS. I was able to get traces into ES, but I don't think the Otel collector liked the formatting of http.url from Jaeger hot rod demo.

What do you mean it didn't like it? Were there errors? Dropped transaction/spans?

Ah, sorry.

This, repeated:

otel-collector_1    | 2020-06-05T02:56:37.310Z  WARN    queuedprocessor/queued_processor.go:190 Sender failed   {"component_kind": "processor", "component_type": "queued_retry", "component_name": "queued_retry", "error": "[parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon; parse \"0.0.0.0:8083\": first path segment in URL cannot contain colon]"}
otel-collector_1    | go.opentelemetry.io/collector/processor/queuedprocessor.(*queuedSpanProcessor).processItemFromQueue
otel-collector_1    |   /go/pkg/mod/go.opentelemetry.io/[email protected]/processor/queuedprocessor/queued_processor.go:190
otel-collector_1    | go.opentelemetry.io/collector/processor/queuedprocessor.(*queuedSpanProcessor).Start.func1
otel-collector_1    |   /go/pkg/mod/go.opentelemetry.io/[email protected]/processor/queuedprocessor/queued_processor.go:86
otel-collector_1    | github.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1
otel-collector_1    |   /go/pkg/mod/github.com/jaegertracing/[email protected]/pkg/queue/bounded_queue.go:77
otel-collector_1    | 2020-06-05T02:56:37.310Z  WARN    queuedprocessor/queued_processor.go:202 Failed to process batch, re-enqueued    {"component_kind": "processor", "component_type": "queued_retry", "component_name": "queued_retry", "batch-size": 33}
otel-collector_1    | go.opentelemetry.io/collector/processor/queuedprocessor.(*queuedSpanProcessor).processItemFromQueue
otel-collector_1    |   /go/pkg/mod/go.opentelemetry.io/[email protected]/processor/queuedprocessor/queued_processor.go:202
otel-collector_1    | go.opentelemetry.io/collector/processor/queuedprocessor.(*queuedSpanProcessor).Start.func1
otel-collector_1    |   /go/pkg/mod/go.opentelemetry.io/[email protected]/processor/queuedprocessor/queued_processor.go:86
otel-collector_1    | github.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1
otel-collector_1    |   /go/pkg/mod/github.com/jaegertracing/[email protected]/pkg/queue/bounded_queue.go:77
otel-collector_1    | 2020-06-05T02:56:37.310Z  WARN    queuedprocessor/queued_processor.go:208 Backing off before next attempt {"component_kind": "processor", "component_type": "queued_retry", "component_name": "queued_retry", "backoff_delay": "5s"}
otel-collector_1    | go.opentelemetry.io/collector/processor/queuedprocessor.(*queuedSpanProcessor).processItemFromQueue
otel-collector_1    |   /go/pkg/mod/go.opentelemetry.io/[email protected]/processor/queuedprocessor/queued_processor.go:208
otel-collector_1    | go.opentelemetry.io/collector/processor/queuedprocessor.(*queuedSpanProcessor).Start.func1
otel-collector_1    |   /go/pkg/mod/go.opentelemetry.io/[email protected]/processor/queuedprocessor/queued_processor.go:86
otel-collector_1    | github.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1
otel-collector_1    |   /go/pkg/mod/github.com/jaegertracing/[email protected]/pkg/queue/bounded_queue.go:77

But I was able to receive data in ESS. Lots of spans though:

Click to expand screenshot

I just figured it was user error or Jaeger not playing nicely with Otel. If you want to try it out, here's the docker-compose. docker-compose up. Then head to localhost:8080 to click some buttons.

[axw]

Looks good overall! Just a handful of comments, nothing major.

I'll have a look at what's going on with Jaeger/HotROD data, sounds like a bug in the exporter. Thanks for highlighting that issue.

[axw]

I've made a change to the exporter to gracefully handle the http.url values that the OpenTracing instrumentation is recording: open-telemetry/opentelemetry-collector-contrib#285. Thanks for picking that up!

[axw]

Looks great, thank you!

[codecov-commenter]

Codecov Report

Merging #3845 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3845   +/-   ##
=======================================
  Coverage   79.70%   79.70%           
=======================================
  Files         135      135           
  Lines        6158     6158           
=======================================
  Hits         4908     4908           
  Misses       1250     1250