Support ingesting OpenTelemetry metrics

[spencergilbert]

Ref: #16121

[arendjr]

I was just trying out Vector for a PoC and was bumped to see it didn't support OTLP metrics yet, but now I'm happy to see you self-assigned, @pront :) Does this mean we can look forward to this soon?

[pront]

Hi @arendjr, yes, I will be working on this in the following weeks!

[pront]

Quick update, unfortunately this has been put on hold for now. We will update this issue if something changes.

[arendjr]

Too bad. Do you think it’s something that could be reasonably picked up by an outside contributor? If you have some pointers, I might have a chance to look at it.

[gaby]

Any updates or alternatives for this?

[gaby]

@pront Any updates on this?

[pront]

@pront Any updates on this?

Hi @gaby, no. Nothing from me. I am focusing on logs only at the moment.

RFCs and contributions are most welcome.

[mayanksingh2298]

Why can't I simply use the http sink in vector to send json encoded data to otel http endpoint?

[gaby]

@mayanksingh2298 I think you can as long as both sides support OTEL via HTTP ?

[mayanksingh2298]

otel definitely does: https://github.com/open-telemetry/opentelemetry-proto/blob/main/examples/README.md

I can configure my source to do so.
For some reason I'm getting a 400 Bad request in vector http sink.

Any tips on debugging this?

[gaby]

Probably something with the encoding, I havent tried this this yet. Default method is POST in Vector

[mayanksingh2298]

I used tcpdump to analyze the requests. Turns out vector sends an array [{},...] of jsons but otel collector expects a single json.

I followed this comment by @EdMcBane #3378 (comment) and it helped me out.

[mayanksingh2298]

In summary this is my workaround config to send otel traces (also works with other telemetry) to an otel collector.

sinks:
  sink_internal_otel_collector:
    type: http
    request:
      headers:
        Content-Type: application/json
    encoding:
      codec: json
    uri: http://localhost:4318/v1/traces
    batch:
      max_size: 1
    framing:
      method: newline_delimited
    inputs:
    - source_FluentD_log_httpnew 

sources:
  source_FluentD_log_httpnew:
    address: 0.0.0.0:49153
    decoding:
      codec: json
    framing:
      method: bytes
    type: http_server

Hopefully this helps someone

[gaby]

@mayanksingh2298 Thanks for sharing, you could also split by newline in a separate step, but yours should work too

[CmdrSharp]

Is there any work being done on this? I feel like the limitations on receiving and sending OpenTelemetry-data is a tad problematic. I'm happy to try to look into this myself, but it'd be helpful to understand why progress was halted initially

[jszwedko]

Is there any work being done on this? I feel like the limitations on receiving and sending OpenTelemetry-data is a tad problematic. I'm happy to try to look into this myself, but it'd be helpful to understand why progress was halted initially

I think the challenge will be mapping OTLP to Vector's internal data model (which might require expanding Vector's internal data model). This is a fairly broad change which is why it hasn't happened just yet, though I recognize OTLP support is particularly important for a tool like Vector and so we will get there eventually.