Add OTEL_EXPORTER_OTLP_PROTOCOL env variable support

[akvlad]

In the opentelemetry documentation you have link to, there is a configuration of the protocol for traces to send: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.7.0/specification/protocol/exporter.md#specify-protocol. It can be grpc or http/protobuf etc.

As far as I see in the code: https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/tracing/tracer.go#L61 you have only grpc supported.

It would be great to have other options like http/protobuf supported as well. Some tracing ingesters have the no GRPC support by different reasons.

[francislavoie]

PRs welcome. Most maintainers don't use tracing or metrics ourselves, so we need to lean on the community to contribute improvements.

[trajano]

I think this may be the reason Caddy doesn't work with grafana since they only expose the HTTPS port

[hairyhenderson]

I think this may be the reason Caddy doesn't work with grafana

Can you elaborate? In what way does Caddy "not work with grafana"?

[trajano]

When you create a free stack at grafana.com and try to set up their OTEL collector it recommends the following environment variables

export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic SECRET"

I also added

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://otlp-gateway-prod-us-east-0.grafana.net/otlp"

So problems with Caddy are

1. http/protobuf is the protocol required but as per the OP of this issue, it does not work because it's hard coded to GRPC tracer only. I tried to work around it with a change in my PR
2. The change unfortunately didn't work as somewhere else in the caddy codebase or one of it's imports it looks at the OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT it parses the value and determines there's no port value (since grafana uses https which has a default port of 443 it doesn't place it)
3. Adding the port i.e. https://otlp-gateway-prod-us-east-0.grafana.net:443/otlp will make Caddy parse out the port as 443/otlp then there'd be an error saying port 443/otlp is not valid. I am unable to ascertain where that is in the code base

[trajano]

Maybe it's the way I am testing

RUN git clone https://github.com/trajano/caddy /mnt/
RUN xcaddy build \
    --replace github.com/caddyserver/caddy/v2/modules/caddyhttp/tracing=/mnt/modules/caddyhttp/tracing

FROM caddy
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

[trajano]

Figured out what was wrong with the Dockerfile

Got a bit further but got

traces export: failed to send to https://otlp-gateway-prod-us-east-0.grafana.net/oltp/v1/traces: 404 Not Found

But posting directly using curl seems to work

[trajano]

Finally got it working, the last bit was a typo might've accidentally transposed two characters in the otlp in https://otlp-gateway-prod-us-east-0.grafana.net/otlp I see data in Grafana now with my PR.

[trajano]

I can't merge #6487 until master builds cleanly. [https://opengraph.githubassets.com/2a261bf798dd7b9f98bb5d496a4efcb97869d4da56424bb702b2c377132fad69/caddyserver/caddy/pull/6487]<https://github.com/caddyserver/caddy/pull/6487> Add Otel Trace client interactions by trajano · Pull Request #6487 · caddyserver/caddy<#6487> Should be okay now for the most part it works well see trajano/google-fonts-proxy-docker#7 (comment) for a screenshot But there&#39;s one context.TODO() line I am not sure which context to pass in ... github.com

…

________________________________ From: 'core2duoe6420' via Open Source Development ***@***.***> Sent: Tuesday, August 27, 2024 9:29 AM To: caddyserver/caddy ***@***.***> Cc: Archimedes Trajano ***@***.***>; Mention ***@***.***> Subject: Re: [caddyserver/caddy] Add OTEL_EXPORTER_OTLP_PROTOCOL env variable support (Issue #5743) @trajano<https://github.com/trajano> Do you have plan to raise a pull request to merge your change? I believe the way you are doing is the canonical way to instrument tracer and exporters. — Reply to this email directly, view it on GitHub<#5743 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAA3AI4O3JOHTNGI4X2DVPLZTR5K7AVCNFSM6AAAAABLSTTJMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJSGU3DOMRYGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>