Support for OpenTelemetry in addition to OpenTracing?

[hairyhenderson]

The OpenTracing support is super useful, however recently OpenTracing has merged with the OpenTelemetry project, and it seems like many more integrations will be possible.

To my knowledge there doesn't yet exist an OpenTelemetry equivalent to the nginx-opentracing library yet, but the original author (@rnburn) is also active in https://github.com/open-telemetry/opentelemetry-cpp.

Given the lack of library it may be premature to ask for this in this repo, but I thought I would at least start the conversation here to see if there's any other interest.

For now my plan is to configure nginx-ingress-controller with the zipkin-collector-host parameter pointing at an instance of opentelemetry-collector, configured with the zipkin receiver.

The downside is that trace propagation won't be seamless - downstream services will have to be configured to support Zipkin (B3/X-B3-*) headers as well as the W3C Trace-Context (Traceparent) header.

/kind feature

[aledbf]

@hairyhenderson interest sure, but I would prefer to know what is going to happen with nginx-opentracing first.
From the three plugins we use, only datadog is being actively maintained. Also, the three use different versions of the same libraries, something that introduces additional time to build the image. Errors like this one are hard to track too.
Link to the request to update zipkin-cpp-opentracing

[hairyhenderson]

@aledbf ok... that may be orthogonal, though obviously it would be good to have the OpenTracing libraries up-to-date. I've noticed that the Jaeger client was recently updated, so it may be useful to bump that.

From an outsider's perspective it seems that a bit more effort is being put towards OpenTelemetry right now, which may explain the lull in OpenTracing development.

[aledbf]

From an outsider's perspective it seems that a bit more effort is being put towards OpenTelemetry right now, which may explain the lull in OpenTracing development.

I agree with that. Now convince @rnburn to write a new nginx module 😉

[hairyhenderson]

@aledbf I have no relationship with @rnburn and I'm sure he's under no obligation to listen to me 😉

There's also a feature request opentracing-contrib/nginx-opentracing#132 that may make it even easier to use ingress-nginx with opentelemetry, as it would solve the propagation issue. I'm just putting it here for reference.

[aledbf]

There's also a feature request opentracing-contrib/nginx-opentracing#132

I am aware of that issue :) #5716

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[project0]

/remove-lifecycle stale

i hate this bots...

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[lalitb]

/remove-lifecycle stale

Keeping alive to look forward on more interest in this requirement in coming days.

[dgzlopes]

Related: https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx

[miry]

With new version of Jaeger v0.7.0
jaegertracing/jaeger-client-cpp@c1d5795
it is possible to use W3C format in ingress-nginx with Jaeger tracer.

Ingress-nginx commit with new changes 71c8ef1

Example of config ingress-config:

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: nginx-load-balancer-conf
data:
  enable-opentracing: "true"
  jaeger-service-name: nginx-ingress
  jaeger-collector-host: otel-collector.observability.svc.cluster.local
  jaeger-propagation-format: w3c
  worker-processes: "1"

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[msschl]

/remove-lifecycle stale

[rikatz]

@msschl can you confirm comment from @miry does not work your for you?

#5883 (comment)

Also there's some instrumentation module for OTEL right now, if someone proposes to work on a PR for this I can review it.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[dmathieu]

/remove-lifecycle stale
I am working on this, though I'm current blocked by base image build timeouts.

[rikatz]

/lifecycle active
@dmathieu let me try to help you unblocking this during this/next week

[iamNoah1]

/triage accepted

[dmathieu]

The latest development of this is #8013, which aims to bring the OpenTelemetry module as a sidecar.

[iamNoah1]

@dmathieu can consider this then as being obsolete with the implementation of #8013 ?

[dmathieu]

Even with that PR, we will need #7621 (rebased and probably adapted) to have actual support.
I wouldn't consider this issue obsolete, as it tracks the need for the feature.

[cskinfill]

Could this be used?
https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx

[rcjsuen]

Could this be used? https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx

@cskinfill It looks like it's being used per #7621.

[cskinfill]

nice! so any more progress being made on support for OpenTelemetry? Looks like #7621 is staled???

[dmathieu]

Ongoing work is happening in #8013.

[strongjz]

@dmathieu looks like there some issues with building the image in prow https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-ingress-nginx-build-opentelemetry/1498713694396223488

[dmathieu]

Sorry about that @strongjz. The nginx version was changed as a PR feedback, and I missed the checksum. #8286 fixes it.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[dmathieu]

/remove-lifecycle stale

[thomaschaaf]

Is this resolved with the latest merged PR #8719?

[Tobrek]

No. At least PR #8735 is still missing, to get a working open telemetry solution (using the sidecar). And after merging #8735 there is no specific configuration of the used open telemetry nginx module possible - enable or disable it at all. Maybe there will be am additional pull request to enable the detail configuration for the module.

[esigo]

WIP in #9016.
This issue probably can be closed.

[hairyhenderson]

#9016 is a duplicate of this, but I'm fine closing this in favour of that one. I no longer use ingress-nginx anyway so I'm happy receiving fewer notifications 😅