Support injecting instrumentation via the OpenTelemetry Operator in Kubernetes

[devrimdemiroz]

Feature Request

Currently, auto-instrumentation is injected via Docker file.

While having it for Docker deployment is fine, for Kubernetes deployment, it is a preferred option to get it injected automatically via opentelemetry-operator.

Describe the solution you'd like:

Instead, agent(s) can be injected via opentelemetry-operator

annotations:
  instrumentation.opentelemetry.io/inject-java: otel-agent-xxxxx

[JaredTan95]

I think it should be solved by #244

becauese the opentelemetry-operator will inject the java agent sidecar/initcontainer and override the JAVA_TOOL_OPTIONS value.

annotations:
  instrumentation.opentelemetry.io/inject-java: otel-agent-xxxxx

[devrimdemiroz]

@JaredTan95 ,
I put java as an example. Same applies for other languages so far supported via operator injection.

[puckpuck]

All the existing services in the demo have manual instrumentation, which requires them to be compiled with the Otel SDKs/Agents for them to run.

There is the potential for this with the addition of new services where we don't do any manual instrumentation in them and have those new services auto-instrumented only using operator injection. In particular, I think the Message queue implementation with 2 new services would be a good candidate for this.

[devrimdemiroz]

@puckpuck

I still did not deep dive into particular services. However, here (except C++) each service is marked with both auto and manual instrumentation. If the page is correct, we will lose the manual instrumentation provided as demo but still have auto instrumentation in hand. Do I get it right?

[puckpuck]

To provide some status on this issue.

As a top-level item, moving to support the Operator is dependent on either adding similar functionality to the existing docker-based dev environment or dropping support completely for docker as a local dev environment which is currently up for discussion in #635

In the SIG meetings, we have decided to add auto-instrumentation via the Operator for the accountingservice and fraudservice components of the demo. Since these services communicate with Kafka, we need to ensure they are written in languages that have Kafka auto-instrumentation available and can be instrumented with the Operator.

Currently, these services are written in Go and Java:

• fraudservice - Java
• accountingservice - Go

Without Go support for Operator based auto-instrumentation, we need re-write the accountingservice in another language. The Operator currently supports:

• Java
• Javascript
• Python
• .NET

Since we already have an auto-instrumented service in Java, a different language would be preferred that also supports Kafka auto-instrumentation. The Otel-Demo already contains 2 services using the JavaScript SDK. We also have 1 service + load generator using the Python SDK. This leaves .NET as the top candidate. Unfortunately the .NET SDK does not yet support Kafka auto-instrumentation. If delivery of Kafka auto-instrumentation with the .NET SDK is not imminent we will look to rewrite the service using Python instead.