Fix (possible) bug where cross-namespace auto-instrumentation configs are not always supported

[jvoravong]

Description:

There seems to be an inconsistency when deploying a target application, the Instrumentation opentelemetry.io/v1alpha1, and the Operator across different namespaces: auto-instrumentation might fail in some configurations. While the current documentation indicates support for such deployments, we need to clearly define which scenarios are supported and rectify any discrepancies.

For clarity, I've expanded and provided some e2e tests in a fork:

• Scenario 1 - Success✅
  • Namespace A:
    • Operator+Collector
    • Target Application
  • Namespace B:
    • Instrumentation
• Scenario 2 - Fail❌
  • Namespace A:
    • Operator+Collector
  • Namespace B:
    • Target Application
    • Instrumentation
• Scenario 3 - Fail❌
  • Namespace A:
    • Operator+Collector
    • Instrumentation
  • Namespace B:
    • Target Application

AC:
Instrumentation configurations from any namespace should seamlessly apply to a pod, irrespective of the namespace housing the target, Instrumentation opentelemetry.io/v1alpha1, and Operator.

Related:

• Clarify namespace requirements for Instrumentation objects when auto-instrumenting #2147
• Cross namespace instrumentation #889

Lastly, I'd like to offer my assistance in addressing the failing scenarios with code changes or facilitating any related discussions

[TylerHelmuth]

@jvoravong if you perform these test cases manually do they fail?

[jvoravong]

These failures occur with manual testing as well. From an initial glance, it looks like the "namespace" value isn't handled properly when looking for the Instrumentation instance in the failing cases.

[TylerHelmuth]

@jvoravong I tested both scenario 2 and 3 locally and got success. Can you provide more details on the environment/setup of your failed scenarios?

[jvoravong]

Hmmm. I actually can't reproduce this issue in recent versions of the Operator. Original I discovered this issue 6 months ago and it was reproducible. Closing this issue as fixed. Thanks @TylerHelmuth.