Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with Python AutoInstrumentation #682

Closed
henrikrexed opened this issue Jan 28, 2022 · 8 comments · Fixed by #684
Closed

Issues with Python AutoInstrumentation #682

henrikrexed opened this issue Jan 28, 2022 · 8 comments · Fixed by #684
Labels
area:auto-instrumentation Issues for auto-instrumentation

Comments

@henrikrexed
Copy link

henrikrexed commented Jan 28, 2022

Hi,

I'm currently building content to promote the Operator.
Side of the content I'm building a tutorial showing how the operator injec the auto instrumentation library.
For the moment the nodejs, java services are very well instrumented , but I'm getting errors with my python services.

Here is the log generated on the container :

Failed to load configured propagators
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration
Instrumenting of requests failed
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/distro.py", line 62, in load_instrumentor
    instrumentor: BaseInstrumentor = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2465, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2471, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/__init__.py", line 67, in <module>
    from opentelemetry.propagate import inject
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 117, in initialize
    _load_instrumentors(distro)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 88, in _load_instrumentors
    raise exc
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/distro.py", line 62, in load_instrumentor
    instrumentor: BaseInstrumentor = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2465, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2471, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/requests/__init__.py", line 67, in <module>
    from opentelemetry.propagate import inject
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration
{"timestamp": 1643390481.8418608, "severity": "INFO", "name": "emailservice-server", "message": "starting the email service in dummy mode."}
Failed to load configured propagators
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration
Traceback (most recent call last):
  File "email_server.py", line 144, in <module>
    start(dummy_mode = True)
  File "email_server.py", line 119, in start
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/grpc/__init__.py", line 166, in server
    server_interceptor(tracer_provider=tracer_provider)
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/grpc/__init__.py", line 252, in server_interceptor
    from . import _server
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/grpc/_server.py", line 31, in <module>
    from opentelemetry.propagate import extract
  File "/otel-auto-instrumentation/opentelemetry/propagate/__init__.py", line 137, in <module>
    iter_entry_points("opentelemetry_propagator", propagator)
StopIteration

How can i resolve this?

@jpkrohling
Copy link
Member

cc @anuraaga, @pavolloffay

@pavolloffay
Copy link
Member

@henrikrexed were you able to resolve this? If yes could you please share what was the problem and if no would it be possible to share a docker image with the reproducer?

I am not working on the python ecosystem but @anuraaga could perhaps help here.

@henrikrexed
Copy link
Author

Hi,
No, I didn't find a solution to my issue.

here is the repo related to the episode :
https://github.com/isItObservable/OpenTelemetryOperator
The tutorial is not finalized, so don't pay attention to the readme.md.

The image generating the issue are :
hrexed/recommendationservice-otel:0.1
and hrexed/emailservice-otel:0.1
Here is the link to the deployment file :
https://github.com/isItObservable/OpenTelemetryOperator/tree/master/hipster-shop-nootel

Best regards,

@pavolloffay
Copy link
Member

Are the images just upstream hipstershop source code?

@henrikrexed
Copy link
Author

No i have created a specific version with for a dedicated content explaining how to instrument your code using openTelemetry.
For the content on the operator I have created a "no otel" version that excludes all the opentelemetry , opensensus, opentracing libraries.

Here is the link to the repo :
https://github.com/isItObservable/OpenTelemetry-Instrumentation/tree/mixotel_nonotel/src

@pavolloffay pavolloffay added the area:auto-instrumentation Issues for auto-instrumentation label Jan 31, 2022
@mat-rumian
Copy link
Contributor

Hey @henrikrexed ,
I'm in the middle of testing of OT-Operator capabilities. I think you've faced an issue with default instrumentation yamls :) I mean they are not accurate. Probably you have configured in your Instrumentation CR propagators which are not supported by Python instrumentation. What I found is OT Operator Python instrumentation supports only tracecontext and baggage. All other propagator packages are not installed. If you cleanup your propagators list then your issue should be solved. List of supported instrumentations and propagators is here: https://github.com/open-telemetry/opentelemetry-operator/blob/v0.42.0/autoinstrumentation/python/requirements.txt

@anuraaga
Copy link
Contributor

Thanks @mat-rumian! I think it makes sense to add the other propagators to the instrumentation image too.

@henrikrexed
Copy link
Author

Hi, I made the changes on my various deployment files and it works perfectly.
Thanks for your support.
If we need to define a specific propagator on a specific language, does it make sense to specify specific propagators or resources directly at the nodejs, java, or python level ( in the instrumentation CRD)? it would avoid defining several instrumentation objects in the namespace. . what do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:auto-instrumentation Issues for auto-instrumentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants