-
Notifications
You must be signed in to change notification settings - Fork 651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
automatic metrics for gRPC client interceptor #917
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! This is looking pretty good, just a couple of questions in the comments and waiting for an update of the changelog before approving.
@@ -21,7 +21,7 @@ | |||
from opentelemetry.ext.grpc.version import __version__ | |||
|
|||
|
|||
def client_interceptor(tracer_provider=None): | |||
def client_interceptor(tracer_provider=None, meter=None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
curious what the reason for passing in a meter and not a meter_provider here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we pass a meter_provider, we would need to create/get an exporter and a controller as well. Passing meter is the only way (as far as i can tell) to get everything needed for metrics in 1 argument
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm ok with either, not sure if @lzchen has a strong preference one way or another here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the purpose of this just to collect the related metrics? Are we leaving it up to the user to choose what to do with it, or is there only one exporter in which you want to use? If it is the former, then using a meter instance would work.
self._tracer = tracer | ||
self._meter = meter | ||
self._metrics_recorder = TimedMetricRecorder(self._meter, "client") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what happens here if the meter passed in is None? Is the TimedMetricRecorder treated as a no-op?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes
) | ||
raise | ||
|
||
ret = self._trace_result(guarded_span, rpc_info, result) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any reason for this return variable? it doesn't appear to be used anywhere other than in the return
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_trace_result
assigned rpc_info.response
, which is why I did it this way, but it does look pretty bad. Moved the byte count to inside _trace_result
and got rid of this variable
d8d2b10
to
105ec0d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -21,7 +21,7 @@ | |||
from opentelemetry.ext.grpc.version import __version__ | |||
|
|||
|
|||
def client_interceptor(tracer_provider=None): | |||
def client_interceptor(tracer_provider=None, meter=None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is client_interceptor
actually used? Is this instantiated by the user? How does this relate to [#788]?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
interceptor = client_interceptor(meter=meter)
self.channel = intercept_channel(
grpc.insecure_channel("localhost:25565"), interceptor
)
I will add a meter parameter to _instrument
if #788 gets merged first (although.. passing a meter makes uninstrumentation difficult. maybe exporter + interval as params is actually a better choice in that case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
An exporter + interval would make more. We can instantiate a specific meter for grpc
metrics under the hood (unique name) and start the pipeline implicitly if an exporter is passed in.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
switched to using exporter + interval which will do for now, but I still think there needs to be a proper solution to this that doesn't require passing two things to every single instrumentation
@@ -21,7 +21,7 @@ | |||
from opentelemetry.ext.grpc.version import __version__ | |||
|
|||
|
|||
def client_interceptor(tracer_provider=None): | |||
def client_interceptor(tracer_provider=None, meter=None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How will the metrics be exposed if the instrumentor is auto-instrumented? There's no way to pass in/access a meter if the instrumentor is instantiated through an entry point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think there is a good way to have the metrics be created on auto instrumentation, since there is no global meterprovider + interval + exporter we can attach to. The meter has to be passed in somehow, and if auto-instrumentation can't pass anything in, then we can't add metrics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. In the above suggestion for export + interval, this only works on explicit instantiation. Seems like we can either expose an API for the user to manually indicate to start exporting metrics, or we can use a default exporter + export interval. Both methods are kind of not great IMO, it either requires the user to know about the implementation/existence of metrics in the first place (and they have to manually call start_pipeline) or it forces them to use an exporter/export interval that is not configurable. Might be something to bring up during the SIG.
964aff1
to
20e0a2b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this PR need to be refactored since [https://github.com//pull/788] was merged?
bc85b1f
to
17633bb
Compare
yes, updated it |
bc375f3
to
bdb5225
Compare
self._tracer = tracer | ||
self._meter = metrics.get_meter(__name__) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the user is not using the default SDK, DefaultMeter
will be used (or some other Meter implementation) which might not have the collect
method defined, which will cause this to crash.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure what you mean here? I took this part from ext/system_metrics, is that wrong too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure about ext/system_metrics, but if you are not explicitly using the meter implementation in opentelemetry-sdk
, then this would crash, since every tick()
in PushController
calls meter.collect() which only exists in the sdk implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also on that note, opentelemetry-sdk
would need to be a dependency now correct in setup,cfg
correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I should create the meterprovider from opentelemetry-sdk explicitly here even though that means the user can't set it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you can simply just take a dependency on the sdk and include in the examples that you must set meterprovider. Also, since users can autoinstrument the grpc instrumentor, you must not have this logic execute (since meterprovider won't be set).
ext/opentelemetry-ext-grpc/src/opentelemetry/ext/grpc/_client.py
Outdated
Show resolved
Hide resolved
ext/opentelemetry-ext-grpc/src/opentelemetry/ext/grpc/_client.py
Outdated
Show resolved
Hide resolved
c56f5db
to
bc5363b
Compare
c140121
to
7278b51
Compare
part 2/2 of #878