You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Seeing an error in async callback deading with prometheus label count. This does not cause any completion requests to fail, but errors like this make me worried that prometheus metrics are not accurate representation of system health and performance. This is a regression as in v1.54.1 this error was not present.
Relevant log output
Task exception was never retrieved
future: <Task finished name='Task-769554' coro=<ServiceLogging.async_service_failure_hook() done, defined at /usr/local/lib/python3.13/site-packages/litellm/_service_logger.py:207> exception=ValueError('Incorrect label count')>
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/litellm/_service_logger.py", line 243, in async_service_failure_hook
await self.prometheusServicesLogger.async_service_failure_hook(
...<2 lines>...
)
File "/usr/local/lib/python3.13/site-packages/litellm/integrations/prometheus_services.py", line 207, in async_service_failure_hook
self.increment_counter(
~~~~~~~~~~~~~~~~~~~~~~^
counter=obj,
^^^^^^^^^^^^
...<3 lines>...
amount=1, # LOG ERROR COUNT TO PROMETHEUS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/litellm/integrations/prometheus_services.py", line 131, in increment_counter
counter.labels(labels, *additional_labels).inc(amount)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/prometheus_client/metrics.py", line 199, in labels
raise ValueError('Incorrect label count')
ValueError: Incorrect label count
### Are you a ML Ops Team?
Yes
### What LiteLLM version are you on ?
v1.56.5
### Twitter / LinkedIn details
_No response_
The text was updated successfully, but these errors were encountered:
* fix(main.py): pass custom llm provider on litellm logging provider update
* fix(cost_calculator.py): don't append provider name to return model if existing llm provider
FixesBerriAI#7607
* fix(prometheus_services.py): fix prometheus system health error logging
FixesBerriAI#7611
What happened?
Seeing an error in async callback deading with prometheus label count. This does not cause any completion requests to fail, but errors like this make me worried that prometheus metrics are not accurate representation of system health and performance. This is a regression as in v1.54.1 this error was not present.
Relevant log output
The text was updated successfully, but these errors were encountered: