Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix custom pricing - separate provider info from model info #7990

Merged
merged 8 commits into from
Jan 26, 2025
Merged

Fix custom pricing - separate provider info from model info #7990

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5a4c304
fix(utils.py): initial commit fixing custom cost tracking
krrishdholakia Jan 25, 2025
aebdb58
fix(utils.py): cleanup `_supports_factory` to check provider info, if…
krrishdholakia Jan 25, 2025
dbedffe
fix(utils.py): refactor to use _supports_factory
krrishdholakia Jan 25, 2025
d751d84
test: update testing
krrishdholakia Jan 25, 2025
2b44ec0
build: Merge branch 'main' into litellm_dev_01_24_2025_p3
krrishdholakia Jan 25, 2025
22f8e62
fix: fix linting errors
krrishdholakia Jan 25, 2025
d939345
test: fix testing
krrishdholakia Jan 25, 2025
dfcc6c9
build: Merge branch 'main' into litellm_dev_01_24_2025_p3
krrishdholakia Jan 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions litellm/cost_calculator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,10 @@
base_model=base_model,
)

verbose_logger.debug(
f"completion_response _select_model_name_for_cost_calc: {model}"

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High

This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expressi

Copilot Autofix AI 20 days ago

To fix the problem, we should avoid logging sensitive information directly. Instead, we can log a sanitized version of the model variable or avoid logging it altogether if it contains sensitive data. The best way to fix this without changing existing functionality is to sanitize the model variable before logging it. We can create a helper function to sanitize the model variable by removing or masking any sensitive information.

Suggested changeset 1
litellm/cost_calculator.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/litellm/cost_calculator.py b/litellm/cost_calculator.py
--- a/litellm/cost_calculator.py
+++ b/litellm/cost_calculator.py
@@ -72,2 +72,8 @@
 
+def sanitize_model(model: Optional[str]) -> Optional[str]:
+    if model is None:
+        return None
+    # Add logic to sanitize the model string, e.g., mask sensitive parts
+    # For simplicity, let's mask any part that looks like an API key
+    return model.replace("os.environ/", "****")
 
@@ -562,4 +568,5 @@
 
+        sanitized_model = sanitize_model(model)
         verbose_logger.debug(
-            f"completion_response _select_model_name_for_cost_calc: {model}"
+            f"completion_response _select_model_name_for_cost_calc: {sanitized_model}"
         )
EOF
@@ -72,2 +72,8 @@

def sanitize_model(model: Optional[str]) -> Optional[str]:
if model is None:
return None
# Add logic to sanitize the model string, e.g., mask sensitive parts
# For simplicity, let's mask any part that looks like an API key
return model.replace("os.environ/", "****")

@@ -562,4 +568,5 @@

sanitized_model = sanitize_model(model)
verbose_logger.debug(
f"completion_response _select_model_name_for_cost_calc: {model}"
f"completion_response _select_model_name_for_cost_calc: {sanitized_model}"
)
Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
)

if completion_response is not None and (
isinstance(completion_response, BaseModel)
or isinstance(completion_response, dict)
Expand Down Expand Up @@ -598,9 +602,6 @@
cache_read_input_tokens = prompt_tokens_details.get("cached_tokens", 0)

total_time = getattr(completion_response, "_response_ms", 0)
verbose_logger.debug(
f"completion_response response ms: {getattr(completion_response, '_response_ms', None)} "
)

hidden_params = getattr(completion_response, "_hidden_params", None)
if hidden_params is not None:
Expand Down
1 change: 1 addition & 0 deletions litellm/litellm_core_utils/litellm_logging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -847,6 +847,7 @@ def _response_cost_calculator(
response_cost = litellm.response_cost_calculator(
**response_cost_calculator_kwargs
)
verbose_logger.debug(f"response_cost: {response_cost}")
return response_cost
except Exception as e: # error calculating cost
debug_info = StandardLoggingModelCostFailureDebugInformation(
Expand Down
10 changes: 4 additions & 6 deletions litellm/llms/base_llm/base_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,15 @@
from openai.lib import _parsing, _pydantic
from pydantic import BaseModel

from litellm.types.utils import ModelInfoBase
from litellm.types.utils import ProviderSpecificModelInfo


class BaseLLMModelInfo(ABC):
@abstractmethod
def get_model_info(
def get_provider_info(
self,
model: str,
existing_model_info: Optional[ModelInfoBase] = None,
) -> Optional[ModelInfoBase]:
pass
) -> Optional[ProviderSpecificModelInfo]:
return None

@abstractmethod
def get_models(self) -> List[str]:
Expand Down
22 changes: 3 additions & 19 deletions litellm/llms/fireworks_ai/chat/transformation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import litellm
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues, ChatCompletionImageObject
from litellm.types.utils import ModelInfoBase, ProviderSpecificModelInfo
from litellm.types.utils import ProviderSpecificModelInfo

from ...openai.chat.gpt_transformation import OpenAIGPTConfig

Expand Down Expand Up @@ -159,30 +159,14 @@ def _transform_messages_helper(
)
return messages

def get_model_info(
self, model: str, existing_model_info: Optional[ModelInfoBase] = None
) -> ModelInfoBase:
def get_provider_info(self, model: str) -> ProviderSpecificModelInfo:
provider_specific_model_info = ProviderSpecificModelInfo(
supports_function_calling=True,
supports_prompt_caching=True, # https://docs.fireworks.ai/guides/prompt-caching
supports_pdf_input=True, # via document inlining
supports_vision=True, # via document inlining
)
if existing_model_info is not None:
return ModelInfoBase(
**{**existing_model_info, **provider_specific_model_info}
)
return ModelInfoBase(
key=model,
litellm_provider="fireworks_ai",
mode="chat",
input_cost_per_token=0.0,
output_cost_per_token=0.0,
max_tokens=None,
max_input_tokens=None,
max_output_tokens=None,
**provider_specific_model_info,
)
return provider_specific_model_info

def transform_request(
self,
Expand Down
19 changes: 1 addition & 18 deletions litellm/llms/openai/chat/gpt_transformation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from litellm.llms.base_llm.chat.transformation import BaseConfig, BaseLLMException
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelInfoBase, ModelResponse
from litellm.types.utils import ModelResponse

from ..common_utils import OpenAIError

Expand Down Expand Up @@ -255,23 +255,6 @@ def get_models(
models = response.json()["data"]
return [model["id"] for model in models]

def get_model_info(
self, model: str, existing_model_info: Optional[ModelInfoBase] = None
) -> ModelInfoBase:

if existing_model_info is not None:
return existing_model_info
return ModelInfoBase(
key=model,
litellm_provider="openai",
mode="chat",
input_cost_per_token=0.0,
output_cost_per_token=0.0,
max_tokens=None,
max_input_tokens=None,
max_output_tokens=None,
)

@staticmethod
def get_api_key(api_key: Optional[str] = None) -> Optional[str]:
return (
Expand Down
6 changes: 0 additions & 6 deletions litellm/llms/topaz/common_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
from typing import List, Optional

from litellm.secret_managers.main import get_secret_str
from litellm.types.utils import ModelInfoBase

from ..base_llm.base_utils import BaseLLMModelInfo
from ..base_llm.chat.transformation import BaseLLMException
Expand All @@ -12,11 +11,6 @@ class TopazException(BaseLLMException):


class TopazModelInfo(BaseLLMModelInfo):
def get_model_info(
self, model: str, existing_model_info: Optional[ModelInfoBase] = None
) -> Optional[ModelInfoBase]:
return existing_model_info

def get_models(self) -> List[str]:
return [
"topaz/Standard V2",
Expand Down
1 change: 0 additions & 1 deletion litellm/proxy/_experimental/out/404.html
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion litellm/proxy/_experimental/out/model_hub.html
View file
Open in desktop

This file was deleted.

Loading