[Bug]: Cannot pass provider-specific parameters to Bedrock Anthropic models

[mrm1001]

What happened?

Passing the top_k parameter to an Anthropic model in the Bedrock platform throws this error:
BadRequestError: litellm.BadRequestError: BedrockException - {"message":"The model returned the following errors: Malformed input request: #: extraneous key [inferenceConfig] is not permitted, please reformat your input and try again."}

I think the issue is with the Bedrock Converse API, sometimes the additional parameters are passed in this way (from the API docs):

additionalModelRequestFields = {
    "inferenceConfig": {
         "topK": 20
    }
}

model_response = client.converse(
    modelId="us.amazon.nova-lite-v1:0", 
    messages=messages, 
    system=system, 
    inferenceConfig=inf_params,
    additionalModelRequestFields=additionalModelRequestFields
)

Sometimes it's passed without the inferenceConfig key, such as this example in the API docs:

additional_model_fields = {"top_k": top_k}

    # Send the message.
    response = bedrock_client.converse(
        modelId=model_id,
        messages=messages,
        system=system_prompts,
        inferenceConfig=inference_config,
        additionalModelRequestFields=additional_model_fields
    )

I managed to fix the error for Anthropic by changing the litellm code here to this:

      if "topK" in inference_params:
            additional_request_params = {
                "top_k": inference_params.pop("topK")
            }
        elif "top_k" in inference_params:
            additional_request_params = {
                "top_k": inference_params.pop("top_k")
            }

However if you do that, then it stops working for the Nova model. So I'm not sure what the solution is here!

Relevant log output

No response

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

1.58.2

Twitter / LinkedIn details

No response

[krrishdholakia]

However if you do that, then it stops working for the Nova model. So I'm not sure what the solution is here!

Thank you for this ticket @mrm1001 - what's the error thrown by nova? @mrm1001

[ishaan-jaff]

fixed in #8131 by @vibhavbhat