Fix passing top_k parameter for Bedrock Anthropic models #8131
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| for k, v in inference_params.items(): | ||
| if ( | ||
| k not in supported_converse_params | ||
| and k not in supported_tool_call_params |
There was a problem hiding this comment.
is this deletion required ? I'm pretty sure we need to filter our specific guardrail params
There was a problem hiding this comment.
I just wanted to simplify the code a bit (using list comprehension instead of adding and then popping from the dict). From my understanding the before and after logic should be functionally equivalent, but if I'm wrong, I can revert it
| "bedrock/mistral.mistral-7b-instruct-v0:2", | ||
| ] | ||
| ) | ||
| def test_bedrock_top_k(model): |
There was a problem hiding this comment.
can you send a screenshot of this test working for all the API calls here ?
There was a problem hiding this comment.
Yeah I have it added it in the description, let me know if you need more screenshots
| top_k=2, | ||
| client=client | ||
| ) | ||
| except Exception as e: |
There was a problem hiding this comment.
why do you catch exceptions in the test and just print it?
There was a problem hiding this comment.
it's a mock call - the test is checking for the input. the mock call isn't returning a response, so for the mock test the translation output error is expected
|
Is this okay to merge? @ishaan-jaff |
Title
Fix passing top_k parameter for Bedrock Anthropic models
Relevant issues
Fixes #7782
Type
🐛 Bug Fix
✅ Test
Changes
The bug arose from the fact that different bedrock models pass in the top_k parameter in different ways.
Specifically the nova model passes in the parameter through
and the anthropic model passes it in through
This PR checks the model types and sets the parameter based on that. Right now, this is a simple if statement, but a long term fix might be to create a new class for each model, and create handling logic for supported / model-specific param within each class. This way the overall converse handler does not need to know about these specifics.
This PR also simplifies the additional_model_fields creation logic
[REQUIRED] Testing - Attach a screenshot of any new tests passing locally
If UI changes, send a screenshot/GIF of working UI fixes
Tested the top_k param for 4 different models w/ real API calls:
Tests still passed w/ mocks:
