Skip to content

Commit

Permalink
Fix/bedrock issues (infiniflow#2718)
Browse files Browse the repository at this point in the history
### What problem does this PR solve?

Adding a Bedrock API key for Claude Sonnet was broken. I find the issue
came up when trying to test the LLM configuration, the system is a
required parameter in boto3.

As well, there were problems in Bedrock implementation for embeddings
when trying to encode queries.

### Type of change

- [X] Bug Fix (non-breaking change which fixes an issue)
  • Loading branch information
Omarlsg98 authored Oct 5, 2024
1 parent c718c41 commit 18be0df
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 3 deletions.
5 changes: 3 additions & 2 deletions rag/llm/chat_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -630,7 +630,7 @@ def chat(self, system, history, gen_conf):
modelId=self.model_name,
messages=history,
inferenceConfig=gen_conf,
system=[{"text": system}] if system else None,
system=[{"text": (system if system else "Answer the user's message.")}] ,
)

# Extract and print the response text.
Expand Down Expand Up @@ -675,7 +675,8 @@ def chat_streamly(self, system, history, gen_conf):
streaming_response = self.client.converse_stream(
modelId=self.model_name,
messages=history,
inferenceConfig=gen_conf
inferenceConfig=gen_conf,
system=[{"text": system if system else ""}],
)

# Extract and print the streamed response text in real-time.
Expand Down
2 changes: 1 addition & 1 deletion rag/llm/embedding_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -443,7 +443,7 @@ def encode_queries(self, text):

response = self.client.invoke_model(modelId=self.model_name, body=json.dumps(body))
model_response = json.loads(response["body"].read())
embeddings.extend([model_response["embedding"]])
embeddings.extend(model_response["embedding"])

return np.array(embeddings), token_count

Expand Down

0 comments on commit 18be0df

Please sign in to comment.