feat: add Amazon Bedrock support #6226

viveksilimkhan1 · 2023-11-03T07:28:47Z

Related Issues

fixes Add Amazon Bedrock support #6225

Proposed Changes:

Added support for Amazon Bedrock Models

Added AmazonBedrockInvocationLayer
All currently available models on Bedrock are supported using model adapters
AWS common functionality with Sagemaker was refactored into AWSBaseInvocationLayer
streaming is supported for all models that support streaming

How did you test it?

added lots of tests
tested manually all models except Amazon Titan as they are currently unavailable

Notes for the reviewer

Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
I documented my code
I ran pre-commit hooks and fixed any issue

CLAassistant · 2023-11-03T07:28:53Z

All committers have signed the CLA.

tstadel · 2023-11-06T10:44:16Z

Thanks @viveksilimkhan1 for this awesome contribution. Could you please sign the CLA?

As we want to get this as fast as possible into haystack, I will support you in writing tests and pushing it over the finish line. Hope that's ok.

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py

tstadel

I left some comments. Functionality-wise we have to:

add anthropic claude support
support streaming
request supported models from bedrock
use DefaultPromptHandler to implement _ensure_token_limit

Quality-wise we have to:

implement tests
improve the params validation

tstadel · 2023-11-06T12:19:32Z

I left some comments. Functionality-wise we have to:

add anthropic claude support

support streaming

request supported models from bedrock

Quality-wise we have to:

implement tests

improve the params validation

@viveksilimkhan1 please let me know if you find time to tackle one or more of those points in the next two days. Otherwise I'm happy to jump on it. :-)

haystack/errors.py

tstadel · 2023-11-07T15:25:13Z

@viveksilimkhan1 I did some refactorings and improved the supports method a bit. Even though we are checking against the available models on Bedrock, we should check against a static list for which we provide correct handling of inputs and outputs.

I will continue writing tests for instantiation and supports. If you like, you can continue improving and completing the invoke method, e.g.:

refactor common expressions
support all params for existing implementations
support claude models

Also make sure to install the precommit hooks running pre-commit install. Thx! 👍

tstadel · 2023-11-14T16:47:48Z

@anakin87 this PR is good to go from my side. Feel free to have a look.

tstadel · 2023-11-14T16:49:12Z

.github/workflows/linting_preview.yml

-            **/*.py
-          files_ignore: |
-            test/**
-            rest_api/test/**
+            haystack/preview/**/*.py


Misconfig tried to lint my changes and failed because of missing dependencies that are not available to in preview.

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py

haystack/nodes/prompt/invocation_layer/aws_base.py

releasenotes/notes/bedrock-support-bce28e3078c85c12.yaml

anakin87 · 2023-11-15T10:35:25Z

@tstadel I had a quick look and it looks good to me...

If you haven't already done so, I would ask you to play around with these changes locally a bit.

In particular, I would check if the PromptNode still detects the appropriate invocation layers
(for example, you can try a local HF model and and another one hosted on SageMaker).
With respect to this PR, there should be no problem, but sometimes seemingly unrelated changes have broken this (fragile) mechanism.

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py

haystack/nodes/prompt/invocation_layer/aws_base.py

releasenotes/notes/bedrock-support-bce28e3078c85c12.yaml

Co-authored-by: Agnieszka Marzec <[email protected]>

tstadel · 2023-11-15T11:58:35Z

@tstadel I had a quick look and it looks good to me...

If you haven't already done so, I would ask you to play around with these changes locally a bit.

In particular, I would check if the PromptNode still detects the appropriate invocation layers (for example, you can try a local HF model and and another one hosted on SageMaker). With respect to this PR, there should be no problem, but sometimes seemingly unrelated changes have broken this (fragile) mechanism.

So I've tested the following manually:

test streaming for all supported Bedrock models (found a bug for cohere and fixed it)
load and prompt a local HF model, that is PromptNode("google/flan-t5-base")
load and prompt a Meta Sagemaker model (llama-2-7b-chat)
load and prompt a non-Meta Sagemaker model (mistral-7b-instruct)

I'll finish the last one in the meantime. @anakin87 would you have one final look?

anakin87

👍

tstadel · 2023-11-15T12:15:30Z

@viveksilimkhan1 thanks a lot for this contribution 🚀 ! I had to change a few things regarding streaming support, but now we're good to merge.

* Add Bedrock * Update supported models for Bedrock * Fix supports and add extract response in Bedrock * fix errors imports * improve and refactor supports * fix install * fix mypy * fix pylint * fix existing tests * Added Anthropic Bedrock * fix tests * fix sagemaker tests * add default prompt handler, constructor and supports tests * more tests * invoke refactoring * refactor model_kwargs * fix mypy * lstrip responses * Add streaming support * bump boto3 version * add class docstrings, better exception names * fix layer name * add tests for anthropic and cohere model adapters * update cohere params * update ai21 args and add tests * support cohere command light model * add tital tests * better class names * support meta llama 2 model * fix streaming support * more future-proof model adapter selection * fix import * fix mypy * fix pylint for preview * add tests for streaming * add release notes * Apply suggestions from code review Co-authored-by: Agnieszka Marzec <[email protected]> * fix format * fix tests after msg changes * fix streaming for cohere --------- Co-authored-by: tstadel <[email protected]> Co-authored-by: tstadel <[email protected]> Co-authored-by: Agnieszka Marzec <[email protected]>

Add Bedrock

7462525

viveksilimkhan1 requested a review from a team as a code owner November 3, 2023 07:28

viveksilimkhan1 requested review from anakin87 and removed request for a team November 3, 2023 07:28

github-actions bot added the type:documentation Improvements on the docs label Nov 3, 2023

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 6, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel requested changes Nov 6, 2023

View reviewed changes

viveksilimkhan1 added 2 commits November 7, 2023 11:57

Update supported models for Bedrock

e3efb9f

Fix supports and add extract response in Bedrock

a3f34ed

tstadel reviewed Nov 7, 2023

View reviewed changes

haystack/errors.py Outdated Show resolved Hide resolved

Merge branch 'main' into add-bedrock

5295bc9

tstadel changed the title ~~Add Amazon Bedrock support~~ feat: add Amazon Bedrock support Nov 7, 2023

tstadel added 2 commits November 7, 2023 11:30

fix errors imports

f8a910c

improve and refactor supports

1cd86c7

tstadel added 3 commits November 7, 2023 16:35

fix install

3a343a7

fix mypy

be0e211

fix pylint

5c57455

fix pylint for preview

ec6b995

github-actions bot added the topic:CI label Nov 14, 2023

tstadel added 2 commits November 14, 2023 17:33

add tests for streaming

2ec9f81

add release notes

d10b91d

tstadel requested a review from a team as a code owner November 14, 2023 16:46

tstadel reviewed Nov 14, 2023

View reviewed changes

agnieszka-m reviewed Nov 15, 2023

View reviewed changes

tstadel reviewed Nov 15, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 15, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 15, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/amazon_bedrock.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 15, 2023

View reviewed changes

haystack/nodes/prompt/invocation_layer/aws_base.py Outdated Show resolved Hide resolved

tstadel reviewed Nov 15, 2023

View reviewed changes

releasenotes/notes/bedrock-support-bce28e3078c85c12.yaml Outdated Show resolved Hide resolved

tstadel and others added 4 commits November 15, 2023 11:59

Apply suggestions from code review

c516d1d

Co-authored-by: Agnieszka Marzec <[email protected]>

fix format

4a30206

fix tests after msg changes

9846fd1

fix streaming for cohere

4d70ce5

anakin87 approved these changes Nov 15, 2023

View reviewed changes

Merge branch 'main' into add-bedrock

1997281

tstadel approved these changes Nov 15, 2023

View reviewed changes

tstadel added type:feature New feature or request topic:LLM labels Nov 15, 2023

tstadel merged commit f998bf4 into deepset-ai:main Nov 15, 2023
64 checks passed

anakin87 mentioned this pull request Nov 15, 2023

fix: change pydoc configuration for TextLanguageRouter #6313

Merged

julian-risch added the 1.x label Dec 18, 2023

julian-risch mentioned this pull request Dec 18, 2023

Add Amazon Bedrock Integration deepset-ai/haystack-core-integrations#113

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Amazon Bedrock support #6226

feat: add Amazon Bedrock support #6226

viveksilimkhan1 commented Nov 3, 2023 •

edited by tstadel

Loading

CLAassistant commented Nov 3, 2023 •

edited

Loading

tstadel commented Nov 6, 2023

tstadel left a comment •

edited

Loading

tstadel commented Nov 6, 2023 •

edited

Loading

tstadel commented Nov 7, 2023

tstadel commented Nov 14, 2023

tstadel Nov 14, 2023

anakin87 commented Nov 15, 2023

tstadel commented Nov 15, 2023 •

edited

Loading

anakin87 left a comment

tstadel commented Nov 15, 2023

feat: add Amazon Bedrock support #6226

feat: add Amazon Bedrock support #6226

Conversation

viveksilimkhan1 commented Nov 3, 2023 • edited by tstadel Loading

Related Issues

Proposed Changes:

How did you test it?

Notes for the reviewer

Checklist

CLAassistant commented Nov 3, 2023 • edited Loading

tstadel commented Nov 6, 2023

tstadel left a comment • edited Loading

Choose a reason for hiding this comment

tstadel commented Nov 6, 2023 • edited Loading

tstadel commented Nov 7, 2023

tstadel commented Nov 14, 2023

tstadel Nov 14, 2023

Choose a reason for hiding this comment

anakin87 commented Nov 15, 2023

tstadel commented Nov 15, 2023 • edited Loading

anakin87 left a comment

Choose a reason for hiding this comment

tstadel commented Nov 15, 2023

viveksilimkhan1 commented Nov 3, 2023 •

edited by tstadel

Loading

CLAassistant commented Nov 3, 2023 •

edited

Loading

tstadel left a comment •

edited

Loading

tstadel commented Nov 6, 2023 •

edited

Loading

tstadel commented Nov 15, 2023 •

edited

Loading