Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add AWS bedrock embeddings to embedding encoder #6406

Merged
merged 29 commits into from
Dec 17, 2023
Merged

feat: Add AWS bedrock embeddings to embedding encoder #6406

merged 29 commits into from
Dec 17, 2023

Conversation

jlonge4
Copy link
Contributor

@jlonge4 jlonge4 commented Nov 23, 2023

Proposed Changes:

Add AWS Bedrock embedding retriever to support embedding queries/documents with titan embeddings model.

How did you test it?

manual verification

Notes for the reviewer

haystack/nodes/retriever/embedding_encoder
haystack/nodes/retriever/dense

Checklist

@jlonge4 jlonge4 requested a review from a team as a code owner November 23, 2023 22:49
@jlonge4 jlonge4 requested review from anakin87 and removed request for a team November 23, 2023 22:49
@jlonge4 jlonge4 requested a review from a team as a code owner November 24, 2023 04:11
@jlonge4 jlonge4 requested review from dfokina and removed request for a team November 24, 2023 04:11
@coveralls
Copy link
Collaborator

coveralls commented Nov 24, 2023

Pull Request Test Coverage Report for Build 6977754646

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 146 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.02%) to 40.094%

Files with Coverage Reduction New Missed Lines %
nodes/retriever/dense.py 54 28.01%
nodes/retriever/_embedding_encoder.py 92 35.41%
Totals Coverage Status
Change from base Build 6973422803: -0.02%
Covered Lines: 10699
Relevant Lines: 26685

💛 - Coveralls

@github-actions github-actions bot added the type:documentation Improvements on the docs label Nov 24, 2023
@anakin87
Copy link
Member

Hey @jlonge4, we are very busy with the upcoming 2.0 release these days.

So probably the review process may be a bit long... Please bear with us.

@anakin87 anakin87 requested a review from tstadel November 24, 2023 09:42
@masci masci changed the base branch from main to v1.x November 24, 2023 12:08
@jlonge4
Copy link
Contributor Author

jlonge4 commented Nov 24, 2023

@anakin87 No worries! I totally understand

@masci masci added the 1.x label Nov 24, 2023
@jlonge4 jlonge4 changed the base branch from v1.x to 1.14 November 25, 2023 22:14
@jlonge4 jlonge4 changed the base branch from 1.14 to v1.x November 25, 2023 22:14
@jlonge4
Copy link
Contributor Author

jlonge4 commented Dec 7, 2023

@tstadel no worries I appreciate the feedback! Everything should be implemented, however some weird failures related to files I hadn't even touched that I didn't get last time.

@jlonge4
Copy link
Contributor Author

jlonge4 commented Dec 7, 2023

@anakin87 was that a merge in the wrong direction? 😅

@jlonge4 jlonge4 requested a review from tstadel December 15, 2023 01:43
@anakin87
Copy link
Member

I fixed mypy and pylint errors.

In general, it looks good, but I haven't tried these changes locally....

@tstadel can you take another look?

@tstadel
Copy link
Member

tstadel commented Dec 17, 2023

@anakin87 @jlonge4 I did some final refactorings and fixed two things:

Also for cohere we use batch mode now. I made some final tests locally to ensure doc and query embeddings are working with titan and cohere models.

So from my side we can merge now :-)

@tstadel
Copy link
Member

tstadel commented Dec 17, 2023

@anakin87 wait I've found another thing to improve:

  • we should throw AWSConfigurationError instead of ValueError if an error occurs during boto3 session init. I'll quickly fix that.

@tstadel
Copy link
Member

tstadel commented Dec 17, 2023

@anakin87 ok now we're good to go.

Copy link
Member

@anakin87 anakin87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jlonge4 @tstadel
Thanks for your efforts!

@anakin87 anakin87 merged commit c812250 into deepset-ai:v1.x Dec 17, 2023
53 checks passed
@jlonge4
Copy link
Contributor Author

jlonge4 commented Dec 17, 2023

@tstadel @anakin87 thank you both so much for the help and patience!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants