fix chroma update_document to embed entire documents, fixes a characer-wise embedding bug #5584
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
June 1, 2023 16:18
added 2 commits
June 1, 2023 16:34
dev2049
reviewed
Jun 1, 2023
|
thanks @cnellington! |
…gs and new content embeddings
Merged
Undertone0809
pushed a commit
to Undertone0809/langchain
that referenced
this pull request
Jun 19, 2023
…r-wise embedding bug (langchain-ai#5584) # Chroma update_document full document embeddings bugfix Chroma update_document takes a single document, but treats the page_content sting of that document as a list when getting the new document embedding. This is a two-fold problem, where the resulting embedding for the updated document is incorrect (it's only an embedding of the first character in the new page_content) and it calls the embedding function for every character in the new page_content string, using many tokens in the process. Fixes langchain-ai#5582 Co-authored-by: Caleb Ellington <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Chroma update_document full document embeddings bugfix
Chroma update_document takes a single document, but treats the page_content sting of that document as a list when getting the new document embedding.
This is a two-fold problem, where the resulting embedding for the updated document is incorrect (it's only an embedding of the first character in the new page_content) and it calls the embedding function for every character in the new page_content string, using many tokens in the process.
Fixes #5582
Before submitting
Who can review?
Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested:
Tagging @dev2049 for vectorstore bugfix