Harrison/llamacpp #5402
Merged
Harrison/llamacpp #5402
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Adds support for counting tokens using the llama.cpp python interface rather than the default huggingface transformers library The current implementation of the `LlamaCpp` LLM defaults to the base `LLM` for token counting. This results in the need for the huggingface transformers library to be loaded. The Llama.cpp python interface provides a method for tokenizing a given string. This PR overloads the `get_num_tokens` method of the base class to use that instead. Using the native tokenizer should yield more accurate token counts dependent on the loaded model. For llama.cpp workflows this PR reduces dependencies. <!-- Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution. Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change. After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost. --> ## Before submitting Wasn't sure how to setup a test for this without spinning up a particular model. But I have tested it in a project. ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: @hwchase17 @agola11 <!-- For a quicker response, figure out the right person to tag with @ @hwchase17 - project lead Tracing / Callbacks - @agola11 Async - @agola11 DataLoaders - @eyurtsev Models - @hwchase17 - @agola11 Agents / Tools / Toolkits - @vowelparrot VectorStores / Retrievers / Memory - @dev2049 -->
hwchase17
added
the
lgtm
vowelparrot
pushed a commit
that referenced
this pull request
May 31, 2023
Co-authored-by: Gavin S <[email protected]>
Merged
Undertone0809
pushed a commit
to Undertone0809/langchain
that referenced
this pull request
Jun 19, 2023
Co-authored-by: Gavin S <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.