Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Harrison/llamacpp #5402

Merged
merged 2 commits into from
May 29, 2023
Merged

Harrison/llamacpp #5402

merged 2 commits into from
May 29, 2023

Commits on May 29, 2023

  1. Add llama.cpp get_num_tokens support (#5352) 

    # Adds support for counting tokens using the llama.cpp python interface
rather than the default huggingface transformers library

The current implementation of the `LlamaCpp` LLM defaults to the base
`LLM` for token counting. This results in the need for the huggingface
transformers library to be loaded.

The Llama.cpp python interface provides a method for tokenizing a given
string. This PR overloads the `get_num_tokens` method of the base class
to use that instead.

Using the native tokenizer should yield more accurate token counts
dependent on the loaded model.

For llama.cpp workflows this PR reduces dependencies.

<!--
Thank you for contributing to LangChain! Your PR will appear in our
release under the title you set. Please make sure it highlights your
valuable contribution.

Replace this with a description of the change, the issue it fixes (if
applicable), and relevant context. List any dependencies required for
this change.

After you're done, someone will review your PR. They may suggest
improvements. If no one reviews your PR within a few days, feel free to
@-mention the same people again, as notifications can get lost.
-->

## Before submitting

Wasn't sure how to setup a test for this without spinning up a
particular model. But I have tested it in a project.

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@hwchase17
@agola11
<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @vowelparrot

  VectorStores / Retrievers / Memory
  - @dev2049
        
 -->
    @s7726
    s7726 authored May 29, 2023
    Configuration menu
    Copy the full SHA
    655007e View commit details
    Browse the repository at this point in the history

  2. cr

    @hwchase17
    hwchase17 committed May 29, 2023
    Configuration menu
    Copy the full SHA
    aeae8fe View commit details
    Browse the repository at this point in the history