Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for using tiktoken tokenizers #610

Merged
merged 66 commits into from
Sep 21, 2023
Merged

Support for using tiktoken tokenizers #610

merged 66 commits into from
Sep 21, 2023

Commits on Sep 17, 2023

  1. unit tests pass

    @dakinggg
    dakinggg committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    2964228 View commit details
    Browse the repository at this point in the history

  2. precommit and add vocab test

    @dakinggg
    dakinggg committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    7904938 View commit details
    Browse the repository at this point in the history

  3. precommit

    @dakinggg
    dakinggg committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    29d0564 View commit details
    Browse the repository at this point in the history

  4. precommit

    @dakinggg
    dakinggg committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    8ba4283 View commit details
    Browse the repository at this point in the history

  5. fix auto stuff

    @dakinggg
    dakinggg committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    194bc59 View commit details
    Browse the repository at this point in the history

  6. precommit

    @dakinggg
    dakinggg committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    da37e8c View commit details
    Browse the repository at this point in the history

  7. add kwargs to dataset conversion

    @dakinggg
    dakinggg committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    246606d View commit details
    Browse the repository at this point in the history

  8. precommit

    @dakinggg
    dakinggg committed Sep 17, 2023
    Configuration menu
    Copy the full SHA
    f821306 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. fix non unicode stuff

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    e01fc57 View commit details
    Browse the repository at this point in the history

  2. attempt to ignore test file

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    067bbb4 View commit details
    Browse the repository at this point in the history

  3. get the ignore right

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    44dcf3e View commit details
    Browse the repository at this point in the history

  4. precommit

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    2acc466 View commit details
    Browse the repository at this point in the history

  5. precommit

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    371185d View commit details
    Browse the repository at this point in the history

  6. remove formatter ignore

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    27be411 View commit details
    Browse the repository at this point in the history

  7. still precommit

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    d00c25b View commit details
    Browse the repository at this point in the history

  8. license

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    d4bf5b7 View commit details
    Browse the repository at this point in the history

  9. more precommit

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    ff30ce8 View commit details
    Browse the repository at this point in the history

  10. pyrigth config change

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    b34eb8f View commit details
    Browse the repository at this point in the history

  11. more tests

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    6fd1874 View commit details
    Browse the repository at this point in the history

  12. add another test

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    c84fee4 View commit details
    Browse the repository at this point in the history

  13. precommit

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    01504eb View commit details
    Browse the repository at this point in the history

  14. fix test

    @dakinggg
    dakinggg committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    fc7a65d View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2023

  1. merge

    @dakinggg
    dakinggg committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    bcf49c3 View commit details
    Browse the repository at this point in the history

  2. rm old implementation

    @dakinggg
    dakinggg committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    2c0d4e6 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. merge with other wrapper

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    caa51d2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into tiktoken

    @dakinggg
    dakinggg authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    4b05efd View commit details
    Browse the repository at this point in the history

  3. docs

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    e6f385f View commit details
    Browse the repository at this point in the history

  4. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    0e5376a View commit details
    Browse the repository at this point in the history

  5. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    0fec0ff View commit details
    Browse the repository at this point in the history

  6. test the encoding path

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    de3bbbc View commit details
    Browse the repository at this point in the history

  7. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    13ce011 View commit details
    Browse the repository at this point in the history

  8. fix error

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    4ad6f87 View commit details
    Browse the repository at this point in the history

  9. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    a67689f View commit details
    Browse the repository at this point in the history

  10. more thorough tests and fix no split tokens setting

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    bf3ac1b View commit details
    Browse the repository at this point in the history

  11. Merge branch 'main' into tiktoken

    @dakinggg
    dakinggg authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    535202b View commit details
    Browse the repository at this point in the history

  12. fix tests

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    e802370 View commit details
    Browse the repository at this point in the history

  13. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    56fe3c1 View commit details
    Browse the repository at this point in the history

  14. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    2ee57b6 View commit details
    Browse the repository at this point in the history

  15. pyright

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    f5acf48 View commit details
    Browse the repository at this point in the history

  16. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    17002a5 View commit details
    Browse the repository at this point in the history

  17. remove unnecessary cast

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    1f48d62 View commit details
    Browse the repository at this point in the history

  18. Merge branch 'main' into tiktoken

    @dakinggg
    dakinggg authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    85e5740 View commit details
    Browse the repository at this point in the history

  19. fix type ignore

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    75c26de View commit details
    Browse the repository at this point in the history

  20. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    fdec6ab View commit details
    Browse the repository at this point in the history

  21. pr nit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    2893eaf View commit details
    Browse the repository at this point in the history

  22. add return types

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    3eb328d View commit details
    Browse the repository at this point in the history

  23. docstring typos

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    efed581 View commit details
    Browse the repository at this point in the history

  24. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    473dd0e View commit details
    Browse the repository at this point in the history

  25. pr comments

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    132b6e5 View commit details
    Browse the repository at this point in the history

  26. pr comments

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    32c7bd9 View commit details
    Browse the repository at this point in the history

  27. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    925355a View commit details
    Browse the repository at this point in the history

  28. use helper

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    40ec2c4 View commit details
    Browse the repository at this point in the history

  29. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    21c7953 View commit details
    Browse the repository at this point in the history

  30. fix import

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    53d0761 View commit details
    Browse the repository at this point in the history

  31. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    295d42b View commit details
    Browse the repository at this point in the history

  32. reprompt tests

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    a0c10c1 View commit details
    Browse the repository at this point in the history

  33. pr comments

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    7e4e878 View commit details
    Browse the repository at this point in the history

  34. fix test

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    a669130 View commit details
    Browse the repository at this point in the history

  35. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    58ecca4 View commit details
    Browse the repository at this point in the history

  36. Merge branch 'main' into tiktoken

    @dakinggg
    dakinggg authored Sep 20, 2023
    Configuration menu
    Copy the full SHA
    9e8a0d1 View commit details
    Browse the repository at this point in the history

  37. fix typo

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    1e01e11 View commit details
    Browse the repository at this point in the history

  38. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    b1a70b8 View commit details
    Browse the repository at this point in the history

  39. clean up tests

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    2a766fc View commit details
    Browse the repository at this point in the history

  40. precommit

    @dakinggg
    dakinggg committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    483b4f8 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2023

  1. Merge branch 'main' into tiktoken

    @dakinggg
    dakinggg authored Sep 21, 2023
    Configuration menu
    Copy the full SHA
    2b5fb5c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into tiktoken

    @dakinggg
    dakinggg authored Sep 21, 2023
    Configuration menu
    Copy the full SHA
    1c5a999 View commit details
    Browse the repository at this point in the history