Skip to content

Latest commit

 

History

History
33 lines (24 loc) · 805 Bytes

CHANGELOG.md

File metadata and controls

33 lines (24 loc) · 805 Bytes

0.4.0

07 Jun 2024

Added

  • O200K encoding

0.3.0

14 Oct 2023

Focused on internal refactoring and API stabilization to enhance usability and facilitate future development.

Breaking Changes

  • Rename Enconding to EncodingConfig.
  • Rework encodings to enhance extensibility:
    • Add a new Encoding interface.
    • Introduce default implementations of Encoding: CL100KBase, P50KBase, R50KBase, and P50KEdit.
  • Rename Tokenizer.encoding and Tokenizer.encodingForModel to Tokenizer.of with overrides.

0.2.0

12 Oct 2023

Changed

  • JVM: use local PBE loader by default
  • Encoding: enable custom encoding

0.1.0

08 Oct 2023

Initial release.

Added

  • Encodings: CL100K_BASE , R50K_BASE, P50K_BASE and P50K_EDIT
  • Custom encoding support