Code

Work on the BPE tokenizer#3252

Code

Merged

goerch merged 35 commits intoggml-org:masterggml-org/llama.cpp:masterfrom goerch:falcon-tokenizergoerch/llama.cpp:falcon-tokenizerCopy head branch name to clipboardOct 3, 2023

+852-227

Commits

Commits on Sep 18, 2023

• Work on the BPE tokenizer

  Show description for bfaab6f
  goerch

  committed
  bfaab6f

  Copy full SHA for bfaab6f

• Try to fix build problem

  goerch

  committed
  89e74c6

  Copy full SHA for 89e74c6

• Fix debug assertion failure

  goerch

  committed
  7770423

  Copy full SHA for 7770423

• Fix MSVC Unicode BOM problem

  goerch

  committed
  37cf135

  Copy full SHA for 37cf135

• Cleanup and an improvement

  goerch

  committed
  91a527a

  Copy full SHA for 91a527a

• Fix compiler warning

  goerch

  committed
  208d3d7

  Copy full SHA for 208d3d7

• Cleanup

  goerch

  committed
  407f76d

  Copy full SHA for 407f76d

• Test doesn't work over the full range of Unicodes

  goerch

  committed
  311fcf1

  Copy full SHA for 311fcf1

• Update .gitignore and Makefile

  goerch

  committed
  c85cb29

  Copy full SHA for c85cb29

• Another Makefile rule

  goerch

  committed
  048e659

  Copy full SHA for 048e659

Commits on Sep 19, 2023

• Testing Aquila

  goerch

  committed
  c0990bb

  Copy full SHA for c0990bb

• Moving byte decoding back to token_to_piece ...

  Show description for 1b7c369
  goerch

  committed
  1b7c369

  Copy full SHA for 1b7c369

• Guarding some unusable code pathes

  goerch

  committed
  a4e9448

  Copy full SHA for a4e9448

• Streamlining code and adding some more assertions

  Show description for 17ca832
  goerch

  committed
  17ca832

  Copy full SHA for 17ca832

• Adding a comment

  goerch

  committed
  4abbfb5

  Copy full SHA for 4abbfb5

• Adding another assertion

  goerch

  committed
  59a30b7

  Copy full SHA for 59a30b7

• Fixed vocabulary guarding assertions

  goerch

  committed
  a6070b7

  Copy full SHA for a6070b7

Commits on Sep 29, 2023

• Merge branch 'master' into falcon-tokenizer

  goerch

  authored
  16c06fe

  Copy full SHA for 16c06fe

• Fix PR for recent change

  goerch

  authored
  c09330e

  Copy full SHA for c09330e

• Fix PR for recent change

  goerch

  authored
  9cfb714

  Copy full SHA for 9cfb714

• Fix for compiler warning

  goerch

  authored
  607e3bf

  Copy full SHA for 607e3bf

• Fix PR for recent change

  goerch

  authored
  fad8a77

  Copy full SHA for fad8a77

• Fix PR for recent change

  goerch

  authored
  6a16c36

  Copy full SHA for 6a16c36

• Fix PR for recent change

  goerch

  authored
  a2ddaad

  Copy full SHA for a2ddaad

• Fix for compiler warning

  goerch

  authored
  3fa8c55

  Copy full SHA for 3fa8c55

• Fixes for more compiler warnings

  goerch

  authored
  d6d7d0f

  Copy full SHA for d6d7d0f

Commits on Oct 1, 2023

• Remove unused code

  goerch

  authored
  37af613

  Copy full SHA for 37af613

• Fix initialization of static maps

  goerch

  authored
  2117e23

  Copy full SHA for 2117e23

Commits on Oct 2, 2023

• Add scores and token types back, adapt gptneox

  goerch

  authored
  28778f8

  Copy full SHA for 28778f8

• Update llama.cpp

  Show description for a9a2af9
  

  goerch

  and

  ggerganov

  authored
  a9a2af9

  Copy full SHA for a9a2af9

• Update unicode.h

  Show description for dccd1db
  

  goerch

  and

  ggerganov

  authored
  dccd1db

  Copy full SHA for dccd1db

• Update unicode.h

  Show description for 02b9ccf
  

  goerch

  and

  ggerganov

  authored
  02b9ccf

  Copy full SHA for 02b9ccf

• Ported Starcoder and added some assertions

  goerch

  authored
  3d162cc

  Copy full SHA for 3d162cc

• Fix coding style

  goerch

  authored
  5aee498

  Copy full SHA for 5aee498

Commits on Oct 3, 2023

• Apply @jploski 's fix for missing tokens

  goerch

  authored
  3e518e2

  Copy full SHA for 3e518e2