Code

Add exllama GPTQ CUDA kernel support#553

Code

Closed

fxmarty wants to merge 27 commits intohuggingface:mainhuggingface/text-generation-inference:mainfrom fxmarty:gptq-cuda-kernelsfxmarty/text-generation-inference:gptq-cuda-kernelsCopy head branch name to clipboard

+3,317-66

Commits

Commits on Jul 5, 2023

• add exllama gptq kernel

  fxmarty

  committed
  ee7ba48

  Copy full SHA for ee7ba48

• add attribution

  fxmarty

  committed
  c858d79

  Copy full SHA for c858d79

• Merge branch 'main' into gptq-cuda-kernels

  fxmarty

  committed
  0ff8219

  Copy full SHA for 0ff8219

• some more cleanup

  fxmarty

  committed
  2272b3a

  Copy full SHA for 2272b3a

• Merge branch 'gptq-cuda-kernels' of https://github.com/fxmarty/text-generation-inference into gptq-cuda-kernels

  fxmarty

  committed
  620ed7d

  Copy full SHA for 620ed7d

• try-catch to load the cuda extension, quite ugly practice tbh

  fxmarty

  committed
  a6e3874

  Copy full SHA for a6e3874

Commits on Jul 12, 2023

• have a single gptq quantization type

  fxmarty

  committed
  4462854

  Copy full SHA for 4462854

• move exllama buffer init to the top level

  fxmarty

  committed
  67a46b7

  Copy full SHA for 67a46b7

• cleanup

  fxmarty

  committed
  67d6876

  Copy full SHA for 67d6876

• support bits different than 4

  fxmarty

  committed
  f90c61a

  Copy full SHA for f90c61a

• tests

  fxmarty

  committed
  8645fd3

  Copy full SHA for 8645fd3

• Merge branch 'main' into gptq-cuda-kernels

  fxmarty

  committed
  faa5b52

  Copy full SHA for faa5b52

• fix test

  fxmarty

  committed
  38c2be5

  Copy full SHA for 38c2be5

Commits on Jul 13, 2023

• fix tests

  fxmarty

  committed
  2ae65b4

  Copy full SHA for 2ae65b4

• support all, test llama

  fxmarty

  committed
  0036084

  Copy full SHA for 0036084

• Merge branch 'main' into gptq-cuda-kernels

  fxmarty

  committed
  9401e10

  Copy full SHA for 9401e10

• fix the usual merge mess

  fxmarty

  committed
  74e6d6e

  Copy full SHA for 74e6d6e

Commits on Jul 19, 2023

• Merge branch 'main' into gptq-cuda-kernels

  fxmarty

  committed
  edfbfdf

  Copy full SHA for edfbfdf

• fix per-column quantization

  fxmarty

  committed
  6bf7090

  Copy full SHA for 6bf7090

Commits on Jul 20, 2023

• Refactored a bit.

  Narsil

  committed
  0860394

  Copy full SHA for 0860394

• Small polish.

  Narsil

  committed
  8cf7c89

  Copy full SHA for 8cf7c89

• Give escape hatch to not use exllama kernels even if available.

  Narsil

  committed
  7faef69

  Copy full SHA for 7faef69

• Fixing GTPQ device santacoder.

  Narsil

  committed
  900ac49

  Copy full SHA for 900ac49

• Fix config.

  Narsil

  committed
  12191b7

  Copy full SHA for 12191b7

• Add kernel target.

  Narsil

  committed
  c6e702f

  Copy full SHA for c6e702f

• Separate build process.

  Narsil

  committed
  3ec3add

  Copy full SHA for 3ec3add

Commits on Jul 21, 2023

• Update starcoder_gptq

  Narsil

  committed
  40be532

  Copy full SHA for 40be532