Code

Use vLLM to load LLMs#230

Code

Merged

ad-astra-video merged 10 commits intolivepeer:mainlivepeer/ai-runner:mainfrom joinhive-ai:llmjoinhive-ai/ai-worker:llmCopy head branch name to clipboardDec 26, 2024

+698-453

Commits

Commits on Dec 19, 2024

• llm: use vLLM

  kyriediculous

  committed
  67a32b7

  Copy full SHA for 67a32b7

• llm: improve API

  kyriediculous

  committed
  dcd60e7

  Copy full SHA for dcd60e7

• llm: regenerate bindings

  kyriediculous

  committed
  59f0a58

  Copy full SHA for 59f0a58

• requirements.in using pip-compile

  kyriediculous

  committed
  484b7c5

  Copy full SHA for 484b7c5

Commits on Dec 20, 2024

• fix: stream_generator helper

  kyriediculous

  committed
  217eeee

  Copy full SHA for 217eeee

• docker: isolated docker image for LLMs

  kyriediculous

  committed
  5f58a29

  Copy full SHA for 5f58a29

• fixup! docker: isolated docker image for LLMs

  kyriediculous

  committed
  7c9343f

  Copy full SHA for 7c9343f

Commits on Dec 26, 2024

• fix: quantization with load_format

  kyriediculous

  committed
  d738705

  Copy full SHA for d738705

• fix: rm sfast from Dockerfile.llm

  kyriediculous

  committed
  92f7c35

  Copy full SHA for 92f7c35

• Merge branch 'main' into llm

  kyriediculous

  authored
  bce039a

  Copy full SHA for bce039a