Skip to content

Commit

Permalink
use env variable for adding model
Browse files Browse the repository at this point in the history
  • Loading branch information
hitpoint6 committed Oct 2, 2024
1 parent 7e6c0c5 commit 8bc1ae0
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions docker-compose-vllm.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
environment:
- LETTA_LLM_ENDPOINT=http://vllm:8000
- LETTA_LLM_ENDPOINT_TYPE=vllm
- LETTA_LLM_MODEL=teknium/OpenHermes-2-Mistral-7B # Replace with your model
- LETTA_LLM_MODEL=${LETTA_LLM_MODEL} # Replace with your model
- LETTA_LLM_CONTEXT_WINDOW=8192
depends_on:
- vllm
Expand All @@ -30,6 +30,6 @@ services:
ports:
- "8000:8000"
command: >
--model teknium/OpenHermes-2-Mistral-7B --max_model_len=8000
--model ${LETTA_LLM_MODEL} --max_model_len=8000
# Replace with your model
ipc: host

0 comments on commit 8bc1ae0

Please sign in to comment.