From 402382286f4488c62206967ef13f2b078a000e6b Mon Sep 17 00:00:00 2001 From: Zhanghao Wu Date: Wed, 18 Sep 2024 14:15:45 -0700 Subject: [PATCH] Qwen 2.5 support (#3959) * Update qwen example for 2.5 release * Add support for qwen 2.5 example --- llm/qwen/README.md | 14 ++++++++------ llm/qwen/{qwen2-72b.yaml => qwen25-72b.yaml} | 2 +- llm/qwen/{qwen2-7b.yaml => qwen25-7b.yaml} | 2 +- 3 files changed, 10 insertions(+), 8 deletions(-) rename llm/qwen/{qwen2-72b.yaml => qwen25-72b.yaml} (95%) rename llm/qwen/{qwen2-7b.yaml => qwen25-7b.yaml} (94%) diff --git a/llm/qwen/README.md b/llm/qwen/README.md index d10b9e59c36..04ecd88e9c0 100644 --- a/llm/qwen/README.md +++ b/llm/qwen/README.md @@ -3,9 +3,11 @@ [Qwen2](https://github.com/QwenLM/Qwen2) is one of the top open LLMs. As of Jun 2024, Qwen1.5-110B-Chat is ranked higher than GPT-4-0613 on the [LMSYS Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). -📰 **Update (Jun 6 2024) -** SkyPilot now also supports the [**Qwen2**](https://qwenlm.github.io/blog/qwen2/) model! It further improves the competitive model, Qwen1.5. +**Update (Sep 18, 2024) -** SkyPilot now supports the [**Qwen2.5**](https://qwenlm.github.io/blog/qwen2.5/) model! -📰 **Update (April 26 2024) -** SkyPilot now also supports the [**Qwen1.5-110B**](https://qwenlm.github.io/blog/qwen1.5-110b/) model! It performs competitively with Llama-3-70B across a [series of evaluations](https://qwenlm.github.io/blog/qwen1.5-110b/#model-quality). Use [qwen15-110b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen15-110b.yaml) to serve the 110B model. +📰 **Update (Jun 6, 2024) -** SkyPilot now also supports the [**Qwen2**](https://qwenlm.github.io/blog/qwen2/) model! It further improves the competitive model, Qwen1.5. + +📰 **Update (April 26, 2024) -** SkyPilot now also supports the [**Qwen1.5-110B**](https://qwenlm.github.io/blog/qwen1.5-110b/) model! It performs competitively with Llama-3-70B across a [series of evaluations](https://qwenlm.github.io/blog/qwen1.5-110b/#model-quality). Use [qwen15-110b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen15-110b.yaml) to serve the 110B model.

qwen @@ -27,7 +29,7 @@ As of Jun 2024, Qwen1.5-110B-Chat is ranked higher than GPT-4-0613 on the [LMSYS After [installing SkyPilot](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html), run your own Qwen model on vLLM with SkyPilot in 1-click: -1. Start serving Qwen 110B on a single instance with any available GPU in the list specified in [qwen15-110b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen15-110b.yaml) with a vLLM powered OpenAI-compatible endpoint (You can also switch to [qwen2-72b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen2-72b.yaml) or [qwen2-7b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen2-7b.yaml) for a smaller model): +1. Start serving Qwen 110B on a single instance with any available GPU in the list specified in [qwen15-110b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen15-110b.yaml) with a vLLM powered OpenAI-compatible endpoint (You can also switch to [qwen25-72b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen25-72b.yaml) or [qwen25-7b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen25-7b.yaml) for a smaller model): ```console sky launch -c qwen qwen15-110b.yaml @@ -69,7 +71,7 @@ curl http://$ENDPOINT/v1/chat/completions \ 1. With [SkyPilot Serving](https://skypilot.readthedocs.io/en/latest/serving/sky-serve.html), a serving library built on top of SkyPilot, scaling up the Qwen service is as simple as running: ```bash -sky serve up -n qwen ./qwen2-72b.yaml +sky serve up -n qwen ./qwen25-72b.yaml ``` This will start the service with multiple replicas on the cheapest available locations and accelerators. SkyServe will automatically manage the replicas, monitor their health, autoscale based on load, and restart them when needed. @@ -101,7 +103,7 @@ ENDPOINT=$(sky serve status --endpoint qwen) curl http://$ENDPOINT/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ - "model": "Qwen/Qwen2-72B-Instruct", + "model": "Qwen/Qwen2.5-72B-Instruct", "messages": [ { "role": "system", @@ -123,7 +125,7 @@ It is also possible to access the Qwen service with a GUI using [vLLM](https://g 1. Start the chat web UI (change the `--env` flag to the model you are running): ```bash -sky launch -c qwen-gui ./gui.yaml --env MODEL_NAME='Qwen/Qwen2-72B-Instruct' --env ENDPOINT=$(sky serve status --endpoint qwen) +sky launch -c qwen-gui ./gui.yaml --env MODEL_NAME='Qwen/Qwen2.5-72B-Instruct' --env ENDPOINT=$(sky serve status --endpoint qwen) ``` 2. Then, we can access the GUI at the returned gradio link: diff --git a/llm/qwen/qwen2-72b.yaml b/llm/qwen/qwen25-72b.yaml similarity index 95% rename from llm/qwen/qwen2-72b.yaml rename to llm/qwen/qwen25-72b.yaml index 00ff41506e9..cfbf1d06a8c 100644 --- a/llm/qwen/qwen2-72b.yaml +++ b/llm/qwen/qwen25-72b.yaml @@ -1,5 +1,5 @@ envs: - MODEL_NAME: Qwen/Qwen2-72B-Instruct + MODEL_NAME: Qwen/Qwen2.5-72B-Instruct service: # Specifying the path to the endpoint to check the readiness of the replicas. diff --git a/llm/qwen/qwen2-7b.yaml b/llm/qwen/qwen25-7b.yaml similarity index 94% rename from llm/qwen/qwen2-7b.yaml rename to llm/qwen/qwen25-7b.yaml index ccf8d62d306..f9065e08579 100644 --- a/llm/qwen/qwen2-7b.yaml +++ b/llm/qwen/qwen25-7b.yaml @@ -1,5 +1,5 @@ envs: - MODEL_NAME: Qwen/Qwen2-7B-Instruct + MODEL_NAME: Qwen/Qwen2.5-7B-Instruct service: # Specifying the path to the endpoint to check the readiness of the replicas.