From a1f9786a14647c2b3e70850668c5d164c9ffc28a Mon Sep 17 00:00:00 2001 From: Zhanghao Wu Date: Wed, 18 Sep 2024 20:57:41 +0000 Subject: [PATCH 1/2] Update qwen example for 2.5 release --- llm/qwen/README.md | 14 ++++++++------ llm/qwen/{qwen2-72b.yaml => qwen25-72b.yaml} | 2 +- llm/qwen/{qwen2-7b.yaml => qwen25-7b.yaml} | 2 +- 3 files changed, 10 insertions(+), 8 deletions(-) rename llm/qwen/{qwen2-72b.yaml => qwen25-72b.yaml} (95%) rename llm/qwen/{qwen2-7b.yaml => qwen25-7b.yaml} (94%) diff --git a/llm/qwen/README.md b/llm/qwen/README.md index d10b9e59c36..7d2d799bc1e 100644 --- a/llm/qwen/README.md +++ b/llm/qwen/README.md @@ -3,9 +3,11 @@ [Qwen2](https://github.com/QwenLM/Qwen2) is one of the top open LLMs. As of Jun 2024, Qwen1.5-110B-Chat is ranked higher than GPT-4-0613 on the [LMSYS Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). -📰 **Update (Jun 6 2024) -** SkyPilot now also supports the [**Qwen2**](https://qwenlm.github.io/blog/qwen2/) model! It further improves the competitive model, Qwen1.5. +**Update (Sep 18, 2024) -** SkyPilot now supports the [**Qwen2.5**](https://qwenlm.github.io/blog/qwen2.5/) model! -📰 **Update (April 26 2024) -** SkyPilot now also supports the [**Qwen1.5-110B**](https://qwenlm.github.io/blog/qwen1.5-110b/) model! It performs competitively with Llama-3-70B across a [series of evaluations](https://qwenlm.github.io/blog/qwen1.5-110b/#model-quality). Use [qwen15-110b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen15-110b.yaml) to serve the 110B model. +📰 **Update (Jun 6, 2024) -** SkyPilot now also supports the [**Qwen2**](https://qwenlm.github.io/blog/qwen2/) model! It further improves the competitive model, Qwen1.5. + +📰 **Update (April 26, 2024) -** SkyPilot now also supports the [**Qwen1.5-110B**](https://qwenlm.github.io/blog/qwen1.5-110b/) model! It performs competitively with Llama-3-70B across a [series of evaluations](https://qwenlm.github.io/blog/qwen1.5-110b/#model-quality). Use [qwen15-110b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen15-110b.yaml) to serve the 110B model.

qwen @@ -27,7 +29,7 @@ As of Jun 2024, Qwen1.5-110B-Chat is ranked higher than GPT-4-0613 on the [LMSYS After [installing SkyPilot](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html), run your own Qwen model on vLLM with SkyPilot in 1-click: -1. Start serving Qwen 110B on a single instance with any available GPU in the list specified in [qwen15-110b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen15-110b.yaml) with a vLLM powered OpenAI-compatible endpoint (You can also switch to [qwen2-72b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen2-72b.yaml) or [qwen2-7b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen2-7b.yaml) for a smaller model): +1. Start serving Qwen 110B on a single instance with any available GPU in the list specified in [qwen15-110b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen15-110b.yaml) with a vLLM powered OpenAI-compatible endpoint (You can also switch to [qwen25-72b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen25-72b.yaml) or [qwen25-7b.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen25-7b.yaml) for a smaller model): ```console sky launch -c qwen qwen15-110b.yaml @@ -69,7 +71,7 @@ curl http://$ENDPOINT/v1/chat/completions \ 1. With [SkyPilot Serving](https://skypilot.readthedocs.io/en/latest/serving/sky-serve.html), a serving library built on top of SkyPilot, scaling up the Qwen service is as simple as running: ```bash -sky serve up -n qwen ./qwen2-72b.yaml +sky serve up -n qwen ./qwen25-72b.yaml ``` This will start the service with multiple replicas on the cheapest available locations and accelerators. SkyServe will automatically manage the replicas, monitor their health, autoscale based on load, and restart them when needed. @@ -101,7 +103,7 @@ ENDPOINT=$(sky serve status --endpoint qwen) curl http://$ENDPOINT/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ - "model": "Qwen/Qwen2-72B-Instruct", + "model": "Qwen/qwen25-72B-Instruct", "messages": [ { "role": "system", @@ -123,7 +125,7 @@ It is also possible to access the Qwen service with a GUI using [vLLM](https://g 1. Start the chat web UI (change the `--env` flag to the model you are running): ```bash -sky launch -c qwen-gui ./gui.yaml --env MODEL_NAME='Qwen/Qwen2-72B-Instruct' --env ENDPOINT=$(sky serve status --endpoint qwen) +sky launch -c qwen-gui ./gui.yaml --env MODEL_NAME='Qwen/qwen25-72B-Instruct' --env ENDPOINT=$(sky serve status --endpoint qwen) ``` 2. Then, we can access the GUI at the returned gradio link: diff --git a/llm/qwen/qwen2-72b.yaml b/llm/qwen/qwen25-72b.yaml similarity index 95% rename from llm/qwen/qwen2-72b.yaml rename to llm/qwen/qwen25-72b.yaml index 00ff41506e9..cfbf1d06a8c 100644 --- a/llm/qwen/qwen2-72b.yaml +++ b/llm/qwen/qwen25-72b.yaml @@ -1,5 +1,5 @@ envs: - MODEL_NAME: Qwen/Qwen2-72B-Instruct + MODEL_NAME: Qwen/Qwen2.5-72B-Instruct service: # Specifying the path to the endpoint to check the readiness of the replicas. diff --git a/llm/qwen/qwen2-7b.yaml b/llm/qwen/qwen25-7b.yaml similarity index 94% rename from llm/qwen/qwen2-7b.yaml rename to llm/qwen/qwen25-7b.yaml index ccf8d62d306..f9065e08579 100644 --- a/llm/qwen/qwen2-7b.yaml +++ b/llm/qwen/qwen25-7b.yaml @@ -1,5 +1,5 @@ envs: - MODEL_NAME: Qwen/Qwen2-7B-Instruct + MODEL_NAME: Qwen/Qwen2.5-7B-Instruct service: # Specifying the path to the endpoint to check the readiness of the replicas. From 0f488cbeda6debb63daf33a01cc7c7605755eac3 Mon Sep 17 00:00:00 2001 From: Zhanghao Wu Date: Wed, 18 Sep 2024 21:04:01 +0000 Subject: [PATCH 2/2] Add support for qwen 2.5 example --- llm/qwen/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/llm/qwen/README.md b/llm/qwen/README.md index 7d2d799bc1e..04ecd88e9c0 100644 --- a/llm/qwen/README.md +++ b/llm/qwen/README.md @@ -103,7 +103,7 @@ ENDPOINT=$(sky serve status --endpoint qwen) curl http://$ENDPOINT/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ - "model": "Qwen/qwen25-72B-Instruct", + "model": "Qwen/Qwen2.5-72B-Instruct", "messages": [ { "role": "system", @@ -125,7 +125,7 @@ It is also possible to access the Qwen service with a GUI using [vLLM](https://g 1. Start the chat web UI (change the `--env` flag to the model you are running): ```bash -sky launch -c qwen-gui ./gui.yaml --env MODEL_NAME='Qwen/qwen25-72B-Instruct' --env ENDPOINT=$(sky serve status --endpoint qwen) +sky launch -c qwen-gui ./gui.yaml --env MODEL_NAME='Qwen/Qwen2.5-72B-Instruct' --env ENDPOINT=$(sky serve status --endpoint qwen) ``` 2. Then, we can access the GUI at the returned gradio link: