Skip to content

Commit

Permalink
Update README.md (#916)
Browse files Browse the repository at this point in the history
  • Loading branch information
keshavkowshik authored Aug 16, 2024
1 parent b04fedd commit 8d91a5a
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion benchmarks/inference/deepspeedometer/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# DeepSpeedometer

NOTE: This is an experimental tool and is not currently being supported since it's not fully functional. Please use the MII benchmark which can be found here:
https://github.com/microsoft/DeepSpeedExamples/tree/master/benchmarks/inference/mii

This benchmark is designed to measure performance of LLM serving solutions. Using a different number of parallel clients sending requests to an inference server, we gather data to plot throughput-latency curves and find the saturation point of an inference server that demonstrates the maximum performance.

## Installation
Expand Down Expand Up @@ -82,4 +85,4 @@ The DeepSpeedometer benchmark was designed to allow easily adding support for ne
3. Define a `*ClientConfig` class that inherits from the `BaseConfigModel` class. Place any configuration options (i.e., user-passed command line arguments) necessary for your defined `*Client` class in here.
4. Import the newly added `*Client` and `*ClientConfig` into `clients/__init__.py` and add them to the `client_config_classes` and `client_classes` dictionaries.

For the simplest example of adding a new client, take a look at the `clients/dummy_client.py` file where we have defined a client that does not stand up a server and only returns a sample of the input prompt after a short sleep cycle. We use this as a light-weight class for unit testing.
For the simplest example of adding a new client, take a look at the `clients/dummy_client.py` file where we have defined a client that does not stand up a server and only returns a sample of the input prompt after a short sleep cycle. We use this as a light-weight class for unit testing.

0 comments on commit 8d91a5a

Please sign in to comment.