Support for frequency_penalty

[richardliaw]

Frequency Penalty is between -2.0 and 2.0 and it impacts how the model penalizes new tokens based on their existing frequency in the text.

Positive values will decrease the likelihood of the model repeating the same line verbatim by penalizing new tokens that have already been used frequently.

This parameter is important for OpenAI compatibility, which is a growing standard for LLM usage.

cc @Yard1 @akshay-anyscale

[juney-nvidia]

@richardliaw

Thanks for reporting this. Can you elaborate a little bit more about the difference between the already supported repetiton_penalty and frequency_penalty here?

Thanks
June

[richardliaw]

Frequency penalty has a specific implementation here: https://platform.openai.com/docs/guides/gpt/parameter-details.

vLLM treats the two differently (repetition_penalty is multiplicative with the logits, whereas frequency penalty is additive): https://github.com/vllm-project/vllm/blob/9f669a9a7c2b2d0a7963a6e29253280e57680adb/vllm/model_executor/layers/sampler.py#L233-L236

[juney-nvidia]

Thanks for sharing this, this issue relates to the other one, both regarding to the control of sampling/decoder process. We will follow up and reply later.

June

[kaiyux]

The support should be included in the latest main branch, please see #754.

Closing. Please feel free to comment if you have any question, thanks!