Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [support logit_bias for OpenAI API compatible] #263

Closed
nguyenhoangthuan99 opened this issue Oct 24, 2024 · 1 comment · Fixed by #270
Closed

feat: [support logit_bias for OpenAI API compatible] #263

nguyenhoangthuan99 opened this issue Oct 24, 2024 · 1 comment · Fixed by #270
Assignees
Labels
Milestone

Comments

@nguyenhoangthuan99
Copy link
Contributor

nguyenhoangthuan99 commented Oct 24, 2024

Problem
logit_bias map Optional Defaults to null.

Modify the likelihood of specified tokens appearing in the completion. Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token.

-> need to confirm llamacpp support this or not, but this might be nice to have feature.

reference: https://platform.openai.com/docs/api-reference/chat/create#chat-create-logit_bias

related issue: https://github.com/janhq/internal/issues/160

@github-project-automation github-project-automation bot moved this to Investigating in Menlo Oct 24, 2024
@nguyenhoangthuan99 nguyenhoangthuan99 self-assigned this Oct 28, 2024
@nguyenhoangthuan99 nguyenhoangthuan99 moved this from Investigating to In Progress in Menlo Oct 28, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Review + QA in Menlo Oct 29, 2024
@gabrielle-ong gabrielle-ong added this to the v1.0.2 milestone Nov 6, 2024
@gabrielle-ong
Copy link

✅ QA: Logit_bias 100 vs -100:
logit_bias = 100 => exclusive selection of the word "under"
logit_bias = 10 => increased selection of the word "under"
logit_bias = 0 => normal response

Image
Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants