Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Potential improvements related to tokens and YAML parameters #25

Open
enorms opened this issue Jan 10, 2023 · 0 comments
Open

Potential improvements related to tokens and YAML parameters #25

enorms opened this issue Jan 10, 2023 · 0 comments

Comments

@enorms
Copy link

enorms commented Jan 10, 2023

Was happy to find this module as it is a more sophisticated version of some formulaic string concatenation I had written. In setting it up for my use case, I made several changes and wanted to see if any would be useful to submit.

  1. updated the over token error message to give the quantity. so

    error - unhandledRejection: Error: Prompt is greater than the configured max tokens. Either shorten context (detail + examples) or increase the max tokens in the model config.
    

    becomes

    error - unhandledRejection: Error: Prompt token length is greater than the configured max tokens (i.e. 4379 > 4000). Either shorten context (detail + examples) or increase the max tokens in the model config.
    

    which was preferred to searching for the number in the response object .

  2. use a dynamic max token value based on prompt size

    This was preferred to using the number above and updating frequently. So if the prompt takes 600 tokens, and the model max is 4000, then 3600 is set as the max tokens to generate in the completion.

    This is for text completion with OpenAI, and it's possible this does not generalize well, but I found it quite useful for a general query which is not an ongoing dialogue (so no history).

  3. moved additional parameters to the YAML

    Since the YAML already specifies max_tokens, it seemed logical to assign other completion parameters there. Specifically, model, temperature. The model name and max tokens in particular would seem to be changed infrequently.

    Would love to know your opinion here, I could see an argument for keeping in the code for tuning access, or moving to an openai module config, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant