You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Was happy to find this module as it is a more sophisticated version of some formulaic string concatenation I had written. In setting it up for my use case, I made several changes and wanted to see if any would be useful to submit.
updated the over token error message to give the quantity. so
error - unhandledRejection: Error: Prompt is greater than the configured max tokens. Either shorten context (detail + examples) or increase the max tokens in the model config.
becomes
error - unhandledRejection: Error: Prompt token length is greater than the configured max tokens (i.e. 4379 > 4000). Either shorten context (detail + examples) or increase the max tokens in the model config.
which was preferred to searching for the number in the response object .
use a dynamic max token value based on prompt size
This was preferred to using the number above and updating frequently. So if the prompt takes 600 tokens, and the model max is 4000, then 3600 is set as the max tokens to generate in the completion.
This is for text completion with OpenAI, and it's possible this does not generalize well, but I found it quite useful for a general query which is not an ongoing dialogue (so no history).
moved additional parameters to the YAML
Since the YAML already specifies max_tokens, it seemed logical to assign other completion parameters there. Specifically, model, temperature. The model name and max tokens in particular would seem to be changed infrequently.
Would love to know your opinion here, I could see an argument for keeping in the code for tuning access, or moving to an openai module config, etc.
The text was updated successfully, but these errors were encountered:
Was happy to find this module as it is a more sophisticated version of some formulaic string concatenation I had written. In setting it up for my use case, I made several changes and wanted to see if any would be useful to submit.
updated the over token error message to give the quantity. so
becomes
which was preferred to searching for the number in the response object .
use a dynamic max token value based on prompt size
This was preferred to using the number above and updating frequently. So if the prompt takes 600 tokens, and the model max is 4000, then 3600 is set as the max tokens to generate in the completion.
This is for text completion with OpenAI, and it's possible this does not generalize well, but I found it quite useful for a general query which is not an ongoing dialogue (so no history).
moved additional parameters to the YAML
Since the YAML already specifies
max_tokens
, it seemed logical to assign other completion parameters there. Specifically,model
,temperature
. The model name and max tokens in particular would seem to be changed infrequently.Would love to know your opinion here, I could see an argument for keeping in the code for tuning access, or moving to an openai module config, etc.
The text was updated successfully, but these errors were encountered: