Skip to content
This repository has been archived by the owner on Feb 5, 2024. It is now read-only.

Wishlist: see if we can use a cheaper model to generate adequate summaries #2

Open
scottleibrand opened this issue Dec 20, 2022 · 0 comments

Comments

@scottleibrand
Copy link
Owner

Below is a screenshot from https://beta.openai.com/account/usage for the two script runs used to generate the https://github.com/scottleibrand/gpt-summarizer/tree/main/examples. At $0.02 per 10k tokens, that totals to $0.76 worth of usage to summarize those two documents. That's worthwhile, if they're documents I'd have had to read otherwise: if my free time is worth $20/hr, that means the summaries would only need to save me at least 1-2 minutes of reading time per document to be worth doing.
image

That said, it would be better if we could figure out how to generate adequate summaries with a smaller model. I tried directly using text-curie-001, but it generated empty summaries with the current prompts: when I ran the same prompts in the playground, it said that is because the generation started with a stop token. Removing the reiterated instructions from the prompt allowed it to generate a brief summary, but not a very good one: it seemed to only briefly be summarizing the last paragraph.

To make it work better with curie or other models, we might need to change the chunking to summarize much smaller chunks, and possibly add another layer of summarization to deal with larger documents. This might warrant a refactor of the main code block.

If anyone is interested in taking this on, please comment below.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant