Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support prompt caching for Claude AI chat models #546

Open
KallynGowdy opened this issue Sep 25, 2024 · 0 comments
Open

Support prompt caching for Claude AI chat models #546

KallynGowdy opened this issue Sep 25, 2024 · 0 comments
Assignees
Labels
area:backend Is related to the backend server. (Everything in aux-records and aux-server/aux-backend) enhancement New feature or request

Comments

@KallynGowdy
Copy link
Member

Claude has added the ability to cache input prompts to increase performance and reduce costs. The caveat is that writing to the cache costs more than a regular input token, but the benefit is that a prompt that hits the cache will save 90% over what it would cost without the cache. CasualOS should support specifying if a message can be cached, and the AnthropicAIChatInterface should pass along the correct values to the Anthropic API.

@KallynGowdy KallynGowdy converted this from a draft issue Sep 25, 2024
@KallynGowdy KallynGowdy added enhancement New feature or request area:backend Is related to the backend server. (Everything in aux-records and aux-server/aux-backend) labels Sep 25, 2024
@TroyceGowdy TroyceGowdy self-assigned this Dec 27, 2024
@TroyceGowdy TroyceGowdy moved this from Backlog to In progress in CasualOS Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:backend Is related to the backend server. (Everything in aux-records and aux-server/aux-backend) enhancement New feature or request
Projects
Status: In progress
Development

No branches or pull requests

2 participants