Support prompt caching for Claude AI chat models #546
Labels
area:backend
Is related to the backend server. (Everything in aux-records and aux-server/aux-backend)
enhancement
New feature or request
Claude has added the ability to cache input prompts to increase performance and reduce costs. The caveat is that writing to the cache costs more than a regular input token, but the benefit is that a prompt that hits the cache will save 90% over what it would cost without the cache. CasualOS should support specifying if a message can be cached, and the
AnthropicAIChatInterface
should pass along the correct values to the Anthropic API.The text was updated successfully, but these errors were encountered: