Skip to content

Commit

Permalink
fix: disable caching on OpenAI-compatible endpoint type
Browse files Browse the repository at this point in the history
  • Loading branch information
@nsarrazin
nsarrazin committed Dec 28, 2024
1 parent 811974e commit f1b5bfd
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions src/lib/server/endpoints/openai/endpointOai.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ export async function endpointOai(
body: { ...body, ...extraBody },
headers: {
"ChatUI-Conversation-ID": conversationId?.toString() ?? "",
"X-use-cache": "false",
},
});

Expand Down Expand Up @@ -267,6 +268,7 @@ export async function endpointOai(
body: { ...body, ...extraBody },
headers: {
"ChatUI-Conversation-ID": conversationId?.toString() ?? "",
"X-use-cache": "false",
},
});

Expand Down

2 comments on commit f1b5bfd

@evalstate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @nsarrazin -- what problem was prompt caching causing?

@nsarrazin
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When retrying to get a new answer for the same prompt, the endpoint was just reusing previous answers from the cache, making it useless 😅

Not sure if this is specific to the HF Inference API or generic to all OpenAI compatible endpoint though. This is fixed in prod for HuggingChat now.

Please sign in to comment.