Having issues while retrieving from GPT-4

[mohit19906]

Until now, our communication with our PDF was facilitated by GPT-3.5 Turbo. The PDF was kept in a Chroma vector store using OpenAI Embeddings and answers were retrieved from the PDF through the RetrievalQA chain. However, we have been encountering an error when attempting the same process with GPT-4. The error message states: "InvalidRequestError: The completion operation is not compatible with the selected model, gpt-4. Please select a different model and try again." More information about compatible models for each operation can be found here: https://go.microsoft.com/fwlink/?linkid=2197993.
Following is the code to retrieve LLM response -
retriever = chatDb.as_retriever(search_type="similarity", search_kwargs={"k": 3})

qa = RetrievalQA.from_chain_type( llm=get_openai_model(), chain_type="stuff", retriever=retriever, return_source_documents=True, verbose = True, chain_type_kwargs={ "verbose" : True, "prompt" : promptHist, } )

result = qa({"query": prompt})

[dosubot[bot]]

🤖

Hello,

Based on the error message you're encountering, it seems that the GPT-4 model is not compatible with the completion operation you're trying to perform. This could be due to the fact that the GPT-4 model might not be supported by the operation or the tiktoken package used by the LangChain framework.

From the similar issues I found in the LangChain repository, there are a few potential solutions you could try:

1. Switch to a model that is supported by the Azure OpenAI API, such as GPT-3 or Codex. This can be done by changing the DEPLOYMENT_NAME variable in your code to the name of the deployment of the supported model. You might also consider using the AzureChatOpenAI class from the langchain.chat_models.azure_openai module, which wraps around the Azure OpenAI Chat Completion API. This class requires a deployed model on Azure OpenAI, and the deployment_name should refer to the "Model deployment name" in the Azure portal. For example:

AzureChatOpenAI(
    deployment_name="35-turbo-dev",
    openai_api_version="2023-03-15-preview",
)

Please note that the API version may change, so always ensure you're using the correct version. You can find more details about this solution in this issue.

2. Replace OpenAI with ChatOpenAI when working with a chat model (like gpt-3.5-turbo). The corrected code is:

qa = ConversationalRetrievalChain.from_llm(ChatOpenAI(model="gpt-4", temperature=0.2, max_tokens=5000),
                                               retriever=retriever, 
                                               return_source_documents=True,
                                               verbose=True,
                                               output_key='answer',
                                               combine_docs_chain_kwargs={'prompt': prompt},
                                               condense_question_llm=ChatOpenAI(model="gpt-3.5-turbo", temperature=0))

You can find more details about this solution in this issue.

3. Use model_kwargs and deployment_name instead of deployment. Also, ensure that you're using an API version later than 2022-03-15-preview to avoid errors. You can find more details about this solution in this issue.

If you're still encountering issues after trying these solutions, please provide more specific details about how you're using the GPT-4 model in the LangChain framework. This will help us better understand the issue and provide a more accurate solution.

Best,
Dosu

Sources

• openai.error.InvalidRequestError: The completion operation does not work with the specified model, gpt-4. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.
• 0.0.229 breaks existing code that works with 0.0.228 for ConverstaionalRetrievalChain
• Issue: RetrievalQA is not working with AzureOpenAI LLM model
• libs/langchain/langchain/chat_models/openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.