Fixed ValueError: Could not parse LLM output

[klimentij]

Sometimes a conversational agent produces a noisy JSON with action and action_input keys that has hard time to be parsed even after cleaning, for example:

Here's a response in the second format, as I don't think using any of the tools would be helpful in this case:

``json
{
    "action": "Final Answer",
    "action_input": "I'm sorry to hear that you're feeling sad. If you'd like, I'm here to chat and listen if you want to talk about what's been bothering you."
}
``

Remember that it's important to take care of your mental health, and it's okay to not be okay sometimes. If you're feeling overwhelmed or need someone to talk to, there are resources available to help you.

This causes a ValueError: Could not parse LLM output.

Related issues: #1657 #1477 #1358

I just added a simple fallback parsing that instead of using json.loads() relies on detecting "action": " and "action_input": " and parsing the value with split(). Also in case LLM failed to produce any JSON at all, I made the agent to return llm_output as a Final Answer instead of raising an error

[itjuba]

Still getting the erorr

File "/home/venv/lib/python3.10/site-packages/langchain/agents/conversational_chat/base.py", line 119, in _extract_tool_and_input
response = self.output_parser.parse(llm_output)
File "/home/venv/lib/python3.10/site-packages/langchain/agents/conversational_chat/base.py", line 61, in parse
raise ValueError("Invalid input format. Unable to extract 'action' and 'action_input' from the text.")
ValueError: Invalid input format. Unable to extract 'action' and 'action_input' from the text.

Using this Parser

   def parse(self, text: str) -> Dict[str, str]:
        cleaned_output = text.strip()
        if "```json" in cleaned_output:
            _, cleaned_output = cleaned_output.split("```json")
        if cleaned_output.startswith("```json"):
            cleaned_output = cleaned_output[len("```json") :]
        if cleaned_output.startswith("```"):
            cleaned_output = cleaned_output[len("```") :]
        if cleaned_output.endswith("```"):
            cleaned_output = cleaned_output[: -len("```")]
        cleaned_output = cleaned_output.strip()

        try:
            response = json.loads(cleaned_output)
        except json.JSONDecodeError:
            response = {}
            cleaned_output = cleaned_output.replace('\n', '')

            try:
                response["action"] = cleaned_output.split('"action": "', 1)[1].split('",', 1)[0].strip()
                response["action_input"] = cleaned_output.split('"action_input": "', 1)[1].split('"}', 1)[0].strip()
            except IndexError:

                raise ValueError("Invalid input format. Unable to extract 'action' and 'action_input' from the text.")

        return {"action": response["action"], "action_input": response["action_input"]}

[tonyabracadabra]

I think maybe we can adjust the prompt to avoid this error

[alexprice12]

Any thoughts on how to improve the prompt? It seems to happen for all agents periodically for gpt-4

[klimentij]

Add a bunch of in-context examples? But still no guarantee, so a robust parser on top is a must IMHO.

[alexprice12]

For some context, is there a reason there are so many different parsers? It seems there is a lot of complexity in the fact that there are different parsers.

I'm not trying to be critical :). There just appears to be a slightly different issue for each of the parsers and it seems like whack-a-mole.

[tiagoefreitas]

This solved the issue for me, but PR is out of sync with latest langchain

[klimentij]

@alexprice12 I agree with you, it's basically a fast and ugly hack. In my project, I ended up replacing this agent with several custom chains, where each chain does its own little thing (but does it very well, each with few-shot examples):

1. Predicts a tool
2. If search, predicts queries
3. Map-reduce style (also custom) response construction

I would suggest replacing the original Langchain's one-step JSON with a two-step JSON-free approach (first generate just the tool name, then generate what the tool needs - in different chains).

[tiagoefreitas]

@klimentij that makes perfect sense, are you willing to share your code or is it already on github in your fork?

Price for the API calls is the same, so I don't understand the design decision of doing more than one step in the same prompt, especially since langchain has chain in the name, and its not using chains properly in agents?? It would be interesting to understand that let to that decision, there may be a good reason.

@hwchase17 can you chime in? Thanks!

[Houss3m]

I am facing the same issue, I am using one tool only, do you think if I force the model to use the tool always would be a good idea? if yes, how can I modify the base.py to do so?

The issue is raising from class ConversationalAgent(Agent) : [dist-packages/langchain/agents/conversational/base.py]

[pieroit]

This error is happening for small, non instruction fine-tuned models (which means, most of non-commercial LLMs)

I suggest when the output cannot be parsed, the chain just returns the latest model output

[extrange]

I modified the prompts:

FORMAT_INSTRUCTIONS = """RESPONSE FORMAT INSTRUCTIONS
----------------------------

When responding please, please output a response in this format:

thought: Reason about what action to take next, and whether to use a tool.
action: The tool to use. Must be one of: {tool_names}
action_input: The input to the tool

For example:

thought: I need to send a message to xxx
action: Telegram
action_input: Send a message to xxx: I don't agree with that idea
"""

The agent responds in the following format:

thought: I can use Telegram to send a message to xxx with a poem.
action: Telegram
action_input: Send a message to xxx with the following poem:

Oh hey there, it's me
(truncated)

The modified parse method of the 'BaseOutputParser`:

    def parse(self, text: str) -> Any:
        """
        More resilient version of parser.

        Searches for lines beginning with 'action:' and 'action_input:', and ignores preceding lines.
        """
        cleaned_output = text.strip()

        # Find line starting with 'action:'
        action_match = re.search("^action: (.*)$", cleaned_output, re.MULTILINE)
        if action_match is None:
            raise ValueError(f"Could not find valid action in {cleaned_output}")
        action = action_match.group(1)

        action_input_match = re.search(
            "^action_input: (.*)", text, flags=re.MULTILINE | re.DOTALL
        )
        if action_input_match is None:
            raise ValueError(f"Could not find valid action_input in {cleaned_output}")
        action_input = action_input_match.group(1)
        return {"action": action, "action_input": action_input}

[jimilp7]

I tried to solve this using semantic parsing as a fail over. Would love to hear feedback: #2958

[Srajangpt1]

How would I be able to call this function from JsonAgent Toolkit?

[seanjaffeg]

I see 'action input' in the response, not 'action_input'. Is there a way to make this fix more general?

[Illapavan]

I am using the langchain agent. I have passed handle_parsing_errors=True this solved my issue
llm_chain = LLMChain(llm=OpenAI(model_name = "gpt-4", temperature=0), prompt=prompt)
agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True)
agent_chain = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory, handle_parsing_errors=True)
response = agent_chain.run(user_input)

[cap-blurr]

I am using the langchain agent. I have passed handle_parsing_errors=True this solved my issue llm_chain = LLMChain(llm=OpenAI(model_name = "gpt-4", temperature=0), prompt=prompt) agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True) agent_chain = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory, handle_parsing_errors=True) response = agent_chain.run(user_input)

this literally worked,thanks!

[qinchaofeng]

check this, worked for me:
https://python.langchain.com/en/latest/modules/agents/agent_executors/examples/handle_parsing_errors.html

[engperini]

only do not have error when using 'gpt-4' model!

[leo-gan]

@klimentij Hi Klimentij, could you, please, resolve the merging issues? After that ping me and I push this PR for the review. Thanks!

[efriis]

Closing because the library's folder structure has changed since this PR was last worked on, and you're welcome to reopen if you get it in line with the new format!