initialize_agent does not accept callable 'handle_parsing_errors', only bool ('value could not be parsed to a boolean')

[necarlson97]

System Info

langchain 0.0.301
Python 3.10.12
Ubuntu 22.04 LTS

Who can help?

@hwchase17

(based on blame of langchain/agents/agent.py", line 622)

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

import yaml
import os
from langchain.chat_models import ChatOpenAI
from langchain.agents import initialize_agent

# https://python.langchain.com/docs/modules/agents/how_to/handle_parsing_errors
def _handle_error(error) -> str:
    return str(error)

# Do whatever to load your api key
with open('secrets.yml', 'r') as f:
    config = yaml.safe_load(f)
    os.environ["OPENAI_API_KEY"] = config['open_ai_key']

llm = ChatOpenAI(model_name='gpt-3.5-turbo')

agentExcecutor = initialize_agent(
    [], llm,
    agent="conversational-react-description",
    handle_parsing_errors=_handle_error,
)

Results in:

Traceback (most recent call last):
  File ".../test_error_handling.py", line 17, in <module>
    agentExcecutor = initialize_agent(
  File ".../langchain/agents/initialize.py", line 64, in initialize_agent
    return AgentExecutor.from_agent_and_tools(
  File ".../langchain/agents/agent.py", line 622, in from_agent_and_tools
    return cls(
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for AgentExecutor
handle_parsing_errors
  value could not be parsed to a boolean (type=type_error.bool)

Expected behavior

It should initialize the agent, and trigger the custom _handle_error as described in:
https://python.langchain.com/docs/modules/agents/how_to/handle_parsing_errors#custom-error-function

I am (like many) experiencing frequent 'Could not parse LLM output' #1358.
My current project is targeting gpt-3.5, and doing system-message/prompt tweaking - so parse failures are to be expected. However, I want to handle the error gracefully - retrying, summarizing failed attempts, apologizing and moving on, etc - depending on the specifics of my project.

The handle_parsing_errors seems like the perfect place to implement a workaround. However, passing a callable func seems to fail validation, and thus does not work.
Note:

• using try/except to handle parse errors is not ideal for us, as, by default, it omits those messages from the memory
• using handle_parsing_errors=true is what I am using now, but I'd like to customize the failure handling

As always, apologies if I misunderstand the documentation, or if this is already reported. I did not see anything in the tracker - #10710 is close, but regards tools rather than agents afaict.
Thank you for any help!

[dosubot]

🤖

Hello,

Thank you for your detailed report. It seems like you've encountered a bug in the initialize_agent function. According to the documentation, the handle_parsing_errors parameter should indeed accept a callable function. However, the error message you're seeing suggests that the function is only accepting boolean values.

This discrepancy between the documentation and the actual behavior of the function is likely due to a bug in the implementation of the initialize_agent function. I've looked into the codebase and found that the initialize_agent function is implemented as follows:

def initialize_agent(
    tools: Sequence[BaseTool],
    llm: BaseLanguageModel,
    agent: Optional[AgentType] = None,
    callback_manager: Optional[BaseCallbackManager] = None,
    agent_path: Optional[str] = None,
    agent_kwargs: Optional[dict] = None,
    *,
    tags: Optional[Sequence[str]] = None,
    **kwargs: Any,
) -> AgentExecutor:
    ...
    return AgentExecutor.from_agent_and_tools(
        agent=agent_obj,
        tools=tools,
        callback_manager=callback_manager,
        tags=tags_,
        **kwargs,
    )

The **kwargs at the end of the AgentExecutor.from_agent_and_tools call is where the handle_parsing_errors parameter should be passed. However, it seems like the AgentExecutor class is expecting a boolean value for handle_parsing_errors, hence the validation error.

I found a similar issue in the repository where the handle_parsing_errors parameter was not being passed correctly to the AgentExecutor (issue #10045). The solution in that case was to pass the handle_parsing_errors parameter using the agent_executor_kwargs argument, like so:

create_pandas_dataframe_agent(
...,
agent_executor_kwargs={"handle_parsing_errors": True}
)

However, this solution might not work in your case since you're trying to pass a function instead of a boolean.

I recommend opening a new issue in the LangChain repository to report this bug. Please include the details you've provided here, as well as the error message you're seeing. The maintainers of the repository should be able to help you resolve this issue.

I hope this helps, and I apologize for the inconvenience.

Sources

• create_pandas_dataframe_agent does not pass the handle_parsing_error variable into the underlying Agent.
• docs/extras/modules/agents/how_to/handle_parsing_errors.ipynb
• libs/langchain/langchain/agents/initialize.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[necarlson97]

Apologies! It seems this resulted from an error in my dev environment that resulted in my machine using langchain-0.0.159, even though pip reported 0.0.301.
After reinstalling 0.0.301, this appears to now be working. Thanks!