Option for ConversationalChatAgent to place format instruction and tools within System Message for ChatGPT 4.0

[arnabbiswas1]

Issue you'd like to raise.

ConversationalChatAgent places FORMAT_INSTRUCTIONS as well as tools section within human message, not within system message.

This is fine as long as ChatGPT-3.5 is being used where LLM gives more weightage to human message than system message. But, that is not the case for ChatGPT-4 where System message has higher weightage than human message.

In fact, with ChatGPT-4, when FORMAT_INSTRUCTIONS is embedded within the human message, the output generated not always follow the format. However, the vice versa is not true.

IMHO, the [JSONDecodeError] (#3455) may vanish for ChatGPT-4 if we place the format instruction inside the System message.

Suggestion:

I have been able to modify the implementation inside langchain.agents.conversational_chat (below), but I guess it would be better if we change the langchain API as well.

On top of my change, I had to use the changes from this PR to handle the cases, when GPT4 is not responding with the json format.

from typing import Sequence, Optional, List, Any

from pydantic import Field

from langchain.tools.base import BaseTool

from langchain.agents import ConversationalChatAgent
from langchain.schema import (
    BaseOutputParser,
)
from langchain.prompts.base import BasePromptTemplate
from langchain.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    MessagesPlaceholder,
    SystemMessagePromptTemplate,
)
from langchain.agents.agent import Agent, AgentOutputParser

class GPT4ConversationalChatAgent(ConversationalChatAgent):

    @classmethod
    def create_prompt(
        cls,
        tools: Sequence[BaseTool],
        system_message: str = PREFIX,
        human_message: str = SUFFIX,
        input_variables: Optional[List[str]] = None,
        output_parser: Optional[BaseOutputParser] = None,
    ) -> BasePromptTemplate:
        tool_strings = "\n".join(
            [f"> {tool.name}: {tool.description}" for tool in tools]
        )
        tool_names = ", ".join([tool.name for tool in tools])
        _output_parser = output_parser or cls._get_default_output_parser()
        format_instructions = system_message.format(
            format_instructions=_output_parser.get_format_instructions()
        )
        final_system_prompt = format_instructions.format(
            tool_names=tool_names, tools=tool_strings
        )

        if input_variables is None:
            input_variables = ["input", "chat_history", "agent_scratchpad"]
        messages = [
            SystemMessagePromptTemplate.from_template(final_system_prompt),
            MessagesPlaceholder(variable_name="chat_history"),
            HumanMessagePromptTemplate.from_template(human_message),
            MessagesPlaceholder(variable_name="agent_scratchpad"),
        ]
        return ChatPromptTemplate(input_variables=input_variables, messages=messages)

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

agent = GPT4ConversationalChatAgent.from_llm_and_tools(
        llm=llm,
        tools=tools,
        system_message=PREFIX,
        human_message=SUFFIX,
    )

agent_chain = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)

[treppers]

I ended up heavily modifying the conversationalchat agent which resulted in creating a new agent all together. I moved the format instructions into the system message as suggested as well as created a third voice known as TOOL. Here's the PR to add the agent.

This agent was specifically designed to have the bot speak in a "character's voice" but with some quick tweaks it could work as a vanilla chat agent.

[dosubot]

Hi, @arnabbiswas1! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue you reported was about the ConversationalChatAgent in ChatGPT 4.0 placing format instructions and tools within the human message instead of the system message, causing inconsistencies in the output. However, treppers has resolved this issue by creating a new agent that modifies the ConversationalChatAgent. The format instructions have been moved to the system message, and a third voice called TOOL has been added. The PR for this change has received positive reactions from you and LGHTNNG.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution and understanding. Let us know if you have any further questions or concerns!