Is the SqlToolkit agent toolkit only supported for OpenAI LLMs?

[mikebars]

I see that in the SqlToolkit constructor it creates a QueryCheckerTool without arguments:

• https://github.com/hwchase17/langchainjs/blob/main/langchain/src/agents/agent_toolkits/sql/sql.ts#L29

Which causes the else branch of the QueryCheckerTool constructor to run:

• https://github.com/hwchase17/langchainjs/blob/main/langchain/src/agents/tools/sql.ts#L104

Which then creates the default OpenAI LLMChain

Here's a reproduction of the issue (attempting to use the HuggingFaceInference LLM):

• https://gist.github.com/mikebars/3c0fad5f9e3cd1cedd31e3fc2e489ae4

And here is a patch I made that appears to allow the functionality to progress further:

• https://gist.github.com/mikebars/c4aca3595c03e1da1ba71e3b0e4edd53

And here is my patched code that is able to reach a different error state:

• https://gist.github.com/mikebars/64951dd57e771bc3a46fec961e117866

However, I still get an error as output, which makes me think perhaps these cases are not yet supported:

Error: Could not parse LLM output: Action: list-tables-sql
Action Input:
Observation: a list of tables
Th
    at ZeroShotAgent.extractToolAndInput (file:///home/user/langchain-projects/projects/langchain-test/node_modules/langchain/src/agents/mrkl/index.ts:111:13)
    at ZeroShotAgent._plan (file:///home/user/langchain-projects/projects/langchain-test/node_modules/langchain/src/agents/agent.ts:159:31)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AgentExecutor._call (file:///home/user/langchain-projects/projects/langchain-test/node_modules/langchain/src/agents/executor.ts:75:22)
    at async AgentExecutor.call (file:///home/user/langchain-projects/projects/langchain-test/node_modules/langchain/src/chains/base.ts:94:22)

[hkd987]

I’ve been running into the same issues as you and came to mostly the same conclusions as you. I’ve found this comment on the langchain repo that makes me think a very custom fine tuning is going to be be needed to get a SQL agent with a huggingface model to work the way I want.

[hkd987]

I’m actually thinking of using OpenAI SQL Agent questions/answers and building a fine tuning data set, that I can then plug into a Huggingface model.

If you have any other ideas or find a better solution I’d love to hear/talk about it.

[dosubot]

Hi, @mikebars! I'm here to help the LangChain team manage their backlog and I wanted to let you know that we are marking this issue as stale.

From what I understand, you reported an issue with the SqlToolkit agent toolkit in the langchainjs repository. It seems that when using the HuggingFaceInference LLMChain with the toolkit, you encountered an error. Another user, @hkd987, also faced similar issues and suggested that a custom fine-tuning may be needed to get a SQL agent with a HuggingFace model to work. They mentioned considering using OpenAI SQL Agent questions/answers and building a fine-tuning dataset.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding and we look forward to your response.