withStructuredOutput is not a function on ChatGoogleGenerativeAI

[yharaskrik]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

const genAi = new ChatGoogleGenerativeAI({});

console.log(genAi.withStructuredOutput); <-- yields undefined, calling it on the call results in below

Error Message and Stack Trace (if applicable)

}).withStructuredOutput(schema);
   ^

TypeError: (intermediate value).withStructuredOutput is not a function

Description

The ChatGoogleGenerativeAI class has the withStructuredOutput function on it, but it is undefined. If it is not implemented for this model then it should not be on the types.

        "@langchain/community": "^0.2.0",
        "@langchain/core": "^0.2.0",
        "@langchain/google-genai": "^0.0.14",
        "@langchain/openai": "^0.0.33",
        "@langchain/redis": "^0.0.5",
        "langchain": "^0.2.0",

System Info

Mac
yarn 1.22.7
node 18.20.2

[dosubot]

Hey there, @yharaskrik! 👋 I'm Dosu, a bot here to lend a hand with bugs, answer questions, and guide you through becoming a contributor while we wait for a human maintainer to join us. Think of me as your friendly digital sidekick in navigating the world of LangChain JS. I'm diving into your issue now and will get back to you with a full answer shortly. Hang tight! 🤖

[yharaskrik]

Oh I see the withStructuredOutput function is optional on the abstract BaseLanguageModel class. My IDE was not reflecting that, so the ChatGoogleGenerativeAI is not implementing it and that is why it is undefined.

[bracesproul]

The withStructuredOutput isn't implemented (or normal tool calling for that matter) in the @langchain/google-genai package. It's on my list to implement both sometime this week though! I'll use this issue as a tracker for when that goes in.

[yharaskrik]

Amazing thank you Brace! That's pretty much what I determined as well. I went to try and do it myself but only got so far as copying the code from the common package for those functions. I'm sure there's a better way just not familiar with the structure of all the packages! Thanks!

…

On Mon, May 20, 2024, 11:11 AM Brace Sproul ***@***.***> wrote: The withStructuredOutput isn't implemented (or normal tool calling for that matter) in the @langchain/google-genai package. It's on my list to implement both sometime this week though! I'll use this issue as a tracker for when that goes in. — Reply to this email directly, view it on GitHub <#5484 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIHZITXNRW2OJXSVPZHXNTZDI4GNAVCNFSM6AAAAABIAD455GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQHE2TIMJZGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bracesproul]

g

That's how I was planning on going about it as well! If you want to take a stab at it, I wouldn't start until tmro. So if you can put up a PR before tmro morning I would love to use your implementation!

[dosubot]

To implement the withStructuredOutput function in the @langchain/google-genai package, follow these steps:

1. Fork and Clone the Repository: Fork the langchainjs repository to your account and clone it locally.

2. Set Up Your Development Environment: Ensure Node.js (v18+) is installed. Run yarn install in the repository's root to install dependencies.

3. Create a New Branch: Use git checkout -b feature/withStructuredOutput to create a branch for your changes.

4. Implement withStructuredOutput:

   • Navigate to langchain-core/src/language_models.
   • Update or define the withStructuredOutput function in the relevant TypeScript files. Follow the project's coding standards and ensure compatibility with existing patterns.

5. Build and Test Your Changes: Run yarn build to build the package and yarn test to ensure your changes don't break existing functionality. Add new tests if necessary.

6. Commit and Push Your Changes: After testing, commit your changes and push them to your fork.

7. Open a Pull Request (PR): From your fork, open a PR to the main langchainjs repository. Describe your changes and their motivation clearly.

8. Review Process: Engage with any review feedback to refine your contribution.

This approach will help you contribute effectively to resolving the TypeError issue with withStructuredOutput in the @langchain/google-genai package.

Details

Sources

• OpenAI Agent returning structured output has type errors
• libs/langchain-google-genai/package.json
• libs/langchain-google-common/src/chat_models.ts
• langchain-core/src/language_models/base.ts
• libs/langchain-openai/src/chat_models.ts
• langchain/src/chains/openai_functions/structured_output.ts
• langchain/src/chains/openai_functions/tests/structured_output.test.ts
• langchain/src/chains/openai_functions/base.ts
• langchain/src/chains/openai_functions/tests/structured_output.int.test.ts
• docs/core_docs/docs/how_to/structured_output.ipynb
• libs/langchain-openai/src/tests/chat_models_structured_output.int.test.ts
• libs/langchain-groq/src/tests/chat_models_structured_output.int.test.ts
• libs/langchain-anthropic/src/tests/chat_models.test.ts
• libs/langchain-google-genai/src/chat_models.ts
• examples/src/models/chat/googlegenerativeai.ts
• libs/langchain-anthropic/src/experimental/tool_calling.ts
• libs/langchain-google-genai/README.md

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[bitsnaps]

The same applies for huggingface inference API, I'm getting the error TypeError: llm.withStructuredOutput is not a function even with models supporting function calling.

[jason-allen-oneal]

same with groq

[pndtr]

Are there anyone is facing the issue that using withStructuredOutput not calling tool

[Moe03]

same with me sometimes works sometimes doesn't on groq

[sstraus]

Same on Ollama