Confusing error message when `llm_config` is not initialized or is `None`. #1522

ekzhu · 2024-02-03T08:04:06Z

Is your feature request related to a problem? Please describe.

Right now, if an agent's llm_config is set to None (default in ConversableAgent and AssistantAgent), or if config_list is empty.

from autogen import UserProxyAgent, AssistantAgent

# assistant = AssistantAgent("assistant", llm_config={"config_list": []})
assistant = AssistantAgent("assistant")
proxy = UserProxyAgent("user")

proxy.initiate_chat(assistant)

We get confusing error message from the openai client:

TypeError: Missing required arguments; Expected either ('messages' and 'model') or ('messages', 'model' and 'stream') arguments to be given

Describe the solution you'd like

We should raise an error early if the llm_config is None or config_list is empty, as it is guaranteed to lead to error as the model is not specified.

Alternatively, we should set llm_config by default to false in ConversableAgent and make llm_config a required argument in AssistantAgent. As the current default None is always going to cause error.

Additional context

The text was updated successfully, but these errors were encountered:

afourney · 2024-02-04T08:10:09Z

Yeah, this is another case where None or empty is taken as a positive (in this case that you want to use an LLM)

Code_execution_config was the same until late last week.

At one point the default was to assume gpt-4 and read the key from the OPENAI_API_KEY. So some users setting llm_config to None and getting billed for GPT-4 use... This error is at least better than that!

ekzhu · 2024-02-12T04:57:04Z

@gunnarku here you go. You can investigate based on the PR description and see what is a good way to fix this.

Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None`. - The `llm_config` is valid, but `config_list` is missing or lacks elements. - The `config_list` is valid, but no `model` is specified. The rest of the changes are code churn to adjust or add the test cases.

Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases.

Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None`. - The `llm_config` is valid, but `config_list` is missing or lacks elements. - The `config_list` is valid, but no `model` is specified. The rest of the changes are code churn to adjust or add the test cases.

Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases.

Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None`. - The `llm_config` is valid, but `config_list` is missing or lacks elements. - The `config_list` is valid, but no `model` is specified. The rest of the changes are code churn to adjust or add the test cases.

Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases.

Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None`. - The `llm_config` is valid, but `config_list` is missing or lacks elements. - The `config_list` is valid, but no `model` is specified. The rest of the changes are code churn to adjust or add the test cases.

Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases.

* Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None`. - The `llm_config` is valid, but `config_list` is missing or lacks elements. - The `config_list` is valid, but no `model` is specified. The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Fix the test_web_surfer issue For anyone reading this: you need to `pip install markdownify` for the `import WebSurferAgent` to succeed. That is needed to run the `test_web_surfer.py` locally. Test logic needs `llm_config` that is not `None` and that is not `False`. Let us pray that this works as part of GitHub actions ... * One more fix for llm_config validation contract

* Logging (#1146) * WIP:logging * serialize request, response and client * Fixed code formatting. * Updated to use a global package, and added some test cases. Still very-much a draft. * Update work in progress. * adding cost * log new agent * update log_completion test in test_agent_telemetry * tests * fix formatting * Added additional telemetry for wrappers and clients. * WIP: add test for oai client and oai wrapper table * update test_telemetry * fix format * More tests, update doc and clean up * small fix for session id - moved to start_logging and return from start_logging * update start_logging type to return str, add notebook to demonstrate use of telemetry * add ability to get log dataframe * precommit formatting fixes * formatting fix * Remove pandas dependency from telemetry and only use in notebook * formatting fixes * log query exceptions * fix formatting * fix ci * fix comment - add notebook link in doc and fix groupchat serialization * small fix * do not serialize Agent * formatting * wip * fix test * serialization bug fix for soc moderator * fix test and clean up * wip: add version table * fix test * fix test * fix test * make the logging interface more general and fix client model logging * fix format * fix formatting and tests * fix * fix comment * Renaming telemetry to logging * update notebook * update doc * formatting * formatting and clean up * fix doc * fix link and title * fix notebook format and fix comment * format * try fixing agent test and update migration guide * fix link * debug print * debug * format * add back tests * fix tests --------- Co-authored-by: Adam Fourney <[email protected]> Co-authored-by: Victor Dibia <[email protected]> Co-authored-by: Chi Wang <[email protected]> * Validate llm_config passed to ConversableAgent (issue #1522) (#1654) * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None`. - The `llm_config` is valid, but `config_list` is missing or lacks elements. - The `config_list` is valid, but no `model` is specified. The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on #1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Fix the test_web_surfer issue For anyone reading this: you need to `pip install markdownify` for the `import WebSurferAgent` to succeed. That is needed to run the `test_web_surfer.py` locally. Test logic needs `llm_config` that is not `None` and that is not `False`. Let us pray that this works as part of GitHub actions ... * One more fix for llm_config validation contract * update test_utils * remove stale files --------- Co-authored-by: Adam Fourney <[email protected]> Co-authored-by: Victor Dibia <[email protected]> Co-authored-by: Chi Wang <[email protected]> Co-authored-by: Gunnar Kudrjavets <[email protected]>

…microsoft#1654) * Validate llm_config passed to ConversableAgent Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None`. - The `llm_config` is valid, but `config_list` is missing or lacks elements. - The `config_list` is valid, but no `model` is specified. The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Validate llm_config passed to ConversableAgent Based on microsoft#1522, this commit implements the additional validation checks in `ConversableAgent.` Add the following validation and `raise ValueError` if: - The `llm_config` is `None` (validated in `ConversableAgent`). - The `llm_config` has no `model` specified and `config_list` is empty (validated in `OpenAIWrapper`). - The `config_list` has at least one entry, but not all the entries have the `model` is specified (validated in `OpenAIWrapper`). The rest of the changes are code churn to adjust or add the test cases. * Fix the test_web_surfer issue For anyone reading this: you need to `pip install markdownify` for the `import WebSurferAgent` to succeed. That is needed to run the `test_web_surfer.py` locally. Test logic needs `llm_config` that is not `None` and that is not `False`. Let us pray that this works as part of GitHub actions ... * One more fix for llm_config validation contract

ekzhu added dev good first issue Good for newcomers labels Feb 3, 2024

ekzhu mentioned this issue Feb 4, 2024

New initiate_chats Interface for Managing Dependent Chats in ConversableAgent #1402

Merged

6 tasks

ekzhu mentioned this issue Feb 12, 2024

Validate the OpenAI API key format #1635

Merged

1 task

ekzhu assigned ekzhu and unassigned ekzhu Feb 12, 2024

gunnarku mentioned this issue Feb 12, 2024

Validate llm_config passed to ConversableAgent (issue #1522) #1654

Merged

1 task

sonichi closed this as completed in #1654 Feb 15, 2024

sonichi mentioned this issue Feb 15, 2024

do model check properly #1686

Merged

3 tasks

gunnarku mentioned this issue Mar 10, 2024

Autogen llm_config error #1863

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Confusing error message when `llm_config` is not initialized or is `None`. #1522

Confusing error message when `llm_config` is not initialized or is `None`. #1522

ekzhu commented Feb 3, 2024 •

edited

Loading

afourney commented Feb 4, 2024

ekzhu commented Feb 12, 2024 •

edited

Loading

Confusing error message when llm_config is not initialized or is None. #1522

Confusing error message when llm_config is not initialized or is None. #1522

Comments

ekzhu commented Feb 3, 2024 • edited Loading

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Additional context

afourney commented Feb 4, 2024

ekzhu commented Feb 12, 2024 • edited Loading

Confusing error message when `llm_config` is not initialized or is `None`. #1522

Confusing error message when `llm_config` is not initialized or is `None`. #1522

ekzhu commented Feb 3, 2024 •

edited

Loading

ekzhu commented Feb 12, 2024 •

edited

Loading