Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

接入自制OCR新工具时出现的问题 #356

Open
SatoriEye opened this issue Jul 9, 2024 · 12 comments
Open

接入自制OCR新工具时出现的问题 #356

SatoriEye opened this issue Jul 9, 2024 · 12 comments
Assignees
@sijunhe

Comments

@SatoriEye
Copy link

我在使用贵平台时出现了pydantic的问题

具体报错如下：
Traceback (most recent call last): File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 26, in <module> class CustomOCRInput(ToolParameterView): File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 200, in __new__ set_model_fields(cls, bases, config_wrapper, types_namespace) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 469, in set_model_fields fields, class_vars = collect_model_fields(cls, bases, config_wrapper, types_namespace, typevars_map=typevars_map) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_fields.py", line 132, in collect_model_fields type_hints = get_cls_type_hints_lenient(cls, types_namespace) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_typing_extra.py", line 228, in get_cls_type_hints_lenient hints[name] = eval_type_lenient(value, globalns, localns) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_typing_extra.py", line 240, in eval_type_lenient return eval_type_backport(value, globalns, localns) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_typing_extra.py", line 264, in eval_type_backport return typing._eval_type( # type: ignore File "/home/aistudio/external-libraries/agent/lib/python3.10/typing.py", line 327, in _eval_type return t._evaluate(globalns, localns, recursive_guard) File "/home/aistudio/external-libraries/agent/lib/python3.10/typing.py", line 693, in _evaluate type_ = _type_check( File "/home/aistudio/external-libraries/agent/lib/python3.10/typing.py", line 176, in _type_check raise TypeError(f"{msg} Got {arg!r:.100}.") TypeError: Forward references must evaluate to types. Got <module 'PIL.Image' from '/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/PIL/I. (/home/aistudio/external-libraries/agent) aistudio@jupyter-2163331-8134354:~/deploy/llm_agent/api/common_tools$ python ocr_tools.py Traceback (most recent call last): File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 7, in <module> from erniebot_agent.file import LocalFile ImportError: cannot import name 'LocalFile' from 'erniebot_agent.file' (/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/file/__init__.py) (/home/aistudio/external-libraries/agent) aistudio@jupyter-2163331-8134354:~/deploy/llm_agent/api/common_tools$ python ocr_tools.py Traceback (most recent call last): File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 26, in <module> class CustomOCRInput(ToolParameterView): File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 205, in __new__ complete_model_class( File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 534, in complete_model_class schema = cls.__get_pydantic_core_schema__(cls, handler) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/main.py", line 642, in __get_pydantic_core_schema__ return handler(source) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in __call__ schema = self._handler(source_type) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 512, in generate_schema schema = self._generate_schema_inner(obj) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 784, in _generate_schema_inner return self._model_schema(obj) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 591, in _model_schema {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()}, File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 591, in <dictcomp> {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()}, File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 947, in _generate_md_field_schema common_field = self._common_field_schema(name, field_info, decorators) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 1134, in _common_field_schema schema = self._apply_annotations( File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 1890, in _apply_annotations schema = get_inner_schema(source_type) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in __call__ schema = self._handler(source_type) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 1871, in inner_handler schema = self._generate_schema_inner(obj) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 789, in _generate_schema_inner return self.match_type(obj) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 875, in match_type return self._unknown_type_schema(obj) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 415, in _unknown_type_schema raise PydanticSchemaGenerationError( **pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'erniebot_agent.file.local_file.LocalFile'>. Set arbitrary_types_allowed=Truein the model_config to ignore this error or implementget_pydantic_core_schema` on your type to fully support it.

If you got this error by calling handler() within __get_pydantic_core_schema__ then you likely need to call handler.generate_schema(<some type>) since we do not call __get_pydantic_core_schema__ on <some type> otherwise to avoid infinite recursion.

For further information visit https://errors.pydantic.dev/2.8/u/schema-for-unknown-type**

`

相关代码如下（具体的token与部分request逻辑我已经隐藏，应该不会影响到问题的判断）
`from future import annotations

import requests
from typing import Any, Dict, Type, List
from pydantic import Field
from erniebot_agent.tools.base import Tool
from erniebot_agent.file.local_file import LocalFile
from erniebot_agent.tools.schema import ToolParameterView
from erniebot_agent.agents.function_agent import FunctionAgent
from erniebot_agent.chat_models import ERNIEBot
from erniebot_agent.memory import WholeMemory
from erniebot_agent.file import GlobalFileManagerHandler

def image_path_to_base64(img):
#转换图像为base64
...

class CustomOCRInput(ToolParameterView):
img: LocalFile = Field(description="待识别的图像")

class CustomOCROutput(ToolParameterView):
result: str = Field(description="识别的文字结果")

class CustomOCRTool(Tool):

description: str = "识别输入图像中的文字"
input_type: Type[ToolParameterView] = CustomOCRInput
output_type: Type[ToolParameterView] = CustomOCROutput

def __init__(self) -> None:
    super().__init__()

async def __call__(self, img: File) -> Dict[str, Any]:
    url = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    data = image_path_to_base64(img)

    # 发送 POST 请求
    data = {
        'img': data,
    }
    response = requests.post(url, data=data)

    # 处理返回的图像数据
    result = ""
    if response.status_code == 200:
        reply = response.json()
        for block in reply:
            result += block["transcription"]
            result += "\n"

    return {"result": result}

import asyncio
import os

async def a():
os.environ["EB_AGENT_ACCESS_TOKEN"] = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
file_manager = GlobalFileManagerHandler().get()
agent = FunctionAgent(ERNIEBot("ernie-3.5"), tools=[CustomOCRTool()], memory=WholeMemory())
data = await file_manager.create_file_from_path(file_path="/home/aistudio/deploy/llm_agent/api/img_12.jpg", file_type="local")
result = await agent.run("告诉我图片上讲了什么？", files=[data])
print(result)

asyncio.run(a())

`

相关环境如下：
平台为百度飞浆AI Studio提供的BML CodeLab配置（CPU）。

下面是我个人的一点看法：
从报错上来说，似乎是pydantic试图对LocalFile类做一点处理（我不是很了解这个），然后该类有一些方法没实现导致出现了问题？
同时 GlobalFileManagerHandler().get()与FunctionAgent(ERNIEBot("ernie-3.5"), tools=[CustomOCRTool()], memory=WholeMemory())
去掉了原教程中的await参数，因为也会报错（判断为GlobalFileManagerHandler().get()用不到异步，去掉后没有报错）

我尝试过声明arbitrary_types_allowed=True在相关的工具Input子类中
但是没有效果，会出现新的错误：
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 195, in run_llm raise e File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 192, in run_llm llm_resp = await self._run_llm(messages, **(llm_opts or {})) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 324, in _run_llm functions = self._tool_manager.get_tool_schemas() File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/tool_manager.py", line 73, in get_tool_schemas return [tool.function_call_schema() for tool in self._tools.values()] File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/tool_manager.py", line 73, in <listcomp> return [tool.function_call_schema() for tool in self._tools.values()] File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/base.py", line 80, in function_call_schema inputs["parameters"] = self.input_type.function_call_schema() File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 339, in function_call_schema return cls.to_openapi_dict() File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 321, in to_openapi_dict properties[field_name] = dict(get_field_openapi_property(field_info)) File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 181, in get_field_openapi_property elif field_info.annotation is not None and issubclass(field_info.annotation, Enum): TypeError: issubclass() arg 1 must be a class
上面的堆栈报错中个人认为比较重要的部分

代码如下：
class CustomOCRInput(ToolParameterView): class Config: arbitrary_types_allowed = True local_file: 'erniebot_agent.file.local_file.LocalFile' img: LocalFile = Field(description="待识别的图像")

我已经黔驴技穷了，希望能够有大佬能够高抬贵手帮个忙，感谢感谢！！！
@SatoriEye
Copy link
Author

貌似github代码块不能很好的处理代码与报错显示？我再重新发看看。

刚开始的堆栈报错：

Traceback (most recent call last):
File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 78, in
asyncio.run(a())
File "/home/aistudio/external-libraries/agent/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/aistudio/external-libraries/agent/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 75, in a
result = await agent.run("告诉我图片上讲了什么？", files=[data])
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 137, in run
raise e
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 134, in run
agent_resp = await self._run(prompt, files)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/function_agent.py", line 163, in _run
curr_step, new_messages = await self._step(chat_history)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/function_agent.py", line 208, in _step
llm_resp = await self.run_llm(messages=input_messages)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 195, in run_llm
raise e
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 192, in run_llm
llm_resp = await self._run_llm(messages, **(llm_opts or {}))
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 324, in _run_llm
functions = self._tool_manager.get_tool_schemas()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/tool_manager.py", line 73, in get_tool_schemas
return [tool.function_call_schema() for tool in self._tools.values()]
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/tool_manager.py", line 73, in
return [tool.function_call_schema() for tool in self._tools.values()]
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/base.py", line 80, in function_call_schema
inputs["parameters"] = self.input_type.function_call_schema()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 339, in function_call_schema
return cls.to_openapi_dict()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 321, in to_openapi_dict
properties[field_name] = dict(get_field_openapi_property(field_info))
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 181, in get_field_openapi_property
elif field_info.annotation is not None and issubclass(field_info.annotation, Enum):
TypeError: issubclass() arg 1 must be a class
(/home/aistudio/external-libraries/agent) aistudio@jupyter-2163331-8134354:~/deploy/llm_agent/api/common_tools$ python ocr_tools.py
Traceback (most recent call last):
File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 26, in
class CustomOCRInput(ToolParameterView):
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 205, in new
complete_model_class(
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 534, in complete_model_class
schema = cls.get_pydantic_core_schema(cls, handler)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/main.py", line 642, in get_pydantic_core_schema
return handler(source)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in call
schema = self._handler(source_type)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 512, in generate_schema
schema = self._generate_schema_inner(obj)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 784, in _generate_schema_inner
return self._model_schema(obj)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 591, in _model_schema
{k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 591, in
{k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 947, in _generate_md_field_schema
common_field = self._common_field_schema(name, field_info, decorators)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 1134, in _common_field_schema
schema = self._apply_annotations(
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 1890, in _apply_annotations
schema = get_inner_schema(source_type)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in call
schema = self._handler(source_type)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 1871, in inner_handler
schema = self._generate_schema_inner(obj)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 789, in _generate_schema_inner
return self.match_type(obj)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 875, in match_type
return self._unknown_type_schema(obj)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 415, in _unknown_type_schema
raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'erniebot_agent.file.local_file.LocalFile'>. Set arbitrary_types_allowed=True in the model_config to ignore this error or implement __get_pydantic_core_schema__ on your type to fully support it.

If you got this error by calling handler() within __get_pydantic_core_schema__ then you likely need to call handler.generate_schema(<some type>) since we do not call __get_pydantic_core_schema__ on <some type> otherwise to avoid infinite recursion.

后面尝试修复时出现的堆栈报错：

Traceback (most recent call last):
File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 78, in
asyncio.run(a())
File "/home/aistudio/external-libraries/agent/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/aistudio/external-libraries/agent/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 75, in a
result = await agent.run("告诉我图片上讲了什么？", files=[data])
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 137, in run
raise e
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 134, in run
agent_resp = await self._run(prompt, files)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/function_agent.py", line 163, in _run
curr_step, new_messages = await self._step(chat_history)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/function_agent.py", line 208, in _step
llm_resp = await self.run_llm(messages=input_messages)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 195, in run_llm
raise e
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 192, in run_llm
llm_resp = await self._run_llm(messages, **(llm_opts or {}))
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 324, in _run_llm
functions = self._tool_manager.get_tool_schemas()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/tool_manager.py", line 73, in get_tool_schemas
return [tool.function_call_schema() for tool in self._tools.values()]
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/tool_manager.py", line 73, in
return [tool.function_call_schema() for tool in self._tools.values()]
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/base.py", line 80, in function_call_schema
inputs["parameters"] = self.input_type.function_call_schema()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 339, in function_call_schema
return cls.to_openapi_dict()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 321, in to_openapi_dict
properties[field_name] = dict(get_field_openapi_property(field_info))
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 181, in get_field_openapi_property
elif field_info.annotation is not None and issubclass(field_info.annotation, Enum):
TypeError: issubclass() arg 1 must be a class

@SatoriEye
Copy link
Author

这个是最早的代码：

from future import annotations

import requests
from typing import Any, Dict, Type, List
from pydantic import Field
from erniebot_agent.tools.base import Tool
from erniebot_agent.file.local_file import LocalFile
from erniebot_agent.tools.schema import ToolParameterView
from erniebot_agent.agents.function_agent import FunctionAgent
from erniebot_agent.chat_models import ERNIEBot
from erniebot_agent.memory import WholeMemory
from erniebot_agent.file import GlobalFileManagerHandler

def image_path_to_base64(img):
#转换图像为base64
...

class CustomOCRInput(ToolParameterView):
img: LocalFile = Field(description="待识别的图像")

class CustomOCROutput(ToolParameterView):
result: str = Field(description="识别的文字结果")

class CustomOCRTool(Tool):

description: str = "识别输入图像中的文字"
input_type: Type[ToolParameterView] = CustomOCRInput
output_type: Type[ToolParameterView] = CustomOCROutput

def __init__(self) -> None:
    super().__init__()

async def __call__(self, img: File) -> Dict[str, Any]:
    url = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    data = image_path_to_base64(img)

    # 发送 POST 请求
    data = {
        'img': data,
    }
    response = requests.post(url, data=data)

    # 处理返回的图像数据
    result = ""
    if response.status_code == 200:
        reply = response.json()
        for block in reply:
            result += block["transcription"]
            result += "\n"

    return {"result": result}

import asyncio
import os

async def a():
os.environ["EB_AGENT_ACCESS_TOKEN"] = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
file_manager = GlobalFileManagerHandler().get()
agent = FunctionAgent(ERNIEBot("ernie-3.5"), tools=[CustomOCRTool()], memory=WholeMemory())
result = await agent.run("告诉我图片上讲了什么？", files=[file_manager.create_file_from_path(file_path="road_sign.jpeg", file_type="local")])
print(result)

asyncio.run(a())

@SatoriEye
Copy link
Author

目前的情况：
我试着让pydantic不处理这个类而是直接调用，即：

class CustomOCRInput(ToolParameterView):
class Config:
arbitrary_types_allowed = True
local_file: 'erniebot_agent.file.local_file.LocalFile'
img: 'erniebot_agent.file.local_file.LocalFile' = Field(description="待识别的图像")

报错：

Traceback (most recent call last):
File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 78, in
asyncio.run(a())
File "/home/aistudio/external-libraries/agent/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/aistudio/external-libraries/agent/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/aistudio/deploy/llm_agent/api/common_tools/ocr_tools.py", line 75, in a
result = await agent.run("告诉我图片上讲了什么？", files=[data])
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 137, in run
raise e
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 134, in run
agent_resp = await self._run(prompt, files)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/function_agent.py", line 163, in _run
curr_step, new_messages = await self._step(chat_history)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/function_agent.py", line 208, in _step
llm_resp = await self.run_llm(messages=input_messages)
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 195, in run_llm
raise e
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 192, in run_llm
llm_resp = await self._run_llm(messages, **(llm_opts or {}))
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/agents/agent.py", line 324, in _run_llm
functions = self._tool_manager.get_tool_schemas()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/tool_manager.py", line 73, in get_tool_schemas
return [tool.function_call_schema() for tool in self._tools.values()]
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/tool_manager.py", line 73, in
return [tool.function_call_schema() for tool in self._tools.values()]
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/base.py", line 80, in function_call_schema
inputs["parameters"] = self.input_type.function_call_schema()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 339, in function_call_schema
return cls.to_openapi_dict()
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 321, in to_openapi_dict
properties[field_name] = dict(get_field_openapi_property(field_info))
File "/home/aistudio/external-libraries/agent/lib/python3.10/site-packages/erniebot_agent/tools/schema.py", line 181, in get_field_openapi_property
elif field_info.annotation is not None and issubclass(field_info.annotation, Enum):
TypeError: issubclass() arg 1 must be a class
(/home/aistudio/external-libraries/agent) aistudi

现在摆在我脸上的有三个选项：

  1. 不使用贵平台的工具链，使用其他方法（目前最现实的一个）
  2. 自制Agent实现（但感觉也有问题，因为任何文件都会经过File类打包，因此这个逻辑绕不过去）
  3. 强行修改环境内包的源代码（我不熟悉pydant这个包，也不大行）

我太菜了，确是已经没办法了。

@SatoriEye
Copy link
Author

防沉，菜狗一只，希望有大佬捞捞

@sijunhe
Copy link
Collaborator

sijunhe commented Jul 10, 2024

hello. 报错的原因是CustomOCRInput 的 img 的type不应该是 LocalFile. 这个img你是希望一个文件路径吗？ 还是希望模型填入一个创建好了的LocalFile?

@SatoriEye
Copy link
Author

SatoriEye commented Jul 10, 2024 via email

@lyyilin
Copy link

lyyilin commented Aug 15, 2024

请问您解决了吗？

@lyyilin
Copy link

lyyilin commented Aug 15, 2024

我好像解了！！！！！！！芜湖。

@SatoriEye
Copy link
Author

我这里没有解决，方便请问您的解决方案吗？

@lyyilin
Copy link

lyyilin commented Aug 31, 2024

我这里没有解决，方便请问您的解决方案吗？

这是我开发的飞桨项目链接：https://aistudio.baidu.com/projectdetail/8226582?contributionType=1，你可以参考一下，我是图片生成，整体和你的很像，你可以看看。

@lyyilin
Copy link

lyyilin commented Aug 31, 2024

我这里没有解决，方便请问您的解决方案吗？

我这个是跑通了的，希望对你有帮助。

@SatoriEye
Copy link
Author

感谢您的解决方案！我想这应该能工作

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sijunhe sijunhe

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sijunhe @lyyilin @SatoriEye