Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Running custom workflows in evaluation #2414

Open
snip2D opened this issue Jan 10, 2025 · 2 comments
Open

[Bug] Running custom workflows in evaluation #2414

snip2D opened this issue Jan 10, 2025 · 2 comments
Assignees

Comments

@snip2D
Copy link

snip2D commented Jan 10, 2025

Describe the bug
I customized a variant through the SDK and generated a test set in the playground. When I perform an evaluation, I encounter the following error message:
Traceback (most recent call last):
File "/app/agenta_backend/tasks/evaluations.py", line 297, in evaluate
type="text", value=app_output.result.value["data"]
KeyError: 'data'

variant project code
from pydantic import BaseModel, Field
import agenta as ag

class MyConfig(BaseModel):
prompt: str = Field(default="nihao")

@ag.route("/", config_schema=MyConfig)
def generate_text(input_text: str):
config = ag.ConfigManager.get_from_route(schema=MyConfig)
# Function implementation

output_text = f"input: {input_text} it's good! config: {config.prompt}"
return output_text

evaluate config
1736494701493


@dosubot dosubot bot added the bug Something isn't working label Jan 10, 2025
@mmabrouk mmabrouk added unconfirmed bug and removed bug Something isn't working labels Jan 10, 2025
@mmabrouk mmabrouk self-assigned this Jan 10, 2025
@Agenta-AI Agenta-AI deleted a comment from dosubot bot Jan 10, 2025
@mmabrouk
Copy link
Member

Hello @snip2D ,

The code you are using looks good to me. I will try to reproduce the error. Can you please make sure to upgrade agenta locally to the latest version:
pip install -U agenta

then check which version are you using before serving the app again:

agenta --version
agenta variant serve app.py

Please let me know if that does not resolve the issue.

If the case persist:

  • Can you please share with me the response of the application that you have created when called from the playground (you can go to the developer tools in your browser, and check the response to the request POST /generate).
  • Can you use the application in the playground, or the issue is solely within evaluation.

As soon as we have that, and the problem is still unresolved, I will try to reproduce it from our side.

@mmabrouk mmabrouk changed the title [Bug] [Bug] Running custom workflows in evaluation Jan 12, 2025
@snip2D
Copy link
Author

snip2D commented Jan 16, 2025

Hi @mmabrouk I checked my agenta version, which was 0.29.0, and then I upgraded it to 0.31.0. After running the original code, a new error occurred:
1736997897749

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants