Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Evaluator respond with Nan values as the first token is a text #3522

Closed
deepakas opened this issue Jul 9, 2024 · 3 comments
Closed
Assignees
Labels
bug Something isn't working no-recent-activity There has been no recent activity on this issue/pull request

Comments

@deepakas
Copy link

deepakas commented Jul 9, 2024

Describe the bug
Similarity Evaluator responds with a response Nan as first token in response is 'Text" and promptflow is not able to convert text to integer.

How To Reproduce the bug
Steps to reproduce the behavior, how frequent can you experience the bug:

  1. Calling promptflow evaluate method passing json file as input. Since it is customer data we can't share the exact data.

Expected behavior
A clear and concise description of what you expected to happen.

It should respond with a number 1 to 5 as per the prompty - https://github.com/microsoft/promptflow/blob/main/src/promptflow-evals/promptflow/evals/evaluators/_similarity/similarity.prompty

Screenshots
If applicable, add screenshots to help explain your problem.

Running Information(please complete the following information):

  • Promptflow Package Version using pf -v: [e.g. 0.0.102309906] :
    {
    "promptflow": "1.12.0",
    "promptflow-azure": "1.13.0",
    "promptflow-core": "1.12.0",
    "promptflow-devkit": "1.12.0",
    "promptflow-evals": "0.3.0",
    "promptflow-tracing": "1.12.0"
    }
  • Operating System: [e.g. Ubuntu 20.04, Windows 11] : Windows 11
  • Python Version using python --version: [e.g. python==3.10.12]
    Python (Windows) 3.11.9 (tags/v3.11.9:de54cf5, Apr 2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]

Additional context
Add any other context about the problem here.

@deepakas deepakas added the bug Something isn't working label Jul 9, 2024
@luigiw
Copy link
Member

luigiw commented Jul 12, 2024

@deepakas Can you share the model version you used? This is usually cased by the use of preview OpenAI models.

Copy link

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

@github-actions github-actions bot added the no-recent-activity There has been no recent activity on this issue/pull request label Aug 11, 2024
@luigiw
Copy link
Member

luigiw commented Aug 15, 2024

Fixed in 0.3.2 version.

@luigiw luigiw closed this as completed Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working no-recent-activity There has been no recent activity on this issue/pull request
Projects
None yet
Development

No branches or pull requests

3 participants