Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set top_k to 5 in SAS to be consistent #2550

Merged
merged 2 commits into from
May 16, 2022

Conversation

ClaMnc
Copy link
Contributor

@ClaMnc ClaMnc commented May 14, 2022

-The change refers to the top_k value of the retriever in the Advanced Evaluation Metrics in the Tutorial5_Evaluation.py.

The actual value is set to 1 which is both suboptimal and inconsistent with all the previous utilized values of top_k for the retriever (equal to 5). Trying the script with other values leads to better SAS results. For example, setting it to 5 (to be consistent with the previous values) results in 0.528 compared to 0.443 using top_k = 1.

@Timoeller Timoeller self-requested a review May 16, 2022 07:53
Copy link
Contributor

@Timoeller Timoeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the change. This configuration really does not make much sense here.

Since we have two Tutorial versions (python script and jupyter notebook) could you also change the other one, please?

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@Timoeller Timoeller self-requested a review May 16, 2022 08:28
Copy link
Contributor

@Timoeller Timoeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG

@julian-risch
Copy link
Member

julian-risch commented May 16, 2022

@ClaMnc congratulations to your first small PR in Haystack! 🎉 Looking forward to more contributions from you!
Small note on that: If you mention closes #2549 in the description of your pull request, it will automatically close the issue once the pull request is merged. Otherwise one of our team members might try to understand and solve the issue when you already have a pull request open. 🙂 Thank you @ZanSara for illustrating that here. 😉
@Timoeller please also not forget to add a label before merging a PR. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants