-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update LFQA with the latest LFQA seq2seq and retriever models #2210
Conversation
Guess we have three options: a) Keep documentation as is, b) Update documentation to use new LFQA, c) showcase both old and new LFQA in one notebook. Although this last one might be cool from my perspective, maybe it would be breaking Haystack's pattern of small, focused documentation examples. LMK @TuanaCelik |
Good question. Let me check with @brandenchan while I am also understanding how we would ideally do these things 👍🏽 |
Thanks for waiting @vblagoje Here's how we suggest to proceed:
Branden went and had a look at our documentation and there's actually no need for change there because we don't specify which model it uses. So just the tutorial :) Hope this helps! Let me know if there's anything else |
Ok thanks @TuanaCelik When you say LFQA tutorial, what exactly are you referring to? |
A bit more change needed - we need to use DPR encoders instead of EmbeddingReetriever |
Hey @vblagoje , yes you're right. You can go ahead with changing the EmbeddingRetreiver to DPR and make the changes so that the LFQA models are used by default (e.g. when you change the EmbeddingRetriever and also the generator to use the new lfqa so yjernite/bart_eli5" to "vblagoje/bart_lfqa") If that and the updates from the previous comment are complete we will make sure on our side to include the relevant info in our documentation here |
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Just a note: please ignore the merge conflicts for docs and json-schemas. Tag me when you are ready to merge and I'll take care of them 🙂 |
@ZanSara this was ready to merge two weeks ago. Apologies, should have been more explicit about it. |
@vblagoje Great, thanks for the contribution! |
Enables the use of the new LFQA model and its DPR-based context and question encoders.
The old LFQA architecture continues to be supported.
Haystack users can choose which architecture to choose (old vs. new LFQA)
Mix/match old/new retriever with old/new seq2seq generator is also possible
The new LFQA architecture has almost 2x better KILT retriever scores on Wikipedia-like text.
Empirically the grounding is better in the new architecture (no concrete metrics available yet)
Motivation for the new LFQA architecture is detailed in this article
Note - do not merge before we update documentation. What would you guys like the new LFQA docs to look like? We can, for example, showcase both architectures in a single notebook? Feedback and suggestions are appreciated.
Status (please check what you already did):