fix: Use use_auth_token in all cases when loading from the HF Hub

[sjrl]

Related Issues

• fixes the issue of not being able to load private models from HF for various models throughout Haystack (e.g. FARMReader and Retriever)

Proposed Changes:

I've passed the use_auth_token to the HuggingFace methods that accept it (e.g. from_petrained() and pipeline).

How did you test it?

I was able to load a private Reader model from HF after the changes were made

from haystack.nodes import FARMReader
reader = FARMReader(model_name_or_path="private/model", use_gpu=True, use_auth_token=True)

Notes for the reviewer

use_auth_token is not consistently used throughout Haystack. This PR tries to use use_auth_token consistently in all places in Haystack when loading models from the HF hub.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added tests that demonstrate the correct behavior of the change
• I've used the conventional commit convention for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

[julian-risch]

Looks very good to me already. Just some small remarks to further improve. Further, I was wondering whether we could add a test case to make sure a problem with auth tokens is recognized faster in future. The auth token could be stored as a GitHub secret.

[ZanSara]

What a change! That's no surprise no one wanted to do this before 😄 Looks good to me, except for the pipes, that need to be reverted. @vblagoje do you want to have a look too?

[sjrl]

Further, I was wondering whether we could add a test case to make sure a problem with auth tokens is recognized faster in future. The auth token could be stored as a GitHub secret.

We could definitely do that. However, to completely test this I think we would need to have a private HF model for all cases we want to test (e.g. Reader, Retriever, Summarizer, AnswerGenerator, etc.).

[sjrl]

Further, I was wondering whether we could add a test case to make sure a problem with auth tokens is recognized faster in future. The auth token could be stored as a GitHub secret.

We could definitely do that. However, to completely test this I think we would need to have a private HF model for all cases we want to test (e.g. Reader, Retriever, Summarizer, AnswerGenerator, etc.).

@julian-risch Is it alright if we open this as a new issue? It may be worth discussing how we would like to best test this across Haystack.

[julian-risch]

Looks good to me now. 👍