Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add BLIP support in TransformersImageToText #4912

Merged
merged 3 commits into from
May 16, 2023
Merged

feat: add BLIP support in TransformersImageToText #4912

merged 3 commits into from
May 16, 2023

Conversation

anakin87
Copy link
Member

In Hugging Face Transformers, support for BLIP in the image-to-text pipeline has recently been added (huggingface/transformers#21904).

BLIP is a powerful model, so this PR is intended to bring this improvement to Haystack.

Proposed Changes:

  • extend the list of supported models, including BLIP
    (this list is still needed because some models that can be loaded via HF pipeline raise errors)
  • make Salesforce/blip-image-captioning-base the default model: it is way better than nlpconnect/vit-gpt2-image-captioning and is about the same size
  • Remove a test that is now irreproducible in the sense that it originally was (due to changes in implementations in HF transformers)

How did you test it?

Existing tests in CI.

Other related issues

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added tests that demonstrate the correct behavior of the change
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@anakin87 anakin87 requested a review from a team as a code owner May 14, 2023 16:55
@anakin87 anakin87 requested review from silvanocerza and removed request for a team May 14, 2023 16:55
@coveralls
Copy link
Collaborator

coveralls commented May 14, 2023

Coverage Status

Coverage: 37.893% (+0.1%) from 37.792% when pulling 150fe00 on anakin87:blip_support into 6eb251d on deepset-ai:main.

Copy link
Contributor

@silvanocerza silvanocerza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good, just a small nitpick in the docstring.

Co-authored-by: Silvano Cerza <[email protected]>
@anakin87
Copy link
Member Author

@silvanocerza feel free to merge 😊

@silvanocerza silvanocerza merged commit 6e00007 into deepset-ai:main May 16, 2023
@anakin87 anakin87 deleted the blip_support branch May 16, 2023 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants