-
Notifications
You must be signed in to change notification settings - Fork 27.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CLIP resources #26534
Add CLIP resources #26534
Conversation
LGTM! Thanks for adding resources for CLIP. |
Thank you for reviewing!! I fixed it! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your contribution! 🤗
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for contributing resources! It's very helpful, however, a lot of resources here are a bit random.
- assigning the "text-to-image" pipeline tag to CLIP is a bit weird since CLIP is only a text encoder, vision encoder model. It can't be used for generating images using text. It is however used as a building block in Stable Diffusion for instance to condition the model on text prompts.
- the "deploy" section includes several resources which are not about deployment. Deployment is about optimizing the model for inference, using tools like ONNX, 🤗 Optimum, quantization, etc.
- the "inference" section includes a notebook about... explainability. Then it makes more sense to include this in a separate "explainability" section. Inference is about showcasing predictions with the model
Some relevant resources for CLIP include:
- fine-tuning CLIP => HF provides a training script for this: https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text.
- fine-tuning CLIP on additional (image, text) pairs: https://huggingface.co/blog/fine-tune-clip-rsicd.
Co-authored-by: Steven Liu <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing, can you also include the training script and blog post @NielsRogge linked to?
Co-authored-by: Steven Liu <[email protected]>
Sorry for the late reply. It seems to appear in 86 and 87 lines. Do you want me to change the description to something else? |
Should be good then! 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few more minor nits sorry! 😅
Co-authored-by: Steven Liu <[email protected]>
* docs: feat: model resources for CLIP * fix: resolve suggestion Co-authored-by: Steven Liu <[email protected]> * fix: resolve suggestion * fix: resolve suggestion * fix: resolve suggestion Co-authored-by: Steven Liu <[email protected]> * fix: resolve suggestion * fix: resolve suggestions Co-authored-by: Steven Liu <[email protected]> --------- Co-authored-by: Steven Liu <[email protected]>
* docs: feat: model resources for CLIP * fix: resolve suggestion Co-authored-by: Steven Liu <[email protected]> * fix: resolve suggestion * fix: resolve suggestion * fix: resolve suggestion Co-authored-by: Steven Liu <[email protected]> * fix: resolve suggestion * fix: resolve suggestions Co-authored-by: Steven Liu <[email protected]> --------- Co-authored-by: Steven Liu <[email protected]>
* docs: feat: model resources for CLIP * fix: resolve suggestion Co-authored-by: Steven Liu <[email protected]> * fix: resolve suggestion * fix: resolve suggestion * fix: resolve suggestion Co-authored-by: Steven Liu <[email protected]> * fix: resolve suggestion * fix: resolve suggestions Co-authored-by: Steven Liu <[email protected]> --------- Co-authored-by: Steven Liu <[email protected]>
What does this PR do?
Adds resources of CLIP according to this issue
Part of #20055
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@stevhliu, @jungnerd, @wonhyeongseo may you please review this PR?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.