Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multimodal example #6313

Closed
mscheong01 opened this issue Mar 26, 2024 · 2 comments
Closed

Add multimodal example #6313

mscheong01 opened this issue Mar 26, 2024 · 2 comments
Labels
enhancement New feature or request llava LLaVa and multimodal

Comments

@mscheong01
Copy link
Collaborator

Feature Description

Add example for multimodal capabilities

Motivation

#5882 took out the multimodal features from the server. Given it's a highly requested feature, our plan would be to reintroduce it at some point (#6168). How about we set up a solid multimodal example elsewhere and then port it to the server example later on?

Possible Implementation

Implementation based on the removed code from https://github.com/ggerganov/llama.cpp/pull/5882/files which had already implemented this feature in the server.cpp example, hopefully with some performance optimization.
For the example, image file could be provided via command line option.

@mscheong01 mscheong01 added the enhancement New feature or request label Mar 26, 2024
@mscheong01 mscheong01 changed the title add multimodal example Add multimodal example Mar 26, 2024
@phymbert
Copy link
Collaborator

@phymbert phymbert added the llava LLaVa and multimodal label Mar 26, 2024
@mscheong01
Copy link
Collaborator Author

@phymbert Ah, I see, I guess llava-cli fits my description for a solid multimodal example. Thanks for pointing it out 🙇‍♂️

@mscheong01 mscheong01 closed this as not planned Won't fix, can't repro, duplicate, stale Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request llava LLaVa and multimodal
Projects
None yet
Development

No branches or pull requests

2 participants