Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add local LLAMA2 support #7

Open
danmohad opened this issue Oct 9, 2023 · 0 comments
Open

Add local LLAMA2 support #7

danmohad opened this issue Oct 9, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@danmohad
Copy link
Owner

danmohad commented Oct 9, 2023

The Interface class handles all back-end interaction with the LLM's API. The base class is currently written specifically to interact with the OpenAI API. It should be reasonably straightforward to inherit from this and create an interface to arbitrary LLM backends. In particular, using llama.cpp it should be possible to make an interface to LLAMA2 that runs on the local GPU (or CPU if none is available).

This approach could then be extended to various other LLMs that are able to run inference locally.

One question is how this interface would interact with the GPU through khod-kaar's Docker environment. That may require some digging into and hacking of llama.cpp.

Having the ability to run khod-kaar locally and independently of any outside parties would be a very powerful feature.

@danmohad danmohad added the enhancement New feature or request label Oct 9, 2023
@danmohad danmohad changed the title Add LLAMA2 support Add local LLAMA2 support Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant