You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Interface class handles all back-end interaction with the LLM's API. The base class is currently written specifically to interact with the OpenAI API. It should be reasonably straightforward to inherit from this and create an interface to arbitrary LLM backends. In particular, using llama.cpp it should be possible to make an interface to LLAMA2 that runs on the local GPU (or CPU if none is available).
This approach could then be extended to various other LLMs that are able to run inference locally.
One question is how this interface would interact with the GPU through khod-kaar's Docker environment. That may require some digging into and hacking of llama.cpp.
Having the ability to run khod-kaar locally and independently of any outside parties would be a very powerful feature.
The text was updated successfully, but these errors were encountered:
The
Interface
class handles all back-end interaction with the LLM's API. The base class is currently written specifically to interact with the OpenAI API. It should be reasonably straightforward to inherit from this and create an interface to arbitrary LLM backends. In particular, using llama.cpp it should be possible to make an interface to LLAMA2 that runs on the local GPU (or CPU if none is available).This approach could then be extended to various other LLMs that are able to run inference locally.
One question is how this interface would interact with the GPU through
khod-kaar
's Docker environment. That may require some digging into and hacking of llama.cpp.Having the ability to run
khod-kaar
locally and independently of any outside parties would be a very powerful feature.The text was updated successfully, but these errors were encountered: