At the time of writing this post the current official KoboldAI branch is outdated and behind in model support.
Want to run the latest models? Want to avoid large downloads and installations?
Check out KoboldCpp, our GGUF based solution.
Need a model for KoboldCpp?
- Go to Huggingface and look for GGUF models if you want the GGUF for a specific model search for a part of the name of your model followed by GGUF to find GGUF releases.
- Go the files tab and pick the file size that best fits your hardware, Q4_K_S is a good balance.
- Click the small download icon right to the filename to download your GGUF.
- Load the GGUF in KoboldCpp, you can now use the AI.
KoboldCpp is available for Windows, Linux and ARM MacOS
(Files attached below are automatically posted by Github and do not work, please use the link above to obtain the release).