Support for local models via ollama

[ViktorAgafonov]

Add (and add documents) how the add-on works with models installed locally via the ollama endpoint (default http://localhost:11434/v1) by overriding the model (does it overlap with adding other models?).

[ViktorAgafonov]

I'll add. When composing the endpoint string from the settings, it would be good to bring it to the proper form, taking into account double slash characters, so as not to display “Request URI: http://localhost:11434//v1/chat/completions” in the output logs.

It's strange (something I don't understand) but the "add summary" command works, but the "explain" command doesn't work.

AddSummary

28.10.2024 22:46:36: Request URI: http://localhost:11434/v1/chat/completions
28.10.2024 22:46:36: Request Method: POST
28.10.2024 22:46:36: Request Headers:
28.10.2024 22:46:36: Accept-Encoding: gzip, deflate
28.10.2024 22:46:36: Connection: keep-alive
28.10.2024 22:46:36: Authorization: Bearer pai-0af1b253-0250-479e2bf2290d
28.10.2024 22:46:36: api-key: pai-0af1b253-0250-479e2bf2290d
28.10.2024 22:46:36: User-Agent: okgodoit/dotnet_openai_api
28.10.2024 22:46:36: Request Content: {"model":"gemma","messages":[{"role":"system","content":"Only write a comment as C# summary format like\r\n\r\n/// <summary>\r\n/// \r\n/// </summary>\r\n\r\n\r\nfor"},{"role":"user","content":"Commands.WipeCreateSeed(false);\r\nCommands.ListAll();"}],"temperature":0.0,"top_p":0.0,"n":1,"stream":false,"stop":["public","private","internal"],"max_tokens":1024,"frequency_penalty":0.0,"presence_penalty":0.0}
28.10.2024 22:46:36: ____________________________________________________________________________________________________
28.10.2024 22:46:44: Response Headers:
28.10.2024 22:46:44: Date: Mon, 28 Oct 2024 19:46:44 GMT
28.10.2024 22:46:44: Response Status Code: OK
28.10.2024 22:46:44: Response Content: {"id":"chatcmpl-573","object":"chat.completion","created":1730144804,"model":"gemma","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"/// \u003csummary\u003e\r\n/// Wipes the existing seed and creates a new one, then lists all available commands.\r\n/// \u003c/summary\u003e"},"finish_reason":"stop"}],"usage":{"prompt_tokens":55,"completion_tokens":29,"total_tokens":84}}

Explain:

28.10.2024 22:50:29: Request URI: http://localhost:11434/v1/chat/completions
28.10.2024 22:50:29: Request Method: POST
28.10.2024 22:50:29: Request Headers:
28.10.2024 22:50:29: Accept-Encoding: gzip, deflate
28.10.2024 22:50:29: Connection: keep-alive
28.10.2024 22:50:29: Authorization: Bearer pai-0af1b253-0250-479e2bf2290d
28.10.2024 22:50:29: api-key: pai-0af1b253-0250-479e2bf2290d
28.10.2024 22:50:29: User-Agent: okgodoit/dotnet_openai_api
28.10.2024 22:50:29: Request Content: {"model":"gemma","messages":[{"role":"system","content":"Explain"},{"role":"user","content":"Commands.WipeCreateSeed(false);\r\nCommands.ListAll();"}],"temperature":0.0,"top_p":0.0,"n":1,"stream":true,"stop":"","max_tokens":1024,"frequency_penalty":0.0,"presence_penalty":0.0}
28.10.2024 22:50:29: ____________________________________________________________________________________________________
28.10.2024 22:50:32: Response Headers:
28.10.2024 22:50:32: Transfer-Encoding: chunked
28.10.2024 22:50:32: Date: Mon, 28 Oct 2024 19:50:32 GMT
28.10.2024 22:50:32: Response Status Code: OK
28.10.2024 22:50:32: Response Content: data: {"id":"chatcmpl-210","object":"chat.completion.chunk","created":1730145032,"model":"gemma","system_fingerprint":"fp_ollama","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":"stop"}]}

[jeffdapaz]

Hi @ViktorAgafonov,

First of all, it's worth mentioning that I've never tested the integration of the extension with LLama myself, but I know it works because I've already received positive feedback about it.

However, the idea is simple: you just need to have an LLM implementation, whatever it may be, whether local or not, but that is compatible with the OpenAI specification, and it will probably work.

To make it work, just do what you already did, override the URL and the model.
The predefined models in the options are those available by OpenAI/Azure, but if you want to go beyond those options, just define the model through the "Model Override" parameter.

"It's strange (something I don't understand) but the "add summary" command works, but the "explain" command doesn't work.":

It’s probably related to LLama and its implementation, but I can’t help much since, like I said, I’ve never tested LLama myself. However, you can try changing the command. If you look at the log, you’ll see that for this command, it’s just sending "Explain," and maybe LLama didn’t quite understand what needs to be done. You could try changing that command in the extension options to something more detailed, like "Explain this code" or something like that, and test different commands until you find one that works well.

[ViktorAgafonov]

Maybe this will help somehow. I don't know what to do with this:
https://github.com/awaescher/OllamaSharp

[jeffdapaz]

@ViktorAgafonov,

The project you mentioned is a library designed to make it easier to integrate .NET applications with the Ollama API. Instead of having to create methods like HTTPClient and entities, etc., on your own, you just add this library to your development application, and you'll have everything ready to go.

My extension also uses a similar library, but it's specific to the OpenAI and Azure OpenAI APIs. However, since the Llama API is compatible with the OpenAI API, this library can also be used with Llama, as well as other APIs that are compatible with OpenAI.