Compute embeddings using local server

[matttrent]

I came across the Smart Connections plugin last week and have been enjoying exploring my vault in a new way. I upgraded to v2.1 and was able to connect it to my local Ollama server and get to test local chat with the new Llama 3 model. It's quite impressive.

With the new local server support in v2.1 for chat, would it be possible to use the same local server for generating embeddings? I'd prefer to keep everything local, and the default CPU-based embedding generation is pretty slow and locks up my Obsidian instance for some time.

I'd love to use the nomic-embed-text model for embeddings, and have it run fast via GPU inference in Ollama. It seems like all the parts are implemented to enable this, but aren't connected in code or available in settings. Any chance this could be added as an option?

[brianpetro]

@matttrent thanks for bringing this up.

There was a recent feature request about this that can be found here #559

Please make sure to comment on that feature request so that I can better prioritize this!

Thanks for participating in the Smart Connections community
🌴

[matttrent]

Done!

[Luke2791]

Please forgive me if I am missing something or beating a dead horse here...I am relatively new to the LLM/GPT world.
Is there any official (or unofficial) documentation on how to connect Ollama (or another local LLM) with Smart Connections? Downloading and setting up llama3.1, for example, from Ollama is well documented. I have done this and am able to interface with it from within terminal on my Mac. However, I am not aware of the correct settings I need for fields such as "Model Name" , "protocol", "hostname", "port", "path", "streaming", or "max input tokens". Is there anyway you could add this to the Readme page or have this up front @brianpetro ?

Given Obsidian's secure reputation and user base, I would suspect that many users would be interested in running local models vs web-based models (like ChatGPT). Perhaps I have misunderstood something somewhere - I am just wanting to guard my data/notes, and feel much better running a local LLM wherever possible. Thanks for all your hard work!

[Reaper176]

The model would be something like " llama3.1:latest " or what ever ollama model you download. THIS NEEDS TO BE IN THE READ ME!

The protocol would be http or https depending on how you are connecting to ollama. THIS NEEDS TO BE IN THE READ ME!

Hostname is the ip address that you are connecting to. I should be something like 127.0.0.1 if it is locally hosted on the same machine. THIS NEEDS TO BE IN THE READ ME!

Port is the port number that ollama is running on. By default it is 11434. This probably should be in the read me.

Path....this is the annoying one that i cant find the answer for. i think it should be something like " /api/chat " but i am not sure. THE ANSWER TO THIS NEEDS TO BE IN THE READ ME!!!!!!

Streaming and max input tokens are things you would google to see if you need to change.

THESE THINGS NEED TO BE IN THE READ ME!

The read me....the read me....da read me.....

[Luke2791]

Thanks, @Reaper176 . I believe the data we/you are looking for can be found in another one of discussion forums. Brian has done an excellent job supporting the plug in so far. But I agree, it would benefit the users to have this in the readme. @brianpetro , would you mind adding something like this (the list below) to the read me, please?

Custom Local Model
Model Name: llama3.1 (or mistral, gemma2, etc.)
Protocol: http (https not really necessary for local)
Hostname: localhost (or perhaps local IP address?)
Port: 11434 (11434 for Ollama. 1234 for LM Studio, I think...)
Path: /api/chat (Works for Ollama - unsure about LM Studio...)
Streaming: Varies - Try Both
Max Input Tokens: 2048 (or 4096, 8192, etc)