Question about the inner workings of embeddings and constructing context for chat

[abhuva]

I have 3 questions that, atleast for me, arent completely clear:

1. i guess the search functionality is purely using the embeddings and therefor doesnt cost tokens (beside the initial cost to calculate the embedding) - in what way is making new notes triggering new api calls for embeddings?
   Is it automatic, can i specify certain intervals to check - is there even a way currently to see how many of my notes have those embeddings?

2. about the chat: i guess it is working like this: based on the question there is a search call (locally with the help of embeddings), then from this search result the context gets constructed and this gets send via api to openai.
   Is this in general correct? If yes, is there a way to finetune in what way the context get constructed? Like how many tokens get used for context vs answer (this is especially interesting for using gpt4 - the big benefit here is the way bigger token amount / context window - but so far i see no way to control this)

3. this is more of a feature request i guess, based on my testings it seems that the chat currently is only context aware for a single question - this works surprisingly well, but as it appears to be a chat, my assumption is of course that its also in some way context aware of my previous questions (and the answers). This seems not to be the case right now.
   I guess this could be an easy fix (maybe as a toggle) - during the construction of the context, just include parts (or all) of the question/answers from the chat --> this way it would feel more natural to chat while still preserving the whole "search through embeddings first and use this to construct the context/prompt" --> again, here it would be interesting to be able to have more control over what amount of tokens are used for (chat-context / search-context / answer)

[brianpetro]

1. Turn on the log_render settings (see screenshot below) and check your console log (CTRL+SHIFT+I).
2. Currently, the ratio is usually higher on the context side. There are ways to improve this, and many configurations are possible. But I think providing a few "presets" would be best instead of exposing every possible setting. So I'd like to know if you have any thoughts on how it would work better for you. This will help me better design options that you might consider fine-tuning the context surfacing process.
3. When looking up context from your notes, only your most recent message is included in the prompt. I know this is less than ideal, and it's quite trivial to fix. But I am still looking for feedback about how much of the chat history should be included. Should we limit the context from your notes to prioritize chat history? Vice versa.
   

This plugin, especially this Smart Chat feature, is still a work-in-progress. So feedback like yours above, and whatever else you can tell me, will play a huge part in future decisions and further development. I appreciate your help!

Thanks,
Brian🌴

[abhuva]

About 2) : i like the way TextGenerator plugin is doing this, in the settings you can specify the (average) amount of tokens reserved for the answer - this is set to something reasonable on default, so people who dont want to tinker with it dont have to. So lets say i use GPT3.5 with a 4k token limit, and i specify 700 for the answer in the settings, this means the plugin can use up to 3.3k tokens for the context.
I like this aproach as it gives me the option to generate a very long answer (from little context) or mid to short answers from high amount of context - and this all with just one additional variable in the settings.

About 3) I can envision this the same like for answer 2) : Have a toggle in the settings that lets you decide wether or not to include chat history and maybe a second variable for the amount of tokens reserved for chat history.
While for the new user this is most likely advanced settings they wont bother touching, it could help finetune the experience for more experienced users while not beeing overly complicated (just 2 variables and a toggle in the settings for a high amount of control over what and to what amount is considered context.

EDIT: just realized that you could get away without the toggle, by setting the token-amount for chat-history to 0 by default.
Less options / clutter in the settings is always good!

Beside these suggestions, i just have to say thanks for the work so far! I was testing DUAL a couple years ago (similar idea, different execution, using gpt2 at this time and finetuning the model and running locally) - the tech wasnt ready to be really useful.
But now its really there and integrations like yours are fantastic to see.

[brezl8]

i personally would like if we could just adapt all possible variables, starting where its simple to implement.
nothing super fancy, just within the settings, or in a settings tab within the chat.

things like system prompts, number of chat history sent as context, api variables, etc.

maybe some more creative things like a simple check box next to the chat messages for inclusion in context.

or a pane (list) with notes to select (checkbox) to include in the context.

or chat windows instead of one unified smart chat window. that would be powerful with the aforementioned features as settings within the chat window.

i think obsidian users are generally more interested in high customizability and most of the users have some sort of technical background.

like this you can have some basic standard settings but also have high customizability for everyone who wants to do something more specific.

i think this is where llm's as well as obsidian really shine. really doubling down on that might be the right approach.

high customizability open up a myriad of use cases.

also users can share their setups and workflows here or in the obsidian forums, which would be awesome.

i cannot code yet, else I'd be helping. keep up the great work, i think this could easily become the greatest obsidian plugin of all times! 🐐✨

[brianpetro]

Thank you for your suggestions and kind words! I really appreciate your feedback.

I agree that having more customization options would be beneficial for users who want to fine-tune their experience. Your ideas about adjusting variables in the settings, including chat history as context, and selecting specific notes to include in the context are all great suggestions. I will definitely take them into consideration as I continue to develop the plugin.

I also agree that Obsidian users value high customizability, and I believe that providing a balance between standard settings and advanced customization options would be the best approach. This way, users can have a seamless experience with default settings, while also having the flexibility to tailor the plugin to their specific needs.

Thank you for your support, and please don't hesitate to reach out if you have any more questions or suggestions!

Brian🌴