Curious omissions

[webby2001]

One of my many uses for the plugin is to remind me if I've already written about something, which generally takes the form of "Have I written about 'x'?) But I am finding that the accuracy of this is spotty - some terms I have only used once get returned, while others with more mentions generate nothing. I have force-refreshed (and am a supporter). What might I be missing?

[webby2001]

As a follow-up, I asked the chat if it was using all of my notes. It responded, "Based on your notes, I have access to all 20 of them." I have a lot more than 20! I don't seem to have any exclusions in the settings, and I am using the 3.5-turbo-16K model

[quelsemme]

I'm interested to know what Brian's response to this is... In my use cases, it has access to more than 20... I wonder if there is a straight forward way to test.

[brianpetro]

One of the two buttons in the bottom right of the AI response's chat bubble copies the notes that were used context to the clipboard. This is one way you can begin to investigate what is being passed to GPT.

For most models in v1, 20 note snippets is the upper limit of what's sent to GPT. The v1 retrieval strategy is known as HyDE, and is how the note snippets are chosen.

The v1 retrieval method is rather rudimentary and will be completely replaced in v2.1 (v2.0 is focused on the embeddings implementation) and complemented by function calls where the GPT can choose to further query notes.

I hope that helps clear things up!

🌴

[quelsemme]

The tip for clicking on Quite interesting and useful to know.

As someone with hundreds of notes across a field (PhD student), I want to be able to find everything I've written about. That makes me think that my discussions with Smart Chat should be as precise as possible, to get it to come back with info from a small series of notes. What's interesting is, I asked it what I should include in my thesis and it used 30 notes to give me an answer, but a lot of them were based on potential thesis outlines I had written, which makes sense for accuracy, but reminds me I should exclude all drafts and outlines from embeddings.

I've seen some references to v2 in the last week of updates - is there a public release date in mind?

[quelsemme]

I've been playing around with it and find again that it's retrieving 30 notes, and looking at what those notes are is helping me hone what gets processed as embeddings and what doesn't...

For my vault, this means excluding the following folders:

• INBOX - this contains notes and citations which I haven't yet parsed or written up in my own language - it is relevant information, but for my purposes is far too general for getting a response on my notes and not the literature I have read.
• PROJECT NOTES - This contains outlines and MOCs. When they are included in the embeddings, the Smart Chat sees it as a source of information. The problem there is, if I ask it how to outline an essay that I have already written an outline for, it will repeat the existing outline back instead of giving me new suggestions.

I have no doubt as I continue to tinker and grow my vault, I'll look at other methods of exclusion.

I think there is also room for methods of 'inclusion'

That is, while I may have a note full of people and a title for how they are related to one another, the embedding would be more accurate if the note includes a description of why they are connected and how they might be useful within the vault, in regards to helping Smart Connections be precise.

I do wonder if there is a better way than the button for revealing source notes (also because 'Copy prompt to clipboard' doesn't describe what that button does, which is why I never clicked it - I hadn't considered it was the full prompt being set to ChatGPT, I just assumed it was my original prompt.

In the description of how the plugin works, it could be interesting to add a description of the process you've mentioned before.

For example (steal if you wish)

How does Smart Chat retrieve information on my notes?

The Smart Connections plug-in uses a retrieval strategy known as HyDE. Put simply, Smart Connections takes your prompt and uses embeddings to decide the top 20-30 notes in your vault that would provide an accurate response to the question. It then feeds these chosen notes to the OpenAI GPT model as a large, amalgamated prompt and returns an answer based on that information.

If you ask questions on a topic which can be found across more than those 20 or 30 notes, the response may be less comprehensive. It is better to be as specific as possible with your questions, so that the HyDE process can feed only the most relevant notes to the GPT AI.

Thanks for everything Brian. This is by far my favourite Obsidian plug-in and it is immensely helpful. I'll be doing a full video about using it for PhD research when I have some time.

[brianpetro]

@quelsemme thank you for the immensely helpful feedback. I already added that snippet to the README 😊

Re: 'inclusion' features, try prefacing your chat message with the folder(s) to include using a forward-slash / followed by the name of the folder(s). Typing a forward slash should also bring up a display for easy folder selection.

Re: v2: a very-much-still-in-beta version was just recently made available to supporters, however, based on your messages you might want to wait a bit to upgrade. The v2.0 (current) updates are focused on implementing local embeddings and improving the software architecture. The main reason people would want to update to v2.0 would be to prevent notes from being sent to OpenAI by using the local embeddings. Version 2.1 is where I plan to make significant updates to the Smart Chat and retrieval methods, which are the features I expect you will find most helpful.

No exact ETA for v2.1 yet. But things have been moving along nicely with v2.0, and once that stabilizes, I'll move that from supporter-beta to public, and move v2.1 into supporter-beta.

One chat improvement I intend to publish with v2.1 is "note tools for chat": so, in addition to the HyDE-type retrieval (a sort of pre-processing), "note tools for chat" will allow the GPT to subsequently query your notes during its output, similar to how ChatGPT uses tools like web browsing, code interpreter, Dall-e, etc. This would allow the GPT to retrieve additional context as needed using both the embeddings and traditional search queries.

I hope that helps clarify some things! Thanks again for your help in making Smart Connections the best it can be 🤓

🌴