The chat doesn't seem to use any of my notes when I ask a question

[dmatt]

I can't seem to get the chat to work. It seems to pull in some notes as reference but then does not use them and claims to not be able to answer my question.

I've tried general prompts which should be answerable like:

• what have I written about [keyword I know I've written about]
• what have I written about recently?
• tell me about myself based on my writings

I'm using custom local model llama3:70b-instruct-q4_0 installed with ollama, running the plugin version 2.1.86.

Here you can see it pulled in some notes for "context" but then did not use them. Why?

[dmatt]

I meant to add: is there something else I should include from a configuration/settings/debug file that would help debug why the chat is only partially working and the context is not used?

[jwhco]

For new content, I've got to prime the model by ensuring Smart Connection Files loads. If it doesn't, then the index isn't up to date. If you open your key source note, and then click the Files it will start embedding. Once that exists, then direct reference is useful to dial in context of response.

[dmatt]

Thanks @jwhco — I think I'm following what you are saying but I still can't get context to be used in the chat. Here's what I tried:

• go to the Files sidebar in Obsidian
• open several recent files/notes
• for each one click on the "Smart Connections Files" icon
• confirm that many "connections" are loaded within the plugin's smart connection sidebar
• then try a generic question like "what have I written about recently?"
• result: it doesn't use any notes as context, despite saying that it is loading some into context. expected: it should be specific about things I've written about in the notes it is pulling into context.

[jwhco]

Something else must be wrong. I followed your steps. The first time, with the same prompt, it broke when an error message appeared. Then I submitted it again, and within a few seconds, this came back.

It's not entirely accurate. This morning I was working on a design for a shipping container datacenter, yesterday a book on establishing credibility, and the previous day on a client project about neuro-feedback therapy.

My vault contains articles from my newsletter that include improving sleep, memory, and reducing stress. But it is NOT a dominant topic in the 22k notes. However, from a last modified perspective, I did touch on the topic a few days ago.

[jwhco]

I dug a little deeper. The prompt lacks the context to provide an accurate response but should have provided something. The only file referencing floating chamber is highlighted in the image below. (Green rectangle.)

Notice that the modified file in this chunk only had minor metadata changes. (Yellow rectangle.) That change was part of a cosmetic change shown below.

When writing up details about a client's neurofeedback device, I modified many files in a different vault area. This plug-in must be pulling context from everywhere. Those modifications were made on 6/21/2024, specifically in two files.

Try this prompt:

Please list the most recent modified files, cluster key ideas and provide a summary. I'd like to see a summary of what I've been writing about for the last 90 days. General topics in a list.

Oddly enough, it tells me it cannot look at file modification dates then repeats what it gave me before.

I've been getting a lot of repeats lately, but this prompt has specific context and tells me it doesn't know when things are modified. Are you getting anything from specific prompts, or does it gives you the same kind of response?

[Garrett3Nelson]

@jwhco or @dmatt did either of you figure out how to fix this? I'm running a local implementation of Gemma2 with settings that appear to be correct, based on getting actual responses:

I'm getting the same issue as described though:

However if I change from local to Open Router Gemma2:9b and just copy/paste the prompt again, it works perfectly fine.

Given this isn't an active issue for many others, I assume I've set up my local model wrong. I did notice this in the console, but I can't find where it would be looking for an API key:

Do you have any advice?

[jwhco]

The prompt we are having trouble with asks about recent notes in general, while your prompt has a specific note reference that asks for a summary. You're right to suspect an error with a local model setup.

I would try the same prompt with a local model and OpenRouter access. Choose the most straightforward configuration while checking the local modern outside of Smart Connections.

API key issues are odd because you got a summary in the chat response. Try the one of the prompts above before you change anything. They API key could be something else.

[Garrett3Nelson]

I wonder if there is something specific in Smart Connections that I'm missing. With a generic prompt like yours above ("Tell me about my recent note activity") I got the following:

However at the same time I'm using Continue for VSCode with the same model and it's providing context and responses properly.

[brianpetro]

The Smart Chat in Obsidian is currently limited to a single semantic search function for retrieval. That means it does not have a temporal vector.

Time-based actions are available via the Smart Connect app (i.e. "get recent notes"), and more will come in the future. Those actions will also become available in a future iteration of the Smart Chat in Obsidian 🌴

[Garrett3Nelson]

Thanks for the reply Brian, does this also mean that the local model is functioning as intended in my earlier post? #707 (comment)

TLDR: It receives no context from the plugin, but an unrelated VSCode plugin has no issue sending context and receiving relevant answers.

Is this an Issue, User Error, or future feature?

[brianpetro]

The local models receive all the same context that any other model receives.

And I'm unaware of the vscode plugin so I can't say what the differences might be.

While I'm optimistic about the future of local models, there are many inconsistencies and insufficiencies between models and platforms. However, the design of the underlying open-source architecture of Smart Chat allows for accounting for these issues via the adapter pattern. The development of these adaptations beyond the most common platforms is something that I believe would best be contributed by the community 🌴

[eugr]

Is there any way to see what API requests are going out? If I open Developer Console in Obsidian (Ctrl-Shift-I) and switch to network tab, it doesn't show any API requests. I want to see what's being passed in the context and the actual prompt. It struggles to answer questions correctly even if it shows the right note in the context (BTW, would be great if we could see the full list of blocks/sources that make it to the context).

If I load the same notes into my Open-WebUI knowledgebase and ask the same model the same question, it gives me correct results. It uses different embedding model though.

I love the idea, but as of now, I can't make it work properly, at least with the local LLM.

[brianpetro]

There is a history file that can be opened from within the Smart Chat view that should contain a list of which notes were used as context 🌴