How many notes does chat incorporate?

[Thomas-Richardson]

If I use the "chat with my notes" function, how many notes does it pass to ChatGPT to answer my question? I'm loosely aware it works by:

• embed question
• find notes with similar embeddings
• send questions alonside full text of similar notes to ChatGPT

How many notes will it typically send?

[brianpetro]

@Thomas-Richardson currently the max is 20, except for gpt-4-turbo, where the max is 40.

It's usually less because the algo stops after the "next most similar" exceeds a standard deviation threshold.

Note: this is how the v1/v2.0 work. The v2.1 release, which is focused on improving the Smart Chat, will work a little differently.

[alexmhoward3]

Hi! Is there any documentation on how this is implemented in v2.1? Interested in reading more, especially with the longer context window APIs available.

[brianpetro]

@alexmhoward3 right now, the implementation is essentially the same.

The current difference is primarily how the retrieval logic is isolated instead of mixed with other code. But it isn't finished.

The new retrieval function is called lookup, and it follows the OpenAI API's "tool call" pattern. That means that the AI "calls" the function. The AI inputs "hypotheticals" that it anticipates would be semantically similar to the content of the notes to be retrieved. Then, cosine similarity is used to retrieve the context.

The function call is still triggered by self-referential pronouns in v2.1. This will change slightly in v2.2 when additional functions become available. In the latter version, "tools" will be able to be toggled such that the AI has access to them anytime (self-referential pronoun included or not).

So, to recap, the v2.1 retrieval is effectively still the same. But, with the improving architecture, the retrieval logic will become much easier to follow. I aim for the lookup function to be completely human-readable. The result of this should be the ability to implement various retrieval strategies in a maintainable way (and there are many possible variations).

In the meantime, my recommendation for maximizing the available context window is a) manually linking notes in the chat input and b) utilizing Dataview to aggregate relevant context and linking to that note in the chat input. "B" is my preferred method.

In the future, I intend to enable a "context review" step in the Smart Chat so that the retrieved context is displayed prior to sending it to the model for analysis. This would allow removing irrelevant context and a "retrieve more" option.

I hope that helps
🌴

[alexmhoward3]

amazing response - thanks!

In the meantime, my recommendation for maximizing the available context window is a) manually linking notes in the chat input and b) utilizing Dataview to aggregate relevant context and linking to that note in the chat input. "B" is my preferred method.

great idea! I hadn’t thought of option B. Cheers!

[jritsema]

@brianpetro I just started using this plugin (awesome btw, thank you!) and noticed that, (based on the contents of smart-chats), the 1st assistant entry has some JSON (hypotheticals) that contains text that appears to come from an LLM. 1/ Is the RAG process it uses documented in detail anywhere? 2/ Is there a way to modify the LLM prompt so that I can constrain it to only use the context from my notes? for example...

{context}
Read the user's question supplied within the tags. Then, using only the contextual information provided above within the tags, generate an answer to the question.
{question}